2024-09-10T15:55:31.471 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T15:55:31.488 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T15:55:31.567 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} email: skanta@redhat.com first_in_suite: false job_id: '7899276' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi164.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTluh4G05URZnRM8auvbuBi5V9GAruFRN9qhzIo/j/Z8wn7mWwGqgl1+mdZA2EDJe1r9Hu/gQaNh7GFy887Dpo= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPhaMDYMV5pBfzzoGVlHP9rBr4/ZybC6d9W8Uc/UO2qd18xBKF7Ra2SELiUTleyyV73kw8SEbF0fBrb0lOh10= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 20 thrash_delay: 1 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T15:55:31.576 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T15:55:31.577 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T15:55:31.577 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T15:55:31.578 INFO:teuthology.task.internal:Checking packages... 2024-09-10T15:55:31.599 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T15:55:31.600 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T15:55:31.600 INFO:teuthology.packaging:ref: None 2024-09-10T15:55:31.600 INFO:teuthology.packaging:tag: None 2024-09-10T15:55:31.600 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T15:55:31.600 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T15:55:31.600 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T15:55:31.744 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858.gbad1f558 2024-09-10T15:55:31.746 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T15:55:31.761 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T15:55:31.761 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T15:55:31.787 INFO:teuthology.task.internal:Saving configuration 2024-09-10T15:55:31.801 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T15:55:31.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T15:55:31.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi164.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 15:50:18.723309', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTluh4G05URZnRM8auvbuBi5V9GAruFRN9qhzIo/j/Z8wn7mWwGqgl1+mdZA2EDJe1r9Hu/gQaNh7GFy887Dpo='} 2024-09-10T15:55:31.859 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 15:50:18.725289', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPhaMDYMV5pBfzzoGVlHP9rBr4/ZybC6d9W8Uc/UO2qd18xBKF7Ra2SELiUTleyyV73kw8SEbF0fBrb0lOh10='} 2024-09-10T15:55:31.859 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T15:55:31.865 INFO:teuthology.task.internal:roles: ubuntu@smithi164.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-10T15:55:31.865 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-10T15:55:31.865 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T15:55:31.942 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f11b8bafbe0>, signals=[15]) 2024-09-10T15:55:31.942 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T15:55:31.953 INFO:teuthology.task.internal:Opening connections... 2024-09-10T15:55:31.953 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T15:55:31.954 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:55:32.074 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T15:55:32.075 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:55:32.156 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T15:55:32.162 DEBUG:teuthology.orchestra.run.smithi164:> uname -m 2024-09-10T15:55:32.181 INFO:teuthology.orchestra.run.smithi164.stdout:x86_64 2024-09-10T15:55:32.181 DEBUG:teuthology.orchestra.run.smithi164:> cat /etc/os-release 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:NAME="CentOS Stream" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION="9" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:ID="centos" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:ID_LIKE="rhel fedora" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:VERSION_ID="9" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:PLATFORM_ID="platform:el9" 2024-09-10T15:55:32.239 INFO:teuthology.orchestra.run.smithi164.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:ANSI_COLOR="0;31" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:LOGO="fedora-logo-icon" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:HOME_URL="https://centos.org/" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T15:55:32.240 INFO:teuthology.orchestra.run.smithi164.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T15:55:32.241 INFO:teuthology.lock.ops:Updating smithi164.front.sepia.ceph.com on lock server 2024-09-10T15:55:32.268 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-10T15:55:32.288 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-10T15:55:32.288 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-10T15:55:32.346 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-09-10T15:55:32.346 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-09-10T15:55:32.346 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-09-10T15:55:32.346 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-09-10T15:55:32.346 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-10T15:55:32.347 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-10T15:55:32.347 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-10T15:55:32.367 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T15:55:32.376 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T15:55:32.394 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T15:55:32.394 DEBUG:teuthology.orchestra.run.smithi164:> test '!' -e /home/ubuntu/cephtest 2024-09-10T15:55:32.396 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-10T15:55:32.412 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T15:55:32.418 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T15:55:32.418 DEBUG:teuthology.orchestra.run.smithi164:> test -z $(ls -A /var/lib/ceph) 2024-09-10T15:55:32.454 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-10T15:55:32.607 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T15:55:32.658 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T15:55:32.673 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T15:55:32.673 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T15:55:32.674 DEBUG:teuthology.orchestra.run.smithi164:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T15:55:32.674 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T15:55:32.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T15:55:32.691 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-10T15:55:32.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T15:55:32.692 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-09-10T15:55:32.748 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-503.el9.x86_64 2024-09-10T15:55:32.748 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-503.el9.x86_64 2024-09-10T15:55:32.748 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-09-10T15:55:32.750 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-503.el9.x86_64 2024-09-10T15:55:32.750 INFO:teuthology.task.kernel:Running kernel on smithi164: 5.14.0-503.el9.x86_64 2024-09-10T15:55:32.750 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum install -y kernel 2024-09-10T15:55:34.927 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.2 MB 00:00 2024-09-10T15:55:37.303 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 16 MB/s | 20 MB 00:01 2024-09-10T15:55:37.987 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 8.2 MB 00:03 2024-09-10T15:55:42.586 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.5 MB 00:01 2024-09-10T15:55:44.054 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 70 kB/s | 19 kB 00:00 2024-09-10T15:55:44.957 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-10T15:55:50.405 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-10T15:55:52.444 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T15:55:52.445 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T15:55:52.530 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T15:55:52.541 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:55:52.542 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T15:55:52.542 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T15:55:52.542 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-10T15:55:52.542 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:55:52.543 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 79 M 2024-09-10T15:55:52.543 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 126 M 2024-09-10T15:55:52.543 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T15:55:53.309 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 41 kB/s | 23 kB 00:00 2024-09-10T15:55:54.315 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 10 MB/s | 16 MB 00:01 2024-09-10T15:55:54.873 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 16 MB/s | 35 MB 00:02 2024-09-10T15:55:55.506 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 13 MB/s | 29 MB 00:02 2024-09-10T15:55:55.507 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T15:55:55.507 INFO:teuthology.orchestra.run.smithi165.stdout:Total 27 MB/s | 79 MB 00:02 2024-09-10T15:55:55.619 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 1.2 MB/s | 20 MB 00:16 2024-09-10T15:55:56.189 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T15:55:56.546 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T15:55:56.547 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T15:55:57.523 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T15:55:57.523 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T15:55:59.984 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T15:56:00.103 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 9.8 MB/s | 6.5 MB 00:00 2024-09-10T15:56:00.692 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T15:56:00.783 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:56:01.637 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 19 kB 00:00 2024-09-10T15:56:02.441 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 33 MB/s | 23 MB 00:00 2024-09-10T15:56:03.576 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:56:03.700 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:56:07.896 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-09-10T15:56:08.543 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:56:08.583 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:09.945 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-10T15:56:09.945 INFO:teuthology.orchestra.run.smithi164.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-10T15:56:10.040 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T15:56:10.048 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T15:56:10.048 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-09-10T15:56:10.048 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout: kernel x86_64 5.14.0-505.el9 baseos 23 k 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core x86_64 5.14.0-505.el9 baseos 16 M 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules x86_64 5.14.0-505.el9 baseos 35 M 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core x86_64 5.14.0-505.el9 baseos 29 M 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:56:10.049 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 79 M 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 126 M 2024-09-10T15:56:10.050 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T15:56:10.840 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): kernel-5.14.0-505.el9.x86_64.rpm 43 kB/s | 23 kB 00:00 2024-09-10T15:56:11.940 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): kernel-core-5.14.0-505.el9.x86_64.rpm 9.6 MB/s | 16 MB 00:01 2024-09-10T15:56:13.199 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): kernel-modules-5.14.0-505.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-10T15:56:13.454 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:13.891 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): kernel-modules-core-5.14.0-505.el9.x86_6 9.4 MB/s | 29 MB 00:03 2024-09-10T15:56:13.891 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T15:56:13.891 INFO:teuthology.orchestra.run.smithi164.stdout:Total 20 MB/s | 79 MB 00:03 2024-09-10T15:56:14.573 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T15:56:14.951 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T15:56:14.951 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T15:56:15.921 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T15:56:15.921 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T15:56:18.373 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T15:56:19.090 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-core-5.14.0-505.el9.x86_64 1/4 2024-09-10T15:56:19.182 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:56:22.006 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:56:22.135 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:56:26.961 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:56:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:31.833 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:51.522 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:51.562 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:51.899 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T15:56:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:56:51.900 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T15:56:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T15:56:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T15:56:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:56:52.772 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T15:56:53.130 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T15:56:53.208 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T15:56:53.209 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T15:56:53.210 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T15:56:53.210 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T15:56:53.210 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T15:56:53.210 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T15:56:53.210 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T15:56:53.210 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T15:56:53.269 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T15:56:53.269 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T15:56:53.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-09-10T15:56:53.302 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 82.el9 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8274670 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T15:56:53.303 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T15:56:53.304 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T15:56:53.305 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T15:56:53.305 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-505.el9.x86_64 2024-09-10T15:56:53.305 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T15:56:53.984 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-10T15:56:55.324 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T15:56:55.358 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-10T15:56:55.360 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-10T15:56:55.376 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.4D36JUBAmu 2024-09-10T15:56:55.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4D36JUBAmu 2024-09-10T15:56:55.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.4D36JUBAmu 2024-09-10T15:56:55.587 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.4D36JUBAmu is 6KB 2024-09-10T15:56:55.635 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.4D36JUBAmu 2024-09-10T15:56:55.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T15:56:55.713 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T15:56:55.713 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T15:56:55.713 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T15:56:55.713 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T15:56:55.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T15:56:55.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-10T15:57:10.245 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:57:10.284 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-modules-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:57:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: kernel-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:57:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-5.14.0-505.el9.x86_64 1/4 2024-09-10T15:57:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-core-5.14.0-505.el9.x86_64 2/4 2024-09-10T15:57:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-5.14.0-505.el9.x86_64 3/4 2024-09-10T15:57:11.520 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : kernel-modules-core-5.14.0-505.el9.x86_64 4/4 2024-09-10T15:57:11.520 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:57:11.520 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T15:57:11.520 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-5.14.0-505.el9.x86_64 2024-09-10T15:57:11.521 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-core-5.14.0-505.el9.x86_64 2024-09-10T15:57:11.521 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-5.14.0-505.el9.x86_64 2024-09-10T15:57:11.521 INFO:teuthology.orchestra.run.smithi164.stdout: kernel-modules-core-5.14.0-505.el9.x86_64 2024-09-10T15:57:11.521 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:57:11.521 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T15:57:11.898 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T15:57:11.994 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T15:57:11.994 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T15:57:11.994 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-505.el9.x86_64 2024-09-10T15:57:11.995 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T15:57:11.995 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T15:57:11.995 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T15:57:11.995 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi164.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T15:57:11.995 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q kernel | sort -rV | head -n 1 2024-09-10T15:57:12.053 INFO:teuthology.orchestra.run.smithi164.stdout:kernel-5.14.0-505.el9.x86_64 2024-09-10T15:57:12.054 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-505.el9.x86_64 2024-09-10T15:57:12.054 DEBUG:teuthology.orchestra.run.smithi164:> sudo rpm -qi grub2-tools 2024-09-10T15:57:12.089 INFO:teuthology.orchestra.run.smithi164.stdout:Name : grub2-tools 2024-09-10T15:57:12.089 INFO:teuthology.orchestra.run.smithi164.stdout:Epoch : 1 2024-09-10T15:57:12.089 INFO:teuthology.orchestra.run.smithi164.stdout:Version : 2.06 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Release : 82.el9 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Architecture: x86_64 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Group : Unspecified 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Size : 8274670 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:License : GPLv3+ 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Packager : builder@centos.org 2024-09-10T15:57:12.090 INFO:teuthology.orchestra.run.smithi164.stdout:Vendor : CentOS 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:Summary : Support tools for GRUB. 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:Description : 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:hardware devices. 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T15:57:12.091 INFO:teuthology.orchestra.run.smithi164.stdout:This subpackage provides tools for support of all platforms. 2024-09-10T15:57:12.092 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-10T15:57:12.092 INFO:teuthology.task.kernel:Updating grub on smithi164 to boot 5.14.0-505.el9.x86_64 2024-09-10T15:57:12.092 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-10T15:57:12.733 INFO:teuthology.orchestra.run.smithi164.stderr:Generating grub configuration file ... 2024-09-10T15:57:14.109 INFO:teuthology.orchestra.run.smithi164.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-10T15:57:14.143 INFO:teuthology.orchestra.run.smithi164.stderr:done 2024-09-10T15:57:14.146 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-09-10T15:57:14.161 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.f2L6Gf6egi 2024-09-10T15:57:14.161 DEBUG:teuthology.orchestra.run.smithi164:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.f2L6Gf6egi 2024-09-10T15:57:14.227 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.f2L6Gf6egi 2024-09-10T15:57:14.363 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.f2L6Gf6egi is 6KB 2024-09-10T15:57:14.413 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.f2L6Gf6egi 2024-09-10T15:57:14.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo /bin/ls /boot/loader/entries || true 2024-09-10T15:57:14.491 INFO:teuthology.orchestra.run.smithi164.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-10T15:57:14.491 INFO:teuthology.orchestra.run.smithi164.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-10T15:57:14.491 INFO:teuthology.orchestra.run.smithi164.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-10T15:57:14.491 INFO:teuthology.orchestra.run.smithi164.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64.conf 2024-09-10T15:57:14.493 DEBUG:teuthology.orchestra.run.smithi164:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-505.el9.x86_64 2024-09-10T15:57:14.643 DEBUG:teuthology.orchestra.run.smithi164:> sudo shutdown -r now 2024-09-10T15:57:25.885 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T15:57:25.885 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T15:57:25.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:57:44.381 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-10T15:57:44.646 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T15:57:44.647 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:57:44.648 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:57:53.391 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T15:57:53.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:57:56.445 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-10T15:58:03.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-10T15:58:08.449 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T15:58:08.449 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:58:12.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:58:12.073 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:58:15.133 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-10T15:58:27.145 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:58:27.146 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:58:45.565 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-10T15:59:00.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:59:00.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:59:03.645 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.164 2024-09-10T15:59:08.456 DEBUG:teuthology.orchestra.remote:timed out 2024-09-10T15:59:21.659 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:59:21.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:59:22.087 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-09-10T15:59:22.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T15:59:22.429 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-505.el9.x86_64"... 2024-09-10T15:59:22.429 DEBUG:teuthology.orchestra.run.smithi164:> uname -r 2024-09-10T15:59:22.445 INFO:teuthology.orchestra.run.smithi164.stdout:5.14.0-505.el9.x86_64 2024-09-10T15:59:22.446 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T15:59:22.446 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T15:59:22.446 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T15:59:23.447 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T15:59:23.447 DEBUG:teuthology.orchestra.run.smithi164:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T15:59:23.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T15:59:23.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T15:59:23.521 INFO:teuthology.orchestra.run.smithi164.stdout:ttyS1 2024-09-10T15:59:23.548 DEBUG:teuthology.parallel:result is None 2024-09-10T15:59:23.934 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-10T15:59:24.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T15:59:24.286 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-505.el9.x86_64"... 2024-09-10T15:59:24.286 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-10T15:59:24.302 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-505.el9.x86_64 2024-09-10T15:59:24.302 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-505.el9.x86_64 vs 5.14.0-505.el9.x86_64 2024-09-10T15:59:24.302 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T15:59:24.302 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-10T15:59:25.303 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T15:59:25.303 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T15:59:25.377 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-10T15:59:25.409 DEBUG:teuthology.parallel:result is None 2024-09-10T15:59:25.409 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T15:59:25.416 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T15:59:25.417 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T15:59:25.420 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T15:59:25.439 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T15:59:25.467 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T15:59:25.514 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T15:59:25.514 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T15:59:25.517 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T15:59:25.550 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T15:59:25.564 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T15:59:25.565 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T15:59:25.585 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T15:59:25.620 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T15:59:25.621 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T15:59:25.632 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T15:59:25.632 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T15:59:25.634 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T15:59:25.641 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T15:59:25.642 DEBUG:teuthology.orchestra.run.smithi164:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T15:59:25.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T15:59:25.705 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T15:59:25.714 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T15:59:25.714 DEBUG:teuthology.orchestra.run.smithi164:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T15:59:25.747 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T15:59:25.763 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T15:59:25.846 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T15:59:25.890 DEBUG:teuthology.orchestra.run.smithi164:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T15:59:25.970 DEBUG:teuthology.orchestra.run.smithi164:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T15:59:26.014 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T15:59:26.014 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T15:59:26.079 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T15:59:26.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T15:59:26.177 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T15:59:26.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T15:59:26.303 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T15:59:26.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T15:59:26.370 DEBUG:teuthology.orchestra.run.smithi164:> sudo service rsyslog restart 2024-09-10T15:59:26.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-10T15:59:26.407 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T15:59:26.456 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T15:59:26.948 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T15:59:26.957 INFO:teuthology.task.internal:Starting timer... 2024-09-10T15:59:26.957 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T15:59:27.001 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T15:59:27.149 DEBUG:teuthology.orchestra.run.smithi164:> sudo service auditd rotate 2024-09-10T15:59:27.192 INFO:teuthology.orchestra.run.smithi164.stdout:Rotating logs: 2024-09-10T15:59:27.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-09-10T15:59:27.237 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-09-10T15:59:27.239 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T15:59:27.239 DEBUG:teuthology.orchestra.run.smithi164:> /usr/sbin/getenforce 2024-09-10T15:59:27.266 INFO:teuthology.orchestra.run.smithi164.stdout:Permissive 2024-09-10T15:59:27.266 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-09-10T15:59:27.310 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-09-10T15:59:27.310 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi164.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-09-10T15:59:27.310 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T15:59:27.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T15:59:27.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T15:59:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T15:59:27.378 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T15:59:27.378 DEBUG:teuthology.orchestra.run.smithi164:> sudo /usr/sbin/setenforce permissive 2024-09-10T15:59:27.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-09-10T15:59:27.456 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T15:59:27.465 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-10T15:59:27.468 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-10T15:59:27.584 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T15:59:27.594 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T15:59:27.594 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi164.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T16:03:08.803 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi164.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-09-10T16:03:08.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T16:03:08.806 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi164.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:03:08.913 DEBUG:teuthology.orchestra.run.smithi164:> true 2024-09-10T16:03:08.988 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi164.front.sepia.ceph.com' 2024-09-10T16:03:08.988 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T16:03:08.989 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T16:03:09.064 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-10T16:03:09.149 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-10T16:03:09.149 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T16:03:09.161 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T16:03:09.161 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T16:03:09.162 DEBUG:teuthology.orchestra.run.smithi164:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T16:03:09.164 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T16:03:09.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T16:03:09.191 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T16:03:09.206 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T16:03:09.220 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-10T16:03:09.231 INFO:teuthology.orchestra.run.smithi164.stderr:sudo: ntpd: command not found 2024-09-10T16:03:09.235 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-10T16:03:09.242 INFO:teuthology.orchestra.run.smithi164.stdout:506 Cannot talk to daemon 2024-09-10T16:03:09.256 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T16:03:09.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-09-10T16:03:09.270 INFO:teuthology.orchestra.run.smithi164.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T16:03:09.271 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-09-10T16:03:09.286 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-10T16:03:09.300 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-10T16:03:09.325 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-09-10T16:03:09.328 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T16:03:09.328 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-09-10T16:03:09.329 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.329 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.329 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.329 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.353 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-09-10T16:03:09.356 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T16:03:09.356 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-09-10T16:03:09.356 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.357 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.357 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.357 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:03:09.358 INFO:teuthology.run_tasks:Running task install... 2024-09-10T16:03:09.366 DEBUG:teuthology.task.install:project ceph 2024-09-10T16:03:09.366 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T16:03:09.366 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T16:03:09.366 INFO:teuthology.task.install:Using flavor: default 2024-09-10T16:03:09.372 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T16:03:09.372 INFO:teuthology.task.install:extra packages: [] 2024-09-10T16:03:09.372 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T16:03:09.373 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:03:09.375 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'tag': None, 'wait_for_package': False} 2024-09-10T16:03:09.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:03:09.519 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T16:03:09.520 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T16:03:09.523 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/ 2024-09-10T16:03:09.523 INFO:teuthology.task.install.rpm:Package version is 19.3.0-4858.gbad1f558 2024-09-10T16:03:09.630 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T16:03:09.630 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:03:09.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T16:03:09.639 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-10T16:03:09.639 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:03:09.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-10T16:03:09.657 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T16:03:09.657 DEBUG:teuthology.orchestra.run.smithi164:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T16:03:09.666 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-10T16:03:09.666 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-10T16:03:09.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T16:03:09.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-10T16:03:09.797 DEBUG:teuthology.orchestra.run.smithi164:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T16:03:09.809 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-10T16:03:09.862 INFO:teuthology.orchestra.run.smithi164.stdout:check_obsoletes = 1 2024-09-10T16:03:09.863 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum clean all 2024-09-10T16:03:09.873 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-09-10T16:03:09.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-09-10T16:03:10.101 INFO:teuthology.orchestra.run.smithi164.stdout:45 files removed 2024-09-10T16:03:10.111 INFO:teuthology.orchestra.run.smithi165.stdout:45 files removed 2024-09-10T16:03:10.129 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-radosgw 2024-09-10T16:03:10.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw 2024-09-10T16:03:10.686 INFO:teuthology.orchestra.run.smithi164.stdout:ceph packages for x86_64 309 kB/s | 84 kB 00:00 2024-09-10T16:03:10.716 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 285 kB/s | 84 kB 00:00 2024-09-10T16:03:10.987 INFO:teuthology.orchestra.run.smithi164.stdout:ceph noarch packages 94 kB/s | 24 kB 00:00 2024-09-10T16:03:11.000 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 100 kB/s | 24 kB 00:00 2024-09-10T16:03:11.271 INFO:teuthology.orchestra.run.smithi164.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-10T16:03:11.301 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 7.4 kB/s | 1.9 kB 00:00 2024-09-10T16:03:11.909 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.2 MB 00:00 2024-09-10T16:03:12.021 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.2 MB 00:00 2024-09-10T16:03:15.494 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 8.2 MB/s | 20 MB 00:02 2024-09-10T16:03:17.580 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - AppStream 4.6 MB/s | 20 MB 00:04 2024-09-10T16:03:19.803 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.5 MB 00:00 2024-09-10T16:03:21.996 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 19 kB 00:00 2024-09-10T16:03:23.125 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-10T16:03:23.816 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - CRB 2.7 MB/s | 6.5 MB 00:02 2024-09-10T16:03:25.342 INFO:teuthology.orchestra.run.smithi164.stdout:CentOS Stream 9 - Extras packages 105 kB/s | 19 kB 00:00 2024-09-10T16:03:27.354 INFO:teuthology.orchestra.run.smithi164.stdout:Extra Packages for Enterprise Linux 12 MB/s | 23 MB 00:01 2024-09-10T16:03:28.481 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-10T16:03:30.857 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-09-10T16:03:30.865 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T16:03:30.866 INFO:teuthology.orchestra.run.smithi165.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T16:03:30.867 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T16:03:30.868 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Install 31 Packages 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 62 M 2024-09-10T16:03:30.869 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:03:31.932 INFO:teuthology.orchestra.run.smithi165.stdout:(1/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 14 MB/s | 11 MB 00:00 2024-09-10T16:03:32.015 INFO:teuthology.orchestra.run.smithi165.stdout:(2/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 6.5 MB/s | 5.5 MB 00:00 2024-09-10T16:03:32.040 INFO:teuthology.orchestra.run.smithi165.stdout:(3/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 232 kB/s | 25 kB 00:00 2024-09-10T16:03:32.083 INFO:teuthology.orchestra.run.smithi165.stdout:(4/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 11 MB/s | 755 kB 00:00 2024-09-10T16:03:32.108 INFO:teuthology.orchestra.run.smithi165.stdout:(5/33): libradosstriper1-19.3.0-4858.gbad1f558. 7.4 MB/s | 509 kB 00:00 2024-09-10T16:03:32.158 INFO:teuthology.orchestra.run.smithi165.stdout:(6/33): python3-ceph-argparse-19.3.0-4858.gbad1 901 kB/s | 45 kB 00:00 2024-09-10T16:03:32.242 INFO:teuthology.orchestra.run.smithi165.stdout:(7/33): python3-ceph-common-19.3.0-4858.gbad1f5 1.7 MB/s | 143 kB 00:00 2024-09-10T16:03:32.476 INFO:teuthology.orchestra.run.smithi165.stdout:(8/33): ceph-common-19.3.0-4858.gbad1f558.el9.x 17 MB/s | 22 MB 00:01 2024-09-10T16:03:32.501 INFO:teuthology.orchestra.run.smithi165.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 624 kB/s | 161 kB 00:00 2024-09-10T16:03:32.526 INFO:teuthology.orchestra.run.smithi165.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 6.3 MB/s | 321 kB 00:00 2024-09-10T16:03:32.560 INFO:teuthology.orchestra.run.smithi165.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 5.1 MB/s | 306 kB 00:00 2024-09-10T16:03:32.585 INFO:teuthology.orchestra.run.smithi165.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.7 MB/s | 99 kB 00:00 2024-09-10T16:03:32.652 INFO:teuthology.orchestra.run.smithi165.stdout:(13/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_ 9.7 MB/s | 5.5 MB 00:00 2024-09-10T16:03:32.719 INFO:teuthology.orchestra.run.smithi165.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 673 kB/s | 107 kB 00:00 2024-09-10T16:03:32.777 INFO:teuthology.orchestra.run.smithi164.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-09-10T16:03:32.778 INFO:teuthology.orchestra.run.smithi165.stdout:(15/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 777 kB/s | 45 kB 00:00 2024-09-10T16:03:32.803 INFO:teuthology.orchestra.run.smithi165.stdout:(16/33): libnbd-1.20.2-2.el9.x86_64.rpm 784 kB/s | 171 kB 00:00 2024-09-10T16:03:32.829 INFO:teuthology.orchestra.run.smithi165.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 911 kB/s | 160 kB 00:00 2024-09-10T16:03:32.879 INFO:teuthology.orchestra.run.smithi165.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-10T16:03:32.905 INFO:teuthology.orchestra.run.smithi165.stdout:(19/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-10T16:03:32.930 INFO:teuthology.orchestra.run.smithi165.stdout:(20/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-10T16:03:32.955 INFO:teuthology.orchestra.run.smithi165.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 713 kB/s | 36 kB 00:00 2024-09-10T16:03:32.980 INFO:teuthology.orchestra.run.smithi165.stdout:(22/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 940 kB/s | 47 kB 00:00 2024-09-10T16:03:33.005 INFO:teuthology.orchestra.run.smithi165.stdout:(23/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-09-10T16:03:33.072 INFO:teuthology.orchestra.run.smithi165.stdout:(24/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 496 kB/s | 26 kB 00:00 2024-09-10T16:03:33.097 INFO:teuthology.orchestra.run.smithi165.stdout:(25/33): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-10T16:03:33.123 INFO:teuthology.orchestra.run.smithi165.stdout:(26/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.8 MB/s | 309 kB 00:00 2024-09-10T16:03:33.148 INFO:teuthology.orchestra.run.smithi165.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T16:03:33.182 INFO:teuthology.orchestra.run.smithi165.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-10T16:03:33.224 INFO:teuthology.orchestra.run.smithi165.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-10T16:03:33.291 INFO:teuthology.orchestra.run.smithi165.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-10T16:03:33.358 INFO:teuthology.orchestra.run.smithi165.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-10T16:03:33.499 INFO:teuthology.orchestra.run.smithi165.stdout:(32/33): librados2-19.3.0-4858.gbad1f558.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-10T16:03:33.558 INFO:teuthology.orchestra.run.smithi165.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-10T16:03:33.563 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:03:33.564 INFO:teuthology.orchestra.run.smithi165.stdout:Total 23 MB/s | 62 MB 00:02 2024-09-10T16:03:33.671 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:03:33.709 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:03:33.709 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:03:34.421 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:03:34.421 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:03:35.145 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 11 M 2024-09-10T16:03:35.161 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrading: 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: librados2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.5 M 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 22 M 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 25 k 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-10T16:03:35.162 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 755 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: libnbd x86_64 1.20.2-2.el9 appstream 171 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 509 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2 x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 5.5 M 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-10T16:03:35.163 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 45 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 143 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 321 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 306 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 99 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout:Installing weak dependencies: 2024-09-10T16:03:35.164 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout:Install 31 Packages 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout:Upgrade 2 Packages 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:35.165 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 62 M 2024-09-10T16:03:35.166 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:03:35.960 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:03:36.051 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T16:03:36.080 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T16:03:36.105 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T16:03:36.128 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:03:36.145 INFO:teuthology.orchestra.run.smithi164.stdout:(1/33): ceph-base-19.3.0-4858.gbad1f558.el9.x86 11 MB/s | 5.5 MB 00:00 2024-09-10T16:03:36.154 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:03:36.228 INFO:teuthology.orchestra.run.smithi164.stdout:(2/33): ceph-selinux-19.3.0-4858.gbad1f558.el9. 303 kB/s | 25 kB 00:00 2024-09-10T16:03:36.312 INFO:teuthology.orchestra.run.smithi164.stdout:(3/33): libcephfs2-19.3.0-4858.gbad1f558.el9.x8 8.9 MB/s | 755 kB 00:00 2024-09-10T16:03:36.348 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:03:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:03:36.412 INFO:teuthology.orchestra.run.smithi164.stdout:(4/33): libradosstriper1-19.3.0-4858.gbad1f558. 5.0 MB/s | 509 kB 00:00 2024-09-10T16:03:36.671 INFO:teuthology.orchestra.run.smithi164.stdout:(5/33): ceph-radosgw-19.3.0-4858.gbad1f558.el9. 11 MB/s | 11 MB 00:01 2024-09-10T16:03:36.821 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T16:03:36.874 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:36.888 INFO:teuthology.orchestra.run.smithi164.stdout:(6/33): ceph-common-19.3.0-4858.gbad1f558.el9.x 18 MB/s | 22 MB 00:01 2024-09-10T16:03:36.913 INFO:teuthology.orchestra.run.smithi164.stdout:(7/33): python3-ceph-argparse-19.3.0-4858.gbad1 187 kB/s | 45 kB 00:00 2024-09-10T16:03:36.939 INFO:teuthology.orchestra.run.smithi164.stdout:(8/33): python3-ceph-common-19.3.0-4858.gbad1f5 2.8 MB/s | 143 kB 00:00 2024-09-10T16:03:36.964 INFO:teuthology.orchestra.run.smithi164.stdout:(9/33): python3-cephfs-19.3.0-4858.gbad1f558.el 3.1 MB/s | 161 kB 00:00 2024-09-10T16:03:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:36.989 INFO:teuthology.orchestra.run.smithi164.stdout:(10/33): python3-rados-19.3.0-4858.gbad1f558.el 6.2 MB/s | 321 kB 00:00 2024-09-10T16:03:37.023 INFO:teuthology.orchestra.run.smithi164.stdout:(11/33): python3-rbd-19.3.0-4858.gbad1f558.el9. 5.1 MB/s | 306 kB 00:00 2024-09-10T16:03:37.040 INFO:teuthology.orchestra.run.smithi164.stdout:(12/33): python3-rgw-19.3.0-4858.gbad1f558.el9. 1.9 MB/s | 99 kB 00:00 2024-09-10T16:03:37.057 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T16:03:37.093 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:03:37.115 INFO:teuthology.orchestra.run.smithi164.stdout:(13/33): librgw2-19.3.0-4858.gbad1f558.el9.x86_ 7.9 MB/s | 5.5 MB 00:00 2024-09-10T16:03:37.155 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:03:37.193 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T16:03:37.236 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T16:03:37.257 INFO:teuthology.orchestra.run.smithi164.stdout:(14/33): boost-program-options-1.75.0-8.el9.x86 457 kB/s | 107 kB 00:00 2024-09-10T16:03:37.282 INFO:teuthology.orchestra.run.smithi164.stdout:(15/33): libnbd-1.20.2-2.el9.x86_64.rpm 705 kB/s | 171 kB 00:00 2024-09-10T16:03:37.307 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:03:37.307 INFO:teuthology.orchestra.run.smithi164.stdout:(16/33): librabbitmq-0.11.0-7.el9.x86_64.rpm 900 kB/s | 45 kB 00:00 2024-09-10T16:03:37.328 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:03:37.333 INFO:teuthology.orchestra.run.smithi164.stdout:(17/33): libpmemobj-1.12.1-1.el9.x86_64.rpm 737 kB/s | 160 kB 00:00 2024-09-10T16:03:37.383 INFO:teuthology.orchestra.run.smithi164.stdout:(18/33): python3-prettytable-0.7.2-27.el9.noarc 848 kB/s | 42 kB 00:00 2024-09-10T16:03:37.396 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:03:37.431 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T16:03:37.442 INFO:teuthology.orchestra.run.smithi164.stdout:(19/33): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-10T16:03:37.467 INFO:teuthology.orchestra.run.smithi164.stdout:(20/33): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.8 MB/s | 292 kB 00:00 2024-09-10T16:03:37.492 INFO:teuthology.orchestra.run.smithi164.stdout:(21/33): qatlib-service-24.02.0-1.el9.x86_64.rp 716 kB/s | 36 kB 00:00 2024-09-10T16:03:37.517 INFO:teuthology.orchestra.run.smithi164.stdout:(22/33): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-09-10T16:03:37.543 INFO:teuthology.orchestra.run.smithi164.stdout:(23/33): qatzip-libs-1.2.0-1.el9.x86_64.rpm 627 kB/s | 47 kB 00:00 2024-09-10T16:03:37.551 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T16:03:37.587 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T16:03:37.601 INFO:teuthology.orchestra.run.smithi164.stdout:(24/33): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-09-10T16:03:37.627 INFO:teuthology.orchestra.run.smithi164.stdout:(25/33): libarrow-doc-9.0.0-11.el9.noarch.rpm 378 kB/s | 26 kB 00:00 2024-09-10T16:03:37.628 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T16:03:37.652 INFO:teuthology.orchestra.run.smithi164.stdout:(26/33): liboath-2.6.7-2.el9.x86_64.rpm 969 kB/s | 49 kB 00:00 2024-09-10T16:03:37.668 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T16:03:37.677 INFO:teuthology.orchestra.run.smithi164.stdout:(27/33): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-10T16:03:37.697 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T16:03:37.711 INFO:teuthology.orchestra.run.smithi164.stdout:(28/33): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-10T16:03:37.739 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:03:37.744 INFO:teuthology.orchestra.run.smithi164.stdout:(29/33): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-10T16:03:37.790 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T16:03:37.828 INFO:teuthology.orchestra.run.smithi164.stdout:(30/33): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-10T16:03:37.895 INFO:teuthology.orchestra.run.smithi164.stdout:(31/33): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-10T16:03:38.020 INFO:teuthology.orchestra.run.smithi164.stdout:(32/33): librados2-19.3.0-4858.gbad1f558.el9.x8 13 MB/s | 3.5 MB 00:00 2024-09-10T16:03:38.095 INFO:teuthology.orchestra.run.smithi164.stdout:(33/33): librbd1-19.3.0-4858.gbad1f558.el9.x86_ 12 MB/s | 3.2 MB 00:00 2024-09-10T16:03:38.100 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:03:38.100 INFO:teuthology.orchestra.run.smithi164.stdout:Total 21 MB/s | 62 MB 00:02 2024-09-10T16:03:38.129 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T16:03:38.178 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:03:38.209 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:03:38.246 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:03:38.246 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:03:38.268 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:03:38.300 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T16:03:38.335 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T16:03:38.828 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T16:03:38.855 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:03:38.927 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:03:38.944 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:03:38.944 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:03:39.005 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T16:03:39.033 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T16:03:40.472 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:03:40.555 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/35 2024-09-10T16:03:40.584 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/35 2024-09-10T16:03:40.609 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/35 2024-09-10T16:03:40.632 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:03:40.661 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/35 2024-09-10T16:03:40.851 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:03:40.904 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/35 2024-09-10T16:03:41.273 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/35 2024-09-10T16:03:41.369 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:41.373 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:41.405 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:41.462 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:41.558 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 8/35 2024-09-10T16:03:41.588 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:03:41.645 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 9/35 2024-09-10T16:03:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/35 2024-09-10T16:03:41.732 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 11/35 2024-09-10T16:03:41.812 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:03:41.840 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:03:41.904 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 13/35 2024-09-10T16:03:41.939 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/35 2024-09-10T16:03:41.984 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:42.017 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:03:42.047 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/35 2024-09-10T16:03:42.082 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/35 2024-09-10T16:03:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:03:42.114 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T16:03:42.114 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T16:03:42.114 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:42.131 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/35 2024-09-10T16:03:42.143 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:42.172 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/35 2024-09-10T16:03:42.218 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/35 2024-09-10T16:03:42.250 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:03:42.302 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/35 2024-09-10T16:03:42.622 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libnbd-1.20.2-2.el9.x86_64 22/35 2024-09-10T16:03:42.672 INFO:teuthology.orchestra.run.smithi164.stdout: Upgrading : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:03:42.763 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 23/35 2024-09-10T16:03:42.794 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 24/35 2024-09-10T16:03:42.830 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 25/35 2024-09-10T16:03:43.324 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 26/35 2024-09-10T16:03:43.350 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:03:43.431 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 27/35 2024-09-10T16:03:43.508 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 28/35 2024-09-10T16:03:43.536 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 29/35 2024-09-10T16:03:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:45.866 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:46.438 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 30/35 2024-09-10T16:03:46.477 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:03:46.573 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 31/35 2024-09-10T16:03:46.573 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-10T16:03:46.573 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-10T16:03:46.573 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:46.604 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-09-10T16:03:52.341 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:53.366 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:03:53.396 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:03:53.396 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:03:53.396 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T16:03:53.397 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:03:53.397 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:03:53.397 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:53.398 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:03:53.494 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:03:53.494 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:54.668 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T16:03:54.669 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T16:03:54.670 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T16:03:54.671 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T16:03:54.671 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T16:03:54.671 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T16:03:54.671 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T16:03:54.671 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T16:03:54.672 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T16:03:54.672 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:54.672 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T16:03:54.672 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T16:03:55.655 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:55.655 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:55.655 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-09-10T16:03:55.655 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.655 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T16:03:55.656 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.657 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:55.658 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:03:56.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-test 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /sys 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /proc 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /mnt 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /var/tmp 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /home 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /root 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout:skipping the directory /tmp 2024-09-10T16:03:56.474 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:56.615 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:03:56.746 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:03:56.747 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:56.747 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:03:56.747 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:56.747 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:03:56.747 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-10T16:03:56.748 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:03:56.749 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 50 M 2024-09-10T16:03:56.749 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 215 M 2024-09-10T16:03:56.749 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:03:57.076 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 728 kB/s | 161 kB 00:00 2024-09-10T16:03:57.101 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.2 MB/s | 304 kB 00:00 2024-09-10T16:03:57.218 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 450 kB/s | 64 kB 00:00 2024-09-10T16:03:57.497 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:03:57.522 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 33/35 2024-09-10T16:03:57.522 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:03:57.523 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-10T16:03:57.523 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:03:57.523 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-10T16:03:57.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:57.525 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:03:57.608 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 34/35 2024-09-10T16:03:57.608 INFO:teuthology.orchestra.run.smithi164.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:58.418 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-10T16:03:58.420 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:03:58.421 INFO:teuthology.orchestra.run.smithi165.stdout:Total 30 MB/s | 50 MB 00:01 2024-09-10T16:03:58.449 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:03:58.470 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:03:58.470 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:03:58.846 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:03:58.847 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 1/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 3/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 4/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 5/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_6 6/35 2024-09-10T16:03:58.850 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 7/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9. 8/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x8 9/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 10/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 11/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 12/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 13/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libnbd-1.20.2-2.el9.x86_64 15/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 16/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 17/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 18/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 19/35 2024-09-10T16:03:58.851 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 20/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 21/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 22/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 23/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 24/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 25/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 26/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 27/35 2024-09-10T16:03:58.852 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 28/35 2024-09-10T16:03:58.853 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 29/35 2024-09-10T16:03:58.853 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : re2-1:20211101-20.el9.x86_64 30/35 2024-09-10T16:03:58.853 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 31/35 2024-09-10T16:03:58.853 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 32/35 2024-09-10T16:03:58.853 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 33/35 2024-09-10T16:03:58.854 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 34/35 2024-09-10T16:03:59.386 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:03:59.479 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T16:03:59.526 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T16:03:59.553 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:03:59.742 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 35/35 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout:Upgraded: 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-radosgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-selinux-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-10T16:03:59.743 INFO:teuthology.orchestra.run.smithi164.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: libnbd-1.20.2-2.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: libradosstriper1-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: librgw2-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-10T16:03:59.744 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-argparse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ceph-common-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: re2-1:20211101-20.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:03:59.745 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:00.154 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-test 2024-09-10T16:04:00.730 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:28 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:00.871 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:00.871 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:00.871 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:00.871 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 50 M 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 161 k 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout:Install 4 Packages 2024-09-10T16:04:00.872 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:00.873 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 50 M 2024-09-10T16:04:00.873 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 215 M 2024-09-10T16:04:00.873 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:01.283 INFO:teuthology.orchestra.run.smithi164.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.9 MB/s | 304 kB 00:00 2024-09-10T16:04:01.325 INFO:teuthology.orchestra.run.smithi164.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-10T16:04:01.367 INFO:teuthology.orchestra.run.smithi164.stdout:(3/4): libcephsqlite-19.3.0-4858.gbad1f558.el9. 680 kB/s | 161 kB 00:00 2024-09-10T16:04:02.675 INFO:teuthology.orchestra.run.smithi164.stdout:(4/4): ceph-test-19.3.0-4858.gbad1f558.el9.x86_ 32 MB/s | 50 MB 00:01 2024-09-10T16:04:02.678 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:02.678 INFO:teuthology.orchestra.run.smithi164.stdout:Total 28 MB/s | 50 MB 00:01 2024-09-10T16:04:02.706 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:02.727 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:02.727 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:03.103 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:03.104 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:03.631 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:03.714 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-10T16:04:03.766 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-10T16:04:03.802 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:04:04.248 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:04:04.335 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:04:06.803 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:04:06.804 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T16:04:06.804 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T16:04:06.804 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T16:04:07.121 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T16:04:07.122 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:07.122 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:07.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph 2024-09-10T16:04:08.021 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:40 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:08.163 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:08.165 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:08.165 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:04:08.165 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:08.165 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T16:04:08.166 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T16:04:08.167 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T16:04:08.168 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout:Install 37 Packages 2024-09-10T16:04:08.169 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:08.170 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 31 M 2024-09-10T16:04:08.170 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 109 M 2024-09-10T16:04:08.170 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:08.530 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 3/4 2024-09-10T16:04:08.610 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:04:09.154 INFO:teuthology.orchestra.run.smithi165.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 52 kB/s | 6.6 kB 00:00 2024-09-10T16:04:09.379 INFO:teuthology.orchestra.run.smithi165.stdout:(2/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 6.8 MB/s | 2.4 MB 00:00 2024-09-10T16:04:09.413 INFO:teuthology.orchestra.run.smithi165.stdout:(3/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 4.0 MB/s | 1.5 MB 00:00 2024-09-10T16:04:09.514 INFO:teuthology.orchestra.run.smithi165.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 2.8 MB/s | 291 kB 00:00 2024-09-10T16:04:09.647 INFO:teuthology.orchestra.run.smithi165.stdout:(5/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 9.7 MB/s | 4.8 MB 00:00 2024-09-10T16:04:09.756 INFO:teuthology.orchestra.run.smithi165.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-09-10T16:04:09.940 INFO:teuthology.orchestra.run.smithi165.stdout:(7/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 31 MB/s | 17 MB 00:00 2024-09-10T16:04:09.965 INFO:teuthology.orchestra.run.smithi165.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 510 kB/s | 106 kB 00:00 2024-09-10T16:04:10.023 INFO:teuthology.orchestra.run.smithi165.stdout:(9/37): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-09-10T16:04:10.074 INFO:teuthology.orchestra.run.smithi165.stdout:(10/37): python3-cryptography-36.0.1-4.el9.x86_ 2.9 MB/s | 1.2 MB 00:00 2024-09-10T16:04:10.099 INFO:teuthology.orchestra.run.smithi165.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 2.8 MB/s | 215 kB 00:00 2024-09-10T16:04:10.158 INFO:teuthology.orchestra.run.smithi165.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 621 kB/s | 135 kB 00:00 2024-09-10T16:04:10.233 INFO:teuthology.orchestra.run.smithi165.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 188 kB 00:00 2024-09-10T16:04:10.250 INFO:teuthology.orchestra.run.smithi165.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 378 kB/s | 35 kB 00:00 2024-09-10T16:04:10.275 INFO:teuthology.orchestra.run.smithi165.stdout:(15/37): python3-mako-1.1.4-6.el9.noarch.rpm 980 kB/s | 172 kB 00:00 2024-09-10T16:04:10.300 INFO:teuthology.orchestra.run.smithi165.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 1.1 MB/s | 77 kB 00:00 2024-09-10T16:04:10.317 INFO:teuthology.orchestra.run.smithi165.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 758 kB/s | 51 kB 00:00 2024-09-10T16:04:10.335 INFO:teuthology.orchestra.run.smithi165.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 707 kB/s | 42 kB 00:00 2024-09-10T16:04:10.351 INFO:teuthology.orchestra.run.smithi165.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 442 kB/s | 22 kB 00:00 2024-09-10T16:04:10.402 INFO:teuthology.orchestra.run.smithi165.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-09-10T16:04:10.427 INFO:teuthology.orchestra.run.smithi165.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 473 kB/s | 43 kB 00:00 2024-09-10T16:04:10.444 INFO:teuthology.orchestra.run.smithi165.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 646 kB/s | 11 kB 00:00 2024-09-10T16:04:10.469 INFO:teuthology.orchestra.run.smithi165.stdout:(23/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-09-10T16:04:10.494 INFO:teuthology.orchestra.run.smithi165.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-09-10T16:04:10.520 INFO:teuthology.orchestra.run.smithi165.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-09-10T16:04:10.545 INFO:teuthology.orchestra.run.smithi165.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-09-10T16:04:10.578 INFO:teuthology.orchestra.run.smithi165.stdout:(27/37): python3-cherrypy-18.6.1-2.el9.noarch.r 2.0 MB/s | 358 kB 00:00 2024-09-10T16:04:10.604 INFO:teuthology.orchestra.run.smithi165.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 235 kB/s | 20 kB 00:00 2024-09-10T16:04:10.629 INFO:teuthology.orchestra.run.smithi165.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 553 kB/s | 46 kB 00:00 2024-09-10T16:04:10.654 INFO:teuthology.orchestra.run.smithi165.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-10T16:04:10.680 INFO:teuthology.orchestra.run.smithi165.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-10T16:04:10.705 INFO:teuthology.orchestra.run.smithi165.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-10T16:04:10.730 INFO:teuthology.orchestra.run.smithi165.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 711 kB/s | 36 kB 00:00 2024-09-10T16:04:10.756 INFO:teuthology.orchestra.run.smithi165.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T16:04:10.781 INFO:teuthology.orchestra.run.smithi165.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 803 kB/s | 20 kB 00:00 2024-09-10T16:04:10.815 INFO:teuthology.orchestra.run.smithi165.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-10T16:04:10.840 INFO:teuthology.orchestra.run.smithi165.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-10T16:04:10.844 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:10.844 INFO:teuthology.orchestra.run.smithi165.stdout:Total 11 MB/s | 31 MB 00:02 2024-09-10T16:04:10.991 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:11.030 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:11.030 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:11.135 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 4/4 2024-09-10T16:04:11.135 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 1/4 2024-09-10T16:04:11.135 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2/4 2024-09-10T16:04:11.135 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-test-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: libcephsqlite-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-10T16:04:11.490 INFO:teuthology.orchestra.run.smithi164.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-10T16:04:11.491 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:11.491 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:11.507 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:11.508 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:11.798 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph 2024-09-10T16:04:12.375 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:40 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:12.519 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: ceph x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 6.6 k 2024-09-10T16:04:12.522 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 2.4 M 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 1.5 M 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 291 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 4.8 M 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 17 M 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-10T16:04:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-10T16:04:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:12.525 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:12.526 INFO:teuthology.orchestra.run.smithi164.stdout:Install 37 Packages 2024-09-10T16:04:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:12.526 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 31 M 2024-09-10T16:04:12.526 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 109 M 2024-09-10T16:04:12.526 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:12.912 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:12.919 INFO:teuthology.orchestra.run.smithi164.stdout:(1/37): ceph-19.3.0-4858.gbad1f558.el9.x86_64.r 50 kB/s | 6.6 kB 00:00 2024-09-10T16:04:12.980 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T16:04:13.013 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T16:04:13.067 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T16:04:13.069 INFO:teuthology.orchestra.run.smithi164.stdout:(2/37): ceph-mgr-19.3.0-4858.gbad1f558.el9.x86_ 5.4 MB/s | 1.5 MB 00:00 2024-09-10T16:04:13.120 INFO:teuthology.orchestra.run.smithi164.stdout:(3/37): ceph-mds-19.3.0-4858.gbad1f558.el9.x86_ 7.2 MB/s | 2.4 MB 00:00 2024-09-10T16:04:13.146 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T16:04:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T16:04:13.220 INFO:teuthology.orchestra.run.smithi164.stdout:(4/37): ceph-mgr-modules-core-19.3.0-4858.gbad1 2.8 MB/s | 291 kB 00:00 2024-09-10T16:04:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T16:04:13.287 INFO:teuthology.orchestra.run.smithi164.stdout:(5/37): ceph-mon-19.3.0-4858.gbad1f558.el9.x86_ 13 MB/s | 4.8 MB 00:00 2024-09-10T16:04:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T16:04:13.410 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T16:04:13.459 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T16:04:13.494 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T16:04:13.496 INFO:teuthology.orchestra.run.smithi164.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 919 kB/s | 253 kB 00:00 2024-09-10T16:04:13.524 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T16:04:13.558 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T16:04:13.592 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T16:04:13.641 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T16:04:13.663 INFO:teuthology.orchestra.run.smithi164.stdout:(7/37): ceph-osd-19.3.0-4858.gbad1f558.el9.x86_ 29 MB/s | 17 MB 00:00 2024-09-10T16:04:13.684 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T16:04:13.688 INFO:teuthology.orchestra.run.smithi164.stdout:(8/37): python3-ply-3.11-14.el9.noarch.rpm 555 kB/s | 106 kB 00:00 2024-09-10T16:04:13.719 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T16:04:13.730 INFO:teuthology.orchestra.run.smithi164.stdout:(9/37): python3-cryptography-36.0.1-4.el9.x86_6 2.8 MB/s | 1.2 MB 00:00 2024-09-10T16:04:13.755 INFO:teuthology.orchestra.run.smithi164.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-10T16:04:13.767 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T16:04:13.789 INFO:teuthology.orchestra.run.smithi164.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-09-10T16:04:13.799 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T16:04:13.880 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T16:04:13.906 INFO:teuthology.orchestra.run.smithi164.stdout:(12/37): python3-pycparser-2.20-6.el9.noarch.rp 557 kB/s | 135 kB 00:00 2024-09-10T16:04:13.931 INFO:teuthology.orchestra.run.smithi164.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-10T16:04:13.973 INFO:teuthology.orchestra.run.smithi164.stdout:(14/37): python3-mako-1.1.4-6.el9.noarch.rpm 936 kB/s | 172 kB 00:00 2024-09-10T16:04:13.998 INFO:teuthology.orchestra.run.smithi164.stdout:(15/37): python3-markupsafe-1.1.1-12.el9.x86_64 378 kB/s | 35 kB 00:00 2024-09-10T16:04:14.031 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T16:04:14.040 INFO:teuthology.orchestra.run.smithi164.stdout:(16/37): python3-packaging-20.9-5.el9.noarch.rp 709 kB/s | 77 kB 00:00 2024-09-10T16:04:14.066 INFO:teuthology.orchestra.run.smithi164.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 552 kB/s | 51 kB 00:00 2024-09-10T16:04:14.091 INFO:teuthology.orchestra.run.smithi164.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 451 kB/s | 42 kB 00:00 2024-09-10T16:04:14.109 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T16:04:14.116 INFO:teuthology.orchestra.run.smithi164.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 295 kB/s | 22 kB 00:00 2024-09-10T16:04:14.167 INFO:teuthology.orchestra.run.smithi164.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-10T16:04:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T16:04:14.192 INFO:teuthology.orchestra.run.smithi164.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-10T16:04:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T16:04:14.217 INFO:teuthology.orchestra.run.smithi164.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 464 kB/s | 11 kB 00:00 2024-09-10T16:04:14.242 INFO:teuthology.orchestra.run.smithi164.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-09-10T16:04:14.268 INFO:teuthology.orchestra.run.smithi164.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-10T16:04:14.273 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T16:04:14.301 INFO:teuthology.orchestra.run.smithi164.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 395 kB/s | 23 kB 00:00 2024-09-10T16:04:14.327 INFO:teuthology.orchestra.run.smithi164.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 333 kB/s | 19 kB 00:00 2024-09-10T16:04:14.337 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T16:04:14.352 INFO:teuthology.orchestra.run.smithi164.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-10T16:04:14.377 INFO:teuthology.orchestra.run.smithi164.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 921 kB/s | 46 kB 00:00 2024-09-10T16:04:14.402 INFO:teuthology.orchestra.run.smithi164.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-10T16:04:14.428 INFO:teuthology.orchestra.run.smithi164.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-10T16:04:14.444 INFO:teuthology.orchestra.run.smithi164.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 393 kB/s | 16 kB 00:00 2024-09-10T16:04:14.461 INFO:teuthology.orchestra.run.smithi164.stdout:(32/37): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-09-10T16:04:14.487 INFO:teuthology.orchestra.run.smithi164.stdout:(33/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-09-10T16:04:14.524 INFO:teuthology.orchestra.run.smithi164.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-10T16:04:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T16:04:14.549 INFO:teuthology.orchestra.run.smithi164.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 799 kB/s | 20 kB 00:00 2024-09-10T16:04:14.575 INFO:teuthology.orchestra.run.smithi164.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.0 MB/s | 230 kB 00:00 2024-09-10T16:04:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T16:04:14.600 INFO:teuthology.orchestra.run.smithi164.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.7 MB/s | 427 kB 00:00 2024-09-10T16:04:14.604 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:14.604 INFO:teuthology.orchestra.run.smithi164.stdout:Total 15 MB/s | 31 MB 00:02 2024-09-10T16:04:14.671 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T16:04:14.752 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:14.760 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T16:04:14.794 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:14.794 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:14.809 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T16:04:14.938 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T16:04:15.109 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T16:04:15.137 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:04:15.164 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:15.298 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:16.638 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:04:16.666 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:16.693 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:16.753 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-10T16:04:16.794 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-10T16:04:16.822 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-10T16:04:16.892 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-10T16:04:16.946 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-10T16:04:16.979 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-10T16:04:17.070 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-10T16:04:17.113 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:04:17.137 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:17.140 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-10T16:04:17.190 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-10T16:04:17.233 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-10T16:04:17.263 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-10T16:04:17.298 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-10T16:04:17.331 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-10T16:04:17.364 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:04:17.371 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:04:17.388 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:17.406 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-10T16:04:17.449 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-10T16:04:17.452 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:04:17.497 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-10T16:04:17.529 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-10T16:04:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-10T16:04:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-10T16:04:17.838 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-10T16:04:17.891 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-10T16:04:17.930 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-10T16:04:17.986 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-10T16:04:18.045 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-10T16:04:18.230 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-10T16:04:18.278 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-10T16:04:18.355 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-10T16:04:18.445 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-10T16:04:18.489 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-10T16:04:18.625 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-10T16:04:18.809 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 32/37 2024-09-10T16:04:18.846 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:04:18.873 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 33/37 2024-09-10T16:04:18.873 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:18.873 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-10T16:04:18.873 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:04:18.873 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-10T16:04:18.874 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:19.203 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:04:19.203 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T16:04:19.203 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T16:04:19.203 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T16:04:19.204 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T16:04:19.205 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T16:04:19.206 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T16:04:19.207 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T16:04:20.234 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T16:04:20.235 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T16:04:20.236 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T16:04:20.237 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:20.238 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:20.330 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:04:20.358 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 34/37 2024-09-10T16:04:20.358 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:20.358 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-10T16:04:20.358 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:04:20.359 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-10T16:04:20.359 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:20.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-base 2024-09-10T16:04:20.815 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 35/37 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-10T16:04:20.839 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:21.073 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 36/37 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-10T16:04:21.097 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:21.163 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:04:21.201 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:21.292 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:04:21.335 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:21.336 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:04:21.336 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:21.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install cephadm 2024-09-10T16:04:21.990 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:22.123 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:22.124 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 784 k 2024-09-10T16:04:22.125 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 790 k 2024-09-10T16:04:22.125 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:22.382 INFO:teuthology.orchestra.run.smithi165.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-10T16:04:22.382 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:22.382 INFO:teuthology.orchestra.run.smithi165.stdout:Total 3.0 MB/s | 784 kB 00:00 2024-09-10T16:04:22.383 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:22.388 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:22.388 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:22.394 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:22.394 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:22.465 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:22.986 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 37/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 1/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 3/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 4/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 5/37 2024-09-10T16:04:23.058 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9. 6/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-10T16:04:23.059 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-10T16:04:23.060 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-10T16:04:23.061 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-10T16:04:23.070 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:23.447 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:23.700 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:23.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mds-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-modules-core-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mon-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-osd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:23.994 INFO:teuthology.orchestra.run.smithi164.stdout: lua-5.4.4-4.el9.x86_64 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-10T16:04:23.995 INFO:teuthology.orchestra.run.smithi164.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-ply-3.11-14.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-10T16:04:23.996 INFO:teuthology.orchestra.run.smithi164.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:23.997 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:24.366 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-base 2024-09-10T16:04:24.510 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:24.647 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:24.648 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:24.648 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-10T16:04:24.648 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:24.648 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 145 k 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 432 k 2024-09-10T16:04:24.649 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:24.856 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 702 kB/s | 145 kB 00:00 2024-09-10T16:04:24.856 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:24.857 INFO:teuthology.orchestra.run.smithi165.stdout:Total 697 kB/s | 145 kB 00:00 2024-09-10T16:04:24.857 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:24.864 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:24.864 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:24.921 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:24.921 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:24.939 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:52 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:25.029 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-base-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:04:25.070 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:25.071 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:04:25.072 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:25.082 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:25.142 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install cephadm 2024-09-10T16:04:25.148 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:25.173 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:25.173 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:25.173 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T16:04:25.174 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:25.708 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:53 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:25.840 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:25.841 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:25.841 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:25.841 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:25.841 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:25.841 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 784 k 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 784 k 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 790 k 2024-09-10T16:04:25.842 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:26.060 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:26.101 INFO:teuthology.orchestra.run.smithi164.stdout:cephadm-19.3.0-4858.gbad1f558.el9.noarch.rpm 3.0 MB/s | 784 kB 00:00 2024-09-10T16:04:26.101 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:26.102 INFO:teuthology.orchestra.run.smithi164.stdout:Total 2.9 MB/s | 784 kB 00:00 2024-09-10T16:04:26.102 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:26.107 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:26.107 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:26.113 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:26.113 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:26.185 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:26.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr 2024-09-10T16:04:26.672 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:26.745 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:26.919 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:27.014 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:04:27.057 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:27.058 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:04:27.058 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:27.099 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:27.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T16:04:27.368 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:04:27.369 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:27.369 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:27.369 INFO:teuthology.orchestra.run.smithi164.stdout: cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:27.369 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:27.369 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:27.563 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-immutable-object-cache 2024-09-10T16:04:27.707 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:27.842 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:27.843 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:27.843 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T16:04:27.844 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-10T16:04:27.845 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:27.846 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 19 M 2024-09-10T16:04:27.846 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 159 M 2024-09-10T16:04:27.846 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:28.139 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:56 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:28.275 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:28.276 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:28.276 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-09-10T16:04:28.276 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:28.276 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:28.276 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 145 k 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 145 k 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 432 k 2024-09-10T16:04:28.277 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:28.385 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 231 kB/s | 34 kB 00:00 2024-09-10T16:04:28.410 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 95 kB/s | 16 kB 00:00 2024-09-10T16:04:28.499 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-immutable-object-cache-19.3.0-4858.gbad1f5 654 kB/s | 145 kB 00:00 2024-09-10T16:04:28.500 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:28.500 INFO:teuthology.orchestra.run.smithi164.stdout:Total 650 kB/s | 145 kB 00:00 2024-09-10T16:04:28.500 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:28.507 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:28.507 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:28.552 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.9 MB/s | 269 kB 00:00 2024-09-10T16:04:28.565 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:28.566 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:28.602 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-10T16:04:28.653 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-10T16:04:28.670 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 1.2 MB/s | 19 kB 00:00 2024-09-10T16:04:28.721 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:28.737 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 31 MB/s | 2.0 MB 00:00 2024-09-10T16:04:28.762 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-10T16:04:28.779 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-10T16:04:28.783 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:28.804 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-10T16:04:28.808 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:28.808 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:04:28.808 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-10T16:04:28.808 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:28.979 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f5 19 MB/s | 14 MB 00:00 2024-09-10T16:04:29.013 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.8 MB/s | 863 kB 00:00 2024-09-10T16:04:29.016 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:29.016 INFO:teuthology.orchestra.run.smithi165.stdout:Total 16 MB/s | 19 MB 00:01 2024-09-10T16:04:29.064 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:29.083 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:29.083 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:29.378 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:29.378 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:29.774 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558. 1/1 2024-09-10T16:04:29.775 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:29.775 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:29.775 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-immutable-object-cache-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:04:29.775 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:29.775 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:30.026 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr 2024-09-10T16:04:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T16:04:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T16:04:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T16:04:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T16:04:30.319 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T16:04:30.452 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T16:04:30.545 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T16:04:30.606 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:58 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:30.606 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T16:04:30.635 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:04:30.667 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T16:04:30.698 INFO:teuthology.orchestra.run.smithi164.stdout:Package ceph-mgr-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:04:30.740 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:30.741 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:04:30.741 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:30.810 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-dashboard 2024-09-10T16:04:31.379 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:00:59 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:31.514 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 14 M 2024-09-10T16:04:31.516 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:31.517 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-10T16:04:31.517 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 34 k 2024-09-10T16:04:31.517 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 16 k 2024-09-10T16:04:31.517 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-10T16:04:31.517 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-10T16:04:31.518 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-10T16:04:31.519 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:31.519 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:31.519 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:31.519 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-09-10T16:04:31.519 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:31.520 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 19 M 2024-09-10T16:04:31.520 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 159 M 2024-09-10T16:04:31.521 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:31.819 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): ceph-grafana-dashboards-19.3.0-4858.gba 237 kB/s | 34 kB 00:00 2024-09-10T16:04:31.836 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): ceph-prometheus-alerts-19.3.0-4858.gbad 102 kB/s | 16 kB 00:00 2024-09-10T16:04:31.986 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.8 MB/s | 269 kB 00:00 2024-09-10T16:04:32.037 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-09-10T16:04:32.087 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-10T16:04:32.112 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 783 kB/s | 19 kB 00:00 2024-09-10T16:04:32.171 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-10T16:04:32.196 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-10T16:04:32.221 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-10T16:04:32.246 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.3 MB/s | 188 kB 00:00 2024-09-10T16:04:32.389 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): ceph-mgr-dashboard-19.3.0-4858.gbad1f5 20 MB/s | 14 MB 00:00 2024-09-10T16:04:32.555 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): protobuf-compiler-3.14.0-13.el9.x86_64 1.5 MB/s | 863 kB 00:00 2024-09-10T16:04:32.558 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:32.558 INFO:teuthology.orchestra.run.smithi164.stdout:Total 19 MB/s | 19 MB 00:01 2024-09-10T16:04:32.606 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:32.624 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:32.624 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:32.912 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:32.912 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:33.532 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:33.614 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-10T16:04:33.658 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-10T16:04:33.705 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-10T16:04:33.735 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-10T16:04:33.835 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T16:04:33.850 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-10T16:04:33.888 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:04:33.979 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-10T16:04:34.073 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-10T16:04:34.124 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-10T16:04:34.160 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:04:34.193 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 10/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T16:04:35.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T16:04:35.797 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:04:35.797 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T16:04:35.797 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:36.114 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:36.115 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:36.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T16:04:36.970 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:08 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:37.106 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:37.107 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-10T16:04:37.107 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:04:37.107 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T16:04:37.108 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout:======================================================================================= 2024-09-10T16:04:37.109 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-10T16:04:37.110 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:37.110 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 38 M 2024-09-10T16:04:37.110 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 212 M 2024-09-10T16:04:37.110 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:37.363 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-10T16:04:37.367 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 11/12 2024-09-10T16:04:37.433 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:04:37.479 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.6 MB/s | 801 kB 00:00 2024-09-10T16:04:37.621 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): ceph-mgr-diskprediction-local-19.3.0-48 17 MB/s | 7.4 MB 00:00 2024-09-10T16:04:37.980 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): flexiblas-3.0.4-8.el9.x86_64.rpm 53 kB/s | 33 kB 00:00 2024-09-10T16:04:38.113 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 37 kB/s | 18 kB 00:00 2024-09-10T16:04:38.530 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 70 kB/s | 38 kB 00:00 2024-09-10T16:04:38.697 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.5 MB/s | 3.0 MB 00:01 2024-09-10T16:04:39.150 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 12/12 2024-09-10T16:04:39.150 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el 1/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noa 2/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9 3/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-10T16:04:39.151 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-10T16:04:39.456 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 265 kB/s | 245 kB 00:00 2024-09-10T16:04:39.460 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-grafana-dashboards-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-dashboard-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:39.461 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-prometheus-alerts-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-10T16:04:39.462 INFO:teuthology.orchestra.run.smithi164.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-10T16:04:39.463 INFO:teuthology.orchestra.run.smithi164.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-10T16:04:39.463 INFO:teuthology.orchestra.run.smithi164.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-10T16:04:39.463 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:39.463 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:39.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-10T16:04:40.339 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 284 kB/s | 250 kB 00:00 2024-09-10T16:04:40.429 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:08 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:40.563 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 7.4 M 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-10T16:04:40.564 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout:======================================================================================= 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout:Install 12 Packages 2024-09-10T16:04:40.565 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:40.566 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 38 M 2024-09-10T16:04:40.566 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 212 M 2024-09-10T16:04:40.566 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:41.212 INFO:teuthology.orchestra.run.smithi164.stdout:(1/12): ceph-mgr-diskprediction-local-19.3.0-48 17 MB/s | 7.4 MB 00:00 2024-09-10T16:04:41.271 INFO:teuthology.orchestra.run.smithi164.stdout:(2/12): libquadmath-11.5.0-2.el9.x86_64.rpm 382 kB/s | 191 kB 00:00 2024-09-10T16:04:41.329 INFO:teuthology.orchestra.run.smithi164.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 280 kB/s | 33 kB 00:00 2024-09-10T16:04:41.380 INFO:teuthology.orchestra.run.smithi164.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-09-10T16:04:41.438 INFO:teuthology.orchestra.run.smithi164.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 660 kB/s | 38 kB 00:00 2024-09-10T16:04:41.547 INFO:teuthology.orchestra.run.smithi164.stdout:(6/12): libgfortran-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 801 kB 00:00 2024-09-10T16:04:41.631 INFO:teuthology.orchestra.run.smithi164.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.3 MB/s | 3.0 MB 00:00 2024-09-10T16:04:41.747 INFO:teuthology.orchestra.run.smithi164.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2024-09-10T16:04:41.806 INFO:teuthology.orchestra.run.smithi164.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 4.2 MB/s | 250 kB 00:00 2024-09-10T16:04:41.832 INFO:teuthology.orchestra.run.smithi164.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 860 kB/s | 245 kB 00:00 2024-09-10T16:04:41.907 INFO:teuthology.orchestra.run.smithi164.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 18 MB/s | 5.1 MB 00:00 2024-09-10T16:04:42.241 INFO:teuthology.orchestra.run.smithi164.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-09-10T16:04:42.241 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:42.241 INFO:teuthology.orchestra.run.smithi164.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-10T16:04:42.596 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:42.622 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:42.622 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:43.064 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:43.065 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:43.826 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:43.904 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T16:04:43.947 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:04:44.046 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:04:44.081 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T16:04:44.267 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T16:04:44.297 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T16:04:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T16:04:44.564 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 784 kB/s | 4.9 MB 00:06 2024-09-10T16:04:44.856 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 848 kB/s | 5.1 MB 00:06 2024-09-10T16:04:45.054 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T16:04:45.177 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T16:04:45.989 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 2.9 MB/s | 16 MB 00:05 2024-09-10T16:04:45.991 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:45.991 INFO:teuthology.orchestra.run.smithi165.stdout:Total 4.3 MB/s | 38 MB 00:08 2024-09-10T16:04:46.349 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:46.376 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:46.376 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:46.429 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:04:46.801 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:46.802 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:47.474 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T16:04:47.506 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:04:47.580 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:47.671 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-10T16:04:47.712 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:04:47.813 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:04:47.855 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-10T16:04:48.041 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-10T16:04:48.079 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-10T16:04:48.156 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-10T16:04:48.832 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-10T16:04:48.961 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-10T16:04:50.027 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:04:50.027 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T16:04:50.027 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:04:50.028 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T16:04:50.235 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T16:04:50.490 INFO:teuthology.orchestra.run.smithi164.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:50.491 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:50.902 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-rook 2024-09-10T16:04:51.275 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-10T16:04:51.305 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:04:51.489 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:19 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:51.625 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T16:04:51.626 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout:Install 13 Packages 2024-09-10T16:04:51.627 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:51.628 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 2.2 M 2024-09-10T16:04:51.628 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 26 M 2024-09-10T16:04:51.628 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:51.949 INFO:teuthology.orchestra.run.smithi164.stdout:(1/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 308 kB/s | 49 kB 00:00 2024-09-10T16:04:52.299 INFO:teuthology.orchestra.run.smithi164.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-10T16:04:52.341 INFO:teuthology.orchestra.run.smithi164.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 46 kB/s | 26 kB 00:00 2024-09-10T16:04:52.799 INFO:teuthology.orchestra.run.smithi164.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 261 kB/s | 222 kB 00:00 2024-09-10T16:04:53.016 INFO:teuthology.orchestra.run.smithi164.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 221 kB/s | 159 kB 00:00 2024-09-10T16:04:53.058 INFO:teuthology.orchestra.run.smithi164.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 963 kB/s | 32 kB 00:00 2024-09-10T16:04:53.075 INFO:teuthology.orchestra.run.smithi164.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 853 kB/s | 14 kB 00:00 2024-09-10T16:04:53.100 INFO:teuthology.orchestra.run.smithi164.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.6 MB/s | 220 kB 00:00 2024-09-10T16:04:53.142 INFO:teuthology.orchestra.run.smithi164.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 25 MB/s | 1.0 MB 00:00 2024-09-10T16:04:53.168 INFO:teuthology.orchestra.run.smithi164.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 337 kB/s | 279 kB 00:00 2024-09-10T16:04:53.186 INFO:teuthology.orchestra.run.smithi164.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-09-10T16:04:53.226 INFO:teuthology.orchestra.run.smithi164.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-09-10T16:04:53.418 INFO:teuthology.orchestra.run.smithi164.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-10T16:04:53.422 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:53.422 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-10T16:04:53.494 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:04:53.511 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:04:53.511 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:04:53.602 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:04:53.604 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:04:53.852 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 12/12 2024-09-10T16:04:53.852 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f 1/12 2024-09-10T16:04:53.852 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-10T16:04:53.852 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-10T16:04:53.852 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-10T16:04:53.853 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-10T16:04:54.121 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:04:54.284 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T16:04:54.337 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T16:04:54.340 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-10T16:04:54.340 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:54.340 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-10T16:04:54.341 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-10T16:04:54.342 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:54.342 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:04:54.417 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T16:04:54.454 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T16:04:54.496 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T16:04:54.599 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T16:04:54.709 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T16:04:54.765 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T16:04:54.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-rook 2024-09-10T16:04:55.357 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:27 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:04:55.492 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:04:55.493 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:55.493 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:04:55.493 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:55.493 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:04:55.493 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 49 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-10T16:04:55.494 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:04:55.495 INFO:teuthology.orchestra.run.smithi165.stdout:Install 13 Packages 2024-09-10T16:04:55.496 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:55.496 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.2 M 2024-09-10T16:04:55.497 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 26 M 2024-09-10T16:04:55.497 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:04:55.498 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T16:04:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T16:04:55.587 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T16:04:55.636 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T16:04:55.658 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:04:56.249 INFO:teuthology.orchestra.run.smithi165.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 317 kB/s | 26 kB 00:00 2024-09-10T16:04:56.274 INFO:teuthology.orchestra.run.smithi165.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 181 kB/s | 19 kB 00:00 2024-09-10T16:04:56.341 INFO:teuthology.orchestra.run.smithi165.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.4 MB/s | 222 kB 00:00 2024-09-10T16:04:56.366 INFO:teuthology.orchestra.run.smithi165.stdout:(4/13): ceph-mgr-rook-19.3.0-4858.gbad1f558.el9 248 kB/s | 49 kB 00:00 2024-09-10T16:04:56.392 INFO:teuthology.orchestra.run.smithi165.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-10T16:04:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:04:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T16:04:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T16:04:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T16:04:56.418 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T16:04:56.419 INFO:teuthology.orchestra.run.smithi165.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-09-10T16:04:56.459 INFO:teuthology.orchestra.run.smithi165.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 482 kB/s | 32 kB 00:00 2024-09-10T16:04:56.485 INFO:teuthology.orchestra.run.smithi165.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 456 kB/s | 54 kB 00:00 2024-09-10T16:04:56.510 INFO:teuthology.orchestra.run.smithi165.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 166 kB/s | 14 kB 00:00 2024-09-10T16:04:56.543 INFO:teuthology.orchestra.run.smithi165.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-09-10T16:04:56.569 INFO:teuthology.orchestra.run.smithi165.stdout:(11/13): python3-google-auth-2.34.0-1.el9.noarc 2.0 MB/s | 220 kB 00:00 2024-09-10T16:04:56.594 INFO:teuthology.orchestra.run.smithi165.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-09-10T16:04:56.636 INFO:teuthology.orchestra.run.smithi165.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.8 MB/s | 1.0 MB 00:00 2024-09-10T16:04:56.639 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:04:56.639 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.9 MB/s | 2.2 MB 00:01 2024-09-10T16:04:56.707 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:04:56.723 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:04:56.723 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:04:56.733 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T16:04:56.733 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:56.733 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:04:56.733 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:56.733 INFO:teuthology.orchestra.run.smithi164.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:56.734 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:04:56.811 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:04:56.813 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:04:57.009 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T16:04:57.320 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:04:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-10T16:04:57.577 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:25 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:04:57.579 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-10T16:04:57.638 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-10T16:04:57.672 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-10T16:04:57.711 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T16:04:57.712 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout:Install 6 Packages 2024-09-10T16:04:57.713 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:04:57.714 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-10T16:04:57.714 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 7.0 M 2024-09-10T16:04:57.714 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 33 M 2024-09-10T16:04:57.714 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:04:57.825 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-10T16:04:57.935 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-10T16:04:57.990 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-10T16:04:58.354 INFO:teuthology.orchestra.run.smithi164.stdout:(1/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 734 kB/s | 154 kB 00:00 2024-09-10T16:04:58.405 INFO:teuthology.orchestra.run.smithi164.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-10T16:04:58.421 INFO:teuthology.orchestra.run.smithi164.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-10T16:04:58.447 INFO:teuthology.orchestra.run.smithi164.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-10T16:04:58.708 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-10T16:04:58.739 INFO:teuthology.orchestra.run.smithi164.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 417 kB/s | 248 kB 00:00 2024-09-10T16:04:58.771 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-10T16:04:58.805 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-10T16:04:58.854 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-10T16:04:58.883 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 13/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 1/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-10T16:04:59.613 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-10T16:04:59.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-10T16:04:59.614 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-10T16:04:59.952 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-10T16:04:59.953 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:04:59.954 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:00.072 INFO:teuthology.orchestra.run.smithi164.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 6.0 MB 00:01 2024-09-10T16:05:00.074 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:00.075 INFO:teuthology.orchestra.run.smithi164.stdout:Total 3.0 MB/s | 7.0 MB 00:02 2024-09-10T16:05:00.149 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:00.161 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:00.161 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:00.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-cephadm 2024-09-10T16:05:00.262 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:00.264 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:00.568 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:00.738 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T16:05:00.783 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T16:05:00.797 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:32 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:00.930 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:00.930 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 154 k 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:00.931 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:00.932 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:00.932 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-09-10T16:05:00.932 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:00.932 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 7.0 M 2024-09-10T16:05:00.932 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 33 M 2024-09-10T16:05:00.933 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:01.312 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T16:05:01.402 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T16:05:01.470 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T16:05:01.495 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:05:01.539 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-10T16:05:01.590 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-10T16:05:01.615 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): ceph-mgr-cephadm-19.3.0-4858.gbad1f558.e 694 kB/s | 154 kB 00:00 2024-09-10T16:05:01.640 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-10T16:05:01.682 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-10T16:05:01.774 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 16 MB/s | 6.0 MB 00:00 2024-09-10T16:05:01.777 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:01.777 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.3 MB/s | 7.0 MB 00:00 2024-09-10T16:05:01.847 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:01.858 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:01.859 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:01.960 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:01.960 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T16:05:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T16:05:02.263 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:02.443 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-10T16:05:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T16:05:02.523 INFO:teuthology.orchestra.run.smithi164.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T16:05:02.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T16:05:02.524 INFO:teuthology.orchestra.run.smithi164.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T16:05:02.524 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:02.524 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:02.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-fuse 2024-09-10T16:05:03.051 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-10T16:05:03.137 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-10T16:05:03.206 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-10T16:05:03.230 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:05:03.348 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:31 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:03.479 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout:Installing dependencies: 2024-09-10T16:05:03.480 INFO:teuthology.orchestra.run.smithi164.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout:Install 2 Packages 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 981 k 2024-09-10T16:05:03.481 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 2.9 M 2024-09-10T16:05:03.482 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:03.749 INFO:teuthology.orchestra.run.smithi164.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 506 kB/s | 80 kB 00:00 2024-09-10T16:05:03.883 INFO:teuthology.orchestra.run.smithi164.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.0 MB/s | 901 kB 00:00 2024-09-10T16:05:03.883 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:03.884 INFO:teuthology.orchestra.run.smithi164.stdout:Total 2.4 MB/s | 981 kB 00:00 2024-09-10T16:05:03.895 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:03.917 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:03.917 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:03.999 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:03.999 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:04.045 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 6/6 2024-09-10T16:05:04.046 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 1/6 2024-09-10T16:05:04.046 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-10T16:05:04.046 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-10T16:05:04.046 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-10T16:05:04.046 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-10T16:05:04.193 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:04.328 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T16:05:04.343 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:04.344 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:04.398 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:05:04.630 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-fuse 2024-09-10T16:05:05.118 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:05:05.118 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T16:05:05.194 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:37 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:05.329 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 901 k 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:05.330 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-09-10T16:05:05.331 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:05.331 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 981 k 2024-09-10T16:05:05.331 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 2.9 M 2024-09-10T16:05:05.331 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:05.387 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:05.644 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install ceph-volume 2024-09-10T16:05:05.727 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 566 kB/s | 80 kB 00:00 2024-09-10T16:05:05.872 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): ceph-fuse-19.3.0-4858.gbad1f558.el9.x86_ 3.1 MB/s | 901 kB 00:00 2024-09-10T16:05:05.872 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:05.873 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.8 MB/s | 981 kB 00:00 2024-09-10T16:05:05.881 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:05.903 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:05.903 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:05.985 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:05.985 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:06.180 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:06.216 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:34 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:06.329 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-10T16:05:06.354 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:06.354 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:06.354 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repository Size 2024-09-10T16:05:06.354 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:06.355 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:06.356 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 285 k 2024-09-10T16:05:06.356 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 1.3 M 2024-09-10T16:05:06.356 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:06.402 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:05:06.579 INFO:teuthology.orchestra.run.smithi164.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-10T16:05:06.580 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:06.580 INFO:teuthology.orchestra.run.smithi164.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-10T16:05:06.580 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:06.589 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:06.589 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:06.618 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:06.618 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:06.835 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:06.900 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:06.914 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:06.914 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:05:06.914 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T16:05:06.914 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2/2 2024-09-10T16:05:07.084 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/2 2024-09-10T16:05:07.376 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-10T16:05:07.377 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:07.377 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:07.377 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-10T16:05:07.377 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:07.377 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:07.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-volume 2024-09-10T16:05:07.952 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:07.952 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:07.952 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:07.953 INFO:teuthology.orchestra.run.smithi164.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:05:07.953 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:07.953 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:08.265 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:40 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:08.297 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados-devel 2024-09-10T16:05:08.404 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.3.0-4858.gbad1f558.el9 ceph-noarch 285 k 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 285 k 2024-09-10T16:05:08.405 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 1.3 M 2024-09-10T16:05:08.406 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:08.633 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-volume-19.3.0-4858.gbad1f558.el9.noarch.rp 1.2 MB/s | 285 kB 00:00 2024-09-10T16:05:08.633 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:08.633 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-10T16:05:08.634 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:08.645 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:08.645 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:08.671 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:08.671 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:08.884 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:36 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:08.887 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:08.968 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:08.984 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:08.984 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:05:08.984 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-10T16:05:08.985 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:09.018 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:09.019 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:09.020 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:09.020 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 126 k 2024-09-10T16:05:09.020 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 456 k 2024-09-10T16:05:09.020 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:09.227 INFO:teuthology.orchestra.run.smithi164.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 612 kB/s | 126 kB 00:00 2024-09-10T16:05:09.228 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:09.228 INFO:teuthology.orchestra.run.smithi164.stdout:Total 608 kB/s | 126 kB 00:00 2024-09-10T16:05:09.228 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:09.233 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:09.233 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:09.297 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:09.297 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:09.449 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:09.559 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 1/1 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.3.0-4858.gbad1f558.el9.noarch 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:09.985 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:10.011 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:10.269 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:10.269 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:10.270 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:10.270 INFO:teuthology.orchestra.run.smithi164.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:10.270 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:10.270 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:10.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados-devel 2024-09-10T16:05:10.522 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs2 2024-09-10T16:05:10.873 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:11.011 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:11.013 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:11.013 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-10T16:05:11.013 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:11.013 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:11.013 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 126 k 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:11.014 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 126 k 2024-09-10T16:05:11.015 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 456 k 2024-09-10T16:05:11.015 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:11.107 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:11.201 INFO:teuthology.orchestra.run.smithi164.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:11.214 INFO:teuthology.orchestra.run.smithi165.stdout:librados-devel-19.3.0-4858.gbad1f558.el9.x86_64 628 kB/s | 126 kB 00:00 2024-09-10T16:05:11.215 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:11.215 INFO:teuthology.orchestra.run.smithi165.stdout:Total 624 kB/s | 126 kB 00:00 2024-09-10T16:05:11.215 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:11.221 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:11.221 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:11.246 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:11.247 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:11.248 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:11.284 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:11.284 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:11.323 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install libcephfs-devel 2024-09-10T16:05:11.436 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:11.553 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:11.894 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:39 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:11.999 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:12.028 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:12.029 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:12.029 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-09-10T16:05:12.029 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:12.029 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:12.030 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T16:05:12.030 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:12.030 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:12.030 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:12.032 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:12.032 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:12.032 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 32 k 2024-09-10T16:05:12.032 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 142 k 2024-09-10T16:05:12.032 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:12.184 INFO:teuthology.orchestra.run.smithi164.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 209 kB/s | 32 kB 00:00 2024-09-10T16:05:12.184 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:12.185 INFO:teuthology.orchestra.run.smithi164.stdout:Total 207 kB/s | 32 kB 00:00 2024-09-10T16:05:12.185 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:12.186 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:12.187 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:12.213 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:12.214 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:12.274 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:12.275 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:12.275 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:12.275 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:12.275 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:12.275 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:12.305 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:12.404 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:12.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs2 2024-09-10T16:05:12.778 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:13.008 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:13.009 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:13.009 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:13.009 INFO:teuthology.orchestra.run.smithi164.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:13.009 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:13.009 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:13.113 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:13.204 INFO:teuthology.orchestra.run.smithi165.stdout:Package libcephfs2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:13.214 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librados2 2024-09-10T16:05:13.249 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:13.250 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:13.250 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:13.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs-devel 2024-09-10T16:05:13.786 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:41 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:13.876 INFO:teuthology.orchestra.run.smithi164.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:13.900 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:13.919 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:13.920 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:13.920 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:13.995 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install librbd1 2024-09-10T16:05:14.036 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 32 k 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 32 k 2024-09-10T16:05:14.037 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 142 k 2024-09-10T16:05:14.038 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:14.194 INFO:teuthology.orchestra.run.smithi165.stdout:libcephfs-devel-19.3.0-4858.gbad1f558.el9.x86_6 202 kB/s | 32 kB 00:00 2024-09-10T16:05:14.195 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:14.195 INFO:teuthology.orchestra.run.smithi165.stdout:Total 201 kB/s | 32 kB 00:00 2024-09-10T16:05:14.195 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:14.199 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:14.199 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:14.221 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:14.222 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:14.308 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:14.425 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:14.568 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:42 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:14.657 INFO:teuthology.orchestra.run.smithi164.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:14.700 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:14.704 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:14.704 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:14.770 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rados 2024-09-10T16:05:14.851 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:15.089 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:15.089 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:15.089 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:15.089 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:15.090 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:15.090 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:15.283 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados2 2024-09-10T16:05:15.339 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:43 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:15.430 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:15.473 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:15.474 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:15.474 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:15.544 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rgw 2024-09-10T16:05:15.850 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:47 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:15.939 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:15.983 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:15.984 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:15.984 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:16.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librbd1 2024-09-10T16:05:16.118 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:16.209 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:16.252 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:16.253 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:16.254 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:16.329 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-cephfs 2024-09-10T16:05:16.633 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:16.722 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:16.766 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:16.767 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:16.767 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:16.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rados 2024-09-10T16:05:16.901 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:44 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:16.991 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:17.036 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:17.036 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:17.037 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:17.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install python3-rbd 2024-09-10T16:05:17.417 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:49 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:17.507 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rados-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:17.550 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:17.551 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:17.551 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:17.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rgw 2024-09-10T16:05:17.682 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:45 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:17.777 INFO:teuthology.orchestra.run.smithi164.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:17.822 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:17.823 INFO:teuthology.orchestra.run.smithi164.stdout:Nothing to do. 2024-09-10T16:05:17.823 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:17.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-fuse 2024-09-10T16:05:18.206 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:18.297 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rgw-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:18.339 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:18.340 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:18.340 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:18.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-cephfs 2024-09-10T16:05:18.493 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:46 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:18.626 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 85 k 2024-09-10T16:05:18.627 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 231 k 2024-09-10T16:05:18.628 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:18.835 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 411 kB/s | 85 kB 00:00 2024-09-10T16:05:18.836 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:18.836 INFO:teuthology.orchestra.run.smithi164.stdout:Total 407 kB/s | 85 kB 00:00 2024-09-10T16:05:18.836 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:18.843 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:18.844 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:18.907 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:18.907 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:18.978 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:50 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:19.049 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:19.068 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-cephfs-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:19.112 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:19.113 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:19.113 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:19.148 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:19.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rbd 2024-09-10T16:05:19.565 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:19.767 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:19.825 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:19.857 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rbd-2:19.3.0-4858.gbad1f558.el9.x86_64 is already installed. 2024-09-10T16:05:19.900 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:19.901 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-10T16:05:19.901 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:19.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-fuse 2024-09-10T16:05:20.077 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-mirror 2024-09-10T16:05:20.552 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:52 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:20.650 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:48 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:20.684 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:20.685 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:20.685 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-10T16:05:20.685 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:20.685 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 85 k 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 85 k 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 231 k 2024-09-10T16:05:20.686 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:20.785 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:20.786 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:20.786 INFO:teuthology.orchestra.run.smithi164.stdout: Package Arch Version Repo Size 2024-09-10T16:05:20.786 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:20.786 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 3.2 M 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 14 M 2024-09-10T16:05:20.787 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:20.873 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-fuse-19.3.0-4858.gbad1f558.el9.x86_64.rpm 456 kB/s | 85 kB 00:00 2024-09-10T16:05:20.874 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:20.874 INFO:teuthology.orchestra.run.smithi165.stdout:Total 453 kB/s | 85 kB 00:00 2024-09-10T16:05:20.874 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:20.880 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:20.881 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:20.945 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:20.945 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:21.086 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:21.139 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.0 MB/s | 3.2 MB 00:00 2024-09-10T16:05:21.140 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:21.141 INFO:teuthology.orchestra.run.smithi164.stdout:Total 9.0 MB/s | 3.2 MB 00:00 2024-09-10T16:05:21.141 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:21.150 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:21.150 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:21.186 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:21.236 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:21.236 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:21.589 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:21.678 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:21.732 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:21.757 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:21.757 INFO:teuthology.orchestra.run.smithi164.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:05:21.758 INFO:teuthology.orchestra.run.smithi164.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T16:05:21.758 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:05:21.758 INFO:teuthology.orchestra.run.smithi164.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:05:21.758 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:21.864 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:21.864 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:21.865 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:21.865 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:22.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-mirror 2024-09-10T16:05:22.741 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:54 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:22.880 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 3.2 M 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:22.881 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:22.882 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.2 M 2024-09-10T16:05:22.882 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 14 M 2024-09-10T16:05:22.882 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:22.915 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:22.916 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:22.916 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:22.916 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:23.161 DEBUG:teuthology.orchestra.run.smithi164:> sudo yum -y install rbd-nbd 2024-09-10T16:05:23.212 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-mirror-19.3.0-4858.gbad1f558.el9.x86_64.rpm 9.6 MB/s | 3.2 MB 00:00 2024-09-10T16:05:23.213 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:23.214 INFO:teuthology.orchestra.run.smithi165.stdout:Total 9.6 MB/s | 3.2 MB 00:00 2024-09-10T16:05:23.214 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:23.222 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:23.222 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:23.309 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:23.309 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:23.749 INFO:teuthology.orchestra.run.smithi164.stdout:Last metadata expiration check: 0:01:51 ago on Tue 10 Sep 2024 04:03:32 PM UTC. 2024-09-10T16:05:23.752 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:23.830 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-10T16:05:23.856 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:23.885 INFO:teuthology.orchestra.run.smithi164.stdout:Dependencies resolved. 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout: Package Architecture Version Repository Size 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:Installing: 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction Summary 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:================================================================================ 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout:Install 1 Package 2024-09-10T16:05:23.886 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:23.887 INFO:teuthology.orchestra.run.smithi164.stdout:Total download size: 170 k 2024-09-10T16:05:23.887 INFO:teuthology.orchestra.run.smithi164.stdout:Installed size: 490 k 2024-09-10T16:05:23.887 INFO:teuthology.orchestra.run.smithi164.stdout:Downloading Packages: 2024-09-10T16:05:24.102 INFO:teuthology.orchestra.run.smithi164.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 793 kB/s | 170 kB 00:00 2024-09-10T16:05:24.103 INFO:teuthology.orchestra.run.smithi164.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:24.103 INFO:teuthology.orchestra.run.smithi164.stdout:Total 788 kB/s | 170 kB 00:00 2024-09-10T16:05:24.103 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction check 2024-09-10T16:05:24.111 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction check succeeded. 2024-09-10T16:05:24.111 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction test 2024-09-10T16:05:24.175 INFO:teuthology.orchestra.run.smithi164.stdout:Transaction test succeeded. 2024-09-10T16:05:24.176 INFO:teuthology.orchestra.run.smithi164.stdout:Running transaction 2024-09-10T16:05:24.335 INFO:teuthology.orchestra.run.smithi164.stdout: Preparing : 1/1 2024-09-10T16:05:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:24.843 INFO:teuthology.orchestra.run.smithi164.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:24.898 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout:Installed: 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:25.101 INFO:teuthology.orchestra.run.smithi164.stdout:Complete! 2024-09-10T16:05:25.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-nbd 2024-09-10T16:05:25.347 DEBUG:teuthology.parallel:result is None 2024-09-10T16:05:25.769 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:57 ago on Tue 10 Sep 2024 04:03:28 PM UTC. 2024-09-10T16:05:25.901 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-10T16:05:25.902 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:25.902 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-10T16:05:25.902 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:25.902 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.3.0-4858.gbad1f558.el9 ceph 170 k 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 170 k 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 490 k 2024-09-10T16:05:25.903 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-10T16:05:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-nbd-19.3.0-4858.gbad1f558.el9.x86_64.rpm 799 kB/s | 170 kB 00:00 2024-09-10T16:05:26.117 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-10T16:05:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:Total 793 kB/s | 170 kB 00:00 2024-09-10T16:05:26.118 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-10T16:05:26.125 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-10T16:05:26.125 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-10T16:05:26.192 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-10T16:05:26.192 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-10T16:05:26.345 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-10T16:05:26.449 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:26.873 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:27.145 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 1/1 2024-09-10T16:05:27.145 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:27.145 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-10T16:05:27.146 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.3.0-4858.gbad1f558.el9.x86_64 2024-09-10T16:05:27.146 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:27.146 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-10T16:05:27.379 DEBUG:teuthology.parallel:result is None 2024-09-10T16:05:27.380 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:05:27.527 DEBUG:teuthology.orchestra.run.smithi164:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T16:05:27.553 INFO:teuthology.orchestra.run.smithi164.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T16:05:27.554 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T16:05:27.554 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T16:05:27.556 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T16:05:27.701 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-10T16:05:27.723 INFO:teuthology.orchestra.run.smithi165.stdout:19.3.0-4858.gbad1f558.el9 2024-09-10T16:05:27.724 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858.gbad1f558.el9 2024-09-10T16:05:27.724 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858.gbad1f558 is installed. 2024-09-10T16:05:27.726 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T16:05:27.726 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:27.726 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T16:05:27.756 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:27.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T16:05:27.791 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T16:05:27.791 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:27.791 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T16:05:27.825 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T16:05:27.893 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:27.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T16:05:27.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T16:05:27.985 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T16:05:27.985 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:27.985 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T16:05:28.012 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T16:05:28.078 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:28.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T16:05:28.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T16:05:28.167 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T16:05:28.167 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:28.167 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T16:05:28.193 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T16:05:28.259 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:28.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T16:05:28.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T16:05:28.349 INFO:teuthology.run_tasks:Running task ceph... 2024-09-10T16:05:28.438 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-10T16:05:28.439 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 777 /var/log/ceph 2024-09-10T16:05:28.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-09-10T16:05:28.466 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-10T16:05:28.466 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T16:05:28.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T16:05:28.531 INFO:tasks.ceph:Creating extra log directories... 2024-09-10T16:05:28.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T16:05:28.572 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T16:05:28.606 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-10T16:05:28.606 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-10T16:05:28.606 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276', 'branch': 'wip-bharath7-testing-2024-09-10-1409', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7899276', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-bharath7-testing-2024-09-10-1409'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath7-testing-2024-09-10-1409', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5245, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath7-testing-2024-09-10-1409', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'targets': {'smithi164.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMTluh4G05URZnRM8auvbuBi5V9GAruFRN9qhzIo/j/Z8wn7mWwGqgl1+mdZA2EDJe1r9Hu/gQaNh7GFy887Dpo=', 'smithi165.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPiPhaMDYMV5pBfzzoGVlHP9rBr4/ZybC6d9W8Uc/UO2qd18xBKF7Ra2SELiUTleyyV73kw8SEbF0fBrb0lOh10='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 20, 'thrash_delay': 1}}, {'rados': {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_15:30:27', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-10T16:05:28.607 DEBUG:teuthology.orchestra.run.smithi164:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T16:05:28.638 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T16:05:28.653 DEBUG:teuthology.orchestra.run.smithi164:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T16:05:28.693 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T16:05:28.718 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:28.718 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T16:05:28.773 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T16:05:28.773 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_1 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-10 16:05:22.008548178 +0000 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-10 16:02:49.406452214 +0000 2024-09-10T16:05:28.829 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-10 16:02:49.406452214 +0000 2024-09-10T16:05:28.830 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-09-10 16:02:49.406452214 +0000 2024-09-10T16:05:28.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T16:05:28.893 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-10T16:05:28.893 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-10T16:05:28.893 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.00020766 s, 2.5 MB/s 2024-09-10T16:05:28.894 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T16:05:28.951 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_2 2024-09-10T16:05:29.006 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T16:05:29.006 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-10 16:05:22.008548178 +0000 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-10 16:02:49.744446869 +0000 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-10 16:02:49.744446869 +0000 2024-09-10T16:05:29.007 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-09-10 16:02:49.744446869 +0000 2024-09-10T16:05:29.007 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T16:05:29.069 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-10T16:05:29.070 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-10T16:05:29.070 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000180274 s, 2.8 MB/s 2024-09-10T16:05:29.071 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T16:05:29.127 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_3 2024-09-10T16:05:29.183 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-10 16:05:22.009548161 +0000 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-10 16:02:50.077441600 +0000 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-10 16:02:50.077441600 +0000 2024-09-10T16:05:29.184 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-09-10 16:02:50.077441600 +0000 2024-09-10T16:05:29.184 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T16:05:29.247 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-10T16:05:29.247 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-10T16:05:29.247 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000207015 s, 2.5 MB/s 2024-09-10T16:05:29.249 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T16:05:29.306 DEBUG:teuthology.orchestra.run.smithi164:> stat /dev/vg_nvme/lv_4 2024-09-10T16:05:29.361 INFO:teuthology.orchestra.run.smithi164.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T16:05:29.361 INFO:teuthology.orchestra.run.smithi164.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Access: 2024-09-10 16:05:22.009548161 +0000 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Modify: 2024-09-10 16:02:50.397436536 +0000 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout:Change: 2024-09-10 16:02:50.397436536 +0000 2024-09-10T16:05:29.362 INFO:teuthology.orchestra.run.smithi164.stdout: Birth: 2024-09-10 16:02:50.397436536 +0000 2024-09-10T16:05:29.362 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T16:05:29.424 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records in 2024-09-10T16:05:29.425 INFO:teuthology.orchestra.run.smithi164.stderr:1+0 records out 2024-09-10T16:05:29.425 INFO:teuthology.orchestra.run.smithi164.stderr:512 bytes copied, 0.000190191 s, 2.7 MB/s 2024-09-10T16:05:29.426 DEBUG:teuthology.orchestra.run.smithi164:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T16:05:29.483 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:29.483 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:29.483 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T16:05:29.501 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T16:05:29.501 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-10T16:05:29.558 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T16:05:29.558 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.558 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-09-10T16:05:29.558 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.558 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.559 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 16:05:24.093217269 +0000 2024-09-10T16:05:29.559 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 16:02:49.422106095 +0000 2024-09-10T16:05:29.559 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 16:02:49.422106095 +0000 2024-09-10T16:05:29.559 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-10 16:02:49.422106095 +0000 2024-09-10T16:05:29.559 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T16:05:29.625 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T16:05:29.625 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T16:05:29.625 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000250989 s, 2.0 MB/s 2024-09-10T16:05:29.626 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T16:05:29.684 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-10T16:05:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T16:05:29.741 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.741 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 16:05:24.093217269 +0000 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 16:02:49.767101110 +0000 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 16:02:49.767101110 +0000 2024-09-10T16:05:29.742 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-10 16:02:49.767101110 +0000 2024-09-10T16:05:29.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T16:05:29.806 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T16:05:29.806 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T16:05:29.807 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000233564 s, 2.2 MB/s 2024-09-10T16:05:29.808 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T16:05:29.866 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-10T16:05:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T16:05:29.922 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:29.922 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-10T16:05:29.922 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:29.922 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:29.923 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 16:05:24.093217269 +0000 2024-09-10T16:05:29.923 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 16:02:50.083096545 +0000 2024-09-10T16:05:29.923 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 16:02:50.083096545 +0000 2024-09-10T16:05:29.923 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-10 16:02:50.083096545 +0000 2024-09-10T16:05:29.923 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T16:05:29.987 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T16:05:29.987 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T16:05:29.987 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000228108 s, 2.2 MB/s 2024-09-10T16:05:29.988 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T16:05:30.045 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-10T16:05:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T16:05:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T16:05:30.102 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-10T16:05:30.102 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T16:05:30.102 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-10T16:05:30.102 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-10 16:05:24.094217254 +0000 2024-09-10T16:05:30.103 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-10 16:02:50.402091937 +0000 2024-09-10T16:05:30.103 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-10 16:02:50.402091937 +0000 2024-09-10T16:05:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-10 16:02:50.402091937 +0000 2024-09-10T16:05:30.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T16:05:30.166 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-10T16:05:30.166 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-10T16:05:30.166 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000182255 s, 2.8 MB/s 2024-09-10T16:05:30.167 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T16:05:30.224 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:30.224 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-10T16:05:30.224 INFO:tasks.ceph:Generating config... 2024-09-10T16:05:30.225 INFO:tasks.ceph:[client] debug monc = 20 2024-09-10T16:05:30.226 INFO:tasks.ceph:[client] debug ms = 1 2024-09-10T16:05:30.226 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-10T16:05:30.226 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-10T16:05:30.226 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-10T16:05:30.227 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-10T16:05:30.227 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-10T16:05:30.227 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-10T16:05:30.227 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-10T16:05:30.227 INFO:tasks.ceph:[global] ms type = async 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-10T16:05:30.227 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-10T16:05:30.228 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-10T16:05:30.228 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-10T16:05:30.229 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-10T16:05:30.230 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-10T16:05:30.230 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-10T16:05:30.230 INFO:tasks.ceph:Setting up mon.a... 2024-09-10T16:05:30.230 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-10T16:05:30.288 INFO:teuthology.orchestra.run.smithi164.stdout:creating /etc/ceph/ceph.keyring 2024-09-10T16:05:30.291 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-10T16:05:30.336 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T16:05:30.361 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.164'), ('mon.c', '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]'), ('mon.b', '172.21.15.165')] 2024-09-10T16:05:30.362 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790],172.21.15.165', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-10T16:05:30.363 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:30.363 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T16:05:30.417 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.164 --addv c '[v2:172.21.15.164:3301,v1:172.21.15.164:6790]' --add b 172.21.15.165 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: generated fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:setting min_mon_release = quincy 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:epoch 0 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:min_mon_release 17 (quincy) 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:election_strategy: 1 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:05:30.503 INFO:teuthology.orchestra.run.smithi164.stdout:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:05:30.504 INFO:teuthology.orchestra.run.smithi164.stdout:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:05:30.504 INFO:teuthology.orchestra.run.smithi164.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-10T16:05:30.505 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T16:05:30.520 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b134de91-f819-41d3-9bbd-da58ec0ed6be... 2024-09-10T16:05:30.521 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T16:05:30.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T16:05:30.603 INFO:teuthology.orchestra.run.smithi164.stdout:[global] 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: chdir = "" 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: auth supported = cephx 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: filestore xattr use omap = true 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: mon clock drift allowed = 1.000 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: osd crush chooseleaf type = 0 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: auth debug = true 2024-09-10T16:05:30.604 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on old message = true 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: ms die on bug = true 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: mon pg warn max object skew = 0 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default size = 2 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow primary affinity = true 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd allow pg remap = true 2024-09-10T16:05:30.605 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on legacy crush tunables = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on crush straw calc version zero = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on no sortbitwise = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on osd down out interval zero = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon warn on too few osds = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon_allow_pool_size_one = true 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: osd default data pool replay window = 5 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: mon allow pool delete = true 2024-09-10T16:05:30.606 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: mon cluster log file level = debug 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: debug asserts on shutdown = true 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: mon health detail to clog = false 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: mon host = "172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790],172.21.15.165" 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: mon client directed command retry = 5 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: mon election default strategy = 1 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay max = 1 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay probability = 0.005 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject delay type = mon 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject internal delays = 0.002 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms inject socket failures = 2500 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: ms type = async 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: fsid = b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:05:30.607 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout:[osd] 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd journal size = 100 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub load threshold = 5.0 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd scrub max interval = 600 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock profile = high_recovery_ops 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd recover clone overlap = true 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd recovery max chunk = 1048576 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug shutdown = true 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug op order = true 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify stray on activate = true 2024-09-10T16:05:30.608 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug trim objects = true 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: osd open classes on start = true 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug pg log writeout = true 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: osd deep scrub update digest min age = 30 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: osd map max advance = 10 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: journal zero on create = true 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: filestore ondisk finisher threads = 3 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: filestore apply finisher threads = 3 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.609 INFO:teuthology.orchestra.run.smithi164.stdout: bdev debug aio = true 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug misdirected ops = true 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore block size = 96636764160 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression algorithm = snappy 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore compression mode = aggressive 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore fsck on mount = True 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore write v2 = True 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: bluestore zero block detection = True 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluefs = 20 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: debug bluestore = 20 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: debug osd = 20 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: debug rocksdb = 10 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T16:05:30.610 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd full ratio = 0.9 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd nearfull ratio = 0.8 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify cached snaps = True 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd debug verify missing on start = True 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd failsafe full ratio = 0.95 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd mclock override recovery settings = True 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd objectstore = bluestore 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue = debug_random 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: osd op queue cut off = debug_random 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr] 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: debug mgr = 20 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-10T16:05:30.611 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mgr/telemetry/nag = false 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 10 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout:[mon] 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: debug mon = 20 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: debug paxos = 20 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: debug auth = 20 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon data avail warn = 5 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon mgr mkfs grace = 240 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min pgs per osd = 4 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd reporter subtree level = osd 2024-09-10T16:05:30.612 INFO:teuthology.orchestra.run.smithi164.stdout: mon osd prime pg temp = true 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon reweight min bytes per osd = 10 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: # 1m isn't quite enough 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon_down_mkfs_grace = 2m 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.613 INFO:teuthology.orchestra.run.smithi164.stdout: mon_warn_on_filestore_osds = false 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon min osdmap epochs = 25 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune interval = 2 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune min = 15 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon osdmap full prune txsize = 2 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon scrub inject crc mismatch = 0.01 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon scrub inject missing keys = 0.05 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: mon scrub interval = 300 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: paxos service trim min = 5 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout:[client] 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: rgw cache enabled = true 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable ops log = true 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: rgw enable usage log = true 2024-09-10T16:05:30.614 INFO:teuthology.orchestra.run.smithi164.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout: debug monc = 20 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout: debug ms = 1 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.a] 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.c] 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi164.stdout:[mon.b] 2024-09-10T16:05:30.615 INFO:teuthology.orchestra.run.smithi165.stdout:[global] 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: chdir = "" 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: auth supported = cephx 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: filestore xattr use omap = true 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: mon clock drift allowed = 1.000 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: osd crush chooseleaf type = 0 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: auth debug = true 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.616 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on old message = true 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: ms die on bug = true 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon pg warn max object skew = 0 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default size = 2 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow primary affinity = true 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd allow pg remap = true 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on legacy crush tunables = false 2024-09-10T16:05:30.617 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on crush straw calc version zero = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on no sortbitwise = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on osd down out interval zero = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon warn on too few osds = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon_allow_pool_size_one = true 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: osd default data pool replay window = 5 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: mon allow pool delete = true 2024-09-10T16:05:30.618 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: mon cluster log file level = debug 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: debug asserts on shutdown = true 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: mon health detail to clog = false 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: mon host = "172.21.15.164,[v2:172.21.15.164:3301,v1:172.21.15.164:6790],172.21.15.165" 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: mon client directed command retry = 5 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: mon election default strategy = 1 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay max = 1 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay probability = 0.005 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject delay type = mon 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject internal delays = 0.002 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms inject socket failures = 2500 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: ms type = async 2024-09-10T16:05:30.619 INFO:teuthology.orchestra.run.smithi165.stdout: fsid = b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout:[osd] 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: osd journal size = 100 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub load threshold = 5.0 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: osd scrub max interval = 600 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock profile = high_recovery_ops 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.620 INFO:teuthology.orchestra.run.smithi165.stdout: osd recover clone overlap = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd recovery max chunk = 1048576 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug shutdown = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug op order = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify stray on activate = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug trim objects = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd open classes on start = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug pg log writeout = true 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd deep scrub update digest min age = 30 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: osd map max advance = 10 2024-09-10T16:05:30.621 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: journal zero on create = true 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: filestore ondisk finisher threads = 3 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: filestore apply finisher threads = 3 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bdev debug aio = true 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug misdirected ops = true 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore block size = 96636764160 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression algorithm = snappy 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore compression mode = aggressive 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore fsck on mount = True 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore write v2 = True 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: bluestore zero block detection = True 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluefs = 20 2024-09-10T16:05:30.622 INFO:teuthology.orchestra.run.smithi165.stdout: debug bluestore = 20 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: debug osd = 20 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: debug rocksdb = 10 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd full ratio = 0.9 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd nearfull ratio = 0.8 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify cached snaps = True 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: osd debug verify missing on start = True 2024-09-10T16:05:30.623 INFO:teuthology.orchestra.run.smithi165.stdout: osd failsafe full ratio = 0.95 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: osd mclock override recovery settings = True 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: osd objectstore = bluestore 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue = debug_random 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: osd op queue cut off = debug_random 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr] 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: debug mgr = 20 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-10T16:05:30.624 INFO:teuthology.orchestra.run.smithi165.stdout: mgr/telemetry/nag = false 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: debug monc = 10 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout:[mon] 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: debug mon = 20 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: debug paxos = 20 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: debug auth = 20 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: mon data avail warn = 5 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: mon mgr mkfs grace = 240 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min pgs per osd = 4 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd reporter subtree level = osd 2024-09-10T16:05:30.625 INFO:teuthology.orchestra.run.smithi165.stdout: mon osd prime pg temp = true 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: mon reweight min bytes per osd = 10 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: # 1m isn't quite enough 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: mon_down_mkfs_grace = 2m 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.626 INFO:teuthology.orchestra.run.smithi165.stdout: mon_warn_on_filestore_osds = false 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon min osdmap epochs = 25 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon osdmap full prune interval = 2 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon osdmap full prune min = 15 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon osdmap full prune txsize = 2 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon scrub inject crc mismatch = 0.01 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon scrub inject missing keys = 0.05 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: mon scrub interval = 300 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: paxos service trim min = 5 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout:[client] 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: rgw cache enabled = true 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable ops log = true 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: rgw enable usage log = true 2024-09-10T16:05:30.627 INFO:teuthology.orchestra.run.smithi165.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout: debug monc = 20 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout: debug ms = 1 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.a] 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.c] 2024-09-10T16:05:30.628 INFO:teuthology.orchestra.run.smithi165.stdout:[mon.b] 2024-09-10T16:05:30.629 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-10T16:05:30.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-10T16:05:30.724 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-10T16:05:30.725 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:30.725 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-10T16:05:30.740 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:30.740 DEBUG:teuthology.orchestra.run.smithi164:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-10T16:05:30.795 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:05:30.796 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:30.796 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T16:05:30.796 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T16:05:30.871 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:30.871 DEBUG:teuthology.orchestra.run.smithi164:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:30.927 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:05:30.927 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:30.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T16:05:30.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T16:05:30.962 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:30.962 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:31.017 INFO:tasks.ceph:Setting up mon nodes... 2024-09-10T16:05:31.018 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-10T16:05:31.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T16:05:31.118 INFO:teuthology.orchestra.run.smithi165.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T16:05:31.121 INFO:tasks.ceph:Setting up mds nodes... 2024-09-10T16:05:31.121 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-10T16:05:31.121 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T16:05:31.163 INFO:teuthology.orchestra.run.smithi165.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-10T16:05:31.175 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-10T16:05:31.175 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi164.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi165.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-10T16:05:31.175 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-10T16:05:31.201 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:31.201 INFO:tasks.ceph:role: osd.0 2024-09-10T16:05:31.201 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:05:31.201 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T16:05:31.273 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:31.274 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:31.774 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-10T16:05:31.776 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-10T16:05:31.776 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-10T16:05:31.906 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-10T16:05:31.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-10T16:05:32.006 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:32.006 INFO:tasks.ceph:role: osd.1 2024-09-10T16:05:32.006 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:05:32.007 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T16:05:32.079 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:32.079 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:32.079 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:32.079 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:32.079 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:32.080 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:32.080 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:32.080 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:32.080 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:32.080 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:32.537 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-10T16:05:32.538 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-10T16:05:32.538 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-10T16:05:32.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-10T16:05:32.661 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-10T16:05:32.726 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:32.726 INFO:tasks.ceph:role: osd.2 2024-09-10T16:05:32.726 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:05:32.726 DEBUG:teuthology.orchestra.run.smithi164:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T16:05:32.798 INFO:teuthology.orchestra.run.smithi164.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:32.798 INFO:teuthology.orchestra.run.smithi164.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:32.798 INFO:teuthology.orchestra.run.smithi164.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:32.799 INFO:teuthology.orchestra.run.smithi164.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:32.799 INFO:teuthology.orchestra.run.smithi164.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:32.799 INFO:teuthology.orchestra.run.smithi164.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:32.799 INFO:teuthology.orchestra.run.smithi164.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:33.283 INFO:teuthology.orchestra.run.smithi164.stdout:Discarding blocks...Done. 2024-09-10T16:05:33.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi164.front.sepia.ceph.com -o noatime 2024-09-10T16:05:33.285 DEBUG:teuthology.orchestra.run.smithi164:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-10T16:05:33.423 DEBUG:teuthology.orchestra.run.smithi164:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-10T16:05:33.452 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:33.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:33.544+0000 7fa663a12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-10T16:05:33.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:33.548+0000 7fa663a12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-10T16:05:33.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:33.548+0000 7fa663a12640 -1 bdev(0x561fc7961800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:33.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:33.548+0000 7fa663a12640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-10T16:05:36.713 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T16:05:36.743 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:36.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:36.832+0000 7fe1f241b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-10T16:05:36.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:36.833+0000 7fe1f241b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-10T16:05:36.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:36.833+0000 7fe1f241b640 -1 bdev(0x559dbcbf3800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:36.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:36.833+0000 7fe1f241b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-10T16:05:40.232 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T16:05:40.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:40.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:40.352+0000 7f812e816640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-10T16:05:40.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:40.353+0000 7f812e816640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-10T16:05:40.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:40.353+0000 7f812e816640 -1 bdev(0x55b3ce039800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:40.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:40.353+0000 7f812e816640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-10T16:05:44.008 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-10T16:05:44.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-10T16:05:44.065 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:44.066 INFO:tasks.ceph:role: osd.3 2024-09-10T16:05:44.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:05:44.066 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T16:05:44.143 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:44.144 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:44.668 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-10T16:05:44.670 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-10T16:05:44.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-10T16:05:44.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-10T16:05:44.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-10T16:05:44.917 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:44.917 INFO:tasks.ceph:role: osd.4 2024-09-10T16:05:44.917 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:05:44.917 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T16:05:44.992 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:44.992 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:44.992 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:44.992 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:44.992 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:44.993 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:44.993 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:44.993 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:44.993 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:44.993 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:45.442 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-10T16:05:45.445 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-10T16:05:45.445 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-10T16:05:45.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-10T16:05:45.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-10T16:05:45.635 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T16:05:45.635 INFO:tasks.ceph:role: osd.5 2024-09-10T16:05:45.635 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:05:45.636 DEBUG:teuthology.orchestra.run.smithi165:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T16:05:45.708 INFO:teuthology.orchestra.run.smithi165.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout: = sunit=0 swidth=0 blks 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T16:05:45.709 INFO:teuthology.orchestra.run.smithi165.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T16:05:46.193 INFO:teuthology.orchestra.run.smithi165.stdout:Discarding blocks...Done. 2024-09-10T16:05:46.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi165.front.sepia.ceph.com -o noatime 2024-09-10T16:05:46.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-10T16:05:46.327 DEBUG:teuthology.orchestra.run.smithi165:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-10T16:05:46.357 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:46.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:46.455+0000 7fd39ee0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-10T16:05:46.461 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:46.460+0000 7fd39ee0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-10T16:05:46.462 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:46.460+0000 7fd39ee0d640 -1 bdev(0x55acc0c21800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:46.462 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:46.460+0000 7fd39ee0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-10T16:05:49.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-10T16:05:49.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:49.740 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:49.737+0000 7f0df722e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-10T16:05:49.740 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:49.738+0000 7f0df722e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-10T16:05:49.740 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:49.738+0000 7f0df722e640 -1 bdev(0x55c7ef809800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:49.740 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:49.738+0000 7f0df722e640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-10T16:05:52.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-10T16:05:52.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:53.033+0000 7f7969dbb640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-10T16:05:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:53.033+0000 7f7969dbb640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-10T16:05:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:53.033+0000 7f7969dbb640 -1 bdev(0x55c51b4f3800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-10T16:05:53.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:05:53.033+0000 7f7969dbb640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-10T16:05:56.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-10T16:05:56.225 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-10T16:05:56.225 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:56.225 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-10T16:05:56.256 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:56.257 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-10T16:05:56.325 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:05:56.325 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-10T16:05:56.394 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:56.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-10T16:05:56.422 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:56.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-10T16:05:56.488 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:56.488 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-10T16:05:56.554 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:56.555 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-10T16:05:56.620 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:05:56.620 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-10T16:05:56.676 INFO:tasks.ceph:Adding keys to all mons... 2024-09-10T16:05:56.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T16:05:56.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T16:05:56.719 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.0] 2024-09-10T16:05:56.719 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBNbuBmY7qBIBAA6AuLJEl11mik+p/V4vr54w== 2024-09-10T16:05:56.719 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.1] 2024-09-10T16:05:56.719 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBQbuBmUPSzMRAAiVB5aSI0jmgygo67Ur5LtA== 2024-09-10T16:05:56.719 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.2] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBUbuBmtp0TFRAAcL1Pz/SZIy+R4B8ETj7N5g== 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout:[mgr.x] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBLbuBmvrEJBxAAPkbCHjGlzgZqMY+EkvS77Q== 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.3] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBabuBmyYNBGxAAeBTzK9bNRN5ufka1YuILIA== 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.4] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBdbuBmCV4KLBAA7E52v9G/5vAN65YcQDhNHQ== 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout:[osd.5] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBhbuBmy7kJAhAAOtbpUOROPVDiXt3YKkH/4g== 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout:[client.0] 2024-09-10T16:05:56.720 INFO:teuthology.orchestra.run.smithi164.stdout: key = AQBLbuBmJ1itCRAATUMq3+NxDJByLXPFFqp9Sw== 2024-09-10T16:05:56.739 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.0] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBNbuBmY7qBIBAA6AuLJEl11mik+p/V4vr54w== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.1] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBQbuBmUPSzMRAAiVB5aSI0jmgygo67Ur5LtA== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.2] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBUbuBmtp0TFRAAcL1Pz/SZIy+R4B8ETj7N5g== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[mgr.x] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBLbuBmvrEJBxAAPkbCHjGlzgZqMY+EkvS77Q== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.3] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBabuBmyYNBGxAAeBTzK9bNRN5ufka1YuILIA== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.4] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBdbuBmCV4KLBAA7E52v9G/5vAN65YcQDhNHQ== 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout:[osd.5] 2024-09-10T16:05:56.740 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBhbuBmy7kJAhAAOtbpUOROPVDiXt3YKkH/4g== 2024-09-10T16:05:56.741 INFO:teuthology.orchestra.run.smithi165.stdout:[client.0] 2024-09-10T16:05:56.741 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQBLbuBmJ1itCRAATUMq3+NxDJByLXPFFqp9Sw== 2024-09-10T16:05:56.742 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.834 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.888 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.941 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T16:05:56.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T16:05:56.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:56.997 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:57.048 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:57.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:57.102 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:57.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T16:05:57.155 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T16:05:57.157 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T16:05:57.211 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-10T16:05:57.211 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-10T16:05:57.236 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T16:05:57.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-10T16:05:57.357 DEBUG:teuthology.orchestra.run.smithi164:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-10T16:05:57.422 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T16:05:57.518 DEBUG:teuthology.orchestra.run.smithi164:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-10T16:05:57.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-10T16:05:57.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T16:05:57.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-10T16:05:57.690 DEBUG:teuthology.orchestra.run.smithi164:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:57.693 DEBUG:teuthology.orchestra.run.smithi165:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T16:05:57.745 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-10T16:05:57.746 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:05:57.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:05:57.748 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:05:57.749 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:05:57.749 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:05:57.752 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:05:57.752 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:05:57.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:05:57.788 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:05:57.789 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-10T16:05:57.789 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-10T16:05:57.789 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-10T16:05:57.791 INFO:tasks.ceph.mgr.x:Started 2024-09-10T16:05:57.791 DEBUG:tasks.ceph:set 0 configs 2024-09-10T16:05:57.791 INFO:tasks.ceph:Setting crush tunables to default 2024-09-10T16:05:57.791 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd crush tunables default 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.888+0000 7f69cd994640 10 monclient: get_monmap_and_config 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.888+0000 7f69cd994640 10 monclient: build_initial_monmap 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.888+0000 7f69cd994640 10 monclient: monmap: 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:57.890646+0000 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:57.890646+0000 2024-09-10T16:05:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:05:57.892 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 Processor -- start 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 -- start start 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient: init 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient: _reopen_session rank -1 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69c809f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-a con 0x7f69c80a2550 addr v2:172.21.15.164:3300/0 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-c con 0x7f69c80a2980 addr v2:172.21.15.165:3300/0 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c809fd10 con 0x7f69c809f930 2024-09-10T16:05:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.889+0000 7f69cd994640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f69c809cd20 con 0x7f69c80a2550 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69cd994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69cd994640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f69c809a0d0 con 0x7f69c80a2980 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69cd994640 10 monclient(hunting): _renew_subs 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69cd994640 10 monclient(hunting): authenticate will time out at 736.378012s 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47952/0 (socket says 172.21.15.164:47952) 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 learned_addr learned my addr 172.21.15.164:0/2285738294 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69c80a2550 auth_method 0 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:05:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69c80a2980 auth_method 0 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.890+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:05:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:57.892+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> v1:172.21.15.165:6789/0 conn(0x7f69c809f930 legacy=0x7f69c80a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:05:58.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69c80a2550 auth_method 0 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69c80a2980 auth_method 0 2024-09-10T16:05:58.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:58.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69c80a2550 auth_method 0 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:58.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69c80a2980 auth_method 0 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:58.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:58.492+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69c80a2550 auth_method 0 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:59.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69c80a2980 auth_method 0 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:05:59.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:05:59.293+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): tick 2024-09-10T16:06:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 msgr2=0x7f69c813db20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.165:3300/0 conn(0x7f69c80a2980 0x7f69c813db20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 -- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 msgr2=0x7f69c813b5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 --2- 172.21.15.164:0/2285738294 >> v2:172.21.15.164:3300/0 conn(0x7f69c80a2550 0x7f69c813b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 -- 172.21.15.164:0/2285738294 >> v1:172.21.15.165:6789/0 conn(0x7f69c809f930 legacy=0x7f69c80a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient: _add_conns ranks=[2,0] 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): picked mon.noname-b con 0x7f69a80036f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f69a8006270 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 -- 172.21.15.164:0/2285738294 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69c809e840 con 0x7f69a8006270 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 1 -- 172.21.15.164:0/2285738294 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f69c809cd20 con 0x7f69a80036f0 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c57fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69a80036f0 auth_method 0 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.890+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:00.892+0000 7f69bffff640 1 -- 172.21.15.164:0/2285738294 >> v1:172.21.15.164:6789/0 conn(0x7f69a8006270 legacy=0x7f69a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:01.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69a80036f0 auth_method 0 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:01.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.091+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:06:01.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69a80036f0 auth_method 0 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:01.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:01.492+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69a80036f0 auth_method 0 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:02.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:02.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:02.294+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1224737705 0 0) 0x7f69c809e840 con 0x7f69a8006270 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 10 monclient(hunting): _negotiate 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 10 monclient(hunting): my global_id is 4098 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.083+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69b4003520 con 0x7f69a8006270 2024-09-10T16:06:03.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1574550743 0 0) 0x7f69b4003520 con 0x7f69a8006270 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 msgr2=0x7f69a8005b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69a80036f0 0x7f69a8005b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69a800b0c0 con 0x7f69a8006270 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f69b000f040 con 0x7f69a8006270 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: dump: 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:06:03.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 >> v1:172.21.15.164:6789/0 conn(0x7f69a8006270 legacy=0x7f69a8008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 1 -- 172.21.15.164:0/2285738294 reap_dead start 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69b4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): picked mon.c con 0x7f69b4003ac0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): picked mon.b con 0x7f69b4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69b400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): picked mon.a con 0x7f69b40053f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f69c809cd20 con 0x7f69b40053f0 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f69b400c340 con 0x7f69b4004800 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f69b400c540 con 0x7f69b4003ac0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c67fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69b4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69b4003ac0 auth_method 0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69b400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69b4004800 auth_method 0 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69b40053f0 auth_method 0 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 msgr2=0x7f69b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 msgr2=0x7f69b400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69b400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 monclient: found mon.c 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69b4003520 con 0x7f69b4003ac0 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69cd994640 5 monclient: authenticate success, global_id 4098 2024-09-10T16:06:03.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.084+0000 7f69cd994640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.087+0000 7f69c6ffd640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 msgr2=0x7f69b4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.087+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69b4003ec0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69b8007380 tx=0x7f69b4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c34f816e2442c4b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b8031070 con 0x7f69b4003ac0 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: dump: 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:03.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69b802b550 con 0x7f69b4003ac0 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 1 -- 172.21.15.164:0/2285738294 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69b80070b0 con 0x7f69b4003ac0 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 4 set_mon_vals no callback set 2024-09-10T16:06:03.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69c67fc640 10 monclient: dump: 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 10 monclient: shutdown 2024-09-10T16:06:03.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 1 -- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 msgr2=0x7f69b4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.088+0000 7f69cd994640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69b4003ec0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f69b8007380 tx=0x7f69b4005830 comp rx=0 tx=0).stop 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 -- 172.21.15.164:0/2285738294 shutdown_connections 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69b400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69b4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 --2- 172.21.15.164:0/2285738294 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69b4003ec0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 -- 172.21.15.164:0/2285738294 >> 172.21.15.164:0/2285738294 conn(0x7f69c80a3110 msgr2=0x7f69c80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 -- 172.21.15.164:0/2285738294 shutdown_connections 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 1 -- 172.21.15.164:0/2285738294 wait complete. 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 10 monclient: build_initial_monmap 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.089+0000 7f69cd994640 10 monclient: monmap: 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:03.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 Processor -- start 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 -- start start 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient: init 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-c con 0x7f69b4003ac0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69c8099570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-b con 0x7f69b4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69c8099ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): picked mon.noname-a con 0x7f69b40053f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f69c809d930 con 0x7f69b40053f0 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f69c809ee00 con 0x7f69b4004800 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f69c809c8b0 con 0x7f69b4003ac0 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.090+0000 7f69cd994640 10 monclient(hunting): authenticate will time out at 741.578976s 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:41888/0 (socket says 172.21.15.164:41888) 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 -- 172.21.15.164:0/3570433432 learned_addr learned my addr 172.21.15.164:0/3570433432 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request con 0x7f69b4003ac0 auth_method 0 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69c8099ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): get_auth_request con 0x7f69b40053f0 auth_method 0 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69c8099570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 msgr2=0x7f69c81ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 msgr2=0x7f69c81ccb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): get_auth_request con 0x7f69b4004800 auth_method 0 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c6ffd640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 msgr2=0x7f69c81ccb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 msgr2=0x7f69c8099570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69c8099570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.091+0000 7f69c77fe640 1 -- 172.21.15.164:0/3570433432 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69c8095e10 con 0x7f69b40053f0 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.093+0000 7f69c77fe640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 msgr2=0x7f69c8099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.093+0000 7f69c77fe640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69c8099ab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69c80a2d60 tx=0x7f69c0015700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.093+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c001b070 con 0x7f69b40053f0 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: dump: 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69c0020680 con 0x7f69b40053f0 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69c0020a70 con 0x7f69b40053f0 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 5 monclient: authenticate success, global_id 4104 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 10 monclient: dump: 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 10 monclient: _renew_subs 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69c809bc90 con 0x7f69b40053f0 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c5ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 10 monclient: _renew_subs 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69c8096440 con 0x7f69b40053f0 2024-09-10T16:06:03.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f69c0005d10 con 0x7f69b40053f0 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.094+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f69c0017b80 con 0x7f69b40053f0 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.095+0000 7f69cd994640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.095+0000 7f69cd994640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.095+0000 7f69cd994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.095+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f69c8096cb0 con 0x7f69b40053f0 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.097+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f69c0025060 con 0x7f69b40053f0 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.097+0000 7f69c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.097+0000 7f69c4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:03.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.162+0000 7f69cd994640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T16:06:03.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.162+0000 7f69cd994640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T16:06:03.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.162+0000 7f69cd994640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.162+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f69c80a0760 con 0x7f69b40053f0 2024-09-10T16:06:03.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.166+0000 7f69c4ff9640 1 -- 172.21.15.164:0/3570433432 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f69c0010050 con 0x7f69b40053f0 2024-09-10T16:06:03.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.166+0000 7f69c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T16:06:03.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.166+0000 7f69c4ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-10T16:06:03.168 INFO:teuthology.orchestra.run.smithi164.stderr:adjusted tunables profile to default 2024-09-10T16:06:03.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 10 monclient: shutdown 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 msgr2=0x7f69c8099ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69c8099ab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69c80a2d60 tx=0x7f69c0015700 comp rx=0 tx=0).stop 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 shutdown_connections 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f69b40053f0 0x7f69c8099ab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f69b4004800 0x7f69c8099570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 --2- 172.21.15.164:0/3570433432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f69b4003ac0 0x7f69c81ccb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 >> 172.21.15.164:0/3570433432 conn(0x7f69c80a3110 msgr2=0x7f69c81be3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 shutdown_connections 2024-09-10T16:06:03.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:03.167+0000 7f69cd994640 1 -- 172.21.15.164:0/3570433432 wait complete. 2024-09-10T16:06:03.181 INFO:tasks.ceph:check_enable_crimson: False 2024-09-10T16:06:03.181 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-10T16:06:03.181 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:06:03.181 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-10T16:06:03.207 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:06:03.207 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-10T16:06:03.273 DEBUG:teuthology.orchestra.run.smithi164:> set -ex 2024-09-10T16:06:03.273 DEBUG:teuthology.orchestra.run.smithi164:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-10T16:06:03.338 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:06:03.338 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-10T16:06:03.364 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:06:03.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-10T16:06:03.429 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-10T16:06:03.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-10T16:06:03.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new d38e4bd2-6da0-4c53-8a8c-26ae27d74b04 0 2024-09-10T16:06:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.631+0000 7fe348eef640 10 monclient: get_monmap_and_config 2024-09-10T16:06:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.631+0000 7fe348eef640 10 monclient: build_initial_monmap 2024-09-10T16:06:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.631+0000 7fe348eef640 10 monclient: monmap: 2024-09-10T16:06:03.633 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:03.632444+0000 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:03.632444+0000 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 Processor -- start 2024-09-10T16:06:03.634 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 -- start start 2024-09-10T16:06:03.635 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient: init 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe34406d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe34406d540 0x7fe3440a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-a con 0x7fe34406d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe3440a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe344135510 con 0x7fe34406d160 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe34409cb90 con 0x7fe3440a4c20 2024-09-10T16:06:03.637 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe34409ce70 con 0x7fe34406d540 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe348eef640 10 monclient(hunting): authenticate will time out at 758.261860s 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.632+0000 7fe342d9d640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fe3440a4c20 0x7fe34413f070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:42896/0 (socket says 172.21.15.165:42896) 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe342d9d640 1 -- 172.21.15.165:0/2046865946 learned_addr learned my addr 172.21.15.165:0/2046865946 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 1 --2- 172.21.15.165:0/2046865946 >> v2:172.21.15.164:3300/0 conn(0x7fe34406d540 0x7fe3440a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): get_auth_request con 0x7fe34406d540 auth_method 0 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.638 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient(hunting): handle_auth_done global_id 4110 payload 274 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 1 -- 172.21.15.165:0/2046865946 >> v1:172.21.15.165:6789/0 conn(0x7fe3440a4c20 legacy=0x7fe34413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 1 -- 172.21.15.165:0/2046865946 >> v1:172.21.15.164:6789/0 conn(0x7fe34406d160 legacy=0x7fe344134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 1 monclient: found mon.noname-a 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:03.639 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.633+0000 7fe341d9b640 1 -- 172.21.15.165:0/2046865946 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34409d150 con 0x7fe34406d540 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe341d9b640 1 -- 172.21.15.165:0/2046865946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 msgr2=0x7fe3440a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe341d9b640 1 --2- 172.21.15.165:0/2046865946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a46e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3380097a0 tx=0x7fe338029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6cb6f8a50e4942b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 1 -- 172.21.15.165:0/2046865946 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338037070 con 0x7fe34406d540 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: dump: 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.646 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 1 -- 172.21.15.165:0/2046865946 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe338032640 con 0x7fe34406d540 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe348eef640 5 monclient: authenticate success, global_id 4110 2024-09-10T16:06:03.647 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe348eef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe348eef640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 1 -- 172.21.15.165:0/2046865946 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe338032a10 con 0x7fe34406d540 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe348eef640 4 set_mon_vals no callback set 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.636+0000 7fe34159a640 10 monclient: dump: 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.648 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 10 monclient: shutdown 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 -- 172.21.15.165:0/2046865946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 msgr2=0x7fe3440a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 --2- 172.21.15.165:0/2046865946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a46e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3380097a0 tx=0x7fe338029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 -- 172.21.15.165:0/2046865946 shutdown_connections 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 --2- 172.21.15.165:0/2046865946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a46e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 -- 172.21.15.165:0/2046865946 >> 172.21.15.165:0/2046865946 conn(0x7fe3440aacb0 msgr2=0x7fe3440ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.649 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 -- 172.21.15.165:0/2046865946 shutdown_connections 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.637+0000 7fe348eef640 1 -- 172.21.15.165:0/2046865946 wait complete. 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient: build_initial_monmap 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient: monmap: 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:03.650 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 Processor -- start 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 -- start start 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient: init 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-b con 0x7fe34406d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-a con 0x7fe34406d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe34409d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): picked mon.noname-c con 0x7fe3440a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.651 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe344140ca0 con 0x7fe34406d540 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe344140b20 con 0x7fe34406d160 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe344140e20 con 0x7fe3440a4c20 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.638+0000 7fe348eef640 10 monclient(hunting): authenticate will time out at 758.267937s 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.652 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49142/0 (socket says 172.21.15.165:49142) 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 learned_addr learned my addr 172.21.15.165:0/608364188 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): get_auth_request con 0x7fe34406d160 auth_method 0 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe34409d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): get_auth_request con 0x7fe3440a4c20 auth_method 0 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.653 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): get_auth_request con 0x7fe34406d540 auth_method 0 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.654 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe341d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient(hunting): handle_auth_done global_id 4105 payload 824 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 msgr2=0x7fe34409d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe34409d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 msgr2=0x7fe3440a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe3440a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 monclient: found mon.noname-b 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.639+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe34412f7d0 con 0x7fe34406d160 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 msgr2=0x7fe3440a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c430 tx=0x7fe33400c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.655 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3340119a0 con 0x7fe34406d160 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: dump: 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.656 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe334017070 con 0x7fe34406d160 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 5 monclient: authenticate success, global_id 4105 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 10 monclient: _renew_subs 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe34406e6b0 con 0x7fe34406d160 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe334011d80 con 0x7fe34406d160 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.657 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe332ffd640 10 monclient: dump: 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 10 monclient: _renew_subs 2024-09-10T16:06:03.658 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe340d99640 4 set_mon_vals no callback set 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.641+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3440a43e0 con 0x7fe34406d160 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe33401c460 con 0x7fe34406d160 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7fe334012920 con 0x7fe34406d160 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe348eef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe348eef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe348eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.642+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3440a46e0 con 0x7fe34406d160 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.646+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fe34406e6b0 con 0x7fe34406d160 2024-09-10T16:06:03.659 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.646+0000 7fe332ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.660 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.646+0000 7fe332ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:03.719 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.715+0000 7fe348eef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0}] 2024-09-10T16:06:03.719 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.715+0000 7fe348eef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0}] 2024-09-10T16:06:03.719 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.715+0000 7fe348eef640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.719 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.715+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0} v 0) -- 0x7fe3440764c0 con 0x7fe34406d160 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 msgr2=0x7fe3440a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 msgr2=0x7fe3440a3960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c430 tx=0x7fe33400c8f0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe33400c430 tx=0x7fe33400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 msgr2=0x7fe3440a3960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe34406d160 0x7fe3440a3960 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe3240077a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): picked mon.c con 0x7fe3440a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe324009fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.720 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): picked mon.a con 0x7fe34406d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 0x7fe32400cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): picked mon.b con 0x7fe32400aa70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe32400da00 con 0x7fe34406d540 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe32400db80 con 0x7fe32400aa70 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe32400dd50 con 0x7fe3440a4c20 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe332ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 0x7fe32400cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): get_auth_request con 0x7fe32400aa70 auth_method 0 2024-09-10T16:06:03.721 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe341d9b640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe324009fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe3240077a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient(hunting): handle_auth_done global_id 4105 payload 293 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 msgr2=0x7fe3240077a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe3240077a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 msgr2=0x7fe324009fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe324009fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.722 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 monclient: found mon.b 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7fe32400df50 con 0x7fe32400aa70 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0}] 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 1 -- 172.21.15.165:0/608364188 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0} v 0) -- 0x7fe33c006cb0 con 0x7fe32400aa70 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe342d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 10 monclient: get_auth_request con 0x7fe3440a4c20 auth_method 0 2024-09-10T16:06:03.723 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.716+0000 7fe34259c640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe3240077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:03.724 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.718+0000 7fe342d9d640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 msgr2=0x7fe32400cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.724 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.718+0000 7fe342d9d640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 0x7fe32400cf00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0074c0 tx=0x7fe33c007720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.724 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.718+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe33c006930 con 0x7fe32400aa70 2024-09-10T16:06:03.724 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.718+0000 7fe332ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe332ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe332ffd640 10 monclient: dump: 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe33c018070 con 0x7fe32400aa70 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe332ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe340d99640 4 set_mon_vals no callback set 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.719+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe33c006260 con 0x7fe32400aa70 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.724+0000 7fe332ffd640 1 -- 172.21.15.165:0/608364188 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7fe33c005550 con 0x7fe32400aa70 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.724+0000 7fe332ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d38e4bd2-6da0-4c53-8a8c-26ae27d74b04", "id": 0}] 2024-09-10T16:06:03.726 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.724+0000 7fe332ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:03.727 INFO:teuthology.orchestra.run.smithi165.stdout:0 2024-09-10T16:06:03.727 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 10 monclient: shutdown 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 msgr2=0x7fe32400cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 0x7fe32400cf00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fe33c0074c0 tx=0x7fe33c007720 comp rx=0 tx=0).stop 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 shutdown_connections 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe32400aa70 0x7fe32400cf00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe34406d540 0x7fe324009fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 --2- 172.21.15.165:0/608364188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3440a4c20 0x7fe3240077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 >> 172.21.15.165:0/608364188 conn(0x7fe3440aacb0 msgr2=0x7fe344132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 shutdown_connections 2024-09-10T16:06:03.728 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.725+0000 7fe348eef640 1 -- 172.21.15.165:0/608364188 wait complete. 2024-09-10T16:06:03.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new f75ad5a2-93bb-4997-9998-7bc4da2f5c07 1 2024-09-10T16:06:03.835 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.834+0000 7f2722a7f640 10 monclient: get_monmap_and_config 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.834+0000 7f2722a7f640 10 monclient: build_initial_monmap 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.834+0000 7f2722a7f640 10 monclient: monmap: 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:03.835194+0000 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:03.835194+0000 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:03.836 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:03.837 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.839 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 Processor -- start 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 -- start start 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient: init 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c06cb70 0x7f271c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f271c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f271c0a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:03.840 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271c0a4a80 0x7f271c1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f271c0a4a80 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271c09ce50 con 0x7f271c0a46a0 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f271c06db60 con 0x7f271c0a4a80 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f271c0a4f10 con 0x7f271c06cb70 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f2722a7f640 10 monclient(hunting): authenticate will time out at 758.464720s 2024-09-10T16:06:03.841 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271c0a4a80 0x7f271c1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271c0a4a80 0x7f271c1383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49156/0 (socket says 172.21.15.165:49156) 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 1 -- 172.21.15.165:0/1471104822 learned_addr learned my addr 172.21.15.165:0/1471104822 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): get_auth_request con 0x7f271c0a4a80 auth_method 0 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272081b640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c06cb70 0x7f271c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.835+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): get_auth_request con 0x7f271c06cb70 auth_method 0 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 10 monclient(hunting): handle_auth_done global_id 4108 payload 274 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 1 -- 172.21.15.165:0/1471104822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c06cb70 msgr2=0x7f271c134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c06cb70 0x7f271c134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 1 -- 172.21.15.165:0/1471104822 >> v1:172.21.15.164:6789/0 conn(0x7f271c0a46a0 legacy=0x7f271c137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.843 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 1 monclient: found mon.noname-c 2024-09-10T16:06:03.844 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.844 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.844 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.836+0000 7f272101c640 1 -- 172.21.15.165:0/1471104822 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c09cb70 con 0x7f271c0a4a80 2024-09-10T16:06:03.850 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f272101c640 1 -- 172.21.15.165:0/1471104822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a4a80 msgr2=0x7f271c1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.850 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f272101c640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a4a80 0x7f271c1383b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2714009e50 tx=0x7f271402a560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c646037aa4080cc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 1 -- 172.21.15.165:0/1471104822 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714036070 con 0x7f271c0a4a80 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: dump: 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.851 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f2722a7f640 5 monclient: authenticate success, global_id 4108 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f2722a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 1 -- 172.21.15.165:0/1471104822 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2714007800 con 0x7f271c0a4a80 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f2722a7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f2722a7f640 4 set_mon_vals no callback set 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 1 -- 172.21.15.165:0/1471104822 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714007bd0 con 0x7f271c0a4a80 2024-09-10T16:06:03.852 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.839+0000 7f271b7fe640 10 monclient: dump: 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.853 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 10 monclient: shutdown 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 -- 172.21.15.165:0/1471104822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a4a80 msgr2=0x7f271c1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a4a80 0x7f271c1383b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2714009e50 tx=0x7f271402a560 comp rx=0 tx=0).stop 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 -- 172.21.15.165:0/1471104822 shutdown_connections 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a4a80 0x7f271c1383b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 --2- 172.21.15.165:0/1471104822 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c06cb70 0x7f271c134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 -- 172.21.15.165:0/1471104822 >> 172.21.15.165:0/1471104822 conn(0x7f271c0aa900 msgr2=0x7f271c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 -- 172.21.15.165:0/1471104822 shutdown_connections 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 1 -- 172.21.15.165:0/1471104822 wait complete. 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 10 monclient: build_initial_monmap 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.840+0000 7f2722a7f640 10 monclient: monmap: 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:03.854 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 Processor -- start 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 -- start start 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient: init 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:03.855 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271c06cb70 0x7f271c139fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f271c06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f271c0a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c0a4a80 0x7f271c1cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f271c0a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f271c1388f0 con 0x7f271c06cb70 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f271c06cf50 con 0x7f271c0a46a0 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f271c138a70 con 0x7f271c0a4a80 2024-09-10T16:06:03.856 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f2722a7f640 10 monclient(hunting): authenticate will time out at 758.470846s 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49172/0 (socket says 172.21.15.165:49172) 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 1 -- 172.21.15.165:0/3218576265 learned_addr learned my addr 172.21.15.165:0/3218576265 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 10 monclient(hunting): get_auth_request con 0x7f271c0a46a0 auth_method 0 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.841+0000 7f271bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271c06cb70 0x7f271c139fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.857 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c0a4a80 0x7f271c1cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): get_auth_request con 0x7f271c06cb70 auth_method 0 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): get_auth_request con 0x7f271c0a4a80 auth_method 0 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:03.858 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f272101c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 -- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c0a4a80 msgr2=0x7f271c1cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c0a4a80 0x7f271c1cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 -- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271c06cb70 msgr2=0x7f271c139fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271c06cb70 0x7f271c139fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 monclient: found mon.noname-b 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.842+0000 7f271bfff640 1 -- 172.21.15.165:0/3218576265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271c13b740 con 0x7f271c0a46a0 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f271bfff640 1 -- 172.21.15.165:0/3218576265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 msgr2=0x7f271c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f271bfff640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f270800c430 tx=0x7f270800c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080119a0 con 0x7f271c0a46a0 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: dump: 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.860 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 5 monclient: authenticate success, global_id 4111 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2708017070 con 0x7f271c0a46a0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708011d80 con 0x7f271c0a46a0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 10 monclient: _renew_subs 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271c13daa0 con 0x7f271c0a46a0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f271affd640 4 set_mon_vals no callback set 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:03.861 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 10 monclient: dump: 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 10 monclient: _renew_subs 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f271c133f40 con 0x7f271c0a46a0 2024-09-10T16:06:03.862 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f270801c460 con 0x7f271c0a46a0 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.844+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f27080128d0 con 0x7f271c0a46a0 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.845+0000 7f2722a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.845+0000 7f2722a7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.845+0000 7f2722a7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.845+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271c134f70 con 0x7f271c0a46a0 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.847+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f271c13b740 con 0x7f271c0a46a0 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.847+0000 7f27197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:03.863 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.847+0000 7f27197fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:03.920 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.918+0000 7f2722a7f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f75ad5a2-93bb-4997-9998-7bc4da2f5c07", "id": 1}] 2024-09-10T16:06:03.920 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.918+0000 7f2722a7f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f75ad5a2-93bb-4997-9998-7bc4da2f5c07", "id": 1}] 2024-09-10T16:06:03.920 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.918+0000 7f2722a7f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:03.920 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.918+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f75ad5a2-93bb-4997-9998-7bc4da2f5c07", "id": 1} v 0) -- 0x7f271c0770f0 con 0x7f271c0a46a0 2024-09-10T16:06:03.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.926+0000 7f27197fa640 1 -- 172.21.15.165:0/3218576265 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f75ad5a2-93bb-4997-9998-7bc4da2f5c07", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f2708012420 con 0x7f271c0a46a0 2024-09-10T16:06:03.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.926+0000 7f27197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f75ad5a2-93bb-4997-9998-7bc4da2f5c07", "id": 1}] 2024-09-10T16:06:03.927 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.926+0000 7f27197fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:03.928 INFO:teuthology.orchestra.run.smithi165.stdout:1 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 10 monclient: shutdown 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 msgr2=0x7f271c13a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f270800c430 tx=0x7f270800c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 shutdown_connections 2024-09-10T16:06:03.929 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271c0a4a80 0x7f271c1cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271c0a46a0 0x7f271c13a500 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 --2- 172.21.15.165:0/3218576265 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271c06cb70 0x7f271c139fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 >> 172.21.15.165:0/3218576265 conn(0x7f271c0aa900 msgr2=0x7f271c12e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 shutdown_connections 2024-09-10T16:06:03.930 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:03.927+0000 7f2722a7f640 1 -- 172.21.15.165:0/3218576265 wait complete. 2024-09-10T16:06:03.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7 2 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.033+0000 7f01e17e2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.033+0000 7f01e17e2640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.033+0000 7f01e17e2640 10 monclient: monmap: 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:04.034703+0000 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:04.034703+0000 2024-09-10T16:06:04.035 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:04.036 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.034+0000 7f01e17e2640 1 Processor -- start 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.034+0000 7f01e17e2640 1 -- start start 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.034+0000 7f01e17e2640 10 monclient: init 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01dc0a0bc0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:04.041 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f01dc136260 0x7f01dc09e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc136260 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc13d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc13b070 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01dc0a1740 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f01dc069660 con 0x7f01dc136260 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01dc0a0fa0 con 0x7f01dc13b070 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01e17e2640 10 monclient(hunting): authenticate will time out at 758.664131s 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f01dc136260 0x7f01dc09e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01daffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f01dc0a0bc0 0x7f01dc09d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:42908/0 (socket says 172.21.15.165:42908) 2024-09-10T16:06:04.042 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01daffd640 1 -- 172.21.15.165:0/4126463819 learned_addr learned my addr 172.21.15.165:0/4126463819 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): get_auth_request con 0x7f01dc136260 auth_method 0 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc13d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): get_auth_request con 0x7f01dc13b070 auth_method 0 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.043 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 -- 172.21.15.165:0/4126463819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 msgr2=0x7f01dc13d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc13d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 -- 172.21.15.165:0/4126463819 >> v1:172.21.15.165:6789/0 conn(0x7f01dc0a0bc0 legacy=0x7f01dc09d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 monclient: found mon.noname-c 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.035+0000 7f01d3fff640 1 -- 172.21.15.165:0/4126463819 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc0690a0 con 0x7f01dc136260 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01d3fff640 1 -- 172.21.15.165:0/4126463819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc136260 msgr2=0x7f01dc09e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.044 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01d3fff640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc136260 0x7f01dc09e070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0697e0 tx=0x7f01c4029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9b1884fc615c6ea server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 1 -- 172.21.15.165:0/4126463819 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4037070 con 0x7f01dc136260 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: dump: 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.045 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 1 -- 172.21.15.165:0/4126463819 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c4032440 con 0x7f01dc136260 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01e17e2640 5 monclient: authenticate success, global_id 4114 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01e17e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01e17e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 1 -- 172.21.15.165:0/4126463819 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4032810 con 0x7f01dc136260 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01e17e2640 4 set_mon_vals no callback set 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:04.046 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.038+0000 7f01da7fc640 10 monclient: dump: 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 10 monclient: shutdown 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:04.047 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 -- 172.21.15.165:0/4126463819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc136260 msgr2=0x7f01dc09e070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc136260 0x7f01dc09e070 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f01dc0697e0 tx=0x7f01c4029fb0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 -- 172.21.15.165:0/4126463819 shutdown_connections 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc13d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 --2- 172.21.15.165:0/4126463819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc136260 0x7f01dc09e070 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 -- 172.21.15.165:0/4126463819 >> 172.21.15.165:0/4126463819 conn(0x7f01dc0a3fb0 msgr2=0x7f01dc0a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 -- 172.21.15.165:0/4126463819 shutdown_connections 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 1 -- 172.21.15.165:0/4126463819 wait complete. 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.039+0000 7f01e17e2640 10 monclient: monmap: 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:04.048 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 Processor -- start 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 -- start start 2024-09-10T16:06:04.049 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient: init 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-b con 0x7f01dc0a0bc0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01dc136260 0x7f01dc074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-a con 0x7f01dc136260 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc070020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): picked mon.noname-c con 0x7f01dc13b070 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.054 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc068b30 con 0x7f01dc136260 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01dc0689b0 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01dc13d9a0 con 0x7f01dc13b070 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01e17e2640 10 monclient(hunting): authenticate will time out at 758.669647s 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49182/0 (socket says 172.21.15.165:49182) 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 1 -- 172.21.15.165:0/461335995 learned_addr learned my addr 172.21.15.165:0/461335995 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): get_auth_request con 0x7f01dc0a0bc0 auth_method 0 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.055 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01dc136260 0x7f01dc074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc070020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): get_auth_request con 0x7f01dc136260 auth_method 0 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): get_auth_request con 0x7f01dc13b070 auth_method 0 2024-09-10T16:06:04.056 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.040+0000 7f01daffd640 1 -- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 msgr2=0x7f01dc070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc070020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 1 -- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01dc136260 msgr2=0x7f01dc074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.057 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01dc136260 0x7f01dc074710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 1 monclient: found mon.noname-b 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.041+0000 7f01daffd640 1 -- 172.21.15.165:0/461335995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01dc072c20 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01daffd640 1 -- 172.21.15.165:0/461335995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 msgr2=0x7f01dc13a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01daffd640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00c460 tx=0x7f01cc00c920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc007070 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:04.058 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: dump: 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.059 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01cc007230 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 5 monclient: authenticate success, global_id 4117 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 10 monclient: _renew_subs 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01dc072f00 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01cc007620 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 10 monclient: dump: 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.060 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d9ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 10 monclient: _renew_subs 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01dc072940 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.061 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f01cc01b400 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.043+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f01cc012ae0 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.044+0000 7f01e17e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.044+0000 7f01e17e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.044+0000 7f01e17e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.044+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01dc09d960 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.046+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f01cc012e00 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.046+0000 7f01d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.062 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.046+0000 7f01d37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:04.121 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.119+0000 7f01e17e2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7", "id": 2}] 2024-09-10T16:06:04.121 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.119+0000 7f01e17e2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7", "id": 2}] 2024-09-10T16:06:04.121 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.119+0000 7f01e17e2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.121 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.119+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7", "id": 2} v 0) -- 0x7f01dc1cb4b0 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.128 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.127+0000 7f01d37fe640 1 -- 172.21.15.165:0/461335995 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f01cc012680 con 0x7f01dc0a0bc0 2024-09-10T16:06:04.128 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.127+0000 7f01d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7", "id": 2}] 2024-09-10T16:06:04.128 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.127+0000 7f01d37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:04.128 INFO:teuthology.orchestra.run.smithi165.stdout:2 2024-09-10T16:06:04.129 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 10 monclient: shutdown 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 msgr2=0x7f01dc13a4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f01cc00c460 tx=0x7f01cc00c920 comp rx=0 tx=0).stop 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 shutdown_connections 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01dc13b070 0x7f01dc070020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01dc136260 0x7f01dc074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 --2- 172.21.15.165:0/461335995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01dc0a0bc0 0x7f01dc13a4e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 >> 172.21.15.165:0/461335995 conn(0x7f01dc0a3fb0 msgr2=0x7f01dc0a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 shutdown_connections 2024-09-10T16:06:04.130 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.128+0000 7f01e17e2640 1 -- 172.21.15.165:0/461335995 wait complete. 2024-09-10T16:06:04.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 48093655-e3b0-45b9-961b-c11d709531a2 3 2024-09-10T16:06:04.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 10 monclient: get_monmap_and_config 2024-09-10T16:06:04.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.238 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 10 monclient: monmap: 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:04.238086+0000 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:04.238086+0000 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:04.239 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 1 Processor -- start 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 1 -- start start 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.237+0000 7fdbcd87f640 10 monclient: init 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 0x7fdbc8134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc806d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:04.242 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 0x7fdbc80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdbc80a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdbc809cb90 con 0x7fdbc80a4c20 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdbc809ce70 con 0x7fdbc806d160 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc809d150 con 0x7fdbc806d540 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbcd87f640 10 monclient(hunting): authenticate will time out at 758.867487s 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc77fe640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fdbc80a4c20 0x7fdbc8137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.165:42914/0 (socket says 172.21.15.165:42914) 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc77fe640 1 -- 172.21.15.165:0/1374393818 learned_addr learned my addr 172.21.15.165:0/1374393818 (peer_addr_for_me v1:172.21.15.165:0/0) 2024-09-10T16:06:04.243 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 1 --2- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 0x7fdbc80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 1 --2- 172.21.15.165:0/1374393818 >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 0x7fdbc8134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): get_auth_request con 0x7fdbc806d540 auth_method 0 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): get_auth_request con 0x7fdbc806d160 auth_method 0 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.244 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 1 -- 172.21.15.165:0/1374393818 >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 msgr2=0x7fdbc8134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 1 --2- 172.21.15.165:0/1374393818 >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 0x7fdbc8134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.238+0000 7fdbbffff640 1 -- 172.21.15.165:0/1374393818 >> v1:172.21.15.165:6789/0 conn(0x7fdbc80a4c20 legacy=0x7fdbc8137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbbffff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbbffff640 1 monclient: found mon.noname-b 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbbffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbbffff640 1 -- 172.21.15.165:0/1374393818 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc8135340 con 0x7fdbc806d540 2024-09-10T16:06:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.239+0000 7fdbc6ffd640 1 --2- 172.21.15.165:0/1374393818 >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 0x7fdbc8134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.241+0000 7fdbbffff640 1 -- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 msgr2=0x7fdbc80a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.241+0000 7fdbbffff640 1 --2- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 0x7fdbc80a46e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00097a0 tx=0x7fdbb0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a83421b74c6e3f3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 1 -- 172.21.15.165:0/1374393818 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0037070 con 0x7fdbc806d540 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: dump: 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.251 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 1 -- 172.21.15.165:0/1374393818 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbb0032440 con 0x7fdbc806d540 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 5 monclient: authenticate success, global_id 4121 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:04.252 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 1 -- 172.21.15.165:0/1374393818 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0032810 con 0x7fdbc806d540 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 4 set_mon_vals no callback set 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbc67fc640 10 monclient: dump: 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.253 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 10 monclient: shutdown 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 -- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 msgr2=0x7fdbc80a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 0x7fdbc80a46e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fdbb00097a0 tx=0x7fdbb0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 -- 172.21.15.165:0/1374393818 shutdown_connections 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/1374393818 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc806d540 0x7fdbc80a46e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/1374393818 >> v2:172.21.15.164:3300/0 conn(0x7fdbc806d160 0x7fdbc8134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 -- 172.21.15.165:0/1374393818 >> 172.21.15.165:0/1374393818 conn(0x7fdbc80aacb0 msgr2=0x7fdbc80ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 -- 172.21.15.165:0/1374393818 shutdown_connections 2024-09-10T16:06:04.254 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.242+0000 7fdbcd87f640 1 -- 172.21.15.165:0/1374393818 wait complete. 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.243+0000 7fdbcd87f640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.243+0000 7fdbcd87f640 10 monclient: monmap: 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.243+0000 7fdbcd87f640 1 Processor -- start 2024-09-10T16:06:04.255 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.243+0000 7fdbcd87f640 1 -- start start 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.243+0000 7fdbcd87f640 10 monclient: init 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-a con 0x7fdbc806d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 0x7fdbc80a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-b con 0x7fdbc806d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80a4c20 0x7fdbc813dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): picked mon.noname-c con 0x7fdbc80a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc8138870 con 0x7fdbc806d160 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.256 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdbc81386f0 con 0x7fdbc806d540 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdbc81389f0 con 0x7fdbc80a4c20 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbcd87f640 10 monclient(hunting): authenticate will time out at 758.873278s 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 0x7fdbc80a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 0x7fdbc80a3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49198/0 (socket says 172.21.15.165:49198) 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.165:45662/0 (socket says 172.21.15.165:45662) 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80a4c20 0x7fdbc813dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 1 -- 172.21.15.165:0/775441995 learned_addr learned my addr 172.21.15.165:0/775441995 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): get_auth_request con 0x7fdbc806d540 auth_method 0 2024-09-10T16:06:04.257 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): get_auth_request con 0x7fdbc806d160 auth_method 0 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): get_auth_request con 0x7fdbc80a4c20 auth_method 0 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.258 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient(hunting): handle_auth_done global_id 4125 payload 824 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 -- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80a4c20 msgr2=0x7fdbc813dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80a4c20 0x7fdbc813dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 -- 172.21.15.165:0/775441995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 msgr2=0x7fdbc80a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 0x7fdbc80a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.259 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.244+0000 7fdbc6ffd640 1 -- 172.21.15.165:0/775441995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdbc813f850 con 0x7fdbc806d160 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc6ffd640 1 -- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 msgr2=0x7fdbc81cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc6ffd640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e900 tx=0x7fdbb800edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8017070 con 0x7fdbc806d160 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: dump: 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.260 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 5 monclient: authenticate success, global_id 4125 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdbb800cc10 con 0x7fdbc806d160 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 10 monclient: _renew_subs 2024-09-10T16:06:04.261 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdbc809ebe0 con 0x7fdbc806d160 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8013470 con 0x7fdbc806d160 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 10 monclient: dump: 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.262 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 10 monclient: _renew_subs 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdbc809dc80 con 0x7fdbc806d160 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc5ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fdbb801a070 con 0x7fdbc806d160 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.247+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7fdbb801cab0 con 0x7fdbc806d160 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.248+0000 7fdbcd87f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.248+0000 7fdbcd87f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.248+0000 7fdbcd87f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:04.263 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.248+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdbc806e710 con 0x7fdbc806d160 2024-09-10T16:06:04.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.250+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fdbb801cdd0 con 0x7fdbc806d160 2024-09-10T16:06:04.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.250+0000 7fdbc4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.264 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.250+0000 7fdbc4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:04.322 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.320+0000 7fdbcd87f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "48093655-e3b0-45b9-961b-c11d709531a2", "id": 3}] 2024-09-10T16:06:04.322 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.320+0000 7fdbcd87f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "48093655-e3b0-45b9-961b-c11d709531a2", "id": 3}] 2024-09-10T16:06:04.322 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.320+0000 7fdbcd87f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:04.322 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.320+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd new", "uuid": "48093655-e3b0-45b9-961b-c11d709531a2", "id": 3} v 0) -- 0x7fdbc8134e00 con 0x7fdbc806d160 2024-09-10T16:06:04.323 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.322+0000 7fdbc4ff9640 1 -- 172.21.15.165:0/775441995 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "48093655-e3b0-45b9-961b-c11d709531a2", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7fdbb801c5e0 con 0x7fdbc806d160 2024-09-10T16:06:04.323 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.322+0000 7fdbc4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "48093655-e3b0-45b9-961b-c11d709531a2", "id": 3}] 2024-09-10T16:06:04.324 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.322+0000 7fdbc4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:04.324 INFO:teuthology.orchestra.run.smithi165.stdout:3 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 10 monclient: shutdown 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 msgr2=0x7fdbc81cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800e900 tx=0x7fdbb800edc0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 shutdown_connections 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdbc80a4c20 0x7fdbc813dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdbc806d540 0x7fdbc80a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 --2- 172.21.15.165:0/775441995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdbc806d160 0x7fdbc81cd2b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.325 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 >> 172.21.15.165:0/775441995 conn(0x7fdbc80aacb0 msgr2=0x7fdbc8136740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 shutdown_connections 2024-09-10T16:06:04.326 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.323+0000 7fdbcd87f640 1 -- 172.21.15.165:0/775441995 wait complete. 2024-09-10T16:06:04.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new b0070757-0f2c-43d9-9fb7-deadfab0ea10 4 2024-09-10T16:06:04.434 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.432+0000 7f6d277c2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:04.434 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.432+0000 7f6d277c2640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.432+0000 7f6d277c2640 10 monclient: monmap: 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:04.433636+0000 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:04.433636+0000 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:04.435 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.436 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 1 Processor -- start 2024-09-10T16:06:04.436 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 1 -- start start 2024-09-10T16:06:04.436 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 10 monclient: init 2024-09-10T16:06:04.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.442 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d2006d160 0x7f6d20134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6d2006d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.433+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d2006d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 0x7f6d2013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6d2013d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d2009cb90 con 0x7f6d2006d540 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6d2009ce70 con 0x7f6d2006d160 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2009d150 con 0x7f6d2013d060 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d277c2640 10 monclient(hunting): authenticate will time out at 759.063060s 2024-09-10T16:06:04.443 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d2006d160 0x7f6d20134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6d2006d160 0x7f6d20134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49208/0 (socket says 172.21.15.165:49208) 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 -- 172.21.15.165:0/3866636220 learned_addr learned my addr 172.21.15.165:0/3866636220 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): get_auth_request con 0x7f6d2006d160 auth_method 0 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 0x7f6d2013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 10 monclient(hunting): get_auth_request con 0x7f6d2013d060 auth_method 0 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.444 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient(hunting): handle_auth_done global_id 4123 payload 274 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 -- 172.21.15.165:0/3866636220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 msgr2=0x7f6d2013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 0x7f6d2013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 -- 172.21.15.165:0/3866636220 >> v1:172.21.15.164:6789/0 conn(0x7f6d2006d540 legacy=0x7f6d200a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 monclient: found mon.noname-c 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d2555e640 1 -- 172.21.15.165:0/3866636220 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d200a4df0 con 0x7f6d2006d160 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.434+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 0x7f6d2013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d2555e640 1 -- 172.21.15.165:0/3866636220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2006d160 msgr2=0x7f6d20134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.445 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d2555e640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2006d160 0x7f6d20134e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009870 tx=0x7f6d1c02a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=45710fb1edb7ed48 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 1 -- 172.21.15.165:0/3866636220 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c037070 con 0x7f6d2006d160 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: dump: 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.446 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 1 -- 172.21.15.165:0/3866636220 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d1c032440 con 0x7f6d2006d160 2024-09-10T16:06:04.447 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 1 -- 172.21.15.165:0/3866636220 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1c032810 con 0x7f6d2006d160 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d277c2640 5 monclient: authenticate success, global_id 4123 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d277c2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d277c2640 4 set_mon_vals no callback set 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.437+0000 7f6d17fff640 10 monclient: dump: 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.448 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 10 monclient: shutdown 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 -- 172.21.15.165:0/3866636220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2006d160 msgr2=0x7f6d20134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2006d160 0x7f6d20134e00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6d1c009870 tx=0x7f6d1c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 -- 172.21.15.165:0/3866636220 shutdown_connections 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2013d060 0x7f6d2013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 --2- 172.21.15.165:0/3866636220 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2006d160 0x7f6d20134e00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.449 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 -- 172.21.15.165:0/3866636220 >> 172.21.15.165:0/3866636220 conn(0x7f6d200aacb0 msgr2=0x7f6d200ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 -- 172.21.15.165:0/3866636220 shutdown_connections 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 1 -- 172.21.15.165:0/3866636220 wait complete. 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 10 monclient: build_initial_monmap 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.438+0000 7f6d277c2640 10 monclient: monmap: 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:04.450 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 Processor -- start 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 -- start start 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient: init 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d2006d160 0x7f6d201c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-a con 0x7f6d2006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 0x7f6d200a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-c con 0x7f6d2006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): picked mon.noname-b con 0x7f6d2013d060 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:04.451 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2013fb10 con 0x7f6d2006d160 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6d2013f990 con 0x7f6d2013d060 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d2013fc90 con 0x7f6d2006d540 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d277c2640 10 monclient(hunting): authenticate will time out at 759.068635s 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49212/0 (socket says 172.21.15.165:49212) 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 1 -- 172.21.15.165:0/527445335 learned_addr learned my addr 172.21.15.165:0/527445335 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): get_auth_request con 0x7f6d2013d060 auth_method 0 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.452 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d25d5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d2006d160 0x7f6d201c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d24d5d640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 0x7f6d200a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 10 monclient(hunting): get_auth_request con 0x7f6d2006d160 auth_method 0 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d24d5d640 10 monclient(hunting): get_auth_request con 0x7f6d2006d540 auth_method 0 2024-09-10T16:06:04.453 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d24d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d24d5d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d24d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.439+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d2555e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 10 monclient(hunting): handle_auth_done global_id 4126 payload 824 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 -- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 msgr2=0x7f6d200a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 0x7f6d200a3de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 -- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d2006d160 msgr2=0x7f6d201c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d2006d160 0x7f6d201c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 10 monclient: _finish_hunting 0 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 monclient: found mon.noname-b 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:04.454 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.455 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d25d5f640 1 -- 172.21.15.165:0/527445335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d2009e400 con 0x7f6d2013d060 2024-09-10T16:06:04.455 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.440+0000 7f6d24d5d640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 0x7f6d200a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:04.456 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d25d5f640 1 -- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 msgr2=0x7f6d201cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009fd0 tx=0x7f6d1800d560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d1801a070 con 0x7f6d2013d060 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: dump: 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.457 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: _finish_auth 0 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 5 monclient: authenticate success, global_id 4126 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d18003bb0 con 0x7f6d2013d060 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d18004030 con 0x7f6d2013d060 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:04.458 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 10 monclient: dump: 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d177fe640 4 set_mon_vals no callback set 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 10 monclient: _renew_subs 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.459 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d2006e6b0 con 0x7f6d2013d060 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6d1801d070 con 0x7f6d2013d060 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 10 monclient: _renew_subs 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d2009f2e0 con 0x7f6d2013d060 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.442+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f6d18012560 con 0x7f6d2013d060 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.443+0000 7f6d277c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.443+0000 7f6d277c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.443+0000 7f6d277c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.443+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d2013f4a0 con 0x7f6d2013d060 2024-09-10T16:06:04.460 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:04.445+0000 7f6d25d5f640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009fd0 tx=0x7f6d1800d560 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f6d2013f4a0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:06:05.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.273+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f6d2013f4a0 con 0x7f6d2013d060 2024-09-10T16:06:05.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.273+0000 7f6d15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.275 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.273+0000 7f6d15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:05.376 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.374+0000 7f6d277c2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b0070757-0f2c-43d9-9fb7-deadfab0ea10", "id": 4}] 2024-09-10T16:06:05.376 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.374+0000 7f6d277c2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b0070757-0f2c-43d9-9fb7-deadfab0ea10", "id": 4}] 2024-09-10T16:06:05.376 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.374+0000 7f6d277c2640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.376 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.374+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b0070757-0f2c-43d9-9fb7-deadfab0ea10", "id": 4} v 0) -- 0x7f6d2009e100 con 0x7f6d2013d060 2024-09-10T16:06:05.380 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.379+0000 7f6d15ffb640 1 -- 172.21.15.165:0/527445335 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b0070757-0f2c-43d9-9fb7-deadfab0ea10", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f6d18024070 con 0x7f6d2013d060 2024-09-10T16:06:05.380 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.379+0000 7f6d15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b0070757-0f2c-43d9-9fb7-deadfab0ea10", "id": 4}] 2024-09-10T16:06:05.380 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.379+0000 7f6d15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:05.380 INFO:teuthology.orchestra.run.smithi165.stdout:4 2024-09-10T16:06:05.381 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 10 monclient: shutdown 2024-09-10T16:06:05.381 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 msgr2=0x7f6d201cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6d18009fd0 tx=0x7f6d1800d560 comp rx=0 tx=0).stop 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 shutdown_connections 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d2013d060 0x7f6d201cd490 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d2006d540 0x7f6d200a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 --2- 172.21.15.165:0/527445335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d2006d160 0x7f6d201c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 >> 172.21.15.165:0/527445335 conn(0x7f6d200aacb0 msgr2=0x7f6d20132700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 shutdown_connections 2024-09-10T16:06:05.382 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.380+0000 7f6d277c2640 1 -- 172.21.15.165:0/527445335 wait complete. 2024-09-10T16:06:05.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo ceph --cluster ceph osd new 8de7e69d-412c-4983-912b-baa09f77d37c 5 2024-09-10T16:06:05.490 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.488+0000 7f2424fad640 10 monclient: get_monmap_and_config 2024-09-10T16:06:05.490 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.488+0000 7f2424fad640 10 monclient: build_initial_monmap 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.488+0000 7f2424fad640 10 monclient: monmap: 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:06:05.489617+0000 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:06:05.489617+0000 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:05.491 INFO:teuthology.orchestra.run.smithi165.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:05.492 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.492 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 1 Processor -- start 2024-09-10T16:06:05.492 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 1 -- start start 2024-09-10T16:06:05.492 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 10 monclient: init 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f242006d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f242006d540 0x7f24200a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.489+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-c con 0x7f242006d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:05.502 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242013f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-b con 0x7f24200a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2420135510 con 0x7f242006d160 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f242009cb90 con 0x7f242006d540 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f242009ce70 con 0x7f24200a4c20 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f2424fad640 10 monclient(hunting): authenticate will time out at 760.119076s 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f242006d540 0x7f24200a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f242006d540 0x7f24200a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49218/0 (socket says 172.21.15.165:49218) 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 -- 172.21.15.165:0/527013634 learned_addr learned my addr 172.21.15.165:0/527013634 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:05.503 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): get_auth_request con 0x7f242006d540 auth_method 0 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241ed9d640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242013f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241ed9d640 10 monclient(hunting): get_auth_request con 0x7f24200a4c20 auth_method 0 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.504 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 -- 172.21.15.165:0/527013634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 msgr2=0x7f242013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242013f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 -- 172.21.15.165:0/527013634 >> v1:172.21.15.164:6789/0 conn(0x7f242006d160 legacy=0x7f2420134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 monclient: found mon.noname-c 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.490+0000 7f241dd9b640 1 -- 172.21.15.165:0/527013634 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242009d150 con 0x7f242006d540 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241dd9b640 1 -- 172.21.15.165:0/527013634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 msgr2=0x7f24200a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:05.505 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241dd9b640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a46e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2414009870 tx=0x7f241402a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10655678988afacd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 1 -- 172.21.15.165:0/527013634 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414037070 con 0x7f242006d540 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: dump: 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.506 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: _finish_auth 0 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f2424fad640 5 monclient: authenticate success, global_id 4129 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 1 -- 172.21.15.165:0/527013634 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2414032440 con 0x7f242006d540 2024-09-10T16:06:05.507 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f2424fad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f2424fad640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f2424fad640 4 set_mon_vals no callback set 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 1 -- 172.21.15.165:0/527013634 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414032810 con 0x7f242006d540 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.493+0000 7f241d59a640 10 monclient: dump: 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.508 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.509 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 10 monclient: shutdown 2024-09-10T16:06:05.512 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:05.512 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 -- 172.21.15.165:0/527013634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 msgr2=0x7f24200a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.512 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a46e0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2414009870 tx=0x7f241402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.512 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 -- 172.21.15.165:0/527013634 shutdown_connections 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 --2- 172.21.15.165:0/527013634 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a46e0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 -- 172.21.15.165:0/527013634 >> 172.21.15.165:0/527013634 conn(0x7f24200aacb0 msgr2=0x7f24200ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 -- 172.21.15.165:0/527013634 shutdown_connections 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 1 -- 172.21.15.165:0/527013634 wait complete. 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 10 monclient: build_initial_monmap 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.494+0000 7f2424fad640 10 monclient: monmap: 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 0 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 0.000000 2024-09-10T16:06:05.513 INFO:teuthology.orchestra.run.smithi165.stderr:created 0.000000 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 Processor -- start 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 -- start start 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient: init 2024-09-10T16:06:05.514 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 0x7f24201cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-a con 0x7f242006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-b con 0x7f242006d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242009e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): picked mon.noname-c con 0x7f24200a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f242013fb20 con 0x7f242006d160 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f242013f9a0 con 0x7f242006d540 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.515 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f242013fca0 con 0x7f24200a4c20 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f2424fad640 10 monclient(hunting): authenticate will time out at 760.124755s 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.165:49222/0 (socket says 172.21.15.165:49222) 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 1 -- 172.21.15.165:0/3863738807 learned_addr learned my addr 172.21.15.165:0/3863738807 (peer_addr_for_me v2:172.21.15.165:0/0) 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): get_auth_request con 0x7f242006d540 auth_method 0 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.516 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.495+0000 7f241dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241e59c640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 0x7f24201cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241ed9d640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242009e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 -- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 msgr2=0x7f242009e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242009e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 -- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 msgr2=0x7f24201cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 0x7f24201cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:05.517 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 monclient: found mon.noname-b 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241dd9b640 1 -- 172.21.15.165:0/3863738807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f242013f0b0 con 0x7f242006d540 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241e59c640 10 monclient: get_auth_request con 0x7f242006d160 auth_method 0 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.496+0000 7f241e59c640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 0x7f24201cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f241dd9b640 1 -- 172.21.15.165:0/3863738807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 msgr2=0x7f24200a3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f241dd9b640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2414009990 tx=0x7f24140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2414037040 con 0x7f242006d540 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: dump: 2024-09-10T16:06:05.518 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: _finish_auth 0 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:05.519 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 5 monclient: authenticate success, global_id 4132 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2414042070 con 0x7f242006d540 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 10 monclient: _renew_subs 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f242006e6b0 con 0x7f242006d540 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241403b900 con 0x7f242006d540 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 10 monclient: dump: 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:epoch 1 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.520 INFO:teuthology.orchestra.run.smithi165.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:election_strategy: 1 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 10 monclient: _renew_subs 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f241cd99640 4 set_mon_vals no callback set 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24200a01e0 con 0x7f242006d540 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f241403baa0 con 0x7f242006d540 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.498+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f2414047c30 con 0x7f242006d540 2024-09-10T16:06:05.521 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.499+0000 7f2424fad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.499+0000 7f2424fad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.499+0000 7f2424fad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.499+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24200aa1e0 con 0x7f242006d540 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.501+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2414047460 con 0x7f242006d540 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.501+0000 7f240effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.522 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.501+0000 7f240effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:05.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.572+0000 7f2424fad640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8de7e69d-412c-4983-912b-baa09f77d37c", "id": 5}] 2024-09-10T16:06:05.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.572+0000 7f2424fad640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8de7e69d-412c-4983-912b-baa09f77d37c", "id": 5}] 2024-09-10T16:06:05.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.572+0000 7f2424fad640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:05.574 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.572+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8de7e69d-412c-4983-912b-baa09f77d37c", "id": 5} v 0) -- 0x7f24200a0d40 con 0x7f242006d540 2024-09-10T16:06:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.576+0000 7f240effd640 1 -- 172.21.15.165:0/3863738807 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8de7e69d-412c-4983-912b-baa09f77d37c", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f2414047640 con 0x7f242006d540 2024-09-10T16:06:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f240effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8de7e69d-412c-4983-912b-baa09f77d37c", "id": 5}] 2024-09-10T16:06:05.579 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f240effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:05.579 INFO:teuthology.orchestra.run.smithi165.stdout:5 2024-09-10T16:06:05.581 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 10 monclient: shutdown 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 msgr2=0x7f24200a3ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2414009990 tx=0x7f24140046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 shutdown_connections 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24200a4c20 0x7f242009e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f242006d540 0x7f24200a3ca0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.577+0000 7f2424fad640 1 --2- 172.21.15.165:0/3863738807 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f242006d160 0x7f24201cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.578+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 >> 172.21.15.165:0/3863738807 conn(0x7f24200aacb0 msgr2=0x7f242012e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.578+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 shutdown_connections 2024-09-10T16:06:05.582 INFO:teuthology.orchestra.run.smithi165.stderr:2024-09-10T16:06:05.578+0000 7f2424fad640 1 -- 172.21.15.165:0/3863738807 wait complete. 2024-09-10T16:06:05.591 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T16:06:05.591 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T16:06:05.594 INFO:tasks.ceph.osd.0:Started 2024-09-10T16:06:05.594 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T16:06:05.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T16:06:05.596 INFO:tasks.ceph.osd.1:Started 2024-09-10T16:06:05.596 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-10T16:06:05.596 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-10T16:06:05.598 INFO:tasks.ceph.osd.2:Started 2024-09-10T16:06:05.598 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T16:06:05.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T16:06:05.601 INFO:tasks.ceph.osd.3:Started 2024-09-10T16:06:05.601 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-10T16:06:05.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-10T16:06:05.603 INFO:tasks.ceph.osd.4:Started 2024-09-10T16:06:05.603 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-10T16:06:05.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-10T16:06:05.606 INFO:tasks.ceph.osd.5:Started 2024-09-10T16:06:05.606 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:05.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.716+0000 7f14271bb640 10 monclient: get_monmap_and_config 2024-09-10T16:06:05.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.716+0000 7f14271bb640 10 monclient: build_initial_monmap 2024-09-10T16:06:05.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.716+0000 7f14271bb640 10 monclient: monmap: 2024-09-10T16:06:05.723 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:05.723 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:05.717106+0000 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:05.717106+0000 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.716+0000 7f14271bb640 1 Processor -- start 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 -- start start 2024-09-10T16:06:05.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient: init 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f142006d160 0x7f1420134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-c con 0x7f142006d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f142006d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-b con 0x7f142013d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f142009cb90 con 0x7f142006d540 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f142009ce70 con 0x7f142006d160 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f142009d150 con 0x7f142013d060 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f14271bb640 10 monclient(hunting): authenticate will time out at 744.204737s 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38794/0 (socket says 172.21.15.164:38794) 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 1 -- 172.21.15.164:0/228204338 learned_addr learned my addr 172.21.15.164:0/228204338 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): get_auth_request con 0x7f142013d060 auth_method 0 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 1 --2- 172.21.15.164:0/228204338 >> v2:172.21.15.165:3300/0 conn(0x7f142006d160 0x7f1420134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): get_auth_request con 0x7f142006d160 auth_method 0 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.717+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 1 -- 172.21.15.164:0/228204338 >> v2:172.21.15.165:3300/0 conn(0x7f142006d160 msgr2=0x7f1420134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 1 --2- 172.21.15.164:0/228204338 >> v2:172.21.15.165:3300/0 conn(0x7f142006d160 0x7f1420134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 1 -- 172.21.15.164:0/228204338 >> v1:172.21.15.165:6789/0 conn(0x7f142006d540 legacy=0x7f14200a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 10 monclient: _finish_hunting 0 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 1 monclient: found mon.noname-b 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.718+0000 7f1425758640 1 -- 172.21.15.164:0/228204338 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14200a4df0 con 0x7f142013d060 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.720+0000 7f1425758640 1 -- 172.21.15.164:0/228204338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 msgr2=0x7f142013f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.720+0000 7f1425758640 1 --2- 172.21.15.164:0/228204338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e50 tx=0x7f141802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90100f8a5c9abcda server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:05.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 1 -- 172.21.15.164:0/228204338 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418036070 con 0x7f142013d060 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: dump: 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.729 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 1 -- 172.21.15.164:0/228204338 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1418007800 con 0x7f142013d060 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 5 monclient: authenticate success, global_id 4145 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 1 -- 172.21.15.164:0/228204338 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418007bd0 con 0x7f142013d060 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 4 set_mon_vals no callback set 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f141f7fe640 10 monclient: dump: 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 10 monclient: shutdown 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:05.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 -- 172.21.15.164:0/228204338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 msgr2=0x7f142013f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 --2- 172.21.15.164:0/228204338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e50 tx=0x7f141802a560 comp rx=0 tx=0).stop 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 -- 172.21.15.164:0/228204338 shutdown_connections 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 --2- 172.21.15.164:0/228204338 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013f450 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 --2- 172.21.15.164:0/228204338 >> v2:172.21.15.165:3300/0 conn(0x7f142006d160 0x7f1420134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 -- 172.21.15.164:0/228204338 >> 172.21.15.164:0/228204338 conn(0x7f14200aacb0 msgr2=0x7f14200ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 -- 172.21.15.164:0/228204338 shutdown_connections 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.721+0000 7f14271bb640 1 -- 172.21.15.164:0/228204338 wait complete. 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.722+0000 7f14271bb640 10 monclient: build_initial_monmap 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.722+0000 7f14271bb640 10 monclient: monmap: 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:05.732 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:05.733 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.722+0000 7f14271bb640 1 Processor -- start 2024-09-10T16:06:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.722+0000 7f14271bb640 1 -- start start 2024-09-10T16:06:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.722+0000 7f14271bb640 10 monclient: init 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-a con 0x7f142006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f142006d540 0x7f142013be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-b con 0x7f142006d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): picked mon.noname-c con 0x7f142013d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f142013fb10 con 0x7f142006d160 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f142013f990 con 0x7f142006d540 2024-09-10T16:06:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f142013fc90 con 0x7f142013d060 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f14271bb640 10 monclient(hunting): authenticate will time out at 744.210414s 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38796/0 (socket says 172.21.15.164:38796) 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49698/0 (socket says 172.21.15.164:49698) 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 -- 172.21.15.164:0/4124085610 learned_addr learned my addr 172.21.15.164:0/4124085610 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): get_auth_request con 0x7f142013d060 auth_method 0 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): get_auth_request con 0x7f142006d160 auth_method 0 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f142006d540 0x7f142013be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): get_auth_request con 0x7f142006d540 auth_method 0 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f141ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient(hunting): handle_auth_done global_id 4148 payload 824 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 -- 172.21.15.164:0/4124085610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f142006d540 msgr2=0x7f142013be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f142006d540 0x7f142013be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 -- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 msgr2=0x7f14200a7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient: _finish_hunting 0 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 monclient: found mon.noname-c 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1425758640 1 -- 172.21.15.164:0/4124085610 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f142013db40 con 0x7f142013d060 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.723+0000 7f1424f57640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f1425758640 1 -- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 msgr2=0x7f142013c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f1425758640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e50 tx=0x7f1418004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418036050 con 0x7f142013d060 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: dump: 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 5 monclient: authenticate success, global_id 4148 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1418042070 con 0x7f142013d060 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 10 monclient: _renew_subs 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1418031d90 con 0x7f142013d060 2024-09-10T16:06:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14201cb790 con 0x7f142013d060 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 10 monclient: dump: 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:05.745 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141effd640 4 set_mon_vals no callback set 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 10 monclient: _renew_subs 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f14201369e0 con 0x7f142013d060 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1418047460 con 0x7f142013d060 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.726+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f141803da40 con 0x7f142013d060 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.727+0000 7f14271bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.727+0000 7f14271bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.727+0000 7f14271bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.727+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1420134e00 con 0x7f142013d060 2024-09-10T16:06:05.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.729+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f141803dc90 con 0x7f142013d060 2024-09-10T16:06:05.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.729+0000 7f141d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:05.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.729+0000 7f141d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:05.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f14271bb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f14271bb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f14271bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14200aa1c0 con 0x7f142013d060 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f141d7fa640 1 -- 172.21.15.164:0/4124085610 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f141803d450 con 0x7f142013d060 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f141d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:05.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.787+0000 7f141d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:05.790 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:05.790 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":8,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:05.574588+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 10 monclient: shutdown 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 msgr2=0x7f142013c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1418009e50 tx=0x7f1418004a40 comp rx=0 tx=0).stop 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 shutdown_connections 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f142013d060 0x7f142013c360 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f142006d540 0x7f142013be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 --2- 172.21.15.164:0/4124085610 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f142006d160 0x7f14200a7770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 >> 172.21.15.164:0/4124085610 conn(0x7f14200aacb0 msgr2=0x7f1420132780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 shutdown_connections 2024-09-10T16:06:05.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:05.788+0000 7f14271bb640 1 -- 172.21.15.164:0/4124085610 wait complete. 2024-09-10T16:06:05.800 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-10T16:06:05.801 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-10T16:06:06.405 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.494+0000 7f13389ec640 10 monclient: get_monmap_and_config 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.494+0000 7f13389ec640 10 monclient: build_initial_monmap 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.494+0000 7f13389ec640 10 monclient: monmap: 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:06:06.495413+0000 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:06:06.495413+0000 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.496 INFO:teuthology.misc.health.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:06.497 INFO:teuthology.misc.health.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:06.497 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:06.497 INFO:teuthology.misc.health.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:06.497 INFO:teuthology.misc.health.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:06.497 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 Processor -- start 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 -- start start 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient: init 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1334135140 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:06.499 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-b con 0x7f133409f650 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1334132ae0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f133409ea40 con 0x7f1334132ae0 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13340a2a60 con 0x7f1334135140 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1334074150 con 0x7f133409f650 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f13389ec640 10 monclient(hunting): authenticate will time out at 744.982808s 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38806/0 (socket says 172.21.15.164:38806) 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 1 -- 172.21.15.164:0/1323875671 learned_addr learned my addr 172.21.15.164:0/1323875671 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 10 monclient(hunting): get_auth_request con 0x7f133409f650 auth_method 0 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:06.500 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.495+0000 7f132bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient(hunting): handle_auth_done global_id 4151 payload 274 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 1 -- 172.21.15.164:0/1323875671 >> v1:172.21.15.165:6789/0 conn(0x7f1334135140 legacy=0x7f13340a1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 1 -- 172.21.15.164:0/1323875671 >> v1:172.21.15.164:6789/0 conn(0x7f1334132ae0 legacy=0x7f13340a2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 1 monclient: found mon.noname-b 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:06.501 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.496+0000 7f132bfff640 1 -- 172.21.15.164:0/1323875671 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f133409ec40 con 0x7f133409f650 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f132bfff640 1 -- 172.21.15.164:0/1323875671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 msgr2=0x7f1334135520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f132bfff640 1 --2- 172.21.15.164:0/1323875671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f131c009880 tx=0x7f131c02a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3dc66741e23161be server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 1 -- 172.21.15.164:0/1323875671 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c004230 con 0x7f133409f650 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: dump: 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.507 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: _finish_auth 0 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 1 -- 172.21.15.164:0/1323875671 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f131c0043d0 con 0x7f133409f650 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 5 monclient: authenticate success, global_id 4151 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:06.508 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 1 -- 172.21.15.164:0/1323875671 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f131c003810 con 0x7f133409f650 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 4 set_mon_vals no callback set 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f1331d9b640 10 monclient: dump: 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:06.509 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 10 monclient: shutdown 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 1 -- 172.21.15.164:0/1323875671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 msgr2=0x7f1334135520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.499+0000 7f13389ec640 1 --2- 172.21.15.164:0/1323875671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f131c009880 tx=0x7f131c02a050 comp rx=0 tx=0).stop 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 1 -- 172.21.15.164:0/1323875671 shutdown_connections 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 1 --2- 172.21.15.164:0/1323875671 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f1334135520 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 1 -- 172.21.15.164:0/1323875671 >> 172.21.15.164:0/1323875671 conn(0x7f13340a68b0 msgr2=0x7f13340a2d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 1 -- 172.21.15.164:0/1323875671 shutdown_connections 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 1 -- 172.21.15.164:0/1323875671 wait complete. 2024-09-10T16:06:06.510 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 10 monclient: build_initial_monmap 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.500+0000 7f13389ec640 10 monclient: monmap: 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:created 0.000000 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:06.511 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 Processor -- start 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 -- start start 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient: init 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-c con 0x7f133409f650 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1334132ae0 0x7f1334131cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1334132ae0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 0x7f1334132200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1334135140 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:06.512 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f13340a36b0 con 0x7f1334135140 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f13340a3830 con 0x7f1334132ae0 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f133413deb0 con 0x7f133409f650 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f13389ec640 10 monclient(hunting): authenticate will time out at 744.988582s 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 0x7f1334132200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38810/0 (socket says 172.21.15.164:38810) 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 1 -- 172.21.15.164:0/3431766861 learned_addr learned my addr 172.21.15.164:0/3431766861 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): get_auth_request con 0x7f133409f650 auth_method 0 2024-09-10T16:06:06.513 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): get_auth_request con 0x7f1334135140 auth_method 0 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1334132ae0 0x7f1334131cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f133259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:06.514 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): get_auth_request con 0x7f1334132ae0 auth_method 0 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f132bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient(hunting): handle_auth_done global_id 4140 payload 824 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 -- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 msgr2=0x7f13341cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 -- 172.21.15.164:0/3431766861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1334132ae0 msgr2=0x7f1334131cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1334132ae0 0x7f1334131cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:06.515 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 1 monclient: found mon.noname-a 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.501+0000 7f1332d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.502+0000 7f1332d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.502+0000 7f1332d9d640 1 -- 172.21.15.164:0/3431766861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13341ce230 con 0x7f1334135140 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.502+0000 7f133259c640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f1332d9d640 1 -- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 msgr2=0x7f1334132200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f1332d9d640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 0x7f1334132200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f132c00c750 tx=0x7f132c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c016020 con 0x7f1334135140 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: dump: 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:06.516 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: _finish_auth 0 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f132c00aa60 con 0x7f1334135140 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 5 monclient: authenticate success, global_id 4140 2024-09-10T16:06:06.517 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 10 monclient: _renew_subs 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13341337f0 con 0x7f1334135140 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f132c00ae50 con 0x7f1334135140 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 10 monclient: dump: 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:06.518 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f133159a640 4 set_mon_vals no callback set 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 10 monclient: _renew_subs 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f133409d420 con 0x7f1334135140 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f132c019020 con 0x7f1334135140 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.504+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f132c012b90 con 0x7f1334135140 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.505+0000 7f13389ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.505+0000 7f13389ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.505+0000 7f13389ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:06.519 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.505+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f133406d4b0 con 0x7f1334135140 2024-09-10T16:06:06.520 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.507+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f132c020030 con 0x7f1334135140 2024-09-10T16:06:06.520 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.507+0000 7f132affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:06.520 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.507+0000 7f132affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:06.568 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.566+0000 7f13389ec640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:06.568 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.566+0000 7f13389ec640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:06.568 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.566+0000 7f13389ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:06.568 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.566+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1334135520 con 0x7f1334135140 2024-09-10T16:06:06.569 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.567+0000 7f132affd640 1 -- 172.21.15.164:0/3431766861 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f132c012560 con 0x7f1334135140 2024-09-10T16:06:06.569 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.567+0000 7f132affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:06.569 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.567+0000 7f132affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:06.569 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-10T16:06:06.569 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":8,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:05.574588+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:06.570 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 10 monclient: shutdown 2024-09-10T16:06:06.570 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:06.570 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 msgr2=0x7f1334132200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:06.570 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 0x7f1334132200 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f132c00c750 tx=0x7f132c00cc10 comp rx=0 tx=0).stop 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 shutdown_connections 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1334135140 0x7f1334132200 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1334132ae0 0x7f1334131cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 --2- 172.21.15.164:0/3431766861 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f133409f650 0x7f13341cd3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 >> 172.21.15.164:0/3431766861 conn(0x7f13340a68b0 msgr2=0x7f13340a4d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 shutdown_connections 2024-09-10T16:06:06.571 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:06.568+0000 7f13389ec640 1 -- 172.21.15.164:0/3431766861 wait complete. 2024-09-10T16:06:06.578 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-10T16:06:06.785 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-10T16:06:06.783+0000 7f96f86b6640 -1 Falling back to public interface 2024-09-10T16:06:06.802 INFO:tasks.ceph.osd.0.smithi164.stderr:2024-09-10T16:06:06.800+0000 7f41f0cd4640 -1 Falling back to public interface 2024-09-10T16:06:07.049 INFO:tasks.ceph.osd.2.smithi164.stderr:2024-09-10T16:06:07.047+0000 7f0837dcc640 -1 Falling back to public interface 2024-09-10T16:06:07.062 INFO:tasks.ceph.osd.1.smithi164.stderr:2024-09-10T16:06:07.060+0000 7f063b211640 -1 Falling back to public interface 2024-09-10T16:06:07.303 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:06:07.301+0000 7f1d8b1c5640 -1 Falling back to public interface 2024-09-10T16:06:07.311 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-09-10T16:06:07.310+0000 7fcf45633640 -1 Falling back to public interface 2024-09-10T16:06:08.365 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-10T16:06:08.363+0000 7f768bea0640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-10T16:06:08.367 INFO:tasks.ceph.mgr.x.smithi165.stderr:2024-09-10T16:06:08.366+0000 7f768bea0640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T16:06:10.346 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-10T16:06:10.344+0000 7f96f86b6640 -1 osd.3 0 log_to_monitors true 2024-09-10T16:06:10.615 INFO:tasks.ceph.osd.0.smithi164.stderr:2024-09-10T16:06:10.614+0000 7f41f0cd4640 -1 osd.0 0 log_to_monitors true 2024-09-10T16:06:10.802 INFO:tasks.ceph.osd.2.smithi164.stderr:2024-09-10T16:06:10.801+0000 7f0837dcc640 -1 osd.2 0 log_to_monitors true 2024-09-10T16:06:11.347 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-09-10T16:06:11.345+0000 7fcf45633640 -1 osd.5 0 log_to_monitors true 2024-09-10T16:06:11.368 INFO:tasks.ceph.osd.1.smithi164.stderr:2024-09-10T16:06:11.366+0000 7f063b211640 -1 osd.1 0 log_to_monitors true 2024-09-10T16:06:11.546 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:06:11.544+0000 7f1d8b1c5640 -1 osd.4 0 log_to_monitors true 2024-09-10T16:06:11.976 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-10T16:06:11.974+0000 7f96f4645640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T16:06:11.986 INFO:tasks.ceph.osd.3.smithi165.stderr:2024-09-10T16:06:11.985+0000 7f96ec464640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:12.386 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-09-10T16:06:12.385+0000 7fcf415c2640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T16:06:12.389 INFO:tasks.ceph.osd.5.smithi165.stderr:2024-09-10T16:06:12.388+0000 7fcf3d3e9640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:13.160 INFO:tasks.ceph.osd.0.smithi164.stderr:2024-09-10T16:06:13.159+0000 7f41ecc63640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T16:06:13.173 INFO:tasks.ceph.osd.0.smithi164.stderr:2024-09-10T16:06:13.172+0000 7f41e4a82640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:13.175 INFO:tasks.ceph.osd.2.smithi164.stderr:2024-09-10T16:06:13.174+0000 7f083456f640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T16:06:13.183 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:13.191 INFO:tasks.ceph.osd.2.smithi164.stderr:2024-09-10T16:06:13.190+0000 7f082bb7a640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 10 monclient: get_monmap_and_config 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 10 monclient: build_initial_monmap 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 10 monclient: monmap: 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:13.274 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:06:13.273227+0000 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:06:13.273227+0000 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 1 Processor -- start 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 1 -- start start 2024-09-10T16:06:13.275 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.272+0000 7f1b40952640 10 monclient: init 2024-09-10T16:06:13.277 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:13.277 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:06:13.277 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b3c068920 0x7f1b3c138250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c068920 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b3c068d00 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b3c13d270 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3c074680 con 0x7f1b3c13d270 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3c0a1ce0 con 0x7f1b3c068d00 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3c0a4c10 con 0x7f1b3c068920 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b40952640 10 monclient(hunting): authenticate will time out at 751.760457s 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b3c068920 0x7f1b3c138250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:13.278 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1b3c13d270 0x7f1b3c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53952/0 (socket says 172.21.15.164:53952) 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3ad9d640 1 -- 172.21.15.164:0/160390686 learned_addr learned my addr 172.21.15.164:0/160390686 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b3c068920 0x7f1b3c138250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49756/0 (socket says 172.21.15.164:49756) 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): get_auth_request con 0x7f1b3c068920 auth_method 0 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_done global_id 4176 payload 274 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 -- 172.21.15.164:0/160390686 >> v1:172.21.15.165:6789/0 conn(0x7f1b3c068d00 legacy=0x7f1b3c069110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:13.279 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 -- 172.21.15.164:0/160390686 >> v1:172.21.15.164:6789/0 conn(0x7f1b3c13d270 legacy=0x7f1b3c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:13.280 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:13.280 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 monclient: found mon.noname-a 2024-09-10T16:06:13.280 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:13.280 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:13.280 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.273+0000 7f1b3a59c640 1 -- 172.21.15.164:0/160390686 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c09dcc0 con 0x7f1b3c068920 2024-09-10T16:06:13.287 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b3a59c640 1 -- 172.21.15.164:0/160390686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c068920 msgr2=0x7f1b3c138250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:13.287 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b3a59c640 1 --2- 172.21.15.164:0/160390686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c068920 0x7f1b3c138250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097a0 tx=0x7f1b2c029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c3c9457496d1d6a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:13.287 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 1 -- 172.21.15.164:0/160390686 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c037070 con 0x7f1b3c068920 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: dump: 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:13.288 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: _finish_auth 0 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b40952640 5 monclient: authenticate success, global_id 4176 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b40952640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 1 -- 172.21.15.164:0/160390686 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c032640 con 0x7f1b3c068920 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 1 -- 172.21.15.164:0/160390686 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c032a10 con 0x7f1b3c068920 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b40952640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b40952640 4 set_mon_vals no callback set 2024-09-10T16:06:13.289 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.276+0000 7f1b39d9b640 10 monclient: dump: 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.290 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 10 monclient: shutdown 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 -- 172.21.15.164:0/160390686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c068920 msgr2=0x7f1b3c138250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 --2- 172.21.15.164:0/160390686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c068920 0x7f1b3c138250 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c0097a0 tx=0x7f1b2c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 -- 172.21.15.164:0/160390686 shutdown_connections 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 --2- 172.21.15.164:0/160390686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c068920 0x7f1b3c138250 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 -- 172.21.15.164:0/160390686 >> 172.21.15.164:0/160390686 conn(0x7f1b3c0a3cd0 msgr2=0x7f1b3c0a40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 -- 172.21.15.164:0/160390686 shutdown_connections 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.277+0000 7f1b40952640 1 -- 172.21.15.164:0/160390686 wait complete. 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.278+0000 7f1b40952640 10 monclient: build_initial_monmap 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.278+0000 7f1b40952640 10 monclient: monmap: 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:13.291 INFO:teuthology.misc.health.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:created 0.000000 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.278+0000 7f1b40952640 1 Processor -- start 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.278+0000 7f1b40952640 1 -- start start 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.278+0000 7f1b40952640 10 monclient: init 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:13.292 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b3c068920 0x7f1b3c1cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-b con 0x7f1b3c068920 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-c con 0x7f1b3c068d00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 0x7f1b3c13a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): picked mon.noname-a con 0x7f1b3c13d270 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c0a1fc0 con 0x7f1b3c13d270 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:13.293 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3c140d80 con 0x7f1b3c068920 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1b3c140f00 con 0x7f1b3c068d00 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b40952640 10 monclient(hunting): authenticate will time out at 751.766314s 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 0x7f1b3c13a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38842/0 (socket says 172.21.15.164:38842) 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 -- 172.21.15.164:0/320497381 learned_addr learned my addr 172.21.15.164:0/320497381 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): get_auth_request con 0x7f1b3c068d00 auth_method 0 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:13.294 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): get_auth_request con 0x7f1b3c13d270 auth_method 0 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b3c068920 0x7f1b3c1cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): get_auth_request con 0x7f1b3c068920 auth_method 0 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:13.295 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 -- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b3c068920 msgr2=0x7f1b3c1cd030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b3c068920 0x7f1b3c1cd030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 -- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 msgr2=0x7f1b3c13a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 0x7f1b3c13a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 monclient: found mon.noname-c 2024-09-10T16:06:13.296 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b31d9b640 1 -- 172.21.15.164:0/320497381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3c11e400 con 0x7f1b3c068d00 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.279+0000 7f1b3ad9d640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 0x7f1b3c13a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b31d9b640 1 -- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 msgr2=0x7f1b3c134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b31d9b640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c430 tx=0x7f1b2400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b240119a0 con 0x7f1b3c068d00 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: dump: 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:13.297 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:13.302 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 5 monclient: authenticate success, global_id 4190 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b24017070 con 0x7f1b3c068d00 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 10 monclient: _renew_subs 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3c09e3c0 con 0x7f1b3c068d00 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b24011d80 con 0x7f1b3c068d00 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b337fe640 10 monclient: dump: 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:13.303 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b3959a640 4 set_mon_vals no callback set 2024-09-10T16:06:13.304 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 10 monclient: _renew_subs 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.282+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3c13bdd0 con 0x7f1b3c068d00 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 4) ==== 50553+0+0 (secure 0 0 0) 0x7f1b2401c460 con 0x7f1b3c068d00 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b40952640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b40952640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b40952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3c138250 con 0x7f1b3c068d00 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b337fe640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 0x7f1b1c040640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(11..11 src has 1..11) ==== 2175+0+0 (secure 0 0 0) 0x7f1b24052100 con 0x7f1b3c068d00 2024-09-10T16:06:13.305 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b3a59c640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 0x7f1b1c040640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:13.306 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.283+0000 7f1b3a59c640 10 monclient: get_auth_request con 0x7f1b1c03e180 auth_method 0 2024-09-10T16:06:13.306 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.286+0000 7f1b3a59c640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 0x7f1b1c040640 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c069820 tx=0x7f1b2c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:13.306 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.288+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1b2401ac60 con 0x7f1b3c068d00 2024-09-10T16:06:13.306 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.288+0000 7f1b337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:13.306 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.288+0000 7f1b337fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:13.382 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:06:13.381+0000 7f1d87154640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T16:06:13.387 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:06:13.385+0000 7f1d82f7b640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:13.478 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.477+0000 7f1b40952640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:13.479 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.477+0000 7f1b40952640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:13.479 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.477+0000 7f1b40952640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:13.479 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.477+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1b3c11e040 con 0x7f1b3c068d00 2024-09-10T16:06:13.479 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.478+0000 7f1b337fe640 1 -- 172.21.15.164:0/320497381 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7596 (secure 0 0 0) 0x7f1b3c11e040 con 0x7f1b3c068d00 2024-09-10T16:06:13.479 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.478+0000 7f1b337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:13.480 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.478+0000 7f1b337fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:13.480 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-10T16:06:13.480 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":12,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:13.365261+0000","last_up_change":"2024-09-10T16:06:13.365261+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 msgr2=0x7f1b1c040640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 0x7f1b1c040640 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c069820 tx=0x7f1b2c02e000 comp rx=0 tx=0).stop 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 10 monclient: shutdown 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 msgr2=0x7f1b3c134d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1b2400c430 tx=0x7f1b2400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.482 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 shutdown_connections 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b1c03e180 0x7f1b1c040640 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b3c13d270 0x7f1b3c13a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b3c068d00 0x7f1b3c134d60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 --2- 172.21.15.164:0/320497381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b3c068920 0x7f1b3c1cd030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 >> 172.21.15.164:0/320497381 conn(0x7f1b3c0a3cd0 msgr2=0x7f1b3c136b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 shutdown_connections 2024-09-10T16:06:13.483 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:13.480+0000 7f1b40952640 1 -- 172.21.15.164:0/320497381 wait complete. 2024-09-10T16:06:13.492 DEBUG:teuthology.misc:4 of 6 OSDs are up 2024-09-10T16:06:13.953 INFO:tasks.ceph.osd.1.smithi164.stderr:2024-09-10T16:06:13.951+0000 7f0636b92640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T16:06:13.962 INFO:tasks.ceph.osd.1.smithi164.stderr:2024-09-10T16:06:13.961+0000 7f062e9b1640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T16:06:20.098 DEBUG:teuthology.orchestra.run.smithi164:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:20.198 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.197+0000 7f7e5635d640 10 monclient: get_monmap_and_config 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.197+0000 7f7e5635d640 10 monclient: build_initial_monmap 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.197+0000 7f7e5635d640 10 monclient: monmap: 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:06:20.198314+0000 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:06:20.198314+0000 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.199 INFO:teuthology.misc.health.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:20.200 INFO:teuthology.misc.health.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:20.200 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:20.200 INFO:teuthology.misc.health.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:20.200 INFO:teuthology.misc.health.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:20.200 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 Processor -- start 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 -- start start 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient: init 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e50131740 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e500a2e70 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7e5013e620 0x7f7e50140a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e5013e620 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:20.211 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e50132230 con 0x7f7e500a2e70 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e500a32a0 con 0x7f7e50131740 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7e500a1bf0 con 0x7f7e5013e620 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.198+0000 7f7e5635d640 10 monclient(hunting): authenticate will time out at 758.685893s 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e4f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7e500a2e70 0x7f7e5013cf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55884/0 (socket says 172.21.15.164:55884) 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e4f7fe640 1 -- 172.21.15.164:0/2200923411 learned_addr learned my addr 172.21.15.164:0/2200923411 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 1 --2- 172.21.15.164:0/2200923411 >> v2:172.21.15.164:3300/0 conn(0x7f7e5013e620 0x7f7e50140a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): get_auth_request con 0x7f7e5013e620 auth_method 0 2024-09-10T16:06:20.212 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 1 -- 172.21.15.164:0/2200923411 >> v1:172.21.15.165:6789/0 conn(0x7f7e50131740 legacy=0x7f7e50131b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 1 -- 172.21.15.164:0/2200923411 >> v1:172.21.15.164:6789/0 conn(0x7f7e500a2e70 legacy=0x7f7e5013cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient: _finish_hunting 0 2024-09-10T16:06:20.213 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 1 monclient: found mon.noname-a 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.199+0000 7f7e548fa640 1 -- 172.21.15.164:0/2200923411 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e500a1ed0 con 0x7f7e5013e620 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e548fa640 1 -- 172.21.15.164:0/2200923411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e5013e620 msgr2=0x7f7e50140a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e548fa640 1 --2- 172.21.15.164:0/2200923411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e5013e620 0x7f7e50140a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75d6efee8cea278f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 1 -- 172.21.15.164:0/2200923411 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48007a30 con 0x7f7e5013e620 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: dump: 2024-09-10T16:06:20.214 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.215 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: _finish_auth 0 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 5 monclient: authenticate success, global_id 4194 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 1 -- 172.21.15.164:0/2200923411 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e48007bd0 con 0x7f7e5013e620 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 1 -- 172.21.15.164:0/2200923411 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48031400 con 0x7f7e5013e620 2024-09-10T16:06:20.216 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 4 set_mon_vals no callback set 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e4effd640 10 monclient: dump: 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.217 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 10 monclient: shutdown 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 -- 172.21.15.164:0/2200923411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e5013e620 msgr2=0x7f7e50140a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 --2- 172.21.15.164:0/2200923411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e5013e620 0x7f7e50140a10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7e48009d90 tx=0x7f7e4802a490 comp rx=0 tx=0).stop 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 -- 172.21.15.164:0/2200923411 shutdown_connections 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 --2- 172.21.15.164:0/2200923411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e5013e620 0x7f7e50140a10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.218 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 -- 172.21.15.164:0/2200923411 >> 172.21.15.164:0/2200923411 conn(0x7f7e500a3d90 msgr2=0x7f7e50068920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 -- 172.21.15.164:0/2200923411 shutdown_connections 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.202+0000 7f7e5635d640 1 -- 172.21.15.164:0/2200923411 wait complete. 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.203+0000 7f7e5635d640 10 monclient: build_initial_monmap 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.203+0000 7f7e5635d640 10 monclient: monmap: 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:epoch 0 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:created 0.000000 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.219 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.203+0000 7f7e5635d640 1 Processor -- start 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 -- start start 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient: init 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e500a2e70 0x7f7e501ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-b con 0x7f7e500a2e70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:20.220 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-a con 0x7f7e50131740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 0x7f7e5013ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): picked mon.noname-c con 0x7f7e5013e620 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7e500a4410 con 0x7f7e50131740 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7e50069030 con 0x7f7e500a2e70 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e500a4590 con 0x7f7e5013e620 2024-09-10T16:06:20.221 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e5635d640 10 monclient(hunting): authenticate will time out at 758.691619s 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35008/0 (socket says 172.21.15.164:35008) 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 1 -- 172.21.15.164:0/3353541842 learned_addr learned my addr 172.21.15.164:0/3353541842 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): get_auth_request con 0x7f7e50131740 auth_method 0 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 0x7f7e5013ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): get_auth_request con 0x7f7e5013e620 auth_method 0 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.222 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e548fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.204+0000 7f7e4ffff640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e500a2e70 0x7f7e501ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e4ffff640 10 monclient(hunting): get_auth_request con 0x7f7e500a2e70 auth_method 0 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.223 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 10 monclient(hunting): handle_auth_done global_id 4202 payload 824 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 -- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e500a2e70 msgr2=0x7f7e501ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e500a2e70 0x7f7e501ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 -- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 msgr2=0x7f7e5013d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 10 monclient: _finish_hunting 0 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 monclient: found mon.noname-c 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.224 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e548fa640 1 -- 172.21.15.164:0/3353541842 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e50138ff0 con 0x7f7e5013e620 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.205+0000 7f7e4f7fe640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e548fa640 1 -- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 msgr2=0x7f7e5013ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e548fa640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 0x7f7e5013ddb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7e48031000 tx=0x7f7e48004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e48040070 con 0x7f7e5013e620 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: dump: 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.225 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:20.226 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e48045680 con 0x7f7e5013e620 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 5 monclient: authenticate success, global_id 4202 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 10 monclient: _renew_subs 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e5013e2f0 con 0x7f7e5013e620 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e480314c0 con 0x7f7e5013e620 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4cff9640 10 monclient: dump: 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:epoch 1 2024-09-10T16:06:20.227 INFO:teuthology.misc.health.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr: 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 10 monclient: _renew_subs 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.228 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e4dffb640 4 set_mon_vals no callback set 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.207+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e501cd510 con 0x7f7e5013e620 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7e48039040 con 0x7f7e5013e620 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e4cff9640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 0x7f7e3c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7f7e48074410 con 0x7f7e5013e620 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e5635d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e5635d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e5635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e50131b20 con 0x7f7e5013e620 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e4ffff640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 0x7f7e3c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.229 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.208+0000 7f7e4ffff640 10 monclient: get_auth_request con 0x7f7e3c03e210 auth_method 0 2024-09-10T16:06:20.230 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.211+0000 7f7e4ffff640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 0x7f7e3c0406d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004ea0 tx=0x7f7e40009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.230 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.213+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7e4803aca0 con 0x7f7e5013e620 2024-09-10T16:06:20.230 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.213+0000 7f7e4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.230 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.213+0000 7f7e4cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:20.393 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.391+0000 7f7e5635d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:20.393 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.391+0000 7f7e5635d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:20.393 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.391+0000 7f7e5635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.393 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.391+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7e5006cdd0 con 0x7f7e5013e620 2024-09-10T16:06:20.394 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.392+0000 7f7e4cff9640 1 -- 172.21.15.164:0/3353541842 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+10533 (secure 0 0 0) 0x7f7e48074180 con 0x7f7e5013e620 2024-09-10T16:06:20.394 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.392+0000 7f7e4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:20.394 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.392+0000 7f7e4cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:20.394 INFO:teuthology.misc.health.smithi164.stdout: 2024-09-10T16:06:20.394 INFO:teuthology.misc.health.smithi164.stdout:{"epoch":15,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:16.396165+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 msgr2=0x7f7e3c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 0x7f7e3c0406d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f7e40004ea0 tx=0x7f7e40009040 comp rx=0 tx=0).stop 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 10 monclient: shutdown 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 msgr2=0x7f7e5013ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 0x7f7e5013ddb0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f7e48031000 tx=0x7f7e48004a40 comp rx=0 tx=0).stop 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 shutdown_connections 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e3c03e210 0x7f7e3c0406d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e5013e620 0x7f7e5013ddb0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.396 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e50131740 0x7f7e5013d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.397 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 --2- 172.21.15.164:0/3353541842 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e500a2e70 0x7f7e501ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.397 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 >> 172.21.15.164:0/3353541842 conn(0x7f7e500a3d90 msgr2=0x7f7e500a4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:20.397 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 shutdown_connections 2024-09-10T16:06:20.397 INFO:teuthology.misc.health.smithi164.stderr:2024-09-10T16:06:20.394+0000 7f7e5635d640 1 -- 172.21.15.164:0/3353541842 wait complete. 2024-09-10T16:06:20.405 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-10T16:06:20.406 INFO:tasks.ceph:Creating RBD pool 2024-09-10T16:06:20.406 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-10T16:06:20.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 10 monclient: get_monmap_and_config 2024-09-10T16:06:20.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 10 monclient: build_initial_monmap 2024-09-10T16:06:20.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 10 monclient: monmap: 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:20.503092+0000 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:20.503092+0000 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:20.504 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 1 Processor -- start 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 1 -- start start 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.502+0000 7f834cb56640 10 monclient: init 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 0x7f8348134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-b con 0x7f834806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:20.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83480a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f83480a4a80 0x7f83481383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-c con 0x7f83480a4a80 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f834809ce50 con 0x7f83480a46a0 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f834806db60 con 0x7f83480a4a80 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f83480a4f10 con 0x7f834806cb70 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834cb56640 10 monclient(hunting): authenticate will time out at 758.990560s 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 0x7f8348134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8345d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f83480a46a0 0x7f8348137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55886/0 (socket says 172.21.15.164:55886) 2024-09-10T16:06:20.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8345d9b640 1 -- 172.21.15.164:0/1398252280 learned_addr learned my addr 172.21.15.164:0/1398252280 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): get_auth_request con 0x7f834806cb70 auth_method 0 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8346d9d640 1 --2- 172.21.15.164:0/1398252280 >> v2:172.21.15.165:3300/0 conn(0x7f83480a4a80 0x7f83481383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8346d9d640 10 monclient(hunting): get_auth_request con 0x7f83480a4a80 auth_method 0 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8346d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8346d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f8346d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient(hunting): handle_auth_done global_id 4205 payload 274 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 -- 172.21.15.164:0/1398252280 >> v2:172.21.15.165:3300/0 conn(0x7f83480a4a80 msgr2=0x7f83481383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 --2- 172.21.15.164:0/1398252280 >> v2:172.21.15.165:3300/0 conn(0x7f83480a4a80 0x7f83481383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 -- 172.21.15.164:0/1398252280 >> v1:172.21.15.164:6789/0 conn(0x7f83480a46a0 legacy=0x7f8348137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 monclient: found mon.noname-b 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:20.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.503+0000 7f834659c640 1 -- 172.21.15.164:0/1398252280 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f834809cb70 con 0x7f834806cb70 2024-09-10T16:06:20.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834659c640 1 -- 172.21.15.164:0/1398252280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 msgr2=0x7f8348134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:20.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834659c640 1 --2- 172.21.15.164:0/1398252280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 0x7f8348134f70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83400097a0 tx=0x7f8340029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41ac9f2dd15b959d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 1 -- 172.21.15.164:0/1398252280 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340037070 con 0x7f834806cb70 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: dump: 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.516 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: _finish_auth 0 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: _check_auth_tickets 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 1 -- 172.21.15.164:0/1398252280 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8340032440 con 0x7f834806cb70 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834cb56640 5 monclient: authenticate success, global_id 4205 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834cb56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834cb56640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 1 -- 172.21.15.164:0/1398252280 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8340032810 con 0x7f834806cb70 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834cb56640 4 set_mon_vals no callback set 2024-09-10T16:06:20.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.506+0000 7f834559a640 10 monclient: dump: 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 10 monclient: shutdown 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 -- 172.21.15.164:0/1398252280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 msgr2=0x7f8348134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 --2- 172.21.15.164:0/1398252280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 0x7f8348134f70 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83400097a0 tx=0x7f8340029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 -- 172.21.15.164:0/1398252280 shutdown_connections 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 --2- 172.21.15.164:0/1398252280 >> v2:172.21.15.165:3300/0 conn(0x7f83480a4a80 0x7f83481383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 --2- 172.21.15.164:0/1398252280 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f834806cb70 0x7f8348134f70 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 -- 172.21.15.164:0/1398252280 >> 172.21.15.164:0/1398252280 conn(0x7f83480aa900 msgr2=0x7f83480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 -- 172.21.15.164:0/1398252280 shutdown_connections 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.507+0000 7f834cb56640 1 -- 172.21.15.164:0/1398252280 wait complete. 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.508+0000 7f834cb56640 10 monclient: build_initial_monmap 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.508+0000 7f834cb56640 10 monclient: monmap: 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:20.519 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.508+0000 7f834cb56640 1 Processor -- start 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.508+0000 7f834cb56640 1 -- start start 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.508+0000 7f834cb56640 10 monclient: init 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:20.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f834806cb70 0x7f83481cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-b con 0x7f834806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-c con 0x7f83480a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 0x7f83481cdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): picked mon.noname-a con 0x7f83480a4a80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f83481388f0 con 0x7f83480a4a80 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f834806cf50 con 0x7f834806cb70 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:20.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8348138a70 con 0x7f83480a46a0 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834cb56640 10 monclient(hunting): authenticate will time out at 758.996082s 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 0x7f83481cdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56214/0 (socket says 172.21.15.164:56214) 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 1 -- 172.21.15.164:0/3686092616 learned_addr learned my addr 172.21.15.164:0/3686092616 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): get_auth_request con 0x7f83480a46a0 auth_method 0 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): get_auth_request con 0x7f83480a4a80 auth_method 0 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f834806cb70 0x7f83481cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): get_auth_request con 0x7f834806cb70 auth_method 0 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:20.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f834659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient(hunting): handle_auth_done global_id 4200 payload 824 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 -- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 msgr2=0x7f83481cd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 -- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f834806cb70 msgr2=0x7f83481cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f834806cb70 0x7f83481cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 monclient: found mon.noname-a 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8346d9d640 1 -- 172.21.15.164:0/3686092616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83480a3400 con 0x7f83480a4a80 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.509+0000 7f8345d9b640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:20.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8346d9d640 1 -- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 msgr2=0x7f83481cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8346d9d640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 0x7f83481cdcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f833800d530 tx=0x7f833800d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338012800 con 0x7f83480a4a80 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: dump: 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.525 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8338019070 con 0x7f83480a4a80 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f834cb56640 5 monclient: authenticate success, global_id 4200 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338012bc0 con 0x7f83480a4a80 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f834cb56640 10 monclient: _renew_subs 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f834cb56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83480a1ee0 con 0x7f83480a4a80 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:20.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8336ffd640 10 monclient: dump: 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.511+0000 7f8344d99640 4 set_mon_vals no callback set 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 10 monclient: _renew_subs 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83480a1380 con 0x7f83480a4a80 2024-09-10T16:06:20.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f833801e460 con 0x7f83480a4a80 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.512+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83480a2b70 con 0x7f83480a4a80 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.513+0000 7f8336ffd640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 0x7f8328040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.513+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(15..15 src has 1..15) ==== 3993+0+0 (secure 0 0 0) 0x7f8338052d50 con 0x7f83480a4a80 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.513+0000 7f834659c640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 0x7f8328040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.513+0000 7f834659c640 10 monclient: get_auth_request con 0x7f832803e240 auth_method 0 2024-09-10T16:06:20.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.515+0000 7f834659c640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 0x7f8328040700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f834806d1b0 tx=0x7f834002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:20.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.517+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8338051a60 con 0x7f83480a4a80 2024-09-10T16:06:20.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.517+0000 7f8336ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:20.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.517+0000 7f8336ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:20.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.691+0000 7f834cb56640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T16:06:20.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.692+0000 7f834cb56640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T16:06:20.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.692+0000 7f834cb56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:20.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:20.692+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7f8348134f70 con 0x7f83480a4a80 2024-09-10T16:06:21.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.412+0000 7f8336ffd640 1 -- 172.21.15.164:0/3686092616 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (secure 0 0 0) 0x7f8338012d60 con 0x7f83480a4a80 2024-09-10T16:06:21.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.412+0000 7f8336ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T16:06:21.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.412+0000 7f8336ffd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-10T16:06:21.413 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'rbd' created 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 msgr2=0x7f8328040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 0x7f8328040700 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f834806d1b0 tx=0x7f834002e000 comp rx=0 tx=0).stop 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 10 monclient: shutdown 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 msgr2=0x7f83481cdcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 0x7f83481cdcf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f833800d530 tx=0x7f833800d9f0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 shutdown_connections 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f832803e240 0x7f8328040700 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83480a4a80 0x7f83481cdcf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83480a46a0 0x7f83481cd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 --2- 172.21.15.164:0/3686092616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f834806cb70 0x7f83481cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 >> 172.21.15.164:0/3686092616 conn(0x7f83480aa900 msgr2=0x7f8348132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 shutdown_connections 2024-09-10T16:06:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.415+0000 7f834cb56640 1 -- 172.21.15.164:0/3686092616 wait complete. 2024-09-10T16:06:21.427 DEBUG:teuthology.orchestra.run.smithi164:> rbd --cluster ceph pool init rbd 2024-09-10T16:06:21.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.554+0000 7feeb759a140 10 monclient: get_monmap_and_config 2024-09-10T16:06:21.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.554+0000 7feeb759a140 10 monclient: build_initial_monmap 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.554+0000 7feeb759a140 10 monclient: monmap: 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:21.555053+0000 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:21.555053+0000 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:21.556 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:21.557 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.554+0000 7feeb759a140 1 Processor -- start 2024-09-10T16:06:21.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 -- start start 2024-09-10T16:06:21.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient: init 2024-09-10T16:06:21.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient: _reopen_session rank -1 2024-09-10T16:06:21.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 --2- >> v2:172.21.15.164:3300/0 conn(0x55df190f3540 0x55df190f0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-a con 0x55df190f3540 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 --2- >> v2:172.21.15.165:3300/0 conn(0x55df190f3920 0x55df190416a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-c con 0x55df190f3920 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-a-legacy con 0x55df19041be0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55df18edcd90 con 0x55df19041be0 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x55df18f2ca80 con 0x55df190f3540 2024-09-10T16:06:21.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x55df18f3d650 con 0x55df190f3920 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): _renew_subs 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb759a140 10 monclient(hunting): authenticate will time out at 760.042604s 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x55df190f3540 0x55df190f0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x55df190f3540 0x55df190f0f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35036/0 (socket says 172.21.15.164:35036) 2024-09-10T16:06:21.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb72e9640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x55df19041be0 0x55df190fa2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55890/0 (socket says 172.21.15.164:55890) 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 1 -- 172.21.15.164:0/1481777041 learned_addr learned my addr 172.21.15.164:0/1481777041 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): get_auth_request con 0x55df190f3540 auth_method 0 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:21.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb5886640 1 --2- 172.21.15.164:0/1481777041 >> v2:172.21.15.165:3300/0 conn(0x55df190f3920 0x55df190416a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb5886640 10 monclient(hunting): get_auth_request con 0x55df190f3920 auth_method 0 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb5886640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb5886640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.555+0000 7feeb5886640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 1 -- 172.21.15.164:0/1481777041 >> v2:172.21.15.165:3300/0 conn(0x55df190f3920 msgr2=0x55df190416a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:21.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 1 --2- 172.21.15.164:0/1481777041 >> v2:172.21.15.165:3300/0 conn(0x55df190f3920 0x55df190416a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 1 -- 172.21.15.164:0/1481777041 >> v1:172.21.15.164:6789/0 conn(0x55df19041be0 legacy=0x55df190fa2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 10 monclient: _finish_hunting 0 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 1 monclient: found mon.noname-a 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:21.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.556+0000 7feeb6087640 1 -- 172.21.15.164:0/1481777041 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df18ef4410 con 0x55df190f3540 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.559+0000 7feeb6087640 1 -- 172.21.15.164:0/1481777041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 msgr2=0x55df190f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.559+0000 7feeb6087640 1 --2- 172.21.15.164:0/1481777041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f0f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feea400a5a0 tx=0x7feea4045960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb030b6ddde7940 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 1 -- 172.21.15.164:0/1481777041 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea405a070 con 0x55df190f3540 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: dump: 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:21.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: _finish_auth 0 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: _check_auth_tickets 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 1 -- 172.21.15.164:0/1481777041 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea4055680 con 0x55df190f3540 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 5 monclient: authenticate success, global_id 4203 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:21.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 1 -- 172.21.15.164:0/1481777041 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4055a70 con 0x55df190f3540 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 10 monclient: get_monmap_and_config success 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 4 set_mon_vals no callback set 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb5085640 10 monclient: dump: 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:21.572 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:21.573 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 10 monclient: shutdown 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 -- 172.21.15.164:0/1481777041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 msgr2=0x55df190f0f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 --2- 172.21.15.164:0/1481777041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f0f50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feea400a5a0 tx=0x7feea4045960 comp rx=0 tx=0).stop 2024-09-10T16:06:21.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 -- 172.21.15.164:0/1481777041 shutdown_connections 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 --2- 172.21.15.164:0/1481777041 >> v2:172.21.15.165:3300/0 conn(0x55df190f3920 0x55df190416a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 --2- 172.21.15.164:0/1481777041 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f0f50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 -- 172.21.15.164:0/1481777041 >> 172.21.15.164:0/1481777041 conn(0x55df19042c70 msgr2=0x55df19043080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 -- 172.21.15.164:0/1481777041 shutdown_connections 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.560+0000 7feeb759a140 1 -- 172.21.15.164:0/1481777041 wait complete. 2024-09-10T16:06:21.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 10 monclient: build_initial_monmap 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 10 monclient: monmap: 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:21.576 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 1 Processor -- start 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 1 -- start start 2024-09-10T16:06:21.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 10 monclient: init 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.561+0000 7feeb759a140 10 monclient: _reopen_session rank -1 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 0x55df191023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-c con 0x55df19041be0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f00d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-a con 0x55df190f3540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df190f3920 0x55df190f0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): picked mon.noname-b con 0x55df190f3920 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x55df18ed2390 con 0x55df190f3540 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:21.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x55df18f30890 con 0x55df190f3920 2024-09-10T16:06:21.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:21.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x55df19054900 con 0x55df19041be0 2024-09-10T16:06:21.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): _renew_subs 2024-09-10T16:06:21.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb759a140 10 monclient(hunting): authenticate will time out at 760.049080s 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 0x55df191023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 0x55df191023c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56218/0 (socket says 172.21.15.164:56218) 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 learned_addr learned my addr 172.21.15.164:0/572682847 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): get_auth_request con 0x55df19041be0 auth_method 0 2024-09-10T16:06:21.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): get_auth_request con 0x55df190f3540 auth_method 0 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:21.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb72e9640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df190f3920 0x55df190f0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2024-09-10T16:06:21.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df190f3920 msgr2=0x55df190f0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df190f3920 0x55df190f0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 msgr2=0x55df190f00d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f00d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient: _finish_hunting 0 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 monclient: found mon.noname-c 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:21.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df18eab720 con 0x55df19041be0 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb72e9640 10 monclient: get_auth_request con 0x55df190f3920 auth_method 0 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb72e9640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x55df190f3920 0x55df190f0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.562+0000 7feeb5886640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x55df190f3540 0x55df190f00d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 msgr2=0x55df191023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:21.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb6087640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 0x55df191023c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feea4000c00 tx=0x7feea4002fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea405a050 con 0x55df19041be0 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: dump: 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:21.586 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:21.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb759a140 5 monclient: authenticate success, global_id 4211 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feea405b040 con 0x55df19041be0 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb759a140 10 monclient: _renew_subs 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feea4055c60 con 0x55df19041be0 2024-09-10T16:06:21.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb759a140 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x55df190deeb0 con 0x55df19041be0 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feea27fc640 10 monclient: dump: 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:21.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:21.590 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.564+0000 7feeb4884640 4 set_mon_vals no callback set 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.565+0000 7feeb759a140 10 monclient: _renew_subs 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.565+0000 7feeb759a140 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.565+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x55df190de760 con 0x55df19041be0 2024-09-10T16:06:21.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.565+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feea4068020 con 0x55df19041be0 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.565+0000 7feea27fc640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee9403cb00 0x7fee9403efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 4335+0+0 (secure 0 0 0) 0x7feea4094da0 con 0x55df19041be0 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb759a140 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb759a140 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb759a140 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55df190df2c0 con 0x55df19041be0 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb5886640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee9403cb00 0x7fee9403efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:21.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.566+0000 7feeb5886640 10 monclient: get_auth_request con 0x7fee9403cb00 auth_method 0 2024-09-10T16:06:21.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:21.568+0000 7feeb5886640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee9403cb00 0x7fee9403efc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x55df190de310 tx=0x7feeac00c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:22.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.416+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (secure 0 0 0) 0x7feea406f020 con 0x55df19041be0 2024-09-10T16:06:22.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.416+0000 7feea27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T16:06:22.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.416+0000 7feea27fc640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-10T16:06:22.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.416+0000 7feeb759a140 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:22.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.416+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7feea4055680 con 0x55df19041be0 2024-09-10T16:06:22.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7feea40549b0 con 0x55df19041be0 2024-09-10T16:06:22.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feea27fc640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-10T16:06:22.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feeb4884640 10 monclient: _renew_subs 2024-09-10T16:06:22.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feeb4884640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:22.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feeb4884640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=17}) -- 0x7fee98002ae0 con 0x55df19041be0 2024-09-10T16:06:22.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.422+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 8 ==== osd_map(17..17 src has 1..17) ==== 674+0+0 (secure 0 0 0) 0x7feea4055e00 con 0x55df19041be0 2024-09-10T16:06:22.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.423+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x55df19109430 0x55df1910b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:22.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.423+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55df1910bd60 con 0x55df19109430 2024-09-10T16:06:22.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.423+0000 7feeb72e9640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x55df19109430 0x55df1910b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:22.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.423+0000 7feeb72e9640 10 monclient: get_auth_request con 0x55df19109430 auth_method 0 2024-09-10T16:06:22.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.427+0000 7feeb72e9640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x55df19109430 0x55df1910b820 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:22.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.429+0000 7feeb72e9640 1 -- 172.21.15.164:0/572682847 <== osd.1 v2:172.21.15.164:6816/1609205011 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7feea8006040 con 0x55df19109430 2024-09-10T16:06:22.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.431+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x55df1915f440 0x55df191618d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:22.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.431+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55df19161e30 con 0x55df1915f440 2024-09-10T16:06:22.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.432+0000 7feeb6087640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x55df1915f440 0x55df191618d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:22.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.432+0000 7feeb6087640 10 monclient: get_auth_request con 0x55df1915f440 auth_method 0 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.436+0000 7feeb6087640 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x55df1915f440 0x55df191618d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.436+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 <== osd.0 v2:172.21.15.164:6800/3366597746 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7feea405d070 con 0x55df1915f440 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.436+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55df1915df20 con 0x55df1915f440 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.437+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 <== osd.0 v2:172.21.15.164:6800/3366597746 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7feea405d070 con 0x55df1915f440 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.437+0000 7feea3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:22.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:22.437+0000 7feea3fff640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7fee8c001f00 con 0x55df19041be0 2024-09-10T16:06:23.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.425+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 9 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (secure 0 0 0) 0x7feea4018c90 con 0x55df19041be0 2024-09-10T16:06:23.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.425+0000 7feea27fc640 10 monclient: _renew_subs 2024-09-10T16:06:23.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.425+0000 7feea27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:23.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.425+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=18}) -- 0x7fee94042d60 con 0x55df19041be0 2024-09-10T16:06:23.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.426+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 10 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (secure 0 0 0) 0x7feea4085020 con 0x55df19041be0 2024-09-10T16:06:23.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.426+0000 7feeb4884640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7fee98005620 con 0x55df1915f440 2024-09-10T16:06:23.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.429+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 <== osd.0 v2:172.21.15.164:6800/3366597746 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7feea405d070 con 0x55df1915f440 2024-09-10T16:06:23.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.429+0000 7feea3fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:23.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:23.429+0000 7feea3fff640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7fee8c003350 con 0x55df19041be0 2024-09-10T16:06:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.427+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 11 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (secure 0 0 0) 0x7feea401af10 con 0x55df19041be0 2024-09-10T16:06:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.427+0000 7feea27fc640 10 monclient: _renew_subs 2024-09-10T16:06:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.427+0000 7feea27fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.427+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=19}) -- 0x7fee94042e10 con 0x55df19041be0 2024-09-10T16:06:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.428+0000 7feea27fc640 1 -- 172.21.15.164:0/572682847 <== mon.2 v2:172.21.15.164:3301/0 12 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (secure 0 0 0) 0x7feea408e070 con 0x55df19041be0 2024-09-10T16:06:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.428+0000 7feeb4884640 1 -- 172.21.15.164:0/572682847 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7fee980059c0 con 0x55df1915f440 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb6087640 1 -- 172.21.15.164:0/572682847 <== osd.0 v2:172.21.15.164:6800/3366597746 4 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7feea405d070 con 0x55df1915f440 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x55df1915f440 msgr2=0x55df191618d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x55df1915f440 0x55df191618d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x55df19109430 msgr2=0x55df1910b820 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x55df19109430 0x55df1910b820 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee9403cb00 msgr2=0x7fee9403efc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee9403cb00 0x7fee9403efc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x55df190de310 tx=0x7feeac00c040 comp rx=0 tx=0).stop 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 10 monclient: shutdown 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 msgr2=0x55df191023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 --2- 172.21.15.164:0/572682847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x55df19041be0 0x55df191023c0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7feea4000c00 tx=0x7feea4002fb0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb72e9640 1 -- 172.21.15.164:0/572682847 reap_dead start 2024-09-10T16:06:24.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 shutdown_connections 2024-09-10T16:06:24.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 >> 172.21.15.164:0/572682847 conn(0x55df19042c70 msgr2=0x55df190f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:24.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 shutdown_connections 2024-09-10T16:06:24.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.433+0000 7feeb759a140 1 -- 172.21.15.164:0/572682847 wait complete. 2024-09-10T16:06:24.439 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-10T16:06:24.440 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-10T16:06:24.441 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.547+0000 7fc002532640 10 monclient: get_monmap_and_config 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.547+0000 7fc002532640 10 monclient: build_initial_monmap 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.547+0000 7fc002532640 10 monclient: monmap: 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.549 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:24.548630+0000 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:24.548630+0000 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:24.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.548+0000 7fc002532640 1 Processor -- start 2024-09-10T16:06:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.548+0000 7fc002532640 1 -- start start 2024-09-10T16:06:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.548+0000 7fc002532640 10 monclient: init 2024-09-10T16:06:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbffc06d490 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbffc06d8b0 0x7fbffc13fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): picked mon.noname-c con 0x7fbffc06d8b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbffc0a45b0 con 0x7fbffc06d490 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbffc06e3b0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc06e690 con 0x7fbffc06cb70 2024-09-10T16:06:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fc002532640 10 monclient(hunting): authenticate will time out at 763.036100s 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47014/0 (socket says 172.21.15.164:47014) 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 -- 172.21.15.164:0/4259497973 learned_addr learned my addr 172.21.15.164:0/4259497973 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): get_auth_request con 0x7fbffc06cb70 auth_method 0 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffb7fe640 1 --1- 172.21.15.164:0/4259497973 >> v1:172.21.15.164:6789/0 conn(0x7fbffc06d490 0x7fbffc13c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59228/0 (socket says 172.21.15.164:59228) 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 -- 172.21.15.164:0/4259497973 >> v2:172.21.15.165:3300/0 conn(0x7fbffc06d8b0 msgr2=0x7fbffc13fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 --2- 172.21.15.164:0/4259497973 >> v2:172.21.15.165:3300/0 conn(0x7fbffc06d8b0 0x7fbffc13fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 -- 172.21.15.164:0/4259497973 >> v1:172.21.15.164:6789/0 conn(0x7fbffc06d490 legacy=0x7fbffc13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 monclient: found mon.noname-b 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.549+0000 7fbffbfff640 1 -- 172.21.15.164:0/4259497973 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc0a47f0 con 0x7fbffc06cb70 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffbfff640 1 -- 172.21.15.164:0/4259497973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 msgr2=0x7fbffc06cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffbfff640 1 --2- 172.21.15.164:0/4259497973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbffc075300 tx=0x7fbfec02a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f6d84004035e28a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 1 -- 172.21.15.164:0/4259497973 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec037070 con 0x7fbffc06cb70 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: dump: 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 1 -- 172.21.15.164:0/4259497973 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfec032640 con 0x7fbffc06cb70 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fc002532640 5 monclient: authenticate success, global_id 4214 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fc002532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fc002532640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 1 -- 172.21.15.164:0/4259497973 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfec032a10 con 0x7fbffc06cb70 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fc002532640 4 set_mon_vals no callback set 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.552+0000 7fbffaffd640 10 monclient: dump: 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 10 monclient: shutdown 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 -- 172.21.15.164:0/4259497973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 msgr2=0x7fbffc06cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 --2- 172.21.15.164:0/4259497973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbffc075300 tx=0x7fbfec02a070 comp rx=0 tx=0).stop 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 -- 172.21.15.164:0/4259497973 shutdown_connections 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 --2- 172.21.15.164:0/4259497973 >> v2:172.21.15.165:3300/0 conn(0x7fbffc06d8b0 0x7fbffc13fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 --2- 172.21.15.164:0/4259497973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06cb70 0x7fbffc06cf50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 -- 172.21.15.164:0/4259497973 >> 172.21.15.164:0/4259497973 conn(0x7fbffc0a4ee0 msgr2=0x7fbffc0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 -- 172.21.15.164:0/4259497973 shutdown_connections 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 1 -- 172.21.15.164:0/4259497973 wait complete. 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 10 monclient: build_initial_monmap 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.553+0000 7fc002532640 10 monclient: monmap: 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 Processor -- start 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 -- start start 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient: init 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbffc06cb70 0x7fbffc09a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): picked mon.noname-b con 0x7fbffc06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): picked mon.noname-a con 0x7fbffc06d490 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 0x7fbffc094440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): picked mon.noname-c con 0x7fbffc06d8b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0a0310 con 0x7fbffc06d490 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc0a0190 con 0x7fbffc06cb70 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbffc09bd30 con 0x7fbffc06d8b0 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc002532640 10 monclient(hunting): authenticate will time out at 763.041876s 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fbffb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.554+0000 7fc000acf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 0x7fbffc094440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35818/0 (socket says 172.21.15.164:35818) 2024-09-10T16:06:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 1 -- 172.21.15.164:0/1501695445 learned_addr learned my addr 172.21.15.164:0/1501695445 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): get_auth_request con 0x7fbffc06d490 auth_method 0 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): get_auth_request con 0x7fbffc06d8b0 auth_method 0 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbffc06cb70 0x7fbffc09a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): get_auth_request con 0x7fbffc06cb70 auth_method 0 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient(hunting): handle_auth_done global_id 4217 payload 824 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 -- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbffc06cb70 msgr2=0x7fbffc09a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbffc06cb70 0x7fbffc09a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 -- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 msgr2=0x7fbffc093f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient: _finish_hunting 0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 monclient: found mon.noname-c 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fc000acf640 1 -- 172.21.15.164:0/1501695445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc097d20 con 0x7fbffc06d8b0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.555+0000 7fbffb7fe640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc000acf640 1 -- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 msgr2=0x7fbffc094440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc000acf640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 0x7fbffc094440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbff400be20 tx=0x7fbff400bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4019070 con 0x7fbffc06d8b0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: dump: 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 5 monclient: authenticate success, global_id 4217 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 10 monclient: _renew_subs 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbff4014660 con 0x7fbffc06d8b0 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc09f5d0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4014a50 con 0x7fbffc06d8b0 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbff8ff9640 10 monclient: dump: 2024-09-10T16:06:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 10 monclient: _renew_subs 2024-09-10T16:06:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc0959c0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.557+0000 7fbffa7fc640 4 set_mon_vals no callback set 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fc002532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbff401c070 con 0x7fbffc06d8b0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fc002532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fc002532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc0a92f0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fbff8ff9640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 0x7fbfe8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.558+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fbff4052fd0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.559+0000 7fbffbfff640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 0x7fbfe8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.559+0000 7fbffbfff640 10 monclient: get_auth_request con 0x7fbfe803e240 auth_method 0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.561+0000 7fbffbfff640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 0x7fbfe8040700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbffc09c190 tx=0x7fbfec02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.563+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbffc0a92f0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.563+0000 7fbff8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.563+0000 7fbff8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.736+0000 7fc002532640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T16:06:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.736+0000 7fc002532640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T16:06:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.736+0000 7fc002532640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.736+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fbffc06cfd0 con 0x7fbffc06d8b0 2024-09-10T16:06:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.737+0000 7fbff8ff9640 1 -- 172.21.15.164:0/1501695445 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (secure 0 0 0) 0x7fbff4051b40 con 0x7fbffc06d8b0 2024-09-10T16:06:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.737+0000 7fbff8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T16:06:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.737+0000 7fbff8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:24.739 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 msgr2=0x7fbfe8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 0x7fbfe8040700 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fbffc09c190 tx=0x7fbfec02e000 comp rx=0 tx=0).stop 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 10 monclient: shutdown 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 msgr2=0x7fbffc094440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 0x7fbffc094440 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbff400be20 tx=0x7fbff400bf10 comp rx=0 tx=0).stop 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 shutdown_connections 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfe803e240 0x7fbfe8040700 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbffc06d8b0 0x7fbffc094440 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbffc06d490 0x7fbffc093f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 --2- 172.21.15.164:0/1501695445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbffc06cb70 0x7fbffc09a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 >> 172.21.15.164:0/1501695445 conn(0x7fbffc0a4ee0 msgr2=0x7fbffc0a1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 shutdown_connections 2024-09-10T16:06:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.739+0000 7fc002532640 1 -- 172.21.15.164:0/1501695445 wait complete. 2024-09-10T16:06:24.751 INFO:teuthology.orchestra.run.smithi164.stdout:[{"version":5,"timestamp":"2024-09-10T16:06:13.953226+0000","name":"","changes":[{"name":"osd.1/osd_mclock_max_capacity_iops_ssd","new_value":"9109.235465"}]}] 2024-09-10T16:06:24.751 INFO:tasks.ceph_manager:config epoch is 5 2024-09-10T16:06:24.751 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T16:06:24.751 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-10T16:06:24.751 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.848+0000 7f4c064c6640 10 monclient: get_monmap_and_config 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.848+0000 7f4c064c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.848+0000 7f4c064c6640 10 monclient: monmap: 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:24.849741+0000 2024-09-10T16:06:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:24.849741+0000 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:24.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.849+0000 7f4c064c6640 1 Processor -- start 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.849+0000 7f4c064c6640 1 -- start start 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.849+0000 7f4c064c6640 10 monclient: init 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c000a4540 0x7f4c000a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a4540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:24.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c001385c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-b con 0x7f4c000a60b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c000a71c0 con 0x7f4c001385c0 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4c0009c9f0 con 0x7f4c000a4540 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4c0009ccd0 con 0x7f4c000a60b0 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c064c6640 10 monclient(hunting): authenticate will time out at 763.337483s 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47026/0 (socket says 172.21.15.164:47026) 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 -- 172.21.15.164:0/4217269650 learned_addr learned my addr 172.21.15.164:0/4217269650 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:24.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): get_auth_request con 0x7f4c000a60b0 auth_method 0 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4bfffff640 1 --2- 172.21.15.164:0/4217269650 >> v2:172.21.15.165:3300/0 conn(0x7f4c000a4540 0x7f4c000a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4bfffff640 10 monclient(hunting): get_auth_request con 0x7f4c000a4540 auth_method 0 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4bfffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient(hunting): handle_auth_done global_id 4220 payload 274 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 -- 172.21.15.164:0/4217269650 >> v2:172.21.15.165:3300/0 conn(0x7f4c000a4540 msgr2=0x7f4c000a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 --2- 172.21.15.164:0/4217269650 >> v2:172.21.15.165:3300/0 conn(0x7f4c000a4540 0x7f4c000a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 -- 172.21.15.164:0/4217269650 >> v1:172.21.15.165:6789/0 conn(0x7f4c001385c0 legacy=0x7f4c001345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient: _finish_hunting 0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 monclient: found mon.noname-b 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.850+0000 7f4c04a63640 1 -- 172.21.15.164:0/4217269650 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c0009cfb0 con 0x7f4c000a60b0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c04a63640 1 -- 172.21.15.164:0/4217269650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 msgr2=0x7f4c00134cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c04a63640 1 --2- 172.21.15.164:0/4217269650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009e80 tx=0x7f4bf802a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5465ee010e42ac75 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 1 -- 172.21.15.164:0/4217269650 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8036070 con 0x7f4c000a60b0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: dump: 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:24.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c064c6640 5 monclient: authenticate success, global_id 4220 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 1 -- 172.21.15.164:0/4217269650 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf8007800 con 0x7f4c000a60b0 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c064c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c064c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 1 -- 172.21.15.164:0/4217269650 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf8007bd0 con 0x7f4c000a60b0 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4c064c6640 4 set_mon_vals no callback set 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.853+0000 7f4bff7fe640 10 monclient: dump: 2024-09-10T16:06:24.866 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 10 monclient: shutdown 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 -- 172.21.15.164:0/4217269650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 msgr2=0x7f4c00134cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 --2- 172.21.15.164:0/4217269650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f4bf8009e80 tx=0x7f4bf802a580 comp rx=0 tx=0).stop 2024-09-10T16:06:24.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 -- 172.21.15.164:0/4217269650 shutdown_connections 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 --2- 172.21.15.164:0/4217269650 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a60b0 0x7f4c00134cf0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 --2- 172.21.15.164:0/4217269650 >> v2:172.21.15.165:3300/0 conn(0x7f4c000a4540 0x7f4c000a4920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 -- 172.21.15.164:0/4217269650 >> 172.21.15.164:0/4217269650 conn(0x7f4c000aaa00 msgr2=0x7f4c000aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 -- 172.21.15.164:0/4217269650 shutdown_connections 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 1 -- 172.21.15.164:0/4217269650 wait complete. 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.854+0000 7f4c064c6640 10 monclient: monmap: 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:24.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 Processor -- start 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 -- start start 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient: init 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-c con 0x7f4c000a4540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:24.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000a60b0 0x7f4c0012eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-a con 0x7f4c000a60b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 0x7f4c001cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): picked mon.noname-b con 0x7f4c001cd110 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00137f50 con 0x7f4c000a60b0 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4c00137dd0 con 0x7f4c001cd110 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4c00135230 con 0x7f4c000a4540 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:24.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4c064c6640 10 monclient(hunting): authenticate will time out at 763.342815s 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47032/0 (socket says 172.21.15.164:47032) 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 1 -- 172.21.15.164:0/1394068485 learned_addr learned my addr 172.21.15.164:0/1394068485 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 10 monclient(hunting): get_auth_request con 0x7f4c000a4540 auth_method 0 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bf7fff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000a60b0 0x7f4c0012eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bf7fff640 10 monclient(hunting): get_auth_request con 0x7f4c000a60b0 auth_method 0 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bf7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bf7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.855+0000 7f4bf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:24.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bf7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bf7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bf7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4c04a63640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 0x7f4c001cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient(hunting): handle_auth_done global_id 4223 payload 824 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 -- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 msgr2=0x7f4c001cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 0x7f4c001cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 -- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000a60b0 msgr2=0x7f4c0012eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000a60b0 0x7f4c0012eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:24.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 monclient: found mon.noname-c 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4bfffff640 1 -- 172.21.15.164:0/1394068485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c000a31f0 con 0x7f4c000a4540 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4c04a63640 10 monclient: get_auth_request con 0x7f4c001cd110 auth_method 0 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.856+0000 7f4c04a63640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 0x7f4c001cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfffff640 1 -- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 msgr2=0x7f4c0012e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfffff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000d830 tx=0x7f4bf000dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf000b8d0 con 0x7f4c000a4540 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: dump: 2024-09-10T16:06:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 5 monclient: authenticate success, global_id 4223 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4bf0019070 con 0x7f4c000a4540 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 10 monclient: _renew_subs 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bf000bc90 con 0x7f4c000a4540 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c000a34d0 con 0x7f4c000a4540 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfeffd640 4 set_mon_vals no callback set 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4bfd7fa640 10 monclient: dump: 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:24.875 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 10 monclient: _renew_subs 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.858+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c000aa530 con 0x7f4c000a4540 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4bf001e460 con 0x7f4c000a4540 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4c064c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4c064c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4c064c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4bfd7fa640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 0x7f4be00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c00050ee0 con 0x7f4c000a4540 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.859+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f4bf00530b0 con 0x7f4c000a4540 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.860+0000 7f4bf7fff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 0x7f4be00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.860+0000 7f4bf7fff640 10 monclient: get_auth_request con 0x7f4be003e210 auth_method 0 2024-09-10T16:06:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.862+0000 7f4bf7fff640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 0x7f4be00406d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c000a1df0 tx=0x7f4be8007e50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:24.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.864+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4c00050ee0 con 0x7f4c000a4540 2024-09-10T16:06:24.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.864+0000 7f4bfd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:24.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:24.864+0000 7f4bfd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:25.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.083+0000 7f4c064c6640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-10T16:06:25.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.083+0000 7f4c064c6640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-10T16:06:25.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.083+0000 7f4c064c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.083+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f4c0006e7b0 con 0x7f4c000a4540 2024-09-10T16:06:25.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.085+0000 7f4bfd7fa640 1 -- 172.21.15.164:0/1394068485 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+97345 (secure 0 0 0) 0x7f4bf0015030 con 0x7f4c000a4540 2024-09-10T16:06:25.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.085+0000 7f4bfd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-10T16:06:25.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.085+0000 7f4bfd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:25.088 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:25.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 msgr2=0x7f4be00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 0x7f4be00406d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4c000a1df0 tx=0x7f4be8007e50 comp rx=0 tx=0).stop 2024-09-10T16:06:25.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 10 monclient: shutdown 2024-09-10T16:06:25.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 msgr2=0x7f4c0012e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4bf000d830 tx=0x7f4bf000dcf0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 shutdown_connections 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4be003e210 0x7f4be00406d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c001cd110 0x7f4c001cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c000a60b0 0x7f4c0012eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 --2- 172.21.15.164:0/1394068485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c000a4540 0x7f4c0012e570 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 >> 172.21.15.164:0/1394068485 conn(0x7f4c000aaa00 msgr2=0x7f4c000a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 shutdown_connections 2024-09-10T16:06:25.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.087+0000 7f4c064c6640 1 -- 172.21.15.164:0/1394068485 wait complete. 2024-09-10T16:06:25.099 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":5,"flags":0,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6824","nonce":3840871261},{"type":"v1","addr":"172.21.15.165:6825","nonce":3840871261}]},"active_addr":"172.21.15.165:6825/3840871261","active_change":"2024-09-10T16:06:07.345574+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":2100351171}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1493597318}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":1031791453}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.165:0","nonce":3374211574}]}]} 2024-09-10T16:06:25.102 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-10T16:06:25.102 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-10T16:06:25.102 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.198+0000 7fdcadb5d640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.198+0000 7fdcadb5d640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.198+0000 7fdcadb5d640 10 monclient: monmap: 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.199545+0000 2024-09-10T16:06:25.200 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.199545+0000 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.201 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 Processor -- start 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 -- start start 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient: init 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdca81348c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 0x7fdca8135100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdca8134ca0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdca8137ba0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdca809ccd0 con 0x7fdca8137ba0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdca809cfb0 con 0x7fdca8134ca0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdca8134620 con 0x7fdca81348c0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.199+0000 7fdcadb5d640 10 monclient(hunting): authenticate will time out at 763.686988s 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47036/0 (socket says 172.21.15.164:47036) 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 0x7fdca8135100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 -- 172.21.15.164:0/3773260640 learned_addr learned my addr 172.21.15.164:0/3773260640 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): get_auth_request con 0x7fdca81348c0 auth_method 0 2024-09-10T16:06:25.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): get_auth_request con 0x7fdca8134ca0 auth_method 0 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient(hunting): handle_auth_done global_id 4226 payload 274 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 -- 172.21.15.164:0/3773260640 >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 msgr2=0x7fdca8135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 --2- 172.21.15.164:0/3773260640 >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 0x7fdca8135100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 -- 172.21.15.164:0/3773260640 >> v1:172.21.15.164:6789/0 conn(0x7fdca8137ba0 legacy=0x7fdca8137ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 monclient: found mon.noname-b 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdca77fe640 1 -- 172.21.15.164:0/3773260640 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca80a3e30 con 0x7fdca81348c0 2024-09-10T16:06:25.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.200+0000 7fdc9effd640 1 --2- 172.21.15.164:0/3773260640 >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 0x7fdca8135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca77fe640 1 -- 172.21.15.164:0/3773260640 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 msgr2=0x7fdca8073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca77fe640 1 --2- 172.21.15.164:0/3773260640 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009870 tx=0x7fdc9802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6ed8d6ef00d77ea server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 1 -- 172.21.15.164:0/3773260640 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98037070 con 0x7fdca81348c0 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: dump: 2024-09-10T16:06:25.213 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 1 -- 172.21.15.164:0/3773260640 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc98032440 con 0x7fdca81348c0 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 5 monclient: authenticate success, global_id 4226 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 1 -- 172.21.15.164:0/3773260640 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98032810 con 0x7fdca81348c0 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 4 set_mon_vals no callback set 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdca6ffd640 10 monclient: dump: 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 10 monclient: shutdown 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 -- 172.21.15.164:0/3773260640 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 msgr2=0x7fdca8073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/3773260640 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdc98009870 tx=0x7fdc9802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 -- 172.21.15.164:0/3773260640 shutdown_connections 2024-09-10T16:06:25.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/3773260640 >> v2:172.21.15.164:3300/0 conn(0x7fdca8134ca0 0x7fdca8135100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/3773260640 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca8073780 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 -- 172.21.15.164:0/3773260640 >> 172.21.15.164:0/3773260640 conn(0x7fdca80a6f80 msgr2=0x7fdca80a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.203+0000 7fdcadb5d640 1 -- 172.21.15.164:0/3773260640 shutdown_connections 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.204+0000 7fdcadb5d640 1 -- 172.21.15.164:0/3773260640 wait complete. 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.204+0000 7fdcadb5d640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.204+0000 7fdcadb5d640 10 monclient: monmap: 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.204+0000 7fdcadb5d640 1 Processor -- start 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 -- start start 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient: init 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-c con 0x7fdca81348c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdca8134ca0 0x7fdca8131620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-b con 0x7fdca8134ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 0x7fdca8131b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): picked mon.noname-a con 0x7fdca8137ba0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdca8138880 con 0x7fdca8137ba0 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdca8138700 con 0x7fdca8134ca0 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdca8138a00 con 0x7fdca81348c0 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdcadb5d640 10 monclient(hunting): authenticate will time out at 763.692612s 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 0x7fdca8131b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47042/0 (socket says 172.21.15.164:47042) 2024-09-10T16:06:25.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 0x7fdca8131b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35858/0 (socket says 172.21.15.164:35858) 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 -- 172.21.15.164:0/2842838915 learned_addr learned my addr 172.21.15.164:0/2842838915 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): get_auth_request con 0x7fdca81348c0 auth_method 0 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): get_auth_request con 0x7fdca8137ba0 auth_method 0 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 -- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdca8134ca0 msgr2=0x7fdca8131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdca8134ca0 0x7fdca8131620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 -- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 msgr2=0x7fdca8131b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 0x7fdca8131b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.205+0000 7fdca77fe640 1 -- 172.21.15.164:0/2842838915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdca813c3f0 con 0x7fdca81348c0 2024-09-10T16:06:25.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca77fe640 1 -- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 msgr2=0x7fdca81310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca77fe640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdc980090b0 tx=0x7fdc980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98041020 con 0x7fdca81348c0 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: dump: 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.222 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 5 monclient: authenticate success, global_id 4229 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 10 monclient: _renew_subs 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdc9803d630 con 0x7fdca81348c0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdca813e750 con 0x7fdca81348c0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc98037040 con 0x7fdca81348c0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca4ff9640 10 monclient: dump: 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 10 monclient: _renew_subs 2024-09-10T16:06:25.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdca67fc640 4 set_mon_vals no callback set 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.208+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdca813eee0 con 0x7fdca81348c0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdc98044020 con 0x7fdca81348c0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdca4ff9640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 0x7fdc88040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fdc98072f00 con 0x7fdca81348c0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdcadb5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdcadb5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdcadb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdca8074d20 con 0x7fdca81348c0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdc9effd640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 0x7fdc88040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.209+0000 7fdc9effd640 10 monclient: get_auth_request con 0x7fdc8803e240 auth_method 0 2024-09-10T16:06:25.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.212+0000 7fdc9effd640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 0x7fdc88040700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdc90006e80 tx=0x7fdc90006dd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.214+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdc9804b030 con 0x7fdca81348c0 2024-09-10T16:06:25.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.214+0000 7fdca4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.214+0000 7fdca4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:25.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.389+0000 7fdcadb5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.389+0000 7fdcadb5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.389+0000 7fdcadb5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.389+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdca8073780 con 0x7fdca81348c0 2024-09-10T16:06:25.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.390+0000 7fdca4ff9640 1 -- 172.21.15.164:0/2842838915 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12439 (secure 0 0 0) 0x7fdc9803b3f0 con 0x7fdca81348c0 2024-09-10T16:06:25.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.390+0000 7fdca4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.390+0000 7fdca4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:25.392 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:25.392 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":19,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:24.423222+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T16:06:20.696007+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 msgr2=0x7fdc88040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 0x7fdc88040700 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdc90006e80 tx=0x7fdc90006dd0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 10 monclient: shutdown 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 msgr2=0x7fdca81310e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdc980090b0 tx=0x7fdc980046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 shutdown_connections 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdc8803e240 0x7fdc88040700 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdca8137ba0 0x7fdca8131b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdca8134ca0 0x7fdca8131620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 --2- 172.21.15.164:0/2842838915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdca81348c0 0x7fdca81310e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 >> 172.21.15.164:0/2842838915 conn(0x7fdca80a6f80 msgr2=0x7fdca8136090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 shutdown_connections 2024-09-10T16:06:25.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.392+0000 7fdcadb5d640 1 -- 172.21.15.164:0/2842838915 wait complete. 2024-09-10T16:06:25.405 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-10T16:06:25.405 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:25.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 10 monclient: monmap: 2024-09-10T16:06:25.503 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.502938+0000 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.502938+0000 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.504 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 1 Processor -- start 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 1 -- start start 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.502+0000 7f4b9d1a1640 10 monclient: init 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b9809d680 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4b9806d1f0 0x7f4b9813cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b9806d1f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b9806d5d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9806e110 con 0x7f4b9809d680 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b9806e3f0 con 0x7f4b9806d5d0 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4b980ab520 con 0x7f4b9806d1f0 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b9d1a1640 10 monclient(hunting): authenticate will time out at 763.990355s 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4b9806d1f0 0x7f4b9813cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b97fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4b9809d680 0x7f4b9809da60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59240/0 (socket says 172.21.15.164:59240) 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b97fff640 1 -- 172.21.15.164:0/458948985 learned_addr learned my addr 172.21.15.164:0/458948985 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:25.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): get_auth_request con 0x7f4b9806d1f0 auth_method 0 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient(hunting): handle_auth_done global_id 4221 payload 274 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 1 -- 172.21.15.164:0/458948985 >> v1:172.21.15.165:6789/0 conn(0x7f4b9806d5d0 legacy=0x7f4b9813f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 1 -- 172.21.15.164:0/458948985 >> v1:172.21.15.164:6789/0 conn(0x7f4b9809d680 legacy=0x7f4b9809da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.503+0000 7f4b977fe640 1 -- 172.21.15.164:0/458948985 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b980a6260 con 0x7f4b9806d1f0 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b977fe640 1 -- 172.21.15.164:0/458948985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9806d1f0 msgr2=0x7f4b9813cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b977fe640 1 --2- 172.21.15.164:0/458948985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9806d1f0 0x7f4b9813cea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b800097a0 tx=0x7f4b80029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6c06bdcaad3bb79 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 1 -- 172.21.15.164:0/458948985 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80037070 con 0x7f4b9806d1f0 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: dump: 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.511 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b9d1a1640 5 monclient: authenticate success, global_id 4221 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 1 -- 172.21.15.164:0/458948985 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b80032640 con 0x7f4b9806d1f0 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 1 -- 172.21.15.164:0/458948985 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b80032a10 con 0x7f4b9806d1f0 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b9d1a1640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b9d1a1640 4 set_mon_vals no callback set 2024-09-10T16:06:25.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.506+0000 7f4b96ffd640 10 monclient: dump: 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 10 monclient: shutdown 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/458948985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9806d1f0 msgr2=0x7f4b9813cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/458948985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9806d1f0 0x7f4b9813cea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f4b800097a0 tx=0x7f4b80029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/458948985 shutdown_connections 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/458948985 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9806d1f0 0x7f4b9813cea0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/458948985 >> 172.21.15.164:0/458948985 conn(0x7f4b980a70e0 msgr2=0x7f4b980a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/458948985 shutdown_connections 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/458948985 wait complete. 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.507+0000 7f4b9d1a1640 10 monclient: monmap: 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.520 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 Processor -- start 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 -- start start 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient: init 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b9806d1f0 0x7f4b9812f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-b con 0x7f4b9806d1f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9806d5d0 0x7f4b9812fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-c con 0x7f4b9806d5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): picked mon.noname-a con 0x7f4b9809d680 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98140ea0 con 0x7f4b9809d680 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b98140d20 con 0x7f4b9806d1f0 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4b98141020 con 0x7f4b9806d5d0 2024-09-10T16:06:25.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9d1a1640 10 monclient(hunting): authenticate will time out at 763.995859s 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35876/0 (socket says 172.21.15.164:35876) 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b977fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9806d5d0 0x7f4b9812fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 1 -- 172.21.15.164:0/318636195 learned_addr learned my addr 172.21.15.164:0/318636195 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 10 monclient(hunting): get_auth_request con 0x7f4b9809d680 auth_method 0 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.508+0000 7f4b9c9a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): get_auth_request con 0x7f4b9806d5d0 auth_method 0 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b97fff640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b9806d1f0 0x7f4b9812f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b97fff640 10 monclient(hunting): get_auth_request con 0x7f4b9806d1f0 auth_method 0 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b97fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b97fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 -- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9806d5d0 msgr2=0x7f4b9812fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9806d5d0 0x7f4b9812fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 -- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b9806d1f0 msgr2=0x7f4b9812f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b9806d1f0 0x7f4b9812f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.509+0000 7f4b9c9a0640 1 -- 172.21.15.164:0/318636195 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b981408b0 con 0x7f4b9809d680 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9c9a0640 1 -- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 msgr2=0x7f4b98130120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9c9a0640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c780 tx=0x7f4b9000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9000ac60 con 0x7f4b9809d680 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: dump: 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.526 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 5 monclient: authenticate success, global_id 4224 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9000ae00 con 0x7f4b9809d680 2024-09-10T16:06:25.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b90004ba0 con 0x7f4b9809d680 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 10 monclient: _renew_subs 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b98131af0 con 0x7f4b9809d680 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b967fc640 4 set_mon_vals no callback set 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b94ff9640 10 monclient: dump: 2024-09-10T16:06:25.528 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 10 monclient: _renew_subs 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.511+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b98140180 con 0x7f4b9809d680 2024-09-10T16:06:25.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4b90005030 con 0x7f4b9809d680 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b9d1a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b9d1a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b9d1a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b9809da60 con 0x7f4b9809d680 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b94ff9640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 0x7f4b78040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.512+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f4b90052da0 con 0x7f4b9809d680 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.513+0000 7f4b97fff640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 0x7f4b78040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.513+0000 7f4b97fff640 10 monclient: get_auth_request con 0x7f4b7803e260 auth_method 0 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.515+0000 7f4b97fff640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 0x7f4b78040720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c7f0 tx=0x7f4b88009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.517+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4b90020070 con 0x7f4b9809d680 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.517+0000 7f4b94ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:25.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.517+0000 7f4b94ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:25.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.692+0000 7f4b9d1a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.692+0000 7f4b9d1a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.692+0000 7f4b9d1a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.692+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4b9813bf60 con 0x7f4b9809d680 2024-09-10T16:06:25.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.693+0000 7f4b94ff9640 1 -- 172.21.15.164:0/318636195 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+12439 (secure 0 0 0) 0x7f4b90005320 con 0x7f4b9809d680 2024-09-10T16:06:25.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.693+0000 7f4b94ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:25.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.693+0000 7f4b94ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:25.695 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:25.695 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":19,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:24.423222+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T16:06:20.696007+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 msgr2=0x7f4b78040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 0x7f4b78040720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f4b8800c7f0 tx=0x7f4b88009040 comp rx=0 tx=0).stop 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 10 monclient: shutdown 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 msgr2=0x7f4b98130120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4b9000c780 tx=0x7f4b9000cc40 comp rx=0 tx=0).stop 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 shutdown_connections 2024-09-10T16:06:25.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b7803e260 0x7f4b78040720 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b9809d680 0x7f4b98130120 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b9806d5d0 0x7f4b9812fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 --2- 172.21.15.164:0/318636195 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b9806d1f0 0x7f4b9812f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 >> 172.21.15.164:0/318636195 conn(0x7f4b980a70e0 msgr2=0x7f4b9809c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 shutdown_connections 2024-09-10T16:06:25.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.695+0000 7f4b9d1a1640 1 -- 172.21.15.164:0/318636195 wait complete. 2024-09-10T16:06:25.708 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T16:06:25.708 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T16:06:25.709 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T16:06:25.709 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T16:06:25.709 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-10T16:06:25.710 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-10T16:06:25.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7f96fad65640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7fdfeb0cd640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7fdfeb0cd640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7fdfeb0cd640 10 monclient: monmap: 2024-09-10T16:06:25.822 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.821645+0000 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.821645+0000 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.823 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7f96fad65640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.820+0000 7f96fad65640 10 monclient: monmap: 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.821645+0000 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.821645+0000 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.824 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7fdfeb0cd640 1 Processor -- start 2024-09-10T16:06:25.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7f96fad65640 1 Processor -- start 2024-09-10T16:06:25.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7fdfeb0cd640 1 -- start start 2024-09-10T16:06:25.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7fdfeb0cd640 10 monclient: init 2024-09-10T16:06:25.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7f96fad65640 1 -- start start 2024-09-10T16:06:25.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.821+0000 7f96fad65640 10 monclient: init 2024-09-10T16:06:25.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:06:25.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdfe406cd00 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdfe4135000 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-10T16:06:25.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-b con 0x7fdfe413e690 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96f406d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f96f406d540 0x7f96f40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfe4135450 con 0x7fdfe4135000 2024-09-10T16:06:25.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47054/0 (socket says 172.21.15.164:47054) 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 1 -- 172.21.15.164:0/2545657117 learned_addr learned my addr 172.21.15.164:0/2545657117 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdfe406df10 con 0x7fdfe406cd00 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe406d0e0 con 0x7fdfe413e690 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-a con 0x7f96f406d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:25.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfeb0cd640 10 monclient(hunting): authenticate will time out at 764.309686s 2024-09-10T16:06:25.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): get_auth_request con 0x7fdfe413e690 auth_method 0 2024-09-10T16:06:25.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-b con 0x7f96f40a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7fdfe966a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96f4135510 con 0x7f96f406d160 2024-09-10T16:06:25.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f96f409cb90 con 0x7f96f406d540 2024-09-10T16:06:25.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 10 monclient(hunting): handle_auth_done global_id 4235 payload 274 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96f9302640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96f9302640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47066/0 (socket says 172.21.15.164:47066) 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96f9302640 1 -- 172.21.15.164:0/307819908 learned_addr learned my addr 172.21.15.164:0/307819908 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96f409ce70 con 0x7f96f40a4c20 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96fad65640 10 monclient(hunting): authenticate will time out at 764.309793s 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 1 --2- 172.21.15.164:0/307819908 >> v2:172.21.15.164:3300/0 conn(0x7f96f406d540 0x7f96f40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): get_auth_request con 0x7f96f406d540 auth_method 0 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.822+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 1 -- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 msgr2=0x7f96f413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 1 -- 172.21.15.164:0/307819908 >> v1:172.21.15.165:6789/0 conn(0x7f96f406d160 legacy=0x7f96f4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96ebfff640 1 -- 172.21.15.164:0/307819908 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f409d150 con 0x7f96f406d540 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96f9302640 10 monclient: get_auth_request con 0x7f96f40a4c20 auth_method 0 2024-09-10T16:06:25.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7f96f9302640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 1 -- 172.21.15.164:0/2545657117 >> v1:172.21.15.165:6789/0 conn(0x7fdfe406cd00 legacy=0x7fdfe40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 1 -- 172.21.15.164:0/2545657117 >> v1:172.21.15.164:6789/0 conn(0x7fdfe4135000 legacy=0x7fdfe413cf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 1 monclient: found mon.noname-b 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.823+0000 7fdfe966a640 1 -- 172.21.15.164:0/2545657117 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe409cb90 con 0x7fdfe413e690 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe966a640 1 -- 172.21.15.164:0/2545657117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 msgr2=0x7fdfe4140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe966a640 1 --2- 172.21.15.164:0/2545657117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009e80 tx=0x7fdfdc02a530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4648197cb5e3622d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96ebfff640 1 -- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d540 msgr2=0x7f96f40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96ebfff640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d540 0x7f96f40a46e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009a00 tx=0x7f96e402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0df3256b7adc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 1 -- 172.21.15.164:0/2545657117 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc036070 con 0x7fdfe413e690 2024-09-10T16:06:25.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 1 -- 172.21.15.164:0/307819908 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4037070 con 0x7f96f406d540 2024-09-10T16:06:25.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: dump: 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96fad65640 5 monclient: authenticate success, global_id 4227 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 1 -- 172.21.15.164:0/307819908 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e4032640 con 0x7f96f406d540 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 1 -- 172.21.15.164:0/307819908 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e402ae60 con 0x7f96f406d540 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96fad65640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96fad65640 4 set_mon_vals no callback set 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7f96eb7fe640 10 monclient: dump: 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.841 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: dump: 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 1 -- 172.21.15.164:0/2545657117 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfdc007800 con 0x7fdfe413e690 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 1 -- 172.21.15.164:0/2545657117 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfdc007bd0 con 0x7fdfe413e690 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfe37fe640 10 monclient: dump: 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfeb0cd640 5 monclient: authenticate success, global_id 4235 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfeb0cd640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.826+0000 7fdfeb0cd640 4 set_mon_vals no callback set 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 10 monclient: shutdown 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 -- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d540 msgr2=0x7f96f40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d540 0x7f96f40a46e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009a00 tx=0x7f96e402a230 comp rx=0 tx=0).stop 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 -- 172.21.15.164:0/307819908 shutdown_connections 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f40a4c20 0x7f96f413f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 --2- 172.21.15.164:0/307819908 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d540 0x7f96f40a46e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 -- 172.21.15.164:0/307819908 >> 172.21.15.164:0/307819908 conn(0x7f96f40aacb0 msgr2=0x7f96f40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 -- 172.21.15.164:0/307819908 shutdown_connections 2024-09-10T16:06:25.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 1 -- 172.21.15.164:0/307819908 wait complete. 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 10 monclient: shutdown 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 msgr2=0x7fdfe4140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 --2- 172.21.15.164:0/2545657117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdfdc009e80 tx=0x7fdfdc02a530 comp rx=0 tx=0).stop 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 shutdown_connections 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 --2- 172.21.15.164:0/2545657117 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe4140a80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 >> 172.21.15.164:0/2545657117 conn(0x7fdfe40aaa70 msgr2=0x7fdfe40aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 shutdown_connections 2024-09-10T16:06:25.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/2545657117 wait complete. 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.827+0000 7f96fad65640 10 monclient: monmap: 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.848 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 1 Processor -- start 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 1 -- start start 2024-09-10T16:06:25.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 10 monclient: init 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7fdfeb0cd640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7fdfeb0cd640 10 monclient: monmap: 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7fdfeb0cd640 1 Processor -- start 2024-09-10T16:06:25.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7fdfeb0cd640 1 -- start start 2024-09-10T16:06:25.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7fdfeb0cd640 10 monclient: init 2024-09-10T16:06:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d160 0x7f96f40a1d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.828+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-a con 0x7f96f406d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 0x7f96f40a2270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 0x7f96f40a2270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d160 0x7f96f40a1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 0x7f96f40a2270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47082/0 (socket says 172.21.15.164:47082) 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d160 0x7f96f40a1d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35894/0 (socket says 172.21.15.164:35894) 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 1 -- 172.21.15.164:0/2017595892 learned_addr learned my addr 172.21.15.164:0/2017595892 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-c con 0x7f96f406d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96f40a4c20 0x7f96f41cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): picked mon.noname-b con 0x7f96f40a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96f413fb20 con 0x7f96f406d160 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96f413f9a0 con 0x7f96f40a4c20 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96f413fca0 con 0x7f96f406d540 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96fad65640 10 monclient(hunting): authenticate will time out at 764.316395s 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 10 monclient(hunting): get_auth_request con 0x7f96f406d160 auth_method 0 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f8b01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): get_auth_request con 0x7f96f406d540 auth_method 0 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f9302640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96f40a4c20 0x7f96f41cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f9302640 10 monclient(hunting): get_auth_request con 0x7f96f40a4c20 auth_method 0 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f9302640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f9302640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7f96f9302640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f9302640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f9302640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f9302640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfe406cd00 0x7fdfe4131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-a con 0x7fdfe406cd00 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 0x7fdfe4131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-b con 0x7fdfe4135000 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe413ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): picked mon.noname-c con 0x7fdfe413e690 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe4140fc0 con 0x7fdfe406cd00 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdfe40a4df0 con 0x7fdfe4135000 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdfe4141140 con 0x7fdfe413e690 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.829+0000 7fdfeb0cd640 10 monclient(hunting): authenticate will time out at 764.316993s 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 0x7fdfe4131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 0x7fdfe4131790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:44198/0 (socket says 172.21.15.164:44198) 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 -- 172.21.15.164:0/3243537584 learned_addr learned my addr 172.21.15.164:0/3243537584 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfe406cd00 0x7fdfe4131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): get_auth_request con 0x7fdfe4135000 auth_method 0 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): get_auth_request con 0x7fdfe406cd00 auth_method 0 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f8b01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f8b01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe413ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 10 monclient(hunting): get_auth_request con 0x7fdfe413e690 auth_method 0 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96f8b01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe8e69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2024-09-10T16:06:25.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96f40a4c20 msgr2=0x7f96f41cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96f40a4c20 0x7f96f41cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d160 msgr2=0x7f96f40a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96f406d160 0x7f96f40a1d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient(hunting): handle_auth_done global_id 4228 payload 824 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 msgr2=0x7fdfe413ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe413ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfe406cd00 msgr2=0x7fdfe4131250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdfe406cd00 0x7fdfe4131250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 monclient: found mon.noname-b 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe3fff640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfe413df70 con 0x7fdfe4135000 2024-09-10T16:06:25.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7fdfe966a640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdfe413e690 0x7fdfe413ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.830+0000 7f96ebfff640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96f4136a50 con 0x7f96f406d540 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96ebfff640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 msgr2=0x7f96f40a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96ebfff640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 0x7f96f40a2270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f96e4034000 tx=0x7f96e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4040020 con 0x7f96f406d540 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe3fff640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 msgr2=0x7fdfe4131790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe3fff640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 0x7fdfe4131790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c430 tx=0x7fdfd000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: dump: 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.866 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00119a0 con 0x7fdfe4135000 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: dump: 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfd0017070 con 0x7fdfe4135000 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0011d80 con 0x7fdfe4135000 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe17fa640 10 monclient: dump: 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7fdfe2ffd640 4 set_mon_vals no callback set 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e4038040 con 0x7f96f406d540 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4037070 con 0x7f96f406d540 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.832+0000 7f96e97fa640 10 monclient: dump: 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 5 monclient: authenticate success, global_id 4238 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 10 monclient: _renew_subs 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96f413f0b0 con 0x7f96f406d540 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96eaffd640 4 set_mon_vals no callback set 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 10 monclient: _renew_subs 2024-09-10T16:06:25.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.833+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96f40a3540 con 0x7f96f406d540 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 5 monclient: authenticate success, global_id 4228 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 10 monclient: _renew_subs 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfe409def0 con 0x7fdfe4135000 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 10 monclient: _renew_subs 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfe406e670 con 0x7fdfe4135000 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7f96fad65640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f96f40aa1e0 con 0x7f96f406d540 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.834+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f96e4043020 con 0x7f96f406d540 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96e003e210 0x7f96e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f96e4072dd0 con 0x7f96f406d540 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7f96e0044080 0x7f96e00464c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96e403ce50 con 0x7f96e0044080 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f96e4077030 con 0x7f96f406d540 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96e97fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-10T16:06:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96f8b01640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96e003e210 0x7f96e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96f8b01640 10 monclient: get_auth_request con 0x7f96e003e210 auth_method 0 2024-09-10T16:06:25.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96f9302640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7f96e0044080 0x7f96e00464c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7fdfeb0cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fdfe40a46e0 con 0x7fdfe4135000 2024-09-10T16:06:25.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7f96f9302640 10 monclient: get_auth_request con 0x7f96e0044080 auth_method 0 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.835+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdfd001c460 con 0x7fdfe4135000 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdfc803e260 0x7fdfc8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fdfd0052a80 con 0x7fdfe4135000 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7fdfc8044150 0x7fdfc8046590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdfd001ee70 con 0x7fdfc8044150 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdfd00572a0 con 0x7fdfe4135000 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe17fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe8e69640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdfc803e260 0x7fdfc8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe966a640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7fdfc8044150 0x7fdfc8046590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe8e69640 10 monclient: get_auth_request con 0x7fdfc803e260 auth_method 0 2024-09-10T16:06:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.836+0000 7fdfe966a640 10 monclient: get_auth_request con 0x7fdfc8044150 auth_method 0 2024-09-10T16:06:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.837+0000 7f96f8b01640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96e003e210 0x7f96e00406d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f96f0007fd0 tx=0x7f96f0003e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.839+0000 7fdfe8e69640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdfc803e260 0x7fdfc8040720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfe413cf30 tx=0x7fdfd8009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.840+0000 7f96f9302640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7f96e0044080 0x7f96e00464c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.840+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== osd.2 v2:172.21.15.164:6808/1539912567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f96e403ce50 con 0x7f96e0044080 2024-09-10T16:06:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.841+0000 7fdfe966a640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7fdfc8044150 0x7fdfc8046590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.841+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== osd.3 v2:172.21.15.165:6800/4161381544 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdfd001ee70 con 0x7fdfc8044150 2024-09-10T16:06:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.856+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 --> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f96f41366d0 con 0x7f96e0044080 2024-09-10T16:06:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96e97fa640 1 -- 172.21.15.164:0/2017595892 <== osd.2 v2:172.21.15.164:6808/1539912567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f96f41366d0 con 0x7f96e0044080 2024-09-10T16:06:25.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7f96e0044080 msgr2=0x7f96e00464c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7f96e0044080 0x7f96e00464c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96e003e210 msgr2=0x7f96e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96e003e210 0x7f96e00406d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f96f0007fd0 tx=0x7f96f0003e60 comp rx=0 tx=0).stop 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 10 monclient: shutdown 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 msgr2=0x7f96f40a2270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.857+0000 7f96fad65640 1 --2- 172.21.15.164:0/2017595892 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96f406d540 0x7f96f40a2270 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f96e4034000 tx=0x7f96e40046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7f96f9302640 1 -- 172.21.15.164:0/2017595892 reap_dead start 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 shutdown_connections 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 >> 172.21.15.164:0/2017595892 conn(0x7f96f40aacb0 msgr2=0x7f96f4132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 shutdown_connections 2024-09-10T16:06:25.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7f96fad65640 1 -- 172.21.15.164:0/2017595892 wait complete. 2024-09-10T16:06:25.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.858+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 --> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdfe413cfe0 con 0x7fdfc8044150 2024-09-10T16:06:25.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.859+0000 7fdfe17fa640 1 -- 172.21.15.164:0/3243537584 <== osd.3 v2:172.21.15.165:6800/4161381544 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdfe413cfe0 con 0x7fdfc8044150 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.859+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7fdfc8044150 msgr2=0x7fdfc8046590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.859+0000 7fdfeb0cd640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7fdfc8044150 0x7fdfc8046590 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdfc803e260 msgr2=0x7fdfc8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdfc803e260 0x7fdfc8040720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fdfe413cf30 tx=0x7fdfd8009060 comp rx=0 tx=0).stop 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 10 monclient: shutdown 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 msgr2=0x7fdfe4131790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 --2- 172.21.15.164:0/3243537584 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdfe4135000 0x7fdfe4131790 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c430 tx=0x7fdfd000c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfe966a640 1 -- 172.21.15.164:0/3243537584 reap_dead start 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 shutdown_connections 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 >> 172.21.15.164:0/3243537584 conn(0x7fdfe40aaa70 msgr2=0x7fdfe413f890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 shutdown_connections 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.860+0000 7fdfeb0cd640 1 -- 172.21.15.164:0/3243537584 wait complete. 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.861+0000 7fcbeadf3640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.861+0000 7fcbeadf3640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.862+0000 7fcbeadf3640 10 monclient: monmap: 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.862902+0000 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.862902+0000 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.887 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.862+0000 7f1409ce2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.862+0000 7f1409ce2640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.862+0000 7f1409ce2640 10 monclient: monmap: 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.863192+0000 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.863192+0000 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7fcbeadf3640 1 Processor -- start 2024-09-10T16:06:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 Processor -- start 2024-09-10T16:06:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 -- start start 2024-09-10T16:06:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient: init 2024-09-10T16:06:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7fcbeadf3640 1 -- start start 2024-09-10T16:06:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7fcbeadf3640 10 monclient: init 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f140409b3a0 0x7f14040a2410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f140409b3a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f140409b780 0x7f1404135ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f140409b780 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f140409bbb0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14040a1aa0 con 0x7f140409bbb0 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.863+0000 7f1409ce2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f14040a1190 con 0x7f140409b3a0 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f1409ce2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f1409ce2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f14040a1470 con 0x7f140409b780 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f1409ce2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f1409ce2640 10 monclient(hunting): authenticate will time out at 764.351019s 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f140409b3a0 0x7f14040a2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f140409b3a0 0x7f14040a2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35912/0 (socket says 172.21.15.164:35912) 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 -- 172.21.15.164:0/4251694868 learned_addr learned my addr 172.21.15.164:0/4251694868 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): get_auth_request con 0x7f140409b3a0 auth_method 0 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 1 --2- 172.21.15.164:0/4251694868 >> v2:172.21.15.165:3300/0 conn(0x7f140409b780 0x7f1404135ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): get_auth_request con 0x7f140409b780 auth_method 0 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient(hunting): handle_auth_done global_id 4236 payload 274 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 -- 172.21.15.164:0/4251694868 >> v2:172.21.15.165:3300/0 conn(0x7f140409b780 msgr2=0x7f1404135ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 --2- 172.21.15.164:0/4251694868 >> v2:172.21.15.165:3300/0 conn(0x7f140409b780 0x7f1404135ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 -- 172.21.15.164:0/4251694868 >> v1:172.21.15.165:6789/0 conn(0x7f140409bbb0 legacy=0x7f1404136010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7f14037fe640 1 -- 172.21.15.164:0/4251694868 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14040a4770 con 0x7f140409b3a0 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcbe406d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcbe406d540 0x7fcbe40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe406d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcbe40a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe4135510 con 0x7fcbe406d160 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe409cb90 con 0x7fcbe40a4c20 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe409ce70 con 0x7fcbe406d540 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbeadf3640 10 monclient(hunting): authenticate will time out at 764.351702s 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbe8b8f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcbe406d160 0x7fcbe4134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59244/0 (socket says 172.21.15.164:59244) 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbe8b8f640 1 -- 172.21.15.164:0/3753000274 learned_addr learned my addr 172.21.15.164:0/3753000274 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbe3fff640 1 --2- 172.21.15.164:0/3753000274 >> v2:172.21.15.165:3300/0 conn(0x7fcbe406d540 0x7fcbe40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.864+0000 7fcbe3fff640 10 monclient(hunting): get_auth_request con 0x7fcbe406d540 auth_method 0 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 1 -- 172.21.15.164:0/3753000274 >> v1:172.21.15.165:6789/0 conn(0x7fcbe40a4c20 legacy=0x7fcbe413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 1 -- 172.21.15.164:0/3753000274 >> v1:172.21.15.164:6789/0 conn(0x7fcbe406d160 legacy=0x7fcbe4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.865+0000 7fcbe3fff640 1 -- 172.21.15.164:0/3753000274 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe409d150 con 0x7fcbe406d540 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 10 monclient: monmap: 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.867021+0000 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.867021+0000 2024-09-10T16:06:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 1 Processor -- start 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 1 -- start start 2024-09-10T16:06:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.866+0000 7fddf3248640 10 monclient: init 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7efffa4c6640 10 monclient: get_monmap_and_config 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7efffa4c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7efffa4c6640 10 monclient: monmap: 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:25.868012+0000 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:25.868012+0000 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:25.902 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:25.903 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:25.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:06:25.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fddec06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fddec134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fddec13e740 0x7fddec140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7f14037fe640 1 -- 172.21.15.164:0/4251694868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 msgr2=0x7f14040a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-a con 0x7fddec13e740 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:25.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7f14037fe640 1 --2- 172.21.15.164:0/4251694868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f14040a2410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1404071030 tx=0x7f13f402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d8579b2d604ecd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddebfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fddec134f70 0x7fddec13d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59260/0 (socket says 172.21.15.164:59260) 2024-09-10T16:06:25.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddebfff640 1 -- 172.21.15.164:0/2894106333 learned_addr learned my addr 172.21.15.164:0/2894106333 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:25.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf17e5640 1 --2- 172.21.15.164:0/2894106333 >> v2:172.21.15.164:3300/0 conn(0x7fddec13e740 0x7fddec140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec09ce50 con 0x7fddec134f70 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fddec135380 con 0x7fddec06cb70 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fddec06db60 con 0x7fddec13e740 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7efffa4c6640 1 Processor -- start 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 1 -- start start 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient: init 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf3248640 10 monclient(hunting): authenticate will time out at 764.354917s 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf17e5640 10 monclient(hunting): get_auth_request con 0x7fddec13e740 auth_method 0 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf17e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf17e5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.867+0000 7fddf17e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe3fff640 1 -- 172.21.15.164:0/3753000274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 msgr2=0x7fcbe40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe3fff640 1 --2- 172.21.15.164:0/3753000274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 0x7fcbe40a46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80097a0 tx=0x7fcbd8029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=faf891ca3e78979b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient(hunting): handle_auth_done global_id 4239 payload 274 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 1 -- 172.21.15.164:0/2894106333 >> v1:172.21.15.165:6789/0 conn(0x7fddec06cb70 legacy=0x7fddec0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 1 -- 172.21.15.164:0/2894106333 >> v1:172.21.15.164:6789/0 conn(0x7fddec134f70 legacy=0x7fddec13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fddf17e5640 1 -- 172.21.15.164:0/2894106333 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec09cb70 con 0x7fddec13e740 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efff406d160 0x7efff4134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-c con 0x7efff406d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efff406d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 1 -- 172.21.15.164:0/3753000274 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8037070 con 0x7fcbe406d540 2024-09-10T16:06:25.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: dump: 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 1 -- 172.21.15.164:0/3753000274 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd8032640 con 0x7fcbe406d540 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 1 -- 172.21.15.164:0/3753000274 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd8032a10 con 0x7fcbe406d540 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbe37fe640 10 monclient: dump: 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.912 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efff413d060 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff409ce70 con 0x7efff406d540 2024-09-10T16:06:25.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbeadf3640 5 monclient: authenticate success, global_id 4234 2024-09-10T16:06:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbeadf3640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7fcbeadf3640 4 set_mon_vals no callback set 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efff409d150 con 0x7efff413d060 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efff37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efff406d540 0x7efff40a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59272/0 (socket says 172.21.15.164:59272) 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efff37fe640 1 -- 172.21.15.164:0/370571274 learned_addr learned my addr 172.21.15.164:0/370571274 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efff3fff640 1 --2- 172.21.15.164:0/370571274 >> v2:172.21.15.165:3300/0 conn(0x7efff406d160 0x7efff4134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efff409cc20 con 0x7efff406d160 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efffa4c6640 10 monclient(hunting): authenticate will time out at 764.355977s 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efff3fff640 10 monclient(hunting): get_auth_request con 0x7efff406d160 auth_method 0 2024-09-10T16:06:25.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.868+0000 7efff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient(hunting): handle_auth_done global_id 4237 payload 274 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 1 -- 172.21.15.164:0/370571274 >> v1:172.21.15.165:6789/0 conn(0x7efff413d060 legacy=0x7efff413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 1 -- 172.21.15.164:0/370571274 >> v1:172.21.15.164:6789/0 conn(0x7efff406d540 legacy=0x7efff40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:25.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7efff3fff640 1 -- 172.21.15.164:0/370571274 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff40a4df0 con 0x7efff406d160 2024-09-10T16:06:25.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 10 monclient: shutdown 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 -- 172.21.15.164:0/3753000274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 msgr2=0x7fcbe40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 --2- 172.21.15.164:0/3753000274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 0x7fcbe40a46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcbd80097a0 tx=0x7fcbd8029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 -- 172.21.15.164:0/3753000274 shutdown_connections 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 --2- 172.21.15.164:0/3753000274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 0x7fcbe40a46e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 -- 172.21.15.164:0/3753000274 >> 172.21.15.164:0/3753000274 conn(0x7fcbe40aacb0 msgr2=0x7fcbe40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 -- 172.21.15.164:0/3753000274 shutdown_connections 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.869+0000 7fcbeadf3640 1 -- 172.21.15.164:0/3753000274 wait complete. 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.870+0000 7fcbeadf3640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.870+0000 7fcbeadf3640 10 monclient: monmap: 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.870+0000 7fcbeadf3640 1 Processor -- start 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.870+0000 7fcbeadf3640 1 -- start start 2024-09-10T16:06:25.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.870+0000 7fcbeadf3640 10 monclient: init 2024-09-10T16:06:25.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf17e5640 1 -- 172.21.15.164:0/2894106333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec13e740 msgr2=0x7fddec140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf17e5640 1 --2- 172.21.15.164:0/2894106333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec13e740 0x7fddec140b30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009dc0 tx=0x7fdde402a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcc690722d542e05 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 1 -- 172.21.15.164:0/4251694868 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4037070 con 0x7f140409b3a0 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: dump: 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.921 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 1 -- 172.21.15.164:0/4251694868 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13f4032640 con 0x7f140409b3a0 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 1 -- 172.21.15.164:0/4251694868 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f4032a10 con 0x7f140409b3a0 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1402ffd640 10 monclient: dump: 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.922 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1409ce2640 5 monclient: authenticate success, global_id 4236 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1409ce2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7f1409ce2640 4 set_mon_vals no callback set 2024-09-10T16:06:25.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 1 -- 172.21.15.164:0/2894106333 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4007a30 con 0x7fddec13e740 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbe406d160 0x7fcbe40a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-c con 0x7fcbe406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 0x7fcbe409dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-b con 0x7fcbe406d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 0x7fcbe409e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): picked mon.noname-a con 0x7fcbe40a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe4140ca0 con 0x7fcbe40a4c20 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe4140b20 con 0x7fcbe406d540 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe4140e20 con 0x7fcbe406d160 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbeadf3640 10 monclient(hunting): authenticate will time out at 764.358612s 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: dump: 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.926 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf3248640 5 monclient: authenticate success, global_id 4239 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf3248640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 1 -- 172.21.15.164:0/2894106333 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde4007bd0 con 0x7fddec13e740 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 1 -- 172.21.15.164:0/2894106333 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde4031400 con 0x7fddec13e740 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf3248640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddf3248640 4 set_mon_vals no callback set 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 0x7fcbe409e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 0x7fcbe409e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35926/0 (socket says 172.21.15.164:35926) 2024-09-10T16:06:25.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 learned_addr learned my addr 172.21.15.164:0/1554684190 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe8b8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbe406d160 0x7fcbe40a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): get_auth_request con 0x7fcbe40a4c20 auth_method 0 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe8b8f640 10 monclient(hunting): get_auth_request con 0x7fcbe406d160 auth_method 0 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe8b8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe8b8f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe8b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fcbe9390640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.871+0000 7fddeb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe8b8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe8b8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe8b8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fddeb7fe640 10 monclient: dump: 2024-09-10T16:06:25.930 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.931 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 10 monclient(hunting): handle_auth_done global_id 4242 payload 824 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbe406d160 msgr2=0x7fcbe40a0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbe406d160 0x7fcbe40a0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 msgr2=0x7fcbe409dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcbe406d540 0x7fcbe409dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe409f620 con 0x7fcbe40a4c20 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fcbe8b8f640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcbe406d160 0x7fcbe40a0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 10 monclient: shutdown 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 -- 172.21.15.164:0/4251694868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 msgr2=0x7f14040a2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 --2- 172.21.15.164:0/4251694868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f14040a2410 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1404071030 tx=0x7f13f402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff3fff640 1 -- 172.21.15.164:0/370571274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff406d160 msgr2=0x7efff4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff3fff640 1 --2- 172.21.15.164:0/370571274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff406d160 0x7efff4134e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7effe40097a0 tx=0x7effe4029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9351a7ebde8ab0e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 -- 172.21.15.164:0/4251694868 shutdown_connections 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 --2- 172.21.15.164:0/4251694868 >> v2:172.21.15.165:3300/0 conn(0x7f140409b780 0x7f1404135ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 --2- 172.21.15.164:0/4251694868 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f14040a2410 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 -- 172.21.15.164:0/4251694868 >> 172.21.15.164:0/4251694868 conn(0x7f14040a5030 msgr2=0x7f14040a5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 -- 172.21.15.164:0/4251694868 shutdown_connections 2024-09-10T16:06:25.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7f1409ce2640 1 -- 172.21.15.164:0/4251694868 wait complete. 2024-09-10T16:06:25.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 1 -- 172.21.15.164:0/370571274 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4037070 con 0x7efff406d160 2024-09-10T16:06:25.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: dump: 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 1 -- 172.21.15.164:0/370571274 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe4032640 con 0x7efff406d160 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 1 -- 172.21.15.164:0/370571274 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4032a10 con 0x7efff406d160 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efff2ffd640 10 monclient: dump: 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.937 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efffa4c6640 5 monclient: authenticate success, global_id 4237 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efffa4c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:25.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7efffa4c6640 4 set_mon_vals no callback set 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fddf3248640 10 monclient: shutdown 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fddf3248640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec13e740 msgr2=0x7fddec140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.872+0000 7fddf3248640 1 --2- 172.21.15.164:0/2894106333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec13e740 0x7fddec140b30 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fdde4009dc0 tx=0x7fdde402a4d0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 shutdown_connections 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 1 --2- 172.21.15.164:0/2894106333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec13e740 0x7fddec140b30 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 >> 172.21.15.164:0/2894106333 conn(0x7fddec0aa900 msgr2=0x7fddec0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 shutdown_connections 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 1 -- 172.21.15.164:0/2894106333 wait complete. 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7f1409ce2640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7f1409ce2640 10 monclient: monmap: 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.940 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 10 monclient: shutdown 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff406d160 msgr2=0x7efff4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 --2- 172.21.15.164:0/370571274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff406d160 0x7efff4134e00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7effe40097a0 tx=0x7effe4029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 shutdown_connections 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 --2- 172.21.15.164:0/370571274 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff406d160 0x7efff4134e00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 >> 172.21.15.164:0/370571274 conn(0x7efff40aacb0 msgr2=0x7efff40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 shutdown_connections 2024-09-10T16:06:25.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7efffa4c6640 1 -- 172.21.15.164:0/370571274 wait complete. 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7f1409ce2640 1 Processor -- start 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7f1409ce2640 1 -- start start 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7f1409ce2640 10 monclient: init 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.873+0000 7fddf3248640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fddf3248640 10 monclient: monmap: 2024-09-10T16:06:25.943 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7efffa4c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7efffa4c6640 10 monclient: monmap: 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:25.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fddf3248640 1 Processor -- start 2024-09-10T16:06:25.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f140409a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-a con 0x7f140409b3a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 0x7f140412d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fddf3248640 1 -- start start 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fddf3248640 10 monclient: init 2024-09-10T16:06:25.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 msgr2=0x7fcbe409e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7fcbe9390640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 0x7fcbe409e0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c7b0 tx=0x7fcbdc00cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-c con 0x7f140409b780 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f140409bbb0 0x7f140412db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f140409a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f13fbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 0x7f140412d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f140409a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35938/0 (socket says 172.21.15.164:35938) 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): picked mon.noname-b con 0x7f140409bbb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 1 -- 172.21.15.164:0/27463156 learned_addr learned my addr 172.21.15.164:0/27463156 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f14041368a0 con 0x7f140409b3a0 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1404136720 con 0x7f140409bbb0 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f140413de10 con 0x7f140409b780 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f1409ce2640 10 monclient(hunting): authenticate will time out at 764.361923s 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 10 monclient(hunting): get_auth_request con 0x7f140409b3a0 auth_method 0 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f14037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f13fbfff640 10 monclient(hunting): get_auth_request con 0x7f140409b780 auth_method 0 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f13fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f13fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7f13fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f140409bbb0 0x7f140412db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f14037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): get_auth_request con 0x7f140409bbb0 auth_method 0 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.874+0000 7efffa4c6640 1 Processor -- start 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7efffa4c6640 1 -- start start 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7efffa4c6640 10 monclient: init 2024-09-10T16:06:25.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 0x7fddec13ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-a con 0x7fddec06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddec134f70 0x7fddec13b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-b con 0x7fddec134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddec13e740 0x7fddec13b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): picked mon.noname-c con 0x7fddec13e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fddec0a4db0 con 0x7fddec06cb70 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fddec06cf50 con 0x7fddec134f70 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fddec141070 con 0x7fddec13e740 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf3248640 10 monclient(hunting): authenticate will time out at 764.362797s 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 0x7fddec13ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 0x7fddec13ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35946/0 (socket says 172.21.15.164:35946) 2024-09-10T16:06:25.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 1 -- 172.21.15.164:0/3606740012 learned_addr learned my addr 172.21.15.164:0/3606740012 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 10 monclient(hunting): get_auth_request con 0x7fddec06cb70 auth_method 0 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddf0fe4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc016070 con 0x7fcbe40a4c20 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: dump: 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbdc006d50 con 0x7fcbe40a4c20 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc00ce60 con 0x7fcbe40a4c20 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe17fa640 10 monclient: dump: 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.956 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbe2ffd640 4 set_mon_vals no callback set 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 5 monclient: authenticate success, global_id 4242 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 10 monclient: _renew_subs 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe412f810 con 0x7fcbe40a4c20 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 10 monclient: _renew_subs 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe40a1980 con 0x7fcbe40a4c20 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddebfff640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddec134f70 0x7fddec13b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7efffa4c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddebfff640 10 monclient(hunting): get_auth_request con 0x7fddec134f70 auth_method 0 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7fddebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:25.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 0x7efff40a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f1403fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-c con 0x7efff406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:25.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efff406d540 0x7efff409efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 0x7efff40a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 0x7efff40a1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47110/0 (socket says 172.21.15.164:47110) 2024-09-10T16:06:25.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 -- 172.21.15.164:0/582728173 learned_addr learned my addr 172.21.15.164:0/582728173 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:25.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2024-09-10T16:06:25.962 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:25.962 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T16:06:25.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-a con 0x7efff406d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:25.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff413d060 0x7efff409f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f140409bbb0 msgr2=0x7f140412db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): picked mon.noname-b con 0x7efff413d060 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:25.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efff4140cf0 con 0x7efff406d540 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efff4140b70 con 0x7efff413d060 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efff4140e70 con 0x7efff406d160 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efffa4c6640 10 monclient(hunting): authenticate will time out at 764.363544s 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): get_auth_request con 0x7efff406d160 auth_method 0 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f140409bbb0 0x7f140412db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 msgr2=0x7f140409a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:25.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:25.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient(hunting): handle_auth_done global_id 4248 payload 824 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddec13e740 msgr2=0x7fddec13b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fddec13e740 0x7fddec13b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddec134f70 msgr2=0x7fddec13b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fddec134f70 0x7fddec13b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 monclient: found mon.noname-a 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fddf0fe4640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddec13c830 con 0x7fddec06cb70 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f140409b3a0 0x7f140409a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.875+0000 7f13fbfff640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14041309e0 con 0x7f140409b780 2024-09-10T16:06:25.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff8a63640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff413d060 0x7efff409f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff8a63640 10 monclient(hunting): get_auth_request con 0x7efff413d060 auth_method 0 2024-09-10T16:06:25.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff8a63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:25.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff8a63640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:25.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff8a63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fcbeadf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fcbe4130ba0 con 0x7fcbe40a4c20 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient(hunting): handle_auth_done global_id 4250 payload 824 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff413d060 msgr2=0x7efff409f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efff413d060 0x7efff409f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efff406d540 msgr2=0x7efff409efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:25.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efff406d540 0x7efff409efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 monclient: found mon.noname-c 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.876+0000 7efff3fff640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efff40a2370 con 0x7efff406d160 2024-09-10T16:06:25.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcbdc004960 con 0x7fcbe40a4c20 2024-09-10T16:06:25.971 INFO:teuthology.orchestra.run.smithi164.stdout:47244640260 2024-09-10T16:06:25.971 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcbd003e290 0x7fcbd0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fcbdc052ad0 con 0x7fcbe40a4c20 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7fcbd0044110 0x7fcbd0046550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbdc015ec0 con 0x7fcbd0044110 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fcbdc0572a0 con 0x7fcbe40a4c20 2024-09-10T16:06:25.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe17fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-10T16:06:25.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7f13fbfff640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 msgr2=0x7f140412d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7f13fbfff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 0x7f140412d5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00e880 tx=0x7f13ec00ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe8b8f640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcbd003e290 0x7fcbd0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe8b8f640 10 monclient: get_auth_request con 0x7fcbd003e290 auth_method 0 2024-09-10T16:06:25.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe3fff640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7fcbd0044110 0x7fcbd0046550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.877+0000 7fcbe3fff640 10 monclient: get_auth_request con 0x7fcbd0044110 auth_method 0 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec017070 con 0x7f140409b780 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: dump: 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.975 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 5 monclient: authenticate success, global_id 4247 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 10 monclient: _renew_subs 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f140409f720 con 0x7f140409b780 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec00cb50 con 0x7f140409b780 2024-09-10T16:06:25.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f14027fc640 4 set_mon_vals no callback set 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec021060 con 0x7f140409b780 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1400ff9640 10 monclient: dump: 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.977 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 10 monclient: _renew_subs 2024-09-10T16:06:25.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f140412fa80 con 0x7f140409b780 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7fddf0fe4640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 msgr2=0x7fddec13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.878+0000 7fddf0fe4640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 0x7fddec13ad70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdde000b3f0 tx=0x7fdde000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0015020 con 0x7fddec06cb70 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: dump: 2024-09-10T16:06:25.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdde0011a60 con 0x7fddec06cb70 2024-09-10T16:06:25.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdde0011e50 con 0x7fddec06cb70 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fdde97fa640 10 monclient: dump: 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.981 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 5 monclient: authenticate success, global_id 4248 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 10 monclient: _renew_subs 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddeaffd640 4 set_mon_vals no callback set 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddec09e3c0 con 0x7fddec06cb70 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 10 monclient: _renew_subs 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddec13d390 con 0x7fddec06cb70 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff3fff640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 msgr2=0x7efff40a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:25.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff3fff640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 0x7efff40a1410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7effe402a5c0 tx=0x7effe40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f13ec01a070 con 0x7f140409b780 2024-09-10T16:06:25.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7f1409ce2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1404131c60 con 0x7f140409b780 2024-09-10T16:06:25.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7f1400ff9640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e4042620 0x7f13e4044ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4042020 con 0x7efff406d160 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: dump: 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.984 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe4030880 con 0x7efff406d160 2024-09-10T16:06:25.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efffa4c6640 5 monclient: authenticate success, global_id 4250 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efffa4c6640 10 monclient: _renew_subs 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efffa4c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efff40a2650 con 0x7efff406d160 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4037040 con 0x7efff406d160 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:25.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f13ec01c450 con 0x7f140409b780 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1400ff9640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f13e4048420 0x7f13e404a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13e404af30 con 0x7f13e4048420 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f14037fe640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e4042620 0x7f13e4044ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f14037fe640 10 monclient: get_auth_request con 0x7f13e4042620 auth_method 0 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1403fff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f13e4048420 0x7f13e404a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1403fff640 10 monclient: get_auth_request con 0x7f13e4048420 auth_method 0 2024-09-10T16:06:25.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7fddf3248640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:25.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fddec0a46a0 con 0x7fddec06cb70 2024-09-10T16:06:25.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7fcbe8b8f640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcbd003e290 0x7fcbd0040750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4007c90 tx=0x7fcbd4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7f13ec01c7e0 con 0x7f140409b780 2024-09-10T16:06:25.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7f1400ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff0ff9640 10 monclient: dump: 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:25.989 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:25.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.879+0000 7efff27fc640 4 set_mon_vals no callback set 2024-09-10T16:06:25.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7efffa4c6640 10 monclient: _renew_subs 2024-09-10T16:06:25.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7efffa4c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:25.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efff40a0b10 con 0x7efff406d160 2024-09-10T16:06:25.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.880+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7effe4045020 con 0x7efff406d160 2024-09-10T16:06:25.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdde0018020 con 0x7fddec06cb70 2024-09-10T16:06:25.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efff0ff9640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effe003e290 0x7effe0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7effe4077170 con 0x7efff406d160 2024-09-10T16:06:25.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fddd003e3b0 0x7fddd0040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efffa4c6640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7efff40a4170 0x7efff40a7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fdde0052a70 con 0x7fddec06cb70 2024-09-10T16:06:25.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efff406a400 con 0x7efff40a4170 2024-09-10T16:06:25.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7fddd0044220 0x7fddd0046660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:25.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdde000de50 con 0x7fddd0044220 2024-09-10T16:06:25.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (secure 0 0 0) 0x7fdde0057220 con 0x7fddec06cb70 2024-09-10T16:06:25.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fdde97fa640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-10T16:06:25.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efff8a63640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7efff40a4170 0x7efff40a7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7efff8a63640 10 monclient: get_auth_request con 0x7efff40a4170 auth_method 0 2024-09-10T16:06:25.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fddf17e5640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7fddd0044220 0x7fddd0046660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.881+0000 7fddf17e5640 10 monclient: get_auth_request con 0x7fddd0044220 auth_method 0 2024-09-10T16:06:25.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.882+0000 7fcbe3fff640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7fcbd0044110 0x7fcbd0046550 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.882+0000 7f14037fe640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e4042620 0x7f13e4044ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009630 tx=0x7f13f402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.882+0000 7efff37fe640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effe003e290 0x7effe0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.882+0000 7efff37fe640 10 monclient: get_auth_request con 0x7effe003e290 auth_method 0 2024-09-10T16:06:25.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.883+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== osd.1 v2:172.21.15.164:6816/1609205011 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fcbdc015ec0 con 0x7fcbd0044110 2024-09-10T16:06:25.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.884+0000 7f1403fff640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f13e4048420 0x7f13e404a860 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.885+0000 7efff37fe640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effe003e290 0x7effe0040750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.885+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== osd.0 v2:172.21.15.164:6800/3366597746 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f13e404af30 con 0x7f13e4048420 2024-09-10T16:06:25.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.886+0000 7efff8a63640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7efff40a4170 0x7efff40a7a20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.886+0000 7fddf17e5640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7fddd0044220 0x7fddd0046660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.886+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== osd.5 v2:172.21.15.165:6816/3196812989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efff406a400 con 0x7efff40a4170 2024-09-10T16:06:25.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.886+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== osd.4 v2:172.21.15.165:6808/592433527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fdde000de50 con 0x7fddd0044220 2024-09-10T16:06:25.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.888+0000 7fddebfff640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fddd003e3b0 0x7fddd0040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:25.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.888+0000 7fddebfff640 10 monclient: get_auth_request con 0x7fddd003e3b0 auth_method 0 2024-09-10T16:06:25.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.890+0000 7fddebfff640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fddd003e3b0 0x7fddd0040870 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007c90 tx=0x7fddd8007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:25.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.896+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 --> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fcbe40a46e0 con 0x7fcbd0044110 2024-09-10T16:06:25.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbe17fa640 1 -- 172.21.15.164:0/1554684190 <== osd.1 v2:172.21.15.164:6816/1609205011 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fcbe40a46e0 con 0x7fcbd0044110 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7fcbd0044110 msgr2=0x7fcbd0046550 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7fcbd0044110 0x7fcbd0046550 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcbd003e290 msgr2=0x7fcbd0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcbd003e290 0x7fcbd0040750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcbd4007c90 tx=0x7fcbd4007be0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 10 monclient: shutdown 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 msgr2=0x7fcbe409e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbeadf3640 1 --2- 172.21.15.164:0/1554684190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcbe40a4c20 0x7fcbe409e0f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc00c7b0 tx=0x7fcbdc00cc70 comp rx=0 tx=0).stop 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.897+0000 7fcbe9390640 1 -- 172.21.15.164:0/1554684190 reap_dead start 2024-09-10T16:06:26.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.898+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 shutdown_connections 2024-09-10T16:06:26.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.898+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 >> 172.21.15.164:0/1554684190 conn(0x7fcbe40aacb0 msgr2=0x7fcbe40a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.898+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 shutdown_connections 2024-09-10T16:06:26.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.898+0000 7fcbeadf3640 1 -- 172.21.15.164:0/1554684190 wait complete. 2024-09-10T16:06:26.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.899+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f1404131e10 con 0x7f13e4048420 2024-09-10T16:06:26.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.899+0000 7f1400ff9640 1 -- 172.21.15.164:0/27463156 <== osd.0 v2:172.21.15.164:6800/3366597746 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f1404131e10 con 0x7f13e4048420 2024-09-10T16:06:26.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f13e4048420 msgr2=0x7f13e404a860 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f13e4048420 0x7f13e404a860 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e4042620 msgr2=0x7f13e4044ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e4042620 0x7f13e4044ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13f4009630 tx=0x7f13f402e000 comp rx=0 tx=0).stop 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 10 monclient: shutdown 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 msgr2=0x7f140412d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 --2- 172.21.15.164:0/27463156 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f140409b780 0x7f140412d5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f13ec00e880 tx=0x7f13ec00ed40 comp rx=0 tx=0).stop 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1403fff640 1 -- 172.21.15.164:0/27463156 reap_dead start 2024-09-10T16:06:26.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 --> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efff4131200 con 0x7efff40a4170 2024-09-10T16:06:26.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 shutdown_connections 2024-09-10T16:06:26.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 >> 172.21.15.164:0/27463156 conn(0x7f14040a5030 msgr2=0x7f14040a6430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 shutdown_connections 2024-09-10T16:06:26.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7f1409ce2640 1 -- 172.21.15.164:0/27463156 wait complete. 2024-09-10T16:06:26.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.900+0000 7efff0ff9640 1 -- 172.21.15.164:0/582728173 <== osd.5 v2:172.21.15.165:6816/3196812989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efff4131200 con 0x7efff40a4170 2024-09-10T16:06:26.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7efff40a4170 msgr2=0x7efff40a7a20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7efff40a4170 0x7efff40a7a20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effe003e290 msgr2=0x7effe0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effe003e290 0x7effe0040750 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7effe8007fd0 tx=0x7effe8007aa0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 10 monclient: shutdown 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 msgr2=0x7efff40a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 --2- 172.21.15.164:0/582728173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efff406d160 0x7efff40a1410 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7effe402a5c0 tx=0x7effe40046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efff8a63640 1 -- 172.21.15.164:0/582728173 reap_dead start 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 shutdown_connections 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 >> 172.21.15.164:0/582728173 conn(0x7efff40aacb0 msgr2=0x7efff40a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 shutdown_connections 2024-09-10T16:06:26.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.901+0000 7efffa4c6640 1 -- 172.21.15.164:0/582728173 wait complete. 2024-09-10T16:06:26.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.902+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 --> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fddec09df90 con 0x7fddd0044220 2024-09-10T16:06:26.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fdde97fa640 1 -- 172.21.15.164:0/3606740012 <== osd.4 v2:172.21.15.165:6808/592433527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fdde4007970 con 0x7fddd0044220 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7fddd0044220 msgr2=0x7fddd0046660 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7fddd0044220 0x7fddd0046660 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fddd003e3b0 msgr2=0x7fddd0040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fddd003e3b0 0x7fddd0040870 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fddd8007c90 tx=0x7fddd8007be0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 10 monclient: shutdown 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 msgr2=0x7fddec13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 --2- 172.21.15.164:0/3606740012 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fddec06cb70 0x7fddec13ad70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fdde000b3f0 tx=0x7fdde000b8b0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf17e5640 1 -- 172.21.15.164:0/3606740012 reap_dead start 2024-09-10T16:06:26.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 shutdown_connections 2024-09-10T16:06:26.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.903+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 >> 172.21.15.164:0/3606740012 conn(0x7fddec0aa900 msgr2=0x7fddec0a9e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.904+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 shutdown_connections 2024-09-10T16:06:26.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:25.904+0000 7fddf3248640 1 -- 172.21.15.164:0/3606740012 wait complete. 2024-09-10T16:06:26.010 INFO:teuthology.orchestra.run.smithi164.stdout:55834574853 2024-09-10T16:06:26.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T16:06:26.011 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:26.011 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T16:06:26.012 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:26.012 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T16:06:26.013 INFO:teuthology.orchestra.run.smithi164.stdout:55834574853 2024-09-10T16:06:26.014 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.166+0000 7f1df7fc2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.166+0000 7f1df7fc2640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.166+0000 7f1df7fc2640 10 monclient: monmap: 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.167256+0000 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.167256+0000 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.168 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.169 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.169 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.169 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.169 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.167+0000 7f1df7fc2640 1 Processor -- start 2024-09-10T16:06:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.167+0000 7f1df7fc2640 1 -- start start 2024-09-10T16:06:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.167+0000 7f1df7fc2640 10 monclient: init 2024-09-10T16:06:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-b con 0x7f1df006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1df006d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1df013d060 0x7f1df013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-c con 0x7f1df013d060 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1df009cb90 con 0x7f1df006d540 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1df009ce70 con 0x7f1df013d060 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1df009d150 con 0x7f1df006d160 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df7fc2640 10 monclient(hunting): authenticate will time out at 764.655529s 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47114/0 (socket says 172.21.15.164:47114) 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 -- 172.21.15.164:0/2072276271 learned_addr learned my addr 172.21.15.164:0/2072276271 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): get_auth_request con 0x7f1df006d160 auth_method 0 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 1 --2- 172.21.15.164:0/2072276271 >> v2:172.21.15.165:3300/0 conn(0x7f1df013d060 0x7f1df013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): get_auth_request con 0x7f1df013d060 auth_method 0 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df655f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 -- 172.21.15.164:0/2072276271 >> v2:172.21.15.165:3300/0 conn(0x7f1df013d060 msgr2=0x7f1df013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 --2- 172.21.15.164:0/2072276271 >> v2:172.21.15.165:3300/0 conn(0x7f1df013d060 0x7f1df013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.168+0000 7f1df5d5e640 1 -- 172.21.15.164:0/2072276271 >> v1:172.21.15.165:6789/0 conn(0x7f1df006d540 legacy=0x7f1df00a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.169+0000 7f1df5d5e640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.169+0000 7f1df5d5e640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.169+0000 7f1df5d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.169+0000 7f1df5d5e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.169+0000 7f1df5d5e640 1 -- 172.21.15.164:0/2072276271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df00a4df0 con 0x7f1df006d160 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df5d5e640 1 -- 172.21.15.164:0/2072276271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 msgr2=0x7f1df0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df5d5e640 1 --2- 172.21.15.164:0/2072276271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009870 tx=0x7f1dec02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91616295db9ba207 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 1 -- 172.21.15.164:0/2072276271 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec037070 con 0x7f1df006d160 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: dump: 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 1 -- 172.21.15.164:0/2072276271 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec032440 con 0x7f1df006d160 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 1 -- 172.21.15.164:0/2072276271 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec032810 con 0x7f1df006d160 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df4d5c640 10 monclient: dump: 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df7fc2640 5 monclient: authenticate success, global_id 4253 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df7fc2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.172+0000 7f1df7fc2640 4 set_mon_vals no callback set 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 10 monclient: shutdown 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 -- 172.21.15.164:0/2072276271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 msgr2=0x7f1df0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/2072276271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009870 tx=0x7f1dec02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 -- 172.21.15.164:0/2072276271 shutdown_connections 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/2072276271 >> v2:172.21.15.165:3300/0 conn(0x7f1df013d060 0x7f1df013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/2072276271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d160 0x7f1df0134e00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 -- 172.21.15.164:0/2072276271 >> 172.21.15.164:0/2072276271 conn(0x7f1df00aacb0 msgr2=0x7f1df00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 -- 172.21.15.164:0/2072276271 shutdown_connections 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.173+0000 7f1df7fc2640 1 -- 172.21.15.164:0/2072276271 wait complete. 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.174+0000 7f1df7fc2640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.174+0000 7f1df7fc2640 10 monclient: monmap: 2024-09-10T16:06:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.174+0000 7f1df7fc2640 1 Processor -- start 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.174+0000 7f1df7fc2640 1 -- start start 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.174+0000 7f1df7fc2640 10 monclient: init 2024-09-10T16:06:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-a con 0x7f1df006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 0x7f1df01cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-c con 0x7f1df006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1df013d060 0x7f1df01cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): picked mon.noname-b con 0x7f1df013d060 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1df013fb10 con 0x7f1df006d160 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1df013f990 con 0x7f1df013d060 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1df013fc90 con 0x7f1df006d540 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df7fc2640 10 monclient(hunting): authenticate will time out at 764.662125s 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35970/0 (socket says 172.21.15.164:35970) 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 -- 172.21.15.164:0/1964660524 learned_addr learned my addr 172.21.15.164:0/1964660524 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): get_auth_request con 0x7f1df006d160 auth_method 0 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 0x7f1df01cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): get_auth_request con 0x7f1df006d540 auth_method 0 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 -- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 msgr2=0x7f1df01cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 0x7f1df01cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 -- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1df013d060 msgr2=0x7f1df01cdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df655f640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1df013d060 0x7f1df01cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1df013d060 0x7f1df01cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 monclient: found mon.noname-a 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df5d5e640 1 -- 172.21.15.164:0/1964660524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1df013a240 con 0x7f1df006d160 2024-09-10T16:06:26.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.175+0000 7f1df555d640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 0x7f1df01cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.176+0000 7fa87f117640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.176+0000 7fa87f117640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.176+0000 7fa87f117640 10 monclient: monmap: 2024-09-10T16:06:26.187 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.187 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.177117+0000 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.177117+0000 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.176+0000 7fa87f117640 1 Processor -- start 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 -- start start 2024-09-10T16:06:26.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient: init 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa87806d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-b con 0x7fa87806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7f6321bbb640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8780a4c20 0x7fa87813f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-c con 0x7fa8780a4c20 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa878135510 con 0x7fa87806d160 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa87809cb90 con 0x7fa8780a4c20 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa87809ce70 con 0x7fa87806d540 2024-09-10T16:06:26.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa87f117640 10 monclient(hunting): authenticate will time out at 764.664864s 2024-09-10T16:06:26.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7f6321bbb640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47126/0 (socket says 172.21.15.164:47126) 2024-09-10T16:06:26.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 1 -- 172.21.15.164:0/4233900587 learned_addr learned my addr 172.21.15.164:0/4233900587 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 10 monclient(hunting): get_auth_request con 0x7fa87806d540 auth_method 0 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7fa877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.177+0000 7f6321bbb640 10 monclient: monmap: 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.178636+0000 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.178636+0000 2024-09-10T16:06:26.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.193 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df5d5e640 1 -- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 msgr2=0x7f1df01cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df5d5e640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009870 tx=0x7f1dec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 1 --2- 172.21.15.164:0/4233900587 >> v2:172.21.15.165:3300/0 conn(0x7fa8780a4c20 0x7fa87813f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): get_auth_request con 0x7fa8780a4c20 auth_method 0 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2024-09-10T16:06:26.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 1 -- 172.21.15.164:0/4233900587 >> v2:172.21.15.165:3300/0 conn(0x7fa8780a4c20 msgr2=0x7fa87813f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec037040 con 0x7f1df006d160 2024-09-10T16:06:26.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:26.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: dump: 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df7fc2640 5 monclient: authenticate success, global_id 4251 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df7fc2640 10 monclient: _renew_subs 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df7fc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1df013b0a0 con 0x7f1df006d160 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1dec042070 con 0x7f1df006d160 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 1 --2- 172.21.15.164:0/4233900587 >> v2:172.21.15.165:3300/0 conn(0x7fa8780a4c20 0x7fa87813f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 1 -- 172.21.15.164:0/4233900587 >> v1:172.21.15.165:6789/0 conn(0x7fa87806d160 legacy=0x7fa878134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7fa877fff640 1 -- 172.21.15.164:0/4233900587 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87809d150 con 0x7fa87806d540 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.178+0000 7f1de7fff640 4 set_mon_vals no callback set 2024-09-10T16:06:26.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1df7fc2640 10 monclient: _renew_subs 2024-09-10T16:06:26.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1df7fc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1df0137cd0 con 0x7f1df006d160 2024-09-10T16:06:26.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f6321bbb640 1 Processor -- start 2024-09-10T16:06:26.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f6321bbb640 1 -- start start 2024-09-10T16:06:26.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f6321bbb640 10 monclient: init 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1dec03c7e0 con 0x7f1df006d160 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1de67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.179+0000 7f1de67fc640 10 monclient: dump: 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.202 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.203 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:06:26.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f631c06d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:26.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f631c06d540 0x7f631c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f631c06d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:26.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1dec03c980 con 0x7f1df006d160 2024-09-10T16:06:26.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c0a4c20 0x7f631c13f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f631c0a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f631c135510 con 0x7f631c06d160 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f631c09cb90 con 0x7f631c06d540 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f631c09ce70 con 0x7f631c0a4c20 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f6321bbb640 10 monclient(hunting): authenticate will time out at 764.667442s 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f631c06d160 0x7f631c134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59286/0 (socket says 172.21.15.164:59286) 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631b7fe640 1 -- 172.21.15.164:0/658942711 learned_addr learned my addr 172.21.15.164:0/658942711 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:26.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631affd640 1 --2- 172.21.15.164:0/658942711 >> v2:172.21.15.165:3300/0 conn(0x7f631c06d540 0x7f631c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f1df7fc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f1df7fc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f1df7fc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631affd640 10 monclient(hunting): get_auth_request con 0x7f631c06d540 auth_method 0 2024-09-10T16:06:26.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f631affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.180+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1df006dfb0 con 0x7f1df006d160 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 1 -- 172.21.15.164:0/658942711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c0a4c20 msgr2=0x7f631c13f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 1 --2- 172.21.15.164:0/658942711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c0a4c20 0x7f631c13f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 1 -- 172.21.15.164:0/658942711 >> v1:172.21.15.164:6789/0 conn(0x7f631c06d160 legacy=0x7f631c134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 1 monclient: found mon.noname-c 2024-09-10T16:06:26.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f631affd640 1 -- 172.21.15.164:0/658942711 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c09d150 con 0x7f631c06d540 2024-09-10T16:06:26.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f1de67fc640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 0x7f1dd40408d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f1dec02ae00 con 0x7f1df006d160 2024-09-10T16:06:26.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa877fff640 1 -- 172.21.15.164:0/4233900587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 msgr2=0x7fa8780a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa877fff640 1 --2- 172.21.15.164:0/4233900587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa864009870 tx=0x7fa86402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b814d2525da7cd5c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f1df555d640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 0x7f1dd40408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7f1df555d640 10 monclient: get_auth_request con 0x7f1dd403e410 auth_method 0 2024-09-10T16:06:26.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 1 -- 172.21.15.164:0/4233900587 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864037070 con 0x7fa87806d540 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: dump: 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 1 -- 172.21.15.164:0/4233900587 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa864032640 con 0x7fa87806d540 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 1 -- 172.21.15.164:0/4233900587 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864032a10 con 0x7fa87806d540 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa8777fe640 10 monclient: dump: 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa87f117640 5 monclient: authenticate success, global_id 4259 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa87f117640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.181+0000 7fa87f117640 4 set_mon_vals no callback set 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 10 monclient: shutdown 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 -- 172.21.15.164:0/4233900587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 msgr2=0x7fa8780a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 --2- 172.21.15.164:0/4233900587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa864009870 tx=0x7fa86402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 -- 172.21.15.164:0/4233900587 shutdown_connections 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 --2- 172.21.15.164:0/4233900587 >> v2:172.21.15.165:3300/0 conn(0x7fa8780a4c20 0x7fa87813f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 --2- 172.21.15.164:0/4233900587 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a46e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 -- 172.21.15.164:0/4233900587 >> 172.21.15.164:0/4233900587 conn(0x7fa8780aacb0 msgr2=0x7fa8780ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 -- 172.21.15.164:0/4233900587 shutdown_connections 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.182+0000 7fa87f117640 1 -- 172.21.15.164:0/4233900587 wait complete. 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.183+0000 7fa87f117640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.183+0000 7fa87f117640 10 monclient: monmap: 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7fa87f117640 1 Processor -- start 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7fa87f117640 1 -- start start 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7fa87f117640 10 monclient: init 2024-09-10T16:06:26.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7f631affd640 1 -- 172.21.15.164:0/658942711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d540 msgr2=0x7f631c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7f631affd640 1 --2- 172.21.15.164:0/658942711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d540 0x7f631c0a46e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63040097a0 tx=0x7f6304029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dba534fd07f5b74 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.184+0000 7f1df555d640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 0x7f1dd40408d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1df013dcf0 tx=0x7f1de0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 1 -- 172.21.15.164:0/658942711 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6304037070 con 0x7f631c06d540 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: dump: 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.219 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f6321bbb640 5 monclient: authenticate success, global_id 4255 2024-09-10T16:06:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f6321bbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-a con 0x7fa87806d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 1 -- 172.21.15.164:0/658942711 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6304032640 con 0x7f631c06d540 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f631a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f6321bbb640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7f6321bbb640 4 set_mon_vals no callback set 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87ceb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87ceb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35978/0 (socket says 172.21.15.164:35978) 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87ceb3640 1 -- 172.21.15.164:0/2345460505 learned_addr learned my addr 172.21.15.164:0/2345460505 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-c con 0x7fa87806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa877fff640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8780a4c20 0x7fa8780a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient(hunting): picked mon.noname-b con 0x7fa8780a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.185+0000 7fa87f117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa87813fb20 con 0x7fa87806d160 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa87813f9a0 con 0x7fa8780a4c20 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa87813fca0 con 0x7fa87806d540 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8780a4c20 0x7fa8780a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 10 monclient: shutdown 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 -- 172.21.15.164:0/658942711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d540 msgr2=0x7f631c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 --2- 172.21.15.164:0/658942711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d540 0x7f631c0a46e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63040097a0 tx=0x7f6304029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f631a7fc640 1 -- 172.21.15.164:0/658942711 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f6304032a10 con 0x7f631c06d540 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f631a7fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 -- 172.21.15.164:0/658942711 shutdown_connections 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 --2- 172.21.15.164:0/658942711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c0a4c20 0x7f631c13f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 --2- 172.21.15.164:0/658942711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d540 0x7f631c0a46e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 -- 172.21.15.164:0/658942711 >> 172.21.15.164:0/658942711 conn(0x7f631c0aacb0 msgr2=0x7f631c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87f117640 10 monclient(hunting): authenticate will time out at 764.673492s 2024-09-10T16:06:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 -- 172.21.15.164:0/658942711 shutdown_connections 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): get_auth_request con 0x7fa87806d160 auth_method 0 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): get_auth_request con 0x7fa87806d540 auth_method 0 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): get_auth_request con 0x7fa8780a4c20 auth_method 0 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87ceb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa87d6b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2024-09-10T16:06:26.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 -- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8780a4c20 msgr2=0x7fa8780a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8780a4c20 0x7fa8780a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 -- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 msgr2=0x7fa8781cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 monclient: found mon.noname-c 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7fa877fff640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8780a4510 con 0x7fa87806d540 2024-09-10T16:06:26.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.187+0000 7fa87ceb3640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.186+0000 7f6321bbb640 1 -- 172.21.15.164:0/658942711 wait complete. 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.187+0000 7f6321bbb640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.187+0000 7f6321bbb640 10 monclient: monmap: 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.230 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f6321bbb640 1 Processor -- start 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f6321bbb640 1 -- start start 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f6321bbb640 10 monclient: init 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1df006dfb0 con 0x7f1df006d160 2024-09-10T16:06:26.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f1de67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.188+0000 7f1de67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:26.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d160 0x7f631c1cd3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-b con 0x7f631c06d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c06d540 0x7f631c0a3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-c con 0x7f631c06d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): picked mon.noname-a con 0x7f631c0a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f631c13fb20 con 0x7f631c0a4c20 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f631c13f9a0 con 0x7f631c06d160 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f631c13fca0 con 0x7f631c06d540 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f6321bbb640 10 monclient(hunting): authenticate will time out at 764.676220s 2024-09-10T16:06:26.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa877fff640 1 -- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 msgr2=0x7fa8780a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa877fff640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a3a90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa864031000 tx=0x7fa8640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35988/0 (socket says 172.21.15.164:35988) 2024-09-10T16:06:26.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 -- 172.21.15.164:0/1193468307 learned_addr learned my addr 172.21.15.164:0/1193468307 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): get_auth_request con 0x7f631c0a4c20 auth_method 0 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631b7fe640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d160 0x7f631c1cd3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631affd640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c06d540 0x7f631c0a3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631b7fe640 10 monclient(hunting): get_auth_request con 0x7f631c06d160 auth_method 0 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631affd640 10 monclient(hunting): get_auth_request con 0x7f631c06d540 auth_method 0 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient(hunting): handle_auth_done global_id 4257 payload 824 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 -- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c06d540 msgr2=0x7f631c0a3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c06d540 0x7f631c0a3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 -- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d160 msgr2=0x7f631c1cd3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d160 0x7f631c1cd3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 monclient: found mon.noname-a 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7f631bfff640 1 -- 172.21.15.164:0/1193468307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f631c1ce020 con 0x7f631c0a4c20 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa86404b070 con 0x7fa87806d540 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: dump: 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8640463f0 con 0x7fa87806d540 2024-09-10T16:06:26.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8640467c0 con 0x7fa87806d540 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa8757fa640 10 monclient: dump: 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.240 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa876ffd640 4 set_mon_vals no callback set 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 5 monclient: authenticate success, global_id 4262 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 10 monclient: _renew_subs 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8780a2560 con 0x7fa87806d540 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 10 monclient: _renew_subs 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.189+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8780a0d60 con 0x7fa87806d540 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.191+0000 7fa87f117640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.191+0000 7fa87f117640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.191+0000 7fa87f117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.191+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8780a46e0 con 0x7fa87806d540 2024-09-10T16:06:26.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa86404f3f0 con 0x7fa87806d540 2024-09-10T16:06:26.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7fa8757fa640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 0x7fa85c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fa86407ae80 con 0x7fa87806d540 2024-09-10T16:06:26.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f631bfff640 1 -- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 msgr2=0x7f631c09e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f631bfff640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f631400b3f0 tx=0x7f631400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314015020 con 0x7f631c0a4c20 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: dump: 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.243 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6314011a60 con 0x7f631c0a4c20 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6314011e50 con 0x7f631c0a4c20 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.192+0000 7f6313fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.193+0000 7f6313fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.193+0000 7f6313fff640 10 monclient: dump: 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.244 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.193+0000 7f6319ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:26.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7fa87ceb3640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 0x7fa85c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7fa87ceb3640 10 monclient: get_auth_request con 0x7fa85c03e260 auth_method 0 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 5 monclient: authenticate success, global_id 4257 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 10 monclient: _renew_subs 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f631c06e6b0 con 0x7f631c0a4c20 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 10 monclient: _renew_subs 2024-09-10T16:06:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.194+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f631c0a1c70 con 0x7f631c0a4c20 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.195+0000 7f6321bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.195+0000 7f6321bbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.195+0000 7f6321bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.195+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f631c0aa1e0 con 0x7f631c0a4c20 2024-09-10T16:06:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.196+0000 7fa87ceb3640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 0x7fa85c040720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0043e0 tx=0x7fa86c007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.197+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6314018020 con 0x7f631c0a4c20 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.198+0000 7f6313fff640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 0x7f62fc0408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.198+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f6314053190 con 0x7f631c0a4c20 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.198+0000 7f631b7fe640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 0x7f62fc0408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.198+0000 7f631b7fe640 10 monclient: get_auth_request con 0x7f62fc03e400 auth_method 0 2024-09-10T16:06:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.200+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa864043070 con 0x7fa87806d540 2024-09-10T16:06:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.200+0000 7fa8757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.200+0000 7fa8757fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.201+0000 7f631b7fe640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 0x7f62fc0408c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f630c007b70 tx=0x7f630c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.202+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f631401f020 con 0x7f631c0a4c20 2024-09-10T16:06:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.202+0000 7f6313fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.202+0000 7f6313fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.203+0000 7fc2380cd640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.203+0000 7fc2380cd640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.203+0000 7fc2380cd640 10 monclient: monmap: 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.204460+0000 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.204460+0000 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 1 Processor -- start 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 1 -- start start 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 10 monclient: init 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc23006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.204+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc230134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc230140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-b con 0x7fc23013e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc23009ce50 con 0x7fc230134f70 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc230135380 con 0x7fc23006cb70 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc23006db60 con 0x7fc23013e740 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc2380cd640 10 monclient(hunting): authenticate will time out at 764.692091s 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc235668640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc230134f70 0x7fc23013d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59294/0 (socket says 172.21.15.164:59294) 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc235668640 1 -- 172.21.15.164:0/1667303049 learned_addr learned my addr 172.21.15.164:0/1667303049 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 1 --2- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc230140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): get_auth_request con 0x7fc23013e740 auth_method 0 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 1 -- 172.21.15.164:0/1667303049 >> v1:172.21.15.165:6789/0 conn(0x7fc23006cb70 legacy=0x7fc2300a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 1 -- 172.21.15.164:0/1667303049 >> v1:172.21.15.164:6789/0 conn(0x7fc230134f70 legacy=0x7fc23013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.205+0000 7fc23666a640 1 -- 172.21.15.164:0/1667303049 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23009cb70 con 0x7fc23013e740 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.207+0000 7fbee1c9e640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.207+0000 7fbee1c9e640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.207+0000 7fbee1c9e640 10 monclient: monmap: 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.208607+0000 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.208607+0000 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.208+0000 7fbee1c9e640 1 Processor -- start 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.208+0000 7fc23666a640 1 -- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 msgr2=0x7fc230140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.208+0000 7fc23666a640 1 --2- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc230140b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc228009df0 tx=0x7fc22802a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a07fc9b7c93e9fc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.208+0000 7fbee1c9e640 1 -- start start 2024-09-10T16:06:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.208+0000 7fbee1c9e640 10 monclient: init 2024-09-10T16:06:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 1 -- 172.21.15.164:0/1667303049 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228007a30 con 0x7fc23013e740 2024-09-10T16:06:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: dump: 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 1 -- 172.21.15.164:0/1667303049 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc228007bd0 con 0x7fc23013e740 2024-09-10T16:06:26.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc234e67640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc2380cd640 5 monclient: authenticate success, global_id 4268 2024-09-10T16:06:26.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc2380cd640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fc2380cd640 4 set_mon_vals no callback set 2024-09-10T16:06:26.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbed40b63a0 0x7fbed40b6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-c con 0x7fbed40b63a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbed40257f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-b con 0x7fbed401fd80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed40209a0 con 0x7fbed40257f0 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbed40207a0 con 0x7fbed40b63a0 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbed4020c20 con 0x7fbed401fd80 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee1c9e640 10 monclient(hunting): authenticate will time out at 764.696517s 2024-09-10T16:06:26.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47172/0 (socket says 172.21.15.164:47172) 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 -- 172.21.15.164:0/3651771547 learned_addr learned my addr 172.21.15.164:0/3651771547 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): get_auth_request con 0x7fbed401fd80 auth_method 0 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 1 --2- 172.21.15.164:0/3651771547 >> v2:172.21.15.165:3300/0 conn(0x7fbed40b63a0 0x7fbed40b6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): get_auth_request con 0x7fbed40b63a0 auth_method 0 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient(hunting): handle_auth_done global_id 4271 payload 274 2024-09-10T16:06:26.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 -- 172.21.15.164:0/3651771547 >> v2:172.21.15.165:3300/0 conn(0x7fbed40b63a0 msgr2=0x7fbed40b6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 10 monclient: shutdown 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc234e67640 1 -- 172.21.15.164:0/1667303049 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc228031400 con 0x7fc23013e740 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc234e67640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc234e67640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc234e67640 10 monclient: dump: 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.265 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 -- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 msgr2=0x7fc230140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 --2- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc230140b30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fc228009df0 tx=0x7fc22802a4b0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 --2- 172.21.15.164:0/3651771547 >> v2:172.21.15.165:3300/0 conn(0x7fbed40b63a0 0x7fbed40b6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 -- 172.21.15.164:0/3651771547 >> v1:172.21.15.165:6789/0 conn(0x7fbed40257f0 legacy=0x7fbed401f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 -- 172.21.15.164:0/1667303049 shutdown_connections 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 --2- 172.21.15.164:0/1667303049 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc230140b30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 -- 172.21.15.164:0/1667303049 >> 172.21.15.164:0/1667303049 conn(0x7fc2300aa900 msgr2=0x7fc2300aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.209+0000 7fbee149d640 1 -- 172.21.15.164:0/3651771547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed4021cc0 con 0x7fbed401fd80 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 -- 172.21.15.164:0/1667303049 shutdown_connections 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.210+0000 7fc2380cd640 1 -- 172.21.15.164:0/1667303049 wait complete. 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.211+0000 7fc2380cd640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.211+0000 7fc2380cd640 10 monclient: monmap: 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.269 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.212+0000 7fc2380cd640 1 Processor -- start 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.212+0000 7fc2380cd640 1 -- start start 2024-09-10T16:06:26.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.212+0000 7fc2380cd640 10 monclient: init 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.212+0000 7fbee149d640 1 -- 172.21.15.164:0/3651771547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 msgr2=0x7fbed40c16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.212+0000 7fbee149d640 1 --2- 172.21.15.164:0/3651771547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbedc07cd80 tx=0x7fbedc0a6b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a360595187913ad5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 1 -- 172.21.15.164:0/3651771547 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0a8b30 con 0x7fbed401fd80 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: dump: 2024-09-10T16:06:26.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbee1c9e640 5 monclient: authenticate success, global_id 4271 2024-09-10T16:06:26.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbee1c9e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 1 -- 172.21.15.164:0/3651771547 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbedc0a8d10 con 0x7fbed401fd80 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 1 -- 172.21.15.164:0/3651771547 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc0a9630 con 0x7fbed401fd80 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbee1c9e640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbee1c9e640 4 set_mon_vals no callback set 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fbedb7fe640 10 monclient: dump: 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.274 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.274 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.274 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.274 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.274 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 0x7fc230131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-b con 0x7fc23006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-a con 0x7fc230134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36000/0 (socket says 172.21.15.164:36000) 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): picked mon.noname-c con 0x7fc23013e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc2300a4db0 con 0x7fc230134f70 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc23006cf50 con 0x7fc23006cb70 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc230141070 con 0x7fc23013e740 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc23666a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc23666a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47180/0 (socket says 172.21.15.164:47180) 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 1 -- 172.21.15.164:0/2153441501 learned_addr learned my addr 172.21.15.164:0/2153441501 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc2380cd640 10 monclient(hunting): authenticate will time out at 764.700966s 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 10 monclient(hunting): get_auth_request con 0x7fc230134f70 auth_method 0 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235668640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.213+0000 7fc235e69640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 0x7fc230131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): get_auth_request con 0x7fc23013e740 auth_method 0 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): get_auth_request con 0x7fc23006cb70 auth_method 0 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235668640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235668640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235668640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc23666a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient(hunting): handle_auth_done global_id 4267 payload 824 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 10 monclient: shutdown 2024-09-10T16:06:26.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 -- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 msgr2=0x7fc23013bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 -- 172.21.15.164:0/3651771547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 msgr2=0x7fbed40c16a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/3651771547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbedc07cd80 tx=0x7fbedc0a6b00 comp rx=0 tx=0).stop 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 -- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 msgr2=0x7fc23013b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fc235e69640 1 -- 172.21.15.164:0/2153441501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc23009eb30 con 0x7fc23006cb70 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 -- 172.21.15.164:0/3651771547 shutdown_connections 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/3651771547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed401fd80 0x7fbed40c16a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/3651771547 >> v2:172.21.15.165:3300/0 conn(0x7fbed40b63a0 0x7fbed40b6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 -- 172.21.15.164:0/3651771547 >> 172.21.15.164:0/3651771547 conn(0x7fbed4006340 msgr2=0x7fbed4006750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 -- 172.21.15.164:0/3651771547 shutdown_connections 2024-09-10T16:06:26.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.214+0000 7fbee1c9e640 1 -- 172.21.15.164:0/3651771547 wait complete. 2024-09-10T16:06:26.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.215+0000 7fc235668640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.215+0000 7fc23666a640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.215+0000 7fbee1c9e640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.215+0000 7fbee1c9e640 10 monclient: monmap: 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.216+0000 7fbee1c9e640 1 Processor -- start 2024-09-10T16:06:26.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.216+0000 7fbee1c9e640 1 -- start start 2024-09-10T16:06:26.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.216+0000 7fbee1c9e640 10 monclient: init 2024-09-10T16:06:26.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc235e69640 1 -- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 msgr2=0x7fc230131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc235e69640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 0x7fc230131710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b3e0 tx=0x7fc22c00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22c015020 con 0x7fc23006cb70 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: dump: 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.287 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.288 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:26.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc22c011a40 con 0x7fc23006cb70 2024-09-10T16:06:26.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-b con 0x7fbed401fd80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 5 monclient: authenticate success, global_id 4267 2024-09-10T16:06:26.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-a con 0x7fbed40257f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 10 monclient: _renew_subs 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 1 -- 172.21.15.164:0/2153441501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc23009de90 con 0x7fc23006cb70 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36008/0 (socket says 172.21.15.164:36008) 2024-09-10T16:06:26.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:44354/0 (socket says 172.21.15.164:44354) 2024-09-10T16:06:26.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc22c011e10 con 0x7fc23006cb70 2024-09-10T16:06:26.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47182/0 (socket says 172.21.15.164:47182) 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2267fc640 10 monclient: dump: 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.293 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.294 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): picked mon.noname-c con 0x7fbed40b63a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbed40275e0 con 0x7fbed40257f0 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbed4027460 con 0x7fbed401fd80 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbed40233a0 con 0x7fbed40b63a0 2024-09-10T16:06:26.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc227fff640 4 set_mon_vals no callback set 2024-09-10T16:06:26.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee1c9e640 10 monclient(hunting): authenticate will time out at 764.705580s 2024-09-10T16:06:26.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 10 monclient: _renew_subs 2024-09-10T16:06:26.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.217+0000 7fc2380cd640 1 -- 172.21.15.164:0/2153441501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc230140b40 con 0x7fc23006cb70 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 1 -- 172.21.15.164:0/1640773964 learned_addr learned my addr 172.21.15.164:0/1640773964 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): get_auth_request con 0x7fbed40257f0 auth_method 0 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): get_auth_request con 0x7fbed401fd80 auth_method 0 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): get_auth_request con 0x7fbed40b63a0 auth_method 0 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbedbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee149d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.218+0000 7fbee0c9c640 10 monclient(hunting): handle_auth_done global_id 4270 payload 824 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 -- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 msgr2=0x7fbed40b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 -- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 msgr2=0x7fbed40b5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fc213fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fc213fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fc213fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc20c005080 con 0x7fc23006cb70 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 monclient: found mon.noname-b 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee0c9c640 1 -- 172.21.15.164:0/1640773964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed40252f0 con 0x7fbed401fd80 2024-09-10T16:06:26.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc22c018020 con 0x7fc23006cb70 2024-09-10T16:06:26.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbedbfff640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.219+0000 7fbee149d640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:26.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.220+0000 7fc2267fc640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 0x7fc214040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.220+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fc22c052a50 con 0x7fc23006cb70 2024-09-10T16:06:26.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.220+0000 7fc235668640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 0x7fc214040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.220+0000 7fc235668640 10 monclient: get_auth_request con 0x7fc21403e240 auth_method 0 2024-09-10T16:06:26.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.221+0000 7fbee0c9c640 1 -- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 msgr2=0x7fbed414e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.221+0000 7fbee0c9c640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009c00 tx=0x7fbed000b510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.222+0000 7fc235668640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 0x7fc214040700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc220007d50 tx=0x7fc220007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed001e070 con 0x7fbed401fd80 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: dump: 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.304 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbed000be30 con 0x7fbed401fd80 2024-09-10T16:06:26.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed0019630 con 0x7fbed401fd80 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbed97fa640 10 monclient: dump: 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.306 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 5 monclient: authenticate success, global_id 4270 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 10 monclient: _renew_subs 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed40a47b0 con 0x7fbed401fd80 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbedaffd640 4 set_mon_vals no callback set 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 10 monclient: _renew_subs 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed40a6270 con 0x7fbed401fd80 2024-09-10T16:06:26.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc22c01f020 con 0x7fc23006cb70 2024-09-10T16:06:26.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fc2267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.225+0000 7fc2267fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.226+0000 7fbee1c9e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.226+0000 7fbee1c9e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.226+0000 7fbee1c9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.226+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed40a5370 con 0x7fbed401fd80 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.227+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbed0021070 con 0x7fbed401fd80 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.227+0000 7fbed97fa640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 0x7fbec0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.227+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fbed0056640 con 0x7fbed401fd80 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.227+0000 7fbedbfff640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 0x7fbec0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.227+0000 7fbedbfff640 10 monclient: get_auth_request con 0x7fbec003e260 auth_method 0 2024-09-10T16:06:26.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.230+0000 7fbedbfff640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 0x7fbec0040720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbed40250c0 tx=0x7fbec8007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.232+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbed00223b0 con 0x7fbed401fd80 2024-09-10T16:06:26.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.232+0000 7fbed97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.232+0000 7fbed97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.238+0000 7f01421c2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.238+0000 7f01421c2640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.238+0000 7f01421c2640 10 monclient: monmap: 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.318 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:26.239770+0000 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:26.239770+0000 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.239+0000 7f01421c2640 1 Processor -- start 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.239+0000 7f01421c2640 1 -- start start 2024-09-10T16:06:26.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.239+0000 7f01421c2640 10 monclient: init 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f013c06d160 0x7f013c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-c con 0x7f013c06d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f013c06d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 0x7f013c13f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-a con 0x7f013c13d060 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f013c09ce70 con 0x7f013c06d540 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f013c09cc20 con 0x7f013c13d060 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f013c09d150 con 0x7f013c06d160 2024-09-10T16:06:26.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f01421c2640 10 monclient(hunting): authenticate will time out at 764.727627s 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 0x7f013c13f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 0x7f013c13f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36018/0 (socket says 172.21.15.164:36018) 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 1 -- 172.21.15.164:0/1994409334 learned_addr learned my addr 172.21.15.164:0/1994409334 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): get_auth_request con 0x7f013c13d060 auth_method 0 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013affd640 1 --1- 172.21.15.164:0/1994409334 >> v1:172.21.15.164:6789/0 conn(0x7f013c06d540 0x7f013c0a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59306/0 (socket says 172.21.15.164:59306) 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 1 --2- 172.21.15.164:0/1994409334 >> v2:172.21.15.165:3300/0 conn(0x7f013c06d160 0x7f013c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): get_auth_request con 0x7f013c06d160 auth_method 0 2024-09-10T16:06:26.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.240+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 10 monclient(hunting): handle_auth_done global_id 4273 payload 274 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 1 -- 172.21.15.164:0/1994409334 >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 msgr2=0x7f013c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 1 --2- 172.21.15.164:0/1994409334 >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 0x7f013c13f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 1 -- 172.21.15.164:0/1994409334 >> v1:172.21.15.164:6789/0 conn(0x7f013c06d540 legacy=0x7f013c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:26.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.241+0000 7f013b7fe640 1 -- 172.21.15.164:0/1994409334 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c0a4df0 con 0x7f013c06d160 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013b7fe640 1 -- 172.21.15.164:0/1994409334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d160 msgr2=0x7f013c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013b7fe640 1 --2- 172.21.15.164:0/1994409334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d160 0x7f013c134e00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f013c09cda0 tx=0x7f012c029fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d7aab194b05eef3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 1 -- 172.21.15.164:0/1994409334 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c037070 con 0x7f013c06d160 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: dump: 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.323 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 1 -- 172.21.15.164:0/1994409334 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f012c032440 con 0x7f013c06d160 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 1 -- 172.21.15.164:0/1994409334 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f012c032810 con 0x7f013c06d160 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f013a7fc640 10 monclient: dump: 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.325 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 5 monclient: authenticate success, global_id 4273 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 4 set_mon_vals no callback set 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 10 monclient: shutdown 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 1 -- 172.21.15.164:0/1994409334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d160 msgr2=0x7f013c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.244+0000 7f01421c2640 1 --2- 172.21.15.164:0/1994409334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d160 0x7f013c134e00 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f013c09cda0 tx=0x7f012c029fe0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 -- 172.21.15.164:0/1994409334 shutdown_connections 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 --2- 172.21.15.164:0/1994409334 >> v2:172.21.15.164:3300/0 conn(0x7f013c13d060 0x7f013c13f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 --2- 172.21.15.164:0/1994409334 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d160 0x7f013c134e00 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 -- 172.21.15.164:0/1994409334 >> 172.21.15.164:0/1994409334 conn(0x7f013c0aacb0 msgr2=0x7f013c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 -- 172.21.15.164:0/1994409334 shutdown_connections 2024-09-10T16:06:26.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 1 -- 172.21.15.164:0/1994409334 wait complete. 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 10 monclient: build_initial_monmap 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.245+0000 7f01421c2640 10 monclient: monmap: 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 Processor -- start 2024-09-10T16:06:26.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 -- start start 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient: init 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 0x7f013c12e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-a con 0x7f013c06d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d540 0x7f013c135d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-b con 0x7f013c06d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 0x7f013c1362b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): picked mon.noname-c con 0x7f013c13d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f013c13fb10 con 0x7f013c06d160 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f013c13f990 con 0x7f013c06d540 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f013c13fc90 con 0x7f013c13d060 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f01421c2640 10 monclient(hunting): authenticate will time out at 764.733821s 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 0x7f013c12e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 0x7f013c1362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 0x7f013c12e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36020/0 (socket says 172.21.15.164:36020) 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 1 -- 172.21.15.164:0/1921226910 learned_addr learned my addr 172.21.15.164:0/1921226910 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 10 monclient(hunting): get_auth_request con 0x7f013c06d160 auth_method 0 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013bfff640 10 monclient(hunting): get_auth_request con 0x7f013c13d060 auth_method 0 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.246+0000 7f013bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013affd640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d540 0x7f013c135d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013affd640 10 monclient(hunting): get_auth_request con 0x7f013c06d540 auth_method 0 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:26.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 -- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d540 msgr2=0x7f013c135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d540 0x7f013c135d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 -- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 msgr2=0x7f013c12e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 0x7f013c12e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 monclient: found mon.noname-c 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.247+0000 7f013bfff640 1 -- 172.21.15.164:0/1921226910 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f013c137590 con 0x7f013c13d060 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f013bfff640 1 -- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 msgr2=0x7f013c1362b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f013bfff640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 0x7f013c1362b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f013400ef90 tx=0x7f013400c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134003990 con 0x7f013c13d060 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 10 monclient: dump: 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f0123fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 5 monclient: authenticate success, global_id 4280 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 10 monclient: _renew_subs 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 1 -- 172.21.15.164:0/1921226910 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f013c138ac0 con 0x7f013c13d060 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 10 monclient: _renew_subs 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.249+0000 7f01421c2640 1 -- 172.21.15.164:0/1921226910 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f013c1367f0 con 0x7f013c13d060 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0134003b30 con 0x7f013c13d060 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0134004030 con 0x7f013c13d060 2024-09-10T16:06:26.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0123fff640 10 monclient: dump: 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.250+0000 7f0139ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:26.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0134005d10 con 0x7f013c13d060 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0121ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0121ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0123fff640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 0x7f0128048ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f01340531f0 con 0x7f013c13d060 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0121ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.251+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f013c06a220 con 0x7f013c13d060 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.252+0000 7f013b7fe640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 0x7f0128048ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.252+0000 7f013b7fe640 10 monclient: get_auth_request con 0x7f0128046a20 auth_method 0 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.254+0000 7f013b7fe640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 0x7f0128048ee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f012c009560 tx=0x7f012c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.260+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f013c06a220 con 0x7f013c13d060 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.260+0000 7f0123fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:26.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.260+0000 7f0123fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:26.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc213fff640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:26.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc213fff640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:26.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc213fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fc20c005370 con 0x7fc23006cb70 2024-09-10T16:06:26.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc2267fc640 1 -- 172.21.15.164:0/2153441501 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc23009eb30 con 0x7fc23006cb70 2024-09-10T16:06:26.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc2267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:26.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.430+0000 7fc2267fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.433 INFO:teuthology.orchestra.run.smithi164.stdout:51539607556 2024-09-10T16:06:26.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 msgr2=0x7fc214040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 0x7fc214040700 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc220007d50 tx=0x7fc220007ca0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 10 monclient: shutdown 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 msgr2=0x7fc230131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 0x7fc230131710 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc22c00b3e0 tx=0x7fc22c00b8a0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 shutdown_connections 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc21403e240 0x7fc214040700 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc220007d50 tx=0x7fc220007ca0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc23013e740 0x7fc23013bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc230134f70 0x7fc23013b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 --2- 172.21.15.164:0/2153441501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc23006cb70 0x7fc230131710 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 >> 172.21.15.164:0/2153441501 conn(0x7fc2300aa900 msgr2=0x7fc230132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 shutdown_connections 2024-09-10T16:06:26.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.433+0000 7fc213fff640 1 -- 172.21.15.164:0/2153441501 wait complete. 2024-09-10T16:06:26.447 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.2 2024-09-10T16:06:26.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.461+0000 7f0121ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:26.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.461+0000 7f0121ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:26.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.461+0000 7f0121ffb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.461+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f013c134e00 con 0x7f013c13d060 2024-09-10T16:06:26.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.462+0000 7f0123fff640 1 -- 172.21.15.164:0/1921226910 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0134022020 con 0x7f013c13d060 2024-09-10T16:06:26.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.462+0000 7f0123fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:26.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.462+0000 7f0123fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.464 INFO:teuthology.orchestra.run.smithi164.stdout:55834574851 2024-09-10T16:06:26.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 msgr2=0x7f0128048ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 0x7f0128048ee0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f012c009560 tx=0x7f012c02e000 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 10 monclient: shutdown 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 msgr2=0x7f013c1362b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 0x7f013c1362b0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f013400ef90 tx=0x7f013400c490 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 shutdown_connections 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0128046a20 0x7f0128048ee0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f013c13d060 0x7f013c1362b0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f013c06d540 0x7f013c135d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 --2- 172.21.15.164:0/1921226910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f013c06d160 0x7f013c12e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 >> 172.21.15.164:0/1921226910 conn(0x7f013c0aacb0 msgr2=0x7f013c132800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 shutdown_connections 2024-09-10T16:06:26.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.464+0000 7f0121ffb640 1 -- 172.21.15.164:0/1921226910 wait complete. 2024-09-10T16:06:26.476 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.4 2024-09-10T16:06:26.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.552+0000 7f6321bbb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:26.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.552+0000 7f6321bbb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:26.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.552+0000 7f6321bbb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.552+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f631c0770b0 con 0x7f631c0a4c20 2024-09-10T16:06:26.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.553+0000 7f6313fff640 1 -- 172.21.15.164:0/1193468307 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6314021d10 con 0x7f631c0a4c20 2024-09-10T16:06:26.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.553+0000 7f6313fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:26.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.553+0000 7f6313fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.555 INFO:teuthology.orchestra.run.smithi164.stdout:55834574851 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 msgr2=0x7f62fc0408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 0x7f62fc0408c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f630c007b70 tx=0x7f630c009040 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 10 monclient: shutdown 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 msgr2=0x7f631c09e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f631400b3f0 tx=0x7f631400b8b0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 shutdown_connections 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62fc03e400 0x7f62fc0408c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f631c0a4c20 0x7f631c09e9e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f631c06d540 0x7f631c0a3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 --2- 172.21.15.164:0/1193468307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f631c06d160 0x7f631c1cd3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 >> 172.21.15.164:0/1193468307 conn(0x7f631c0aacb0 msgr2=0x7f631c12e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 shutdown_connections 2024-09-10T16:06:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.555+0000 7f6321bbb640 1 -- 172.21.15.164:0/1193468307 wait complete. 2024-09-10T16:06:26.567 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.1 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.577+0000 7fbee1c9e640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.577+0000 7fbee1c9e640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.577+0000 7fbee1c9e640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.577+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbed40b5da0 con 0x7fbed401fd80 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.578+0000 7fbed97fa640 1 -- 172.21.15.164:0/1640773964 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbed0056030 con 0x7fbed401fd80 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.578+0000 7fbed97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.578+0000 7fbed97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.580 INFO:teuthology.orchestra.run.smithi164.stdout:47244640259 2024-09-10T16:06:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 msgr2=0x7fbec0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 0x7fbec0040720 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fbed40250c0 tx=0x7fbec8007ef0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 10 monclient: shutdown 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 msgr2=0x7fbed414e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fbed0009c00 tx=0x7fbed000b510 comp rx=0 tx=0).stop 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 shutdown_connections 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbec003e260 0x7fbec0040720 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed40b63a0 0x7fbed40b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed40257f0 0x7fbed40b5110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 --2- 172.21.15.164:0/1640773964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed401fd80 0x7fbed414e580 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 >> 172.21.15.164:0/1640773964 conn(0x7fbed4006340 msgr2=0x7fbed40b9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 shutdown_connections 2024-09-10T16:06:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.580+0000 7fbee1c9e640 1 -- 172.21.15.164:0/1640773964 wait complete. 2024-09-10T16:06:26.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.3 2024-09-10T16:06:26.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1df7fc2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:26.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1df7fc2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:26.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1df7fc2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f1df0134e00 con 0x7f1df006d160 2024-09-10T16:06:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1de67fc640 1 -- 172.21.15.164:0/1964660524 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1df006dfb0 con 0x7f1df006d160 2024-09-10T16:06:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1de67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.599+0000 7f1de67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.601 INFO:teuthology.orchestra.run.smithi164.stdout:51539607556 2024-09-10T16:06:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 msgr2=0x7f1dd40408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 0x7f1dd40408d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1df013dcf0 tx=0x7f1de0009040 comp rx=0 tx=0).stop 2024-09-10T16:06:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 10 monclient: shutdown 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 msgr2=0x7f1df01cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1dec009870 tx=0x7f1dec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 shutdown_connections 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1dd403e410 0x7f1dd40408d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1df013d060 0x7f1df01cdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1df006d540 0x7f1df01cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 --2- 172.21.15.164:0/1964660524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1df006d160 0x7f1df01cd170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 >> 172.21.15.164:0/1964660524 conn(0x7f1df00aacb0 msgr2=0x7f1df0132760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 shutdown_connections 2024-09-10T16:06:26.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.601+0000 7f1df7fc2640 1 -- 172.21.15.164:0/1964660524 wait complete. 2024-09-10T16:06:26.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa87f117640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:26.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa87f117640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:26.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa87f117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:26.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fa8780a1f00 con 0x7fa87806d540 2024-09-10T16:06:26.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa8757fa640 1 -- 172.21.15.164:0/2345460505 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa86404e360 con 0x7fa87806d540 2024-09-10T16:06:26.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa8757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:26.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.607+0000 7fa8757fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:26.610 INFO:teuthology.orchestra.run.smithi164.stdout:51539607556 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 msgr2=0x7fa85c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 0x7fa85c040720 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa86c0043e0 tx=0x7fa86c007ef0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 10 monclient: shutdown 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 msgr2=0x7fa8780a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a3a90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa864031000 tx=0x7fa8640046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 shutdown_connections 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e260 0x7fa85c040720 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8780a4c20 0x7fa8780a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87806d540 0x7fa8780a3a90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 --2- 172.21.15.164:0/2345460505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa87806d160 0x7fa8781cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 >> 172.21.15.164:0/2345460505 conn(0x7fa8780aacb0 msgr2=0x7fa87812e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 shutdown_connections 2024-09-10T16:06:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:26.610+0000 7fa87f117640 1 -- 172.21.15.164:0/2345460505 wait complete. 2024-09-10T16:06:26.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.5 2024-09-10T16:06:26.622 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607556 for osd.0 2024-09-10T16:06:27.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T16:06:27.477 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T16:06:27.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.554+0000 7f2310465640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.554+0000 7f2310465640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.554+0000 7f2310465640 10 monclient: monmap: 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.555135+0000 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.555135+0000 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.556 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.554+0000 7f2310465640 1 Processor -- start 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 1 -- start start 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient: init 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f230806cb70 0x7f2308134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient(hunting): picked mon.noname-a con 0x7f230806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23080a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23080a4ab0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:27.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f230809ce50 con 0x7f23080a46a0 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.555+0000 7f2310465640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2308138ac0 con 0x7f23080a4ab0 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f2310465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f2310465640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f230806db60 con 0x7f230806cb70 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f2310465640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f2310465640 10 monclient(hunting): authenticate will time out at 766.043016s 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230da00640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f23080a46a0 0x7f2308137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59320/0 (socket says 172.21.15.164:59320) 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230da00640 1 -- 172.21.15.164:0/2527199965 learned_addr learned my addr 172.21.15.164:0/2527199965 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 1 --2- 172.21.15.164:0/2527199965 >> v2:172.21.15.164:3300/0 conn(0x7f230806cb70 0x7f2308134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): get_auth_request con 0x7f230806cb70 auth_method 0 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 1 -- 172.21.15.164:0/2527199965 >> v1:172.21.15.165:6789/0 conn(0x7f23080a4ab0 legacy=0x7f23081383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 1 -- 172.21.15.164:0/2527199965 >> v1:172.21.15.164:6789/0 conn(0x7f23080a46a0 legacy=0x7f2308137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.556+0000 7f230e201640 1 -- 172.21.15.164:0/2527199965 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230809cb70 con 0x7f230806cb70 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230e201640 1 -- 172.21.15.164:0/2527199965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230806cb70 msgr2=0x7f2308134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230e201640 1 --2- 172.21.15.164:0/2527199965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230806cb70 0x7f2308134f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23000097d0 tx=0x7f230002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=472dfa1f1a497992 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 1 -- 172.21.15.164:0/2527199965 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300037070 con 0x7f230806cb70 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: dump: 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.566 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 1 -- 172.21.15.164:0/2527199965 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2300032440 con 0x7f230806cb70 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f2310465640 5 monclient: authenticate success, global_id 4272 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f2310465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f2310465640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 1 -- 172.21.15.164:0/2527199965 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2300032810 con 0x7f230806cb70 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f2310465640 4 set_mon_vals no callback set 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.559+0000 7f230d1ff640 10 monclient: dump: 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 10 monclient: shutdown 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 -- 172.21.15.164:0/2527199965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230806cb70 msgr2=0x7f2308134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 --2- 172.21.15.164:0/2527199965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230806cb70 0x7f2308134f70 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f23000097d0 tx=0x7f230002a000 comp rx=0 tx=0).stop 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 -- 172.21.15.164:0/2527199965 shutdown_connections 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 --2- 172.21.15.164:0/2527199965 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f230806cb70 0x7f2308134f70 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 -- 172.21.15.164:0/2527199965 >> 172.21.15.164:0/2527199965 conn(0x7f23080aa900 msgr2=0x7f23080aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 -- 172.21.15.164:0/2527199965 shutdown_connections 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.560+0000 7f2310465640 1 -- 172.21.15.164:0/2527199965 wait complete. 2024-09-10T16:06:27.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient: monmap: 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 Processor -- start 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 -- start start 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient: init 2024-09-10T16:06:27.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230806cb70 0x7f23081312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): picked mon.noname-b con 0x7f230806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 0x7f2308131800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): picked mon.noname-c con 0x7f23080a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): picked mon.noname-a con 0x7f23080a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2308141050 con 0x7f23080a4ab0 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f230806cf50 con 0x7f230806cb70 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23081411d0 con 0x7f23080a46a0 2024-09-10T16:06:27.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f2310465640 10 monclient(hunting): authenticate will time out at 766.048871s 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f230ea02640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.561+0000 7f230da00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 0x7f2308131800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36040/0 (socket says 172.21.15.164:36040) 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 -- 172.21.15.164:0/1011511749 learned_addr learned my addr 172.21.15.164:0/1011511749 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): get_auth_request con 0x7f23080a4ab0 auth_method 0 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): get_auth_request con 0x7f23080a46a0 auth_method 0 2024-09-10T16:06:27.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230e201640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230806cb70 0x7f23081312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230e201640 10 monclient(hunting): get_auth_request con 0x7f230806cb70 auth_method 0 2024-09-10T16:06:27.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230e201640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230e201640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230e201640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient(hunting): handle_auth_done global_id 4275 payload 824 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 -- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 msgr2=0x7f2308131800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 0x7f2308131800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 -- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230806cb70 msgr2=0x7f23081312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230806cb70 0x7f23081312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230ea02640 1 -- 172.21.15.164:0/1011511749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f230813a030 con 0x7f23080a4ab0 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.562+0000 7f230da00640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 0x7f2308131800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f230ea02640 1 -- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 msgr2=0x7f23080a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f230ea02640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c410 tx=0x7f22f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f80119a0 con 0x7f23080a4ab0 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: dump: 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f8017070 con 0x7f23080a4ab0 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 5 monclient: authenticate success, global_id 4275 2024-09-10T16:06:27.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f8011d80 con 0x7f23080a4ab0 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 10 monclient: _renew_subs 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f230806e720 con 0x7f23080a4ab0 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2306ffd640 10 monclient: dump: 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f230c9fe640 4 set_mon_vals no callback set 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 10 monclient: _renew_subs 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.564+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2308138db0 con 0x7f23080a4ab0 2024-09-10T16:06:27.578 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f22f801c460 con 0x7f23080a4ab0 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2306ffd640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 0x7f22f40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f22f8052750 con 0x7f23080a4ab0 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2310465640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2310465640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2310465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.565+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2308134f70 con 0x7f23080a4ab0 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.566+0000 7f230e201640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 0x7f22f40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.566+0000 7f230e201640 10 monclient: get_auth_request con 0x7f22f403e210 auth_method 0 2024-09-10T16:06:27.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.568+0000 7f230e201640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 0x7f22f40406d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f23000097a0 tx=0x7f230002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.570+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f22f8021020 con 0x7f23080a4ab0 2024-09-10T16:06:27.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.570+0000 7f2306ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.570+0000 7f2306ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.594 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T16:06:27.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.594+0000 7fdd3aca9640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.594+0000 7fdd3aca9640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.594+0000 7fdd3aca9640 10 monclient: monmap: 2024-09-10T16:06:27.596 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.595503+0000 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.595503+0000 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.597 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.598 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.598 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 1 Processor -- start 2024-09-10T16:06:27.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 1 -- start start 2024-09-10T16:06:27.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient: init 2024-09-10T16:06:27.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 0x7fdd34134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd3406d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-b con 0x7fdd3406d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd340a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:27.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.595+0000 7fdd3aca9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd3409cb90 con 0x7fdd340a4c20 2024-09-10T16:06:27.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdd3409ce70 con 0x7fdd3406d160 2024-09-10T16:06:27.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdd3409d150 con 0x7fdd3406d540 2024-09-10T16:06:27.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd3aca9640 10 monclient(hunting): authenticate will time out at 766.083021s 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47212/0 (socket says 172.21.15.164:47212) 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 0x7fdd34134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 -- 172.21.15.164:0/1465800874 learned_addr learned my addr 172.21.15.164:0/1465800874 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): get_auth_request con 0x7fdd3406d540 auth_method 0 2024-09-10T16:06:27.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): get_auth_request con 0x7fdd3406d160 auth_method 0 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient(hunting): handle_auth_done global_id 4286 payload 274 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 -- 172.21.15.164:0/1465800874 >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 msgr2=0x7fdd34134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 --2- 172.21.15.164:0/1465800874 >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 0x7fdd34134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 -- 172.21.15.164:0/1465800874 >> v1:172.21.15.164:6789/0 conn(0x7fdd340a4c20 legacy=0x7fdd34137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 monclient: found mon.noname-b 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd33fff640 1 -- 172.21.15.164:0/1465800874 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd34135340 con 0x7fdd3406d540 2024-09-10T16:06:27.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.596+0000 7fdd38a45640 1 --2- 172.21.15.164:0/1465800874 >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 0x7fdd34134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:27.616 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T16:06:27.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd33fff640 1 -- 172.21.15.164:0/1465800874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 msgr2=0x7fdd340a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd33fff640 1 --2- 172.21.15.164:0/1465800874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009870 tx=0x7fdd2002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f0d2b607701af5e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 1 -- 172.21.15.164:0/1465800874 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20037070 con 0x7fdd3406d540 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: dump: 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.618 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd3aca9640 5 monclient: authenticate success, global_id 4286 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd3aca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 1 -- 172.21.15.164:0/1465800874 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd20032440 con 0x7fdd3406d540 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd3aca9640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 1 -- 172.21.15.164:0/1465800874 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd20032810 con 0x7fdd3406d540 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd3aca9640 4 set_mon_vals no callback set 2024-09-10T16:06:27.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.599+0000 7fdd337fe640 10 monclient: dump: 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 10 monclient: shutdown 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 -- 172.21.15.164:0/1465800874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 msgr2=0x7fdd340a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/1465800874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdd20009870 tx=0x7fdd2002a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 -- 172.21.15.164:0/1465800874 shutdown_connections 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/1465800874 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd3406d540 0x7fdd340a46e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/1465800874 >> v2:172.21.15.164:3300/0 conn(0x7fdd3406d160 0x7fdd34134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 -- 172.21.15.164:0/1465800874 >> 172.21.15.164:0/1465800874 conn(0x7fdd340aacb0 msgr2=0x7fdd340ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 -- 172.21.15.164:0/1465800874 shutdown_connections 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 1 -- 172.21.15.164:0/1465800874 wait complete. 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.600+0000 7fdd3aca9640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.601+0000 7fdd3aca9640 10 monclient: monmap: 2024-09-10T16:06:27.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.601+0000 7fdd3aca9640 1 Processor -- start 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.601+0000 7fdd3aca9640 1 -- start start 2024-09-10T16:06:27.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.601+0000 7fdd3aca9640 10 monclient: init 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-a con 0x7fdd3406d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd3406d540 0x7fdd340a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-b con 0x7fdd3406d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd340a4c20 0x7fdd3413dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): picked mon.noname-c con 0x7fdd340a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdd34138870 con 0x7fdd3406d160 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdd341386f0 con 0x7fdd3406d540 2024-09-10T16:06:27.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdd341389f0 con 0x7fdd340a4c20 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd3aca9640 10 monclient(hunting): authenticate will time out at 766.089218s 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36060/0 (socket says 172.21.15.164:36060) 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 -- 172.21.15.164:0/2445767787 learned_addr learned my addr 172.21.15.164:0/2445767787 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): get_auth_request con 0x7fdd3406d160 auth_method 0 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd3406d540 0x7fdd340a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): get_auth_request con 0x7fdd3406d540 auth_method 0 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd340a4c20 0x7fdd3413dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): get_auth_request con 0x7fdd340a4c20 auth_method 0 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd39246640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-10T16:06:27.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 -- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd340a4c20 msgr2=0x7fdd3413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd340a4c20 0x7fdd3413dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 -- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd3406d540 msgr2=0x7fdd340a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd3406d540 0x7fdd340a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.602+0000 7fdd38a45640 1 -- 172.21.15.164:0/2445767787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd3413f850 con 0x7fdd3406d160 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd38a45640 1 -- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 msgr2=0x7fdd341cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd38a45640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800ebe0 tx=0x7fdd2800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28017070 con 0x7fdd3406d160 2024-09-10T16:06:27.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: dump: 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 5 monclient: authenticate success, global_id 4281 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd2800ce40 con 0x7fdd3406d160 2024-09-10T16:06:27.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd28013690 con 0x7fdd3406d160 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 10 monclient: _renew_subs 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd3409ebe0 con 0x7fdd3406d160 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd32ffd640 4 set_mon_vals no callback set 2024-09-10T16:06:27.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd317fa640 10 monclient: dump: 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.634 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.635 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 10 monclient: _renew_subs 2024-09-10T16:06:27.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.605+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd3409dc80 con 0x7fdd3406d160 2024-09-10T16:06:27.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdd2801a070 con 0x7fdd3406d160 2024-09-10T16:06:27.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd3aca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd3aca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd3aca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd34134e00 con 0x7fdd3406d160 2024-09-10T16:06:27.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.606+0000 7fdd317fa640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 0x7fdd18040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.607+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7fdd28052f80 con 0x7fdd3406d160 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.607+0000 7fdd33fff640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 0x7fdd18040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.607+0000 7fdd33fff640 10 monclient: get_auth_request con 0x7fdd1803e240 auth_method 0 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.610+0000 7fdd33fff640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 0x7fdd18040700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd340773f0 tx=0x7fdd2002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.615+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdd28051af0 con 0x7fdd3406d160 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.615+0000 7fdd317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.615+0000 7fdd317fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.639 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.703+0000 7f16ba6bd640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.703+0000 7f16ba6bd640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.703+0000 7f16ba6bd640 10 monclient: monmap: 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.705 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.704677+0000 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.704677+0000 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.706 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.704+0000 7f16ba6bd640 1 Processor -- start 2024-09-10T16:06:27.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.704+0000 7f16ba6bd640 1 -- start start 2024-09-10T16:06:27.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.704+0000 7f16ba6bd640 10 monclient: init 2024-09-10T16:06:27.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 0x7f16b4134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f16b406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f16b406d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 0x7f16b413f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-c con 0x7f16b413d060 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16ba6bd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16b409cb90 con 0x7f16b406d540 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16b37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f16b406d540 0x7f16b40a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59352/0 (socket says 172.21.15.164:59352) 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16b37fe640 1 -- 172.21.15.164:0/1872930124 learned_addr learned my addr 172.21.15.164:0/1872930124 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 0x7f16b4134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.705+0000 7f16b8c5a640 1 --2- 172.21.15.164:0/1872930124 >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 0x7f16b413f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f16b409ce70 con 0x7f16b413d060 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f16b409d150 con 0x7f16b406d160 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16ba6bd640 10 monclient(hunting): authenticate will time out at 766.193125s 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): get_auth_request con 0x7f16b406d160 auth_method 0 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b8c5a640 10 monclient(hunting): get_auth_request con 0x7f16b413d060 auth_method 0 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b8c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b8c5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b8c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 1 -- 172.21.15.164:0/1872930124 >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 msgr2=0x7f16b413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1872930124 >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 0x7f16b413f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 1 -- 172.21.15.164:0/1872930124 >> v1:172.21.15.164:6789/0 conn(0x7f16b406d540 legacy=0x7f16b40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 1 monclient: found mon.noname-b 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b3fff640 1 -- 172.21.15.164:0/1872930124 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b40a4df0 con 0x7f16b406d160 2024-09-10T16:06:27.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.706+0000 7f16b8c5a640 1 --2- 172.21.15.164:0/1872930124 >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 0x7f16b413f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b3fff640 1 -- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 msgr2=0x7f16b4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 0x7f16b4134e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16a40097a0 tx=0x7f16a4029fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29eb73fb9a69e77f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 1 -- 172.21.15.164:0/1872930124 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4037070 con 0x7f16b406d160 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: dump: 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.717 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16ba6bd640 5 monclient: authenticate success, global_id 4292 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16ba6bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 1 -- 172.21.15.164:0/1872930124 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16a4032440 con 0x7f16b406d160 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 1 -- 172.21.15.164:0/1872930124 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16a4032810 con 0x7f16b406d160 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16b2ffd640 10 monclient: dump: 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16ba6bd640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.709+0000 7f16ba6bd640 4 set_mon_vals no callback set 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 10 monclient: shutdown 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 msgr2=0x7f16b4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 0x7f16b4134e00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f16a40097a0 tx=0x7f16a4029fc0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 shutdown_connections 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1872930124 >> v2:172.21.15.165:3300/0 conn(0x7f16b413d060 0x7f16b413f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1872930124 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b406d160 0x7f16b4134e00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.710+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 >> 172.21.15.164:0/1872930124 conn(0x7f16b40aacb0 msgr2=0x7f16b40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.711+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 shutdown_connections 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.711+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1872930124 wait complete. 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.711+0000 7f16ba6bd640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.711+0000 7f16ba6bd640 10 monclient: monmap: 2024-09-10T16:06:27.720 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.712+0000 7f16ba6bd640 1 Processor -- start 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.712+0000 7f16ba6bd640 1 -- start start 2024-09-10T16:06:27.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.712+0000 7f16ba6bd640 10 monclient: init 2024-09-10T16:06:27.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-b con 0x7f16b406d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-a con 0x7f16b406d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16b37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36066/0 (socket says 172.21.15.164:36066) 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16b8c5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16b8c5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47250/0 (socket says 172.21.15.164:47250) 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16b3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): picked mon.noname-c con 0x7f16b413d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f16b413fb10 con 0x7f16b406d540 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f16b413f990 con 0x7f16b406d160 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f16b413fc90 con 0x7f16b413d060 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.713+0000 7f16ba6bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16ba6bd640 10 monclient(hunting): authenticate will time out at 766.201012s 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:44414/0 (socket says 172.21.15.164:44414) 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 1 -- 172.21.15.164:0/1788656555 learned_addr learned my addr 172.21.15.164:0/1788656555 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): get_auth_request con 0x7f16b406d160 auth_method 0 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): get_auth_request con 0x7f16b406d540 auth_method 0 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): get_auth_request con 0x7f16b413d060 auth_method 0 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b8c5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.714+0000 7f16b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 -- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 msgr2=0x7f16b40a7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 -- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 msgr2=0x7f16b4135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 monclient: found mon.noname-c 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b8c5a640 1 -- 172.21.15.164:0/1788656555 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16b40a7e30 con 0x7f16b413d060 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.715+0000 7f16b37fe640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.717+0000 7f4e3d324640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.717+0000 7f4e3d324640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.717+0000 7f4e3d324640 10 monclient: monmap: 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.718377+0000 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.718377+0000 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.717+0000 7f16b8c5a640 1 -- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 msgr2=0x7f16b41362b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b8c5a640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00d400 tx=0x7f16ac00d8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 1 Processor -- start 2024-09-10T16:06:27.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 1 -- start start 2024-09-10T16:06:27.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 10 monclient: init 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac01a070 con 0x7f16b413d060 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 10 monclient: dump: 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.734 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ac003b90 con 0x7f16b413d060 2024-09-10T16:06:27.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16ba6bd640 5 monclient: authenticate success, global_id 4295 2024-09-10T16:06:27.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16ba6bd640 10 monclient: _renew_subs 2024-09-10T16:06:27.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16ba6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16b413db80 con 0x7f16b413d060 2024-09-10T16:06:27.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.718+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e3806d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4e3806d540 0x7f4e380a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-c con 0x7f4e3806d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e380a4c20 0x7f4e3813f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-a con 0x7f4e380a4c20 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e38135510 con 0x7f4e3806d160 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e36ffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4e3806d160 0x7f4e38134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:54782/0 (socket says 172.21.15.164:54782) 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3340145794 learned_addr learned my addr 172.21.15.164:0/3340145794 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 --2- 172.21.15.164:0/3340145794 >> v2:172.21.15.164:3300/0 conn(0x7f4e380a4c20 0x7f4e3813f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4e3809cb90 con 0x7f4e380a4c20 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4e3809ce70 con 0x7f4e3806d540 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e3d324640 10 monclient(hunting): authenticate will time out at 766.206402s 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): get_auth_request con 0x7f4e380a4c20 auth_method 0 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 1 --2- 172.21.15.164:0/3340145794 >> v2:172.21.15.165:3300/0 conn(0x7f4e3806d540 0x7f4e380a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): get_auth_request con 0x7f4e3806d540 auth_method 0 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient(hunting): handle_auth_done global_id 4287 payload 274 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 -- 172.21.15.164:0/3340145794 >> v2:172.21.15.165:3300/0 conn(0x7f4e3806d540 msgr2=0x7f4e380a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 --2- 172.21.15.164:0/3340145794 >> v2:172.21.15.165:3300/0 conn(0x7f4e3806d540 0x7f4e380a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 -- 172.21.15.164:0/3340145794 >> v1:172.21.15.165:6789/0 conn(0x7f4e3806d160 legacy=0x7f4e38134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ac004030 con 0x7f16b413d060 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 10 monclient: dump: 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.743 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f16ac01d070 con 0x7f16b413d060 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16ba6bd640 10 monclient: _renew_subs 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16ba6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f16b4138640 con 0x7f16b413d060 2024-09-10T16:06:27.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.719+0000 7f4e377fe640 1 -- 172.21.15.164:0/3340145794 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3809d150 con 0x7f4e380a4c20 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 0x7f16a0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b27fc640 4 set_mon_vals no callback set 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.720+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f16ac058060 con 0x7f16b413d060 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 0x7f16a0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16b3fff640 10 monclient: get_auth_request con 0x7f16a003e240 auth_method 0 2024-09-10T16:06:27.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16ba6bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.722+0000 7f4e377fe640 1 -- 172.21.15.164:0/3340145794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e380a4c20 msgr2=0x7f4e3813f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.722+0000 7f4e377fe640 1 --2- 172.21.15.164:0/3340145794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e380a4c20 0x7f4e3813f070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a9d40 tx=0x7f4e3002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb7474ebc7815db1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16ba6bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16ba6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.721+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16b4136f00 con 0x7f16b413d060 2024-09-10T16:06:27.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 1 -- 172.21.15.164:0/3340145794 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e30007740 con 0x7f4e380a4c20 2024-09-10T16:06:27.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: dump: 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 1 -- 172.21.15.164:0/3340145794 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e300078e0 con 0x7f4e380a4c20 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e3d324640 5 monclient: authenticate success, global_id 4287 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e3d324640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e3d324640 4 set_mon_vals no callback set 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 1 -- 172.21.15.164:0/3340145794 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e30007cb0 con 0x7f4e380a4c20 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.723+0000 7f4e35ffb640 10 monclient: dump: 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.751 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 10 monclient: shutdown 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e380a4c20 msgr2=0x7f4e3813f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3340145794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e380a4c20 0x7f4e3813f070 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a9d40 tx=0x7f4e3002a730 comp rx=0 tx=0).stop 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 shutdown_connections 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3340145794 >> v2:172.21.15.165:3300/0 conn(0x7f4e3806d540 0x7f4e380a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3340145794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e380a4c20 0x7f4e3813f070 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 >> 172.21.15.164:0/3340145794 conn(0x7f4e380aacb0 msgr2=0x7f4e380ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 shutdown_connections 2024-09-10T16:06:27.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.724+0000 7f4e3d324640 1 -- 172.21.15.164:0/3340145794 wait complete. 2024-09-10T16:06:27.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.725+0000 7f16b3fff640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 0x7f16a0040700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f16a402a000 tx=0x7f16a402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.725+0000 7f4e3d324640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.725+0000 7f4e3d324640 10 monclient: monmap: 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.726+0000 7f4e3d324640 1 Processor -- start 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.726+0000 7f4e3d324640 1 -- start start 2024-09-10T16:06:27.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.726+0000 7f4e3d324640 10 monclient: init 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-a con 0x7f4e3806d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e36ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e36ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36090/0 (socket says 172.21.15.164:36090) 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3359776459 learned_addr learned my addr 172.21.15.164:0/3359776459 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e3806d540 0x7f4e380a3e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-b con 0x7f4e3806d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e380a4c20 0x7f4e3809e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): picked mon.noname-c con 0x7f4e380a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3813f9e0 con 0x7f4e3806d160 2024-09-10T16:06:27.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3813f860 con 0x7f4e3806d540 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.727+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e3813fb60 con 0x7f4e380a4c20 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e3d324640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e3d324640 10 monclient(hunting): authenticate will time out at 766.215033s 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): get_auth_request con 0x7f4e3806d160 auth_method 0 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e380a4c20 0x7f4e3809e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): get_auth_request con 0x7f4e380a4c20 auth_method 0 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e380a4c20 msgr2=0x7f4e3809e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e380a4c20 0x7f4e3809e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e3806d540 msgr2=0x7f4e380a3e40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e3806d540 0x7f4e380a3e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.728+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e380a4380 con 0x7f4e3806d160 2024-09-10T16:06:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.730+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f16b413db80 con 0x7f16b413d060 2024-09-10T16:06:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.730+0000 7f16b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.730+0000 7f16b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.730+0000 7f4e36ffd640 1 -- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 msgr2=0x7f4e380a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.730+0000 7f4e36ffd640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800c490 tx=0x7f4e2800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28007070 con 0x7f4e3806d160 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: dump: 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.763 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e28007230 con 0x7f4e3806d160 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28007620 con 0x7f4e3806d160 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e1f7fe640 10 monclient: dump: 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.764 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 5 monclient: authenticate success, global_id 4290 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e357fa640 4 set_mon_vals no callback set 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 10 monclient: _renew_subs 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e380a0bd0 con 0x7f4e3806d160 2024-09-10T16:06:27.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 10 monclient: _renew_subs 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.731+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e380a1b30 con 0x7f4e3806d160 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4e2801b400 con 0x7f4e3806d160 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e1f7fe640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 0x7f4e24040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f4e280527c0 con 0x7f4e3806d160 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e3d324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e3d324640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.732+0000 7f4e3d324640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.733+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e380a46e0 con 0x7f4e3806d160 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.733+0000 7f4e367fc640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 0x7f4e24040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.733+0000 7f4e367fc640 10 monclient: get_auth_request con 0x7f4e2403e260 auth_method 0 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.735+0000 7f4e367fc640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 0x7f4e24040720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a0b60 tx=0x7f4e2c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.738+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4e2801ac60 con 0x7f4e3806d160 2024-09-10T16:06:27.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.738+0000 7f4e1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.738+0000 7f4e1f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.764+0000 7f5f8738a640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.764+0000 7f5f8738a640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.764+0000 7f5f8738a640 10 monclient: monmap: 2024-09-10T16:06:27.772 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.765157+0000 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.765157+0000 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.773 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.764+0000 7f5f8738a640 1 Processor -- start 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.765+0000 7f5f8738a640 1 -- start start 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.765+0000 7f5f8738a640 10 monclient: init 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.765+0000 7f5f8738a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 0x7f5f780bb8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-b con 0x7f5f78021a50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5f780bbe00 0x7f5f78021e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-a con 0x7f5f780bbe00 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5f78022370 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f86388640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 0x7f5f780bb8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f86388640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 0x7f5f780bb8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47278/0 (socket says 172.21.15.164:47278) 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f86388640 1 -- 172.21.15.164:0/481174582 learned_addr learned my addr 172.21.15.164:0/481174582 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f85b87640 1 --2- 172.21.15.164:0/481174582 >> v2:172.21.15.164:3300/0 conn(0x7f5f780bbe00 0x7f5f78021e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7801fa50 con 0x7f5f78022370 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5f780202f0 con 0x7f5f780bbe00 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5f7801fd80 con 0x7f5f78021a50 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.766+0000 7f5f8738a640 10 monclient(hunting): authenticate will time out at 766.253983s 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): get_auth_request con 0x7f5f78021a50 auth_method 0 2024-09-10T16:06:27.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): get_auth_request con 0x7f5f780bbe00 auth_method 0 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient(hunting): handle_auth_done global_id 4293 payload 274 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 1 -- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 msgr2=0x7f5f780bb8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 1 --2- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 0x7f5f780bb8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 1 -- 172.21.15.164:0/481174582 >> v1:172.21.15.164:6789/0 conn(0x7f5f78022370 legacy=0x7f5f78007690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.767+0000 7f5f85b87640 1 -- 172.21.15.164:0/481174582 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f78020010 con 0x7f5f780bbe00 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85b87640 1 -- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780bbe00 msgr2=0x7f5f78021e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85b87640 1 --2- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780bbe00 0x7f5f78021e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f700097a0 tx=0x7f5f7002a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ead3ff965d4d0d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 1 -- 172.21.15.164:0/481174582 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70037070 con 0x7f5f780bbe00 2024-09-10T16:06:27.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: dump: 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 1 -- 172.21.15.164:0/481174582 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f70032440 con 0x7f5f780bbe00 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 1 -- 172.21.15.164:0/481174582 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70032810 con 0x7f5f780bbe00 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f85386640 10 monclient: dump: 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f8738a640 5 monclient: authenticate success, global_id 4293 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f8738a640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.770+0000 7f5f8738a640 4 set_mon_vals no callback set 2024-09-10T16:06:27.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 10 monclient: shutdown 2024-09-10T16:06:27.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780bbe00 msgr2=0x7f5f78021e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 --2- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780bbe00 0x7f5f78021e30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5f700097a0 tx=0x7f5f7002a050 comp rx=0 tx=0).stop 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 shutdown_connections 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 --2- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f780bbe00 0x7f5f78021e30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 --2- 172.21.15.164:0/481174582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f78021a50 0x7f5f780bb8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.771+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 >> 172.21.15.164:0/481174582 conn(0x7f5f78006340 msgr2=0x7f5f78006750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.772+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 shutdown_connections 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.772+0000 7f5f8738a640 1 -- 172.21.15.164:0/481174582 wait complete. 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.773+0000 7f5f8738a640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.773+0000 7f5f8738a640 10 monclient: monmap: 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.786 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.773+0000 7f5f8738a640 1 Processor -- start 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 1 -- start start 2024-09-10T16:06:27.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 10 monclient: init 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.774+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-a con 0x7f5f78021a50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 0x7f5f780a1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-b con 0x7f5f78022370 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36104/0 (socket says 172.21.15.164:36104) 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 0x7f5f780a1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 0x7f5f780a1e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:44440/0 (socket says 172.21.15.164:44440) 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47284/0 (socket says 172.21.15.164:47284) 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): picked mon.noname-c con 0x7f5f780bbe00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 1 -- 172.21.15.164:0/3334955764 learned_addr learned my addr 172.21.15.164:0/3334955764 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5f78008130 con 0x7f5f78021a50 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5f78007fb0 con 0x7f5f78022370 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5f780082b0 con 0x7f5f780bbe00 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f8738a640 10 monclient(hunting): authenticate will time out at 766.262756s 2024-09-10T16:06:27.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): get_auth_request con 0x7f5f78021a50 auth_method 0 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 10 monclient(hunting): get_auth_request con 0x7f5f78022370 auth_method 0 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f85b87640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 10 monclient(hunting): get_auth_request con 0x7f5f780bbe00 auth_method 0 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.775+0000 7f5f86b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f85b87640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 10 monclient(hunting): handle_auth_done global_id 4296 payload 824 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 -- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 msgr2=0x7f5f780a23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 -- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 msgr2=0x7f5f780a1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 0x7f5f780a1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 monclient: found mon.noname-a 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86388640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f780a5a70 con 0x7f5f78021a50 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.776+0000 7f5f86b89640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 10 monclient: get_monmap_and_config 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 10 monclient: monmap: 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:27.778962+0000 2024-09-10T16:06:27.792 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:27.778962+0000 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f86388640 1 -- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 msgr2=0x7f5f780b7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f86388640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00ebb0 tx=0x7f5f7c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 1 Processor -- start 2024-09-10T16:06:27.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c017070 con 0x7f5f78021a50 2024-09-10T16:06:27.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 1 -- start start 2024-09-10T16:06:27.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:27.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: dump: 2024-09-10T16:06:27.795 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.795 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.795 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7c00cc30 con 0x7f5f78021a50 2024-09-10T16:06:27.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7c013470 con 0x7f5f78021a50 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f76ffd640 10 monclient: dump: 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.797 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f84b85640 4 set_mon_vals no callback set 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f8738a640 5 monclient: authenticate success, global_id 4296 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f8738a640 10 monclient: _renew_subs 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f5f8738a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f780228a0 con 0x7f5f78021a50 2024-09-10T16:06:27.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.778+0000 7f2ffcc16640 10 monclient: init 2024-09-10T16:06:27.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f5f8738a640 10 monclient: _renew_subs 2024-09-10T16:06:27.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f5f8738a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f780b89b0 con 0x7f5f78021a50 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ff806d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ff806d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2ff813e070 0x7f2ff8140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff813e070 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff809cb90 con 0x7f2ff806d160 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ff809ce70 con 0x7f2ff806d540 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2ff809d150 con 0x7f2ff813e070 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5f7c01a070 con 0x7f5f78021a50 2024-09-10T16:06:27.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.779+0000 7f2ffcc16640 10 monclient(hunting): authenticate will time out at 766.266850s 2024-09-10T16:06:27.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2ff806d160 0x7f2ff8134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59384/0 (socket says 172.21.15.164:59384) 2024-09-10T16:06:27.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f8738a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f8738a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f8738a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f780bb8a0 con 0x7f5f78021a50 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f76ffd640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 0x7f5f64040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f5f7c052b00 con 0x7f5f78021a50 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f5f85b87640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 0x7f5f64040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.781+0000 7f5f85b87640 10 monclient: get_auth_request con 0x7f5f6403e260 auth_method 0 2024-09-10T16:06:27.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff659c640 1 -- 172.21.15.164:0/3028273595 learned_addr learned my addr 172.21.15.164:0/3028273595 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 1 --2- 172.21.15.164:0/3028273595 >> v2:172.21.15.165:3300/0 conn(0x7f2ff813e070 0x7f2ff8140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): get_auth_request con 0x7f2ff813e070 auth_method 0 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 1 -- 172.21.15.164:0/3028273595 >> v1:172.21.15.165:6789/0 conn(0x7f2ff806d540 legacy=0x7f2ff80a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 1 -- 172.21.15.164:0/3028273595 >> v1:172.21.15.164:6789/0 conn(0x7f2ff806d160 legacy=0x7f2ff8134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 1 monclient: found mon.noname-c 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.780+0000 7f2ff6d9d640 1 -- 172.21.15.164:0/3028273595 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff80a4df0 con 0x7f2ff813e070 2024-09-10T16:06:27.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.783+0000 7f5f85b87640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 0x7f5f64040720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f78020710 tx=0x7f5f7002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff6d9d640 1 -- 172.21.15.164:0/3028273595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 msgr2=0x7f2ff8140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff6d9d640 1 --2- 172.21.15.164:0/3028273595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff8140460 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00097a0 tx=0x7f2ff0029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0b7b0c0148f208d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 1 -- 172.21.15.164:0/3028273595 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0037070 con 0x7f2ff813e070 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:27.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: dump: 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 1 -- 172.21.15.164:0/3028273595 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ff0032640 con 0x7f2ff813e070 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 1 -- 172.21.15.164:0/3028273595 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff0032a10 con 0x7f2ff813e070 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ff5d9b640 10 monclient: dump: 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.809 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ffcc16640 5 monclient: authenticate success, global_id 4297 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ffcc16640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.784+0000 7f2ffcc16640 4 set_mon_vals no callback set 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 10 monclient: shutdown 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3028273595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 msgr2=0x7f2ff8140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3028273595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff8140460 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2ff00097a0 tx=0x7f2ff0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3028273595 shutdown_connections 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3028273595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff8140460 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3028273595 >> 172.21.15.164:0/3028273595 conn(0x7f2ff80aacb0 msgr2=0x7f2ff80ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3028273595 shutdown_connections 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.785+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3028273595 wait complete. 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.786+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5f7c051670 con 0x7f5f78021a50 2024-09-10T16:06:27.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.786+0000 7f5f76ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.786+0000 7f5f76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.786+0000 7f2ffcc16640 10 monclient: build_initial_monmap 2024-09-10T16:06:27.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.786+0000 7f2ffcc16640 10 monclient: monmap: 2024-09-10T16:06:27.816 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.787+0000 7f2ffcc16640 1 Processor -- start 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.787+0000 7f2ffcc16640 1 -- start start 2024-09-10T16:06:27.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.787+0000 7f2ffcc16640 10 monclient: init 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-a con 0x7f2ff806d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 0x7f2ff812d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-c con 0x7f2ff806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff809eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): picked mon.noname-b con 0x7f2ff813e070 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8140d30 con 0x7f2ff806d160 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2ff8140bb0 con 0x7f2ff813e070 2024-09-10T16:06:27.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2ff8140eb0 con 0x7f2ff806d540 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ffcc16640 10 monclient(hunting): authenticate will time out at 766.275119s 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36120/0 (socket says 172.21.15.164:36120) 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 1 -- 172.21.15.164:0/3006142080 learned_addr learned my addr 172.21.15.164:0/3006142080 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): get_auth_request con 0x7f2ff806d160 auth_method 0 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 0x7f2ff812d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): get_auth_request con 0x7f2ff806d540 auth_method 0 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:27.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff6d9d640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff809eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 -- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 msgr2=0x7f2ff809eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff809eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 -- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 msgr2=0x7f2ff812d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 monclient: found mon.noname-c 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2fedd9b640 1 -- 172.21.15.164:0/3006142080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ff80a47e0 con 0x7f2ff806d540 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.788+0000 7f2ff659c640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fedd9b640 1 -- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 msgr2=0x7f2ff812d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fedd9b640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 0x7f2ff812d750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c780 tx=0x7f2fe000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe0016020 con 0x7f2ff806d540 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: dump: 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:27.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 5 monclient: authenticate success, global_id 4307 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 10 monclient: _renew_subs 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ff80a0e00 con 0x7f2ff806d540 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 10 monclient: _renew_subs 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ff80a1d60 con 0x7f2ff806d540 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fe000a840 con 0x7f2ff806d540 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fe000ac30 con 0x7f2ff806d540 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:27.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2fef7fe640 10 monclient: dump: 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.792+0000 7f2ffcc16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.791+0000 7f2ff559a640 4 set_mon_vals no callback set 2024-09-10T16:06:27.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.793+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2fe0019020 con 0x7f2ff806d540 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2fef7fe640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 0x7f2fd80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 4402+0+0 (secure 0 0 0) 0x7f2fe0052b90 con 0x7f2ff806d540 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2ffcc16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2ffcc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ff806df60 con 0x7f2ff806d540 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2ff659c640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 0x7f2fd80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.794+0000 7f2ff659c640 10 monclient: get_auth_request con 0x7f2fd803e210 auth_method 0 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.797+0000 7f2ff659c640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 0x7f2fd80406d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800d660 tx=0x7f2fe8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.803+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2ff806df60 con 0x7f2ff806d540 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.803+0000 7f2fef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:27.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.803+0000 7f2fef7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:27.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.929+0000 7f2310465640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.929+0000 7f2310465640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.929+0000 7f2310465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.929+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f23080770f0 con 0x7f23080a4ab0 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.930+0000 7f2306ffd640 1 -- 172.21.15.164:0/1011511749 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f230813a030 con 0x7f23080a4ab0 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.930+0000 7f2306ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.930+0000 7f2306ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:27.932 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:27.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 msgr2=0x7f22f40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 0x7f22f40406d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f23000097a0 tx=0x7f230002e000 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 10 monclient: shutdown 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 msgr2=0x7f23080a23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f22f800c410 tx=0x7f22f800c8d0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 shutdown_connections 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.934+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22f403e210 0x7f22f40406d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23080a4ab0 0x7f23080a23f0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23080a46a0 0x7f2308131800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 --2- 172.21.15.164:0/1011511749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f230806cb70 0x7f23081312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 >> 172.21.15.164:0/1011511749 conn(0x7f23080aa900 msgr2=0x7f2308137980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 shutdown_connections 2024-09-10T16:06:27.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.935+0000 7f2310465640 1 -- 172.21.15.164:0/1011511749 wait complete. 2024-09-10T16:06:27.952 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.2 2024-09-10T16:06:27.953 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:27.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.963+0000 7f16ba6bd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:27.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.963+0000 7f16ba6bd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:27.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.963+0000 7f16ba6bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:27.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.963+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f16b4134e00 con 0x7f16b413d060 2024-09-10T16:06:27.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.963+0000 7f16b0ff9640 1 -- 172.21.15.164:0/1788656555 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f16ac010070 con 0x7f16b413d060 2024-09-10T16:06:27.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.964+0000 7f16b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T16:06:27.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.964+0000 7f16b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:27.966 INFO:teuthology.orchestra.run.smithi164.stdout:55834574853 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 msgr2=0x7f16a0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 0x7f16a0040700 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f16a402a000 tx=0x7f16a402e000 comp rx=0 tx=0).stop 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 10 monclient: shutdown 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 msgr2=0x7f16b41362b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f16ac00d400 tx=0x7f16ac00d8c0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 shutdown_connections 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f16a003e240 0x7f16a0040700 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f16b413d060 0x7f16b41362b0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f16b406d540 0x7f16b4135d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 --2- 172.21.15.164:0/1788656555 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f16b406d160 0x7f16b40a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 >> 172.21.15.164:0/1788656555 conn(0x7f16b40aacb0 msgr2=0x7f16b4132760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 shutdown_connections 2024-09-10T16:06:27.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.966+0000 7f16ba6bd640 1 -- 172.21.15.164:0/1788656555 wait complete. 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd3aca9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd3aca9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd3aca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fdd3406e6b0 con 0x7fdd3406d160 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd317fa640 1 -- 172.21.15.164:0/2445767787 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fdd280123d0 con 0x7fdd3406d160 2024-09-10T16:06:27.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T16:06:27.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.975+0000 7fdd317fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:27.978 INFO:teuthology.orchestra.run.smithi164.stdout:55834574853 2024-09-10T16:06:27.979 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2024-09-10T16:06:27.980 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:27.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 msgr2=0x7fdd18040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 0x7fdd18040700 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fdd340773f0 tx=0x7fdd2002e000 comp rx=0 tx=0).stop 2024-09-10T16:06:27.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 10 monclient: shutdown 2024-09-10T16:06:27.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 msgr2=0x7fdd341cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdd2800ebe0 tx=0x7fdd2800c520 comp rx=0 tx=0).stop 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 shutdown_connections 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdd1803e240 0x7fdd18040700 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdd340a4c20 0x7fdd3413dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdd3406d540 0x7fdd340a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 --2- 172.21.15.164:0/2445767787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdd3406d160 0x7fdd341cd2b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 >> 172.21.15.164:0/2445767787 conn(0x7fdd340aacb0 msgr2=0x7fdd34136740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:27.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 shutdown_connections 2024-09-10T16:06:27.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:27.978+0000 7fdd3aca9640 1 -- 172.21.15.164:0/2445767787 wait complete. 2024-09-10T16:06:27.990 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.4 2024-09-10T16:06:27.990 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:28.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f8738a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f8738a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f8738a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f5f7800a500 con 0x7f5f78021a50 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f76ffd640 1 -- 172.21.15.164:0/3334955764 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f5f7c013dc0 con 0x7f5f78021a50 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f76ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.006+0000 7f5f76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:28.008 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:28.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 msgr2=0x7f5f64040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 0x7f5f64040720 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5f78020710 tx=0x7f5f7002e000 comp rx=0 tx=0).stop 2024-09-10T16:06:28.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 10 monclient: shutdown 2024-09-10T16:06:28.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 msgr2=0x7f5f780b7fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c00ebb0 tx=0x7f5f7c00c520 comp rx=0 tx=0).stop 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 shutdown_connections 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f6403e260 0x7f5f64040720 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5f780bbe00 0x7f5f780a23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5f78022370 0x7f5f780a1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 --2- 172.21.15.164:0/3334955764 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5f78021a50 0x7f5f780b7fa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 >> 172.21.15.164:0/3334955764 conn(0x7f5f78006340 msgr2=0x7f5f780b9e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 shutdown_connections 2024-09-10T16:06:28.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.008+0000 7f5f8738a640 1 -- 172.21.15.164:0/3334955764 wait complete. 2024-09-10T16:06:28.020 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-10T16:06:28.020 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:28.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e3d324640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:28.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e3d324640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:28.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e3d324640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f4e380aa1e0 con 0x7f4e3806d160 2024-09-10T16:06:28.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e1f7fe640 1 -- 172.21.15.164:0/3359776459 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f4e2801b8d0 con 0x7f4e3806d160 2024-09-10T16:06:28.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T16:06:28.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.062+0000 7f4e1f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:28.064 INFO:teuthology.orchestra.run.smithi164.stdout:47244640260 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 msgr2=0x7f4e24040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 0x7f4e24040720 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4e380a0b60 tx=0x7f4e2c009190 comp rx=0 tx=0).stop 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 10 monclient: shutdown 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 msgr2=0x7f4e380a3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e2800c490 tx=0x7f4e2800c950 comp rx=0 tx=0).stop 2024-09-10T16:06:28.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 shutdown_connections 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e2403e260 0x7f4e24040720 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e380a4c20 0x7f4e3809e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e3806d540 0x7f4e380a3e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 --2- 172.21.15.164:0/3359776459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e3806d160 0x7f4e380a3900 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 >> 172.21.15.164:0/3359776459 conn(0x7f4e380aacb0 msgr2=0x7f4e3812e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 shutdown_connections 2024-09-10T16:06:28.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.064+0000 7f4e3d324640 1 -- 172.21.15.164:0/3359776459 wait complete. 2024-09-10T16:06:28.076 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.3 2024-09-10T16:06:28.077 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.100+0000 7f2ffcc16640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.100+0000 7f2ffcc16640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.100+0000 7f2ffcc16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.100+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f2ff80aa1c0 con 0x7f2ff806d540 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.101+0000 7f2fef7fe640 1 -- 172.21.15.164:0/3006142080 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2fe0020020 con 0x7f2ff806d540 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.101+0000 7f2fef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T16:06:28.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.101+0000 7f2fef7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:28.104 INFO:teuthology.orchestra.run.smithi164.stdout:51539607557 2024-09-10T16:06:28.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 msgr2=0x7f2fd80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 0x7f2fd80406d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2fe800d660 tx=0x7f2fe8009040 comp rx=0 tx=0).stop 2024-09-10T16:06:28.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 10 monclient: shutdown 2024-09-10T16:06:28.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 msgr2=0x7f2ff812d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 0x7f2ff812d750 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2fe000c780 tx=0x7f2fe000cc40 comp rx=0 tx=0).stop 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 shutdown_connections 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2fd803e210 0x7f2fd80406d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ff813e070 0x7f2ff809eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ff806d540 0x7f2ff812d750 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 --2- 172.21.15.164:0/3006142080 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ff806d160 0x7f2ff812d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 >> 172.21.15.164:0/3006142080 conn(0x7f2ff80aacb0 msgr2=0x7f2ff813eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 shutdown_connections 2024-09-10T16:06:28.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.103+0000 7f2ffcc16640 1 -- 172.21.15.164:0/3006142080 wait complete. 2024-09-10T16:06:28.115 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.0 2024-09-10T16:06:28.115 DEBUG:teuthology.parallel:result is None 2024-09-10T16:06:28.115 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T16:06:28.116 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:06:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.251+0000 7f90e09ad640 10 monclient: get_monmap_and_config 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.251+0000 7f90e09ad640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.251+0000 7f90e09ad640 10 monclient: monmap: 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:28.252333+0000 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:28.252333+0000 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:28.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 Processor -- start 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 -- start start 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient: init 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-10T16:06:28.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90dc09d930 0x7f90dc09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-a con 0x7f90dc09d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90dc09ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f90dc09e250 0x7f90dc134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-c con 0x7f90dc09e250 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90dc0a0870 con 0x7f90dc09ab50 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f90dc09d3e0 con 0x7f90dc09d930 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f90dc074c70 con 0x7f90dc09e250 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90e09ad640 10 monclient(hunting): authenticate will time out at 766.739692s 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90dc09d930 0x7f90dc09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90dc09d930 0x7f90dc09dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36122/0 (socket says 172.21.15.164:36122) 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 1 -- 172.21.15.164:0/2617028606 learned_addr learned my addr 172.21.15.164:0/2617028606 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): get_auth_request con 0x7f90dc09d930 auth_method 0 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.252+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 10 monclient(hunting): handle_auth_done global_id 4302 payload 274 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 1 -- 172.21.15.164:0/2617028606 >> v2:172.21.15.165:3300/0 conn(0x7f90dc09e250 msgr2=0x7f90dc134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 1 --2- 172.21.15.164:0/2617028606 >> v2:172.21.15.165:3300/0 conn(0x7f90dc09e250 0x7f90dc134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 1 -- 172.21.15.164:0/2617028606 >> v1:172.21.15.165:6789/0 conn(0x7f90dc09ab50 legacy=0x7f90dc134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 1 monclient: found mon.noname-a 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.253+0000 7f90da59c640 1 -- 172.21.15.164:0/2617028606 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90dc09ecb0 con 0x7f90dc09d930 2024-09-10T16:06:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.255+0000 7f90da59c640 1 -- 172.21.15.164:0/2617028606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 msgr2=0x7f90dc09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.255+0000 7f90da59c640 1 --2- 172.21.15.164:0/2617028606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc09dd10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90dc09a5a0 tx=0x7f90cc02b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64c03bb97e22dfb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.255+0000 7f90d9d9b640 1 -- 172.21.15.164:0/2617028606 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc038070 con 0x7f90dc09d930 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: dump: 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 1 -- 172.21.15.164:0/2617028606 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90cc033640 con 0x7f90dc09d930 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 5 monclient: authenticate success, global_id 4302 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 1 -- 172.21.15.164:0/2617028606 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90cc033a10 con 0x7f90dc09d930 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 4 set_mon_vals no callback set 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90d9d9b640 10 monclient: dump: 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 10 monclient: shutdown 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 -- 172.21.15.164:0/2617028606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 msgr2=0x7f90dc09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 --2- 172.21.15.164:0/2617028606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc09dd10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f90dc09a5a0 tx=0x7f90cc02b1c0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 -- 172.21.15.164:0/2617028606 shutdown_connections 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 --2- 172.21.15.164:0/2617028606 >> v2:172.21.15.165:3300/0 conn(0x7f90dc09e250 0x7f90dc134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 --2- 172.21.15.164:0/2617028606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc09dd10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 -- 172.21.15.164:0/2617028606 >> 172.21.15.164:0/2617028606 conn(0x7f90dc0a3110 msgr2=0x7f90dc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 -- 172.21.15.164:0/2617028606 shutdown_connections 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.256+0000 7f90e09ad640 1 -- 172.21.15.164:0/2617028606 wait complete. 2024-09-10T16:06:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.257+0000 7f90e09ad640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.257+0000 7f90e09ad640 10 monclient: monmap: 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.257+0000 7f90e09ad640 1 Processor -- start 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.257+0000 7f90e09ad640 1 -- start start 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.257+0000 7f90e09ad640 10 monclient: init 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 0x7f90dc099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-c con 0x7f90dc09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-a con 0x7f90dc09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90dc09e250 0x7f90dc12b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): picked mon.noname-b con 0x7f90dc09e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc09ef30 con 0x7f90dc09d930 2024-09-10T16:06:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90dc09e6e0 con 0x7f90dc09e250 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f90dc0a2940 con 0x7f90dc09ab50 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90e09ad640 10 monclient(hunting): authenticate will time out at 766.745209s 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 0x7f90dc099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36132/0 (socket says 172.21.15.164:36132) 2024-09-10T16:06:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 -- 172.21.15.164:0/1018326964 learned_addr learned my addr 172.21.15.164:0/1018326964 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:28.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): get_auth_request con 0x7f90dc09d930 auth_method 0 2024-09-10T16:06:28.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): get_auth_request con 0x7f90dc09ab50 auth_method 0 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90dc09e250 0x7f90dc12b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): get_auth_request con 0x7f90dc09e250 auth_method 0 2024-09-10T16:06:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90dad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 -- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 msgr2=0x7f90dc099380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 0x7f90dc099380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 -- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90dc09e250 msgr2=0x7f90dc12b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90dc09e250 0x7f90dc12b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 monclient: found mon.noname-a 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90d3fff640 1 -- 172.21.15.164:0/1018326964 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90dc12dfc0 con 0x7f90dc09d930 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.258+0000 7f90da59c640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 0x7f90dc099380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.260+0000 7f90d3fff640 1 -- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 msgr2=0x7f90dc0998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.260+0000 7f90d3fff640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c410 tx=0x7f90c400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.260+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4017070 con 0x7f90dc09d930 2024-09-10T16:06:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: dump: 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90c40037f0 con 0x7f90dc09d930 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 5 monclient: authenticate success, global_id 4305 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90c4003be0 con 0x7f90dc09d930 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 10 monclient: _renew_subs 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90dc12caa0 con 0x7f90dc09d930 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d959a640 4 set_mon_vals no callback set 2024-09-10T16:06:28.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 10 monclient: dump: 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 10 monclient: _renew_subs 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90dc12c7f0 con 0x7f90dc09d930 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.261+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f90c401a070 con 0x7f90dc09d930 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90e09ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90e09ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90e09ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90d2ffd640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 0x7f90bc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90dc09dd10 con 0x7f90dc09d930 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f90c40520b0 con 0x7f90dc09d930 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90da59c640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 0x7f90bc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.262+0000 7f90da59c640 10 monclient: get_auth_request con 0x7f90bc03e260 auth_method 0 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.264+0000 7f90da59c640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 0x7f90bc040720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f90dc06de20 tx=0x7f90cc02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.266+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f90c401bbb0 con 0x7f90dc09d930 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.266+0000 7f90d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.266+0000 7f90d2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.436+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f90dc06dac0 con 0x7f90bc03e260 2024-09-10T16:06:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.438+0000 7f90d2ffd640 1 -- 172.21.15.164:0/1018326964 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34190 (secure 0 0 0) 0x7f90dc06dac0 con 0x7f90bc03e260 2024-09-10T16:06:28.439 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:28.440 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:06:28.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.439+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 msgr2=0x7f90bc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.439+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 0x7f90bc040720 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f90dc06de20 tx=0x7f90cc02f000 comp rx=0 tx=0).stop 2024-09-10T16:06:28.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 10 monclient: shutdown 2024-09-10T16:06:28.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 msgr2=0x7f90dc0998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f90c400c410 tx=0x7f90c400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 shutdown_connections 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90bc03e260 0x7f90bc040720 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90dc09e250 0x7f90dc12b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90dc09d930 0x7f90dc0998c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 --2- 172.21.15.164:0/1018326964 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90dc09ab50 0x7f90dc099380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 >> 172.21.15.164:0/1018326964 conn(0x7f90dc0a3110 msgr2=0x7f90dc1bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 shutdown_connections 2024-09-10T16:06:28.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.440+0000 7f90e09ad640 1 -- 172.21.15.164:0/1018326964 wait complete. 2024-09-10T16:06:28.452 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-10T16:06:27.837822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163376,"kb_used_data":1184,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067664,"statfs":{"total":579820584960,"available":579653287936,"internally_reserved":0,"allocated":1212416,"data_stored":1415842,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":165927587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.422616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428562+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:06:24.428562+0000","last_peered":"2024-09-10T16:06:24.428562+0000","last_clean":"2024-09-10T16:06:24.428562+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:06:24.428562+0000","last_undegraded":"2024-09-10T16:06:24.428562+0000","last_fullsized":"2024-09-10T16:06:24.428562+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.437270+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:06:24.437270+0000","last_peered":"2024-09-10T16:06:24.437270+0000","last_clean":"2024-09-10T16:06:24.437270+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:06:24.437270+0000","last_undegraded":"2024-09-10T16:06:24.437270+0000","last_fullsized":"2024-09-10T16:06:24.437270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428235+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:06:24.428235+0000","last_peered":"2024-09-10T16:06:24.428235+0000","last_clean":"2024-09-10T16:06:24.428235+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:06:24.428235+0000","last_undegraded":"2024-09-10T16:06:24.428235+0000","last_fullsized":"2024-09-10T16:06:24.428235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.520319+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:06:24.520319+0000","last_peered":"2024-09-10T16:06:24.520319+0000","last_clean":"2024-09-10T16:06:24.520319+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:06:24.520319+0000","last_undegraded":"2024-09-10T16:06:24.520319+0000","last_fullsized":"2024-09-10T16:06:24.520319+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.432116+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:06:24.432116+0000","last_peered":"2024-09-10T16:06:24.432116+0000","last_clean":"2024-09-10T16:06:24.432116+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:06:24.432116+0000","last_undegraded":"2024-09-10T16:06:24.432116+0000","last_fullsized":"2024-09-10T16:06:24.432116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.429954+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:06:24.429954+0000","last_peered":"2024-09-10T16:06:24.429954+0000","last_clean":"2024-09-10T16:06:24.429954+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:06:24.429954+0000","last_undegraded":"2024-09-10T16:06:24.429954+0000","last_fullsized":"2024-09-10T16:06:24.429954+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427455+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:06:24.427455+0000","last_peered":"2024-09-10T16:06:24.427455+0000","last_clean":"2024-09-10T16:06:24.427455+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:06:24.427455+0000","last_undegraded":"2024-09-10T16:06:24.427455+0000","last_fullsized":"2024-09-10T16:06:24.427455+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428654+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:06:24.428654+0000","last_peered":"2024-09-10T16:06:24.428654+0000","last_clean":"2024-09-10T16:06:24.428654+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:06:24.428654+0000","last_undegraded":"2024-09-10T16:06:24.428654+0000","last_fullsized":"2024-09-10T16:06:24.428654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":112,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427527+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:06:24.427527+0000","last_peered":"2024-09-10T16:06:24.427527+0000","last_clean":"2024-09-10T16:06:24.427527+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:06:24.427527+0000","last_undegraded":"2024-09-10T16:06:24.427527+0000","last_fullsized":"2024-09-10T16:06:24.427527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:06:28.453 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:06:28.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.549+0000 7f687b7c6640 10 monclient: get_monmap_and_config 2024-09-10T16:06:28.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.549+0000 7f687b7c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.549+0000 7f687b7c6640 10 monclient: monmap: 2024-09-10T16:06:28.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:28.550704+0000 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:28.550704+0000 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:28.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:28.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:28.553 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:28.553 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:28.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.550+0000 7f687b7c6640 1 Processor -- start 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.550+0000 7f687b7c6640 1 -- start start 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.550+0000 7f687b7c6640 10 monclient: init 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f68740a3e60 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f687406d0b0 0x7f687406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f687406d0b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 0x7f687413e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f687413c5e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f68740a49b0 con 0x7f68740a3e60 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f68740a4240 con 0x7f687406d0b0 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f68740a4600 con 0x7f687413c5e0 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f687b7c6640 10 monclient(hunting): authenticate will time out at 767.038178s 2024-09-10T16:06:28.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 0x7f687413e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36140/0 (socket says 172.21.15.164:36140) 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 1 -- 172.21.15.164:0/1673491632 learned_addr learned my addr 172.21.15.164:0/1673491632 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): get_auth_request con 0x7f68740a3e60 auth_method 0 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): get_auth_request con 0x7f687413c5e0 auth_method 0 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6878d61640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.165:3300/0 conn(0x7f687406d0b0 0x7f687406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6878d61640 10 monclient(hunting): get_auth_request con 0x7f687406d0b0 auth_method 0 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6878d61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6878d61640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6878d61640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient(hunting): handle_auth_done global_id 4313 payload 274 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 -- 172.21.15.164:0/1673491632 >> v2:172.21.15.165:3300/0 conn(0x7f687406d0b0 msgr2=0x7f687406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.165:3300/0 conn(0x7f687406d0b0 0x7f687406d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 -- 172.21.15.164:0/1673491632 >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 msgr2=0x7f687406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 monclient: found mon.noname-b 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879d63640 1 -- 172.21.15.164:0/1673491632 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68740a5d70 con 0x7f687413c5e0 2024-09-10T16:06:28.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.551+0000 7f6879562640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f6879d63640 1 -- 172.21.15.164:0/1673491632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 msgr2=0x7f687413e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f6879d63640 1 --2- 172.21.15.164:0/1673491632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 0x7f687413e9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68740a2b10 tx=0x7f686c02a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b61a3c8b10936929 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 1 -- 172.21.15.164:0/1673491632 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c007950 con 0x7f687413c5e0 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: dump: 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f687b7c6640 5 monclient: authenticate success, global_id 4313 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f687b7c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:28.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 1 -- 172.21.15.164:0/1673491632 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f686c007af0 con 0x7f687413c5e0 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 1 -- 172.21.15.164:0/1673491632 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f686c031400 con 0x7f687413c5e0 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f687b7c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f687b7c6640 4 set_mon_vals no callback set 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.554+0000 7f686bfff640 10 monclient: dump: 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.569 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 10 monclient: shutdown 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 -- 172.21.15.164:0/1673491632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 msgr2=0x7f687413e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 --2- 172.21.15.164:0/1673491632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 0x7f687413e9d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f68740a2b10 tx=0x7f686c02a4e0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 -- 172.21.15.164:0/1673491632 shutdown_connections 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 --2- 172.21.15.164:0/1673491632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687413c5e0 0x7f687413e9d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.165:3300/0 conn(0x7f687406d0b0 0x7f687406d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 --2- 172.21.15.164:0/1673491632 >> v2:172.21.15.164:3300/0 conn(0x7f68740a3e60 0x7f687406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 -- 172.21.15.164:0/1673491632 >> 172.21.15.164:0/1673491632 conn(0x7f68740a4f50 msgr2=0x7f68740a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 -- 172.21.15.164:0/1673491632 shutdown_connections 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 1 -- 172.21.15.164:0/1673491632 wait complete. 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.555+0000 7f687b7c6640 10 monclient: monmap: 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 Processor -- start 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 -- start start 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient: init 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-c con 0x7f687406d0b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68740a3e60 0x7f68740738e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-b con 0x7f68740a3e60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 0x7f6874073e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): picked mon.noname-a con 0x7f687413c5e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:28.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f687406d9d0 con 0x7f687413c5e0 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f687409bd30 con 0x7f68740a3e60 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f687406db50 con 0x7f687406d0b0 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f687b7c6640 10 monclient(hunting): authenticate will time out at 767.043661s 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47314/0 (socket says 172.21.15.164:47314) 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 0x7f6874073e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 1 -- 172.21.15.164:0/894253224 learned_addr learned my addr 172.21.15.164:0/894253224 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): get_auth_request con 0x7f687406d0b0 auth_method 0 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): get_auth_request con 0x7f687413c5e0 auth_method 0 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879d63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68740a3e60 0x7f68740738e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): get_auth_request con 0x7f68740a3e60 auth_method 0 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6878d61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.556+0000 7f6879562640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 -- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68740a3e60 msgr2=0x7f68740738e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68740a3e60 0x7f68740738e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 -- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 msgr2=0x7f6874073e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 0x7f6874073e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 monclient: found mon.noname-c 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879562640 1 -- 172.21.15.164:0/894253224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68740944e0 con 0x7f687406d0b0 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.557+0000 7f6879d63640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 0x7f6874073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6879562640 1 -- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 msgr2=0x7f687409a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6879562640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f687000e9b0 tx=0x7f687000ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870017070 con 0x7f687406d0b0 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: dump: 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.577 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f687000cbb0 con 0x7f687406d0b0 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 5 monclient: authenticate success, global_id 4316 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 10 monclient: _renew_subs 2024-09-10T16:06:28.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68740a67a0 con 0x7f687406d0b0 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870013470 con 0x7f687406d0b0 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f6869ffb640 10 monclient: dump: 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.579 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f686b7fe640 4 set_mon_vals no callback set 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 10 monclient: _renew_subs 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.559+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f68740947c0 con 0x7f687406d0b0 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f687001a070 con 0x7f687406d0b0 2024-09-10T16:06:28.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f6869ffb640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 0x7f68580406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f687b7c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f687b7c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f687b7c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f6870052a30 con 0x7f687406d0b0 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.560+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f687406d490 con 0x7f687406d0b0 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.561+0000 7f6878d61640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 0x7f68580406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.561+0000 7f6878d61640 10 monclient: get_auth_request con 0x7f685803e210 auth_method 0 2024-09-10T16:06:28.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.563+0000 7f6878d61640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 0x7f68580406d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f687409ac30 tx=0x7f6864008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.565+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6870023d00 con 0x7f687406d0b0 2024-09-10T16:06:28.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.565+0000 7f6869ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.565+0000 7f6869ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:28.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.738+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f68740770e0 con 0x7f685803e210 2024-09-10T16:06:28.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.739+0000 7f6869ffb640 1 -- 172.21.15.164:0/894253224 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34190 (secure 0 0 0) 0x7f68740770e0 con 0x7f685803e210 2024-09-10T16:06:28.741 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:28.741 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 msgr2=0x7f68580406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 0x7f68580406d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f687409ac30 tx=0x7f6864008040 comp rx=0 tx=0).stop 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 10 monclient: shutdown 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 msgr2=0x7f687409a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f687000e9b0 tx=0x7f687000ee70 comp rx=0 tx=0).stop 2024-09-10T16:06:28.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 shutdown_connections 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f685803e210 0x7f68580406d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f687413c5e0 0x7f6874073e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68740a3e60 0x7f68740738e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 --2- 172.21.15.164:0/894253224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f687406d0b0 0x7f687409a160 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 >> 172.21.15.164:0/894253224 conn(0x7f68740a4f50 msgr2=0x7f687413bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 shutdown_connections 2024-09-10T16:06:28.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.741+0000 7f687b7c6640 1 -- 172.21.15.164:0/894253224 wait complete. 2024-09-10T16:06:28.753 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-10T16:06:27.837822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163376,"kb_used_data":1184,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067664,"statfs":{"total":579820584960,"available":579653287936,"internally_reserved":0,"allocated":1212416,"data_stored":1415842,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":165927587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.422616"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428562+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:06:24.428562+0000","last_peered":"2024-09-10T16:06:24.428562+0000","last_clean":"2024-09-10T16:06:24.428562+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:06:24.428562+0000","last_undegraded":"2024-09-10T16:06:24.428562+0000","last_fullsized":"2024-09-10T16:06:24.428562+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.437270+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:06:24.437270+0000","last_peered":"2024-09-10T16:06:24.437270+0000","last_clean":"2024-09-10T16:06:24.437270+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:06:24.437270+0000","last_undegraded":"2024-09-10T16:06:24.437270+0000","last_fullsized":"2024-09-10T16:06:24.437270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428235+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:06:24.428235+0000","last_peered":"2024-09-10T16:06:24.428235+0000","last_clean":"2024-09-10T16:06:24.428235+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:06:24.428235+0000","last_undegraded":"2024-09-10T16:06:24.428235+0000","last_fullsized":"2024-09-10T16:06:24.428235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.520319+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:06:24.520319+0000","last_peered":"2024-09-10T16:06:24.520319+0000","last_clean":"2024-09-10T16:06:24.520319+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:06:24.520319+0000","last_undegraded":"2024-09-10T16:06:24.520319+0000","last_fullsized":"2024-09-10T16:06:24.520319+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.432116+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:06:24.432116+0000","last_peered":"2024-09-10T16:06:24.432116+0000","last_clean":"2024-09-10T16:06:24.432116+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:06:24.432116+0000","last_undegraded":"2024-09-10T16:06:24.432116+0000","last_fullsized":"2024-09-10T16:06:24.432116+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.429954+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:06:24.429954+0000","last_peered":"2024-09-10T16:06:24.429954+0000","last_clean":"2024-09-10T16:06:24.429954+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:06:24.429954+0000","last_undegraded":"2024-09-10T16:06:24.429954+0000","last_fullsized":"2024-09-10T16:06:24.429954+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427455+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:06:24.427455+0000","last_peered":"2024-09-10T16:06:24.427455+0000","last_clean":"2024-09-10T16:06:24.427455+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:06:24.427455+0000","last_undegraded":"2024-09-10T16:06:24.427455+0000","last_fullsized":"2024-09-10T16:06:24.427455+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428654+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:06:24.428654+0000","last_peered":"2024-09-10T16:06:24.428654+0000","last_clean":"2024-09-10T16:06:24.428654+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:06:24.428654+0000","last_undegraded":"2024-09-10T16:06:24.428654+0000","last_fullsized":"2024-09-10T16:06:24.428654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":112,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427527+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:06:24.427527+0000","last_peered":"2024-09-10T16:06:24.427527+0000","last_clean":"2024-09-10T16:06:24.427527+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:06:24.427527+0000","last_undegraded":"2024-09-10T16:06:24.427527+0000","last_fullsized":"2024-09-10T16:06:24.427527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:06:28.754 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T16:06:28.754 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T16:06:28.754 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-10T16:06:28.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-10T16:06:28.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.850+0000 7fc7e9d82640 10 monclient: get_monmap_and_config 2024-09-10T16:06:28.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.850+0000 7fc7e9d82640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.850+0000 7fc7e9d82640 10 monclient: monmap: 2024-09-10T16:06:28.852 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:28.851806+0000 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:28.851806+0000 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:28.853 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.851+0000 7fc7e9d82640 1 Processor -- start 2024-09-10T16:06:28.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.851+0000 7fc7e9d82640 1 -- start start 2024-09-10T16:06:28.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.851+0000 7fc7e9d82640 10 monclient: init 2024-09-10T16:06:28.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc7e406d160 0x7fc7e4134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-c con 0x7fc7e406d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7e406d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7e413d060 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7e409ce70 con 0x7fc7e406d540 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7e409d150 con 0x7fc7e413d060 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc7e409cc20 con 0x7fc7e406d160 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e9d82640 10 monclient(hunting): authenticate will time out at 767.339378s 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc7e406d540 0x7fc7e40a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59390/0 (socket says 172.21.15.164:59390) 2024-09-10T16:06:28.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e2ffd640 1 -- 172.21.15.164:0/3535768854 learned_addr learned my addr 172.21.15.164:0/3535768854 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 1 --2- 172.21.15.164:0/3535768854 >> v2:172.21.15.165:3300/0 conn(0x7fc7e406d160 0x7fc7e4134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): get_auth_request con 0x7fc7e406d160 auth_method 0 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.852+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.854+0000 7fc7e2ffd640 1 -- 172.21.15.164:0/3535768854 >> v1:172.21.15.164:6789/0 conn(0x7fc7e406d540 legacy=0x7fc7e40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.854+0000 7fc7e3fff640 1 -- 172.21.15.164:0/3535768854 >> v1:172.21.15.165:6789/0 conn(0x7fc7e413d060 legacy=0x7fc7e413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 1 -- 172.21.15.164:0/3535768854 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098740852 0 0) 0x7fc7e409ce70 con 0x7fc7e406d540 2024-09-10T16:06:28.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 10 monclient(hunting): _negotiate 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 10 monclient(hunting): my global_id is 4314 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e27fc640 1 -- 172.21.15.164:0/3535768854 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7d0003520 con 0x7fc7e406d540 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_done global_id 4309 payload 274 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 1 -- 172.21.15.164:0/3535768854 >> v1:172.21.15.165:6789/0 conn(0x7fc7e413d060 legacy=0x7fc7e413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 1 -- 172.21.15.164:0/3535768854 >> v1:172.21.15.164:6789/0 conn(0x7fc7e406d540 legacy=0x7fc7e40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:06:28.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.855+0000 7fc7e37fe640 1 -- 172.21.15.164:0/3535768854 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e40a4df0 con 0x7fc7e406d160 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e37fe640 1 -- 172.21.15.164:0/3535768854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e406d160 msgr2=0x7fc7e4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e37fe640 1 --2- 172.21.15.164:0/3535768854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e406d160 0x7fc7e4134e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40098a0 tx=0x7fc7d402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=844bf5e81ccd204f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 1 -- 172.21.15.164:0/3535768854 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4037070 con 0x7fc7e406d160 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: dump: 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.867 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 1 -- 172.21.15.164:0/3535768854 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d4032640 con 0x7fc7e406d160 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 1 -- 172.21.15.164:0/3535768854 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4032a10 con 0x7fc7e406d160 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e27fc640 10 monclient: dump: 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.869 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 5 monclient: authenticate success, global_id 4309 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 4 set_mon_vals no callback set 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 10 monclient: shutdown 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 -- 172.21.15.164:0/3535768854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e406d160 msgr2=0x7fc7e4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/3535768854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e406d160 0x7fc7e4134e00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc7d40098a0 tx=0x7fc7d402a210 comp rx=0 tx=0).stop 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 -- 172.21.15.164:0/3535768854 shutdown_connections 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/3535768854 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e406d160 0x7fc7e4134e00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 -- 172.21.15.164:0/3535768854 >> 172.21.15.164:0/3535768854 conn(0x7fc7e40aacb0 msgr2=0x7fc7e40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 -- 172.21.15.164:0/3535768854 shutdown_connections 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.858+0000 7fc7e9d82640 1 -- 172.21.15.164:0/3535768854 wait complete. 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.859+0000 7fc7e9d82640 10 monclient: build_initial_monmap 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.859+0000 7fc7e9d82640 10 monclient: monmap: 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:28.871 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.859+0000 7fc7e9d82640 1 Processor -- start 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 -- start start 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient: init 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-c con 0x7fc7e406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:28.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-a con 0x7fc7e406d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e413d060 0x7fc7e409e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): picked mon.noname-b con 0x7fc7e413d060 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4140cf0 con 0x7fc7e406d540 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc7e4140b70 con 0x7fc7e413d060 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc7e4140e70 con 0x7fc7e406d160 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e9d82640 10 monclient(hunting): authenticate will time out at 767.347466s 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47330/0 (socket says 172.21.15.164:47330) 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36156/0 (socket says 172.21.15.164:36156) 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 -- 172.21.15.164:0/1813756420 learned_addr learned my addr 172.21.15.164:0/1813756420 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): get_auth_request con 0x7fc7e406d160 auth_method 0 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): get_auth_request con 0x7fc7e406d540 auth_method 0 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e3fff640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e413d060 0x7fc7e409e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 -- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e413d060 msgr2=0x7fc7e409e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e413d060 0x7fc7e409e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 -- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 msgr2=0x7fc7e412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e37fe640 1 -- 172.21.15.164:0/1813756420 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7e406e6b0 con 0x7fc7e406d160 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.860+0000 7fc7e2ffd640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e37fe640 1 -- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 msgr2=0x7fc7e412e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e37fe640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4035040 tx=0x7fc7d40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4041030 con 0x7fc7e406d160 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: dump: 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.876 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: _finish_auth 0 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7d403d630 con 0x7fc7e406d160 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 5 monclient: authenticate success, global_id 4319 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 10 monclient: _renew_subs 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7e409ff30 con 0x7fc7e406d160 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7d4037040 con 0x7fc7e406d160 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7dbfff640 10 monclient: dump: 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:28.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e17fa640 4 set_mon_vals no callback set 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 10 monclient: _renew_subs 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.863+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7e40a19f0 con 0x7fc7e406d160 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc7d4044070 con 0x7fc7e406d160 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e9d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e9d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e9d82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7e40a4720 con 0x7fc7e406d160 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7dbfff640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 0x7fc7c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fc7d4073020 con 0x7fc7e406d160 2024-09-10T16:06:28.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e2ffd640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 0x7fc7c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:28.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.864+0000 7fc7e2ffd640 10 monclient: get_auth_request con 0x7fc7c403e210 auth_method 0 2024-09-10T16:06:28.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.867+0000 7fc7e2ffd640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 0x7fc7c40406d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc001590 tx=0x7fc7cc007b70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:28.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.868+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc7e406e6b0 con 0x7fc7e406d160 2024-09-10T16:06:28.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.868+0000 7fc7dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:28.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:28.868+0000 7fc7dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:29.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.080+0000 7fc7e9d82640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-10T16:06:29.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.080+0000 7fc7e9d82640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-10T16:06:29.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.080+0000 7fc7e9d82640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.080+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fc7e4134e00 con 0x7fc7e406d160 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.081+0000 7fc7dbfff640 1 -- 172.21.15.164:0/1813756420 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fc7d4046410 con 0x7fc7e406d160 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.081+0000 7fc7dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.081+0000 7fc7dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:29.083 INFO:teuthology.orchestra.run.smithi164.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 msgr2=0x7fc7c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 0x7fc7c40406d0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc001590 tx=0x7fc7cc007b70 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 10 monclient: shutdown 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 msgr2=0x7fc7e412e320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc7d4035040 tx=0x7fc7d40046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 shutdown_connections 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7c403e210 0x7fc7c40406d0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7e413d060 0x7fc7e409e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7e406d540 0x7fc7e412e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 --2- 172.21.15.164:0/1813756420 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7e406d160 0x7fc7e412e320 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 >> 172.21.15.164:0/1813756420 conn(0x7fc7e40aacb0 msgr2=0x7fc7e412f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 shutdown_connections 2024-09-10T16:06:29.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.083+0000 7fc7e9d82640 1 -- 172.21.15.164:0/1813756420 wait complete. 2024-09-10T16:06:29.095 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-10T16:06:29.095 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-10T16:06:29.140 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-10T16:06:29.140 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:06:29.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 10 monclient: get_monmap_and_config 2024-09-10T16:06:29.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 10 monclient: monmap: 2024-09-10T16:06:29.237 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:29.236983+0000 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:29.236983+0000 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:29.238 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 1 Processor -- start 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 1 -- start start 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.236+0000 7fce8ff5d640 10 monclient: init 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:06:29.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce8809d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce8809ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce8809e420 0x7fce8813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fce8809e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce880a0870 con 0x7fce8809d930 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce88074c70 con 0x7fce8809ab50 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fce8809d3e0 con 0x7fce8809e420 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8ff5d640 10 monclient(hunting): authenticate will time out at 767.724391s 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8dcf9640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fce8809d930 0x7fce8809dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59396/0 (socket says 172.21.15.164:59396) 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8dcf9640 1 -- 172.21.15.164:0/4247741079 learned_addr learned my addr 172.21.15.164:0/4247741079 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:29.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/4247741079 >> v2:172.21.15.165:3300/0 conn(0x7fce8809e420 0x7fce8813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request con 0x7fce8809e420 auth_method 0 2024-09-10T16:06:29.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8e4fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.237+0000 7fce8e4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8dcf9640 1 -- 172.21.15.164:0/4247741079 >> v1:172.21.15.164:6789/0 conn(0x7fce8809d930 legacy=0x7fce8809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8d4f8640 1 -- 172.21.15.164:0/4247741079 >> v1:172.21.15.165:6789/0 conn(0x7fce8809ab50 legacy=0x7fce881397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1460341363 0 0) 0x7fce880a0870 con 0x7fce8809d930 2024-09-10T16:06:29.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8ccf7640 10 monclient(hunting): _negotiate 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8ccf7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8ccf7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.239+0000 7fce8ccf7640 10 monclient(hunting): my global_id is 4320 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce74003520 con 0x7fce8809d930 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 396178110 0 0) 0x7fce74003520 con 0x7fce8809d930 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 >> v2:172.21.15.165:3300/0 conn(0x7fce8809e420 msgr2=0x7fce8813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 --2- 172.21.15.164:0/4247741079 >> v2:172.21.15.165:3300/0 conn(0x7fce8809e420 0x7fce8813cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 >> v1:172.21.15.165:6789/0 conn(0x7fce8809ab50 legacy=0x7fce881397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:06:29.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce8809ecb0 con 0x7fce8809d930 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fce7c001a60 con 0x7fce8809d930 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: dump: 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.253 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 >> v1:172.21.15.164:6789/0 conn(0x7fce8809d930 legacy=0x7fce8809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce74003ac0 0x7fce74003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): picked mon.b con 0x7fce74003ac0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:29.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce74004800 0x7fce74004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): picked mon.c con 0x7fce74004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce7400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): picked mon.a con 0x7fce740053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce8809d3e0 con 0x7fce740053c0 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce7400c2a0 con 0x7fce74003ac0 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fce7400d5b0 con 0x7fce74004800 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ccf7640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:06:29.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce74004800 0x7fce74004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce7400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce74003ac0 0x7fce74003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request con 0x7fce740053c0 auth_method 0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): get_auth_request con 0x7fce74004800 auth_method 0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): get_auth_request con 0x7fce74003ac0 auth_method 0 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_done global_id 4320 payload 293 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 -- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce74004800 msgr2=0x7fce74004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce74004800 0x7fce74004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 -- 172.21.15.164:0/4247741079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce74003ac0 msgr2=0x7fce74003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce74003ac0 0x7fce74003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 monclient: found mon.a 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 1 -- 172.21.15.164:0/4247741079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce74003520 con 0x7fce740053c0 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8e4fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ff5d640 5 monclient: authenticate success, global_id 4320 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.240+0000 7fce8ff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.241+0000 7fce8e4fa640 1 -- 172.21.15.164:0/4247741079 reap_dead start 2024-09-10T16:06:29.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8e4fa640 1 -- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 msgr2=0x7fce7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce7400b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce80009fc0 tx=0x7fce74005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e6ba1339e9af85c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce80035020 con 0x7fce740053c0 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: dump: 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.259 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ff5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce80003b90 con 0x7fce740053c0 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ff5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ff5d640 4 set_mon_vals no callback set 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 1 -- 172.21.15.164:0/4247741079 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce80009c10 con 0x7fce740053c0 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.243+0000 7fce8ccf7640 10 monclient: dump: 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.260 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 10 monclient: shutdown 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 -- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 msgr2=0x7fce7400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce7400b9c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce80009fc0 tx=0x7fce74005830 comp rx=0 tx=0).stop 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 -- 172.21.15.164:0/4247741079 shutdown_connections 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/4247741079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce7400b9c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 -- 172.21.15.164:0/4247741079 >> 172.21.15.164:0/4247741079 conn(0x7fce880a3110 msgr2=0x7fce880a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 -- 172.21.15.164:0/4247741079 shutdown_connections 2024-09-10T16:06:29.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.244+0000 7fce8ff5d640 1 -- 172.21.15.164:0/4247741079 wait complete. 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient: monmap: 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:29.262 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 Processor -- start 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 -- start start 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient: init 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce8812f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-a con 0x7fce740053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-c con 0x7fce8812fe40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881d1a20 0x7fce881d3e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): picked mon.noname-b con 0x7fce881d1a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce8809ef30 con 0x7fce740053c0 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce880a0090 con 0x7fce881d1a20 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fce880a2940 con 0x7fce8812fe40 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.245+0000 7fce8ff5d640 10 monclient(hunting): authenticate will time out at 767.732983s 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47360/0 (socket says 172.21.15.164:47360) 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 -- 172.21.15.164:0/1141622125 learned_addr learned my addr 172.21.15.164:0/1141622125 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): get_auth_request con 0x7fce8812fe40 auth_method 0 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce8812f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): get_auth_request con 0x7fce740053c0 auth_method 0 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881d1a20 0x7fce881d3e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request con 0x7fce881d1a20 auth_method 0 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8e4fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 -- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881d1a20 msgr2=0x7fce881d3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881d1a20 0x7fce881d3e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 -- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 msgr2=0x7fce8812f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce8812f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 monclient: found mon.noname-c 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8d4f8640 1 -- 172.21.15.164:0/1141622125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce880a5f30 con 0x7fce8812fe40 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.246+0000 7fce8dcf9640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce8812f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8d4f8640 1 -- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 msgr2=0x7fce881302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8d4f8640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce84006e20 tx=0x7fce840019c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84015070 con 0x7fce8812fe40 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: dump: 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.267 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce84011b20 con 0x7fce8812fe40 2024-09-10T16:06:29.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 5 monclient: authenticate success, global_id 4322 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce84010430 con 0x7fce8812fe40 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7a7fc640 10 monclient: dump: 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.269 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 10 monclient: _renew_subs 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce8809d620 con 0x7fce8812fe40 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce7bfff640 4 set_mon_vals no callback set 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 10 monclient: _renew_subs 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.248+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fce88073d70 con 0x7fce8812fe40 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fce84018070 con 0x7fce8812fe40 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce8ff5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce8ff5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce8ff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce88132130 con 0x7fce8812fe40 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.249+0000 7fce7a7fc640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 0x7fce6c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.250+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fce84052f60 con 0x7fce8812fe40 2024-09-10T16:06:29.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.250+0000 7fce8dcf9640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 0x7fce6c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.250+0000 7fce8dcf9640 10 monclient: get_auth_request con 0x7fce6c03e210 auth_method 0 2024-09-10T16:06:29.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.252+0000 7fce8dcf9640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 0x7fce6c0406d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0014a0 tx=0x7fce7c007b80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.254+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fce880a5f30 con 0x7fce8812fe40 2024-09-10T16:06:29.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.254+0000 7fce7a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.254+0000 7fce7a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:29.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.431+0000 7fce8ff5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:29.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.431+0000 7fce8ff5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:29.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.431+0000 7fce8ff5d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.431+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fce8809dd10 con 0x7fce8812fe40 2024-09-10T16:06:29.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.432+0000 7fce7a7fc640 1 -- 172.21.15.164:0/1141622125 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+12396 (secure 0 0 0) 0x7fce8400f360 con 0x7fce8812fe40 2024-09-10T16:06:29.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.432+0000 7fce7a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:06:29.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.432+0000 7fce7a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:29.434 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:29.434 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":20,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:06:27.831164+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T16:06:20.696007+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 msgr2=0x7fce6c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 0x7fce6c0406d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fce7c0014a0 tx=0x7fce7c007b80 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 10 monclient: shutdown 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 msgr2=0x7fce881302c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fce84006e20 tx=0x7fce840019c0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 shutdown_connections 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce6c03e210 0x7fce6c0406d0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce881d1a20 0x7fce881d3e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce8812fe40 0x7fce881302c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 --2- 172.21.15.164:0/1141622125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce740053c0 0x7fce8812f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 >> 172.21.15.164:0/1141622125 conn(0x7fce880a3110 msgr2=0x7fce881bad80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 shutdown_connections 2024-09-10T16:06:29.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.435+0000 7fce8ff5d640 1 -- 172.21.15.164:0/1141622125 wait complete. 2024-09-10T16:06:29.447 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T16:06:13.375364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-10T16:06:20.696007+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T16:06:29.447 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.547+0000 7f677dc5a640 10 monclient: get_monmap_and_config 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.547+0000 7f677dc5a640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.547+0000 7f677dc5a640 10 monclient: monmap: 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:29.548537+0000 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:29.548537+0000 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:29.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 Processor -- start 2024-09-10T16:06:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 -- start start 2024-09-10T16:06:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient: init 2024-09-10T16:06:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-a con 0x7f6778135ce0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f67780a40d0 0x7f67781360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-c con 0x7f67780a40d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-b con 0x7f6778136600 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f67780a1400 con 0x7f6778135ce0 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f67780a4ac0 con 0x7f67780a40d0 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f67780a1120 con 0x7f6778136600 2024-09-10T16:06:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.548+0000 7f677dc5a640 10 monclient(hunting): authenticate will time out at 768.035970s 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47364/0 (socket says 172.21.15.164:47364) 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36180/0 (socket says 172.21.15.164:36180) 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 -- 172.21.15.164:0/818997445 learned_addr learned my addr 172.21.15.164:0/818997445 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): get_auth_request con 0x7f6778136600 auth_method 0 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): get_auth_request con 0x7f6778135ce0 auth_method 0 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.165:3300/0 conn(0x7f67780a40d0 0x7f67781360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): get_auth_request con 0x7f67780a40d0 auth_method 0 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 -- 172.21.15.164:0/818997445 >> v2:172.21.15.165:3300/0 conn(0x7f67780a40d0 msgr2=0x7f67781360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.165:3300/0 conn(0x7f67780a40d0 0x7f67781360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 -- 172.21.15.164:0/818997445 >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 msgr2=0x7f67780a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 monclient: found mon.noname-b 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f6777fff640 1 -- 172.21.15.164:0/818997445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67780a28e0 con 0x7f6778136600 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.549+0000 7f67777fe640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f6777fff640 1 -- 172.21.15.164:0/818997445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 msgr2=0x7f677809b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f6777fff640 1 --2- 172.21.15.164:0/818997445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6778070a90 tx=0x7f677002a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66c3e5d8733febbd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 1 -- 172.21.15.164:0/818997445 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770037070 con 0x7f6778136600 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: dump: 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 1 -- 172.21.15.164:0/818997445 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6770032640 con 0x7f6778136600 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 5 monclient: authenticate success, global_id 4325 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 1 -- 172.21.15.164:0/818997445 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6770032a10 con 0x7f6778136600 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 4 set_mon_vals no callback set 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f67767fc640 10 monclient: dump: 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 10 monclient: shutdown 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 1 -- 172.21.15.164:0/818997445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 msgr2=0x7f677809b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.552+0000 7f677dc5a640 1 --2- 172.21.15.164:0/818997445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6778070a90 tx=0x7f677002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 -- 172.21.15.164:0/818997445 shutdown_connections 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 --2- 172.21.15.164:0/818997445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778136600 0x7f677809b0e0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.165:3300/0 conn(0x7f67780a40d0 0x7f67781360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 --2- 172.21.15.164:0/818997445 >> v2:172.21.15.164:3300/0 conn(0x7f6778135ce0 0x7f67780a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 -- 172.21.15.164:0/818997445 >> 172.21.15.164:0/818997445 conn(0x7f67780a4f50 msgr2=0x7f67780a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 -- 172.21.15.164:0/818997445 shutdown_connections 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 1 -- 172.21.15.164:0/818997445 wait complete. 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.553+0000 7f677dc5a640 10 monclient: monmap: 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:29.564 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 Processor -- start 2024-09-10T16:06:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 -- start start 2024-09-10T16:06:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient: init 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-a con 0x7f67780a40d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 0x7f67781cd910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-c con 0x7f6778135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6778136600 0x7f6778073c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): picked mon.noname-b con 0x7f6778136600 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f677809b7a0 con 0x7f67780a40d0 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f677809b620 con 0x7f6778136600 2024-09-10T16:06:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f677809b920 con 0x7f6778135ce0 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f677dc5a640 10 monclient(hunting): authenticate will time out at 768.041712s 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 0x7f67781cd910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36194/0 (socket says 172.21.15.164:36194) 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 1 -- 172.21.15.164:0/3329734998 learned_addr learned my addr 172.21.15.164:0/3329734998 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): get_auth_request con 0x7f67780a40d0 auth_method 0 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): get_auth_request con 0x7f6778135ce0 auth_method 0 2024-09-10T16:06:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f67777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6777fff640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6778136600 0x7f6778073c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6776ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6777fff640 10 monclient(hunting): get_auth_request con 0x7f6778136600 auth_method 0 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6777fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6777fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.554+0000 7f6777fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f6777fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 -- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 msgr2=0x7f67781cd910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 0x7f67781cd910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 -- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6778136600 msgr2=0x7f6778073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6778136600 0x7f6778073c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f67777fe640 1 -- 172.21.15.164:0/3329734998 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f677806e3f0 con 0x7f67780a40d0 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.555+0000 7f6776ffd640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 0x7f67781cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f67777fe640 1 -- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 msgr2=0x7f67781cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f67777fe640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f676800efc0 tx=0x7f676800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6768018020 con 0x7f67780a40d0 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: dump: 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.573 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6768014890 con 0x7f67780a40d0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 5 monclient: authenticate success, global_id 4329 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 10 monclient: _renew_subs 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f677806e6d0 con 0x7f67780a40d0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6768014c60 con 0x7f67780a40d0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f675ffff640 10 monclient: dump: 2024-09-10T16:06:29.574 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f6775ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 10 monclient: _renew_subs 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.557+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f677809c140 con 0x7f67780a40d0 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6768005d10 con 0x7f67780a40d0 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f675ffff640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 0x7f67640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f677dc5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f6768053a70 con 0x7f67780a40d0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f677dc5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f677dc5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677809bd20 con 0x7f67780a40d0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f6776ffd640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 0x7f67640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.558+0000 7f6776ffd640 10 monclient: get_auth_request con 0x7f676403e210 auth_method 0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.561+0000 7f6776ffd640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 0x7f67640406d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f677806ecd0 tx=0x7f676c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.563+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6768022030 con 0x7f67780a40d0 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.563+0000 7f675ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.563+0000 7f675ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:29.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.739+0000 7f677dc5a640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.739+0000 7f677dc5a640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.739+0000 7f677dc5a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.739+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f67781cde50 con 0x7f67780a40d0 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.740+0000 7f675ffff640 1 -- 172.21.15.164:0/3329734998 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f6768010040 con 0x7f67780a40d0 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.740+0000 7f675ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.740+0000 7f675ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:29.742 INFO:teuthology.orchestra.run.smithi164.stdout:pg_num: 1 2024-09-10T16:06:29.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 msgr2=0x7f67640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 0x7f67640406d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f677806ecd0 tx=0x7f676c009190 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 10 monclient: shutdown 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 msgr2=0x7f67781cd3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f676800efc0 tx=0x7f676800c490 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 shutdown_connections 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f676403e210 0x7f67640406d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6778136600 0x7f6778073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6778135ce0 0x7f67781cd910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 --2- 172.21.15.164:0/3329734998 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f67780a40d0 0x7f67781cd3d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 >> 172.21.15.164:0/3329734998 conn(0x7f67780a4f50 msgr2=0x7f677809c350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 shutdown_connections 2024-09-10T16:06:29.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.742+0000 7f677dc5a640 1 -- 172.21.15.164:0/3329734998 wait complete. 2024-09-10T16:06:29.754 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-10T16:06:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.854+0000 7f7c0b47f640 10 monclient: get_monmap_and_config 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.854+0000 7f7c0b47f640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.854+0000 7f7c0b47f640 10 monclient: monmap: 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:29.855333+0000 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:29.855333+0000 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:29.857 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 Processor -- start 2024-09-10T16:06:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 -- start start 2024-09-10T16:06:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient: init 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7c0406cd00 0x7f7c04135000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c0406cd00 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c040a46e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c04137bf0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c0406df10 con 0x7f7c04137bf0 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7c0406d0e0 con 0x7f7c0406cd00 2024-09-10T16:06:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c0409cb90 con 0x7f7c040a46e0 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c0b47f640 10 monclient(hunting): authenticate will time out at 768.342848s 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47376/0 (socket says 172.21.15.164:47376) 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 1 -- 172.21.15.164:0/1106506445 learned_addr learned my addr 172.21.15.164:0/1106506445 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 10 monclient(hunting): get_auth_request con 0x7f7c040a46e0 auth_method 0 2024-09-10T16:06:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.855+0000 7f7c08a1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c0921b640 1 --2- 172.21.15.164:0/1106506445 >> v2:172.21.15.165:3300/0 conn(0x7f7c0406cd00 0x7f7c04135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 1 -- 172.21.15.164:0/1106506445 >> v2:172.21.15.165:3300/0 conn(0x7f7c0406cd00 msgr2=0x7f7c04135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 1 --2- 172.21.15.164:0/1106506445 >> v2:172.21.15.165:3300/0 conn(0x7f7c0406cd00 0x7f7c04135000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 1 -- 172.21.15.164:0/1106506445 >> v1:172.21.15.165:6789/0 conn(0x7f7c04137bf0 legacy=0x7f7c04138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 1 monclient: found mon.noname-b 2024-09-10T16:06:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.856+0000 7f7c08a1a640 1 -- 172.21.15.164:0/1106506445 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0409ce70 con 0x7f7c040a46e0 2024-09-10T16:06:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c08a1a640 1 -- 172.21.15.164:0/1106506445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 msgr2=0x7f7c040a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c08a1a640 1 --2- 172.21.15.164:0/1106506445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009870 tx=0x7f7bf402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cc0ae2b08d4113a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 1 -- 172.21.15.164:0/1106506445 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4037070 con 0x7f7c040a46e0 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: dump: 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 5 monclient: authenticate success, global_id 4331 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 1 -- 172.21.15.164:0/1106506445 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf4032640 con 0x7f7c040a46e0 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 4 set_mon_vals no callback set 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 1 -- 172.21.15.164:0/1106506445 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4032a10 con 0x7f7c040a46e0 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7bfbfff640 10 monclient: dump: 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.872 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 10 monclient: shutdown 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 -- 172.21.15.164:0/1106506445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 msgr2=0x7f7c040a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/1106506445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7bf4009870 tx=0x7f7bf402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 -- 172.21.15.164:0/1106506445 shutdown_connections 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/1106506445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a4b00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/1106506445 >> v2:172.21.15.165:3300/0 conn(0x7f7c0406cd00 0x7f7c04135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 -- 172.21.15.164:0/1106506445 >> 172.21.15.164:0/1106506445 conn(0x7f7c040aaa70 msgr2=0x7f7c040aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 -- 172.21.15.164:0/1106506445 shutdown_connections 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.859+0000 7f7c0b47f640 1 -- 172.21.15.164:0/1106506445 wait complete. 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.860+0000 7f7c0b47f640 10 monclient: build_initial_monmap 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.860+0000 7f7c0b47f640 10 monclient: monmap: 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:29.873 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.860+0000 7f7c0b47f640 1 Processor -- start 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.860+0000 7f7c0b47f640 1 -- start start 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.860+0000 7f7c0b47f640 10 monclient: init 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:29.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c0406cd00 0x7f7c040a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c0406cd00 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c040a46e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 0x7f7c0413dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c04137bf0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c041388d0 con 0x7f7c04137bf0 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7c04138750 con 0x7f7c0406cd00 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c04138a50 con 0x7f7c040a46e0 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:29.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0b47f640 10 monclient(hunting): authenticate will time out at 768.348301s 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47386/0 (socket says 172.21.15.164:47386) 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 0x7f7c0413dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 -- 172.21.15.164:0/181801485 learned_addr learned my addr 172.21.15.164:0/181801485 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): get_auth_request con 0x7f7c040a46e0 auth_method 0 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): get_auth_request con 0x7f7c04137bf0 auth_method 0 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c0406cd00 0x7f7c040a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): get_auth_request con 0x7f7c0406cd00 auth_method 0 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c0921b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:29.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 -- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c0406cd00 msgr2=0x7f7c040a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c0406cd00 0x7f7c040a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 -- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 msgr2=0x7f7c0413dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 0x7f7c0413dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 monclient: found mon.noname-c 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c08a1a640 1 -- 172.21.15.164:0/181801485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c0406e670 con 0x7f7c040a46e0 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.861+0000 7f7c09a1c640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 0x7f7c0413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.863+0000 7f7c08a1a640 1 -- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 msgr2=0x7f7c040a3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.863+0000 7f7c08a1a640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40090b0 tx=0x7f7bf40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4041020 con 0x7f7c040a46e0 2024-09-10T16:06:29.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: dump: 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:06:29.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf403d830 con 0x7f7c040a46e0 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 5 monclient: authenticate success, global_id 4334 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4037040 con 0x7f7c040a46e0 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bf9ffb640 10 monclient: dump: 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:29.880 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 10 monclient: _renew_subs 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7bfb7fe640 4 set_mon_vals no callback set 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c0413f670 con 0x7f7c040a46e0 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 10 monclient: _renew_subs 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.864+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c0409e9f0 con 0x7f7c040a46e0 2024-09-10T16:06:29.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7bf4044020 con 0x7f7c040a46e0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0b47f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0b47f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0b47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c04135000 con 0x7f7c040a46e0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7bf9ffb640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 0x7f7be8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f7bf4072c70 con 0x7f7c040a46e0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0921b640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 0x7f7be8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.865+0000 7f7c0921b640 10 monclient: get_auth_request con 0x7f7be803e240 auth_method 0 2024-09-10T16:06:29.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.868+0000 7f7c0921b640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 0x7f7be8040700 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000aa20 tx=0x7f7c00008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:29.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.870+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7bf404b070 con 0x7f7c040a46e0 2024-09-10T16:06:29.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.870+0000 7f7bf9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:29.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:29.870+0000 7f7bf9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.044+0000 7f7c0b47f640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.044+0000 7f7c0b47f640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.044+0000 7f7c0b47f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.044+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f7c0409ecd0 con 0x7f7c040a46e0 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.045+0000 7f7bf9ffb640 1 -- 172.21.15.164:0/181801485 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f7bf4071800 con 0x7f7c040a46e0 2024-09-10T16:06:30.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.045+0000 7f7bf9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T16:06:30.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.045+0000 7f7bf9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:30.048 INFO:teuthology.orchestra.run.smithi164.stdout:pg_num: 8 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 msgr2=0x7f7be8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 0x7f7be8040700 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f7c0000aa20 tx=0x7f7c00008040 comp rx=0 tx=0).stop 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 10 monclient: shutdown 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 msgr2=0x7f7c040a3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7bf40090b0 tx=0x7f7bf40046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 shutdown_connections 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7be803e240 0x7f7be8040700 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c04137bf0 0x7f7c0413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c040a46e0 0x7f7c040a3d30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 --2- 172.21.15.164:0/181801485 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c0406cd00 0x7f7c040a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 >> 172.21.15.164:0/181801485 conn(0x7f7c040aaa70 msgr2=0x7f7c040a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 shutdown_connections 2024-09-10T16:06:30.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.047+0000 7f7c0b47f640 1 -- 172.21.15.164:0/181801485 wait complete. 2024-09-10T16:06:30.060 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-10T16:06:30.060 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:06:30.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.155+0000 7f28683ad640 10 monclient: get_monmap_and_config 2024-09-10T16:06:30.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.155+0000 7f28683ad640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.155+0000 7f28683ad640 10 monclient: monmap: 2024-09-10T16:06:30.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.159 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:30.156580+0000 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:30.156580+0000 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 Processor -- start 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 -- start start 2024-09-10T16:06:30.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient: init 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f286009c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f286009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f286013b5d0 0x7f286013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-a con 0x7f286013b5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286009a430 con 0x7f286009c180 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286009f860 con 0x7f286009d160 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f28600a24e0 con 0x7f286013b5d0 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.156+0000 7f28683ad640 10 monclient(hunting): authenticate will time out at 768.643966s 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f286013b5d0 0x7f286013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f286013b5d0 0x7f286013d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36212/0 (socket says 172.21.15.164:36212) 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f2866149640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f286009c180 0x7f286009ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59412/0 (socket says 172.21.15.164:59412) 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 -- 172.21.15.164:0/3618874349 learned_addr learned my addr 172.21.15.164:0/3618874349 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): get_auth_request con 0x7f286013b5d0 auth_method 0 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient(hunting): handle_auth_done global_id 4335 payload 274 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 -- 172.21.15.164:0/3618874349 >> v1:172.21.15.165:6789/0 conn(0x7f286009d160 legacy=0x7f286009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 -- 172.21.15.164:0/3618874349 >> v1:172.21.15.164:6789/0 conn(0x7f286009c180 legacy=0x7f286009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.157+0000 7f286694a640 1 -- 172.21.15.164:0/3618874349 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28600a2bf0 con 0x7f286013b5d0 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f286694a640 1 -- 172.21.15.164:0/3618874349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286013b5d0 msgr2=0x7f286013d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f286694a640 1 --2- 172.21.15.164:0/3618874349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286013b5d0 0x7f286013d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f28580098a0 tx=0x7f285802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d06a3b809ee384 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 1 -- 172.21.15.164:0/3618874349 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858037070 con 0x7f286013b5d0 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: dump: 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.169 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 5 monclient: authenticate success, global_id 4335 2024-09-10T16:06:30.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 1 -- 172.21.15.164:0/3618874349 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2858032640 con 0x7f286013b5d0 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 4 set_mon_vals no callback set 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 1 -- 172.21.15.164:0/3618874349 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2858032a10 con 0x7f286013b5d0 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f2865147640 10 monclient: dump: 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.171 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 10 monclient: shutdown 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 -- 172.21.15.164:0/3618874349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286013b5d0 msgr2=0x7f286013d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 --2- 172.21.15.164:0/3618874349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286013b5d0 0x7f286013d9c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f28580098a0 tx=0x7f285802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 -- 172.21.15.164:0/3618874349 shutdown_connections 2024-09-10T16:06:30.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 --2- 172.21.15.164:0/3618874349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286013b5d0 0x7f286013d9c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 -- 172.21.15.164:0/3618874349 >> 172.21.15.164:0/3618874349 conn(0x7f28600a3110 msgr2=0x7f28600a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.160+0000 7f28683ad640 1 -- 172.21.15.164:0/3618874349 shutdown_connections 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 1 -- 172.21.15.164:0/3618874349 wait complete. 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 10 monclient: monmap: 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:30.173 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 1 Processor -- start 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 1 -- start start 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.161+0000 7f28683ad640 10 monclient: init 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:30.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-c con 0x7f286009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 0x7f2860072bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-a con 0x7f286009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 0x7f28600730f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): picked mon.noname-b con 0x7f286013b5d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28600a2230 con 0x7f286009d160 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f28600a20b0 con 0x7f286013b5d0 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2860074e10 con 0x7f286009c180 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f28683ad640 10 monclient(hunting): authenticate will time out at 768.649309s 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 0x7f2860072bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47400/0 (socket says 172.21.15.164:47400) 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 1 -- 172.21.15.164:0/2416791606 learned_addr learned my addr 172.21.15.164:0/2416791606 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): get_auth_request con 0x7f286009c180 auth_method 0 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): get_auth_request con 0x7f286009d160 auth_method 0 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 0x7f28600730f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 10 monclient(hunting): get_auth_request con 0x7f286013b5d0 auth_method 0 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient(hunting): handle_auth_done global_id 4338 payload 824 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 -- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 msgr2=0x7f2860072670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 -- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 msgr2=0x7f28600730f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 0x7f28600730f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2865948640 1 -- 172.21.15.164:0/2416791606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286012c080 con 0x7f286009d160 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f286694a640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 0x7f28600730f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.162+0000 7f2866149640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.164+0000 7f2865948640 1 -- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 msgr2=0x7f2860072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.164+0000 7f2865948640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 0x7f2860072bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f285400c3e0 tx=0x7f285400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28540119a0 con 0x7f286009d160 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: dump: 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2854017070 con 0x7f286009d160 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854011d80 con 0x7f286009d160 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 5 monclient: authenticate success, global_id 4338 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 10 monclient: _renew_subs 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f286012c360 con 0x7f286009d160 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f285effd640 10 monclient: dump: 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.180 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f2864946640 4 set_mon_vals no callback set 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 10 monclient: _renew_subs 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.165+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f286009dbd0 con 0x7f286009d160 2024-09-10T16:06:30.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f28683ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f28683ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f28683ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f285401c460 con 0x7f286009d160 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f286009ea40 con 0x7f286009d160 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f285effd640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 0x7f284c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f28540528b0 con 0x7f286009d160 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f2866149640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 0x7f284c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.166+0000 7f2866149640 10 monclient: get_auth_request con 0x7f284c03e260 auth_method 0 2024-09-10T16:06:30.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.169+0000 7f2866149640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 0x7f284c040720 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28500058d0 tx=0x7f2850005020 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.170+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2854051440 con 0x7f286009d160 2024-09-10T16:06:30.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.170+0000 7f285effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.170+0000 7f285effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:30.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.341+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f28600a5070 con 0x7f284c03e260 2024-09-10T16:06:30.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.343+0000 7f285effd640 1 -- 172.21.15.164:0/2416791606 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34160 (secure 0 0 0) 0x7f28500059c0 con 0x7f284c03e260 2024-09-10T16:06:30.345 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:30.345 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:06:30.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 msgr2=0x7f284c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 0x7f284c040720 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f28500058d0 tx=0x7f2850005020 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 10 monclient: shutdown 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 msgr2=0x7f2860072bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 0x7f2860072bb0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f285400c3e0 tx=0x7f285400c8a0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 shutdown_connections 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284c03e260 0x7f284c040720 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f286013b5d0 0x7f28600730f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d160 0x7f2860072bb0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 --2- 172.21.15.164:0/2416791606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286009c180 0x7f2860072670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 >> 172.21.15.164:0/2416791606 conn(0x7f28600a3110 msgr2=0x7f28600a0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 shutdown_connections 2024-09-10T16:06:30.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.345+0000 7f28683ad640 1 -- 172.21.15.164:0/2416791606 wait complete. 2024-09-10T16:06:30.357 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-10T16:06:29.355455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163384,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067656,"statfs":{"total":579820584960,"available":579653279744,"internally_reserved":0,"allocated":1236992,"data_stored":1425046,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":165927587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001250"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428562+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:06:24.428562+0000","last_peered":"2024-09-10T16:06:24.428562+0000","last_clean":"2024-09-10T16:06:24.428562+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:06:24.428562+0000","last_undegraded":"2024-09-10T16:06:24.428562+0000","last_fullsized":"2024-09-10T16:06:24.428562+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.437270+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:06:24.437270+0000","last_peered":"2024-09-10T16:06:24.437270+0000","last_clean":"2024-09-10T16:06:24.437270+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:06:24.437270+0000","last_undegraded":"2024-09-10T16:06:24.437270+0000","last_fullsized":"2024-09-10T16:06:24.437270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428235+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:06:24.428235+0000","last_peered":"2024-09-10T16:06:24.428235+0000","last_clean":"2024-09-10T16:06:24.428235+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:06:24.428235+0000","last_undegraded":"2024-09-10T16:06:24.428235+0000","last_fullsized":"2024-09-10T16:06:24.428235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:28.378608+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:06:28.378608+0000","last_peered":"2024-09-10T16:06:28.378608+0000","last_clean":"2024-09-10T16:06:28.378608+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:06:28.378608+0000","last_undegraded":"2024-09-10T16:06:28.378608+0000","last_fullsized":"2024-09-10T16:06:28.378608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:27.836653+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:06:27.836653+0000","last_peered":"2024-09-10T16:06:27.836653+0000","last_clean":"2024-09-10T16:06:27.836653+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:06:27.836653+0000","last_undegraded":"2024-09-10T16:06:27.836653+0000","last_fullsized":"2024-09-10T16:06:27.836653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.429954+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:06:24.429954+0000","last_peered":"2024-09-10T16:06:24.429954+0000","last_clean":"2024-09-10T16:06:24.429954+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:06:24.429954+0000","last_undegraded":"2024-09-10T16:06:24.429954+0000","last_fullsized":"2024-09-10T16:06:24.429954+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:27.836167+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:06:27.836167+0000","last_peered":"2024-09-10T16:06:27.836167+0000","last_clean":"2024-09-10T16:06:27.836167+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:06:27.836167+0000","last_undegraded":"2024-09-10T16:06:27.836167+0000","last_fullsized":"2024-09-10T16:06:27.836167+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428654+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:06:24.428654+0000","last_peered":"2024-09-10T16:06:24.428654+0000","last_clean":"2024-09-10T16:06:24.428654+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:06:24.428654+0000","last_undegraded":"2024-09-10T16:06:24.428654+0000","last_fullsized":"2024-09-10T16:06:24.428654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":112,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427527+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:06:24.427527+0000","last_peered":"2024-09-10T16:06:24.427527+0000","last_clean":"2024-09-10T16:06:24.427527+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:06:24.427527+0000","last_undegraded":"2024-09-10T16:06:24.427527+0000","last_fullsized":"2024-09-10T16:06:24.427527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":64260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":64279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:06:30.358 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:06:30.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.454+0000 7f61ae756640 10 monclient: get_monmap_and_config 2024-09-10T16:06:30.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.454+0000 7f61ae756640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.454+0000 7f61ae756640 10 monclient: monmap: 2024-09-10T16:06:30.456 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.456 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:30.455677+0000 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:30.455677+0000 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:30.457 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 1 Processor -- start 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 1 -- start start 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 10 monclient: init 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.455+0000 7f61ae756640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f61a809c180 0x7f61a809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-c con 0x7f61a809c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61a809d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-b con 0x7f61a813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61a809a430 con 0x7f61a809d160 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f61a80a24e0 con 0x7f61a809c180 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f61a80a2bf0 con 0x7f61a813a5c0 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61ae756640 10 monclient(hunting): authenticate will time out at 768.943058s 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47414/0 (socket says 172.21.15.164:47414) 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 -- 172.21.15.164:0/1184211835 learned_addr learned my addr 172.21.15.164:0/1184211835 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): get_auth_request con 0x7f61a813a5c0 auth_method 0 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61a7fff640 1 --2- 172.21.15.164:0/1184211835 >> v2:172.21.15.165:3300/0 conn(0x7f61a809c180 0x7f61a809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61a7fff640 10 monclient(hunting): get_auth_request con 0x7f61a809c180 auth_method 0 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 -- 172.21.15.164:0/1184211835 >> v2:172.21.15.165:3300/0 conn(0x7f61a809c180 msgr2=0x7f61a809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 --2- 172.21.15.164:0/1184211835 >> v2:172.21.15.165:3300/0 conn(0x7f61a809c180 0x7f61a809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 -- 172.21.15.164:0/1184211835 >> v1:172.21.15.165:6789/0 conn(0x7f61a809d160 legacy=0x7f61a809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 monclient: found mon.noname-b 2024-09-10T16:06:30.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:30.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.456+0000 7f61accf3640 1 -- 172.21.15.164:0/1184211835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a809c560 con 0x7f61a813a5c0 2024-09-10T16:06:30.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61accf3640 1 -- 172.21.15.164:0/1184211835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 msgr2=0x7f61a813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61accf3640 1 --2- 172.21.15.164:0/1184211835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009e80 tx=0x7f61a002a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=411e0c4d32fc3ea6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 1 -- 172.21.15.164:0/1184211835 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0036070 con 0x7f61a813a5c0 2024-09-10T16:06:30.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: dump: 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 1 -- 172.21.15.164:0/1184211835 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a0007800 con 0x7f61a813a5c0 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 5 monclient: authenticate success, global_id 4340 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 1 -- 172.21.15.164:0/1184211835 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a0007bd0 con 0x7f61a813a5c0 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 4 set_mon_vals no callback set 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61a6ffd640 10 monclient: dump: 2024-09-10T16:06:30.470 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 10 monclient: shutdown 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 1 -- 172.21.15.164:0/1184211835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 msgr2=0x7f61a813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.459+0000 7f61ae756640 1 --2- 172.21.15.164:0/1184211835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f61a0009e80 tx=0x7f61a002a580 comp rx=0 tx=0).stop 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 -- 172.21.15.164:0/1184211835 shutdown_connections 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 --2- 172.21.15.164:0/1184211835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a813a5c0 0x7f61a813c9b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 --2- 172.21.15.164:0/1184211835 >> v2:172.21.15.165:3300/0 conn(0x7f61a809c180 0x7f61a809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 -- 172.21.15.164:0/1184211835 >> 172.21.15.164:0/1184211835 conn(0x7f61a80a3110 msgr2=0x7f61a80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 -- 172.21.15.164:0/1184211835 shutdown_connections 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 -- 172.21.15.164:0/1184211835 wait complete. 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 10 monclient: monmap: 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:30.472 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.460+0000 7f61ae756640 1 Processor -- start 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 -- start start 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient: init 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-a con 0x7f61a809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-c con 0x7f61a809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61a813a5c0 0x7f61a81c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): picked mon.noname-b con 0x7f61a813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f61a8074f90 con 0x7f61a809c180 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f61a8074e10 con 0x7f61a813a5c0 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f61a809f690 con 0x7f61a809d160 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61ae756640 10 monclient(hunting): authenticate will time out at 768.948606s 2024-09-10T16:06:30.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36244/0 (socket says 172.21.15.164:36244) 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47418/0 (socket says 172.21.15.164:47418) 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 1 -- 172.21.15.164:0/3336914749 learned_addr learned my addr 172.21.15.164:0/3336914749 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): get_auth_request con 0x7f61a809c180 auth_method 0 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): get_auth_request con 0x7f61a809d160 auth_method 0 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61a813a5c0 0x7f61a81c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): get_auth_request con 0x7f61a813a5c0 auth_method 0 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61accf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.461+0000 7f61a7fff640 10 monclient(hunting): handle_auth_done global_id 4341 payload 824 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 -- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 msgr2=0x7f61a81c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 -- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61a813a5c0 msgr2=0x7f61a81c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61a813a5c0 0x7f61a81c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a7fff640 1 -- 172.21.15.164:0/3336914749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61a809d620 con 0x7f61a809c180 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.462+0000 7f61a77fe640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:30.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a7fff640 1 -- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 msgr2=0x7f61a8099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a7fff640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f619800d800 tx=0x7f619800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619800b8d0 con 0x7f61a809c180 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: dump: 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.478 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 5 monclient: authenticate success, global_id 4341 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6198019070 con 0x7f61a809c180 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 10 monclient: _renew_subs 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f619800bcd0 con 0x7f61a809c180 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61a8092e70 con 0x7f61a809c180 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a67fc640 4 set_mon_vals no callback set 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61a4ff9640 10 monclient: dump: 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 10 monclient: _renew_subs 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.464+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61a8099c70 con 0x7f61a809c180 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61ae756640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61ae756640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61ae756640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61a8092920 con 0x7f61a809c180 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f619801e460 con 0x7f61a809c180 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61a4ff9640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 0x7f6194040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.465+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f61980530a0 con 0x7f61a809c180 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.466+0000 7f61a77fe640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 0x7f6194040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.466+0000 7f61a77fe640 10 monclient: get_auth_request con 0x7f619403e240 auth_method 0 2024-09-10T16:06:30.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.468+0000 7f61a77fe640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 0x7f6194040700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61a80a5c70 tx=0x7f619c007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.470+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f619801dce0 con 0x7f61a809c180 2024-09-10T16:06:30.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.470+0000 7f61a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.470+0000 7f61a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:30.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.642+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f61a8094da0 con 0x7f619403e240 2024-09-10T16:06:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.644+0000 7f61a4ff9640 1 -- 172.21.15.164:0/3336914749 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34160 (secure 0 0 0) 0x7f619c00fcb0 con 0x7f619403e240 2024-09-10T16:06:30.645 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:06:30.646 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:06:30.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 msgr2=0x7f6194040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 0x7f6194040700 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f61a80a5c70 tx=0x7f619c007e30 comp rx=0 tx=0).stop 2024-09-10T16:06:30.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 10 monclient: shutdown 2024-09-10T16:06:30.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 msgr2=0x7f61a8099730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f619800d800 tx=0x7f619800dcc0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 shutdown_connections 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f619403e240 0x7f6194040700 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61a813a5c0 0x7f61a81c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f61a809d160 0x7f61a81c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 --2- 172.21.15.164:0/3336914749 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f61a809c180 0x7f61a8099730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 >> 172.21.15.164:0/3336914749 conn(0x7f61a80a3110 msgr2=0x7f61a8139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 shutdown_connections 2024-09-10T16:06:30.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.646+0000 7f61ae756640 1 -- 172.21.15.164:0/3336914749 wait complete. 2024-09-10T16:06:30.658 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2024-09-10T16:06:29.355455+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163384,"kb_used_data":1208,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067656,"statfs":{"total":579820584960,"available":579653279744,"internally_reserved":0,"allocated":1236992,"data_stored":1425046,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":165927587},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001250"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428562+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:06:24.428562+0000","last_peered":"2024-09-10T16:06:24.428562+0000","last_clean":"2024-09-10T16:06:24.428562+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:06:24.428562+0000","last_undegraded":"2024-09-10T16:06:24.428562+0000","last_fullsized":"2024-09-10T16:06:24.428562+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.437270+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:06:24.437270+0000","last_peered":"2024-09-10T16:06:24.437270+0000","last_clean":"2024-09-10T16:06:24.437270+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:06:24.437270+0000","last_undegraded":"2024-09-10T16:06:24.437270+0000","last_fullsized":"2024-09-10T16:06:24.437270+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428235+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:06:24.428235+0000","last_peered":"2024-09-10T16:06:24.428235+0000","last_clean":"2024-09-10T16:06:24.428235+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:06:24.428235+0000","last_undegraded":"2024-09-10T16:06:24.428235+0000","last_fullsized":"2024-09-10T16:06:24.428235+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:28.378608+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:06:28.378608+0000","last_peered":"2024-09-10T16:06:28.378608+0000","last_clean":"2024-09-10T16:06:28.378608+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:06:28.378608+0000","last_undegraded":"2024-09-10T16:06:28.378608+0000","last_fullsized":"2024-09-10T16:06:28.378608+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":24,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:27.836653+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:06:27.836653+0000","last_peered":"2024-09-10T16:06:27.836653+0000","last_clean":"2024-09-10T16:06:27.836653+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:06:27.836653+0000","last_undegraded":"2024-09-10T16:06:27.836653+0000","last_fullsized":"2024-09-10T16:06:27.836653+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.429954+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:06:24.429954+0000","last_peered":"2024-09-10T16:06:24.429954+0000","last_clean":"2024-09-10T16:06:24.429954+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:06:24.429954+0000","last_undegraded":"2024-09-10T16:06:24.429954+0000","last_fullsized":"2024-09-10T16:06:24.429954+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T16:06:27.836167+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:06:27.836167+0000","last_peered":"2024-09-10T16:06:27.836167+0000","last_clean":"2024-09-10T16:06:27.836167+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:06:27.836167+0000","last_undegraded":"2024-09-10T16:06:27.836167+0000","last_fullsized":"2024-09-10T16:06:27.836167+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.428654+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:06:24.428654+0000","last_peered":"2024-09-10T16:06:24.428654+0000","last_clean":"2024-09-10T16:06:24.428654+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:06:24.428654+0000","last_undegraded":"2024-09-10T16:06:24.428654+0000","last_fullsized":"2024-09-10T16:06:24.428654+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":112,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-10T16:06:24.427527+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:06:24.427527+0000","last_peered":"2024-09-10T16:06:24.427527+0000","last_clean":"2024-09-10T16:06:24.427527+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:06:24.427527+0000","last_undegraded":"2024-09-10T16:06:24.427527+0000","last_fullsized":"2024-09-10T16:06:24.427527+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27208,"kb_used_data":180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344632,"statfs":{"total":96636764160,"available":96608903168,"internally_reserved":0,"allocated":184320,"data_stored":59677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607558,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":241664,"data_stored":588586,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607558,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":192512,"data_stored":64260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607558,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":64279,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27204,"kb_used_data":176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344636,"statfs":{"total":96636764160,"available":96608907264,"internally_reserved":0,"allocated":180224,"data_stored":59658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:06:30.659 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-10T16:06:30.660 DEBUG:tasks.mon_thrash:Yielding 2024-09-10T16:06:30.660 INFO:teuthology.run_tasks:Running task rados... 2024-09-10T16:06:30.671 INFO:tasks.rados:Beginning rados... 2024-09-10T16:06:30.671 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-10T16:06:30.671 INFO:tasks.rados:overrides is {'admin_socket': {'branch': 'wip-bharath7-testing-2024-09-10-1409'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath7-testing-2024-09-10-1409', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T16:06:30.671 INFO:tasks.rados:config is {'clients': ['client.0'], 'objects': 50, 'op_weights': {'copy_from': 50, 'delete': 50, 'read': 100, 'rollback': 50, 'snap_create': 50, 'snap_remove': 50, 'write': 100}, 'ops': 4000} 2024-09-10T16:06:30.671 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-10T16:06:30.705 INFO:tasks.rados:joining rados 2024-09-10T16:06:30.706 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-10T16:06:30.706 INFO:tasks.mon_thrash.mon_thrasher:seed: 1725984390, revive delay: 20.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-10T16:06:30.706 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:06:30.706 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:06:30.706 INFO:tasks.rados:clients are ['client.0'] 2024-09-10T16:06:30.706 INFO:tasks.rados:starting run 0 out of 1 2024-09-10T16:06:30.707 INFO:tasks.ceph.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-10T16:06:30.707 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create unique_pool_0 16 2024-09-10T16:06:30.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 10 monclient: get_monmap_and_config 2024-09-10T16:06:30.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 10 monclient: monmap: 2024-09-10T16:06:30.805 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:30.804983+0000 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:30.804983+0000 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:30.806 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 1 Processor -- start 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 1 -- start start 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.804+0000 7f2788a4f640 10 monclient: init 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 0x7f2784134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f278406cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27840a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27840a4ab0 0x7f27841383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f27840a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f278409ce50 con 0x7f27840a46a0 2024-09-10T16:06:30.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f278406db60 con 0x7f27840a4ab0 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f278406cf50 con 0x7f278406cb70 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2788a4f640 10 monclient(hunting): authenticate will time out at 769.292472s 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27840a4ab0 0x7f27841383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27840a4ab0 0x7f27841383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36258/0 (socket says 172.21.15.164:36258) 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 -- 172.21.15.164:0/997567575 learned_addr learned my addr 172.21.15.164:0/997567575 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): get_auth_request con 0x7f27840a4ab0 auth_method 0 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 1 --2- 172.21.15.164:0/997567575 >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 0x7f2784134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 10 monclient(hunting): get_auth_request con 0x7f278406cb70 auth_method 0 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient(hunting): handle_auth_done global_id 4344 payload 274 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 -- 172.21.15.164:0/997567575 >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 msgr2=0x7f2784134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 --2- 172.21.15.164:0/997567575 >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 0x7f2784134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 -- 172.21.15.164:0/997567575 >> v1:172.21.15.165:6789/0 conn(0x7f27840a46a0 legacy=0x7f2784137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f2782d9d640 1 -- 172.21.15.164:0/997567575 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f278409cb70 con 0x7f27840a4ab0 2024-09-10T16:06:30.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.805+0000 7f278259c640 1 --2- 172.21.15.164:0/997567575 >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 0x7f2784134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:30.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f5052ae2640 10 monclient: get_monmap_and_config 2024-09-10T16:06:30.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f5052ae2640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f5052ae2640 10 monclient: monmap: 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:30.809194+0000 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:30.809194+0000 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:30.813 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f2782d9d640 1 -- 172.21.15.164:0/997567575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a4ab0 msgr2=0x7f27841383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f2782d9d640 1 --2- 172.21.15.164:0/997567575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a4ab0 0x7f27841383b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f277c009e80 tx=0x7f277c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ecc6990c125b60 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 1 -- 172.21.15.164:0/997567575 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c036070 con 0x7f27840a4ab0 2024-09-10T16:06:30.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: dump: 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f2788a4f640 5 monclient: authenticate success, global_id 4344 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 1 -- 172.21.15.164:0/997567575 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f277c007800 con 0x7f27840a4ab0 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 1 -- 172.21.15.164:0/997567575 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277c007bd0 con 0x7f27840a4ab0 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f2788a4f640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f2788a4f640 4 set_mon_vals no callback set 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.808+0000 7f278159a640 10 monclient: dump: 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.816 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 Processor -- start 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 -- start start 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient: init 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 10 monclient: shutdown 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 -- 172.21.15.164:0/997567575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a4ab0 msgr2=0x7f27841383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 --2- 172.21.15.164:0/997567575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a4ab0 0x7f27841383b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f277c009e80 tx=0x7f277c02a5d0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 -- 172.21.15.164:0/997567575 shutdown_connections 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 --2- 172.21.15.164:0/997567575 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a4ab0 0x7f27841383b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 --2- 172.21.15.164:0/997567575 >> v2:172.21.15.165:3300/0 conn(0x7f278406cb70 0x7f2784134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 -- 172.21.15.164:0/997567575 >> 172.21.15.164:0/997567575 conn(0x7f27840aa900 msgr2=0x7f27840aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-10T16:06:30.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 0x7f504c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 -- 172.21.15.164:0/997567575 shutdown_connections 2024-09-10T16:06:30.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f504c06d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:30.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f2788a4f640 1 -- 172.21.15.164:0/997567575 wait complete. 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f504c06d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f504c0a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f504c09cb90 con 0x7f504c0a4c20 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f504c09ce70 con 0x7f504c06d160 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f504c09d150 con 0x7f504c06d540 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f5052ae2640 10 monclient(hunting): authenticate will time out at 769.296710s 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f505107f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f504c0a4c20 0x7f504c137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59416/0 (socket says 172.21.15.164:59416) 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f505107f640 1 -- 172.21.15.164:0/1061406538 learned_addr learned my addr 172.21.15.164:0/1061406538 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): get_auth_request con 0x7f504c06d540 auth_method 0 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.809+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f505087e640 1 --2- 172.21.15.164:0/1061406538 >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 0x7f504c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 10 monclient(hunting): handle_auth_done global_id 4346 payload 274 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 1 -- 172.21.15.164:0/1061406538 >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 msgr2=0x7f504c134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 1 --2- 172.21.15.164:0/1061406538 >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 0x7f504c134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 1 -- 172.21.15.164:0/1061406538 >> v1:172.21.15.164:6789/0 conn(0x7f504c0a4c20 legacy=0x7f504c137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 1 monclient: found mon.noname-b 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f504bfff640 1 -- 172.21.15.164:0/1061406538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504c135340 con 0x7f504c06d540 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f505087e640 10 monclient: get_auth_request con 0x7f504c06d160 auth_method 0 2024-09-10T16:06:30.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f505087e640 1 --2- 172.21.15.164:0/1061406538 >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 0x7f504c134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f2788a4f640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f2788a4f640 10 monclient: monmap: 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:30.827 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f2788a4f640 1 Processor -- start 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f2788a4f640 1 -- start start 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.810+0000 7f2788a4f640 10 monclient: init 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f278406cb70 0x7f278412e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-c con 0x7f278406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-a con 0x7f27840a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27840a4ab0 0x7f27841cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): picked mon.noname-b con 0x7f27840a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2784138a70 con 0x7f27840a46a0 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27841388f0 con 0x7f27840a4ab0 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2784140040 con 0x7f278406cb70 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2788a4f640 10 monclient(hunting): authenticate will time out at 769.298303s 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36262/0 (socket says 172.21.15.164:36262) 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 -- 172.21.15.164:0/779836691 learned_addr learned my addr 172.21.15.164:0/779836691 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): get_auth_request con 0x7f27840a46a0 auth_method 0 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2782d9d640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27840a4ab0 0x7f27841cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2782d9d640 10 monclient(hunting): get_auth_request con 0x7f27840a4ab0 auth_method 0 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2782d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2782d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2782d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 -- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f278406cb70 msgr2=0x7f278412e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f278406cb70 0x7f278412e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 -- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27840a4ab0 msgr2=0x7f27841cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27840a4ab0 0x7f27841cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.811+0000 7f2781d9b640 1 -- 172.21.15.164:0/779836691 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27840a34c0 con 0x7f27840a46a0 2024-09-10T16:06:30.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.812+0000 7f504bfff640 1 -- 172.21.15.164:0/1061406538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 msgr2=0x7f504c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.812+0000 7f504bfff640 1 --2- 172.21.15.164:0/1061406538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c0a46e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f5038029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87e3ba2c501aaba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 1 -- 172.21.15.164:0/1061406538 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038037070 con 0x7f504c06d540 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: dump: 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.833 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 1 -- 172.21.15.164:0/1061406538 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5038032640 con 0x7f504c06d540 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 5 monclient: authenticate success, global_id 4346 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 1 -- 172.21.15.164:0/1061406538 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5038032a10 con 0x7f504c06d540 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 4 set_mon_vals no callback set 2024-09-10T16:06:30.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f504b7fe640 10 monclient: dump: 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 10 monclient: shutdown 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:30.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 -- 172.21.15.164:0/1061406538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 msgr2=0x7f504c0a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 --2- 172.21.15.164:0/1061406538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c0a46e0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f50380097a0 tx=0x7f5038029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 -- 172.21.15.164:0/1061406538 shutdown_connections 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 --2- 172.21.15.164:0/1061406538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c0a46e0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 --2- 172.21.15.164:0/1061406538 >> v2:172.21.15.165:3300/0 conn(0x7f504c06d160 0x7f504c134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 -- 172.21.15.164:0/1061406538 >> 172.21.15.164:0/1061406538 conn(0x7f504c0aacb0 msgr2=0x7f504c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 -- 172.21.15.164:0/1061406538 shutdown_connections 2024-09-10T16:06:30.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f5052ae2640 1 -- 172.21.15.164:0/1061406538 wait complete. 2024-09-10T16:06:30.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f2781d9b640 1 -- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 msgr2=0x7f278412e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.813+0000 7f2781d9b640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c430 tx=0x7f276c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0119a0 con 0x7f27840a46a0 2024-09-10T16:06:30.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: dump: 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 5 monclient: authenticate success, global_id 4347 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 10 monclient: _renew_subs 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2784134390 con 0x7f27840a46a0 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 10 monclient: _renew_subs 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f5052ae2640 10 monclient: build_initial_monmap 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f5052ae2640 10 monclient: monmap: 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:30.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27840a2f00 con 0x7f27840a46a0 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276c017070 con 0x7f27840a46a0 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f2780d99640 4 set_mon_vals no callback set 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f5052ae2640 1 Processor -- start 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c011d80 con 0x7f27840a46a0 2024-09-10T16:06:30.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f277affd640 10 monclient: dump: 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f5052ae2640 1 -- start start 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.814+0000 7f5052ae2640 10 monclient: init 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f2788a4f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f2788a4f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f2788a4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2768005080 con 0x7f27840a46a0 2024-09-10T16:06:30.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:30.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:30.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-a con 0x7f504c06d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:30.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-c con 0x7f504c06d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504c0a4c20 0x7f504c13dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): picked mon.noname-b con 0x7f504c0a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f504c138870 con 0x7f504c06d160 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f504c1386f0 con 0x7f504c0a4c20 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f504c1389f0 con 0x7f504c06d540 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f5052ae2640 10 monclient(hunting): authenticate will time out at 769.302893s 2024-09-10T16:06:30.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36276/0 (socket says 172.21.15.164:36276) 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 1 -- 172.21.15.164:0/2893472592 learned_addr learned my addr 172.21.15.164:0/2893472592 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 10 monclient(hunting): get_auth_request con 0x7f504c06d160 auth_method 0 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f505087e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f276c01c460 con 0x7f27840a46a0 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f504bfff640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): get_auth_request con 0x7f504c06d540 auth_method 0 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.815+0000 7f277affd640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 0x7f2764040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f276c052a80 con 0x7f27840a46a0 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505107f640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504c0a4c20 0x7f504c13dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f504bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:30.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505107f640 10 monclient(hunting): get_auth_request con 0x7f504c0a4c20 auth_method 0 2024-09-10T16:06:30.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505107f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:30.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505107f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:30.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f278259c640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 0x7f2764040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505107f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 -- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 msgr2=0x7f504c131790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c131790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 -- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504c0a4c20 msgr2=0x7f504c13dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504c0a4c20 0x7f504c13dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient: _finish_hunting 0 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 monclient: found mon.noname-a 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f505087e640 1 -- 172.21.15.164:0/2893472592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504c0a4290 con 0x7f504c06d160 2024-09-10T16:06:30.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.816+0000 7f278259c640 10 monclient: get_auth_request con 0x7f276403e240 auth_method 0 2024-09-10T16:06:30.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.818+0000 7f505087e640 1 -- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 msgr2=0x7f504c131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:30.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.818+0000 7f505087e640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f504000c7f0 tx=0x7f504000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.818+0000 7f278259c640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 0x7f2764040700 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2774006e80 tx=0x7f2774008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5040015070 con 0x7f504c06d160 2024-09-10T16:06:30.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: dump: 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.854 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: _finish_auth 0 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: _check_auth_tickets 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5040007c40 con 0x7f504c06d160 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 5 monclient: authenticate success, global_id 4350 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 10 monclient: _renew_subs 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f504c09f330 con 0x7f504c06d160 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f504000ce40 con 0x7f504c06d160 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:30.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f50497fa640 10 monclient: dump: 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f504affd640 4 set_mon_vals no callback set 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 10 monclient: _renew_subs 2024-09-10T16:06:30.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f504c13f720 con 0x7f504c06d160 2024-09-10T16:06:30.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f276c012420 con 0x7f27840a46a0 2024-09-10T16:06:30.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f277affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.820+0000 7f277affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5040018070 con 0x7f504c06d160 2024-09-10T16:06:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f5052ae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f5052ae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f5052ae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:30.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f504c134e00 con 0x7f504c06d160 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f50497fa640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 0x7f5030040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f5040052cd0 con 0x7f504c06d160 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f504bfff640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 0x7f5030040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.821+0000 7f504bfff640 10 monclient: get_auth_request con 0x7f503003e240 auth_method 0 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.824+0000 7f504bfff640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 0x7f5030040700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f504c0773f0 tx=0x7f503802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.826+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5040051860 con 0x7f504c06d160 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.826+0000 7f50497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:30.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:30.826+0000 7f50497fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:31.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.035+0000 7f5052ae2640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-10T16:06:31.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.035+0000 7f5052ae2640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-10T16:06:31.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.035+0000 7f5052ae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.035+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16} v 0) -- 0x7f504c0aa1e0 con 0x7f504c06d160 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.103+0000 7f2788a4f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.103+0000 7f2788a4f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.103+0000 7f2788a4f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.103+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2768005370 con 0x7f27840a46a0 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.104+0000 7f277affd640 1 -- 172.21.15.164:0/779836691 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f276c01ac80 con 0x7f27840a46a0 2024-09-10T16:06:31.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.104+0000 7f277affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:31.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.104+0000 7f277affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:31.106 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 msgr2=0x7f2764040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 0x7f2764040700 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2774006e80 tx=0x7f2774008040 comp rx=0 tx=0).stop 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 10 monclient: shutdown 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 msgr2=0x7f278412e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f276c00c430 tx=0x7f276c00c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 shutdown_connections 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f276403e240 0x7f2764040700 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27840a4ab0 0x7f27841cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f27840a46a0 0x7f278412e8b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 --2- 172.21.15.164:0/779836691 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f278406cb70 0x7f278412e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 >> 172.21.15.164:0/779836691 conn(0x7f27840aa900 msgr2=0x7f27840a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 shutdown_connections 2024-09-10T16:06:31.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.106+0000 7f2788a4f640 1 -- 172.21.15.164:0/779836691 wait complete. 2024-09-10T16:06:31.119 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:06:31.119 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:06:31.119 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:06:31.119 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:06:31.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 10 monclient: get_monmap_and_config 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 10 monclient: monmap: 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:31.218098+0000 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:31.218098+0000 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:31.219 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:31.220 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:31.220 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:31.220 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 1 Processor -- start 2024-09-10T16:06:31.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 1 -- start start 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.217+0000 7fecf7cd4640 10 monclient: init 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fecf006d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7fecf006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fecf00a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf0135510 con 0x7fecf006d160 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecf009cb90 con 0x7fecf00a4c20 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fecf009cec0 con 0x7fecf006d540 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf7cd4640 10 monclient(hunting): authenticate will time out at 769.705528s 2024-09-10T16:06:31.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf5a70640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fecf006d160 0x7fecf0134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59420/0 (socket says 172.21.15.164:59420) 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf5a70640 1 -- 172.21.15.164:0/660496967 learned_addr learned my addr 172.21.15.164:0/660496967 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): get_auth_request con 0x7fecf006d540 auth_method 0 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 1 -- 172.21.15.164:0/660496967 >> v1:172.21.15.165:6789/0 conn(0x7fecf00a4c20 legacy=0x7fecf013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 1 -- 172.21.15.164:0/660496967 >> v1:172.21.15.164:6789/0 conn(0x7fecf006d160 legacy=0x7fecf0134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 1 monclient: found mon.noname-b 2024-09-10T16:06:31.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.218+0000 7fecf526f640 1 -- 172.21.15.164:0/660496967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf009d150 con 0x7fecf006d540 2024-09-10T16:06:31.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.221+0000 7fecf526f640 1 -- 172.21.15.164:0/660496967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 msgr2=0x7fecf00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.221+0000 7fecf526f640 1 --2- 172.21.15.164:0/660496967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf00a46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fece4009dc0 tx=0x7fece402a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fe2a73b7b1807209 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 1 -- 172.21.15.164:0/660496967 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4007a30 con 0x7fecf006d540 2024-09-10T16:06:31.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: dump: 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 1 -- 172.21.15.164:0/660496967 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4007bd0 con 0x7fecf006d540 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 5 monclient: authenticate success, global_id 4352 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 1 -- 172.21.15.164:0/660496967 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4031400 con 0x7fecf006d540 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 4 set_mon_vals no callback set 2024-09-10T16:06:31.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf4a6e640 10 monclient: dump: 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 10 monclient: shutdown 2024-09-10T16:06:31.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 -- 172.21.15.164:0/660496967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 msgr2=0x7fecf00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/660496967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf00a46e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fece4009dc0 tx=0x7fece402a480 comp rx=0 tx=0).stop 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 -- 172.21.15.164:0/660496967 shutdown_connections 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/660496967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf00a46e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 -- 172.21.15.164:0/660496967 >> 172.21.15.164:0/660496967 conn(0x7fecf00aacb0 msgr2=0x7fecf00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 -- 172.21.15.164:0/660496967 shutdown_connections 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.222+0000 7fecf7cd4640 1 -- 172.21.15.164:0/660496967 wait complete. 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 10 monclient: monmap: 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:31.234 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 1 Processor -- start 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 1 -- start start 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 10 monclient: init 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.223+0000 7fecf7cd4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:31.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf006d160 0x7fecf01cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-a con 0x7fecf006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf01ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-c con 0x7fecf006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf00a4c20 0x7fecf00a09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): picked mon.noname-b con 0x7fecf00a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fecf0140cd0 con 0x7fecf006d160 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fecf0140b50 con 0x7fecf00a4c20 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fecf0140e50 con 0x7fecf006d540 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf7cd4640 10 monclient(hunting): authenticate will time out at 769.711063s 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf006d160 0x7fecf01cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf01ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf01ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47478/0 (socket says 172.21.15.164:47478) 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf006d160 0x7fecf01cdb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36278/0 (socket says 172.21.15.164:36278) 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 -- 172.21.15.164:0/2726531091 learned_addr learned my addr 172.21.15.164:0/2726531091 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): get_auth_request con 0x7fecf006d540 auth_method 0 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): get_auth_request con 0x7fecf006d160 auth_method 0 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf00a4c20 0x7fecf00a09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 10 monclient(hunting): get_auth_request con 0x7fecf00a4c20 auth_method 0 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient(hunting): handle_auth_done global_id 4355 payload 824 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf00a4c20 msgr2=0x7fecf00a09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf00a4c20 0x7fecf00a09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf006d160 msgr2=0x7fecf01cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf006d160 0x7fecf01cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 monclient: found mon.noname-c 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf526f640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecf00a4320 con 0x7fecf006d540 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.224+0000 7fecf6271640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fecf00a4c20 0x7fecf00a09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf526f640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 msgr2=0x7fecf01ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf526f640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf01ce0b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fece4000f80 tx=0x7fece4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece4040070 con 0x7fecf006d540 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: dump: 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.240 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 5 monclient: authenticate success, global_id 4355 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece4045660 con 0x7fecf006d540 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 10 monclient: _renew_subs 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecf0133670 con 0x7fecf006d540 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece40314c0 con 0x7fecf006d540 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecee7fc640 10 monclient: dump: 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7feceffff640 4 set_mon_vals no callback set 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 10 monclient: _renew_subs 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.226+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fecf0133b30 con 0x7fecf006d540 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fece4039040 con 0x7fecf006d540 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf7cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 0x7fecf002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fecf00a2300 con 0x7fecf00a16d0 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 0x7fecf002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient: get_auth_request con 0x7fecf00a16d0 auth_method 0 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecee7fc640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecdc03e210 0x7fecdc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.227+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fece4074630 con 0x7fecf006d540 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecf5a70640 10 monclient(hunting): handle_auth_done global_id 4356 payload 824 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecf5a70640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fecf00a46e0 con 0x7fecf00a16d0 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecf6271640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecdc03e210 0x7fecdc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.228+0000 7fecf6271640 10 monclient: get_auth_request con 0x7fecdc03e210 auth_method 0 2024-09-10T16:06:31.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecf5a70640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 msgr2=0x7fecf002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecf5a70640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 0x7fecf002d440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fece000ea70 tx=0x7fece000ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece001a070 con 0x7fecf00a16d0 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 10 monclient: dump: 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.247 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecf6271640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecdc03e210 0x7fecdc0406d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecf006d9a0 tx=0x7fece8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fece000cc50 con 0x7fecf00a16d0 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 msgr2=0x7fecf002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 0x7fecf002d440 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fece000ea70 tx=0x7fece000ef30 comp rx=0 tx=0).stop 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 msgr2=0x7fecf002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.230+0000 7fecee7fc640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a16d0 0x7fecf002d440 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fece000ea70 tx=0x7fece000ef30 comp rx=0 tx=0).stop 2024-09-10T16:06:31.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.236+0000 7fecf7cd4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.236+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 0x7fecf002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.236+0000 7fecf7cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.236+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fecf00aa1e0 con 0x7fecf00a1ab0 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.236+0000 7fecf5a70640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 0x7fecf002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient: get_auth_request con 0x7fecf00a1ab0 auth_method 0 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2024-09-10T16:06:31.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.237+0000 7fecf5a70640 1 -- 172.21.15.164:0/2726531091 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fecf00a8130 con 0x7fecf00a1ab0 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecf5a70640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 msgr2=0x7fecf002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecf5a70640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 0x7fecf002d980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece0017000 tx=0x7fece0004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece001a070 con 0x7fecf00a1ab0 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 10 monclient: dump: 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.250 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fece001b040 con 0x7fecf00a1ab0 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 msgr2=0x7fecf002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 0x7fecf002d980 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece0017000 tx=0x7fece0004940 comp rx=0 tx=0).stop 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 msgr2=0x7fecf002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.239+0000 7fecee7fc640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fecf00a1ab0 0x7fecf002d980 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fece0017000 tx=0x7fece0004940 comp rx=0 tx=0).stop 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 8, 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 33414, 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.252 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 28, 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.253 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.254 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:06:31.255 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.256 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:31.257 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.259 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.262 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.264 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:06:31.265 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.265 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.265 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:06:31.265 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:06:31.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecdc03e210 msgr2=0x7fecdc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecdc03e210 0x7fecdc0406d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fecf006d9a0 tx=0x7fece8009040 comp rx=0 tx=0).stop 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 10 monclient: shutdown 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 msgr2=0x7fecf01ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 --2- 172.21.15.164:0/2726531091 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fecf006d540 0x7fecf01ce0b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fece4000f80 tx=0x7fece4004a40 comp rx=0 tx=0).stop 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf6271640 1 -- 172.21.15.164:0/2726531091 reap_dead start 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf6271640 1 -- 172.21.15.164:0/2726531091 reap_dead start 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 shutdown_connections 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 >> 172.21.15.164:0/2726531091 conn(0x7fecf00aacb0 msgr2=0x7fecf00a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 shutdown_connections 2024-09-10T16:06:31.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.240+0000 7fecf7cd4640 1 -- 172.21.15.164:0/2726531091 wait complete. 2024-09-10T16:06:31.268 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:06:31.268 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:06:31.268 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.365+0000 7f20c9b7d640 10 monclient: get_monmap_and_config 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.365+0000 7f20c9b7d640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.365+0000 7f20c9b7d640 10 monclient: monmap: 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:31.366766+0000 2024-09-10T16:06:31.367 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:31.366766+0000 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:31.368 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.366+0000 7f20c9b7d640 1 Processor -- start 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.366+0000 7f20c9b7d640 1 -- start start 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.366+0000 7f20c9b7d640 10 monclient: init 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20c4074af0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-b con 0x7f20c40a1090 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f20c413d190 0x7f20c413f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-c con 0x7f20c413d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20c409ebc0 con 0x7f20c4074af0 2024-09-10T16:06:31.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f20c40698a0 con 0x7f20c413d190 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20c406bfb0 con 0x7f20c40a1090 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c9b7d640 10 monclient(hunting): authenticate will time out at 769.854162s 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47488/0 (socket says 172.21.15.164:47488) 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 -- 172.21.15.164:0/1499139887 learned_addr learned my addr 172.21.15.164:0/1499139887 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request con 0x7f20c40a1090 auth_method 0 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 1 --2- 172.21.15.164:0/1499139887 >> v2:172.21.15.165:3300/0 conn(0x7f20c413d190 0x7f20c413f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): get_auth_request con 0x7f20c413d190 auth_method 0 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 -- 172.21.15.164:0/1499139887 >> v2:172.21.15.165:3300/0 conn(0x7f20c413d190 msgr2=0x7f20c413f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/1499139887 >> v2:172.21.15.165:3300/0 conn(0x7f20c413d190 0x7f20c413f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 -- 172.21.15.164:0/1499139887 >> v1:172.21.15.165:6789/0 conn(0x7f20c4074af0 legacy=0x7f20c4137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 monclient: found mon.noname-b 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.367+0000 7f20c2ffd640 1 -- 172.21.15.164:0/1499139887 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c4138580 con 0x7f20c40a1090 2024-09-10T16:06:31.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c2ffd640 1 -- 172.21.15.164:0/1499139887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 msgr2=0x7f20c40a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/1499139887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009870 tx=0x7f20b802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2d94e1e45cda97f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 1 -- 172.21.15.164:0/1499139887 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8037070 con 0x7f20c40a1090 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: dump: 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.382 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c9b7d640 5 monclient: authenticate success, global_id 4358 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 1 -- 172.21.15.164:0/1499139887 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b8032440 con 0x7f20c40a1090 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c9b7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c9b7d640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c9b7d640 4 set_mon_vals no callback set 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 1 -- 172.21.15.164:0/1499139887 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8032810 con 0x7f20c40a1090 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.370+0000 7f20c27fc640 10 monclient: dump: 2024-09-10T16:06:31.383 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 10 monclient: shutdown 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 -- 172.21.15.164:0/1499139887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 msgr2=0x7f20c40a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/1499139887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f20b8009870 tx=0x7f20b802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 -- 172.21.15.164:0/1499139887 shutdown_connections 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/1499139887 >> v2:172.21.15.165:3300/0 conn(0x7f20c413d190 0x7f20c413f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/1499139887 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c40a14d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 -- 172.21.15.164:0/1499139887 >> 172.21.15.164:0/1499139887 conn(0x7f20c40a3fb0 msgr2=0x7f20c40a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 -- 172.21.15.164:0/1499139887 shutdown_connections 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 1 -- 172.21.15.164:0/1499139887 wait complete. 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.371+0000 7f20c9b7d640 10 monclient: monmap: 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 Processor -- start 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 -- start start 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient: init 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-a con 0x7f20c4074af0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c413c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-c con 0x7f20c40a1090 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20c413d190 0x7f20c409b840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): picked mon.noname-b con 0x7f20c413d190 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20c4074ed0 con 0x7f20c4074af0 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f20c4069370 con 0x7f20c413d190 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20c413fcd0 con 0x7f20c40a1090 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c9b7d640 10 monclient(hunting): authenticate will time out at 769.859899s 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c413c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.372+0000 7f20c37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36318/0 (socket says 172.21.15.164:36318) 2024-09-10T16:06:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c413c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47494/0 (socket says 172.21.15.164:47494) 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 -- 172.21.15.164:0/91962669 learned_addr learned my addr 172.21.15.164:0/91962669 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): get_auth_request con 0x7f20c4074af0 auth_method 0 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request con 0x7f20c40a1090 auth_method 0 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20c413d190 0x7f20c409b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 10 monclient(hunting): get_auth_request con 0x7f20c413d190 auth_method 0 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 msgr2=0x7f20c413c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c413c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20c413d190 msgr2=0x7f20c409b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20c413d190 0x7f20c409b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 monclient: found mon.noname-a 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c37fe640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20c41398c0 con 0x7f20c4074af0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c3fff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20c413d190 0x7f20c409b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.373+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c40a1090 0x7f20c413c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c37fe640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 msgr2=0x7f20c413c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c37fe640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20b400c430 tx=0x7f20b400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b40119a0 con 0x7f20c4074af0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: dump: 2024-09-10T16:06:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 5 monclient: authenticate success, global_id 4362 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20b4017070 con 0x7f20c4074af0 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 10 monclient: _renew_subs 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20c409dc60 con 0x7f20c4074af0 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b4011d80 con 0x7f20c4074af0 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20abfff640 10 monclient: dump: 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 10 monclient: _renew_subs 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20c409e6a0 con 0x7f20c4074af0 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.375+0000 7f20c1ffb640 4 set_mon_vals no callback set 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c9b7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 0x7f20c402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20c40a14d0 con 0x7f20c41cd320 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 0x7f20c402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient: get_auth_request con 0x7f20c41cd320 auth_method 0 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c37fe640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20b400c430 tx=0x7f20b400c8f0 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f20b401c460 mgrmap(e 5) 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f20b401c460 con 0x7f20c4074af0 2024-09-10T16:06:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_done global_id 4364 payload 824 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.376+0000 7f20c2ffd640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20c41395e0 con 0x7f20c41cd320 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.377+0000 7f20abfff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f20b003e210 0x7f20b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.377+0000 7f20c3fff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f20b003e210 0x7f20b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.377+0000 7f20c3fff640 10 monclient: get_auth_request con 0x7f20b003e210 auth_method 0 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20c2ffd640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 msgr2=0x7f20c402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 0x7f20c402d440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20c40a3ed0 tx=0x7f20b80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8043070 con 0x7f20c41cd320 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 10 monclient: dump: 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20b803c860 con 0x7f20c41cd320 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 msgr2=0x7f20c402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 0x7f20c402d440 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20c40a3ed0 tx=0x7f20b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 msgr2=0x7f20c402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20abfff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c41cd320 0x7f20c402d440 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f20c40a3ed0 tx=0x7f20b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.379+0000 7f20c3fff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f20b003e210 0x7f20b00406d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20c409dad0 tx=0x7f20bc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c9b7d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:06:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 0x7f20c402dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c9b7d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20c4137170 con 0x7f20c402d980 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 0x7f20c402dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient: get_auth_request con 0x7f20c402d980 auth_method 0 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.385+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.386+0000 7f20c2ffd640 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.386+0000 7f20c2ffd640 1 -- 172.21.15.164:0/91962669 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20c40772c0 con 0x7f20c402d980 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c2ffd640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 msgr2=0x7f20c402dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c2ffd640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 0x7f20c402dd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20b8000c00 tx=0x7f20b8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b8037040 con 0x7f20c402d980 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 10 monclient: dump: 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f20b8004480 con 0x7f20c402d980 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 msgr2=0x7f20c402dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 0x7f20c402dd60 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20b8000c00 tx=0x7f20b8009240 comp rx=0 tx=0).stop 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 msgr2=0x7f20c402dd60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f20c402d980 0x7f20c402dd60 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f20b8000c00 tx=0x7f20b8009240 comp rx=0 tx=0).stop 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 8, 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 33560, 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:06:31.402 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 28, 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.403 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.404 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:06:31.405 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.406 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.407 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.409 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:31.411 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.413 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.414 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:06:31.415 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f20b003e210 msgr2=0x7f20b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f20b003e210 0x7f20b00406d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20c409dad0 tx=0x7f20bc00b040 comp rx=0 tx=0).stop 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 10 monclient: shutdown 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 msgr2=0x7f20c413c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 --2- 172.21.15.164:0/91962669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20c4074af0 0x7f20c413c3c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20b400c430 tx=0x7f20b400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c3fff640 1 -- 172.21.15.164:0/91962669 reap_dead start 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20abfff640 1 -- 172.21.15.164:0/91962669 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (unknown 0 0 0) 0x7f20b4052a10 con 0x7f20c4074af0 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c3fff640 1 -- 172.21.15.164:0/91962669 reap_dead start 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 shutdown_connections 2024-09-10T16:06:31.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.388+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 >> 172.21.15.164:0/91962669 conn(0x7f20c40a3fb0 msgr2=0x7f20c40a54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.389+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 shutdown_connections 2024-09-10T16:06:31.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.389+0000 7f20c9b7d640 1 -- 172.21.15.164:0/91962669 wait complete. 2024-09-10T16:06:31.418 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:06:31.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:06:31.418 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.516+0000 7fb8a5f94640 10 monclient: get_monmap_and_config 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.516+0000 7fb8a5f94640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.516+0000 7fb8a5f94640 10 monclient: monmap: 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:31.517476+0000 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:31.517476+0000 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:31.518 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:31.519 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:31.519 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:31.519 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:31.519 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 Processor -- start 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 -- start start 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient: init 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb8a0137bb0 0x7fb8a00a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a0137bb0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:31.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 0x7fb8a01383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a0137f90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb8a01346c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8a0073820 con 0x7fb8a01346c0 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb8a00a1c90 con 0x7fb8a0137bb0 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a00a1f70 con 0x7fb8a0137f90 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.517+0000 7fb8a5f94640 10 monclient(hunting): authenticate will time out at 770.004944s 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb8a01346c0 0x7fb8a0134b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59428/0 (socket says 172.21.15.164:59428) 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89ffff640 1 -- 172.21.15.164:0/2760468795 learned_addr learned my addr 172.21.15.164:0/2760468795 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:31.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 --2- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 0x7fb8a01383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): get_auth_request con 0x7fb8a0137f90 auth_method 0 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89f7fe640 1 --2- 172.21.15.164:0/2760468795 >> v2:172.21.15.165:3300/0 conn(0x7fb8a0137bb0 0x7fb8a00a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89f7fe640 10 monclient(hunting): get_auth_request con 0x7fb8a0137bb0 auth_method 0 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient(hunting): handle_auth_done global_id 4370 payload 274 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 -- 172.21.15.164:0/2760468795 >> v2:172.21.15.165:3300/0 conn(0x7fb8a0137bb0 msgr2=0x7fb8a00a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 --2- 172.21.15.164:0/2760468795 >> v2:172.21.15.165:3300/0 conn(0x7fb8a0137bb0 0x7fb8a00a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 -- 172.21.15.164:0/2760468795 >> v1:172.21.15.164:6789/0 conn(0x7fb8a01346c0 legacy=0x7fb8a0134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 monclient: found mon.noname-b 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.518+0000 7fb89effd640 1 -- 172.21.15.164:0/2760468795 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a00a2250 con 0x7fb8a0137f90 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89effd640 1 -- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 msgr2=0x7fb8a01383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89effd640 1 --2- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 0x7fb8a01383a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb8940097a0 tx=0x7fb89402a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bb984a4d2832190 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 1 -- 172.21.15.164:0/2760468795 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894037070 con 0x7fb8a0137f90 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: dump: 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.528 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 1 -- 172.21.15.164:0/2760468795 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb894032440 con 0x7fb8a0137f90 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 5 monclient: authenticate success, global_id 4370 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 1 -- 172.21.15.164:0/2760468795 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894032810 con 0x7fb8a0137f90 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 4 set_mon_vals no callback set 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb89e7fc640 10 monclient: dump: 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.530 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 10 monclient: shutdown 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 -- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 msgr2=0x7fb8a01383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 0x7fb8a01383a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb8940097a0 tx=0x7fb89402a050 comp rx=0 tx=0).stop 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 -- 172.21.15.164:0/2760468795 shutdown_connections 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/2760468795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a0137f90 0x7fb8a01383a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/2760468795 >> v2:172.21.15.165:3300/0 conn(0x7fb8a0137bb0 0x7fb8a00a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 -- 172.21.15.164:0/2760468795 >> 172.21.15.164:0/2760468795 conn(0x7fb8a00a7320 msgr2=0x7fb8a00a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 -- 172.21.15.164:0/2760468795 shutdown_connections 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.521+0000 7fb8a5f94640 1 -- 172.21.15.164:0/2760468795 wait complete. 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.522+0000 7fb8a5f94640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.522+0000 7fb8a5f94640 10 monclient: monmap: 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.522+0000 7fb8a5f94640 1 Processor -- start 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 -- start start 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient: init 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-c con 0x7fb8a01346c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a0137bb0 0x7fb8a0121110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-b con 0x7fb8a0137bb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8a0137f90 0x7fb8a0121650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): picked mon.noname-a con 0x7fb8a0137f90 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0077060 con 0x7fb8a0137f90 2024-09-10T16:06:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0076ee0 con 0x7fb8a0137bb0 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb8a0135260 con 0x7fb8a01346c0 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb8a5f94640 10 monclient(hunting): authenticate will time out at 770.010470s 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8a0137f90 0x7fb8a0121650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47526/0 (socket says 172.21.15.164:47526) 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 -- 172.21.15.164:0/1174624849 learned_addr learned my addr 172.21.15.164:0/1174624849 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): get_auth_request con 0x7fb8a01346c0 auth_method 0 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): get_auth_request con 0x7fb8a0137f90 auth_method 0 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient(hunting): handle_auth_done global_id 4373 payload 824 2024-09-10T16:06:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a0137bb0 msgr2=0x7fb8a0121110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a0137bb0 0x7fb8a0121110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8a0137f90 msgr2=0x7fb8a0121650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb8a0137f90 0x7fb8a0121650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.523+0000 7fb89f7fe640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a00a5770 con 0x7fb8a01346c0 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.526+0000 7fb89f7fe640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 msgr2=0x7fb8a01cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.526+0000 7fb89f7fe640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb89000d800 tx=0x7fb89000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000b8d0 con 0x7fb8a01346c0 2024-09-10T16:06:31.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:31.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: dump: 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb890019070 con 0x7fb8a01346c0 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 5 monclient: authenticate success, global_id 4373 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 10 monclient: _renew_subs 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a0122bd0 con 0x7fb8a01346c0 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000bc90 con 0x7fb8a01346c0 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb887fff640 10 monclient: dump: 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 10 monclient: _renew_subs 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb89dffb640 4 set_mon_vals no callback set 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.528+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a0124f30 con 0x7fb8a01346c0 2024-09-10T16:06:31.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb8a5f94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 0x7fb8a0121b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a009e0f0 con 0x7fb8a01cda90 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb89001e460 con 0x7fb8a01346c0 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb887fff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb88c03e260 0x7fb88c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fb890052ff0 con 0x7fb8a01346c0 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 0x7fb8a0121b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient: get_auth_request con 0x7fb8a01cda90 auth_method 0 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89ffff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb88c03e260 0x7fb88c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89ffff640 10 monclient: get_auth_request con 0x7fb88c03e260 auth_method 0 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 10 monclient(hunting): handle_auth_done global_id 4360 payload 824 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.529+0000 7fb89effd640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb8a0136fe0 con 0x7fb8a01cda90 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb89effd640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 msgr2=0x7fb8a0121b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb89effd640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 0x7fb8a0121b90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb894034000 tx=0x7fb894032cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb89ffff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb88c03e260 0x7fb88c040720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8980055a0 tx=0x7fb898005530 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894040020 con 0x7fb8a01cda90 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:31.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 10 monclient: dump: 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb894038040 con 0x7fb8a01cda90 2024-09-10T16:06:31.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 msgr2=0x7fb8a0121b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 0x7fb8a0121b90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb894034000 tx=0x7fb894032cd0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 msgr2=0x7fb8a0121b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.532+0000 7fb887fff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01cda90 0x7fb8a0121b90 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb894034000 tx=0x7fb894032cd0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb8a5f94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 0x7fb8a011d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb8a5f94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a0124800 con 0x7fb8a01220d0 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb89effd640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 0x7fb8a011d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb89effd640 10 monclient: get_auth_request con 0x7fb8a01220d0 auth_method 0 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb89effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb89effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.538+0000 7fb89effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.539+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.539+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.539+0000 7fb89effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.539+0000 7fb89effd640 10 monclient(hunting): handle_auth_done global_id 4363 payload 824 2024-09-10T16:06:31.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.539+0000 7fb89effd640 1 -- 172.21.15.164:0/1174624849 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb8a00a2e60 con 0x7fb8a01220d0 2024-09-10T16:06:31.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb89effd640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 msgr2=0x7fb8a011d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb89effd640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 0x7fb8a011d550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8940097a0 tx=0x7fb894008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb894040030 con 0x7fb8a01220d0 2024-09-10T16:06:31.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 10 monclient: dump: 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7fb894045420 con 0x7fb8a01220d0 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 msgr2=0x7fb8a011d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 0x7fb8a011d550 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8940097a0 tx=0x7fb894008030 comp rx=0 tx=0).stop 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 msgr2=0x7fb8a011d550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.541+0000 7fb887fff640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb8a01220d0 0x7fb8a011d550 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8940097a0 tx=0x7fb894008030 comp rx=0 tx=0).stop 2024-09-10T16:06:31.553 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 8, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 33675, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 28, 2024-09-10T16:06:31.554 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.555 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.556 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:06:31.557 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:31.558 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:31.559 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.560 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.561 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:06:31.563 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:31.564 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:06:31.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:06:31.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:06:31.567 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:06:31.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb88c03e260 msgr2=0x7fb88c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb88c03e260 0x7fb88c040720 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb8980055a0 tx=0x7fb898005530 comp rx=0 tx=0).stop 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 10 monclient: shutdown 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 msgr2=0x7fb8a01cd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb89000d800 tx=0x7fb89000dcc0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb89ffff640 1 -- 172.21.15.164:0/1174624849 reap_dead start 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 shutdown_connections 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 --2- 172.21.15.164:0/1174624849 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb8a01346c0 0x7fb8a01cd550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 >> 172.21.15.164:0/1174624849 conn(0x7fb8a00a7320 msgr2=0x7fb8a0137330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 shutdown_connections 2024-09-10T16:06:31.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.542+0000 7fb8a5f94640 1 -- 172.21.15.164:0/1174624849 wait complete. 2024-09-10T16:06:31.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:06:31.570 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:06:31.570 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:06:31.570 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:06:31.570 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:06:31.571 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:06:31.571 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:06:31.585 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:06:31.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:06:31.585 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.682+0000 7f02fc7d3640 10 monclient: get_monmap_and_config 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.682+0000 7f02fc7d3640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.682+0000 7f02fc7d3640 10 monclient: monmap: 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:31.683414+0000 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:31.683414+0000 2024-09-10T16:06:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:31.686 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 Processor -- start 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 -- start start 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient: init 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f02f406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 0x7f02f406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-b con 0x7f02f409bcc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02f40a40d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:31.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f406e900 con 0x7f02f406cb70 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02f406e340 con 0x7f02f40a40d0 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f02f40a4840 con 0x7f02f409bcc0 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fc7d3640 10 monclient(hunting): authenticate will time out at 770.170787s 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02f9d6e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 0x7f02f406d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fa56f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f02f406cb70 0x7f02f406cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59442/0 (socket says 172.21.15.164:59442) 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fa56f640 1 -- 172.21.15.164:0/2893374956 learned_addr learned my addr 172.21.15.164:0/2893374956 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02f9d6e640 10 monclient(hunting): get_auth_request con 0x7f02f409bcc0 auth_method 0 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02f9d6e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02f9d6e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02f9d6e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.683+0000 7f02fad70640 1 -- 172.21.15.164:0/2893374956 >> v1:172.21.15.165:6789/0 conn(0x7f02f40a40d0 legacy=0x7f02f413f600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient(hunting): handle_auth_done global_id 4376 payload 274 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 1 -- 172.21.15.164:0/2893374956 >> v1:172.21.15.165:6789/0 conn(0x7f02f40a40d0 legacy=0x7f02f413f600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 1 -- 172.21.15.164:0/2893374956 >> v1:172.21.15.164:6789/0 conn(0x7f02f406cb70 legacy=0x7f02f406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 1 monclient: found mon.noname-b 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.684+0000 7f02f9d6e640 1 -- 172.21.15.164:0/2893374956 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f40a01e0 con 0x7f02f409bcc0 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.686+0000 7f02f9d6e640 1 -- 172.21.15.164:0/2893374956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 msgr2=0x7f02f406d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.686+0000 7f02f9d6e640 1 --2- 172.21.15.164:0/2893374956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 0x7f02f406d660 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02e40097d0 tx=0x7f02e4029fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8892ef60aaf9eb07 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 1 -- 172.21.15.164:0/2893374956 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e4037070 con 0x7f02f409bcc0 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: dump: 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.695 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 1 -- 172.21.15.164:0/2893374956 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02e4032440 con 0x7f02f409bcc0 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 1 -- 172.21.15.164:0/2893374956 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02e40327f0 con 0x7f02f409bcc0 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 5 monclient: authenticate success, global_id 4376 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 4 set_mon_vals no callback set 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02f956d640 10 monclient: dump: 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 10 monclient: shutdown 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 -- 172.21.15.164:0/2893374956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 msgr2=0x7f02f406d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/2893374956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 0x7f02f406d660 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02e40097d0 tx=0x7f02e4029fa0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 -- 172.21.15.164:0/2893374956 shutdown_connections 2024-09-10T16:06:31.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/2893374956 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f409bcc0 0x7f02f406d660 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 -- 172.21.15.164:0/2893374956 >> 172.21.15.164:0/2893374956 conn(0x7f02f40a4f50 msgr2=0x7f02f40a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 -- 172.21.15.164:0/2893374956 shutdown_connections 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.687+0000 7f02fc7d3640 1 -- 172.21.15.164:0/2893374956 wait complete. 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.688+0000 7f02fc7d3640 10 monclient: build_initial_monmap 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.688+0000 7f02fc7d3640 10 monclient: monmap: 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:31.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.688+0000 7f02fc7d3640 1 Processor -- start 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.688+0000 7f02fc7d3640 1 -- start start 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.688+0000 7f02fc7d3640 10 monclient: init 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-c con 0x7f02f406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 0x7f02f413d090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-b con 0x7f02f40a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 0x7f02f41cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): picked mon.noname-a con 0x7f02f413d5d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f02f4140f30 con 0x7f02f413d5d0 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f02f41410b0 con 0x7f02f40a40d0 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f02f4141230 con 0x7f02f406cb70 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fc7d3640 10 monclient(hunting): authenticate will time out at 770.176175s 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47542/0 (socket says 172.21.15.164:47542) 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 0x7f02f41cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 1 -- 172.21.15.164:0/1224458309 learned_addr learned my addr 172.21.15.164:0/1224458309 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:31.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): get_auth_request con 0x7f02f406cb70 auth_method 0 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02f9d6e640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 msgr2=0x7f02f413d090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02f9d6e640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 0x7f02f413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): get_auth_request con 0x7f02f413d5d0 auth_method 0 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient(hunting): handle_auth_done global_id 4368 payload 824 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 msgr2=0x7f02f413cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 msgr2=0x7f02f413d090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 0x7f02f413d090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient: _finish_hunting 0 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 monclient: found mon.noname-a 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fad70640 1 -- 172.21.15.164:0/1224458309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02f409c140 con 0x7f02f413d5d0 2024-09-10T16:06:31.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.689+0000 7f02fa56f640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fad70640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 msgr2=0x7f02f41cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fad70640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 0x7f02f41cd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00bd80 tx=0x7f02ec009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec015670 con 0x7f02f413d5d0 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:31.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: dump: 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:31.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ec010070 con 0x7f02f413d5d0 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 5 monclient: authenticate success, global_id 4368 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec015a90 con 0x7f02f413d5d0 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 10 monclient: _renew_subs 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02f4140320 con 0x7f02f413d5d0 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02eaffd640 10 monclient: dump: 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:31.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02f8d6c640 4 set_mon_vals no callback set 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 10 monclient: _renew_subs 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.691+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02f40a6710 con 0x7f02f413d5d0 2024-09-10T16:06:31.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.692+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f02ec015cf0 con 0x7f02f413d5d0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.692+0000 7f02fc7d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.692+0000 7f02fc7d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.692+0000 7f02fc7d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.692+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02f406cf50 con 0x7f02f413d5d0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.693+0000 7f02eaffd640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 0x7f02d8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.693+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f02ec052e90 con 0x7f02f413d5d0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.693+0000 7f02fa56f640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 0x7f02d8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.693+0000 7f02fa56f640 10 monclient: get_auth_request con 0x7f02d803e1c0 auth_method 0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.695+0000 7f02fa56f640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 0x7f02d8040680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f02f0007d50 tx=0x7f02f0007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.697+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f02f409c140 con 0x7f02f413d5d0 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.697+0000 7f02eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:31.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.697+0000 7f02eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:31.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02fc7d3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:06:31.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02fc7d3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:31.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02fc7d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:31.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f02f40a6350 con 0x7f02f413d5d0 2024-09-10T16:06:31.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02eaffd640 1 -- 172.21.15.164:0/1224458309 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f02ec051a20 con 0x7f02f413d5d0 2024-09-10T16:06:31.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:31.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.940+0000 7f02eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:31.943 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":28,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 msgr2=0x7f02d8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 0x7f02d8040680 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f02f0007d50 tx=0x7f02f0007ca0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 10 monclient: shutdown 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 msgr2=0x7f02f41cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:31.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 0x7f02f41cd510 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00bd80 tx=0x7f02ec009ef0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 shutdown_connections 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f02d803e1c0 0x7f02d8040680 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f02f413d5d0 0x7f02f41cd510 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02f40a40d0 0x7f02f413d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 --2- 172.21.15.164:0/1224458309 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02f406cb70 0x7f02f413cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 >> 172.21.15.164:0/1224458309 conn(0x7f02f40a4f50 msgr2=0x7f02f409e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 shutdown_connections 2024-09-10T16:06:31.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:31.942+0000 7f02fc7d3640 1 -- 172.21.15.164:0/1224458309 wait complete. 2024-09-10T16:06:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:33.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:33.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:33.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:33.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:33.816+0000 7f5037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:34.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:34.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:34.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:34.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:34.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:34.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:34.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:06:35.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 10 monclient: get_monmap_and_config 2024-09-10T16:06:35.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 10 monclient: build_initial_monmap 2024-09-10T16:06:35.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 10 monclient: monmap: 2024-09-10T16:06:35.065 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:35.064963+0000 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:35.064963+0000 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:35.066 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 1 Processor -- start 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 1 -- start start 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.064+0000 7f3b9bff3640 10 monclient: init 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b9406d160 0x7f3b94134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-a con 0x7f3b9406d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:35.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 0x7f3b940a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9406d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b940a4c20 0x7f3b94137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-b con 0x7f3b940a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3b9409cb90 con 0x7f3b9406d160 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3b9409ce70 con 0x7f3b9406d540 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3b9409d150 con 0x7f3b940a4c20 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9bff3640 10 monclient(hunting): authenticate will time out at 773.552482s 2024-09-10T16:06:35.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b9406d160 0x7f3b94134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b9406d160 0x7f3b94134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49856/0 (socket says 172.21.15.164:49856) 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 -- 172.21.15.164:0/1497785220 learned_addr learned my addr 172.21.15.164:0/1497785220 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): get_auth_request con 0x7f3b9406d160 auth_method 0 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9958e640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 msgr2=0x7f3b940a46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b940a4c20 0x7f3b94137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9958e640 1 --2- 172.21.15.164:0/1497785220 >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 0x7f3b940a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): get_auth_request con 0x7f3b940a4c20 auth_method 0 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:35.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 -- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b940a4c20 msgr2=0x7f3b94137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b940a4c20 0x7f3b94137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 -- 172.21.15.164:0/1497785220 >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 msgr2=0x7f3b940a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/1497785220 >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 0x7f3b940a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient: _finish_hunting 0 2024-09-10T16:06:35.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 monclient: found mon.noname-a 2024-09-10T16:06:35.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:35.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.065+0000 7f3b99d8f640 1 -- 172.21.15.164:0/1497785220 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b94135340 con 0x7f3b9406d160 2024-09-10T16:06:35.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b99d8f640 1 -- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b9406d160 msgr2=0x7f3b94134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:35.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b9406d160 0x7f3b94134e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b940a9d40 tx=0x7f3b9002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ba79de92443348d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 1 -- 172.21.15.164:0/1497785220 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90037070 con 0x7f3b9406d160 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: dump: 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:35.079 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: _finish_auth 0 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: _check_auth_tickets 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b9bff3640 5 monclient: authenticate success, global_id 4371 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b9bff3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 1 -- 172.21.15.164:0/1497785220 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b90032640 con 0x7f3b9406d160 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b9bff3640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b9bff3640 4 set_mon_vals no callback set 2024-09-10T16:06:35.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 1 -- 172.21.15.164:0/1497785220 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90032a10 con 0x7f3b9406d160 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.068+0000 7f3b98d8d640 10 monclient: dump: 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:35.081 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 10 monclient: shutdown 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 -- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b9406d160 msgr2=0x7f3b94134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b9406d160 0x7f3b94134e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3b940a9d40 tx=0x7f3b9002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 -- 172.21.15.164:0/1497785220 shutdown_connections 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b940a4c20 0x7f3b94137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/1497785220 >> v2:172.21.15.165:3300/0 conn(0x7f3b9406d540 0x7f3b940a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/1497785220 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b9406d160 0x7f3b94134e00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 -- 172.21.15.164:0/1497785220 >> 172.21.15.164:0/1497785220 conn(0x7f3b940aacb0 msgr2=0x7f3b940ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 -- 172.21.15.164:0/1497785220 shutdown_connections 2024-09-10T16:06:35.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.069+0000 7f3b9bff3640 1 -- 172.21.15.164:0/1497785220 wait complete. 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.070+0000 7f3b9bff3640 10 monclient: build_initial_monmap 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.070+0000 7f3b9bff3640 10 monclient: monmap: 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.070+0000 7f3b9bff3640 1 Processor -- start 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.070+0000 7f3b9bff3640 1 -- start start 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.070+0000 7f3b9bff3640 10 monclient: init 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-c con 0x7f3b9406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 0x7f3b94130570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-b con 0x7f3b9406d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 0x7f3b94130ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): picked mon.noname-a con 0x7f3b940a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3b941385b0 con 0x7f3b940a4c20 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b94138430 con 0x7f3b9406d540 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3b94138730 con 0x7f3b9406d160 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9bff3640 10 monclient(hunting): authenticate will time out at 773.558115s 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43346/0 (socket says 172.21.15.164:43346) 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 1 -- 172.21.15.164:0/3012874240 learned_addr learned my addr 172.21.15.164:0/3012874240 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 0x7f3b94130ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): get_auth_request con 0x7f3b9406d160 auth_method 0 2024-09-10T16:06:35.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): get_auth_request con 0x7f3b940a4c20 auth_method 0 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9958e640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 msgr2=0x7f3b94130570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9958e640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 0x7f3b94130570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:35.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 msgr2=0x7f3b94130030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 msgr2=0x7f3b94130570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 0x7f3b94130570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient: _finish_hunting 0 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 monclient: found mon.noname-a 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b9a590640 1 -- 172.21.15.164:0/3012874240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b940a41f0 con 0x7f3b940a4c20 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.071+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9a590640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 msgr2=0x7f3b94130ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9a590640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 0x7f3b94130ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00efc0 tx=0x7f3b8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c018020 con 0x7f3b940a4c20 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: dump: 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.088 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b8c0146a0 con 0x7f3b940a4c20 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 5 monclient: authenticate success, global_id 4374 2024-09-10T16:06:35.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 10 monclient: _renew_subs 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b9409f2b0 con 0x7f3b940a4c20 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b8c014a90 con 0x7f3b940a4c20 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8a7fc640 10 monclient: dump: 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:35.090 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b8bfff640 4 set_mon_vals no callback set 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 10 monclient: _renew_subs 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.073+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b9409efd0 con 0x7f3b940a4c20 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3b8c005d10 con 0x7f3b940a4c20 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b9bff3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b9bff3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b9bff3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b8a7fc640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 0x7f3b78040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.074+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b940aa1e0 con 0x7f3b940a4c20 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.075+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f3b8c053a00 con 0x7f3b940a4c20 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.075+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 0x7f3b78040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.075+0000 7f3b99d8f640 10 monclient: get_auth_request con 0x7f3b7803e1c0 auth_method 0 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.078+0000 7f3b99d8f640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 0x7f3b78040680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002a110 tx=0x7f3b9002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.079+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3b8c022030 con 0x7f3b940a4c20 2024-09-10T16:06:35.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.079+0000 7f3b8a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:35.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.079+0000 7f3b8a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b9bff3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:06:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b9bff3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b9bff3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3b940a4660 con 0x7f3b940a4c20 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b8a7fc640 1 -- 172.21.15.164:0/3012874240 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f3b8c010040 con 0x7f3b940a4c20 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b8a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.320+0000 7f3b8a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:35.322 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":32,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 msgr2=0x7f3b78040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 0x7f3b78040680 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3b9002a110 tx=0x7f3b9002e000 comp rx=0 tx=0).stop 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 10 monclient: shutdown 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 msgr2=0x7f3b94130ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 0x7f3b94130ab0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3b8c00efc0 tx=0x7f3b8c00c490 comp rx=0 tx=0).stop 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 shutdown_connections 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b7803e1c0 0x7f3b78040680 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b940a4c20 0x7f3b94130ab0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b9406d540 0x7f3b94130570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 --2- 172.21.15.164:0/3012874240 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b9406d160 0x7f3b94130030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 >> 172.21.15.164:0/3012874240 conn(0x7f3b940aacb0 msgr2=0x7f3b94132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 shutdown_connections 2024-09-10T16:06:35.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.322+0000 7f3b9bff3640 1 -- 172.21.15.164:0/3012874240 wait complete. 2024-09-10T16:06:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:35.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:36.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:36.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:36.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:36.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:36.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:36.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:37.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:37.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:37.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:38.335 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.444+0000 7fb7660d4640 10 monclient: get_monmap_and_config 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.444+0000 7fb7660d4640 10 monclient: build_initial_monmap 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.444+0000 7fb7660d4640 10 monclient: monmap: 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:38.445755+0000 2024-09-10T16:06:38.446 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:38.445755+0000 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:38.447 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.445+0000 7fb7660d4640 1 Processor -- start 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.445+0000 7fb7660d4640 1 -- start start 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.445+0000 7fb7660d4640 10 monclient: init 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb76009c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb76009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-c con 0x7fb76013b5d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb76009a430 con 0x7fb76009c180 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb76009f860 con 0x7fb76009d160 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:38.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb7600a24e0 con 0x7fb76013b5d0 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb7660d4640 10 monclient(hunting): authenticate will time out at 776.933233s 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb75ffff640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 msgr2=0x7fb76013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb75effd640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb76009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb75f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb76009c180 0x7fb76009ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:40222/0 (socket says 172.21.15.164:40222) 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb75ffff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.446+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 learned_addr learned my addr 172.21.15.164:0/2683586894 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:38.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.448+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.164:6789/0 conn(0x7fb76009c180 legacy=0x7fb76009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:38.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.647+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 msgr2=0x7fb76013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:38.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.647+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb76009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:38.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.647+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:06:38.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:38.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:38.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:38.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:39.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.048+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb76009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:39.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.048+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 msgr2=0x7fb76013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:39.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.048+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:06:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:39.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.850+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 msgr2=0x7fb76013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:39.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.850+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb76009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:39.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:39.850+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:40.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:40.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:40.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:40.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:40.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:40.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:41.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): tick 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 msgr2=0x7fb76013d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb76013b5d0 0x7fb76013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb76009f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.164:6789/0 conn(0x7fb76009c180 legacy=0x7fb76009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient: _add_conns ranks=[3,2] 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:41.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb74c005980 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb74c008490 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb7600a24e0 con 0x7fb74c005980 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb74c00b030 con 0x7fb74c008490 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75d7fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 10 monclient(hunting): get_auth_request con 0x7fb74c008490 auth_method 0 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:41.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request con 0x7fb74c005980 auth_method 0 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:41.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.446+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request con 0x7fb74c005980 auth_method 0 2024-09-10T16:06:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 10 monclient(hunting): get_auth_request con 0x7fb74c008490 auth_method 0 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:41.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:06:41.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.647+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:06:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.816+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.816+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:41.816+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request con 0x7fb74c005980 auth_method 0 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 10 monclient(hunting): get_auth_request con 0x7fb74c008490 auth_method 0 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:42.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:06:42.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.048+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:06:42.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:42.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:42.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 10 monclient(hunting): get_auth_request con 0x7fb74c008490 auth_method 0 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:42.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request con 0x7fb74c005980 auth_method 0 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:42.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:42.849+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:43.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:43.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:43.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:43.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:43.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:43.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:43.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:43.817+0000 7f5037fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:44.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): tick 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 msgr2=0x7fb74c00a950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb74c008490 0x7fb74c00a950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 msgr2=0x7fb74c007e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.164:3300/0 conn(0x7fb74c005980 0x7fb74c007e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:06:44.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 reap_dead start 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb76009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb74c005df0 addr v2:172.21.15.165:3300/0 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb74c008e80 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7600a2bf0 con 0x7fb74c008e80 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb76009f860 con 0x7fb76009d160 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 1 -- 172.21.15.164:0/2683586894 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb7600a24e0 con 0x7fb74c005df0 2024-09-10T16:06:44.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75d7fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:44.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb74c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:44.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 msgr2=0x7fb74c008940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:44.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.446+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:44.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.449+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.164:6789/0 conn(0x7fb74c008e80 legacy=0x7fb74c009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:44.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.647+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 msgr2=0x7fb74c008940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:44.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.647+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb74c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:44.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.647+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:06:44.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.817+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:44.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.817+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:44.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:44.817+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:45.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.048+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb74c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:45.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.048+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 msgr2=0x7fb74c008940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:45.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.048+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:06:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.818+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.818+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:45.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.818+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:45.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.849+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb74c00d4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:45.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.849+0000 7fb75effd640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 msgr2=0x7fb74c008940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:45.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.849+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:06:45.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2005826784 0 0) 0x7fb7600a2bf0 con 0x7fb74c008e80 2024-09-10T16:06:45.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:06:45.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:45.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:45.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 10 monclient(hunting): my global_id is 4377 2024-09-10T16:06:45.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.857+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb748003520 con 0x7fb74c008e80 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2073632512 0 0) 0x7fb748003520 con 0x7fb74c008e80 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 msgr2=0x7fb74c008940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 --2- 172.21.15.164:0/2683586894 >> v2:172.21.15.165:3300/0 conn(0x7fb74c005df0 0x7fb74c008940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.165:6789/0 conn(0x7fb76009d160 legacy=0x7fb74c00d4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:45.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74c00ab30 con 0x7fb74c008e80 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fb7580042d0 con 0x7fb74c008e80 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: dump: 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.863 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 >> v1:172.21.15.164:6789/0 conn(0x7fb74c008e80 legacy=0x7fb74c009300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb748003a20 0x7fb748003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): picked mon.b con 0x7fb748003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:45.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748004800 0x7fb748004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): picked mon.a con 0x7fb748004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb74800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): picked mon.c con 0x7fb7480053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb7600a24e0 con 0x7fb748004800 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb74800c280 con 0x7fb748003a20 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb74800c480 con 0x7fb7480053c0 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.858+0000 7fb75e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75f7fe640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb748003a20 msgr2=0x7fb748003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:45.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb74800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb748003a20 0x7fb748003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): get_auth_request con 0x7fb7480053c0 auth_method 0 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748004800 0x7fb748004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): get_auth_request con 0x7fb748004800 auth_method 0 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:45.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient(hunting): handle_auth_done global_id 4377 payload 293 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb748003a20 msgr2=0x7fb748003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb748003a20 0x7fb748003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748004800 msgr2=0x7fb748004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb748004800 0x7fb748004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 monclient: found mon.c 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb748003520 con 0x7fb7480053c0 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient: _finish_auth 0 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb75ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb7660d4640 5 monclient: authenticate success, global_id 4377 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.859+0000 7fb7660d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.860+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 reap_dead start 2024-09-10T16:06:45.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75ffff640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 msgr2=0x7fb74800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:45.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75ffff640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb74800b9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb76009ab50 tx=0x7fb748005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f364a5f8dc26a7c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb758007d80 con 0x7fb7480053c0 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: dump: 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:45.877 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb758006290 con 0x7fb7480053c0 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb7660d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb7660d4640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 1 -- 172.21.15.164:0/2683586894 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7580046b0 con 0x7fb7480053c0 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb7660d4640 4 set_mon_vals no callback set 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.862+0000 7fb75e7fc640 10 monclient: dump: 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:45.878 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 10 monclient: shutdown 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 -- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 msgr2=0x7fb74800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:45.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb74800b9c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb76009ab50 tx=0x7fb748005830 comp rx=0 tx=0).stop 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 -- 172.21.15.164:0/2683586894 shutdown_connections 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 --2- 172.21.15.164:0/2683586894 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb74800b9c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 -- 172.21.15.164:0/2683586894 >> 172.21.15.164:0/2683586894 conn(0x7fb7600a3110 msgr2=0x7fb7600a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 -- 172.21.15.164:0/2683586894 shutdown_connections 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.863+0000 7fb7660d4640 1 -- 172.21.15.164:0/2683586894 wait complete. 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.864+0000 7fb7660d4640 10 monclient: build_initial_monmap 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.864+0000 7fb7660d4640 10 monclient: monmap: 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:45.880 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.864+0000 7fb7660d4640 1 Processor -- start 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 -- start start 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient: init 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-c con 0x7fb7480053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 0x7fb7601c9270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-b con 0x7fb7601c8df0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 0x7fb760137a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): picked mon.noname-a con 0x7fb7601375b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb7600a2230 con 0x7fb7601375b0 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb7600a20b0 con 0x7fb7601c8df0 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb760074e10 con 0x7fb7480053c0 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb7660d4640 10 monclient(hunting): authenticate will time out at 784.352618s 2024-09-10T16:06:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75effd640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 msgr2=0x7fb7601c9270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 0x7fb760137a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52678/0 (socket says 172.21.15.164:52678) 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 1 -- 172.21.15.164:0/360837043 learned_addr learned my addr 172.21.15.164:0/360837043 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request con 0x7fb7480053c0 auth_method 0 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75effd640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 0x7fb7601c9270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): get_auth_request con 0x7fb7601375b0 auth_method 0 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.865+0000 7fb75ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 10 monclient(hunting): handle_auth_done global_id 4388 payload 824 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 -- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 msgr2=0x7fb7601c9270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 0x7fb7601c9270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 -- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 msgr2=0x7fb760137a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 0x7fb760137a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75f7fe640 1 -- 172.21.15.164:0/360837043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb76009f4d0 con 0x7fb7480053c0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.866+0000 7fb75ffff640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 0x7fb760137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb75f7fe640 1 -- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 msgr2=0x7fb7601c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb75f7fe640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb750007ba0 tx=0x7fb7500020f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb75001a030 con 0x7fb7480053c0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: dump: 2024-09-10T16:06:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 5 monclient: authenticate success, global_id 4388 2024-09-10T16:06:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb750013e50 con 0x7fb7480053c0 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb750016450 con 0x7fb7480053c0 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb747fff640 10 monclient: dump: 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb75dffb640 4 set_mon_vals no callback set 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 10 monclient: _renew_subs 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7600905a0 con 0x7fb7480053c0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 10 monclient: _renew_subs 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.868+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7600a5840 con 0x7fb7480053c0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.869+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb75001d030 con 0x7fb7480053c0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.869+0000 7fb7660d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.869+0000 7fb7660d4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.869+0000 7fb7660d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.869+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb76009d930 con 0x7fb7480053c0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.870+0000 7fb747fff640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 0x7fb74c043ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.870+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7fb750053500 con 0x7fb7480053c0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.870+0000 7fb75effd640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 0x7fb74c043ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.870+0000 7fb75effd640 10 monclient: get_auth_request con 0x7fb74c0419e0 auth_method 0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.872+0000 7fb75effd640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 0x7fb74c043ea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb754006e50 tx=0x7fb754008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.874+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb76009f4d0 con 0x7fb7480053c0 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.875+0000 7fb747fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:45.875+0000 7fb747fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.119+0000 7fb7660d4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:06:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.119+0000 7fb7660d4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.119+0000 7fb7660d4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.120+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb760070640 con 0x7fb7480053c0 2024-09-10T16:06:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.120+0000 7fb747fff640 1 -- 172.21.15.164:0/360837043 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fb750052090 con 0x7fb7480053c0 2024-09-10T16:06:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.120+0000 7fb747fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:06:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.120+0000 7fb747fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:46.122 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 msgr2=0x7fb74c043ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 0x7fb74c043ea0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb754006e50 tx=0x7fb754008040 comp rx=0 tx=0).stop 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 10 monclient: shutdown 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 msgr2=0x7fb7601c88b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb750007ba0 tx=0x7fb7500020f0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 shutdown_connections 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb74c0419e0 0x7fb74c043ea0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7601375b0 0x7fb760137a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7601c8df0 0x7fb7601c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 --2- 172.21.15.164:0/360837043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7480053c0 0x7fb7601c88b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 >> 172.21.15.164:0/360837043 conn(0x7fb7600a3110 msgr2=0x7fb76013a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 shutdown_connections 2024-09-10T16:06:46.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.122+0000 7fb7660d4640 1 -- 172.21.15.164:0/360837043 wait complete. 2024-09-10T16:06:46.134 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:06:46.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:06:46.135 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:06:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 10 monclient: get_monmap_and_config 2024-09-10T16:06:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 10 monclient: monmap: 2024-09-10T16:06:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:46.233073+0000 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:46.233073+0000 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:46.234 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 1 Processor -- start 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 1 -- start start 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.232+0000 7f5365c47640 10 monclient: init 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 0x7f5360134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-b con 0x7f536006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53600a46a0 0x7f53600a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-a con 0x7f53600a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:46.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5360137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f536009ce50 con 0x7f5360137ca0 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f536006db60 con 0x7f53600a46a0 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f536006cf50 con 0x7f536006cb70 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f5365c47640 10 monclient(hunting): authenticate will time out at 784.720508s 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 0x7f5360134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53600a46a0 0x7f53600a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 0x7f5360134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52686/0 (socket says 172.21.15.164:52686) 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53600a46a0 0x7f53600a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37398/0 (socket says 172.21.15.164:37398) 2024-09-10T16:06:46.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 1 -- 172.21.15.164:0/3423259113 learned_addr learned my addr 172.21.15.164:0/3423259113 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): get_auth_request con 0x7f536006cb70 auth_method 0 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f53600a46a0 auth_method 0 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient(hunting): handle_auth_done global_id 4383 payload 274 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 -- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 msgr2=0x7f5360134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 --2- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 0x7f5360134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 -- 172.21.15.164:0/3423259113 >> v1:172.21.15.164:6789/0 conn(0x7f5360137ca0 legacy=0x7f5360138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 monclient: found mon.noname-a 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:46.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.233+0000 7f535effd640 1 -- 172.21.15.164:0/3423259113 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536009cb70 con 0x7f53600a46a0 2024-09-10T16:06:46.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535effd640 1 -- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 msgr2=0x7f53600a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535effd640 1 --2- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f53600a4a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53500097a0 tx=0x7f5350029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13f57859618aec7f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 1 -- 172.21.15.164:0/3423259113 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350037070 con 0x7f53600a46a0 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: dump: 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.246 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 1 -- 172.21.15.164:0/3423259113 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5350032440 con 0x7f53600a46a0 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f5365c47640 5 monclient: authenticate success, global_id 4383 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f5365c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f5365c47640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 1 -- 172.21.15.164:0/3423259113 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350032810 con 0x7f53600a46a0 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f5365c47640 4 set_mon_vals no callback set 2024-09-10T16:06:46.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.236+0000 7f535e7fc640 10 monclient: dump: 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 10 monclient: shutdown 2024-09-10T16:06:46.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 -- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 msgr2=0x7f53600a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 --2- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f53600a4a80 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f53500097a0 tx=0x7f5350029fd0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 -- 172.21.15.164:0/3423259113 shutdown_connections 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 --2- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f53600a4a80 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 --2- 172.21.15.164:0/3423259113 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536006cb70 0x7f5360134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 -- 172.21.15.164:0/3423259113 >> 172.21.15.164:0/3423259113 conn(0x7f53600aa900 msgr2=0x7f53600aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 -- 172.21.15.164:0/3423259113 shutdown_connections 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.237+0000 7f5365c47640 1 -- 172.21.15.164:0/3423259113 wait complete. 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.238+0000 7f5365c47640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.238+0000 7f5365c47640 10 monclient: monmap: 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:46.249 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.238+0000 7f5365c47640 1 Processor -- start 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.238+0000 7f5365c47640 1 -- start start 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.238+0000 7f5365c47640 10 monclient: init 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536006cb70 0x7f53601312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-b con 0x7f536006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-a con 0x7f53600a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5360137ca0 0x7f536009e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): picked mon.noname-c con 0x7f5360137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53601389c0 con 0x7f53600a46a0 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5360138840 con 0x7f536006cb70 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f536006ed80 con 0x7f5360137ca0 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f5365c47640 10 monclient(hunting): authenticate will time out at 784.726210s 2024-09-10T16:06:46.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37410/0 (socket says 172.21.15.164:37410) 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 -- 172.21.15.164:0/1650626139 learned_addr learned my addr 172.21.15.164:0/1650626139 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f53600a46a0 auth_method 0 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535f7fe640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536006cb70 msgr2=0x7f53601312c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5360137ca0 0x7f536009e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535f7fe640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536006cb70 0x7f53601312c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:46.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): get_auth_request con 0x7f5360137ca0 auth_method 0 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient(hunting): handle_auth_done global_id 4386 payload 824 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5360137ca0 msgr2=0x7f536009e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5360137ca0 0x7f536009e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536006cb70 msgr2=0x7f53601312c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536006cb70 0x7f53601312c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 monclient: found mon.noname-a 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.239+0000 7f535effd640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536006e720 con 0x7f53600a46a0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f535effd640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 msgr2=0x7f5360131800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f535effd640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5350000c00 tx=0x7f53500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5350037040 con 0x7f53600a46a0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 10 monclient: dump: 2024-09-10T16:06:46.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5350042070 con 0x7f53600a46a0 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.241+0000 7f5365c47640 5 monclient: authenticate success, global_id 4386 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 monclient: _renew_subs 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536009fdf0 con 0x7f53600a46a0 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5347fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535003bb00 con 0x7f53600a46a0 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5347fff640 10 monclient: dump: 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 monclient: _renew_subs 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5360131d40 con 0x7f53600a46a0 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f535dffb640 4 set_mon_vals no callback set 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:46.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 0x7f536009f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.242+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53600aa5b0 con 0x7f53600a0950 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 0x7f536009f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5350047460 con 0x7f53600a46a0 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient: get_auth_request con 0x7f53600a0950 auth_method 0 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 10 monclient(hunting): handle_auth_done global_id 4389 payload 824 2024-09-10T16:06:46.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535f7fe640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5360134f70 con 0x7f53600a0950 2024-09-10T16:06:46.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f5347fff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534c03e1c0 0x7f534c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f5350073360 con 0x7f53600a46a0 2024-09-10T16:06:46.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535ffff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534c03e1c0 0x7f534c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.243+0000 7f535ffff640 10 monclient: get_auth_request con 0x7f534c03e1c0 auth_method 0 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f535f7fe640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 msgr2=0x7f536009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f535f7fe640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 0x7f536009f080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5354006c80 tx=0x7f5354009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f535401b790 con 0x7f53600a0950 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 10 monclient: dump: 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5354021070 con 0x7f53600a0950 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 msgr2=0x7f536009f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 0x7f536009f080 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5354006c80 tx=0x7f5354009780 comp rx=0 tx=0).stop 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 msgr2=0x7f536009f080 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.245+0000 7f5347fff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a0950 0x7f536009f080 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5354006c80 tx=0x7f5354009780 comp rx=0 tx=0).stop 2024-09-10T16:06:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.246+0000 7f535ffff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534c03e1c0 0x7f534c040680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f53580055e0 tx=0x7f5358003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.251+0000 7f5365c47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.251+0000 7f5365c47640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 0x7f536002d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f5365c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5360070870 con 0x7f536002d440 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 0x7f536002d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient: get_auth_request con 0x7f536002d440 auth_method 0 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 10 monclient(hunting): handle_auth_done global_id 4392 payload 824 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.252+0000 7f535f7fe640 1 -- 172.21.15.164:0/1650626139 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53600770f0 con 0x7f536002d440 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f535f7fe640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 msgr2=0x7f536002d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f535f7fe640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 0x7f536002d850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5354020000 tx=0x7f5354004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5354026de0 con 0x7f536002d440 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 10 monclient: dump: 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f5354026de0 con 0x7f536002d440 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 msgr2=0x7f536002d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 0x7f536002d850 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5354020000 tx=0x7f5354004730 comp rx=0 tx=0).stop 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 msgr2=0x7f536002d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.254+0000 7f5347fff640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536002d440 0x7f536002d850 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5354020000 tx=0x7f5354004730 comp rx=0 tx=0).stop 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 12, 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 48429, 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:46.267 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.268 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.269 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:06:46.270 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.271 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.272 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.273 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:06:46.274 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:06:46.276 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:06:46.277 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.278 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:06:46.279 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:06:46.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534c03e1c0 msgr2=0x7f534c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534c03e1c0 0x7f534c040680 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f53580055e0 tx=0x7f5358003d10 comp rx=0 tx=0).stop 2024-09-10T16:06:46.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 10 monclient: shutdown 2024-09-10T16:06:46.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 msgr2=0x7f5360131800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5350000c00 tx=0x7f53500046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f535ffff640 1 -- 172.21.15.164:0/1650626139 reap_dead start 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 shutdown_connections 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 --2- 172.21.15.164:0/1650626139 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53600a46a0 0x7f5360131800 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 >> 172.21.15.164:0/1650626139 conn(0x7f53600aa900 msgr2=0x7f53600a95c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 shutdown_connections 2024-09-10T16:06:46.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.255+0000 7f5365c47640 1 -- 172.21.15.164:0/1650626139 wait complete. 2024-09-10T16:06:46.281 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:06:46.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:06:46.282 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.378+0000 7f60dbec6640 10 monclient: get_monmap_and_config 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.378+0000 7f60dbec6640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.378+0000 7f60dbec6640 10 monclient: monmap: 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:46.379509+0000 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:46.379509+0000 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.380 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:46.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:46.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:46.381 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:46.381 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:46.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 Processor -- start 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 -- start start 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient: init 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:06:46.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60d409d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60d409ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 0x7f60d413cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-b con 0x7f60d409e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d40a0870 con 0x7f60d409ab50 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d4074c70 con 0x7f60d409d930 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d409d3e0 con 0x7f60d409e420 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60dbec6640 10 monclient(hunting): authenticate will time out at 784.866870s 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60d9461640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f60d409ab50 0x7f60d41397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55894/0 (socket says 172.21.15.164:55894) 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60d9461640 1 -- 172.21.15.164:0/2311590628 learned_addr learned my addr 172.21.15.164:0/2311590628 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60d9c62640 1 -- 172.21.15.164:0/2311590628 >> v1:172.21.15.165:6789/0 conn(0x7f60d409d930 legacy=0x7f60d409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:46.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.379+0000 7f60da463640 1 --2- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 0x7f60d413cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): get_auth_request con 0x7f60d409e420 auth_method 0 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient(hunting): handle_auth_done global_id 4397 payload 274 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 1 -- 172.21.15.164:0/2311590628 >> v1:172.21.15.165:6789/0 conn(0x7f60d409d930 legacy=0x7f60d409dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 1 -- 172.21.15.164:0/2311590628 >> v1:172.21.15.164:6789/0 conn(0x7f60d409ab50 legacy=0x7f60d41397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 1 monclient: found mon.noname-b 2024-09-10T16:06:46.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.380+0000 7f60da463640 1 -- 172.21.15.164:0/2311590628 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d409ecb0 con 0x7f60d409e420 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60da463640 1 -- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 msgr2=0x7f60d413cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60da463640 1 --2- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 0x7f60d413cef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00a970 tx=0x7f60cc02b360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5c85b1bf28c6156 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 1 -- 172.21.15.164:0/2311590628 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc038070 con 0x7f60d409e420 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:06:46.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: dump: 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 1 -- 172.21.15.164:0/2311590628 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60cc0306b0 con 0x7f60d409e420 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60dbec6640 5 monclient: authenticate success, global_id 4397 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60dbec6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60dbec6640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 1 -- 172.21.15.164:0/2311590628 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60cc00b040 con 0x7f60d409e420 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60dbec6640 4 set_mon_vals no callback set 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.383+0000 7f60d8c60640 10 monclient: dump: 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.395 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 10 monclient: shutdown 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 -- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 msgr2=0x7f60d413cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 --2- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 0x7f60d413cef0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f60cc00a970 tx=0x7f60cc02b360 comp rx=0 tx=0).stop 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 -- 172.21.15.164:0/2311590628 shutdown_connections 2024-09-10T16:06:46.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 --2- 172.21.15.164:0/2311590628 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409e420 0x7f60d413cef0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 -- 172.21.15.164:0/2311590628 >> 172.21.15.164:0/2311590628 conn(0x7f60d40a3110 msgr2=0x7f60d40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 -- 172.21.15.164:0/2311590628 shutdown_connections 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 1 -- 172.21.15.164:0/2311590628 wait complete. 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.384+0000 7f60dbec6640 10 monclient: monmap: 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 Processor -- start 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 -- start start 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient: init 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409ab50 0x7f60d4099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-a con 0x7f60d409ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 0x7f60d40998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-c con 0x7f60d409d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d409e420 0x7f60d4137680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): picked mon.noname-b con 0x7f60d409e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f60d40a2940 con 0x7f60d409ab50 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f60d409ef30 con 0x7f60d409e420 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d40a2ac0 con 0x7f60d409d930 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60dbec6640 10 monclient(hunting): authenticate will time out at 784.872643s 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409ab50 0x7f60d4099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 0x7f60d40998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409ab50 0x7f60d4099380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37442/0 (socket says 172.21.15.164:37442) 2024-09-10T16:06:46.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60da463640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d409e420 msgr2=0x7f60d4137680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 1 -- 172.21.15.164:0/1448444837 learned_addr learned my addr 172.21.15.164:0/1448444837 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): get_auth_request con 0x7f60d409ab50 auth_method 0 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60da463640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d409e420 0x7f60d4137680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): get_auth_request con 0x7f60d409d930 auth_method 0 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient(hunting): handle_auth_done global_id 4400 payload 824 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d409e420 msgr2=0x7f60d4137680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d409e420 0x7f60d4137680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409ab50 msgr2=0x7f60d4099380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409ab50 0x7f60d4099380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 monclient: found mon.noname-c 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.385+0000 7f60d9461640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d409d620 con 0x7f60d409d930 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60d9461640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 msgr2=0x7f60d40998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60d9461640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 0x7f60d40998c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c3e0 tx=0x7f60c400c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4017070 con 0x7f60d409d930 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: dump: 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.402 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 5 monclient: authenticate success, global_id 4400 2024-09-10T16:06:46.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c40070d0 con 0x7f60d409d930 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 10 monclient: _renew_subs 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d413b100 con 0x7f60d409d930 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c40074c0 con 0x7f60d409d930 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60ca7fc640 10 monclient: dump: 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.404 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60cbfff640 4 set_mon_vals no callback set 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 10 monclient: _renew_subs 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.388+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d4139be0 con 0x7f60d409d930 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60dbec6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60dbec6640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 0x7f60d4137bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d40a4310 con 0x7f60d413bb70 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 0x7f60d4137bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient: get_auth_request con 0x7f60d413bb70 auth_method 0 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f60c401a070 con 0x7f60d409d930 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2024-09-10T16:06:46.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60d9c62640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60d40a46f0 con 0x7f60d413bb70 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60ca7fc640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60b803e1c0 0x7f60b8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 4374+0+0 (secure 0 0 0) 0x7f60c4052000 con 0x7f60d409d930 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60da463640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60b803e1c0 0x7f60b8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.389+0000 7f60da463640 10 monclient: get_auth_request con 0x7f60b803e1c0 auth_method 0 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60d9c62640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 msgr2=0x7f60d4137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60d9c62640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 0x7f60d4137bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d00054e0 con 0x7f60d413bb70 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.391+0000 7f60ca7fc640 10 monclient: dump: 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.407 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60d0018070 con 0x7f60d413bb70 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 msgr2=0x7f60d4137bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 0x7f60d4137bc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0007f90 comp rx=0 tx=0).stop 2024-09-10T16:06:46.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 msgr2=0x7f60d4137bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:46.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60ca7fc640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d413bb70 0x7f60d4137bc0 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0007f90 comp rx=0 tx=0).stop 2024-09-10T16:06:46.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.392+0000 7f60da463640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60b803e1c0 0x7f60b8040680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f60d406de20 tx=0x7f60cc00c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60dbec6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60dbec6640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 0x7f60d40717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60dbec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d4138510 con 0x7f60d4138100 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 0x7f60d40717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient: get_auth_request con 0x7f60d4138100 auth_method 0 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 10 monclient(hunting): handle_auth_done global_id 4406 payload 824 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.398+0000 7f60d9c62640 1 -- 172.21.15.164:0/1448444837 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60d413ab40 con 0x7f60d4138100 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60d9c62640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 msgr2=0x7f60d40717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60d9c62640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 0x7f60d40717f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0005d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60d0013600 con 0x7f60d4138100 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.400+0000 7f60ca7fc640 10 monclient: dump: 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.412 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f60d0013600 con 0x7f60d4138100 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 msgr2=0x7f60d40717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 0x7f60d40717f0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0005d00 comp rx=0 tx=0).stop 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 msgr2=0x7f60d40717f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:06:46.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60ca7fc640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4138100 0x7f60d40717f0 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f60d0007f60 tx=0x7f60d0005d00 comp rx=0 tx=0).stop 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 12, 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 48572, 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:06:46.414 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.415 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.416 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:06:46.417 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:06:46.418 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:06:46.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.420 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:06:46.421 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:06:46.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:06:46.424 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.425 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.426 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:06:46.427 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60b803e1c0 msgr2=0x7f60b8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60b803e1c0 0x7f60b8040680 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f60d406de20 tx=0x7f60cc00c040 comp rx=0 tx=0).stop 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 10 monclient: shutdown 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 msgr2=0x7f60d40998c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 --2- 172.21.15.164:0/1448444837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d930 0x7f60d40998c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c3e0 tx=0x7f60c400c8a0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60da463640 1 -- 172.21.15.164:0/1448444837 reap_dead start 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60da463640 1 -- 172.21.15.164:0/1448444837 reap_dead start 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 shutdown_connections 2024-09-10T16:06:46.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 >> 172.21.15.164:0/1448444837 conn(0x7f60d40a3110 msgr2=0x7f60d409b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 shutdown_connections 2024-09-10T16:06:46.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.401+0000 7f60dbec6640 1 -- 172.21.15.164:0/1448444837 wait complete. 2024-09-10T16:06:46.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:06:46.430 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:06:46.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.818+0000 7f5037fff640 10 monclient: tick 2024-09-10T16:06:46.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.818+0000 7f5037fff640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.818+0000 7f5037fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.855+0000 7f50497fa640 1 -- 172.21.15.164:0/2893472592 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]=0 pool 'unique_pool_0' created v21) ==== 130+0+0 (secure 0 0 0) 0x7f5040011bf0 con 0x7f504c06d160 2024-09-10T16:06:46.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.855+0000 7f50497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}] 2024-09-10T16:06:46.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.855+0000 7f50497fa640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' created 2024-09-10T16:06:46.857 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'unique_pool_0' created 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 msgr2=0x7f5030040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 0x7f5030040700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f504c0773f0 tx=0x7f503802e000 comp rx=0 tx=0).stop 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 10 monclient: shutdown 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 msgr2=0x7f504c131250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.857+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f504000c7f0 tx=0x7f504000cbf0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 shutdown_connections 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f503003e240 0x7f5030040700 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f504c0a4c20 0x7f504c13dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f504c06d540 0x7f504c131790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 --2- 172.21.15.164:0/2893472592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f504c06d160 0x7f504c131250 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 >> 172.21.15.164:0/2893472592 conn(0x7f504c0aacb0 msgr2=0x7f504c136740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 shutdown_connections 2024-09-10T16:06:46.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.858+0000 7f5052ae2640 1 -- 172.21.15.164:0/2893472592 wait complete. 2024-09-10T16:06:46.873 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.973+0000 7f3618d3a640 10 monclient: get_monmap_and_config 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.973+0000 7f3618d3a640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.973+0000 7f3618d3a640 10 monclient: monmap: 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.975 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:06:46.974710+0000 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:06:46.974710+0000 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:06:46.976 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.974+0000 7f3618d3a640 1 Processor -- start 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.974+0000 7f3618d3a640 1 -- start start 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.974+0000 7f3618d3a640 10 monclient: init 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36140a0520 addr v1:172.21.15.165:6789/0 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f361409d070 0x7f36140a2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-a con 0x7f361409d070 addr v2:172.21.15.164:3300/0 2024-09-10T16:06:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 0x7f361413daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f361413b6b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f361409e8b0 con 0x7f36140a0520 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f361409fb50 con 0x7f361409d070 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f361409eb90 con 0x7f361413b6b0 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3618d3a640 10 monclient(hunting): authenticate will time out at 785.462155s 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f361259c640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f36140a0520 legacy=0x7f36140a1f20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f361409d070 0x7f36140a2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 0x7f361413daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f361409d070 0x7f36140a2630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37452/0 (socket says 172.21.15.164:37452) 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 -- 172.21.15.164:0/4208059487 learned_addr learned my addr 172.21.15.164:0/4208059487 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): get_auth_request con 0x7f361409d070 auth_method 0 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): get_auth_request con 0x7f361413b6b0 auth_method 0 2024-09-10T16:06:46.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient(hunting): handle_auth_done global_id 4398 payload 274 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 -- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 msgr2=0x7f361413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 0x7f361413daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 -- 172.21.15.164:0/4208059487 >> v1:172.21.15.165:6789/0 conn(0x7f36140a0520 legacy=0x7f36140a1f20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 monclient: found mon.noname-a 2024-09-10T16:06:46.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:06:46.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3611d9b640 1 -- 172.21.15.164:0/4208059487 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f361409da10 con 0x7f361409d070 2024-09-10T16:06:46.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.975+0000 7f3612d9d640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 0x7f361413daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3611d9b640 1 -- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361409d070 msgr2=0x7f36140a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3611d9b640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361409d070 0x7f36140a2630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3608009870 tx=0x7f360802a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee4f18635c5bbdd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 1 -- 172.21.15.164:0/4208059487 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608037070 con 0x7f361409d070 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: dump: 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 1 -- 172.21.15.164:0/4208059487 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3608032640 con 0x7f361409d070 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3618d3a640 5 monclient: authenticate success, global_id 4398 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3618d3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:06:46.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3618d3a640 10 monclient: get_monmap_and_config success 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 1 -- 172.21.15.164:0/4208059487 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608032a10 con 0x7f361409d070 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f3618d3a640 4 set_mon_vals no callback set 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.978+0000 7f361159a640 10 monclient: dump: 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.991 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 10 monclient: shutdown 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 -- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361409d070 msgr2=0x7f36140a2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361409d070 0x7f36140a2630 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3608009870 tx=0x7f360802a130 comp rx=0 tx=0).stop 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 -- 172.21.15.164:0/4208059487 shutdown_connections 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f361413b6b0 0x7f361413daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 --2- 172.21.15.164:0/4208059487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361409d070 0x7f36140a2630 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 -- 172.21.15.164:0/4208059487 >> 172.21.15.164:0/4208059487 conn(0x7f36140a31f0 msgr2=0x7f36140a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 -- 172.21.15.164:0/4208059487 shutdown_connections 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 1 -- 172.21.15.164:0/4208059487 wait complete. 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 10 monclient: build_initial_monmap 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.979+0000 7f3618d3a640 10 monclient: monmap: 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:06:46.993 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 Processor -- start 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 -- start start 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient: init 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient: _reopen_session rank -1 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 0x7f36141cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-b con 0x7f361409d070 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-c con 0x7f36140a0520 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): picked mon.noname-a con 0x7f361413b6b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:06:46.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36140a0a80 con 0x7f361413b6b0 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f36140a0900 con 0x7f361409d070 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f361409a060 con 0x7f36140a0520 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): _renew_subs 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3618d3a640 10 monclient(hunting): authenticate will time out at 785.467689s 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52746/0 (socket says 172.21.15.164:52746) 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37462/0 (socket says 172.21.15.164:37462) 2024-09-10T16:06:46.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 1 -- 172.21.15.164:0/2579694762 learned_addr learned my addr 172.21.15.164:0/2579694762 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): get_auth_request con 0x7f36140a0520 auth_method 0 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f361259c640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 msgr2=0x7f36141cada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): get_auth_request con 0x7f361413b6b0 auth_method 0 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f361259c640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 0x7f36141cada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3611d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.980+0000 7f3612d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 10 monclient(hunting): handle_auth_done global_id 4412 payload 824 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 msgr2=0x7f36141cada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 0x7f36141cada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 msgr2=0x7f3614099910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 monclient: found mon.noname-c 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:06:46.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3611d9b640 1 -- 172.21.15.164:0/2579694762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3614092e50 con 0x7f36140a0520 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.981+0000 7f3612d9d640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3611d9b640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 msgr2=0x7f36140993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3611d9b640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f36080090b0 tx=0x7f36080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608041020 con 0x7f36140a0520 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: dump: 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:46.998 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: _finish_auth 0 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f360803d630 con 0x7f36140a0520 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 5 monclient: authenticate success, global_id 4412 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 10 monclient: _renew_subs 2024-09-10T16:06:46.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3608037040 con 0x7f36140a0520 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3614092890 con 0x7f36140a0520 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3602ffd640 10 monclient: dump: 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:06:47.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3610d99640 4 set_mon_vals no callback set 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 10 monclient: _renew_subs 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.983+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3614091650 con 0x7f36140a0520 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3608044020 con 0x7f36140a0520 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3618d3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3618d3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3618d3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36140a2ce0 con 0x7f36140a0520 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3602ffd640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 0x7f35f40406e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:06:47.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 4698+0+0 (secure 0 0 0) 0x7f3608073160 con 0x7f36140a0520 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f361259c640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 0x7f35f40406e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.984+0000 7f361259c640 10 monclient: get_auth_request con 0x7f35f403e220 auth_method 0 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.987+0000 7f361259c640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 0x7f35f40406e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3604006fd0 tx=0x7f3604006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.989+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f360804b030 con 0x7f36140a0520 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.989+0000 7f3602ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:06:47.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:46.989+0000 7f3602ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:06:47.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.163+0000 7f3618d3a640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-10T16:06:47.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.164+0000 7f3618d3a640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-10T16:06:47.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.164+0000 7f3618d3a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:06:47.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.164+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true} v 0) -- 0x7f3614090b30 con 0x7f36140a0520 2024-09-10T16:06:47.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.863+0000 7f3602ffd640 1 -- 172.21.15.164:0/2579694762 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]=0 enabled application 'rados' on pool 'unique_pool_0' v22) ==== 197+0+0 (secure 0 0 0) 0x7f3608071bb0 con 0x7f36140a0520 2024-09-10T16:06:47.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.863+0000 7f3602ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}] 2024-09-10T16:06:47.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.863+0000 7f3602ffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rados' on pool 'unique_pool_0' 2024-09-10T16:06:47.865 INFO:teuthology.orchestra.run.smithi164.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 msgr2=0x7f35f40406e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 0x7f35f40406e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3604006fd0 tx=0x7f3604006d80 comp rx=0 tx=0).stop 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 10 monclient: shutdown 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 msgr2=0x7f36140993d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f36080090b0 tx=0x7f36080046c0 comp rx=0 tx=0).stop 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 shutdown_connections 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403e220 0x7f35f40406e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f361413b6b0 0x7f3614099910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36140a0520 0x7f36140993d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:47.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 --2- 172.21.15.164:0/2579694762 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f361409d070 0x7f36141cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:06:47.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 >> 172.21.15.164:0/2579694762 conn(0x7f36140a31f0 msgr2=0x7f3614138940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:06:47.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 shutdown_connections 2024-09-10T16:06:47.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:06:47.866+0000 7f3618d3a640 1 -- 172.21.15.164:0/2579694762 wait complete. 2024-09-10T16:06:48.885 DEBUG:teuthology.orchestra.run.smithi165:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-10T16:06:48.919 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight read -> 100 2024-09-10T16:06:48.919 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight write -> 50 2024-09-10T16:06:48.919 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight delete -> 50 2024-09-10T16:06:48.920 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight snap_create -> 50 2024-09-10T16:06:48.920 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight snap_remove -> 50 2024-09-10T16:06:48.920 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight rollback -> 50 2024-09-10T16:06:48.920 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight copy_from -> 50 2024-09-10T16:06:48.920 INFO:tasks.rados.rados.0.smithi165.stdout:adding op weight write_excl -> 50 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout:ceph version 19.3.0-4858-gbad1f558 (bad1f558c327094897e4a32e99dd5cb4ae6ccd46) squid (dev) 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout:Configuration: 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout: Number of operations: 4000 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout: Number of objects: 50 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout: Max in flight operations: 16 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout: Object size (in bytes): 4000000 2024-09-10T16:06:48.921 INFO:tasks.rados.rados.0.smithi165.stdout: Write stride min: 400000 2024-09-10T16:06:48.922 INFO:tasks.rados.rados.0.smithi165.stdout: Write stride max: 800000 2024-09-10T16:06:49.884 INFO:tasks.rados.rados.0.smithi165.stdout:1: write initial oid 1 2024-09-10T16:06:49.884 INFO:tasks.rados.rados.0.smithi165.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-10T16:06:49.889 INFO:tasks.rados.rados.0.smithi165.stdout:1: writing smithi16534841-1 from 416807 to 892056 tid 1 2024-09-10T16:06:49.896 INFO:tasks.rados.rados.0.smithi165.stdout:1: writing smithi16534841-1 from 1542129 to 2016807 tid 2 2024-09-10T16:06:49.896 INFO:tasks.rados.rados.0.smithi165.stdout:2: write initial oid 2 2024-09-10T16:06:49.896 INFO:tasks.rados.rados.0.smithi165.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-10T16:06:49.901 INFO:tasks.rados.rados.0.smithi165.stdout:2: writing smithi16534841-2 from 416807 to 892056 tid 1 2024-09-10T16:06:49.908 INFO:tasks.rados.rados.0.smithi165.stdout:2: writing smithi16534841-2 from 1542129 to 2016807 tid 2 2024-09-10T16:06:49.908 INFO:tasks.rados.rados.0.smithi165.stdout:3: write initial oid 3 2024-09-10T16:06:49.908 INFO:tasks.rados.rados.0.smithi165.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-10T16:06:49.914 INFO:tasks.rados.rados.0.smithi165.stdout:3: writing smithi16534841-3 from 433614 to 984112 tid 1 2024-09-10T16:06:49.920 INFO:tasks.rados.rados.0.smithi165.stdout:3: writing smithi16534841-3 from 1600611 to 2033614 tid 2 2024-09-10T16:06:49.920 INFO:tasks.rados.rados.0.smithi165.stdout:4: write initial oid 4 2024-09-10T16:06:49.920 INFO:tasks.rados.rados.0.smithi165.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-10T16:06:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:4: writing smithi16534841-4 from 450421 to 1076168 tid 1 2024-09-10T16:06:49.932 INFO:tasks.rados.rados.0.smithi165.stdout:4: writing smithi16534841-4 from 1659093 to 2050421 tid 2 2024-09-10T16:06:49.932 INFO:tasks.rados.rados.0.smithi165.stdout:5: write initial oid 5 2024-09-10T16:06:49.932 INFO:tasks.rados.rados.0.smithi165.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-10T16:06:49.938 INFO:tasks.rados.rados.0.smithi165.stdout:5: writing smithi16534841-5 from 467228 to 1168224 tid 1 2024-09-10T16:06:49.943 INFO:tasks.rados.rados.0.smithi165.stdout:5: writing smithi16534841-5 from 1717575 to 2067228 tid 2 2024-09-10T16:06:49.943 INFO:tasks.rados.rados.0.smithi165.stdout:6: write initial oid 6 2024-09-10T16:06:49.944 INFO:tasks.rados.rados.0.smithi165.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-10T16:06:49.950 INFO:tasks.rados.rados.0.smithi165.stdout:6: writing smithi16534841-6 from 484035 to 1260280 tid 1 2024-09-10T16:06:49.955 INFO:tasks.rados.rados.0.smithi165.stdout:6: writing smithi16534841-6 from 2059704 to 2084035 tid 2 2024-09-10T16:06:49.955 INFO:tasks.rados.rados.0.smithi165.stdout:7: write initial oid 7 2024-09-10T16:06:49.955 INFO:tasks.rados.rados.0.smithi165.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-10T16:06:49.960 INFO:tasks.rados.rados.0.smithi165.stdout:7: writing smithi16534841-7 from 500842 to 952336 tid 1 2024-09-10T16:06:49.966 INFO:tasks.rados.rados.0.smithi165.stdout:7: writing smithi16534841-7 from 1718186 to 2100842 tid 2 2024-09-10T16:06:49.966 INFO:tasks.rados.rados.0.smithi165.stdout:8: write initial oid 8 2024-09-10T16:06:49.966 INFO:tasks.rados.rados.0.smithi165.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-10T16:06:49.972 INFO:tasks.rados.rados.0.smithi165.stdout:8: writing smithi16534841-8 from 517649 to 1044392 tid 1 2024-09-10T16:06:49.980 INFO:tasks.rados.rados.0.smithi165.stdout:8: writing smithi16534841-8 from 1776668 to 2117649 tid 2 2024-09-10T16:06:49.981 INFO:tasks.rados.rados.0.smithi165.stdout:9: write initial oid 9 2024-09-10T16:06:49.981 INFO:tasks.rados.rados.0.smithi165.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-10T16:06:49.985 INFO:tasks.rados.rados.0.smithi165.stdout:9: writing smithi16534841-9 from 534456 to 1252801 tid 1 2024-09-10T16:06:49.990 INFO:tasks.rados.rados.0.smithi165.stdout:9: writing smithi16534841-9 from 1951503 to 2134456 tid 2 2024-09-10T16:06:49.990 INFO:tasks.rados.rados.0.smithi165.stdout:10: write initial oid 10 2024-09-10T16:06:49.990 INFO:tasks.rados.rados.0.smithi165.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-10T16:06:49.997 INFO:tasks.rados.rados.0.smithi165.stdout:10: writing smithi16534841-10 from 551263 to 1344857 tid 1 2024-09-10T16:06:50.001 INFO:tasks.rados.rados.0.smithi165.stdout:10: writing smithi16534841-10 from 1893632 to 2151263 tid 2 2024-09-10T16:06:50.001 INFO:tasks.rados.rados.0.smithi165.stdout:11: write initial oid 11 2024-09-10T16:06:50.002 INFO:tasks.rados.rados.0.smithi165.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-10T16:06:50.007 INFO:tasks.rados.rados.0.smithi165.stdout:11: writing smithi16534841-11 from 568070 to 1036913 tid 1 2024-09-10T16:06:50.013 INFO:tasks.rados.rados.0.smithi165.stdout:11: writing smithi16534841-11 from 1552114 to 2168070 tid 2 2024-09-10T16:06:50.013 INFO:tasks.rados.rados.0.smithi165.stdout:12: write initial oid 12 2024-09-10T16:06:50.013 INFO:tasks.rados.rados.0.smithi165.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-10T16:06:50.019 INFO:tasks.rados.rados.0.smithi165.stdout:12: writing smithi16534841-12 from 584877 to 1128969 tid 1 2024-09-10T16:06:50.025 INFO:tasks.rados.rados.0.smithi165.stdout:12: writing smithi16534841-12 from 1610596 to 2172599 tid 2 2024-09-10T16:06:50.025 INFO:tasks.rados.rados.0.smithi165.stdout:13: write initial oid 13 2024-09-10T16:06:50.025 INFO:tasks.rados.rados.0.smithi165.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-10T16:06:50.032 INFO:tasks.rados.rados.0.smithi165.stdout:13: writing smithi16534841-13 from 601684 to 1221025 tid 1 2024-09-10T16:06:50.038 INFO:tasks.rados.rados.0.smithi165.stdout:13: writing smithi16534841-13 from 1669078 to 2201684 tid 2 2024-09-10T16:06:50.038 INFO:tasks.rados.rados.0.smithi165.stdout:14: write initial oid 14 2024-09-10T16:06:50.038 INFO:tasks.rados.rados.0.smithi165.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-10T16:06:50.044 INFO:tasks.rados.rados.0.smithi165.stdout:14: writing smithi16534841-14 from 618491 to 1313081 tid 1 2024-09-10T16:06:50.050 INFO:tasks.rados.rados.0.smithi165.stdout:14: writing smithi16534841-14 from 2011207 to 2218491 tid 2 2024-09-10T16:06:50.050 INFO:tasks.rados.rados.0.smithi165.stdout:15: write initial oid 15 2024-09-10T16:06:50.050 INFO:tasks.rados.rados.0.smithi165.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-10T16:06:50.057 INFO:tasks.rados.rados.0.smithi165.stdout:15: writing smithi16534841-15 from 635298 to 1405137 tid 1 2024-09-10T16:06:50.062 INFO:tasks.rados.rados.0.smithi165.stdout:15: writing smithi16534841-15 from 2069689 to 2235298 tid 2 2024-09-10T16:06:50.062 INFO:tasks.rados.rados.0.smithi165.stdout:16: write initial oid 16 2024-09-10T16:06:50.062 INFO:tasks.rados.rados.0.smithi165.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-10T16:06:50.068 INFO:tasks.rados.rados.0.smithi165.stdout:16: writing smithi16534841-16 from 652105 to 1097193 tid 1 2024-09-10T16:06:50.076 INFO:tasks.rados.rados.0.smithi165.stdout:16: writing smithi16534841-16 from 1728171 to 2181159 tid 2 2024-09-10T16:06:50.076 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.076 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 1 to smithi16534841-1 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 2 to smithi16534841-1 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 3 to smithi16534841-1 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: finishing write tid 4 to smithi16534841-1 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: oid 1 updating version 0 to 2 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: oid 1 version 2 is already newer than 1 2024-09-10T16:06:50.077 INFO:tasks.rados.rados.0.smithi165.stdout:1: oid 1 updating version 2 to 3 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 1 to smithi16534841-2 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 2 to smithi16534841-2 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 3 to smithi16534841-2 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: finishing write tid 4 to smithi16534841-2 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: oid 2 updating version 0 to 1 2024-09-10T16:06:50.078 INFO:tasks.rados.rados.0.smithi165.stdout:2: oid 2 updating version 1 to 3 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:2: oid 2 version 3 is already newer than 2 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 1 to smithi16534841-3 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 2 to smithi16534841-3 2024-09-10T16:06:50.079 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 3 to smithi16534841-3 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:3: finishing write tid 4 to smithi16534841-3 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:3: oid 3 updating version 0 to 3 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:3: oid 3 version 3 is already newer than 1 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:3: oid 3 version 3 is already newer than 2 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-10T16:06:50.080 INFO:tasks.rados.rados.0.smithi165.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 1 to smithi16534841-4 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 2 to smithi16534841-4 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 3 to smithi16534841-4 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: finishing write tid 4 to smithi16534841-4 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: oid 4 updating version 0 to 2 2024-09-10T16:06:50.081 INFO:tasks.rados.rados.0.smithi165.stdout:4: oid 4 updating version 2 to 3 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:4: oid 4 version 3 is already newer than 1 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 1 to smithi16534841-5 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 2 to smithi16534841-5 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:1: done (15 left) 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:2: done (14 left) 2024-09-10T16:06:50.082 INFO:tasks.rados.rados.0.smithi165.stdout:3: done (13 left) 2024-09-10T16:06:50.083 INFO:tasks.rados.rados.0.smithi165.stdout:4: done (12 left) 2024-09-10T16:06:50.083 INFO:tasks.rados.rados.0.smithi165.stdout:17: write initial oid 17 2024-09-10T16:06:50.083 INFO:tasks.rados.rados.0.smithi165.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-10T16:06:50.083 INFO:tasks.rados.rados.0.smithi165.stdout:17: writing smithi16534841-17 from 668912 to 1305602 tid 1 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:17: writing smithi16534841-17 from 1903006 to 2268912 tid 2 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:18: write initial oid 18 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 3 to smithi16534841-5 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:5: finishing write tid 4 to smithi16534841-5 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:5: oid 5 updating version 0 to 3 2024-09-10T16:06:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:5: oid 5 version 3 is already newer than 2 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:5: oid 5 version 3 is already newer than 1 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 1 to smithi16534841-6 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 2 to smithi16534841-6 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 3 to smithi16534841-6 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: finishing write tid 4 to smithi16534841-6 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: oid 6 updating version 0 to 5 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: oid 6 version 5 is already newer than 4 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: oid 6 updating version 5 to 6 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 6 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-10T16:06:50.088 INFO:tasks.rados.rados.0.smithi165.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 1 to smithi16534841-7 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 2 to smithi16534841-7 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 3 to smithi16534841-7 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: finishing write tid 4 to smithi16534841-7 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: oid 7 updating version 0 to 1 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: oid 7 updating version 1 to 3 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: oid 7 version 3 is already newer than 2 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 1 to smithi16534841-8 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 2 to smithi16534841-8 2024-09-10T16:06:50.089 INFO:tasks.rados.rados.0.smithi165.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-10T16:06:50.094 INFO:tasks.rados.rados.0.smithi165.stdout:18: writing smithi16534841-18 from 685719 to 1397658 tid 1 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:18: writing smithi16534841-18 from 1845135 to 2285719 tid 2 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:5: done (13 left) 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:6: done (12 left) 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:7: done (11 left) 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:19: write initial oid 19 2024-09-10T16:06:50.099 INFO:tasks.rados.rados.0.smithi165.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-10T16:06:50.107 INFO:tasks.rados.rados.0.smithi165.stdout:19: writing smithi16534841-19 from 702526 to 1489714 tid 1 2024-09-10T16:06:50.112 INFO:tasks.rados.rados.0.smithi165.stdout:19: writing smithi16534841-19 from 1903617 to 2302526 tid 2 2024-09-10T16:06:50.112 INFO:tasks.rados.rados.0.smithi165.stdout:20: write initial oid 20 2024-09-10T16:06:50.112 INFO:tasks.rados.rados.0.smithi165.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-10T16:06:50.117 INFO:tasks.rados.rados.0.smithi165.stdout:20: writing smithi16534841-20 from 719333 to 1181770 tid 1 2024-09-10T16:06:50.124 INFO:tasks.rados.rados.0.smithi165.stdout:20: writing smithi16534841-20 from 1962099 to 2319333 tid 2 2024-09-10T16:06:50.124 INFO:tasks.rados.rados.0.smithi165.stdout:21: write initial oid 21 2024-09-10T16:06:50.124 INFO:tasks.rados.rados.0.smithi165.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-10T16:06:50.131 INFO:tasks.rados.rados.0.smithi165.stdout:21: writing smithi16534841-21 from 736140 to 1273826 tid 1 2024-09-10T16:06:50.137 INFO:tasks.rados.rados.0.smithi165.stdout:21: writing smithi16534841-21 from 2020581 to 2336140 tid 2 2024-09-10T16:06:50.137 INFO:tasks.rados.rados.0.smithi165.stdout:22: write initial oid 22 2024-09-10T16:06:50.137 INFO:tasks.rados.rados.0.smithi165.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-10T16:06:50.144 INFO:tasks.rados.rados.0.smithi165.stdout:22: writing smithi16534841-22 from 752947 to 1365882 tid 1 2024-09-10T16:06:50.149 INFO:tasks.rados.rados.0.smithi165.stdout:22: writing smithi16534841-22 from 1962710 to 2352947 tid 2 2024-09-10T16:06:50.150 INFO:tasks.rados.rados.0.smithi165.stdout:23: write initial oid 23 2024-09-10T16:06:50.150 INFO:tasks.rados.rados.0.smithi165.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-10T16:06:50.157 INFO:tasks.rados.rados.0.smithi165.stdout:23: writing smithi16534841-23 from 769754 to 1457938 tid 1 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:23: writing smithi16534841-23 from 2021192 to 2369754 tid 2 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 3 to smithi16534841-8 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: finishing write tid 4 to smithi16534841-8 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: oid 8 updating version 0 to 4 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: oid 8 updating version 4 to 5 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: oid 8 updating version 5 to 6 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 1 to smithi16534841-9 2024-09-10T16:06:50.163 INFO:tasks.rados.rados.0.smithi165.stdout:8: done (15 left) 2024-09-10T16:06:50.164 INFO:tasks.rados.rados.0.smithi165.stdout:24: write initial oid 24 2024-09-10T16:06:50.164 INFO:tasks.rados.rados.0.smithi165.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-10T16:06:50.169 INFO:tasks.rados.rados.0.smithi165.stdout:24: writing smithi16534841-24 from 786561 to 1266347 tid 1 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:24: writing smithi16534841-24 from 1796027 to 2263691 tid 2 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 2 to smithi16534841-9 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 3 to smithi16534841-9 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: finishing write tid 4 to smithi16534841-9 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: oid 9 updating version 0 to 1 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: oid 9 updating version 1 to 3 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:9: oid 9 version 3 is already newer than 2 2024-09-10T16:06:50.181 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 1 to smithi16534841-10 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 2 to smithi16534841-10 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 3 to smithi16534841-10 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: finishing write tid 4 to smithi16534841-10 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: oid 10 updating version 0 to 1 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: oid 10 updating version 1 to 3 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: oid 10 version 3 is already newer than 2 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 3 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 1 to smithi16534841-11 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 2 to smithi16534841-11 2024-09-10T16:06:50.182 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 3 to smithi16534841-11 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:11: finishing write tid 4 to smithi16534841-11 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:11: oid 11 updating version 0 to 1 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:11: oid 11 updating version 1 to 3 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:11: oid 11 version 3 is already newer than 2 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 1 to smithi16534841-12 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 2 to smithi16534841-12 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 3 to smithi16534841-12 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:12: finishing write tid 4 to smithi16534841-12 2024-09-10T16:06:50.183 INFO:tasks.rados.rados.0.smithi165.stdout:12: oid 12 updating version 0 to 3 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:12: oid 12 version 3 is already newer than 2 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:12: oid 12 version 3 is already newer than 1 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 1 to smithi16534841-13 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 2 to smithi16534841-13 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 3 to smithi16534841-13 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: finishing write tid 4 to smithi16534841-13 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: oid 13 updating version 0 to 1 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: oid 13 updating version 1 to 2 2024-09-10T16:06:50.184 INFO:tasks.rados.rados.0.smithi165.stdout:13: oid 13 updating version 2 to 3 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 1 to smithi16534841-14 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 2 to smithi16534841-14 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 3 to smithi16534841-14 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: finishing write tid 4 to smithi16534841-14 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: oid 14 updating version 0 to 5 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: oid 14 version 5 is already newer than 4 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: oid 14 updating version 5 to 6 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-10T16:06:50.185 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 1 to smithi16534841-15 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 2 to smithi16534841-15 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 3 to smithi16534841-15 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: finishing write tid 4 to smithi16534841-15 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: oid 15 updating version 0 to 5 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: oid 15 updating version 5 to 6 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: oid 15 version 6 is already newer than 4 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 6 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 1 to smithi16534841-16 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:9: done (15 left) 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:10: done (14 left) 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:11: done (13 left) 2024-09-10T16:06:50.186 INFO:tasks.rados.rados.0.smithi165.stdout:12: done (12 left) 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:13: done (11 left) 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:14: done (10 left) 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:15: done (9 left) 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:25: write initial oid 25 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-10T16:06:50.187 INFO:tasks.rados.rados.0.smithi165.stdout:25: writing smithi16534841-25 from 403368 to 958403 tid 1 2024-09-10T16:06:50.188 INFO:tasks.rados.rados.0.smithi165.stdout:25: writing smithi16534841-25 from 1454509 to 2182184 tid 2 2024-09-10T16:06:50.188 INFO:tasks.rados.rados.0.smithi165.stdout:26: write initial oid 26 2024-09-10T16:06:50.188 INFO:tasks.rados.rados.0.smithi165.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-10T16:06:50.193 INFO:tasks.rados.rados.0.smithi165.stdout:26: writing smithi16534841-26 from 420175 to 1050459 tid 1 2024-09-10T16:06:50.200 INFO:tasks.rados.rados.0.smithi165.stdout:26: writing smithi16534841-26 from 1796638 to 2267971 tid 2 2024-09-10T16:06:50.200 INFO:tasks.rados.rados.0.smithi165.stdout:27: write initial oid 27 2024-09-10T16:06:50.200 INFO:tasks.rados.rados.0.smithi165.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-10T16:06:50.206 INFO:tasks.rados.rados.0.smithi165.stdout:27: writing smithi16534841-27 from 436982 to 1142515 tid 1 2024-09-10T16:06:50.213 INFO:tasks.rados.rados.0.smithi165.stdout:27: writing smithi16534841-27 from 1855120 to 2436982 tid 2 2024-09-10T16:06:50.213 INFO:tasks.rados.rados.0.smithi165.stdout:28: write initial oid 28 2024-09-10T16:06:50.213 INFO:tasks.rados.rados.0.smithi165.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-10T16:06:50.220 INFO:tasks.rados.rados.0.smithi165.stdout:28: writing smithi16534841-28 from 453789 to 1234571 tid 1 2024-09-10T16:06:50.226 INFO:tasks.rados.rados.0.smithi165.stdout:28: writing smithi16534841-28 from 1913602 to 2388604 tid 2 2024-09-10T16:06:50.227 INFO:tasks.rados.rados.0.smithi165.stdout:29: write initial oid 29 2024-09-10T16:06:50.227 INFO:tasks.rados.rados.0.smithi165.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-10T16:06:50.231 INFO:tasks.rados.rados.0.smithi165.stdout:29: writing smithi16534841-29 from 470596 to 926627 tid 1 2024-09-10T16:06:50.239 INFO:tasks.rados.rados.0.smithi165.stdout:29: writing smithi16534841-29 from 1572084 to 2190744 tid 2 2024-09-10T16:06:50.239 INFO:tasks.rados.rados.0.smithi165.stdout:30: write initial oid 30 2024-09-10T16:06:50.239 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 2 to smithi16534841-16 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 3 to smithi16534841-16 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: finishing write tid 4 to smithi16534841-16 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: oid 16 updating version 0 to 5 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: oid 16 version 5 is already newer than 4 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: oid 16 updating version 5 to 6 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 1 to smithi16534841-17 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 2 to smithi16534841-17 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 3 to smithi16534841-17 2024-09-10T16:06:50.240 INFO:tasks.rados.rados.0.smithi165.stdout:17: finishing write tid 4 to smithi16534841-17 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:17: oid 17 updating version 0 to 4 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:17: oid 17 updating version 4 to 6 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:17: oid 17 version 6 is already newer than 5 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 6 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 1 to smithi16534841-18 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 2 to smithi16534841-18 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 3 to smithi16534841-18 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:18: finishing write tid 4 to smithi16534841-18 2024-09-10T16:06:50.241 INFO:tasks.rados.rados.0.smithi165.stdout:18: oid 18 updating version 0 to 6 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:18: oid 18 version 6 is already newer than 5 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:18: oid 18 version 6 is already newer than 4 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 6 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 1 to smithi16534841-19 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 2 to smithi16534841-19 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 3 to smithi16534841-19 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: finishing write tid 4 to smithi16534841-19 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: oid 19 updating version 0 to 8 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: oid 19 version 8 is already newer than 7 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:19: oid 19 updating version 8 to 9 2024-09-10T16:06:50.242 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 9 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 1 to smithi16534841-20 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 2 to smithi16534841-20 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 3 to smithi16534841-20 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: finishing write tid 4 to smithi16534841-20 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: oid 20 updating version 0 to 4 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: oid 20 updating version 4 to 5 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: oid 20 updating version 5 to 6 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 6 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-10T16:06:50.243 INFO:tasks.rados.rados.0.smithi165.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-10T16:06:50.244 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 1 to smithi16534841-21 2024-09-10T16:06:50.244 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 2 to smithi16534841-21 2024-09-10T16:06:50.244 INFO:tasks.rados.rados.0.smithi165.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-10T16:06:50.244 INFO:tasks.rados.rados.0.smithi165.stdout:30: writing smithi16534841-30 from 487403 to 1018683 tid 1 2024-09-10T16:06:50.249 INFO:tasks.rados.rados.0.smithi165.stdout:30: writing smithi16534841-30 from 1514213 to 1992884 tid 2 2024-09-10T16:06:50.249 INFO:tasks.rados.rados.0.smithi165.stdout:16: done (14 left) 2024-09-10T16:06:50.249 INFO:tasks.rados.rados.0.smithi165.stdout:17: done (13 left) 2024-09-10T16:06:50.250 INFO:tasks.rados.rados.0.smithi165.stdout:18: done (12 left) 2024-09-10T16:06:50.250 INFO:tasks.rados.rados.0.smithi165.stdout:19: done (11 left) 2024-09-10T16:06:50.250 INFO:tasks.rados.rados.0.smithi165.stdout:20: done (10 left) 2024-09-10T16:06:50.250 INFO:tasks.rados.rados.0.smithi165.stdout:31: write initial oid 31 2024-09-10T16:06:50.250 INFO:tasks.rados.rados.0.smithi165.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-10T16:06:50.255 INFO:tasks.rados.rados.0.smithi165.stdout:31: writing smithi16534841-31 from 504210 to 1110739 tid 1 2024-09-10T16:06:50.261 INFO:tasks.rados.rados.0.smithi165.stdout:31: writing smithi16534841-31 from 1572695 to 2195024 tid 2 2024-09-10T16:06:50.261 INFO:tasks.rados.rados.0.smithi165.stdout:32: write initial oid 32 2024-09-10T16:06:50.261 INFO:tasks.rados.rados.0.smithi165.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-10T16:06:50.268 INFO:tasks.rados.rados.0.smithi165.stdout:32: writing smithi16534841-32 from 521017 to 1319148 tid 1 2024-09-10T16:06:50.273 INFO:tasks.rados.rados.0.smithi165.stdout:32: writing smithi16534841-32 from 1747530 to 2229870 tid 2 2024-09-10T16:06:50.274 INFO:tasks.rados.rados.0.smithi165.stdout:33: write initial oid 33 2024-09-10T16:06:50.274 INFO:tasks.rados.rados.0.smithi165.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-10T16:06:50.279 INFO:tasks.rados.rados.0.smithi165.stdout:33: writing smithi16534841-33 from 537824 to 1011204 tid 1 2024-09-10T16:06:50.287 INFO:tasks.rados.rados.0.smithi165.stdout:33: writing smithi16534841-33 from 1806012 to 2432010 tid 2 2024-09-10T16:06:50.287 INFO:tasks.rados.rados.0.smithi165.stdout:34: write initial oid 34 2024-09-10T16:06:50.287 INFO:tasks.rados.rados.0.smithi165.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-10T16:06:50.292 INFO:tasks.rados.rados.0.smithi165.stdout:34: writing smithi16534841-34 from 554631 to 1103260 tid 1 2024-09-10T16:06:50.299 INFO:tasks.rados.rados.0.smithi165.stdout:34: writing smithi16534841-34 from 1748141 to 2234150 tid 2 2024-09-10T16:06:50.299 INFO:tasks.rados.rados.0.smithi165.stdout:35: write initial oid 35 2024-09-10T16:06:50.299 INFO:tasks.rados.rados.0.smithi165.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-10T16:06:50.305 INFO:tasks.rados.rados.0.smithi165.stdout:35: writing smithi16534841-35 from 571438 to 1195316 tid 1 2024-09-10T16:06:50.313 INFO:tasks.rados.rados.0.smithi165.stdout:35: writing smithi16534841-35 from 1806623 to 2436290 tid 2 2024-09-10T16:06:50.313 INFO:tasks.rados.rados.0.smithi165.stdout:36: write initial oid 36 2024-09-10T16:06:50.313 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 3 to smithi16534841-21 2024-09-10T16:06:50.313 INFO:tasks.rados.rados.0.smithi165.stdout:21: finishing write tid 4 to smithi16534841-21 2024-09-10T16:06:50.313 INFO:tasks.rados.rados.0.smithi165.stdout:21: oid 21 updating version 0 to 8 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:21: oid 21 version 8 is already newer than 7 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:21: oid 21 updating version 8 to 9 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 1 to smithi16534841-22 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 2 to smithi16534841-22 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 3 to smithi16534841-22 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: finishing write tid 4 to smithi16534841-22 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: oid 22 updating version 0 to 5 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: oid 22 updating version 5 to 6 2024-09-10T16:06:50.314 INFO:tasks.rados.rados.0.smithi165.stdout:22: oid 22 version 6 is already newer than 4 2024-09-10T16:06:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:06:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:06:50.315 INFO:tasks.rados.rados.0.smithi165.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-10T16:06:50.319 INFO:tasks.rados.rados.0.smithi165.stdout:36: writing smithi16534841-36 from 588245 to 1287372 tid 1 2024-09-10T16:06:50.325 INFO:tasks.rados.rados.0.smithi165.stdout:36: writing smithi16534841-36 from 1865105 to 2354783 tid 2 2024-09-10T16:06:50.325 INFO:tasks.rados.rados.0.smithi165.stdout:21: done (15 left) 2024-09-10T16:06:50.325 INFO:tasks.rados.rados.0.smithi165.stdout:22: done (14 left) 2024-09-10T16:06:50.325 INFO:tasks.rados.rados.0.smithi165.stdout:37: write initial oid 37 2024-09-10T16:06:50.325 INFO:tasks.rados.rados.0.smithi165.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-10T16:06:50.332 INFO:tasks.rados.rados.0.smithi165.stdout:37: writing smithi16534841-37 from 605052 to 1379428 tid 1 2024-09-10T16:06:50.338 INFO:tasks.rados.rados.0.smithi165.stdout:37: writing smithi16534841-37 from 1923587 to 2556923 tid 2 2024-09-10T16:06:50.339 INFO:tasks.rados.rados.0.smithi165.stdout:38: write initial oid 38 2024-09-10T16:06:50.339 INFO:tasks.rados.rados.0.smithi165.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-10T16:06:50.344 INFO:tasks.rados.rados.0.smithi165.stdout:38: writing smithi16534841-38 from 621859 to 1071484 tid 1 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:38: writing smithi16534841-38 from 1865716 to 2621859 tid 2 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 1 to smithi16534841-23 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 2 to smithi16534841-23 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 3 to smithi16534841-23 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: finishing write tid 4 to smithi16534841-23 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: oid 23 updating version 0 to 9 2024-09-10T16:06:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:23: oid 23 version 9 is already newer than 8 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:23: oid 23 version 9 is already newer than 7 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 9 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 1 to smithi16534841-24 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:23: done (15 left) 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:39: write initial oid 39 2024-09-10T16:06:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-10T16:06:50.359 INFO:tasks.rados.rados.0.smithi165.stdout:39: writing smithi16534841-39 from 638666 to 1163540 tid 1 2024-09-10T16:06:50.366 INFO:tasks.rados.rados.0.smithi165.stdout:39: writing smithi16534841-39 from 1924198 to 2561203 tid 2 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 2 to smithi16534841-24 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 3 to smithi16534841-24 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: finishing write tid 4 to smithi16534841-24 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: oid 24 updating version 0 to 7 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: oid 24 updating version 7 to 9 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: oid 24 version 9 is already newer than 8 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 9 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 1 to smithi16534841-25 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:24: done (15 left) 2024-09-10T16:06:50.367 INFO:tasks.rados.rados.0.smithi165.stdout:40: write initial oid 40 2024-09-10T16:06:50.368 INFO:tasks.rados.rados.0.smithi165.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-10T16:06:50.373 INFO:tasks.rados.rados.0.smithi165.stdout:40: writing smithi16534841-40 from 655473 to 1371949 tid 1 2024-09-10T16:06:50.381 INFO:tasks.rados.rados.0.smithi165.stdout:40: writing smithi16534841-40 from 2099033 to 2655473 tid 2 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 2 to smithi16534841-25 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 3 to smithi16534841-25 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: finishing write tid 4 to smithi16534841-25 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: oid 25 updating version 0 to 10 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: oid 25 updating version 10 to 11 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: oid 25 updating version 11 to 12 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 12 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 1 to smithi16534841-26 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:25: done (15 left) 2024-09-10T16:06:50.382 INFO:tasks.rados.rados.0.smithi165.stdout:41: write initial oid 41 2024-09-10T16:06:50.384 INFO:tasks.rados.rados.0.smithi165.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-10T16:06:50.389 INFO:tasks.rados.rados.0.smithi165.stdout:41: writing smithi16534841-41 from 672280 to 1464005 tid 1 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:41: writing smithi16534841-41 from 2157515 to 2672280 tid 2 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 2 to smithi16534841-26 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 3 to smithi16534841-26 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: finishing write tid 4 to smithi16534841-26 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: oid 26 updating version 0 to 12 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: oid 26 version 12 is already newer than 11 2024-09-10T16:06:50.396 INFO:tasks.rados.rados.0.smithi165.stdout:26: oid 26 version 12 is already newer than 10 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 12 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 1 to smithi16534841-27 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:26: done (15 left) 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:42: write initial oid 42 2024-09-10T16:06:50.397 INFO:tasks.rados.rados.0.smithi165.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-10T16:06:50.402 INFO:tasks.rados.rados.0.smithi165.stdout:42: writing smithi16534841-42 from 689087 to 1156061 tid 1 2024-09-10T16:06:50.409 INFO:tasks.rados.rados.0.smithi165.stdout:42: writing smithi16534841-42 from 1699644 to 2483976 tid 2 2024-09-10T16:06:50.409 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 2 to smithi16534841-27 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 3 to smithi16534841-27 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: finishing write tid 4 to smithi16534841-27 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: oid 27 updating version 0 to 10 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: oid 27 updating version 10 to 12 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: oid 27 version 12 is already newer than 11 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 12 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 1 to smithi16534841-28 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:27: done (15 left) 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:43: write initial oid 43 2024-09-10T16:06:50.410 INFO:tasks.rados.rados.0.smithi165.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-10T16:06:50.416 INFO:tasks.rados.rados.0.smithi165.stdout:43: writing smithi16534841-43 from 705894 to 1248117 tid 1 2024-09-10T16:06:50.422 INFO:tasks.rados.rados.0.smithi165.stdout:43: writing smithi16534841-43 from 1758126 to 2402469 tid 2 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 2 to smithi16534841-28 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 3 to smithi16534841-28 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: finishing write tid 4 to smithi16534841-28 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: oid 28 updating version 0 to 3 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: oid 28 version 3 is already newer than 1 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: oid 28 version 3 is already newer than 2 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 3 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 1 to smithi16534841-29 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:28: done (15 left) 2024-09-10T16:06:50.423 INFO:tasks.rados.rados.0.smithi165.stdout:44: write initial oid 44 2024-09-10T16:06:50.424 INFO:tasks.rados.rados.0.smithi165.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-10T16:06:50.429 INFO:tasks.rados.rados.0.smithi165.stdout:44: writing smithi16534841-44 from 722701 to 1340173 tid 1 2024-09-10T16:06:50.437 INFO:tasks.rados.rados.0.smithi165.stdout:44: writing smithi16534841-44 from 1816608 to 2604609 tid 2 2024-09-10T16:06:50.437 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 2 to smithi16534841-29 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 3 to smithi16534841-29 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: finishing write tid 4 to smithi16534841-29 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: oid 29 updating version 0 to 15 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: oid 29 version 15 is already newer than 14 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: oid 29 version 15 is already newer than 13 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 15 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 1 to smithi16534841-30 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 2 to smithi16534841-30 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 3 to smithi16534841-30 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:30: finishing write tid 4 to smithi16534841-30 2024-09-10T16:06:50.438 INFO:tasks.rados.rados.0.smithi165.stdout:30: oid 30 updating version 0 to 13 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:30: oid 30 updating version 13 to 14 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:30: oid 30 updating version 14 to 15 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 15 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 1 to smithi16534841-31 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 2 to smithi16534841-31 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 3 to smithi16534841-31 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: finishing write tid 4 to smithi16534841-31 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: oid 31 updating version 0 to 16 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: oid 31 updating version 16 to 18 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: oid 31 version 18 is already newer than 17 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 18 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-10T16:06:50.439 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 1 to smithi16534841-32 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 2 to smithi16534841-32 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 3 to smithi16534841-32 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: finishing write tid 4 to smithi16534841-32 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: oid 32 updating version 0 to 8 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: oid 32 version 8 is already newer than 7 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: oid 32 updating version 8 to 9 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 9 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 1 to smithi16534841-33 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:29: done (15 left) 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:30: done (14 left) 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:31: done (13 left) 2024-09-10T16:06:50.440 INFO:tasks.rados.rados.0.smithi165.stdout:32: done (12 left) 2024-09-10T16:06:50.441 INFO:tasks.rados.rados.0.smithi165.stdout:45: write initial oid 45 2024-09-10T16:06:50.441 INFO:tasks.rados.rados.0.smithi165.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-10T16:06:50.444 INFO:tasks.rados.rados.0.smithi165.stdout:45: writing smithi16534841-45 from 739508 to 1432229 tid 1 2024-09-10T16:06:50.450 INFO:tasks.rados.rados.0.smithi165.stdout:45: writing smithi16534841-45 from 1875090 to 2523102 tid 2 2024-09-10T16:06:50.451 INFO:tasks.rados.rados.0.smithi165.stdout:46: write initial oid 46 2024-09-10T16:06:50.451 INFO:tasks.rados.rados.0.smithi165.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-10T16:06:50.459 INFO:tasks.rados.rados.0.smithi165.stdout:46: writing smithi16534841-46 from 756315 to 1524285 tid 1 2024-09-10T16:06:50.466 INFO:tasks.rados.rados.0.smithi165.stdout:46: writing smithi16534841-46 from 1933572 to 2725242 tid 2 2024-09-10T16:06:50.466 INFO:tasks.rados.rados.0.smithi165.stdout:47: write initial oid 47 2024-09-10T16:06:50.466 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 2 to smithi16534841-33 2024-09-10T16:06:50.466 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 3 to smithi16534841-33 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:33: finishing write tid 4 to smithi16534841-33 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:33: oid 33 updating version 0 to 13 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:33: oid 33 updating version 13 to 14 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:33: oid 33 updating version 14 to 15 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 15 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 1 to smithi16534841-34 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 2 to smithi16534841-34 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 3 to smithi16534841-34 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: finishing write tid 4 to smithi16534841-34 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: oid 34 updating version 0 to 3 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: oid 34 version 3 is already newer than 2 2024-09-10T16:06:50.467 INFO:tasks.rados.rados.0.smithi165.stdout:34: oid 34 version 3 is already newer than 1 2024-09-10T16:06:50.468 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 3 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-10T16:06:50.468 INFO:tasks.rados.rados.0.smithi165.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-10T16:06:50.468 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 1 to smithi16534841-35 2024-09-10T16:06:50.468 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 2 to smithi16534841-35 2024-09-10T16:06:50.468 INFO:tasks.rados.rados.0.smithi165.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-10T16:06:50.473 INFO:tasks.rados.rados.0.smithi165.stdout:47: writing smithi16534841-47 from 773122 to 1332694 tid 1 2024-09-10T16:06:50.480 INFO:tasks.rados.rados.0.smithi165.stdout:47: writing smithi16534841-47 from 1992054 to 2643735 tid 2 2024-09-10T16:06:50.480 INFO:tasks.rados.rados.0.smithi165.stdout:33: done (14 left) 2024-09-10T16:06:50.481 INFO:tasks.rados.rados.0.smithi165.stdout:34: done (13 left) 2024-09-10T16:06:50.481 INFO:tasks.rados.rados.0.smithi165.stdout:48: write initial oid 48 2024-09-10T16:06:50.481 INFO:tasks.rados.rados.0.smithi165.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-10T16:06:50.488 INFO:tasks.rados.rados.0.smithi165.stdout:48: writing smithi16534841-48 from 789929 to 1424750 tid 1 2024-09-10T16:06:50.496 INFO:tasks.rados.rados.0.smithi165.stdout:48: writing smithi16534841-48 from 2050536 to 2789929 tid 2 2024-09-10T16:06:50.496 INFO:tasks.rados.rados.0.smithi165.stdout:49: write initial oid 49 2024-09-10T16:06:50.496 INFO:tasks.rados.rados.0.smithi165.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-10T16:06:50.502 INFO:tasks.rados.rados.0.smithi165.stdout:49: writing smithi16534841-49 from 406736 to 1116806 tid 1 2024-09-10T16:06:50.508 INFO:tasks.rados.rados.0.smithi165.stdout:49: writing smithi16534841-49 from 1709018 to 2364368 tid 2 2024-09-10T16:06:50.509 INFO:tasks.rados.rados.0.smithi165.stdout:50: write initial oid 50 2024-09-10T16:06:50.509 INFO:tasks.rados.rados.0.smithi165.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-10T16:06:50.515 INFO:tasks.rados.rados.0.smithi165.stdout:50: writing smithi16534841-50 from 423543 to 1208862 tid 1 2024-09-10T16:06:50.525 INFO:tasks.rados.rados.0.smithi165.stdout:50: writing smithi16534841-50 from 1767500 to 2566508 tid 2 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 3 to smithi16534841-35 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: finishing write tid 4 to smithi16534841-35 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: oid 35 updating version 0 to 7 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: oid 35 updating version 7 to 9 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: oid 35 version 9 is already newer than 8 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 1 to smithi16534841-36 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:35: done (15 left) 2024-09-10T16:06:50.526 INFO:tasks.rados.rados.0.smithi165.stdout:51: read oid 1 snap -1 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:51: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 16 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 2 to smithi16534841-36 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 3 to smithi16534841-36 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: finishing write tid 4 to smithi16534841-36 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: oid 36 updating version 0 to 3 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: oid 36 version 3 is already newer than 1 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: oid 36 version 3 is already newer than 2 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 3 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-10T16:06:50.527 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 1 to smithi16534841-37 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 2 to smithi16534841-37 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 3 to smithi16534841-37 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: finishing write tid 4 to smithi16534841-37 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: oid 37 updating version 0 to 4 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: oid 37 updating version 4 to 6 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: oid 37 version 6 is already newer than 5 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 6 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 1 to smithi16534841-38 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:36: done (15 left) 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:37: done (14 left) 2024-09-10T16:06:50.528 INFO:tasks.rados.rados.0.smithi165.stdout:52: delete oid 23 current snap is 0 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 2 to smithi16534841-38 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 3 to smithi16534841-38 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: finishing write tid 4 to smithi16534841-38 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: oid 38 updating version 0 to 5 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: oid 38 updating version 5 to 6 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: oid 38 version 6 is already newer than 4 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 6 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 1 to smithi16534841-39 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 2 to smithi16534841-39 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 3 to smithi16534841-39 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: finishing write tid 4 to smithi16534841-39 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: oid 39 updating version 0 to 11 2024-09-10T16:06:50.529 INFO:tasks.rados.rados.0.smithi165.stdout:39: oid 39 version 11 is already newer than 10 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:39: oid 39 updating version 11 to 12 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 12 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 1 to smithi16534841-40 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 2 to smithi16534841-40 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 3 to smithi16534841-40 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: finishing write tid 4 to smithi16534841-40 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: oid 40 updating version 0 to 8 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: oid 40 version 8 is already newer than 7 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: oid 40 updating version 8 to 9 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-10T16:06:50.530 INFO:tasks.rados.rados.0.smithi165.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 1 to smithi16534841-41 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 2 to smithi16534841-41 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 3 to smithi16534841-41 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: finishing write tid 4 to smithi16534841-41 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: oid 41 updating version 0 to 14 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: oid 41 version 14 is already newer than 13 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:41: oid 41 updating version 14 to 15 2024-09-10T16:06:50.531 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 15 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 1 to smithi16534841-42 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 2 to smithi16534841-42 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 3 to smithi16534841-42 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: finishing write tid 4 to smithi16534841-42 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: oid 42 updating version 0 to 5 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: oid 42 version 5 is already newer than 4 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: oid 42 updating version 5 to 6 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 6 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 1 to smithi16534841-43 2024-09-10T16:06:50.532 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 2 to smithi16534841-43 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 3 to smithi16534841-43 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: finishing write tid 4 to smithi16534841-43 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: oid 43 updating version 0 to 1 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: oid 43 updating version 1 to 2 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: oid 43 updating version 2 to 3 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 3 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 1 to smithi16534841-44 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 2 to smithi16534841-44 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 3 to smithi16534841-44 2024-09-10T16:06:50.533 INFO:tasks.rados.rados.0.smithi165.stdout:44: finishing write tid 4 to smithi16534841-44 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:44: oid 44 updating version 0 to 2 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:44: oid 44 version 2 is already newer than 1 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:44: oid 44 updating version 2 to 3 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 3 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 1 to smithi16534841-45 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 2 to smithi16534841-45 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 3 to smithi16534841-45 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: finishing write tid 4 to smithi16534841-45 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: oid 45 updating version 0 to 5 2024-09-10T16:06:50.534 INFO:tasks.rados.rados.0.smithi165.stdout:45: oid 45 updating version 5 to 6 2024-09-10T16:06:50.535 INFO:tasks.rados.rados.0.smithi165.stdout:45: oid 45 version 6 is already newer than 4 2024-09-10T16:06:50.535 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-10T16:06:50.535 INFO:tasks.rados.rados.0.smithi165.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-10T16:06:50.535 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 1 to smithi16534841-46 2024-09-10T16:06:50.535 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 2 to smithi16534841-46 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 3 to smithi16534841-46 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: finishing write tid 4 to smithi16534841-46 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: oid 46 updating version 0 to 5 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: oid 46 version 5 is already newer than 4 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: oid 46 updating version 5 to 6 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 6 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-10T16:06:50.536 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 1 to smithi16534841-47 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 2 to smithi16534841-47 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 3 to smithi16534841-47 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: finishing write tid 4 to smithi16534841-47 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: oid 47 updating version 0 to 17 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: oid 47 updating version 17 to 18 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: oid 47 version 18 is already newer than 16 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 18 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 1 to smithi16534841-48 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 2 to smithi16534841-48 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 3 to smithi16534841-48 2024-09-10T16:06:50.537 INFO:tasks.rados.rados.0.smithi165.stdout:48: finishing write tid 4 to smithi16534841-48 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 updating version 0 to 19 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 updating version 19 to 20 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:48: oid 48 updating version 20 to 21 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 21 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 1 to smithi16534841-49 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 2 to smithi16534841-49 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 3 to smithi16534841-49 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: finishing write tid 4 to smithi16534841-49 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 updating version 0 to 9 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 version 9 is already newer than 7 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: oid 49 version 9 is already newer than 8 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-10T16:06:50.538 INFO:tasks.rados.rados.0.smithi165.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-10T16:06:50.539 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 1 to smithi16534841-50 2024-09-10T16:06:50.539 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 2 to smithi16534841-50 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 3 to smithi16534841-50 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:38: done (14 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:39: done (13 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:40: done (12 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:41: done (11 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:42: done (10 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:43: done (9 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:44: done (8 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:45: done (7 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:46: done (6 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:47: done (5 left) 2024-09-10T16:06:50.540 INFO:tasks.rados.rados.0.smithi165.stdout:48: done (4 left) 2024-09-10T16:06:50.541 INFO:tasks.rados.rados.0.smithi165.stdout:49: done (3 left) 2024-09-10T16:06:50.541 INFO:tasks.rados.rados.0.smithi165.stdout:52: done (2 left) 2024-09-10T16:06:50.541 INFO:tasks.rados.rados.0.smithi165.stdout:53: snap_create 2024-09-10T16:06:50.541 INFO:tasks.rados.rados.0.smithi165.stdout:51: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-10T16:06:50.541 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:50.573 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:50: finishing write tid 4 to smithi16534841-50 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 updating version 0 to 9 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 version 9 is already newer than 8 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:50: oid 50 version 9 is already newer than 7 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 9 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-10T16:06:50.595 INFO:tasks.rados.rados.0.smithi165.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-10T16:06:50.886 INFO:tasks.rados.rados.0.smithi165.stdout:50: done (2 left) 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:51: done (1 left) 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:53: done (0 left) 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:54: copy_from oid 48 from oid 7 current snap is 1 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:55: rollback oid 36 current snap is 1 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 0 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:56: read oid 12 snap -1 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:56: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:57: read oid 4 snap 0 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:57: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:58: write oid 2 current snap is 1 2024-09-10T16:06:50.887 INFO:tasks.rados.rados.0.smithi165.stdout:58: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-10T16:06:50.892 INFO:tasks.rados.rados.0.smithi165.stdout:58: writing smithi16534841-2 from 457157 to 992974 tid 1 2024-09-10T16:06:50.899 INFO:tasks.rados.rados.0.smithi165.stdout:58: writing smithi16534841-2 from 1768111 to 2170788 tid 2 2024-09-10T16:06:50.899 INFO:tasks.rados.rados.0.smithi165.stdout:59: snap_create 2024-09-10T16:06:50.899 INFO:tasks.rados.rados.0.smithi165.stdout:55: finishing rollback tid 0 to smithi16534841-36 2024-09-10T16:06:50.899 INFO:tasks.rados.rados.0.smithi165.stdout:56: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:06:50.964 INFO:tasks.rados.rados.0.smithi165.stdout:55: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:06:50.964 INFO:tasks.rados.rados.0.smithi165.stdout:55: finishing rollback tid 2 to smithi16534841-36 2024-09-10T16:06:50.965 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 10 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-10T16:06:50.965 INFO:tasks.rados.rados.0.smithi165.stdout:57: expect (ObjNum 3 snap 0 seq_num 3) 2024-09-10T16:06:50.965 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:51.029 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing write tid 1 to smithi16534841-2 2024-09-10T16:06:51.029 INFO:tasks.rados.rados.0.smithi165.stdout:54: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing write tid 2 to smithi16534841-2 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing write tid 3 to smithi16534841-2 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: finishing write tid 4 to smithi16534841-2 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: oid 2 updating version 0 to 22 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: oid 2 version 22 is already newer than 21 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: oid 2 version 22 is already newer than 20 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 22 (ObjNum 51 snap 1 seq_num 51) dirty exists 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:58: left oid 2 (ObjNum 51 snap 1 seq_num 51) 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:54: finishing copy_from to smithi16534841-48 2024-09-10T16:06:51.030 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:54: done (5 left) 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:55: done (4 left) 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:56: done (3 left) 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:57: done (2 left) 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:58: done (1 left) 2024-09-10T16:06:51.894 INFO:tasks.rados.rados.0.smithi165.stdout:59: done (0 left) 2024-09-10T16:06:51.895 INFO:tasks.rados.rados.0.smithi165.stdout:60: delete oid 2 current snap is 2 2024-09-10T16:06:51.899 INFO:tasks.rados.rados.0.smithi165.stdout:60: done (0 left) 2024-09-10T16:06:51.899 INFO:tasks.rados.rados.0.smithi165.stdout:61: copy_from oid 44 from oid 48 current snap is 2 2024-09-10T16:06:51.899 INFO:tasks.rados.rados.0.smithi165.stdout:62: copy_from oid 11 from oid 3 current snap is 2 2024-09-10T16:06:51.899 INFO:tasks.rados.rados.0.smithi165.stdout:63: delete oid 37 current snap is 2 2024-09-10T16:06:51.903 INFO:tasks.rados.rados.0.smithi165.stdout:63: done (2 left) 2024-09-10T16:06:51.904 INFO:tasks.rados.rados.0.smithi165.stdout:64: snap_remove snap 1 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:64: done (2 left) 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:65: snap_create 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:61: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:62: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:61: finishing copy_from to smithi16534841-44 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:62: finishing copy_from to smithi16534841-11 2024-09-10T16:06:52.899 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-10T16:06:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:61: done (2 left) 2024-09-10T16:06:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:62: done (1 left) 2024-09-10T16:06:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:65: done (0 left) 2024-09-10T16:06:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:66: write (excl) oid 19 current snap is 3 2024-09-10T16:06:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:66: seq_num 54 ranges {507578=477917,1543557=550004,2833665=73913} 2024-09-10T16:06:53.909 INFO:tasks.rados.rados.0.smithi165.stdout:66: writing smithi16534841-19 from 507578 to 985495 tid 1 2024-09-10T16:06:53.915 INFO:tasks.rados.rados.0.smithi165.stdout:66: writing smithi16534841-19 from 1543557 to 2093561 tid 2 2024-09-10T16:06:53.920 INFO:tasks.rados.rados.0.smithi165.stdout:66: writing smithi16534841-19 from 2833665 to 2907578 tid 3 2024-09-10T16:06:53.920 INFO:tasks.rados.rados.0.smithi165.stdout:67: read oid 13 snap -1 2024-09-10T16:06:53.920 INFO:tasks.rados.rados.0.smithi165.stdout:67: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-10T16:06:53.920 INFO:tasks.rados.rados.0.smithi165.stdout:68: read oid 18 snap -1 2024-09-10T16:06:53.920 INFO:tasks.rados.rados.0.smithi165.stdout:68: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:69: read oid 28 snap -1 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:69: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:70: delete oid 15 current snap is 3 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 1 to smithi16534841-19 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 2 to smithi16534841-19 2024-09-10T16:06:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 3 to smithi16534841-19 2024-09-10T16:06:53.924 INFO:tasks.rados.rados.0.smithi165.stdout:68: expect (ObjNum 17 snap 0 seq_num 17) 2024-09-10T16:06:53.924 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:53.954 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1673058 2024-09-10T16:06:53.989 INFO:tasks.rados.rados.0.smithi165.stdout:69: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:06:53.990 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 48669 2024-09-10T16:06:54.057 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 4 to smithi16534841-19 2024-09-10T16:06:54.057 INFO:tasks.rados.rados.0.smithi165.stdout:66: finishing write tid 5 to smithi16534841-19 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 19 updating version 0 to 28 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 19 version 28 is already newer than 26 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 19 version 28 is already newer than 25 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:66: oid 19 version 28 is already newer than 27 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 28 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:66: left oid 19 (ObjNum 54 snap 3 seq_num 54) 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:67: expect (ObjNum 12 snap 0 seq_num 12) 2024-09-10T16:06:54.058 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:54.081 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:54.129 INFO:tasks.rados.rados.0.smithi165.stdout:66: done (4 left) 2024-09-10T16:06:54.129 INFO:tasks.rados.rados.0.smithi165.stdout:67: done (3 left) 2024-09-10T16:06:54.129 INFO:tasks.rados.rados.0.smithi165.stdout:68: done (2 left) 2024-09-10T16:06:54.129 INFO:tasks.rados.rados.0.smithi165.stdout:69: done (1 left) 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:70: done (0 left) 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:71: read oid 1 snap -1 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:71: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:72: rollback oid 20 current snap is 3 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 2 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:73: read oid 49 snap -1 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:73: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:74: read oid 10 snap -1 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:74: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:75: copy_from oid 26 from oid 21 current snap is 3 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:76: rollback oid 5 current snap is 3 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 2 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:77: write oid 33 current snap is 3 2024-09-10T16:06:54.130 INFO:tasks.rados.rados.0.smithi165.stdout:77: seq_num 56 ranges {541192=628415,1660521=553673,2888511=52681} 2024-09-10T16:06:54.135 INFO:tasks.rados.rados.0.smithi165.stdout:77: writing smithi16534841-33 from 541192 to 1169607 tid 1 2024-09-10T16:06:54.141 INFO:tasks.rados.rados.0.smithi165.stdout:77: writing smithi16534841-33 from 1660521 to 2214194 tid 2 2024-09-10T16:06:54.145 INFO:tasks.rados.rados.0.smithi165.stdout:77: writing smithi16534841-33 from 2888511 to 2941192 tid 3 2024-09-10T16:06:54.145 INFO:tasks.rados.rados.0.smithi165.stdout:78: read oid 35 snap -1 2024-09-10T16:06:54.145 INFO:tasks.rados.rados.0.smithi165.stdout:78: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-10T16:06:54.145 INFO:tasks.rados.rados.0.smithi165.stdout:79: snap_remove snap 0 2024-09-10T16:06:54.914 INFO:tasks.rados.rados.0.smithi165.stdout:71: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-10T16:06:54.984 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:06:54.984 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:06:54.984 INFO:tasks.rados.rados.0.smithi165.stdout:72: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 19 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing write tid 1 to smithi16534841-33 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing write tid 2 to smithi16534841-33 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:71: done (8 left) 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:72: done (7 left) 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:79: done (6 left) 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:80: write (excl) oid 6 current snap is 3 2024-09-10T16:06:54.985 INFO:tasks.rados.rados.0.smithi165.stdout:80: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-10T16:06:54.990 INFO:tasks.rados.rados.0.smithi165.stdout:80: writing smithi16534841-6 from 557999 to 1261663 tid 1 2024-09-10T16:06:54.995 INFO:tasks.rados.rados.0.smithi165.stdout:80: writing smithi16534841-6 from 1719003 to 2132687 tid 2 2024-09-10T16:06:55.000 INFO:tasks.rados.rados.0.smithi165.stdout:80: writing smithi16534841-6 from 2632287 to 2957999 tid 3 2024-09-10T16:06:55.000 INFO:tasks.rados.rados.0.smithi165.stdout:81: write oid 30 current snap is 3 2024-09-10T16:06:55.000 INFO:tasks.rados.rados.0.smithi165.stdout:81: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-10T16:06:55.007 INFO:tasks.rados.rados.0.smithi165.stdout:81: writing smithi16534841-30 from 574806 to 1353719 tid 1 2024-09-10T16:06:55.014 INFO:tasks.rados.rados.0.smithi165.stdout:81: writing smithi16534841-30 from 2061132 to 2618474 tid 2 2024-09-10T16:06:55.014 INFO:tasks.rados.rados.0.smithi165.stdout:82: read oid 31 snap -1 2024-09-10T16:06:55.014 INFO:tasks.rados.rados.0.smithi165.stdout:82: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-10T16:06:55.014 INFO:tasks.rados.rados.0.smithi165.stdout:83: read oid 23 snap -1 2024-09-10T16:06:55.014 INFO:tasks.rados.rados.0.smithi165.stdout:83: expect deleted 2024-09-10T16:06:55.015 INFO:tasks.rados.rados.0.smithi165.stdout:84: write (excl) oid 44 current snap is 3 2024-09-10T16:06:55.015 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing write tid 3 to smithi16534841-33 2024-09-10T16:06:55.015 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing write tid 4 to smithi16534841-33 2024-09-10T16:06:55.015 INFO:tasks.rados.rados.0.smithi165.stdout:84: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-10T16:06:55.020 INFO:tasks.rados.rados.0.smithi165.stdout:84: writing smithi16534841-44 from 591613 to 1045775 tid 1 2024-09-10T16:06:55.026 INFO:tasks.rados.rados.0.smithi165.stdout:84: writing smithi16534841-44 from 1719614 to 2136967 tid 2 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:84: writing smithi16534841-44 from 2570780 to 2991613 tid 3 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:85: copy_from oid 43 from oid 8 current snap is 3 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:77: finishing write tid 5 to smithi16534841-33 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:77: oid 33 updating version 0 to 21 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:77: oid 33 updating version 21 to 23 2024-09-10T16:06:55.031 INFO:tasks.rados.rados.0.smithi165.stdout:77: oid 33 updating version 23 to 24 2024-09-10T16:06:55.032 INFO:tasks.rados.rados.0.smithi165.stdout:77: oid 33 version 24 is already newer than 22 2024-09-10T16:06:55.032 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 24 (ObjNum 56 snap 3 seq_num 56) dirty exists 2024-09-10T16:06:55.032 INFO:tasks.rados.rados.0.smithi165.stdout:77: left oid 33 (ObjNum 56 snap 3 seq_num 56) 2024-09-10T16:06:55.032 INFO:tasks.rados.rados.0.smithi165.stdout:78: expect (ObjNum 34 snap 0 seq_num 34) 2024-09-10T16:06:55.044 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2424475 2024-09-10T16:06:55.061 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:76: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:06:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:76: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:06:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:76: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:06:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 13 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-10T16:06:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:73: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-10T16:06:55.139 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 190512 2024-09-10T16:06:55.168 INFO:tasks.rados.rados.0.smithi165.stdout:74: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-10T16:06:55.191 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:55.215 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:75: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:06:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:75: finishing copy_from to smithi16534841-26 2024-09-10T16:06:55.238 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 9 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: finishing write tid 1 to smithi16534841-6 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: finishing write tid 2 to smithi16534841-6 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: finishing write tid 3 to smithi16534841-6 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: finishing write tid 4 to smithi16534841-6 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: finishing write tid 5 to smithi16534841-6 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: oid 6 updating version 0 to 18 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: oid 6 updating version 18 to 19 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: oid 6 updating version 19 to 20 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: oid 6 version 20 is already newer than 17 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 20 (ObjNum 57 snap 3 seq_num 57) dirty exists 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:80: left oid 6 (ObjNum 57 snap 3 seq_num 57) 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing write tid 1 to smithi16534841-30 2024-09-10T16:06:55.239 INFO:tasks.rados.rados.0.smithi165.stdout:73: done (11 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:74: done (10 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:75: done (9 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:76: done (8 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:77: done (7 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:78: done (6 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:80: done (5 left) 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:86: snap_create 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing write tid 2 to smithi16534841-30 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing write tid 3 to smithi16534841-30 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: finishing write tid 4 to smithi16534841-30 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: oid 30 updating version 0 to 38 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: oid 30 updating version 38 to 39 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:81: oid 30 updating version 39 to 40 2024-09-10T16:06:55.240 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 40 (ObjNum 58 snap 3 seq_num 58) dirty exists 2024-09-10T16:06:55.241 INFO:tasks.rados.rados.0.smithi165.stdout:81: left oid 30 (ObjNum 58 snap 3 seq_num 58) 2024-09-10T16:06:55.241 INFO:tasks.rados.rados.0.smithi165.stdout:82: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-10T16:06:55.265 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:55.295 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1149500 2024-09-10T16:06:55.313 INFO:tasks.rados.rados.0.smithi165.stdout:84: finishing write tid 1 to smithi16534841-44 2024-09-10T16:06:55.313 INFO:tasks.rados.rados.0.smithi165.stdout:84: finishing write tid 2 to smithi16534841-44 2024-09-10T16:06:55.313 INFO:tasks.rados.rados.0.smithi165.stdout:84: finishing write tid 3 to smithi16534841-44 2024-09-10T16:06:55.313 INFO:tasks.rados.rados.0.smithi165.stdout:84: finishing write tid 4 to smithi16534841-44 2024-09-10T16:06:55.313 INFO:tasks.rados.rados.0.smithi165.stdout:84: finishing write tid 5 to smithi16534841-44 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:84: oid 44 updating version 0 to 14 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:84: oid 44 updating version 14 to 15 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:84: oid 44 updating version 15 to 16 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:84: oid 44 updating version 16 to 17 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 17 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:84: left oid 44 (ObjNum 59 snap 3 seq_num 59) 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:85: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:85: finishing copy_from to smithi16534841-43 2024-09-10T16:06:55.314 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:81: done (5 left) 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:82: done (4 left) 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:83: done (3 left) 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:84: done (2 left) 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:85: done (1 left) 2024-09-10T16:06:55.919 INFO:tasks.rados.rados.0.smithi165.stdout:86: done (0 left) 2024-09-10T16:06:55.920 INFO:tasks.rados.rados.0.smithi165.stdout:87: read oid 20 snap -1 2024-09-10T16:06:55.920 INFO:tasks.rados.rados.0.smithi165.stdout:87: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-10T16:06:55.920 INFO:tasks.rados.rados.0.smithi165.stdout:88: snap_create 2024-09-10T16:06:55.926 INFO:tasks.rados.rados.0.smithi165.stdout:87: expect (ObjNum 19 snap 0 seq_num 19) 2024-09-10T16:06:55.926 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:55.953 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:56.918 INFO:tasks.rados.rados.0.smithi165.stdout:87: done (1 left) 2024-09-10T16:06:56.918 INFO:tasks.rados.rados.0.smithi165.stdout:88: done (0 left) 2024-09-10T16:06:56.919 INFO:tasks.rados.rados.0.smithi165.stdout:89: write oid 42 current snap is 5 2024-09-10T16:06:56.919 INFO:tasks.rados.rados.0.smithi165.stdout:89: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-10T16:06:56.924 INFO:tasks.rados.rados.0.smithi165.stdout:89: writing smithi16534841-42 from 608420 to 1137831 tid 1 2024-09-10T16:06:56.931 INFO:tasks.rados.rados.0.smithi165.stdout:89: writing smithi16534841-42 from 1778096 to 2339107 tid 2 2024-09-10T16:06:56.935 INFO:tasks.rados.rados.0.smithi165.stdout:89: writing smithi16534841-42 from 2881850 to 3008420 tid 3 2024-09-10T16:06:56.935 INFO:tasks.rados.rados.0.smithi165.stdout:90: snap_remove snap 3 2024-09-10T16:06:57.925 INFO:tasks.rados.rados.0.smithi165.stdout:90: done (1 left) 2024-09-10T16:06:57.925 INFO:tasks.rados.rados.0.smithi165.stdout:91: read oid 6 snap 2 2024-09-10T16:06:57.925 INFO:tasks.rados.rados.0.smithi165.stdout:91: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-10T16:06:57.925 INFO:tasks.rados.rados.0.smithi165.stdout:92: write oid 32 current snap is 5 2024-09-10T16:06:57.925 INFO:tasks.rados.rados.0.smithi165.stdout:92: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-10T16:06:57.932 INFO:tasks.rados.rados.0.smithi165.stdout:92: writing smithi16534841-32 from 625227 to 1346240 tid 1 2024-09-10T16:06:57.939 INFO:tasks.rados.rados.0.smithi165.stdout:92: writing smithi16534841-32 from 1952931 to 2657600 tid 2 2024-09-10T16:06:57.939 INFO:tasks.rados.rados.0.smithi165.stdout:93: write oid 3 current snap is 5 2024-09-10T16:06:57.940 INFO:tasks.rados.rados.0.smithi165.stdout:93: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-10T16:06:57.947 INFO:tasks.rados.rados.0.smithi165.stdout:93: writing smithi16534841-3 from 642034 to 1438296 tid 1 2024-09-10T16:06:57.957 INFO:tasks.rados.rados.0.smithi165.stdout:93: writing smithi16534841-3 from 1895060 to 2459740 tid 2 2024-09-10T16:06:57.957 INFO:tasks.rados.rados.0.smithi165.stdout:94: read oid 27 snap -1 2024-09-10T16:06:57.957 INFO:tasks.rados.rados.0.smithi165.stdout:94: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-10T16:06:57.957 INFO:tasks.rados.rados.0.smithi165.stdout:95: write oid 10 current snap is 5 2024-09-10T16:06:57.958 INFO:tasks.rados.rados.0.smithi165.stdout:95: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-10T16:06:57.960 INFO:tasks.rados.rados.0.smithi165.stdout:95: writing smithi16534841-10 from 658841 to 1130352 tid 1 2024-09-10T16:06:57.967 INFO:tasks.rados.rados.0.smithi165.stdout:95: writing smithi16534841-10 from 1553542 to 2261880 tid 2 2024-09-10T16:06:57.972 INFO:tasks.rados.rados.0.smithi165.stdout:95: writing smithi16534841-10 from 2964119 to 3058841 tid 3 2024-09-10T16:06:57.972 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing write tid 1 to smithi16534841-42 2024-09-10T16:06:57.972 INFO:tasks.rados.rados.0.smithi165.stdout:96: read oid 36 snap -1 2024-09-10T16:06:57.972 INFO:tasks.rados.rados.0.smithi165.stdout:96: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-10T16:06:57.972 INFO:tasks.rados.rados.0.smithi165.stdout:97: snap_remove snap 4 2024-09-10T16:06:58.961 INFO:tasks.rados.rados.0.smithi165.stdout:97: done (7 left) 2024-09-10T16:06:58.961 INFO:tasks.rados.rados.0.smithi165.stdout:98: write (excl) oid 20 current snap is 5 2024-09-10T16:06:58.961 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing write tid 2 to smithi16534841-42 2024-09-10T16:06:58.961 INFO:tasks.rados.rados.0.smithi165.stdout:98: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-10T16:06:58.968 INFO:tasks.rados.rados.0.smithi165.stdout:98: writing smithi16534841-20 from 675648 to 1222408 tid 1 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:98: writing smithi16534841-20 from 2012024 to 2580373 tid 2 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:99: snap_create 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing write tid 3 to smithi16534841-42 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing write tid 4 to smithi16534841-42 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: finishing write tid 5 to smithi16534841-42 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: oid 42 updating version 0 to 20 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: oid 42 version 20 is already newer than 18 2024-09-10T16:06:58.977 INFO:tasks.rados.rados.0.smithi165.stdout:89: oid 42 version 20 is already newer than 19 2024-09-10T16:06:58.978 INFO:tasks.rados.rados.0.smithi165.stdout:89: oid 42 version 20 is already newer than 17 2024-09-10T16:06:58.978 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 20 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-09-10T16:06:58.978 INFO:tasks.rados.rados.0.smithi165.stdout:89: left oid 42 (ObjNum 60 snap 5 seq_num 60) 2024-09-10T16:06:58.978 INFO:tasks.rados.rados.0.smithi165.stdout:91: expect (ObjNum 5 snap 0 seq_num 5) 2024-09-10T16:06:58.978 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:59.024 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:59.048 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 1 to smithi16534841-32 2024-09-10T16:06:59.048 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 2 to smithi16534841-32 2024-09-10T16:06:59.048 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 3 to smithi16534841-32 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:92: finishing write tid 4 to smithi16534841-32 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 32 updating version 0 to 24 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 32 version 24 is already newer than 22 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:92: oid 32 version 24 is already newer than 23 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 24 (ObjNum 61 snap 5 seq_num 61) dirty exists 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:92: left oid 32 (ObjNum 61 snap 5 seq_num 61) 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 1 to smithi16534841-3 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 2 to smithi16534841-3 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 3 to smithi16534841-3 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: finishing write tid 4 to smithi16534841-3 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 3 updating version 0 to 31 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 3 version 31 is already newer than 30 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:93: oid 3 updating version 31 to 32 2024-09-10T16:06:59.049 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 32 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-10T16:06:59.050 INFO:tasks.rados.rados.0.smithi165.stdout:93: left oid 3 (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:06:59.050 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing write tid 1 to smithi16534841-10 2024-09-10T16:06:59.050 INFO:tasks.rados.rados.0.smithi165.stdout:94: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-10T16:06:59.074 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:59.099 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing write tid 2 to smithi16534841-10 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing write tid 3 to smithi16534841-10 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing write tid 4 to smithi16534841-10 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: finishing write tid 5 to smithi16534841-10 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: oid 10 updating version 0 to 12 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: oid 10 updating version 12 to 14 2024-09-10T16:06:59.126 INFO:tasks.rados.rados.0.smithi165.stdout:95: oid 10 version 14 is already newer than 13 2024-09-10T16:06:59.127 INFO:tasks.rados.rados.0.smithi165.stdout:95: oid 10 version 14 is already newer than 11 2024-09-10T16:06:59.127 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 14 (ObjNum 63 snap 5 seq_num 63) dirty exists 2024-09-10T16:06:59.127 INFO:tasks.rados.rados.0.smithi165.stdout:95: left oid 10 (ObjNum 63 snap 5 seq_num 63) 2024-09-10T16:06:59.127 INFO:tasks.rados.rados.0.smithi165.stdout:96: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-10T16:06:59.140 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 566031 2024-09-10T16:06:59.164 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: finishing write tid 1 to smithi16534841-20 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: finishing write tid 2 to smithi16534841-20 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: finishing write tid 3 to smithi16534841-20 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: finishing write tid 4 to smithi16534841-20 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: oid 20 updating version 0 to 36 2024-09-10T16:06:59.193 INFO:tasks.rados.rados.0.smithi165.stdout:98: oid 20 updating version 36 to 37 2024-09-10T16:06:59.194 INFO:tasks.rados.rados.0.smithi165.stdout:98: oid 20 version 37 is already newer than 33 2024-09-10T16:06:59.194 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 37 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-10T16:06:59.194 INFO:tasks.rados.rados.0.smithi165.stdout:98: left oid 20 (ObjNum 64 snap 5 seq_num 64) 2024-09-10T16:06:59.966 INFO:tasks.rados.rados.0.smithi165.stdout:89: done (8 left) 2024-09-10T16:06:59.966 INFO:tasks.rados.rados.0.smithi165.stdout:91: done (7 left) 2024-09-10T16:06:59.966 INFO:tasks.rados.rados.0.smithi165.stdout:92: done (6 left) 2024-09-10T16:06:59.966 INFO:tasks.rados.rados.0.smithi165.stdout:93: done (5 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:94: done (4 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:95: done (3 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:96: done (2 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:98: done (1 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:99: done (0 left) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:100: read oid 40 snap -1 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:100: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-10T16:06:59.967 INFO:tasks.rados.rados.0.smithi165.stdout:101: snap_create 2024-09-10T16:06:59.976 INFO:tasks.rados.rados.0.smithi165.stdout:100: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-10T16:07:00.018 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2421281 2024-09-10T16:07:00.970 INFO:tasks.rados.rados.0.smithi165.stdout:100: done (1 left) 2024-09-10T16:07:00.970 INFO:tasks.rados.rados.0.smithi165.stdout:101: done (0 left) 2024-09-10T16:07:00.970 INFO:tasks.rados.rados.0.smithi165.stdout:102: read oid 27 snap -1 2024-09-10T16:07:00.970 INFO:tasks.rados.rados.0.smithi165.stdout:102: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-10T16:07:00.971 INFO:tasks.rados.rados.0.smithi165.stdout:103: read oid 28 snap -1 2024-09-10T16:07:00.971 INFO:tasks.rados.rados.0.smithi165.stdout:103: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:00.971 INFO:tasks.rados.rados.0.smithi165.stdout:104: delete oid 34 current snap is 7 2024-09-10T16:07:00.977 INFO:tasks.rados.rados.0.smithi165.stdout:104: done (2 left) 2024-09-10T16:07:00.977 INFO:tasks.rados.rados.0.smithi165.stdout:105: write oid 21 current snap is 7 2024-09-10T16:07:00.978 INFO:tasks.rados.rados.0.smithi165.stdout:105: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2024-09-10T16:07:00.985 INFO:tasks.rados.rados.0.smithi165.stdout:105: writing smithi16534841-21 from 709262 to 1406520 tid 1 2024-09-10T16:07:00.992 INFO:tasks.rados.rados.0.smithi165.stdout:105: writing smithi16534841-21 from 2012635 to 2584653 tid 2 2024-09-10T16:07:00.995 INFO:tasks.rados.rados.0.smithi165.stdout:105: writing smithi16534841-21 from 3046388 to 3109262 tid 3 2024-09-10T16:07:00.995 INFO:tasks.rados.rados.0.smithi165.stdout:106: snap_create 2024-09-10T16:07:00.996 INFO:tasks.rados.rados.0.smithi165.stdout:102: expect (ObjNum 26 snap 0 seq_num 26) 2024-09-10T16:07:00.996 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:01.066 INFO:tasks.rados.rados.0.smithi165.stdout:103: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:01.115 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:01.141 INFO:tasks.rados.rados.0.smithi165.stdout:105: finishing write tid 1 to smithi16534841-21 2024-09-10T16:07:01.141 INFO:tasks.rados.rados.0.smithi165.stdout:105: finishing write tid 2 to smithi16534841-21 2024-09-10T16:07:01.141 INFO:tasks.rados.rados.0.smithi165.stdout:105: finishing write tid 3 to smithi16534841-21 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: finishing write tid 4 to smithi16534841-21 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: finishing write tid 5 to smithi16534841-21 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: oid 21 updating version 0 to 40 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: oid 21 updating version 40 to 42 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: oid 21 version 42 is already newer than 41 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: oid 21 version 42 is already newer than 39 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 42 (ObjNum 66 snap 7 seq_num 66) dirty exists 2024-09-10T16:07:01.142 INFO:tasks.rados.rados.0.smithi165.stdout:105: left oid 21 (ObjNum 66 snap 7 seq_num 66) 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:102: done (3 left) 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:103: done (2 left) 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:105: done (1 left) 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:106: done (0 left) 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:107: write (excl) oid 25 current snap is 8 2024-09-10T16:07:01.988 INFO:tasks.rados.rados.0.smithi165.stdout:107: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-10T16:07:01.996 INFO:tasks.rados.rados.0.smithi165.stdout:107: writing smithi16534841-25 from 726069 to 1498576 tid 1 2024-09-10T16:07:02.003 INFO:tasks.rados.rados.0.smithi165.stdout:107: writing smithi16534841-25 from 2071117 to 2786793 tid 2 2024-09-10T16:07:02.004 INFO:tasks.rados.rados.0.smithi165.stdout:108: read oid 37 snap -1 2024-09-10T16:07:02.004 INFO:tasks.rados.rados.0.smithi165.stdout:108: expect deleted 2024-09-10T16:07:02.004 INFO:tasks.rados.rados.0.smithi165.stdout:109: write oid 33 current snap is 8 2024-09-10T16:07:02.004 INFO:tasks.rados.rados.0.smithi165.stdout:109: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-10T16:07:02.010 INFO:tasks.rados.rados.0.smithi165.stdout:109: writing smithi16534841-33 from 742876 to 1190632 tid 1 2024-09-10T16:07:02.017 INFO:tasks.rados.rados.0.smithi165.stdout:109: writing smithi16534841-33 from 1729599 to 2305286 tid 2 2024-09-10T16:07:02.021 INFO:tasks.rados.rados.0.smithi165.stdout:109: writing smithi16534841-33 from 3101234 to 3142876 tid 3 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:110: snap_create 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: finishing write tid 1 to smithi16534841-25 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: finishing write tid 2 to smithi16534841-25 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: finishing write tid 3 to smithi16534841-25 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: finishing write tid 4 to smithi16534841-25 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: oid 25 updating version 0 to 45 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: oid 25 version 45 is already newer than 44 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: oid 25 updating version 45 to 46 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 46 (ObjNum 67 snap 8 seq_num 67) dirty exists 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:107: left oid 25 (ObjNum 67 snap 8 seq_num 67) 2024-09-10T16:07:02.022 INFO:tasks.rados.rados.0.smithi165.stdout:109: finishing write tid 1 to smithi16534841-33 2024-09-10T16:07:02.023 INFO:tasks.rados.rados.0.smithi165.stdout:109: finishing write tid 2 to smithi16534841-33 2024-09-10T16:07:02.025 INFO:tasks.rados.rados.0.smithi165.stdout:109: finishing write tid 3 to smithi16534841-33 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: finishing write tid 4 to smithi16534841-33 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: finishing write tid 5 to smithi16534841-33 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: oid 33 updating version 0 to 49 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: oid 33 version 49 is already newer than 48 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: oid 33 updating version 49 to 50 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:109: oid 33 updating version 50 to 51 2024-09-10T16:07:02.028 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 51 (ObjNum 68 snap 8 seq_num 68) dirty exists 2024-09-10T16:07:02.029 INFO:tasks.rados.rados.0.smithi165.stdout:109: left oid 33 (ObjNum 68 snap 8 seq_num 68) 2024-09-10T16:07:02.840 INFO:tasks.rados.rados.0.smithi165.stdout:107: done (3 left) 2024-09-10T16:07:02.840 INFO:tasks.rados.rados.0.smithi165.stdout:108: done (2 left) 2024-09-10T16:07:02.840 INFO:tasks.rados.rados.0.smithi165.stdout:109: done (1 left) 2024-09-10T16:07:02.840 INFO:tasks.rados.rados.0.smithi165.stdout:110: done (0 left) 2024-09-10T16:07:02.840 INFO:tasks.rados.rados.0.smithi165.stdout:111: snap_create 2024-09-10T16:07:03.844 INFO:tasks.rados.rados.0.smithi165.stdout:111: done (0 left) 2024-09-10T16:07:03.844 INFO:tasks.rados.rados.0.smithi165.stdout:112: copy_from oid 43 from oid 49 current snap is 10 2024-09-10T16:07:03.844 INFO:tasks.rados.rados.0.smithi165.stdout:113: write (excl) oid 11 current snap is 10 2024-09-10T16:07:03.844 INFO:tasks.rados.rados.0.smithi165.stdout:113: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-10T16:07:03.851 INFO:tasks.rados.rados.0.smithi165.stdout:113: writing smithi16534841-11 from 759683 to 1399041 tid 1 2024-09-10T16:07:03.859 INFO:tasks.rados.rados.0.smithi165.stdout:113: writing smithi16534841-11 from 1904434 to 2623779 tid 2 2024-09-10T16:07:03.862 INFO:tasks.rados.rados.0.smithi165.stdout:113: writing smithi16534841-11 from 3128657 to 3159683 tid 3 2024-09-10T16:07:03.862 INFO:tasks.rados.rados.0.smithi165.stdout:114: read oid 28 snap 7 2024-09-10T16:07:03.863 INFO:tasks.rados.rados.0.smithi165.stdout:114: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:03.863 INFO:tasks.rados.rados.0.smithi165.stdout:115: write (excl) oid 46 current snap is 10 2024-09-10T16:07:03.863 INFO:tasks.rados.rados.0.smithi165.stdout:115: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-10T16:07:03.870 INFO:tasks.rados.rados.0.smithi165.stdout:115: writing smithi16534841-46 from 776490 to 1491097 tid 1 2024-09-10T16:07:03.878 INFO:tasks.rados.rados.0.smithi165.stdout:115: writing smithi16534841-46 from 2246563 to 2825919 tid 2 2024-09-10T16:07:03.878 INFO:tasks.rados.rados.0.smithi165.stdout:116: write (excl) oid 15 current snap is 10 2024-09-10T16:07:03.879 INFO:tasks.rados.rados.0.smithi165.stdout:112: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:07:03.879 INFO:tasks.rados.rados.0.smithi165.stdout:116: seq_num 71 ranges {793297=789856,2305045=723014} 2024-09-10T16:07:03.888 INFO:tasks.rados.rados.0.smithi165.stdout:116: writing smithi16534841-15 from 793297 to 1583153 tid 1 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:116: writing smithi16534841-15 from 2305045 to 3028059 tid 2 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:117: snap_create 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 1 to smithi16534841-11 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:112: finishing copy_from to smithi16534841-43 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 9 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 2 to smithi16534841-11 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 3 to smithi16534841-11 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 4 to smithi16534841-11 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: finishing write tid 5 to smithi16534841-11 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 11 updating version 0 to 26 2024-09-10T16:07:03.896 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 11 updating version 26 to 27 2024-09-10T16:07:03.897 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 11 updating version 27 to 29 2024-09-10T16:07:03.897 INFO:tasks.rados.rados.0.smithi165.stdout:113: oid 11 version 29 is already newer than 28 2024-09-10T16:07:03.897 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 29 (ObjNum 69 snap 10 seq_num 69) dirty exists 2024-09-10T16:07:03.897 INFO:tasks.rados.rados.0.smithi165.stdout:113: left oid 11 (ObjNum 69 snap 10 seq_num 69) 2024-09-10T16:07:03.897 INFO:tasks.rados.rados.0.smithi165.stdout:114: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing write tid 1 to smithi16534841-46 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing write tid 2 to smithi16534841-46 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing write tid 3 to smithi16534841-46 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: finishing write tid 4 to smithi16534841-46 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: oid 46 updating version 0 to 20 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: oid 46 updating version 20 to 21 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: oid 46 version 21 is already newer than 19 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 21 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:07:03.968 INFO:tasks.rados.rados.0.smithi165.stdout:115: left oid 46 (ObjNum 70 snap 10 seq_num 70) 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: finishing write tid 1 to smithi16534841-15 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: finishing write tid 2 to smithi16534841-15 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: finishing write tid 3 to smithi16534841-15 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: finishing write tid 4 to smithi16534841-15 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: oid 15 updating version 0 to 12 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: oid 15 updating version 12 to 13 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: oid 15 version 13 is already newer than 11 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 13 (ObjNum 71 snap 10 seq_num 71) dirty exists 2024-09-10T16:07:03.969 INFO:tasks.rados.rados.0.smithi165.stdout:116: left oid 15 (ObjNum 71 snap 10 seq_num 71) 2024-09-10T16:07:04.843 INFO:tasks.rados.rados.0.smithi165.stdout:112: done (5 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:113: done (4 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:114: done (3 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:115: done (2 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:116: done (1 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:117: done (0 left) 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:118: write (excl) oid 36 current snap is 11 2024-09-10T16:07:04.844 INFO:tasks.rados.rados.0.smithi165.stdout:118: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-10T16:07:04.847 INFO:tasks.rados.rados.0.smithi165.stdout:118: writing smithi16534841-36 from 410104 to 875209 tid 1 2024-09-10T16:07:04.854 INFO:tasks.rados.rados.0.smithi165.stdout:118: writing smithi16534841-36 from 1563527 to 2146552 tid 2 2024-09-10T16:07:04.860 INFO:tasks.rados.rados.0.smithi165.stdout:118: writing smithi16534841-36 from 2810926 to 3210104 tid 3 2024-09-10T16:07:04.860 INFO:tasks.rados.rados.0.smithi165.stdout:119: delete oid 39 current snap is 11 2024-09-10T16:07:04.860 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing write tid 1 to smithi16534841-36 2024-09-10T16:07:04.861 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing write tid 2 to smithi16534841-36 2024-09-10T16:07:04.865 INFO:tasks.rados.rados.0.smithi165.stdout:119: done (1 left) 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:120: rollback oid 21 current snap is 11 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 2 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:121: snap_create 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing write tid 3 to smithi16534841-36 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing write tid 4 to smithi16534841-36 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: finishing write tid 5 to smithi16534841-36 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: oid 36 updating version 0 to 15 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: oid 36 version 15 is already newer than 14 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: oid 36 updating version 15 to 17 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:118: oid 36 version 17 is already newer than 16 2024-09-10T16:07:04.866 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 17 (ObjNum 72 snap 11 seq_num 72) dirty exists 2024-09-10T16:07:04.867 INFO:tasks.rados.rados.0.smithi165.stdout:118: left oid 36 (ObjNum 72 snap 11 seq_num 72) 2024-09-10T16:07:04.871 INFO:tasks.rados.rados.0.smithi165.stdout:120: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:07:04.871 INFO:tasks.rados.rados.0.smithi165.stdout:120: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:07:04.873 INFO:tasks.rados.rados.0.smithi165.stdout:120: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:07:04.873 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 55 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-10T16:07:05.858 INFO:tasks.rados.rados.0.smithi165.stdout:118: done (2 left) 2024-09-10T16:07:05.858 INFO:tasks.rados.rados.0.smithi165.stdout:120: done (1 left) 2024-09-10T16:07:05.859 INFO:tasks.rados.rados.0.smithi165.stdout:121: done (0 left) 2024-09-10T16:07:05.859 INFO:tasks.rados.rados.0.smithi165.stdout:122: snap_create 2024-09-10T16:07:06.431 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:07:06.431 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:07:06.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:07:06.435 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:07:06.435 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:07:06.435 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:07:06.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.543+0000 7f4ea270e640 10 monclient: get_monmap_and_config 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.543+0000 7f4ea270e640 10 monclient: build_initial_monmap 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.543+0000 7f4ea270e640 10 monclient: monmap: 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:06.544430+0000 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:06.544430+0000 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:06.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:06.546 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:06.546 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:06.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 Processor -- start 2024-09-10T16:07:06.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 -- start start 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient: init 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e9c09c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e9c09d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e9c09f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e9c09fbc0 con 0x7f4e9c09f4c0 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4e9c0a2a20 con 0x7f4e9c09c310 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:06.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c09c6f0 con 0x7f4e9c09d190 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.544+0000 7f4ea270e640 10 monclient(hunting): authenticate will time out at 805.031961s 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48214/0 (socket says 172.21.15.164:48214) 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 1 -- 172.21.15.164:0/1164209677 learned_addr learned my addr 172.21.15.164:0/1164209677 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33052/0 (socket says 172.21.15.164:33052) 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): get_auth_request con 0x7f4e9c09c310 auth_method 0 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:06.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4ea0cab640 1 --1- 172.21.15.164:0/1164209677 >> v1:172.21.15.164:6789/0 conn(0x7f4e9c09f4c0 0x7f4e9c13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55474/0 (socket says 172.21.15.164:55474) 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): get_auth_request con 0x7f4e9c09d190 auth_method 0 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/1164209677 >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 msgr2=0x7f4e9c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 --2- 172.21.15.164:0/1164209677 >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/1164209677 >> v1:172.21.15.164:6789/0 conn(0x7f4e9c09f4c0 legacy=0x7f4e9c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 monclient: found mon.noname-b 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/1164209677 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c09a4d0 con 0x7f4e9c09d190 2024-09-10T16:07:06.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.545+0000 7f4e9bfff640 1 --2- 172.21.15.164:0/1164209677 >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/1164209677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 msgr2=0x7f4e9c09ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9b7fe640 1 --2- 172.21.15.164:0/1164209677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4dc3e8ffbb08c55 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 1 -- 172.21.15.164:0/1164209677 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c037070 con 0x7f4e9c09d190 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: dump: 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:06.558 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: _finish_auth 0 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: _check_auth_tickets 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 1 -- 172.21.15.164:0/1164209677 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c032440 con 0x7f4e9c09d190 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 5 monclient: authenticate success, global_id 4418 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:06.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 1 -- 172.21.15.164:0/1164209677 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c032810 con 0x7f4e9c09d190 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 4 set_mon_vals no callback set 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4e9affd640 10 monclient: dump: 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 10 monclient: shutdown 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 -- 172.21.15.164:0/1164209677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 msgr2=0x7f4e9c09ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 --2- 172.21.15.164:0/1164209677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c0097a0 tx=0x7f4e8c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 -- 172.21.15.164:0/1164209677 shutdown_connections 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 --2- 172.21.15.164:0/1164209677 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09d190 0x7f4e9c09ef80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 --2- 172.21.15.164:0/1164209677 >> v2:172.21.15.164:3300/0 conn(0x7f4e9c09c310 0x7f4e9c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 -- 172.21.15.164:0/1164209677 >> 172.21.15.164:0/1164209677 conn(0x7f4e9c0a3110 msgr2=0x7f4e9c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.548+0000 7f4ea270e640 1 -- 172.21.15.164:0/1164209677 shutdown_connections 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.549+0000 7f4ea270e640 1 -- 172.21.15.164:0/1164209677 wait complete. 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.549+0000 7f4ea270e640 10 monclient: build_initial_monmap 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.549+0000 7f4ea270e640 10 monclient: monmap: 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:06.562 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.549+0000 7f4ea270e640 1 Processor -- start 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 -- start start 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient: init 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 0x7f4e9c0973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-b con 0x7f4e9c09c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-c con 0x7f4e9c09f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:06.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 0x7f4e9c08e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): picked mon.noname-a con 0x7f4e9c090230 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c0a2230 con 0x7f4e9c090230 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9c0a20b0 con 0x7f4e9c09c310 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9c074e00 con 0x7f4e9c09f4c0 2024-09-10T16:07:06.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea270e640 10 monclient(hunting): authenticate will time out at 805.037680s 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 0x7f4e9c08e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33066/0 (socket says 172.21.15.164:33066) 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): get_auth_request con 0x7f4e9c090230 auth_method 0 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/3298765414 learned_addr learned my addr 172.21.15.164:0/3298765414 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): get_auth_request con 0x7f4e9c09f4c0 auth_method 0 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:06.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4ea0cab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.550+0000 7f4e9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 10 monclient(hunting): handle_auth_done global_id 4413 payload 824 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 msgr2=0x7f4e9c0978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9bfff640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 0x7f4e9c0973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9b7fe640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 msgr2=0x7f4e9c0978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 msgr2=0x7f4e9c0978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:06.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9b7fe640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9b7fe640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 msgr2=0x7f4e9c0973b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 0x7f4e9c0973b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 10 monclient: _finish_hunting 0 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 monclient: found mon.noname-a 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4ea0cab640 1 -- 172.21.15.164:0/3298765414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9c0915c0 con 0x7f4e9c090230 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9bfff640 10 monclient: get_auth_request con 0x7f4e9c09c310 auth_method 0 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.551+0000 7f4e9bfff640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 0x7f4e9c0973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea0cab640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 msgr2=0x7f4e9c08e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:06.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea0cab640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 0x7f4e9c08e3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400eaf0 tx=0x7f4e9400c060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94019070 con 0x7f4e9c090230 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: dump: 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:06.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: _finish_auth 0 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e94014400 con 0x7f4e9c090230 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 5 monclient: authenticate success, global_id 4413 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e940147f0 con 0x7f4e9c090230 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 10 monclient: _renew_subs 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e9c0918a0 con 0x7f4e9c090230 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:06.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e98ff9640 10 monclient: dump: 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4e9a7fc640 4 set_mon_vals no callback set 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 10 monclient: _renew_subs 2024-09-10T16:07:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.553+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e9c1c8bd0 con 0x7f4e9c090230 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4e9401c070 con 0x7f4e9c090230 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4ea270e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4ea270e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4ea270e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9c1c8ef0 con 0x7f4e9c090230 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4e98ff9640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 0x7f4e88040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.554+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4739+0+0 (secure 0 0 0) 0x7f4e940549b0 con 0x7f4e9c090230 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.555+0000 7f4e9bfff640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 0x7f4e88040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.555+0000 7f4e9bfff640 10 monclient: get_auth_request con 0x7f4e8803e240 auth_method 0 2024-09-10T16:07:06.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.557+0000 7f4e9bfff640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 0x7f4e88040700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009630 tx=0x7f4e90008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:06.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.559+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4e940533d0 con 0x7f4e9c090230 2024-09-10T16:07:06.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.559+0000 7f4e98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:06.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.559+0000 7f4e98ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4ea270e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:07:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4ea270e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4ea270e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e9c09d930 con 0x7f4e9c090230 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4e98ff9640 1 -- 172.21.15.164:0/3298765414 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f4e9401dd90 con 0x7f4e9c090230 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4e98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.803+0000 7f4e98ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:06.805 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":12,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:07:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 msgr2=0x7f4e88040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 0x7f4e88040700 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e90009630 tx=0x7f4e90008040 comp rx=0 tx=0).stop 2024-09-10T16:07:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 10 monclient: shutdown 2024-09-10T16:07:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 msgr2=0x7f4e9c08e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 0x7f4e9c08e3c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400eaf0 tx=0x7f4e9400c060 comp rx=0 tx=0).stop 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.805+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 shutdown_connections 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e8803e240 0x7f4e88040700 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e9c090230 0x7f4e9c08e3c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e9c09f4c0 0x7f4e9c0978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 --2- 172.21.15.164:0/3298765414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9c09c310 0x7f4e9c0973b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 >> 172.21.15.164:0/3298765414 conn(0x7f4e9c0a3110 msgr2=0x7f4e9c06e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 shutdown_connections 2024-09-10T16:07:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:06.806+0000 7f4ea270e640 1 -- 172.21.15.164:0/3298765414 wait complete. 2024-09-10T16:07:06.855 INFO:tasks.rados.rados.0.smithi165.stdout:122: done (0 left) 2024-09-10T16:07:06.855 INFO:tasks.rados.rados.0.smithi165.stdout:123: snap_create 2024-09-10T16:07:07.841 INFO:tasks.rados.rados.0.smithi165.stdout:123: done (0 left) 2024-09-10T16:07:07.841 INFO:tasks.rados.rados.0.smithi165.stdout:124: delete oid 36 current snap is 14 2024-09-10T16:07:07.845 INFO:tasks.rados.rados.0.smithi165.stdout:124: done (0 left) 2024-09-10T16:07:07.845 INFO:tasks.rados.rados.0.smithi165.stdout:125: read oid 19 snap -1 2024-09-10T16:07:07.845 INFO:tasks.rados.rados.0.smithi165.stdout:125: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-10T16:07:07.845 INFO:tasks.rados.rados.0.smithi165.stdout:126: snap_create 2024-09-10T16:07:07.854 INFO:tasks.rados.rados.0.smithi165.stdout:125: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-10T16:07:07.906 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1114832 2024-09-10T16:07:07.934 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:08.847 INFO:tasks.rados.rados.0.smithi165.stdout:125: done (1 left) 2024-09-10T16:07:08.847 INFO:tasks.rados.rados.0.smithi165.stdout:126: done (0 left) 2024-09-10T16:07:08.847 INFO:tasks.rados.rados.0.smithi165.stdout:127: read oid 30 snap -1 2024-09-10T16:07:08.847 INFO:tasks.rados.rados.0.smithi165.stdout:127: expect (ObjNum 58 snap 3 seq_num 58) 2024-09-10T16:07:08.847 INFO:tasks.rados.rados.0.smithi165.stdout:128: snap_create 2024-09-10T16:07:08.858 INFO:tasks.rados.rados.0.smithi165.stdout:127: expect (ObjNum 58 snap 3 seq_num 58) 2024-09-10T16:07:08.878 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2221595 2024-09-10T16:07:08.891 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:09.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:127: done (1 left) 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:128: done (0 left) 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:129: read oid 19 snap -1 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:129: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:130: write (excl) oid 6 current snap is 16 2024-09-10T16:07:09.851 INFO:tasks.rados.rados.0.smithi165.stdout:130: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2024-09-10T16:07:09.857 INFO:tasks.rados.rados.0.smithi165.stdout:130: writing smithi16534841-6 from 460525 to 1151377 tid 1 2024-09-10T16:07:09.863 INFO:tasks.rados.rados.0.smithi165.stdout:130: writing smithi16534841-6 from 1622620 to 2352972 tid 2 2024-09-10T16:07:09.868 INFO:tasks.rados.rados.0.smithi165.stdout:130: writing smithi16534841-6 from 3060489 to 3260525 tid 3 2024-09-10T16:07:09.868 INFO:tasks.rados.rados.0.smithi165.stdout:131: delete oid 11 current snap is 16 2024-09-10T16:07:09.868 INFO:tasks.rados.rados.0.smithi165.stdout:129: expect (ObjNum 54 snap 3 seq_num 54) 2024-09-10T16:07:09.911 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.932+0000 7f1b6dda1640 10 monclient: get_monmap_and_config 2024-09-10T16:07:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.932+0000 7f1b6dda1640 10 monclient: build_initial_monmap 2024-09-10T16:07:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.932+0000 7f1b6dda1640 10 monclient: monmap: 2024-09-10T16:07:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:09.933 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:09.933050+0000 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:09.933050+0000 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:09.934 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.932+0000 7f1b6dda1640 1 Processor -- start 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 -- start start 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient: init 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient: _add_conns ranks=[4,0,2,3,1] 2024-09-10T16:07:09.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-c con 0x7f1b681354c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b6809d450 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b681383e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6809d830 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b6806e180 con 0x7f1b6809d450 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1b680a65d0 con 0x7f1b681354c0 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1b68135090 con 0x7f1b6809d830 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6dda1640 10 monclient(hunting): authenticate will time out at 808.420580s 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b67fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1b6809d450 0x7f1b68137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55480/0 (socket says 172.21.15.164:55480) 2024-09-10T16:07:09.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b67fff640 1 -- 172.21.15.164:0/1665761953 learned_addr learned my addr 172.21.15.164:0/1665761953 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 1 --2- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b681383e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): get_auth_request con 0x7f1b6809d830 auth_method 0 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 10 monclient(hunting): get_auth_request con 0x7f1b681354c0 auth_method 0 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:09.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 1 -- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 msgr2=0x7f1b680a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 1 -- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 msgr2=0x7f1b680a5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.933+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_done global_id 4424 payload 274 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 1 -- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 msgr2=0x7f1b680a5ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 1 --2- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 1 -- 172.21.15.164:0/1665761953 >> v1:172.21.15.164:6789/0 conn(0x7f1b6809d450 legacy=0x7f1b68137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:09.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 10 monclient: _finish_hunting 0 2024-09-10T16:07:09.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 1 monclient: found mon.noname-b 2024-09-10T16:07:09.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:09.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:09.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.934+0000 7f1b6d5a0640 1 -- 172.21.15.164:0/1665761953 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6806e460 con 0x7f1b6809d830 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6d5a0640 1 -- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 msgr2=0x7f1b681383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6d5a0640 1 --2- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b681383e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009d90 tx=0x7f1b6002a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff0f167772df7cce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 1 -- 172.21.15.164:0/1665761953 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b60007950 con 0x7f1b6809d830 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: dump: 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:09.948 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:09.949 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: _finish_auth 0 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: _check_auth_tickets 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:09.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6dda1640 5 monclient: authenticate success, global_id 4424 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 1 -- 172.21.15.164:0/1665761953 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b60007af0 con 0x7f1b6809d830 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6dda1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6dda1640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 1 -- 172.21.15.164:0/1665761953 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b60031400 con 0x7f1b6809d830 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b6dda1640 4 set_mon_vals no callback set 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.936+0000 7f1b677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b677fe640 10 monclient: dump: 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:09.951 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 10 monclient: shutdown 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 -- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 msgr2=0x7f1b681383e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b681383e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b60009d90 tx=0x7f1b6002a490 comp rx=0 tx=0).stop 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 -- 172.21.15.164:0/1665761953 shutdown_connections 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/1665761953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b681383e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:09.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/1665761953 >> v2:172.21.15.165:3300/0 conn(0x7f1b681354c0 0x7f1b680a5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 -- 172.21.15.164:0/1665761953 >> 172.21.15.164:0/1665761953 conn(0x7f1b680a70e0 msgr2=0x7f1b680a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 -- 172.21.15.164:0/1665761953 shutdown_connections 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.937+0000 7f1b6dda1640 1 -- 172.21.15.164:0/1665761953 wait complete. 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.938+0000 7f1b6dda1640 10 monclient: build_initial_monmap 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.938+0000 7f1b6dda1640 10 monclient: monmap: 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:09.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.938+0000 7f1b6dda1640 1 Processor -- start 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.938+0000 7f1b6dda1640 1 -- start start 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.938+0000 7f1b6dda1640 10 monclient: init 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6809d450 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6809d830 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): picked mon.noname-a con 0x7f1b681354c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6809dec0 con 0x7f1b681354c0 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1b68134dc0 con 0x7f1b6809d450 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1b6809e040 con 0x7f1b6809d830 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6dda1640 10 monclient(hunting): authenticate will time out at 808.426188s 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33088/0 (socket says 172.21.15.164:33088) 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 -- 172.21.15.164:0/3065250182 learned_addr learned my addr 172.21.15.164:0/3065250182 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48234/0 (socket says 172.21.15.164:48234) 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): get_auth_request con 0x7f1b6809d830 auth_method 0 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): get_auth_request con 0x7f1b681354c0 auth_method 0 2024-09-10T16:07:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 10 monclient(hunting): get_auth_request con 0x7f1b6809d450 auth_method 0 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 msgr2=0x7f1b6809a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 msgr2=0x7f1b6809a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:09.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient(hunting): handle_auth_done global_id 4427 payload 824 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 msgr2=0x7f1b6809a390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 msgr2=0x7f1b681cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient: _finish_hunting 0 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 monclient: found mon.noname-c 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b67fff640 1 -- 172.21.15.164:0/3065250182 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b681c0be0 con 0x7f1b6809d830 2024-09-10T16:07:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.939+0000 7f1b6d5a0640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.941+0000 7f1b67fff640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 msgr2=0x7f1b6809a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.941+0000 7f1b67fff640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b460 tx=0x7f1b5400b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.941+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b54015020 con 0x7f1b6809d830 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: dump: 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: _finish_auth 0 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: _check_auth_tickets 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 5 monclient: authenticate success, global_id 4427 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b54011ac0 con 0x7f1b6809d830 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _renew_subs 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b680a17d0 con 0x7f1b6809d830 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:09.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b540103e0 con 0x7f1b6809d830 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 10 monclient: dump: 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:09.962 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _renew_subs 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b66ffd640 4 set_mon_vals no callback set 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b68134710 con 0x7f1b6809d830 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1b54018020 con 0x7f1b6809d830 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.942+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b68138ad0 con 0x7f1b6809d830 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.943+0000 7f1b657fa640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 0x7f1b4c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.943+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(44..44 src has 1..44) ==== 4711+0+0 (secure 0 0 0) 0x7f1b54052b50 con 0x7f1b6809d830 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.943+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 0x7f1b4c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:09.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.943+0000 7f1b6cd9f640 10 monclient: get_auth_request con 0x7f1b4c03e210 auth_method 0 2024-09-10T16:07:09.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.945+0000 7f1b6cd9f640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 0x7f1b4c0406d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b681bf370 tx=0x7f1b5c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:09.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.947+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1b5401f030 con 0x7f1b6809d830 2024-09-10T16:07:09.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.947+0000 7f1b657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:09.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:09.947+0000 7f1b657fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:09.964 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:09.996 INFO:tasks.rados.rados.0.smithi165.stdout:130: finishing write tid 1 to smithi16534841-6 2024-09-10T16:07:09.996 INFO:tasks.rados.rados.0.smithi165.stdout:130: finishing write tid 2 to smithi16534841-6 2024-09-10T16:07:09.996 INFO:tasks.rados.rados.0.smithi165.stdout:130: finishing write tid 3 to smithi16534841-6 2024-09-10T16:07:09.996 INFO:tasks.rados.rados.0.smithi165.stdout:130: finishing write tid 4 to smithi16534841-6 2024-09-10T16:07:09.996 INFO:tasks.rados.rados.0.smithi165.stdout:130: finishing write tid 5 to smithi16534841-6 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: oid 6 updating version 0 to 30 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: oid 6 updating version 30 to 31 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: oid 6 version 31 is already newer than 29 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: oid 6 version 31 is already newer than 28 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 31 (ObjNum 75 snap 16 seq_num 75) dirty exists 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: left oid 6 (ObjNum 75 snap 16 seq_num 75) 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:129: done (2 left) 2024-09-10T16:07:09.997 INFO:tasks.rados.rados.0.smithi165.stdout:130: done (1 left) 2024-09-10T16:07:09.998 INFO:tasks.rados.rados.0.smithi165.stdout:131: done (0 left) 2024-09-10T16:07:09.998 INFO:tasks.rados.rados.0.smithi165.stdout:132: snap_remove snap 2 2024-09-10T16:07:10.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b6dda1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:07:10.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b6dda1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b6dda1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b6809ae10 con 0x7f1b6809d830 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b657fa640 1 -- 172.21.15.164:0/3065250182 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f1b5400d040 con 0x7f1b6809d830 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.189+0000 7f1b657fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:10.191 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:07:10.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 msgr2=0x7f1b4c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 0x7f1b4c0406d0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1b681bf370 tx=0x7f1b5c008040 comp rx=0 tx=0).stop 2024-09-10T16:07:10.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 10 monclient: shutdown 2024-09-10T16:07:10.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 msgr2=0x7f1b6809a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1b5400b460 tx=0x7f1b5400b920 comp rx=0 tx=0).stop 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 shutdown_connections 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1b4c03e210 0x7f1b4c0406d0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b681354c0 0x7f1b681cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b6809d830 0x7f1b6809a8d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 --2- 172.21.15.164:0/3065250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b6809d450 0x7f1b6809a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.191+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 >> 172.21.15.164:0/3065250182 conn(0x7f1b680a70e0 msgr2=0x7f1b681378d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.192+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 shutdown_connections 2024-09-10T16:07:10.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.192+0000 7f1b6dda1640 1 -- 172.21.15.164:0/3065250182 wait complete. 2024-09-10T16:07:10.203 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:07:10.203 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:07:10.204 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:07:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 10 monclient: get_monmap_and_config 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 10 monclient: monmap: 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:10.301224+0000 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:10.301224+0000 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:10.303 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 1 Processor -- start 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 1 -- start start 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.300+0000 7f0d806a9640 10 monclient: init 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T16:07:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0d780a0bc0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 0x7f0d7809e070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0d78136260 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d7813b070 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d780a1740 con 0x7f0d780a0bc0 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d780690a0 con 0x7f0d7813b070 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0d78069660 con 0x7f0d78136260 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d806a9640 10 monclient(hunting): authenticate will time out at 808.788627s 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d7e445640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0d780a0bc0 0x7f0d7809d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55490/0 (socket says 172.21.15.164:55490) 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 learned_addr learned my addr 172.21.15.164:0/1608432231 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.301+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 0x7f0d7809e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request con 0x7f0d78136260 auth_method 0 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 1 -- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 msgr2=0x7f0d7809e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 1 -- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 msgr2=0x7f0d7809e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 0x7f0d7809e070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.302+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 0x7f0d7809e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.303+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 >> v1:172.21.15.164:6789/0 conn(0x7f0d780a0bc0 legacy=0x7f0d7809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 390015210 0 0) 0x7f0d780a1740 con 0x7f0d780a0bc0 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): _negotiate 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): my global_id is 4419 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0d64003520 con 0x7f0d780a0bc0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1608432231 >> v1:172.21.15.165:6789/0 conn(0x7f0d7813b070 legacy=0x7f0d7813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2456977077 0 0) 0x7f0d64003520 con 0x7f0d780a0bc0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 msgr2=0x7f0d7809e070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 --2- 172.21.15.164:0/1608432231 >> v2:172.21.15.165:3300/0 conn(0x7f0d78136260 0x7f0d7809e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 >> v1:172.21.15.165:6789/0 conn(0x7f0d7813b070 legacy=0x7f0d7813d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d78069380 con 0x7f0d780a0bc0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f0d740019a0 con 0x7f0d780a0bc0 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d806a9640 5 monclient: authenticate success, global_id 4419 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d806a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: dump: 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 >> v1:172.21.15.164:6789/0 conn(0x7f0d780a0bc0 legacy=0x7f0d7809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 0x7f0d64003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): picked mon.a con 0x7f0d64003b10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64004800 0x7f0d64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): picked mon.b con 0x7f0d64004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d640099b0 0x7f0d6400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): picked mon.c con 0x7f0d640099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78069660 con 0x7f0d64003b10 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0d6400c3c0 con 0x7f0d64004800 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0d6400c5c0 con 0x7f0d640099b0 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7d443640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d806a9640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 0x7f0d64003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): get_auth_request con 0x7f0d64003b10 auth_method 0 2024-09-10T16:07:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d640099b0 0x7f0d6400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): get_auth_request con 0x7f0d640099b0 auth_method 0 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64004800 0x7f0d64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request con 0x7f0d64004800 auth_method 0 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient(hunting): handle_auth_done global_id 4419 payload 293 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d640099b0 msgr2=0x7f0d6400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d640099b0 0x7f0d6400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64004800 msgr2=0x7f0d64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64004800 0x7f0d64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 monclient: found mon.a 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d64003520 con 0x7f0d64003b10 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7e445640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.304+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1608432231 reap_dead start 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7e445640 1 -- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 msgr2=0x7f0d64003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 0x7f0d64003f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d740014d0 tx=0x7f0d64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5bf100bae733d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74036070 con 0x7f0d64003b10 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: dump: 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.326 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d806a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d74005af0 con 0x7f0d64003b10 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d806a9640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 1 -- 172.21.15.164:0/1608432231 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7400c0e0 con 0x7f0d64003b10 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d806a9640 4 set_mon_vals no callback set 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.307+0000 7f0d7d443640 10 monclient: dump: 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 10 monclient: shutdown 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 -- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 msgr2=0x7f0d64003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 0x7f0d64003f30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f0d740014d0 tx=0x7f0d64005830 comp rx=0 tx=0).stop 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 -- 172.21.15.164:0/1608432231 shutdown_connections 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1608432231 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d64003b10 0x7f0d64003f30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 -- 172.21.15.164:0/1608432231 >> 172.21.15.164:0/1608432231 conn(0x7f0d780a3fb0 msgr2=0x7f0d780a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 -- 172.21.15.164:0/1608432231 shutdown_connections 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.308+0000 7f0d806a9640 1 -- 172.21.15.164:0/1608432231 wait complete. 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.309+0000 7f0d806a9640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.309+0000 7f0d806a9640 10 monclient: monmap: 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.329 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.309+0000 7f0d806a9640 1 Processor -- start 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.309+0000 7f0d806a9640 1 -- start start 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.309+0000 7f0d806a9640 10 monclient: init 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64003b10 0x7f0d7809e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-b con 0x7f0d64003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d7809eef0 0x7f0d7809f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-c con 0x7f0d7809eef0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): picked mon.noname-a con 0x7f0d781cd230 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0d7809d960 con 0x7f0d781cd230 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0d780a0fa0 con 0x7f0d64003b10 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0d7809dae0 con 0x7f0d7809eef0 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d806a9640 10 monclient(hunting): authenticate will time out at 808.797254s 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d7809eef0 0x7f0d7809f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48250/0 (socket says 172.21.15.164:48250) 2024-09-10T16:07:10.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 learned_addr learned my addr 172.21.15.164:0/1192189734 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): get_auth_request con 0x7f0d781cd230 auth_method 0 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request con 0x7f0d7809eef0 auth_method 0 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64003b10 0x7f0d7809e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): get_auth_request con 0x7f0d64003b10 auth_method 0 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient(hunting): handle_auth_done global_id 4422 payload 824 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d7809eef0 msgr2=0x7f0d7809f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d7809eef0 0x7f0d7809f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64003b10 msgr2=0x7f0d7809e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0d64003b10 0x7f0d7809e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d78131e40 con 0x7f0d781cd230 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.310+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0d7809eef0 0x7f0d7809f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.312+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 msgr2=0x7f0d781cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.312+0000 7f0d7ec46640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000f80 tx=0x7f0d700077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.312+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70017020 con 0x7f0d781cd230 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.312+0000 7f0d6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: dump: 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 5 monclient: authenticate success, global_id 4422 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d70007a20 con 0x7f0d781cd230 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 monclient: _renew_subs 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d78132120 con 0x7f0d781cd230 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d7000fd40 con 0x7f0d781cd230 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 10 monclient: dump: 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.336 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7cc42640 4 set_mon_vals no callback set 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 monclient: _renew_subs 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d780a09c0 con 0x7f0d781cd230 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 0x7f0d78138e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0d780a6e40 con 0x7f0d781cdbd0 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 0x7f0d78138e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7e445640 10 monclient: get_auth_request con 0x7f0d781cdbd0 auth_method 0 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7e445640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d7e445640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.313+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0d7001a020 con 0x7f0d781cd230 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7e445640 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7e445640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d78130980 con 0x7f0d781cdbd0 2024-09-10T16:07:10.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d6effd640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0d5c03e210 0x7f0d5c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4711+0+0 (secure 0 0 0) 0x7f0d70052a80 con 0x7f0d781cd230 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0d5c03e210 0x7f0d5c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.314+0000 7f0d7dc44640 10 monclient: get_auth_request con 0x7f0d5c03e210 auth_method 0 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d7e445640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 msgr2=0x7f0d78138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 0x7f0d78138e00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d74033000 tx=0x7f0d78069550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74042070 con 0x7f0d781cdbd0 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 10 monclient: dump: 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.339 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d74037040 con 0x7f0d781cdbd0 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 msgr2=0x7f0d78138e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 0x7f0d78138e00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f0d74033000 tx=0x7f0d78069550 comp rx=0 tx=0).stop 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 msgr2=0x7f0d78138e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.316+0000 7f0d6effd640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cdbd0 0x7f0d78138e00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.317+0000 7f0d7dc44640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0d5c03e210 0x7f0d5c0406d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d780a1a50 tx=0x7f0d68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d806a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 0x7f0d78139750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d806a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78132460 con 0x7f0d78139340 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 0x7f0d78139750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d7e445640 10 monclient: get_auth_request con 0x7f0d78139340 auth_method 0 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d7e445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d7e445640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.322+0000 7f0d7e445640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.323+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.323+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.323+0000 7f0d7e445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.323+0000 7f0d7e445640 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2024-09-10T16:07:10.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.323+0000 7f0d7e445640 1 -- 172.21.15.164:0/1192189734 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d781cdfb0 con 0x7f0d78139340 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d7e445640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 msgr2=0x7f0d78139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d7e445640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 0x7f0d78139750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0d74007a40 tx=0x7f0d7402e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d74036030 con 0x7f0d78139340 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 10 monclient: dump: 2024-09-10T16:07:10.343 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f0d74007690 con 0x7f0d78139340 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:10.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 msgr2=0x7f0d78139750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 0x7f0d78139750 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0d74007a40 tx=0x7f0d7402e000 comp rx=0 tx=0).stop 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 msgr2=0x7f0d78139750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.325+0000 7f0d6effd640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d78139340 0x7f0d78139750 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f0d74007a40 tx=0x7f0d7402e000 comp rx=0 tx=0).stop 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:07:10.345 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 72500, 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.346 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.347 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:10.348 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:10.349 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.350 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:10.351 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:10.352 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:10.354 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.356 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:10.357 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0d5c03e210 msgr2=0x7f0d5c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0d5c03e210 0x7f0d5c0406d0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0d780a1a50 tx=0x7f0d68008040 comp rx=0 tx=0).stop 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 10 monclient: shutdown 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 msgr2=0x7f0d781cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d7ec46640 1 -- 172.21.15.164:0/1192189734 reap_dead start 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f0d70000f80 tx=0x7f0d700077d0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 shutdown_connections 2024-09-10T16:07:10.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 --2- 172.21.15.164:0/1192189734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0d781cd230 0x7f0d781cd690 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 >> 172.21.15.164:0/1192189734 conn(0x7f0d780a3fb0 msgr2=0x7f0d78134380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 shutdown_connections 2024-09-10T16:07:10.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.326+0000 7f0d806a9640 1 -- 172.21.15.164:0/1192189734 wait complete. 2024-09-10T16:07:10.360 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:10.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:07:10.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:07:10.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.458+0000 7f5b1e22a640 10 monclient: get_monmap_and_config 2024-09-10T16:07:10.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.458+0000 7f5b1e22a640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.458+0000 7f5b1e22a640 10 monclient: monmap: 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:10.459076+0000 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:10.459076+0000 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:10.460 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.458+0000 7f5b1e22a640 1 Processor -- start 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 1 -- start start 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient: init 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5b1806cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:10.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5b18134f70 0x7f5b1813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-a con 0x7f5b18134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b1813d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b1809ce50 con 0x7f5b1813d560 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b18135380 con 0x7f5b1806cb70 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5b1806db60 con 0x7f5b18134f70 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b1e22a640 10 monclient(hunting): authenticate will time out at 808.946556s 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5b18134f70 0x7f5b1813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b17fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5b1813d560 0x7f5b1813f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55504/0 (socket says 172.21.15.164:55504) 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b17fff640 1 -- 172.21.15.164:0/1305274829 learned_addr learned my addr 172.21.15.164:0/1305274829 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): get_auth_request con 0x7f5b18134f70 auth_method 0 2024-09-10T16:07:10.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient(hunting): handle_auth_done global_id 4431 payload 274 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 1 -- 172.21.15.164:0/1305274829 >> v1:172.21.15.165:6789/0 conn(0x7f5b1806cb70 legacy=0x7f5b180a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 1 -- 172.21.15.164:0/1305274829 >> v1:172.21.15.164:6789/0 conn(0x7f5b1813d560 legacy=0x7f5b1813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.459+0000 7f5b0effd640 1 -- 172.21.15.164:0/1305274829 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1809cb70 con 0x7f5b18134f70 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.462+0000 7f5b0effd640 1 -- 172.21.15.164:0/1305274829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 msgr2=0x7f5b1813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.462+0000 7f5b0effd640 1 --2- 172.21.15.164:0/1305274829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813d020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097d0 tx=0x7f5b0002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5f20d23f4b506d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 1 -- 172.21.15.164:0/1305274829 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00037070 con 0x7f5b18134f70 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: dump: 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.468 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 1 -- 172.21.15.164:0/1305274829 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b00032440 con 0x7f5b18134f70 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 5 monclient: authenticate success, global_id 4431 2024-09-10T16:07:10.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 1 -- 172.21.15.164:0/1305274829 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00032810 con 0x7f5b18134f70 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 4 set_mon_vals no callback set 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b16ffd640 10 monclient: dump: 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.470 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 10 monclient: shutdown 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 -- 172.21.15.164:0/1305274829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 msgr2=0x7f5b1813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/1305274829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813d020 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f5b000097d0 tx=0x7f5b0002a000 comp rx=0 tx=0).stop 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 -- 172.21.15.164:0/1305274829 shutdown_connections 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/1305274829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813d020 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 -- 172.21.15.164:0/1305274829 >> 172.21.15.164:0/1305274829 conn(0x7f5b180aa900 msgr2=0x7f5b180aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 -- 172.21.15.164:0/1305274829 shutdown_connections 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.463+0000 7f5b1e22a640 1 -- 172.21.15.164:0/1305274829 wait complete. 2024-09-10T16:07:10.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.464+0000 7f5b1e22a640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.464+0000 7f5b1e22a640 10 monclient: monmap: 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.464+0000 7f5b1e22a640 1 Processor -- start 2024-09-10T16:07:10.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.464+0000 7f5b1e22a640 1 -- start start 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.464+0000 7f5b1e22a640 10 monclient: init 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1806cb70 0x7f5b18131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-c con 0x7f5b1806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-a con 0x7f5b18134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b1813d560 0x7f5b1813a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): picked mon.noname-b con 0x7f5b1813d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5b180a4db0 con 0x7f5b18134f70 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1806cf50 con 0x7f5b1813d560 2024-09-10T16:07:10.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5b18141070 con 0x7f5b1806cb70 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b1e22a640 10 monclient(hunting): authenticate will time out at 808.952334s 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1806cb70 0x7f5b18131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48296/0 (socket says 172.21.15.164:48296) 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 -- 172.21.15.164:0/972371873 learned_addr learned my addr 172.21.15.164:0/972371873 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): get_auth_request con 0x7f5b18134f70 auth_method 0 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): get_auth_request con 0x7f5b1806cb70 auth_method 0 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b17fff640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b1813d560 0x7f5b1813a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b17fff640 10 monclient(hunting): get_auth_request con 0x7f5b1813d560 auth_method 0 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b17fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient(hunting): handle_auth_done global_id 4434 payload 824 2024-09-10T16:07:10.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1806cb70 msgr2=0x7f5b18131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1806cb70 0x7f5b18131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b1813d560 msgr2=0x7f5b1813a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5b1813d560 0x7f5b1813a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.465+0000 7f5b0effd640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b1809ef10 con 0x7f5b18134f70 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.467+0000 7f5b0effd640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 msgr2=0x7f5b1813a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.467+0000 7f5b0effd640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009720 tx=0x7f5b000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00041070 con 0x7f5b18134f70 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: dump: 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.480 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b0003c670 con 0x7f5b18134f70 2024-09-10T16:07:10.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 5 monclient: authenticate success, global_id 4434 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 monclient: _renew_subs 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b1809e270 con 0x7f5b18134f70 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00037040 con 0x7f5b18134f70 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b14ff9640 10 monclient: dump: 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.482 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b167fc640 4 set_mon_vals no callback set 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 monclient: _renew_subs 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b1813b9a0 con 0x7f5b18134f70 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 0x7f5b1813aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.468+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1809def0 con 0x7f5b18131b50 2024-09-10T16:07:10.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 0x7f5b1813aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5b00044070 con 0x7f5b18134f70 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient: get_auth_request con 0x7f5b18131b50 auth_method 0 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 10 monclient(hunting): handle_auth_done global_id 4436 payload 824 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b177fe640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5b180008d0 con 0x7f5b18131b50 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b14ff9640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5af803e210 0x7f5af80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4711+0+0 (secure 0 0 0) 0x7f5b00073420 con 0x7f5b18134f70 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b17fff640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5af803e210 0x7f5af80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.469+0000 7f5b17fff640 10 monclient: get_auth_request con 0x7f5af803e210 auth_method 0 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b177fe640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 msgr2=0x7f5b1813aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b177fe640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 0x7f5b1813aaa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e770 tx=0x7f5b0800ec30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08017070 con 0x7f5b18131b50 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 10 monclient: dump: 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5b0800c9b0 con 0x7f5b18131b50 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 msgr2=0x7f5b1813aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 0x7f5b1813aaa0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e770 tx=0x7f5b0800ec30 comp rx=0 tx=0).stop 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 msgr2=0x7f5b1813aaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b14ff9640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b18131b50 0x7f5b1813aaa0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5b0800e770 tx=0x7f5b0800ec30 comp rx=0 tx=0).stop 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.471+0000 7f5b17fff640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5af803e210 0x7f5af80406d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b1813c490 tx=0x7f5b10009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b1e22a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 0x7f5b18135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b1e22a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5b1813b3f0 con 0x7f5b1813afe0 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 0x7f5b18135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient: get_auth_request con 0x7f5b1813afe0 auth_method 0 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2024-09-10T16:07:10.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.478+0000 7f5b177fe640 1 -- 172.21.15.164:0/972371873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5b180770f0 con 0x7f5b1813afe0 2024-09-10T16:07:10.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b177fe640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 msgr2=0x7f5b18135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b177fe640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 0x7f5b18135870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b08019000 tx=0x7f5b080046f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b08017070 con 0x7f5b1813afe0 2024-09-10T16:07:10.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 10 monclient: dump: 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f5b08006540 con 0x7f5b1813afe0 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 msgr2=0x7f5b18135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 0x7f5b18135870 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b08019000 tx=0x7f5b080046f0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 msgr2=0x7f5b18135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.480+0000 7f5b14ff9640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5b1813afe0 0x7f5b18135870 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b08019000 tx=0x7f5b080046f0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:10.490 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 72652, 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.491 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.492 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:10.493 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.494 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:10.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:10.497 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:10.499 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.500 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.501 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.502 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:10.503 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:10.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5af803e210 msgr2=0x7f5af80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5af803e210 0x7f5af80406d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f5b1813c490 tx=0x7f5b10009040 comp rx=0 tx=0).stop 2024-09-10T16:07:10.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 10 monclient: shutdown 2024-09-10T16:07:10.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 msgr2=0x7f5b1813a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b17fff640 1 -- 172.21.15.164:0/972371873 reap_dead start 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009720 tx=0x7f5b000046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 shutdown_connections 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 --2- 172.21.15.164:0/972371873 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5b18134f70 0x7f5b1813a020 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 >> 172.21.15.164:0/972371873 conn(0x7f5b180aa900 msgr2=0x7f5b18134070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 shutdown_connections 2024-09-10T16:07:10.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.481+0000 7f5b1e22a640 1 -- 172.21.15.164:0/972371873 wait complete. 2024-09-10T16:07:10.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:10.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:07:10.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:07:10.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.602+0000 7f2cf774b640 10 monclient: get_monmap_and_config 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.602+0000 7f2cf774b640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.602+0000 7f2cf774b640 10 monclient: monmap: 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:10.603836+0000 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:10.603836+0000 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.603+0000 7f2cf774b640 1 Processor -- start 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.603+0000 7f2cf774b640 1 -- start start 2024-09-10T16:07:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.603+0000 7f2cf774b640 10 monclient: init 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2cf0074af0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2cf00a1090 0x7f2cf00a1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf00a1090 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 0x7f2cf013f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf013d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cf009ebc0 con 0x7f2cf0074af0 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf00698a0 con 0x7f2cf00a1090 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2cf006bfb0 con 0x7f2cf013d190 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf774b640 10 monclient(hunting): authenticate will time out at 809.091250s 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2cf00a1090 0x7f2cf00a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf54e7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2cf0074af0 0x7f2cf0137e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55506/0 (socket says 172.21.15.164:55506) 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf54e7640 1 -- 172.21.15.164:0/3977554791 learned_addr learned my addr 172.21.15.164:0/3977554791 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): get_auth_request con 0x7f2cf00a1090 auth_method 0 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf5ce8640 1 --2- 172.21.15.164:0/3977554791 >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 0x7f2cf013f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_done global_id 4437 payload 274 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 -- 172.21.15.164:0/3977554791 >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 msgr2=0x7f2cf013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 --2- 172.21.15.164:0/3977554791 >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 0x7f2cf013f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 -- 172.21.15.164:0/3977554791 >> v1:172.21.15.164:6789/0 conn(0x7f2cf0074af0 legacy=0x7f2cf0137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf4ce6640 1 -- 172.21.15.164:0/3977554791 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf0138580 con 0x7f2cf00a1090 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf5ce8640 10 monclient: get_auth_request con 0x7f2cf013d190 auth_method 0 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.604+0000 7f2cf5ce8640 1 --2- 172.21.15.164:0/3977554791 >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 0x7f2cf013f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf4ce6640 1 -- 172.21.15.164:0/3977554791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf00a1090 msgr2=0x7f2cf00a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf4ce6640 1 --2- 172.21.15.164:0/3977554791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf00a1090 0x7f2cf00a1500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2cf006fc40 tx=0x7f2cd802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe903b11e3ef2075 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 1 -- 172.21.15.164:0/3977554791 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8037070 con 0x7f2cf00a1090 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: dump: 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf774b640 5 monclient: authenticate success, global_id 4437 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf774b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 1 -- 172.21.15.164:0/3977554791 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cd8032640 con 0x7f2cf00a1090 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf774b640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:10.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2cf774b640 4 set_mon_vals no callback set 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 1 -- 172.21.15.164:0/3977554791 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8032a10 con 0x7f2cf00a1090 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.607+0000 7f2ceffff640 10 monclient: dump: 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 10 monclient: shutdown 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 -- 172.21.15.164:0/3977554791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf00a1090 msgr2=0x7f2cf00a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 --2- 172.21.15.164:0/3977554791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf00a1090 0x7f2cf00a1500 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2cf006fc40 tx=0x7f2cd802a000 comp rx=0 tx=0).stop 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 -- 172.21.15.164:0/3977554791 shutdown_connections 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 --2- 172.21.15.164:0/3977554791 >> v2:172.21.15.165:3300/0 conn(0x7f2cf013d190 0x7f2cf013f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 --2- 172.21.15.164:0/3977554791 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf00a1090 0x7f2cf00a1500 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 -- 172.21.15.164:0/3977554791 >> 172.21.15.164:0/3977554791 conn(0x7f2cf00a3fb0 msgr2=0x7f2cf00a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 -- 172.21.15.164:0/3977554791 shutdown_connections 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 1 -- 172.21.15.164:0/3977554791 wait complete. 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.608+0000 7f2cf774b640 10 monclient: monmap: 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 Processor -- start 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 -- start start 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient: init 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 0x7f2cf01cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-c con 0x7f2cf0074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf00a1090 0x7f2cf013c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-b con 0x7f2cf00a1090 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): picked mon.noname-a con 0x7f2cf013d190 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0074ed0 con 0x7f2cf013d190 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf0069370 con 0x7f2cf00a1090 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2cf013fcd0 con 0x7f2cf0074af0 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf774b640 10 monclient(hunting): authenticate will time out at 809.096659s 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 0x7f2cf01cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48326/0 (socket says 172.21.15.164:48326) 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 0x7f2cf01cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33150/0 (socket says 172.21.15.164:33150) 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 learned_addr learned my addr 172.21.15.164:0/2272695717 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): get_auth_request con 0x7f2cf013d190 auth_method 0 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): get_auth_request con 0x7f2cf0074af0 auth_method 0 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf00a1090 0x7f2cf013c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): get_auth_request con 0x7f2cf00a1090 auth_method 0 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf5ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.609+0000 7f2cf4ce6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 msgr2=0x7f2cf01cd1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 0x7f2cf01cd1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf00a1090 msgr2=0x7f2cf013c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf00a1090 0x7f2cf013c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf00a3050 con 0x7f2cf013d190 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.610+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2cf0074af0 0x7f2cf01cd1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 msgr2=0x7f2cf013cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf5ce8640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800cea0 tx=0x7f2ce800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8003b90 con 0x7f2cf013d190 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: dump: 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ce8003d50 con 0x7f2cf013d190 2024-09-10T16:07:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 5 monclient: authenticate success, global_id 4440 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 10 monclient: _renew_subs 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf00a3660 con 0x7f2cf013d190 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800da90 con 0x7f2cf013d190 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cedffb640 10 monclient: dump: 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 10 monclient: _renew_subs 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cef7fe640 4 set_mon_vals no callback set 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.612+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf009d390 con 0x7f2cf013d190 2024-09-10T16:07:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf774b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 0x7f2cf009bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf00a5ad0 con 0x7f2cf009b970 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2ce800dc30 con 0x7f2cf013d190 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 0x7f2cd00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4711+0+0 (secure 0 0 0) 0x7f2ce80536e0 con 0x7f2cf013d190 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 0x7f2cf009bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf54e7640 10 monclient: get_auth_request con 0x7f2cf009b970 auth_method 0 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf54e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf54e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.613+0000 7f2cf54e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf4ce6640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 0x7f2cd00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf4ce6640 10 monclient: get_auth_request con 0x7f2cd003e210 auth_method 0 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_done global_id 14107 payload 824 2024-09-10T16:07:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.614+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cf00a1500 con 0x7f2cf009b970 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 msgr2=0x7f2cf009bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 0x7f2cf009bd50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007270 tx=0x7f2ce0007730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cf4ce6640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 0x7f2cd00406d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00a3ed0 tx=0x7f2cd802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0017020 con 0x7f2cf009b970 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 10 monclient: dump: 2024-09-10T16:07:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2ce000fd00 con 0x7f2cf009b970 2024-09-10T16:07:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 msgr2=0x7f2cf009bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 0x7f2cf009bd50 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007270 tx=0x7f2ce0007730 comp rx=0 tx=0).stop 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 msgr2=0x7f2cf009bd50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.616+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009b970 0x7f2cf009bd50 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007270 tx=0x7f2ce0007730 comp rx=0 tx=0).stop 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.622+0000 7f2cf774b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.622+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.622+0000 7f2cf774b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.622+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf01cd730 con 0x7f2cf009c290 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient: get_auth_request con 0x7f2cf009c290 auth_method 0 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_done global_id 14110 payload 824 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.623+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cf00772c0 con 0x7f2cf009c290 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.625+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 msgr2=0x7f2cf00a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.625+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0018000 tx=0x7f2ce0004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 msgr2=0x7f2cf00a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 msgr2=0x7f2cf00a1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0018000 tx=0x7f2ce0004940 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0018000 tx=0x7f2ce0004940 comp rx=0 tx=0).stop 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 msgr2=0x7f2cf00a1db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 0x7f2cd0046770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 msgr2=0x7f2cf00a1db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cf009c290 0x7f2cf00a1db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd0046d80 con 0x7f2cd0044380 2024-09-10T16:07:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 0x7f2cd0046770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient: get_auth_request con 0x7f2cd0044380 auth_method 0 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 10 monclient(hunting): handle_auth_done global_id 14113 payload 824 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.626+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2ce000fd00 con 0x7f2cd0044380 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cf54e7640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 msgr2=0x7f2cd0046770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cf54e7640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 0x7f2cd0046770 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007920 tx=0x7f2ce0006a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0037020 con 0x7f2cd0044380 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.628+0000 7f2cedffb640 10 monclient: dump: 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f2ce002c030 con 0x7f2cd0044380 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 msgr2=0x7f2cd0046770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 0x7f2cd0046770 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0007920 tx=0x7f2ce0006a70 comp rx=0 tx=0).stop 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 msgr2=0x7f2cd0046770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cedffb640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2cd0044380 0x7f2cd0046770 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf5ce8640 1 -- 172.21.15.164:0/2272695717 reap_dead start 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 4102, 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:10.647 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.648 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.649 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:10.650 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:10.651 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:10.652 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:10.653 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:10.654 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:10.655 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:10.657 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:10.658 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.659 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:10.660 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:10.661 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:10.661 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:10.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:10.661 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:10.661 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:10.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 msgr2=0x7f2cd00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 0x7f2cd00406d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2cf00a3ed0 tx=0x7f2cd802e000 comp rx=0 tx=0).stop 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 10 monclient: shutdown 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 msgr2=0x7f2cf013cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800cea0 tx=0x7f2ce800cf90 comp rx=0 tx=0).stop 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 shutdown_connections 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cd003e210 0x7f2cd00406d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 --2- 172.21.15.164:0/2272695717 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2cf013d190 0x7f2cf013cb70 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 >> 172.21.15.164:0/2272695717 conn(0x7f2cf00a3fb0 msgr2=0x7f2cf00a08c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 shutdown_connections 2024-09-10T16:07:10.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.629+0000 7f2cf774b640 1 -- 172.21.15.164:0/2272695717 wait complete. 2024-09-10T16:07:10.664 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:10.664 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:07:10.664 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.760+0000 7f86e0190640 10 monclient: get_monmap_and_config 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.760+0000 7f86e0190640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.760+0000 7f86e0190640 10 monclient: monmap: 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:10.761832+0000 2024-09-10T16:07:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:10.761832+0000 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:10.763 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.761+0000 7f86e0190640 1 Processor -- start 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.761+0000 7f86e0190640 1 -- start start 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.761+0000 7f86e0190640 10 monclient: init 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 0x7f86d806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-b con 0x7f86d80a3e60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f86d806d0b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:10.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f86d813d5f0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86d80a4600 con 0x7f86d813d5f0 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86d806dba0 con 0x7f86d806d0b0 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f86d80a49b0 con 0x7f86d80a3e60 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86e0190640 10 monclient(hunting): authenticate will time out at 809.249265s 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 0x7f86d806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86de72d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f86d813d5f0 0x7f86d813f9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55520/0 (socket says 172.21.15.164:55520) 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86de72d640 1 -- 172.21.15.164:0/3277424469 learned_addr learned my addr 172.21.15.164:0/3277424469 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): get_auth_request con 0x7f86d80a3e60 auth_method 0 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_done global_id 4445 payload 274 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 1 -- 172.21.15.164:0/3277424469 >> v1:172.21.15.165:6789/0 conn(0x7f86d806d0b0 legacy=0x7f86d806d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 1 -- 172.21.15.164:0/3277424469 >> v1:172.21.15.164:6789/0 conn(0x7f86d813d5f0 legacy=0x7f86d813f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 1 monclient: found mon.noname-b 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:10.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.762+0000 7f86ddf2c640 1 -- 172.21.15.164:0/3277424469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d80a5d70 con 0x7f86d80a3e60 2024-09-10T16:07:10.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86ddf2c640 1 -- 172.21.15.164:0/3277424469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 msgr2=0x7f86d806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86ddf2c640 1 --2- 172.21.15.164:0/3277424469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 0x7f86d806cb70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c802a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ff5aad7737111d2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 1 -- 172.21.15.164:0/3277424469 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c8007a30 con 0x7f86d80a3e60 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: dump: 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.773 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86e0190640 5 monclient: authenticate success, global_id 4445 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86e0190640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 1 -- 172.21.15.164:0/3277424469 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86c8007bd0 con 0x7f86d80a3e60 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86e0190640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 1 -- 172.21.15.164:0/3277424469 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86c8031400 con 0x7f86d80a3e60 2024-09-10T16:07:10.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86e0190640 4 set_mon_vals no callback set 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.765+0000 7f86dcf2a640 10 monclient: dump: 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 10 monclient: shutdown 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 -- 172.21.15.164:0/3277424469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 msgr2=0x7f86d806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 --2- 172.21.15.164:0/3277424469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 0x7f86d806cb70 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009d90 tx=0x7f86c802a500 comp rx=0 tx=0).stop 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 -- 172.21.15.164:0/3277424469 shutdown_connections 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 --2- 172.21.15.164:0/3277424469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d80a3e60 0x7f86d806cb70 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 -- 172.21.15.164:0/3277424469 >> 172.21.15.164:0/3277424469 conn(0x7f86d80a4f50 msgr2=0x7f86d80a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 -- 172.21.15.164:0/3277424469 shutdown_connections 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 1 -- 172.21.15.164:0/3277424469 wait complete. 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 10 monclient: build_initial_monmap 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.766+0000 7f86e0190640 10 monclient: monmap: 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:10.776 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 Processor -- start 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 -- start start 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient: init 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:07:10.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-c con 0x7f86d806d0b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86d80a3e60 0x7f86d8073070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-b con 0x7f86d80a3e60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 0x7f86d80735b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): picked mon.noname-a con 0x7f86d813d5f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f86d809bd30 con 0x7f86d813d5f0 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f86d80a4240 con 0x7f86d80a3e60 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f86d8141100 con 0x7f86d806d0b0 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:10.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86e0190640 10 monclient(hunting): authenticate will time out at 809.254560s 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 0x7f86d80735b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33164/0 (socket says 172.21.15.164:33164) 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 1 -- 172.21.15.164:0/1147939232 learned_addr learned my addr 172.21.15.164:0/1147939232 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): get_auth_request con 0x7f86d806d0b0 auth_method 0 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): get_auth_request con 0x7f86d813d5f0 auth_method 0 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86d80a3e60 0x7f86d8073070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): get_auth_request con 0x7f86d80a3e60 auth_method 0 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86dd72b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:10.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient(hunting): handle_auth_done global_id 4443 payload 824 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 -- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 msgr2=0x7f86d809a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 -- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86d80a3e60 msgr2=0x7f86d8073070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86d80a3e60 0x7f86d8073070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient: _finish_hunting 0 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 monclient: found mon.noname-a 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86de72d640 1 -- 172.21.15.164:0/1147939232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86d809c140 con 0x7f86d813d5f0 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.767+0000 7f86ddf2c640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86de72d640 1 -- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 msgr2=0x7f86d80735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86de72d640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 0x7f86d80735b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f86d000c3e0 tx=0x7f86d000c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d0017070 con 0x7f86d813d5f0 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: dump: 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: _finish_auth 0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 5 monclient: authenticate success, global_id 4443 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86d00070d0 con 0x7f86d813d5f0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 10 monclient: _renew_subs 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86d8093f60 con 0x7f86d813d5f0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86d00074c0 con 0x7f86d813d5f0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d67fc640 10 monclient: dump: 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:10.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86d7fff640 4 set_mon_vals no callback set 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 10 monclient: _renew_subs 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.770+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86d8093c80 con 0x7f86d813d5f0 2024-09-10T16:07:10.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f86d001a070 con 0x7f86d813d5f0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86e0190640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86e0190640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86e0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86d806cb70 con 0x7f86d813d5f0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86d67fc640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 0x7f86c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4711+0+0 (secure 0 0 0) 0x7f86d0052090 con 0x7f86d813d5f0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86ddf2c640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 0x7f86c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.771+0000 7f86ddf2c640 10 monclient: get_auth_request con 0x7f86c403e210 auth_method 0 2024-09-10T16:07:10.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.774+0000 7f86ddf2c640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 0x7f86c40406d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009bd0 tx=0x7f86c802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:10.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.775+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f86d809c140 con 0x7f86d813d5f0 2024-09-10T16:07:10.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.776+0000 7f86d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:10.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:10.776+0000 7f86d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:11.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.017+0000 7f86e0190640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.017+0000 7f86e0190640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.017+0000 7f86e0190640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.017+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f86d813f0e0 con 0x7f86d813d5f0 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.018+0000 7f86d67fc640 1 -- 172.21.15.164:0/1147939232 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f86d0050ad0 con 0x7f86d813d5f0 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.018+0000 7f86d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:07:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.018+0000 7f86d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 msgr2=0x7f86c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 0x7f86c40406d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f86c8009bd0 tx=0x7f86c802f000 comp rx=0 tx=0).stop 2024-09-10T16:07:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 10 monclient: shutdown 2024-09-10T16:07:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 msgr2=0x7f86d80735b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 0x7f86d80735b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f86d000c3e0 tx=0x7f86d000c7e0 comp rx=0 tx=0).stop 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 shutdown_connections 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f86c403e210 0x7f86c40406d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f86d813d5f0 0x7f86d80735b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86d80a3e60 0x7f86d8073070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 --2- 172.21.15.164:0/1147939232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f86d806d0b0 0x7f86d809a7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 >> 172.21.15.164:0/1147939232 conn(0x7f86d80a4f50 msgr2=0x7f86d809e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 shutdown_connections 2024-09-10T16:07:11.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:11.020+0000 7f86e0190640 1 -- 172.21.15.164:0/1147939232 wait complete. 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=11,kv=12,logm=68} crc {auth=875599525,config=1747058506,health=2676676013,kv=805391194,logm=2826578418}) 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=11,kv=11,logm=68} crc {auth=3345878838,config=2586702516,health=632208798,kv=3293307581,logm=720246678}) 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=1,health=11,kv=12,logm=68} crc {auth=875599525,config=1747058506,health=2676676013,kv=805391194,logm=2826578418}) 2024-09-10T16:07:11.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.021+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=8,config=1,health=11,kv=12,logm=68} crc {auth=472549471,config=1747058506,health=632208798,kv=805391194,logm=463175128}) 2024-09-10T16:07:11.026 INFO:tasks.rados.rados.0.smithi165.stdout:132: done (0 left) 2024-09-10T16:07:11.026 INFO:tasks.rados.rados.0.smithi165.stdout:133: snap_create 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445272424}) 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3931856277}) 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=445272424}) 2024-09-10T16:07:11.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.024+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3553430550}) 2024-09-10T16:07:11.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:07:11.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.044+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656352733}) 2024-09-10T16:07:11.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116191621}) 2024-09-10T16:07:11.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656352733}) 2024-09-10T16:07:11.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=852609340}) 2024-09-10T16:07:11.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.045+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676070207}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648298364}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=676070207}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752407465}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.046+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.047+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6,osdmap=4} crc {logm=4240922920,mdsmap=931935893,mgr=2441926605,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1562798213,mon_config_key=2447630413,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=97454253,osd_snap=4112931697,osdmap=4018456517}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.047+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4,osdmap=9} crc {logm=564533690,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3531664815,mon_config_key=1707259822,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=97454253,osd_snap=3266179576,osdmap=2449330546}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.047+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.047+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=6,osdmap=4} crc {logm=4240922920,mdsmap=931935893,mgr=2441926605,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1562798213,mon_config_key=2447630413,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=97454253,osd_snap=4112931697,osdmap=4018456517}) 2024-09-10T16:07:11.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.047+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=35,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=5,osdmap=10} crc {logm=1897598904,mdsmap=204415920,mgr=1154067016,mgr_metadata=2169216343,mgrstat=547679778,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=97454253,osd_snap=2317845079,osdmap=3313512977}) 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=4060532289}) 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=80} crc {osdmap=511115876}) 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=76} crc {osdmap=4060532289}) 2024-09-10T16:07:11.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:07:11.147+0000 7f90ac923640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=80} crc {osdmap=927000963}) 2024-09-10T16:07:12.033 INFO:tasks.rados.rados.0.smithi165.stdout:133: done (0 left) 2024-09-10T16:07:12.033 INFO:tasks.rados.rados.0.smithi165.stdout:134: rollback oid 27 current snap is 17 2024-09-10T16:07:12.033 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 9 2024-09-10T16:07:12.033 INFO:tasks.rados.rados.0.smithi165.stdout:135: delete oid 31 current snap is 17 2024-09-10T16:07:12.038 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:07:12.039 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:07:12.039 INFO:tasks.rados.rados.0.smithi165.stdout:134: finishing rollback tid 0 to smithi16534841-27 2024-09-10T16:07:12.040 INFO:tasks.rados.rados.0.smithi165.stdout:134: finishing rollback tid 1 to smithi16534841-27 2024-09-10T16:07:12.041 INFO:tasks.rados.rados.0.smithi165.stdout:135: done (1 left) 2024-09-10T16:07:12.041 INFO:tasks.rados.rados.0.smithi165.stdout:136: write (excl) oid 1 current snap is 17 2024-09-10T16:07:12.041 INFO:tasks.rados.rados.0.smithi165.stdout:136: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-10T16:07:12.047 INFO:tasks.rados.rados.0.smithi165.stdout:136: writing smithi16534841-1 from 510946 to 1143898 tid 1 2024-09-10T16:07:12.054 INFO:tasks.rados.rados.0.smithi165.stdout:136: writing smithi16534841-1 from 1798066 to 2275745 tid 2 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:136: writing smithi16534841-1 from 2742758 to 3199975 tid 3 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:137: read oid 30 snap 10 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:137: expect (ObjNum 58 snap 3 seq_num 58) 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:138: rollback oid 44 current snap is 17 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 8 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:134: finishing rollback tid 2 to smithi16534841-27 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 32 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing write tid 1 to smithi16534841-1 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing write tid 2 to smithi16534841-1 2024-09-10T16:07:12.060 INFO:tasks.rados.rados.0.smithi165.stdout:134: done (3 left) 2024-09-10T16:07:12.061 INFO:tasks.rados.rados.0.smithi165.stdout:139: snap_create 2024-09-10T16:07:12.063 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing write tid 3 to smithi16534841-1 2024-09-10T16:07:12.064 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing write tid 4 to smithi16534841-1 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: finishing write tid 5 to smithi16534841-1 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: oid 1 updating version 0 to 25 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: oid 1 version 25 is already newer than 24 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: oid 1 version 25 is already newer than 23 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: oid 1 version 25 is already newer than 22 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 25 (ObjNum 78 snap 17 seq_num 78) dirty exists 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:136: left oid 1 (ObjNum 78 snap 17 seq_num 78) 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:138: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:138: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:138: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:07:12.065 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 29 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-10T16:07:12.066 INFO:tasks.rados.rados.0.smithi165.stdout:137: expect (ObjNum 58 snap 3 seq_num 58) 2024-09-10T16:07:12.105 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 75481 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.144+0000 7f0480eed640 10 monclient: get_monmap_and_config 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.144+0000 7f0480eed640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.144+0000 7f0480eed640 10 monclient: monmap: 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:12.145693+0000 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:12.145693+0000 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:12.147 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.145+0000 7f0480eed640 1 Processor -- start 2024-09-10T16:07:12.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.145+0000 7f0480eed640 1 -- start start 2024-09-10T16:07:12.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.145+0000 7f0480eed640 10 monclient: init 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.145+0000 7f0480eed640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.145+0000 7f0480eed640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f047c09c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f047c09d190 0x7f047c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-a con 0x7f047c09d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-b con 0x7f047c13a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f047c09fbc0 con 0x7f047c09c310 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f047c0a2a20 con 0x7f047c09d190 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f047c09c6f0 con 0x7f047c13a5c0 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0480eed640 10 monclient(hunting): authenticate will time out at 810.633097s 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f047c09d190 0x7f047c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047a59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f047c09c310 0x7f047c09ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55532/0 (socket says 172.21.15.164:55532) 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047a59c640 1 -- 172.21.15.164:0/1425837996 learned_addr learned my addr 172.21.15.164:0/1425837996 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): get_auth_request con 0x7f047c09d190 auth_method 0 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): get_auth_request con 0x7f047c13a5c0 auth_method 0 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient(hunting): handle_auth_done global_id 4446 payload 274 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 -- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 msgr2=0x7f047c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 -- 172.21.15.164:0/1425837996 >> v1:172.21.15.164:6789/0 conn(0x7f047c09c310 legacy=0x7f047c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 monclient: found mon.noname-a 2024-09-10T16:07:12.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:12.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f0473fff640 1 -- 172.21.15.164:0/1425837996 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047c09a4d0 con 0x7f047c09d190 2024-09-10T16:07:12.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.146+0000 7f047ad9d640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:12.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0473fff640 1 -- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c09d190 msgr2=0x7f047c09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0473fff640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c09d190 0x7f047c09f150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f04640097a0 tx=0x7f0464029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d7dc1d8388be406b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 1 -- 172.21.15.164:0/1425837996 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464037070 con 0x7f047c09d190 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: dump: 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.159 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0480eed640 5 monclient: authenticate success, global_id 4446 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 1 -- 172.21.15.164:0/1425837996 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0464032440 con 0x7f047c09d190 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0480eed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0480eed640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0480eed640 4 set_mon_vals no callback set 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 1 -- 172.21.15.164:0/1425837996 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464032810 con 0x7f047c09d190 2024-09-10T16:07:12.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.149+0000 7f0479d9b640 10 monclient: dump: 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 10 monclient: shutdown 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 -- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c09d190 msgr2=0x7f047c09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c09d190 0x7f047c09f150 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f04640097a0 tx=0x7f0464029fd0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 -- 172.21.15.164:0/1425837996 shutdown_connections 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 --2- 172.21.15.164:0/1425837996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c09d190 0x7f047c09f150 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 -- 172.21.15.164:0/1425837996 >> 172.21.15.164:0/1425837996 conn(0x7f047c0a3110 msgr2=0x7f047c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 -- 172.21.15.164:0/1425837996 shutdown_connections 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 1 -- 172.21.15.164:0/1425837996 wait complete. 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.150+0000 7f0480eed640 10 monclient: monmap: 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:12.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 Processor -- start 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 -- start start 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient: init 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f047c09c310 0x7f047c139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-b con 0x7f047c09c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-c con 0x7f047c13a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): picked mon.noname-a con 0x7f047c06ed10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f047c0a2230 con 0x7f047c06ed10 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f047c0a20b0 con 0x7f047c09c310 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f047c074e00 con 0x7f047c13a5c0 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0480eed640 10 monclient(hunting): authenticate will time out at 810.638790s 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48350/0 (socket says 172.21.15.164:48350) 2024-09-10T16:07:12.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33180/0 (socket says 172.21.15.164:33180) 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 1 -- 172.21.15.164:0/115624721 learned_addr learned my addr 172.21.15.164:0/115624721 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): get_auth_request con 0x7f047c13a5c0 auth_method 0 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 10 monclient(hunting): get_auth_request con 0x7f047c06ed10 auth_method 0 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f0473fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047a59c640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f047c09c310 0x7f047c139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.151+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): get_auth_request con 0x7f047c09c310 auth_method 0 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 10 monclient(hunting): handle_auth_done global_id 4454 payload 824 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 -- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f047c09c310 msgr2=0x7f047c139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f047c09c310 0x7f047c139f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 -- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 msgr2=0x7f047c06f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 monclient: found mon.noname-c 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f0473fff640 1 -- 172.21.15.164:0/115624721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f047c098220 con 0x7f047c13a5c0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.152+0000 7f047ad9d640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0473fff640 1 -- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 msgr2=0x7f047c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0473fff640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0464031000 tx=0x7f04640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464041020 con 0x7f047c13a5c0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: dump: 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f046403b8c0 con 0x7f047c13a5c0 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 5 monclient: authenticate success, global_id 4454 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0464037040 con 0x7f047c13a5c0 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 10 monclient: _renew_subs 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f047c09d620 con 0x7f047c13a5c0 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f047959a640 4 set_mon_vals no callback set 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0472ffd640 10 monclient: dump: 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 10 monclient: _renew_subs 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.154+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f047c096cf0 con 0x7f047c13a5c0 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.155+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0464046400 con 0x7f047c13a5c0 2024-09-10T16:07:12.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.155+0000 7f0480eed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.155+0000 7f0480eed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.155+0000 7f0480eed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.155+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f047c09f7b0 con 0x7f047c13a5c0 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.156+0000 7f0472ffd640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 0x7f045c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.156+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 4739+0+0 (secure 0 0 0) 0x7f0464072ee0 con 0x7f047c13a5c0 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.156+0000 7f047a59c640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 0x7f045c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.156+0000 7f047a59c640 10 monclient: get_auth_request con 0x7f045c03e210 auth_method 0 2024-09-10T16:07:12.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.158+0000 7f047a59c640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 0x7f045c0406d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f046c007c90 tx=0x7f046c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.160+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f046404b060 con 0x7f047c13a5c0 2024-09-10T16:07:12.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.160+0000 7f0472ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.160+0000 7f0472ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:12.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.402+0000 7f0480eed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:07:12.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.402+0000 7f0480eed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:12.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.402+0000 7f0480eed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.402+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f047c08f690 con 0x7f047c13a5c0 2024-09-10T16:07:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.403+0000 7f0472ffd640 1 -- 172.21.15.164:0/115624721 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f047c09fbc0 con 0x7f047c13a5c0 2024-09-10T16:07:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.403+0000 7f0472ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:12.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.403+0000 7f0472ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:12.407 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:07:12.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 msgr2=0x7f045c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 0x7f045c0406d0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f046c007c90 tx=0x7f046c007be0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 10 monclient: shutdown 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 msgr2=0x7f047c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f0464031000 tx=0x7f04640046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 shutdown_connections 2024-09-10T16:07:12.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f045c03e210 0x7f045c0406d0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f047c06ed10 0x7f047c06f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f047c13a5c0 0x7f047c06e7d0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 --2- 172.21.15.164:0/115624721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f047c09c310 0x7f047c139f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 >> 172.21.15.164:0/115624721 conn(0x7f047c0a3110 msgr2=0x7f047c073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 shutdown_connections 2024-09-10T16:07:12.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.405+0000 7f0480eed640 1 -- 172.21.15.164:0/115624721 wait complete. 2024-09-10T16:07:12.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:07:12.417 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:07:12.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:07:12.418 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:07:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.513+0000 7faa9fbbc640 10 monclient: get_monmap_and_config 2024-09-10T16:07:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.513+0000 7faa9fbbc640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.513+0000 7faa9fbbc640 10 monclient: monmap: 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:12.514446+0000 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:12.514446+0000 2024-09-10T16:07:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:12.518 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 Processor -- start 2024-09-10T16:07:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 -- start start 2024-09-10T16:07:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient: init 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa9809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 0x7faa9809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-b con 0x7faa9809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faa9813a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa9809a430 con 0x7faa9809c180 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa980a24e0 con 0x7faa9813a5c0 2024-09-10T16:07:12.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa980a2bf0 con 0x7faa9809d160 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9fbbc640 10 monclient(hunting): authenticate will time out at 811.001888s 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9d958640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7faa9809c180 0x7faa9809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55534/0 (socket says 172.21.15.164:55534) 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9d958640 1 -- 172.21.15.164:0/2616727709 learned_addr learned my addr 172.21.15.164:0/2616727709 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.514+0000 7faa9d157640 1 --2- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 0x7faa9809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): get_auth_request con 0x7faa9809d160 auth_method 0 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient(hunting): handle_auth_done global_id 4457 payload 274 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 1 -- 172.21.15.164:0/2616727709 >> v1:172.21.15.165:6789/0 conn(0x7faa9813a5c0 legacy=0x7faa9813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 1 -- 172.21.15.164:0/2616727709 >> v1:172.21.15.164:6789/0 conn(0x7faa9809c180 legacy=0x7faa9809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 1 monclient: found mon.noname-b 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.515+0000 7faa9d157640 1 -- 172.21.15.164:0/2616727709 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa9809c560 con 0x7faa9809d160 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9d157640 1 -- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 msgr2=0x7faa9809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9d157640 1 --2- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 0x7faa9809f150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa880097a0 tx=0x7faa88029f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17708de5df26102e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 1 -- 172.21.15.164:0/2616727709 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88037070 con 0x7faa9809d160 2024-09-10T16:07:12.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: dump: 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 1 -- 172.21.15.164:0/2616727709 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa88032440 con 0x7faa9809d160 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 5 monclient: authenticate success, global_id 4457 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 1 -- 172.21.15.164:0/2616727709 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa88032810 con 0x7faa9809d160 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 4 set_mon_vals no callback set 2024-09-10T16:07:12.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9c956640 10 monclient: dump: 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.544 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 10 monclient: shutdown 2024-09-10T16:07:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 1 -- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 msgr2=0x7faa9809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 0x7faa9809f150 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa880097a0 tx=0x7faa88029f90 comp rx=0 tx=0).stop 2024-09-10T16:07:12.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 1 -- 172.21.15.164:0/2616727709 shutdown_connections 2024-09-10T16:07:12.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/2616727709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9809d160 0x7faa9809f150 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.518+0000 7faa9fbbc640 1 -- 172.21.15.164:0/2616727709 >> 172.21.15.164:0/2616727709 conn(0x7faa980a3110 msgr2=0x7faa980a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 1 -- 172.21.15.164:0/2616727709 shutdown_connections 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 1 -- 172.21.15.164:0/2616727709 wait complete. 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 10 monclient: monmap: 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 1 Processor -- start 2024-09-10T16:07:12.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 1 -- start start 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.519+0000 7faa9fbbc640 10 monclient: init 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa9809c180 0x7faa981395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-b con 0x7faa9809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 0x7faa98139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-a con 0x7faa9809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 0x7faa9813a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): picked mon.noname-c con 0x7faa9813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa98074f90 con 0x7faa9809d160 2024-09-10T16:07:12.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faa98074e10 con 0x7faa9809c180 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa9809f690 con 0x7faa9813a5c0 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9fbbc640 10 monclient(hunting): authenticate will time out at 811.007414s 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 0x7faa98139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 0x7faa98139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48366/0 (socket says 172.21.15.164:48366) 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 1 -- 172.21.15.164:0/1311626268 learned_addr learned my addr 172.21.15.164:0/1311626268 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): get_auth_request con 0x7faa9809d160 auth_method 0 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 0x7faa9813a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): get_auth_request con 0x7faa9813a5c0 auth_method 0 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa9809c180 0x7faa981395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient(hunting): handle_auth_done global_id 4460 payload 824 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa9809c180 msgr2=0x7faa981395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa9809c180 0x7faa981395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 msgr2=0x7faa98139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 0x7faa98139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 monclient: found mon.noname-c 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa980991c0 con 0x7faa9813a5c0 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d958640 10 monclient: get_auth_request con 0x7faa9809c180 auth_method 0 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa9809c180 0x7faa981395d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.520+0000 7faa9d157640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa9809d160 0x7faa98139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 msgr2=0x7faa9813a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9e159640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 0x7faa9813a050 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c780 tx=0x7faa9000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa90017070 con 0x7faa9813a5c0 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: dump: 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa90003da0 con 0x7faa9813a5c0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 5 monclient: authenticate success, global_id 4460 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 10 monclient: _renew_subs 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa98098590 con 0x7faa9813a5c0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa900052b0 con 0x7faa9813a5c0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8e7fc640 10 monclient: dump: 2024-09-10T16:07:12.554 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 10 monclient: _renew_subs 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa8ffff640 4 set_mon_vals no callback set 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.523+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faa98097d10 con 0x7faa9813a5c0 2024-09-10T16:07:12.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faa9001a070 con 0x7faa9813a5c0 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9fbbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 0x7faa98091a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa98098140 con 0x7faa98091680 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 0x7faa98091a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient: get_auth_request con 0x7faa98091680 auth_method 0 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa8e7fc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa7c03e210 0x7faa7c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(46..46 src has 1..46) ==== 4739+0+0 (secure 0 0 0) 0x7faa90052470 con 0x7faa9813a5c0 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 10 monclient(hunting): handle_auth_done global_id 4455 payload 824 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d958640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa98097ee0 con 0x7faa98091680 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d157640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa7c03e210 0x7faa7c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.524+0000 7faa9d157640 10 monclient: get_auth_request con 0x7faa7c03e210 auth_method 0 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa9d958640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 msgr2=0x7faa98091a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 0x7faa98091a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa9400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa940070f0 con 0x7faa98091680 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.526+0000 7faa8e7fc640 10 monclient: dump: 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.557 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa94017070 con 0x7faa98091680 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 msgr2=0x7faa98091a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 0x7faa98091a60 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa9400d9b0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 msgr2=0x7faa98091a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa8e7fc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091680 0x7faa98091a60 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa9400d9b0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.527+0000 7faa9d157640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa7c03e210 0x7faa7c0406d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa980a5480 tx=0x7faa8802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9fbbc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 0x7faa9808df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9fbbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa9809ea10 con 0x7faa98091fa0 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 0x7faa9808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient: get_auth_request con 0x7faa98091fa0 auth_method 0 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 10 monclient(hunting): handle_auth_done global_id 4458 payload 824 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.533+0000 7faa9d958640 1 -- 172.21.15.164:0/1311626268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa98077260 con 0x7faa98091fa0 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa9d958640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 msgr2=0x7faa9808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa9d958640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 0x7faa9808df80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa94004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa94007790 con 0x7faa98091fa0 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.535+0000 7faa8e7fc640 10 monclient: dump: 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7faa94007790 con 0x7faa98091fa0 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 msgr2=0x7faa9808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 0x7faa9808df80 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa94004940 comp rx=0 tx=0).stop 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 msgr2=0x7faa9808df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:12.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa8e7fc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa98091fa0 0x7faa9808df80 secure :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7faa9400d4f0 tx=0x7faa94004940 comp rx=0 tx=0).stop 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 74710, 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:12.562 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.563 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.564 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:12.566 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.567 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.568 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.569 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.570 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:12.571 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.572 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.573 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.574 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:12.575 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:12.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa7c03e210 msgr2=0x7faa7c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa7c03e210 0x7faa7c0406d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faa980a5480 tx=0x7faa8802e000 comp rx=0 tx=0).stop 2024-09-10T16:07:12.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 10 monclient: shutdown 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 msgr2=0x7faa9813a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 --2- 172.21.15.164:0/1311626268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa9813a5c0 0x7faa9813a050 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7faa9000c780 tx=0x7faa9000cc40 comp rx=0 tx=0).stop 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 reap_dead start 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9e159640 1 -- 172.21.15.164:0/1311626268 reap_dead start 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 shutdown_connections 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 >> 172.21.15.164:0/1311626268 conn(0x7faa980a3110 msgr2=0x7faa980710e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 shutdown_connections 2024-09-10T16:07:12.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.536+0000 7faa9fbbc640 1 -- 172.21.15.164:0/1311626268 wait complete. 2024-09-10T16:07:12.577 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:12.578 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:07:12.578 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.675+0000 7f810688c640 10 monclient: get_monmap_and_config 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.675+0000 7f810688c640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.675+0000 7f810688c640 10 monclient: monmap: 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:12.676536+0000 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:12.676536+0000 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.677 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:12.678 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 Processor -- start 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 -- start start 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient: init 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): picked mon.noname-a con 0x7f810006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810006d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): picked mon.noname-b con 0x7f810009bcc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81000a40d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:12.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f810006e900 con 0x7f81000a40d0 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f81000a4600 con 0x7f810006cb70 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f810006e340 con 0x7f810009bcc0 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.676+0000 7f810688c640 10 monclient(hunting): authenticate will time out at 811.163961s 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810006d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48386/0 (socket says 172.21.15.164:48386) 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 1 -- 172.21.15.164:0/1085295316 learned_addr learned my addr 172.21.15.164:0/1085295316 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request con 0x7f810009bcc0 auth_method 0 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): get_auth_request con 0x7f810006cb70 auth_method 0 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_done global_id 4463 payload 274 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1085295316 >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 msgr2=0x7f810006cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1085295316 >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1085295316 >> v1:172.21.15.165:6789/0 conn(0x7f81000a40d0 legacy=0x7f810013e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 monclient: found mon.noname-b 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1085295316 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81000a01e0 con 0x7f810009bcc0 2024-09-10T16:07:12.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.677+0000 7f80fffff640 1 --2- 172.21.15.164:0/1085295316 >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:12.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1085295316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 msgr2=0x7f810006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1085295316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810006d490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80f40097a0 tx=0x7f80f4029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1314230351bd3ebc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 1 -- 172.21.15.164:0/1085295316 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4037070 con 0x7f810009bcc0 2024-09-10T16:07:12.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: dump: 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 1 -- 172.21.15.164:0/1085295316 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f4032440 con 0x7f810009bcc0 2024-09-10T16:07:12.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 5 monclient: authenticate success, global_id 4463 2024-09-10T16:07:12.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:12.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 1 -- 172.21.15.164:0/1085295316 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4032810 con 0x7f810009bcc0 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 4 set_mon_vals no callback set 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f80feffd640 10 monclient: dump: 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 10 monclient: shutdown 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 -- 172.21.15.164:0/1085295316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 msgr2=0x7f810006d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 --2- 172.21.15.164:0/1085295316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810006d490 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f80f40097a0 tx=0x7f80f4029fd0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 -- 172.21.15.164:0/1085295316 shutdown_connections 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 --2- 172.21.15.164:0/1085295316 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810006d490 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 --2- 172.21.15.164:0/1085295316 >> v2:172.21.15.164:3300/0 conn(0x7f810006cb70 0x7f810006cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 -- 172.21.15.164:0/1085295316 >> 172.21.15.164:0/1085295316 conn(0x7f81000a4f50 msgr2=0x7f81000a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 -- 172.21.15.164:0/1085295316 shutdown_connections 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.680+0000 7f810688c640 1 -- 172.21.15.164:0/1085295316 wait complete. 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.681+0000 7f810688c640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.681+0000 7f810688c640 10 monclient: monmap: 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.681+0000 7f810688c640 1 Processor -- start 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.681+0000 7f810688c640 1 -- start start 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.681+0000 7f810688c640 10 monclient: init 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 0x7f810013a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): picked mon.noname-a con 0x7f810006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000a40d0 0x7f810013ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): picked mon.noname-c con 0x7f81000a40d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f810013cbf0 0x7f810013d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): picked mon.noname-b con 0x7f810013cbf0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f810013fe90 con 0x7f810006cb70 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f810013fd10 con 0x7f810013cbf0 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8100140010 con 0x7f81000a40d0 2024-09-10T16:07:12.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f810688c640 10 monclient(hunting): authenticate will time out at 811.169401s 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 0x7f810013a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 0x7f810013a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48400/0 (socket says 172.21.15.164:48400) 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000a40d0 0x7f810013ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 -- 172.21.15.164:0/1767262850 learned_addr learned my addr 172.21.15.164:0/1767262850 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): get_auth_request con 0x7f810006cb70 auth_method 0 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request con 0x7f81000a40d0 auth_method 0 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000a40d0 msgr2=0x7f810013ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81000a40d0 0x7f810013ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f810013cbf0 msgr2=0x7f810013d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f8104e29640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f810013cbf0 0x7f810013d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f810013cbf0 0x7f810013d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 monclient: found mon.noname-a 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f80fffff640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81000a6330 con 0x7f810006cb70 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f8104e29640 10 monclient: get_auth_request con 0x7f810013cbf0 auth_method 0 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.682+0000 7f8104e29640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f810013cbf0 0x7f810013d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fffff640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 msgr2=0x7f810013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fffff640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 0x7f810013a850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e9d0 tx=0x7f80f000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0017070 con 0x7f810006cb70 2024-09-10T16:07:12.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: dump: 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80f000cc70 con 0x7f810006cb70 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 5 monclient: authenticate success, global_id 4464 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 10 monclient: _renew_subs 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f0013670 con 0x7f810006cb70 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8100093120 con 0x7f810006cb70 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fcff9640 10 monclient: dump: 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.706 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f80fe7fc640 4 set_mon_vals no callback set 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 10 monclient: _renew_subs 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.685+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f810013b2d0 con 0x7f810006cb70 2024-09-10T16:07:12.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f810688c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f810688c640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810013d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f810009c140 con 0x7f810009bcc0 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f80f001a070 con 0x7f810006cb70 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810013d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient: get_auth_request con 0x7f810009bcc0 auth_method 0 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80fcff9640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80ec03e210 0x7f80ec0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 4739+0+0 (secure 0 0 0) 0x7f80f0052cc0 con 0x7f810006cb70 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_done global_id 4469 payload 824 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81000a6760 con 0x7f810009bcc0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f8104e29640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80ec03e210 0x7f80ec0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.686+0000 7f8104e29640 10 monclient: get_auth_request con 0x7f80ec03e210 auth_method 0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.688+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 msgr2=0x7f810013d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.688+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810013d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80f4034000 tx=0x7f80f40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4043070 con 0x7f810009bcc0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 10 monclient: dump: 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f8104e29640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80ec03e210 0x7f80ec0406d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80f80044a0 tx=0x7f80f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80f403c850 con 0x7f810009bcc0 2024-09-10T16:07:12.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 msgr2=0x7f810013d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810013d580 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80f4034000 tx=0x7f80f40046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 msgr2=0x7f810013d580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:12.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.689+0000 7f80fcff9640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810009bcc0 0x7f810013d580 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f80f4034000 tx=0x7f80f40046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f810688c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f810688c640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 0x7f8100072160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f810688c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8100092cf0 con 0x7f810013dac0 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 0x7f8100072160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient: get_auth_request con 0x7f810013dac0 auth_method 0 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 10 monclient(hunting): handle_auth_done global_id 4472 payload 824 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.695+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1767262850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f810006cf50 con 0x7f810013dac0 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.697+0000 7f80ff7fe640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 msgr2=0x7f8100072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.697+0000 7f80ff7fe640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 0x7f8100072160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80f402a010 tx=0x7f80f4009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80f4037040 con 0x7f810013dac0 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 10 monclient: dump: 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f80f4004030 con 0x7f810013dac0 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 msgr2=0x7f8100072160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 0x7f8100072160 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80f402a010 tx=0x7f80f4009190 comp rx=0 tx=0).stop 2024-09-10T16:07:12.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 msgr2=0x7f8100072160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f80fcff9640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f810013dac0 0x7f8100072160 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f80f402a010 tx=0x7f80f4009190 comp rx=0 tx=0).stop 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 74870, 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:12.716 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.717 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.718 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:12.719 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:12.720 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:12.721 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.724 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:07:12.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:12.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:12.729 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:12.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:12.729 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:12.729 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80ec03e210 msgr2=0x7f80ec0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80ec03e210 0x7f80ec0406d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f80f80044a0 tx=0x7f80f8009040 comp rx=0 tx=0).stop 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 10 monclient: shutdown 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 msgr2=0x7f810013a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 --2- 172.21.15.164:0/1767262850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f810006cb70 0x7f810013a850 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f80f000e9d0 tx=0x7f80f000ee90 comp rx=0 tx=0).stop 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f8104e29640 1 -- 172.21.15.164:0/1767262850 reap_dead start 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f8104e29640 1 -- 172.21.15.164:0/1767262850 reap_dead start 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 shutdown_connections 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 >> 172.21.15.164:0/1767262850 conn(0x7f81000a4f50 msgr2=0x7f810009e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 shutdown_connections 2024-09-10T16:07:12.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.698+0000 7f810688c640 1 -- 172.21.15.164:0/1767262850 wait complete. 2024-09-10T16:07:12.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:12.731 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:07:12.732 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 10 monclient: get_monmap_and_config 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 10 monclient: monmap: 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:12.830070+0000 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:12.830070+0000 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 1 Processor -- start 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 1 -- start start 2024-09-10T16:07:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.829+0000 7f0235079640 10 monclient: init 2024-09-10T16:07:12.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T16:07:12.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0230135ce0 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:12.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f02300a40d0 0x7f02301360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): picked mon.noname-c con 0x7f02300a40d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 0x7f023013d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): picked mon.noname-b con 0x7f0230136600 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02300a4ac0 con 0x7f0230135ce0 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f02300a1400 con 0x7f02300a40d0 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f02300a1120 con 0x7f0230136600 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f0235079640 10 monclient(hunting): authenticate will time out at 811.317463s 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 0x7f023013d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022ed9d640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f0230135ce0 0x7f02300a23a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:40596/0 (socket says 172.21.15.164:40596) 2024-09-10T16:07:12.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022ed9d640 1 -- 172.21.15.164:0/3947737626 learned_addr learned my addr 172.21.15.164:0/3947737626 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 1 --2- 172.21.15.164:0/3947737626 >> v2:172.21.15.165:3300/0 conn(0x7f02300a40d0 0x7f02301360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): get_auth_request con 0x7f0230136600 auth_method 0 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): get_auth_request con 0x7f02300a40d0 auth_method 0 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient(hunting): handle_auth_done global_id 4475 payload 274 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 -- 172.21.15.164:0/3947737626 >> v2:172.21.15.165:3300/0 conn(0x7f02300a40d0 msgr2=0x7f02301360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 --2- 172.21.15.164:0/3947737626 >> v2:172.21.15.165:3300/0 conn(0x7f02300a40d0 0x7f02301360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 -- 172.21.15.164:0/3947737626 >> v1:172.21.15.165:6789/0 conn(0x7f0230135ce0 legacy=0x7f02300a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 monclient: found mon.noname-b 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.830+0000 7f022f59e640 1 -- 172.21.15.164:0/3947737626 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0230136a50 con 0x7f0230136600 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022f59e640 1 -- 172.21.15.164:0/3947737626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 msgr2=0x7f023013d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022f59e640 1 --2- 172.21.15.164:0/3947737626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 0x7f023013d290 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0228009d90 tx=0x7f022802a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91ffa718866a37e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 1 -- 172.21.15.164:0/3947737626 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228007950 con 0x7f0230136600 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: dump: 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 1 -- 172.21.15.164:0/3947737626 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0228007af0 con 0x7f0230136600 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 1 -- 172.21.15.164:0/3947737626 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0228031400 con 0x7f0230136600 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f0235079640 5 monclient: authenticate success, global_id 4475 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f0235079640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f0235079640 4 set_mon_vals no callback set 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.833+0000 7f022dd9b640 10 monclient: dump: 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 10 monclient: shutdown 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 -- 172.21.15.164:0/3947737626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 msgr2=0x7f023013d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 --2- 172.21.15.164:0/3947737626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 0x7f023013d290 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0228009d90 tx=0x7f022802a490 comp rx=0 tx=0).stop 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 -- 172.21.15.164:0/3947737626 shutdown_connections 2024-09-10T16:07:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 --2- 172.21.15.164:0/3947737626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230136600 0x7f023013d290 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 --2- 172.21.15.164:0/3947737626 >> v2:172.21.15.165:3300/0 conn(0x7f02300a40d0 0x7f02301360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 -- 172.21.15.164:0/3947737626 >> 172.21.15.164:0/3947737626 conn(0x7f02300a4f50 msgr2=0x7f02300a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 -- 172.21.15.164:0/3947737626 shutdown_connections 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.834+0000 7f0235079640 1 -- 172.21.15.164:0/3947737626 wait complete. 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.835+0000 7f0235079640 10 monclient: build_initial_monmap 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.835+0000 7f0235079640 10 monclient: monmap: 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:12.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:136: done (3 left) 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:137: done (2 left) 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:138: done (1 left) 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:139: done (0 left) 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:140: write (excl) oid 5 current snap is 18 2024-09-10T16:07:12.845 INFO:tasks.rados.rados.0.smithi165.stdout:140: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2024-09-10T16:07:12.849 INFO:tasks.rados.rados.0.smithi165.stdout:140: writing smithi16534841-5 from 527753 to 1235954 tid 1 2024-09-10T16:07:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.835+0000 7f0235079640 1 Processor -- start 2024-09-10T16:07:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.835+0000 7f0235079640 1 -- start start 2024-09-10T16:07:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.835+0000 7f0235079640 10 monclient: init 2024-09-10T16:07:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02300a40d0 0x7f023012d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): picked mon.noname-b con 0x7f02300a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 0x7f023012da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): picked mon.noname-c con 0x7f0230135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0230136600 0x7f023012df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): picked mon.noname-a con 0x7f0230136600 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f023013db60 con 0x7f0230136600 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f023013d9e0 con 0x7f02300a40d0 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f023013dce0 con 0x7f0230135ce0 2024-09-10T16:07:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f0235079640 10 monclient(hunting): authenticate will time out at 811.323236s 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 0x7f023012da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0230136600 0x7f023012df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 0x7f023012da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33276/0 (socket says 172.21.15.164:33276) 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 -- 172.21.15.164:0/1934939365 learned_addr learned my addr 172.21.15.164:0/1934939365 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): get_auth_request con 0x7f0230135ce0 auth_method 0 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): get_auth_request con 0x7f0230136600 auth_method 0 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02300a40d0 0x7f023012d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): get_auth_request con 0x7f02300a40d0 auth_method 0 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient(hunting): handle_auth_done global_id 4478 payload 824 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02300a40d0 msgr2=0x7f023012d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02300a40d0 0x7f023012d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0230136600 msgr2=0x7f023012df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0230136600 0x7f023012df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 monclient: found mon.noname-c 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022e59c640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023012fc60 con 0x7f0230135ce0 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.836+0000 7f022f59e640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0230136600 0x7f023012df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:07:12.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.838+0000 7f022e59c640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 msgr2=0x7f023012da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.838+0000 7f022e59c640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 0x7f023012da50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f021800e9b0 tx=0x7f021800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218017070 con 0x7f0230135ce0 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: dump: 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.855 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: _finish_auth 0 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: _check_auth_tickets 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f021800cbb0 con 0x7f0230135ce0 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 5 monclient: authenticate success, global_id 4478 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 10 monclient: _renew_subs 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02300a5560 con 0x7f0230135ce0 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0218013470 con 0x7f0230135ce0 2024-09-10T16:07:12.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f02277fe640 10 monclient: dump: 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 10 monclient: _renew_subs 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f023012f980 con 0x7f0230135ce0 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.839+0000 7f022d59a640 4 set_mon_vals no callback set 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f0235079640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f0235079640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 0x7f023009b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0230075000 con 0x7f023009b050 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f021801a070 con 0x7f0230135ce0 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 0x7f023009b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient: get_auth_request con 0x7f023009b050 auth_method 0 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f02277fe640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f021003e290 0x7f0210040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 10 monclient(hunting): handle_auth_done global_id 14128 payload 824 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022ed9d640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02301360c0 con 0x7f023009b050 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022f59e640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f021003e290 0x7f0210040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.840+0000 7f022f59e640 10 monclient: get_auth_request con 0x7f021003e290 auth_method 0 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f022ed9d640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 msgr2=0x7f023009b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 0x7f023009b430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f022000ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022001a070 con 0x7f023009b050 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:12.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 10 monclient: dump: 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f022f59e640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f021003e290 0x7f0210040750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0230070570 tx=0x7f022802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f022000cc50 con 0x7f023009b050 2024-09-10T16:07:12.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 msgr2=0x7f023009b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 0x7f023009b430 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f022000ef30 comp rx=0 tx=0).stop 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 msgr2=0x7f023009b430 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.843+0000 7f02277fe640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b050 0x7f023009b430 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f022000ef30 comp rx=0 tx=0).stop 2024-09-10T16:07:12.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.845+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(47..47 src has 1..47) ==== 4739+0+0 (secure 0 0 0) 0x7f0218023d00 con 0x7f0230135ce0 2024-09-10T16:07:12.862 INFO:tasks.rados.rados.0.smithi165.stdout:140: writing smithi16534841-5 from 1856548 to 2594238 tid 2 2024-09-10T16:07:12.862 INFO:tasks.rados.rados.0.smithi165.stdout:140: writing smithi16534841-5 from 3286534 to 3327753 tid 3 2024-09-10T16:07:12.862 INFO:tasks.rados.rados.0.smithi165.stdout:141: snap_remove snap 17 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f0235079640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f0235079640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f0235079640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f02300770e0 con 0x7f023009b970 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient: get_auth_request con 0x7f023009b970 auth_method 0 2024-09-10T16:07:12.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.850+0000 7f022ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.851+0000 7f022ed9d640 10 monclient(hunting): handle_auth_done global_id 14131 payload 824 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.851+0000 7f022ed9d640 1 -- 172.21.15.164:0/1934939365 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f023009c080 con 0x7f023009b970 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.851+0000 7f022ed9d640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 msgr2=0x7f023009e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.851+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f0220004940 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-10T16:07:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:12.851+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: get_auth_request con 0x7f023009b970 auth_method 0 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: _init_auth method 2 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: _init_auth already have auth, reseting 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T16:07:13.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.053+0000 7f022ed9d640 10 monclient: handle_auth_done global_id 14131 payload 293 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f022ed9d640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 msgr2=0x7f023009e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f022ed9d640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f0220014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f022001a070 con 0x7f023009b970 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:13.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 10 monclient: dump: 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:13.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f022001b040 con 0x7f023009b970 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 msgr2=0x7f023009e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f022000ea70 tx=0x7f0220014000 comp rx=0 tx=0).stop 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 msgr2=0x7f023009e360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:13.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.056+0000 7f02277fe640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f023009b970 0x7f023009e360 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 14, 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 6530, 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:13.060 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:13.061 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:13.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:13.063 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:13.064 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:13.065 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.067 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.068 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:13.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:13.070 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:13.071 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:13.072 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:13.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f021003e290 msgr2=0x7f0210040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f021003e290 0x7f0210040750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0230070570 tx=0x7f022802f000 comp rx=0 tx=0).stop 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 10 monclient: shutdown 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 msgr2=0x7f023012da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 --2- 172.21.15.164:0/1934939365 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0230135ce0 0x7f023012da50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f021800e9b0 tx=0x7f021800ee70 comp rx=0 tx=0).stop 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f022f59e640 1 -- 172.21.15.164:0/1934939365 reap_dead start 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f022f59e640 1 -- 172.21.15.164:0/1934939365 reap_dead start 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 shutdown_connections 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 >> 172.21.15.164:0/1934939365 conn(0x7f02300a4f50 msgr2=0x7f02300a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 shutdown_connections 2024-09-10T16:07:13.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:13.057+0000 7f0235079640 1 -- 172.21.15.164:0/1934939365 wait complete. 2024-09-10T16:07:13.075 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:07:13.075 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:07:13.075 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-10T16:07:13.075 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:07:13.075 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:07:13.075 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:07:13.075 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:07:13.087 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:07:13.087 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-10T16:07:13.087 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-10T16:07:13.087 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:07:28.088 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-10T16:07:28.089 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-10T16:07:28.089 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:07:28.090 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:07:28.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.197+0000 7f4f37ab1640 10 monclient: get_monmap_and_config 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.197+0000 7f4f37ab1640 10 monclient: build_initial_monmap 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.197+0000 7f4f37ab1640 10 monclient: monmap: 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:28.198287+0000 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:28.198287+0000 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:28.199 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:28.200 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:28.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.197+0000 7f4f37ab1640 1 Processor -- start 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 1 -- start start 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient: init 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f3009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f3009ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f300a0870 con 0x7f4f3009d930 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f30074c70 con 0x7f4f3009ab50 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3009d3e0 con 0x7f4f3009e420 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:28.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f37ab1640 10 monclient(hunting): authenticate will time out at 826.685725s 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.198+0000 7f4f3584d640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f4f3009d930 legacy=0x7f4f3009dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3504c640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f4f3009ab50 0x7f4f301397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:59740/0 (socket says 172.21.15.164:59740) 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 learned_addr learned my addr 172.21.15.164:0/453922831 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 10 monclient(hunting): get_auth_request con 0x7f4f3009e420 auth_method 0 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:07:28.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:28.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:28.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.199+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:28.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.201+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.165:6789/0 conn(0x7f4f3009ab50 legacy=0x7f4f301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:28.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.399+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.164:6789/0 conn(0x7f4f3009d930 legacy=0x7f4f3009dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.400+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 10 monclient(hunting): get_auth_request con 0x7f4f3009e420 auth_method 0 2024-09-10T16:07:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:28.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:28.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:28.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:28.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:28.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.401+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:07:28.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.800+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.164:6789/0 conn(0x7f4f3009d930 legacy=0x7f4f3009dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:28.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:28.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 10 monclient(hunting): get_auth_request con 0x7f4f3009e420 auth_method 0 2024-09-10T16:07:28.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:28.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:28.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:28.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:28.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:28.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:28.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:28.803+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:07:29.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.601+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.164:6789/0 conn(0x7f4f3009d930 legacy=0x7f4f3009dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 10 monclient(hunting): get_auth_request con 0x7f4f3009e420 auth_method 0 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:29.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:29.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:29.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:29.604+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:07:31.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): tick 2024-09-10T16:07:31.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:07:31.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 msgr2=0x7f4f3013cef0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f3009e420 0x7f4f3013cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.165:6789/0 conn(0x7f4f3009ab50 legacy=0x7f4f301397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 >> v1:172.21.15.164:6789/0 conn(0x7f4f3009d930 legacy=0x7f4f3009dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient: _add_conns ranks=[2,3] 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4f1c005930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4f1c008460 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3009d3e0 con 0x7f4f1c008460 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:31.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1c00afc0 con 0x7f4f1c005930 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f2f7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 msgr2=0x7f4f1c00a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 10 monclient(hunting): get_auth_request con 0x7f4f1c005930 auth_method 0 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:31.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.199+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:31.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 msgr2=0x7f4f1c00a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:31.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:07:31.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3584d640 10 monclient(hunting): get_auth_request con 0x7f4f1c005930 auth_method 0 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3584d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3584d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.400+0000 7f4f3584d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.401+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.401+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.401+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:31.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.401+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:07:31.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.801+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 msgr2=0x7f4f1c00a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:31.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.801+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:07:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.802+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.802+0000 7f4f3584d640 10 monclient(hunting): get_auth_request con 0x7f4f1c005930 auth_method 0 2024-09-10T16:07:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.802+0000 7f4f3584d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.802+0000 7f4f3584d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:31.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.802+0000 7f4f3584d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:31.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.803+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:31.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.803+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:31.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.803+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:31.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:31.803+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:07:32.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.602+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 msgr2=0x7f4f1c00a900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:32.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.602+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 10 monclient(hunting): get_auth_request con 0x7f4f1c005930 auth_method 0 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:32.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:32.605+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:07:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:141: done (1 left) 2024-09-10T16:07:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:142: read oid 20 snap -1 2024-09-10T16:07:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:142: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-10T16:07:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:143: write oid 4 current snap is 18 2024-09-10T16:07:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:140: finishing write tid 1 to smithi16534841-5 2024-09-10T16:07:33.990 INFO:tasks.rados.rados.0.smithi165.stdout:143: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-10T16:07:33.996 INFO:tasks.rados.rados.0.smithi165.stdout:143: writing smithi16534841-4 from 544560 to 1328010 tid 1 2024-09-10T16:07:34.002 INFO:tasks.rados.rados.0.smithi165.stdout:143: writing smithi16534841-4 from 1915030 to 2396378 tid 2 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:143: writing smithi16534841-4 from 2797604 to 3277365 tid 3 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:144: rollback oid 41 current snap is 18 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 12 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:140: finishing write tid 2 to smithi16534841-5 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:140: finishing write tid 3 to smithi16534841-5 2024-09-10T16:07:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:140: finishing write tid 4 to smithi16534841-5 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: finishing write tid 5 to smithi16534841-5 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: oid 5 updating version 0 to 41 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: oid 5 version 41 is already newer than 39 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: oid 5 version 41 is already newer than 40 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: oid 5 updating version 41 to 42 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 42 (ObjNum 79 snap 18 seq_num 79) dirty exists 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:140: left oid 5 (ObjNum 79 snap 18 seq_num 79) 2024-09-10T16:07:34.009 INFO:tasks.rados.rados.0.smithi165.stdout:142: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-10T16:07:34.011 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1591231 2024-09-10T16:07:34.094 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:34.143 INFO:tasks.rados.rados.0.smithi165.stdout:143: finishing write tid 1 to smithi16534841-4 2024-09-10T16:07:34.143 INFO:tasks.rados.rados.0.smithi165.stdout:143: finishing write tid 2 to smithi16534841-4 2024-09-10T16:07:34.143 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 0 to smithi16534841-41 2024-09-10T16:07:34.143 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:07:34.143 INFO:tasks.rados.rados.0.smithi165.stdout:144: finishing rollback tid 2 to smithi16534841-41 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 29 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: finishing write tid 3 to smithi16534841-4 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: finishing write tid 4 to smithi16534841-4 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: finishing write tid 5 to smithi16534841-4 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: oid 4 updating version 0 to 20 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: oid 4 version 20 is already newer than 18 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: oid 4 version 20 is already newer than 17 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: oid 4 version 20 is already newer than 19 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 20 (ObjNum 80 snap 18 seq_num 80) dirty exists 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: left oid 4 (ObjNum 80 snap 18 seq_num 80) 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:140: done (3 left) 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:142: done (2 left) 2024-09-10T16:07:34.144 INFO:tasks.rados.rados.0.smithi165.stdout:143: done (1 left) 2024-09-10T16:07:34.145 INFO:tasks.rados.rados.0.smithi165.stdout:144: done (0 left) 2024-09-10T16:07:34.145 INFO:tasks.rados.rados.0.smithi165.stdout:145: write oid 26 current snap is 18 2024-09-10T16:07:34.145 INFO:tasks.rados.rados.0.smithi165.stdout:145: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-10T16:07:34.148 INFO:tasks.rados.rados.0.smithi165.stdout:145: writing smithi16534841-26 from 561367 to 1020066 tid 1 2024-09-10T16:07:34.155 INFO:tasks.rados.rados.0.smithi165.stdout:145: writing smithi16534841-26 from 1573512 to 2314871 tid 2 2024-09-10T16:07:34.161 INFO:tasks.rados.rados.0.smithi165.stdout:145: writing smithi16534841-26 from 2941380 to 3361367 tid 3 2024-09-10T16:07:34.161 INFO:tasks.rados.rados.0.smithi165.stdout:146: write (excl) oid 31 current snap is 18 2024-09-10T16:07:34.161 INFO:tasks.rados.rados.0.smithi165.stdout:146: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-10T16:07:34.167 INFO:tasks.rados.rados.0.smithi165.stdout:146: writing smithi16534841-31 from 578174 to 1112122 tid 1 2024-09-10T16:07:34.172 INFO:tasks.rados.rados.0.smithi165.stdout:146: writing smithi16534841-31 from 1515641 to 2000658 tid 2 2024-09-10T16:07:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:146: writing smithi16534841-31 from 2736097 to 3238402 tid 3 2024-09-10T16:07:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:147: read oid 10 snap -1 2024-09-10T16:07:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:147: expect (ObjNum 63 snap 5 seq_num 63) 2024-09-10T16:07:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:148: read oid 38 snap 9 2024-09-10T16:07:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:148: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-10T16:07:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:149: read oid 34 snap -1 2024-09-10T16:07:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:149: expect deleted 2024-09-10T16:07:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:150: write (excl) oid 47 current snap is 18 2024-09-10T16:07:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:150: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2024-09-10T16:07:34.186 INFO:tasks.rados.rados.0.smithi165.stdout:150: writing smithi16534841-47 from 594981 to 1204178 tid 1 2024-09-10T16:07:34.194 INFO:tasks.rados.rados.0.smithi165.stdout:150: writing smithi16534841-47 from 1974123 to 2719151 tid 2 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:150: writing smithi16534841-47 from 3279873 to 3394981 tid 3 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:151: read oid 28 snap 5 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:151: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:152: snap_create 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:145: finishing write tid 1 to smithi16534841-26 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:145: finishing write tid 2 to smithi16534841-26 2024-09-10T16:07:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:145: finishing write tid 3 to smithi16534841-26 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: finishing write tid 4 to smithi16534841-26 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: finishing write tid 5 to smithi16534841-26 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: oid 26 updating version 0 to 52 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: oid 26 version 52 is already newer than 51 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: oid 26 updating version 52 to 53 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: oid 26 version 53 is already newer than 50 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 53 (ObjNum 81 snap 18 seq_num 81) dirty exists 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:145: left oid 26 (ObjNum 81 snap 18 seq_num 81) 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:146: finishing write tid 1 to smithi16534841-31 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:146: finishing write tid 2 to smithi16534841-31 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:146: finishing write tid 3 to smithi16534841-31 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:146: finishing write tid 4 to smithi16534841-31 2024-09-10T16:07:34.200 INFO:tasks.rados.rados.0.smithi165.stdout:146: finishing write tid 5 to smithi16534841-31 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:146: oid 31 updating version 0 to 55 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:146: oid 31 updating version 55 to 56 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:146: oid 31 version 56 is already newer than 54 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:146: oid 31 updating version 56 to 57 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 57 (ObjNum 82 snap 18 seq_num 82) dirty exists 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:146: left oid 31 (ObjNum 82 snap 18 seq_num 82) 2024-09-10T16:07:34.201 INFO:tasks.rados.rados.0.smithi165.stdout:147: expect (ObjNum 63 snap 5 seq_num 63) 2024-09-10T16:07:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): tick 2024-09-10T16:07:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:07:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:07:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:07:34.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 msgr2=0x7f4f1c007d80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c005930 0x7f4f1c007d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 msgr2=0x7f4f1c00a900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c008460 0x7f4f1c00a900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 reap_dead start 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f1c005d70 0x7f4f1c0088f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4f1c005d70 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 0x7f4f1c009230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4f1c008e30 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 0x7f4f1c009c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4f1c0097a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4f3009d3e0 con 0x7f4f1c0097a0 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4f1c00afc0 con 0x7f4f1c005d70 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 1 -- 172.21.15.164:0/453922831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f1c00a390 con 0x7f4f1c008e30 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f2f7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3604e640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 msgr2=0x7f4f1c009c50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3604e640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 0x7f4f1c009c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 0x7f4f1c009230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): get_auth_request con 0x7f4f1c008e30 auth_method 0 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3584d640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f1c005d70 0x7f4f1c0088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3584d640 10 monclient(hunting): get_auth_request con 0x7f4f1c005d70 auth_method 0 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3584d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3584d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3584d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient(hunting): handle_auth_done global_id 4481 payload 274 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f1c005d70 msgr2=0x7f4f1c0088f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f1c005d70 0x7f4f1c0088f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 msgr2=0x7f4f1c009c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 0x7f4f1c009c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 monclient: found mon.noname-b 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.199+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f3009ecb0 con 0x7f4f1c008e30 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.202+0000 7f4f3504c640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 msgr2=0x7f4f1c009230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.202+0000 7f4f3504c640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 0x7f4f1c009230 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0083d0 tx=0x7f4f2402a4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ba04b6342eb7c51 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.202+0000 7f4f3484b640 1 -- 172.21.15.164:0/453922831 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24004aa0 con 0x7f4f1c008e30 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: dump: 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.213 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 1 -- 172.21.15.164:0/453922831 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f24004c40 con 0x7f4f1c008e30 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 5 monclient: authenticate success, global_id 4481 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 4 set_mon_vals no callback set 2024-09-10T16:07:34.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 1 -- 172.21.15.164:0/453922831 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f24007070 con 0x7f4f1c008e30 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f3484b640 10 monclient: dump: 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 10 monclient: shutdown 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 -- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 msgr2=0x7f4f1c009230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 0x7f4f1c009230 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0083d0 tx=0x7f4f2402a4f0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 -- 172.21.15.164:0/453922831 shutdown_connections 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.164:3300/0 conn(0x7f4f1c0097a0 0x7f4f1c009c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/453922831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c008e30 0x7f4f1c009230 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/453922831 >> v2:172.21.15.165:3300/0 conn(0x7f4f1c005d70 0x7f4f1c0088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.203+0000 7f4f37ab1640 1 -- 172.21.15.164:0/453922831 >> 172.21.15.164:0/453922831 conn(0x7f4f300a3110 msgr2=0x7f4f300a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.204+0000 7f4f37ab1640 1 -- 172.21.15.164:0/453922831 shutdown_connections 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.204+0000 7f4f37ab1640 1 -- 172.21.15.164:0/453922831 wait complete. 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.204+0000 7f4f37ab1640 10 monclient: build_initial_monmap 2024-09-10T16:07:34.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.204+0000 7f4f37ab1640 10 monclient: monmap: 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 Processor -- start 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 -- start start 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient: init 2024-09-10T16:07:34.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 0x7f4f3012fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-a con 0x7f4f1c005d70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f1c008e30 0x7f4f3012db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-b con 0x7f4f1c008e30 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): picked mon.noname-c con 0x7f4f1c0097a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3584d640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 msgr2=0x7f4f3012fff0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4f3009ef30 con 0x7f4f1c005d70 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4f300a0090 con 0x7f4f1c008e30 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f300a2940 con 0x7f4f1c0097a0 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3584d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 0x7f4f3012fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f37ab1640 10 monclient(hunting): authenticate will time out at 832.692793s 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:42494/0 (socket says 172.21.15.164:42494) 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 1 -- 172.21.15.164:0/3035217734 learned_addr learned my addr 172.21.15.164:0/3035217734 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): get_auth_request con 0x7f4f1c0097a0 auth_method 0 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3504c640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f1c008e30 0x7f4f3012db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3504c640 10 monclient(hunting): get_auth_request con 0x7f4f1c008e30 auth_method 0 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3504c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3504c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3504c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.205+0000 7f4f3604e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3504c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 10 monclient(hunting): handle_auth_done global_id 4484 payload 824 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 -- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f1c008e30 msgr2=0x7f4f3012db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f1c008e30 0x7f4f3012db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 -- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 msgr2=0x7f4f3012fff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 0x7f4f3012fff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 monclient: found mon.noname-c 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.206+0000 7f4f3604e640 1 -- 172.21.15.164:0/3035217734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f301317f0 con 0x7f4f1c0097a0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f3604e640 1 -- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 msgr2=0x7f4f3012e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f3604e640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00af60 tx=0x7f4f20008dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2001a020 con 0x7f4f1c0097a0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: dump: 2024-09-10T16:07:34.221 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f2000b530 con 0x7f4f1c0097a0 2024-09-10T16:07:34.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 5 monclient: authenticate success, global_id 4484 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 10 monclient: _renew_subs 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f3012ee30 con 0x7f4f1c0097a0 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f20007a00 con 0x7f4f1c0097a0 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2e7fc640 10 monclient: dump: 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.223 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 10 monclient: _renew_subs 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f3012f490 con 0x7f4f1c0097a0 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.208+0000 7f4f2ffff640 4 set_mon_vals no callback set 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.209+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4f2001d020 con 0x7f4f1c0097a0 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.209+0000 7f4f37ab1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.209+0000 7f4f37ab1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.209+0000 7f4f37ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.209+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f3009dda0 con 0x7f4f1c0097a0 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.210+0000 7f4f2e7fc640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 0x7f4f1c0446c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.210+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 4783+0+0 (secure 0 0 0) 0x7f4f20053270 con 0x7f4f1c0097a0 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.210+0000 7f4f3584d640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 0x7f4f1c0446c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.210+0000 7f4f3584d640 10 monclient: get_auth_request con 0x7f4f1c042200 auth_method 0 2024-09-10T16:07:34.225 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2555632 2024-09-10T16:07:34.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.212+0000 7f4f3584d640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 0x7f4f1c0446c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f28003980 tx=0x7f4f2800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.215+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4f20024030 con 0x7f4f1c0097a0 2024-09-10T16:07:34.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.215+0000 7f4f2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.215+0000 7f4f2e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:34.232 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:34.310 INFO:tasks.rados.rados.0.smithi165.stdout:148: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-10T16:07:34.310 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:34.395 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing write tid 1 to smithi16534841-47 2024-09-10T16:07:34.396 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing write tid 2 to smithi16534841-47 2024-09-10T16:07:34.396 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing write tid 3 to smithi16534841-47 2024-09-10T16:07:34.396 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing write tid 4 to smithi16534841-47 2024-09-10T16:07:34.396 INFO:tasks.rados.rados.0.smithi165.stdout:151: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-10T16:07:34.397 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1528571 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:150: finishing write tid 5 to smithi16534841-47 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:150: oid 47 updating version 0 to 34 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:150: oid 47 updating version 34 to 37 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:150: oid 47 version 37 is already newer than 36 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:150: oid 47 version 37 is already newer than 35 2024-09-10T16:07:34.453 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 37 (ObjNum 83 snap 18 seq_num 83) dirty exists 2024-09-10T16:07:34.454 INFO:tasks.rados.rados.0.smithi165.stdout:150: left oid 47 (ObjNum 83 snap 18 seq_num 83) 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f37ab1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f37ab1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f37ab1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f300a4bd0 con 0x7f4f1c0097a0 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f2e7fc640 1 -- 172.21.15.164:0/3035217734 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f4f20051c70 con 0x7f4f1c0097a0 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:07:34.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.462+0000 7f4f2e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:34.465 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:07:34.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 msgr2=0x7f4f1c0446c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 0x7f4f1c0446c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4f28003980 tx=0x7f4f2800d040 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 10 monclient: shutdown 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 msgr2=0x7f4f3012e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00af60 tx=0x7f4f20008dd0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 shutdown_connections 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f1c042200 0x7f4f1c0446c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f1c0097a0 0x7f4f3012e0c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f1c008e30 0x7f4f3012db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 --2- 172.21.15.164:0/3035217734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f1c005d70 0x7f4f3012fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 >> 172.21.15.164:0/3035217734 conn(0x7f4f300a3110 msgr2=0x7f4f300707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 shutdown_connections 2024-09-10T16:07:34.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.465+0000 7f4f37ab1640 1 -- 172.21.15.164:0/3035217734 wait complete. 2024-09-10T16:07:34.477 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:07:34.478 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:07:34.478 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:07:34.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.574+0000 7fb91613d640 10 monclient: get_monmap_and_config 2024-09-10T16:07:34.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.574+0000 7fb91613d640 10 monclient: build_initial_monmap 2024-09-10T16:07:34.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.574+0000 7fb91613d640 10 monclient: monmap: 2024-09-10T16:07:34.576 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:34.575655+0000 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:34.575655+0000 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:34.577 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 1 Processor -- start 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 1 -- start start 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 10 monclient: init 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.575+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-b con 0x7fb91009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb91009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb91013a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:34.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91009a430 con 0x7fb91013a5c0 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9100a24e0 con 0x7fb91009d160 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb9100a2bf0 con 0x7fb91009c180 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb91613d640 10 monclient(hunting): authenticate will time out at 833.063053s 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90ffff640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fb91013a5c0 legacy=0x7fb91013c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49284/0 (socket says 172.21.15.164:49284) 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3429115972 learned_addr learned my addr 172.21.15.164:0/3429115972 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): get_auth_request con 0x7fb91009c180 auth_method 0 2024-09-10T16:07:34.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_done global_id 4487 payload 274 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3429115972 >> v1:172.21.15.165:6789/0 conn(0x7fb91009d160 legacy=0x7fb91009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3429115972 >> v1:172.21.15.164:6789/0 conn(0x7fb91013a5c0 legacy=0x7fb91013c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 monclient: found mon.noname-b 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.576+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3429115972 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb91009c560 con 0x7fb91009c180 2024-09-10T16:07:34.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3429115972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 msgr2=0x7fb91009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90f7fe640 1 --2- 172.21.15.164:0/3429115972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb900009e50 tx=0x7fb90002a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fee66f551998c6d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 1 -- 172.21.15.164:0/3429115972 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900036070 con 0x7fb91009c180 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: dump: 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 5 monclient: authenticate success, global_id 4487 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 1 -- 172.21.15.164:0/3429115972 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb900007800 con 0x7fb91009c180 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 4 set_mon_vals no callback set 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 1 -- 172.21.15.164:0/3429115972 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900007bd0 con 0x7fb91009c180 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb90e7fc640 10 monclient: dump: 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 10 monclient: shutdown 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 -- 172.21.15.164:0/3429115972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 msgr2=0x7fb91009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 --2- 172.21.15.164:0/3429115972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb900009e50 tx=0x7fb90002a500 comp rx=0 tx=0).stop 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 -- 172.21.15.164:0/3429115972 shutdown_connections 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 --2- 172.21.15.164:0/3429115972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb91009ea40 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 -- 172.21.15.164:0/3429115972 >> 172.21.15.164:0/3429115972 conn(0x7fb9100a3110 msgr2=0x7fb9100a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 -- 172.21.15.164:0/3429115972 shutdown_connections 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.579+0000 7fb91613d640 1 -- 172.21.15.164:0/3429115972 wait complete. 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.580+0000 7fb91613d640 10 monclient: build_initial_monmap 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.580+0000 7fb91613d640 10 monclient: monmap: 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:34.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:34.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:34.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.580+0000 7fb91613d640 1 Processor -- start 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 -- start start 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient: init 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-c con 0x7fb91009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb91009d160 0x7fb9100941c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-a con 0x7fb91009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91013a5c0 0x7fb910094700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): picked mon.noname-b con 0x7fb91013a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb910074f90 con 0x7fb91009d160 2024-09-10T16:07:34.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb910074e10 con 0x7fb91013a5c0 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb91009f690 con 0x7fb91009c180 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb91613d640 10 monclient(hunting): authenticate will time out at 833.068499s 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90effd640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb91009d160 msgr2=0x7fb9100941c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb91009d160 0x7fb9100941c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49298/0 (socket says 172.21.15.164:49298) 2024-09-10T16:07:34.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3292338766 learned_addr learned my addr 172.21.15.164:0/3292338766 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): get_auth_request con 0x7fb91009c180 auth_method 0 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91013a5c0 0x7fb910094700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 10 monclient(hunting): get_auth_request con 0x7fb91013a5c0 auth_method 0 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient(hunting): handle_auth_done global_id 4490 payload 824 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91013a5c0 msgr2=0x7fb910094700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91013a5c0 0x7fb910094700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb91009d160 msgr2=0x7fb9100941c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb91009d160 0x7fb9100941c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb910097440 con 0x7fb91009c180 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.581+0000 7fb90ffff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb91013a5c0 0x7fb910094700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:07:34.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb90f7fe640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 msgr2=0x7fb910093c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb90f7fe640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9000093e0 tx=0x7fb900004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900041020 con 0x7fb91009c180 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: dump: 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 5 monclient: authenticate success, global_id 4490 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9000319c0 con 0x7fb91009c180 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900036040 con 0x7fb91009c180 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 10 monclient: _renew_subs 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb910095f20 con 0x7fb91009c180 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb8f7fff640 10 monclient: dump: 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.602 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb90dffb640 4 set_mon_vals no callback set 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 10 monclient: _renew_subs 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.584+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb910096af0 con 0x7fb91009c180 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb91613d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb900044020 con 0x7fb91009c180 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb91613d640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 0x7fb91002e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb910095420 con 0x7fb910096e80 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 0x7fb91002e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient: get_auth_request con 0x7fb910096e80 auth_method 0 2024-09-10T16:07:34.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb8f7fff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb8fc03e1c0 0x7fb8fc040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 4783+0+0 (secure 0 0 0) 0x7fb900073c60 con 0x7fb91009c180 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90effd640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb91009d930 con 0x7fb910096e80 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90ffff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb8fc03e1c0 0x7fb8fc040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.585+0000 7fb90ffff640 10 monclient: get_auth_request con 0x7fb8fc03e1c0 auth_method 0 2024-09-10T16:07:34.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb90effd640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 msgr2=0x7fb91002e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb90effd640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 0x7fb91002e5c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb9100962f0 tx=0x7fb90400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904017070 con 0x7fb910096e80 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.587+0000 7fb8f7fff640 10 monclient: dump: 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb90ffff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb8fc03e1c0 0x7fb8fc040680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb908003930 tx=0x7fb908006eb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb9040057c0 con 0x7fb910096e80 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 msgr2=0x7fb91002e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 0x7fb91002e5c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb9100962f0 tx=0x7fb90400c810 comp rx=0 tx=0).stop 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 msgr2=0x7fb91002e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:34.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.588+0000 7fb8f7fff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910096e80 0x7fb91002e5c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb9100962f0 tx=0x7fb90400c810 comp rx=0 tx=0).stop 2024-09-10T16:07:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb91613d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb91613d640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 0x7fb910063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb91613d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb910097780 con 0x7fb910094c40 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 0x7fb910063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient: get_auth_request con 0x7fb910094c40 auth_method 0 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 10 monclient(hunting): handle_auth_done global_id 4496 payload 824 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.594+0000 7fb90effd640 1 -- 172.21.15.164:0/3292338766 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb910072f00 con 0x7fb910094c40 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb90effd640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 msgr2=0x7fb910063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb90effd640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 0x7fb910063d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb9100962f0 tx=0x7fb9040037e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904017040 con 0x7fb910094c40 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 10 monclient: dump: 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fb904012400 con 0x7fb910094c40 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 msgr2=0x7fb910063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.596+0000 7fb8f7fff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 0x7fb910063d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb9100962f0 tx=0x7fb9040037e0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb8f7fff640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 msgr2=0x7fb910063d50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb8f7fff640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb910094c40 0x7fb910063d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:34.611 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 16, 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 96768, 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.612 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.613 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:34.614 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:34.615 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:34.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.618 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:34.620 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.621 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.622 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.623 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 7 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:34.624 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb8fc03e1c0 msgr2=0x7fb8fc040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb8fc03e1c0 0x7fb8fc040680 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb908003930 tx=0x7fb908006eb0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 10 monclient: shutdown 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 msgr2=0x7fb910093c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb9000093e0 tx=0x7fb900004a40 comp rx=0 tx=0).stop 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb90ffff640 1 -- 172.21.15.164:0/3292338766 reap_dead start 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 shutdown_connections 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 --2- 172.21.15.164:0/3292338766 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb91009c180 0x7fb910093c80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 >> 172.21.15.164:0/3292338766 conn(0x7fb9100a3110 msgr2=0x7fb9101399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 shutdown_connections 2024-09-10T16:07:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.597+0000 7fb91613d640 1 -- 172.21.15.164:0/3292338766 wait complete. 2024-09-10T16:07:34.627 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:07:34.627 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:07:34.627 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.724+0000 7fb530062640 10 monclient: get_monmap_and_config 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.724+0000 7fb530062640 10 monclient: build_initial_monmap 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.724+0000 7fb530062640 10 monclient: monmap: 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.726 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:07:34.725517+0000 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:07:34.725517+0000 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:07:34.727 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.725+0000 7fb530062640 1 Processor -- start 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.725+0000 7fb530062640 1 -- start start 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.725+0000 7fb530062640 10 monclient: init 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb5280a3a50 0x7fb52809c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): picked mon.noname-c con 0x7fb5280a3a50 addr v2:172.21.15.165:3300/0 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): picked mon.noname-b con 0x7fb5280a2320 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb52809ceb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5280748f0 con 0x7fb52809ceb0 2024-09-10T16:07:34.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb5280a2b20 con 0x7fb5280a3a50 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb52809d370 con 0x7fb5280a2320 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb530062640 10 monclient(hunting): authenticate will time out at 833.213245s 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52e5ff640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fb52809ceb0 legacy=0x7fb528138240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49320/0 (socket says 172.21.15.164:49320) 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 -- 172.21.15.164:0/2855700820 learned_addr learned my addr 172.21.15.164:0/2855700820 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): get_auth_request con 0x7fb5280a2320 auth_method 0 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/2855700820 >> v2:172.21.15.165:3300/0 conn(0x7fb5280a3a50 0x7fb52809c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request con 0x7fb5280a3a50 auth_method 0 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient(hunting): handle_auth_done global_id 4499 payload 274 2024-09-10T16:07:34.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 -- 172.21.15.164:0/2855700820 >> v2:172.21.15.165:3300/0 conn(0x7fb5280a3a50 msgr2=0x7fb52809c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 --2- 172.21.15.164:0/2855700820 >> v2:172.21.15.165:3300/0 conn(0x7fb5280a3a50 0x7fb52809c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 -- 172.21.15.164:0/2855700820 >> v1:172.21.15.164:6789/0 conn(0x7fb52809ceb0 legacy=0x7fb528138240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 monclient: found mon.noname-b 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.726+0000 7fb52d5fd640 1 -- 172.21.15.164:0/2855700820 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb528074bd0 con 0x7fb5280a2320 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52d5fd640 1 -- 172.21.15.164:0/2855700820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 msgr2=0x7fb528137d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52d5fd640 1 --2- 172.21.15.164:0/2855700820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb518009870 tx=0x7fb51802a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1b73a36a1c68e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 1 -- 172.21.15.164:0/2855700820 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518037070 con 0x7fb5280a2320 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:07:34.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: dump: 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 1 -- 172.21.15.164:0/2855700820 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518032640 con 0x7fb5280a2320 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb530062640 5 monclient: authenticate success, global_id 4499 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb530062640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb530062640 10 monclient: get_monmap_and_config success 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb530062640 4 set_mon_vals no callback set 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 1 -- 172.21.15.164:0/2855700820 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518032a10 con 0x7fb5280a2320 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.729+0000 7fb52cdfc640 10 monclient: dump: 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 10 monclient: shutdown 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 -- 172.21.15.164:0/2855700820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 msgr2=0x7fb528137d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 --2- 172.21.15.164:0/2855700820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb518009870 tx=0x7fb51802a130 comp rx=0 tx=0).stop 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 -- 172.21.15.164:0/2855700820 shutdown_connections 2024-09-10T16:07:34.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 --2- 172.21.15.164:0/2855700820 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5280a2320 0x7fb528137d00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 --2- 172.21.15.164:0/2855700820 >> v2:172.21.15.165:3300/0 conn(0x7fb5280a3a50 0x7fb52809c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 -- 172.21.15.164:0/2855700820 >> 172.21.15.164:0/2855700820 conn(0x7fb5280a2f30 msgr2=0x7fb5280a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 -- 172.21.15.164:0/2855700820 shutdown_connections 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 1 -- 172.21.15.164:0/2855700820 wait complete. 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 10 monclient: build_initial_monmap 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.730+0000 7fb530062640 10 monclient: monmap: 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.739 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:07:34.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:07:34.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:07:34.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 Processor -- start 2024-09-10T16:07:34.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 -- start start 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient: init 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient: _reopen_session rank -1 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809ceb0 0x7fb5280a01b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): picked mon.noname-b con 0x7fb52809ceb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5280a3a50 0x7fb52812c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): picked mon.noname-a con 0x7fb5280a3a50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 0x7fb52812cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): picked mon.noname-c con 0x7fb52812ca80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb528134cb0 con 0x7fb5280a3a50 2024-09-10T16:07:34.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb528138b60 con 0x7fb52809ceb0 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb528134e30 con 0x7fb52812ca80 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): _renew_subs 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb530062640 10 monclient(hunting): authenticate will time out at 833.218601s 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52d5fd640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5280a3a50 msgr2=0x7fb52812c540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52d5fd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5280a3a50 0x7fb52812c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 0x7fb52812cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 0x7fb52812cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49328/0 (socket says 172.21.15.164:49328) 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 learned_addr learned my addr 172.21.15.164:0/1454941532 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): get_auth_request con 0x7fb52812ca80 auth_method 0 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809ceb0 0x7fb5280a01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request con 0x7fb52809ceb0 auth_method 0 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809ceb0 msgr2=0x7fb5280a01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809ceb0 0x7fb5280a01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5280a3a50 msgr2=0x7fb52812c540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5280a3a50 0x7fb52812c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient: _finish_hunting 0 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 monclient: found mon.noname-c 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb52812f640 con 0x7fb52812ca80 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.731+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809ceb0 0x7fb5280a01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.734+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 msgr2=0x7fb52812cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.734+0000 7fb52e5ff640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 0x7fb52812cf30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb52000be60 tx=0x7fb52000bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.734+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520016630 con 0x7fb52812ca80 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: dump: 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:07:34.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 5 monclient: authenticate success, global_id 4502 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb520010030 con 0x7fb52812ca80 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 10 monclient: _renew_subs 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb52809d620 con 0x7fb52812ca80 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520016a10 con 0x7fb52812ca80 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51e7fc640 10 monclient: dump: 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.751 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 10 monclient: _renew_subs 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb52809bfd0 con 0x7fb52812ca80 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.736+0000 7fb51ffff640 4 set_mon_vals no callback set 2024-09-10T16:07:34.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb530062640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb530062640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 0x7fb52809d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb52809c970 con 0x7fb5280a2320 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb5200153e0 con 0x7fb52812ca80 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 0x7fb52809d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient: get_auth_request con 0x7fb5280a2320 auth_method 0 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb51e7fc640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb50c03e210 0x7fb50c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(48..48 src has 1..48) ==== 4783+0+0 (secure 0 0 0) 0x7fb520053570 con 0x7fb52812ca80 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52d5fd640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb50c03e210 0x7fb50c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52d5fd640 10 monclient: get_auth_request con 0x7fb50c03e210 auth_method 0 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_done global_id 14149 payload 824 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.737+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb528132a20 con 0x7fb5280a2320 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.739+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 msgr2=0x7fb52809d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.739+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 0x7fb52809d830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb524011830 tx=0x7fb524011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb52d5fd640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb50c03e210 0x7fb50c0406d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb518034000 tx=0x7fb51802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb52400fc40 con 0x7fb5280a2320 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 10 monclient: dump: 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb52400fc40 con 0x7fb5280a2320 2024-09-10T16:07:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:07:34.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:07:34.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 msgr2=0x7fb52809d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 0x7fb52809d830 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb524011830 tx=0x7fb524011cf0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 msgr2=0x7fb52809d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:34.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.740+0000 7fb51e7fc640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5280a2320 0x7fb52809d830 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb524011830 tx=0x7fb524011cf0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb530062640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:07:34.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb530062640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:07:34.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb530062640 10 monclient(hunting): start opening mon connection 2024-09-10T16:07:34.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb5281345e0 con 0x7fb52809dd70 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient: get_auth_request con 0x7fb52809dd70 auth_method 0 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 msgr2=0x7fb52809e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 msgr2=0x7fb52809e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:07:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.746+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:07:34.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:07:34.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient: get_auth_request con 0x7fb52809dd70 auth_method 0 2024-09-10T16:07:34.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 10 monclient(hunting): handle_auth_done global_id 14155 payload 824 2024-09-10T16:07:34.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.947+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb528077260 con 0x7fb52809dd70 2024-09-10T16:07:34.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb52ddfe640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 msgr2=0x7fb52809e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:07:34.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb52ddfe640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb524008010 tx=0x7fb5240046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:07:34.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5240066e0 con 0x7fb52809dd70 2024-09-10T16:07:34.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 10 monclient: dump: 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:07:34.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:07:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7fb5240066e0 con 0x7fb52809dd70 2024-09-10T16:07:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:07:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:07:34.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 msgr2=0x7fb52809e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb524008010 tx=0x7fb5240046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 msgr2=0x7fb52809e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.950+0000 7fb51e7fc640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb52809dd70 0x7fb52809e180 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb524008010 tx=0x7fb5240046c0 comp rx=0 tx=0).stop 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 16, 2024-09-10T16:07:34.954 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 28424, 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.955 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.956 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:07:34.957 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:07:34.958 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:07:34.959 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.960 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.961 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.962 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.963 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.965 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.966 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:07:34.967 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:07:34.968 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:07:34.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb50c03e210 msgr2=0x7fb50c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb50c03e210 0x7fb50c0406d0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb518034000 tx=0x7fb51802e000 comp rx=0 tx=0).stop 2024-09-10T16:07:34.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 10 monclient: shutdown 2024-09-10T16:07:34.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 msgr2=0x7fb52812cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 --2- 172.21.15.164:0/1454941532 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb52812ca80 0x7fb52812cf30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb52000be60 tx=0x7fb52000bf50 comp rx=0 tx=0).stop 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 reap_dead start 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb52e5ff640 1 -- 172.21.15.164:0/1454941532 reap_dead start 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 shutdown_connections 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 >> 172.21.15.164:0/1454941532 conn(0x7fb5280a2f30 msgr2=0x7fb5280a5690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 shutdown_connections 2024-09-10T16:07:34.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:07:34.951+0000 7fb530062640 1 -- 172.21.15.164:0/1454941532 wait complete. 2024-09-10T16:07:34.970 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:07:34.971 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:145: done (7 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:146: done (6 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:147: done (5 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:148: done (4 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:149: done (3 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:150: done (2 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:151: done (1 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:152: done (0 left) 2024-09-10T16:07:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:153: copy_from oid 21 from oid 3 current snap is 19 2024-09-10T16:07:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:154: snap_create 2024-09-10T16:07:35.111 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:07:35.115 INFO:tasks.rados.rados.0.smithi165.stdout:153: finishing copy_from to smithi16534841-21 2024-09-10T16:07:35.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 32 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-10T16:07:36.099 INFO:tasks.rados.rados.0.smithi165.stdout:153: done (1 left) 2024-09-10T16:07:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:154: done (0 left) 2024-09-10T16:07:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:155: write (excl) oid 30 current snap is 20 2024-09-10T16:07:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:155: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-10T16:07:36.104 INFO:tasks.rados.rados.0.smithi165.stdout:155: writing smithi16534841-30 from 611788 to 1012587 tid 1 2024-09-10T16:07:36.111 INFO:tasks.rados.rados.0.smithi165.stdout:155: writing smithi16534841-30 from 1748958 to 2237644 tid 2 2024-09-10T16:07:36.118 INFO:tasks.rados.rados.0.smithi165.stdout:155: writing smithi16534841-30 from 2907296 to 3411788 tid 3 2024-09-10T16:07:36.118 INFO:tasks.rados.rados.0.smithi165.stdout:156: write oid 29 current snap is 20 2024-09-10T16:07:36.118 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing write tid 1 to smithi16534841-30 2024-09-10T16:07:36.118 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing write tid 2 to smithi16534841-30 2024-09-10T16:07:36.119 INFO:tasks.rados.rados.0.smithi165.stdout:156: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-10T16:07:36.124 INFO:tasks.rados.rados.0.smithi165.stdout:156: writing smithi16534841-29 from 628595 to 1104643 tid 1 2024-09-10T16:07:36.132 INFO:tasks.rados.rados.0.smithi165.stdout:156: writing smithi16534841-29 from 1807440 to 2439784 tid 2 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:156: writing smithi16534841-29 from 2934719 to 3387193 tid 3 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:157: rollback oid 9 current snap is 20 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 19 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing write tid 3 to smithi16534841-30 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing write tid 4 to smithi16534841-30 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: finishing write tid 5 to smithi16534841-30 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: oid 30 updating version 0 to 61 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: oid 30 version 61 is already newer than 59 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: oid 30 version 61 is already newer than 60 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:155: oid 30 updating version 61 to 62 2024-09-10T16:07:36.138 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 62 (ObjNum 84 snap 20 seq_num 84) dirty exists 2024-09-10T16:07:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:155: left oid 30 (ObjNum 84 snap 20 seq_num 84) 2024-09-10T16:07:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:156: finishing write tid 1 to smithi16534841-29 2024-09-10T16:07:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:156: finishing write tid 2 to smithi16534841-29 2024-09-10T16:07:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:155: done (2 left) 2024-09-10T16:07:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:158: delete oid 1 current snap is 20 2024-09-10T16:07:36.140 INFO:tasks.rados.rados.0.smithi165.stdout:156: finishing write tid 3 to smithi16534841-29 2024-09-10T16:07:36.140 INFO:tasks.rados.rados.0.smithi165.stdout:156: finishing write tid 4 to smithi16534841-29 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:156: finishing write tid 5 to smithi16534841-29 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:156: oid 29 updating version 0 to 42 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:156: oid 29 version 42 is already newer than 39 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:156: oid 29 version 42 is already newer than 41 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:156: oid 29 version 42 is already newer than 40 2024-09-10T16:07:36.141 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 42 (ObjNum 85 snap 20 seq_num 85) dirty exists 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:156: left oid 29 (ObjNum 85 snap 20 seq_num 85) 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:157: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:156: done (2 left) 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:158: done (1 left) 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:159: delete oid 6 current snap is 20 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:157: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:157: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:07:36.142 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 13 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-10T16:07:36.144 INFO:tasks.rados.rados.0.smithi165.stdout:157: done (1 left) 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:159: done (0 left) 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:160: read oid 4 snap -1 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:160: expect (ObjNum 80 snap 18 seq_num 80) 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:161: copy_from oid 3 from oid 39 current snap is 20 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:162: copy_from oid 36 from oid 13 current snap is 20 2024-09-10T16:07:36.145 INFO:tasks.rados.rados.0.smithi165.stdout:163: snap_remove snap 15 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:163: done (3 left) 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:164: copy_from oid 2 from oid 11 current snap is 20 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:161: finishing copy_from to smithi16534841-3 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:161: got expected ENOENT (src dne) 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:161: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:07:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:160: expect (ObjNum 80 snap 18 seq_num 80) 2024-09-10T16:07:36.694 INFO:tasks.rados.rados.0.smithi165.stdout:162: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:162: finishing copy_from to smithi16534841-36 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 3 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:160: done (3 left) 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:161: done (2 left) 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:162: done (1 left) 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:165: read oid 26 snap 8 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:165: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:166: write oid 39 current snap is 20 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing copy_from to smithi16534841-2 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:164: got expected ENOENT (src dne) 2024-09-10T16:07:36.695 INFO:tasks.rados.rados.0.smithi165.stdout:164: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:07:36.696 INFO:tasks.rados.rados.0.smithi165.stdout:166: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-10T16:07:36.701 INFO:tasks.rados.rados.0.smithi165.stdout:166: writing smithi16534841-39 from 679016 to 1380811 tid 1 2024-09-10T16:07:36.707 INFO:tasks.rados.rados.0.smithi165.stdout:166: writing smithi16534841-39 from 1866533 to 2362557 tid 2 2024-09-10T16:07:36.713 INFO:tasks.rados.rados.0.smithi165.stdout:166: writing smithi16534841-39 from 2900635 to 3303278 tid 3 2024-09-10T16:07:36.713 INFO:tasks.rados.rados.0.smithi165.stdout:164: done (2 left) 2024-09-10T16:07:36.713 INFO:tasks.rados.rados.0.smithi165.stdout:167: delete oid 42 current snap is 20 2024-09-10T16:07:36.713 INFO:tasks.rados.rados.0.smithi165.stdout:165: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-10T16:07:36.713 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:36.752 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2301461 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:166: finishing write tid 1 to smithi16534841-39 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:166: finishing write tid 2 to smithi16534841-39 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:165: done (2 left) 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:167: done (1 left) 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:168: rollback oid 24 current snap is 20 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 11 2024-09-10T16:07:36.780 INFO:tasks.rados.rados.0.smithi165.stdout:166: finishing write tid 3 to smithi16534841-39 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: finishing write tid 4 to smithi16534841-39 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: finishing write tid 5 to smithi16534841-39 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: oid 39 updating version 0 to 36 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: oid 39 updating version 36 to 37 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: oid 39 updating version 37 to 38 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: oid 39 updating version 38 to 39 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 39 (ObjNum 88 snap 20 seq_num 88) dirty exists 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: left oid 39 (ObjNum 88 snap 20 seq_num 88) 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:166: done (1 left) 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:169: write oid 6 current snap is 20 2024-09-10T16:07:36.781 INFO:tasks.rados.rados.0.smithi165.stdout:169: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-10T16:07:36.785 INFO:tasks.rados.rados.0.smithi165.stdout:169: writing smithi16534841-6 from 712630 to 1164923 tid 1 2024-09-10T16:07:36.790 INFO:tasks.rados.rados.0.smithi165.stdout:169: writing smithi16534841-6 from 1583497 to 2083190 tid 2 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:169: writing smithi16534841-6 from 2555481 to 2980668 tid 3 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:170: read oid 20 snap -1 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:170: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:171: snap_create 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:168: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:168: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:168: finishing rollback tid 2 to smithi16534841-24 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 52 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-10T16:07:36.796 INFO:tasks.rados.rados.0.smithi165.stdout:169: finishing write tid 1 to smithi16534841-6 2024-09-10T16:07:36.798 INFO:tasks.rados.rados.0.smithi165.stdout:169: finishing write tid 2 to smithi16534841-6 2024-09-10T16:07:36.798 INFO:tasks.rados.rados.0.smithi165.stdout:169: finishing write tid 3 to smithi16534841-6 2024-09-10T16:07:36.799 INFO:tasks.rados.rados.0.smithi165.stdout:169: finishing write tid 4 to smithi16534841-6 2024-09-10T16:07:36.799 INFO:tasks.rados.rados.0.smithi165.stdout:169: finishing write tid 5 to smithi16534841-6 2024-09-10T16:07:36.799 INFO:tasks.rados.rados.0.smithi165.stdout:169: oid 6 updating version 0 to 52 2024-09-10T16:07:36.800 INFO:tasks.rados.rados.0.smithi165.stdout:169: oid 6 updating version 52 to 54 2024-09-10T16:07:36.800 INFO:tasks.rados.rados.0.smithi165.stdout:169: oid 6 version 54 is already newer than 53 2024-09-10T16:07:36.800 INFO:tasks.rados.rados.0.smithi165.stdout:169: oid 6 version 54 is already newer than 51 2024-09-10T16:07:36.800 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 54 (ObjNum 90 snap 20 seq_num 90) dirty exists 2024-09-10T16:07:36.800 INFO:tasks.rados.rados.0.smithi165.stdout:169: left oid 6 (ObjNum 90 snap 20 seq_num 90) 2024-09-10T16:07:36.802 INFO:tasks.rados.rados.0.smithi165.stdout:170: expect (ObjNum 64 snap 5 seq_num 64) 2024-09-10T16:07:36.827 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2738180 2024-09-10T16:07:36.881 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:37.980 INFO:tasks.rados.rados.0.smithi165.stdout:168: done (3 left) 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:169: done (2 left) 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:170: done (1 left) 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:171: done (0 left) 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:172: rollback oid 15 current snap is 21 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 18 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:173: read oid 21 snap -1 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:173: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:07:37.981 INFO:tasks.rados.rados.0.smithi165.stdout:174: snap_remove snap 12 2024-09-10T16:07:38.982 INFO:tasks.rados.rados.0.smithi165.stdout:174: done (2 left) 2024-09-10T16:07:38.982 INFO:tasks.rados.rados.0.smithi165.stdout:175: snap_remove snap 5 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:175: done (2 left) 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:176: snap_create 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:172: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:172: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:172: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:07:39.997 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 26 (ObjNum 71 snap 10 seq_num 71) dirty exists 2024-09-10T16:07:39.998 INFO:tasks.rados.rados.0.smithi165.stdout:173: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:07:41.000 INFO:tasks.rados.rados.0.smithi165.stdout:172: done (2 left) 2024-09-10T16:07:41.000 INFO:tasks.rados.rados.0.smithi165.stdout:173: done (1 left) 2024-09-10T16:07:41.000 INFO:tasks.rados.rados.0.smithi165.stdout:176: done (0 left) 2024-09-10T16:07:41.000 INFO:tasks.rados.rados.0.smithi165.stdout:177: snap_create 2024-09-10T16:07:41.541 INFO:tasks.rados.rados.0.smithi165.stdout:177: done (0 left) 2024-09-10T16:07:41.541 INFO:tasks.rados.rados.0.smithi165.stdout:178: copy_from oid 45 from oid 26 current snap is 23 2024-09-10T16:07:41.541 INFO:tasks.rados.rados.0.smithi165.stdout:179: write (excl) oid 8 current snap is 23 2024-09-10T16:07:41.541 INFO:tasks.rados.rados.0.smithi165.stdout:179: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2024-09-10T16:07:41.548 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16534841-8 from 729437 to 1256979 tid 1 2024-09-10T16:07:41.556 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16534841-8 from 1925626 to 2568977 tid 2 2024-09-10T16:07:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:179: writing smithi16534841-8 from 3266551 to 3529437 tid 3 2024-09-10T16:07:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:180: write oid 23 current snap is 23 2024-09-10T16:07:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 1 to smithi16534841-8 2024-09-10T16:07:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 2 to smithi16534841-8 2024-09-10T16:07:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:07:41.563 INFO:tasks.rados.rados.0.smithi165.stdout:180: seq_num 92 ranges {746244=719144,2100461=503362,3126680=419564} 2024-09-10T16:07:41.570 INFO:tasks.rados.rados.0.smithi165.stdout:180: writing smithi16534841-23 from 746244 to 1465388 tid 1 2024-09-10T16:07:41.576 INFO:tasks.rados.rados.0.smithi165.stdout:180: writing smithi16534841-23 from 2100461 to 2603823 tid 2 2024-09-10T16:07:41.582 INFO:tasks.rados.rados.0.smithi165.stdout:180: writing smithi16534841-23 from 3126680 to 3546244 tid 3 2024-09-10T16:07:41.582 INFO:tasks.rados.rados.0.smithi165.stdout:181: rollback oid 42 current snap is 23 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 16 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:178: finishing copy_from to smithi16534841-45 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 53 (ObjNum 81 snap 18 seq_num 81) dirty exists 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 3 to smithi16534841-8 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 4 to smithi16534841-8 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: finishing write tid 5 to smithi16534841-8 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 8 updating version 0 to 71 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 8 updating version 71 to 72 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 8 version 72 is already newer than 70 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: oid 8 updating version 72 to 73 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 73 (ObjNum 91 snap 23 seq_num 91) dirty exists 2024-09-10T16:07:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:179: left oid 8 (ObjNum 91 snap 23 seq_num 91) 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing write tid 1 to smithi16534841-23 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing write tid 2 to smithi16534841-23 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:178: done (3 left) 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:179: done (2 left) 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:182: copy_from oid 21 from oid 3 current snap is 23 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:183: write oid 35 current snap is 23 2024-09-10T16:07:41.584 INFO:tasks.rados.rados.0.smithi165.stdout:183: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-10T16:07:41.591 INFO:tasks.rados.rados.0.smithi165.stdout:183: writing smithi16534841-35 from 763051 to 1557444 tid 1 2024-09-10T16:07:41.598 INFO:tasks.rados.rados.0.smithi165.stdout:183: writing smithi16534841-35 from 2158943 to 2805963 tid 2 2024-09-10T16:07:41.602 INFO:tasks.rados.rados.0.smithi165.stdout:183: writing smithi16534841-35 from 3437750 to 3563051 tid 3 2024-09-10T16:07:41.602 INFO:tasks.rados.rados.0.smithi165.stdout:184: read oid 17 snap 18 2024-09-10T16:07:41.603 INFO:tasks.rados.rados.0.smithi165.stdout:184: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-10T16:07:41.603 INFO:tasks.rados.rados.0.smithi165.stdout:185: snap_remove snap 13 2024-09-10T16:07:42.542 INFO:tasks.rados.rados.0.smithi165.stdout:185: done (5 left) 2024-09-10T16:07:42.542 INFO:tasks.rados.rados.0.smithi165.stdout:186: snap_remove snap 22 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:186: done (5 left) 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:187: delete oid 6 current snap is 23 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:181: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing write tid 3 to smithi16534841-23 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:182: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing write tid 4 to smithi16534841-23 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:181: finishing rollback tid 2 to smithi16534841-42 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 49 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-09-10T16:07:43.544 INFO:tasks.rados.rados.0.smithi165.stdout:180: finishing write tid 5 to smithi16534841-23 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:180: oid 23 updating version 0 to 52 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:180: oid 23 version 52 is already newer than 50 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:180: oid 23 updating version 52 to 53 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:180: oid 23 version 53 is already newer than 51 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 53 (ObjNum 92 snap 23 seq_num 92) dirty exists 2024-09-10T16:07:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:180: left oid 23 (ObjNum 92 snap 23 seq_num 92) 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:182: finishing copy_from to smithi16534841-21 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 32 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: finishing write tid 1 to smithi16534841-35 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: finishing write tid 2 to smithi16534841-35 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: finishing write tid 3 to smithi16534841-35 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: finishing write tid 4 to smithi16534841-35 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: finishing write tid 5 to smithi16534841-35 2024-09-10T16:07:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:183: oid 35 updating version 0 to 13 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:183: oid 35 updating version 13 to 14 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:183: oid 35 version 14 is already newer than 11 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:183: oid 35 version 14 is already newer than 12 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 14 (ObjNum 93 snap 23 seq_num 93) dirty exists 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:183: left oid 35 (ObjNum 93 snap 23 seq_num 93) 2024-09-10T16:07:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:184: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-10T16:07:43.554 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2089294 2024-09-10T16:07:43.560 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1858748 2024-09-10T16:07:43.570 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:180: done (5 left) 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:181: done (4 left) 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:182: done (3 left) 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:183: done (2 left) 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:184: done (1 left) 2024-09-10T16:07:43.596 INFO:tasks.rados.rados.0.smithi165.stdout:187: done (0 left) 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:188: copy_from oid 11 from oid 34 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:189: copy_from oid 35 from oid 2 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:190: copy_from oid 30 from oid 12 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:191: rollback oid 42 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 8 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:192: rollback oid 44 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 18 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:193: write oid 28 current snap is 23 2024-09-10T16:07:43.597 INFO:tasks.rados.rados.0.smithi165.stdout:193: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-10T16:07:43.603 INFO:tasks.rados.rados.0.smithi165.stdout:193: writing smithi16534841-28 from 796665 to 1341556 tid 1 2024-09-10T16:07:43.609 INFO:tasks.rados.rados.0.smithi165.stdout:193: writing smithi16534841-28 from 1759554 to 2410243 tid 2 2024-09-10T16:07:43.616 INFO:tasks.rados.rados.0.smithi165.stdout:193: writing smithi16534841-28 from 2976243 to 3596665 tid 3 2024-09-10T16:07:43.616 INFO:tasks.rados.rados.0.smithi165.stdout:194: snap_remove snap 11 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:194: done (6 left) 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:195: snap_create 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:191: finishing rollback tid 0 to smithi16534841-42 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:188: finishing copy_from to smithi16534841-11 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:188: got expected ENOENT (src dne) 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:188: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:189: finishing copy_from to smithi16534841-35 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:189: got expected ENOENT (src dne) 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:189: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:191: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:07:44.544 INFO:tasks.rados.rados.0.smithi165.stdout:191: finishing rollback tid 2 to smithi16534841-42 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 56 (ObjNum 60 snap 5 seq_num 60) dirty exists 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:192: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 43 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:190: finishing copy_from to smithi16534841-30 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 3 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing write tid 1 to smithi16534841-28 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing write tid 2 to smithi16534841-28 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing write tid 3 to smithi16534841-28 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing write tid 4 to smithi16534841-28 2024-09-10T16:07:44.545 INFO:tasks.rados.rados.0.smithi165.stdout:193: finishing write tid 5 to smithi16534841-28 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:193: oid 28 updating version 0 to 5 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:193: oid 28 updating version 5 to 8 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:193: oid 28 version 8 is already newer than 6 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:193: oid 28 version 8 is already newer than 7 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 8 (ObjNum 95 snap 23 seq_num 95) dirty exists 2024-09-10T16:07:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:193: left oid 28 (ObjNum 95 snap 23 seq_num 95) 2024-09-10T16:07:45.556 INFO:tasks.rados.rados.0.smithi165.stdout:188: done (6 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:189: done (5 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:190: done (4 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:191: done (3 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:192: done (2 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:193: done (1 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:195: done (0 left) 2024-09-10T16:07:45.557 INFO:tasks.rados.rados.0.smithi165.stdout:196: read oid 37 snap -1 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:196: expect deleted 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:197: read oid 46 snap 7 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:197: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:198: read oid 34 snap -1 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:198: expect deleted 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:199: copy_from oid 27 from oid 33 current snap is 24 2024-09-10T16:07:45.558 INFO:tasks.rados.rados.0.smithi165.stdout:200: delete oid 36 current snap is 24 2024-09-10T16:07:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:196: done (4 left) 2024-09-10T16:07:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:198: done (3 left) 2024-09-10T16:07:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:200: done (2 left) 2024-09-10T16:07:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:201: write oid 23 current snap is 24 2024-09-10T16:07:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:201: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-10T16:07:45.568 INFO:tasks.rados.rados.0.smithi165.stdout:201: writing smithi16534841-23 from 430279 to 1125668 tid 1 2024-09-10T16:07:45.577 INFO:tasks.rados.rados.0.smithi165.stdout:201: writing smithi16534841-23 from 1876518 to 2530876 tid 2 2024-09-10T16:07:45.583 INFO:tasks.rados.rados.0.smithi165.stdout:201: writing smithi16534841-23 from 3031089 to 3567886 tid 3 2024-09-10T16:07:45.583 INFO:tasks.rados.rados.0.smithi165.stdout:202: copy_from oid 31 from oid 35 current snap is 24 2024-09-10T16:07:45.584 INFO:tasks.rados.rados.0.smithi165.stdout:197: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-10T16:07:45.587 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 780972 2024-09-10T16:07:45.609 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 511540 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:199: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:201: finishing write tid 1 to smithi16534841-23 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:199: finishing copy_from to smithi16534841-27 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 51 (ObjNum 68 snap 8 seq_num 68) dirty exists 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:201: finishing write tid 2 to smithi16534841-23 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:201: finishing write tid 3 to smithi16534841-23 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:201: finishing write tid 4 to smithi16534841-23 2024-09-10T16:07:45.662 INFO:tasks.rados.rados.0.smithi165.stdout:197: done (3 left) 2024-09-10T16:07:45.663 INFO:tasks.rados.rados.0.smithi165.stdout:199: done (2 left) 2024-09-10T16:07:45.663 INFO:tasks.rados.rados.0.smithi165.stdout:203: snap_remove snap 16 2024-09-10T16:07:46.541 INFO:tasks.rados.rados.0.smithi165.stdout:203: done (2 left) 2024-09-10T16:07:46.541 INFO:tasks.rados.rados.0.smithi165.stdout:204: snap_create 2024-09-10T16:07:46.541 INFO:tasks.rados.rados.0.smithi165.stdout:201: finishing write tid 5 to smithi16534841-23 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:201: oid 23 updating version 0 to 64 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:201: oid 23 version 64 is already newer than 63 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:201: oid 23 updating version 64 to 65 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:201: oid 23 updating version 65 to 66 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 66 (ObjNum 97 snap 24 seq_num 97) dirty exists 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:201: left oid 23 (ObjNum 97 snap 24 seq_num 97) 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:202: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:202: finishing copy_from to smithi16534841-31 2024-09-10T16:07:46.542 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 14 (ObjNum 93 snap 23 seq_num 93) dirty exists 2024-09-10T16:07:47.548 INFO:tasks.rados.rados.0.smithi165.stdout:201: done (2 left) 2024-09-10T16:07:47.549 INFO:tasks.rados.rados.0.smithi165.stdout:202: done (1 left) 2024-09-10T16:07:47.549 INFO:tasks.rados.rados.0.smithi165.stdout:204: done (0 left) 2024-09-10T16:07:47.549 INFO:tasks.rados.rados.0.smithi165.stdout:205: write oid 34 current snap is 25 2024-09-10T16:07:47.549 INFO:tasks.rados.rados.0.smithi165.stdout:205: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-10T16:07:47.555 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16534841-34 from 447086 to 1217724 tid 1 2024-09-10T16:07:47.563 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16534841-34 from 1935000 to 2733016 tid 2 2024-09-10T16:07:47.568 INFO:tasks.rados.rados.0.smithi165.stdout:205: writing smithi16534841-34 from 3458512 to 3647086 tid 3 2024-09-10T16:07:47.568 INFO:tasks.rados.rados.0.smithi165.stdout:206: rollback oid 33 current snap is 25 2024-09-10T16:07:47.568 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 8 2024-09-10T16:07:47.568 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 1 to smithi16534841-34 2024-09-10T16:07:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 2 to smithi16534841-34 2024-09-10T16:07:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:207: snap_remove snap 9 2024-09-10T16:07:48.564 INFO:tasks.rados.rados.0.smithi165.stdout:207: done (2 left) 2024-09-10T16:07:48.564 INFO:tasks.rados.rados.0.smithi165.stdout:208: snap_create 2024-09-10T16:07:48.564 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 3 to smithi16534841-34 2024-09-10T16:07:48.564 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 4 to smithi16534841-34 2024-09-10T16:07:48.564 INFO:tasks.rados.rados.0.smithi165.stdout:205: finishing write tid 5 to smithi16534841-34 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 34 updating version 0 to 21 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 34 updating version 21 to 22 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 34 version 22 is already newer than 19 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:205: oid 34 version 22 is already newer than 20 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 22 (ObjNum 98 snap 25 seq_num 98) dirty exists 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:205: left oid 34 (ObjNum 98 snap 25 seq_num 98) 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:206: finishing rollback tid 0 to smithi16534841-33 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:206: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:206: finishing rollback tid 2 to smithi16534841-33 2024-09-10T16:07:48.565 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 89 (ObjNum 68 snap 8 seq_num 68) dirty exists 2024-09-10T16:07:49.571 INFO:tasks.rados.rados.0.smithi165.stdout:205: done (2 left) 2024-09-10T16:07:49.571 INFO:tasks.rados.rados.0.smithi165.stdout:206: done (1 left) 2024-09-10T16:07:49.571 INFO:tasks.rados.rados.0.smithi165.stdout:208: done (0 left) 2024-09-10T16:07:49.571 INFO:tasks.rados.rados.0.smithi165.stdout:209: snap_create 2024-09-10T16:07:50.569 INFO:tasks.rados.rados.0.smithi165.stdout:209: done (0 left) 2024-09-10T16:07:50.569 INFO:tasks.rados.rados.0.smithi165.stdout:210: delete oid 18 current snap is 27 2024-09-10T16:07:50.579 INFO:tasks.rados.rados.0.smithi165.stdout:210: done (0 left) 2024-09-10T16:07:50.579 INFO:tasks.rados.rados.0.smithi165.stdout:211: delete oid 46 current snap is 27 2024-09-10T16:07:50.583 INFO:tasks.rados.rados.0.smithi165.stdout:211: done (0 left) 2024-09-10T16:07:50.583 INFO:tasks.rados.rados.0.smithi165.stdout:212: read oid 29 snap -1 2024-09-10T16:07:50.584 INFO:tasks.rados.rados.0.smithi165.stdout:212: expect (ObjNum 85 snap 20 seq_num 85) 2024-09-10T16:07:50.584 INFO:tasks.rados.rados.0.smithi165.stdout:213: delete oid 14 current snap is 27 2024-09-10T16:07:50.586 INFO:tasks.rados.rados.0.smithi165.stdout:213: done (1 left) 2024-09-10T16:07:50.586 INFO:tasks.rados.rados.0.smithi165.stdout:214: snap_remove snap 8 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:214: done (1 left) 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:215: read oid 49 snap -1 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:215: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:216: rollback oid 8 current snap is 27 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 19 2024-09-10T16:07:51.542 INFO:tasks.rados.rados.0.smithi165.stdout:212: expect (ObjNum 85 snap 20 seq_num 85) 2024-09-10T16:07:51.600 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:51.654 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:51.707 INFO:tasks.rados.rados.0.smithi165.stdout:215: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:216: finishing rollback tid 0 to smithi16534841-8 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:216: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:216: finishing rollback tid 2 to smithi16534841-8 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 121 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:212: done (2 left) 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:215: done (1 left) 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:216: done (0 left) 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:217: rollback oid 19 current snap is 27 2024-09-10T16:07:51.794 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 26 2024-09-10T16:07:51.795 INFO:tasks.rados.rados.0.smithi165.stdout:218: write oid 43 current snap is 27 2024-09-10T16:07:51.795 INFO:tasks.rados.rados.0.smithi165.stdout:218: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-10T16:07:51.800 INFO:tasks.rados.rados.0.smithi165.stdout:218: writing smithi16534841-43 from 514314 to 1302301 tid 1 2024-09-10T16:07:51.806 INFO:tasks.rados.rados.0.smithi165.stdout:218: writing smithi16534841-43 from 1768928 to 2174282 tid 2 2024-09-10T16:07:51.812 INFO:tasks.rados.rados.0.smithi165.stdout:218: writing smithi16534841-43 from 2768204 to 3277714 tid 3 2024-09-10T16:07:51.812 INFO:tasks.rados.rados.0.smithi165.stdout:219: snap_remove snap 24 2024-09-10T16:07:52.582 INFO:tasks.rados.rados.0.smithi165.stdout:219: done (2 left) 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:220: snap_create 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:217: finishing rollback tid 0 to smithi16534841-19 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:217: finishing rollback tid 1 to smithi16534841-19 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:217: finishing rollback tid 2 to smithi16534841-19 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 118 (ObjNum 54 snap 3 seq_num 54) dirty exists 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: finishing write tid 1 to smithi16534841-43 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: finishing write tid 2 to smithi16534841-43 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: finishing write tid 3 to smithi16534841-43 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: finishing write tid 4 to smithi16534841-43 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: finishing write tid 5 to smithi16534841-43 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: oid 43 updating version 0 to 55 2024-09-10T16:07:52.583 INFO:tasks.rados.rados.0.smithi165.stdout:218: oid 43 updating version 55 to 56 2024-09-10T16:07:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:218: oid 43 updating version 56 to 57 2024-09-10T16:07:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:218: oid 43 version 57 is already newer than 54 2024-09-10T16:07:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 57 (ObjNum 102 snap 27 seq_num 102) dirty exists 2024-09-10T16:07:52.584 INFO:tasks.rados.rados.0.smithi165.stdout:218: left oid 43 (ObjNum 102 snap 27 seq_num 102) 2024-09-10T16:07:53.582 INFO:tasks.rados.rados.0.smithi165.stdout:217: done (2 left) 2024-09-10T16:07:53.582 INFO:tasks.rados.rados.0.smithi165.stdout:218: done (1 left) 2024-09-10T16:07:53.582 INFO:tasks.rados.rados.0.smithi165.stdout:220: done (0 left) 2024-09-10T16:07:53.582 INFO:tasks.rados.rados.0.smithi165.stdout:221: read oid 3 snap 20 2024-09-10T16:07:53.582 INFO:tasks.rados.rados.0.smithi165.stdout:221: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:07:53.583 INFO:tasks.rados.rados.0.smithi165.stdout:222: copy_from oid 4 from oid 11 current snap is 28 2024-09-10T16:07:53.583 INFO:tasks.rados.rados.0.smithi165.stdout:223: write (excl) oid 29 current snap is 28 2024-09-10T16:07:53.583 INFO:tasks.rados.rados.0.smithi165.stdout:223: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2024-09-10T16:07:53.587 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16534841-29 from 531121 to 994357 tid 1 2024-09-10T16:07:53.595 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16534841-29 from 1711057 to 2376422 tid 2 2024-09-10T16:07:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:223: writing smithi16534841-29 from 3079274 to 3731121 tid 3 2024-09-10T16:07:53.603 INFO:tasks.rados.rados.0.smithi165.stdout:224: rollback oid 20 current snap is 28 2024-09-10T16:07:53.603 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 27 2024-09-10T16:07:53.603 INFO:tasks.rados.rados.0.smithi165.stdout:221: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:07:53.609 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1548424 2024-09-10T16:07:53.682 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 522202 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:222: finishing copy_from to smithi16534841-4 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:222: got expected ENOENT (src dne) 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:222: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 1 to smithi16534841-29 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 2 to smithi16534841-29 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:221: done (3 left) 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:222: done (2 left) 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:225: copy_from oid 19 from oid 22 current snap is 28 2024-09-10T16:07:53.721 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 3 to smithi16534841-29 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:224: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:224: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:224: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 105 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 4 to smithi16534841-29 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: finishing write tid 5 to smithi16534841-29 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 29 updating version 0 to 138 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 29 version 138 is already newer than 136 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 29 version 138 is already newer than 137 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: oid 29 version 138 is already newer than 135 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 138 (ObjNum 103 snap 28 seq_num 103) dirty exists 2024-09-10T16:07:53.722 INFO:tasks.rados.rados.0.smithi165.stdout:223: left oid 29 (ObjNum 103 snap 28 seq_num 103) 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:223: done (2 left) 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:224: done (1 left) 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:226: read oid 5 snap -1 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:226: expect (ObjNum 79 snap 18 seq_num 79) 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:227: copy_from oid 28 from oid 26 current snap is 28 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:228: rollback oid 44 current snap is 28 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 20 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:229: write oid 32 current snap is 28 2024-09-10T16:07:53.723 INFO:tasks.rados.rados.0.smithi165.stdout:229: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-10T16:07:53.725 INFO:tasks.rados.rados.0.smithi165.stdout:229: writing smithi16534841-32 from 547928 to 1086413 tid 1 2024-09-10T16:07:53.731 INFO:tasks.rados.rados.0.smithi165.stdout:229: writing smithi16534841-32 from 1769539 to 2178562 tid 2 2024-09-10T16:07:53.737 INFO:tasks.rados.rados.0.smithi165.stdout:229: writing smithi16534841-32 from 2706697 to 3238751 tid 3 2024-09-10T16:07:53.737 INFO:tasks.rados.rados.0.smithi165.stdout:230: write (excl) oid 27 current snap is 28 2024-09-10T16:07:53.737 INFO:tasks.rados.rados.0.smithi165.stdout:228: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:07:53.737 INFO:tasks.rados.rados.0.smithi165.stdout:230: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-10T16:07:53.743 INFO:tasks.rados.rados.0.smithi165.stdout:230: writing smithi16534841-27 from 564735 to 1178469 tid 1 2024-09-10T16:07:53.751 INFO:tasks.rados.rados.0.smithi165.stdout:230: writing smithi16534841-27 from 1828021 to 2497055 tid 2 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:230: writing smithi16534841-27 from 3134120 to 3764735 tid 3 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:231: write oid 10 current snap is 28 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:228: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:228: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 63 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-10T16:07:53.758 INFO:tasks.rados.rados.0.smithi165.stdout:231: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-10T16:07:53.765 INFO:tasks.rados.rados.0.smithi165.stdout:231: writing smithi16534841-10 from 581542 to 1270525 tid 1 2024-09-10T16:07:53.770 INFO:tasks.rados.rados.0.smithi165.stdout:231: writing smithi16534841-10 from 1886503 to 2299195 tid 2 2024-09-10T16:07:53.776 INFO:tasks.rados.rados.0.smithi165.stdout:231: writing smithi16534841-10 from 2761543 to 3432494 tid 3 2024-09-10T16:07:53.777 INFO:tasks.rados.rados.0.smithi165.stdout:228: done (6 left) 2024-09-10T16:07:53.777 INFO:tasks.rados.rados.0.smithi165.stdout:232: snap_create 2024-09-10T16:07:53.777 INFO:tasks.rados.rados.0.smithi165.stdout:226: expect (ObjNum 79 snap 18 seq_num 79) 2024-09-10T16:07:53.783 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 814739 2024-09-10T16:07:53.852 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3293044 2024-09-10T16:07:53.852 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:07:53.901 INFO:tasks.rados.rados.0.smithi165.stdout:227: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: finishing write tid 1 to smithi16534841-32 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:225: finishing copy_from racing read to smithi16534841-19 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:227: finishing copy_from to smithi16534841-28 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 53 (ObjNum 81 snap 18 seq_num 81) dirty exists 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: finishing write tid 2 to smithi16534841-32 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:225: finishing copy_from to smithi16534841-19 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: finishing write tid 3 to smithi16534841-32 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: finishing write tid 4 to smithi16534841-32 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: finishing write tid 5 to smithi16534841-32 2024-09-10T16:07:53.902 INFO:tasks.rados.rados.0.smithi165.stdout:229: oid 32 updating version 0 to 88 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:229: oid 32 version 88 is already newer than 87 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:229: oid 32 updating version 88 to 89 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:229: oid 32 version 89 is already newer than 86 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 89 (ObjNum 104 snap 28 seq_num 104) dirty exists 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:229: left oid 32 (ObjNum 104 snap 28 seq_num 104) 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing write tid 1 to smithi16534841-27 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing write tid 2 to smithi16534841-27 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing write tid 3 to smithi16534841-27 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing write tid 4 to smithi16534841-27 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: finishing write tid 5 to smithi16534841-27 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: oid 27 updating version 0 to 142 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: oid 27 version 142 is already newer than 140 2024-09-10T16:07:53.903 INFO:tasks.rados.rados.0.smithi165.stdout:230: oid 27 updating version 142 to 143 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:230: oid 27 version 143 is already newer than 141 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 143 (ObjNum 105 snap 28 seq_num 105) dirty exists 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:230: left oid 27 (ObjNum 105 snap 28 seq_num 105) 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: finishing write tid 1 to smithi16534841-10 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: finishing write tid 2 to smithi16534841-10 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: finishing write tid 3 to smithi16534841-10 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: finishing write tid 4 to smithi16534841-10 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: finishing write tid 5 to smithi16534841-10 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: oid 10 updating version 0 to 22 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: oid 10 version 22 is already newer than 20 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: oid 10 updating version 22 to 23 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:231: oid 10 version 23 is already newer than 21 2024-09-10T16:07:53.904 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 23 (ObjNum 106 snap 28 seq_num 106) dirty exists 2024-09-10T16:07:53.905 INFO:tasks.rados.rados.0.smithi165.stdout:231: left oid 10 (ObjNum 106 snap 28 seq_num 106) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:225: done (6 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:226: done (5 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:227: done (4 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:229: done (3 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:230: done (2 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:231: done (1 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:232: done (0 left) 2024-09-10T16:07:54.583 INFO:tasks.rados.rados.0.smithi165.stdout:233: rollback oid 31 current snap is 29 2024-09-10T16:07:54.584 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 23 2024-09-10T16:07:54.584 INFO:tasks.rados.rados.0.smithi165.stdout:234: snap_remove snap 7 2024-09-10T16:07:54.971 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:07:54.971 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:07:54.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:07:54.975 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:07:54.975 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-10T16:07:54.975 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-10T16:07:54.975 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:08:09.976 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-10T16:08:09.977 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-10T16:08:09.977 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:08:09.978 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.084+0000 7f83a76fe640 10 monclient: get_monmap_and_config 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.084+0000 7f83a76fe640 10 monclient: build_initial_monmap 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.084+0000 7f83a76fe640 10 monclient: monmap: 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:10.085551+0000 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:10.085551+0000 2024-09-10T16:08:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:10.087 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 1 Processor -- start 2024-09-10T16:08:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 1 -- start start 2024-09-10T16:08:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 10 monclient: init 2024-09-10T16:08:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.085+0000 7f83a76fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-c con 0x7f83a0073c80 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83a009d170 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83a00a5f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a00745a0 con 0x7f83a009d170 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a00a0110 con 0x7f83a00a5f70 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009cfa0 con 0x7f83a0073c80 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a76fe640 10 monclient(hunting): authenticate will time out at 868.573120s 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a4c99640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f83a009d170 0x7f83a00a5860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59846/0 (socket says 172.21.15.164:59846) 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 learned_addr learned my addr 172.21.15.164:0/2145620601 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:08:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83a0073c80 auth_method 0 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.088+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.164:6789/0 conn(0x7f83a009d170 legacy=0x7f83a00a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.088+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.165:6789/0 conn(0x7f83a00a5f70 legacy=0x7f83a013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83a0073c80 auth_method 0 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:10.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.288+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83a0073c80 auth_method 0 2024-09-10T16:08:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:10.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83a0073c80 auth_method 0 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:11.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:11.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:11.491+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:13.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): tick 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 msgr2=0x7f83a0074060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f83a0073c80 0x7f83a0074060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.165:6789/0 conn(0x7f83a00a5f70 legacy=0x7f83a013dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.164:6789/0 conn(0x7f83a009d170 legacy=0x7f83a00a5860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient: _add_conns ranks=[3,2] 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f83840037f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:13.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f8384006350 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009cfa0 con 0x7f83840037f0 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8384008f20 con 0x7f8384006350 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f8396ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83840037f0 auth_method 0 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f8384006350 auth_method 0 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:13.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.086+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83840037f0 auth_method 0 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f8384006350 auth_method 0 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:13.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.287+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f8384006350 auth_method 0 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83840037f0 auth_method 0 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:13.689+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f8384006350 auth_method 0 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f83840037f0 auth_method 0 2024-09-10T16:08:14.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:14.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:14.490+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): tick 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 msgr2=0x7f83840087f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8384006350 0x7f83840087f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 msgr2=0x7f8384005c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f83840037f0 0x7f8384005c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:16.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 reap_dead start 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8384003c60 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8384007f00 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f838400c3a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a00a03f0 con 0x7f8384007f00 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838400fce0 con 0x7f8384003c60 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f83a009cfa0 con 0x7f838400c3a0 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f8396ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f838400c3a0 auth_method 0 2024-09-10T16:08:16.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.086+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.088+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.164:6789/0 conn(0x7f8384007f00 legacy=0x7f83840082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:16.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.089+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.165:6789/0 conn(0x7f8384003c60 legacy=0x7f83840067e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:16.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:16.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f838400c3a0 auth_method 0 2024-09-10T16:08:16.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:16.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:16.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:16.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:16.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:16.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:16.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.287+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:16.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f838400c3a0 auth_method 0 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:16.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:16.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:16.688+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:17.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:17.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f838400c3a0 auth_method 0 2024-09-10T16:08:17.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:17.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:17.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:17.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:17.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:17.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:17.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:17.489+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:19.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): tick 2024-09-10T16:08:19.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:19.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:19.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 msgr2=0x7f838400e790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400c3a0 0x7f838400e790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.165:6789/0 conn(0x7f8384003c60 legacy=0x7f83840067e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v1:172.21.15.164:6789/0 conn(0x7f8384007f00 legacy=0x7f83840082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f8384010190 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f838400ecd0 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009cfa0 con 0x7f8384010190 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8384008f20 con 0x7f838400ecd0 2024-09-10T16:08:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.086+0000 7f8396ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:19.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:19.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:19.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.289+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:19.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:19.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:19.691+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:20.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:20.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:20.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:20.492+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:20.595 INFO:tasks.rados.rados.0.smithi165.stdout:234: done (1 left) 2024-09-10T16:08:20.595 INFO:tasks.rados.rados.0.smithi165.stdout:235: write oid 18 current snap is 29 2024-09-10T16:08:20.595 INFO:tasks.rados.rados.0.smithi165.stdout:235: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-10T16:08:20.600 INFO:tasks.rados.rados.0.smithi165.stdout:235: writing smithi16534841-18 from 598349 to 1078934 tid 1 2024-09-10T16:08:20.607 INFO:tasks.rados.rados.0.smithi165.stdout:235: writing smithi16534841-18 from 1544985 to 2217688 tid 2 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:235: writing smithi16534841-18 from 2905319 to 3387542 tid 3 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:236: snap_create 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:233: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 126 (ObjNum 82 snap 18 seq_num 82) dirty exists 2024-09-10T16:08:20.614 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing write tid 1 to smithi16534841-18 2024-09-10T16:08:20.615 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing write tid 2 to smithi16534841-18 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing write tid 3 to smithi16534841-18 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing write tid 4 to smithi16534841-18 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: finishing write tid 5 to smithi16534841-18 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: oid 18 updating version 0 to 97 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: oid 18 updating version 97 to 98 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: oid 18 updating version 98 to 100 2024-09-10T16:08:20.618 INFO:tasks.rados.rados.0.smithi165.stdout:235: oid 18 version 100 is already newer than 99 2024-09-10T16:08:20.619 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 100 (ObjNum 107 snap 29 seq_num 107) dirty exists 2024-09-10T16:08:20.619 INFO:tasks.rados.rados.0.smithi165.stdout:235: left oid 18 (ObjNum 107 snap 29 seq_num 107) 2024-09-10T16:08:21.600 INFO:tasks.rados.rados.0.smithi165.stdout:233: done (2 left) 2024-09-10T16:08:21.600 INFO:tasks.rados.rados.0.smithi165.stdout:235: done (1 left) 2024-09-10T16:08:21.600 INFO:tasks.rados.rados.0.smithi165.stdout:236: done (0 left) 2024-09-10T16:08:21.600 INFO:tasks.rados.rados.0.smithi165.stdout:237: write (excl) oid 15 current snap is 30 2024-09-10T16:08:21.600 INFO:tasks.rados.rados.0.smithi165.stdout:237: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2024-09-10T16:08:21.606 INFO:tasks.rados.rados.0.smithi165.stdout:237: writing smithi16534841-15 from 615156 to 1170990 tid 1 2024-09-10T16:08:21.610 INFO:tasks.rados.rados.0.smithi165.stdout:237: writing smithi16534841-15 from 1603467 to 2019828 tid 2 2024-09-10T16:08:21.619 INFO:tasks.rados.rados.0.smithi165.stdout:237: writing smithi16534841-15 from 2816389 to 3509884 tid 3 2024-09-10T16:08:21.619 INFO:tasks.rados.rados.0.smithi165.stdout:238: read oid 3 snap -1 2024-09-10T16:08:21.619 INFO:tasks.rados.rados.0.smithi165.stdout:238: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:08:21.619 INFO:tasks.rados.rados.0.smithi165.stdout:239: write (excl) oid 1 current snap is 30 2024-09-10T16:08:21.619 INFO:tasks.rados.rados.0.smithi165.stdout:237: finishing write tid 1 to smithi16534841-15 2024-09-10T16:08:21.620 INFO:tasks.rados.rados.0.smithi165.stdout:237: finishing write tid 2 to smithi16534841-15 2024-09-10T16:08:21.620 INFO:tasks.rados.rados.0.smithi165.stdout:239: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-10T16:08:21.626 INFO:tasks.rados.rados.0.smithi165.stdout:239: writing smithi16534841-1 from 631963 to 1263046 tid 1 2024-09-10T16:08:21.634 INFO:tasks.rados.rados.0.smithi165.stdout:239: writing smithi16534841-1 from 2061949 to 2621968 tid 2 2024-09-10T16:08:21.640 INFO:tasks.rados.rados.0.smithi165.stdout:239: writing smithi16534841-1 from 3243812 to 3748579 tid 3 2024-09-10T16:08:21.640 INFO:tasks.rados.rados.0.smithi165.stdout:240: delete oid 9 current snap is 30 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: finishing write tid 3 to smithi16534841-15 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: finishing write tid 4 to smithi16534841-15 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: finishing write tid 5 to smithi16534841-15 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: oid 15 updating version 0 to 54 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: oid 15 version 54 is already newer than 53 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: oid 15 updating version 54 to 55 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: oid 15 version 55 is already newer than 52 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 55 (ObjNum 108 snap 30 seq_num 108) dirty exists 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:237: left oid 15 (ObjNum 108 snap 30 seq_num 108) 2024-09-10T16:08:21.641 INFO:tasks.rados.rados.0.smithi165.stdout:238: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:08:21.730 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing write tid 1 to smithi16534841-1 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing write tid 2 to smithi16534841-1 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing write tid 3 to smithi16534841-1 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing write tid 4 to smithi16534841-1 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: finishing write tid 5 to smithi16534841-1 2024-09-10T16:08:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:239: oid 1 updating version 0 to 104 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:239: oid 1 version 104 is already newer than 103 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:239: oid 1 version 104 is already newer than 101 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:239: oid 1 version 104 is already newer than 102 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 104 (ObjNum 109 snap 30 seq_num 109) dirty exists 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:239: left oid 1 (ObjNum 109 snap 30 seq_num 109) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:237: done (3 left) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:238: done (2 left) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:239: done (1 left) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:240: done (0 left) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:241: read oid 41 snap 10 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:241: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-10T16:08:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:242: delete oid 14 current snap is 30 2024-09-10T16:08:21.777 INFO:tasks.rados.rados.0.smithi165.stdout:242: done (1 left) 2024-09-10T16:08:21.777 INFO:tasks.rados.rados.0.smithi165.stdout:243: read oid 44 snap 18 2024-09-10T16:08:21.777 INFO:tasks.rados.rados.0.smithi165.stdout:243: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-10T16:08:21.777 INFO:tasks.rados.rados.0.smithi165.stdout:244: write oid 13 current snap is 30 2024-09-10T16:08:21.777 INFO:tasks.rados.rados.0.smithi165.stdout:244: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-10T16:08:21.782 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16534841-13 from 682384 to 1139214 tid 1 2024-09-10T16:08:21.788 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16534841-13 from 1721042 to 2144741 tid 2 2024-09-10T16:08:21.794 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16534841-13 from 2809728 to 3264664 tid 3 2024-09-10T16:08:21.798 INFO:tasks.rados.rados.0.smithi165.stdout:244: writing smithi16534841-13 from 3731357 to 3882384 tid 4 2024-09-10T16:08:21.798 INFO:tasks.rados.rados.0.smithi165.stdout:245: snap_create 2024-09-10T16:08:21.798 INFO:tasks.rados.rados.0.smithi165.stdout:243: expect (ObjNum 59 snap 3 seq_num 59) 2024-09-10T16:08:21.917 INFO:tasks.rados.rados.0.smithi165.stdout:241: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-10T16:08:21.945 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:21.974 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 1 to smithi16534841-13 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 2 to smithi16534841-13 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 3 to smithi16534841-13 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 4 to smithi16534841-13 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 5 to smithi16534841-13 2024-09-10T16:08:22.003 INFO:tasks.rados.rados.0.smithi165.stdout:244: finishing write tid 6 to smithi16534841-13 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 13 updating version 0 to 32 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 13 version 32 is already newer than 30 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 13 version 32 is already newer than 31 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 13 updating version 32 to 34 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: oid 13 version 34 is already newer than 33 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 34 (ObjNum 112 snap 30 seq_num 112) dirty exists 2024-09-10T16:08:22.004 INFO:tasks.rados.rados.0.smithi165.stdout:244: left oid 13 (ObjNum 112 snap 30 seq_num 112) 2024-09-10T16:08:22.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): tick 2024-09-10T16:08:22.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:22.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 msgr2=0x7f83840105a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 1 -- 172.21.15.164:0/2145620601 reap_dead start 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f838400ecd0 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f8384010190 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:22.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400f7c0 0x7f83840082e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f838400f7c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009cfa0 con 0x7f8384010190 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8384008f20 con 0x7f838400ecd0 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 1 -- 172.21.15.164:0/2145620601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8384008820 con 0x7f838400f7c0 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f8396ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400f7c0 0x7f83840082e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.164:3300/0 conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f838400f7c0 auth_method 0 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient(hunting): handle_auth_done global_id 24099 payload 274 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400f7c0 msgr2=0x7f83840082e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400f7c0 0x7f83840082e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 msgr2=0x7f838400f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.087+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8384010fb0 con 0x7f8384010190 2024-09-10T16:08:22.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.090+0000 7f83a4c99640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384010190 msgr2=0x7f83840105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.090+0000 7f83a4c99640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384010190 0x7f83840105a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8384008c40 tx=0x7f839002c110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87d54750672725a5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 1 -- 172.21.15.164:0/2145620601 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390032070 con 0x7f8384010190 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: dump: 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 5 monclient: authenticate success, global_id 24099 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 1 -- 172.21.15.164:0/2145620601 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839002ca10 con 0x7f8384010190 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 1 -- 172.21.15.164:0/2145620601 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839002cde0 con 0x7f8384010190 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 4 set_mon_vals no callback set 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f8397fff640 10 monclient: dump: 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 10 monclient: shutdown 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 1 -- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384010190 msgr2=0x7f83840105a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.091+0000 7f83a76fe640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384010190 0x7f83840105a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8384008c40 tx=0x7f839002c110 comp rx=0 tx=0).stop 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 -- 172.21.15.164:0/2145620601 shutdown_connections 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400f7c0 0x7f83840082e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 --2- 172.21.15.164:0/2145620601 >> v2:172.21.15.165:3300/0 conn(0x7f838400ecd0 0x7f838400f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 --2- 172.21.15.164:0/2145620601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384010190 0x7f83840105a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 -- 172.21.15.164:0/2145620601 >> 172.21.15.164:0/2145620601 conn(0x7f83a00a6f80 msgr2=0x7f83a00a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 -- 172.21.15.164:0/2145620601 shutdown_connections 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.092+0000 7f83a76fe640 1 -- 172.21.15.164:0/2145620601 wait complete. 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.093+0000 7f83a76fe640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.093+0000 7f83a76fe640 10 monclient: monmap: 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:22.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.093+0000 7f83a76fe640 1 Processor -- start 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.093+0000 7f83a76fe640 1 -- start start 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.093+0000 7f83a76fe640 10 monclient: init 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:08:22.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-c con 0x7f838400ecd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 0x7f83a012f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-a con 0x7f838400f7c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 0x7f83a01bb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): picked mon.noname-b con 0x7f8384010190 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00a3f50 con 0x7f838400f7c0 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00a3dd0 con 0x7f8384010190 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f83a01329a0 con 0x7f838400ecd0 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a76fe640 10 monclient(hunting): authenticate will time out at 880.581315s 2024-09-10T16:08:22.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 0x7f83a012f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34054/0 (socket says 172.21.15.164:34054) 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 -- 172.21.15.164:0/4184224403 learned_addr learned my addr 172.21.15.164:0/4184224403 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): get_auth_request con 0x7f838400ecd0 auth_method 0 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): get_auth_request con 0x7f838400f7c0 auth_method 0 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 0x7f83a01bb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request con 0x7f8384010190 auth_method 0 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 -- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 msgr2=0x7f83a01bb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 0x7f83a01bb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 -- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 msgr2=0x7f83a012f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 0x7f83a012f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 monclient: found mon.noname-c 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a549a640 1 -- 172.21.15.164:0/4184224403 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a01bf020 con 0x7f838400ecd0 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a4c99640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 0x7f83a012f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.094+0000 7f83a5c9b640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 0x7f83a01bb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a549a640 1 -- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 msgr2=0x7f83a012f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a549a640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f839c018aa0 tx=0x7f839c018ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c029070 con 0x7f838400ecd0 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: dump: 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.110 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839c025430 con 0x7f838400ecd0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 5 monclient: authenticate success, global_id 4508 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 10 monclient: _renew_subs 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a01bf300 con 0x7f838400ecd0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839c025800 con 0x7f838400ecd0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f8395ffb640 10 monclient: dump: 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 10 monclient: _renew_subs 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83977fe640 4 set_mon_vals no callback set 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.097+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a01be3a0 con 0x7f838400ecd0 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.098+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f839c02c070 con 0x7f838400ecd0 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f8395ffb640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 0x7f83840435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(71..71 src has 1..71) ==== 4771+0+0 (secure 0 0 0) 0x7f839c0550e0 con 0x7f838400ecd0 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a76fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a76fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a76fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83a00a6400 con 0x7f838400ecd0 2024-09-10T16:08:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a4c99640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 0x7f83840435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.099+0000 7f83a4c99640 10 monclient: get_auth_request con 0x7f8384041130 auth_method 0 2024-09-10T16:08:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.102+0000 7f83a4c99640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 0x7f83840435f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8384010160 tx=0x7f8390032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.104+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f839c01f040 con 0x7f838400ecd0 2024-09-10T16:08:22.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.104+0000 7f8395ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.104+0000 7f8395ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.349+0000 7f83a76fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.349+0000 7f83a76fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.349+0000 7f83a76fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.349+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f83a0073a60 con 0x7f838400ecd0 2024-09-10T16:08:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.350+0000 7f8395ffb640 1 -- 172.21.15.164:0/4184224403 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f839c023d10 con 0x7f838400ecd0 2024-09-10T16:08:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.350+0000 7f8395ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.350+0000 7f8395ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:22.352 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:22.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 msgr2=0x7f83840435f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 0x7f83840435f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f8384010160 tx=0x7f8390032000 comp rx=0 tx=0).stop 2024-09-10T16:08:22.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 10 monclient: shutdown 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 msgr2=0x7f83a012f4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f839c018aa0 tx=0x7f839c018ea0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 shutdown_connections 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8384041130 0x7f83840435f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384010190 0x7f83a01bb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f838400f7c0 0x7f83a012f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 --2- 172.21.15.164:0/4184224403 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f838400ecd0 0x7f83a012f4a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 >> 172.21.15.164:0/4184224403 conn(0x7f83a00a6f80 msgr2=0x7f83a01303e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 shutdown_connections 2024-09-10T16:08:22.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.352+0000 7f83a76fe640 1 -- 172.21.15.164:0/4184224403 wait complete. 2024-09-10T16:08:22.365 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:08:22.365 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:08:22.365 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.463+0000 7fe74aa7f640 10 monclient: get_monmap_and_config 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.463+0000 7fe74aa7f640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.463+0000 7fe74aa7f640 10 monclient: monmap: 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:22.464556+0000 2024-09-10T16:08:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:22.464556+0000 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:22.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 Processor -- start 2024-09-10T16:08:22.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 -- start start 2024-09-10T16:08:22.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient: init 2024-09-10T16:08:22.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe74409c310 0x7fe74409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-a con 0x7fe74409c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-b con 0x7fe74409d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe74409f4c0 0x7fe74413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-c con 0x7fe74409f4c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe74409fbc0 con 0x7fe74409c310 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe7440a2a20 con 0x7fe74409f4c0 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe74409c6f0 con 0x7fe74409d190 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74aa7f640 10 monclient(hunting): authenticate will time out at 880.951877s 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe743fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.464+0000 7fe74881b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe74409c310 0x7fe74409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34064/0 (socket says 172.21.15.164:34064) 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 1 -- 172.21.15.164:0/4252691686 learned_addr learned my addr 172.21.15.164:0/4252691686 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): get_auth_request con 0x7fe74409c310 auth_method 0 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): get_auth_request con 0x7fe74409d190 auth_method 0 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74901c640 1 --2- 172.21.15.164:0/4252691686 >> v2:172.21.15.165:3300/0 conn(0x7fe74409f4c0 0x7fe74413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74901c640 10 monclient(hunting): get_auth_request con 0x7fe74409f4c0 auth_method 0 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74901c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74901c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74901c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient(hunting): handle_auth_done global_id 24105 payload 274 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 -- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 msgr2=0x7fe74409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 -- 172.21.15.164:0/4252691686 >> v2:172.21.15.165:3300/0 conn(0x7fe74409f4c0 msgr2=0x7fe74413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 --2- 172.21.15.164:0/4252691686 >> v2:172.21.15.165:3300/0 conn(0x7fe74409f4c0 0x7fe74413b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe74881b640 1 -- 172.21.15.164:0/4252691686 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe74409a4d0 con 0x7fe74409c310 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.465+0000 7fe743fff640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74881b640 1 -- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409c310 msgr2=0x7fe74409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74881b640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409c310 0x7fe74409ea40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe744074d80 tx=0x7fe73802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb626958665e80aa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 1 -- 172.21.15.164:0/4252691686 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738037070 con 0x7fe74409c310 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: dump: 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 5 monclient: authenticate success, global_id 24105 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 1 -- 172.21.15.164:0/4252691686 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe738032440 con 0x7fe74409c310 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:22.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 1 -- 172.21.15.164:0/4252691686 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738032810 con 0x7fe74409c310 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 4 set_mon_vals no callback set 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe7437fe640 10 monclient: dump: 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 10 monclient: shutdown 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 -- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409c310 msgr2=0x7fe74409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409c310 0x7fe74409ea40 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe744074d80 tx=0x7fe73802a000 comp rx=0 tx=0).stop 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 -- 172.21.15.164:0/4252691686 shutdown_connections 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/4252691686 >> v2:172.21.15.165:3300/0 conn(0x7fe74409f4c0 0x7fe74413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409d190 0x7fe74409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/4252691686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409c310 0x7fe74409ea40 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 -- 172.21.15.164:0/4252691686 >> 172.21.15.164:0/4252691686 conn(0x7fe7440a3110 msgr2=0x7fe7440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 -- 172.21.15.164:0/4252691686 shutdown_connections 2024-09-10T16:08:22.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.468+0000 7fe74aa7f640 1 -- 172.21.15.164:0/4252691686 wait complete. 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.469+0000 7fe74aa7f640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.469+0000 7fe74aa7f640 10 monclient: monmap: 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:22.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.469+0000 7fe74aa7f640 1 Processor -- start 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.469+0000 7fe74aa7f640 1 -- start start 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.469+0000 7fe74aa7f640 10 monclient: init 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe74409c310 0x7fe744099470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-b con 0x7fe74409c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 0x7fe7440999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-c con 0x7fe74409f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): picked mon.noname-a con 0x7fe7440922c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe7440a2230 con 0x7fe7440922c0 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe7440a20b0 con 0x7fe74409c310 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe744074e00 con 0x7fe74409f4c0 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74aa7f640 10 monclient(hunting): authenticate will time out at 880.957418s 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 0x7fe7440999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 0x7fe7440999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34070/0 (socket says 172.21.15.164:34070) 2024-09-10T16:08:22.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57362/0 (socket says 172.21.15.164:57362) 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 1 -- 172.21.15.164:0/3235149599 learned_addr learned my addr 172.21.15.164:0/3235149599 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): get_auth_request con 0x7fe74409f4c0 auth_method 0 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe74409c310 0x7fe744099470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): get_auth_request con 0x7fe7440922c0 auth_method 0 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): get_auth_request con 0x7fe74409c310 auth_method 0 2024-09-10T16:08:22.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient(hunting): handle_auth_done global_id 24108 payload 824 2024-09-10T16:08:22.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 msgr2=0x7fe7440999b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 0x7fe7440999b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe74409c310 msgr2=0x7fe744099470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe74409c310 0x7fe744099470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74901c640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7441cb0e0 con 0x7fe7440922c0 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe743fff640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe74409f4c0 0x7fe7440999b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.470+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe74409c310 0x7fe744099470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74901c640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 msgr2=0x7fe744092740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74901c640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe73c007d90 tx=0x7fe73c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c003990 con 0x7fe7440922c0 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: dump: 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.491 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe73c003b30 con 0x7fe7440922c0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 5 monclient: authenticate success, global_id 24108 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 10 monclient: _renew_subs 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7440a5060 con 0x7fe7440922c0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe73c004020 con 0x7fe7440922c0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe7417fa640 10 monclient: dump: 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe742ffd640 4 set_mon_vals no callback set 2024-09-10T16:08:22.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 10 monclient: _renew_subs 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.473+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe74409d620 con 0x7fe7440922c0 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74aa7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 0x7fe744092c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe7441cbb60 con 0x7fe74409d190 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe73c01b020 con 0x7fe7440922c0 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 0x7fe744092c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient: get_auth_request con 0x7fe74409d190 auth_method 0 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 10 monclient(hunting): handle_auth_done global_id 24111 payload 824 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe7417fa640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe72803e210 0x7fe7280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe74881b640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe74409ea40 con 0x7fe74409d190 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 4771+0+0 (secure 0 0 0) 0x7fe73c052e40 con 0x7fe7440922c0 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe743fff640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe72803e210 0x7fe7280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.474+0000 7fe743fff640 10 monclient: get_auth_request con 0x7fe72803e210 auth_method 0 2024-09-10T16:08:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe74881b640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 msgr2=0x7fe744092c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 0x7fe744092c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe74406d4d0 tx=0x7fe7380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738037040 con 0x7fe74409d190 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe7417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe7417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.476+0000 7fe7417fa640 10 monclient: dump: 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.498 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe73803c830 con 0x7fe74409d190 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 msgr2=0x7fe744092c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 0x7fe744092c80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe74406d4d0 tx=0x7fe7380046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 msgr2=0x7fe744092c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe7417fa640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe74409d190 0x7fe744092c80 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fe74406d4d0 tx=0x7fe7380046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.477+0000 7fe743fff640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe72803e210 0x7fe7280406d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe744095120 tx=0x7fe730009240 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74aa7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 0x7fe744072330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74aa7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe7440947e0 con 0x7fe744071f50 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 0x7fe744072330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient: get_auth_request con 0x7fe744071f50 auth_method 0 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 10 monclient(hunting): handle_auth_done global_id 24114 payload 824 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.483+0000 7fe74881b640 1 -- 172.21.15.164:0/3235149599 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe744093710 con 0x7fe744071f50 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe74881b640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 msgr2=0x7fe744072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe74881b640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 0x7fe744072330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe73802a040 tx=0x7fe738008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe738046b60 con 0x7fe744071f50 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 10 monclient: dump: 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.501 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fe73804f430 con 0x7fe744071f50 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 msgr2=0x7fe744072330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.485+0000 7fe7417fa640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 0x7fe744072330 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe73802a040 tx=0x7fe738008030 comp rx=0 tx=0).stop 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe7417fa640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 msgr2=0x7fe744072330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe7417fa640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe744071f50 0x7fe744072330 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe73802a040 tx=0x7fe738008030 comp rx=0 tx=0).stop 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:22.502 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27418, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:22.503 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.504 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.505 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:22.506 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.507 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.508 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.509 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:22.511 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.512 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.513 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:22.514 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe72803e210 msgr2=0x7fe7280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe72803e210 0x7fe7280406d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fe744095120 tx=0x7fe730009240 comp rx=0 tx=0).stop 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 10 monclient: shutdown 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 msgr2=0x7fe744092740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe73c007d90 tx=0x7fe73c00e4b0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74901c640 1 -- 172.21.15.164:0/3235149599 reap_dead start 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 shutdown_connections 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 --2- 172.21.15.164:0/3235149599 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7440922c0 0x7fe744092740 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 >> 172.21.15.164:0/3235149599 conn(0x7fe7440a3110 msgr2=0x7fe74409af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 shutdown_connections 2024-09-10T16:08:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.486+0000 7fe74aa7f640 1 -- 172.21.15.164:0/3235149599 wait complete. 2024-09-10T16:08:22.517 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:22.517 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:08:22.517 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:08:22.605 INFO:tasks.rados.rados.0.smithi165.stdout:241: done (3 left) 2024-09-10T16:08:22.605 INFO:tasks.rados.rados.0.smithi165.stdout:243: done (2 left) 2024-09-10T16:08:22.605 INFO:tasks.rados.rados.0.smithi165.stdout:244: done (1 left) 2024-09-10T16:08:22.605 INFO:tasks.rados.rados.0.smithi165.stdout:245: done (0 left) 2024-09-10T16:08:22.605 INFO:tasks.rados.rados.0.smithi165.stdout:246: write oid 26 current snap is 31 2024-09-10T16:08:22.606 INFO:tasks.rados.rados.0.smithi165.stdout:246: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-10T16:08:22.611 INFO:tasks.rados.rados.0.smithi165.stdout:246: writing smithi16534841-26 from 699191 to 1231270 tid 1 2024-09-10T16:08:22.617 INFO:tasks.rados.rados.0.smithi165.stdout:246: writing smithi16534841-26 from 1779524 to 2346881 tid 2 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.616+0000 7fb6fb0b0640 10 monclient: get_monmap_and_config 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.616+0000 7fb6fb0b0640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.616+0000 7fb6fb0b0640 10 monclient: monmap: 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:22.617709+0000 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:22.617709+0000 2024-09-10T16:08:22.618 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:22.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.617+0000 7fb6fb0b0640 1 Processor -- start 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.617+0000 7fb6fb0b0640 1 -- start start 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.617+0000 7fb6fb0b0640 10 monclient: init 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f40a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6f40a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6f40a27e0 0x7fb6f413dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f40a27e0 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6f40a0210 con 0x7fb6f40a2380 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb6f409d3f0 con 0x7fb6f40a27e0 2024-09-10T16:08:22.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f409ed90 con 0x7fb6f40a0410 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6fb0b0640 10 monclient(hunting): authenticate will time out at 881.105451s 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6f40a27e0 0x7fb6f413dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34082/0 (socket says 172.21.15.164:34082) 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/3958436773 learned_addr learned my addr 172.21.15.164:0/3958436773 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): get_auth_request con 0x7fb6f40a0410 auth_method 0 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): get_auth_request con 0x7fb6f40a27e0 auth_method 0 2024-09-10T16:08:22.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient(hunting): handle_auth_done global_id 24117 payload 274 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 -- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 msgr2=0x7fb6f40a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 -- 172.21.15.164:0/3958436773 >> v1:172.21.15.164:6789/0 conn(0x7fb6f40a2380 legacy=0x7fb6f413a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f964d640 1 -- 172.21.15.164:0/3958436773 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f409d930 con 0x7fb6f40a27e0 2024-09-10T16:08:22.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.618+0000 7fb6f8e4c640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.626 INFO:tasks.rados.rados.0.smithi165.stdout:246: writing smithi16534841-26 from 2837151 to 3503359 tid 3 2024-09-10T16:08:22.626 INFO:tasks.rados.rados.0.smithi165.stdout:247: write (excl) oid 43 current snap is 31 2024-09-10T16:08:22.626 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing write tid 1 to smithi16534841-26 2024-09-10T16:08:22.626 INFO:tasks.rados.rados.0.smithi165.stdout:247: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-10T16:08:22.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f964d640 1 -- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a27e0 msgr2=0x7fb6f413dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f964d640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a27e0 0x7fb6f413dcf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4074db0 tx=0x7fb6ec029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d98ec740733f09cb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 1 -- 172.21.15.164:0/3958436773 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec037070 con 0x7fb6f40a27e0 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: dump: 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.633 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6fb0b0640 5 monclient: authenticate success, global_id 24117 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 1 -- 172.21.15.164:0/3958436773 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6ec032440 con 0x7fb6f40a27e0 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6fb0b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6fb0b0640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 1 -- 172.21.15.164:0/3958436773 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6ec032810 con 0x7fb6f40a27e0 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6fb0b0640 4 set_mon_vals no callback set 2024-09-10T16:08:22.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.621+0000 7fb6f37fe640 10 monclient: dump: 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 10 monclient: shutdown 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a27e0 msgr2=0x7fb6f413dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a27e0 0x7fb6f413dcf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb6f4074db0 tx=0x7fb6ec029fb0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/3958436773 shutdown_connections 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a27e0 0x7fb6f413dcf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/3958436773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a0410 0x7fb6f40a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/3958436773 >> 172.21.15.164:0/3958436773 conn(0x7fb6f40a3110 msgr2=0x7fb6f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/3958436773 shutdown_connections 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.622+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/3958436773 wait complete. 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.623+0000 7fb6fb0b0640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.623+0000 7fb6fb0b0640 10 monclient: monmap: 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.623+0000 7fb6fb0b0640 1 Processor -- start 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.623+0000 7fb6fb0b0640 1 -- start start 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.623+0000 7fb6fb0b0640 10 monclient: init 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 0x7fb6f41bac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-a con 0x7fb6f40a0410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a2380 0x7fb6f41b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-c con 0x7fb6f40a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6f40a27e0 0x7fb6f41b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): picked mon.noname-b con 0x7fb6f40a27e0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f409c910 con 0x7fb6f40a0410 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6f409eab0 con 0x7fb6f40a27e0 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f40a07f0 con 0x7fb6f40a2380 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6fb0b0640 10 monclient(hunting): authenticate will time out at 881.111222s 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 0x7fb6f41bac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a2380 0x7fb6f41b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 0x7fb6f41bac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57396/0 (socket says 172.21.15.164:57396) 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/2639066969 learned_addr learned my addr 172.21.15.164:0/2639066969 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): get_auth_request con 0x7fb6f40a2380 auth_method 0 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): get_auth_request con 0x7fb6f40a0410 auth_method 0 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f964d640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6f40a27e0 0x7fb6f41b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f964d640 10 monclient(hunting): get_auth_request con 0x7fb6f40a27e0 auth_method 0 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f964d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f964d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f964d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient(hunting): handle_auth_done global_id 24120 payload 824 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a2380 msgr2=0x7fb6f41b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f40a2380 0x7fb6f41b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6f40a27e0 msgr2=0x7fb6f41b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6f40a27e0 0x7fb6f41b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.624+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6f41b4ee0 con 0x7fb6f40a0410 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.626+0000 7fb6f8e4c640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 msgr2=0x7fb6f41bac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.626+0000 7fb6f8e4c640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 0x7fb6f41bac40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400efc0 tx=0x7fb6e400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4014680 con 0x7fb6f40a0410 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: dump: 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.641 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e4019070 con 0x7fb6f40a0410 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 5 monclient: authenticate success, global_id 24120 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 10 monclient: _renew_subs 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6f41b51c0 con 0x7fb6f40a0410 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e4014a20 con 0x7fb6f40a0410 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f17fa640 10 monclient: dump: 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.643 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 10 monclient: _renew_subs 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6f41b33e0 con 0x7fb6f40a0410 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.627+0000 7fb6f2ffd640 4 set_mon_vals no callback set 2024-09-10T16:08:22.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6fb0b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 0x7fb6f41b26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f41b2be0 con 0x7fb6f41b22c0 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb6e4005d10 con 0x7fb6f40a0410 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 0x7fb6f41b26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient: get_auth_request con 0x7fb6f41b22c0 auth_method 0 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f17fa640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6e003e260 0x7fb6e0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4771+0+0 (secure 0 0 0) 0x7fb6e4053280 con 0x7fb6f40a0410 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_done global_id 4523 payload 824 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f3fff640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6f41bb250 con 0x7fb6f41b22c0 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f964d640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6e003e260 0x7fb6e0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.628+0000 7fb6f964d640 10 monclient: get_auth_request con 0x7fb6e003e260 auth_method 0 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f3fff640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 msgr2=0x7fb6f41b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f3fff640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 0x7fb6f41b26a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb6f41b3c30 tx=0x7fb6e800fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8018030 con 0x7fb6f41b22c0 2024-09-10T16:08:22.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.630+0000 7fb6f17fa640 10 monclient: dump: 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f964d640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6e003e260 0x7fb6e0040720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb6f406dad0 tx=0x7fb6ec02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6e800d980 con 0x7fb6f41b22c0 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 msgr2=0x7fb6f41b26a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 0x7fb6f41b26a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb6f41b3c30 tx=0x7fb6e800fc00 comp rx=0 tx=0).stop 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 msgr2=0x7fb6f41b26a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.631+0000 7fb6f17fa640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f41b22c0 0x7fb6f41b26a0 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fb6f41b3c30 tx=0x7fb6e800fc00 comp rx=0 tx=0).stop 2024-09-10T16:08:22.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6fb0b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:22.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 0x7fb6f4071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6fb0b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6f409faa0 con 0x7fb6f4071430 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 0x7fb6f4071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient: get_auth_request con 0x7fb6f4071430 auth_method 0 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.639+0000 7fb6f3fff640 10 monclient(hunting): handle_auth_done global_id 4526 payload 824 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.640+0000 7fb6f3fff640 1 -- 172.21.15.164:0/2639066969 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6f4074710 con 0x7fb6f4071430 2024-09-10T16:08:22.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f3fff640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 msgr2=0x7fb6f4071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f3fff640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 0x7fb6f4071840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8018000 tx=0x7fb6e8003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8007440 con 0x7fb6f4071430 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 10 monclient: dump: 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.653 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb6e8007440 con 0x7fb6f4071430 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 msgr2=0x7fb6f4071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 0x7fb6f4071840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8018000 tx=0x7fb6e8003760 comp rx=0 tx=0).stop 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 msgr2=0x7fb6f4071840 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f17fa640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6f4071430 0x7fb6f4071840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.654 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 144814, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.655 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.656 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.657 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:22.658 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.659 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.660 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.661 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.662 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:22.663 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.664 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.665 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.666 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:22.667 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:247: writing smithi16534841-43 from 715998 to 1323326 tid 1 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:247: writing smithi16534841-43 from 1838006 to 2265374 tid 2 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:247: writing smithi16534841-43 from 2864574 to 3342054 tid 3 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:247: writing smithi16534841-43 from 3863835 to 3915998 tid 4 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:248: rollback oid 23 current snap is 31 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 27 2024-09-10T16:08:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing write tid 2 to smithi16534841-26 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing write tid 3 to smithi16534841-26 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing write tid 4 to smithi16534841-26 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: finishing write tid 5 to smithi16534841-26 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: oid 26 updating version 0 to 139 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: oid 26 version 139 is already newer than 136 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: oid 26 version 139 is already newer than 137 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: oid 26 version 139 is already newer than 138 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 139 (ObjNum 113 snap 31 seq_num 113) dirty exists 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: left oid 26 (ObjNum 113 snap 31 seq_num 113) 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 1 to smithi16534841-43 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 2 to smithi16534841-43 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:246: done (2 left) 2024-09-10T16:08:22.668 INFO:tasks.rados.rados.0.smithi165.stdout:249: read oid 28 snap -1 2024-09-10T16:08:22.669 INFO:tasks.rados.rados.0.smithi165.stdout:249: expect (ObjNum 81 snap 18 seq_num 81) 2024-09-10T16:08:22.669 INFO:tasks.rados.rados.0.smithi165.stdout:250: write (excl) oid 1 current snap is 31 2024-09-10T16:08:22.669 INFO:tasks.rados.rados.0.smithi165.stdout:250: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-10T16:08:22.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6e003e260 msgr2=0x7fb6e0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6e003e260 0x7fb6e0040720 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb6f406dad0 tx=0x7fb6ec02e000 comp rx=0 tx=0).stop 2024-09-10T16:08:22.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 10 monclient: shutdown 2024-09-10T16:08:22.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 msgr2=0x7fb6f41bac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6fb0b0640 1 --2- 172.21.15.164:0/2639066969 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6f40a0410 0x7fb6f41bac40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb6e400efc0 tx=0x7fb6e400c490 comp rx=0 tx=0).stop 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.642+0000 7fb6f964d640 1 -- 172.21.15.164:0/2639066969 reap_dead start 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.643+0000 7fb6f964d640 1 -- 172.21.15.164:0/2639066969 reap_dead start 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.643+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 shutdown_connections 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.643+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 >> 172.21.15.164:0/2639066969 conn(0x7fb6f40a3110 msgr2=0x7fb6f40a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.643+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 shutdown_connections 2024-09-10T16:08:22.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.643+0000 7fb6fb0b0640 1 -- 172.21.15.164:0/2639066969 wait complete. 2024-09-10T16:08:22.671 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:22.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:08:22.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:250: writing smithi16534841-1 from 732805 to 1531735 tid 1 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:250: writing smithi16534841-1 from 2296488 to 2867514 tid 2 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:250: writing smithi16534841-1 from 3291997 to 3697102 tid 3 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:251: copy_from oid 21 from oid 44 current snap is 31 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:248: finishing rollback tid 0 to smithi16534841-23 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:248: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:248: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 108 (ObjNum 97 snap 24 seq_num 97) dirty exists 2024-09-10T16:08:22.673 INFO:tasks.rados.rados.0.smithi165.stdout:249: expect (ObjNum 81 snap 18 seq_num 81) 2024-09-10T16:08:22.674 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 497625 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.772+0000 7f270f90e640 10 monclient: get_monmap_and_config 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.772+0000 7f270f90e640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.772+0000 7f270f90e640 10 monclient: monmap: 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:22.773142+0000 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:22.773142+0000 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.774 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.772+0000 7f270f90e640 1 Processor -- start 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 -- start start 2024-09-10T16:08:22.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient: init 2024-09-10T16:08:22.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-b con 0x7f2708137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2708137f90 0x7f2708138370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-c con 0x7f2708137f90 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27081346c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2708073820 con 0x7f27081346c0 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f27080a1c90 con 0x7f2708137f90 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f27080a1f70 con 0x7f2708137bb0 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270f90e640 10 monclient(hunting): authenticate will time out at 881.260576s 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34126/0 (socket says 172.21.15.164:34126) 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 -- 172.21.15.164:0/3847866219 learned_addr learned my addr 172.21.15.164:0/3847866219 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): get_auth_request con 0x7f2708137bb0 auth_method 0 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 1 --2- 172.21.15.164:0/3847866219 >> v2:172.21.15.165:3300/0 conn(0x7f2708137f90 0x7f2708138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): get_auth_request con 0x7f2708137f90 auth_method 0 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 -- 172.21.15.164:0/3847866219 >> v2:172.21.15.165:3300/0 conn(0x7f2708137f90 msgr2=0x7f2708138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 --2- 172.21.15.164:0/3847866219 >> v2:172.21.15.165:3300/0 conn(0x7f2708137f90 0x7f2708138370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.773+0000 7f270d6aa640 1 -- 172.21.15.164:0/3847866219 >> v1:172.21.15.165:6789/0 conn(0x7f27081346c0 legacy=0x7f2708134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.774+0000 7f270d6aa640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.774+0000 7f270d6aa640 1 monclient: found mon.noname-b 2024-09-10T16:08:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.774+0000 7f270d6aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.774+0000 7f270d6aa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.774+0000 7f270d6aa640 1 -- 172.21.15.164:0/3847866219 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27080a2250 con 0x7f2708137bb0 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270d6aa640 1 -- 172.21.15.164:0/3847866219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 msgr2=0x7f27080a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270d6aa640 1 --2- 172.21.15.164:0/3847866219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd5f61f18cb3f565 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 1 -- 172.21.15.164:0/3847866219 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8037070 con 0x7f2708137bb0 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: dump: 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 1 -- 172.21.15.164:0/3847866219 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26f8032440 con 0x7f2708137bb0 2024-09-10T16:08:22.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270f90e640 5 monclient: authenticate success, global_id 4529 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270f90e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270f90e640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 1 -- 172.21.15.164:0/3847866219 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8032810 con 0x7f2708137bb0 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f270f90e640 4 set_mon_vals no callback set 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.776+0000 7f26fffff640 10 monclient: dump: 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 10 monclient: shutdown 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 -- 172.21.15.164:0/3847866219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 msgr2=0x7f27080a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 --2- 172.21.15.164:0/3847866219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f26f8009870 tx=0x7f26f802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 -- 172.21.15.164:0/3847866219 shutdown_connections 2024-09-10T16:08:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 --2- 172.21.15.164:0/3847866219 >> v2:172.21.15.165:3300/0 conn(0x7f2708137f90 0x7f2708138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 --2- 172.21.15.164:0/3847866219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137bb0 0x7f27080a2e80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 -- 172.21.15.164:0/3847866219 >> 172.21.15.164:0/3847866219 conn(0x7f27080a7320 msgr2=0x7f27080a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 -- 172.21.15.164:0/3847866219 shutdown_connections 2024-09-10T16:08:22.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.777+0000 7f270f90e640 1 -- 172.21.15.164:0/3847866219 wait complete. 2024-09-10T16:08:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient: monmap: 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 1 Processor -- start 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 1 -- start start 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient: init 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081346c0 0x7f27081cd190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-b con 0x7f27081346c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2708137bb0 0x7f27081cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-a con 0x7f2708137bb0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.778+0000 7f270f90e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): picked mon.noname-c con 0x7f2708137f90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2708077060 con 0x7f2708137bb0 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2708076ee0 con 0x7f27081346c0 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2708135260 con 0x7f2708137f90 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270f90e640 10 monclient(hunting): authenticate will time out at 881.266027s 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2708137bb0 0x7f27081cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34132/0 (socket says 172.21.15.164:34132) 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 learned_addr learned my addr 172.21.15.164:0/638167666 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): get_auth_request con 0x7f2708137f90 auth_method 0 2024-09-10T16:08:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): get_auth_request con 0x7f2708137bb0 auth_method 0 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081346c0 0x7f27081cd190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): get_auth_request con 0x7f27081346c0 auth_method 0 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081346c0 msgr2=0x7f27081cd190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081346c0 0x7f27081cd190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2708137bb0 msgr2=0x7f27081cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2708137bb0 0x7f27081cd6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 monclient: found mon.noname-c 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2708134310 con 0x7f2708137f90 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081346c0 0x7f27081cd190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.779+0000 7f270cea9640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2708137bb0 0x7f27081cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 msgr2=0x7f2708120f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270deab640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f270000c7b0 tx=0x7f270000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270000ac60 con 0x7f2708137f90 2024-09-10T16:08:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: dump: 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270000ae00 con 0x7f2708137f90 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270f90e640 5 monclient: authenticate success, global_id 4532 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270f90e640 10 monclient: _renew_subs 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270f90e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2708124d50 con 0x7f2708137f90 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2700004ba0 con 0x7f2708137f90 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26fdffb640 10 monclient: dump: 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.781+0000 7f26ff7fe640 4 set_mon_vals no callback set 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 10 monclient: _renew_subs 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2708124a70 con 0x7f2708137f90 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 0x7f27081218a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.782+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27080a2e80 con 0x7f27081214c0 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 0x7f27081218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient: get_auth_request con 0x7f27081214c0 auth_method 0 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2700005030 con 0x7f2708137f90 2024-09-10T16:08:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 10 monclient(hunting): handle_auth_done global_id 14179 payload 824 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f26fdffb640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26ec03e210 0x7f26ec0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270d6aa640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2708123550 con 0x7f27081214c0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(72..72 src has 1..72) ==== 4771+0+0 (secure 0 0 0) 0x7f2700052f40 con 0x7f2708137f90 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270cea9640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26ec03e210 0x7f26ec0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.783+0000 7f270cea9640 10 monclient: get_auth_request con 0x7f26ec03e210 auth_method 0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f270d6aa640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 msgr2=0x7f27081218a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 0x7f27081218a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26f8034000 tx=0x7f26f80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8037040 con 0x7f27081214c0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f270cea9640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26ec03e210 0x7f26ec0406d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f27080a5590 tx=0x7f2704009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.785+0000 7f26fdffb640 10 monclient: dump: 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26f803c860 con 0x7f27081214c0 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 msgr2=0x7f27081218a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 0x7f27081218a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26f8034000 tx=0x7f26f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 msgr2=0x7f27081218a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.786+0000 7f26fdffb640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27081214c0 0x7f27081218a0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f26f8034000 tx=0x7f26f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270f90e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270f90e640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 0x7f270811d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270f90e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27081cdc10 con 0x7f2708121de0 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 0x7f270811d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient: get_auth_request con 0x7f2708121de0 auth_method 0 2024-09-10T16:08:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 10 monclient(hunting): handle_auth_done global_id 14182 payload 824 2024-09-10T16:08:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.792+0000 7f270d6aa640 1 -- 172.21.15.164:0/638167666 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27081221f0 con 0x7f2708121de0 2024-09-10T16:08:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.794+0000 7f270d6aa640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 msgr2=0x7f270811d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270d6aa640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 0x7f270811d550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26f8000c00 tx=0x7f26f8009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26f8037030 con 0x7f2708121de0 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 10 monclient: dump: 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f26f8047e20 con 0x7f2708121de0 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 msgr2=0x7f270811d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 0x7f270811d550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26f8000c00 tx=0x7f26f8009240 comp rx=0 tx=0).stop 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 msgr2=0x7f270811d550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f26fdffb640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2708121de0 0x7f270811d550 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f26f8000c00 tx=0x7f26f8009240 comp rx=0 tx=0).stop 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 76269, 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:22.813 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.814 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:22.816 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:22.817 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:22.818 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.819 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.821 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:22.825 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3143872 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 3 to smithi16534841-43 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 4 to smithi16534841-43 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 5 to smithi16534841-43 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: finishing write tid 6 to smithi16534841-43 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: oid 43 updating version 0 to 61 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: oid 43 updating version 61 to 62 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: oid 43 version 62 is already newer than 60 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: oid 43 version 62 is already newer than 59 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: oid 43 version 62 is already newer than 58 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 62 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:247: left oid 43 (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing write tid 1 to smithi16534841-1 2024-09-10T16:08:22.826 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing write tid 2 to smithi16534841-1 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:247: done (4 left) 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:248: done (3 left) 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:249: done (2 left) 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:252: read oid 3 snap 23 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:252: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:08:22.827 INFO:tasks.rados.rados.0.smithi165.stdout:253: snap_remove snap 14 2024-09-10T16:08:22.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26ec03e210 msgr2=0x7f26ec0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26ec03e210 0x7f26ec0406d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f27080a5590 tx=0x7f2704009190 comp rx=0 tx=0).stop 2024-09-10T16:08:22.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 10 monclient: shutdown 2024-09-10T16:08:22.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 msgr2=0x7f2708120f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f270000c7b0 tx=0x7f270000cc70 comp rx=0 tx=0).stop 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270deab640 1 -- 172.21.15.164:0/638167666 reap_dead start 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 shutdown_connections 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.795+0000 7f270f90e640 1 --2- 172.21.15.164:0/638167666 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2708137f90 0x7f2708120f80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.796+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 >> 172.21.15.164:0/638167666 conn(0x7f27080a7320 msgr2=0x7f2708135780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.796+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 shutdown_connections 2024-09-10T16:08:22.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.796+0000 7f270f90e640 1 -- 172.21.15.164:0/638167666 wait complete. 2024-09-10T16:08:22.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:22.829 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:08:22.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.926+0000 7f3750bad640 10 monclient: get_monmap_and_config 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.926+0000 7f3750bad640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.926+0000 7f3750bad640 10 monclient: monmap: 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:22.927461+0000 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:22.927461+0000 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.928 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:22.929 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 Processor -- start 2024-09-10T16:08:22.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 -- start start 2024-09-10T16:08:22.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient: init 2024-09-10T16:08:22.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f374c137bb0 0x7f374c0a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-a con 0x7f374c137bb0 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f374c137f90 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f374c13d140 0x7f374c13f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-c con 0x7f374c13d140 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374c073820 con 0x7f374c137f90 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f374c0a1c90 con 0x7f374c137bb0 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f374c0a1f70 con 0x7f374c13d140 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f3750bad640 10 monclient(hunting): authenticate will time out at 881.414785s 2024-09-10T16:08:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f374c137bb0 0x7f374c0a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f374c137bb0 0x7f374c0a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57422/0 (socket says 172.21.15.164:57422) 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 1 -- 172.21.15.164:0/1227701410 learned_addr learned my addr 172.21.15.164:0/1227701410 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 10 monclient(hunting): get_auth_request con 0x7f374c137bb0 auth_method 0 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.927+0000 7f374a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374ad9d640 1 --2- 172.21.15.164:0/1227701410 >> v2:172.21.15.165:3300/0 conn(0x7f374c13d140 0x7f374c13f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374ad9d640 10 monclient(hunting): get_auth_request con 0x7f374c13d140 auth_method 0 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient(hunting): handle_auth_done global_id 24126 payload 274 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 1 -- 172.21.15.164:0/1227701410 >> v2:172.21.15.165:3300/0 conn(0x7f374c13d140 msgr2=0x7f374c13f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 1 --2- 172.21.15.164:0/1227701410 >> v2:172.21.15.165:3300/0 conn(0x7f374c13d140 0x7f374c13f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 1 -- 172.21.15.164:0/1227701410 >> v1:172.21.15.165:6789/0 conn(0x7f374c137f90 legacy=0x7f374c1383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.928+0000 7f374a59c640 1 -- 172.21.15.164:0/1227701410 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c0a2250 con 0x7f374c137bb0 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.930+0000 7f374a59c640 1 -- 172.21.15.164:0/1227701410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 msgr2=0x7f374c0a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.930+0000 7f374a59c640 1 --2- 172.21.15.164:0/1227701410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c0a2e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f374c138b70 tx=0x7f373c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9987e320ea83f391 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 1 -- 172.21.15.164:0/1227701410 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c037070 con 0x7f374c137bb0 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: dump: 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.937 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 1 -- 172.21.15.164:0/1227701410 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c032640 con 0x7f374c137bb0 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 5 monclient: authenticate success, global_id 24126 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 1 -- 172.21.15.164:0/1227701410 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c032a10 con 0x7f374c137bb0 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3749d9b640 10 monclient: dump: 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 4 set_mon_vals no callback set 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 10 monclient: shutdown 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 -- 172.21.15.164:0/1227701410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 msgr2=0x7f374c0a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 --2- 172.21.15.164:0/1227701410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c0a2e80 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f374c138b70 tx=0x7f373c02a0b0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 -- 172.21.15.164:0/1227701410 shutdown_connections 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 --2- 172.21.15.164:0/1227701410 >> v2:172.21.15.165:3300/0 conn(0x7f374c13d140 0x7f374c13f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 --2- 172.21.15.164:0/1227701410 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c0a2e80 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 -- 172.21.15.164:0/1227701410 >> 172.21.15.164:0/1227701410 conn(0x7f374c0a7320 msgr2=0x7f374c0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 -- 172.21.15.164:0/1227701410 shutdown_connections 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.931+0000 7f3750bad640 1 -- 172.21.15.164:0/1227701410 wait complete. 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.932+0000 7f3750bad640 10 monclient: build_initial_monmap 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.932+0000 7f3750bad640 10 monclient: monmap: 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:22.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:22.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:22.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:22.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.932+0000 7f3750bad640 1 Processor -- start 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 -- start start 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient: init 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-a con 0x7f374c137bb0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:22.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f374c137f90 0x7f374c11d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-c con 0x7f374c137f90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f374c13d140 0x7f374c134820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): picked mon.noname-b con 0x7f374c13d140 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f374c077060 con 0x7f374c137bb0 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f374c076ee0 con 0x7f374c13d140 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f374c13fa70 con 0x7f374c137f90 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3750bad640 10 monclient(hunting): authenticate will time out at 881.420500s 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57432/0 (socket says 172.21.15.164:57432) 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f374c137f90 0x7f374c11d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 -- 172.21.15.164:0/4093733837 learned_addr learned my addr 172.21.15.164:0/4093733837 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): get_auth_request con 0x7f374c137bb0 auth_method 0 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): get_auth_request con 0x7f374c137f90 auth_method 0 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f374c13d140 0x7f374c134820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f3741d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): get_auth_request con 0x7f374c13d140 auth_method 0 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient(hunting): handle_auth_done global_id 24129 payload 824 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 -- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f374c137f90 msgr2=0x7f374c11d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f374c137f90 0x7f374c11d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 -- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f374c13d140 msgr2=0x7f374c134820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f374c13d140 0x7f374c134820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 monclient: found mon.noname-a 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.933+0000 7f374a59c640 1 -- 172.21.15.164:0/4093733837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374c11e910 con 0x7f374c137bb0 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f374a59c640 1 -- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 msgr2=0x7f374c13b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f374a59c640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f373c000c00 tx=0x7f373c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c041030 con 0x7f374c137bb0 2024-09-10T16:08:22.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: dump: 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f373c03d6f0 con 0x7f374c137bb0 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 5 monclient: authenticate success, global_id 24129 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f373c037040 con 0x7f374c137bb0 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f37437fe640 10 monclient: dump: 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.951 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 10 monclient: _renew_subs 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f374959a640 4 set_mon_vals no callback set 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f374c09e0f0 con 0x7f374c137bb0 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 10 monclient: _renew_subs 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.936+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f374c11f1b0 con 0x7f374c137bb0 2024-09-10T16:08:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f373c044070 con 0x7f374c137bb0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3750bad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3750bad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3750bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f374c0a2e60 con 0x7f374c137bb0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f37437fe640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 0x7f372c040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 4771+0+0 (secure 0 0 0) 0x7f373c072a50 con 0x7f374c137bb0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3741d9b640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 0x7f372c040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.937+0000 7f3741d9b640 10 monclient: get_auth_request con 0x7f372c03e290 auth_method 0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.940+0000 7f3741d9b640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 0x7f372c040750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f374c1355e0 tx=0x7f3734007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.942+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f373c04b020 con 0x7f374c137bb0 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.942+0000 7f37437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:22.942+0000 7f37437fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:23.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.185+0000 7f3750bad640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:08:23.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.185+0000 7f3750bad640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:08:23.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.185+0000 7f3750bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:23.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.185+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f374c06d1c0 con 0x7f374c137bb0 2024-09-10T16:08:23.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f37437fe640 1 -- 172.21.15.164:0/4093733837 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f374c11e910 con 0x7f374c137bb0 2024-09-10T16:08:23.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f37437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:08:23.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f37437fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:23.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=952659461,config=2586702516,health=3897446751,kv=728585597,logm=151941675}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=2,health=11,kv=12,logm=67} crc {auth=3884764479,config=2586702516,health=344943315,kv=805391194,logm=3582040855}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=952659461,config=2586702516,health=3897446751,kv=728585597,logm=151941675}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.186+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=1,health=12,kv=11,logm=67} crc {auth=333236420,config=1228700967,health=1247533675,kv=3185102672,logm=3923536736}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075461719}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451043283}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075461719}) 2024-09-10T16:08:23.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.187+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684577153}) 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645886836}) 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3183648278}) 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2645886836}) 2024-09-10T16:08:23.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2462721413}) 2024-09-10T16:08:23.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 msgr2=0x7f372c040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 0x7f372c040750 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f374c1355e0 tx=0x7f3734007e30 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 10 monclient: shutdown 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 msgr2=0x7f374c13b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f373c000c00 tx=0x7f373c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 shutdown_connections 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f372c03e290 0x7f372c040750 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f374c13d140 0x7f374c134820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f374c137f90 0x7f374c11d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 --2- 172.21.15.164:0/4093733837 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f374c137bb0 0x7f374c13b490 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 >> 172.21.15.164:0/4093733837 conn(0x7f374c0a7320 msgr2=0x7f374c136950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:23.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 shutdown_connections 2024-09-10T16:08:23.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:23.188+0000 7f3750bad640 1 -- 172.21.15.164:0/4093733837 wait complete. 2024-09-10T16:08:23.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624272724}) 2024-09-10T16:08:23.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2408928615}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3624272724}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2970777271}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201990098}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3600822679}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4201990098}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.189+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=55882361}) 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625902672}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=218062969}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=625902672}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.190+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=494961026}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=16} crc {logm=2363976616,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1820443717,mon_config_key=1452972387,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3645997971,osd_snap=426084571,osdmap=2842447316}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=16,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=13,osdmap=19} crc {logm=1009751039,mdsmap=931935893,mgr=2441926604,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=60558857,mon_config_key=665552247,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3014334576,osd_pg_creating=3645997971,osd_snap=426084571,osdmap=3695968412}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=16} crc {logm=2363976616,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1820443717,mon_config_key=1452972387,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3645997971,osd_snap=426084571,osdmap=2842447316}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.191+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=12,osdmap=18} crc {logm=2363976616,mdsmap=931935893,mgr=3066406838,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3201637268,mon_config_key=665552247,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=633904022,osd_pg_creating=3645997971,osd_snap=4152513199,osdmap=671105589}) 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2852487386}) 2024-09-10T16:08:23.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3457395212}) 2024-09-10T16:08:23.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:08:23.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2852487386}) 2024-09-10T16:08:23.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:08:23.193+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=98} crc {osdmap=3133691053}) 2024-09-10T16:08:23.201 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:253: done (3 left) 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:254: snap_create 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing write tid 3 to smithi16534841-1 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing write tid 4 to smithi16534841-1 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:250: finishing write tid 5 to smithi16534841-1 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:250: oid 1 updating version 0 to 113 2024-09-10T16:08:23.612 INFO:tasks.rados.rados.0.smithi165.stdout:250: oid 1 version 113 is already newer than 110 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:250: oid 1 version 113 is already newer than 111 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:250: oid 1 version 113 is already newer than 112 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 113 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:250: left oid 1 (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:251: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:251: finishing copy_from to smithi16534841-21 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 63 (ObjNum 59 snap 3 seq_num 59) dirty exists 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:252: expect (ObjNum 62 snap 5 seq_num 62) 2024-09-10T16:08:23.613 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:24.202 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:08:24.203 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:24.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 10 monclient: get_monmap_and_config 2024-09-10T16:08:24.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 10 monclient: monmap: 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:24.302982+0000 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:24.302982+0000 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:24.304 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:24.305 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:24.305 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:24.305 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 1 Processor -- start 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 1 -- start start 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.302+0000 7fefdd95d640 10 monclient: init 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fefd8137bb0 0x7fefd80a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-a con 0x7fefd8137bb0 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefd8137f90 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefd813d140 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefd8073820 con 0x7fefd813d140 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefd80a1c90 con 0x7fefd8137f90 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fefd80a1f70 con 0x7fefd8137bb0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefdd95d640 10 monclient(hunting): authenticate will time out at 882.790319s 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fefd8137bb0 0x7fefd80a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd77fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fefd813d140 0x7fefd813f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34700/0 (socket says 172.21.15.164:34700) 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd77fe640 1 -- 172.21.15.164:0/3598577152 learned_addr learned my addr 172.21.15.164:0/3598577152 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): get_auth_request con 0x7fefd8137bb0 auth_method 0 2024-09-10T16:08:24.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_done global_id 24132 payload 274 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 1 -- 172.21.15.164:0/3598577152 >> v1:172.21.15.165:6789/0 conn(0x7fefd8137f90 legacy=0x7fefd81383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 1 -- 172.21.15.164:0/3598577152 >> v1:172.21.15.164:6789/0 conn(0x7fefd813d140 legacy=0x7fefd813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.303+0000 7fefd6ffd640 1 -- 172.21.15.164:0/3598577152 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd80a2250 con 0x7fefd8137bb0 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd6ffd640 1 -- 172.21.15.164:0/3598577152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 msgr2=0x7fefd80a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd6ffd640 1 --2- 172.21.15.164:0/3598577152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd80a2e80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fefc80097a0 tx=0x7fefc8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622509a9d2a83469 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 1 -- 172.21.15.164:0/3598577152 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8037070 con 0x7fefd8137bb0 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: dump: 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 1 -- 172.21.15.164:0/3598577152 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc8032640 con 0x7fefd8137bb0 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefdd95d640 5 monclient: authenticate success, global_id 24132 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefdd95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefdd95d640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:24.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 1 -- 172.21.15.164:0/3598577152 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8032a10 con 0x7fefd8137bb0 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefdd95d640 4 set_mon_vals no callback set 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.306+0000 7fefd5ffb640 10 monclient: dump: 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.314 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 10 monclient: shutdown 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 -- 172.21.15.164:0/3598577152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 msgr2=0x7fefd80a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 --2- 172.21.15.164:0/3598577152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd80a2e80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fefc80097a0 tx=0x7fefc8029fd0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 -- 172.21.15.164:0/3598577152 shutdown_connections 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 --2- 172.21.15.164:0/3598577152 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd80a2e80 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 -- 172.21.15.164:0/3598577152 >> 172.21.15.164:0/3598577152 conn(0x7fefd80a7320 msgr2=0x7fefd80a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 -- 172.21.15.164:0/3598577152 shutdown_connections 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 1 -- 172.21.15.164:0/3598577152 wait complete. 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.307+0000 7fefdd95d640 10 monclient: monmap: 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:24.316 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 Processor -- start 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 -- start start 2024-09-10T16:08:24.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient: init 2024-09-10T16:08:24.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-a con 0x7fefd8137bb0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefd8137f90 0x7fefd81344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-c con 0x7fefd8137f90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 0x7fefd8139c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): picked mon.noname-b con 0x7fefd813d140 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8077060 con 0x7fefd8137bb0 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fefd8076ee0 con 0x7fefd813d140 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefd8140c50 con 0x7fefd8137f90 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefdd95d640 10 monclient(hunting): authenticate will time out at 882.795768s 2024-09-10T16:08:24.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42190/0 (socket says 172.21.15.164:42190) 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 1 -- 172.21.15.164:0/2343780418 learned_addr learned my addr 172.21.15.164:0/2343780418 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): get_auth_request con 0x7fefd8137bb0 auth_method 0 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd67fc640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefd8137f90 0x7fefd81344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd67fc640 10 monclient(hunting): get_auth_request con 0x7fefd8137f90 auth_method 0 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.308+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 0x7fefd8139c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 10 monclient(hunting): get_auth_request con 0x7fefd813d140 auth_method 0 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 10 monclient(hunting): handle_auth_done global_id 24135 payload 824 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 -- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefd8137f90 msgr2=0x7fefd81344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefd8137f90 0x7fefd81344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 -- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 msgr2=0x7fefd8139c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 0x7fefd8139c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd6ffd640 1 -- 172.21.15.164:0/2343780418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefd813be20 con 0x7fefd8137bb0 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.309+0000 7fefd77fe640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 0x7fefd8139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefd6ffd640 1 -- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 msgr2=0x7fefd8133f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefd6ffd640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fefc80097a0 tx=0x7fefc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8041070 con 0x7fefd8137bb0 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: dump: 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.329 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 5 monclient: authenticate success, global_id 24135 2024-09-10T16:08:24.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 10 monclient: _renew_subs 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc803c880 con 0x7fefd8137bb0 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefd811d8d0 con 0x7fefd8137bb0 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc8037040 con 0x7fefd8137bb0 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefbffff640 10 monclient: dump: 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.331 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 10 monclient: _renew_subs 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefd81405e0 con 0x7fefd8137bb0 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.311+0000 7fefd57fa640 4 set_mon_vals no callback set 2024-09-10T16:08:24.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefdd95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefdd95d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefdd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefd80a2f00 con 0x7fefd8137bb0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fefc8044070 con 0x7fefd8137bb0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.312+0000 7fefbffff640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 0x7fefc4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.313+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 4815+0+0 (secure 0 0 0) 0x7fefc8073770 con 0x7fefd8137bb0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.313+0000 7fefd67fc640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 0x7fefc4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.313+0000 7fefd67fc640 10 monclient: get_auth_request con 0x7fefc403e240 auth_method 0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.315+0000 7fefd67fc640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 0x7fefc4040700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fefd811dae0 tx=0x7fefcc007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.317+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fefc8044c60 con 0x7fefd8137bb0 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.317+0000 7fefbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.317+0000 7fefbffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:24.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.559+0000 7fefdd95d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:24.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.559+0000 7fefdd95d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.559+0000 7fefdd95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.559+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fefd8134a00 con 0x7fefd8137bb0 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.560+0000 7fefbffff640 1 -- 172.21.15.164:0/2343780418 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fefc804ed60 con 0x7fefd8137bb0 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.560+0000 7fefbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.560+0000 7fefbffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:24.562 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 msgr2=0x7fefc4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 0x7fefc4040700 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fefd811dae0 tx=0x7fefcc007e30 comp rx=0 tx=0).stop 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 10 monclient: shutdown 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 msgr2=0x7fefd8133f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fefc80097a0 tx=0x7fefc80046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 shutdown_connections 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc403e240 0x7fefc4040700 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefd813d140 0x7fefd8139c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefd8137f90 0x7fefd81344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 --2- 172.21.15.164:0/2343780418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefd8137bb0 0x7fefd8133f80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 >> 172.21.15.164:0/2343780418 conn(0x7fefd80a7320 msgr2=0x7fefd8136730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 shutdown_connections 2024-09-10T16:08:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.562+0000 7fefdd95d640 1 -- 172.21.15.164:0/2343780418 wait complete. 2024-09-10T16:08:24.574 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:08:24.574 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:08:24.574 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:08:24.574 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:08:24.611 INFO:tasks.rados.rados.0.smithi165.stdout:250: done (3 left) 2024-09-10T16:08:24.611 INFO:tasks.rados.rados.0.smithi165.stdout:251: done (2 left) 2024-09-10T16:08:24.612 INFO:tasks.rados.rados.0.smithi165.stdout:252: done (1 left) 2024-09-10T16:08:24.612 INFO:tasks.rados.rados.0.smithi165.stdout:254: done (0 left) 2024-09-10T16:08:24.612 INFO:tasks.rados.rados.0.smithi165.stdout:255: snap_create 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.673+0000 7f87b2a7f640 10 monclient: get_monmap_and_config 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.673+0000 7f87b2a7f640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.673+0000 7f87b2a7f640 10 monclient: monmap: 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:24.674517+0000 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:24.674517+0000 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:24.675 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:24.676 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:24.676 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:24.676 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 Processor -- start 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 -- start start 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient: init 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:08:24.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 0x7f87ac09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f87ac09d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f87ac09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87ac134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87ac0a0870 con 0x7f87ac134580 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f87ac09d3e0 con 0x7f87ac09d930 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac09e790 con 0x7f87ac09ab50 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b2a7f640 10 monclient(hunting): authenticate will time out at 883.161894s 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87abfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.674+0000 7f87b081b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 0x7f87ac09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 0x7f87ac09dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42192/0 (socket says 172.21.15.164:42192) 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39564/0 (socket says 172.21.15.164:39564) 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 1 -- 172.21.15.164:0/2530737111 learned_addr learned my addr 172.21.15.164:0/2530737111 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): get_auth_request con 0x7f87ac09d930 auth_method 0 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): get_auth_request con 0x7f87ac09ab50 auth_method 0 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient(hunting): handle_auth_done global_id 4541 payload 274 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 -- 172.21.15.164:0/2530737111 >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 msgr2=0x7f87ac09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 --2- 172.21.15.164:0/2530737111 >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 0x7f87ac09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 -- 172.21.15.164:0/2530737111 >> v1:172.21.15.165:6789/0 conn(0x7f87ac134580 legacy=0x7f87ac134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 monclient: found mon.noname-b 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:08:24.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.675+0000 7f87abfff640 1 -- 172.21.15.164:0/2530737111 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac074c70 con 0x7f87ac09ab50 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87abfff640 1 -- 172.21.15.164:0/2530737111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 msgr2=0x7f87ac09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87abfff640 1 --2- 172.21.15.164:0/2530737111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87980097a0 tx=0x7f879802a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=488296ec48fbb531 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 1 -- 172.21.15.164:0/2530737111 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798037070 con 0x7f87ac09ab50 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: dump: 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 1 -- 172.21.15.164:0/2530737111 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8798032440 con 0x7f87ac09ab50 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87b2a7f640 5 monclient: authenticate success, global_id 4541 2024-09-10T16:08:24.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87b2a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 1 -- 172.21.15.164:0/2530737111 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798032810 con 0x7f87ac09ab50 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87b2a7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87b2a7f640 4 set_mon_vals no callback set 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.678+0000 7f87ab7fe640 10 monclient: dump: 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.689 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 10 monclient: shutdown 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 -- 172.21.15.164:0/2530737111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 msgr2=0x7f87ac09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/2530737111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f87980097a0 tx=0x7f879802a050 comp rx=0 tx=0).stop 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 -- 172.21.15.164:0/2530737111 shutdown_connections 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/2530737111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09ab50 0x7f87ac09e250 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/2530737111 >> v2:172.21.15.164:3300/0 conn(0x7f87ac09d930 0x7f87ac09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 -- 172.21.15.164:0/2530737111 >> 172.21.15.164:0/2530737111 conn(0x7f87ac0a3110 msgr2=0x7f87ac0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 -- 172.21.15.164:0/2530737111 shutdown_connections 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 1 -- 172.21.15.164:0/2530737111 wait complete. 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.679+0000 7f87b2a7f640 10 monclient: monmap: 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:24.691 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 Processor -- start 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 -- start start 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient: init 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 0x7f87ac1365a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f87ac09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 0x7f87ac136ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f87ac09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ac134580 0x7f87ac137020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f87ac134580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:24.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac09ef30 con 0x7f87ac09ab50 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac0a02b0 con 0x7f87ac134580 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87ac0a2940 con 0x7f87ac09d930 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b2a7f640 10 monclient(hunting): authenticate will time out at 883.167729s 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 0x7f87ac1365a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 0x7f87ac136ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 0x7f87ac1365a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42206/0 (socket says 172.21.15.164:42206) 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 1 -- 172.21.15.164:0/3467006345 learned_addr learned my addr 172.21.15.164:0/3467006345 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): get_auth_request con 0x7f87ac09d930 auth_method 0 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): get_auth_request con 0x7f87ac09ab50 auth_method 0 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b101c640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ac134580 0x7f87ac137020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.680+0000 7f87b101c640 10 monclient(hunting): get_auth_request con 0x7f87ac134580 auth_method 0 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87b101c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87b101c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87b101c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 10 monclient(hunting): handle_auth_done global_id 4544 payload 824 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ac134580 msgr2=0x7f87ac137020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87ac134580 0x7f87ac137020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 msgr2=0x7f87ac1365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 0x7f87ac1365a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 monclient: found mon.noname-c 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87abfff640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87ac13a9c0 con 0x7f87ac09d930 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.681+0000 7f87b081b640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac09ab50 0x7f87ac1365a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87abfff640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 msgr2=0x7f87ac136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87abfff640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 0x7f87ac136ae0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8798031000 tx=0x7f87980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798040020 con 0x7f87ac09d930 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: dump: 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.696 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8798038040 con 0x7f87ac09d930 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 5 monclient: authenticate success, global_id 4544 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 10 monclient: _renew_subs 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:24.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87ac139750 con 0x7f87ac09d930 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8798037070 con 0x7f87ac09d930 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87a97fa640 10 monclient: dump: 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.698 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 10 monclient: _renew_subs 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87aaffd640 4 set_mon_vals no callback set 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.683+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87ac137560 con 0x7f87ac09d930 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b2a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 0x7f87ac063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac02e9a0 con 0x7f87ac02e5c0 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 0x7f87ac063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient: get_auth_request con 0x7f87ac02e5c0 auth_method 0 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8798043020 con 0x7f87ac09d930 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 10 monclient(hunting): handle_auth_done global_id 24144 payload 824 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b081b640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87ac0a1e40 con 0x7f87ac02e5c0 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87a97fa640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879003e270 0x7f8790040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7f8798072ea0 con 0x7f87ac09d930 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b101c640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879003e270 0x7f8790040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.684+0000 7f87b101c640 10 monclient: get_auth_request con 0x7f879003e270 auth_method 0 2024-09-10T16:08:24.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.686+0000 7f87b081b640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 msgr2=0x7f87ac063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.686+0000 7f87b081b640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 0x7f87ac063d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009f70 tx=0x7f87a000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0004020 con 0x7f87ac02e5c0 2024-09-10T16:08:24.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 10 monclient: dump: 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87a0004020 con 0x7f87ac02e5c0 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 msgr2=0x7f87ac063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 0x7f87ac063d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009f70 tx=0x7f87a000f610 comp rx=0 tx=0).stop 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 msgr2=0x7f87ac063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87a97fa640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac02e5c0 0x7f87ac063d50 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f87a0009f70 tx=0x7f87a000f610 comp rx=0 tx=0).stop 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.687+0000 7f87b101c640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879003e270 0x7f8790040730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f87ac139d50 tx=0x7f87a4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b2a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 0x7f87ac1d58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b2a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87ac06dac0 con 0x7f87ac064290 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 0x7f87ac1d58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient: get_auth_request con 0x7f87ac064290 auth_method 0 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 10 monclient(hunting): handle_auth_done global_id 24147 payload 824 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.693+0000 7f87b081b640 1 -- 172.21.15.164:0/3467006345 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87ac071ef0 con 0x7f87ac064290 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87b081b640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 msgr2=0x7f87ac1d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87b081b640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 0x7f87ac1d58d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f87a001b000 tx=0x7f87a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0003a20 con 0x7f87ac064290 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87a97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87a97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.695+0000 7f87a97fa640 10 monclient: dump: 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f87a0003a20 con 0x7f87ac064290 2024-09-10T16:08:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 msgr2=0x7f87ac1d58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 0x7f87ac1d58d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f87a001b000 tx=0x7f87a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 msgr2=0x7f87ac1d58d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87a97fa640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87ac064290 0x7f87ac1d58d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:08:24.708 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 29628, 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.709 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.710 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:24.711 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:24.712 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:24.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:24.714 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:24.716 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:24.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:24.719 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.720 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.720 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:24.720 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879003e270 msgr2=0x7f8790040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879003e270 0x7f8790040730 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f87ac139d50 tx=0x7f87a4009040 comp rx=0 tx=0).stop 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 10 monclient: shutdown 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 msgr2=0x7f87ac136ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 --2- 172.21.15.164:0/3467006345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87ac09d930 0x7f87ac136ae0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f8798031000 tx=0x7f87980046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b101c640 1 -- 172.21.15.164:0/3467006345 reap_dead start 2024-09-10T16:08:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b101c640 1 -- 172.21.15.164:0/3467006345 reap_dead start 2024-09-10T16:08:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 shutdown_connections 2024-09-10T16:08:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 >> 172.21.15.164:0/3467006345 conn(0x7f87ac0a3110 msgr2=0x7f87ac099850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 shutdown_connections 2024-09-10T16:08:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.696+0000 7f87b2a7f640 1 -- 172.21.15.164:0/3467006345 wait complete. 2024-09-10T16:08:24.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:24.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:08:24.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 10 monclient: get_monmap_and_config 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 10 monclient: monmap: 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:24.821075+0000 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:24.821075+0000 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 1 Processor -- start 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 1 -- start start 2024-09-10T16:08:24.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.820+0000 7f8a79447640 10 monclient: init 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a7406d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a7406d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a7413e070 0x7f8a74140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7413e070 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7409cb90 con 0x7f8a7406d160 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a7409ce70 con 0x7f8a7406d540 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8a7409d150 con 0x7f8a7413e070 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a79447640 10 monclient(hunting): authenticate will time out at 883.308352s 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a72ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f8a7406d160 0x7f8a74134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34706/0 (socket says 172.21.15.164:34706) 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a72ffd640 1 -- 172.21.15.164:0/946526756 learned_addr learned my addr 172.21.15.164:0/946526756 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 1 --2- 172.21.15.164:0/946526756 >> v2:172.21.15.165:3300/0 conn(0x7f8a7413e070 0x7f8a74140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a7413e070 auth_method 0 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient(hunting): handle_auth_done global_id 14197 payload 274 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 1 -- 172.21.15.164:0/946526756 >> v1:172.21.15.165:6789/0 conn(0x7f8a7406d540 legacy=0x7f8a740a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 1 -- 172.21.15.164:0/946526756 >> v1:172.21.15.164:6789/0 conn(0x7f8a7406d160 legacy=0x7f8a74134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 1 monclient: found mon.noname-c 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:08:24.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.821+0000 7f8a737fe640 1 -- 172.21.15.164:0/946526756 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a740a4df0 con 0x7f8a7413e070 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a737fe640 1 -- 172.21.15.164:0/946526756 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 msgr2=0x7f8a74140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a737fe640 1 --2- 172.21.15.164:0/946526756 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a74140460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0097a0 tx=0x7f8a6c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8e888caf161f61 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 1 -- 172.21.15.164:0/946526756 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c037070 con 0x7f8a7413e070 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: dump: 2024-09-10T16:08:24.835 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 1 -- 172.21.15.164:0/946526756 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a6c032640 con 0x7f8a7413e070 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 1 -- 172.21.15.164:0/946526756 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a6c032a10 con 0x7f8a7413e070 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a79447640 5 monclient: authenticate success, global_id 14197 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a79447640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a79447640 4 set_mon_vals no callback set 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.825+0000 7f8a71ffb640 10 monclient: dump: 2024-09-10T16:08:24.837 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 10 monclient: shutdown 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 -- 172.21.15.164:0/946526756 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 msgr2=0x7f8a74140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 --2- 172.21.15.164:0/946526756 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a74140460 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c0097a0 tx=0x7f8a6c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 -- 172.21.15.164:0/946526756 shutdown_connections 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 --2- 172.21.15.164:0/946526756 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a74140460 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 -- 172.21.15.164:0/946526756 >> 172.21.15.164:0/946526756 conn(0x7f8a740aacb0 msgr2=0x7f8a740ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 -- 172.21.15.164:0/946526756 shutdown_connections 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 1 -- 172.21.15.164:0/946526756 wait complete. 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.826+0000 7f8a79447640 10 monclient: monmap: 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 Processor -- start 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 -- start start 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient: init 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 0x7f8a7412d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-a con 0x7f8a7406d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7406d540 0x7f8a7412d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-c con 0x7f8a7406d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a7409eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): picked mon.noname-b con 0x7f8a7413e070 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8a74140d30 con 0x7f8a7406d160 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8a74140bb0 con 0x7f8a7413e070 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a74140eb0 con 0x7f8a7406d540 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a79447640 10 monclient(hunting): authenticate will time out at 883.314809s 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 0x7f8a7412d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 0x7f8a7412d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42242/0 (socket says 172.21.15.164:42242) 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 1 -- 172.21.15.164:0/2409350073 learned_addr learned my addr 172.21.15.164:0/2409350073 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request con 0x7f8a7406d160 auth_method 0 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7406d540 0x7f8a7412d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a727fc640 10 monclient(hunting): get_auth_request con 0x7f8a7406d540 auth_method 0 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a727fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.827+0000 7f8a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a7409eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 10 monclient(hunting): get_auth_request con 0x7f8a7413e070 auth_method 0 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient(hunting): handle_auth_done global_id 24150 payload 824 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7406d540 msgr2=0x7f8a7412d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7406d540 0x7f8a7412d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 msgr2=0x7f8a7409eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a7409eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 monclient: found mon.noname-a 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a72ffd640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a740a47e0 con 0x7f8a7406d160 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7406d540 0x7f8a7412d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.828+0000 7f8a737fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a7413e070 0x7f8a7409eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a72ffd640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 msgr2=0x7f8a7412d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a72ffd640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 0x7f8a7412d210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400cc10 tx=0x7f8a6400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a64012400 con 0x7f8a7406d160 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: dump: 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.846 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 5 monclient: authenticate success, global_id 24150 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a640125e0 con 0x7f8a7406d160 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 10 monclient: _renew_subs 2024-09-10T16:08:24.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a740a0e00 con 0x7f8a7406d160 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a640129d0 con 0x7f8a7406d160 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a6b7fe640 10 monclient: dump: 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.848 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a717fa640 4 set_mon_vals no callback set 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 10 monclient: _renew_subs 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.830+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a740a1d60 con 0x7f8a7406d160 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a79447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a79447640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 0x7f8a7409f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a740a24b0 con 0x7f8a740a02e0 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 0x7f8a7409f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient: get_auth_request con 0x7f8a740a02e0 auth_method 0 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8a64012bf0 con 0x7f8a7406d160 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 10 monclient(hunting): handle_auth_done global_id 4550 payload 824 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a727fc640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a7406df60 con 0x7f8a740a02e0 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.831+0000 7f8a6b7fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a5403e210 0x7f8a540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.832+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7f8a640534a0 con 0x7f8a7406d160 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.832+0000 7f8a737fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a5403e210 0x7f8a540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.832+0000 7f8a737fe640 10 monclient: get_auth_request con 0x7f8a5403e210 auth_method 0 2024-09-10T16:08:24.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.834+0000 7f8a727fc640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 msgr2=0x7f8a7409f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.834+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 0x7f8a7409f830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a1b50 tx=0x7f8a5c00ff70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.834+0000 7f8a737fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a5403e210 0x7f8a540406d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c031000 tx=0x7f8a6c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c00dd60 con 0x7f8a740a02e0 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 10 monclient: dump: 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.855 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a5c00dd60 con 0x7f8a740a02e0 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 msgr2=0x7f8a7409f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 0x7f8a7409f830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a1b50 tx=0x7f8a5c00ff70 comp rx=0 tx=0).stop 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 msgr2=0x7f8a7409f830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.836+0000 7f8a6b7fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a740a02e0 0x7f8a7409f830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a79447640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a79447640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 0x7f8a7402d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a79447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7412f8d0 con 0x7f8a7402d440 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 0x7f8a7402d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient: get_auth_request con 0x7f8a7402d440 auth_method 0 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 10 monclient(hunting): handle_auth_done global_id 4553 payload 824 2024-09-10T16:08:24.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.842+0000 7f8a727fc640 1 -- 172.21.15.164:0/2409350073 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a740aa1c0 con 0x7f8a7402d440 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a727fc640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 msgr2=0x7f8a7402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a727fc640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 0x7f8a7402d850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a1b50 tx=0x7f8a5c003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5c01e030 con 0x7f8a7402d440 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 10 monclient: dump: 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.858 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f8a5c01e030 con 0x7f8a7402d440 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 msgr2=0x7f8a7402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 0x7f8a7402d850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a1b50 tx=0x7f8a5c003760 comp rx=0 tx=0).stop 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 msgr2=0x7f8a7402d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:24.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a6b7fe640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a7402d440 0x7f8a7402d850 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a740a1b50 tx=0x7f8a5c003760 comp rx=0 tx=0).stop 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 147017, 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.860 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.861 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.862 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.862 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.863 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:24.864 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:24.865 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:24.866 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.867 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.868 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.869 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:24.870 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:24.871 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:24.872 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:24.873 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:24.874 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:08:24.875 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:24.875 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:24.875 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:24.875 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:24.875 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a5403e210 msgr2=0x7f8a540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a5403e210 0x7f8a540406d0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8a6c031000 tx=0x7f8a6c02e000 comp rx=0 tx=0).stop 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 10 monclient: shutdown 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 msgr2=0x7f8a7412d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 --2- 172.21.15.164:0/2409350073 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a7406d160 0x7f8a7412d210 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8a6400cc10 tx=0x7f8a6400a430 comp rx=0 tx=0).stop 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a737fe640 1 -- 172.21.15.164:0/2409350073 reap_dead start 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a737fe640 1 -- 172.21.15.164:0/2409350073 reap_dead start 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 shutdown_connections 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 >> 172.21.15.164:0/2409350073 conn(0x7f8a740aacb0 msgr2=0x7f8a7413eda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 shutdown_connections 2024-09-10T16:08:24.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.845+0000 7f8a79447640 1 -- 172.21.15.164:0/2409350073 wait complete. 2024-09-10T16:08:24.877 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:24.877 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:08:24.878 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.975+0000 7fa64a74b640 10 monclient: get_monmap_and_config 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.975+0000 7fa64a74b640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.975+0000 7fa64a74b640 10 monclient: monmap: 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:24.976681+0000 2024-09-10T16:08:24.977 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:24.976681+0000 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:24.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.976+0000 7fa64a74b640 1 Processor -- start 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.976+0000 7fa64a74b640 1 -- start start 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.976+0000 7fa64a74b640 10 monclient: init 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.976+0000 7fa64a74b640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.976+0000 7fa64a74b640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-b con 0x7fa64409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa64409d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:24.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa64413a5c0 0x7fa64413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-c con 0x7fa64413a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64409a430 con 0x7fa64409d160 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa6440a24e0 con 0x7fa64413a5c0 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa6440a2bf0 con 0x7fa64409c180 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa64a74b640 10 monclient(hunting): authenticate will time out at 883.464068s 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39600/0 (socket says 172.21.15.164:39600) 2024-09-10T16:08:24.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 -- 172.21.15.164:0/390633051 learned_addr learned my addr 172.21.15.164:0/390633051 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): get_auth_request con 0x7fa64409c180 auth_method 0 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 1 --2- 172.21.15.164:0/390633051 >> v2:172.21.15.165:3300/0 conn(0x7fa64413a5c0 0x7fa64413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): get_auth_request con 0x7fa64413a5c0 auth_method 0 2024-09-10T16:08:24.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient(hunting): handle_auth_done global_id 4556 payload 274 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 -- 172.21.15.164:0/390633051 >> v2:172.21.15.165:3300/0 conn(0x7fa64413a5c0 msgr2=0x7fa64413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 --2- 172.21.15.164:0/390633051 >> v2:172.21.15.165:3300/0 conn(0x7fa64413a5c0 0x7fa64413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 -- 172.21.15.164:0/390633051 >> v1:172.21.15.165:6789/0 conn(0x7fa64409d160 legacy=0x7fa64409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient: _finish_hunting 0 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 monclient: found mon.noname-b 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:24.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.977+0000 7fa643fff640 1 -- 172.21.15.164:0/390633051 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64409c560 con 0x7fa64409c180 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa643fff640 1 -- 172.21.15.164:0/390633051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 msgr2=0x7fa64409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa643fff640 1 --2- 172.21.15.164:0/390633051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa64409abb0 tx=0x7fa63402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9b3ab5f9ea57c05 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 1 -- 172.21.15.164:0/390633051 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634037070 con 0x7fa64409c180 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: dump: 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.987 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: _finish_auth 0 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 1 -- 172.21.15.164:0/390633051 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa634032440 con 0x7fa64409c180 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 5 monclient: authenticate success, global_id 4556 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:24.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 4 set_mon_vals no callback set 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 1 -- 172.21.15.164:0/390633051 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6340327f0 con 0x7fa64409c180 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa642ffd640 10 monclient: dump: 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:24.989 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 10 monclient: shutdown 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 -- 172.21.15.164:0/390633051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 msgr2=0x7fa64409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 --2- 172.21.15.164:0/390633051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa64409abb0 tx=0x7fa63402a080 comp rx=0 tx=0).stop 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 -- 172.21.15.164:0/390633051 shutdown_connections 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 --2- 172.21.15.164:0/390633051 >> v2:172.21.15.165:3300/0 conn(0x7fa64413a5c0 0x7fa64413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 --2- 172.21.15.164:0/390633051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa64409ea40 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.980+0000 7fa64a74b640 1 -- 172.21.15.164:0/390633051 >> 172.21.15.164:0/390633051 conn(0x7fa6440a3110 msgr2=0x7fa6440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.981+0000 7fa64a74b640 1 -- 172.21.15.164:0/390633051 shutdown_connections 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.981+0000 7fa64a74b640 1 -- 172.21.15.164:0/390633051 wait complete. 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.981+0000 7fa64a74b640 10 monclient: build_initial_monmap 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.981+0000 7fa64a74b640 10 monclient: monmap: 2024-09-10T16:08:24.990 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.981+0000 7fa64a74b640 1 Processor -- start 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 -- start start 2024-09-10T16:08:24.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient: init 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa6441c8b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-c con 0x7fa64409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64409d160 0x7fa644099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-b con 0x7fa64409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64413a5c0 0x7fa644099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): picked mon.noname-a con 0x7fa64413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa644074f90 con 0x7fa64413a5c0 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa644074e10 con 0x7fa64409d160 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa64409f690 con 0x7fa64409c180 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa64a74b640 10 monclient(hunting): authenticate will time out at 883.469518s 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa6441c8b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa6441c8b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39604/0 (socket says 172.21.15.164:39604) 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 -- 172.21.15.164:0/1728360615 learned_addr learned my addr 172.21.15.164:0/1728360615 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): get_auth_request con 0x7fa64409c180 auth_method 0 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64413a5c0 0x7fa644099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): get_auth_request con 0x7fa64413a5c0 auth_method 0 2024-09-10T16:08:24.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64409d160 0x7fa644099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient(hunting): handle_auth_done global_id 4559 payload 824 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64409d160 msgr2=0x7fa644099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64409d160 0x7fa644099510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64413a5c0 msgr2=0x7fa644099a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64413a5c0 0x7fa644099a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient: _finish_hunting 0 2024-09-10T16:08:25.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 monclient: found mon.noname-c 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa643fff640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa644091ca0 con 0x7fa64409c180 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa6437fe640 10 monclient: get_auth_request con 0x7fa64409d160 auth_method 0 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64409d160 0x7fa644099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.982+0000 7fa648ce8640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64413a5c0 0x7fa644099a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa643fff640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 msgr2=0x7fa6441c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa643fff640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa6441c8b30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa6340090b0 tx=0x7fa6340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634041020 con 0x7fa64409c180 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: dump: 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.003 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: _finish_auth 0 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa63403d830 con 0x7fa64409c180 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 5 monclient: authenticate success, global_id 4559 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 10 monclient: _renew_subs 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa634037040 con 0x7fa64409c180 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa644091f80 con 0x7fa64409c180 2024-09-10T16:08:25.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa640ff9640 10 monclient: dump: 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.005 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa6427fc640 4 set_mon_vals no callback set 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 10 monclient: _renew_subs 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.985+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64408fc20 con 0x7fa64409c180 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa64a74b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa64a74b640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 0x7fa64408e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa64409d680 con 0x7fa6440916e0 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa634044020 con 0x7fa64409c180 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 0x7fa64408e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient: get_auth_request con 0x7fa6440916e0 auth_method 0 2024-09-10T16:08:25.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa640ff9640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa63003e240 0x7fa630040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7fa6340732f0 con 0x7fa64409c180 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa648ce8640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa63003e240 0x7fa630040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa648ce8640 10 monclient: get_auth_request con 0x7fa63003e240 auth_method 0 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 10 monclient(hunting): handle_auth_done global_id 14206 payload 824 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.986+0000 7fa6437fe640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa64409d930 con 0x7fa6440916e0 2024-09-10T16:08:25.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa6437fe640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 msgr2=0x7fa64408e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 0x7fa64408e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa644090290 tx=0x7fa63800d950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa648ce8640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa63003e240 0x7fa630040700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6440906c0 tx=0x7fa63c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638012a10 con 0x7fa6440916e0 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 10 monclient: dump: 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.008 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa638012a10 con 0x7fa6440916e0 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 msgr2=0x7fa64408e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 0x7fa64408e9a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa644090290 tx=0x7fa63800d950 comp rx=0 tx=0).stop 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 msgr2=0x7fa64408e9a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.989+0000 7fa640ff9640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa6440916e0 0x7fa64408e9a0 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa644090290 tx=0x7fa63800d950 comp rx=0 tx=0).stop 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.995+0000 7fa64a74b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.995+0000 7fa64a74b640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 0x7fa64402d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.995+0000 7fa64a74b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.995+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa644071ef0 con 0x7fa64402d440 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.995+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 0x7fa64402d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient: get_auth_request con 0x7fa64402d440 auth_method 0 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 10 monclient(hunting): handle_auth_done global_id 14209 payload 824 2024-09-10T16:08:25.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.996+0000 7fa6437fe640 1 -- 172.21.15.164:0/1728360615 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa64408eee0 con 0x7fa64402d440 2024-09-10T16:08:25.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa6437fe640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 msgr2=0x7fa64402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:25.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa6437fe640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 0x7fa64402d850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa644090290 tx=0x7fa638003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa638006bf0 con 0x7fa64402d440 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa640ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa640ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.998+0000 7fa640ff9640 10 monclient: dump: 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.012 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fa638006bf0 con 0x7fa64402d440 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 msgr2=0x7fa64402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 0x7fa64402d850 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa644090290 tx=0x7fa638003760 comp rx=0 tx=0).stop 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 msgr2=0x7fa64402d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa640ff9640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64402d440 0x7fa64402d850 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.013 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 24, 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 78472, 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:25.014 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:25.015 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:25.016 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:25.017 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:25.018 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:25.019 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.020 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.021 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.022 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:25.023 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:25.024 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.025 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:25.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:25.027 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:25.027 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa63003e240 msgr2=0x7fa630040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa63003e240 0x7fa630040700 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa6440906c0 tx=0x7fa63c00b040 comp rx=0 tx=0).stop 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 10 monclient: shutdown 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 msgr2=0x7fa6441c8b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 --2- 172.21.15.164:0/1728360615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64409c180 0x7fa6441c8b30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fa6340090b0 tx=0x7fa6340046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa648ce8640 1 -- 172.21.15.164:0/1728360615 reap_dead start 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa648ce8640 1 -- 172.21.15.164:0/1728360615 reap_dead start 2024-09-10T16:08:25.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 shutdown_connections 2024-09-10T16:08:25.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 >> 172.21.15.164:0/1728360615 conn(0x7fa6440a3110 msgr2=0x7fa644138860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:25.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 shutdown_connections 2024-09-10T16:08:25.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:24.999+0000 7fa64a74b640 1 -- 172.21.15.164:0/1728360615 wait complete. 2024-09-10T16:08:25.029 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:08:25.029 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:08:25.029 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:08:25.029 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:08:25.029 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:08:25.030 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:08:25.030 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:08:25.042 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:08:25.042 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:08:25.043 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:25.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.138+0000 7feceb1a0640 10 monclient: get_monmap_and_config 2024-09-10T16:08:25.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.138+0000 7feceb1a0640 10 monclient: build_initial_monmap 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.138+0000 7feceb1a0640 10 monclient: monmap: 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:25.139165+0000 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:25.139165+0000 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:25.140 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:25.141 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:25.141 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:25.141 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:25.141 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 Processor -- start 2024-09-10T16:08:25.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 -- start start 2024-09-10T16:08:25.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient: init 2024-09-10T16:08:25.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:25.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fece409c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fece409d160 0x7fece409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-a con 0x7fece409d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 0x7fece413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fece413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece409a430 con 0x7fece409c180 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fece40a24e0 con 0x7fece409d160 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fece40a2bf0 con 0x7fece413a5c0 2024-09-10T16:08:25.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7feceb1a0640 10 monclient(hunting): authenticate will time out at 883.626595s 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece8f3c640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fece409c180 legacy=0x7fece409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fece409d160 0x7fece409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 0x7fece413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fece409d160 0x7fece409f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42268/0 (socket says 172.21.15.164:42268) 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 -- 172.21.15.164:0/1303108962 learned_addr learned my addr 172.21.15.164:0/1303108962 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): get_auth_request con 0x7fece409d160 auth_method 0 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): get_auth_request con 0x7fece413a5c0 auth_method 0 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece973d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient(hunting): handle_auth_done global_id 24156 payload 274 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 -- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 msgr2=0x7fece413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:25.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 0x7fece413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 -- 172.21.15.164:0/1303108962 >> v1:172.21.15.165:6789/0 conn(0x7fece409c180 legacy=0x7fece409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.139+0000 7fece3fff640 1 monclient: found mon.noname-a 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.140+0000 7fece3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.140+0000 7fece3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.140+0000 7fece3fff640 1 -- 172.21.15.164:0/1303108962 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece409c560 con 0x7fece409d160 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.140+0000 7fece973d640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 0x7fece413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece3fff640 1 -- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409d160 msgr2=0x7fece409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece3fff640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409d160 0x7fece409f150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feccc0098a0 tx=0x7feccc02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d055d1b0a31f54b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 1 -- 172.21.15.164:0/1303108962 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc037070 con 0x7fece409d160 2024-09-10T16:08:25.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: dump: 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.157 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 1 -- 172.21.15.164:0/1303108962 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feccc032440 con 0x7fece409d160 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7feceb1a0640 5 monclient: authenticate success, global_id 24156 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 1 -- 172.21.15.164:0/1303108962 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feccc032810 con 0x7fece409d160 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7feceb1a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7feceb1a0640 4 set_mon_vals no callback set 2024-09-10T16:08:25.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.142+0000 7fece37fe640 10 monclient: dump: 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 10 monclient: shutdown 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 -- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409d160 msgr2=0x7fece409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409d160 0x7fece409f150 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feccc0098a0 tx=0x7feccc02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 -- 172.21.15.164:0/1303108962 shutdown_connections 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece413a5c0 0x7fece413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1303108962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409d160 0x7fece409f150 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 -- 172.21.15.164:0/1303108962 >> 172.21.15.164:0/1303108962 conn(0x7fece40a3110 msgr2=0x7fece40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 -- 172.21.15.164:0/1303108962 shutdown_connections 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.143+0000 7feceb1a0640 1 -- 172.21.15.164:0/1303108962 wait complete. 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient: build_initial_monmap 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient: monmap: 2024-09-10T16:08:25.160 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 1 Processor -- start 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 1 -- start start 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient: init 2024-09-10T16:08:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-a con 0x7fece409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 0x7fece409de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.144+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-c con 0x7fece409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 0x7fece409e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): picked mon.noname-b con 0x7fece413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fece4074f90 con 0x7fece409c180 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fece4074e10 con 0x7fece413a5c0 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fece409f690 con 0x7fece409d160 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7feceb1a0640 10 monclient(hunting): authenticate will time out at 883.632040s 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece973d640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 msgr2=0x7fece409e3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42270/0 (socket says 172.21.15.164:42270) 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece973d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 0x7fece409e3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 -- 172.21.15.164:0/1092576977 learned_addr learned my addr 172.21.15.164:0/1092576977 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): get_auth_request con 0x7fece409c180 auth_method 0 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 0x7fece409de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): get_auth_request con 0x7fece409d160 auth_method 0 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient(hunting): handle_auth_done global_id 24159 payload 824 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 -- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 msgr2=0x7fece409de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 0x7fece409de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 -- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 msgr2=0x7fece409e3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 0x7fece409e3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient: _finish_hunting 0 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 monclient: found mon.noname-a 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece8f3c640 1 -- 172.21.15.164:0/1092576977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece412d130 con 0x7fece409c180 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.145+0000 7fece3fff640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 0x7fece409de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece8f3c640 1 -- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 msgr2=0x7fece409d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece8f3c640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fecd400b580 tx=0x7fecd400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd4013020 con 0x7fece409c180 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: dump: 2024-09-10T16:08:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 5 monclient: authenticate success, global_id 24159 2024-09-10T16:08:25.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecd4005070 con 0x7fece409c180 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecd4005480 con 0x7fece409c180 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 10 monclient: _renew_subs 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece409d620 con 0x7fece409c180 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece17fa640 10 monclient: dump: 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:25.167 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7fece2ffd640 4 set_mon_vals no callback set 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 10 monclient: _renew_subs 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.147+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fece412ce50 con 0x7fece409c180 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.148+0000 7feceb1a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.148+0000 7feceb1a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.148+0000 7feceb1a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.148+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fecd4016020 con 0x7fece409c180 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.148+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece409ea40 con 0x7fece409c180 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.149+0000 7fece17fa640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 0x7fecc4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.149+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7fecd40527d0 con 0x7fece409c180 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.149+0000 7fece3fff640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 0x7fecc4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:25.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.149+0000 7fece3fff640 10 monclient: get_auth_request con 0x7fecc403e1c0 auth_method 0 2024-09-10T16:08:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.151+0000 7fece3fff640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 0x7fecc4040680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fece406d4a0 tx=0x7feccc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.153+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fecd401d030 con 0x7fece409c180 2024-09-10T16:08:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.153+0000 7fece17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:25.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.153+0000 7fece17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:25.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7feceb1a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:25.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7feceb1a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7feceb1a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fece412ba70 con 0x7fece409c180 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7fece17fa640 1 -- 172.21.15.164:0/1092576977 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fecd40173b0 con 0x7fece409c180 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7fece17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.399+0000 7fece17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:25.401 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:25.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 msgr2=0x7fecc4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 0x7fecc4040680 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fece406d4a0 tx=0x7feccc02e000 comp rx=0 tx=0).stop 2024-09-10T16:08:25.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 10 monclient: shutdown 2024-09-10T16:08:25.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:25.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 msgr2=0x7fece409d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fecd400b580 tx=0x7fecd400ba40 comp rx=0 tx=0).stop 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 shutdown_connections 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecc403e1c0 0x7fecc4040680 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fece413a5c0 0x7fece409e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fece409d160 0x7fece409de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 --2- 172.21.15.164:0/1092576977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fece409c180 0x7fece409d930 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 >> 172.21.15.164:0/1092576977 conn(0x7fece40a3110 msgr2=0x7fece4139d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 shutdown_connections 2024-09-10T16:08:25.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:25.401+0000 7feceb1a0640 1 -- 172.21.15.164:0/1092576977 wait complete. 2024-09-10T16:08:28.414 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.521+0000 7ff3588b1640 10 monclient: get_monmap_and_config 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.521+0000 7ff3588b1640 10 monclient: build_initial_monmap 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.522+0000 7ff3588b1640 10 monclient: monmap: 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:28.523 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:28.522870+0000 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:28.522870+0000 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:28.524 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.522+0000 7ff3588b1640 1 Processor -- start 2024-09-10T16:08:28.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.522+0000 7ff3588b1640 1 -- start start 2024-09-10T16:08:28.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.522+0000 7ff3588b1640 10 monclient: init 2024-09-10T16:08:28.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 0x7ff354134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-c con 0x7ff35406cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3540a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3540a4ab0 0x7ff3541383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff3540a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff35409ce50 con 0x7ff3540a46a0 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff35406db60 con 0x7ff3540a4ab0 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff35406cf50 con 0x7ff35406cb70 2024-09-10T16:08:28.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff3588b1640 10 monclient(hunting): authenticate will time out at 887.010333s 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3540a4ab0 0x7ff3541383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff35259c640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 msgr2=0x7ff354134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff3540a4ab0 0x7ff3541383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42278/0 (socket says 172.21.15.164:42278) 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 -- 172.21.15.164:0/2342036353 learned_addr learned my addr 172.21.15.164:0/2342036353 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): get_auth_request con 0x7ff3540a4ab0 auth_method 0 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff35259c640 1 --2- 172.21.15.164:0/2342036353 >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 0x7ff354134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:28.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient(hunting): handle_auth_done global_id 24162 payload 274 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 -- 172.21.15.164:0/2342036353 >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 msgr2=0x7ff354134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 --2- 172.21.15.164:0/2342036353 >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 0x7ff354134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 -- 172.21.15.164:0/2342036353 >> v1:172.21.15.164:6789/0 conn(0x7ff3540a46a0 legacy=0x7ff354137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 monclient: found mon.noname-a 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.523+0000 7ff352d9d640 1 -- 172.21.15.164:0/2342036353 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35409cb70 con 0x7ff3540a4ab0 2024-09-10T16:08:28.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff352d9d640 1 -- 172.21.15.164:0/2342036353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 msgr2=0x7ff3541383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:28.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff352d9d640 1 --2- 172.21.15.164:0/2342036353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff3541383b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009e80 tx=0x7ff34c02a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b098ab2aabbafc1e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:28.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 1 -- 172.21.15.164:0/2342036353 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c036070 con 0x7ff3540a4ab0 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: dump: 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:28.536 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: _finish_auth 0 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: _check_auth_tickets 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff3588b1640 5 monclient: authenticate success, global_id 24162 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 1 -- 172.21.15.164:0/2342036353 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34c007800 con 0x7ff3540a4ab0 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff3588b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff3588b1640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff3588b1640 4 set_mon_vals no callback set 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 1 -- 172.21.15.164:0/2342036353 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c007bd0 con 0x7ff3540a4ab0 2024-09-10T16:08:28.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.526+0000 7ff35159a640 10 monclient: dump: 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:28.538 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 10 monclient: shutdown 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 -- 172.21.15.164:0/2342036353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 msgr2=0x7ff3541383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 --2- 172.21.15.164:0/2342036353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff3541383b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff34c009e80 tx=0x7ff34c02a5f0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 -- 172.21.15.164:0/2342036353 shutdown_connections 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 --2- 172.21.15.164:0/2342036353 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff3541383b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 --2- 172.21.15.164:0/2342036353 >> v2:172.21.15.165:3300/0 conn(0x7ff35406cb70 0x7ff354134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 -- 172.21.15.164:0/2342036353 >> 172.21.15.164:0/2342036353 conn(0x7ff3540aa900 msgr2=0x7ff3540aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 -- 172.21.15.164:0/2342036353 shutdown_connections 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.527+0000 7ff3588b1640 1 -- 172.21.15.164:0/2342036353 wait complete. 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient: build_initial_monmap 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient: monmap: 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:28.539 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 1 Processor -- start 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 1 -- start start 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient: init 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:08:28.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-c con 0x7ff35406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 0x7ff35409ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.528+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-b con 0x7ff3540a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff3540a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff354138a70 con 0x7ff3540a4ab0 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff3541388f0 con 0x7ff3540a46a0 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff354140040 con 0x7ff35406cb70 2024-09-10T16:08:28.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff3588b1640 10 monclient(hunting): authenticate will time out at 887.016042s 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42280/0 (socket says 172.21.15.164:42280) 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39642/0 (socket says 172.21.15.164:39642) 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 -- 172.21.15.164:0/3683306840 learned_addr learned my addr 172.21.15.164:0/3683306840 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): get_auth_request con 0x7ff3540a4ab0 auth_method 0 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff351d9b640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 msgr2=0x7ff35409ee20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): get_auth_request con 0x7ff35406cb70 auth_method 0 2024-09-10T16:08:28.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff351d9b640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 0x7ff35409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient(hunting): handle_auth_done global_id 24165 payload 824 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 msgr2=0x7ff35409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 msgr2=0x7ff35409ee20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 0x7ff35409ee20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 monclient: found mon.noname-a 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff352d9d640 1 -- 172.21.15.164:0/3683306840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff35406e720 con 0x7ff3540a4ab0 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.529+0000 7ff35259c640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff352d9d640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 msgr2=0x7ff35409f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:28.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff352d9d640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0093e0 tx=0x7ff34c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c036040 con 0x7ff3540a4ab0 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: dump: 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: _finish_auth 0 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 5 monclient: authenticate success, global_id 24165 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff34c003990 con 0x7ff3540a4ab0 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 10 monclient: _renew_subs 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3540a1b90 con 0x7ff3540a4ab0 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff34c003d80 con 0x7ff3540a4ab0 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff342ffd640 10 monclient: dump: 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:28.547 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 10 monclient: _renew_subs 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff350d99640 4 set_mon_vals no callback set 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.531+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3541cd030 con 0x7ff3540a4ab0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.532+0000 7ff3588b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.532+0000 7ff3588b1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.532+0000 7ff3588b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.532+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff354134f70 con 0x7ff3540a4ab0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.532+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff34c0318c0 con 0x7ff3540a4ab0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.533+0000 7ff342ffd640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 0x7ff334040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.533+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7ff34c074130 con 0x7ff3540a4ab0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.533+0000 7ff35259c640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 0x7ff334040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:28.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.533+0000 7ff35259c640 10 monclient: get_auth_request con 0x7ff33403e240 auth_method 0 2024-09-10T16:08:28.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.536+0000 7ff35259c640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 0x7ff334040700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff348006e70 tx=0x7ff348006dc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:28.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.537+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff35409ce50 con 0x7ff3540a4ab0 2024-09-10T16:08:28.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.537+0000 7ff342ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:28.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.537+0000 7ff342ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:28.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.776+0000 7ff3588b1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:28.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.776+0000 7ff3588b1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:28.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.776+0000 7ff3588b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:28.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.776+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff354133e50 con 0x7ff3540a4ab0 2024-09-10T16:08:28.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.777+0000 7ff342ffd640 1 -- 172.21.15.164:0/3683306840 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7ff34c046d00 con 0x7ff3540a4ab0 2024-09-10T16:08:28.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.777+0000 7ff342ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:28.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.777+0000 7ff342ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:28.779 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 msgr2=0x7ff334040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 0x7ff334040700 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ff348006e70 tx=0x7ff348006dc0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 10 monclient: shutdown 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 msgr2=0x7ff35409f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff34c0093e0 tx=0x7ff34c004a40 comp rx=0 tx=0).stop 2024-09-10T16:08:28.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 shutdown_connections 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff33403e240 0x7ff334040700 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff3540a4ab0 0x7ff35409f360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff3540a46a0 0x7ff35409ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 --2- 172.21.15.164:0/3683306840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff35406cb70 0x7ff35409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 >> 172.21.15.164:0/3683306840 conn(0x7ff3540aa900 msgr2=0x7ff35412e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 shutdown_connections 2024-09-10T16:08:28.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:28.779+0000 7ff3588b1640 1 -- 172.21.15.164:0/3683306840 wait complete. 2024-09-10T16:08:31.793 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:31.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.901+0000 7f9350616640 10 monclient: get_monmap_and_config 2024-09-10T16:08:31.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.901+0000 7f9350616640 10 monclient: build_initial_monmap 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.901+0000 7f9350616640 10 monclient: monmap: 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:31.902782+0000 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:31.902782+0000 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:31.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.902+0000 7f9350616640 1 Processor -- start 2024-09-10T16:08:31.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.902+0000 7f9350616640 1 -- start start 2024-09-10T16:08:31.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.902+0000 7f9350616640 10 monclient: init 2024-09-10T16:08:31.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:31.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f934809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 0x7f934809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): picked mon.noname-c con 0x7f934809d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 0x7f934813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): picked mon.noname-b con 0x7f934813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f934809a430 con 0x7f934809c180 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f93480a24e0 con 0x7f934809d160 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f93480a2bf0 con 0x7f934813a5c0 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f9350616640 10 monclient(hunting): authenticate will time out at 890.390204s 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934dbb1640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 msgr2=0x7f934809f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:31.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 0x7f934813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934e3b2640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f934809c180 0x7f934809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34718/0 (socket says 172.21.15.164:34718) 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934dbb1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 0x7f934809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934e3b2640 1 -- 172.21.15.164:0/3749496068 learned_addr learned my addr 172.21.15.164:0/3749496068 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): get_auth_request con 0x7f934813a5c0 auth_method 0 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient(hunting): handle_auth_done global_id 4571 payload 274 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 -- 172.21.15.164:0/3749496068 >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 msgr2=0x7f934809f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 --2- 172.21.15.164:0/3749496068 >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 0x7f934809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:31.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 -- 172.21.15.164:0/3749496068 >> v1:172.21.15.164:6789/0 conn(0x7f934809c180 legacy=0x7f934809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:31.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient: _finish_hunting 0 2024-09-10T16:08:31.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 monclient: found mon.noname-b 2024-09-10T16:08:31.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:31.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:08:31.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.903+0000 7f934ebb3640 1 -- 172.21.15.164:0/3749496068 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934809c560 con 0x7f934813a5c0 2024-09-10T16:08:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934ebb3640 1 -- 172.21.15.164:0/3749496068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 msgr2=0x7f934813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934ebb3640 1 --2- 172.21.15.164:0/3749496068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 0x7f934813c9b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9340009dc0 tx=0x7f934002a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db376488b6789bf4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 1 -- 172.21.15.164:0/3749496068 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340007a30 con 0x7f934813a5c0 2024-09-10T16:08:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:31.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: dump: 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: _finish_auth 0 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: _check_auth_tickets 2024-09-10T16:08:31.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 1 -- 172.21.15.164:0/3749496068 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9340007bd0 con 0x7f934813a5c0 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 1 -- 172.21.15.164:0/3749496068 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340031400 con 0x7f934813a5c0 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f9350616640 5 monclient: authenticate success, global_id 4571 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f9350616640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f9350616640 4 set_mon_vals no callback set 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.906+0000 7f934d3b0640 10 monclient: dump: 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.917 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 10 monclient: shutdown 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 -- 172.21.15.164:0/3749496068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 msgr2=0x7f934813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 --2- 172.21.15.164:0/3749496068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 0x7f934813c9b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f9340009dc0 tx=0x7f934002a500 comp rx=0 tx=0).stop 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 -- 172.21.15.164:0/3749496068 shutdown_connections 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 --2- 172.21.15.164:0/3749496068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934813a5c0 0x7f934813c9b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 --2- 172.21.15.164:0/3749496068 >> v2:172.21.15.165:3300/0 conn(0x7f934809d160 0x7f934809f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 -- 172.21.15.164:0/3749496068 >> 172.21.15.164:0/3749496068 conn(0x7f93480a3110 msgr2=0x7f93480a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 -- 172.21.15.164:0/3749496068 shutdown_connections 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 1 -- 172.21.15.164:0/3749496068 wait complete. 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 10 monclient: build_initial_monmap 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.907+0000 7f9350616640 10 monclient: monmap: 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:31.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 Processor -- start 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 -- start start 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient: init 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): picked mon.noname-a con 0x7f934809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 0x7f934813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): picked mon.noname-c con 0x7f934809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 0x7f934813d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): picked mon.noname-b con 0x7f934813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9348074f90 con 0x7f934809c180 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9348074e10 con 0x7f934813a5c0 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f934809f690 con 0x7f934809d160 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f9350616640 10 monclient(hunting): authenticate will time out at 890.395709s 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 0x7f934813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42288/0 (socket says 172.21.15.164:42288) 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 1 -- 172.21.15.164:0/2817230527 learned_addr learned my addr 172.21.15.164:0/2817230527 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): get_auth_request con 0x7f934809c180 auth_method 0 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:31.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): get_auth_request con 0x7f934809d160 auth_method 0 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934ebb3640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 msgr2=0x7f934813d430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934ebb3640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 0x7f934813d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.908+0000 7f934dbb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 10 monclient(hunting): handle_auth_done global_id 4574 payload 824 2024-09-10T16:08:31.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 msgr2=0x7f934813d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 0x7f934813d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 msgr2=0x7f9348099280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 10 monclient: _finish_hunting 0 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 monclient: found mon.noname-c 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934dbb1640 1 -- 172.21.15.164:0/2817230527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934809d620 con 0x7f934809d160 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.909+0000 7f934e3b2640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f934dbb1640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 msgr2=0x7f934813cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f934dbb1640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 0x7f934813cef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f934400b450 tx=0x7f934400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344005070 con 0x7f934809d160 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:31.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: dump: 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: _finish_auth 0 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: _check_auth_tickets 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:31.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9344005210 con 0x7f934809d160 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 5 monclient: authenticate success, global_id 4574 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 10 monclient: _renew_subs 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9348093240 con 0x7f934809d160 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:31.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9344005600 con 0x7f934809d160 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f933affd640 10 monclient: dump: 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:31.926 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:31.927 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f934cbaf640 4 set_mon_vals no callback set 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 10 monclient: _renew_subs 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.911+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9348092390 con 0x7f934809d160 2024-09-10T16:08:31.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f9350616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:31.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f9350616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:31.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f9350616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:31.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f934809d930 con 0x7f934809d160 2024-09-10T16:08:31.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f934401a400 con 0x7f934809d160 2024-09-10T16:08:31.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f933affd640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 0x7f932c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:31.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.912+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(74..74 src has 1..74) ==== 4787+0+0 (secure 0 0 0) 0x7f93440527e0 con 0x7f934809d160 2024-09-10T16:08:31.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.913+0000 7f934e3b2640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 0x7f932c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:31.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.913+0000 7f934e3b2640 10 monclient: get_auth_request con 0x7f932c03e210 auth_method 0 2024-09-10T16:08:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.915+0000 7f934e3b2640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 0x7f932c0406d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f933c007d50 tx=0x7f933c007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.917+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f93440057a0 con 0x7f934809d160 2024-09-10T16:08:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.917+0000 7f933affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:31.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:31.917+0000 7f933affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:32.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.159+0000 7f9350616640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.159+0000 7f9350616640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.159+0000 7f9350616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.159+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9348071600 con 0x7f934809d160 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.160+0000 7f933affd640 1 -- 172.21.15.164:0/2817230527 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f9344011400 con 0x7f934809d160 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.160+0000 7f933affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.160+0000 7f933affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:32.162 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 msgr2=0x7f932c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 0x7f932c0406d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f933c007d50 tx=0x7f933c007ca0 comp rx=0 tx=0).stop 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 10 monclient: shutdown 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 msgr2=0x7f934813cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:32.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 0x7f934813cef0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f934400b450 tx=0x7f934400b910 comp rx=0 tx=0).stop 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 shutdown_connections 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932c03e210 0x7f932c0406d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934813a5c0 0x7f934813d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934809d160 0x7f934813cef0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 --2- 172.21.15.164:0/2817230527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934809c180 0x7f9348099280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 >> 172.21.15.164:0/2817230527 conn(0x7f93480a3110 msgr2=0x7f934806e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 shutdown_connections 2024-09-10T16:08:32.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:32.162+0000 7f9350616640 1 -- 172.21.15.164:0/2817230527 wait complete. 2024-09-10T16:08:35.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:08:35.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.283+0000 7fed2af33640 10 monclient: get_monmap_and_config 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.283+0000 7fed2af33640 10 monclient: build_initial_monmap 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.283+0000 7fed2af33640 10 monclient: monmap: 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:35.284339+0000 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:35.284339+0000 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:35.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:35.286 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:35.286 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:35.286 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:35.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 Processor -- start 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 -- start start 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient: init 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed2409d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-b con 0x7fed2409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:35.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-a con 0x7fed2409e420 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed240a0870 con 0x7fed2409d930 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fed2409d3e0 con 0x7fed2409e420 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed24074c70 con 0x7fed2409ab50 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed2af33640 10 monclient(hunting): authenticate will time out at 893.771773s 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed28ccf640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fed2409d930 0x7fed2409dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57152/0 (socket says 172.21.15.164:57152) 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed23fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed294d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 learned_addr learned my addr 172.21.15.164:0/541122163 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed294d0640 10 monclient(hunting): get_auth_request con 0x7fed2409e420 auth_method 0 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed294d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed294d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed294d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed23fff640 10 monclient(hunting): get_auth_request con 0x7fed2409ab50 auth_method 0 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.284+0000 7fed23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.285+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.287+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.164:6789/0 conn(0x7fed2409d930 legacy=0x7fed2409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:35.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 10 monclient(hunting): get_auth_request con 0x7fed2409e420 auth_method 0 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 10 monclient(hunting): get_auth_request con 0x7fed2409ab50 auth_method 0 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:35.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:35.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.486+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 10 monclient(hunting): get_auth_request con 0x7fed2409e420 auth_method 0 2024-09-10T16:08:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 10 monclient(hunting): get_auth_request con 0x7fed2409ab50 auth_method 0 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:35.887+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:36.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 10 monclient(hunting): get_auth_request con 0x7fed2409e420 auth_method 0 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 10 monclient(hunting): get_auth_request con 0x7fed2409ab50 auth_method 0 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:36.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:36.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:08:36.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:08:36.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:08:36.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:36.688+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): tick 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 msgr2=0x7fed241397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed2409ab50 0x7fed241397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 msgr2=0x7fed2413bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.164:3300/0 conn(0x7fed2409e420 0x7fed2413bd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.164:6789/0 conn(0x7fed2409d930 legacy=0x7fed2409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient: _add_conns ranks=[1,4] 2024-09-10T16:08:38.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed080059a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): picked mon.noname-c con 0x7fed08009610 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed2409ecb0 con 0x7fed080059a0 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fed2409d3e0 con 0x7fed08009610 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed227fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:38.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.285+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:38.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.486+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:38.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.486+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:38.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.486+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:08:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.887+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.887+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:38.887+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:08:39.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:39.688+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:39.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:39.688+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007de0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:39.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:39.688+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:08:40.626 INFO:tasks.rados.rados.0.smithi165.stdout:255: done (0 left) 2024-09-10T16:08:40.626 INFO:tasks.rados.rados.0.smithi165.stdout:256: delete oid 30 current snap is 33 2024-09-10T16:08:40.636 INFO:tasks.rados.rados.0.smithi165.stdout:256: done (0 left) 2024-09-10T16:08:40.636 INFO:tasks.rados.rados.0.smithi165.stdout:257: write (excl) oid 2 current snap is 33 2024-09-10T16:08:40.636 INFO:tasks.rados.rados.0.smithi165.stdout:257: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-10T16:08:40.643 INFO:tasks.rados.rados.0.smithi165.stdout:257: writing smithi16534841-2 from 766419 to 1315847 tid 1 2024-09-10T16:08:40.650 INFO:tasks.rados.rados.0.smithi165.stdout:257: writing smithi16534841-2 from 2013452 to 2588147 tid 2 2024-09-10T16:08:40.657 INFO:tasks.rados.rados.0.smithi165.stdout:257: writing smithi16534841-2 from 3346843 to 3774492 tid 3 2024-09-10T16:08:40.657 INFO:tasks.rados.rados.0.smithi165.stdout:258: snap_remove snap 30 2024-09-10T16:08:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): tick 2024-09-10T16:08:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:08:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:08:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed08009610 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007de0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 reap_dead start 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed080059a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed0800c160 0x7fed0800bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): picked mon.noname-c con 0x7fed0800c160 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed0800c680 addr v1:172.21.15.164:6789/0 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed0800d1e0 con 0x7fed0800c680 2024-09-10T16:08:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed2409ecb0 con 0x7fed080059a0 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 1 -- 172.21.15.164:0/541122163 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fed2409d3e0 con 0x7fed0800c160 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed227fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed0800c160 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.285+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed0800c160 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.287+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.164:6789/0 conn(0x7fed0800c680 legacy=0x7fed0800f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3772918373 0 0) 0x7fed0800d1e0 con 0x7fed0800c680 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): _negotiate 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): my global_id is 24171 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fed14003520 con 0x7fed0800c680 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4067578251 0 0) 0x7fed14003520 con 0x7fed0800c680 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed0800c160 msgr2=0x7fed0800bad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 --2- 172.21.15.164:0/541122163 >> v2:172.21.15.165:3300/0 conn(0x7fed0800c160 0x7fed0800bad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.165:6789/0 conn(0x7fed080059a0 legacy=0x7fed08007dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed0800cd90 con 0x7fed0800c680 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fed1c007670 con 0x7fed0800c680 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed2af33640 5 monclient: authenticate success, global_id 24171 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed2af33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: dump: 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:08:41.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 >> v1:172.21.15.164:6789/0 conn(0x7fed0800c680 legacy=0x7fed0800f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed14003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): picked mon.c con 0x7fed14003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed14004800 0x7fed14004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): picked mon.b con 0x7fed14004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed140053c0 0x7fed1400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): picked mon.a con 0x7fed140053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed2409d3e0 con 0x7fed140053c0 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed1400c280 con 0x7fed14004800 2024-09-10T16:08:41.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed1400c480 con 0x7fed14003a20 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed237fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed2af33640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed14003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed23fff640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed14004800 msgr2=0x7fed14004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed294d0640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed140053c0 0x7fed1400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): get_auth_request con 0x7fed14003a20 auth_method 0 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:41.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed294d0640 10 monclient(hunting): get_auth_request con 0x7fed140053c0 auth_method 0 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed294d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed294d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed294d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed23fff640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed14004800 0x7fed14004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.288+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed294d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed294d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed294d0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 10 monclient(hunting): handle_auth_done global_id 24171 payload 293 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed14004800 msgr2=0x7fed14004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed14004800 0x7fed14004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed140053c0 msgr2=0x7fed1400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed140053c0 0x7fed1400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 monclient: found mon.c 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed14003520 con 0x7fed14003a20 2024-09-10T16:08:41.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed294d0640 1 -- 172.21.15.164:0/541122163 reap_dead start 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.289+0000 7fed28ccf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed28ccf640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 msgr2=0x7fed14003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed28ccf640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed14003e20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fed180014a0 tx=0x7fed14005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccc61fd89394609e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180017b0 con 0x7fed14003a20 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: dump: 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.302 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed2af33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed1802e070 con 0x7fed14003a20 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed2af33640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 1 -- 172.21.15.164:0/541122163 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180048a0 con 0x7fed14003a20 2024-09-10T16:08:41.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed2af33640 4 set_mon_vals no callback set 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.292+0000 7fed237fe640 10 monclient: dump: 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.304 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 10 monclient: shutdown 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 -- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 msgr2=0x7fed14003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed14003e20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fed180014a0 tx=0x7fed14005830 comp rx=0 tx=0).stop 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 -- 172.21.15.164:0/541122163 shutdown_connections 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 --2- 172.21.15.164:0/541122163 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed14003e20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 -- 172.21.15.164:0/541122163 >> 172.21.15.164:0/541122163 conn(0x7fed240a3110 msgr2=0x7fed240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 -- 172.21.15.164:0/541122163 shutdown_connections 2024-09-10T16:08:41.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.293+0000 7fed2af33640 1 -- 172.21.15.164:0/541122163 wait complete. 2024-09-10T16:08:41.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.294+0000 7fed2af33640 10 monclient: build_initial_monmap 2024-09-10T16:08:41.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.294+0000 7fed2af33640 10 monclient: monmap: 2024-09-10T16:08:41.310 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:41.310 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:41.310 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.294+0000 7fed2af33640 1 Processor -- start 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.294+0000 7fed2af33640 1 -- start start 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.294+0000 7fed2af33640 10 monclient: init 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-c con 0x7fed14003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed2412fe40 0x7fed241302c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-a con 0x7fed2412fe40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 0x7fed241cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): picked mon.noname-b con 0x7fed241cd7f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed2409ef30 con 0x7fed2412fe40 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed240a0030 con 0x7fed241cd7f0 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed240a2940 con 0x7fed14003a20 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed2af33640 10 monclient(hunting): authenticate will time out at 899.782538s 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52396/0 (socket says 172.21.15.164:52396) 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 1 -- 172.21.15.164:0/191345949 learned_addr learned my addr 172.21.15.164:0/191345949 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): get_auth_request con 0x7fed14003a20 auth_method 0 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed2412fe40 0x7fed241302c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): get_auth_request con 0x7fed2412fe40 auth_method 0 2024-09-10T16:08:41.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed294d0640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 msgr2=0x7fed241cdc80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed294d0640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 0x7fed241cdc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 10 monclient(hunting): handle_auth_done global_id 4577 payload 824 2024-09-10T16:08:41.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 msgr2=0x7fed241cdc80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.295+0000 7fed28ccf640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 0x7fed241cdc80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed2412fe40 msgr2=0x7fed241302c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed2412fe40 0x7fed241302c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 1 monclient: found mon.noname-c 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.296+0000 7fed28ccf640 1 -- 172.21.15.164:0/191345949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed24130f20 con 0x7fed14003a20 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed28ccf640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 msgr2=0x7fed2412f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed28ccf640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed1802a870 tx=0x7fed180015c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed18044070 con 0x7fed14003a20 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: dump: 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.316 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed18004e00 con 0x7fed14003a20 2024-09-10T16:08:41.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed180072b0 con 0x7fed14003a20 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 5 monclient: authenticate success, global_id 4577 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed217fa640 10 monclient: dump: 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.318 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 10 monclient: _renew_subs 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed2409c9e0 con 0x7fed14003a20 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed22ffd640 4 set_mon_vals no callback set 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 10 monclient: _renew_subs 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.298+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed24132ba0 con 0x7fed14003a20 2024-09-10T16:08:41.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fed18047070 con 0x7fed14003a20 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed2af33640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed2af33640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed2af33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed240a5ca0 con 0x7fed14003a20 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.299+0000 7fed217fa640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 0x7fed080417c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.300+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(76..76 src has 1..76) ==== 4755+0+0 (secure 0 0 0) 0x7fed18072170 con 0x7fed14003a20 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.300+0000 7fed23fff640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 0x7fed080417c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.300+0000 7fed23fff640 10 monclient: get_auth_request con 0x7fed0803f300 auth_method 0 2024-09-10T16:08:41.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.302+0000 7fed23fff640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 0x7fed080417c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed10006fd0 tx=0x7fed100040c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.304+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fed2409c9e0 con 0x7fed14003a20 2024-09-10T16:08:41.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.304+0000 7fed217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.304+0000 7fed217fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:41.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.548+0000 7fed2af33640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:08:41.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.548+0000 7fed2af33640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.548+0000 7fed2af33640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.548+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed2409dd10 con 0x7fed14003a20 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.549+0000 7fed217fa640 1 -- 172.21.15.164:0/191345949 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fed1803e3e0 con 0x7fed14003a20 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.549+0000 7fed217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.549+0000 7fed217fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:41.551 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 msgr2=0x7fed080417c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 0x7fed080417c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fed10006fd0 tx=0x7fed100040c0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 10 monclient: shutdown 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 msgr2=0x7fed2412f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fed1802a870 tx=0x7fed180015c0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 shutdown_connections 2024-09-10T16:08:41.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed0803f300 0x7fed080417c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed241cd7f0 0x7fed241cdc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed2412fe40 0x7fed241302c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 --2- 172.21.15.164:0/191345949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed14003a20 0x7fed2412f900 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 >> 172.21.15.164:0/191345949 conn(0x7fed240a3110 msgr2=0x7fed241bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 shutdown_connections 2024-09-10T16:08:41.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.551+0000 7fed2af33640 1 -- 172.21.15.164:0/191345949 wait complete. 2024-09-10T16:08:41.563 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:08:41.563 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:08:41.564 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.662+0000 7f7f6d639640 10 monclient: get_monmap_and_config 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.662+0000 7f7f6d639640 10 monclient: build_initial_monmap 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.662+0000 7f7f6d639640 10 monclient: monmap: 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:41.663541+0000 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:41.663541+0000 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:41.665 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 Processor -- start 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 -- start start 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient: init 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f6806cb70 0x7f7f68134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-a con 0x7f7f6806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 0x7f7f680a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-b con 0x7f7f680a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 0x7f7f68138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-c con 0x7f7f68137ca0 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:41.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7f6806db60 con 0x7f7f6806cb70 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7f6806cf50 con 0x7f7f68137ca0 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7f6809ce50 con 0x7f7f680a46a0 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.663+0000 7f7f6d639640 10 monclient(hunting): authenticate will time out at 900.150986s 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f6806cb70 0x7f7f68134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 0x7f7f680a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f6806cb70 0x7f7f68134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60676/0 (socket says 172.21.15.164:60676) 2024-09-10T16:08:41.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f677fe640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 msgr2=0x7f7f68138130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 -- 172.21.15.164:0/773095160 learned_addr learned my addr 172.21.15.164:0/773095160 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): get_auth_request con 0x7f7f680a46a0 auth_method 0 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f677fe640 1 --2- 172.21.15.164:0/773095160 >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 0x7f7f68138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): get_auth_request con 0x7f7f6806cb70 auth_method 0 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_done global_id 24177 payload 274 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 -- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 msgr2=0x7f7f680a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 0x7f7f680a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 -- 172.21.15.164:0/773095160 >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 msgr2=0x7f7f68138130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/773095160 >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 0x7f7f68138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 monclient: found mon.noname-a 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f66ffd640 1 -- 172.21.15.164:0/773095160 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6809cb70 con 0x7f7f6806cb70 2024-09-10T16:08:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.664+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 0x7f7f680a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:08:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f66ffd640 1 -- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 msgr2=0x7f7f68134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f68134f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f68074d10 tx=0x7f7f5802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a3eaac0e77fe46d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 1 -- 172.21.15.164:0/773095160 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58037070 con 0x7f7f6806cb70 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: dump: 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.679 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 1 -- 172.21.15.164:0/773095160 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f58032640 con 0x7f7f6806cb70 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 5 monclient: authenticate success, global_id 24177 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 1 -- 172.21.15.164:0/773095160 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58032a10 con 0x7f7f6806cb70 2024-09-10T16:08:41.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 4 set_mon_vals no callback set 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f667fc640 10 monclient: dump: 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 10 monclient: shutdown 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 1 -- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 msgr2=0x7f7f68134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.667+0000 7f7f6d639640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f68134f70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7f68074d10 tx=0x7f7f5802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 -- 172.21.15.164:0/773095160 shutdown_connections 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 --2- 172.21.15.164:0/773095160 >> v2:172.21.15.165:3300/0 conn(0x7f7f68137ca0 0x7f7f68138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f680a46a0 0x7f7f680a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 --2- 172.21.15.164:0/773095160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f68134f70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 -- 172.21.15.164:0/773095160 >> 172.21.15.164:0/773095160 conn(0x7f7f680aa900 msgr2=0x7f7f680aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 -- 172.21.15.164:0/773095160 shutdown_connections 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 -- 172.21.15.164:0/773095160 wait complete. 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 10 monclient: build_initial_monmap 2024-09-10T16:08:41.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 10 monclient: monmap: 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.668+0000 7f7f6d639640 1 Processor -- start 2024-09-10T16:08:41.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 -- start start 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient: init 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f681cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-a con 0x7f7f6806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f680a46a0 0x7f7f681ce370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-b con 0x7f7f680a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f68137ca0 0x7f7f680a2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): picked mon.noname-c con 0x7f7f68137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681387f0 con 0x7f7f6806cb70 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68138670 con 0x7f7f680a46a0 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7f68138970 con 0x7f7f68137ca0 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f6d639640 10 monclient(hunting): authenticate will time out at 900.156477s 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f681cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f681cde30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60684/0 (socket says 172.21.15.164:60684) 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 -- 172.21.15.164:0/1891388186 learned_addr learned my addr 172.21.15.164:0/1891388186 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): get_auth_request con 0x7f7f6806cb70 auth_method 0 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f68137ca0 0x7f7f680a2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): get_auth_request con 0x7f7f68137ca0 auth_method 0 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f5ffff640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f680a46a0 msgr2=0x7f7f681ce370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f680a46a0 0x7f7f681ce370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient(hunting): handle_auth_done global_id 24180 payload 824 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f68137ca0 msgr2=0x7f7f680a2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f68137ca0 0x7f7f680a2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f680a46a0 msgr2=0x7f7f681ce370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f680a46a0 0x7f7f681ce370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 monclient: found mon.noname-a 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.669+0000 7f7f66ffd640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f68121ff0 con 0x7f7f6806cb70 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f66ffd640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 msgr2=0x7f7f681cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f66ffd640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f681cde30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f580090b0 tx=0x7f7f580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58041020 con 0x7f7f6806cb70 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: dump: 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f5803d850 con 0x7f7f6806cb70 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 5 monclient: authenticate success, global_id 24180 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 10 monclient: _renew_subs 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f68123140 con 0x7f7f6806cb70 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58037040 con 0x7f7f6806cb70 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f5f7fe640 10 monclient: dump: 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f65ffb640 4 set_mon_vals no callback set 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 10 monclient: _renew_subs 2024-09-10T16:08:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f680a99f0 con 0x7f7f6806cb70 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.672+0000 7f7f6d639640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f6d639640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 0x7f7f680a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7f68132600 con 0x7f7f680a3290 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 0x7f7f680a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient: get_auth_request con 0x7f7f680a3290 auth_method 0 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7f58044020 con 0x7f7f6806cb70 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_done global_id 24183 payload 824 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5ffff640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f6806e720 con 0x7f7f680a3290 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5f7fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f4803e1c0 0x7f7f48040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 4799+0+0 (secure 0 0 0) 0x7f7f58072fc0 con 0x7f7f6806cb70 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f677fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f4803e1c0 0x7f7f48040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.673+0000 7f7f677fe640 10 monclient: get_auth_request con 0x7f7f4803e1c0 auth_method 0 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5ffff640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 msgr2=0x7f7f680a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 0x7f7f680a3670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7f68121730 tx=0x7f7f5000eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5000cb90 con 0x7f7f680a3290 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:41.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 10 monclient: dump: 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f5000cb90 con 0x7f7f680a3290 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:41.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 msgr2=0x7f7f680a3670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 0x7f7f680a3670 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7f68121730 tx=0x7f7f5000eb90 comp rx=0 tx=0).stop 2024-09-10T16:08:41.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 msgr2=0x7f7f680a3670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:41.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.675+0000 7f7f5f7fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3290 0x7f7f680a3670 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7f68121730 tx=0x7f7f5000eb90 comp rx=0 tx=0).stop 2024-09-10T16:08:41.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.676+0000 7f7f677fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f4803e1c0 0x7f7f48040680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f60003cb0 tx=0x7f7f600095f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.681+0000 7f7f6d639640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.681+0000 7f7f6d639640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 0x7f7f68135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.681+0000 7f7f6d639640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.681+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7f681343d0 con 0x7f7f680a3bb0 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 0x7f7f68135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient: get_auth_request con 0x7f7f680a3bb0 auth_method 0 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 10 monclient(hunting): handle_auth_done global_id 24186 payload 824 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.682+0000 7f7f5ffff640 1 -- 172.21.15.164:0/1891388186 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f680a4ab0 con 0x7f7f680a3bb0 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5ffff640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 msgr2=0x7f7f68135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5ffff640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 0x7f7f68135870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7f50018000 tx=0x7f7f50004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f500044c0 con 0x7f7f680a3bb0 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 10 monclient: dump: 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f7f500044c0 con 0x7f7f680a3bb0 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 msgr2=0x7f7f68135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 0x7f7f68135870 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7f50018000 tx=0x7f7f50004730 comp rx=0 tx=0).stop 2024-09-10T16:08:41.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 msgr2=0x7f7f68135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.684+0000 7f7f5f7fe640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f680a3bb0 0x7f7f68135870 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f7f50018000 tx=0x7f7f50004730 comp rx=0 tx=0).stop 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 28, 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 46617, 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:41.699 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.700 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:41.701 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.702 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:41.703 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.704 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.705 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.706 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.707 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.708 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.710 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:41.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:41.712 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f4803e1c0 msgr2=0x7f7f48040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f4803e1c0 0x7f7f48040680 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f7f60003cb0 tx=0x7f7f600095f0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 10 monclient: shutdown 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 msgr2=0x7f7f681cde30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 --2- 172.21.15.164:0/1891388186 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f6806cb70 0x7f7f681cde30 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7f580090b0 tx=0x7f7f580046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f677fe640 1 -- 172.21.15.164:0/1891388186 reap_dead start 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f677fe640 1 -- 172.21.15.164:0/1891388186 reap_dead start 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 shutdown_connections 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 >> 172.21.15.164:0/1891388186 conn(0x7f7f680aa900 msgr2=0x7f7f68133360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 shutdown_connections 2024-09-10T16:08:41.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.685+0000 7f7f6d639640 1 -- 172.21.15.164:0/1891388186 wait complete. 2024-09-10T16:08:41.715 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:08:41.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:08:41.715 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.811+0000 7f7cc7065640 10 monclient: get_monmap_and_config 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.811+0000 7f7cc7065640 10 monclient: build_initial_monmap 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.811+0000 7f7cc7065640 10 monclient: monmap: 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:08:41.812817+0000 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:08:41.812817+0000 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:08:41.814 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.812+0000 7f7cc7065640 1 Processor -- start 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.812+0000 7f7cc7065640 1 -- start start 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.812+0000 7f7cc7065640 10 monclient: init 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc0074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 0x7f7cc00a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc00a1090 addr v2:172.21.15.165:3300/0 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cc00a2ec0 addr v1:172.21.15.165:6789/0 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cc009ebc0 con 0x7f7cc00a2ec0 2024-09-10T16:08:41.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7cc00698a0 con 0x7f7cc00a1090 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc006bfb0 con 0x7f7cc0074af0 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc7065640 10 monclient(hunting): authenticate will time out at 900.300199s 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cbffff640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 msgr2=0x7f7cc00a14d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc5602640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f7cc00a2ec0 legacy=0x7f7cc00a3330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cbffff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 0x7f7cc00a14d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52438/0 (socket says 172.21.15.164:52438) 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 -- 172.21.15.164:0/2862776505 learned_addr learned my addr 172.21.15.164:0/2862776505 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): get_auth_request con 0x7f7cc0074af0 auth_method 0 2024-09-10T16:08:41.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 -- 172.21.15.164:0/2862776505 >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 msgr2=0x7f7cc00a14d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/2862776505 >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 0x7f7cc00a14d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 -- 172.21.15.164:0/2862776505 >> v1:172.21.15.165:6789/0 conn(0x7f7cc00a2ec0 legacy=0x7f7cc00a3330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 monclient: found mon.noname-b 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.813+0000 7f7cc4e01640 1 -- 172.21.15.164:0/2862776505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc01383b0 con 0x7f7cc0074af0 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc4e01640 1 -- 172.21.15.164:0/2862776505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 msgr2=0x7f7cc0137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/2862776505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63116c1d8d5e06e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 1 -- 172.21.15.164:0/2862776505 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8037070 con 0x7f7cc0074af0 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: dump: 2024-09-10T16:08:41.825 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 1 -- 172.21.15.164:0/2862776505 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca8032640 con 0x7f7cc0074af0 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc7065640 5 monclient: authenticate success, global_id 4586 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc7065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc7065640 10 monclient: get_monmap_and_config success 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 1 -- 172.21.15.164:0/2862776505 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8032a10 con 0x7f7cc0074af0 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cc7065640 4 set_mon_vals no callback set 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.816+0000 7f7cbf7fe640 10 monclient: dump: 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.827 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 10 monclient: shutdown 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 -- 172.21.15.164:0/2862776505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 msgr2=0x7f7cc0137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 --2- 172.21.15.164:0/2862776505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 -- 172.21.15.164:0/2862776505 shutdown_connections 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 --2- 172.21.15.164:0/2862776505 >> v2:172.21.15.165:3300/0 conn(0x7f7cc00a1090 0x7f7cc00a14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 --2- 172.21.15.164:0/2862776505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc0137e70 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 -- 172.21.15.164:0/2862776505 >> 172.21.15.164:0/2862776505 conn(0x7f7cc00a3fb0 msgr2=0x7f7cc00a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 -- 172.21.15.164:0/2862776505 shutdown_connections 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 1 -- 172.21.15.164:0/2862776505 wait complete. 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 10 monclient: build_initial_monmap 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.817+0000 7f7cc7065640 10 monclient: monmap: 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:08:41.829 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 Processor -- start 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 -- start start 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient: init 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient: _reopen_session rank -1 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc0074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc00a1090 0x7f7cc0131700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc00a1090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cc00a2ec0 0x7f7cc0131c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc00a2ec0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:08:41.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc00a3c50 con 0x7f7cc00a1090 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7cc0068d70 con 0x7f7cc00a2ec0 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc00a52b0 con 0x7f7cc0074af0 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): _renew_subs 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc7065640 10 monclient(hunting): authenticate will time out at 900.305857s 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52442/0 (socket says 172.21.15.164:52442) 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 1 -- 172.21.15.164:0/79189407 learned_addr learned my addr 172.21.15.164:0/79189407 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 10 monclient(hunting): get_auth_request con 0x7f7cc0074af0 auth_method 0 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cc4e01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cbffff640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc00a1090 0x7f7cc0131700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cbffff640 10 monclient(hunting): get_auth_request con 0x7f7cc00a1090 auth_method 0 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.818+0000 7f7cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc5602640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cc00a2ec0 msgr2=0x7f7cc0131c40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc5602640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cc00a2ec0 0x7f7cc0131c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:08:41.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient(hunting): handle_auth_done global_id 4589 payload 824 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cc00a2ec0 msgr2=0x7f7cc0131c40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7cc00a2ec0 0x7f7cc0131c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc00a1090 msgr2=0x7f7cc0131700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7cc00a1090 0x7f7cc0131700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient: _finish_hunting 0 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 monclient: found mon.noname-c 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.819+0000 7f7cc4e01640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cc0132180 con 0x7f7cc0074af0 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc4e01640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 msgr2=0x7f7cc01cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8041070 con 0x7f7cc0074af0 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: dump: 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.834 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc4e01640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca80046c0 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f7ca803c880 config(0 keys) 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 5 monclient: authenticate success, global_id 4589 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 10 monclient: _renew_subs 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cc00a45c0 con 0x7f7cc0074af0 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ca803c880 con 0x7f7cc0074af0 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:08:41.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cbeffd640 4 set_mon_vals no callback set 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 10 monclient: _renew_subs 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.821+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cc01c16a0 con 0x7f7cc0074af0 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cc7065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cc7065640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 0x7f7cc01be280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc00a1a10 con 0x7f7cc01bdea0 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ca8037040 con 0x7f7cc0074af0 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 10 monclient: dump: 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.836 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7ca8044070 con 0x7f7cc0074af0 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 0x7f7cc01be280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient: get_auth_request con 0x7f7cc01bdea0 auth_method 0 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbd7fa640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ca003e1c0 0x7f7ca0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 10 monclient(hunting): handle_auth_done global_id 4592 payload 824 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.822+0000 7f7cbffff640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cc01c1270 con 0x7f7cc01bdea0 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.823+0000 7f7cc5602640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ca003e1c0 0x7f7ca0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.823+0000 7f7cc5602640 10 monclient: get_auth_request con 0x7f7ca003e1c0 auth_method 0 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbffff640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 msgr2=0x7f7cc01be280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbffff640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 0x7f7cc01be280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb000b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb001a070 con 0x7f7cc01bdea0 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 10 monclient: dump: 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7cb00055d0 con 0x7f7cc01bdea0 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 msgr2=0x7f7cc01be280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 0x7f7cc01be280 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb000b860 comp rx=0 tx=0).stop 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 msgr2=0x7f7cc01be280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cbd7fa640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01bdea0 0x7f7cc01be280 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb000b860 comp rx=0 tx=0).stop 2024-09-10T16:08:41.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.825+0000 7f7cc5602640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ca003e1c0 0x7f7ca0040680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8003930 tx=0x7f7cb8006c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cc7065640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cc7065640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 0x7f7cc009b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cc7065640 10 monclient(hunting): start opening mon connection 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7cc0137e70 con 0x7f7cc01be7c0 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 0x7f7cc009b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient: get_auth_request con 0x7f7cc01be7c0 auth_method 0 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.831+0000 7f7cbffff640 10 monclient(hunting): handle_auth_done global_id 4595 payload 824 2024-09-10T16:08:41.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.832+0000 7f7cbffff640 1 -- 172.21.15.164:0/79189407 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7cc00772c0 con 0x7f7cc01be7c0 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbffff640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 msgr2=0x7f7cc009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbffff640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 0x7f7cc009b3b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb0005d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb001a050 con 0x7f7cc01be7c0 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 10 monclient: dump: 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:08:41.844 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f7cb001b040 con 0x7f7cc01be7c0 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 msgr2=0x7f7cc009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 0x7f7cc009b3b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb0005d40 comp rx=0 tx=0).stop 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 msgr2=0x7f7cc009b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cbd7fa640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc01be7c0 0x7f7cc009b3b0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0009fb0 tx=0x7f7cb0005d40 comp rx=0 tx=0).stop 2024-09-10T16:08:41.845 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 28, 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 164006, 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.846 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.847 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.848 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:08:41.849 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:08:41.850 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.851 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.852 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:08:41.853 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.854 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:08:41.855 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:08:41.856 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.857 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 7 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:08:41.858 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:08:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:08:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:08:41.859 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:08:41.859 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:08:41.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ca003e1c0 msgr2=0x7f7ca0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ca003e1c0 0x7f7ca0040680 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8003930 tx=0x7f7cb8006c70 comp rx=0 tx=0).stop 2024-09-10T16:08:41.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 10 monclient: shutdown 2024-09-10T16:08:41.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:08:41.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 msgr2=0x7f7cc01cd370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc7065640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ca8009870 tx=0x7f7ca80046c0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.834+0000 7f7cc5602640 1 -- 172.21.15.164:0/79189407 reap_dead start 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cbd7fa640 1 -- 172.21.15.164:0/79189407 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(77..77 src has 1..77) ==== 4799+0+0 (secure 0 0 0) 0x7f7ca8073640 con 0x7f7cc0074af0 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 shutdown_connections 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cc7065640 1 --2- 172.21.15.164:0/79189407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7cc0074af0 0x7f7cc01cd370 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 >> 172.21.15.164:0/79189407 conn(0x7f7cc00a3fb0 msgr2=0x7f7cc00a59c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 shutdown_connections 2024-09-10T16:08:41.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:08:41.835+0000 7f7cc7065640 1 -- 172.21.15.164:0/79189407 wait complete. 2024-09-10T16:08:41.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:08:41.862 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:258: done (1 left) 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:259: snap_create 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:257: finishing write tid 1 to smithi16534841-2 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:257: finishing write tid 2 to smithi16534841-2 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:257: finishing write tid 3 to smithi16534841-2 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:257: finishing write tid 4 to smithi16534841-2 2024-09-10T16:08:42.282 INFO:tasks.rados.rados.0.smithi165.stdout:257: finishing write tid 5 to smithi16534841-2 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:257: oid 2 updating version 0 to 164 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:257: oid 2 updating version 164 to 166 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:257: oid 2 version 166 is already newer than 165 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:257: oid 2 updating version 166 to 167 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 167 (ObjNum 117 snap 33 seq_num 117) dirty exists 2024-09-10T16:08:42.283 INFO:tasks.rados.rados.0.smithi165.stdout:257: left oid 2 (ObjNum 117 snap 33 seq_num 117) 2024-09-10T16:08:42.645 INFO:tasks.rados.rados.0.smithi165.stdout:257: done (1 left) 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:259: done (0 left) 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:260: read oid 32 snap -1 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:260: expect (ObjNum 104 snap 28 seq_num 104) 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:261: rollback oid 47 current snap is 34 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 31 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:262: write (excl) oid 49 current snap is 34 2024-09-10T16:08:42.646 INFO:tasks.rados.rados.0.smithi165.stdout:262: seq_num 118 ranges {783226=624677,2071934=434706,2974266=638921} 2024-09-10T16:08:42.653 INFO:tasks.rados.rados.0.smithi165.stdout:262: writing smithi16534841-49 from 783226 to 1407903 tid 1 2024-09-10T16:08:42.659 INFO:tasks.rados.rados.0.smithi165.stdout:262: writing smithi16534841-49 from 2071934 to 2506640 tid 2 2024-09-10T16:08:42.665 INFO:tasks.rados.rados.0.smithi165.stdout:262: writing smithi16534841-49 from 2974266 to 3613187 tid 3 2024-09-10T16:08:42.665 INFO:tasks.rados.rados.0.smithi165.stdout:263: snap_remove snap 28 2024-09-10T16:08:43.659 INFO:tasks.rados.rados.0.smithi165.stdout:263: done (3 left) 2024-09-10T16:08:43.659 INFO:tasks.rados.rados.0.smithi165.stdout:264: delete oid 50 current snap is 34 2024-09-10T16:08:43.659 INFO:tasks.rados.rados.0.smithi165.stdout:261: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:08:43.660 INFO:tasks.rados.rados.0.smithi165.stdout:261: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:08:43.660 INFO:tasks.rados.rados.0.smithi165.stdout:261: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:08:43.660 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 171 (ObjNum 83 snap 18 seq_num 83) dirty exists 2024-09-10T16:08:43.660 INFO:tasks.rados.rados.0.smithi165.stdout:260: expect (ObjNum 104 snap 28 seq_num 104) 2024-09-10T16:08:43.700 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2953787 2024-09-10T16:08:43.778 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: finishing write tid 1 to smithi16534841-49 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: finishing write tid 2 to smithi16534841-49 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: finishing write tid 3 to smithi16534841-49 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: finishing write tid 4 to smithi16534841-49 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: finishing write tid 5 to smithi16534841-49 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: oid 49 updating version 0 to 30 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: oid 49 version 30 is already newer than 29 2024-09-10T16:08:43.848 INFO:tasks.rados.rados.0.smithi165.stdout:262: oid 49 updating version 30 to 32 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:262: oid 49 version 32 is already newer than 31 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 32 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:262: left oid 49 (ObjNum 118 snap 34 seq_num 118) 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:260: done (3 left) 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:261: done (2 left) 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:262: done (1 left) 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:264: done (0 left) 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:265: write oid 47 current snap is 34 2024-09-10T16:08:43.849 INFO:tasks.rados.rados.0.smithi165.stdout:265: seq_num 120 ranges {416840=775175,1672545=344295} 2024-09-10T16:08:43.854 INFO:tasks.rados.rados.0.smithi165.stdout:265: writing smithi16534841-47 from 416840 to 1192015 tid 1 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:265: writing smithi16534841-47 from 1672545 to 2016840 tid 2 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:266: read oid 19 snap -1 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:266: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:267: rollback oid 18 current snap is 34 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 32 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing write tid 1 to smithi16534841-47 2024-09-10T16:08:43.859 INFO:tasks.rados.rados.0.smithi165.stdout:268: delete oid 3 current snap is 34 2024-09-10T16:08:43.863 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing rollback tid 0 to smithi16534841-18 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing write tid 2 to smithi16534841-47 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing rollback tid 1 to smithi16534841-18 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:268: done (3 left) 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:269: snap_create 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:267: finishing rollback tid 2 to smithi16534841-18 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 129 (ObjNum 107 snap 29 seq_num 107) dirty exists 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing write tid 3 to smithi16534841-47 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:265: finishing write tid 4 to smithi16534841-47 2024-09-10T16:08:43.864 INFO:tasks.rados.rados.0.smithi165.stdout:265: oid 47 updating version 0 to 174 2024-09-10T16:08:43.865 INFO:tasks.rados.rados.0.smithi165.stdout:265: oid 47 updating version 174 to 175 2024-09-10T16:08:43.865 INFO:tasks.rados.rados.0.smithi165.stdout:265: oid 47 updating version 175 to 176 2024-09-10T16:08:43.865 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 176 (ObjNum 120 snap 34 seq_num 120) dirty exists 2024-09-10T16:08:43.865 INFO:tasks.rados.rados.0.smithi165.stdout:265: left oid 47 (ObjNum 120 snap 34 seq_num 120) 2024-09-10T16:08:43.868 INFO:tasks.rados.rados.0.smithi165.stdout:266: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:08:43.892 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1246642 2024-09-10T16:08:43.904 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:265: done (3 left) 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:266: done (2 left) 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:267: done (1 left) 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:269: done (0 left) 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:270: write oid 28 current snap is 35 2024-09-10T16:08:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:270: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-10T16:08:44.670 INFO:tasks.rados.rados.0.smithi165.stdout:270: writing smithi16534841-28 from 450454 to 1092480 tid 1 2024-09-10T16:08:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:270: writing smithi16534841-28 from 1505862 to 2050454 tid 2 2024-09-10T16:08:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:271: write oid 26 current snap is 35 2024-09-10T16:08:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:271: seq_num 123 ranges {467261=717275,1847991=219270} 2024-09-10T16:08:44.681 INFO:tasks.rados.rados.0.smithi165.stdout:271: writing smithi16534841-26 from 467261 to 1184536 tid 1 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:271: writing smithi16534841-26 from 1847991 to 2067261 tid 2 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:272: delete oid 40 current snap is 35 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: finishing write tid 1 to smithi16534841-28 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: finishing write tid 2 to smithi16534841-28 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: finishing write tid 3 to smithi16534841-28 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: finishing write tid 4 to smithi16534841-28 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: oid 28 updating version 0 to 55 2024-09-10T16:08:44.687 INFO:tasks.rados.rados.0.smithi165.stdout:270: oid 28 updating version 55 to 56 2024-09-10T16:08:44.688 INFO:tasks.rados.rados.0.smithi165.stdout:270: oid 28 version 56 is already newer than 54 2024-09-10T16:08:44.688 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 56 (ObjNum 122 snap 35 seq_num 122) dirty exists 2024-09-10T16:08:44.688 INFO:tasks.rados.rados.0.smithi165.stdout:270: left oid 28 (ObjNum 122 snap 35 seq_num 122) 2024-09-10T16:08:44.688 INFO:tasks.rados.rados.0.smithi165.stdout:271: finishing write tid 1 to smithi16534841-26 2024-09-10T16:08:44.689 INFO:tasks.rados.rados.0.smithi165.stdout:271: finishing write tid 2 to smithi16534841-26 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:270: done (2 left) 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:272: done (1 left) 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:273: delete oid 6 current snap is 35 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:271: finishing write tid 3 to smithi16534841-26 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:271: finishing write tid 4 to smithi16534841-26 2024-09-10T16:08:44.691 INFO:tasks.rados.rados.0.smithi165.stdout:271: oid 26 updating version 0 to 163 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:271: oid 26 version 163 is already newer than 162 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:271: oid 26 version 163 is already newer than 161 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 163 (ObjNum 123 snap 35 seq_num 123) dirty exists 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:271: left oid 26 (ObjNum 123 snap 35 seq_num 123) 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:271: done (1 left) 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:273: done (0 left) 2024-09-10T16:08:44.692 INFO:tasks.rados.rados.0.smithi165.stdout:274: snap_remove snap 29 2024-09-10T16:08:45.083 INFO:tasks.rados.rados.0.smithi165.stdout:274: done (0 left) 2024-09-10T16:08:45.083 INFO:tasks.rados.rados.0.smithi165.stdout:275: write (excl) oid 27 current snap is 35 2024-09-10T16:08:45.083 INFO:tasks.rados.rados.0.smithi165.stdout:275: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-10T16:08:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:275: writing smithi16534841-27 from 517682 to 1060704 tid 1 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:275: writing smithi16534841-27 from 1623437 to 2117682 tid 2 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:276: copy_from oid 43 from oid 25 current snap is 35 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:277: read oid 12 snap -1 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:277: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:278: write oid 28 current snap is 35 2024-09-10T16:08:45.096 INFO:tasks.rados.rados.0.smithi165.stdout:278: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-10T16:08:45.102 INFO:tasks.rados.rados.0.smithi165.stdout:278: writing smithi16534841-28 from 534489 to 1152760 tid 1 2024-09-10T16:08:45.109 INFO:tasks.rados.rados.0.smithi165.stdout:278: writing smithi16534841-28 from 1565566 to 2134489 tid 2 2024-09-10T16:08:45.109 INFO:tasks.rados.rados.0.smithi165.stdout:279: write (excl) oid 48 current snap is 35 2024-09-10T16:08:45.109 INFO:tasks.rados.rados.0.smithi165.stdout:279: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-10T16:08:45.115 INFO:tasks.rados.rados.0.smithi165.stdout:279: writing smithi16534841-48 from 551296 to 1244816 tid 1 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:279: writing smithi16534841-48 from 2024048 to 2151296 tid 2 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:280: read oid 13 snap 33 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:280: expect (ObjNum 112 snap 30 seq_num 112) 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:281: snap_create 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing write tid 1 to smithi16534841-27 2024-09-10T16:08:45.121 INFO:tasks.rados.rados.0.smithi165.stdout:277: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:08:45.124 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1458246 2024-09-10T16:08:45.137 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:45.164 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1312711 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing write tid 2 to smithi16534841-27 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing write tid 3 to smithi16534841-27 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: finishing write tid 4 to smithi16534841-27 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: oid 27 updating version 0 to 178 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: oid 27 updating version 178 to 181 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: oid 27 updating version 181 to 182 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 182 (ObjNum 126 snap 35 seq_num 126) dirty exists 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:275: left oid 27 (ObjNum 126 snap 35 seq_num 126) 2024-09-10T16:08:45.177 INFO:tasks.rados.rados.0.smithi165.stdout:278: finishing write tid 1 to smithi16534841-28 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:276: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: finishing write tid 2 to smithi16534841-28 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: finishing write tid 3 to smithi16534841-28 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: finishing write tid 4 to smithi16534841-28 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: oid 28 updating version 0 to 57 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: oid 28 updating version 57 to 59 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: oid 28 version 59 is already newer than 58 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 59 (ObjNum 127 snap 35 seq_num 127) dirty exists 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:278: left oid 28 (ObjNum 127 snap 35 seq_num 127) 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:276: finishing copy_from to smithi16534841-43 2024-09-10T16:08:45.178 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 46 (ObjNum 67 snap 8 seq_num 67) dirty exists 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 1 to smithi16534841-48 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 2 to smithi16534841-48 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 3 to smithi16534841-48 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: finishing write tid 4 to smithi16534841-48 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 48 updating version 0 to 170 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 48 version 170 is already newer than 169 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: oid 48 updating version 170 to 171 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 171 (ObjNum 128 snap 35 seq_num 128) dirty exists 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:279: left oid 48 (ObjNum 128 snap 35 seq_num 128) 2024-09-10T16:08:45.179 INFO:tasks.rados.rados.0.smithi165.stdout:280: expect (ObjNum 112 snap 30 seq_num 112) 2024-09-10T16:08:45.285 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:275: done (6 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:276: done (5 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:277: done (4 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:278: done (3 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:279: done (2 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:280: done (1 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:281: done (0 left) 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:282: read oid 14 snap -1 2024-09-10T16:08:46.117 INFO:tasks.rados.rados.0.smithi165.stdout:282: expect deleted 2024-09-10T16:08:46.118 INFO:tasks.rados.rados.0.smithi165.stdout:283: delete oid 27 current snap is 36 2024-09-10T16:08:46.121 INFO:tasks.rados.rados.0.smithi165.stdout:282: done (1 left) 2024-09-10T16:08:46.121 INFO:tasks.rados.rados.0.smithi165.stdout:283: done (0 left) 2024-09-10T16:08:46.122 INFO:tasks.rados.rados.0.smithi165.stdout:284: write (excl) oid 20 current snap is 36 2024-09-10T16:08:46.122 INFO:tasks.rados.rados.0.smithi165.stdout:284: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-10T16:08:46.127 INFO:tasks.rados.rados.0.smithi165.stdout:284: writing smithi16534841-20 from 584910 to 1145281 tid 1 2024-09-10T16:08:46.133 INFO:tasks.rados.rados.0.smithi165.stdout:284: writing smithi16534841-20 from 1857365 to 2184910 tid 2 2024-09-10T16:08:46.134 INFO:tasks.rados.rados.0.smithi165.stdout:285: write oid 30 current snap is 36 2024-09-10T16:08:46.134 INFO:tasks.rados.rados.0.smithi165.stdout:285: seq_num 131 ranges {601717=635620,1799494=402223} 2024-09-10T16:08:46.140 INFO:tasks.rados.rados.0.smithi165.stdout:285: writing smithi16534841-30 from 601717 to 1237337 tid 1 2024-09-10T16:08:46.146 INFO:tasks.rados.rados.0.smithi165.stdout:285: writing smithi16534841-30 from 1799494 to 2201717 tid 2 2024-09-10T16:08:46.146 INFO:tasks.rados.rados.0.smithi165.stdout:286: snap_create 2024-09-10T16:08:46.146 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing write tid 1 to smithi16534841-20 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing write tid 2 to smithi16534841-20 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing write tid 3 to smithi16534841-20 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: finishing write tid 4 to smithi16534841-20 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: oid 20 updating version 0 to 134 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: oid 20 updating version 134 to 135 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: oid 20 version 135 is already newer than 133 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 135 (ObjNum 130 snap 36 seq_num 130) dirty exists 2024-09-10T16:08:46.147 INFO:tasks.rados.rados.0.smithi165.stdout:284: left oid 20 (ObjNum 130 snap 36 seq_num 130) 2024-09-10T16:08:46.148 INFO:tasks.rados.rados.0.smithi165.stdout:285: finishing write tid 1 to smithi16534841-30 2024-09-10T16:08:46.149 INFO:tasks.rados.rados.0.smithi165.stdout:285: finishing write tid 2 to smithi16534841-30 2024-09-10T16:08:46.149 INFO:tasks.rados.rados.0.smithi165.stdout:285: finishing write tid 3 to smithi16534841-30 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:285: finishing write tid 4 to smithi16534841-30 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:285: oid 30 updating version 0 to 172 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:285: oid 30 updating version 172 to 174 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:285: oid 30 version 174 is already newer than 173 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 174 (ObjNum 131 snap 36 seq_num 131) dirty exists 2024-09-10T16:08:46.150 INFO:tasks.rados.rados.0.smithi165.stdout:285: left oid 30 (ObjNum 131 snap 36 seq_num 131) 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:284: done (2 left) 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:285: done (1 left) 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:286: done (0 left) 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:287: copy_from oid 29 from oid 39 current snap is 37 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:288: rollback oid 48 current snap is 37 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 33 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:289: write oid 4 current snap is 37 2024-09-10T16:08:47.120 INFO:tasks.rados.rados.0.smithi165.stdout:289: seq_num 132 ranges {618524=710869,1857976=360548} 2024-09-10T16:08:47.127 INFO:tasks.rados.rados.0.smithi165.stdout:289: writing smithi16534841-4 from 618524 to 1329393 tid 1 2024-09-10T16:08:47.133 INFO:tasks.rados.rados.0.smithi165.stdout:289: writing smithi16534841-4 from 1857976 to 2218524 tid 2 2024-09-10T16:08:47.133 INFO:tasks.rados.rados.0.smithi165.stdout:290: snap_create 2024-09-10T16:08:47.133 INFO:tasks.rados.rados.0.smithi165.stdout:288: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:08:47.133 INFO:tasks.rados.rados.0.smithi165.stdout:288: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:08:47.133 INFO:tasks.rados.rados.0.smithi165.stdout:288: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:08:47.134 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 178 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-10T16:08:47.137 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing write tid 1 to smithi16534841-4 2024-09-10T16:08:47.137 INFO:tasks.rados.rados.0.smithi165.stdout:287: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:08:47.138 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing write tid 2 to smithi16534841-4 2024-09-10T16:08:47.140 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing write tid 3 to smithi16534841-4 2024-09-10T16:08:47.140 INFO:tasks.rados.rados.0.smithi165.stdout:289: finishing write tid 4 to smithi16534841-4 2024-09-10T16:08:47.140 INFO:tasks.rados.rados.0.smithi165.stdout:289: oid 4 updating version 0 to 65 2024-09-10T16:08:47.140 INFO:tasks.rados.rados.0.smithi165.stdout:289: oid 4 updating version 65 to 66 2024-09-10T16:08:47.141 INFO:tasks.rados.rados.0.smithi165.stdout:289: oid 4 updating version 66 to 67 2024-09-10T16:08:47.141 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 67 (ObjNum 132 snap 37 seq_num 132) dirty exists 2024-09-10T16:08:47.141 INFO:tasks.rados.rados.0.smithi165.stdout:289: left oid 4 (ObjNum 132 snap 37 seq_num 132) 2024-09-10T16:08:47.141 INFO:tasks.rados.rados.0.smithi165.stdout:287: finishing copy_from to smithi16534841-29 2024-09-10T16:08:47.141 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 39 (ObjNum 88 snap 20 seq_num 88) dirty exists 2024-09-10T16:08:48.128 INFO:tasks.rados.rados.0.smithi165.stdout:287: done (3 left) 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:288: done (2 left) 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:289: done (1 left) 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:290: done (0 left) 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:291: read oid 42 snap -1 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:291: expect (ObjNum 60 snap 5 seq_num 60) 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:292: write oid 46 current snap is 38 2024-09-10T16:08:48.129 INFO:tasks.rados.rados.0.smithi165.stdout:292: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-10T16:08:48.136 INFO:tasks.rados.rados.0.smithi165.stdout:292: writing smithi16534841-46 from 635331 to 1421449 tid 1 2024-09-10T16:08:48.141 INFO:tasks.rados.rados.0.smithi165.stdout:292: writing smithi16534841-46 from 1916458 to 2235331 tid 2 2024-09-10T16:08:48.141 INFO:tasks.rados.rados.0.smithi165.stdout:293: snap_remove snap 10 2024-09-10T16:08:49.133 INFO:tasks.rados.rados.0.smithi165.stdout:293: done (2 left) 2024-09-10T16:08:49.133 INFO:tasks.rados.rados.0.smithi165.stdout:294: read oid 15 snap -1 2024-09-10T16:08:49.133 INFO:tasks.rados.rados.0.smithi165.stdout:294: expect (ObjNum 108 snap 30 seq_num 108) 2024-09-10T16:08:49.133 INFO:tasks.rados.rados.0.smithi165.stdout:295: write (excl) oid 23 current snap is 38 2024-09-10T16:08:49.133 INFO:tasks.rados.rados.0.smithi165.stdout:295: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-10T16:08:49.140 INFO:tasks.rados.rados.0.smithi165.stdout:295: writing smithi16534841-23 from 652138 to 1113505 tid 1 2024-09-10T16:08:49.145 INFO:tasks.rados.rados.0.smithi165.stdout:295: writing smithi16534841-23 from 1574940 to 2252138 tid 2 2024-09-10T16:08:49.146 INFO:tasks.rados.rados.0.smithi165.stdout:296: write (excl) oid 17 current snap is 38 2024-09-10T16:08:49.146 INFO:tasks.rados.rados.0.smithi165.stdout:296: seq_num 135 ranges {668945=536616,1633422=491363} 2024-09-10T16:08:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:296: writing smithi16534841-17 from 668945 to 1205561 tid 1 2024-09-10T16:08:49.157 INFO:tasks.rados.rados.0.smithi165.stdout:296: writing smithi16534841-17 from 1633422 to 2124785 tid 2 2024-09-10T16:08:49.157 INFO:tasks.rados.rados.0.smithi165.stdout:297: snap_create 2024-09-10T16:08:49.158 INFO:tasks.rados.rados.0.smithi165.stdout:291: expect (ObjNum 60 snap 5 seq_num 60) 2024-09-10T16:08:49.196 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:49.272 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3007741 2024-09-10T16:08:49.272 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 1 to smithi16534841-46 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 2 to smithi16534841-46 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 3 to smithi16534841-46 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: finishing write tid 4 to smithi16534841-46 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 46 updating version 0 to 73 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 46 updating version 73 to 74 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: oid 46 version 74 is already newer than 72 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 74 (ObjNum 133 snap 38 seq_num 133) dirty exists 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:292: left oid 46 (ObjNum 133 snap 38 seq_num 133) 2024-09-10T16:08:49.273 INFO:tasks.rados.rados.0.smithi165.stdout:294: expect (ObjNum 108 snap 30 seq_num 108) 2024-09-10T16:08:49.328 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: finishing write tid 1 to smithi16534841-23 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: finishing write tid 2 to smithi16534841-23 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: finishing write tid 3 to smithi16534841-23 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: finishing write tid 4 to smithi16534841-23 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: oid 23 updating version 0 to 144 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: oid 23 updating version 144 to 145 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: oid 23 version 145 is already newer than 143 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 145 (ObjNum 134 snap 38 seq_num 134) dirty exists 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:295: left oid 23 (ObjNum 134 snap 38 seq_num 134) 2024-09-10T16:08:49.427 INFO:tasks.rados.rados.0.smithi165.stdout:296: finishing write tid 1 to smithi16534841-17 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: finishing write tid 2 to smithi16534841-17 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: finishing write tid 3 to smithi16534841-17 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: finishing write tid 4 to smithi16534841-17 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: oid 17 updating version 0 to 44 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: oid 17 version 44 is already newer than 42 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: oid 17 version 44 is already newer than 43 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 44 (ObjNum 135 snap 38 seq_num 135) dirty exists 2024-09-10T16:08:49.428 INFO:tasks.rados.rados.0.smithi165.stdout:296: left oid 17 (ObjNum 135 snap 38 seq_num 135) 2024-09-10T16:08:50.086 INFO:tasks.rados.rados.0.smithi165.stdout:291: done (5 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:292: done (4 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:294: done (3 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:295: done (2 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:296: done (1 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:297: done (0 left) 2024-09-10T16:08:50.087 INFO:tasks.rados.rados.0.smithi165.stdout:298: snap_remove snap 25 2024-09-10T16:08:51.095 INFO:tasks.rados.rados.0.smithi165.stdout:298: done (0 left) 2024-09-10T16:08:51.096 INFO:tasks.rados.rados.0.smithi165.stdout:299: write oid 33 current snap is 39 2024-09-10T16:08:51.096 INFO:tasks.rados.rados.0.smithi165.stdout:299: seq_num 136 ranges {685752=611865,1975551=310201} 2024-09-10T16:08:51.102 INFO:tasks.rados.rados.0.smithi165.stdout:299: writing smithi16534841-33 from 685752 to 1297617 tid 1 2024-09-10T16:08:51.108 INFO:tasks.rados.rados.0.smithi165.stdout:299: writing smithi16534841-33 from 1975551 to 2285752 tid 2 2024-09-10T16:08:51.109 INFO:tasks.rados.rados.0.smithi165.stdout:300: rollback oid 39 current snap is 39 2024-09-10T16:08:51.109 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 27 2024-09-10T16:08:51.109 INFO:tasks.rados.rados.0.smithi165.stdout:301: snap_create 2024-09-10T16:08:51.112 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing write tid 1 to smithi16534841-33 2024-09-10T16:08:51.116 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing write tid 2 to smithi16534841-33 2024-09-10T16:08:51.116 INFO:tasks.rados.rados.0.smithi165.stdout:300: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:08:51.118 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing write tid 3 to smithi16534841-33 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:299: finishing write tid 4 to smithi16534841-33 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:299: oid 33 updating version 0 to 147 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:299: oid 33 version 147 is already newer than 146 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:299: oid 33 version 147 is already newer than 145 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 147 (ObjNum 136 snap 39 seq_num 136) dirty exists 2024-09-10T16:08:51.119 INFO:tasks.rados.rados.0.smithi165.stdout:299: left oid 33 (ObjNum 136 snap 39 seq_num 136) 2024-09-10T16:08:51.120 INFO:tasks.rados.rados.0.smithi165.stdout:300: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:08:51.120 INFO:tasks.rados.rados.0.smithi165.stdout:300: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:08:51.120 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 153 (ObjNum 88 snap 20 seq_num 88) dirty exists 2024-09-10T16:08:52.102 INFO:tasks.rados.rados.0.smithi165.stdout:299: done (2 left) 2024-09-10T16:08:52.102 INFO:tasks.rados.rados.0.smithi165.stdout:300: done (1 left) 2024-09-10T16:08:52.103 INFO:tasks.rados.rados.0.smithi165.stdout:301: done (0 left) 2024-09-10T16:08:52.103 INFO:tasks.rados.rados.0.smithi165.stdout:302: delete oid 41 current snap is 40 2024-09-10T16:08:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:302: done (0 left) 2024-09-10T16:08:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:303: read oid 19 snap -1 2024-09-10T16:08:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:303: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:304: read oid 6 snap -1 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:304: expect deleted 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:305: copy_from oid 30 from oid 49 current snap is 40 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:306: read oid 46 snap -1 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:306: expect (ObjNum 133 snap 38 seq_num 133) 2024-09-10T16:08:52.109 INFO:tasks.rados.rados.0.smithi165.stdout:307: snap_create 2024-09-10T16:08:52.111 INFO:tasks.rados.rados.0.smithi165.stdout:303: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:08:52.179 INFO:tasks.rados.rados.0.smithi165.stdout:306: expect (ObjNum 133 snap 38 seq_num 133) 2024-09-10T16:08:52.185 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1325179 2024-09-10T16:08:52.243 INFO:tasks.rados.rados.0.smithi165.stdout:305: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:08:52.243 INFO:tasks.rados.rados.0.smithi165.stdout:305: finishing copy_from to smithi16534841-30 2024-09-10T16:08:52.243 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 32 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:08:53.105 INFO:tasks.rados.rados.0.smithi165.stdout:303: done (4 left) 2024-09-10T16:08:53.105 INFO:tasks.rados.rados.0.smithi165.stdout:304: done (3 left) 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:305: done (2 left) 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:306: done (1 left) 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:307: done (0 left) 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:308: read oid 42 snap 6 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:308: expect (ObjNum 60 snap 5 seq_num 60) 2024-09-10T16:08:53.106 INFO:tasks.rados.rados.0.smithi165.stdout:309: snap_remove snap 36 2024-09-10T16:08:54.474 INFO:tasks.rados.rados.0.smithi165.stdout:309: done (1 left) 2024-09-10T16:08:54.474 INFO:tasks.rados.rados.0.smithi165.stdout:310: rollback oid 3 current snap is 41 2024-09-10T16:08:54.474 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 21 2024-09-10T16:08:54.474 INFO:tasks.rados.rados.0.smithi165.stdout:308: expect (ObjNum 60 snap 5 seq_num 60) 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:310: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:310: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 155 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:308: done (1 left) 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:310: done (0 left) 2024-09-10T16:08:54.616 INFO:tasks.rados.rados.0.smithi165.stdout:311: delete oid 40 current snap is 41 2024-09-10T16:08:54.617 INFO:tasks.rados.rados.0.smithi165.stdout:311: done (0 left) 2024-09-10T16:08:54.617 INFO:tasks.rados.rados.0.smithi165.stdout:312: snap_create 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:312: done (0 left) 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:313: copy_from oid 42 from oid 22 current snap is 42 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:314: rollback oid 46 current snap is 42 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 21 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:315: write (excl) oid 33 current snap is 42 2024-09-10T16:08:55.089 INFO:tasks.rados.rados.0.smithi165.stdout:315: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-10T16:08:55.096 INFO:tasks.rados.rados.0.smithi165.stdout:315: writing smithi16534841-33 from 736173 to 1290138 tid 1 2024-09-10T16:08:55.102 INFO:tasks.rados.rados.0.smithi165.stdout:315: writing smithi16534841-33 from 1867350 to 2336173 tid 2 2024-09-10T16:08:55.102 INFO:tasks.rados.rados.0.smithi165.stdout:316: write oid 31 current snap is 42 2024-09-10T16:08:55.102 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:08:55.102 INFO:tasks.rados.rados.0.smithi165.stdout:316: seq_num 140 ranges {752980=629214,1809479=543501} 2024-09-10T16:08:55.110 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16534841-31 from 752980 to 1382194 tid 1 2024-09-10T16:08:55.116 INFO:tasks.rados.rados.0.smithi165.stdout:316: writing smithi16534841-31 from 1809479 to 2352980 tid 2 2024-09-10T16:08:55.116 INFO:tasks.rados.rados.0.smithi165.stdout:317: rollback oid 11 current snap is 42 2024-09-10T16:08:55.116 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 37 2024-09-10T16:08:55.116 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:314: finishing rollback tid 2 to smithi16534841-46 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 86 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: finishing write tid 1 to smithi16534841-33 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: finishing write tid 2 to smithi16534841-33 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: finishing write tid 3 to smithi16534841-33 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: finishing write tid 4 to smithi16534841-33 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: oid 33 updating version 0 to 160 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: oid 33 updating version 160 to 161 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: oid 33 version 161 is already newer than 159 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 161 (ObjNum 139 snap 42 seq_num 139) dirty exists 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:315: left oid 33 (ObjNum 139 snap 42 seq_num 139) 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:313: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:08:55.117 INFO:tasks.rados.rados.0.smithi165.stdout:314: done (4 left) 2024-09-10T16:08:55.118 INFO:tasks.rados.rados.0.smithi165.stdout:315: done (3 left) 2024-09-10T16:08:55.118 INFO:tasks.rados.rados.0.smithi165.stdout:318: copy_from oid 43 from oid 4 current snap is 42 2024-09-10T16:08:55.118 INFO:tasks.rados.rados.0.smithi165.stdout:319: snap_remove snap 23 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:319: done (4 left) 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:320: read oid 32 snap -1 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:320: expect (ObjNum 104 snap 28 seq_num 104) 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:321: write oid 33 current snap is 42 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 1 to smithi16534841-31 2024-09-10T16:08:56.099 INFO:tasks.rados.rados.0.smithi165.stdout:321: seq_num 141 ranges {769787=704463,2267961=101826} 2024-09-10T16:08:56.107 INFO:tasks.rados.rados.0.smithi165.stdout:321: writing smithi16534841-33 from 769787 to 1474250 tid 1 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:321: writing smithi16534841-33 from 2267961 to 2369787 tid 2 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:322: copy_from oid 23 from oid 20 current snap is 42 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:313: finishing copy_from to smithi16534841-42 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 6 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:317: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 31 (ObjNum 33 snap 0 seq_num 1127806872) dirty dne 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 2 to smithi16534841-31 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 3 to smithi16534841-31 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:316: finishing write tid 4 to smithi16534841-31 2024-09-10T16:08:56.115 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 31 updating version 0 to 207 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 31 version 207 is already newer than 206 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:316: oid 31 updating version 207 to 208 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 208 (ObjNum 140 snap 42 seq_num 140) dirty exists 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:316: left oid 31 (ObjNum 140 snap 42 seq_num 140) 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:318: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:318: finishing copy_from to smithi16534841-43 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 67 (ObjNum 132 snap 37 seq_num 132) dirty exists 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:313: done (6 left) 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:316: done (5 left) 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:317: done (4 left) 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:318: done (3 left) 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:323: write (excl) oid 14 current snap is 42 2024-09-10T16:08:56.116 INFO:tasks.rados.rados.0.smithi165.stdout:323: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-10T16:08:56.123 INFO:tasks.rados.rados.0.smithi165.stdout:323: writing smithi16534841-14 from 786594 to 1566306 tid 1 2024-09-10T16:08:56.128 INFO:tasks.rados.rados.0.smithi165.stdout:323: writing smithi16534841-14 from 2326443 to 2386594 tid 2 2024-09-10T16:08:56.128 INFO:tasks.rados.rados.0.smithi165.stdout:324: read oid 15 snap -1 2024-09-10T16:08:56.129 INFO:tasks.rados.rados.0.smithi165.stdout:324: expect (ObjNum 108 snap 30 seq_num 108) 2024-09-10T16:08:56.129 INFO:tasks.rados.rados.0.smithi165.stdout:325: snap_remove snap 40 2024-09-10T16:08:57.111 INFO:tasks.rados.rados.0.smithi165.stdout:325: done (5 left) 2024-09-10T16:08:57.111 INFO:tasks.rados.rados.0.smithi165.stdout:326: rollback oid 24 current snap is 42 2024-09-10T16:08:57.111 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 6 2024-09-10T16:08:57.111 INFO:tasks.rados.rados.0.smithi165.stdout:320: expect (ObjNum 104 snap 28 seq_num 104) 2024-09-10T16:08:57.111 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:57.182 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:57.258 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 789108 2024-09-10T16:08:57.316 INFO:tasks.rados.rados.0.smithi165.stdout:321: finishing write tid 1 to smithi16534841-33 2024-09-10T16:08:57.316 INFO:tasks.rados.rados.0.smithi165.stdout:321: finishing write tid 2 to smithi16534841-33 2024-09-10T16:08:57.316 INFO:tasks.rados.rados.0.smithi165.stdout:321: finishing write tid 3 to smithi16534841-33 2024-09-10T16:08:57.316 INFO:tasks.rados.rados.0.smithi165.stdout:321: finishing write tid 4 to smithi16534841-33 2024-09-10T16:08:57.316 INFO:tasks.rados.rados.0.smithi165.stdout:321: oid 33 updating version 0 to 162 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:321: oid 33 updating version 162 to 163 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:321: oid 33 updating version 163 to 164 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 164 (ObjNum 141 snap 42 seq_num 141) dirty exists 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:321: left oid 33 (ObjNum 141 snap 42 seq_num 141) 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:323: finishing write tid 1 to smithi16534841-14 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:322: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:323: finishing write tid 2 to smithi16534841-14 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:320: done (5 left) 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:321: done (4 left) 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:327: write (excl) oid 13 current snap is 42 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:323: finishing write tid 3 to smithi16534841-14 2024-09-10T16:08:57.317 INFO:tasks.rados.rados.0.smithi165.stdout:327: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-10T16:08:57.320 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16534841-13 from 403401 to 858362 tid 1 2024-09-10T16:08:57.327 INFO:tasks.rados.rados.0.smithi165.stdout:327: writing smithi16534841-13 from 1584925 to 2090964 tid 2 2024-09-10T16:08:57.327 INFO:tasks.rados.rados.0.smithi165.stdout:328: write oid 3 current snap is 42 2024-09-10T16:08:57.327 INFO:tasks.rados.rados.0.smithi165.stdout:323: finishing write tid 4 to smithi16534841-14 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:323: oid 14 updating version 0 to 74 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:323: oid 14 updating version 74 to 76 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:323: oid 14 version 76 is already newer than 75 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 76 (ObjNum 142 snap 42 seq_num 142) dirty exists 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:323: left oid 14 (ObjNum 142 snap 42 seq_num 142) 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:322: finishing copy_from to smithi16534841-23 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 135 (ObjNum 130 snap 36 seq_num 130) dirty exists 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:324: expect (ObjNum 108 snap 30 seq_num 108) 2024-09-10T16:08:57.328 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:57.396 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1512817 2024-09-10T16:08:57.434 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:57.490 INFO:tasks.rados.rados.0.smithi165.stdout:326: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:08:57.490 INFO:tasks.rados.rados.0.smithi165.stdout:326: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:08:57.490 INFO:tasks.rados.rados.0.smithi165.stdout:326: finishing rollback tid 2 to smithi16534841-24 2024-09-10T16:08:57.490 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 218 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-10T16:08:57.491 INFO:tasks.rados.rados.0.smithi165.stdout:328: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-10T16:08:57.495 INFO:tasks.rados.rados.0.smithi165.stdout:328: writing smithi16534841-3 from 420208 to 950418 tid 1 2024-09-10T16:08:57.503 INFO:tasks.rados.rados.0.smithi165.stdout:328: writing smithi16534841-3 from 1527054 to 2293104 tid 2 2024-09-10T16:08:57.503 INFO:tasks.rados.rados.0.smithi165.stdout:322: done (5 left) 2024-09-10T16:08:57.503 INFO:tasks.rados.rados.0.smithi165.stdout:323: done (4 left) 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:324: done (3 left) 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:326: done (2 left) 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:329: delete oid 50 current snap is 42 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 1 to smithi16534841-13 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 2 to smithi16534841-13 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 3 to smithi16534841-13 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: finishing write tid 4 to smithi16534841-13 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 13 updating version 0 to 52 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 13 updating version 52 to 53 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: oid 13 updating version 53 to 54 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 54 (ObjNum 143 snap 42 seq_num 143) dirty exists 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:327: left oid 13 (ObjNum 143 snap 42 seq_num 143) 2024-09-10T16:08:57.504 INFO:tasks.rados.rados.0.smithi165.stdout:328: finishing write tid 1 to smithi16534841-3 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:327: done (2 left) 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:329: done (1 left) 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:330: rollback oid 45 current snap is 42 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 37 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:331: write oid 15 current snap is 42 2024-09-10T16:08:57.508 INFO:tasks.rados.rados.0.smithi165.stdout:331: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-10T16:08:57.514 INFO:tasks.rados.rados.0.smithi165.stdout:331: writing smithi16534841-15 from 453822 to 1250883 tid 1 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:331: writing smithi16534841-15 from 1760371 to 2413737 tid 2 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:332: copy_from oid 20 from oid 46 current snap is 42 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: finishing write tid 2 to smithi16534841-3 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: finishing write tid 3 to smithi16534841-3 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: finishing write tid 4 to smithi16534841-3 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: oid 3 updating version 0 to 174 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: oid 3 updating version 174 to 177 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: oid 3 updating version 177 to 178 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 178 (ObjNum 144 snap 42 seq_num 144) dirty exists 2024-09-10T16:08:57.522 INFO:tasks.rados.rados.0.smithi165.stdout:328: left oid 3 (ObjNum 144 snap 42 seq_num 144) 2024-09-10T16:08:57.523 INFO:tasks.rados.rados.0.smithi165.stdout:328: done (3 left) 2024-09-10T16:08:57.523 INFO:tasks.rados.rados.0.smithi165.stdout:333: read oid 13 snap -1 2024-09-10T16:08:57.523 INFO:tasks.rados.rados.0.smithi165.stdout:333: expect (ObjNum 143 snap 42 seq_num 143) 2024-09-10T16:08:57.523 INFO:tasks.rados.rados.0.smithi165.stdout:334: write (excl) oid 1 current snap is 42 2024-09-10T16:08:57.523 INFO:tasks.rados.rados.0.smithi165.stdout:334: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-10T16:08:57.526 INFO:tasks.rados.rados.0.smithi165.stdout:334: writing smithi16534841-1 from 470629 to 942939 tid 1 2024-09-10T16:08:57.533 INFO:tasks.rados.rados.0.smithi165.stdout:334: writing smithi16534841-1 from 1418853 to 1932230 tid 2 2024-09-10T16:08:57.533 INFO:tasks.rados.rados.0.smithi165.stdout:335: write (excl) oid 28 current snap is 42 2024-09-10T16:08:57.533 INFO:tasks.rados.rados.0.smithi165.stdout:331: finishing write tid 1 to smithi16534841-15 2024-09-10T16:08:57.533 INFO:tasks.rados.rados.0.smithi165.stdout:335: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-10T16:08:57.538 INFO:tasks.rados.rados.0.smithi165.stdout:335: writing smithi16534841-28 from 487436 to 1034995 tid 1 2024-09-10T16:08:57.546 INFO:tasks.rados.rados.0.smithi165.stdout:335: writing smithi16534841-28 from 1760982 to 2418017 tid 2 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:336: copy_from oid 11 from oid 5 current snap is 42 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: finishing write tid 2 to smithi16534841-15 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:330: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: finishing write tid 3 to smithi16534841-15 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: finishing write tid 4 to smithi16534841-15 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: oid 15 updating version 0 to 84 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: oid 15 updating version 84 to 85 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: oid 15 version 85 is already newer than 83 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 85 (ObjNum 146 snap 42 seq_num 146) dirty exists 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:331: left oid 15 (ObjNum 146 snap 42 seq_num 146) 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:330: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:08:57.547 INFO:tasks.rados.rados.0.smithi165.stdout:330: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:08:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 77 (ObjNum 81 snap 18 seq_num 81) dirty exists 2024-09-10T16:08:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:334: finishing write tid 1 to smithi16534841-1 2024-09-10T16:08:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:330: done (6 left) 2024-09-10T16:08:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:331: done (5 left) 2024-09-10T16:08:57.548 INFO:tasks.rados.rados.0.smithi165.stdout:337: snap_remove snap 32 2024-09-10T16:08:58.111 INFO:tasks.rados.rados.0.smithi165.stdout:337: done (5 left) 2024-09-10T16:08:58.111 INFO:tasks.rados.rados.0.smithi165.stdout:338: snap_create 2024-09-10T16:08:58.111 INFO:tasks.rados.rados.0.smithi165.stdout:333: expect (ObjNum 143 snap 42 seq_num 143) 2024-09-10T16:08:58.133 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2041552 2024-09-10T16:08:58.186 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:08:58.234 INFO:tasks.rados.rados.0.smithi165.stdout:332: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:08:58.234 INFO:tasks.rados.rados.0.smithi165.stdout:332: finishing copy_from to smithi16534841-20 2024-09-10T16:08:58.234 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 86 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:08:58.234 INFO:tasks.rados.rados.0.smithi165.stdout:334: finishing write tid 2 to smithi16534841-1 2024-09-10T16:08:58.234 INFO:tasks.rados.rados.0.smithi165.stdout:334: finishing write tid 3 to smithi16534841-1 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:334: finishing write tid 4 to smithi16534841-1 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:334: oid 1 updating version 0 to 175 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:334: oid 1 updating version 175 to 177 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:334: oid 1 version 177 is already newer than 176 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 177 (ObjNum 147 snap 42 seq_num 147) dirty exists 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:334: left oid 1 (ObjNum 147 snap 42 seq_num 147) 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: finishing write tid 1 to smithi16534841-28 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: finishing write tid 2 to smithi16534841-28 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: finishing write tid 3 to smithi16534841-28 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: finishing write tid 4 to smithi16534841-28 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: oid 28 updating version 0 to 61 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: oid 28 version 61 is already newer than 60 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: oid 28 updating version 61 to 62 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 62 (ObjNum 148 snap 42 seq_num 148) dirty exists 2024-09-10T16:08:58.235 INFO:tasks.rados.rados.0.smithi165.stdout:335: left oid 28 (ObjNum 148 snap 42 seq_num 148) 2024-09-10T16:08:58.236 INFO:tasks.rados.rados.0.smithi165.stdout:336: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:08:58.236 INFO:tasks.rados.rados.0.smithi165.stdout:336: finishing copy_from to smithi16534841-11 2024-09-10T16:08:58.236 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 42 (ObjNum 79 snap 18 seq_num 79) dirty exists 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:332: done (5 left) 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:333: done (4 left) 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:334: done (3 left) 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:335: done (2 left) 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:336: done (1 left) 2024-09-10T16:08:59.124 INFO:tasks.rados.rados.0.smithi165.stdout:338: done (0 left) 2024-09-10T16:08:59.125 INFO:tasks.rados.rados.0.smithi165.stdout:339: write oid 34 current snap is 43 2024-09-10T16:08:59.125 INFO:tasks.rados.rados.0.smithi165.stdout:339: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-10T16:08:59.130 INFO:tasks.rados.rados.0.smithi165.stdout:339: writing smithi16534841-34 from 504243 to 1127051 tid 1 2024-09-10T16:08:59.137 INFO:tasks.rados.rados.0.smithi165.stdout:339: writing smithi16534841-34 from 1819464 to 2336510 tid 2 2024-09-10T16:08:59.137 INFO:tasks.rados.rados.0.smithi165.stdout:340: copy_from oid 5 from oid 48 current snap is 43 2024-09-10T16:08:59.137 INFO:tasks.rados.rados.0.smithi165.stdout:339: finishing write tid 1 to smithi16534841-34 2024-09-10T16:08:59.137 INFO:tasks.rados.rados.0.smithi165.stdout:341: snap_remove snap 26 2024-09-10T16:09:00.088 INFO:tasks.rados.rados.0.smithi165.stdout:341: done (2 left) 2024-09-10T16:09:00.088 INFO:tasks.rados.rados.0.smithi165.stdout:342: read oid 25 snap 37 2024-09-10T16:09:00.088 INFO:tasks.rados.rados.0.smithi165.stdout:342: expect (ObjNum 67 snap 8 seq_num 67) 2024-09-10T16:09:00.088 INFO:tasks.rados.rados.0.smithi165.stdout:343: delete oid 1 current snap is 43 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: finishing write tid 2 to smithi16534841-34 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: finishing write tid 3 to smithi16534841-34 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: finishing write tid 4 to smithi16534841-34 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: oid 34 updating version 0 to 25 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: oid 34 version 25 is already newer than 24 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: oid 34 updating version 25 to 26 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 26 (ObjNum 149 snap 43 seq_num 149) dirty exists 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:339: left oid 34 (ObjNum 149 snap 43 seq_num 149) 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:340: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:340: finishing copy_from to smithi16534841-5 2024-09-10T16:09:00.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 178 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-10T16:09:00.093 INFO:tasks.rados.rados.0.smithi165.stdout:342: expect (ObjNum 67 snap 8 seq_num 67) 2024-09-10T16:09:00.218 INFO:tasks.rados.rados.0.smithi165.stdout:339: done (3 left) 2024-09-10T16:09:00.218 INFO:tasks.rados.rados.0.smithi165.stdout:340: done (2 left) 2024-09-10T16:09:00.218 INFO:tasks.rados.rados.0.smithi165.stdout:342: done (1 left) 2024-09-10T16:09:00.218 INFO:tasks.rados.rados.0.smithi165.stdout:343: done (0 left) 2024-09-10T16:09:00.218 INFO:tasks.rados.rados.0.smithi165.stdout:344: delete oid 32 current snap is 43 2024-09-10T16:09:00.221 INFO:tasks.rados.rados.0.smithi165.stdout:344: done (0 left) 2024-09-10T16:09:00.221 INFO:tasks.rados.rados.0.smithi165.stdout:345: write oid 45 current snap is 43 2024-09-10T16:09:00.221 INFO:tasks.rados.rados.0.smithi165.stdout:345: seq_num 152 ranges {554664=448555,1478557=664373} 2024-09-10T16:09:00.226 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16534841-45 from 554664 to 1003219 tid 1 2024-09-10T16:09:00.232 INFO:tasks.rados.rados.0.smithi165.stdout:345: writing smithi16534841-45 from 1478557 to 2142930 tid 2 2024-09-10T16:09:00.233 INFO:tasks.rados.rados.0.smithi165.stdout:346: snap_remove snap 6 2024-09-10T16:09:01.129 INFO:tasks.rados.rados.0.smithi165.stdout:346: done (1 left) 2024-09-10T16:09:01.129 INFO:tasks.rados.rados.0.smithi165.stdout:347: snap_remove snap 34 2024-09-10T16:09:01.862 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:09:01.862 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:09:01.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:09:01.866 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:09:01.866 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:09:01.866 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:01.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.973+0000 7fba1b10e640 10 monclient: get_monmap_and_config 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.973+0000 7fba1b10e640 10 monclient: build_initial_monmap 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.973+0000 7fba1b10e640 10 monclient: monmap: 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:01.975188+0000 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:01.975188+0000 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:01.976 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:01.977 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:01.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 Processor -- start 2024-09-10T16:09:01.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 -- start start 2024-09-10T16:09:01.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient: init 2024-09-10T16:09:01.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-c con 0x7fba1406cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba140a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba140a4ab0 0x7fba141383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-a con 0x7fba140a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba1409ce50 con 0x7fba140a46a0 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fba1406db60 con 0x7fba140a4ab0 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fba1406cf50 con 0x7fba1406cb70 2024-09-10T16:09:01.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba1b10e640 10 monclient(hunting): authenticate will time out at 920.462726s 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba140a4ab0 0x7fba141383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fba140a4ab0 0x7fba141383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39286/0 (socket says 172.21.15.164:39286) 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 1 -- 172.21.15.164:0/84316157 learned_addr learned my addr 172.21.15.164:0/84316157 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): get_auth_request con 0x7fba140a4ab0 auth_method 0 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 1 --2- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 10 monclient(hunting): get_auth_request con 0x7fba1406cb70 auth_method 0 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:01.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 1 -- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 msgr2=0x7fba14134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 1 -- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 msgr2=0x7fba14134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 1 --2- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.974+0000 7fba18eaa640 1 --2- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 10 monclient(hunting): handle_auth_done global_id 24192 payload 274 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 1 -- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 msgr2=0x7fba14134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 1 --2- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 1 -- 172.21.15.164:0/84316157 >> v1:172.21.15.165:6789/0 conn(0x7fba140a46a0 legacy=0x7fba14137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 10 monclient: _finish_hunting 0 2024-09-10T16:09:01.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 1 monclient: found mon.noname-a 2024-09-10T16:09:01.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:01.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:01.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.975+0000 7fba196ab640 1 -- 172.21.15.164:0/84316157 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba1409cb70 con 0x7fba140a4ab0 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.977+0000 7fba196ab640 1 -- 172.21.15.164:0/84316157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 msgr2=0x7fba141383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.977+0000 7fba196ab640 1 --2- 172.21.15.164:0/84316157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba141383b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009e80 tx=0x7fba0c02a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb86c65d2c79fd13 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 1 -- 172.21.15.164:0/84316157 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c036070 con 0x7fba140a4ab0 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: dump: 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:01.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 1 -- 172.21.15.164:0/84316157 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c007800 con 0x7fba140a4ab0 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 5 monclient: authenticate success, global_id 24192 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 1 -- 172.21.15.164:0/84316157 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c007bd0 con 0x7fba140a4ab0 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 4 set_mon_vals no callback set 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba137fe640 10 monclient: dump: 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:01.991 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 10 monclient: shutdown 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 -- 172.21.15.164:0/84316157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 msgr2=0x7fba141383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:01.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 --2- 172.21.15.164:0/84316157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba141383b0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba0c009e80 tx=0x7fba0c02a5d0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 -- 172.21.15.164:0/84316157 shutdown_connections 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 --2- 172.21.15.164:0/84316157 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba141383b0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 --2- 172.21.15.164:0/84316157 >> v2:172.21.15.165:3300/0 conn(0x7fba1406cb70 0x7fba14134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 -- 172.21.15.164:0/84316157 >> 172.21.15.164:0/84316157 conn(0x7fba140aa900 msgr2=0x7fba140aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 -- 172.21.15.164:0/84316157 shutdown_connections 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.978+0000 7fba1b10e640 1 -- 172.21.15.164:0/84316157 wait complete. 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.979+0000 7fba1b10e640 10 monclient: build_initial_monmap 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.979+0000 7fba1b10e640 10 monclient: monmap: 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:01.993 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.979+0000 7fba1b10e640 1 Processor -- start 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.979+0000 7fba1b10e640 1 -- start start 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.979+0000 7fba1b10e640 10 monclient: init 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 0x7fba1413f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-c con 0x7fba1406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:01.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-b con 0x7fba140a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): picked mon.noname-a con 0x7fba140a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fba14138a70 con 0x7fba140a4ab0 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fba141388f0 con 0x7fba140a46a0 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fba14140040 con 0x7fba1406cb70 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba1b10e640 10 monclient(hunting): authenticate will time out at 920.468248s 2024-09-10T16:09:01.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 0x7fba1413f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39288/0 (socket says 172.21.15.164:39288) 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 -- 172.21.15.164:0/2476827793 learned_addr learned my addr 172.21.15.164:0/2476827793 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): get_auth_request con 0x7fba140a4ab0 auth_method 0 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): get_auth_request con 0x7fba1406cb70 auth_method 0 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:01.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 10 monclient(hunting): get_auth_request con 0x7fba140a46a0 auth_method 0 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:01.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient(hunting): handle_auth_done global_id 24195 payload 824 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 msgr2=0x7fba1413f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 msgr2=0x7fba140a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 0x7fba1413f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 msgr2=0x7fba140a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 msgr2=0x7fba140a27c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba13fff640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient: _finish_hunting 0 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 monclient: found mon.noname-a 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:01.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba196ab640 1 -- 172.21.15.164:0/2476827793 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba141ce0f0 con 0x7fba140a4ab0 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.980+0000 7fba18eaa640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 0x7fba1413f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.982+0000 7fba196ab640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 msgr2=0x7fba140a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.982+0000 7fba196ab640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba0c034040 tx=0x7fba0c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c036050 con 0x7fba140a4ab0 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: dump: 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:01.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: _finish_auth 0 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: _check_auth_tickets 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba0c042070 con 0x7fba140a4ab0 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 5 monclient: authenticate success, global_id 24195 2024-09-10T16:09:02.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 10 monclient: _renew_subs 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba1413ad20 con 0x7fba140a4ab0 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba0c031d90 con 0x7fba140a4ab0 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba117fa640 10 monclient: dump: 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:02.001 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba12ffd640 4 set_mon_vals no callback set 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 10 monclient: _renew_subs 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.983+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba14133f70 con 0x7fba140a4ab0 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fba0c047460 con 0x7fba140a4ab0 2024-09-10T16:09:02.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba1b10e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba1b10e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba1b10e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba14134f70 con 0x7fba140a4ab0 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba117fa640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 0x7fb9f80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 4847+0+0 (secure 0 0 0) 0x7fba0c073c70 con 0x7fba140a4ab0 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba18eaa640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 0x7fb9f80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.984+0000 7fba18eaa640 10 monclient: get_auth_request con 0x7fb9f803e210 auth_method 0 2024-09-10T16:09:02.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.987+0000 7fba18eaa640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 0x7fb9f80406d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba08006fd0 tx=0x7fba08008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:02.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.988+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fba1409ce50 con 0x7fba140a4ab0 2024-09-10T16:09:02.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.988+0000 7fba117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:02.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:01.988+0000 7fba117fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:02.133 INFO:tasks.rados.rados.0.smithi165.stdout:347: done (1 left) 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:348: snap_create 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 1 to smithi16534841-45 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 2 to smithi16534841-45 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 3 to smithi16534841-45 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: finishing write tid 4 to smithi16534841-45 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 45 updating version 0 to 87 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 45 updating version 87 to 88 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:345: oid 45 updating version 88 to 89 2024-09-10T16:09:02.134 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 89 (ObjNum 152 snap 43 seq_num 152) dirty exists 2024-09-10T16:09:02.135 INFO:tasks.rados.rados.0.smithi165.stdout:345: left oid 45 (ObjNum 152 snap 43 seq_num 152) 2024-09-10T16:09:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.236+0000 7fba1b10e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba1b10e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba1b10e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fba1409ed20 con 0x7fba140a4ab0 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba117fa640 1 -- 172.21.15.164:0/2476827793 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fba0c072630 con 0x7fba140a4ab0 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.237+0000 7fba117fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:02.240 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:02.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 msgr2=0x7fb9f80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:02.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 0x7fb9f80406d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fba08006fd0 tx=0x7fba08008040 comp rx=0 tx=0).stop 2024-09-10T16:09:02.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 10 monclient: shutdown 2024-09-10T16:09:02.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 msgr2=0x7fba140a2d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba0c034040 tx=0x7fba0c004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 shutdown_connections 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb9f803e210 0x7fb9f80406d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fba140a4ab0 0x7fba140a2d00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fba140a46a0 0x7fba140a27c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 --2- 172.21.15.164:0/2476827793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fba1406cb70 0x7fba1413f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 >> 172.21.15.164:0/2476827793 conn(0x7fba140aa900 msgr2=0x7fba14137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 shutdown_connections 2024-09-10T16:09:02.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:02.239+0000 7fba1b10e640 1 -- 172.21.15.164:0/2476827793 wait complete. 2024-09-10T16:09:03.143 INFO:tasks.rados.rados.0.smithi165.stdout:345: done (1 left) 2024-09-10T16:09:03.143 INFO:tasks.rados.rados.0.smithi165.stdout:348: done (0 left) 2024-09-10T16:09:03.143 INFO:tasks.rados.rados.0.smithi165.stdout:349: write (excl) oid 15 current snap is 44 2024-09-10T16:09:03.143 INFO:tasks.rados.rados.0.smithi165.stdout:349: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-10T16:09:03.149 INFO:tasks.rados.rados.0.smithi165.stdout:349: writing smithi16534841-15 from 571471 to 1211628 tid 1 2024-09-10T16:09:03.155 INFO:tasks.rados.rados.0.smithi165.stdout:349: writing smithi16534841-15 from 1653392 to 2177776 tid 2 2024-09-10T16:09:03.155 INFO:tasks.rados.rados.0.smithi165.stdout:350: read oid 3 snap -1 2024-09-10T16:09:03.155 INFO:tasks.rados.rados.0.smithi165.stdout:350: expect (ObjNum 144 snap 42 seq_num 144) 2024-09-10T16:09:03.155 INFO:tasks.rados.rados.0.smithi165.stdout:351: rollback oid 49 current snap is 44 2024-09-10T16:09:03.156 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 38 2024-09-10T16:09:03.156 INFO:tasks.rados.rados.0.smithi165.stdout:352: delete oid 22 current snap is 44 2024-09-10T16:09:03.159 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing write tid 1 to smithi16534841-15 2024-09-10T16:09:03.160 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:09:03.160 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:09:03.161 INFO:tasks.rados.rados.0.smithi165.stdout:351: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:09:03.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 80 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:09:03.161 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing write tid 2 to smithi16534841-15 2024-09-10T16:09:03.162 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing write tid 3 to smithi16534841-15 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:351: done (3 left) 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:352: done (2 left) 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:353: delete oid 21 current snap is 44 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:349: finishing write tid 4 to smithi16534841-15 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:349: oid 15 updating version 0 to 101 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:349: oid 15 updating version 101 to 103 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:349: oid 15 version 103 is already newer than 102 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 103 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:09:03.163 INFO:tasks.rados.rados.0.smithi165.stdout:349: left oid 15 (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:09:03.165 INFO:tasks.rados.rados.0.smithi165.stdout:350: expect (ObjNum 144 snap 42 seq_num 144) 2024-09-10T16:09:03.179 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1770939 2024-09-10T16:09:03.228 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:349: done (2 left) 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:350: done (1 left) 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:353: done (0 left) 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:354: read oid 10 snap -1 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:354: expect (ObjNum 106 snap 28 seq_num 106) 2024-09-10T16:09:03.265 INFO:tasks.rados.rados.0.smithi165.stdout:355: read oid 4 snap -1 2024-09-10T16:09:03.266 INFO:tasks.rados.rados.0.smithi165.stdout:355: expect (ObjNum 132 snap 37 seq_num 132) 2024-09-10T16:09:03.266 INFO:tasks.rados.rados.0.smithi165.stdout:356: snap_remove snap 41 2024-09-10T16:09:04.154 INFO:tasks.rados.rados.0.smithi165.stdout:356: done (2 left) 2024-09-10T16:09:04.154 INFO:tasks.rados.rados.0.smithi165.stdout:357: rollback oid 43 current snap is 44 2024-09-10T16:09:04.154 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 33 2024-09-10T16:09:04.155 INFO:tasks.rados.rados.0.smithi165.stdout:355: expect (ObjNum 132 snap 37 seq_num 132) 2024-09-10T16:09:04.189 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:04.251 INFO:tasks.rados.rados.0.smithi165.stdout:354: expect (ObjNum 106 snap 28 seq_num 106) 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:357: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 101 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:354: done (2 left) 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:355: done (1 left) 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:357: done (0 left) 2024-09-10T16:09:04.449 INFO:tasks.rados.rados.0.smithi165.stdout:358: snap_create 2024-09-10T16:09:05.088 INFO:tasks.rados.rados.0.smithi165.stdout:358: done (0 left) 2024-09-10T16:09:05.089 INFO:tasks.rados.rados.0.smithi165.stdout:359: rollback oid 12 current snap is 45 2024-09-10T16:09:05.089 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 20 2024-09-10T16:09:05.089 INFO:tasks.rados.rados.0.smithi165.stdout:360: snap_remove snap 21 2024-09-10T16:09:05.253 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.360+0000 7f355f88c640 10 monclient: get_monmap_and_config 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.360+0000 7f355f88c640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.360+0000 7f355f88c640 10 monclient: monmap: 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:05.362435+0000 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:05.362435+0000 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:05.363 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:05.364 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:05.364 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:05.364 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:05.364 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 Processor -- start 2024-09-10T16:09:05.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 -- start start 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient: init 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-c con 0x7f355809c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f355809d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-b con 0x7f355813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355809a430 con 0x7f355809d160 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f35580a24e0 con 0x7f355809c180 2024-09-10T16:09:05.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35580a2bf0 con 0x7f355813a5c0 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.361+0000 7f355f88c640 10 monclient(hunting): authenticate will time out at 923.849883s 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51344/0 (socket says 172.21.15.164:51344) 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 -- 172.21.15.164:0/2501296263 learned_addr learned my addr 172.21.15.164:0/2501296263 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): get_auth_request con 0x7f355813a5c0 auth_method 0 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 1 --2- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 10 monclient(hunting): get_auth_request con 0x7f355809c180 auth_method 0 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 1 -- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 msgr2=0x7f355809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 1 -- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 msgr2=0x7f355809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 1 --2- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355d628640 1 --2- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient(hunting): handle_auth_done global_id 4601 payload 274 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 -- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 msgr2=0x7f355809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 --2- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 -- 172.21.15.164:0/2501296263 >> v1:172.21.15.165:6789/0 conn(0x7f355809d160 legacy=0x7f355809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:05.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 monclient: found mon.noname-b 2024-09-10T16:09:05.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.362+0000 7f355de29640 1 -- 172.21.15.164:0/2501296263 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355809c560 con 0x7f355813a5c0 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355de29640 1 -- 172.21.15.164:0/2501296263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 msgr2=0x7f355813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355de29640 1 --2- 172.21.15.164:0/2501296263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3550009e80 tx=0x7f355002a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d568757c2468dc7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 1 -- 172.21.15.164:0/2501296263 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550036070 con 0x7f355813a5c0 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: dump: 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.376 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 1 -- 172.21.15.164:0/2501296263 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3550007800 con 0x7f355813a5c0 2024-09-10T16:09:05.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 5 monclient: authenticate success, global_id 4601 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 1 -- 172.21.15.164:0/2501296263 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550007bd0 con 0x7f355813a5c0 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 4 set_mon_vals no callback set 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f354ffff640 10 monclient: dump: 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.378 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 10 monclient: shutdown 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 -- 172.21.15.164:0/2501296263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 msgr2=0x7f355813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 --2- 172.21.15.164:0/2501296263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3550009e80 tx=0x7f355002a580 comp rx=0 tx=0).stop 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 -- 172.21.15.164:0/2501296263 shutdown_connections 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 --2- 172.21.15.164:0/2501296263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355813a5c0 0x7f355813c9b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 --2- 172.21.15.164:0/2501296263 >> v2:172.21.15.165:3300/0 conn(0x7f355809c180 0x7f355809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.365+0000 7f355f88c640 1 -- 172.21.15.164:0/2501296263 >> 172.21.15.164:0/2501296263 conn(0x7f35580a3110 msgr2=0x7f35580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.366+0000 7f355f88c640 1 -- 172.21.15.164:0/2501296263 shutdown_connections 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.366+0000 7f355f88c640 1 -- 172.21.15.164:0/2501296263 wait complete. 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.366+0000 7f355f88c640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.366+0000 7f355f88c640 10 monclient: monmap: 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:05.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.366+0000 7f355f88c640 1 Processor -- start 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 -- start start 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient: init 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-c con 0x7f355809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:05.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355809d160 0x7f3558090300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-b con 0x7f355809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 0x7f3558090840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): picked mon.noname-a con 0x7f355813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3558074f90 con 0x7f355813a5c0 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3558074e10 con 0x7f355809d160 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f355809f690 con 0x7f355809c180 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355f88c640 10 monclient(hunting): authenticate will time out at 923.855483s 2024-09-10T16:09:05.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51350/0 (socket says 172.21.15.164:51350) 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 -- 172.21.15.164:0/805881331 learned_addr learned my addr 172.21.15.164:0/805881331 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): get_auth_request con 0x7f355809c180 auth_method 0 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 0x7f3558090840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): get_auth_request con 0x7f355813a5c0 auth_method 0 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355ce27640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355809d160 0x7f3558090300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355ce27640 10 monclient(hunting): get_auth_request con 0x7f355809d160 auth_method 0 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355ce27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355ce27640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355ce27640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient(hunting): handle_auth_done global_id 4604 payload 824 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 -- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355809d160 msgr2=0x7f3558090300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355809d160 0x7f3558090300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 -- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 msgr2=0x7f3558090840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 0x7f3558090840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 monclient: found mon.noname-c 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355d628640 1 -- 172.21.15.164:0/805881331 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35580a42a0 con 0x7f355809c180 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.367+0000 7f355de29640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 0x7f3558090840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355d628640 1 -- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 msgr2=0x7f35580999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355d628640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f355400d7d0 tx=0x7f355400dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355400b8d0 con 0x7f355809c180 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: dump: 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3554019070 con 0x7f355809c180 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 5 monclient: authenticate success, global_id 4604 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 10 monclient: _renew_subs 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355809d620 con 0x7f355809c180 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f355400bcd0 con 0x7f355809c180 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354dffb640 10 monclient: dump: 2024-09-10T16:09:05.387 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f354f7fe640 4 set_mon_vals no callback set 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 10 monclient: _renew_subs 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.370+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3558093810 con 0x7f355809c180 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3554014430 con 0x7f355809c180 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f354dffb640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 0x7f353c040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f355f88c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 4863+0+0 (secure 0 0 0) 0x7f3554052f90 con 0x7f355809c180 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f355f88c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f355f88c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.371+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f355809d930 con 0x7f355809c180 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.372+0000 7f355ce27640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 0x7f353c040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.372+0000 7f355ce27640 10 monclient: get_auth_request con 0x7f353c03e270 auth_method 0 2024-09-10T16:09:05.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.374+0000 7f355ce27640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 0x7f353c040730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3558092560 tx=0x7f3548009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.376+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3554014900 con 0x7f355809c180 2024-09-10T16:09:05.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.376+0000 7f354dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.376+0000 7f354dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.620+0000 7f355f88c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.620+0000 7f355f88c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.620+0000 7f355f88c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.620+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f355813e5e0 con 0x7f355809c180 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.621+0000 7f354dffb640 1 -- 172.21.15.164:0/805881331 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f355401d410 con 0x7f355809c180 2024-09-10T16:09:05.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.621+0000 7f354dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:05.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.621+0000 7f354dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:05.624 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 msgr2=0x7f353c040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 0x7f353c040730 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3558092560 tx=0x7f3548009040 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 10 monclient: shutdown 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 msgr2=0x7f35580999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f355400d7d0 tx=0x7f355400dc90 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 shutdown_connections 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f353c03e270 0x7f353c040730 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f355813a5c0 0x7f3558090840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f355809d160 0x7f3558090300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 --2- 172.21.15.164:0/805881331 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f355809c180 0x7f35580999c0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 >> 172.21.15.164:0/805881331 conn(0x7f35580a3110 msgr2=0x7f3558072be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 shutdown_connections 2024-09-10T16:09:05.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.623+0000 7f355f88c640 1 -- 172.21.15.164:0/805881331 wait complete. 2024-09-10T16:09:05.636 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:09:05.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:09:05.636 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:09:05.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 10 monclient: get_monmap_and_config 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 10 monclient: monmap: 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:05.734085+0000 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:05.734085+0000 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:05.735 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:05.736 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:05.736 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:05.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 1 Processor -- start 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 1 -- start start 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.732+0000 7f2867b5d640 10 monclient: init 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 0x7f2860134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f286006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 0x7f28600a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f28600a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:05.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2860137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f286009ce50 con 0x7f2860137ca0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f286006db60 con 0x7f286006cb70 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f286006cf50 con 0x7f28600a46a0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f2867b5d640 10 monclient(hunting): authenticate will time out at 924.221525s 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 0x7f2860134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 0x7f28600a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28660fa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2860137ca0 0x7f2860138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39184/0 (socket says 172.21.15.164:39184) 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28660fa640 1 -- 172.21.15.164:0/919972999 learned_addr learned my addr 172.21.15.164:0/919972999 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): get_auth_request con 0x7f286006cb70 auth_method 0 2024-09-10T16:09:05.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): get_auth_request con 0x7f28600a46a0 auth_method 0 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient(hunting): handle_auth_done global_id 4607 payload 274 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 -- 172.21.15.164:0/919972999 >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 msgr2=0x7f2860134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 --2- 172.21.15.164:0/919972999 >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 0x7f2860134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 -- 172.21.15.164:0/919972999 >> v1:172.21.15.164:6789/0 conn(0x7f2860137ca0 legacy=0x7f2860138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 monclient: found mon.noname-b 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28650f8640 1 -- 172.21.15.164:0/919972999 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f286009cb70 con 0x7f28600a46a0 2024-09-10T16:09:05.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.733+0000 7f28658f9640 1 --2- 172.21.15.164:0/919972999 >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 0x7f2860134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28650f8640 1 -- 172.21.15.164:0/919972999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 msgr2=0x7f28600a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28650f8640 1 --2- 172.21.15.164:0/919972999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 0x7f28600a4ab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097a0 tx=0x7f285c029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32aa5f3fd3fdffc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 1 -- 172.21.15.164:0/919972999 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c037070 con 0x7f28600a46a0 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: dump: 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.748 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 1 -- 172.21.15.164:0/919972999 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c032440 con 0x7f28600a46a0 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f2867b5d640 5 monclient: authenticate success, global_id 4607 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f2867b5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:05.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 1 -- 172.21.15.164:0/919972999 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c032810 con 0x7f28600a46a0 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f28648f7640 10 monclient: dump: 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f2867b5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.736+0000 7f2867b5d640 4 set_mon_vals no callback set 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 10 monclient: shutdown 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 -- 172.21.15.164:0/919972999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 msgr2=0x7f28600a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 --2- 172.21.15.164:0/919972999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 0x7f28600a4ab0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f285c0097a0 tx=0x7f285c029fb0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 -- 172.21.15.164:0/919972999 shutdown_connections 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 --2- 172.21.15.164:0/919972999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f28600a46a0 0x7f28600a4ab0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 --2- 172.21.15.164:0/919972999 >> v2:172.21.15.164:3300/0 conn(0x7f286006cb70 0x7f2860134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 -- 172.21.15.164:0/919972999 >> 172.21.15.164:0/919972999 conn(0x7f28600aa900 msgr2=0x7f28600aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 -- 172.21.15.164:0/919972999 shutdown_connections 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.737+0000 7f2867b5d640 1 -- 172.21.15.164:0/919972999 wait complete. 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient: monmap: 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 Processor -- start 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 -- start start 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient: init 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-c con 0x7f286006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 0x7f28600a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-a con 0x7f28600a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860137ca0 0x7f28600a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): picked mon.noname-b con 0x7f2860137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28601389c0 con 0x7f28600a46a0 2024-09-10T16:09:05.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2860138840 con 0x7f2860137ca0 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2860140040 con 0x7f286006cb70 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.738+0000 7f2867b5d640 10 monclient(hunting): authenticate will time out at 924.226934s 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51374/0 (socket says 172.21.15.164:51374) 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 learned_addr learned my addr 172.21.15.164:0/2117517988 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 10 monclient(hunting): get_auth_request con 0x7f286006cb70 auth_method 0 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 0x7f28600a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): get_auth_request con 0x7f28600a46a0 auth_method 0 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 msgr2=0x7f2860131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 msgr2=0x7f2860131620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28660fa640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860137ca0 0x7f28600a3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient(hunting): handle_auth_done global_id 24204 payload 824 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 msgr2=0x7f2860131620 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f286006cb70 0x7f2860131620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860137ca0 msgr2=0x7f28600a3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860137ca0 0x7f28600a3fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 monclient: found mon.noname-a 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28650f8640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28601cd790 con 0x7f28600a46a0 2024-09-10T16:09:05.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28660fa640 10 monclient: get_auth_request con 0x7f2860137ca0 auth_method 0 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.739+0000 7f28660fa640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2860137ca0 0x7f28600a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28650f8640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 msgr2=0x7f28600a3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28650f8640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 0x7f28600a3aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f285c031000 tx=0x7f285c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c041020 con 0x7f28600a46a0 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: dump: 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.757 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 5 monclient: authenticate success, global_id 24204 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f285c03b8c0 con 0x7f28600a46a0 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 10 monclient: _renew_subs 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28600a0290 con 0x7f28600a46a0 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285c037040 con 0x7f28600a46a0 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f28527fc640 10 monclient: dump: 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.759 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2853fff640 4 set_mon_vals no callback set 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 10 monclient: _renew_subs 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.741+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28600a95b0 con 0x7f28600a46a0 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f2867b5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f2867b5d640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 0x7f286009da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2860134f70 con 0x7f286009d630 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f285c02ad10 con 0x7f28600a46a0 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 0x7f286009da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient: get_auth_request con 0x7f286009d630 auth_method 0 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28527fc640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e210 0x7f28440406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 10 monclient(hunting): handle_auth_done global_id 24207 payload 824 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f286009ed70 con 0x7f286009d630 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.742+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 4863+0+0 (secure 0 0 0) 0x7f285c072c30 con 0x7f28600a46a0 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.743+0000 7f28660fa640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e210 0x7f28440406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.743+0000 7f28660fa640 10 monclient: get_auth_request con 0x7f284403e210 auth_method 0 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 msgr2=0x7f286009da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 0x7f286009da10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f2854011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f285400fc40 con 0x7f286009d630 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 10 monclient: dump: 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f285400fc40 con 0x7f286009d630 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 msgr2=0x7f286009da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 0x7f286009da10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f2854011cc0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 msgr2=0x7f286009da10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28527fc640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009d630 0x7f286009da10 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f2854011cc0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.745+0000 7f28660fa640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e210 0x7f28440406d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2858005ac0 tx=0x7f2858009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f2867b5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f2867b5d640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 0x7f2860135aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f2867b5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28600770f0 con 0x7f286009df50 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 0x7f2860135aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient: get_auth_request con 0x7f286009df50 auth_method 0 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 10 monclient(hunting): handle_auth_done global_id 24210 payload 824 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.751+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f286009e360 con 0x7f286009df50 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28658f9640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 msgr2=0x7f2860135aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28658f9640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 0x7f2860135aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f28540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2854003610 con 0x7f286009df50 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 10 monclient: dump: 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.766 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f2854026070 con 0x7f286009df50 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 msgr2=0x7f2860135aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 0x7f2860135aa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f28540046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 msgr2=0x7f2860135aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28527fc640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f286009df50 0x7f2860135aa0 secure :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2854011800 tx=0x7f28540046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 70687, 2024-09-10T16:09:05.768 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:05.769 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.770 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.771 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:05.772 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.773 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.774 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:05.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:05.776 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:05.779 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:05.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.781 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:05.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.782 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.782 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:05.782 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:05.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e210 msgr2=0x7f28440406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e210 0x7f28440406d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2858005ac0 tx=0x7f2858009040 comp rx=0 tx=0).stop 2024-09-10T16:09:05.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 10 monclient: shutdown 2024-09-10T16:09:05.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 msgr2=0x7f28600a3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 0x7f28600a3aa0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f285c031000 tx=0x7f285c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f28660fa640 1 -- 172.21.15.164:0/2117517988 reap_dead start 2024-09-10T16:09:05.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 shutdown_connections 2024-09-10T16:09:05.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 --2- 172.21.15.164:0/2117517988 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f28600a46a0 0x7f28600a3aa0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.754+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 >> 172.21.15.164:0/2117517988 conn(0x7f28600aa900 msgr2=0x7f28600a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.755+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 shutdown_connections 2024-09-10T16:09:05.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.755+0000 7f2867b5d640 1 -- 172.21.15.164:0/2117517988 wait complete. 2024-09-10T16:09:05.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:05.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:09:05.785 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 10 monclient: get_monmap_and_config 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 10 monclient: monmap: 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:05.883012+0000 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:05.883012+0000 2024-09-10T16:09:05.884 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 1 Processor -- start 2024-09-10T16:09:05.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 1 -- start start 2024-09-10T16:09:05.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.881+0000 7f4bb0dc6640 10 monclient: init 2024-09-10T16:09:05.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:09:05.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4bac06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:05.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bac134f70 0x7f4bac13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4bac13d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac09ce50 con 0x7f4bac13d560 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac135380 con 0x7f4bac06cb70 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4bac06db60 con 0x7f4bac134f70 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4bb0dc6640 10 monclient(hunting): authenticate will time out at 924.370551s 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4bac134f70 0x7f4bac13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4baad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4bac13d560 0x7f4bac13f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39200/0 (socket says 172.21.15.164:39200) 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4baad9d640 1 -- 172.21.15.164:0/184332079 learned_addr learned my addr 172.21.15.164:0/184332079 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): get_auth_request con 0x7f4bac134f70 auth_method 0 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_done global_id 24213 payload 274 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 1 -- 172.21.15.164:0/184332079 >> v1:172.21.15.165:6789/0 conn(0x7f4bac06cb70 legacy=0x7f4bac0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 1 -- 172.21.15.164:0/184332079 >> v1:172.21.15.164:6789/0 conn(0x7f4bac13d560 legacy=0x7f4bac13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:05.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 1 monclient: found mon.noname-a 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.882+0000 7f4ba9d9b640 1 -- 172.21.15.164:0/184332079 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac09cb70 con 0x7f4bac134f70 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.885+0000 7f4ba9d9b640 1 -- 172.21.15.164:0/184332079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac134f70 msgr2=0x7f4bac13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.885+0000 7f4ba9d9b640 1 --2- 172.21.15.164:0/184332079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac134f70 0x7f4bac13d020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4b940097d0 tx=0x7f4b9402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd62bbdc44aff949 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 1 -- 172.21.15.164:0/184332079 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b94037070 con 0x7f4bac134f70 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: dump: 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 1 -- 172.21.15.164:0/184332079 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b94032440 con 0x7f4bac134f70 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 5 monclient: authenticate success, global_id 24213 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 4 set_mon_vals no callback set 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 1 -- 172.21.15.164:0/184332079 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b94032810 con 0x7f4bac134f70 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4ba959a640 10 monclient: dump: 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 10 monclient: shutdown 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/184332079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac134f70 msgr2=0x7f4bac13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/184332079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac134f70 0x7f4bac13d020 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4b940097d0 tx=0x7f4b9402a000 comp rx=0 tx=0).stop 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/184332079 shutdown_connections 2024-09-10T16:09:05.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/184332079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac134f70 0x7f4bac13d020 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/184332079 >> 172.21.15.164:0/184332079 conn(0x7f4bac0aa900 msgr2=0x7f4bac0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/184332079 shutdown_connections 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.886+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/184332079 wait complete. 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.887+0000 7f4bb0dc6640 10 monclient: build_initial_monmap 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.887+0000 7f4bb0dc6640 10 monclient: monmap: 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:05.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.887+0000 7f4bb0dc6640 1 Processor -- start 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.887+0000 7f4bb0dc6640 1 -- start start 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.887+0000 7f4bb0dc6640 10 monclient: init 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac06cb70 0x7f4bac131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-b con 0x7f4bac06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 0x7f4bac13a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-c con 0x7f4bac134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 0x7f4bac13a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac13d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac0a4db0 con 0x7f4bac13d560 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac06cf50 con 0x7f4bac06cb70 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac141070 con 0x7f4bac134f70 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4bb0dc6640 10 monclient(hunting): authenticate will time out at 924.376261s 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 0x7f4bac13a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 0x7f4bac13a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 0x7f4bac13a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51382/0 (socket says 172.21.15.164:51382) 2024-09-10T16:09:05.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 1 -- 172.21.15.164:0/307578544 learned_addr learned my addr 172.21.15.164:0/307578544 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:05.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): get_auth_request con 0x7f4bac13d560 auth_method 0 2024-09-10T16:09:05.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): get_auth_request con 0x7f4bac134f70 auth_method 0 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baa59c640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac06cb70 0x7f4bac131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baa59c640 10 monclient(hunting): get_auth_request con 0x7f4bac06cb70 auth_method 0 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient(hunting): handle_auth_done global_id 24216 payload 824 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 msgr2=0x7f4bac13a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 0x7f4bac13a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac06cb70 msgr2=0x7f4bac131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4bac06cb70 0x7f4bac131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 monclient: found mon.noname-a 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4baad9d640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac09ef10 con 0x7f4bac13d560 2024-09-10T16:09:05.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.888+0000 7f4ba9d9b640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac134f70 0x7f4bac13a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.890+0000 7f4baad9d640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 msgr2=0x7f4bac13a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.890+0000 7f4baad9d640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 0x7f4bac13a560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0a7380 tx=0x7f4b9c00c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c017070 con 0x7f4bac13d560 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: dump: 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.912 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b9c0070d0 con 0x7f4bac13d560 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 5 monclient: authenticate success, global_id 24216 2024-09-10T16:09:05.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 monclient: _renew_subs 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac09e270 con 0x7f4bac13d560 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b9c0074c0 con 0x7f4bac13d560 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba2ffd640 10 monclient: dump: 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.914 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4ba8d99640 4 set_mon_vals no callback set 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 monclient: _renew_subs 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac13b9a0 con 0x7f4bac13d560 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 0x7f4bac13aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.891+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac09def0 con 0x7f4bac131b50 2024-09-10T16:09:05.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 0x7f4bac13aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient: get_auth_request con 0x7f4bac131b50 auth_method 0 2024-09-10T16:09:05.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4b9c01a070 con 0x7f4bac13d560 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 10 monclient(hunting): handle_auth_done global_id 4613 payload 824 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4baa59c640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4bac0008d0 con 0x7f4bac131b50 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4ba2ffd640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b8c03e210 0x7f4b8c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 4863+0+0 (secure 0 0 0) 0x7f4b9c0520f0 con 0x7f4bac13d560 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4ba9d9b640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b8c03e210 0x7f4b8c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.892+0000 7f4ba9d9b640 10 monclient: get_auth_request con 0x7f4b8c03e210 auth_method 0 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4baa59c640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 msgr2=0x7f4bac13aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4baa59c640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 0x7f4bac13aaa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ce30 tx=0x7f4ba4007660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba401a070 con 0x7f4bac131b50 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 10 monclient: dump: 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ba400a800 con 0x7f4bac131b50 2024-09-10T16:09:05.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 msgr2=0x7f4bac13aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 0x7f4bac13aaa0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ce30 tx=0x7f4ba4007660 comp rx=0 tx=0).stop 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 msgr2=0x7f4bac13aaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.894+0000 7f4ba2ffd640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac131b50 0x7f4bac13aaa0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.895+0000 7f4ba9d9b640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b8c03e210 0x7f4b8c0406d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b94000c00 tx=0x7f4b9402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.900+0000 7f4bb0dc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.900+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 0x7f4bac135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.900+0000 7f4bb0dc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.900+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac13b3f0 con 0x7f4bac13afe0 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.900+0000 7f4baa59c640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 0x7f4bac135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient: get_auth_request con 0x7f4bac13afe0 auth_method 0 2024-09-10T16:09:05.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 10 monclient(hunting): handle_auth_done global_id 4616 payload 824 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.901+0000 7f4baa59c640 1 -- 172.21.15.164:0/307578544 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4bac0770f0 con 0x7f4bac13afe0 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4baa59c640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 msgr2=0x7f4bac135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4baa59c640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 0x7f4bac135870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ce30 tx=0x7f4ba4004a70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba401a050 con 0x7f4bac13afe0 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:05.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 10 monclient: dump: 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f4ba401b040 con 0x7f4bac13afe0 2024-09-10T16:09:05.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 msgr2=0x7f4bac135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 0x7f4bac135870 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ce30 tx=0x7f4ba4004a70 comp rx=0 tx=0).stop 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 msgr2=0x7f4bac135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.903+0000 7f4ba2ffd640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4bac13afe0 0x7f4bac135870 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4ba400ce30 tx=0x7f4ba4004a70 comp rx=0 tx=0).stop 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:05.922 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 188075, 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.923 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.924 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.925 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:05.926 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:05.927 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.929 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:05.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.934 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:05.935 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:05.936 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:05.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b8c03e210 msgr2=0x7f4b8c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b8c03e210 0x7f4b8c0406d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4b94000c00 tx=0x7f4b9402e000 comp rx=0 tx=0).stop 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 10 monclient: shutdown 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 msgr2=0x7f4bac13a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4baad9d640 1 -- 172.21.15.164:0/307578544 reap_dead start 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 0x7f4bac13a560 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0a7380 tx=0x7f4b9c00c7b0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 shutdown_connections 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 --2- 172.21.15.164:0/307578544 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4bac13d560 0x7f4bac13a560 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 >> 172.21.15.164:0/307578544 conn(0x7f4bac0aa900 msgr2=0x7f4bac134070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 shutdown_connections 2024-09-10T16:09:05.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:05.904+0000 7f4bb0dc6640 1 -- 172.21.15.164:0/307578544 wait complete. 2024-09-10T16:09:05.939 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:05.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:09:05.939 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.035+0000 7f03ab647640 10 monclient: get_monmap_and_config 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.035+0000 7f03ab647640 10 monclient: build_initial_monmap 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.035+0000 7f03ab647640 10 monclient: monmap: 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:06.037697+0000 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:06.037697+0000 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.038 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:06.039 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:06.039 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:06.039 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:06.039 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:06.039 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.036+0000 7f03ab647640 1 Processor -- start 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.036+0000 7f03ab647640 1 -- start start 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.036+0000 7f03ab647640 10 monclient: init 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-a con 0x7f03a4133a00 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-c con 0x7f03a4133de0 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a409d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-b con 0x7f03a4134270 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f03a4073ee0 con 0x7f03a4133a00 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f03a40a1c90 con 0x7f03a4133de0 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f03a40a1f70 con 0x7f03a4134270 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03ab647640 10 monclient(hunting): authenticate will time out at 924.525112s 2024-09-10T16:09:06.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a409d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60496/0 (socket says 172.21.15.164:60496) 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): get_auth_request con 0x7f03a4134270 auth_method 0 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 1 -- 172.21.15.164:0/3302927563 learned_addr learned my addr 172.21.15.164:0/3302927563 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): get_auth_request con 0x7f03a4133a00 auth_method 0 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 10 monclient(hunting): get_auth_request con 0x7f03a4133de0 auth_method 0 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient(hunting): handle_auth_done global_id 4619 payload 274 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 -- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 msgr2=0x7f03a409ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 1 -- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 msgr2=0x7f03a409ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 1 -- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 msgr2=0x7f03a409ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a8be2640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 -- 172.21.15.164:0/3302927563 >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 msgr2=0x7f03a40a2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient: _finish_hunting 0 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 monclient: found mon.noname-b 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:06.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:06.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a9be4640 1 -- 172.21.15.164:0/3302927563 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a40a2250 con 0x7f03a4134270 2024-09-10T16:09:06.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.037+0000 7f03a93e3640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:06.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03a9be4640 1 -- 172.21.15.164:0/3302927563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 msgr2=0x7f03a409d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:06.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03a9be4640 1 --2- 172.21.15.164:0/3302927563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a409d410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03a40742b0 tx=0x7f039c02a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9d7b55ada2b6d3f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:06.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 1 -- 172.21.15.164:0/3302927563 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c007950 con 0x7f03a4134270 2024-09-10T16:09:06.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: dump: 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:06.054 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: _finish_auth 0 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03ab647640 5 monclient: authenticate success, global_id 4619 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03ab647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 1 -- 172.21.15.164:0/3302927563 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f039c007af0 con 0x7f03a4134270 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03ab647640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f03ab647640 4 set_mon_vals no callback set 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 1 -- 172.21.15.164:0/3302927563 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c031400 con 0x7f03a4134270 2024-09-10T16:09:06.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.040+0000 7f039bfff640 10 monclient: dump: 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:06.056 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 10 monclient: shutdown 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 -- 172.21.15.164:0/3302927563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 msgr2=0x7f03a409d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 --2- 172.21.15.164:0/3302927563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a409d410 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f03a40742b0 tx=0x7f039c02a4b0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 -- 172.21.15.164:0/3302927563 shutdown_connections 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 --2- 172.21.15.164:0/3302927563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a409d410 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.165:3300/0 conn(0x7f03a4133de0 0x7f03a409ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 --2- 172.21.15.164:0/3302927563 >> v2:172.21.15.164:3300/0 conn(0x7f03a4133a00 0x7f03a40a2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 -- 172.21.15.164:0/3302927563 >> 172.21.15.164:0/3302927563 conn(0x7f03a40a7320 msgr2=0x7f03a40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 -- 172.21.15.164:0/3302927563 shutdown_connections 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 1 -- 172.21.15.164:0/3302927563 wait complete. 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 10 monclient: build_initial_monmap 2024-09-10T16:09:06.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.041+0000 7f03ab647640 10 monclient: monmap: 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 Processor -- start 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 -- start start 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient: init 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:06.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 0x7f03a41ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-a con 0x7f03a4133a00 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4133de0 0x7f03a4132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-b con 0x7f03a4133de0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a4132e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): picked mon.noname-c con 0x7f03a4134270 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40a36b0 con 0x7f03a4133a00 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40a3530 con 0x7f03a4133de0 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f03a409d950 con 0x7f03a4134270 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:06.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03ab647640 10 monclient(hunting): authenticate will time out at 924.530634s 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 0x7f03a41ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 0x7f03a41ccdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60498/0 (socket says 172.21.15.164:60498) 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 -- 172.21.15.164:0/248705166 learned_addr learned my addr 172.21.15.164:0/248705166 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): get_auth_request con 0x7f03a4133a00 auth_method 0 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a4132e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): get_auth_request con 0x7f03a4134270 auth_method 0 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:06.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a9be4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4133de0 0x7f03a4132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient(hunting): handle_auth_done global_id 24222 payload 824 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 msgr2=0x7f03a4132e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f03a4134270 0x7f03a4132e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4133de0 msgr2=0x7f03a4132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4133de0 0x7f03a4132930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient: _finish_hunting 0 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 1 monclient: found mon.noname-a 2024-09-10T16:09:06.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.042+0000 7f03a93e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.043+0000 7f03a93e3640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a4125000 con 0x7f03a4133a00 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.043+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4133de0 auth_method 0 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.043+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4133de0 0x7f03a4132930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03a93e3640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 msgr2=0x7f03a41ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03a93e3640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 0x7f03a41ccdc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03a000e9d0 tx=0x7f03a000ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0017070 con 0x7f03a4133a00 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: dump: 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:06.062 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 5 monclient: authenticate success, global_id 24222 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 10 monclient: _renew_subs 2024-09-10T16:09:06.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f03a000cc10 con 0x7f03a4133a00 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03a409e0f0 con 0x7f03a4133a00 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a0013470 con 0x7f03a4133a00 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f0399ffb640 10 monclient: dump: 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:06.064 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f039b7fe640 4 set_mon_vals no callback set 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 10 monclient: _renew_subs 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.045+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a4127360 con 0x7f03a4133a00 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03ab647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03ab647640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f03a4126f30 con 0x7f03a4123540 2024-09-10T16:09:06.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f03a001a070 con 0x7f03a4133a00 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f0399ffb640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f038803e240 0x7f0388040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4123540 auth_method 0 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 4863+0+0 (secure 0 0 0) 0x7f03a0052d00 con 0x7f03a4133a00 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a9be4640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f038803e240 0x7f0388040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.046+0000 7f03a9be4640 10 monclient: get_auth_request con 0x7f038803e240 auth_method 0 2024-09-10T16:09:06.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.049+0000 7f03a9be4640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f038803e240 0x7f0388040700 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f03a4073c30 tx=0x7f039c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:360: done (1 left) 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:361: rollback oid 14 current snap is 45 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 27 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:359: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:359: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:359: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 7 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:359: done (1 left) 2024-09-10T16:09:06.156 INFO:tasks.rados.rados.0.smithi165.stdout:362: snap_create 2024-09-10T16:09:06.170 INFO:tasks.rados.rados.0.smithi165.stdout:361: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:09:06.170 INFO:tasks.rados.rados.0.smithi165.stdout:361: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:09:06.170 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 93 (ObjNum 129 snap 0 seq_num 1162980328) dirty dne 2024-09-10T16:09:06.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4123540 auth_method 0 2024-09-10T16:09:06.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:06.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.248+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:09:06.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:06.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4123540 auth_method 0 2024-09-10T16:09:06.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:06.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:06.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:06.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:06.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:06.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:06.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:06.649+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:09:07.159 INFO:tasks.rados.rados.0.smithi165.stdout:361: done (1 left) 2024-09-10T16:09:07.159 INFO:tasks.rados.rados.0.smithi165.stdout:362: done (0 left) 2024-09-10T16:09:07.160 INFO:tasks.rados.rados.0.smithi165.stdout:363: delete oid 47 current snap is 46 2024-09-10T16:09:07.165 INFO:tasks.rados.rados.0.smithi165.stdout:363: done (0 left) 2024-09-10T16:09:07.165 INFO:tasks.rados.rados.0.smithi165.stdout:364: write (excl) oid 46 current snap is 46 2024-09-10T16:09:07.165 INFO:tasks.rados.rados.0.smithi165.stdout:364: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-10T16:09:07.171 INFO:tasks.rados.rados.0.smithi165.stdout:364: writing smithi16534841-46 from 638699 to 1179852 tid 1 2024-09-10T16:09:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:364: writing smithi16534841-46 from 1770967 to 2302689 tid 2 2024-09-10T16:09:07.178 INFO:tasks.rados.rados.0.smithi165.stdout:365: read oid 23 snap 31 2024-09-10T16:09:07.178 INFO:tasks.rados.rados.0.smithi165.stdout:365: expect (ObjNum 97 snap 24 seq_num 97) 2024-09-10T16:09:07.178 INFO:tasks.rados.rados.0.smithi165.stdout:366: delete oid 18 current snap is 46 2024-09-10T16:09:07.178 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing write tid 1 to smithi16534841-46 2024-09-10T16:09:07.183 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing write tid 2 to smithi16534841-46 2024-09-10T16:09:07.184 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing write tid 3 to smithi16534841-46 2024-09-10T16:09:07.184 INFO:tasks.rados.rados.0.smithi165.stdout:364: finishing write tid 4 to smithi16534841-46 2024-09-10T16:09:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:364: oid 46 updating version 0 to 109 2024-09-10T16:09:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:364: oid 46 version 109 is already newer than 108 2024-09-10T16:09:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:364: oid 46 updating version 109 to 110 2024-09-10T16:09:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 110 (ObjNum 157 snap 46 seq_num 157) dirty exists 2024-09-10T16:09:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:364: left oid 46 (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:07.190 INFO:tasks.rados.rados.0.smithi165.stdout:365: expect (ObjNum 97 snap 24 seq_num 97) 2024-09-10T16:09:07.190 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 130478 2024-09-10T16:09:07.250 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1020318 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:364: done (2 left) 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:365: done (1 left) 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:366: done (0 left) 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:367: rollback oid 23 current snap is 46 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 35 2024-09-10T16:09:07.352 INFO:tasks.rados.rados.0.smithi165.stdout:368: delete oid 44 current snap is 46 2024-09-10T16:09:07.355 INFO:tasks.rados.rados.0.smithi165.stdout:367: finishing rollback tid 0 to smithi16534841-23 2024-09-10T16:09:07.355 INFO:tasks.rados.rados.0.smithi165.stdout:368: done (1 left) 2024-09-10T16:09:07.355 INFO:tasks.rados.rados.0.smithi165.stdout:369: snap_remove snap 18 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4123540 auth_method 0 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 10 monclient(hunting): handle_auth_done global_id 34099 payload 824 2024-09-10T16:09:07.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.451+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03a40a2ff0 con 0x7f03a4123540 2024-09-10T16:09:07.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:07.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394014b10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:07.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039401d070 con 0x7f03a4123540 2024-09-10T16:09:07.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 10 monclient: dump: 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.456 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0394010cc0 con 0x7f03a4123540 2024-09-10T16:09:07.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:07.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:07.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394014b10 comp rx=0 tx=0).stop 2024-09-10T16:09:07.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 msgr2=0x7f03a4123920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:07.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.454+0000 7f0399ffb640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123540 0x7f03a4123920 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f0394014b10 comp rx=0 tx=0).stop 2024-09-10T16:09:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.460+0000 7f03ab647640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.460+0000 7f03ab647640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 0x7f03a411f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.460+0000 7f03ab647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.460+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f03a406d1c0 con 0x7f03a4123e60 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 0x7f03a411f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient: get_auth_request con 0x7f03a4123e60 auth_method 0 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 10 monclient(hunting): handle_auth_done global_id 34102 payload 824 2024-09-10T16:09:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.461+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f03a4126ac0 con 0x7f03a4123e60 2024-09-10T16:09:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f03a8be2640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 msgr2=0x7f03a411f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f03a8be2640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 0x7f03a411f560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f03940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039400f040 con 0x7f03a4123e60 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 10 monclient: dump: 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f0394010e70 con 0x7f03a4123e60 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 msgr2=0x7f03a411f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 0x7f03a411f560 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f03940046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 msgr2=0x7f03a411f560 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.464+0000 7f0399ffb640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f03a4123e60 0x7f03a411f560 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0394006fd0 tx=0x7f03940046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 5510, 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:07.469 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:07.470 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:07.471 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:07.472 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:07.473 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.474 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:07.475 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:07.476 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:07.477 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:07.478 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:07.479 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.480 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:07.481 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:07.482 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f038803e240 msgr2=0x7f0388040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f038803e240 0x7f0388040700 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f03a4073c30 tx=0x7f039c02f000 comp rx=0 tx=0).stop 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 10 monclient: shutdown 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 msgr2=0x7f03a41ccdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 --2- 172.21.15.164:0/248705166 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f03a4133a00 0x7f03a41ccdc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f03a000e9d0 tx=0x7f03a000ee90 comp rx=0 tx=0).stop 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03a9be4640 1 -- 172.21.15.164:0/248705166 reap_dead start 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03a9be4640 1 -- 172.21.15.164:0/248705166 reap_dead start 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 shutdown_connections 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 >> 172.21.15.164:0/248705166 conn(0x7f03a40a7320 msgr2=0x7f03a409e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 shutdown_connections 2024-09-10T16:09:07.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.465+0000 7f03ab647640 1 -- 172.21.15.164:0/248705166 wait complete. 2024-09-10T16:09:07.485 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:07.485 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:09:07.485 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:09:07.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.589+0000 7f7f99e8c640 10 monclient: get_monmap_and_config 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.589+0000 7f7f99e8c640 10 monclient: build_initial_monmap 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.589+0000 7f7f99e8c640 10 monclient: monmap: 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:07.591237+0000 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:07.591237+0000 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:07.592 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:07.593 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:07.593 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 Processor -- start 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 -- start start 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient: init 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f9406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f9409bcc0 0x7f7f9406d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9409bcc0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f940a40d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f9406e900 con 0x7f7f940a40d0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7f940a4600 con 0x7f7f9409bcc0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7f9406e340 con 0x7f7f9406cb70 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f99e8c640 10 monclient(hunting): authenticate will time out at 926.078695s 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f9409bcc0 0x7f7f9406d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f9409bcc0 0x7f7f9406d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60502/0 (socket says 172.21.15.164:60502) 2024-09-10T16:09:07.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 1 -- 172.21.15.164:0/839612469 learned_addr learned my addr 172.21.15.164:0/839612469 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): get_auth_request con 0x7f7f9409bcc0 auth_method 0 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51412/0 (socket says 172.21.15.164:51412) 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): get_auth_request con 0x7f7f9406cb70 auth_method 0 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f92ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.590+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 10 monclient(hunting): handle_auth_done global_id 24225 payload 274 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 1 -- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 msgr2=0x7f7f9406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 1 -- 172.21.15.164:0/839612469 >> v1:172.21.15.165:6789/0 conn(0x7f7f940a40d0 legacy=0x7f7f9413e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 1 monclient: found mon.noname-a 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f92ffd640 1 -- 172.21.15.164:0/839612469 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f940a01e0 con 0x7f7f9409bcc0 2024-09-10T16:09:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.591+0000 7f7f937fe640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.593+0000 7f7f92ffd640 1 -- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9409bcc0 msgr2=0x7f7f9406d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.593+0000 7f7f92ffd640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9409bcc0 0x7f7f9406d490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009870 tx=0x7f7f7c02a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223f8b48eaa502e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 1 -- 172.21.15.164:0/839612469 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c037070 con 0x7f7f9409bcc0 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: dump: 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.604 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 1 -- 172.21.15.164:0/839612469 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f7c032640 con 0x7f7f9409bcc0 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 5 monclient: authenticate success, global_id 24225 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:07.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 4 set_mon_vals no callback set 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 1 -- 172.21.15.164:0/839612469 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7c032a10 con 0x7f7f9409bcc0 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f927fc640 10 monclient: dump: 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.606 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 10 monclient: shutdown 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 -- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9409bcc0 msgr2=0x7f7f9406d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9409bcc0 0x7f7f9406d490 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7f7c009870 tx=0x7f7f7c02a130 comp rx=0 tx=0).stop 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 -- 172.21.15.164:0/839612469 shutdown_connections 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9409bcc0 0x7f7f9406d490 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/839612469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9406cb70 0x7f7f9406cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 -- 172.21.15.164:0/839612469 >> 172.21.15.164:0/839612469 conn(0x7f7f940a4f50 msgr2=0x7f7f940a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 -- 172.21.15.164:0/839612469 shutdown_connections 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.594+0000 7f7f99e8c640 1 -- 172.21.15.164:0/839612469 wait complete. 2024-09-10T16:09:07.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.595+0000 7f7f99e8c640 10 monclient: build_initial_monmap 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.595+0000 7f7f99e8c640 10 monclient: monmap: 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.595+0000 7f7f99e8c640 1 Processor -- start 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.595+0000 7f7f99e8c640 1 -- start start 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.595+0000 7f7f99e8c640 10 monclient: init 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-a con 0x7f7f9406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f940a40d0 0x7f7f9413d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-b con 0x7f7f940a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 0x7f7f9413dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): picked mon.noname-c con 0x7f7f9413d710 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9413fe90 con 0x7f7f9406cb70 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f9413fd10 con 0x7f7f940a40d0 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7f94140010 con 0x7f7f9413d710 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f99e8c640 10 monclient(hunting): authenticate will time out at 926.084311s 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60514/0 (socket says 172.21.15.164:60514) 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 0x7f7f9413dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 1 -- 172.21.15.164:0/3439536794 learned_addr learned my addr 172.21.15.164:0/3439536794 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): get_auth_request con 0x7f7f9406cb70 auth_method 0 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): get_auth_request con 0x7f7f9413d710 auth_method 0 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient(hunting): handle_auth_done global_id 4628 payload 824 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f92ffd640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f940a40d0 0x7f7f9413d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 -- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f940a40d0 msgr2=0x7f7f9413d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f940a40d0 0x7f7f9413d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 -- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 msgr2=0x7f7f9409aec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 monclient: found mon.noname-c 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f93fff640 1 -- 172.21.15.164:0/3439536794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f94093670 con 0x7f7f9413d710 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.596+0000 7f7f937fe640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.598+0000 7f7f93fff640 1 -- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 msgr2=0x7f7f9413dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.598+0000 7f7f93fff640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 0x7f7f9413dbc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c410 tx=0x7f7f8400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f840119a0 con 0x7f7f9413d710 2024-09-10T16:09:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: dump: 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: _finish_auth 0 2024-09-10T16:09:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f84017070 con 0x7f7f9413d710 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 5 monclient: authenticate success, global_id 4628 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 10 monclient: _renew_subs 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f940a1c20 con 0x7f7f9413d710 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84011d80 con 0x7f7f9413d710 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f8bfff640 10 monclient: dump: 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f91ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 10 monclient: _renew_subs 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f9413ea50 con 0x7f7f9413d710 2024-09-10T16:09:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.599+0000 7f7f99e8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f99e8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f99e8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7f8401c460 con 0x7f7f9413d710 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f9406de10 con 0x7f7f9413d710 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f8bfff640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 0x7f7f740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(103..103 src has 1..103) ==== 4803+0+0 (secure 0 0 0) 0x7f7f840528c0 con 0x7f7f9413d710 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f937fe640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 0x7f7f740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.600+0000 7f7f937fe640 10 monclient: get_auth_request con 0x7f7f7403e210 auth_method 0 2024-09-10T16:09:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.603+0000 7f7f937fe640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 0x7f7f740406d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009bd0 tx=0x7f7f8c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.604+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7f9406de10 con 0x7f7f9413d710 2024-09-10T16:09:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.604+0000 7f7f8bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.604+0000 7f7f8bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:07.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.844+0000 7f7f99e8c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:09:07.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.844+0000 7f7f99e8c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:09:07.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.844+0000 7f7f99e8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:07.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.844+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7f940a7980 con 0x7f7f9413d710 2024-09-10T16:09:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f7f8bfff640 1 -- 172.21.15.164:0/3439536794 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7f940a7980 con 0x7f7f9413d710 2024-09-10T16:09:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f7f8bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:09:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f7f8bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=503463072,config=2586702516,health=2157093694,kv=1350217199,logm=349104896}) 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=11,kv=10,logm=68} crc {auth=1664314236,config=2586702516,health=1886805112,kv=2192483863,logm=2974862096}) 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=503463072,config=2586702516,health=2157093694,kv=1350217199,logm=349104896}) 2024-09-10T16:09:07.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.846+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=10,config=2,health=11,kv=11,logm=66} crc {auth=201455788,config=2586702516,health=1886805112,kv=3293307581,logm=3275134825}) 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197534291}) 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3756142898}) 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1197534291}) 2024-09-10T16:09:07.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.847+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3707004388}) 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 msgr2=0x7f7f740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 0x7f7f740406d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7f8c009bd0 tx=0x7f7f8c00b040 comp rx=0 tx=0).stop 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 10 monclient: shutdown 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 msgr2=0x7f7f9413dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 0x7f7f9413dbc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7f8400c410 tx=0x7f7f8400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 shutdown_connections 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f7403e210 0x7f7f740406d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f9413d710 0x7f7f9413dbc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f940a40d0 0x7f7f9413d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 --2- 172.21.15.164:0/3439536794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f9406cb70 0x7f7f9409aec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 >> 172.21.15.164:0/3439536794 conn(0x7f7f940a4f50 msgr2=0x7f7f9409e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476182854}) 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=114293897}) 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2476182854}) 2024-09-10T16:09:07.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323684486}) 2024-09-10T16:09:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 shutdown_connections 2024-09-10T16:09:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:07.848+0000 7f7f99e8c640 1 -- 172.21.15.164:0/3439536794 wait complete. 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172953742}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3406939531}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172953742}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.849+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735548801}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211913054}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2433570873}) 2024-09-10T16:09:07.857 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211913054}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.850+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4218613632}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442068462}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4015682603}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2442068462}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.851+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3548275488}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879949856}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970705940}) 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2879949856}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.852+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3189804441}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=2387715987,mdsmap=931935893,mgr=3878148497,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3565528127,mon_config_key=2350018942}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=1354764649,mdsmap=931935893,mgr=2354521040,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1044748599,mon_config_key=1867747704}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=2387715987,mdsmap=931935893,mgr=3878148497,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3565528127,mon_config_key=2350018942}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.853+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=2} crc {logm=2234567494,mdsmap=3140514009,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1044748599,mon_config_key=3427581358}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20,osdmap=55} crc {mon_config_key=1974895890,monmap=1433040830,nvmeofgw=598337686,osd_metadata=633904022,osd_pg_creating=3271782483,osd_snap=1330478176,osdmap=3049253498}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=21,osdmap=54} crc {mon_config_key=1974895890,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3271782483,osd_snap=2230640128,osdmap=1080215545}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=20,osdmap=55} crc {mon_config_key=1974895890,monmap=1433040830,nvmeofgw=598337686,osd_metadata=633904022,osd_pg_creating=3271782483,osd_snap=1330478176,osdmap=3049253498}) 2024-09-10T16:09:07.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.854+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=11,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=22,osdmap=54} crc {mon_config_key=1478468289,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3271782483,osd_snap=2087653371,osdmap=3035948151}) 2024-09-10T16:09:07.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1375273485}) 2024-09-10T16:09:07.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3755780429}) 2024-09-10T16:09:07.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1375273485}) 2024-09-10T16:09:07.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.855+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2408304654}) 2024-09-10T16:09:07.864 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:09:07.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=7} crc {osdmap=4246387398}) 2024-09-10T16:09:07.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=9} crc {osdmap=2888397177}) 2024-09-10T16:09:07.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:07.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=7} crc {osdmap=4246387398}) 2024-09-10T16:09:07.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:07.864+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=9} crc {osdmap=2888397177}) 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:369: done (1 left) 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:370: read oid 15 snap -1 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:370: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:371: snap_create 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:367: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:09:08.170 INFO:tasks.rados.rados.0.smithi165.stdout:367: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:09:08.171 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 229 (ObjNum 97 snap 24 seq_num 97) dirty exists 2024-09-10T16:09:08.186 INFO:tasks.rados.rados.0.smithi165.stdout:370: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:09:08.865 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:09:08.865 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:08.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.965+0000 7f512676b640 10 monclient: get_monmap_and_config 2024-09-10T16:09:08.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.965+0000 7f512676b640 10 monclient: build_initial_monmap 2024-09-10T16:09:08.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.966+0000 7f512676b640 10 monclient: monmap: 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:08.967871+0000 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:08.967871+0000 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:08.969 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.966+0000 7f512676b640 1 Processor -- start 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.966+0000 7f512676b640 1 -- start start 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.966+0000 7f512676b640 10 monclient: init 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5120135ce0 0x7f51200a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): picked mon.noname-a con 0x7f5120135ce0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51200a40d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51201367d0 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51200a4ac0 con 0x7f51201367d0 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51200a1120 con 0x7f51200a40d0 2024-09-10T16:09:08.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f51200a1400 con 0x7f5120135ce0 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f512676b640 10 monclient(hunting): authenticate will time out at 927.455404s 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5120135ce0 0x7f51200a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f5124d08640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f51201367d0 0x7f512013d290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39202/0 (socket says 172.21.15.164:39202) 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5120135ce0 0x7f51200a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60536/0 (socket says 172.21.15.164:60536) 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f5124d08640 1 -- 172.21.15.164:0/2288102813 learned_addr learned my addr 172.21.15.164:0/2288102813 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): get_auth_request con 0x7f5120135ce0 auth_method 0 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:08.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient(hunting): handle_auth_done global_id 24231 payload 274 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 -- 172.21.15.164:0/2288102813 >> v1:172.21.15.165:6789/0 conn(0x7f51200a40d0 legacy=0x7f51201360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 -- 172.21.15.164:0/2288102813 >> v1:172.21.15.164:6789/0 conn(0x7f51201367d0 legacy=0x7f512013d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 monclient: found mon.noname-a 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:08.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.967+0000 7f511ffff640 1 -- 172.21.15.164:0/2288102813 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51200a28e0 con 0x7f5120135ce0 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511ffff640 1 -- 172.21.15.164:0/2288102813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5120135ce0 msgr2=0x7f51200a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511ffff640 1 --2- 172.21.15.164:0/2288102813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5120135ce0 0x7f51200a23a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f5110029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a15584c365cf47 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 1 -- 172.21.15.164:0/2288102813 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110037070 con 0x7f5120135ce0 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: dump: 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:08.981 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:08.982 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: _finish_auth 0 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 1 -- 172.21.15.164:0/2288102813 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5110032640 con 0x7f5120135ce0 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f512676b640 5 monclient: authenticate success, global_id 24231 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f512676b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:08.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 1 -- 172.21.15.164:0/2288102813 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110032a10 con 0x7f5120135ce0 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f512676b640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f512676b640 4 set_mon_vals no callback set 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.970+0000 7f511effd640 10 monclient: dump: 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:08.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 10 monclient: shutdown 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 -- 172.21.15.164:0/2288102813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5120135ce0 msgr2=0x7f51200a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 --2- 172.21.15.164:0/2288102813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5120135ce0 0x7f51200a23a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f5110029fd0 comp rx=0 tx=0).stop 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 -- 172.21.15.164:0/2288102813 shutdown_connections 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 --2- 172.21.15.164:0/2288102813 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5120135ce0 0x7f51200a23a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 -- 172.21.15.164:0/2288102813 >> 172.21.15.164:0/2288102813 conn(0x7f51200a4f50 msgr2=0x7f51200a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 -- 172.21.15.164:0/2288102813 shutdown_connections 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 1 -- 172.21.15.164:0/2288102813 wait complete. 2024-09-10T16:09:08.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 10 monclient: build_initial_monmap 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.971+0000 7f512676b640 10 monclient: monmap: 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 Processor -- start 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 -- start start 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient: init 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51200a40d0 0x7f512009b410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): picked mon.noname-b con 0x7f51200a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5120135ce0 0x7f512009b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): picked mon.noname-c con 0x7f5120135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): picked mon.noname-a con 0x7f51201367d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:08.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f512013ed40 con 0x7f51201367d0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f512013ebc0 con 0x7f51200a40d0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f512013eec0 con 0x7f5120135ce0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.972+0000 7f512676b640 10 monclient(hunting): authenticate will time out at 927.460964s 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60546/0 (socket says 172.21.15.164:60546) 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 -- 172.21.15.164:0/2725910228 learned_addr learned my addr 172.21.15.164:0/2725910228 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5120135ce0 0x7f512009b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): get_auth_request con 0x7f51201367d0 auth_method 0 2024-09-10T16:09:08.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): get_auth_request con 0x7f5120135ce0 auth_method 0 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51200a40d0 0x7f512009b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): get_auth_request con 0x7f51200a40d0 auth_method 0 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:08.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f511ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient(hunting): handle_auth_done global_id 24234 payload 824 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 -- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5120135ce0 msgr2=0x7f512009b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:08.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5120135ce0 0x7f512009b950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 -- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51200a40d0 msgr2=0x7f512009b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51200a40d0 0x7f512009b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient: _finish_hunting 0 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 monclient: found mon.noname-a 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.973+0000 7f5124d08640 1 -- 172.21.15.164:0/2725910228 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51200746d0 con 0x7f51201367d0 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f5124d08640 1 -- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 msgr2=0x7f512006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f5124d08640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f511800cb20 tx=0x7f511800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5118012400 con 0x7f51201367d0 2024-09-10T16:09:08.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: dump: 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.993 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: _finish_auth 0 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51180125c0 con 0x7f51201367d0 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 5 monclient: authenticate success, global_id 24234 2024-09-10T16:09:08.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 10 monclient: _renew_subs 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f512013e290 con 0x7f51201367d0 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51180129b0 con 0x7f51201367d0 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511cff9640 10 monclient: dump: 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:08.995 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 10 monclient: _renew_subs 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f512009be90 con 0x7f51201367d0 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.975+0000 7f511e7fc640 4 set_mon_vals no callback set 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.976+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5118012b90 con 0x7f51201367d0 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.976+0000 7f512676b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:08.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.976+0000 7f512676b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.976+0000 7f512676b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.976+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51200a23a0 con 0x7f51201367d0 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.977+0000 7f511cff9640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 0x7f510c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.977+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 4847+0+0 (secure 0 0 0) 0x7f51180533e0 con 0x7f51201367d0 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.977+0000 7f511ffff640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 0x7f510c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:08.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.977+0000 7f511ffff640 10 monclient: get_auth_request con 0x7f510c03e260 auth_method 0 2024-09-10T16:09:08.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.979+0000 7f511ffff640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 0x7f510c040720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f511002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:08.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.981+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5118051da0 con 0x7f51201367d0 2024-09-10T16:09:08.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.981+0000 7f511cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:08.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:08.981+0000 7f511cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:367: done (2 left) 2024-09-10T16:09:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:370: done (1 left) 2024-09-10T16:09:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:371: done (0 left) 2024-09-10T16:09:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:372: read oid 33 snap -1 2024-09-10T16:09:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:372: expect (ObjNum 141 snap 42 seq_num 141) 2024-09-10T16:09:09.175 INFO:tasks.rados.rados.0.smithi165.stdout:373: write (excl) oid 7 current snap is 47 2024-09-10T16:09:09.175 INFO:tasks.rados.rados.0.smithi165.stdout:373: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-10T16:09:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:373: writing smithi16534841-7 from 689120 to 1172373 tid 1 2024-09-10T16:09:09.189 INFO:tasks.rados.rados.0.smithi165.stdout:373: writing smithi16534841-7 from 1946413 to 2625462 tid 2 2024-09-10T16:09:09.189 INFO:tasks.rados.rados.0.smithi165.stdout:374: read oid 9 snap -1 2024-09-10T16:09:09.189 INFO:tasks.rados.rados.0.smithi165.stdout:374: expect deleted 2024-09-10T16:09:09.189 INFO:tasks.rados.rados.0.smithi165.stdout:375: snap_create 2024-09-10T16:09:09.189 INFO:tasks.rados.rados.0.smithi165.stdout:372: expect (ObjNum 141 snap 42 seq_num 141) 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f512676b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f512676b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f512676b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f51200770e0 con 0x7f51201367d0 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f511cff9640 1 -- 172.21.15.164:0/2725910228 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5118010d10 con 0x7f51201367d0 2024-09-10T16:09:09.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f511cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:09.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.227+0000 7f511cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:09.230 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:09.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 msgr2=0x7f510c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 0x7f510c040720 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5110034000 tx=0x7f511002e000 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 10 monclient: shutdown 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 msgr2=0x7f512006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f511800cb20 tx=0x7f511800a060 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 shutdown_connections 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f510c03e260 0x7f510c040720 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51201367d0 0x7f512006cb70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5120135ce0 0x7f512009b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 --2- 172.21.15.164:0/2725910228 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51200a40d0 0x7f512009b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 >> 172.21.15.164:0/2725910228 conn(0x7f51200a4f50 msgr2=0x7f51200a2d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 shutdown_connections 2024-09-10T16:09:09.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.230+0000 7f512676b640 1 -- 172.21.15.164:0/2725910228 wait complete. 2024-09-10T16:09:09.243 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:09:09.243 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:09:09.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:09:09.243 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.341+0000 7f630332c640 10 monclient: get_monmap_and_config 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.341+0000 7f630332c640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.341+0000 7f630332c640 10 monclient: monmap: 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.343 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:09.342457+0000 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:09.342457+0000 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 Processor -- start 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 -- start start 2024-09-10T16:09:09.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient: init 2024-09-10T16:09:09.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f62fc09ce10 0x7f62fc0a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): picked mon.noname-a con 0x7f62fc09ce10 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62fc0a3450 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f62fc09d900 0x7f62fc13f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): picked mon.noname-c con 0x7f62fc09d900 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62fc0a1c90 con 0x7f62fc0a3450 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc0a22e0 con 0x7f62fc09ce10 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f62fc073ee0 con 0x7f62fc09d900 2024-09-10T16:09:09.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f630332c640 10 monclient(hunting): authenticate will time out at 927.829837s 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f63010c8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f62fc09ce10 0x7f62fc0a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.342+0000 7f63010c8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f62fc09ce10 0x7f62fc0a2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60556/0 (socket says 172.21.15.164:60556) 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 -- 172.21.15.164:0/244010587 learned_addr learned my addr 172.21.15.164:0/244010587 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): get_auth_request con 0x7f62fc09ce10 auth_method 0 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63018c9640 1 --2- 172.21.15.164:0/244010587 >> v2:172.21.15.165:3300/0 conn(0x7f62fc09d900 0x7f62fc13f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63018c9640 10 monclient(hunting): get_auth_request con 0x7f62fc09d900 auth_method 0 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63018c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63018c9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63018c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient(hunting): handle_auth_done global_id 24237 payload 274 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 -- 172.21.15.164:0/244010587 >> v2:172.21.15.165:3300/0 conn(0x7f62fc09d900 msgr2=0x7f62fc13f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 --2- 172.21.15.164:0/244010587 >> v2:172.21.15.165:3300/0 conn(0x7f62fc09d900 0x7f62fc13f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 -- 172.21.15.164:0/244010587 >> v1:172.21.15.165:6789/0 conn(0x7f62fc0a3450 legacy=0x7f62fc09d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 monclient: found mon.noname-a 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.343+0000 7f63010c8640 1 -- 172.21.15.164:0/244010587 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc0a1f70 con 0x7f62fc09ce10 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f63010c8640 1 -- 172.21.15.164:0/244010587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09ce10 msgr2=0x7f62fc0a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f63010c8640 1 --2- 172.21.15.164:0/244010587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09ce10 0x7f62fc0a2f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62fc074060 tx=0x7f62f802a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3804ead8b5c4bf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 1 -- 172.21.15.164:0/244010587 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8037070 con 0x7f62fc09ce10 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: dump: 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.358 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 1 -- 172.21.15.164:0/244010587 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62f8032440 con 0x7f62fc09ce10 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 1 -- 172.21.15.164:0/244010587 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8032810 con 0x7f62fc09ce10 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f62f3fff640 10 monclient: dump: 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.360 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 5 monclient: authenticate success, global_id 24237 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 4 set_mon_vals no callback set 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 10 monclient: shutdown 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 -- 172.21.15.164:0/244010587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09ce10 msgr2=0x7f62fc0a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 --2- 172.21.15.164:0/244010587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09ce10 0x7f62fc0a2f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f62fc074060 tx=0x7f62f802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 -- 172.21.15.164:0/244010587 shutdown_connections 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 --2- 172.21.15.164:0/244010587 >> v2:172.21.15.165:3300/0 conn(0x7f62fc09d900 0x7f62fc13f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 --2- 172.21.15.164:0/244010587 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09ce10 0x7f62fc0a2f10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 -- 172.21.15.164:0/244010587 >> 172.21.15.164:0/244010587 conn(0x7f62fc0a7320 msgr2=0x7f62fc0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 -- 172.21.15.164:0/244010587 shutdown_connections 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.346+0000 7f630332c640 1 -- 172.21.15.164:0/244010587 wait complete. 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.347+0000 7f630332c640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.347+0000 7f630332c640 10 monclient: monmap: 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:09.362 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.347+0000 7f630332c640 1 Processor -- start 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.347+0000 7f630332c640 1 -- start start 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.347+0000 7f630332c640 10 monclient: init 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62fc09ce10 0x7f62fc1ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): picked mon.noname-b con 0x7f62fc09ce10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 0x7f62fc133970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): picked mon.noname-a con 0x7f62fc09d900 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62fc133eb0 0x7f62fc13c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): picked mon.noname-c con 0x7f62fc133eb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc13f990 con 0x7f62fc09d900 2024-09-10T16:09:09.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc077050 con 0x7f62fc09ce10 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f62fc13fb10 con 0x7f62fc133eb0 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f630332c640 10 monclient(hunting): authenticate will time out at 927.835292s 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 0x7f62fc133970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 0x7f62fc133970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60568/0 (socket says 172.21.15.164:60568) 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 -- 172.21.15.164:0/1354331342 learned_addr learned my addr 172.21.15.164:0/1354331342 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62fc133eb0 0x7f62fc13c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): get_auth_request con 0x7f62fc09d900 auth_method 0 2024-09-10T16:09:09.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): get_auth_request con 0x7f62fc133eb0 auth_method 0 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62fc09ce10 0x7f62fc1ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): get_auth_request con 0x7f62fc09ce10 auth_method 0 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient(hunting): handle_auth_done global_id 24240 payload 824 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62fc133eb0 msgr2=0x7f62fc13c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62fc133eb0 0x7f62fc13c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62fc09ce10 msgr2=0x7f62fc1ccff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f62fc09ce10 0x7f62fc1ccff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 monclient: found mon.noname-a 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63008c7640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62fc11d670 con 0x7f62fc09d900 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.348+0000 7f63018c9640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62fc133eb0 0x7f62fc13c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.350+0000 7f63008c7640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 msgr2=0x7f62fc133970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.350+0000 7f63008c7640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 0x7f62fc133970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c430 tx=0x7f62ec00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec0119a0 con 0x7f62fc09d900 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: dump: 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.367 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f62ec017070 con 0x7f62fc09d900 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 5 monclient: authenticate success, global_id 24240 2024-09-10T16:09:09.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 10 monclient: _renew_subs 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62fc11e300 con 0x7f62fc09d900 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62ec011d80 con 0x7f62fc09d900 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f1ffb640 10 monclient: dump: 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.369 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f62f37fe640 4 set_mon_vals no callback set 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 10 monclient: _renew_subs 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.351+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f62fc13de70 con 0x7f62fc09d900 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f630332c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f62ec01c460 con 0x7f62fc09d900 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f630332c640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 0x7f62fc13c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc0a2f10 con 0x7f62fc0a3450 2024-09-10T16:09:09.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 0x7f62fc13c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient: get_auth_request con 0x7f62fc0a3450 auth_method 0 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f62f1ffb640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62e003e210 0x7f62e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 10 monclient(hunting): handle_auth_done global_id 24243 payload 824 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (secure 0 0 0) 0x7f62ec052be0 con 0x7f62fc09d900 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63010c8640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f62fc0008d0 con 0x7f62fc0a3450 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63018c9640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62e003e210 0x7f62e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.352+0000 7f63018c9640 10 monclient: get_auth_request con 0x7f62e003e210 auth_method 0 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f63010c8640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 msgr2=0x7f62fc13c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f63010c8640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 0x7f62fc13c8f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f62f8034000 tx=0x7f62f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8043070 con 0x7f62fc0a3450 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f62f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f62f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f62f1ffb640 10 monclient: dump: 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.372 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.354+0000 7f63018c9640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62e003e210 0x7f62e00406d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62fc1342b0 tx=0x7f62f4009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f62f8048450 con 0x7f62fc0a3450 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 msgr2=0x7f62fc13c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 0x7f62fc13c8f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f62f8034000 tx=0x7f62f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 msgr2=0x7f62fc13c8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.355+0000 7f62f1ffb640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc0a3450 0x7f62fc13c8f0 secure :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f62f8034000 tx=0x7f62f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: finishing write tid 1 to smithi16534841-7 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: finishing write tid 2 to smithi16534841-7 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: finishing write tid 3 to smithi16534841-7 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: finishing write tid 4 to smithi16534841-7 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: oid 7 updating version 0 to 278 2024-09-10T16:09:09.374 INFO:tasks.rados.rados.0.smithi165.stdout:373: oid 7 updating version 278 to 279 2024-09-10T16:09:09.375 INFO:tasks.rados.rados.0.smithi165.stdout:373: oid 7 updating version 279 to 280 2024-09-10T16:09:09.375 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 280 (ObjNum 160 snap 47 seq_num 160) dirty exists 2024-09-10T16:09:09.375 INFO:tasks.rados.rados.0.smithi165.stdout:373: left oid 7 (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:09:09.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f630332c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:09.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f630332c640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 0x7f62fc09fd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f630332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f62fc13d240 con 0x7f62fc13ce30 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 0x7f62fc09fd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient: get_auth_request con 0x7f62fc13ce30 auth_method 0 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 10 monclient(hunting): handle_auth_done global_id 24246 payload 824 2024-09-10T16:09:09.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.361+0000 7f63010c8640 1 -- 172.21.15.164:0/1354331342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f62fc11df80 con 0x7f62fc13ce30 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f63010c8640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 msgr2=0x7f62fc09fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f63010c8640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 0x7f62fc09fd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62f802a0e0 tx=0x7f62f8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62f8037040 con 0x7f62fc13ce30 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 10 monclient: dump: 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.378 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f62f8003810 con 0x7f62fc13ce30 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 msgr2=0x7f62fc09fd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 0x7f62fc09fd00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f62f802a0e0 tx=0x7f62f8008030 comp rx=0 tx=0).stop 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 msgr2=0x7f62fc09fd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.363+0000 7f62f1ffb640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc13ce30 0x7f62fc09fd00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 74296, 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:09.380 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.381 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.382 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:09.383 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.384 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.385 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.386 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.388 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.389 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.390 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.391 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:09.392 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62e003e210 msgr2=0x7f62e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f62e003e210 0x7f62e00406d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f62fc1342b0 tx=0x7f62f4009820 comp rx=0 tx=0).stop 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 10 monclient: shutdown 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 msgr2=0x7f62fc133970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 --2- 172.21.15.164:0/1354331342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f62fc09d900 0x7f62fc133970 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f62ec00c430 tx=0x7f62ec00c8f0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f63018c9640 1 -- 172.21.15.164:0/1354331342 reap_dead start 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f63018c9640 1 -- 172.21.15.164:0/1354331342 reap_dead start 2024-09-10T16:09:09.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 shutdown_connections 2024-09-10T16:09:09.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 >> 172.21.15.164:0/1354331342 conn(0x7f62fc0a7320 msgr2=0x7f62fc0a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 shutdown_connections 2024-09-10T16:09:09.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.364+0000 7f630332c640 1 -- 172.21.15.164:0/1354331342 wait complete. 2024-09-10T16:09:09.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:09.395 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:09:09.395 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:09:09.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.493+0000 7f4eda789640 10 monclient: get_monmap_and_config 2024-09-10T16:09:09.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.493+0000 7f4eda789640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.493+0000 7f4eda789640 10 monclient: monmap: 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:09.494086+0000 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:09.494086+0000 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:09.495 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:09.496 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.493+0000 7f4eda789640 1 Processor -- start 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 -- start start 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient: init 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4ed406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:09.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4ed4134f70 0x7f4ed413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed4134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ed413d560 0x7f4ed413f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed413d560 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ed409ce50 con 0x7f4ed406cb70 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed406db60 con 0x7f4ed413d560 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4ed406cf50 con 0x7f4ed4134f70 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4eda789640 10 monclient(hunting): authenticate will time out at 927.981568s 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4ed413d560 0x7f4ed413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed3fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4ed406cb70 0x7f4ed40a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39210/0 (socket says 172.21.15.164:39210) 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed3fff640 1 -- 172.21.15.164:0/1384452175 learned_addr learned my addr 172.21.15.164:0/1384452175 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): get_auth_request con 0x7f4ed413d560 auth_method 0 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 1 --2- 172.21.15.164:0/1384452175 >> v2:172.21.15.165:3300/0 conn(0x7f4ed4134f70 0x7f4ed413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): get_auth_request con 0x7f4ed4134f70 auth_method 0 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_done global_id 24249 payload 274 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 -- 172.21.15.164:0/1384452175 >> v2:172.21.15.165:3300/0 conn(0x7f4ed4134f70 msgr2=0x7f4ed413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 --2- 172.21.15.164:0/1384452175 >> v2:172.21.15.165:3300/0 conn(0x7f4ed4134f70 0x7f4ed413d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 -- 172.21.15.164:0/1384452175 >> v1:172.21.15.164:6789/0 conn(0x7f4ed406cb70 legacy=0x7f4ed40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 monclient: found mon.noname-a 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.494+0000 7f4ed8d26640 1 -- 172.21.15.164:0/1384452175 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed4135380 con 0x7f4ed413d560 2024-09-10T16:09:09.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed8d26640 1 -- 172.21.15.164:0/1384452175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed413d560 msgr2=0x7f4ed413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed8d26640 1 --2- 172.21.15.164:0/1384452175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed413d560 0x7f4ed413f950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009d90 tx=0x7f4ecc02a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beee98b5491fd601 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 1 -- 172.21.15.164:0/1384452175 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc007950 con 0x7f4ed413d560 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: dump: 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.510 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4eda789640 5 monclient: authenticate success, global_id 24249 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 1 -- 172.21.15.164:0/1384452175 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ecc007af0 con 0x7f4ed413d560 2024-09-10T16:09:09.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4eda789640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 1 -- 172.21.15.164:0/1384452175 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ecc031400 con 0x7f4ed413d560 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4eda789640 4 set_mon_vals no callback set 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.497+0000 7f4ed2ffd640 10 monclient: dump: 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.512 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 10 monclient: shutdown 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 -- 172.21.15.164:0/1384452175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed413d560 msgr2=0x7f4ed413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 --2- 172.21.15.164:0/1384452175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed413d560 0x7f4ed413f950 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009d90 tx=0x7f4ecc02a4e0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 -- 172.21.15.164:0/1384452175 shutdown_connections 2024-09-10T16:09:09.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 --2- 172.21.15.164:0/1384452175 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed413d560 0x7f4ed413f950 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 --2- 172.21.15.164:0/1384452175 >> v2:172.21.15.165:3300/0 conn(0x7f4ed4134f70 0x7f4ed413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 -- 172.21.15.164:0/1384452175 >> 172.21.15.164:0/1384452175 conn(0x7f4ed40aa900 msgr2=0x7f4ed40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 -- 172.21.15.164:0/1384452175 shutdown_connections 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.498+0000 7f4eda789640 1 -- 172.21.15.164:0/1384452175 wait complete. 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.499+0000 7f4eda789640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.499+0000 7f4eda789640 10 monclient: monmap: 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.514 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.499+0000 7f4eda789640 1 Processor -- start 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.499+0000 7f4eda789640 1 -- start start 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.499+0000 7f4eda789640 10 monclient: init 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed406cb70 0x7f4ed41ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-c con 0x7f4ed406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-a con 0x7f4ed4134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ed413d560 0x7f4ed41cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): picked mon.noname-b con 0x7f4ed413d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed40a4db0 con 0x7f4ed4134f70 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4ed409d130 con 0x7f4ed413d560 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed413fe90 con 0x7f4ed406cb70 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4eda789640 10 monclient(hunting): authenticate will time out at 927.987145s 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed406cb70 0x7f4ed41ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60608/0 (socket says 172.21.15.164:60608) 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 -- 172.21.15.164:0/426138719 learned_addr learned my addr 172.21.15.164:0/426138719 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): get_auth_request con 0x7f4ed4134f70 auth_method 0 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): get_auth_request con 0x7f4ed406cb70 auth_method 0 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ed413d560 0x7f4ed41cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): get_auth_request con 0x7f4ed413d560 auth_method 0 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed8d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient(hunting): handle_auth_done global_id 24252 payload 824 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed406cb70 msgr2=0x7f4ed41ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed406cb70 0x7f4ed41ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ed413d560 msgr2=0x7f4ed41cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4ed413d560 0x7f4ed41cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 monclient: found mon.noname-a 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.500+0000 7f4ed37fe640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ed406e720 con 0x7f4ed4134f70 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed37fe640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 msgr2=0x7f4ed41cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed37fe640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800e9b0 tx=0x7f4ec800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8017070 con 0x7f4ed4134f70 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: dump: 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ec800cbb0 con 0x7f4ed4134f70 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4eda789640 5 monclient: authenticate success, global_id 24252 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4eda789640 10 monclient: _renew_subs 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4eda789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ed413b1a0 con 0x7f4ed4134f70 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec8013470 con 0x7f4ed4134f70 2024-09-10T16:09:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed0ff9640 10 monclient: dump: 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.502+0000 7f4ed27fc640 4 set_mon_vals no callback set 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 10 monclient: _renew_subs 2024-09-10T16:09:09.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ed4137ce0 con 0x7f4ed4134f70 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 0x7f4ed412e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed40a46a0 con 0x7f4ed412e020 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed37fe640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800e9b0 tx=0x7f4ec800ee70 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f4ec80531d0 osd_map(105..105 src has 1..105) 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 0x7f4ed412e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4ec801a070 con 0x7f4ed4134f70 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient: get_auth_request con 0x7f4ed412e020 auth_method 0 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.503+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.504+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_done global_id 4640 payload 824 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.504+0000 7f4ed3fff640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4ed40008d0 con 0x7f4ed412e020 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.504+0000 7f4ed0ff9640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4ec003e210 0x7f4ec00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.504+0000 7f4ed8d26640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4ec003e210 0x7f4ec00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.504+0000 7f4ed8d26640 10 monclient: get_auth_request con 0x7f4ec003e210 auth_method 0 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed3fff640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 msgr2=0x7f4ed412e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed3fff640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 0x7f4ed412e400 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e7a0 tx=0x7f4ec400ec60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4017070 con 0x7f4ed412e020 2024-09-10T16:09:09.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 10 monclient: dump: 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ec400ee70 con 0x7f4ed412e020 2024-09-10T16:09:09.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 msgr2=0x7f4ed412e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 0x7f4ed412e400 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f4ec400e7a0 tx=0x7f4ec400ec60 comp rx=0 tx=0).stop 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 msgr2=0x7f4ed412e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed0ff9640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e020 0x7f4ed412e400 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.506+0000 7f4ed8d26640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4ec003e210 0x7f4ec00406d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406e110 tx=0x7f4ecc02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4eda789640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4eda789640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 0x7f4ed4130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4eda789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4ed413a340 con 0x7f4ed412e940 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 0x7f4ed4130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient: get_auth_request con 0x7f4ed412e940 auth_method 0 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.512+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.513+0000 7f4ed3fff640 10 monclient(hunting): handle_auth_done global_id 4643 payload 824 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.513+0000 7f4ed3fff640 1 -- 172.21.15.164:0/426138719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4ed412ed50 con 0x7f4ed412e940 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed3fff640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 msgr2=0x7f4ed4130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed3fff640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 0x7f4ed4130250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4019000 tx=0x7f4ec40058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ec4017050 con 0x7f4ed412e940 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 10 monclient: dump: 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f4ec40066e0 con 0x7f4ed412e940 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 msgr2=0x7f4ed4130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 0x7f4ed4130250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4019000 tx=0x7f4ec40058d0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 msgr2=0x7f4ed4130250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.515+0000 7f4ed0ff9640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4ed412e940 0x7f4ed4130250 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4ec4019000 tx=0x7f4ec40058d0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 191687, 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:09.531 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.532 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.533 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:09.534 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.535 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.536 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:09.537 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.538 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.542 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.543 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:09.544 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.545 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.545 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:09.545 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4ec003e210 msgr2=0x7f4ec00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4ec003e210 0x7f4ec00406d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f4ed406e110 tx=0x7f4ecc02f000 comp rx=0 tx=0).stop 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 10 monclient: shutdown 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 msgr2=0x7f4ed41cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4ec800e9b0 tx=0x7f4ec800ee70 comp rx=0 tx=0).stop 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4ed8d26640 1 -- 172.21.15.164:0/426138719 reap_dead start 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4ed0ff9640 1 -- 172.21.15.164:0/426138719 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (unknown 0 0 0) 0x7f4ec80531d0 con 0x7f4ed4134f70 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 shutdown_connections 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 --2- 172.21.15.164:0/426138719 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4ed4134f70 0x7f4ed41cd4e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 >> 172.21.15.164:0/426138719 conn(0x7f4ed40aa900 msgr2=0x7f4ed40a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 shutdown_connections 2024-09-10T16:09:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.516+0000 7f4eda789640 1 -- 172.21.15.164:0/426138719 wait complete. 2024-09-10T16:09:09.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:09.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:09:09.548 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:09:09.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.644+0000 7f5c146c6640 10 monclient: get_monmap_and_config 2024-09-10T16:09:09.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.644+0000 7f5c146c6640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.644+0000 7f5c146c6640 10 monclient: monmap: 2024-09-10T16:09:09.646 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.646 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:09.645840+0000 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:09.645840+0000 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:09.647 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.645+0000 7f5c146c6640 1 Processor -- start 2024-09-10T16:09:09.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.645+0000 7f5c146c6640 1 -- start start 2024-09-10T16:09:09.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.645+0000 7f5c146c6640 10 monclient: init 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 0x7f5c0c0a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c0c06cf50 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c0c06da40 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0a0190 con 0x7f5c0c06cf50 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c0c0a4960 con 0x7f5c0c06da40 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c06ea00 con 0x7f5c0c06cb70 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c146c6640 10 monclient(hunting): authenticate will time out at 928.133257s 2024-09-10T16:09:09.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c11c61640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5c0c06cf50 0x7f5c0c06d330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39224/0 (socket says 172.21.15.164:39224) 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c11c61640 1 -- 172.21.15.164:0/4019410778 learned_addr learned my addr 172.21.15.164:0/4019410778 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 1 --2- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 0x7f5c0c0a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): get_auth_request con 0x7f5c0c06cb70 auth_method 0 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient(hunting): handle_auth_done global_id 4646 payload 274 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 1 -- 172.21.15.164:0/4019410778 >> v1:172.21.15.165:6789/0 conn(0x7f5c0c06da40 legacy=0x7f5c0c13f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 1 -- 172.21.15.164:0/4019410778 >> v1:172.21.15.164:6789/0 conn(0x7f5c0c06cf50 legacy=0x7f5c0c06d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 1 monclient: found mon.noname-b 2024-09-10T16:09:09.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.646+0000 7f5c12462640 1 -- 172.21.15.164:0/4019410778 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c0a45b0 con 0x7f5c0c06cb70 2024-09-10T16:09:09.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c12462640 1 -- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 msgr2=0x7f5c0c0a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c12462640 1 --2- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 0x7f5c0c0a3fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c080097d0 tx=0x7f5c08029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=396a2819eb35a3b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 1 -- 172.21.15.164:0/4019410778 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08037070 con 0x7f5c0c06cb70 2024-09-10T16:09:09.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: dump: 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 1 -- 172.21.15.164:0/4019410778 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c08032440 con 0x7f5c0c06cb70 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 1 -- 172.21.15.164:0/4019410778 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08032810 con 0x7f5c0c06cb70 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c11460640 10 monclient: dump: 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.660 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c146c6640 5 monclient: authenticate success, global_id 4646 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c146c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.649+0000 7f5c146c6640 4 set_mon_vals no callback set 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 10 monclient: shutdown 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 -- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 msgr2=0x7f5c0c0a3fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 --2- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 0x7f5c0c0a3fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c080097d0 tx=0x7f5c08029fb0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 -- 172.21.15.164:0/4019410778 shutdown_connections 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 --2- 172.21.15.164:0/4019410778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cb70 0x7f5c0c0a3fd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 -- 172.21.15.164:0/4019410778 >> 172.21.15.164:0/4019410778 conn(0x7f5c0c0a4ee0 msgr2=0x7f5c0c0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 -- 172.21.15.164:0/4019410778 shutdown_connections 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 1 -- 172.21.15.164:0/4019410778 wait complete. 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.650+0000 7f5c146c6640 10 monclient: monmap: 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 Processor -- start 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 -- start start 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient: init 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c06cb70 0x7f5c0c13ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c0c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 0x7f5c0c13d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c0c06cf50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c0c06da40 0x7f5c0c13d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c0c06da40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c140e30 con 0x7f5c0c06da40 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c140cb0 con 0x7f5c0c06cb70 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0c140fb0 con 0x7f5c0c06cf50 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c146c6640 10 monclient(hunting): authenticate will time out at 928.138599s 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c0c06da40 0x7f5c0c13d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 0x7f5c0c13d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 0x7f5c0c13d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51488/0 (socket says 172.21.15.164:51488) 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c0c06da40 0x7f5c0c13d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60616/0 (socket says 172.21.15.164:60616) 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 -- 172.21.15.164:0/1980963232 learned_addr learned my addr 172.21.15.164:0/1980963232 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): get_auth_request con 0x7f5c0c06cf50 auth_method 0 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): get_auth_request con 0x7f5c0c06da40 auth_method 0 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c06cb70 0x7f5c0c13ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12c63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): get_auth_request con 0x7f5c0c06cb70 auth_method 0 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient(hunting): handle_auth_done global_id 4649 payload 824 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c06cb70 msgr2=0x7f5c0c13ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c06cb70 0x7f5c0c13ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c0c06da40 msgr2=0x7f5c0c13d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c0c06da40 0x7f5c0c13d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 monclient: found mon.noname-c 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.651+0000 7f5c11c61640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c0a6370 con 0x7f5c0c06cf50 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c11c61640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 msgr2=0x7f5c0c13d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c11c61640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 0x7f5c0c13d210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c460 tx=0x7f5bfc00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc007070 con 0x7f5c0c06cf50 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: dump: 2024-09-10T16:09:09.667 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5bfc007230 con 0x7f5c0c06cf50 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 5 monclient: authenticate success, global_id 4649 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc007620 con 0x7f5c0c06cf50 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c02ffd640 10 monclient: dump: 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.669 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 10 monclient: _renew_subs 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c09f5d0 con 0x7f5c0c06cf50 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c10c5f640 4 set_mon_vals no callback set 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 10 monclient: _renew_subs 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.654+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c13e5c0 con 0x7f5c0c06cf50 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5bfc01b400 con 0x7f5c0c06cf50 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c146c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c146c6640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 0x7f5c0c072140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c0a92f0 con 0x7f5c0c13dc90 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c02ffd640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 0x7f5c0c072140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (secure 0 0 0) 0x7f5bfc0529b0 con 0x7f5c0c06cf50 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient: get_auth_request con 0x7f5c0c13dc90 auth_method 0 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12c63640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12c63640 10 monclient: get_auth_request con 0x7f5bf003e210 auth_method 0 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 10 monclient(hunting): handle_auth_done global_id 34123 payload 824 2024-09-10T16:09:09.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.655+0000 7f5c12462640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c0c13e070 con 0x7f5c0c13dc90 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c12c63640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5c040055e0 tx=0x7f5c04003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c12462640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 msgr2=0x7f5c0c072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c12462640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 0x7f5c0c072140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0a5f80 tx=0x7f5c080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08037040 con 0x7f5c0c13dc90 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 10 monclient: dump: 2024-09-10T16:09:09.674 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c08046640 con 0x7f5c0c13dc90 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 msgr2=0x7f5c0c072140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 0x7f5c0c072140 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0a5f80 tx=0x7f5c080046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 msgr2=0x7f5c0c072140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.658+0000 7f5c02ffd640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c13dc90 0x7f5c0c072140 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c146c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c146c6640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 0x7f5c0c072a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c146c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c0a4050 con 0x7f5c0c072680 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 0x7f5c0c072a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient: get_auth_request con 0x7f5c0c072680 auth_method 0 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 10 monclient(hunting): handle_auth_done global_id 34126 payload 824 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.664+0000 7f5c12462640 1 -- 172.21.15.164:0/1980963232 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c0c1406a0 con 0x7f5c0c072680 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c12462640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 msgr2=0x7f5c0c072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c12462640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 0x7f5c0c072a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c08000c00 tx=0x7f5c08008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08037070 con 0x7f5c0c072680 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 10 monclient: dump: 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f5c0804bc70 con 0x7f5c0c072680 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 msgr2=0x7f5c0c072a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 0x7f5c0c072a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c08000c00 tx=0x7f5c08008030 comp rx=0 tx=0).stop 2024-09-10T16:09:09.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 msgr2=0x7f5c0c072a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.667+0000 7f5c02ffd640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c0c072680 0x7f5c0c072a60 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c08000c00 tx=0x7f5c08008030 comp rx=0 tx=0).stop 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 30, 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7712, 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:09.679 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.680 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.681 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:09.682 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:09.683 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:09.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.685 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.686 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.687 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:09.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.689 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:09.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:09.691 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 msgr2=0x7f5bf00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f5c040055e0 tx=0x7f5c04003d10 comp rx=0 tx=0).stop 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 10 monclient: shutdown 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 msgr2=0x7f5c0c13d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 --2- 172.21.15.164:0/1980963232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c0c06cf50 0x7f5c0c13d210 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc00c460 tx=0x7f5bfc00c920 comp rx=0 tx=0).stop 2024-09-10T16:09:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c12c63640 1 -- 172.21.15.164:0/1980963232 reap_dead start 2024-09-10T16:09:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c12c63640 1 -- 172.21.15.164:0/1980963232 reap_dead start 2024-09-10T16:09:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 shutdown_connections 2024-09-10T16:09:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 >> 172.21.15.164:0/1980963232 conn(0x7f5c0c0a4ee0 msgr2=0x7f5c0c09e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 shutdown_connections 2024-09-10T16:09:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.668+0000 7f5c146c6640 1 -- 172.21.15.164:0/1980963232 wait complete. 2024-09-10T16:09:09.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:09.694 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:09:09.694 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:09:09.694 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:09:09.695 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:09:09.695 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:09:09.695 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:09:09.704 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:09:09.704 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:09:09.704 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:09.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.800+0000 7fafbf216640 10 monclient: get_monmap_and_config 2024-09-10T16:09:09.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.800+0000 7fafbf216640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.800+0000 7fafbf216640 10 monclient: monmap: 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:09.801136+0000 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:09.801136+0000 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:09.802 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.800+0000 7fafbf216640 1 Processor -- start 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 -- start start 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient: init 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafb809c180 0x7fafb809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-a con 0x7fafb809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-b con 0x7fafb809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 0x7fafb813b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-c con 0x7fafb809f4c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fafb809a430 con 0x7fafb809c180 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fafb80a24e0 con 0x7fafb809f4c0 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fafb80a2bf0 con 0x7fafb809d160 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbf216640 10 monclient(hunting): authenticate will time out at 928.288562s 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafb809c180 0x7fafb809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbd7b3640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 msgr2=0x7fafb813b5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51492/0 (socket says 172.21.15.164:51492) 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafb809c180 0x7fafb809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60624/0 (socket says 172.21.15.164:60624) 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 1 -- 172.21.15.164:0/1796904165 learned_addr learned my addr 172.21.15.164:0/1796904165 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): get_auth_request con 0x7fafb809d160 auth_method 0 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbd7b3640 1 --2- 172.21.15.164:0/1796904165 >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 0x7fafb813b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): get_auth_request con 0x7fafb809c180 auth_method 0 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient(hunting): handle_auth_done global_id 24258 payload 274 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 -- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 msgr2=0x7fafb809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 -- 172.21.15.164:0/1796904165 >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 msgr2=0x7fafb813b5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 --2- 172.21.15.164:0/1796904165 >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 0x7fafb813b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.801+0000 7fafbcfb2640 1 monclient: found mon.noname-a 2024-09-10T16:09:09.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.802+0000 7fafbcfb2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.802+0000 7fafbcfb2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:09.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.802+0000 7fafbcfb2640 1 -- 172.21.15.164:0/1796904165 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb809c560 con 0x7fafb809c180 2024-09-10T16:09:09.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.802+0000 7fafb7fff640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:09.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafbcfb2640 1 -- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809c180 msgr2=0x7fafb809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafbcfb2640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809c180 0x7fafb809ea40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafb809d5b0 tx=0x7fafa802e390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc80ffb020245b22 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafb77fe640 1 -- 172.21.15.164:0/1796904165 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa800b650 con 0x7fafb809c180 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafb77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.804+0000 7fafb77fe640 10 monclient: dump: 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.815 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 1 -- 172.21.15.164:0/1796904165 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa800b7f0 con 0x7fafb809c180 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 5 monclient: authenticate success, global_id 24258 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 1 -- 172.21.15.164:0/1796904165 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa800bbc0 con 0x7fafb809c180 2024-09-10T16:09:09.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 4 set_mon_vals no callback set 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafb77fe640 10 monclient: dump: 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.817 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 10 monclient: shutdown 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 -- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809c180 msgr2=0x7fafb809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809c180 0x7fafb809ea40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fafb809d5b0 tx=0x7fafa802e390 comp rx=0 tx=0).stop 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 -- 172.21.15.164:0/1796904165 shutdown_connections 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 --2- 172.21.15.164:0/1796904165 >> v2:172.21.15.165:3300/0 conn(0x7fafb809f4c0 0x7fafb813b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 --2- 172.21.15.164:0/1796904165 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809c180 0x7fafb809ea40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 -- 172.21.15.164:0/1796904165 >> 172.21.15.164:0/1796904165 conn(0x7fafb80a3110 msgr2=0x7fafb80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 -- 172.21.15.164:0/1796904165 shutdown_connections 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.805+0000 7fafbf216640 1 -- 172.21.15.164:0/1796904165 wait complete. 2024-09-10T16:09:09.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.806+0000 7fafbf216640 10 monclient: build_initial_monmap 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.806+0000 7fafbf216640 10 monclient: monmap: 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.806+0000 7fafbf216640 1 Processor -- start 2024-09-10T16:09:09.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.806+0000 7fafbf216640 1 -- start start 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.806+0000 7fafbf216640 10 monclient: init 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 0x7fafb81b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-b con 0x7fafb809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-c con 0x7fafb809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 0x7fafb81ba500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): picked mon.noname-a con 0x7fafb809f4c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8074f90 con 0x7fafb809f4c0 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fafb8074e10 con 0x7fafb809c180 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fafb809d930 con 0x7fafb809d160 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbf216640 10 monclient(hunting): authenticate will time out at 928.294116s 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbcfb2640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 msgr2=0x7fafb81b2160 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbcfb2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 0x7fafb81b2160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 0x7fafb81ba500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51506/0 (socket says 172.21.15.164:51506) 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 -- 172.21.15.164:0/1654411819 learned_addr learned my addr 172.21.15.164:0/1654411819 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): get_auth_request con 0x7fafb809d160 auth_method 0 2024-09-10T16:09:09.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): get_auth_request con 0x7fafb809f4c0 auth_method 0 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:09.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient(hunting): handle_auth_done global_id 4655 payload 824 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 -- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 msgr2=0x7fafb81b2160 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 0x7fafb81b2160 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 -- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 msgr2=0x7fafb81ba500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 0x7fafb81ba500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 monclient: found mon.noname-c 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafb7fff640 1 -- 172.21.15.164:0/1654411819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb81b39e0 con 0x7fafb809d160 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.807+0000 7fafbd7b3640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 0x7fafb81ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb7fff640 1 -- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 msgr2=0x7fafb81b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:09.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb7fff640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fafa000fde0 tx=0x7fafa000d720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa001d070 con 0x7fafb809d160 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: dump: 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.824 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: _finish_auth 0 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafa0010a60 con 0x7fafb809d160 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafbf216640 5 monclient: authenticate success, global_id 4655 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafbf216640 10 monclient: _renew_subs 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafbf216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafb81b3cc0 con 0x7fafb809d160 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafa0010e10 con 0x7fafb809d160 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:09.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb57fa640 10 monclient: dump: 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.810+0000 7fafbf216640 10 monclient: _renew_subs 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.810+0000 7fafbf216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.809+0000 7fafb6ffd640 4 set_mon_vals no callback set 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.810+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fafb81baa40 con 0x7fafb809d160 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbf216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbf216640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbf216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafb809dd10 con 0x7fafb809d160 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fafa0020070 con 0x7fafb809d160 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafb57fa640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 0x7faf980406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (secure 0 0 0) 0x7fafa0053410 con 0x7fafb809d160 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbcfb2640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 0x7faf980406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:09.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.811+0000 7fafbcfb2640 10 monclient: get_auth_request con 0x7faf9803e1f0 auth_method 0 2024-09-10T16:09:09.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.814+0000 7fafbcfb2640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 0x7faf980406b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafb806d4a0 tx=0x7fafa8032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:09.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.815+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fafa000be20 con 0x7fafb809d160 2024-09-10T16:09:09.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.815+0000 7fafb57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:09.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:09.815+0000 7fafb57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:10.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.057+0000 7fafbf216640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:10.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafbf216640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafbf216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fafb81b3e90 con 0x7fafb809d160 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafb57fa640 1 -- 172.21.15.164:0/1654411819 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fafa0021da0 con 0x7fafb809d160 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafb57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.058+0000 7fafb57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:10.060 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 msgr2=0x7faf980406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 0x7faf980406b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fafb806d4a0 tx=0x7fafa8032000 comp rx=0 tx=0).stop 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 10 monclient: shutdown 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 msgr2=0x7fafb81b0970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fafa000fde0 tx=0x7fafa000d720 comp rx=0 tx=0).stop 2024-09-10T16:09:10.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 shutdown_connections 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf9803e1f0 0x7faf980406b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafb809f4c0 0x7fafb81ba500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafb809d160 0x7fafb81b0970 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 --2- 172.21.15.164:0/1654411819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafb809c180 0x7fafb81b2160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 >> 172.21.15.164:0/1654411819 conn(0x7fafb80a3110 msgr2=0x7fafb81bc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 shutdown_connections 2024-09-10T16:09:10.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:10.060+0000 7fafbf216640 1 -- 172.21.15.164:0/1654411819 wait complete. 2024-09-10T16:09:13.073 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.180+0000 7f4e9ed9f640 10 monclient: get_monmap_and_config 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.180+0000 7f4e9ed9f640 10 monclient: build_initial_monmap 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.180+0000 7f4e9ed9f640 10 monclient: monmap: 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:13.181551+0000 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:13.181551+0000 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.182 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:13.183 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:13.183 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:13.183 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:13.183 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:13.183 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 Processor -- start 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 -- start start 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient: init 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e981354c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e980a5ff0 0x7f4e9813cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:13.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e980a5ff0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e9813d530 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e9806e180 con 0x7f4e981354c0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e98135090 con 0x7f4e9813d530 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4e980a63d0 con 0x7f4e980a5ff0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.181+0000 7f4e9ed9f640 10 monclient(hunting): authenticate will time out at 931.668997s 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e980a5ff0 0x7f4e9813cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9dd9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4e981354c0 0x7f4e9809d450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39228/0 (socket says 172.21.15.164:39228) 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9e59e640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f4e9813d530 legacy=0x7f4e9813f920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e980a5ff0 0x7f4e9813cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60642/0 (socket says 172.21.15.164:60642) 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/3790958891 learned_addr learned my addr 172.21.15.164:0/3790958891 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:13.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): get_auth_request con 0x7f4e980a5ff0 auth_method 0 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_done global_id 24264 payload 274 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 -- 172.21.15.164:0/3790958891 >> v1:172.21.15.165:6789/0 conn(0x7f4e9813d530 legacy=0x7f4e9813f920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 -- 172.21.15.164:0/3790958891 >> v1:172.21.15.164:6789/0 conn(0x7f4e981354c0 legacy=0x7f4e9809d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient: _finish_hunting 0 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 monclient: found mon.noname-a 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.182+0000 7f4e9d59c640 1 -- 172.21.15.164:0/3790958891 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9806e460 con 0x7f4e980a5ff0 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9d59c640 1 -- 172.21.15.164:0/3790958891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 msgr2=0x7f4e9813cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9d59c640 1 --2- 172.21.15.164:0/3790958891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e9813cff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e88029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc5f8e68d981b3c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 1 -- 172.21.15.164:0/3790958891 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88037070 con 0x7f4e980a5ff0 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: dump: 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:13.193 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: _finish_auth 0 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 1 -- 172.21.15.164:0/3790958891 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e88032640 con 0x7f4e980a5ff0 2024-09-10T16:09:13.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9ed9f640 5 monclient: authenticate success, global_id 24264 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9ed9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9ed9f640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 1 -- 172.21.15.164:0/3790958891 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88032a10 con 0x7f4e980a5ff0 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9ed9f640 4 set_mon_vals no callback set 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.185+0000 7f4e9cd9b640 10 monclient: dump: 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 10 monclient: shutdown 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/3790958891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 msgr2=0x7f4e9813cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/3790958891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e9813cff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f4e880097a0 tx=0x7f4e88029fd0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/3790958891 shutdown_connections 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/3790958891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e9813cff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/3790958891 >> 172.21.15.164:0/3790958891 conn(0x7f4e980a70e0 msgr2=0x7f4e980a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/3790958891 shutdown_connections 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/3790958891 wait complete. 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 10 monclient: build_initial_monmap 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.186+0000 7f4e9ed9f640 10 monclient: monmap: 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:13.197 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 Processor -- start 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 -- start start 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient: init 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-a con 0x7f4e980a5ff0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e981354c0 0x7f4e9809a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-c con 0x7f4e981354c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 0x7f4e9809f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): picked mon.noname-b con 0x7f4e9813d530 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e98134d80 con 0x7f4e980a5ff0 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4e9806e7d0 con 0x7f4e9813d530 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e9809dec0 con 0x7f4e981354c0 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9ed9f640 10 monclient(hunting): authenticate will time out at 931.674741s 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e981354c0 0x7f4e9809a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60658/0 (socket says 172.21.15.164:60658) 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/2137529817 learned_addr learned my addr 172.21.15.164:0/2137529817 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): get_auth_request con 0x7f4e980a5ff0 auth_method 0 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): get_auth_request con 0x7f4e981354c0 auth_method 0 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9e59e640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 msgr2=0x7f4e9809f530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9e59e640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 0x7f4e9809f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9dd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.187+0000 7f4e9d59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 10 monclient(hunting): handle_auth_done global_id 24267 payload 824 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e981354c0 msgr2=0x7f4e9809a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e981354c0 0x7f4e9809a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 msgr2=0x7f4e9809f530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 0x7f4e9809f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 10 monclient: _finish_hunting 0 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 monclient: found mon.noname-a 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.188+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/2137529817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e980a0ea0 con 0x7f4e980a5ff0 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9dd9d640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 msgr2=0x7f4e980a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9dd9d640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400b430 tx=0x7f4e9400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94015020 con 0x7f4e980a5ff0 2024-09-10T16:09:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: dump: 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:13.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 5 monclient: authenticate success, global_id 24267 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e94011a60 con 0x7f4e980a5ff0 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e94011e50 con 0x7f4e980a5ff0 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 10 monclient: _renew_subs 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e980a3660 con 0x7f4e980a5ff0 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8e7fc640 10 monclient: dump: 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.203 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e8ffff640 4 set_mon_vals no callback set 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 10 monclient: _renew_subs 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.190+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e980a2dc0 con 0x7f4e980a5ff0 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4e94018020 con 0x7f4e980a5ff0 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e9ed9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e9ed9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:13.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e9ed9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9809d450 con 0x7f4e980a5ff0 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e8e7fc640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 0x7f4e7c040850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.191+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (secure 0 0 0) 0x7f4e94052650 con 0x7f4e980a5ff0 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.192+0000 7f4e9d59c640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 0x7f4e7c040850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.192+0000 7f4e9d59c640 10 monclient: get_auth_request con 0x7f4e7c03e390 auth_method 0 2024-09-10T16:09:13.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.194+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:09:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.194+0000 7f4e9d59c640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 0x7f4e7c040850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e98075230 tx=0x7f4e8802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.196+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4e9401f070 con 0x7f4e980a5ff0 2024-09-10T16:09:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.196+0000 7f4e8e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:13.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.196+0000 7f4e8e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e9ed9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e9ed9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e9ed9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4e98076eb0 con 0x7f4e980a5ff0 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e8e7fc640 1 -- 172.21.15.164:0/2137529817 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f4e94051020 con 0x7f4e980a5ff0 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e8e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:13.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.439+0000 7f4e8e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:13.442 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 msgr2=0x7f4e7c040850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 0x7f4e7c040850 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4e98075230 tx=0x7f4e8802e000 comp rx=0 tx=0).stop 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 10 monclient: shutdown 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 msgr2=0x7f4e980a3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400b430 tx=0x7f4e9400b8f0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 shutdown_connections 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e7c03e390 0x7f4e7c040850 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e9813d530 0x7f4e9809f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e981354c0 0x7f4e9809a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 --2- 172.21.15.164:0/2137529817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e980a5ff0 0x7f4e980a3df0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:13.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 >> 172.21.15.164:0/2137529817 conn(0x7f4e980a70e0 msgr2=0x7f4e9813f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:13.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 shutdown_connections 2024-09-10T16:09:13.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:13.442+0000 7f4e9ed9f640 1 -- 172.21.15.164:0/2137529817 wait complete. 2024-09-10T16:09:16.455 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.563+0000 7f285e310640 10 monclient: get_monmap_and_config 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.563+0000 7f285e310640 10 monclient: build_initial_monmap 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.563+0000 7f285e310640 10 monclient: monmap: 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:16.564574+0000 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:16.564574+0000 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:16.565 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:16.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.564+0000 7f285e310640 1 Processor -- start 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.564+0000 7f285e310640 1 -- start start 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.564+0000 7f285e310640 10 monclient: init 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.564+0000 7f285e310640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f285809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 0x7f285809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): picked mon.noname-c con 0x7f285809d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:16.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f285813a5c0 0x7f285813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): picked mon.noname-a con 0x7f285813a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:16.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f285809fbc0 con 0x7f285809c310 2024-09-10T16:09:16.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:16.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f28580a2a20 con 0x7f285813a5c0 2024-09-10T16:09:16.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f285809c6f0 con 0x7f285809d190 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285e310640 10 monclient(hunting): authenticate will time out at 935.052120s 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f2857fff640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f285809c310 legacy=0x7f285809ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f28577fe640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 msgr2=0x7f285809f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f285813a5c0 0x7f285813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:16.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f28577fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 0x7f285809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f285813a5c0 0x7f285813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55568/0 (socket says 172.21.15.164:55568) 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 -- 172.21.15.164:0/3744540973 learned_addr learned my addr 172.21.15.164:0/3744540973 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): get_auth_request con 0x7f285813a5c0 auth_method 0 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:16.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient(hunting): handle_auth_done global_id 24270 payload 274 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 -- 172.21.15.164:0/3744540973 >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 msgr2=0x7f285809f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 --2- 172.21.15.164:0/3744540973 >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 0x7f285809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 -- 172.21.15.164:0/3744540973 >> v1:172.21.15.165:6789/0 conn(0x7f285809c310 legacy=0x7f285809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient: _finish_hunting 0 2024-09-10T16:09:16.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 monclient: found mon.noname-a 2024-09-10T16:09:16.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:16.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.565+0000 7f285c8ad640 1 -- 172.21.15.164:0/3744540973 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f285809a4d0 con 0x7f285813a5c0 2024-09-10T16:09:16.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285c8ad640 1 -- 172.21.15.164:0/3744540973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f285813a5c0 msgr2=0x7f285813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:16.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285c8ad640 1 --2- 172.21.15.164:0/3744540973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f285813a5c0 0x7f285813c9b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f28580a0280 tx=0x7f285002a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bc37778cce94 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:16.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 1 -- 172.21.15.164:0/3744540973 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850036070 con 0x7f285813a5c0 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: dump: 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:16.582 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 1 -- 172.21.15.164:0/3744540973 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2850007800 con 0x7f285813a5c0 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285e310640 5 monclient: authenticate success, global_id 24270 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285e310640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285e310640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 1 -- 172.21.15.164:0/3744540973 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850007bd0 con 0x7f285813a5c0 2024-09-10T16:09:16.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f285e310640 4 set_mon_vals no callback set 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.568+0000 7f2856ffd640 10 monclient: dump: 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:16.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 10 monclient: shutdown 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 -- 172.21.15.164:0/3744540973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f285813a5c0 msgr2=0x7f285813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 --2- 172.21.15.164:0/3744540973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f285813a5c0 0x7f285813c9b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f28580a0280 tx=0x7f285002a590 comp rx=0 tx=0).stop 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 -- 172.21.15.164:0/3744540973 shutdown_connections 2024-09-10T16:09:16.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 --2- 172.21.15.164:0/3744540973 >> v2:172.21.15.165:3300/0 conn(0x7f285809d190 0x7f285809f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 --2- 172.21.15.164:0/3744540973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f285813a5c0 0x7f285813c9b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 -- 172.21.15.164:0/3744540973 >> 172.21.15.164:0/3744540973 conn(0x7f28580a3110 msgr2=0x7f28580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 -- 172.21.15.164:0/3744540973 shutdown_connections 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 1 -- 172.21.15.164:0/3744540973 wait complete. 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 10 monclient: build_initial_monmap 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.569+0000 7f285e310640 10 monclient: monmap: 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:16.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 1 Processor -- start 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 1 -- start start 2024-09-10T16:09:16.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 10 monclient: init 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 10 monclient(hunting): picked mon.noname-c con 0x7f285809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.570+0000 7f285e310640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 0x7f2858090210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): picked mon.noname-b con 0x7f285813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): picked mon.noname-a con 0x7f2858090750 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:16.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f28580a2230 con 0x7f2858090750 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f28580a20b0 con 0x7f285813a5c0 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2858074e00 con 0x7f285809c310 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285e310640 10 monclient(hunting): authenticate will time out at 935.058053s 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:16.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37750/0 (socket says 172.21.15.164:37750) 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55580/0 (socket says 172.21.15.164:55580) 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 1 -- 172.21.15.164:0/221501797 learned_addr learned my addr 172.21.15.164:0/221501797 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): get_auth_request con 0x7f285809c310 auth_method 0 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f28577fe640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 msgr2=0x7f2858090210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:16.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): get_auth_request con 0x7f2858090750 auth_method 0 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f28577fe640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 0x7f2858090210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:16.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient(hunting): handle_auth_done global_id 24273 payload 824 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 msgr2=0x7f2858139d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 msgr2=0x7f2858090210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 0x7f2858090210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient: _finish_hunting 0 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 monclient: found mon.noname-a 2024-09-10T16:09:16.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f285c8ad640 1 -- 172.21.15.164:0/221501797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2858093910 con 0x7f2858090750 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.571+0000 7f2857fff640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285c8ad640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 msgr2=0x7f2858090be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285c8ad640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2850030000 tx=0x7f2850004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850041020 con 0x7f2858090750 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: dump: 2024-09-10T16:09:16.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:16.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: _finish_auth 0 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285e310640 5 monclient: authenticate success, global_id 24273 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2850031940 con 0x7f2858090750 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2850036040 con 0x7f2858090750 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285e310640 10 monclient: _renew_subs 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285e310640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2858093bf0 con 0x7f2858090750 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f2854ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.574+0000 7f2854ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.574+0000 7f2854ff9640 10 monclient: dump: 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:16.597 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.573+0000 7f28567fc640 4 set_mon_vals no callback set 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.574+0000 7f285e310640 10 monclient: _renew_subs 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.574+0000 7f285e310640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.574+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2858091b70 con 0x7f2858090750 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2850044020 con 0x7f2858090750 2024-09-10T16:09:16.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f285e310640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f285e310640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f285e310640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f285809f7b0 con 0x7f2858090750 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f2854ff9640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 0x7f28440406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(105..105 src has 1..105) ==== 4819+0+0 (secure 0 0 0) 0x7f2850073be0 con 0x7f2858090750 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f2857fff640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 0x7f28440406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.575+0000 7f2857fff640 10 monclient: get_auth_request con 0x7f284403e1f0 auth_method 0 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.578+0000 7f2857fff640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 0x7f28440406b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2848009930 tx=0x7f2848003b40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:16.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.580+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f285004b030 con 0x7f2858090750 2024-09-10T16:09:16.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.580+0000 7f2854ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:16.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.580+0000 7f2854ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.822+0000 7f285e310640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.822+0000 7f285e310640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.822+0000 7f285e310640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.822+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2858073610 con 0x7f2858090750 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.823+0000 7f2854ff9640 1 -- 172.21.15.164:0/221501797 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f285809fbc0 con 0x7f2858090750 2024-09-10T16:09:16.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.823+0000 7f2854ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:16.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.823+0000 7f2854ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:16.825 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 msgr2=0x7f28440406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 0x7f28440406b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2848009930 tx=0x7f2848003b40 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 10 monclient: shutdown 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 msgr2=0x7f2858090be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2850030000 tx=0x7f2850004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 shutdown_connections 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f284403e1f0 0x7f28440406b0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2858090750 0x7f2858090be0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f285813a5c0 0x7f2858090210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 --2- 172.21.15.164:0/221501797 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f285809c310 0x7f2858139d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:16.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 >> 172.21.15.164:0/221501797 conn(0x7f28580a3110 msgr2=0x7f28580707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:16.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 shutdown_connections 2024-09-10T16:09:16.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:16.825+0000 7f285e310640 1 -- 172.21.15.164:0/221501797 wait complete. 2024-09-10T16:09:19.840 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.948+0000 7f1800833640 10 monclient: get_monmap_and_config 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.948+0000 7f1800833640 10 monclient: build_initial_monmap 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.948+0000 7f1800833640 10 monclient: monmap: 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:19.949664+0000 2024-09-10T16:09:19.950 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:19.949664+0000 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:19.951 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.949+0000 7f1800833640 1 Processor -- start 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.949+0000 7f1800833640 1 -- start start 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.949+0000 7f1800833640 10 monclient: init 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): picked mon.noname-b con 0x7f17fc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): picked mon.noname-c con 0x7f17fc09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): picked mon.noname-a con 0x7f17fc09f4c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:19.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f17fc09a430 con 0x7f17fc09f4c0 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f17fc0a24e0 con 0x7f17fc09d160 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17fc0a2bf0 con 0x7f17fc09c180 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f1800833640 10 monclient(hunting): authenticate will time out at 938.437164s 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17f9d9b640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 msgr2=0x7f17fc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:19.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55590/0 (socket says 172.21.15.164:55590) 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37760/0 (socket says 172.21.15.164:37760) 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 learned_addr learned my addr 172.21.15.164:0/1003383011 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17fc09f4c0 auth_method 0 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 10 monclient(hunting): get_auth_request con 0x7f17fc09c180 auth_method 0 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:19.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:19.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:19.950+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.151+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 msgr2=0x7f17fc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.151+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17fc09f4c0 auth_method 0 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:20.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 10 monclient(hunting): get_auth_request con 0x7f17fc09c180 auth_method 0 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:20.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:09:20.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.152+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 msgr2=0x7f17fc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17fc09f4c0 auth_method 0 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:20.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.553+0000 7f17fa59c640 10 monclient(hunting): get_auth_request con 0x7f17fc09c180 auth_method 0 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:20.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:20.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:20.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:20.554+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.354+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 msgr2=0x7f17fc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.354+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.354+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17fc09f4c0 auth_method 0 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:21.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:21.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:21.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:21.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.355+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:09:21.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 10 monclient(hunting): get_auth_request con 0x7f17fc09c180 auth_method 0 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:21.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:21.356+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): tick 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 msgr2=0x7f17fc09ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17fc09c180 0x7f17fc09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 msgr2=0x7f17fc09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:22.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.165:3300/0 conn(0x7f17fc09d160 0x7f17fc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 -- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 msgr2=0x7f17fc13b5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 --2- 172.21.15.164:0/1003383011 >> v2:172.21.15.164:3300/0 conn(0x7f17fc09f4c0 0x7f17fc13b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient: _add_conns ranks=[0,1] 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f17dc003690 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17dc007340 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 -- 172.21.15.164:0/1003383011 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17fc09c560 con 0x7f17dc003690 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 1 -- 172.21.15.164:0/1003383011 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17dc00c060 con 0x7f17dc007340 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17ebfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:22.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.950+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.165:6789/0 conn(0x7f17dc007340 legacy=0x7f17dc0097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:09:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:22.952+0000 7f17fa59c640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.164:6789/0 conn(0x7f17dc003690 legacy=0x7f17dc005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:23.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:23.151+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.165:6789/0 conn(0x7f17dc007340 legacy=0x7f17dc0097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:09:23.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:23.552+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.165:6789/0 conn(0x7f17dc007340 legacy=0x7f17dc0097e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:09:24.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.180+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1699879504 0 0) 0x7f17fc09c560 con 0x7f17dc003690 2024-09-10T16:09:24.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.180+0000 7f17f959a640 10 monclient(hunting): _negotiate 2024-09-10T16:09:24.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.180+0000 7f17f959a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.180+0000 7f17f959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.180+0000 7f17f959a640 10 monclient(hunting): my global_id is 24276 2024-09-10T16:09:24.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e4003520 con 0x7f17dc003690 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2041716453 0 0) 0x7f17e4003520 con 0x7f17dc003690 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.165:6789/0 conn(0x7f17dc007340 legacy=0x7f17dc0097e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17dc00d260 con 0x7f17dc003690 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f17f000d680 con 0x7f17dc003690 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f1800833640 5 monclient: authenticate success, global_id 24276 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f1800833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:09:24.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.181+0000 7f17f959a640 10 monclient: dump: 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.187 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 >> v1:172.21.15.164:6789/0 conn(0x7f17dc003690 legacy=0x7f17dc005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 reap_dead start 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4003a50 0x7f17e4003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): picked mon.a con 0x7f17e4003a50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 0x7f17e4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): picked mon.b con 0x7f17e4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17e400ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): picked mon.c con 0x7f17e40051c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17e4007bd0 con 0x7f17e4003a50 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f17e4007d80 con 0x7f17e4004800 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17e4007f80 con 0x7f17e40051c0 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f959a640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4003a50 0x7f17e4003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f1800833640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): get_auth_request con 0x7f17e4003a50 auth_method 0 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17e400ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17e40051c0 auth_method 0 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:24.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 msgr2=0x7f17e4004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:24.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 0x7f17e4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:24.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 10 monclient(hunting): handle_auth_done global_id 24276 payload 293 2024-09-10T16:09:24.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 msgr2=0x7f17e4004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.182+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 0x7f17e4004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4003a50 msgr2=0x7f17e4003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4003a50 0x7f17e4003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 1 monclient: found mon.c 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17e4003520 con 0x7f17e40051c0 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.183+0000 7f17fad9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.186+0000 7f17fad9d640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 msgr2=0x7f17e400ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.186+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17e400ffd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f17fc09a720 tx=0x7f17e4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3577c910c0848c0f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f4009700 con 0x7f17e40051c0 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: dump: 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.200 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f1800833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f402c720 con 0x7f17e40051c0 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 1 -- 172.21.15.164:0/1003383011 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f402cb10 con 0x7f17e40051c0 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f1800833640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f1800833640 4 set_mon_vals no callback set 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.189+0000 7f17f959a640 10 monclient: dump: 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.201 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 10 monclient: shutdown 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 -- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 msgr2=0x7f17e400ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17e400ffd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f17fc09a720 tx=0x7f17e4005830 comp rx=0 tx=0).stop 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 -- 172.21.15.164:0/1003383011 shutdown_connections 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17e400ffd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4004800 0x7f17e4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 --2- 172.21.15.164:0/1003383011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4003a50 0x7f17e4003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.190+0000 7f1800833640 1 -- 172.21.15.164:0/1003383011 >> 172.21.15.164:0/1003383011 conn(0x7f17fc0a3110 msgr2=0x7f17fc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.191+0000 7f1800833640 1 -- 172.21.15.164:0/1003383011 shutdown_connections 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.191+0000 7f1800833640 1 -- 172.21.15.164:0/1003383011 wait complete. 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.192+0000 7f1800833640 10 monclient: build_initial_monmap 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.192+0000 7f1800833640 10 monclient: monmap: 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:24.203 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 Processor -- start 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 -- start start 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient: init 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 0x7f17fc1bc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): picked mon.noname-b con 0x7f17e4003a50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): picked mon.noname-a con 0x7f17e4004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17fc08fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): picked mon.noname-c con 0x7f17e40051c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc074f90 con 0x7f17e4004800 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f17fc074e10 con 0x7f17e4003a50 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17fc09d930 con 0x7f17e40051c0 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f1800833640 10 monclient(hunting): authenticate will time out at 942.680612s 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fa59c640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 msgr2=0x7f17fc1bc530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fa59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 0x7f17fc1bc530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17fc08fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55662/0 (socket says 172.21.15.164:55662) 2024-09-10T16:09:24.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1622690931 learned_addr learned my addr 172.21.15.164:0/1622690931 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): get_auth_request con 0x7f17e4004800 auth_method 0 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): get_auth_request con 0x7f17e40051c0 auth_method 0 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.193+0000 7f17f9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 10 monclient(hunting): handle_auth_done global_id 24279 payload 824 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 msgr2=0x7f17fc08fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17fc08fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 msgr2=0x7f17fc1bc530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 0x7f17fc1bc530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 monclient: found mon.noname-a 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1622690931 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc09d620 con 0x7f17e4004800 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.194+0000 7f17fad9d640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17fc08fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17f9d9b640 1 -- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 msgr2=0x7f17fc1bca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17f9d9b640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00b920 tx=0x7f17ec00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec00cde0 con 0x7f17e4004800 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: dump: 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 5 monclient: authenticate success, global_id 24279 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17ec010030 con 0x7f17e4004800 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec015660 con 0x7f17e4004800 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 10 monclient: _renew_subs 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17fc0917a0 con 0x7f17e4004800 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17f8d99640 4 set_mon_vals no callback set 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f17eaffd640 10 monclient: dump: 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 10 monclient: _renew_subs 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.196+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17fc1bcfb0 con 0x7f17e4004800 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f1800833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f1800833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f1800833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f17ec0143d0 con 0x7f17e4004800 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17fc09dcc0 con 0x7f17e4004800 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.197+0000 7f17eaffd640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 0x7f17dc040eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.198+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4831+0+0 (secure 0 0 0) 0x7f17ec052880 con 0x7f17e4004800 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.198+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 0x7f17dc040eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.198+0000 7f17fa59c640 10 monclient: get_auth_request con 0x7f17dc03e9f0 auth_method 0 2024-09-10T16:09:24.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.200+0000 7f17fa59c640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 0x7f17dc040eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17f000d310 tx=0x7f17f0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.202+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f17ec051250 con 0x7f17e4004800 2024-09-10T16:09:24.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.202+0000 7f17eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.202+0000 7f17eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:24.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.448+0000 7f1800833640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:24.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.448+0000 7f1800833640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:24.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.448+0000 7f1800833640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.448+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f17fc093310 con 0x7f17e4004800 2024-09-10T16:09:24.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.449+0000 7f17eaffd640 1 -- 172.21.15.164:0/1622690931 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f17ec01d350 con 0x7f17e4004800 2024-09-10T16:09:24.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.449+0000 7f17eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:24.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.449+0000 7f17eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:24.451 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:24.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 msgr2=0x7f17dc040eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 0x7f17dc040eb0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f17f000d310 tx=0x7f17f0008040 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 10 monclient: shutdown 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 msgr2=0x7f17fc1bca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f17ec00b920 tx=0x7f17ec00bde0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 shutdown_connections 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17dc03e9f0 0x7f17dc040eb0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17e40051c0 0x7f17fc08fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17e4004800 0x7f17fc1bca70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 --2- 172.21.15.164:0/1622690931 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17e4003a50 0x7f17fc1bc530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 >> 172.21.15.164:0/1622690931 conn(0x7f17fc0a3110 msgr2=0x7f17fc0707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 shutdown_connections 2024-09-10T16:09:24.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.451+0000 7f1800833640 1 -- 172.21.15.164:0/1622690931 wait complete. 2024-09-10T16:09:24.463 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:09:24.463 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:09:24.464 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:09:24.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 10 monclient: get_monmap_and_config 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 10 monclient: build_initial_monmap 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 10 monclient: monmap: 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:24.562911+0000 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:24.562911+0000 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:24.564 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 1 Processor -- start 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 1 -- start start 2024-09-10T16:09:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.562+0000 7f3d45516640 10 monclient: init 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d40134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-b con 0x7f3d4006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 0x7f3d400a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-c con 0x7f3d4006d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d400a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d4009cb90 con 0x7f3d400a4c20 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3d4009ce70 con 0x7f3d4006d540 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d4009d150 con 0x7f3d4006d160 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d45516640 10 monclient(hunting): authenticate will time out at 943.050215s 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d40134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3d400a4c20 0x7f3d40137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:50554/0 (socket says 172.21.15.164:50554) 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3678841206 learned_addr learned my addr 172.21.15.164:0/3678841206 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3e7fc640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 msgr2=0x7f3d400a46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): get_auth_request con 0x7f3d4006d160 auth_method 0 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3e7fc640 1 --2- 172.21.15.164:0/3678841206 >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 0x7f3d400a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient(hunting): handle_auth_done global_id 4667 payload 274 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 -- 172.21.15.164:0/3678841206 >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 msgr2=0x7f3d400a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3678841206 >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 0x7f3d400a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 -- 172.21.15.164:0/3678841206 >> v1:172.21.15.164:6789/0 conn(0x7f3d400a4c20 legacy=0x7f3d40137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 monclient: found mon.noname-b 2024-09-10T16:09:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.563+0000 7f3d3effd640 1 -- 172.21.15.164:0/3678841206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40135340 con 0x7f3d4006d160 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3effd640 1 -- 172.21.15.164:0/3678841206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 msgr2=0x7f3d40134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3678841206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d40134e00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009d90 tx=0x7f3d3802a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2977a2c434e0067d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 1 -- 172.21.15.164:0/3678841206 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38007a30 con 0x7f3d4006d160 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: dump: 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 1 -- 172.21.15.164:0/3678841206 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d38007bd0 con 0x7f3d4006d160 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d45516640 5 monclient: authenticate success, global_id 4667 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d45516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 1 -- 172.21.15.164:0/3678841206 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d38031400 con 0x7f3d4006d160 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d45516640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d45516640 4 set_mon_vals no callback set 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.566+0000 7f3d3dffb640 10 monclient: dump: 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.574 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 10 monclient: shutdown 2024-09-10T16:09:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 -- 172.21.15.164:0/3678841206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 msgr2=0x7f3d40134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 --2- 172.21.15.164:0/3678841206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d40134e00 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3d38009d90 tx=0x7f3d3802a4b0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 -- 172.21.15.164:0/3678841206 shutdown_connections 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 --2- 172.21.15.164:0/3678841206 >> v2:172.21.15.165:3300/0 conn(0x7f3d4006d540 0x7f3d400a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 --2- 172.21.15.164:0/3678841206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d40134e00 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 -- 172.21.15.164:0/3678841206 >> 172.21.15.164:0/3678841206 conn(0x7f3d400aacb0 msgr2=0x7f3d400ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 -- 172.21.15.164:0/3678841206 shutdown_connections 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 1 -- 172.21.15.164:0/3678841206 wait complete. 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 10 monclient: build_initial_monmap 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.567+0000 7f3d45516640 10 monclient: monmap: 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:24.581 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 Processor -- start 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 -- start start 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient: init 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:24.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d400a2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-c con 0x7f3d4006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d4006d540 0x7f3d400a31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-b con 0x7f3d4006d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 0x7f3d400a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): picked mon.noname-a con 0x7f3d400a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40138830 con 0x7f3d400a4c20 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d401386b0 con 0x7f3d4006d540 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d401389b0 con 0x7f3d4006d160 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d45516640 10 monclient(hunting): authenticate will time out at 943.055744s 2024-09-10T16:09:24.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d400a2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3e7fc640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d4006d540 msgr2=0x7f3d400a31c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d400a2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36342/0 (socket says 172.21.15.164:36342) 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 0x7f3d400a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 1 -- 172.21.15.164:0/3070718597 learned_addr learned my addr 172.21.15.164:0/3070718597 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): get_auth_request con 0x7f3d4006d160 auth_method 0 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3e7fc640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d4006d540 0x7f3d400a31c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f3d400a4c20 auth_method 0 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.568+0000 7f3d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 10 monclient(hunting): handle_auth_done global_id 24282 payload 824 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 msgr2=0x7f3d400a2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d400a2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d4006d540 msgr2=0x7f3d400a31c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:24.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d4006d540 0x7f3d400a31c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40138cf0 con 0x7f3d400a4c20 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.569+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d4006d160 0x7f3d400a2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 msgr2=0x7f3d400a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d3f7fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 0x7f3d400a3700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3d4006d9a0 tx=0x7f3d3000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30017070 con 0x7f3d400a4c20 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: dump: 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d300070d0 con 0x7f3d400a4c20 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 5 monclient: authenticate success, global_id 24282 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d300074c0 con 0x7f3d400a4c20 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 10 monclient: _renew_subs 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d4006e6b0 con 0x7f3d400a4c20 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d377fe640 10 monclient: dump: 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d3d7fa640 4 set_mon_vals no callback set 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 10 monclient: _renew_subs 2024-09-10T16:09:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.571+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d4013a630 con 0x7f3d400a4c20 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d45516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d45516640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 0x7f3d401cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d4013ac30 con 0x7f3d401cd440 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3d3001a070 con 0x7f3d400a4c20 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 0x7f3d401cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient: get_auth_request con 0x7f3d401cd440 auth_method 0 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 10 monclient(hunting): handle_auth_done global_id 24285 payload 824 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d3effd640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d40134e00 con 0x7f3d401cd440 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d377fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d2003e1c0 0x7f3d20040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.572+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4831+0+0 (secure 0 0 0) 0x7f3d30052090 con 0x7f3d400a4c20 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.573+0000 7f3d3e7fc640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d2003e1c0 0x7f3d20040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.573+0000 7f3d3e7fc640 10 monclient: get_auth_request con 0x7f3d2003e1c0 auth_method 0 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.574+0000 7f3d3effd640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 msgr2=0x7f3d401cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.574+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 0x7f3d401cd820 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3d3802a4f0 tx=0x7f3d38004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3803c620 con 0x7f3d401cd440 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 10 monclient: dump: 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d3803c620 con 0x7f3d401cd440 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 msgr2=0x7f3d401cd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 0x7f3d401cd820 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3d3802a4f0 tx=0x7f3d38004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 msgr2=0x7f3d401cd820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:24.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d377fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cd440 0x7f3d401cd820 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f3d3802a4f0 tx=0x7f3d38004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:24.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.575+0000 7f3d3e7fc640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d2003e1c0 0x7f3d20040680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d400775f0 tx=0x7f3d28006d50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d45516640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:24.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d45516640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 0x7f3d4009d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d45516640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d400aa1c0 con 0x7f3d401cdd60 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 0x7f3d4009d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient: get_auth_request con 0x7f3d401cdd60 auth_method 0 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 10 monclient(hunting): handle_auth_done global_id 24288 payload 824 2024-09-10T16:09:24.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.581+0000 7f3d3effd640 1 -- 172.21.15.164:0/3070718597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d400770b0 con 0x7f3d401cdd60 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d3effd640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 msgr2=0x7f3d4009d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d3effd640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 0x7f3d4009d670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d38043000 tx=0x7f3d38030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d3804a040 con 0x7f3d401cdd60 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 10 monclient: dump: 2024-09-10T16:09:24.596 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f3d3803ce40 con 0x7f3d401cdd60 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:24.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 msgr2=0x7f3d4009d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 0x7f3d4009d670 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d38043000 tx=0x7f3d38030000 comp rx=0 tx=0).stop 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 msgr2=0x7f3d4009d670 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d377fe640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d401cdd60 0x7f3d4009d670 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f3d38043000 tx=0x7f3d38030000 comp rx=0 tx=0).stop 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 34, 2024-09-10T16:09:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 89517, 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.606 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.610 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.611 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:24.611 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:24.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d2003e1c0 msgr2=0x7f3d20040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d2003e1c0 0x7f3d20040680 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f3d400775f0 tx=0x7f3d28006d50 comp rx=0 tx=0).stop 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 10 monclient: shutdown 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 msgr2=0x7f3d400a3700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d3f7fe640 1 -- 172.21.15.164:0/3070718597 reap_dead start 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 0x7f3d400a3700 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3d4006d9a0 tx=0x7f3d3000c7b0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 shutdown_connections 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 --2- 172.21.15.164:0/3070718597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d400a4c20 0x7f3d400a3700 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 >> 172.21.15.164:0/3070718597 conn(0x7f3d400aacb0 msgr2=0x7f3d401379b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 shutdown_connections 2024-09-10T16:09:24.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.584+0000 7f3d45516640 1 -- 172.21.15.164:0/3070718597 wait complete. 2024-09-10T16:09:24.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:09:24.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:09:24.614 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.711+0000 7f6475df8640 10 monclient: get_monmap_and_config 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.711+0000 7f6475df8640 10 monclient: build_initial_monmap 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.711+0000 7f6475df8640 10 monclient: monmap: 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:24.712404+0000 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:24.712404+0000 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:24.714 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:24.714 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:24.714 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:24.714 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:24.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 Processor -- start 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 -- start start 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient: init 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f6470137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-b con 0x7f6470074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f64700a1090 0x7f64700a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-a con 0x7f64700a1090 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64700a2ec0 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f647009ebc0 con 0x7f64700a2ec0 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f64700698a0 con 0x7f64700a1090 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f647006bfb0 con 0x7f6470074af0 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f6475df8640 10 monclient(hunting): authenticate will time out at 943.199780s 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f64700a1090 0x7f64700a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f64700a2ec0 0x7f64700a3330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:50558/0 (socket says 172.21.15.164:50558) 2024-09-10T16:09:24.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646ffff640 1 -- 172.21.15.164:0/2048547579 learned_addr learned my addr 172.21.15.164:0/2048547579 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646effd640 10 monclient(hunting): get_auth_request con 0x7f64700a1090 auth_method 0 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646f7fe640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f6470137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.712+0000 7f646f7fe640 10 monclient(hunting): get_auth_request con 0x7f6470074af0 auth_method 0 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient(hunting): handle_auth_done global_id 24291 payload 274 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 1 -- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 msgr2=0x7f6470137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f6470137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 1 -- 172.21.15.164:0/2048547579 >> v1:172.21.15.164:6789/0 conn(0x7f64700a2ec0 legacy=0x7f64700a3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 1 monclient: found mon.noname-a 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.713+0000 7f646effd640 1 -- 172.21.15.164:0/2048547579 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64701383b0 con 0x7f64700a1090 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.715+0000 7f646effd640 1 -- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 msgr2=0x7f64700a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.715+0000 7f646effd640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f64700a14d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f64640097a0 tx=0x7f6464029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47352688c8164c98 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 1 -- 172.21.15.164:0/2048547579 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464037070 con 0x7f64700a1090 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: dump: 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 5 monclient: authenticate success, global_id 24291 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 1 -- 172.21.15.164:0/2048547579 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6464032440 con 0x7f64700a1090 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 4 set_mon_vals no callback set 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 1 -- 172.21.15.164:0/2048547579 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464032810 con 0x7f64700a1090 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f646e7fc640 10 monclient: dump: 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 10 monclient: shutdown 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 -- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 msgr2=0x7f64700a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f64700a14d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f64640097a0 tx=0x7f6464029fd0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 -- 172.21.15.164:0/2048547579 shutdown_connections 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f64700a14d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 --2- 172.21.15.164:0/2048547579 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f6470137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 -- 172.21.15.164:0/2048547579 >> 172.21.15.164:0/2048547579 conn(0x7f64700a3fb0 msgr2=0x7f64700a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 -- 172.21.15.164:0/2048547579 shutdown_connections 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.716+0000 7f6475df8640 1 -- 172.21.15.164:0/2048547579 wait complete. 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.717+0000 7f6475df8640 10 monclient: build_initial_monmap 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.717+0000 7f6475df8640 10 monclient: monmap: 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.717+0000 7f6475df8640 1 Processor -- start 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.717+0000 7f6475df8640 1 -- start start 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.717+0000 7f6475df8640 10 monclient: init 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f64701cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-c con 0x7f6470074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f6470131930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-a con 0x7f64700a1090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64700a2ec0 0x7f6470131e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): picked mon.noname-b con 0x7f64700a2ec0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f64700a3c50 con 0x7f64700a1090 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6470068d70 con 0x7f64700a2ec0 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f64700a7110 con 0x7f6470074af0 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f6475df8640 10 monclient(hunting): authenticate will time out at 943.205255s 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f6470131930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f64701cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f6470131930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:40744/0 (socket says 172.21.15.164:40744) 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 -- 172.21.15.164:0/2518811558 learned_addr learned my addr 172.21.15.164:0/2518811558 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646ffff640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64700a2ec0 msgr2=0x7f6470131e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:09:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): get_auth_request con 0x7f64700a1090 auth_method 0 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): get_auth_request con 0x7f6470074af0 auth_method 0 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646ffff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64700a2ec0 0x7f6470131e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient(hunting): handle_auth_done global_id 24294 payload 824 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 msgr2=0x7f64701cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6470074af0 0x7f64701cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64700a2ec0 msgr2=0x7f6470131e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64700a2ec0 0x7f6470131e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 monclient: found mon.noname-a 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.718+0000 7f646effd640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64700a0d30 con 0x7f64700a1090 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f646effd640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 msgr2=0x7f6470131930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f646effd640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f6470131930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6464000c00 tx=0x7f64640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6464037040 con 0x7f64700a1090 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: dump: 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6464042070 con 0x7f64700a1090 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6475df8640 5 monclient: authenticate success, global_id 24294 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6475df8640 10 monclient: _renew_subs 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f646403bb00 con 0x7f64700a1090 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6475df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f647009e6a0 con 0x7f64700a1090 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6457fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6457fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6457fff640 10 monclient: dump: 2024-09-10T16:09:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.720+0000 7f646dffb640 4 set_mon_vals no callback set 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 10 monclient: _renew_subs 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64701323b0 con 0x7f64700a1090 2024-09-10T16:09:24.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 0x7f64701be2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.721+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f64700a1a10 con 0x7f64701bded0 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6464047460 con 0x7f64700a1090 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 0x7f64701be2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient: get_auth_request con 0x7f64701bded0 auth_method 0 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 10 monclient(hunting): handle_auth_done global_id 4679 payload 824 2024-09-10T16:09:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f6457fff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f645c03e1c0 0x7f645c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646f7fe640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64701c18e0 con 0x7f64701bded0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(106..106 src has 1..106) ==== 4831+0+0 (secure 0 0 0) 0x7f64640734e0 con 0x7f64700a1090 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646ffff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f645c03e1c0 0x7f645c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.722+0000 7f646ffff640 10 monclient: get_auth_request con 0x7f645c03e1c0 auth_method 0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f646f7fe640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 msgr2=0x7f64701be2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f646f7fe640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 0x7f64701be2b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6460011890 tx=0x7f6460011d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460019070 con 0x7f64701bded0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 10 monclient: dump: 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f646000f890 con 0x7f64701bded0 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 msgr2=0x7f64701be2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 0x7f64701be2b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6460011890 tx=0x7f6460011d50 comp rx=0 tx=0).stop 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 msgr2=0x7f64701be2b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.724+0000 7f6457fff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701bded0 0x7f64701be2b0 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f6460011890 tx=0x7f6460011d50 comp rx=0 tx=0).stop 2024-09-10T16:09:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.725+0000 7f646ffff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f645c03e1c0 0x7f645c040680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6468004b90 tx=0x7f6468005080 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.730+0000 7f6475df8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.730+0000 7f6475df8640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 0x7f647009b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.730+0000 7f6475df8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.730+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6470137df0 con 0x7f64701be7f0 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 0x7f647009b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient: get_auth_request con 0x7f64701be7f0 auth_method 0 2024-09-10T16:09:24.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 10 monclient(hunting): handle_auth_done global_id 4682 payload 824 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.731+0000 7f646f7fe640 1 -- 172.21.15.164:0/2518811558 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64700772c0 con 0x7f64701be7f0 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f646f7fe640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 msgr2=0x7f647009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f646f7fe640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 0x7f647009b3b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f646001b000 tx=0x7f64600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6460019040 con 0x7f64701be7f0 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 10 monclient: dump: 2024-09-10T16:09:24.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f646001e450 con 0x7f64701be7f0 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:24.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 msgr2=0x7f647009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 0x7f647009b3b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f646001b000 tx=0x7f64600046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 msgr2=0x7f647009b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.733+0000 7f6457fff640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64701be7f0 0x7f647009b3b0 secure :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f646001b000 tx=0x7f64600046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 34, 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 206905, 2024-09-10T16:09:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.756 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:24.757 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:24.758 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:24.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f645c03e1c0 msgr2=0x7f645c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f645c03e1c0 0x7f645c040680 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f6468004b90 tx=0x7f6468005080 comp rx=0 tx=0).stop 2024-09-10T16:09:24.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 10 monclient: shutdown 2024-09-10T16:09:24.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:24.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 msgr2=0x7f6470131930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 --2- 172.21.15.164:0/2518811558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64700a1090 0x7f6470131930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6464000c00 tx=0x7f64640046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f646ffff640 1 -- 172.21.15.164:0/2518811558 reap_dead start 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f646ffff640 1 -- 172.21.15.164:0/2518811558 reap_dead start 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 shutdown_connections 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 >> 172.21.15.164:0/2518811558 conn(0x7f64700a3fb0 msgr2=0x7f64701328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 shutdown_connections 2024-09-10T16:09:24.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:24.734+0000 7f6475df8640 1 -- 172.21.15.164:0/2518811558 wait complete. 2024-09-10T16:09:24.760 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:09:24.760 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:09:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:372: done (3 left) 2024-09-10T16:09:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:373: done (2 left) 2024-09-10T16:09:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:374: done (1 left) 2024-09-10T16:09:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:375: done (0 left) 2024-09-10T16:09:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:376: copy_from oid 45 from oid 4 current snap is 48 2024-09-10T16:09:25.089 INFO:tasks.rados.rados.0.smithi165.stdout:377: snap_create 2024-09-10T16:09:25.100 INFO:tasks.rados.rados.0.smithi165.stdout:376: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:09:25.104 INFO:tasks.rados.rados.0.smithi165.stdout:376: finishing copy_from to smithi16534841-45 2024-09-10T16:09:25.104 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 67 (ObjNum 132 snap 37 seq_num 132) dirty exists 2024-09-10T16:09:26.745 INFO:tasks.rados.rados.0.smithi165.stdout:376: done (1 left) 2024-09-10T16:09:26.745 INFO:tasks.rados.rados.0.smithi165.stdout:377: done (0 left) 2024-09-10T16:09:26.745 INFO:tasks.rados.rados.0.smithi165.stdout:378: snap_create 2024-09-10T16:09:27.188 INFO:tasks.rados.rados.0.smithi165.stdout:378: done (0 left) 2024-09-10T16:09:27.189 INFO:tasks.rados.rados.0.smithi165.stdout:379: read oid 37 snap -1 2024-09-10T16:09:27.189 INFO:tasks.rados.rados.0.smithi165.stdout:379: expect deleted 2024-09-10T16:09:27.189 INFO:tasks.rados.rados.0.smithi165.stdout:380: rollback oid 3 current snap is 50 2024-09-10T16:09:27.189 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 33 2024-09-10T16:09:27.189 INFO:tasks.rados.rados.0.smithi165.stdout:381: delete oid 47 current snap is 50 2024-09-10T16:09:27.193 INFO:tasks.rados.rados.0.smithi165.stdout:379: done (2 left) 2024-09-10T16:09:27.193 INFO:tasks.rados.rados.0.smithi165.stdout:381: done (1 left) 2024-09-10T16:09:27.193 INFO:tasks.rados.rados.0.smithi165.stdout:382: snap_create 2024-09-10T16:09:27.194 INFO:tasks.rados.rados.0.smithi165.stdout:380: finishing rollback tid 0 to smithi16534841-3 2024-09-10T16:09:27.195 INFO:tasks.rados.rados.0.smithi165.stdout:380: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:09:27.198 INFO:tasks.rados.rados.0.smithi165.stdout:380: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:09:27.198 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 238 (ObjNum 62 snap 5 seq_num 62) dirty exists 2024-09-10T16:09:28.191 INFO:tasks.rados.rados.0.smithi165.stdout:380: done (1 left) 2024-09-10T16:09:28.191 INFO:tasks.rados.rados.0.smithi165.stdout:382: done (0 left) 2024-09-10T16:09:28.191 INFO:tasks.rados.rados.0.smithi165.stdout:383: delete oid 1 current snap is 51 2024-09-10T16:09:28.195 INFO:tasks.rados.rados.0.smithi165.stdout:383: done (0 left) 2024-09-10T16:09:28.195 INFO:tasks.rados.rados.0.smithi165.stdout:384: delete oid 26 current snap is 51 2024-09-10T16:09:28.198 INFO:tasks.rados.rados.0.smithi165.stdout:384: done (0 left) 2024-09-10T16:09:28.198 INFO:tasks.rados.rados.0.smithi165.stdout:385: copy_from oid 6 from oid 22 current snap is 51 2024-09-10T16:09:28.199 INFO:tasks.rados.rados.0.smithi165.stdout:386: rollback oid 9 current snap is 51 2024-09-10T16:09:28.199 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 44 2024-09-10T16:09:28.199 INFO:tasks.rados.rados.0.smithi165.stdout:387: write oid 48 current snap is 51 2024-09-10T16:09:28.199 INFO:tasks.rados.rados.0.smithi165.stdout:387: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-10T16:09:28.207 INFO:tasks.rados.rados.0.smithi165.stdout:387: writing smithi16534841-48 from 756348 to 1540597 tid 1 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:387: writing smithi16534841-48 from 2063988 to 2750375 tid 2 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:388: copy_from oid 5 from oid 44 current snap is 51 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:385: finishing copy_from to smithi16534841-6 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:385: got expected ENOENT (src dne) 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:385: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:386: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:09:28.214 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 41 (ObjNum 129 snap 0 seq_num 1162599888) dirty dne 2024-09-10T16:09:28.215 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing write tid 1 to smithi16534841-48 2024-09-10T16:09:28.215 INFO:tasks.rados.rados.0.smithi165.stdout:385: done (3 left) 2024-09-10T16:09:28.215 INFO:tasks.rados.rados.0.smithi165.stdout:386: done (2 left) 2024-09-10T16:09:28.215 INFO:tasks.rados.rados.0.smithi165.stdout:389: snap_create 2024-09-10T16:09:28.215 INFO:tasks.rados.rados.0.smithi165.stdout:388: finishing copy_from to smithi16534841-5 2024-09-10T16:09:28.216 INFO:tasks.rados.rados.0.smithi165.stdout:388: got expected ENOENT (src dne) 2024-09-10T16:09:28.216 INFO:tasks.rados.rados.0.smithi165.stdout:388: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:09:28.217 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing write tid 2 to smithi16534841-48 2024-09-10T16:09:28.220 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing write tid 3 to smithi16534841-48 2024-09-10T16:09:28.220 INFO:tasks.rados.rados.0.smithi165.stdout:387: finishing write tid 4 to smithi16534841-48 2024-09-10T16:09:28.220 INFO:tasks.rados.rados.0.smithi165.stdout:387: oid 48 updating version 0 to 285 2024-09-10T16:09:28.220 INFO:tasks.rados.rados.0.smithi165.stdout:387: oid 48 updating version 285 to 286 2024-09-10T16:09:28.220 INFO:tasks.rados.rados.0.smithi165.stdout:387: oid 48 version 286 is already newer than 284 2024-09-10T16:09:28.221 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 286 (ObjNum 164 snap 51 seq_num 164) dirty exists 2024-09-10T16:09:28.221 INFO:tasks.rados.rados.0.smithi165.stdout:387: left oid 48 (ObjNum 164 snap 51 seq_num 164) 2024-09-10T16:09:29.192 INFO:tasks.rados.rados.0.smithi165.stdout:387: done (2 left) 2024-09-10T16:09:29.192 INFO:tasks.rados.rados.0.smithi165.stdout:388: done (1 left) 2024-09-10T16:09:29.192 INFO:tasks.rados.rados.0.smithi165.stdout:389: done (0 left) 2024-09-10T16:09:29.192 INFO:tasks.rados.rados.0.smithi165.stdout:390: write oid 18 current snap is 52 2024-09-10T16:09:29.192 INFO:tasks.rados.rados.0.smithi165.stdout:390: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-10T16:09:29.198 INFO:tasks.rados.rados.0.smithi165.stdout:390: writing smithi16534841-18 from 773155 to 1232653 tid 1 2024-09-10T16:09:29.203 INFO:tasks.rados.rados.0.smithi165.stdout:390: writing smithi16534841-18 from 1722470 to 2152515 tid 2 2024-09-10T16:09:29.207 INFO:tasks.rados.rados.0.smithi165.stdout:390: writing smithi16534841-18 from 2648676 to 2773155 tid 3 2024-09-10T16:09:29.207 INFO:tasks.rados.rados.0.smithi165.stdout:391: delete oid 36 current snap is 52 2024-09-10T16:09:29.207 INFO:tasks.rados.rados.0.smithi165.stdout:390: finishing write tid 1 to smithi16534841-18 2024-09-10T16:09:29.207 INFO:tasks.rados.rados.0.smithi165.stdout:390: finishing write tid 2 to smithi16534841-18 2024-09-10T16:09:29.209 INFO:tasks.rados.rados.0.smithi165.stdout:390: finishing write tid 3 to smithi16534841-18 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: finishing write tid 4 to smithi16534841-18 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: finishing write tid 5 to smithi16534841-18 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: oid 18 updating version 0 to 237 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: oid 18 updating version 237 to 238 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: oid 18 updating version 238 to 239 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: oid 18 updating version 239 to 240 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 240 (ObjNum 165 snap 52 seq_num 165) dirty exists 2024-09-10T16:09:29.210 INFO:tasks.rados.rados.0.smithi165.stdout:390: left oid 18 (ObjNum 165 snap 52 seq_num 165) 2024-09-10T16:09:29.211 INFO:tasks.rados.rados.0.smithi165.stdout:390: done (1 left) 2024-09-10T16:09:29.211 INFO:tasks.rados.rados.0.smithi165.stdout:391: done (0 left) 2024-09-10T16:09:29.211 INFO:tasks.rados.rados.0.smithi165.stdout:392: rollback oid 9 current snap is 52 2024-09-10T16:09:29.211 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 35 2024-09-10T16:09:29.211 INFO:tasks.rados.rados.0.smithi165.stdout:393: delete oid 36 current snap is 52 2024-09-10T16:09:29.213 INFO:tasks.rados.rados.0.smithi165.stdout:393: done (1 left) 2024-09-10T16:09:29.213 INFO:tasks.rados.rados.0.smithi165.stdout:394: write oid 44 current snap is 52 2024-09-10T16:09:29.213 INFO:tasks.rados.rados.0.smithi165.stdout:394: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-10T16:09:29.217 INFO:tasks.rados.rados.0.smithi165.stdout:394: writing smithi16534841-44 from 423576 to 825174 tid 1 2024-09-10T16:09:29.225 INFO:tasks.rados.rados.0.smithi165.stdout:394: writing smithi16534841-44 from 1497916 to 2191641 tid 2 2024-09-10T16:09:29.225 INFO:tasks.rados.rados.0.smithi165.stdout:395: snap_remove snap 49 2024-09-10T16:09:30.090 INFO:tasks.rados.rados.0.smithi165.stdout:395: done (2 left) 2024-09-10T16:09:30.091 INFO:tasks.rados.rados.0.smithi165.stdout:396: write (excl) oid 30 current snap is 52 2024-09-10T16:09:30.091 INFO:tasks.rados.rados.0.smithi165.stdout:396: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-10T16:09:30.096 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16534841-30 from 440383 to 917230 tid 1 2024-09-10T16:09:30.102 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16534841-30 from 1556398 to 1993781 tid 2 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:396: writing smithi16534841-30 from 2474721 to 2840383 tid 3 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:392: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 41 (ObjNum 177 snap 0 seq_num 326582889) dirty dne 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:394: finishing write tid 1 to smithi16534841-44 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:394: finishing write tid 2 to smithi16534841-44 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:397: read oid 1 snap -1 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:397: expect deleted 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:392: done (3 left) 2024-09-10T16:09:30.107 INFO:tasks.rados.rados.0.smithi165.stdout:398: write oid 14 current snap is 52 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: finishing write tid 3 to smithi16534841-44 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: finishing write tid 4 to smithi16534841-44 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: oid 44 updating version 0 to 118 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: oid 44 updating version 118 to 119 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: oid 44 version 119 is already newer than 117 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 119 (ObjNum 168 snap 52 seq_num 168) dirty exists 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:394: left oid 44 (ObjNum 168 snap 52 seq_num 168) 2024-09-10T16:09:30.108 INFO:tasks.rados.rados.0.smithi165.stdout:398: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2024-09-10T16:09:30.112 INFO:tasks.rados.rados.0.smithi165.stdout:398: writing smithi16534841-14 from 457190 to 1009286 tid 1 2024-09-10T16:09:30.119 INFO:tasks.rados.rados.0.smithi165.stdout:398: writing smithi16534841-14 from 1614880 to 2195921 tid 2 2024-09-10T16:09:30.123 INFO:tasks.rados.rados.0.smithi165.stdout:398: writing smithi16534841-14 from 2785791 to 2857190 tid 3 2024-09-10T16:09:30.123 INFO:tasks.rados.rados.0.smithi165.stdout:394: done (3 left) 2024-09-10T16:09:30.123 INFO:tasks.rados.rados.0.smithi165.stdout:399: write oid 28 current snap is 52 2024-09-10T16:09:30.123 INFO:tasks.rados.rados.0.smithi165.stdout:399: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-10T16:09:30.128 INFO:tasks.rados.rados.0.smithi165.stdout:399: writing smithi16534841-28 from 473997 to 1101342 tid 1 2024-09-10T16:09:30.134 INFO:tasks.rados.rados.0.smithi165.stdout:399: writing smithi16534841-28 from 1673362 to 2114414 tid 2 2024-09-10T16:09:30.138 INFO:tasks.rados.rados.0.smithi165.stdout:399: writing smithi16534841-28 from 2529567 to 2873997 tid 3 2024-09-10T16:09:30.138 INFO:tasks.rados.rados.0.smithi165.stdout:400: write oid 6 current snap is 52 2024-09-10T16:09:30.139 INFO:tasks.rados.rados.0.smithi165.stdout:400: seq_num 172 ranges {490804=702594,1615491=584710,2724284=166520} 2024-09-10T16:09:30.145 INFO:tasks.rados.rados.0.smithi165.stdout:400: writing smithi16534841-6 from 490804 to 1193398 tid 1 2024-09-10T16:09:30.150 INFO:tasks.rados.rados.0.smithi165.stdout:400: writing smithi16534841-6 from 1615491 to 2200201 tid 2 2024-09-10T16:09:30.154 INFO:tasks.rados.rados.0.smithi165.stdout:400: writing smithi16534841-6 from 2724284 to 2890804 tid 3 2024-09-10T16:09:30.154 INFO:tasks.rados.rados.0.smithi165.stdout:401: write (excl) oid 2 current snap is 52 2024-09-10T16:09:30.154 INFO:tasks.rados.rados.0.smithi165.stdout:401: seq_num 173 ranges {507611=777843,2073973=444721} 2024-09-10T16:09:30.161 INFO:tasks.rados.rados.0.smithi165.stdout:401: writing smithi16534841-2 from 507611 to 1285454 tid 1 2024-09-10T16:09:30.168 INFO:tasks.rados.rados.0.smithi165.stdout:401: writing smithi16534841-2 from 2073973 to 2518694 tid 2 2024-09-10T16:09:30.168 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 1 to smithi16534841-30 2024-09-10T16:09:30.168 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 2 to smithi16534841-30 2024-09-10T16:09:30.168 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 3 to smithi16534841-30 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 4 to smithi16534841-30 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: finishing write tid 5 to smithi16534841-30 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 30 updating version 0 to 290 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 30 version 290 is already newer than 289 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 30 updating version 290 to 291 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: oid 30 version 291 is already newer than 288 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 291 (ObjNum 169 snap 52 seq_num 169) dirty exists 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: left oid 30 (ObjNum 169 snap 52 seq_num 169) 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:398: finishing write tid 1 to smithi16534841-14 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:398: finishing write tid 2 to smithi16534841-14 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:402: read oid 23 snap -1 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:402: expect (ObjNum 97 snap 24 seq_num 97) 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:396: done (6 left) 2024-09-10T16:09:30.169 INFO:tasks.rados.rados.0.smithi165.stdout:397: done (5 left) 2024-09-10T16:09:30.170 INFO:tasks.rados.rados.0.smithi165.stdout:403: write oid 32 current snap is 52 2024-09-10T16:09:30.170 INFO:tasks.rados.rados.0.smithi165.stdout:403: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2024-09-10T16:09:30.173 INFO:tasks.rados.rados.0.smithi165.stdout:403: writing smithi16534841-32 from 524418 to 977510 tid 1 2024-09-10T16:09:30.180 INFO:tasks.rados.rados.0.smithi165.stdout:403: writing smithi16534841-32 from 1732455 to 2320834 tid 2 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:403: writing smithi16534841-32 from 2779130 to 2924418 tid 3 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:404: copy_from oid 10 from oid 18 current snap is 52 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:398: finishing write tid 3 to smithi16534841-14 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:398: finishing write tid 4 to smithi16534841-14 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:398: finishing write tid 5 to smithi16534841-14 2024-09-10T16:09:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:398: oid 14 updating version 0 to 108 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:398: oid 14 updating version 108 to 111 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:398: oid 14 version 111 is already newer than 109 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:398: oid 14 version 111 is already newer than 110 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 111 (ObjNum 170 snap 52 seq_num 170) dirty exists 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:398: left oid 14 (ObjNum 170 snap 52 seq_num 170) 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:399: finishing write tid 1 to smithi16534841-28 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:399: finishing write tid 2 to smithi16534841-28 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:399: finishing write tid 3 to smithi16534841-28 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:399: finishing write tid 4 to smithi16534841-28 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:398: done (6 left) 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:405: read oid 46 snap -1 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:405: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:406: delete oid 39 current snap is 52 2024-09-10T16:09:30.186 INFO:tasks.rados.rados.0.smithi165.stdout:399: finishing write tid 5 to smithi16534841-28 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:399: oid 28 updating version 0 to 64 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:399: oid 28 version 64 is already newer than 63 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:399: oid 28 updating version 64 to 66 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:399: oid 28 version 66 is already newer than 65 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 66 (ObjNum 171 snap 52 seq_num 171) dirty exists 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:399: left oid 28 (ObjNum 171 snap 52 seq_num 171) 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: finishing write tid 1 to smithi16534841-6 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: finishing write tid 2 to smithi16534841-6 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: finishing write tid 3 to smithi16534841-6 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: finishing write tid 4 to smithi16534841-6 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: finishing write tid 5 to smithi16534841-6 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: oid 6 updating version 0 to 183 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: oid 6 version 183 is already newer than 181 2024-09-10T16:09:30.187 INFO:tasks.rados.rados.0.smithi165.stdout:400: oid 6 version 183 is already newer than 180 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:400: oid 6 version 183 is already newer than 182 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 183 (ObjNum 172 snap 52 seq_num 172) dirty exists 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:400: left oid 6 (ObjNum 172 snap 52 seq_num 172) 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: finishing write tid 1 to smithi16534841-2 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: finishing write tid 2 to smithi16534841-2 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: finishing write tid 3 to smithi16534841-2 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: finishing write tid 4 to smithi16534841-2 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: oid 2 updating version 0 to 280 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: oid 2 updating version 280 to 281 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: oid 2 version 281 is already newer than 279 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 281 (ObjNum 173 snap 52 seq_num 173) dirty exists 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:401: left oid 2 (ObjNum 173 snap 52 seq_num 173) 2024-09-10T16:09:30.188 INFO:tasks.rados.rados.0.smithi165.stdout:402: expect (ObjNum 97 snap 24 seq_num 97) 2024-09-10T16:09:30.193 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1245076 2024-09-10T16:09:30.290 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:30.346 INFO:tasks.rados.rados.0.smithi165.stdout:403: finishing write tid 1 to smithi16534841-32 2024-09-10T16:09:30.347 INFO:tasks.rados.rados.0.smithi165.stdout:403: finishing write tid 2 to smithi16534841-32 2024-09-10T16:09:30.347 INFO:tasks.rados.rados.0.smithi165.stdout:403: finishing write tid 3 to smithi16534841-32 2024-09-10T16:09:30.347 INFO:tasks.rados.rados.0.smithi165.stdout:405: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:30.347 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 458414 2024-09-10T16:09:30.477 INFO:tasks.rados.rados.0.smithi165.stdout:403: finishing write tid 4 to smithi16534841-32 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: finishing write tid 5 to smithi16534841-32 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: oid 32 updating version 0 to 185 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: oid 32 version 185 is already newer than 184 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: oid 32 updating version 185 to 187 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: oid 32 version 187 is already newer than 186 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 187 (ObjNum 174 snap 52 seq_num 174) dirty exists 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:403: left oid 32 (ObjNum 174 snap 52 seq_num 174) 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:404: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:09:30.478 INFO:tasks.rados.rados.0.smithi165.stdout:404: finishing copy_from to smithi16534841-10 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 240 (ObjNum 165 snap 52 seq_num 165) dirty exists 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:399: done (7 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:400: done (6 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:401: done (5 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:402: done (4 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:403: done (3 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:404: done (2 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:405: done (1 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:406: done (0 left) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:407: read oid 33 snap -1 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:407: expect (ObjNum 141 snap 42 seq_num 141) 2024-09-10T16:09:30.479 INFO:tasks.rados.rados.0.smithi165.stdout:408: snap_create 2024-09-10T16:09:30.480 INFO:tasks.rados.rados.0.smithi165.stdout:407: expect (ObjNum 141 snap 42 seq_num 141) 2024-09-10T16:09:30.501 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1360923 2024-09-10T16:09:30.534 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:30.595 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 245286 2024-09-10T16:09:31.206 INFO:tasks.rados.rados.0.smithi165.stdout:407: done (1 left) 2024-09-10T16:09:31.206 INFO:tasks.rados.rados.0.smithi165.stdout:408: done (0 left) 2024-09-10T16:09:31.206 INFO:tasks.rados.rados.0.smithi165.stdout:409: read oid 18 snap -1 2024-09-10T16:09:31.206 INFO:tasks.rados.rados.0.smithi165.stdout:409: expect (ObjNum 165 snap 52 seq_num 165) 2024-09-10T16:09:31.206 INFO:tasks.rados.rados.0.smithi165.stdout:410: delete oid 3 current snap is 53 2024-09-10T16:09:31.210 INFO:tasks.rados.rados.0.smithi165.stdout:410: done (1 left) 2024-09-10T16:09:31.210 INFO:tasks.rados.rados.0.smithi165.stdout:411: write (excl) oid 48 current snap is 53 2024-09-10T16:09:31.210 INFO:tasks.rados.rados.0.smithi165.stdout:411: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-10T16:09:31.217 INFO:tasks.rados.rados.0.smithi165.stdout:411: writing smithi16534841-48 from 574839 to 1370031 tid 1 2024-09-10T16:09:31.223 INFO:tasks.rados.rados.0.smithi165.stdout:411: writing smithi16534841-48 from 1907901 to 2359960 tid 2 2024-09-10T16:09:31.223 INFO:tasks.rados.rados.0.smithi165.stdout:412: snap_create 2024-09-10T16:09:31.224 INFO:tasks.rados.rados.0.smithi165.stdout:409: expect (ObjNum 165 snap 52 seq_num 165) 2024-09-10T16:09:31.224 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:31.256 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:31.288 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:31.319 INFO:tasks.rados.rados.0.smithi165.stdout:411: finishing write tid 1 to smithi16534841-48 2024-09-10T16:09:31.319 INFO:tasks.rados.rados.0.smithi165.stdout:411: finishing write tid 2 to smithi16534841-48 2024-09-10T16:09:31.320 INFO:tasks.rados.rados.0.smithi165.stdout:411: finishing write tid 3 to smithi16534841-48 2024-09-10T16:09:31.320 INFO:tasks.rados.rados.0.smithi165.stdout:411: finishing write tid 4 to smithi16534841-48 2024-09-10T16:09:31.320 INFO:tasks.rados.rados.0.smithi165.stdout:411: oid 48 updating version 0 to 297 2024-09-10T16:09:31.321 INFO:tasks.rados.rados.0.smithi165.stdout:411: oid 48 updating version 297 to 299 2024-09-10T16:09:31.321 INFO:tasks.rados.rados.0.smithi165.stdout:411: oid 48 version 299 is already newer than 298 2024-09-10T16:09:31.321 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 299 (ObjNum 177 snap 53 seq_num 177) dirty exists 2024-09-10T16:09:31.321 INFO:tasks.rados.rados.0.smithi165.stdout:411: left oid 48 (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:09:32.209 INFO:tasks.rados.rados.0.smithi165.stdout:409: done (2 left) 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:411: done (1 left) 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:412: done (0 left) 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:413: rollback oid 12 current snap is 54 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 19 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:414: read oid 45 snap 43 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:414: expect (ObjNum 152 snap 43 seq_num 152) 2024-09-10T16:09:32.210 INFO:tasks.rados.rados.0.smithi165.stdout:415: delete oid 44 current snap is 54 2024-09-10T16:09:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:415: done (2 left) 2024-09-10T16:09:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:416: read oid 22 snap -1 2024-09-10T16:09:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:416: expect deleted 2024-09-10T16:09:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:417: rollback oid 11 current snap is 54 2024-09-10T16:09:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 52 2024-09-10T16:09:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:418: copy_from oid 2 from oid 37 current snap is 54 2024-09-10T16:09:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:419: snap_remove snap 39 2024-09-10T16:09:33.219 INFO:tasks.rados.rados.0.smithi165.stdout:419: done (5 left) 2024-09-10T16:09:33.219 INFO:tasks.rados.rados.0.smithi165.stdout:420: copy_from oid 42 from oid 49 current snap is 54 2024-09-10T16:09:33.219 INFO:tasks.rados.rados.0.smithi165.stdout:413: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:09:33.220 INFO:tasks.rados.rados.0.smithi165.stdout:414: expect (ObjNum 152 snap 43 seq_num 152) 2024-09-10T16:09:33.273 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:33.373 INFO:tasks.rados.rados.0.smithi165.stdout:413: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:09:33.373 INFO:tasks.rados.rados.0.smithi165.stdout:413: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:09:33.373 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 15 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-10T16:09:33.373 INFO:tasks.rados.rados.0.smithi165.stdout:417: finishing rollback tid 0 to smithi16534841-11 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:417: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:417: finishing rollback tid 2 to smithi16534841-11 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 129 (ObjNum 79 snap 18 seq_num 79) dirty exists 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:418: finishing copy_from to smithi16534841-2 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:418: got expected ENOENT (src dne) 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:418: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:420: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:420: finishing copy_from to smithi16534841-42 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 80 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:413: done (5 left) 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:414: done (4 left) 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:416: done (3 left) 2024-09-10T16:09:33.374 INFO:tasks.rados.rados.0.smithi165.stdout:417: done (2 left) 2024-09-10T16:09:33.375 INFO:tasks.rados.rados.0.smithi165.stdout:418: done (1 left) 2024-09-10T16:09:33.375 INFO:tasks.rados.rados.0.smithi165.stdout:420: done (0 left) 2024-09-10T16:09:33.375 INFO:tasks.rados.rados.0.smithi165.stdout:421: snap_remove snap 19 2024-09-10T16:09:34.230 INFO:tasks.rados.rados.0.smithi165.stdout:421: done (0 left) 2024-09-10T16:09:34.230 INFO:tasks.rados.rados.0.smithi165.stdout:422: write oid 23 current snap is 54 2024-09-10T16:09:34.230 INFO:tasks.rados.rados.0.smithi165.stdout:422: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-10T16:09:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:422: writing smithi16534841-23 from 608453 to 1154143 tid 1 2024-09-10T16:09:34.242 INFO:tasks.rados.rados.0.smithi165.stdout:422: writing smithi16534841-23 from 1624865 to 2080593 tid 2 2024-09-10T16:09:34.248 INFO:tasks.rados.rados.0.smithi165.stdout:422: writing smithi16534841-23 from 2632598 to 3008453 tid 3 2024-09-10T16:09:34.248 INFO:tasks.rados.rados.0.smithi165.stdout:423: delete oid 31 current snap is 54 2024-09-10T16:09:34.250 INFO:tasks.rados.rados.0.smithi165.stdout:422: finishing write tid 1 to smithi16534841-23 2024-09-10T16:09:34.252 INFO:tasks.rados.rados.0.smithi165.stdout:422: finishing write tid 2 to smithi16534841-23 2024-09-10T16:09:34.256 INFO:tasks.rados.rados.0.smithi165.stdout:422: finishing write tid 3 to smithi16534841-23 2024-09-10T16:09:34.256 INFO:tasks.rados.rados.0.smithi165.stdout:422: finishing write tid 4 to smithi16534841-23 2024-09-10T16:09:34.256 INFO:tasks.rados.rados.0.smithi165.stdout:423: done (1 left) 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:424: copy_from oid 41 from oid 39 current snap is 54 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: finishing write tid 5 to smithi16534841-23 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: oid 23 updating version 0 to 258 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: oid 23 updating version 258 to 259 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: oid 23 updating version 259 to 263 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: oid 23 version 263 is already newer than 262 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 263 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: left oid 23 (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:422: done (1 left) 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:425: copy_from oid 21 from oid 11 current snap is 54 2024-09-10T16:09:34.257 INFO:tasks.rados.rados.0.smithi165.stdout:426: delete oid 47 current snap is 54 2024-09-10T16:09:34.258 INFO:tasks.rados.rados.0.smithi165.stdout:426: done (2 left) 2024-09-10T16:09:34.258 INFO:tasks.rados.rados.0.smithi165.stdout:427: copy_from oid 3 from oid 27 current snap is 54 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:428: read oid 6 snap -1 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:428: expect (ObjNum 172 snap 52 seq_num 172) 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:429: read oid 26 snap -1 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:429: expect deleted 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:430: read oid 47 snap -1 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:430: expect deleted 2024-09-10T16:09:34.259 INFO:tasks.rados.rados.0.smithi165.stdout:431: delete oid 45 current snap is 54 2024-09-10T16:09:34.260 INFO:tasks.rados.rados.0.smithi165.stdout:427: finishing copy_from to smithi16534841-3 2024-09-10T16:09:34.260 INFO:tasks.rados.rados.0.smithi165.stdout:427: got expected ENOENT (src dne) 2024-09-10T16:09:34.260 INFO:tasks.rados.rados.0.smithi165.stdout:427: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:424: finishing copy_from to smithi16534841-41 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:424: got expected ENOENT (src dne) 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:427: done (6 left) 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:429: done (5 left) 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:430: done (4 left) 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:431: done (3 left) 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:432: snap_create 2024-09-10T16:09:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:424: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:09:34.271 INFO:tasks.rados.rados.0.smithi165.stdout:425: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:09:34.271 INFO:tasks.rados.rados.0.smithi165.stdout:428: expect (ObjNum 172 snap 52 seq_num 172) 2024-09-10T16:09:34.276 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1076757 2024-09-10T16:09:34.296 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:34.327 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:34.358 INFO:tasks.rados.rados.0.smithi165.stdout:425: finishing copy_from to smithi16534841-21 2024-09-10T16:09:34.359 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 129 (ObjNum 79 snap 18 seq_num 79) dirty exists 2024-09-10T16:09:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:424: done (3 left) 2024-09-10T16:09:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:425: done (2 left) 2024-09-10T16:09:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:428: done (1 left) 2024-09-10T16:09:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:432: done (0 left) 2024-09-10T16:09:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:433: delete oid 6 current snap is 55 2024-09-10T16:09:35.101 INFO:tasks.rados.rados.0.smithi165.stdout:433: done (0 left) 2024-09-10T16:09:35.101 INFO:tasks.rados.rados.0.smithi165.stdout:434: snap_create 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:434: done (0 left) 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:435: read oid 46 snap -1 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:435: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:436: read oid 5 snap -1 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:436: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-10T16:09:36.095 INFO:tasks.rados.rados.0.smithi165.stdout:437: delete oid 18 current snap is 56 2024-09-10T16:09:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:437: done (2 left) 2024-09-10T16:09:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:438: write oid 11 current snap is 56 2024-09-10T16:09:36.100 INFO:tasks.rados.rados.0.smithi165.stdout:438: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-10T16:09:36.108 INFO:tasks.rados.rados.0.smithi165.stdout:438: writing smithi16534841-11 from 709295 to 1422832 tid 1 2024-09-10T16:09:36.115 INFO:tasks.rados.rados.0.smithi165.stdout:438: writing smithi16534841-11 from 1859404 to 2609786 tid 2 2024-09-10T16:09:36.115 INFO:tasks.rados.rados.0.smithi165.stdout:439: write oid 27 current snap is 56 2024-09-10T16:09:36.115 INFO:tasks.rados.rados.0.smithi165.stdout:439: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-10T16:09:36.123 INFO:tasks.rados.rados.0.smithi165.stdout:439: writing smithi16534841-27 from 726102 to 1514888 tid 1 2024-09-10T16:09:36.129 INFO:tasks.rados.rados.0.smithi165.stdout:439: writing smithi16534841-27 from 1917886 to 2528279 tid 2 2024-09-10T16:09:36.132 INFO:tasks.rados.rados.0.smithi165.stdout:439: writing smithi16534841-27 from 3108206 to 3126102 tid 3 2024-09-10T16:09:36.132 INFO:tasks.rados.rados.0.smithi165.stdout:440: write (excl) oid 50 current snap is 56 2024-09-10T16:09:36.133 INFO:tasks.rados.rados.0.smithi165.stdout:440: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-10T16:09:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:440: writing smithi16534841-50 from 742909 to 1206944 tid 1 2024-09-10T16:09:36.147 INFO:tasks.rados.rados.0.smithi165.stdout:440: writing smithi16534841-50 from 1976368 to 2730419 tid 2 2024-09-10T16:09:36.148 INFO:tasks.rados.rados.0.smithi165.stdout:441: snap_create 2024-09-10T16:09:36.148 INFO:tasks.rados.rados.0.smithi165.stdout:436: expect (ObjNum 6 snap 0 seq_num 6) 2024-09-10T16:09:36.150 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1059579 2024-09-10T16:09:36.168 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1632531 2024-09-10T16:09:36.195 INFO:tasks.rados.rados.0.smithi165.stdout:435: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:36.286 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: finishing write tid 1 to smithi16534841-11 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: finishing write tid 2 to smithi16534841-11 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: finishing write tid 3 to smithi16534841-11 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: finishing write tid 4 to smithi16534841-11 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: oid 11 updating version 0 to 141 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: oid 11 version 141 is already newer than 140 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:438: oid 11 version 141 is already newer than 139 2024-09-10T16:09:36.335 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 141 (ObjNum 185 snap 56 seq_num 185) dirty exists 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:438: left oid 11 (ObjNum 185 snap 56 seq_num 185) 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: finishing write tid 1 to smithi16534841-27 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: finishing write tid 2 to smithi16534841-27 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: finishing write tid 3 to smithi16534841-27 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: finishing write tid 4 to smithi16534841-27 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: finishing write tid 5 to smithi16534841-27 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: oid 27 updating version 0 to 302 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: oid 27 version 302 is already newer than 299 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: oid 27 version 302 is already newer than 300 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: oid 27 version 302 is already newer than 301 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 302 (ObjNum 186 snap 56 seq_num 186) dirty exists 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:439: left oid 27 (ObjNum 186 snap 56 seq_num 186) 2024-09-10T16:09:36.336 INFO:tasks.rados.rados.0.smithi165.stdout:440: finishing write tid 1 to smithi16534841-50 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: finishing write tid 2 to smithi16534841-50 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: finishing write tid 3 to smithi16534841-50 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: finishing write tid 4 to smithi16534841-50 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: oid 50 updating version 0 to 144 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: oid 50 version 144 is already newer than 142 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: oid 50 version 144 is already newer than 143 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 144 (ObjNum 187 snap 56 seq_num 187) dirty exists 2024-09-10T16:09:36.337 INFO:tasks.rados.rados.0.smithi165.stdout:440: left oid 50 (ObjNum 187 snap 56 seq_num 187) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:435: done (5 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:436: done (4 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:438: done (3 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:439: done (2 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:440: done (1 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:441: done (0 left) 2024-09-10T16:09:37.109 INFO:tasks.rados.rados.0.smithi165.stdout:442: write oid 40 current snap is 57 2024-09-10T16:09:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:442: seq_num 188 ranges {759716=539284,2034850=614062} 2024-09-10T16:09:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:442: writing smithi16534841-40 from 759716 to 1299000 tid 1 2024-09-10T16:09:37.124 INFO:tasks.rados.rados.0.smithi165.stdout:442: writing smithi16534841-40 from 2034850 to 2648912 tid 2 2024-09-10T16:09:37.124 INFO:tasks.rados.rados.0.smithi165.stdout:442: finishing write tid 1 to smithi16534841-40 2024-09-10T16:09:37.124 INFO:tasks.rados.rados.0.smithi165.stdout:443: read oid 17 snap -1 2024-09-10T16:09:37.124 INFO:tasks.rados.rados.0.smithi165.stdout:443: expect (ObjNum 135 snap 38 seq_num 135) 2024-09-10T16:09:37.124 INFO:tasks.rados.rados.0.smithi165.stdout:444: snap_create 2024-09-10T16:09:37.127 INFO:tasks.rados.rados.0.smithi165.stdout:442: finishing write tid 2 to smithi16534841-40 2024-09-10T16:09:37.127 INFO:tasks.rados.rados.0.smithi165.stdout:442: finishing write tid 3 to smithi16534841-40 2024-09-10T16:09:37.127 INFO:tasks.rados.rados.0.smithi165.stdout:442: finishing write tid 4 to smithi16534841-40 2024-09-10T16:09:37.127 INFO:tasks.rados.rados.0.smithi165.stdout:442: oid 40 updating version 0 to 118 2024-09-10T16:09:37.128 INFO:tasks.rados.rados.0.smithi165.stdout:442: oid 40 updating version 118 to 120 2024-09-10T16:09:37.128 INFO:tasks.rados.rados.0.smithi165.stdout:442: oid 40 version 120 is already newer than 119 2024-09-10T16:09:37.128 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 120 (ObjNum 188 snap 57 seq_num 188) dirty exists 2024-09-10T16:09:37.128 INFO:tasks.rados.rados.0.smithi165.stdout:442: left oid 40 (ObjNum 188 snap 57 seq_num 188) 2024-09-10T16:09:37.135 INFO:tasks.rados.rados.0.smithi165.stdout:443: expect (ObjNum 135 snap 38 seq_num 135) 2024-09-10T16:09:37.135 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:38.113 INFO:tasks.rados.rados.0.smithi165.stdout:442: done (2 left) 2024-09-10T16:09:38.114 INFO:tasks.rados.rados.0.smithi165.stdout:443: done (1 left) 2024-09-10T16:09:38.114 INFO:tasks.rados.rados.0.smithi165.stdout:444: done (0 left) 2024-09-10T16:09:38.114 INFO:tasks.rados.rados.0.smithi165.stdout:445: write oid 41 current snap is 58 2024-09-10T16:09:38.114 INFO:tasks.rados.rados.0.smithi165.stdout:445: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-10T16:09:38.120 INFO:tasks.rados.rados.0.smithi165.stdout:445: writing smithi16534841-41 from 776523 to 1391056 tid 1 2024-09-10T16:09:38.128 INFO:tasks.rados.rados.0.smithi165.stdout:445: writing smithi16534841-41 from 1976979 to 2734699 tid 2 2024-09-10T16:09:38.128 INFO:tasks.rados.rados.0.smithi165.stdout:446: write (excl) oid 32 current snap is 58 2024-09-10T16:09:38.128 INFO:tasks.rados.rados.0.smithi165.stdout:446: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2024-09-10T16:09:38.136 INFO:tasks.rados.rados.0.smithi165.stdout:446: writing smithi16534841-32 from 793330 to 1483112 tid 1 2024-09-10T16:09:38.142 INFO:tasks.rados.rados.0.smithi165.stdout:446: writing smithi16534841-32 from 2035461 to 2653192 tid 2 2024-09-10T16:09:38.145 INFO:tasks.rados.rados.0.smithi165.stdout:446: writing smithi16534841-32 from 3101545 to 3193330 tid 3 2024-09-10T16:09:38.146 INFO:tasks.rados.rados.0.smithi165.stdout:447: snap_remove snap 50 2024-09-10T16:09:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:447: done (2 left) 2024-09-10T16:09:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:448: snap_create 2024-09-10T16:09:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:445: finishing write tid 1 to smithi16534841-41 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: finishing write tid 2 to smithi16534841-41 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: finishing write tid 3 to smithi16534841-41 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: finishing write tid 4 to smithi16534841-41 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: oid 41 updating version 0 to 267 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: oid 41 updating version 267 to 268 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: oid 41 version 268 is already newer than 266 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 268 (ObjNum 189 snap 58 seq_num 189) dirty exists 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:445: left oid 41 (ObjNum 189 snap 58 seq_num 189) 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: finishing write tid 1 to smithi16534841-32 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: finishing write tid 2 to smithi16534841-32 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: finishing write tid 3 to smithi16534841-32 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: finishing write tid 4 to smithi16534841-32 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: finishing write tid 5 to smithi16534841-32 2024-09-10T16:09:39.115 INFO:tasks.rados.rados.0.smithi165.stdout:446: oid 32 updating version 0 to 192 2024-09-10T16:09:39.116 INFO:tasks.rados.rados.0.smithi165.stdout:446: oid 32 version 192 is already newer than 190 2024-09-10T16:09:39.116 INFO:tasks.rados.rados.0.smithi165.stdout:446: oid 32 version 192 is already newer than 191 2024-09-10T16:09:39.116 INFO:tasks.rados.rados.0.smithi165.stdout:446: oid 32 version 192 is already newer than 189 2024-09-10T16:09:39.116 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 192 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:09:39.116 INFO:tasks.rados.rados.0.smithi165.stdout:446: left oid 32 (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:09:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:445: done (2 left) 2024-09-10T16:09:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:446: done (1 left) 2024-09-10T16:09:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:448: done (0 left) 2024-09-10T16:09:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:449: copy_from oid 11 from oid 29 current snap is 59 2024-09-10T16:09:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:450: snap_remove snap 56 2024-09-10T16:09:41.543 INFO:tasks.rados.rados.0.smithi165.stdout:450: done (1 left) 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:451: read oid 17 snap -1 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:451: expect (ObjNum 135 snap 38 seq_num 135) 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:452: read oid 7 snap -1 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:452: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:453: copy_from oid 38 from oid 24 current snap is 59 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:449: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:449: finishing copy_from to smithi16534841-11 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 39 (ObjNum 88 snap 20 seq_num 88) dirty exists 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:449: done (3 left) 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:454: write oid 13 current snap is 59 2024-09-10T16:09:41.544 INFO:tasks.rados.rados.0.smithi165.stdout:454: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-10T16:09:41.548 INFO:tasks.rados.rados.0.smithi165.stdout:454: writing smithi16534841-13 from 410137 to 891521 tid 1 2024-09-10T16:09:41.555 INFO:tasks.rados.rados.0.smithi165.stdout:454: writing smithi16534841-13 from 1410296 to 2171685 tid 2 2024-09-10T16:09:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:454: writing smithi16534841-13 from 2728968 to 3210137 tid 3 2024-09-10T16:09:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:455: rollback oid 30 current snap is 59 2024-09-10T16:09:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 38 2024-09-10T16:09:41.562 INFO:tasks.rados.rados.0.smithi165.stdout:451: expect (ObjNum 135 snap 38 seq_num 135) 2024-09-10T16:09:41.589 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1424265 2024-09-10T16:09:41.652 INFO:tasks.rados.rados.0.smithi165.stdout:452: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:09:41.722 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:41.764 INFO:tasks.rados.rados.0.smithi165.stdout:453: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:09:41.764 INFO:tasks.rados.rados.0.smithi165.stdout:454: finishing write tid 1 to smithi16534841-13 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:453: finishing copy_from to smithi16534841-38 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 218 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: finishing write tid 2 to smithi16534841-13 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: finishing write tid 3 to smithi16534841-13 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: finishing write tid 4 to smithi16534841-13 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: finishing write tid 5 to smithi16534841-13 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: oid 13 updating version 0 to 96 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: oid 13 version 96 is already newer than 95 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: oid 13 version 96 is already newer than 94 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: oid 13 updating version 96 to 97 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 97 (ObjNum 191 snap 59 seq_num 191) dirty exists 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:454: left oid 13 (ObjNum 191 snap 59 seq_num 191) 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:455: finishing rollback tid 0 to smithi16534841-30 2024-09-10T16:09:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:455: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:455: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 335 (ObjNum 131 snap 36 seq_num 131) dirty exists 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:451: done (4 left) 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:452: done (3 left) 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:453: done (2 left) 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:454: done (1 left) 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:455: done (0 left) 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:456: delete oid 39 current snap is 59 2024-09-10T16:09:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:456: done (0 left) 2024-09-10T16:09:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:457: rollback oid 33 current snap is 59 2024-09-10T16:09:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 37 2024-09-10T16:09:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:458: delete oid 43 current snap is 59 2024-09-10T16:09:41.768 INFO:tasks.rados.rados.0.smithi165.stdout:458: done (1 left) 2024-09-10T16:09:41.769 INFO:tasks.rados.rados.0.smithi165.stdout:459: delete oid 6 current snap is 59 2024-09-10T16:09:41.769 INFO:tasks.rados.rados.0.smithi165.stdout:457: finishing rollback tid 0 to smithi16534841-33 2024-09-10T16:09:41.770 INFO:tasks.rados.rados.0.smithi165.stdout:459: done (1 left) 2024-09-10T16:09:41.770 INFO:tasks.rados.rados.0.smithi165.stdout:460: snap_remove snap 51 2024-09-10T16:09:42.552 INFO:tasks.rados.rados.0.smithi165.stdout:460: done (1 left) 2024-09-10T16:09:42.552 INFO:tasks.rados.rados.0.smithi165.stdout:461: snap_remove snap 46 2024-09-10T16:09:43.561 INFO:tasks.rados.rados.0.smithi165.stdout:461: done (1 left) 2024-09-10T16:09:43.561 INFO:tasks.rados.rados.0.smithi165.stdout:462: snap_create 2024-09-10T16:09:43.561 INFO:tasks.rados.rados.0.smithi165.stdout:457: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:09:43.561 INFO:tasks.rados.rados.0.smithi165.stdout:457: finishing rollback tid 2 to smithi16534841-33 2024-09-10T16:09:43.561 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 263 (ObjNum 68 snap 8 seq_num 68) dirty exists 2024-09-10T16:09:44.559 INFO:tasks.rados.rados.0.smithi165.stdout:457: done (1 left) 2024-09-10T16:09:44.559 INFO:tasks.rados.rados.0.smithi165.stdout:462: done (0 left) 2024-09-10T16:09:44.559 INFO:tasks.rados.rados.0.smithi165.stdout:463: write oid 30 current snap is 60 2024-09-10T16:09:44.560 INFO:tasks.rados.rados.0.smithi165.stdout:463: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-10T16:09:44.565 INFO:tasks.rados.rados.0.smithi165.stdout:463: writing smithi16534841-30 from 477365 to 1259745 tid 1 2024-09-10T16:09:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:463: writing smithi16534841-30 from 1927871 to 2696598 tid 2 2024-09-10T16:09:44.577 INFO:tasks.rados.rados.0.smithi165.stdout:463: writing smithi16534841-30 from 3122307 to 3277365 tid 3 2024-09-10T16:09:44.577 INFO:tasks.rados.rados.0.smithi165.stdout:464: rollback oid 19 current snap is 60 2024-09-10T16:09:44.577 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 44 2024-09-10T16:09:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:463: finishing write tid 1 to smithi16534841-30 2024-09-10T16:09:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:465: write (excl) oid 8 current snap is 60 2024-09-10T16:09:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:465: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-10T16:09:44.582 INFO:tasks.rados.rados.0.smithi165.stdout:465: writing smithi16534841-8 from 494172 to 951801 tid 1 2024-09-10T16:09:44.589 INFO:tasks.rados.rados.0.smithi165.stdout:465: writing smithi16534841-8 from 1586353 to 2098738 tid 2 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:465: writing smithi16534841-8 from 2749730 to 3294172 tid 3 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:466: delete oid 2 current snap is 60 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:463: finishing write tid 2 to smithi16534841-30 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:463: finishing write tid 3 to smithi16534841-30 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:463: finishing write tid 4 to smithi16534841-30 2024-09-10T16:09:44.596 INFO:tasks.rados.rados.0.smithi165.stdout:463: finishing write tid 5 to smithi16534841-30 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:463: oid 30 updating version 0 to 355 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:463: oid 30 updating version 355 to 356 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:463: oid 30 version 356 is already newer than 354 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:463: oid 30 version 356 is already newer than 353 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 356 (ObjNum 195 snap 60 seq_num 195) dirty exists 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:463: left oid 30 (ObjNum 195 snap 60 seq_num 195) 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:464: finishing rollback tid 0 to smithi16534841-19 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:464: finishing rollback tid 1 to smithi16534841-19 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:464: finishing rollback tid 2 to smithi16534841-19 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 360 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:465: finishing write tid 1 to smithi16534841-8 2024-09-10T16:09:44.597 INFO:tasks.rados.rados.0.smithi165.stdout:465: finishing write tid 2 to smithi16534841-8 2024-09-10T16:09:44.600 INFO:tasks.rados.rados.0.smithi165.stdout:465: finishing write tid 3 to smithi16534841-8 2024-09-10T16:09:44.600 INFO:tasks.rados.rados.0.smithi165.stdout:465: finishing write tid 4 to smithi16534841-8 2024-09-10T16:09:44.600 INFO:tasks.rados.rados.0.smithi165.stdout:465: finishing write tid 5 to smithi16534841-8 2024-09-10T16:09:44.600 INFO:tasks.rados.rados.0.smithi165.stdout:465: oid 8 updating version 0 to 310 2024-09-10T16:09:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:465: oid 8 updating version 310 to 311 2024-09-10T16:09:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:465: oid 8 updating version 311 to 312 2024-09-10T16:09:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:465: oid 8 updating version 312 to 313 2024-09-10T16:09:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 313 (ObjNum 196 snap 60 seq_num 196) dirty exists 2024-09-10T16:09:44.601 INFO:tasks.rados.rados.0.smithi165.stdout:465: left oid 8 (ObjNum 196 snap 60 seq_num 196) 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:463: done (3 left) 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:464: done (2 left) 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:465: done (1 left) 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:466: done (0 left) 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:467: copy_from oid 38 from oid 49 current snap is 60 2024-09-10T16:09:44.602 INFO:tasks.rados.rados.0.smithi165.stdout:468: delete oid 40 current snap is 60 2024-09-10T16:09:44.603 INFO:tasks.rados.rados.0.smithi165.stdout:468: done (1 left) 2024-09-10T16:09:44.604 INFO:tasks.rados.rados.0.smithi165.stdout:469: copy_from oid 45 from oid 15 current snap is 60 2024-09-10T16:09:44.604 INFO:tasks.rados.rados.0.smithi165.stdout:470: copy_from oid 37 from oid 17 current snap is 60 2024-09-10T16:09:44.604 INFO:tasks.rados.rados.0.smithi165.stdout:471: snap_create 2024-09-10T16:09:44.614 INFO:tasks.rados.rados.0.smithi165.stdout:469: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:09:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:470: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:09:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:467: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:09:44.625 INFO:tasks.rados.rados.0.smithi165.stdout:469: finishing copy_from to smithi16534841-45 2024-09-10T16:09:44.625 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 103 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:09:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:470: finishing copy_from to smithi16534841-37 2024-09-10T16:09:44.627 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 44 (ObjNum 135 snap 38 seq_num 135) dirty exists 2024-09-10T16:09:44.628 INFO:tasks.rados.rados.0.smithi165.stdout:467: finishing copy_from to smithi16534841-38 2024-09-10T16:09:44.628 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 80 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:09:44.761 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:09:44.762 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:09:44.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:09:44.765 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:09:44.765 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:09:44.765 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.863+0000 7f23b9cd4640 10 monclient: get_monmap_and_config 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.863+0000 7f23b9cd4640 10 monclient: build_initial_monmap 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.863+0000 7f23b9cd4640 10 monclient: monmap: 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:44.864199+0000 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:44.864199+0000 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:44.866 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 Processor -- start 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 -- start start 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient: init 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23b406d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 0x7f23b40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f23b406d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23b40a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23b4135510 con 0x7f23b406d160 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23b409cb90 con 0x7f23b40a4c20 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f23b409ce70 con 0x7f23b406d540 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b9cd4640 10 monclient(hunting): authenticate will time out at 963.351629s 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f23b406d160 0x7f23b4134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39280/0 (socket says 172.21.15.164:39280) 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b37fe640 1 -- 172.21.15.164:0/1227429812 learned_addr learned my addr 172.21.15.164:0/1227429812 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 0x7f23b40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request con 0x7f23b406d540 auth_method 0 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 msgr2=0x7f23b40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 msgr2=0x7f23b40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 0x7f23b40a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.864+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 0x7f23b40a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.866+0000 7f23b37fe640 1 -- 172.21.15.164:0/1227429812 >> v1:172.21.15.164:6789/0 conn(0x7f23b406d160 legacy=0x7f23b4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 1 -- 172.21.15.164:0/1227429812 >> v1:172.21.15.165:6789/0 conn(0x7f23b40a4c20 legacy=0x7f23b413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1451075066 0 0) 0x7f23b4135510 con 0x7f23b406d160 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): _negotiate 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): my global_id is 24297 2024-09-10T16:09:44.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f23a0003520 con 0x7f23b406d160 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3247098667 0 0) 0x7f23a0003520 con 0x7f23b406d160 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 msgr2=0x7f23b40a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 --2- 172.21.15.164:0/1227429812 >> v2:172.21.15.165:3300/0 conn(0x7f23b406d540 0x7f23b40a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 >> v1:172.21.15.165:6789/0 conn(0x7f23b40a4c20 legacy=0x7f23b413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _finish_hunting 0 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b409d150 con 0x7f23b406d160 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f23ac001a60 con 0x7f23b406d160 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b9cd4640 5 monclient: authenticate success, global_id 24297 2024-09-10T16:09:44.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b9cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: dump: 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:09:44.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 >> v1:172.21.15.164:6789/0 conn(0x7f23b406d160 legacy=0x7f23b4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0003ac0 0x7f23a0003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): picked mon.b con 0x7f23a0003ac0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): picked mon.c con 0x7f23a0004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23a0005410 0x7f23a000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): picked mon.a con 0x7f23a0005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23b409ce70 con 0x7f23a0005410 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23a000c350 con 0x7f23a0003ac0 2024-09-10T16:09:44.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23a000c550 con 0x7f23a0004800 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b9cd4640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0003ac0 0x7f23a0003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request con 0x7f23a0004800 auth_method 0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23a0005410 0x7f23a000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): get_auth_request con 0x7f23a0003ac0 auth_method 0 2024-09-10T16:09:44.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): get_auth_request con 0x7f23a0005410 auth_method 0 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_done global_id 24297 payload 293 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0003ac0 msgr2=0x7f23a0003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0003ac0 0x7f23a0003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23a0005410 msgr2=0x7f23a000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23a0005410 0x7f23a000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 monclient: found mon.c 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b3fff640 1 -- 172.21.15.164:0/1227429812 reap_dead start 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23a0003520 con 0x7f23a0004800 2024-09-10T16:09:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:44.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:44.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.867+0000 7f23b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b2ffd640 1 -- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 msgr2=0x7f23a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b2ffd640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 0x7f23a0004c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f239c009fd0 tx=0x7f23a0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76b6cba85b0fda39 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c033070 con 0x7f23a0004800 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:44.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: dump: 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b9cd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f239c02f860 con 0x7f23a0004800 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:44.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 1 -- 172.21.15.164:0/1227429812 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f239c009bd0 con 0x7f23a0004800 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b9cd4640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b9cd4640 4 set_mon_vals no callback set 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.870+0000 7f23b27fc640 10 monclient: dump: 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:44.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 10 monclient: shutdown 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 -- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 msgr2=0x7f23a0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 0x7f23a0004c80 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f239c009fd0 tx=0x7f23a0005830 comp rx=0 tx=0).stop 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 -- 172.21.15.164:0/1227429812 shutdown_connections 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/1227429812 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23a0004800 0x7f23a0004c80 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 -- 172.21.15.164:0/1227429812 >> 172.21.15.164:0/1227429812 conn(0x7f23b40aacb0 msgr2=0x7f23b40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 -- 172.21.15.164:0/1227429812 shutdown_connections 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 1 -- 172.21.15.164:0/1227429812 wait complete. 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 10 monclient: build_initial_monmap 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.871+0000 7f23b9cd4640 10 monclient: monmap: 2024-09-10T16:09:44.889 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 Processor -- start 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 -- start start 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient: init 2024-09-10T16:09:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-b con 0x7f23a0004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 0x7f23b40a3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-c con 0x7f23b40a2fe0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): picked mon.noname-a con 0x7f23b41ccff0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23b4134e00 con 0x7f23b41ccff0 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23b40a4950 con 0x7f23a0004800 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23b4134f80 con 0x7f23b40a2fe0 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b9cd4640 10 monclient(hunting): authenticate will time out at 963.359833s 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 0x7f23b40a3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38696/0 (socket says 172.21.15.164:38696) 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 0x7f23b40a3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36996/0 (socket says 172.21.15.164:36996) 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 1 -- 172.21.15.164:0/122059576 learned_addr learned my addr 172.21.15.164:0/122059576 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 10 monclient(hunting): get_auth_request con 0x7f23b41ccff0 auth_method 0 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request con 0x7f23b40a2fe0 auth_method 0 2024-09-10T16:09:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.872+0000 7f23b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 10 monclient(hunting): get_auth_request con 0x7f23a0004800 auth_method 0 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 msgr2=0x7f23b40a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 msgr2=0x7f23b40a2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient(hunting): handle_auth_done global_id 24300 payload 824 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b37fe640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 msgr2=0x7f23b40a3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 0x7f23b40a3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 msgr2=0x7f23b40a2aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 monclient: found mon.noname-a 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.873+0000 7f23b3fff640 1 -- 172.21.15.164:0/122059576 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23b412fc90 con 0x7f23b41ccff0 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b3fff640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 msgr2=0x7f23b41cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b3fff640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f23b40aa5b0 tx=0x7f23a40019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4012770 con 0x7f23b41ccff0 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: dump: 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: _finish_auth 0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23a4018070 con 0x7f23b41ccff0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 5 monclient: authenticate success, global_id 24300 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 10 monclient: _renew_subs 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23b406e6b0 con 0x7f23b41ccff0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23a4012b50 con 0x7f23b41ccff0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23abfff640 10 monclient: dump: 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 10 monclient: _renew_subs 2024-09-10T16:09:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b1ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.875+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23b41bbf50 con 0x7f23b41ccff0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f23a401d460 con 0x7f23b41ccff0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b9cd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b9cd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b9cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23b4135360 con 0x7f23b41ccff0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23abfff640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 0x7f23940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(126..126 src has 1..126) ==== 4803+0+0 (secure 0 0 0) 0x7f23a4052940 con 0x7f23b41ccff0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b37fe640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 0x7f23940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.876+0000 7f23b37fe640 10 monclient: get_auth_request con 0x7f239403e210 auth_method 0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.879+0000 7f23b37fe640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 0x7f23940406d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001eb0 tx=0x7f23ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.881+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f23a4051320 con 0x7f23b41ccff0 2024-09-10T16:09:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.881+0000 7f23abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:44.881+0000 7f23abfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:467: done (3 left) 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:469: done (2 left) 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:470: done (1 left) 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:471: done (0 left) 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:472: rollback oid 49 current snap is 61 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 53 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:473: rollback oid 34 current snap is 61 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 20 2024-09-10T16:09:45.094 INFO:tasks.rados.rados.0.smithi165.stdout:474: snap_remove snap 38 2024-09-10T16:09:45.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23b9cd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:45.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23b9cd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:45.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23b9cd4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:45.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f23b40aa1e0 con 0x7f23b41ccff0 2024-09-10T16:09:45.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23abfff640 1 -- 172.21.15.164:0/122059576 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f23a4012d70 con 0x7f23b41ccff0 2024-09-10T16:09:45.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:45.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.126+0000 7f23abfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:45.128 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:45.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 msgr2=0x7f23940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 0x7f23940406d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f23ac001eb0 tx=0x7f23ac009040 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 10 monclient: shutdown 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 msgr2=0x7f23b41cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f23b40aa5b0 tx=0x7f23a40019c0 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.128+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 shutdown_connections 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f239403e210 0x7f23940406d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23b41ccff0 0x7f23b41cd460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23b40a2fe0 0x7f23b40a3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 --2- 172.21.15.164:0/122059576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23a0004800 0x7f23b40a2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 >> 172.21.15.164:0/122059576 conn(0x7f23b40aacb0 msgr2=0x7f23b40a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 shutdown_connections 2024-09-10T16:09:45.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:45.129+0000 7f23b9cd4640 1 -- 172.21.15.164:0/122059576 wait complete. 2024-09-10T16:09:48.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.249+0000 7fdabe3f8640 10 monclient: get_monmap_and_config 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.249+0000 7fdabe3f8640 10 monclient: build_initial_monmap 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.249+0000 7fdabe3f8640 10 monclient: monmap: 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:48.250398+0000 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:48.250398+0000 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:48.251 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:48.252 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:48.252 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:48.252 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:48.252 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 Processor -- start 2024-09-10T16:09:48.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 -- start start 2024-09-10T16:09:48.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient: init 2024-09-10T16:09:48.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:48.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-b con 0x7fdab80a0bc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdab8136260 0x7fdab809dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-c con 0x7fdab8136260 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdab80a2ec0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdab80a1740 con 0x7fdab80a2ec0 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdab8069660 con 0x7fdab8136260 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdab80a0fa0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.250+0000 7fdabe3f8640 10 monclient(hunting): authenticate will time out at 966.737961s 2024-09-10T16:09:48.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37004/0 (socket says 172.21.15.164:37004) 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 -- 172.21.15.164:0/348779014 learned_addr learned my addr 172.21.15.164:0/348779014 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): get_auth_request con 0x7fdab80a0bc0 auth_method 0 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab77fe640 1 --2- 172.21.15.164:0/348779014 >> v2:172.21.15.165:3300/0 conn(0x7fdab8136260 0x7fdab809dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab77fe640 10 monclient(hunting): get_auth_request con 0x7fdab8136260 auth_method 0 2024-09-10T16:09:48.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient(hunting): handle_auth_done global_id 4688 payload 274 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 -- 172.21.15.164:0/348779014 >> v2:172.21.15.165:3300/0 conn(0x7fdab8136260 msgr2=0x7fdab809dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 --2- 172.21.15.164:0/348779014 >> v2:172.21.15.165:3300/0 conn(0x7fdab8136260 0x7fdab809dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 -- 172.21.15.164:0/348779014 >> v1:172.21.15.165:6789/0 conn(0x7fdab80a2ec0 legacy=0x7fdab80a32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 monclient: found mon.noname-b 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.251+0000 7fdab7fff640 1 -- 172.21.15.164:0/348779014 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80690a0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab7fff640 1 -- 172.21.15.164:0/348779014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 msgr2=0x7fdab809d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab7fff640 1 --2- 172.21.15.164:0/348779014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009870 tx=0x7fdaa802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b17ec0672b3e799 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 1 -- 172.21.15.164:0/348779014 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037070 con 0x7fdab80a0bc0 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: dump: 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:48.263 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:48.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 1 -- 172.21.15.164:0/348779014 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa8032440 con 0x7fdab80a0bc0 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 5 monclient: authenticate success, global_id 4688 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 1 -- 172.21.15.164:0/348779014 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8032810 con 0x7fdab80a0bc0 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 4 set_mon_vals no callback set 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdab6ffd640 10 monclient: dump: 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:48.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 10 monclient: shutdown 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 -- 172.21.15.164:0/348779014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 msgr2=0x7fdab809d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/348779014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fdaa8009870 tx=0x7fdaa802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 -- 172.21.15.164:0/348779014 shutdown_connections 2024-09-10T16:09:48.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/348779014 >> v2:172.21.15.165:3300/0 conn(0x7fdab8136260 0x7fdab809dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/348779014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab809d960 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 -- 172.21.15.164:0/348779014 >> 172.21.15.164:0/348779014 conn(0x7fdab80a3fb0 msgr2=0x7fdab80a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.254+0000 7fdabe3f8640 1 -- 172.21.15.164:0/348779014 shutdown_connections 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.255+0000 7fdabe3f8640 1 -- 172.21.15.164:0/348779014 wait complete. 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.255+0000 7fdabe3f8640 10 monclient: build_initial_monmap 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.255+0000 7fdabe3f8640 10 monclient: monmap: 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.267 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.255+0000 7fdabe3f8640 1 Processor -- start 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 -- start start 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient: init 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-c con 0x7fdab80a0bc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdab80a2ec0 0x7fdab809a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-a con 0x7fdab80a2ec0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab8136260 0x7fdab809aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): picked mon.noname-b con 0x7fdab8136260 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdab80689b0 con 0x7fdab80a2ec0 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdab809e3e0 con 0x7fdab8136260 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdab8068b30 con 0x7fdab80a0bc0 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabe3f8640 10 monclient(hunting): authenticate will time out at 966.743623s 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdab80a2ec0 0x7fdab809a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37014/0 (socket says 172.21.15.164:37014) 2024-09-10T16:09:48.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 1 -- 172.21.15.164:0/3789474851 learned_addr learned my addr 172.21.15.164:0/3789474851 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): get_auth_request con 0x7fdab80a0bc0 auth_method 0 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): get_auth_request con 0x7fdab80a2ec0 auth_method 0 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:48.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdabc995640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab8136260 0x7fdab809aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 10 monclient(hunting): handle_auth_done global_id 4691 payload 824 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 1 -- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab8136260 msgr2=0x7fdab809aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.256+0000 7fdab7fff640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab8136260 0x7fdab809aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 1 -- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdab80a2ec0 msgr2=0x7fdab809a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdab80a2ec0 0x7fdab809a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 1 monclient: found mon.noname-c 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.257+0000 7fdab7fff640 1 -- 172.21.15.164:0/3789474851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdab80a06f0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab7fff640 1 -- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 msgr2=0x7fdab81cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab7fff640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80090b0 tx=0x7fdaa80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8041020 con 0x7fdab80a0bc0 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: dump: 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:48.272 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdaa803d630 con 0x7fdab80a0bc0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 5 monclient: authenticate success, global_id 4691 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 10 monclient: _renew_subs 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdab809f030 con 0x7fdab80a0bc0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdaa8037040 con 0x7fdab80a0bc0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:48.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab4ff9640 10 monclient: dump: 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 10 monclient: _renew_subs 2024-09-10T16:09:48.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdab809f4b0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.259+0000 7fdab5ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdabe3f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdabe3f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdabe3f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdaa8044020 con 0x7fdab80a0bc0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdab80a3b60 con 0x7fdab80a0bc0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdab4ff9640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 0x7fdaa40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.260+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(127..127 src has 1..127) ==== 4815+0+0 (secure 0 0 0) 0x7fdaa8072d80 con 0x7fdab80a0bc0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.261+0000 7fdab77fe640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 0x7fdaa40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:48.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.261+0000 7fdab77fe640 10 monclient: get_auth_request con 0x7fdaa403e210 auth_method 0 2024-09-10T16:09:48.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.263+0000 7fdab77fe640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 0x7fdaa40406d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdab809f440 tx=0x7fdaac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:48.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.265+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdaa804b070 con 0x7fdab80a0bc0 2024-09-10T16:09:48.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.265+0000 7fdab4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:48.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.265+0000 7fdab4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:48.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.508+0000 7fdabe3f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:48.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.508+0000 7fdabe3f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:48.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.508+0000 7fdabe3f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:48.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.508+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdab80772c0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.509+0000 7fdab4ff9640 1 -- 172.21.15.164:0/3789474851 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fdaa803b3f0 con 0x7fdab80a0bc0 2024-09-10T16:09:48.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.509+0000 7fdab4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:48.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.509+0000 7fdab4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:48.511 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":34,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:48.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 msgr2=0x7fdaa40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:48.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 0x7fdaa40406d0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fdab809f440 tx=0x7fdaac008040 comp rx=0 tx=0).stop 2024-09-10T16:09:48.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 10 monclient: shutdown 2024-09-10T16:09:48.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:48.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 msgr2=0x7fdab81cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fdaa80090b0 tx=0x7fdaa80046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 shutdown_connections 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdaa403e210 0x7fdaa40406d0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdab8136260 0x7fdab809aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdab80a2ec0 0x7fdab809a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 --2- 172.21.15.164:0/3789474851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdab80a0bc0 0x7fdab81cd1b0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 >> 172.21.15.164:0/3789474851 conn(0x7fdab80a3fb0 msgr2=0x7fdab809c480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 shutdown_connections 2024-09-10T16:09:48.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:48.511+0000 7fdabe3f8640 1 -- 172.21.15.164:0/3789474851 wait complete. 2024-09-10T16:09:51.524 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:51.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.632+0000 7f53ef02a640 10 monclient: get_monmap_and_config 2024-09-10T16:09:51.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.632+0000 7f53ef02a640 10 monclient: build_initial_monmap 2024-09-10T16:09:51.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.632+0000 7f53ef02a640 10 monclient: monmap: 2024-09-10T16:09:51.634 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:51.633725+0000 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:51.633725+0000 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:51.635 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:51.636 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:51.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.633+0000 7f53ef02a640 1 Processor -- start 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.633+0000 7f53ef02a640 1 -- start start 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.633+0000 7f53ef02a640 10 monclient: init 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-c con 0x7f53e809d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53e809ab50 0x7f53e809e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-a con 0x7f53e809ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53e8134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53e80a0870 con 0x7f53e8134580 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f53e809d3e0 con 0x7f53e809ab50 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f53e809e790 con 0x7f53e809d930 2024-09-10T16:09:51.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ef02a640 10 monclient(hunting): authenticate will time out at 970.121194s 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53e809ab50 0x7f53e809e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53e809ab50 0x7f53e809e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38722/0 (socket says 172.21.15.164:38722) 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 -- 172.21.15.164:0/2888925826 learned_addr learned my addr 172.21.15.164:0/2888925826 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): get_auth_request con 0x7f53e809ab50 auth_method 0 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:51.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 10 monclient(hunting): get_auth_request con 0x7f53e809d930 auth_method 0 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient(hunting): handle_auth_done global_id 24306 payload 274 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 1 -- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 msgr2=0x7f53e809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 1 -- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 msgr2=0x7f53e809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 -- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 msgr2=0x7f53e809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 --2- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 -- 172.21.15.164:0/2888925826 >> v1:172.21.15.165:6789/0 conn(0x7f53e8134580 legacy=0x7f53e8134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 monclient: found mon.noname-a 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.634+0000 7f53e7fff640 1 -- 172.21.15.164:0/2888925826 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e8074c70 con 0x7f53e809ab50 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e7fff640 1 -- 172.21.15.164:0/2888925826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 msgr2=0x7f53e809e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:51.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e7fff640 1 --2- 172.21.15.164:0/2888925826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e809e250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f53e806d500 tx=0x7f53d002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdaabb335128dad7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 1 -- 172.21.15.164:0/2888925826 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0037070 con 0x7f53e809ab50 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: dump: 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:51.652 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 1 -- 172.21.15.164:0/2888925826 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d0032640 con 0x7f53e809ab50 2024-09-10T16:09:51.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 1 -- 172.21.15.164:0/2888925826 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0032a10 con 0x7f53e809ab50 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53ef02a640 5 monclient: authenticate success, global_id 24306 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53ef02a640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53ef02a640 4 set_mon_vals no callback set 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.637+0000 7f53e77fe640 10 monclient: dump: 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.654 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 10 monclient: shutdown 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 -- 172.21.15.164:0/2888925826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 msgr2=0x7f53e809e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 --2- 172.21.15.164:0/2888925826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e809e250 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f53e806d500 tx=0x7f53d002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 -- 172.21.15.164:0/2888925826 shutdown_connections 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 --2- 172.21.15.164:0/2888925826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e809e250 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 --2- 172.21.15.164:0/2888925826 >> v2:172.21.15.165:3300/0 conn(0x7f53e809d930 0x7f53e809dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 -- 172.21.15.164:0/2888925826 >> 172.21.15.164:0/2888925826 conn(0x7f53e80a3110 msgr2=0x7f53e80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 -- 172.21.15.164:0/2888925826 shutdown_connections 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 1 -- 172.21.15.164:0/2888925826 wait complete. 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 10 monclient: build_initial_monmap 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.638+0000 7f53ef02a640 10 monclient: monmap: 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:51.656 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 Processor -- start 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 -- start start 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient: init 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-a con 0x7f53e809ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:51.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-b con 0x7f53e809d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 0x7f53e812c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): picked mon.noname-c con 0x7f53e8134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53e809ef30 con 0x7f53e809ab50 2024-09-10T16:09:51.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f53e80a02b0 con 0x7f53e809d930 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53e80a2940 con 0x7f53e8134580 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.639+0000 7f53ef02a640 10 monclient(hunting): authenticate will time out at 970.126992s 2024-09-10T16:09:51.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38728/0 (socket says 172.21.15.164:38728) 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 -- 172.21.15.164:0/4286869740 learned_addr learned my addr 172.21.15.164:0/4286869740 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): get_auth_request con 0x7f53e809ab50 auth_method 0 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:51.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 0x7f53e812c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): get_auth_request con 0x7f53e8134580 auth_method 0 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:51.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 10 monclient(hunting): get_auth_request con 0x7f53e809d930 auth_method 0 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 msgr2=0x7f53e812be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 msgr2=0x7f53e812be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53e7fff640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient(hunting): handle_auth_done global_id 24309 payload 824 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 msgr2=0x7f53e812c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 0x7f53e812c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 msgr2=0x7f53e812be70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient: _finish_hunting 0 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 monclient: found mon.noname-a 2024-09-10T16:09:51.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ecdc6640 1 -- 172.21.15.164:0/4286869740 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53e812d3a0 con 0x7f53e809ab50 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.640+0000 7f53ed5c7640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 0x7f53e812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ecdc6640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 msgr2=0x7f53e812b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ecdc6640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007ea0 tx=0x7f53d8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8013790 con 0x7f53e809ab50 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: dump: 2024-09-10T16:09:51.665 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: _finish_auth 0 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:09:51.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 5 monclient: authenticate success, global_id 24309 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53d8019070 con 0x7f53e809ab50 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8013b90 con 0x7f53e809ab50 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 10 monclient: _renew_subs 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53e812f700 con 0x7f53e809ab50 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e6ffd640 4 set_mon_vals no callback set 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53e57fa640 10 monclient: dump: 2024-09-10T16:09:51.667 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 10 monclient: _renew_subs 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.642+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53e812df00 con 0x7f53e809ab50 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.643+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f53d801e460 con 0x7f53e809ab50 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.643+0000 7f53ef02a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.643+0000 7f53ef02a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.643+0000 7f53ef02a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.643+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53e80a1e40 con 0x7f53e809ab50 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.644+0000 7f53e57fa640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 0x7f53c8040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.644+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 4783+0+0 (secure 0 0 0) 0x7f53d8053630 con 0x7f53e809ab50 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.644+0000 7f53e7fff640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 0x7f53c8040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:51.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.644+0000 7f53e7fff640 10 monclient: get_auth_request con 0x7f53c803e290 auth_method 0 2024-09-10T16:09:51.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.647+0000 7f53e7fff640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 0x7f53c8040750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f53d0034000 tx=0x7f53d002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:51.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.648+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f53d801cc60 con 0x7f53e809ab50 2024-09-10T16:09:51.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.648+0000 7f53e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:51.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.648+0000 7f53e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53ef02a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53ef02a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53ef02a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f53e806dac0 con 0x7f53e809ab50 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53e57fa640 1 -- 172.21.15.164:0/4286869740 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f53d8052030 con 0x7f53e809ab50 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.894+0000 7f53e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:51.896 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 msgr2=0x7f53c8040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 0x7f53c8040750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f53d0034000 tx=0x7f53d002e000 comp rx=0 tx=0).stop 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 10 monclient: shutdown 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 msgr2=0x7f53e812b930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.896+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f53d8007ea0 tx=0x7f53d8007f90 comp rx=0 tx=0).stop 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 shutdown_connections 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53c803e290 0x7f53c8040750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53e8134580 0x7f53e812c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53e809d930 0x7f53e812be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 --2- 172.21.15.164:0/4286869740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53e809ab50 0x7f53e812b930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:51.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 >> 172.21.15.164:0/4286869740 conn(0x7f53e80a3110 msgr2=0x7f53e8099600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:51.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 shutdown_connections 2024-09-10T16:09:51.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:51.897+0000 7f53ef02a640 1 -- 172.21.15.164:0/4286869740 wait complete. 2024-09-10T16:09:51.909 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:09:51.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:09:51.909 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:09:52.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.006+0000 7f75f09e2640 10 monclient: get_monmap_and_config 2024-09-10T16:09:52.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.006+0000 7f75f09e2640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.006+0000 7f75f09e2640 10 monclient: monmap: 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:52.007610+0000 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:52.007610+0000 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 Processor -- start 2024-09-10T16:09:52.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 -- start start 2024-09-10T16:09:52.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient: init 2024-09-10T16:09:52.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f75ec09c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75ec09d160 0x7f75ec09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-a con 0x7f75ec09d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75ec13a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ec09a430 con 0x7f75ec13a5c0 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75ec0a24e0 con 0x7f75ec09c180 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f75ec0a2bf0 con 0x7f75ec09d160 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.007+0000 7f75f09e2640 10 monclient(hunting): authenticate will time out at 970.494989s 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75ec09d160 0x7f75ec09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75ead9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f75ec13a5c0 0x7f75ec13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39284/0 (socket says 172.21.15.164:39284) 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75ead9d640 1 -- 172.21.15.164:0/1497154608 learned_addr learned my addr 172.21.15.164:0/1497154608 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75ec09d160 0x7f75ec09f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38742/0 (socket says 172.21.15.164:38742) 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): get_auth_request con 0x7f75ec09d160 auth_method 0 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient(hunting): handle_auth_done global_id 24312 payload 274 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1497154608 >> v1:172.21.15.165:6789/0 conn(0x7f75ec09c180 legacy=0x7f75ec09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1497154608 >> v1:172.21.15.164:6789/0 conn(0x7f75ec13a5c0 legacy=0x7f75ec13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.008+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1497154608 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec09c560 con 0x7f75ec09d160 2024-09-10T16:09:52.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1497154608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec09d160 msgr2=0x7f75ec09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1497154608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec09d160 0x7f75ec09f150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f75e00097d0 tx=0x7f75e002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ede630c80f8437f9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 1 -- 172.21.15.164:0/1497154608 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0037070 con 0x7f75ec09d160 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: dump: 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 1 -- 172.21.15.164:0/1497154608 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e0032440 con 0x7f75ec09d160 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 5 monclient: authenticate success, global_id 24312 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 1 -- 172.21.15.164:0/1497154608 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e0032810 con 0x7f75ec09d160 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 4 set_mon_vals no callback set 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75e959a640 10 monclient: dump: 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.022 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 10 monclient: shutdown 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 -- 172.21.15.164:0/1497154608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec09d160 msgr2=0x7f75ec09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1497154608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec09d160 0x7f75ec09f150 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f75e00097d0 tx=0x7f75e002a000 comp rx=0 tx=0).stop 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 -- 172.21.15.164:0/1497154608 shutdown_connections 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1497154608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec09d160 0x7f75ec09f150 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 -- 172.21.15.164:0/1497154608 >> 172.21.15.164:0/1497154608 conn(0x7f75ec0a3110 msgr2=0x7f75ec0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 -- 172.21.15.164:0/1497154608 shutdown_connections 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.011+0000 7f75f09e2640 1 -- 172.21.15.164:0/1497154608 wait complete. 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.012+0000 7f75f09e2640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.012+0000 7f75f09e2640 10 monclient: monmap: 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:52.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.012+0000 7f75f09e2640 1 Processor -- start 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 -- start start 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient: init 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:52.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 0x7f75ec131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-c con 0x7f75ec09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 0x7f75ec06e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-b con 0x7f75ec09d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): picked mon.noname-a con 0x7f75ec13a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec074f90 con 0x7f75ec13a5c0 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec074e10 con 0x7f75ec09d160 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f75ec09f690 con 0x7f75ec09c180 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75f09e2640 10 monclient(hunting): authenticate will time out at 970.500472s 2024-09-10T16:09:52.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 0x7f75ec131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38752/0 (socket says 172.21.15.164:38752) 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 0x7f75ec131090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37040/0 (socket says 172.21.15.164:37040) 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 learned_addr learned my addr 172.21.15.164:0/1326709197 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): get_auth_request con 0x7f75ec13a5c0 auth_method 0 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): get_auth_request con 0x7f75ec09c180 auth_method 0 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 0x7f75ec06e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 10 monclient(hunting): get_auth_request con 0x7f75ec09d160 auth_method 0 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 msgr2=0x7f75ec06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 msgr2=0x7f75ec06e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:52.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 0x7f75ec06e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 0x7f75ec06e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient(hunting): handle_auth_done global_id 24315 payload 824 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 msgr2=0x7f75ec131090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 0x7f75ec131090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 msgr2=0x7f75ec06e800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75ec09d160 0x7f75ec06e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ec06fb20 con 0x7f75ec13a5c0 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.013+0000 7f75ea59c640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75ec09c180 0x7f75ec131090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 msgr2=0x7f75ec06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75ead9d640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f75e400cac0 tx=0x7f75e400cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e400ae40 con 0x7f75ec13a5c0 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: dump: 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.029 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75e4012400 con 0x7f75ec13a5c0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 5 monclient: authenticate success, global_id 24315 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 10 monclient: _renew_subs 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75ec130de0 con 0x7f75ec13a5c0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75e40127f0 con 0x7f75ec13a5c0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75daffd640 10 monclient: dump: 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75e8d99640 4 set_mon_vals no callback set 2024-09-10T16:09:52.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 10 monclient: _renew_subs 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.016+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75ec131ff0 con 0x7f75ec13a5c0 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75f09e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 0x7f75ec02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec1329b0 con 0x7f75ec02d440 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 0x7f75ec02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f75e4012a10 con 0x7f75ec13a5c0 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient: get_auth_request con 0x7f75ec02d440 auth_method 0 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75daffd640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75cc03e210 0x7f75cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 10 monclient(hunting): handle_auth_done global_id 24318 payload 824 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(128..128 src has 1..128) ==== 4783+0+0 (secure 0 0 0) 0x7f75e40533d0 con 0x7f75ec13a5c0 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75ea59c640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f75ec09ea40 con 0x7f75ec02d440 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.017+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75cc03e210 0x7f75cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.018+0000 7f75e9d9b640 10 monclient: get_auth_request con 0x7f75cc03e210 auth_method 0 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75ea59c640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 msgr2=0x7f75ec02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75ea59c640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 0x7f75ec02d820 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00fb10 tx=0x7f75dc0044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc018020 con 0x7f75ec02d440 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.019+0000 7f75daffd640 10 monclient: dump: 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.036 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f75dc00dd90 con 0x7f75ec02d440 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 msgr2=0x7f75ec02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 0x7f75ec02d820 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00fb10 tx=0x7f75dc0044c0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 msgr2=0x7f75ec02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75daffd640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02d440 0x7f75ec02d820 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.020+0000 7f75e9d9b640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75cc03e210 0x7f75cc0406d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75ec09a720 tx=0x7f75e002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75f09e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 0x7f75ec02e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75f09e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75ec077260 con 0x7f75ec02dd60 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 0x7f75ec02e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient: get_auth_request con 0x7f75ec02dd60 auth_method 0 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 10 monclient(hunting): handle_auth_done global_id 24321 payload 824 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.026+0000 7f75ea59c640 1 -- 172.21.15.164:0/1326709197 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f75ec072090 con 0x7f75ec02dd60 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75ea59c640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 msgr2=0x7f75ec02e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75ea59c640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 0x7f75ec02e140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00fb10 tx=0x7f75dc0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75dc018030 con 0x7f75ec02dd60 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75daffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.028+0000 7f75daffd640 10 monclient: dump: 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.039 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7f75dc01d420 con 0x7f75ec02dd60 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 msgr2=0x7f75ec02e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 0x7f75ec02e140 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00fb10 tx=0x7f75dc0036f0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 msgr2=0x7f75ec02e140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75daffd640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec02dd60 0x7f75ec02e140 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f75dc00fb10 tx=0x7f75dc0036f0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 116961, 2024-09-10T16:09:52.041 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.042 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.043 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:52.044 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.045 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.046 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:52.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.049 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:52.050 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.051 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.052 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:52.053 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:52.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75cc03e210 msgr2=0x7f75cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75cc03e210 0x7f75cc0406d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f75ec09a720 tx=0x7f75e002e000 comp rx=0 tx=0).stop 2024-09-10T16:09:52.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 10 monclient: shutdown 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 msgr2=0x7f75ec06ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f75e400cac0 tx=0x7f75e400cf80 comp rx=0 tx=0).stop 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75ead9d640 1 -- 172.21.15.164:0/1326709197 reap_dead start 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 shutdown_connections 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 --2- 172.21.15.164:0/1326709197 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75ec13a5c0 0x7f75ec06ed40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 >> 172.21.15.164:0/1326709197 conn(0x7f75ec0a3110 msgr2=0x7f75ec139b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 shutdown_connections 2024-09-10T16:09:52.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.029+0000 7f75f09e2640 1 -- 172.21.15.164:0/1326709197 wait complete. 2024-09-10T16:09:52.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:52.056 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:09:52.056 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:09:52.099 INFO:tasks.rados.rados.0.smithi165.stdout:474: done (2 left) 2024-09-10T16:09:52.099 INFO:tasks.rados.rados.0.smithi165.stdout:475: delete oid 30 current snap is 61 2024-09-10T16:09:52.099 INFO:tasks.rados.rados.0.smithi165.stdout:473: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:09:52.099 INFO:tasks.rados.rados.0.smithi165.stdout:472: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:09:52.099 INFO:tasks.rados.rados.0.smithi165.stdout:473: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:09:52.100 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 46 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:09:52.100 INFO:tasks.rados.rados.0.smithi165.stdout:472: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:09:52.100 INFO:tasks.rados.rados.0.smithi165.stdout:472: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:09:52.100 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 243 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:09:52.107 INFO:tasks.rados.rados.0.smithi165.stdout:472: done (2 left) 2024-09-10T16:09:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:473: done (1 left) 2024-09-10T16:09:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:475: done (0 left) 2024-09-10T16:09:52.108 INFO:tasks.rados.rados.0.smithi165.stdout:476: snap_remove snap 55 2024-09-10T16:09:52.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 10 monclient: get_monmap_and_config 2024-09-10T16:09:52.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 10 monclient: monmap: 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:52.161075+0000 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:52.161075+0000 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:52.162 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 1 Processor -- start 2024-09-10T16:09:52.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 1 -- start start 2024-09-10T16:09:52.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.160+0000 7fd8502b6640 10 monclient: init 2024-09-10T16:09:52.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd84806cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 0x7fd84813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd848134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd84813d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd84809ce50 con 0x7fd84813d560 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd848135380 con 0x7fd84806cb70 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd84806db60 con 0x7fd848134f70 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd8502b6640 10 monclient(hunting): authenticate will time out at 970.648512s 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84e853640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd84813d560 0x7fd84813f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39286/0 (socket says 172.21.15.164:39286) 2024-09-10T16:09:52.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84e853640 1 -- 172.21.15.164:0/346825337 learned_addr learned my addr 172.21.15.164:0/346825337 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 1 --2- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 0x7fd84813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 10 monclient(hunting): get_auth_request con 0x7fd848134f70 auth_method 0 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 1 -- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 msgr2=0x7fd84813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 1 -- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 msgr2=0x7fd84813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 1 --2- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 0x7fd84813d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.161+0000 7fd84d851640 1 --2- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 0x7fd84813d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84e853640 1 -- 172.21.15.164:0/346825337 >> v1:172.21.15.164:6789/0 conn(0x7fd84813d560 legacy=0x7fd84813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523174764 0 0) 0x7fd8400017d0 con 0x7fd84813d560 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 10 monclient(hunting): _negotiate 2024-09-10T16:09:52.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 10 monclient(hunting): my global_id is 24324 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84e052640 1 -- 172.21.15.164:0/346825337 >> v1:172.21.15.165:6789/0 conn(0x7fd84806cb70 legacy=0x7fd8480a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.163+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd834003520 con 0x7fd84813d560 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2321797743 0 0) 0x7fd834003520 con 0x7fd84813d560 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 msgr2=0x7fd84813d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 --2- 172.21.15.164:0/346825337 >> v2:172.21.15.165:3300/0 conn(0x7fd848134f70 0x7fd84813d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 >> v1:172.21.15.165:6789/0 conn(0x7fd84806cb70 legacy=0x7fd8480a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84809cb70 con 0x7fd84813d560 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fd840001ef0 con 0x7fd84813d560 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd8502b6640 5 monclient: authenticate success, global_id 24324 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd8502b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:09:52.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: dump: 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 >> v1:172.21.15.164:6789/0 conn(0x7fd84813d560 legacy=0x7fd84813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 0x7fd834003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): picked mon.a con 0x7fd834003b40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834004800 0x7fd834004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): picked mon.c con 0x7fd834004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8340099b0 0x7fd83400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): picked mon.b con 0x7fd8340099b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd83400c3c0 con 0x7fd834003b40 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd83400c570 con 0x7fd8340099b0 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd83400c770 con 0x7fd834004800 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d050640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd8502b6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 0x7fd834003f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834004800 0x7fd834004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): get_auth_request con 0x7fd834003b40 auth_method 0 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): get_auth_request con 0x7fd834004800 auth_method 0 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8340099b0 0x7fd83400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): get_auth_request con 0x7fd8340099b0 auth_method 0 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient(hunting): handle_auth_done global_id 24324 payload 293 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 -- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834004800 msgr2=0x7fd834004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834004800 0x7fd834004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 -- 172.21.15.164:0/346825337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8340099b0 msgr2=0x7fd83400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8340099b0 0x7fd83400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 monclient: found mon.a 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e853640 1 -- 172.21.15.164:0/346825337 reap_dead start 2024-09-10T16:09:52.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 1 -- 172.21.15.164:0/346825337 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd834003520 con 0x7fd834003b40 2024-09-10T16:09:52.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.164+0000 7fd84e052640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84e052640 1 -- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 msgr2=0x7fd834003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84e052640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 0x7fd834003f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd844001990 tx=0x7fd834005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2f2354e2b68046 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844001690 con 0x7fd834003b40 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: dump: 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.184 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd84402e070 con 0x7fd834003b40 2024-09-10T16:09:52.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd8502b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 1 -- 172.21.15.164:0/346825337 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd844007070 con 0x7fd834003b40 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd8502b6640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd8502b6640 4 set_mon_vals no callback set 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.167+0000 7fd84d050640 10 monclient: dump: 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.186 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 10 monclient: shutdown 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 -- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 msgr2=0x7fd834003f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 0x7fd834003f60 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd844001990 tx=0x7fd834005830 comp rx=0 tx=0).stop 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 -- 172.21.15.164:0/346825337 shutdown_connections 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 --2- 172.21.15.164:0/346825337 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd834003b40 0x7fd834003f60 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 -- 172.21.15.164:0/346825337 >> 172.21.15.164:0/346825337 conn(0x7fd8480aa900 msgr2=0x7fd8480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 -- 172.21.15.164:0/346825337 shutdown_connections 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.168+0000 7fd8502b6640 1 -- 172.21.15.164:0/346825337 wait complete. 2024-09-10T16:09:52.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient: monmap: 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 Processor -- start 2024-09-10T16:09:52.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 -- start start 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient: init 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834003b40 0x7fd84809dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-c con 0x7fd834003b40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 0x7fd84809e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-a con 0x7fd84809e500 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8481cd700 0x7fd8481cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): picked mon.noname-b con 0x7fd8481cd700 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd8480a46a0 con 0x7fd84809e500 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd84806cf50 con 0x7fd8481cd700 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd8480a4820 con 0x7fd834003b40 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.169+0000 7fd8502b6640 10 monclient(hunting): authenticate will time out at 970.656964s 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834003b40 0x7fd84809dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 0x7fd84809e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 0x7fd84809e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38782/0 (socket says 172.21.15.164:38782) 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834003b40 0x7fd84809dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37062/0 (socket says 172.21.15.164:37062) 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 -- 172.21.15.164:0/3717851188 learned_addr learned my addr 172.21.15.164:0/3717851188 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): get_auth_request con 0x7fd84809e500 auth_method 0 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): get_auth_request con 0x7fd834003b40 auth_method 0 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8481cd700 0x7fd8481cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 10 monclient(hunting): get_auth_request con 0x7fd8481cd700 auth_method 0 2024-09-10T16:09:52.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient(hunting): handle_auth_done global_id 24327 payload 824 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834003b40 msgr2=0x7fd84809dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd834003b40 0x7fd84809dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8481cd700 msgr2=0x7fd8481cdb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8481cd700 0x7fd8481cdb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84d851640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8480a1a60 con 0x7fd84809e500 2024-09-10T16:09:52.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.170+0000 7fd84e853640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd8481cd700 0x7fd8481cdb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd84d851640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 msgr2=0x7fd84809e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd84d851640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 0x7fd84809e980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd838010820 tx=0x7fd838010ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83801a020 con 0x7fd84809e500 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: dump: 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.193 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd83800ebd0 con 0x7fd84809e500 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 5 monclient: authenticate success, global_id 24327 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 10 monclient: _renew_subs 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8480a1d40 con 0x7fd84809e500 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd838016480 con 0x7fd84809e500 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd83effd640 10 monclient: dump: 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.195 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 10 monclient: _renew_subs 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd848134010 con 0x7fd84809e500 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.172+0000 7fd84c84f640 4 set_mon_vals no callback set 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd8502b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd8502b6640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 0x7fd8481ce0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd84802e9a0 con 0x7fd84802e5c0 2024-09-10T16:09:52.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd83801d020 con 0x7fd84809e500 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 0x7fd8481ce0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient: get_auth_request con 0x7fd84802e5c0 auth_method 0 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd84e052640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd8480a4c50 con 0x7fd84802e5c0 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.173+0000 7fd83effd640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd82c03e240 0x7fd82c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.174+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4783+0+0 (secure 0 0 0) 0x7fd838053590 con 0x7fd84809e500 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.174+0000 7fd84e853640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd82c03e240 0x7fd82c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.174+0000 7fd84e853640 10 monclient: get_auth_request con 0x7fd82c03e240 auth_method 0 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd84e052640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 msgr2=0x7fd8481ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd84e052640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 0x7fd8481ce0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd844032000 tx=0x7fd8440014b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8440059c0 con 0x7fd84802e5c0 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 10 monclient: dump: 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd84402e040 con 0x7fd84802e5c0 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 msgr2=0x7fd8481ce0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 0x7fd8481ce0d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fd844032000 tx=0x7fd8440014b0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 msgr2=0x7fd8481ce0d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd83effd640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd84802e5c0 0x7fd8481ce0d0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.176+0000 7fd84e853640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd82c03e240 0x7fd82c040700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd8480a5330 tx=0x7fd84000a040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd8502b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd8502b6640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 0x7fd848064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd8502b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd8480a1840 con 0x7fd848063d50 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 0x7fd848064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient: get_auth_request con 0x7fd848063d50 auth_method 0 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 10 monclient(hunting): handle_auth_done global_id 4706 payload 824 2024-09-10T16:09:52.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.182+0000 7fd84e052640 1 -- 172.21.15.164:0/3717851188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd84813a4c0 con 0x7fd848063d50 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd84e052640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 msgr2=0x7fd848064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd84e052640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 0x7fd848064160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd844001990 tx=0x7fd844007c20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8440462b0 con 0x7fd848063d50 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 10 monclient: dump: 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.202 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fd8440462b0 con 0x7fd848063d50 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 msgr2=0x7fd848064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 0x7fd848064160 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd844001990 tx=0x7fd844007c20 comp rx=0 tx=0).stop 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 msgr2=0x7fd848064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd83effd640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd848063d50 0x7fd848064160 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fd844001990 tx=0x7fd844007c20 comp rx=0 tx=0).stop 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 234357, 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.204 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.205 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.206 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:52.207 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.208 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:52.209 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.210 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.211 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.212 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.213 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.214 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.215 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.216 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:52.217 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:52.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd82c03e240 msgr2=0x7fd82c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd82c03e240 0x7fd82c040700 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd8480a5330 tx=0x7fd84000a040 comp rx=0 tx=0).stop 2024-09-10T16:09:52.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 10 monclient: shutdown 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 msgr2=0x7fd84809e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 --2- 172.21.15.164:0/3717851188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd84809e500 0x7fd84809e980 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd838010820 tx=0x7fd838010ce0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd84e853640 1 -- 172.21.15.164:0/3717851188 reap_dead start 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd84e853640 1 -- 172.21.15.164:0/3717851188 reap_dead start 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 shutdown_connections 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 >> 172.21.15.164:0/3717851188 conn(0x7fd8480aa900 msgr2=0x7fd848137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 shutdown_connections 2024-09-10T16:09:52.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.185+0000 7fd8502b6640 1 -- 172.21.15.164:0/3717851188 wait complete. 2024-09-10T16:09:52.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:52.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:09:52.220 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:09:52.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 10 monclient: get_monmap_and_config 2024-09-10T16:09:52.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 10 monclient: monmap: 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:52.317973+0000 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:52.317973+0000 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:52.319 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 1 Processor -- start 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 1 -- start start 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.317+0000 7f05f75ec640 10 monclient: init 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-b con 0x7f05f006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f05f006d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 0x7f05f013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-a con 0x7f05f013d060 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05f009cb90 con 0x7f05f006d540 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f05f009ce70 con 0x7f05f013d060 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05f009d150 con 0x7f05f006d160 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f75ec640 10 monclient(hunting): authenticate will time out at 970.805541s 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f4b87640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f05f006d540 0x7f05f00a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39300/0 (socket says 172.21.15.164:39300) 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f4b87640 1 -- 172.21.15.164:0/3141249716 learned_addr learned my addr 172.21.15.164:0/3141249716 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 1 --2- 172.21.15.164:0/3141249716 >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 0x7f05f013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): get_auth_request con 0x7f05f006d160 auth_method 0 2024-09-10T16:09:52.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): get_auth_request con 0x7f05f013d060 auth_method 0 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient(hunting): handle_auth_done global_id 4709 payload 274 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 -- 172.21.15.164:0/3141249716 >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 msgr2=0x7f05f013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 --2- 172.21.15.164:0/3141249716 >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 0x7f05f013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 -- 172.21.15.164:0/3141249716 >> v1:172.21.15.164:6789/0 conn(0x7f05f006d540 legacy=0x7f05f00a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 monclient: found mon.noname-b 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5388640 1 -- 172.21.15.164:0/3141249716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f00a4df0 con 0x7f05f006d160 2024-09-10T16:09:52.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.318+0000 7f05f5b89640 1 --2- 172.21.15.164:0/3141249716 >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 0x7f05f013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f5388640 1 -- 172.21.15.164:0/3141249716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 msgr2=0x7f05f0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f5388640 1 --2- 172.21.15.164:0/3141249716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f0134e00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0097a0 tx=0x7f05ec029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f30d86b8addd2647 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 1 -- 172.21.15.164:0/3141249716 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec037070 con 0x7f05f006d160 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: dump: 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.331 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f75ec640 5 monclient: authenticate success, global_id 4709 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 1 -- 172.21.15.164:0/3141249716 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05ec032440 con 0x7f05f006d160 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f75ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f75ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05f75ec640 4 set_mon_vals no callback set 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 1 -- 172.21.15.164:0/3141249716 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec032810 con 0x7f05f006d160 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.321+0000 7f05e7fff640 10 monclient: dump: 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.333 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 10 monclient: shutdown 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 -- 172.21.15.164:0/3141249716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 msgr2=0x7f05f0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 --2- 172.21.15.164:0/3141249716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f0134e00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0097a0 tx=0x7f05ec029fd0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 -- 172.21.15.164:0/3141249716 shutdown_connections 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 --2- 172.21.15.164:0/3141249716 >> v2:172.21.15.164:3300/0 conn(0x7f05f013d060 0x7f05f013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 --2- 172.21.15.164:0/3141249716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f0134e00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 -- 172.21.15.164:0/3141249716 >> 172.21.15.164:0/3141249716 conn(0x7f05f00aacb0 msgr2=0x7f05f00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 -- 172.21.15.164:0/3141249716 shutdown_connections 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.322+0000 7f05f75ec640 1 -- 172.21.15.164:0/3141249716 wait complete. 2024-09-10T16:09:52.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient: monmap: 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 Processor -- start 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 -- start start 2024-09-10T16:09:52.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient: init 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f00a7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-c con 0x7f05f006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f006d540 0x7f05f013bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-b con 0x7f05f006d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 0x7f05f01cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): picked mon.noname-a con 0x7f05f013d060 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05f013fb10 con 0x7f05f013d060 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05f013f990 con 0x7f05f006d540 2024-09-10T16:09:52.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05f013fc90 con 0x7f05f006d160 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.323+0000 7f05f75ec640 10 monclient(hunting): authenticate will time out at 970.810966s 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f00a7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f00a7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37090/0 (socket says 172.21.15.164:37090) 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 learned_addr learned my addr 172.21.15.164:0/1424965929 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): get_auth_request con 0x7f05f006d160 auth_method 0 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 0x7f05f01cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): get_auth_request con 0x7f05f013d060 auth_method 0 2024-09-10T16:09:52.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f4b87640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f006d540 0x7f05f013bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient(hunting): handle_auth_done global_id 24333 payload 824 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 msgr2=0x7f05f00a7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f00a7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f006d540 msgr2=0x7f05f013bb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f006d540 0x7f05f013bb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5b89640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05f00a7e30 con 0x7f05f013d060 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f4b87640 10 monclient: get_auth_request con 0x7f05f006d540 auth_method 0 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05f006d160 0x7f05f00a7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.324+0000 7f05f4b87640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f006d540 0x7f05f013bb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f5b89640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 msgr2=0x7f05f01cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f5b89640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 0x7f05f01cd100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009e50 tx=0x7f05e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8018030 con 0x7f05f013d060 2024-09-10T16:09:52.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: dump: 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05e8003990 con 0x7f05f013d060 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 5 monclient: authenticate success, global_id 24333 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 10 monclient: _renew_subs 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05f006e6b0 con 0x7f05f013d060 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05e8003d60 con 0x7f05f013d060 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e5ffb640 10 monclient: dump: 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.341 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 10 monclient: _renew_subs 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05e77fe640 4 set_mon_vals no callback set 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.326+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05f0137da0 con 0x7f05f013d060 2024-09-10T16:09:52.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.327+0000 7f05f75ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.327+0000 7f05f75ec640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.327+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.327+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05f0136ea0 con 0x7f05f013c0b0 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.327+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f05e8005d10 con 0x7f05f013d060 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 10 monclient: get_auth_request con 0x7f05f013c0b0 auth_method 0 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05e5ffb640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05d403e240 0x7f05d4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 msgr2=0x7f05f01cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 msgr2=0x7f05f01cd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:52.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:52.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4783+0+0 (secure 0 0 0) 0x7f05e8053730 con 0x7f05f013d060 2024-09-10T16:09:52.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f4b87640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05d403e240 0x7f05d4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.328+0000 7f05f4b87640 10 monclient: get_auth_request con 0x7f05d403e240 auth_method 0 2024-09-10T16:09:52.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.330+0000 7f05f4b87640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05d403e240 0x7f05d4040700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05f0136b10 tx=0x7f05e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.529+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.529+0000 7f05f5388640 10 monclient: get_auth_request con 0x7f05f013c0b0 auth_method 0 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.529+0000 7f05f5388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.529+0000 7f05f5388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.529+0000 7f05f5388640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.530+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.530+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.530+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.530+0000 7f05f5388640 10 monclient(hunting): handle_auth_done global_id 44101 payload 824 2024-09-10T16:09:52.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.530+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05f0134e00 con 0x7f05f013c0b0 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 msgr2=0x7f05f01cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0097a0 tx=0x7f05ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec042070 con 0x7f05f013c0b0 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.532+0000 7f05e5ffb640 10 monclient: dump: 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.535 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05ec03ca30 con 0x7f05f013c0b0 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 msgr2=0x7f05f01cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0097a0 tx=0x7f05ec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 msgr2=0x7f05f01cd640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.533+0000 7f05e5ffb640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f013c0b0 0x7f05f01cd640 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f05ec0097a0 tx=0x7f05ec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.539+0000 7f05f75ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:52.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.539+0000 7f05f75ec640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 0x7f05f012d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.539+0000 7f05f75ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.539+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05f01cdf90 con 0x7f05f01cdb80 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.539+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 0x7f05f012d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient: get_auth_request con 0x7f05f01cdb80 auth_method 0 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 10 monclient(hunting): handle_auth_done global_id 44104 payload 824 2024-09-10T16:09:52.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.540+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05f00770b0 con 0x7f05f01cdb80 2024-09-10T16:09:52.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.542+0000 7f05f5388640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 msgr2=0x7f05f012d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.542+0000 7f05f5388640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 0x7f05f012d1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f05f0137020 tx=0x7f05ec009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.542+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05ec037040 con 0x7f05f01cdb80 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.542+0000 7f05e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.542+0000 7f05e5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 10 monclient: dump: 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f05ec04da80 con 0x7f05f01cdb80 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 msgr2=0x7f05f012d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 0x7f05f012d1b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f05f0137020 tx=0x7f05ec009190 comp rx=0 tx=0).stop 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 msgr2=0x7f05f012d1b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05e5ffb640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05f01cdb80 0x7f05f012d1b0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f05f0137020 tx=0x7f05ec009190 comp rx=0 tx=0).stop 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:52.546 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7694, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:09:52.547 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.548 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.549 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:52.550 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:52.551 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:52.552 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.554 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.555 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:52.556 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:52.557 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:52.558 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05d403e240 msgr2=0x7f05d4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05d403e240 0x7f05d4040700 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05f0136b10 tx=0x7f05e0009040 comp rx=0 tx=0).stop 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 10 monclient: shutdown 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 msgr2=0x7f05f01cd100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f5b89640 1 -- 172.21.15.164:0/1424965929 reap_dead start 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 0x7f05f01cd100 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f05e8009e50 tx=0x7f05e800c490 comp rx=0 tx=0).stop 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 shutdown_connections 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 --2- 172.21.15.164:0/1424965929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05f013d060 0x7f05f01cd100 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 >> 172.21.15.164:0/1424965929 conn(0x7f05f00aacb0 msgr2=0x7f05f0132760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 shutdown_connections 2024-09-10T16:09:52.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.543+0000 7f05f75ec640 1 -- 172.21.15.164:0/1424965929 wait complete. 2024-09-10T16:09:52.561 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:52.561 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:09:52.561 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:09:52.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 10 monclient: get_monmap_and_config 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 10 monclient: monmap: 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:52.657051+0000 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:52.657051+0000 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.658 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:52.659 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:52.659 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:52.659 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:52.659 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:52.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 1 Processor -- start 2024-09-10T16:09:52.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 1 -- start start 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.656+0000 7fd7bf8fe640 10 monclient: init 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd7b806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd7b8134f70 0x7fd7b813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b8134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd7b813d560 0x7fd7b813f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b813d560 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7b809ce50 con 0x7fd7b806cb70 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b806db60 con 0x7fd7b8134f70 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd7b806cf50 con 0x7fd7b813d560 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bf8fe640 10 monclient(hunting): authenticate will time out at 971.144553s 2024-09-10T16:09:52.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd7b8134f70 0x7fd7b813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bd69a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd7b806cb70 0x7fd7b80a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39304/0 (socket says 172.21.15.164:39304) 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bd69a640 1 -- 172.21.15.164:0/2436082467 learned_addr learned my addr 172.21.15.164:0/2436082467 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): get_auth_request con 0x7fd7b8134f70 auth_method 0 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bde9b640 1 --2- 172.21.15.164:0/2436082467 >> v2:172.21.15.165:3300/0 conn(0x7fd7b813d560 0x7fd7b813f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bde9b640 10 monclient(hunting): get_auth_request con 0x7fd7b813d560 auth_method 0 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bde9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bde9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bde9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient(hunting): handle_auth_done global_id 24336 payload 274 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 -- 172.21.15.164:0/2436082467 >> v2:172.21.15.165:3300/0 conn(0x7fd7b813d560 msgr2=0x7fd7b813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 --2- 172.21.15.164:0/2436082467 >> v2:172.21.15.165:3300/0 conn(0x7fd7b813d560 0x7fd7b813f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 -- 172.21.15.164:0/2436082467 >> v1:172.21.15.164:6789/0 conn(0x7fd7b806cb70 legacy=0x7fd7b80a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.657+0000 7fd7bce99640 1 -- 172.21.15.164:0/2436082467 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b8135380 con 0x7fd7b8134f70 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.660+0000 7fd7bce99640 1 -- 172.21.15.164:0/2436082467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b8134f70 msgr2=0x7fd7b813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.660+0000 7fd7bce99640 1 --2- 172.21.15.164:0/2436082467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b8134f70 0x7fd7b813d020 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097a0 tx=0x7fd7ac02a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38ec336ad2975b9e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 1 -- 172.21.15.164:0/2436082467 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac037070 con 0x7fd7b8134f70 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: dump: 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.677 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 5 monclient: authenticate success, global_id 24336 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 1 -- 172.21.15.164:0/2436082467 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7ac032440 con 0x7fd7b8134f70 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 1 -- 172.21.15.164:0/2436082467 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7ac032810 con 0x7fd7b8134f70 2024-09-10T16:09:52.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 4 set_mon_vals no callback set 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7b7fff640 10 monclient: dump: 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.679 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 10 monclient: shutdown 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/2436082467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b8134f70 msgr2=0x7fd7b813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/2436082467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b8134f70 0x7fd7b813d020 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd7ac0097a0 tx=0x7fd7ac02a010 comp rx=0 tx=0).stop 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/2436082467 shutdown_connections 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/2436082467 >> v2:172.21.15.165:3300/0 conn(0x7fd7b813d560 0x7fd7b813f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/2436082467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b8134f70 0x7fd7b813d020 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/2436082467 >> 172.21.15.164:0/2436082467 conn(0x7fd7b80aa900 msgr2=0x7fd7b80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/2436082467 shutdown_connections 2024-09-10T16:09:52.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.661+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/2436082467 wait complete. 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.662+0000 7fd7bf8fe640 10 monclient: build_initial_monmap 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.662+0000 7fd7bf8fe640 10 monclient: monmap: 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:52.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.662+0000 7fd7bf8fe640 1 Processor -- start 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.662+0000 7fd7bf8fe640 1 -- start start 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.662+0000 7fd7bf8fe640 10 monclient: init 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-a con 0x7fd7b806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7b8134f70 0x7fd7b813b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-b con 0x7fd7b8134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7b813d560 0x7fd7b813ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): picked mon.noname-c con 0x7fd7b813d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b80a4db0 con 0x7fd7b806cb70 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd7b809d130 con 0x7fd7b8134f70 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd7b813fe90 con 0x7fd7b813d560 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bf8fe640 10 monclient(hunting): authenticate will time out at 971.150139s 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38826/0 (socket says 172.21.15.164:38826) 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 -- 172.21.15.164:0/3849862418 learned_addr learned my addr 172.21.15.164:0/3849862418 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): get_auth_request con 0x7fd7b806cb70 auth_method 0 2024-09-10T16:09:52.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7b813d560 0x7fd7b813ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): get_auth_request con 0x7fd7b813d560 auth_method 0 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7b8134f70 0x7fd7b813b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bde9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): get_auth_request con 0x7fd7b8134f70 auth_method 0 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bce99640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient(hunting): handle_auth_done global_id 24339 payload 824 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 -- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7b813d560 msgr2=0x7fd7b813ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7b813d560 0x7fd7b813ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 -- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7b8134f70 msgr2=0x7fd7b813b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7b8134f70 0x7fd7b813b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient: _finish_hunting 0 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 monclient: found mon.noname-a 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.663+0000 7fd7bd69a640 1 -- 172.21.15.164:0/3849862418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7b80a9aa0 con 0x7fd7b806cb70 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bd69a640 1 -- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 msgr2=0x7fd7b80a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bd69a640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800ba50 tx=0x7fd7a800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8019070 con 0x7fd7b806cb70 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: dump: 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:52.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7a800ce10 con 0x7fd7b806cb70 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bf8fe640 5 monclient: authenticate success, global_id 24339 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bf8fe640 10 monclient: _renew_subs 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7a8014680 con 0x7fd7b806cb70 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bf8fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7b809d690 con 0x7fd7b806cb70 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b5ffb640 10 monclient: dump: 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:52.688 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.665+0000 7fd7b77fe640 4 set_mon_vals no callback set 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.666+0000 7fd7bf8fe640 10 monclient: _renew_subs 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.666+0000 7fd7bf8fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.666+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7b809e460 con 0x7fd7b806cb70 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.666+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd7a801c070 con 0x7fd7b806cb70 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bf8fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bf8fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bf8fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7b813cdb0 con 0x7fd7b806cb70 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7b5ffb640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 0x7fd7a4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(129..129 src has 1..129) ==== 4783+0+0 (secure 0 0 0) 0x7fd7a8052b80 con 0x7fd7b806cb70 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bce99640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 0x7fd7a4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.667+0000 7fd7bce99640 10 monclient: get_auth_request con 0x7fd7a403e260 auth_method 0 2024-09-10T16:09:52.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.669+0000 7fd7bce99640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 0x7fd7a4040720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7b806e110 tx=0x7fd7ac02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:52.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.671+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd7a8051580 con 0x7fd7b806cb70 2024-09-10T16:09:52.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.671+0000 7fd7b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:52.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.671+0000 7fd7b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.920+0000 7fd7bf8fe640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.920+0000 7fd7bf8fe640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.920+0000 7fd7bf8fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.920+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd7b80a46a0 con 0x7fd7b806cb70 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.921+0000 7fd7b5ffb640 1 -- 172.21.15.164:0/3849862418 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd7a801dd90 con 0x7fd7b806cb70 2024-09-10T16:09:52.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.921+0000 7fd7b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:09:52.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.921+0000 7fd7b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:52.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 msgr2=0x7fd7a4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 0x7fd7a4040720 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fd7b806e110 tx=0x7fd7ac02e000 comp rx=0 tx=0).stop 2024-09-10T16:09:52.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 10 monclient: shutdown 2024-09-10T16:09:52.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 msgr2=0x7fd7b80a3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd7a800ba50 tx=0x7fd7a800bf10 comp rx=0 tx=0).stop 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 shutdown_connections 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7a403e260 0x7fd7a4040720 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7b813d560 0x7fd7b813ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7b8134f70 0x7fd7b813b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 --2- 172.21.15.164:0/3849862418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7b806cb70 0x7fd7b80a3200 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 >> 172.21.15.164:0/3849862418 conn(0x7fd7b80aa900 msgr2=0x7fd7b813f210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.923+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 shutdown_connections 2024-09-10T16:09:52.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:52.924+0000 7fd7bf8fe640 1 -- 172.21.15.164:0/3849862418 wait complete. 2024-09-10T16:09:52.929 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.929 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=2262052658,config=2586702516,health=1363224884,kv=805391194,logm=360511585}) 2024-09-10T16:09:52.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=1,health=12,kv=12,logm=64} crc {auth=521271316,config=1747058506,health=1201404789,kv=805391194,logm=1787302398}) 2024-09-10T16:09:52.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=10,kv=12,logm=66} crc {auth=2262052658,config=2586702516,health=1363224884,kv=805391194,logm=360511585}) 2024-09-10T16:09:52.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.928+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=12,kv=12,logm=63} crc {auth=1557165054,config=2586702516,health=1535303502,kv=805391194,logm=2464138842}) 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73635867}) 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3848430142}) 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=73635867}) 2024-09-10T16:09:52.931 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.929+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614807218}) 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102492965}) 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675425489}) 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2102492965}) 2024-09-10T16:09:52.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.930+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2158271786}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379841728}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685084006}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379841728}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.931+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1140862739}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486182120}) 2024-09-10T16:09:52.934 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975646026}) 2024-09-10T16:09:52.935 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.935 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2486182120}) 2024-09-10T16:09:52.935 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.932+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3185468690}) 2024-09-10T16:09:52.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977940656}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322980863}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=977940656}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.933+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140979542}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700867399}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928521764}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2700867399}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.934+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3433442261}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901101270}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3875361676}) 2024-09-10T16:09:52.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901101270}) 2024-09-10T16:09:52.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.935+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742273506}) 2024-09-10T16:09:52.939 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:09:52.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927097500}) 2024-09-10T16:09:52.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=985159704}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1927097500}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3838147185}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.936+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5} crc {logm=1405942228,mdsmap=931935893,mgr=3878148497,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3032517829,mon_config_key=1946044851,monmap=765578399,nvmeofgw=598337686,osd_metadata=141423795}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=2} crc {logm=423685949,mdsmap=3140514009,mgr=639680456,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=729740788,mon_config_key=3145760012,monmap=1433040829,nvmeofgw=1596038039}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=52,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5} crc {logm=1405942228,mdsmap=931935893,mgr=3878148497,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3032517829,mon_config_key=1946044851,monmap=765578399,nvmeofgw=598337686,osd_metadata=141423795}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=54,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=4} crc {logm=244140037,mdsmap=931935893,mgr=2880672550,mgr_command_descs=3192186153,mgrstat=3894083707,mon_config_key=605496218,monmap=1433040829,nvmeofgw=598337686,osd_metadata=2836976987}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=28,osdmap=70} crc {osd_metadata=2076698823,osd_pg_creating=1501543992,osd_snap=1336192506,osdmap=3956762352}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=28,osdmap=70} crc {osd_metadata=2076698823,osd_pg_creating=1501543992,osd_snap=3602816214,osdmap=3129893181}) 2024-09-10T16:09:52.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=28,osdmap=70} crc {osd_metadata=2076698823,osd_pg_creating=1501543992,osd_snap=1336192506,osdmap=3956762352}) 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.937+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_metadata=1,osd_pg_creating=1,osd_snap=26,osdmap=72} crc {osd_metadata=2076698823,osd_pg_creating=1501543992,osd_snap=750313556,osdmap=1013490161}) 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3708335605}) 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1939479429}) 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3708335605}) 2024-09-10T16:09:52.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.938+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=120194544}) 2024-09-10T16:09:52.947 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.947 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=1051581949}) 2024-09-10T16:09:52.947 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=32} crc {osdmap=627162339}) 2024-09-10T16:09:52.947 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:09:52.948 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=1051581949}) 2024-09-10T16:09:52.948 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:09:52.946+0000 7f55338a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=33} crc {osdmap=2364961876}) 2024-09-10T16:09:53.113 INFO:tasks.rados.rados.0.smithi165.stdout:476: done (0 left) 2024-09-10T16:09:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:477: write (excl) oid 5 current snap is 61 2024-09-10T16:09:53.114 INFO:tasks.rados.rados.0.smithi165.stdout:477: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-10T16:09:53.119 INFO:tasks.rados.rados.0.smithi165.stdout:477: writing smithi16534841-5 from 561400 to 1036378 tid 1 2024-09-10T16:09:53.126 INFO:tasks.rados.rados.0.smithi165.stdout:477: writing smithi16534841-5 from 1820281 to 2340004 tid 2 2024-09-10T16:09:53.132 INFO:tasks.rados.rados.0.smithi165.stdout:477: writing smithi16534841-5 from 2859422 to 3317001 tid 3 2024-09-10T16:09:53.132 INFO:tasks.rados.rados.0.smithi165.stdout:478: delete oid 11 current snap is 61 2024-09-10T16:09:53.132 INFO:tasks.rados.rados.0.smithi165.stdout:477: finishing write tid 1 to smithi16534841-5 2024-09-10T16:09:53.132 INFO:tasks.rados.rados.0.smithi165.stdout:477: finishing write tid 2 to smithi16534841-5 2024-09-10T16:09:53.137 INFO:tasks.rados.rados.0.smithi165.stdout:477: finishing write tid 3 to smithi16534841-5 2024-09-10T16:09:53.137 INFO:tasks.rados.rados.0.smithi165.stdout:477: finishing write tid 4 to smithi16534841-5 2024-09-10T16:09:53.137 INFO:tasks.rados.rados.0.smithi165.stdout:478: done (1 left) 2024-09-10T16:09:53.137 INFO:tasks.rados.rados.0.smithi165.stdout:479: write (excl) oid 22 current snap is 61 2024-09-10T16:09:53.137 INFO:tasks.rados.rados.0.smithi165.stdout:479: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-10T16:09:53.143 INFO:tasks.rados.rados.0.smithi165.stdout:479: writing smithi16534841-22 from 595014 to 1220490 tid 1 2024-09-10T16:09:53.150 INFO:tasks.rados.rados.0.smithi165.stdout:479: writing smithi16534841-22 from 1820892 to 2344284 tid 2 2024-09-10T16:09:53.156 INFO:tasks.rados.rados.0.smithi165.stdout:479: writing smithi16534841-22 from 2797915 to 3394391 tid 3 2024-09-10T16:09:53.156 INFO:tasks.rados.rados.0.smithi165.stdout:480: write (excl) oid 15 current snap is 61 2024-09-10T16:09:53.156 INFO:tasks.rados.rados.0.smithi165.stdout:477: finishing write tid 5 to smithi16534841-5 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:477: oid 5 updating version 0 to 193 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:477: oid 5 updating version 193 to 196 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:477: oid 5 version 196 is already newer than 194 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:477: oid 5 version 196 is already newer than 195 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 196 (ObjNum 200 snap 61 seq_num 200) dirty exists 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:477: left oid 5 (ObjNum 200 snap 61 seq_num 200) 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:479: finishing write tid 1 to smithi16534841-22 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:479: finishing write tid 2 to smithi16534841-22 2024-09-10T16:09:53.157 INFO:tasks.rados.rados.0.smithi165.stdout:480: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-10T16:09:53.163 INFO:tasks.rados.rados.0.smithi165.stdout:480: writing smithi16534841-15 from 611821 to 1312546 tid 1 2024-09-10T16:09:53.171 INFO:tasks.rados.rados.0.smithi165.stdout:480: writing smithi16534841-15 from 1879374 to 2662777 tid 2 2024-09-10T16:09:53.175 INFO:tasks.rados.rados.0.smithi165.stdout:480: writing smithi16534841-15 from 3341691 to 3411821 tid 3 2024-09-10T16:09:53.175 INFO:tasks.rados.rados.0.smithi165.stdout:477: done (2 left) 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:481: copy_from oid 21 from oid 45 current snap is 61 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: finishing write tid 3 to smithi16534841-22 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: finishing write tid 4 to smithi16534841-22 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: finishing write tid 5 to smithi16534841-22 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: oid 22 updating version 0 to 108 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: oid 22 updating version 108 to 111 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: oid 22 version 111 is already newer than 109 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: oid 22 version 111 is already newer than 110 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 111 (ObjNum 202 snap 61 seq_num 202) dirty exists 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:479: left oid 22 (ObjNum 202 snap 61 seq_num 202) 2024-09-10T16:09:53.176 INFO:tasks.rados.rados.0.smithi165.stdout:480: finishing write tid 1 to smithi16534841-15 2024-09-10T16:09:53.177 INFO:tasks.rados.rados.0.smithi165.stdout:480: finishing write tid 2 to smithi16534841-15 2024-09-10T16:09:53.177 INFO:tasks.rados.rados.0.smithi165.stdout:479: done (2 left) 2024-09-10T16:09:53.177 INFO:tasks.rados.rados.0.smithi165.stdout:482: write oid 17 current snap is 61 2024-09-10T16:09:53.177 INFO:tasks.rados.rados.0.smithi165.stdout:482: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2024-09-10T16:09:53.183 INFO:tasks.rados.rados.0.smithi165.stdout:482: writing smithi16534841-17 from 628628 to 1404602 tid 1 2024-09-10T16:09:53.189 INFO:tasks.rados.rados.0.smithi165.stdout:482: writing smithi16534841-17 from 1937856 to 2464917 tid 2 2024-09-10T16:09:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:482: writing smithi16534841-17 from 3252761 to 3428628 tid 3 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:483: read oid 49 snap -1 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:483: expect (ObjNum 118 snap 34 seq_num 118) 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:484: read oid 32 snap -1 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:484: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:485: read oid 9 snap -1 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:485: expect deleted 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:486: copy_from oid 4 from oid 23 current snap is 61 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:480: finishing write tid 3 to smithi16534841-15 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:480: finishing write tid 4 to smithi16534841-15 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:480: finishing write tid 5 to smithi16534841-15 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:480: oid 15 updating version 0 to 126 2024-09-10T16:09:53.196 INFO:tasks.rados.rados.0.smithi165.stdout:480: oid 15 version 126 is already newer than 124 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:480: oid 15 version 126 is already newer than 123 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:480: oid 15 version 126 is already newer than 125 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 126 (ObjNum 203 snap 61 seq_num 203) dirty exists 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:480: left oid 15 (ObjNum 203 snap 61 seq_num 203) 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:481: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:482: finishing write tid 1 to smithi16534841-17 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:480: done (6 left) 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:487: copy_from oid 42 from oid 28 current snap is 61 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:481: finishing copy_from to smithi16534841-21 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 103 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:482: finishing write tid 2 to smithi16534841-17 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:481: done (6 left) 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:488: write (excl) oid 41 current snap is 61 2024-09-10T16:09:53.197 INFO:tasks.rados.rados.0.smithi165.stdout:488: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-10T16:09:53.200 INFO:tasks.rados.rados.0.smithi165.stdout:488: writing smithi16534841-41 from 645435 to 1096658 tid 1 2024-09-10T16:09:53.211 INFO:tasks.rados.rados.0.smithi165.stdout:488: writing smithi16534841-41 from 1879985 to 2667057 tid 2 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:488: writing smithi16534841-41 from 3280184 to 3445435 tid 3 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:488: finishing write tid 1 to smithi16534841-41 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:482: finishing write tid 3 to smithi16534841-17 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:482: finishing write tid 4 to smithi16534841-17 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:489: read oid 2 snap -1 2024-09-10T16:09:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:489: expect deleted 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:485: done (7 left) 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:490: copy_from oid 26 from oid 30 current snap is 61 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:491: read oid 46 snap -1 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:491: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:492: delete oid 6 current snap is 61 2024-09-10T16:09:53.217 INFO:tasks.rados.rados.0.smithi165.stdout:484: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:09:53.232 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1949946 2024-09-10T16:09:53.256 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:53.313 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1409170 2024-09-10T16:09:53.344 INFO:tasks.rados.rados.0.smithi165.stdout:482: finishing write tid 5 to smithi16534841-17 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: oid 17 updating version 0 to 245 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: oid 17 updating version 245 to 246 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: oid 17 version 246 is already newer than 244 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: oid 17 updating version 246 to 247 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 247 (ObjNum 204 snap 61 seq_num 204) dirty exists 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: left oid 17 (ObjNum 204 snap 61 seq_num 204) 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:482: done (9 left) 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:484: done (8 left) 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:492: done (7 left) 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:493: read oid 40 snap -1 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:493: expect deleted 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:494: rollback oid 1 current snap is 61 2024-09-10T16:09:53.345 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 31 2024-09-10T16:09:53.346 INFO:tasks.rados.rados.0.smithi165.stdout:487: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:09:53.346 INFO:tasks.rados.rados.0.smithi165.stdout:488: finishing write tid 2 to smithi16534841-41 2024-09-10T16:09:53.346 INFO:tasks.rados.rados.0.smithi165.stdout:491: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:09:53.346 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:53.393 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:09:53.490 INFO:tasks.rados.rados.0.smithi165.stdout:483: expect (ObjNum 118 snap 34 seq_num 118) 2024-09-10T16:09:53.605 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 234842 2024-09-10T16:09:53.665 INFO:tasks.rados.rados.0.smithi165.stdout:486: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:09:53.665 INFO:tasks.rados.rados.0.smithi165.stdout:490: finishing copy_from to smithi16534841-26 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:490: got expected ENOENT (src dne) 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:483: done (8 left) 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:491: done (7 left) 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:495: read oid 13 snap 42 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:495: expect (ObjNum 143 snap 42 seq_num 143) 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:496: read oid 46 snap 27 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:496: expect deleted 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:497: snap_create 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:490: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:488: finishing write tid 3 to smithi16534841-41 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:488: finishing write tid 4 to smithi16534841-41 2024-09-10T16:09:53.666 INFO:tasks.rados.rados.0.smithi165.stdout:488: finishing write tid 5 to smithi16534841-41 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:488: oid 41 updating version 0 to 295 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:488: oid 41 updating version 295 to 298 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:488: oid 41 version 298 is already newer than 296 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:488: oid 41 version 298 is already newer than 297 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 298 (ObjNum 205 snap 61 seq_num 205) dirty exists 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:488: left oid 41 (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:487: finishing copy_from to smithi16534841-42 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 66 (ObjNum 171 snap 52 seq_num 171) dirty exists 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:486: finishing copy_from to smithi16534841-4 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 263 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:494: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:494: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:09:53.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 300 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:09:53.669 INFO:tasks.rados.rados.0.smithi165.stdout:495: expect (ObjNum 143 snap 42 seq_num 143) 2024-09-10T16:09:53.735 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2164284 2024-09-10T16:09:53.753 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1424998 2024-09-10T16:09:53.940 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:09:53.940 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:54.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.039+0000 7fafde0a3640 10 monclient: get_monmap_and_config 2024-09-10T16:09:54.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.039+0000 7fafde0a3640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.039+0000 7fafde0a3640 10 monclient: monmap: 2024-09-10T16:09:54.041 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.041 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:54.040579+0000 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:54.040579+0000 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:54.042 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 Processor -- start 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 -- start start 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient: init 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fafd809f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fafd80a2550 0x7fafd813b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-c con 0x7fafd80a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafd80a2980 0x7fafd813db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-a con 0x7fafd80a2980 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd809fd10 con 0x7fafd809f930 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fafd809cd20 con 0x7fafd80a2980 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fafd809a0d0 con 0x7fafd80a2550 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.040+0000 7fafde0a3640 10 monclient(hunting): authenticate will time out at 972.527995s 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafd80a2980 0x7fafd813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fafd80a2980 0x7fafd813db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38842/0 (socket says 172.21.15.164:38842) 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 -- 172.21.15.164:0/1146598968 learned_addr learned my addr 172.21.15.164:0/1146598968 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): get_auth_request con 0x7fafd80a2980 auth_method 0 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 1 --2- 172.21.15.164:0/1146598968 >> v2:172.21.15.165:3300/0 conn(0x7fafd80a2550 0x7fafd813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): get_auth_request con 0x7fafd80a2550 auth_method 0 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient(hunting): handle_auth_done global_id 24342 payload 274 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 -- 172.21.15.164:0/1146598968 >> v2:172.21.15.165:3300/0 conn(0x7fafd80a2550 msgr2=0x7fafd813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1146598968 >> v2:172.21.15.165:3300/0 conn(0x7fafd80a2550 0x7fafd813b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 -- 172.21.15.164:0/1146598968 >> v1:172.21.15.165:6789/0 conn(0x7fafd809f930 legacy=0x7fafd80a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.041+0000 7fafd7fff640 1 -- 172.21.15.164:0/1146598968 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd809e840 con 0x7fafd80a2980 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd7fff640 1 -- 172.21.15.164:0/1146598968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd80a2980 msgr2=0x7fafd813db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1146598968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd80a2980 0x7fafd813db20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafd809f500 tx=0x7fafd002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=884e98f61f007078 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 1 -- 172.21.15.164:0/1146598968 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0037070 con 0x7fafd80a2980 2024-09-10T16:09:54.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: dump: 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 1 -- 172.21.15.164:0/1146598968 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd0032640 con 0x7fafd80a2980 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 1 -- 172.21.15.164:0/1146598968 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0032a10 con 0x7fafd80a2980 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafd67fc640 10 monclient: dump: 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.055 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 5 monclient: authenticate success, global_id 24342 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 4 set_mon_vals no callback set 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 10 monclient: shutdown 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 -- 172.21.15.164:0/1146598968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd80a2980 msgr2=0x7fafd813db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1146598968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd80a2980 0x7fafd813db20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fafd809f500 tx=0x7fafd002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 -- 172.21.15.164:0/1146598968 shutdown_connections 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1146598968 >> v2:172.21.15.165:3300/0 conn(0x7fafd80a2550 0x7fafd813b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1146598968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd80a2980 0x7fafd813db20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.044+0000 7fafde0a3640 1 -- 172.21.15.164:0/1146598968 >> 172.21.15.164:0/1146598968 conn(0x7fafd80a3110 msgr2=0x7fafd80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.045+0000 7fafde0a3640 1 -- 172.21.15.164:0/1146598968 shutdown_connections 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.045+0000 7fafde0a3640 1 -- 172.21.15.164:0/1146598968 wait complete. 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.045+0000 7fafde0a3640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.045+0000 7fafde0a3640 10 monclient: monmap: 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.045+0000 7fafde0a3640 1 Processor -- start 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 -- start start 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient: init 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-a con 0x7fafd809f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafd8091910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-c con 0x7fafd80a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafd8091e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): picked mon.noname-b con 0x7fafd80a2980 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fafd809d930 con 0x7fafd809f930 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fafd809eb80 con 0x7fafd80a2980 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fafd809c8b0 con 0x7fafd80a2550 2024-09-10T16:09:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafde0a3640 10 monclient(hunting): authenticate will time out at 972.533609s 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38844/0 (socket says 172.21.15.164:38844) 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 learned_addr learned my addr 172.21.15.164:0/1811799084 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): get_auth_request con 0x7fafd809f930 auth_method 0 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafd8091910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): get_auth_request con 0x7fafd80a2550 auth_method 0 2024-09-10T16:09:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafd8091e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient(hunting): handle_auth_done global_id 24345 payload 824 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 msgr2=0x7fafd8091910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafd8091910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 msgr2=0x7fafd8091e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafd8091e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.046+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd8094d80 con 0x7fafd809f930 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.047+0000 7fafd7fff640 10 monclient: get_auth_request con 0x7fafd80a2980 auth_method 0 2024-09-10T16:09:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.047+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafd8091e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:09:54.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 msgr2=0x7fafd80913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c7c0 tx=0x7fafc800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8016020 con 0x7fafd809f930 2024-09-10T16:09:54.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: dump: 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 5 monclient: authenticate success, global_id 24345 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc800ac00 con 0x7fafd809f930 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 10 monclient: _renew_subs 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd8095060 con 0x7fafd809f930 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8012420 con 0x7fafd809f930 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafbffff640 10 monclient: dump: 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.072 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 10 monclient: _renew_subs 2024-09-10T16:09:54.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd806ebe0 con 0x7fafd809f930 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.049+0000 7fafd5ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fafc8019020 con 0x7fafd809f930 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafde0a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafde0a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafde0a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd8074e90 con 0x7fafd809f930 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 msgr2=0x7fafd80913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 msgr2=0x7fafd80913d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:09:54.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c7c0 tx=0x7fafc800cc80 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c7c0 tx=0x7fafc800cc80 comp rx=0 tx=0).stop 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 0x7fafc4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 msgr2=0x7fafd80913d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafd809f930 0x7fafd80913d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafc4042e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): picked mon.b con 0x7fafd80a2980 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 0x7fafc4045b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): picked mon.a con 0x7fafc40436b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafc40485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): picked mon.c con 0x7fafd80a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fafc4048f50 con 0x7fafc40436b0 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fafc40490d0 con 0x7fafd80a2980 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fafc4049250 con 0x7fafd80a2550 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafbffff640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 0x7fafc4045b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafc40485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): get_auth_request con 0x7fafd80a2550 auth_method 0 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): get_auth_request con 0x7fafc40436b0 auth_method 0 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd6ffd640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 0x7fafc4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafc4042e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): get_auth_request con 0x7fafd80a2980 auth_method 0 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd6ffd640 10 monclient(hunting): get_auth_request con 0x7fafc403e240 auth_method 0 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:09:54.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 10 monclient(hunting): handle_auth_done global_id 24345 payload 293 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 msgr2=0x7fafc4042e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.050+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafc4042e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 msgr2=0x7fafc4045b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 0x7fafc4045b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 1 monclient: found mon.c 2024-09-10T16:09:54.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7fafc4049400 con 0x7fafd80a2550 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd806ebe0 con 0x7fafd80a2550 2024-09-10T16:09:54.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.051+0000 7fafd7fff640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 0x7fafc4045b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafd6ffd640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 0x7fafc4040700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009b40 tx=0x7fafcc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafd77fe640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 msgr2=0x7fafc40485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafd77fe640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafc40485d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fafc8051430 tx=0x7fafc8003e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8051460 con 0x7fafd80a2550 2024-09-10T16:09:54.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 10 monclient: dump: 2024-09-10T16:09:54.082 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.084 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.084 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.084 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.084 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc8058080 con 0x7fafd80a2550 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafbffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.053+0000 7fafd57fa640 4 set_mon_vals no callback set 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.054+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fafc8021cb0 con 0x7fafd80a2550 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.054+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 4 ==== osd_map(130..130 src has 1..130) ==== 4799+0+0 (secure 0 0 0) 0x7fafc8052af0 con 0x7fafd80a2550 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.058+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fafd8074e90 con 0x7fafd80a2550 2024-09-10T16:09:54.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.058+0000 7fafbffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.058+0000 7fafbffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:54.120 INFO:tasks.rados.rados.0.smithi165.stdout:486: done (9 left) 2024-09-10T16:09:54.120 INFO:tasks.rados.rados.0.smithi165.stdout:487: done (8 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:488: done (7 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:489: done (6 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:490: done (5 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:493: done (4 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:494: done (3 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:495: done (2 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:496: done (1 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:497: done (0 left) 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:498: write oid 44 current snap is 62 2024-09-10T16:09:54.121 INFO:tasks.rados.rados.0.smithi165.stdout:498: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-10T16:09:54.127 INFO:tasks.rados.rados.0.smithi165.stdout:498: writing smithi16534841-44 from 679049 to 1397123 tid 1 2024-09-10T16:09:54.135 INFO:tasks.rados.rados.0.smithi165.stdout:498: writing smithi16534841-44 from 2113302 to 2787690 tid 2 2024-09-10T16:09:54.139 INFO:tasks.rados.rados.0.smithi165.stdout:498: writing smithi16534841-44 from 3335030 to 3479049 tid 3 2024-09-10T16:09:54.139 INFO:tasks.rados.rados.0.smithi165.stdout:499: snap_remove snap 53 2024-09-10T16:09:54.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.304+0000 7fafde0a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:54.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.304+0000 7fafde0a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:54.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.304+0000 7fafde0a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.304+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fafd80a0760 con 0x7fafd80a2550 2024-09-10T16:09:54.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.305+0000 7fafbffff640 1 -- 172.21.15.164:0/1811799084 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fafc8020070 con 0x7fafd80a2550 2024-09-10T16:09:54.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.305+0000 7fafbffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:54.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.305+0000 7fafbffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:54.307 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:54.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 msgr2=0x7fafc4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 0x7fafc4040700 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009b40 tx=0x7fafcc009190 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 10 monclient: shutdown 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 msgr2=0x7fafc40485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafc40485d0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fafc8051430 tx=0x7fafc8003e60 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 shutdown_connections 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fafd80a2550 0x7fafc40485d0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fafc40436b0 0x7fafc4045b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fafd80a2980 0x7fafc4042e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 --2- 172.21.15.164:0/1811799084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fafc403e240 0x7fafc4040700 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 >> 172.21.15.164:0/1811799084 conn(0x7fafd80a3110 msgr2=0x7fafd81be6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 shutdown_connections 2024-09-10T16:09:54.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.307+0000 7fafde0a3640 1 -- 172.21.15.164:0/1811799084 wait complete. 2024-09-10T16:09:54.319 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:09:54.319 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:09:54.319 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:09:54.320 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:09:54.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.417+0000 7f428a0e2640 10 monclient: get_monmap_and_config 2024-09-10T16:09:54.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.417+0000 7f428a0e2640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.417+0000 7f428a0e2640 10 monclient: monmap: 2024-09-10T16:09:54.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:54.418697+0000 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:54.418697+0000 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:54.420 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.418+0000 7f428a0e2640 1 Processor -- start 2024-09-10T16:09:54.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.418+0000 7f428a0e2640 1 -- start start 2024-09-10T16:09:54.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.418+0000 7f428a0e2640 10 monclient: init 2024-09-10T16:09:54.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f428409ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4284134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4284134ed0 0x7f428413c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4284134ed0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42840a0870 con 0x7f428409ab50 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f428409d3e0 con 0x7f4284134ed0 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42840a2550 con 0x7f4284134580 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f428a0e2640 10 monclient(hunting): authenticate will time out at 972.906125s 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4284134ed0 0x7f428413c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4284134ed0 0x7f428413c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34900/0 (socket says 172.21.15.164:34900) 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44720/0 (socket says 172.21.15.164:44720) 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 -- 172.21.15.164:0/1578904119 learned_addr learned my addr 172.21.15.164:0/1578904119 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f4284134ed0 auth_method 0 2024-09-10T16:09:54.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): get_auth_request con 0x7f4284134580 auth_method 0 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient(hunting): handle_auth_done global_id 24348 payload 274 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 -- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 msgr2=0x7f4284134990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 -- 172.21.15.164:0/1578904119 >> v1:172.21.15.165:6789/0 conn(0x7f428409ab50 legacy=0x7f42840a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4283fff640 1 -- 172.21.15.164:0/1578904119 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4284074c70 con 0x7f4284134ed0 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.419+0000 7f4282ffd640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:54.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f4283fff640 1 -- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134ed0 msgr2=0x7f428413c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f4283fff640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134ed0 0x7f428413c8e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e80 tx=0x7f427c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec34c109460b2c06 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 1 -- 172.21.15.164:0/1578904119 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c036070 con 0x7f4284134ed0 2024-09-10T16:09:54.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: dump: 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.438 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f428a0e2640 5 monclient: authenticate success, global_id 24348 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 1 -- 172.21.15.164:0/1578904119 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427c007800 con 0x7f4284134ed0 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f428a0e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 1 -- 172.21.15.164:0/1578904119 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f427c007bd0 con 0x7f4284134ed0 2024-09-10T16:09:54.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f428a0e2640 4 set_mon_vals no callback set 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.422+0000 7f42827fc640 10 monclient: dump: 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.441 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.442 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 10 monclient: shutdown 2024-09-10T16:09:54.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 -- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134ed0 msgr2=0x7f428413c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134ed0 0x7f428413c8e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f427c009e80 tx=0x7f427c02a570 comp rx=0 tx=0).stop 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 -- 172.21.15.164:0/1578904119 shutdown_connections 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4284134580 0x7f4284134990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 --2- 172.21.15.164:0/1578904119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134ed0 0x7f428413c8e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 -- 172.21.15.164:0/1578904119 >> 172.21.15.164:0/1578904119 conn(0x7f42840a3110 msgr2=0x7f42840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 -- 172.21.15.164:0/1578904119 shutdown_connections 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 1 -- 172.21.15.164:0/1578904119 wait complete. 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.423+0000 7f428a0e2640 10 monclient: monmap: 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 1 Processor -- start 2024-09-10T16:09:54.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 1 -- start start 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 10 monclient: init 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f428409ab50 0x7f4284133c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-c con 0x7f428409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-a con 0x7f4284134580 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.424+0000 7f428a0e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4284134ed0 0x7f42841c91c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): picked mon.noname-b con 0x7f4284134ed0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42840a02b0 con 0x7f4284134580 2024-09-10T16:09:54.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f428409ffd0 con 0x7f4284134ed0 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f428409ef30 con 0x7f428409ab50 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f428a0e2640 10 monclient(hunting): authenticate will time out at 972.912028s 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f428409ab50 0x7f4284133c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34914/0 (socket says 172.21.15.164:34914) 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 -- 172.21.15.164:0/3713409185 learned_addr learned my addr 172.21.15.164:0/3713409185 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): get_auth_request con 0x7f4284134580 auth_method 0 2024-09-10T16:09:54.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): get_auth_request con 0x7f428409ab50 auth_method 0 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4284134ed0 0x7f42841c91c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): get_auth_request con 0x7f4284134ed0 auth_method 0 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4283fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient(hunting): handle_auth_done global_id 24351 payload 824 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f428409ab50 msgr2=0x7f4284133c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f428409ab50 0x7f4284133c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4284134ed0 msgr2=0x7f42841c91c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4284134ed0 0x7f42841c91c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f4282ffd640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42841c9700 con 0x7f4284134580 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.425+0000 7f42837fe640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f428409ab50 0x7f4284133c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f4282ffd640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 msgr2=0x7f42841c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f4282ffd640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f427800ebe0 tx=0x7f427800c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278017070 con 0x7f4284134580 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: dump: 2024-09-10T16:09:54.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 5 monclient: authenticate success, global_id 24351 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f427800ce40 con 0x7f4284134580 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278013690 con 0x7f4284134580 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f426bfff640 10 monclient: dump: 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 10 monclient: _renew_subs 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f4281ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f428412ba20 con 0x7f4284134580 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 10 monclient: _renew_subs 2024-09-10T16:09:54.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.427+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f428412d100 con 0x7f4284134580 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f428a0e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f428a0e2640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 0x7f42841c5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42840a29d0 con 0x7f4284134160 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f4282ffd640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f427800ebe0 tx=0x7f427800c520 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f4278052d50 osd_map(131..131 src has 1..131) 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f427801a070 con 0x7f4284134580 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 0x7f42841c5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient: get_auth_request con 0x7f4284134160 auth_method 0 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 10 monclient(hunting): handle_auth_done global_id 24354 payload 824 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f42837fe640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f428406dac0 con 0x7f4284134160 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.428+0000 7f426bfff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f427003e210 0x7f42700406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.429+0000 7f4283fff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f427003e210 0x7f42700406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.429+0000 7f4283fff640 10 monclient: get_auth_request con 0x7f427003e210 auth_method 0 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f42837fe640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 msgr2=0x7f42841c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f42837fe640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 0x7f42841c5f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f427400fae0 tx=0x7f42740044e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274018020 con 0x7f4284134160 2024-09-10T16:09:54.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 10 monclient: dump: 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.456 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f427400dd60 con 0x7f4284134160 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 msgr2=0x7f42841c5f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 0x7f42841c5f40 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f427400fae0 tx=0x7f42740044e0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 msgr2=0x7f42841c5f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f426bfff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134160 0x7f42841c5f40 secure :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f427400fae0 tx=0x7f42740044e0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.431+0000 7f4283fff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f427003e210 0x7f42700406d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f428406d500 tx=0x7f427c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f428a0e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:54.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f428a0e2640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 0x7f428402d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f428a0e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f428406fec0 con 0x7f428402d440 2024-09-10T16:09:54.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 0x7f428402d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient: get_auth_request con 0x7f428402d440 auth_method 0 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.437+0000 7f42837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.438+0000 7f42837fe640 10 monclient(hunting): handle_auth_done global_id 24357 payload 824 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.438+0000 7f42837fe640 1 -- 172.21.15.164:0/3713409185 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4284077260 con 0x7f428402d440 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f42837fe640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 msgr2=0x7f428402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f42837fe640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 0x7f428402d850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4274019000 tx=0x7f4274003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4274018030 con 0x7f428402d440 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 10 monclient: dump: 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.461 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7f42740067d0 con 0x7f428402d440 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 msgr2=0x7f428402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 0x7f428402d850 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4274019000 tx=0x7f4274003760 comp rx=0 tx=0).stop 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 msgr2=0x7f428402d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f426bfff640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f428402d440 0x7f428402d850 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f4274019000 tx=0x7f4274003760 comp rx=0 tx=0).stop 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 119372, 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.463 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.464 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.465 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.466 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.467 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:54.468 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.471 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.473 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.474 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:54.475 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f427003e210 msgr2=0x7f42700406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f427003e210 0x7f42700406d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f428406d500 tx=0x7f427c009000 comp rx=0 tx=0).stop 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 10 monclient: shutdown 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 msgr2=0x7f42841c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f428a0e2640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f427800ebe0 tx=0x7f427800c520 comp rx=0 tx=0).stop 2024-09-10T16:09:54.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.440+0000 7f4283fff640 1 -- 172.21.15.164:0/3713409185 reap_dead start 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f426bfff640 1 -- 172.21.15.164:0/3713409185 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (unknown 0 0 0) 0x7f4278052d50 con 0x7f4284134580 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 shutdown_connections 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f428a0e2640 1 --2- 172.21.15.164:0/3713409185 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4284134580 0x7f42841c8c80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 >> 172.21.15.164:0/3713409185 conn(0x7f42840a3110 msgr2=0x7f4284099460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 shutdown_connections 2024-09-10T16:09:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.441+0000 7f428a0e2640 1 -- 172.21.15.164:0/3713409185 wait complete. 2024-09-10T16:09:54.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:54.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:09:54.479 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:09:54.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.576+0000 7f266eae2640 10 monclient: get_monmap_and_config 2024-09-10T16:09:54.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.576+0000 7f266eae2640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.576+0000 7f266eae2640 10 monclient: monmap: 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:54.577836+0000 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:54.577836+0000 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:54.579 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:54.580 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:54.580 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:54.580 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.577+0000 7f266eae2640 1 Processor -- start 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.577+0000 7f266eae2640 1 -- start start 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.577+0000 7f266eae2640 10 monclient: init 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f266809c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:54.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f266809d190 0x7f266809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-c con 0x7f266809d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f266813a5c0 0x7f266813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-a con 0x7f266813a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f266809fbc0 con 0x7f266809c310 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f26680a2a20 con 0x7f266813a5c0 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f266809c6f0 con 0x7f266809d190 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266eae2640 10 monclient(hunting): authenticate will time out at 973.065225s 2024-09-10T16:09:54.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f266813a5c0 0x7f266813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266c87e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f266809c310 0x7f266809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:37238/0 (socket says 172.21.15.164:37238) 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266c87e640 1 -- 172.21.15.164:0/3929099696 learned_addr learned my addr 172.21.15.164:0/3929099696 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): get_auth_request con 0x7f266813a5c0 auth_method 0 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 1 --2- 172.21.15.164:0/3929099696 >> v2:172.21.15.165:3300/0 conn(0x7f266809d190 0x7f266809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): get_auth_request con 0x7f266809d190 auth_method 0 2024-09-10T16:09:54.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f2667fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient(hunting): handle_auth_done global_id 24360 payload 274 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 -- 172.21.15.164:0/3929099696 >> v2:172.21.15.165:3300/0 conn(0x7f266809d190 msgr2=0x7f266809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 --2- 172.21.15.164:0/3929099696 >> v2:172.21.15.165:3300/0 conn(0x7f266809d190 0x7f266809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 -- 172.21.15.164:0/3929099696 >> v1:172.21.15.164:6789/0 conn(0x7f266809c310 legacy=0x7f266809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.578+0000 7f266d07f640 1 -- 172.21.15.164:0/3929099696 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266809a4d0 con 0x7f266813a5c0 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266d07f640 1 -- 172.21.15.164:0/3929099696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f266813a5c0 msgr2=0x7f266813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266d07f640 1 --2- 172.21.15.164:0/3929099696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f266813a5c0 0x7f266813c9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26680a0280 tx=0x7f266002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e9e7b5c12f06fb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 1 -- 172.21.15.164:0/3929099696 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660037070 con 0x7f266813a5c0 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: dump: 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.596 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266eae2640 5 monclient: authenticate success, global_id 24360 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266eae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 1 -- 172.21.15.164:0/3929099696 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2660032440 con 0x7f266813a5c0 2024-09-10T16:09:54.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266eae2640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 1 -- 172.21.15.164:0/3929099696 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660032810 con 0x7f266813a5c0 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f266eae2640 4 set_mon_vals no callback set 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.581+0000 7f26677fe640 10 monclient: dump: 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 10 monclient: shutdown 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 -- 172.21.15.164:0/3929099696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f266813a5c0 msgr2=0x7f266813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 --2- 172.21.15.164:0/3929099696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f266813a5c0 0x7f266813c9b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f26680a0280 tx=0x7f266002a000 comp rx=0 tx=0).stop 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 -- 172.21.15.164:0/3929099696 shutdown_connections 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 --2- 172.21.15.164:0/3929099696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f266813a5c0 0x7f266813c9b0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 --2- 172.21.15.164:0/3929099696 >> v2:172.21.15.165:3300/0 conn(0x7f266809d190 0x7f266809f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 -- 172.21.15.164:0/3929099696 >> 172.21.15.164:0/3929099696 conn(0x7f26680a3110 msgr2=0x7f26680a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 -- 172.21.15.164:0/3929099696 shutdown_connections 2024-09-10T16:09:54.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 1 -- 172.21.15.164:0/3929099696 wait complete. 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.582+0000 7f266eae2640 10 monclient: monmap: 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:54.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 Processor -- start 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 -- start start 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient: init 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 0x7f266809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-c con 0x7f266809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f266813a5c0 0x7f2668099580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-b con 0x7f266813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): picked mon.noname-a con 0x7f2668099ac0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f26680a2230 con 0x7f2668099ac0 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f26680a20b0 con 0x7f266813a5c0 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2668074e00 con 0x7f266809c310 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266eae2640 10 monclient(hunting): authenticate will time out at 973.070810s 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 0x7f266809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34944/0 (socket says 172.21.15.164:34944) 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 0x7f266809d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44728/0 (socket says 172.21.15.164:44728) 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 learned_addr learned my addr 172.21.15.164:0/1476705381 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): get_auth_request con 0x7f2668099ac0 auth_method 0 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 10 monclient(hunting): get_auth_request con 0x7f266809c310 auth_method 0 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266c87e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.583+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f2667fff640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f266813a5c0 0x7f2668099580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f2667fff640 10 monclient(hunting): get_auth_request con 0x7f266813a5c0 auth_method 0 2024-09-10T16:09:54.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f2667fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f2667fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f2667fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 10 monclient(hunting): handle_auth_done global_id 24363 payload 824 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 msgr2=0x7f266809d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 0x7f266809d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f266813a5c0 msgr2=0x7f2668099580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f266813a5c0 0x7f2668099580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f266809d620 con 0x7f2668099ac0 2024-09-10T16:09:54.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.584+0000 7f266c87e640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809c310 0x7f266809d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 msgr2=0x7f2668092690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266d07f640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f266002a040 tx=0x7f26600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660041070 con 0x7f2668099ac0 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: dump: 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.606 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f266003c660 con 0x7f2668099ac0 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 5 monclient: authenticate success, global_id 24363 2024-09-10T16:09:54.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 10 monclient: _renew_subs 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2668094880 con 0x7f2668099ac0 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2660037040 con 0x7f2668099ac0 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f26657fa640 10 monclient: dump: 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.608 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 10 monclient: _renew_subs 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f2666ffd640 4 set_mon_vals no callback set 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.586+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2668093920 con 0x7f2668099ac0 2024-09-10T16:09:54.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266eae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266eae2640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 0x7f266802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2668099f20 con 0x7f266809d190 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2660044070 con 0x7f2668099ac0 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 0x7f266802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient: get_auth_request con 0x7f266809d190 auth_method 0 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 10 monclient(hunting): handle_auth_done global_id 4730 payload 824 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f266c87e640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f266809f720 con 0x7f266809d190 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f26657fa640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f264803e210 0x7f26480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.587+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (secure 0 0 0) 0x7f2660073410 con 0x7f2668099ac0 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.588+0000 7f2667fff640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f264803e210 0x7f26480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.588+0000 7f2667fff640 10 monclient: get_auth_request con 0x7f264803e210 auth_method 0 2024-09-10T16:09:54.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f266c87e640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 msgr2=0x7f266802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f266c87e640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 0x7f266802d440 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2658007270 tx=0x7f2658007730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658017020 con 0x7f266809d190 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 10 monclient: dump: 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f265800fd00 con 0x7f266809d190 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 msgr2=0x7f266802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 0x7f266802d440 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2658007270 tx=0x7f2658007730 comp rx=0 tx=0).stop 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 msgr2=0x7f266802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f26657fa640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266809d190 0x7f266802d440 secure :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2658007270 tx=0x7f2658007730 comp rx=0 tx=0).stop 2024-09-10T16:09:54.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.590+0000 7f2667fff640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f264803e210 0x7f26480406d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f266806e780 tx=0x7f265000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266eae2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:54.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266eae2640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 0x7f266802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266eae2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f266809e370 con 0x7f266802d980 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 0x7f266802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient: get_auth_request con 0x7f266802d980 auth_method 0 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.596+0000 7f266c87e640 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.597+0000 7f266c87e640 1 -- 172.21.15.164:0/1476705381 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2668073340 con 0x7f266802d980 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266c87e640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 msgr2=0x7f266802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266c87e640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 0x7f266802dd90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2658007270 tx=0x7f2658004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658013880 con 0x7f266802d980 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 10 monclient: dump: 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f2658013880 con 0x7f266802d980 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 msgr2=0x7f266802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 0x7f266802dd90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2658007270 tx=0x7f2658004940 comp rx=0 tx=0).stop 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:54.620 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 236771, 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:54.621 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.622 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.623 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:54.624 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.625 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:54.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.627 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:54.629 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:54.630 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:54.632 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:09:54.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.634 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:54.635 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 msgr2=0x7f266802dd90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f26657fa640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f266802d980 0x7f266802dd90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f264803e210 msgr2=0x7f26480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f264803e210 0x7f26480406d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f266806e780 tx=0x7f265000b040 comp rx=0 tx=0).stop 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 10 monclient: shutdown 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 msgr2=0x7f2668092690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266d07f640 1 -- 172.21.15.164:0/1476705381 reap_dead start 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.599+0000 7f266eae2640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f266002a040 tx=0x7f26600046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.600+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 shutdown_connections 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.600+0000 7f266eae2640 1 --2- 172.21.15.164:0/1476705381 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2668099ac0 0x7f2668092690 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.600+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 >> 172.21.15.164:0/1476705381 conn(0x7f26680a3110 msgr2=0x7f26681397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.600+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 shutdown_connections 2024-09-10T16:09:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.600+0000 7f266eae2640 1 -- 172.21.15.164:0/1476705381 wait complete. 2024-09-10T16:09:54.640 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:54.640 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:09:54.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:09:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.737+0000 7f4c273b4640 10 monclient: get_monmap_and_config 2024-09-10T16:09:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.737+0000 7f4c273b4640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.737+0000 7f4c273b4640 10 monclient: monmap: 2024-09-10T16:09:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:54.738746+0000 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:54.738746+0000 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:54.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.738+0000 7f4c273b4640 1 Processor -- start 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.738+0000 7f4c273b4640 1 -- start start 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.738+0000 7f4c273b4640 10 monclient: init 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4c2009c310 0x7f4c2009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2009c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4c2009d190 0x7f4c2009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2009d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4c2009f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2009fbc0 con 0x7f4c2009f4c0 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4c200a2a20 con 0x7f4c2009c310 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4c2009c6f0 con 0x7f4c2009d190 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c273b4640 10 monclient(hunting): authenticate will time out at 973.226136s 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4c2009c310 0x7f4c2009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4c2009c310 0x7f4c2009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34950/0 (socket says 172.21.15.164:34950) 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 -- 172.21.15.164:0/1623917078 learned_addr learned my addr 172.21.15.164:0/1623917078 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): get_auth_request con 0x7f4c2009c310 auth_method 0 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1623917078 >> v2:172.21.15.165:3300/0 conn(0x7f4c2009d190 0x7f4c2009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c2494f640 10 monclient(hunting): get_auth_request con 0x7f4c2009d190 auth_method 0 2024-09-10T16:09:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c2494f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c2494f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c2494f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient(hunting): handle_auth_done global_id 24366 payload 274 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 -- 172.21.15.164:0/1623917078 >> v2:172.21.15.165:3300/0 conn(0x7f4c2009d190 msgr2=0x7f4c2009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 --2- 172.21.15.164:0/1623917078 >> v2:172.21.15.165:3300/0 conn(0x7f4c2009d190 0x7f4c2009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 -- 172.21.15.164:0/1623917078 >> v1:172.21.15.165:6789/0 conn(0x7f4c2009f4c0 legacy=0x7f4c2013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 monclient: found mon.noname-a 2024-09-10T16:09:54.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:09:54.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.739+0000 7f4c25150640 1 -- 172.21.15.164:0/1623917078 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2009a4d0 con 0x7f4c2009c310 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c25150640 1 -- 172.21.15.164:0/1623917078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c2009c310 msgr2=0x7f4c2009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c25150640 1 --2- 172.21.15.164:0/1623917078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c2009c310 0x7f4c2009ea40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c200a0280 tx=0x7f4c1002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41c209d44d1f9d54 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 1 -- 172.21.15.164:0/1623917078 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10037070 con 0x7f4c2009c310 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: dump: 2024-09-10T16:09:54.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 1 -- 172.21.15.164:0/1623917078 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c10032640 con 0x7f4c2009c310 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 1 -- 172.21.15.164:0/1623917078 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10032a10 con 0x7f4c2009c310 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c273b4640 5 monclient: authenticate success, global_id 24366 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c273b4640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c273b4640 4 set_mon_vals no callback set 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.742+0000 7f4c17fff640 10 monclient: dump: 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.756 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 10 monclient: shutdown 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 -- 172.21.15.164:0/1623917078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c2009c310 msgr2=0x7f4c2009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1623917078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c2009c310 0x7f4c2009ea40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f4c200a0280 tx=0x7f4c1002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 -- 172.21.15.164:0/1623917078 shutdown_connections 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1623917078 >> v2:172.21.15.165:3300/0 conn(0x7f4c2009d190 0x7f4c2009ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1623917078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c2009c310 0x7f4c2009ea40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 -- 172.21.15.164:0/1623917078 >> 172.21.15.164:0/1623917078 conn(0x7f4c200a3110 msgr2=0x7f4c200a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 -- 172.21.15.164:0/1623917078 shutdown_connections 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 1 -- 172.21.15.164:0/1623917078 wait complete. 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.743+0000 7f4c273b4640 10 monclient: monmap: 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:54.758 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 Processor -- start 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 -- start start 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient: init 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:54.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 0x7f4c201c8bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-c con 0x7f4c2009c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009f4c0 0x7f4c20099340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2009f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 0x7f4c2008efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): picked mon.noname-a con 0x7f4c20099880 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4c200a2230 con 0x7f4c20099880 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4c200a20b0 con 0x7f4c2009f4c0 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4c20074e00 con 0x7f4c2009c310 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c273b4640 10 monclient(hunting): authenticate will time out at 973.231569s 2024-09-10T16:09:54.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 0x7f4c2008efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 0x7f4c201c8bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 0x7f4c201c8bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44754/0 (socket says 172.21.15.164:44754) 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 -- 172.21.15.164:0/1650697539 learned_addr learned my addr 172.21.15.164:0/1650697539 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 0x7f4c2008efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34960/0 (socket says 172.21.15.164:34960) 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): get_auth_request con 0x7f4c2009c310 auth_method 0 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): get_auth_request con 0x7f4c20099880 auth_method 0 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25951640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009f4c0 0x7f4c20099340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c2494f640 10 monclient(hunting): get_auth_request con 0x7f4c2009f4c0 auth_method 0 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c2494f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c2494f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c2494f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 10 monclient(hunting): handle_auth_done global_id 4736 payload 824 2024-09-10T16:09:54.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009f4c0 msgr2=0x7f4c20099340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009f4c0 0x7f4c20099340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 msgr2=0x7f4c2008efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.744+0000 7f4c25150640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 0x7f4c2008efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25150640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25150640 1 monclient: found mon.noname-c 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25150640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25150640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25150640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c20091390 con 0x7f4c2009c310 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.745+0000 7f4c25951640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4c20099880 0x7f4c2008efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c25150640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 msgr2=0x7f4c201c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c25150640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 0x7f4c201c8bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c100090b0 tx=0x7f4c100046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10041020 con 0x7f4c2009c310 2024-09-10T16:09:54.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: dump: 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.764 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c1003d910 con 0x7f4c2009c310 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 5 monclient: authenticate success, global_id 4736 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 10 monclient: _renew_subs 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c2009d620 con 0x7f4c2009c310 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c10037040 con 0x7f4c2009c310 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c15ffb640 10 monclient: dump: 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 10 monclient: _renew_subs 2024-09-10T16:09:54.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c177fe640 4 set_mon_vals no callback set 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.747+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c200928b0 con 0x7f4c2009c310 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4c10044020 con 0x7f4c2009c310 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c273b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 0x7f4c2008f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4c2009d930 con 0x7f4c2009d190 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c15ffb640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4c0403e210 0x7f4c040406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (secure 0 0 0) 0x7f4c10072fe0 con 0x7f4c2009c310 2024-09-10T16:09:54.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 0x7f4c2008f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 10 monclient: get_auth_request con 0x7f4c2009d190 auth_method 0 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c25951640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4c0403e210 0x7f4c040406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c25951640 10 monclient: get_auth_request con 0x7f4c0403e210 auth_method 0 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.748+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.749+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.749+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.749+0000 7f4c2494f640 10 monclient(hunting): handle_auth_done global_id 44131 payload 824 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.749+0000 7f4c2494f640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c200008d0 con 0x7f4c2009d190 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c25951640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4c0403e210 0x7f4c040406d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c180042a0 tx=0x7f4c18007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c2494f640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 msgr2=0x7f4c2008f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 0x7f4c2008f510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006e780 tx=0x7f4c1c007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c013780 con 0x7f4c2009d190 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 10 monclient: dump: 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.769 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4c1c019070 con 0x7f4c2009d190 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 msgr2=0x7f4c2008f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 0x7f4c2008f510 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006e780 tx=0x7f4c1c007700 comp rx=0 tx=0).stop 2024-09-10T16:09:54.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 msgr2=0x7f4c2008f510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.751+0000 7f4c15ffb640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c2009d190 0x7f4c2008f510 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.757+0000 7f4c273b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:09:54.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.757+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 0x7f4c2008fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.757+0000 7f4c273b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.757+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4c20092af0 con 0x7f4c200922f0 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 0x7f4c2008fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient: get_auth_request con 0x7f4c200922f0 auth_method 0 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 10 monclient(hunting): handle_auth_done global_id 44134 payload 824 2024-09-10T16:09:54.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.758+0000 7f4c2494f640 1 -- 172.21.15.164:0/1650697539 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c20092ca0 con 0x7f4c200922f0 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.760+0000 7f4c2494f640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 msgr2=0x7f4c2008fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.760+0000 7f4c2494f640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 0x7f4c2008fa50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006e780 tx=0x7f4c1c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1c014610 con 0x7f4c200922f0 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 10 monclient: dump: 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.774 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f4c1c013e20 con 0x7f4c200922f0 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 msgr2=0x7f4c2008fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 0x7f4c2008fa50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006e780 tx=0x7f4c1c004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 msgr2=0x7f4c2008fa50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:09:54.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c15ffb640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4c200922f0 0x7f4c2008fa50 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4c2006e780 tx=0x7f4c1c004a40 comp rx=0 tx=0).stop 2024-09-10T16:09:54.776 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:09:54.776 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:54.776 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:54.776 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 36, 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 9912, 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:09:54.777 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.779 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:09:54.780 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:09:54.781 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:09:54.782 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.783 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.785 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:09:54.788 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.789 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:09:54.790 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:09:54.791 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4c0403e210 msgr2=0x7f4c040406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4c0403e210 0x7f4c040406d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f4c180042a0 tx=0x7f4c18007db0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 10 monclient: shutdown 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 msgr2=0x7f4c201c8bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c273b4640 1 --2- 172.21.15.164:0/1650697539 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4c2009c310 0x7f4c201c8bf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f4c100090b0 tx=0x7f4c100046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.761+0000 7f4c25951640 1 -- 172.21.15.164:0/1650697539 reap_dead start 2024-09-10T16:09:54.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.762+0000 7f4c25951640 1 -- 172.21.15.164:0/1650697539 reap_dead start 2024-09-10T16:09:54.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.762+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 shutdown_connections 2024-09-10T16:09:54.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.762+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 >> 172.21.15.164:0/1650697539 conn(0x7f4c200a3110 msgr2=0x7f4c201bc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.762+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 shutdown_connections 2024-09-10T16:09:54.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.762+0000 7f4c273b4640 1 -- 172.21.15.164:0/1650697539 wait complete. 2024-09-10T16:09:54.794 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:09:54.794 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:09:54.795 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:09:54.795 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:09:54.795 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:09:54.795 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:09:54.795 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:09:54.805 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:09:54.806 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:09:54.806 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.940+0000 7fda9b94b640 10 monclient: get_monmap_and_config 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.940+0000 7fda9b94b640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.940+0000 7fda9b94b640 10 monclient: monmap: 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:54.942383+0000 2024-09-10T16:09:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:54.942383+0000 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:54.946 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 Processor -- start 2024-09-10T16:09:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 -- start start 2024-09-10T16:09:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient: init 2024-09-10T16:09:54.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-b con 0x7fda94073c80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda9409d170 addr v1:172.21.15.164:6789/0 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 0x7fda9413dd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-c con 0x7fda940a5f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda940745a0 con 0x7fda9409d170 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fda9409cfa0 con 0x7fda940a5f70 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fda940a0110 con 0x7fda94073c80 2024-09-10T16:09:54.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda9b94b640 10 monclient(hunting): authenticate will time out at 973.429896s 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda98ee6640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fda9409d170 legacy=0x7fda940a5860 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.941+0000 7fda996e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44762/0 (socket says 172.21.15.164:44762) 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 -- 172.21.15.164:0/3089877869 learned_addr learned my addr 172.21.15.164:0/3089877869 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): get_auth_request con 0x7fda94073c80 auth_method 0 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda99ee8640 1 --2- 172.21.15.164:0/3089877869 >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 0x7fda9413dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient(hunting): handle_auth_done global_id 4739 payload 274 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 -- 172.21.15.164:0/3089877869 >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 msgr2=0x7fda9413dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 --2- 172.21.15.164:0/3089877869 >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 0x7fda9413dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 -- 172.21.15.164:0/3089877869 >> v1:172.21.15.164:6789/0 conn(0x7fda9409d170 legacy=0x7fda940a5860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:54.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 monclient: found mon.noname-b 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda996e7640 1 -- 172.21.15.164:0/3089877869 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda940a03f0 con 0x7fda94073c80 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda99ee8640 10 monclient: get_auth_request con 0x7fda940a5f70 auth_method 0 2024-09-10T16:09:54.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.942+0000 7fda99ee8640 1 --2- 172.21.15.164:0/3089877869 >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 0x7fda9413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda996e7640 1 -- 172.21.15.164:0/3089877869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 msgr2=0x7fda94074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda996e7640 1 --2- 172.21.15.164:0/3089877869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009870 tx=0x7fda8c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a93067f22d319cb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 1 -- 172.21.15.164:0/3089877869 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c037070 con 0x7fda94073c80 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:54.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: dump: 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.969 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 5 monclient: authenticate success, global_id 4739 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 1 -- 172.21.15.164:0/3089877869 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8c032640 con 0x7fda94073c80 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 1 -- 172.21.15.164:0/3089877869 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c032a10 con 0x7fda94073c80 2024-09-10T16:09:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 4 set_mon_vals no callback set 2024-09-10T16:09:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda93fff640 10 monclient: dump: 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 10 monclient: shutdown 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:54.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 -- 172.21.15.164:0/3089877869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 msgr2=0x7fda94074060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3089877869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fda8c009870 tx=0x7fda8c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 -- 172.21.15.164:0/3089877869 shutdown_connections 2024-09-10T16:09:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3089877869 >> v2:172.21.15.165:3300/0 conn(0x7fda940a5f70 0x7fda9413dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3089877869 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda94074060 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 -- 172.21.15.164:0/3089877869 >> 172.21.15.164:0/3089877869 conn(0x7fda940a6f80 msgr2=0x7fda940a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 -- 172.21.15.164:0/3089877869 shutdown_connections 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.945+0000 7fda9b94b640 1 -- 172.21.15.164:0/3089877869 wait complete. 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.946+0000 7fda9b94b640 10 monclient: build_initial_monmap 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.946+0000 7fda9b94b640 10 monclient: monmap: 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.946+0000 7fda9b94b640 1 Processor -- start 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.946+0000 7fda9b94b640 1 -- start start 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.946+0000 7fda9b94b640 10 monclient: init 2024-09-10T16:09:54.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-c con 0x7fda94073c80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9409d170 0x7fda9409db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-b con 0x7fda9409d170 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:54.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 0x7fda9409e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): picked mon.noname-a con 0x7fda940a5f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fda940a3f50 con 0x7fda940a5f70 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fda940a3dd0 con 0x7fda9409d170 2024-09-10T16:09:54.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fda941329a0 con 0x7fda94073c80 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda9b94b640 10 monclient(hunting): authenticate will time out at 973.435339s 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda99ee8640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 msgr2=0x7fda9409e070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda99ee8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 0x7fda9409e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44766/0 (socket says 172.21.15.164:44766) 2024-09-10T16:09:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 -- 172.21.15.164:0/3183068356 learned_addr learned my addr 172.21.15.164:0/3183068356 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): get_auth_request con 0x7fda94073c80 auth_method 0 2024-09-10T16:09:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9409d170 0x7fda9409db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): get_auth_request con 0x7fda9409d170 auth_method 0 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda98ee6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient(hunting): handle_auth_done global_id 4742 payload 824 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 -- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9409d170 msgr2=0x7fda9409db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9409d170 0x7fda9409db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 -- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 msgr2=0x7fda9409e070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 0x7fda9409e070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient: _finish_hunting 0 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 monclient: found mon.noname-c 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.947+0000 7fda996e7640 1 -- 172.21.15.164:0/3183068356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda94074d40 con 0x7fda94073c80 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.949+0000 7fda996e7640 1 -- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 msgr2=0x7fda941bed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.949+0000 7fda996e7640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0090b0 tx=0x7fda8c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c041020 con 0x7fda94073c80 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: dump: 2024-09-10T16:09:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: _finish_auth 0 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:09:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda8c03d8f0 con 0x7fda94073c80 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 5 monclient: authenticate success, global_id 4742 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 10 monclient: _renew_subs 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda8c037040 con 0x7fda94073c80 2024-09-10T16:09:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda940a1c30 con 0x7fda94073c80 2024-09-10T16:09:54.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:54.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:54.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda91ffb640 10 monclient: dump: 2024-09-10T16:09:54.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda937fe640 4 set_mon_vals no callback set 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 10 monclient: _renew_subs 2024-09-10T16:09:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.950+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda941bd3a0 con 0x7fda94073c80 2024-09-10T16:09:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fda8c044020 con 0x7fda94073c80 2024-09-10T16:09:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda9b94b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda9b94b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda9b94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda940a63c0 con 0x7fda94073c80 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda91ffb640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 0x7fda80040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (secure 0 0 0) 0x7fda8c072f50 con 0x7fda94073c80 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda98ee6640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 0x7fda80040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.951+0000 7fda98ee6640 10 monclient: get_auth_request con 0x7fda8003e1c0 auth_method 0 2024-09-10T16:09:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.954+0000 7fda98ee6640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 0x7fda80040680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fda9409f1a0 tx=0x7fda88006f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.955+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fda8c04b070 con 0x7fda94073c80 2024-09-10T16:09:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.955+0000 7fda91ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:54.955+0000 7fda91ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:55.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.196+0000 7fda9b94b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:55.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.196+0000 7fda9b94b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:55.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.196+0000 7fda9b94b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:55.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.196+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda94072490 con 0x7fda94073c80 2024-09-10T16:09:55.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.197+0000 7fda91ffb640 1 -- 172.21.15.164:0/3183068356 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fda94074d40 con 0x7fda94073c80 2024-09-10T16:09:55.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.197+0000 7fda91ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:55.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.197+0000 7fda91ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:55.200 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 msgr2=0x7fda80040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 0x7fda80040680 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fda9409f1a0 tx=0x7fda88006f90 comp rx=0 tx=0).stop 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 10 monclient: shutdown 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 msgr2=0x7fda941bed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fda8c0090b0 tx=0x7fda8c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 shutdown_connections 2024-09-10T16:09:55.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fda8003e1c0 0x7fda80040680 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fda940a5f70 0x7fda9409e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fda9409d170 0x7fda9409db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 --2- 172.21.15.164:0/3183068356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fda94073c80 0x7fda941bed20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 >> 172.21.15.164:0/3183068356 conn(0x7fda940a6f80 msgr2=0x7fda94131540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 shutdown_connections 2024-09-10T16:09:55.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:55.199+0000 7fda9b94b640 1 -- 172.21.15.164:0/3183068356 wait complete. 2024-09-10T16:09:58.213 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.320+0000 7fea3a7bc640 10 monclient: get_monmap_and_config 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.320+0000 7fea3a7bc640 10 monclient: build_initial_monmap 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.320+0000 7fea3a7bc640 10 monclient: monmap: 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:09:58.322393+0000 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:09:58.322393+0000 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:09:58.323 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:09:58.324 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:09:58.324 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:09:58.324 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:09:58.324 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 Processor -- start 2024-09-10T16:09:58.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 -- start start 2024-09-10T16:09:58.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient: init 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 0x7fea3409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-a con 0x7fea3409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-b con 0x7fea3409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fea3409f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea3409a430 con 0x7fea3409f4c0 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fea340a24e0 con 0x7fea3409c180 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:58.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fea340a2bf0 con 0x7fea3409d160 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea3a7bc640 10 monclient(hunting): authenticate will time out at 976.809946s 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea33fff640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 msgr2=0x7fea3409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.321+0000 7fea33fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 0x7fea3409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44770/0 (socket says 172.21.15.164:44770) 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 -- 172.21.15.164:0/1958475094 learned_addr learned my addr 172.21.15.164:0/1958475094 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3409d160 auth_method 0 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:58.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient(hunting): handle_auth_done global_id 4745 payload 274 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 -- 172.21.15.164:0/1958475094 >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 msgr2=0x7fea3409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 --2- 172.21.15.164:0/1958475094 >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 0x7fea3409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 -- 172.21.15.164:0/1958475094 >> v1:172.21.15.165:6789/0 conn(0x7fea3409f4c0 legacy=0x7fea3413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 monclient: found mon.noname-b 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:09:58.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.322+0000 7fea337fe640 1 -- 172.21.15.164:0/1958475094 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea3409c560 con 0x7fea3409d160 2024-09-10T16:09:58.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea337fe640 1 -- 172.21.15.164:0/1958475094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 msgr2=0x7fea3409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea337fe640 1 --2- 172.21.15.164:0/1958475094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fea28009870 tx=0x7fea2802a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=644eab53406aae20 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 1 -- 172.21.15.164:0/1958475094 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28037070 con 0x7fea3409d160 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: dump: 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.335 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: _finish_auth 0 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 1 -- 172.21.15.164:0/1958475094 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea28032640 con 0x7fea3409d160 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 5 monclient: authenticate success, global_id 4745 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 10 monclient: get_monmap_and_config success 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 1 -- 172.21.15.164:0/1958475094 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28032a10 con 0x7fea3409d160 2024-09-10T16:09:58.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 4 set_mon_vals no callback set 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea32ffd640 10 monclient: dump: 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:58.337 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 10 monclient: shutdown 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 -- 172.21.15.164:0/1958475094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 msgr2=0x7fea3409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/1958475094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fea28009870 tx=0x7fea2802a150 comp rx=0 tx=0).stop 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 -- 172.21.15.164:0/1958475094 shutdown_connections 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/1958475094 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea3409ef80 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/1958475094 >> v2:172.21.15.164:3300/0 conn(0x7fea3409c180 0x7fea3409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.325+0000 7fea3a7bc640 1 -- 172.21.15.164:0/1958475094 >> 172.21.15.164:0/1958475094 conn(0x7fea340a3110 msgr2=0x7fea340a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.326+0000 7fea3a7bc640 1 -- 172.21.15.164:0/1958475094 shutdown_connections 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.326+0000 7fea3a7bc640 1 -- 172.21.15.164:0/1958475094 wait complete. 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.326+0000 7fea3a7bc640 10 monclient: build_initial_monmap 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.326+0000 7fea3a7bc640 10 monclient: monmap: 2024-09-10T16:09:58.338 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.326+0000 7fea3a7bc640 1 Processor -- start 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 -- start start 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient: init 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient: _reopen_session rank -1 2024-09-10T16:09:58.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 0x7fea34137a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-a con 0x7fea3409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-c con 0x7fea3409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea3409f4c0 0x7fea3408df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): picked mon.noname-b con 0x7fea3409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fea34074f90 con 0x7fea3409c180 2024-09-10T16:09:58.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fea34074e10 con 0x7fea3409f4c0 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fea3413cce0 con 0x7fea3409d160 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): _renew_subs 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea3a7bc640 10 monclient(hunting): authenticate will time out at 976.815628s 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea33fff640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 msgr2=0x7fea34137a00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea33fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 0x7fea34137a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44784/0 (socket says 172.21.15.164:44784) 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 -- 172.21.15.164:0/3939222624 learned_addr learned my addr 172.21.15.164:0/3939222624 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:09:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): get_auth_request con 0x7fea3409d160 auth_method 0 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea38d59640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea3409f4c0 0x7fea3408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea38d59640 10 monclient(hunting): get_auth_request con 0x7fea3409f4c0 auth_method 0 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea38d59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea38d59640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea38d59640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 10 monclient(hunting): handle_auth_done global_id 4748 payload 824 2024-09-10T16:09:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 -- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea3409f4c0 msgr2=0x7fea3408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea3409f4c0 0x7fea3408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.327+0000 7fea337fe640 1 -- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 msgr2=0x7fea34137a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 0x7fea34137a00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 10 monclient: _finish_hunting 0 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 1 monclient: found mon.noname-c 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.328+0000 7fea337fe640 1 -- 172.21.15.164:0/3939222624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea34090c20 con 0x7fea3409d160 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea337fe640 1 -- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 msgr2=0x7fea34137f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea337fe640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fea280090b0 tx=0x7fea28004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28041070 con 0x7fea3409d160 2024-09-10T16:09:58.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: dump: 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: _finish_auth 0 2024-09-10T16:09:58.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea2803c880 con 0x7fea3409d160 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 5 monclient: authenticate success, global_id 4748 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 10 monclient: _renew_subs 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea34073ea0 con 0x7fea3409d160 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea28037040 con 0x7fea3409d160 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea30ff9640 10 monclient: dump: 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:09:58.345 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 10 monclient: _renew_subs 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fea340918c0 con 0x7fea3409d160 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.330+0000 7fea31ffb640 4 set_mon_vals no callback set 2024-09-10T16:09:58.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea3a7bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea3a7bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea3a7bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fea28044070 con 0x7fea3409d160 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea3409ea40 con 0x7fea3409d160 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea30ff9640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 0x7fea20040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.331+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (secure 0 0 0) 0x7fea28073120 con 0x7fea3409d160 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.332+0000 7fea33fff640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 0x7fea20040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:09:58.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.332+0000 7fea33fff640 10 monclient: get_auth_request con 0x7fea2003e1c0 auth_method 0 2024-09-10T16:09:58.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.334+0000 7fea33fff640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 0x7fea20040680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea24003980 tx=0x7fea24006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:09:58.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:58.336+0000 7fea337fe640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fea280090b0 tx=0x7fea28004730 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7fea2804b070 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:09:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.165+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fea2804b070 con 0x7fea3409d160 2024-09-10T16:09:59.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.165+0000 7fea30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:09:59.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.165+0000 7fea30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:09:59.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.409+0000 7fea3a7bc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea3a7bc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea3a7bc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fea3409a0c0 con 0x7fea3409d160 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea30ff9640 1 -- 172.21.15.164:0/3939222624 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fea28030e00 con 0x7fea3409d160 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.410+0000 7fea30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:09:59.413 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":36,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 msgr2=0x7fea20040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 0x7fea20040680 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fea24003980 tx=0x7fea24006d30 comp rx=0 tx=0).stop 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 10 monclient: shutdown 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 msgr2=0x7fea34137f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:09:59.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fea280090b0 tx=0x7fea28004730 comp rx=0 tx=0).stop 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 shutdown_connections 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea2003e1c0 0x7fea20040680 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fea3409f4c0 0x7fea3408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fea3409d160 0x7fea34137f40 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 --2- 172.21.15.164:0/3939222624 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fea3409c180 0x7fea34137a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 >> 172.21.15.164:0/3939222624 conn(0x7fea340a3110 msgr2=0x7fea3413ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 shutdown_connections 2024-09-10T16:09:59.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:09:59.412+0000 7fea3a7bc640 1 -- 172.21.15.164:0/3939222624 wait complete. 2024-09-10T16:10:02.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:02.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 10 monclient: get_monmap_and_config 2024-09-10T16:10:02.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 10 monclient: build_initial_monmap 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 10 monclient: monmap: 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:02.537999+0000 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:02.537999+0000 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:02.539 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:02.540 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:02.540 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:02.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 1 Processor -- start 2024-09-10T16:10:02.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 1 -- start start 2024-09-10T16:10:02.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.536+0000 7f318c1c2640 10 monclient: init 2024-09-10T16:10:02.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f318409d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-a con 0x7f318409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f318409e420 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31840a0870 con 0x7f318409d930 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3184074c70 con 0x7f318409e420 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f318409d3e0 con 0x7f318409ab50 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318c1c2640 10 monclient(hunting): authenticate will time out at 981.025474s 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f3189f5e640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f318409d930 legacy=0x7f318409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:10:02.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318975d640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f31841397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:02.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318975d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:02.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318a75f640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f318409e420 0x7f318413bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:44136/0 (socket says 172.21.15.164:44136) 2024-09-10T16:10:02.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.537+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 learned_addr learned my addr 172.21.15.164:0/2519713601 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:02.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.541+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.165:6789/0 conn(0x7f318409e420 legacy=0x7f318413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:02.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.738+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f31841397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:02.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.738+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.164:6789/0 conn(0x7f318409d930 legacy=0x7f318409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:10:02.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:02.738+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:03.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.139+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f31841397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:03.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.139+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.164:6789/0 conn(0x7f318409d930 legacy=0x7f318409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:10:03.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.139+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:03.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.940+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f31841397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:03.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.940+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.164:6789/0 conn(0x7f318409d930 legacy=0x7f318409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:10:03.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:03.940+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:05.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): tick 2024-09-10T16:10:05.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:05.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f31841397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f31841397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.165:6789/0 conn(0x7f318409e420 legacy=0x7f318413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 >> v1:172.21.15.164:6789/0 conn(0x7f318409d930 legacy=0x7f318409dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient: _add_conns ranks=[4,2] 2024-09-10T16:10:05.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f3168004850 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f3168007360 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f318409d3e0 con 0x7f3168004850 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3168009f00 con 0x7f3168007360 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f317b7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:05.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.538+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:05.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:05.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:05.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:05.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:05.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:05.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:05.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f3189f5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.740+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.741+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.741+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.741+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:05.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:05.741+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:06.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:06.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:06.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:06.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:06.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:06.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:06.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:06.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:06.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.141+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:06.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.143+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.942+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.942+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.942+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.942+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.942+0000 7f318975d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.943+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.943+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.943+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.943+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:06.944+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 10 monclient(hunting): tick 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f3168009820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f3168009820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.538+0000 7f317b7fe640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-10T16:10:08.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f3168007360 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 reap_dead start 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f3168004850 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f318409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f318409d3e0 con 0x7f318409ab50 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3168009f00 con 0x7f3168004850 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:08.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 1 -- 172.21.15.164:0/2519713601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3168003db0 con 0x7f3168007360 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f317b7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f3168003870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318a75f640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:08.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:08.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.539+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:08.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f3168003870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:08.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f318a75f640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.740+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:08.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:08.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:08.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:08.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:08.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:08.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:08.741+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:09.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.141+0000 7f318a75f640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f3168003870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:09.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.141+0000 7f318a75f640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.142+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 10 monclient(hunting): handle_auth_done global_id 4751 payload 274 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 msgr2=0x7f3168006c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 msgr2=0x7f3168003870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 monclient: found mon.noname-b 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318409ecb0 con 0x7f3168007360 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f318975d640 10 monclient: get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:09.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.143+0000 7f318975d640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:10:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3189f5e640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3168009ea0 tx=0x7f318002b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3559043c78e6e902 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 1 -- 172.21.15.164:0/2519713601 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180038020 con 0x7f3168007360 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: dump: 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 1 -- 172.21.15.164:0/2519713601 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3180034400 con 0x7f3168007360 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 1 -- 172.21.15.164:0/2519713601 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f318000f040 con 0x7f3168007360 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f318c1c2640 5 monclient: authenticate success, global_id 4751 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f318c1c2640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f318c1c2640 4 set_mon_vals no callback set 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.146+0000 7f3188f5c640 10 monclient: dump: 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 10 monclient: shutdown 2024-09-10T16:10:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 -- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 msgr2=0x7f31680097a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3168009ea0 tx=0x7f318002b8e0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 -- 172.21.15.164:0/2519713601 shutdown_connections 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.165:3300/0 conn(0x7f3168004850 0x7f3168006c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2519713601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168007360 0x7f31680097a0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2519713601 >> v2:172.21.15.164:3300/0 conn(0x7f318409ab50 0x7f3168003870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 -- 172.21.15.164:0/2519713601 >> 172.21.15.164:0/2519713601 conn(0x7f31840a3110 msgr2=0x7f31840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 -- 172.21.15.164:0/2519713601 shutdown_connections 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.147+0000 7f318c1c2640 1 -- 172.21.15.164:0/2519713601 wait complete. 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.148+0000 7f318c1c2640 10 monclient: build_initial_monmap 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.148+0000 7f318c1c2640 10 monclient: monmap: 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:09.157 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:09.157 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:09.157 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:09.157 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.148+0000 7f318c1c2640 1 Processor -- start 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.148+0000 7f318c1c2640 1 -- start start 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.148+0000 7f318c1c2640 10 monclient: init 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-c con 0x7f3168004850 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3168007360 0x7f31841cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-b con 0x7f3168007360 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:09.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 0x7f3184131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): picked mon.noname-a con 0x7f318409ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f318409ef30 con 0x7f318409ab50 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f31840a0090 con 0x7f3168007360 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f31840a2940 con 0x7f3168004850 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318c1c2640 10 monclient(hunting): authenticate will time out at 987.636305s 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318a75f640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 msgr2=0x7f3184131f80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318a75f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 0x7f3184131f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44584/0 (socket says 172.21.15.164:44584) 2024-09-10T16:10:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 -- 172.21.15.164:0/2219585729 learned_addr learned my addr 172.21.15.164:0/2219585729 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): get_auth_request con 0x7f3168004850 auth_method 0 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3168007360 0x7f31841cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): get_auth_request con 0x7f3168007360 auth_method 0 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f318975d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient(hunting): handle_auth_done global_id 4754 payload 824 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 -- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3168007360 msgr2=0x7f31841cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3168007360 0x7f31841cdd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 -- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 msgr2=0x7f3184131f80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 0x7f3184131f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 monclient: found mon.noname-c 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.149+0000 7f3189f5e640 1 -- 172.21.15.164:0/2219585729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f318408e390 con 0x7f3168004850 2024-09-10T16:10:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f3189f5e640 1 -- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 msgr2=0x7f31841cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f3189f5e640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3180003980 tx=0x7f3180005a90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180042030 con 0x7f3168004850 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: dump: 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.164 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31800037b0 con 0x7f3168004850 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3180038040 con 0x7f3168004850 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 5 monclient: authenticate success, global_id 4754 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 10 monclient: _renew_subs 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3184133b80 con 0x7f3168004850 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317a7fc640 10 monclient: dump: 2024-09-10T16:10:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f317bfff640 4 set_mon_vals no callback set 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 10 monclient: _renew_subs 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.152+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f31840a5ec0 con 0x7f3168004850 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3180045070 con 0x7f3168004850 2024-09-10T16:10:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f317a7fc640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 0x7f3168041590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(131..131 src has 1..131) ==== 4771+0+0 (secure 0 0 0) 0x7f3180073a00 con 0x7f3168004850 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318c1c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318c1c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318c1c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f318409bfb0 con 0x7f3168004850 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318975d640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 0x7f3168041590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.153+0000 7f318975d640 10 monclient: get_auth_request con 0x7f316803f0d0 auth_method 0 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.156+0000 7f318975d640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 0x7f3168041590 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3174003890 tx=0x7f317400d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.158+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f318004f080 con 0x7f3168004850 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.158+0000 7f317a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.158+0000 7f317a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:09.295 INFO:tasks.rados.rados.0.smithi165.stdout:499: done (1 left) 2024-09-10T16:10:09.296 INFO:tasks.rados.rados.0.smithi165.stdout:500: read oid 47 snap -1 2024-09-10T16:10:09.296 INFO:tasks.rados.rados.0.smithi165.stdout:500: expect deleted 2024-09-10T16:10:09.296 INFO:tasks.rados.rados.0.smithi165.stdout:501: write (excl) oid 16 current snap is 62 2024-09-10T16:10:09.296 INFO:tasks.rados.rados.0.smithi165.stdout:498: finishing write tid 1 to smithi16534841-44 2024-09-10T16:10:09.296 INFO:tasks.rados.rados.0.smithi165.stdout:501: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2024-09-10T16:10:09.304 INFO:tasks.rados.rados.0.smithi165.stdout:501: writing smithi16534841-16 from 695856 to 1489179 tid 1 2024-09-10T16:10:09.311 INFO:tasks.rados.rados.0.smithi165.stdout:501: writing smithi16534841-16 from 2171784 to 2706183 tid 2 2024-09-10T16:10:09.315 INFO:tasks.rados.rados.0.smithi165.stdout:501: writing smithi16534841-16 from 3362453 to 3495856 tid 3 2024-09-10T16:10:09.315 INFO:tasks.rados.rados.0.smithi165.stdout:502: snap_remove snap 31 2024-09-10T16:10:09.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f318c1c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f318c1c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f318c1c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f318406dac0 con 0x7f3168004850 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f317a7fc640 1 -- 172.21.15.164:0/2219585729 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f318003ddb0 con 0x7f3168004850 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f317a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.412+0000 7f317a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:09.414 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 msgr2=0x7f3168041590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 0x7f3168041590 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3174003890 tx=0x7f317400d040 comp rx=0 tx=0).stop 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 10 monclient: shutdown 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 msgr2=0x7f31841cd850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3180003980 tx=0x7f3180005a90 comp rx=0 tx=0).stop 2024-09-10T16:10:09.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 shutdown_connections 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f316803f0d0 0x7f3168041590 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f318409ab50 0x7f3184131f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3168007360 0x7f31841cdd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 --2- 172.21.15.164:0/2219585729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3168004850 0x7f31841cd850 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 >> 172.21.15.164:0/2219585729 conn(0x7f31840a3110 msgr2=0x7f31840707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 shutdown_connections 2024-09-10T16:10:09.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.414+0000 7f318c1c2640 1 -- 172.21.15.164:0/2219585729 wait complete. 2024-09-10T16:10:09.427 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:10:09.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:10:09.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:10:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.525+0000 7fcdc6dbb640 10 monclient: get_monmap_and_config 2024-09-10T16:10:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.525+0000 7fcdc6dbb640 10 monclient: build_initial_monmap 2024-09-10T16:10:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.525+0000 7fcdc6dbb640 10 monclient: monmap: 2024-09-10T16:10:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:09.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:09.526784+0000 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:09.526784+0000 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:09.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.526+0000 7fcdc6dbb640 1 Processor -- start 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.526+0000 7fcdc6dbb640 1 -- start start 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.526+0000 7fcdc6dbb640 10 monclient: init 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdc009f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcdc00a2550 0x7fcdc013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc00a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:09.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc00a2980 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc009fd10 con 0x7fcdc009f930 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcdc009cd20 con 0x7fcdc00a2550 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc4b57640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fcdc009f930 legacy=0x7fcdc00a1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc009a0d0 con 0x7fcdc00a2980 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc6dbb640 10 monclient(hunting): authenticate will time out at 988.014187s 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44598/0 (socket says 172.21.15.164:44598) 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 -- 172.21.15.164:0/550778461 learned_addr learned my addr 172.21.15.164:0/550778461 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:09.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): get_auth_request con 0x7fcdc00a2980 auth_method 0 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdb7fff640 1 --2- 172.21.15.164:0/550778461 >> v2:172.21.15.165:3300/0 conn(0x7fcdc00a2550 0x7fcdc013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdb7fff640 10 monclient(hunting): get_auth_request con 0x7fcdc00a2550 auth_method 0 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 -- 172.21.15.164:0/550778461 >> v2:172.21.15.165:3300/0 conn(0x7fcdc00a2550 msgr2=0x7fcdc013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 --2- 172.21.15.164:0/550778461 >> v2:172.21.15.165:3300/0 conn(0x7fcdc00a2550 0x7fcdc013b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 -- 172.21.15.164:0/550778461 >> v1:172.21.15.164:6789/0 conn(0x7fcdc009f930 legacy=0x7fcdc00a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 monclient: found mon.noname-b 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.527+0000 7fcdc5358640 1 -- 172.21.15.164:0/550778461 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc009e840 con 0x7fcdc00a2980 2024-09-10T16:10:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc5358640 1 -- 172.21.15.164:0/550778461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 msgr2=0x7fcdc013db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc5358640 1 --2- 172.21.15.164:0/550778461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009e50 tx=0x7fcdb802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8eeaddc2ff614182 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 1 -- 172.21.15.164:0/550778461 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8036070 con 0x7fcdc00a2980 2024-09-10T16:10:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: dump: 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc6dbb640 5 monclient: authenticate success, global_id 4757 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 1 -- 172.21.15.164:0/550778461 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb8007800 con 0x7fcdc00a2980 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc6dbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc6dbb640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdc6dbb640 4 set_mon_vals no callback set 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 1 -- 172.21.15.164:0/550778461 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb8007bd0 con 0x7fcdc00a2980 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.530+0000 7fcdb77fe640 10 monclient: dump: 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 10 monclient: shutdown 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/550778461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 msgr2=0x7fcdc013db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/550778461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fcdb8009e50 tx=0x7fcdb802a560 comp rx=0 tx=0).stop 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/550778461 shutdown_connections 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/550778461 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2980 0x7fcdc013db20 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/550778461 >> v2:172.21.15.165:3300/0 conn(0x7fcdc00a2550 0x7fcdc013b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/550778461 >> 172.21.15.164:0/550778461 conn(0x7fcdc00a3110 msgr2=0x7fcdc00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/550778461 shutdown_connections 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/550778461 wait complete. 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 10 monclient: build_initial_monmap 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.531+0000 7fcdc6dbb640 10 monclient: monmap: 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:09.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 Processor -- start 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 -- start start 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient: init 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc009f930 0x7fcdc009a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc009f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 0x7fcdc009a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc00a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc00a2980 0x7fcdc009aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc00a2980 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:09.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc009d930 con 0x7fcdc00a2980 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc009ebe0 con 0x7fcdc009f930 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc009c8b0 con 0x7fcdc00a2550 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc6dbb640 10 monclient(hunting): authenticate will time out at 988.019842s 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc5358640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc00a2980 msgr2=0x7fcdc009aee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdc5358640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc00a2980 0x7fcdc009aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 0x7fcdc009a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 0x7fcdc009a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44612/0 (socket says 172.21.15.164:44612) 2024-09-10T16:10:09.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 1 -- 172.21.15.164:0/2036582469 learned_addr learned my addr 172.21.15.164:0/2036582469 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 10 monclient(hunting): get_auth_request con 0x7fcdc00a2550 auth_method 0 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.532+0000 7fcdb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc009f930 0x7fcdc009a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): get_auth_request con 0x7fcdc009f930 auth_method 0 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc009f930 msgr2=0x7fcdc009a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc009f930 0x7fcdc009a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc00a2980 msgr2=0x7fcdc009aee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc00a2980 0x7fcdc009aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 monclient: found mon.noname-c 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.533+0000 7fcdb7fff640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc0096050 con 0x7fcdc00a2550 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb7fff640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 msgr2=0x7fcdc009a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb7fff640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 0x7fcdc009a9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d830 tx=0x7fcdb000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000b8d0 con 0x7fcdc00a2550 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: dump: 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.550 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 5 monclient: authenticate success, global_id 4760 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdb0019070 con 0x7fcdc00a2550 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 10 monclient: _renew_subs 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb000bc90 con 0x7fcdc00a2550 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc0096330 con 0x7fcdc00a2550 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb57fa640 10 monclient: dump: 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.552 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdb6ffd640 4 set_mon_vals no callback set 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 10 monclient: _renew_subs 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.535+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc006ec90 con 0x7fcdc00a2550 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc6dbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 0x7fcdc00723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc00a0760 con 0x7fcdc009b420 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 0x7fcdc00723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcdb001e460 con 0x7fcdc00a2550 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient: get_auth_request con 0x7fcdc009b420 auth_method 0 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdb57fa640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcdac03e1f0 0x7fcdac0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_done global_id 4763 payload 824 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdc4b57640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcdc009b800 con 0x7fcdc009b420 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.536+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(132..132 src has 1..132) ==== 4815+0+0 (secure 0 0 0) 0x7fcdb0053190 con 0x7fcdc00a2550 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.537+0000 7fcdc5358640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcdac03e1f0 0x7fcdac0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.537+0000 7fcdc5358640 10 monclient: get_auth_request con 0x7fcdac03e1f0 auth_method 0 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdc4b57640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 msgr2=0x7fcdc00723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdc4b57640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 0x7fcdc00723c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009610 tx=0x7fcdbc009ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc016070 con 0x7fcdc009b420 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 10 monclient: dump: 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcdbc005a40 con 0x7fcdc009b420 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 msgr2=0x7fcdc00723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 0x7fcdc00723c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc009610 tx=0x7fcdbc009ad0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 msgr2=0x7fcdc00723c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:09.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdb57fa640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc009b420 0x7fcdc00723c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.539+0000 7fcdc5358640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcdac03e1f0 0x7fcdac0406b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcdc006d470 tx=0x7fcdb8009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc6dbb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 0x7fcdc0072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc6dbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc0095e30 con 0x7fcdc0072900 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 0x7fcdc0072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient: get_auth_request con 0x7fcdc0072900 auth_method 0 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.545+0000 7fcdc4b57640 10 monclient(hunting): handle_auth_done global_id 4766 payload 824 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.546+0000 7fcdc4b57640 1 -- 172.21.15.164:0/2036582469 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcdc0077260 con 0x7fcdc0072900 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.548+0000 7fcdc4b57640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 msgr2=0x7fcdc0072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.548+0000 7fcdc4b57640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 0x7fcdc0072d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc017000 tx=0x7fcdbc0059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc016050 con 0x7fcdc0072900 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 10 monclient: dump: 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fcdbc025600 con 0x7fcdc0072900 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 msgr2=0x7fcdc0072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 0x7fcdc0072d10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc017000 tx=0x7fcdbc0059d0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 msgr2=0x7fcdc0072d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:09.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.550+0000 7fcdb57fa640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc0072900 0x7fcdc0072d10 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc017000 tx=0x7fcdbc0059d0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 40, 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 251722, 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:10:09.561 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.562 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.563 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:09.564 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.565 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:09.566 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.568 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:10:09.569 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.570 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:10:09.571 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:10:09.572 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:09.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.574 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.574 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:10:09.574 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcdac03e1f0 msgr2=0x7fcdac0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcdac03e1f0 0x7fcdac0406b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcdc006d470 tx=0x7fcdb8009000 comp rx=0 tx=0).stop 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 10 monclient: shutdown 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 msgr2=0x7fcdc009a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 --2- 172.21.15.164:0/2036582469 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a2550 0x7fcdc009a9a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000d830 tx=0x7fcdb000dc30 comp rx=0 tx=0).stop 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc5358640 1 -- 172.21.15.164:0/2036582469 reap_dead start 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc5358640 1 -- 172.21.15.164:0/2036582469 reap_dead start 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 shutdown_connections 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 >> 172.21.15.164:0/2036582469 conn(0x7fcdc00a3110 msgr2=0x7fcdc01bd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 shutdown_connections 2024-09-10T16:10:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.551+0000 7fcdc6dbb640 1 -- 172.21.15.164:0/2036582469 wait complete. 2024-09-10T16:10:09.576 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:10:09.576 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:10:09.577 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.674+0000 7fac983a0640 10 monclient: get_monmap_and_config 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.674+0000 7fac983a0640 10 monclient: build_initial_monmap 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.674+0000 7fac983a0640 10 monclient: monmap: 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:09.675468+0000 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:09.675468+0000 2024-09-10T16:10:09.676 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:09.677 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 Processor -- start 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 -- start start 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient: init 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac9009ce10 0x7fac900a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-c con 0x7fac9009ce10 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 0x7fac9009d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-a con 0x7fac900a3450 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac9009d730 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac900a1c90 con 0x7fac9009d730 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fac90073ee0 con 0x7fac900a3450 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fac900a1f70 con 0x7fac9009ce10 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac983a0640 10 monclient(hunting): authenticate will time out at 988.162859s 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac9593b640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 msgr2=0x7fac9009d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:09.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.675+0000 7fac9593b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 0x7fac9009d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac9009ce10 0x7fac900a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9693d640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fac9009d730 0x7fac901336b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:37268/0 (socket says 172.21.15.164:37268) 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9693d640 1 -- 172.21.15.164:0/2796083794 learned_addr learned my addr 172.21.15.164:0/2796083794 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): get_auth_request con 0x7fac9009ce10 auth_method 0 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient(hunting): handle_auth_done global_id 44152 payload 274 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 -- 172.21.15.164:0/2796083794 >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 msgr2=0x7fac9009d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:09.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 --2- 172.21.15.164:0/2796083794 >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 0x7fac9009d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 -- 172.21.15.164:0/2796083794 >> v1:172.21.15.165:6789/0 conn(0x7fac9009d730 legacy=0x7fac901336b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 monclient: found mon.noname-c 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:10:09.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.676+0000 7fac9613c640 1 -- 172.21.15.164:0/2796083794 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac9009db90 con 0x7fac9009ce10 2024-09-10T16:10:09.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9613c640 1 -- 172.21.15.164:0/2796083794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009ce10 msgr2=0x7fac900a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9613c640 1 --2- 172.21.15.164:0/2796083794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009ce10 0x7fac900a2f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac90073e70 tx=0x7fac8c02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61b0f266c9323194 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 1 -- 172.21.15.164:0/2796083794 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c037070 con 0x7fac9009ce10 2024-09-10T16:10:09.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: dump: 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 1 -- 172.21.15.164:0/2796083794 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8c032640 con 0x7fac9009ce10 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 1 -- 172.21.15.164:0/2796083794 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c032a10 con 0x7fac9009ce10 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac9513a640 10 monclient: dump: 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac983a0640 5 monclient: authenticate success, global_id 44152 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac983a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.679+0000 7fac983a0640 4 set_mon_vals no callback set 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 10 monclient: shutdown 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 -- 172.21.15.164:0/2796083794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009ce10 msgr2=0x7fac900a2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 --2- 172.21.15.164:0/2796083794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009ce10 0x7fac900a2f10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac90073e70 tx=0x7fac8c02a000 comp rx=0 tx=0).stop 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 -- 172.21.15.164:0/2796083794 shutdown_connections 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 --2- 172.21.15.164:0/2796083794 >> v2:172.21.15.164:3300/0 conn(0x7fac900a3450 0x7fac9009d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 --2- 172.21.15.164:0/2796083794 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009ce10 0x7fac900a2f10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 -- 172.21.15.164:0/2796083794 >> 172.21.15.164:0/2796083794 conn(0x7fac900a7320 msgr2=0x7fac900a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 -- 172.21.15.164:0/2796083794 shutdown_connections 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.680+0000 7fac983a0640 1 -- 172.21.15.164:0/2796083794 wait complete. 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient: build_initial_monmap 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient: monmap: 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 Processor -- start 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 -- start start 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient: init 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac9009ce10 0x7fac901cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-a con 0x7fac9009ce10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009d730 0x7fac901cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-b con 0x7fac9009d730 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): picked mon.noname-c con 0x7fac90120730 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fac90133f40 con 0x7fac9009ce10 2024-09-10T16:10:09.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac90133dc0 con 0x7fac9009d730 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fac901340c0 con 0x7fac90120730 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac983a0640 10 monclient(hunting): authenticate will time out at 988.168894s 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac9613c640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac9009ce10 msgr2=0x7fac901cd280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac9613c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac9009ce10 0x7fac901cd280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.681+0000 7fac9693d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009d730 0x7fac901cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44640/0 (socket says 172.21.15.164:44640) 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 learned_addr learned my addr 172.21.15.164:0/3159391707 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): get_auth_request con 0x7fac90120730 auth_method 0 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): get_auth_request con 0x7fac9009d730 auth_method 0 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9593b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient(hunting): handle_auth_done global_id 4769 payload 824 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009d730 msgr2=0x7fac901cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac9009d730 0x7fac901cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac9009ce10 msgr2=0x7fac901cd280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac9009ce10 0x7fac901cd280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient: _finish_hunting 0 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 monclient: found mon.noname-c 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.682+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac90133500 con 0x7fac90120730 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 msgr2=0x7fac90120be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac9693d640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac8800cb20 tx=0x7fac8800cf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88017070 con 0x7fac90120730 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: dump: 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: _finish_auth 0 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:10:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 5 monclient: authenticate success, global_id 4769 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac8800ade0 con 0x7fac90120730 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac88012640 con 0x7fac90120730 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 10 monclient: _renew_subs 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac90124460 con 0x7fac90120730 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac86ffd640 10 monclient: dump: 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac94939640 4 set_mon_vals no callback set 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 10 monclient: _renew_subs 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.684+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac90123bc0 con 0x7fac90120730 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fac8801a070 con 0x7fac90120730 2024-09-10T16:10:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac983a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac983a0640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 0x7fac9002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac90123dc0 con 0x7fac900a3450 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac86ffd640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fac7403e1c0 0x7fac74040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 0x7fac9002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient: get_auth_request con 0x7fac900a3450 auth_method 0 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(132..132 src has 1..132) ==== 4815+0+0 (secure 0 0 0) 0x7fac88052ad0 con 0x7fac90120730 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.685+0000 7fac9593b640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fac7403e1c0 0x7fac74040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.686+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.686+0000 7fac9593b640 10 monclient: get_auth_request con 0x7fac7403e1c0 auth_method 0 2024-09-10T16:10:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.686+0000 7fac9613c640 10 monclient(hunting): handle_auth_done global_id 44158 payload 824 2024-09-10T16:10:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.686+0000 7fac9613c640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac900a2f10 con 0x7fac900a3450 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac9613c640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 msgr2=0x7fac9002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac9613c640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 0x7fac9002d440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac900747f0 tx=0x7fac8c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac9593b640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fac7403e1c0 0x7fac74040680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac90133d40 tx=0x7fac80009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c042020 con 0x7fac900a3450 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 10 monclient: dump: 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.703 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac8c038040 con 0x7fac900a3450 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 msgr2=0x7fac9002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 0x7fac9002d440 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fac900747f0 tx=0x7fac8c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 msgr2=0x7fac9002d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:09.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.688+0000 7fac86ffd640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac900a3450 0x7fac9002d440 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.694+0000 7fac983a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.694+0000 7fac983a0640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 0x7fac9002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.694+0000 7fac983a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.694+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac901384f0 con 0x7fac901218e0 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 0x7fac9002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient: get_auth_request con 0x7fac901218e0 auth_method 0 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:09.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 10 monclient(hunting): handle_auth_done global_id 44161 payload 824 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.695+0000 7fac9613c640 1 -- 172.21.15.164:0/3159391707 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac90122c60 con 0x7fac901218e0 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac9613c640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 msgr2=0x7fac9002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac9613c640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 0x7fac9002d980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fac8c000c00 tx=0x7fac8c009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac8c042040 con 0x7fac901218e0 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.697+0000 7fac86ffd640 10 monclient: dump: 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:09.706 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7fac8c051450 con 0x7fac901218e0 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 msgr2=0x7fac9002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 0x7fac9002d980 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fac8c000c00 tx=0x7fac8c009190 comp rx=0 tx=0).stop 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 msgr2=0x7fac9002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac86ffd640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac901218e0 0x7fac9002d980 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fac8c000c00 tx=0x7fac8c009190 comp rx=0 tx=0).stop 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:10:09.707 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 40, 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 24849, 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.708 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.709 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:10:09.710 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:09.711 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:10:09.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.713 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:09.714 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.715 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:09.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:10:09.717 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:10:09.719 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fac7403e1c0 msgr2=0x7fac74040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fac7403e1c0 0x7fac74040680 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fac90133d40 tx=0x7fac80009110 comp rx=0 tx=0).stop 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 10 monclient: shutdown 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 msgr2=0x7fac90120be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fac8800cb20 tx=0x7fac8800cf20 comp rx=0 tx=0).stop 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac9693d640 1 -- 172.21.15.164:0/3159391707 reap_dead start 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 shutdown_connections 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 --2- 172.21.15.164:0/3159391707 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac90120730 0x7fac90120be0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 >> 172.21.15.164:0/3159391707 conn(0x7fac900a7320 msgr2=0x7fac901356c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 shutdown_connections 2024-09-10T16:10:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:09.698+0000 7fac983a0640 1 -- 172.21.15.164:0/3159391707 wait complete. 2024-09-10T16:10:09.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:10:09.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:10:09.860 INFO:tasks.rados.rados.0.smithi165.stdout:502: done (3 left) 2024-09-10T16:10:09.860 INFO:tasks.rados.rados.0.smithi165.stdout:503: snap_remove snap 20 2024-09-10T16:10:10.861 INFO:tasks.rados.rados.0.smithi165.stdout:503: done (3 left) 2024-09-10T16:10:10.861 INFO:tasks.rados.rados.0.smithi165.stdout:504: snap_remove snap 58 2024-09-10T16:10:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:504: done (3 left) 2024-09-10T16:10:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:505: snap_create 2024-09-10T16:10:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:498: finishing write tid 2 to smithi16534841-44 2024-09-10T16:10:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:498: finishing write tid 3 to smithi16534841-44 2024-09-10T16:10:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:498: finishing write tid 4 to smithi16534841-44 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: finishing write tid 5 to smithi16534841-44 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: oid 44 updating version 0 to 136 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: oid 44 updating version 136 to 137 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: oid 44 version 137 is already newer than 134 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: oid 44 version 137 is already newer than 135 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 137 (ObjNum 207 snap 62 seq_num 207) dirty exists 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:498: left oid 44 (ObjNum 207 snap 62 seq_num 207) 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: finishing write tid 1 to smithi16534841-16 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: finishing write tid 2 to smithi16534841-16 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: finishing write tid 3 to smithi16534841-16 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: finishing write tid 4 to smithi16534841-16 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: finishing write tid 5 to smithi16534841-16 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: oid 16 updating version 0 to 384 2024-09-10T16:10:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:501: oid 16 updating version 384 to 389 2024-09-10T16:10:12.248 INFO:tasks.rados.rados.0.smithi165.stdout:501: oid 16 version 389 is already newer than 385 2024-09-10T16:10:12.248 INFO:tasks.rados.rados.0.smithi165.stdout:501: oid 16 version 389 is already newer than 388 2024-09-10T16:10:12.248 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 389 (ObjNum 208 snap 62 seq_num 208) dirty exists 2024-09-10T16:10:12.248 INFO:tasks.rados.rados.0.smithi165.stdout:501: left oid 16 (ObjNum 208 snap 62 seq_num 208) 2024-09-10T16:10:13.249 INFO:tasks.rados.rados.0.smithi165.stdout:498: done (3 left) 2024-09-10T16:10:13.250 INFO:tasks.rados.rados.0.smithi165.stdout:500: done (2 left) 2024-09-10T16:10:13.250 INFO:tasks.rados.rados.0.smithi165.stdout:501: done (1 left) 2024-09-10T16:10:13.250 INFO:tasks.rados.rados.0.smithi165.stdout:505: done (0 left) 2024-09-10T16:10:13.250 INFO:tasks.rados.rados.0.smithi165.stdout:506: write oid 21 current snap is 63 2024-09-10T16:10:13.250 INFO:tasks.rados.rados.0.smithi165.stdout:506: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2024-09-10T16:10:13.256 INFO:tasks.rados.rados.0.smithi165.stdout:506: writing smithi16534841-21 from 712663 to 1181235 tid 1 2024-09-10T16:10:13.263 INFO:tasks.rados.rados.0.smithi165.stdout:506: writing smithi16534841-21 from 1713913 to 2391970 tid 2 2024-09-10T16:10:13.269 INFO:tasks.rados.rados.0.smithi165.stdout:506: writing smithi16534841-21 from 2873523 to 3465256 tid 3 2024-09-10T16:10:13.269 INFO:tasks.rados.rados.0.smithi165.stdout:507: snap_remove snap 60 2024-09-10T16:10:14.259 INFO:tasks.rados.rados.0.smithi165.stdout:507: done (1 left) 2024-09-10T16:10:14.259 INFO:tasks.rados.rados.0.smithi165.stdout:508: rollback oid 6 current snap is 63 2024-09-10T16:10:14.259 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 57 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: finishing write tid 1 to smithi16534841-21 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: finishing write tid 2 to smithi16534841-21 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: finishing write tid 3 to smithi16534841-21 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: finishing write tid 4 to smithi16534841-21 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: finishing write tid 5 to smithi16534841-21 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: oid 21 updating version 0 to 306 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: oid 21 version 306 is already newer than 304 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: oid 21 version 306 is already newer than 305 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: oid 21 version 306 is already newer than 303 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 306 (ObjNum 209 snap 63 seq_num 209) dirty exists 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: left oid 21 (ObjNum 209 snap 63 seq_num 209) 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:506: done (1 left) 2024-09-10T16:10:14.260 INFO:tasks.rados.rados.0.smithi165.stdout:509: delete oid 36 current snap is 63 2024-09-10T16:10:14.264 INFO:tasks.rados.rados.0.smithi165.stdout:508: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:10:14.264 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 188 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:10:14.267 INFO:tasks.rados.rados.0.smithi165.stdout:508: done (1 left) 2024-09-10T16:10:14.267 INFO:tasks.rados.rados.0.smithi165.stdout:509: done (0 left) 2024-09-10T16:10:14.267 INFO:tasks.rados.rados.0.smithi165.stdout:510: write oid 15 current snap is 63 2024-09-10T16:10:14.267 INFO:tasks.rados.rados.0.smithi165.stdout:510: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-10T16:10:14.275 INFO:tasks.rados.rados.0.smithi165.stdout:510: writing smithi16534841-15 from 746277 to 1365347 tid 1 2024-09-10T16:10:14.282 INFO:tasks.rados.rados.0.smithi165.stdout:510: writing smithi16534841-15 from 1830877 to 2512603 tid 2 2024-09-10T16:10:14.287 INFO:tasks.rados.rados.0.smithi165.stdout:510: writing smithi16534841-15 from 2928369 to 3546277 tid 3 2024-09-10T16:10:14.287 INFO:tasks.rados.rados.0.smithi165.stdout:511: snap_remove snap 62 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:511: done (1 left) 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:512: snap_create 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: finishing write tid 1 to smithi16534841-15 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: finishing write tid 2 to smithi16534841-15 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: finishing write tid 3 to smithi16534841-15 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: finishing write tid 4 to smithi16534841-15 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: finishing write tid 5 to smithi16534841-15 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: oid 15 updating version 0 to 265 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: oid 15 version 265 is already newer than 264 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: oid 15 updating version 265 to 266 2024-09-10T16:10:14.862 INFO:tasks.rados.rados.0.smithi165.stdout:510: oid 15 updating version 266 to 267 2024-09-10T16:10:14.863 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 267 (ObjNum 211 snap 63 seq_num 211) dirty exists 2024-09-10T16:10:14.863 INFO:tasks.rados.rados.0.smithi165.stdout:510: left oid 15 (ObjNum 211 snap 63 seq_num 211) 2024-09-10T16:10:15.865 INFO:tasks.rados.rados.0.smithi165.stdout:510: done (1 left) 2024-09-10T16:10:15.866 INFO:tasks.rados.rados.0.smithi165.stdout:512: done (0 left) 2024-09-10T16:10:15.866 INFO:tasks.rados.rados.0.smithi165.stdout:513: rollback oid 39 current snap is 64 2024-09-10T16:10:15.866 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 35 2024-09-10T16:10:15.866 INFO:tasks.rados.rados.0.smithi165.stdout:514: delete oid 18 current snap is 64 2024-09-10T16:10:15.868 INFO:tasks.rados.rados.0.smithi165.stdout:513: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:10:15.869 INFO:tasks.rados.rados.0.smithi165.stdout:513: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:10:15.869 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 326 (ObjNum 88 snap 20 seq_num 88) dirty exists 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:513: done (1 left) 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:514: done (0 left) 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:515: rollback oid 6 current snap is 64 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 54 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:516: write oid 17 current snap is 64 2024-09-10T16:10:15.870 INFO:tasks.rados.rados.0.smithi165.stdout:516: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-10T16:10:15.877 INFO:tasks.rados.rados.0.smithi165.stdout:516: writing smithi16534841-17 from 779891 to 1265812 tid 1 2024-09-10T16:10:15.884 INFO:tasks.rados.rados.0.smithi165.stdout:516: writing smithi16534841-17 from 1947841 to 2633236 tid 2 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:516: writing smithi16534841-17 from 3383215 to 3579891 tid 3 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:517: snap_create 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:515: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:515: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 209 (ObjNum 172 snap 52 seq_num 172) dirty exists 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:516: finishing write tid 1 to smithi16534841-17 2024-09-10T16:10:15.890 INFO:tasks.rados.rados.0.smithi165.stdout:516: finishing write tid 2 to smithi16534841-17 2024-09-10T16:10:15.893 INFO:tasks.rados.rados.0.smithi165.stdout:516: finishing write tid 3 to smithi16534841-17 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: finishing write tid 4 to smithi16534841-17 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: finishing write tid 5 to smithi16534841-17 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: oid 17 updating version 0 to 248 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: oid 17 updating version 248 to 250 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: oid 17 updating version 250 to 251 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: oid 17 version 251 is already newer than 249 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 251 (ObjNum 213 snap 64 seq_num 213) dirty exists 2024-09-10T16:10:15.897 INFO:tasks.rados.rados.0.smithi165.stdout:516: left oid 17 (ObjNum 213 snap 64 seq_num 213) 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:515: done (2 left) 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:516: done (1 left) 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:517: done (0 left) 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:518: rollback oid 43 current snap is 65 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 37 2024-09-10T16:10:16.868 INFO:tasks.rados.rados.0.smithi165.stdout:519: snap_create 2024-09-10T16:10:16.873 INFO:tasks.rados.rados.0.smithi165.stdout:518: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:10:16.875 INFO:tasks.rados.rados.0.smithi165.stdout:518: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:10:16.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 160 (ObjNum 67 snap 8 seq_num 67) dirty exists 2024-09-10T16:10:17.865 INFO:tasks.rados.rados.0.smithi165.stdout:518: done (1 left) 2024-09-10T16:10:17.866 INFO:tasks.rados.rados.0.smithi165.stdout:519: done (0 left) 2024-09-10T16:10:17.866 INFO:tasks.rados.rados.0.smithi165.stdout:520: write oid 42 current snap is 66 2024-09-10T16:10:17.866 INFO:tasks.rados.rados.0.smithi165.stdout:520: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2024-09-10T16:10:17.873 INFO:tasks.rados.rados.0.smithi165.stdout:520: writing smithi16534841-42 from 796698 to 1357868 tid 1 2024-09-10T16:10:17.879 INFO:tasks.rados.rados.0.smithi165.stdout:520: writing smithi16534841-42 from 2006323 to 2551729 tid 2 2024-09-10T16:10:17.885 INFO:tasks.rados.rados.0.smithi165.stdout:520: writing smithi16534841-42 from 3126991 to 3596698 tid 3 2024-09-10T16:10:17.886 INFO:tasks.rados.rados.0.smithi165.stdout:521: write oid 39 current snap is 66 2024-09-10T16:10:17.886 INFO:tasks.rados.rados.0.smithi165.stdout:521: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2024-09-10T16:10:17.891 INFO:tasks.rados.rados.0.smithi165.stdout:521: writing smithi16534841-39 from 413505 to 1049924 tid 1 2024-09-10T16:10:17.899 INFO:tasks.rados.rados.0.smithi165.stdout:521: writing smithi16534841-39 from 1664805 to 2353869 tid 2 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:521: writing smithi16534841-39 from 3038061 to 3530132 tid 3 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:522: delete oid 33 current snap is 66 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: finishing write tid 1 to smithi16534841-42 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: finishing write tid 2 to smithi16534841-42 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: finishing write tid 3 to smithi16534841-42 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: finishing write tid 4 to smithi16534841-42 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: finishing write tid 5 to smithi16534841-42 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: oid 42 updating version 0 to 188 2024-09-10T16:10:17.906 INFO:tasks.rados.rados.0.smithi165.stdout:520: oid 42 updating version 188 to 190 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:520: oid 42 updating version 190 to 191 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:520: oid 42 version 191 is already newer than 189 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 191 (ObjNum 214 snap 66 seq_num 214) dirty exists 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:520: left oid 42 (ObjNum 214 snap 66 seq_num 214) 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:521: finishing write tid 1 to smithi16534841-39 2024-09-10T16:10:17.907 INFO:tasks.rados.rados.0.smithi165.stdout:521: finishing write tid 2 to smithi16534841-39 2024-09-10T16:10:17.911 INFO:tasks.rados.rados.0.smithi165.stdout:520: done (2 left) 2024-09-10T16:10:17.911 INFO:tasks.rados.rados.0.smithi165.stdout:522: done (1 left) 2024-09-10T16:10:17.911 INFO:tasks.rados.rados.0.smithi165.stdout:523: delete oid 8 current snap is 66 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:521: finishing write tid 3 to smithi16534841-39 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:523: done (1 left) 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:524: read oid 12 snap 37 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:524: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:525: snap_create 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:521: finishing write tid 4 to smithi16534841-39 2024-09-10T16:10:17.919 INFO:tasks.rados.rados.0.smithi165.stdout:521: finishing write tid 5 to smithi16534841-39 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:521: oid 39 updating version 0 to 332 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:521: oid 39 version 332 is already newer than 329 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:521: oid 39 version 332 is already newer than 331 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:521: oid 39 version 332 is already newer than 330 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 332 (ObjNum 215 snap 66 seq_num 215) dirty exists 2024-09-10T16:10:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:521: left oid 39 (ObjNum 215 snap 66 seq_num 215) 2024-09-10T16:10:17.921 INFO:tasks.rados.rados.0.smithi165.stdout:524: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:10:17.948 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1409110 2024-09-10T16:10:18.880 INFO:tasks.rados.rados.0.smithi165.stdout:521: done (2 left) 2024-09-10T16:10:18.880 INFO:tasks.rados.rados.0.smithi165.stdout:524: done (1 left) 2024-09-10T16:10:18.880 INFO:tasks.rados.rados.0.smithi165.stdout:525: done (0 left) 2024-09-10T16:10:18.880 INFO:tasks.rados.rados.0.smithi165.stdout:526: read oid 2 snap -1 2024-09-10T16:10:18.881 INFO:tasks.rados.rados.0.smithi165.stdout:526: expect deleted 2024-09-10T16:10:18.881 INFO:tasks.rados.rados.0.smithi165.stdout:527: snap_remove snap 52 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:527: done (1 left) 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:528: rollback oid 38 current snap is 67 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 66 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:526: done (1 left) 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:529: read oid 48 snap -1 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:529: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:530: read oid 18 snap -1 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:530: expect deleted 2024-09-10T16:10:19.863 INFO:tasks.rados.rados.0.smithi165.stdout:531: snap_remove snap 59 2024-09-10T16:10:20.864 INFO:tasks.rados.rados.0.smithi165.stdout:531: done (3 left) 2024-09-10T16:10:20.864 INFO:tasks.rados.rados.0.smithi165.stdout:532: snap_remove snap 65 2024-09-10T16:10:21.866 INFO:tasks.rados.rados.0.smithi165.stdout:532: done (3 left) 2024-09-10T16:10:21.866 INFO:tasks.rados.rados.0.smithi165.stdout:533: read oid 12 snap -1 2024-09-10T16:10:21.866 INFO:tasks.rados.rados.0.smithi165.stdout:533: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:534: rollback oid 22 current snap is 67 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 47 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:528: finishing rollback tid 0 to smithi16534841-38 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:528: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:528: finishing rollback tid 2 to smithi16534841-38 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 221 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:529: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:22.046 INFO:tasks.rados.rados.0.smithi165.stdout:534: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:10:22.046 INFO:tasks.rados.rados.0.smithi165.stdout:533: expect (ObjNum 11 snap 0 seq_num 11) 2024-09-10T16:10:22.087 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1074068 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:534: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 130 (ObjNum 129 snap 0 seq_num 1162719696) dirty dne 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:528: done (4 left) 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:529: done (3 left) 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:530: done (2 left) 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:533: done (1 left) 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:534: done (0 left) 2024-09-10T16:10:22.099 INFO:tasks.rados.rados.0.smithi165.stdout:535: delete oid 40 current snap is 67 2024-09-10T16:10:22.101 INFO:tasks.rados.rados.0.smithi165.stdout:535: done (0 left) 2024-09-10T16:10:22.101 INFO:tasks.rados.rados.0.smithi165.stdout:536: read oid 45 snap -1 2024-09-10T16:10:22.101 INFO:tasks.rados.rados.0.smithi165.stdout:536: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:10:22.101 INFO:tasks.rados.rados.0.smithi165.stdout:537: snap_create 2024-09-10T16:10:22.107 INFO:tasks.rados.rados.0.smithi165.stdout:536: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:10:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:22.892 INFO:tasks.rados.rados.0.smithi165.stdout:536: done (1 left) 2024-09-10T16:10:22.893 INFO:tasks.rados.rados.0.smithi165.stdout:537: done (0 left) 2024-09-10T16:10:22.893 INFO:tasks.rados.rados.0.smithi165.stdout:538: snap_remove snap 61 2024-09-10T16:10:23.901 INFO:tasks.rados.rados.0.smithi165.stdout:538: done (0 left) 2024-09-10T16:10:23.901 INFO:tasks.rados.rados.0.smithi165.stdout:539: write (excl) oid 50 current snap is 68 2024-09-10T16:10:23.901 INFO:tasks.rados.rados.0.smithi165.stdout:539: seq_num 219 ranges {480733=537415,1782380=696402,3031400=537159} 2024-09-10T16:10:23.906 INFO:tasks.rados.rados.0.smithi165.stdout:539: writing smithi16534841-50 from 480733 to 1018148 tid 1 2024-09-10T16:10:23.914 INFO:tasks.rados.rados.0.smithi165.stdout:539: writing smithi16534841-50 from 1782380 to 2478782 tid 2 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:539: writing smithi16534841-50 from 3031400 to 3568559 tid 3 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:540: copy_from oid 19 from oid 8 current snap is 68 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:539: finishing write tid 1 to smithi16534841-50 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:539: finishing write tid 2 to smithi16534841-50 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:541: write (excl) oid 25 current snap is 68 2024-09-10T16:10:23.921 INFO:tasks.rados.rados.0.smithi165.stdout:541: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-10T16:10:23.927 INFO:tasks.rados.rados.0.smithi165.stdout:541: writing smithi16534841-25 from 497540 to 1110204 tid 1 2024-09-10T16:10:23.934 INFO:tasks.rados.rados.0.smithi165.stdout:541: writing smithi16534841-25 from 1840862 to 2280922 tid 2 2024-09-10T16:10:23.941 INFO:tasks.rados.rados.0.smithi165.stdout:541: writing smithi16534841-25 from 3058823 to 3523607 tid 3 2024-09-10T16:10:23.941 INFO:tasks.rados.rados.0.smithi165.stdout:542: copy_from oid 40 from oid 1 current snap is 68 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: finishing write tid 3 to smithi16534841-50 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: finishing write tid 4 to smithi16534841-50 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: finishing write tid 5 to smithi16534841-50 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: oid 50 updating version 0 to 208 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: oid 50 version 208 is already newer than 206 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: oid 50 version 208 is already newer than 207 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: oid 50 version 208 is already newer than 205 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 208 (ObjNum 219 snap 68 seq_num 219) dirty exists 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:539: left oid 50 (ObjNum 219 snap 68 seq_num 219) 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:540: finishing copy_from to smithi16534841-19 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:540: got expected ENOENT (src dne) 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:540: finishing copy_from racing read to smithi16534841-19 2024-09-10T16:10:23.942 INFO:tasks.rados.rados.0.smithi165.stdout:541: finishing write tid 1 to smithi16534841-25 2024-09-10T16:10:23.943 INFO:tasks.rados.rados.0.smithi165.stdout:539: done (3 left) 2024-09-10T16:10:23.943 INFO:tasks.rados.rados.0.smithi165.stdout:540: done (2 left) 2024-09-10T16:10:23.943 INFO:tasks.rados.rados.0.smithi165.stdout:543: rollback oid 45 current snap is 68 2024-09-10T16:10:23.943 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 67 2024-09-10T16:10:23.943 INFO:tasks.rados.rados.0.smithi165.stdout:544: snap_remove snap 66 2024-09-10T16:10:24.867 INFO:tasks.rados.rados.0.smithi165.stdout:544: done (3 left) 2024-09-10T16:10:24.867 INFO:tasks.rados.rados.0.smithi165.stdout:545: read oid 19 snap -1 2024-09-10T16:10:24.867 INFO:tasks.rados.rados.0.smithi165.stdout:545: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:546: delete oid 9 current snap is 68 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: finishing write tid 2 to smithi16534841-25 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:543: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:543: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:543: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 168 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: finishing write tid 3 to smithi16534841-25 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: finishing write tid 4 to smithi16534841-25 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: finishing write tid 5 to smithi16534841-25 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: oid 25 updating version 0 to 324 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: oid 25 updating version 324 to 331 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: oid 25 version 331 is already newer than 329 2024-09-10T16:10:24.868 INFO:tasks.rados.rados.0.smithi165.stdout:541: oid 25 version 331 is already newer than 330 2024-09-10T16:10:24.869 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 331 (ObjNum 220 snap 68 seq_num 220) dirty exists 2024-09-10T16:10:24.869 INFO:tasks.rados.rados.0.smithi165.stdout:541: left oid 25 (ObjNum 220 snap 68 seq_num 220) 2024-09-10T16:10:24.869 INFO:tasks.rados.rados.0.smithi165.stdout:542: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:10:24.869 INFO:tasks.rados.rados.0.smithi165.stdout:542: finishing copy_from to smithi16534841-40 2024-09-10T16:10:24.869 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 300 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:541: done (4 left) 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:542: done (3 left) 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:543: done (2 left) 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:546: done (1 left) 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:547: read oid 3 snap -1 2024-09-10T16:10:24.874 INFO:tasks.rados.rados.0.smithi165.stdout:547: expect deleted 2024-09-10T16:10:24.875 INFO:tasks.rados.rados.0.smithi165.stdout:548: write (excl) oid 5 current snap is 68 2024-09-10T16:10:24.875 INFO:tasks.rados.rados.0.smithi165.stdout:548: seq_num 222 ranges {531154=479515,1557826=443729,2713669=487328,3679295=51859} 2024-09-10T16:10:24.880 INFO:tasks.rados.rados.0.smithi165.stdout:548: writing smithi16534841-5 from 531154 to 1010669 tid 1 2024-09-10T16:10:24.886 INFO:tasks.rados.rados.0.smithi165.stdout:548: writing smithi16534841-5 from 1557826 to 2001555 tid 2 2024-09-10T16:10:24.892 INFO:tasks.rados.rados.0.smithi165.stdout:548: writing smithi16534841-5 from 2713669 to 3200997 tid 3 2024-09-10T16:10:24.895 INFO:tasks.rados.rados.0.smithi165.stdout:548: writing smithi16534841-5 from 3679295 to 3731154 tid 4 2024-09-10T16:10:24.895 INFO:tasks.rados.rados.0.smithi165.stdout:549: copy_from oid 16 from oid 47 current snap is 68 2024-09-10T16:10:24.895 INFO:tasks.rados.rados.0.smithi165.stdout:545: expect (ObjNum 21 snap 0 seq_num 21) 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 1 to smithi16534841-5 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 2 to smithi16534841-5 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:545: done (3 left) 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:547: done (2 left) 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:550: read oid 18 snap -1 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:550: expect deleted 2024-09-10T16:10:24.968 INFO:tasks.rados.rados.0.smithi165.stdout:551: read oid 35 snap -1 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:551: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:552: snap_create 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 3 to smithi16534841-5 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 4 to smithi16534841-5 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 5 to smithi16534841-5 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: finishing write tid 6 to smithi16534841-5 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: oid 5 updating version 0 to 220 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: oid 5 updating version 220 to 222 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: oid 5 version 222 is already newer than 221 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: oid 5 updating version 222 to 223 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:548: oid 5 version 223 is already newer than 219 2024-09-10T16:10:24.969 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 223 (ObjNum 222 snap 68 seq_num 222) dirty exists 2024-09-10T16:10:24.970 INFO:tasks.rados.rados.0.smithi165.stdout:548: left oid 5 (ObjNum 222 snap 68 seq_num 222) 2024-09-10T16:10:24.970 INFO:tasks.rados.rados.0.smithi165.stdout:549: finishing copy_from to smithi16534841-16 2024-09-10T16:10:24.970 INFO:tasks.rados.rados.0.smithi165.stdout:549: got expected ENOENT (src dne) 2024-09-10T16:10:24.970 INFO:tasks.rados.rados.0.smithi165.stdout:549: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:10:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:551: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-10T16:10:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:25.037 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1766217 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:548: done (4 left) 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:549: done (3 left) 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:550: done (2 left) 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:551: done (1 left) 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:552: done (0 left) 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:553: write (excl) oid 39 current snap is 69 2024-09-10T16:10:25.903 INFO:tasks.rados.rados.0.smithi165.stdout:553: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-10T16:10:25.908 INFO:tasks.rados.rados.0.smithi165.stdout:553: writing smithi16534841-39 from 547961 to 1102725 tid 1 2024-09-10T16:10:25.915 INFO:tasks.rados.rados.0.smithi165.stdout:553: writing smithi16534841-39 from 1616308 to 2320048 tid 2 2024-09-10T16:10:25.922 INFO:tasks.rados.rados.0.smithi165.stdout:553: writing smithi16534841-39 from 2741092 to 3439692 tid 3 2024-09-10T16:10:25.922 INFO:tasks.rados.rados.0.smithi165.stdout:554: write oid 37 current snap is 69 2024-09-10T16:10:25.922 INFO:tasks.rados.rados.0.smithi165.stdout:554: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-10T16:10:25.928 INFO:tasks.rados.rados.0.smithi165.stdout:554: writing smithi16534841-37 from 564768 to 1194781 tid 1 2024-09-10T16:10:25.933 INFO:tasks.rados.rados.0.smithi165.stdout:554: writing smithi16534841-37 from 1674790 to 2122188 tid 2 2024-09-10T16:10:25.940 INFO:tasks.rados.rados.0.smithi165.stdout:554: writing smithi16534841-37 from 2768515 to 3394740 tid 3 2024-09-10T16:10:25.940 INFO:tasks.rados.rados.0.smithi165.stdout:555: snap_remove snap 54 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:555: done (2 left) 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:556: copy_from oid 47 from oid 17 current snap is 69 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: finishing write tid 1 to smithi16534841-39 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: finishing write tid 2 to smithi16534841-39 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: finishing write tid 3 to smithi16534841-39 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: finishing write tid 4 to smithi16534841-39 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: finishing write tid 5 to smithi16534841-39 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: oid 39 updating version 0 to 344 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: oid 39 updating version 344 to 345 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: oid 39 version 345 is already newer than 343 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:553: oid 39 version 345 is already newer than 342 2024-09-10T16:10:26.909 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 345 (ObjNum 223 snap 69 seq_num 223) dirty exists 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:553: left oid 39 (ObjNum 223 snap 69 seq_num 223) 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: finishing write tid 1 to smithi16534841-37 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: finishing write tid 2 to smithi16534841-37 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: finishing write tid 3 to smithi16534841-37 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: finishing write tid 4 to smithi16534841-37 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: finishing write tid 5 to smithi16534841-37 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: oid 37 updating version 0 to 59 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: oid 37 updating version 59 to 60 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: oid 37 updating version 60 to 61 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: oid 37 version 61 is already newer than 58 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 61 (ObjNum 224 snap 69 seq_num 224) dirty exists 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: left oid 37 (ObjNum 224 snap 69 seq_num 224) 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:553: done (2 left) 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:554: done (1 left) 2024-09-10T16:10:26.910 INFO:tasks.rados.rados.0.smithi165.stdout:557: delete oid 44 current snap is 69 2024-09-10T16:10:26.920 INFO:tasks.rados.rados.0.smithi165.stdout:557: done (1 left) 2024-09-10T16:10:26.920 INFO:tasks.rados.rados.0.smithi165.stdout:558: delete oid 35 current snap is 69 2024-09-10T16:10:26.926 INFO:tasks.rados.rados.0.smithi165.stdout:558: done (1 left) 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:559: rollback oid 14 current snap is 69 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 43 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:560: read oid 2 snap -1 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:560: expect deleted 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:561: read oid 43 snap -1 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:561: expect (ObjNum 67 snap 8 seq_num 67) 2024-09-10T16:10:26.927 INFO:tasks.rados.rados.0.smithi165.stdout:562: read oid 46 snap -1 2024-09-10T16:10:26.928 INFO:tasks.rados.rados.0.smithi165.stdout:562: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:10:26.928 INFO:tasks.rados.rados.0.smithi165.stdout:563: write oid 13 current snap is 69 2024-09-10T16:10:26.929 INFO:tasks.rados.rados.0.smithi165.stdout:563: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2024-09-10T16:10:26.934 INFO:tasks.rados.rados.0.smithi165.stdout:563: writing smithi16534841-13 from 615189 to 1070949 tid 1 2024-09-10T16:10:26.942 INFO:tasks.rados.rados.0.smithi165.stdout:563: writing smithi16534841-13 from 1733883 to 2444961 tid 2 2024-09-10T16:10:26.948 INFO:tasks.rados.rados.0.smithi165.stdout:563: writing smithi16534841-13 from 3134431 to 3594472 tid 3 2024-09-10T16:10:26.948 INFO:tasks.rados.rados.0.smithi165.stdout:564: write (excl) oid 12 current snap is 69 2024-09-10T16:10:26.949 INFO:tasks.rados.rados.0.smithi165.stdout:564: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-10T16:10:26.955 INFO:tasks.rados.rados.0.smithi165.stdout:564: writing smithi16534841-12 from 631996 to 1163005 tid 1 2024-09-10T16:10:26.961 INFO:tasks.rados.rados.0.smithi165.stdout:564: writing smithi16534841-12 from 1792365 to 2247101 tid 2 2024-09-10T16:10:26.968 INFO:tasks.rados.rados.0.smithi165.stdout:564: writing smithi16534841-12 from 2761854 to 3433167 tid 3 2024-09-10T16:10:26.968 INFO:tasks.rados.rados.0.smithi165.stdout:565: rollback oid 19 current snap is 69 2024-09-10T16:10:26.968 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 47 2024-09-10T16:10:26.968 INFO:tasks.rados.rados.0.smithi165.stdout:559: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:10:26.968 INFO:tasks.rados.rados.0.smithi165.stdout:562: expect (ObjNum 157 snap 46 seq_num 157) 2024-09-10T16:10:26.993 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2213233 2024-09-10T16:10:27.004 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:27.057 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 704131 2024-09-10T16:10:27.097 INFO:tasks.rados.rados.0.smithi165.stdout:559: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:10:27.098 INFO:tasks.rados.rados.0.smithi165.stdout:559: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:10:27.098 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 303 (ObjNum 142 snap 42 seq_num 142) dirty exists 2024-09-10T16:10:27.098 INFO:tasks.rados.rados.0.smithi165.stdout:561: expect (ObjNum 67 snap 8 seq_num 67) 2024-09-10T16:10:27.118 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2587666 2024-09-10T16:10:27.150 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2123432 2024-09-10T16:10:27.207 INFO:tasks.rados.rados.0.smithi165.stdout:556: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:10:27.207 INFO:tasks.rados.rados.0.smithi165.stdout:556: finishing copy_from to smithi16534841-47 2024-09-10T16:10:27.207 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 251 (ObjNum 213 snap 64 seq_num 213) dirty exists 2024-09-10T16:10:27.207 INFO:tasks.rados.rados.0.smithi165.stdout:563: finishing write tid 1 to smithi16534841-13 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: finishing write tid 2 to smithi16534841-13 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:556: done (7 left) 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:559: done (6 left) 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:560: done (5 left) 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:561: done (4 left) 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:562: done (3 left) 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:566: write (excl) oid 27 current snap is 69 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: finishing write tid 3 to smithi16534841-13 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: finishing write tid 4 to smithi16534841-13 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: finishing write tid 5 to smithi16534841-13 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: oid 13 updating version 0 to 141 2024-09-10T16:10:27.208 INFO:tasks.rados.rados.0.smithi165.stdout:563: oid 13 updating version 141 to 142 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:563: oid 13 updating version 142 to 143 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:563: oid 13 version 143 is already newer than 140 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 143 (ObjNum 227 snap 69 seq_num 227) dirty exists 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:563: left oid 13 (ObjNum 227 snap 69 seq_num 227) 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:564: finishing write tid 1 to smithi16534841-12 2024-09-10T16:10:27.209 INFO:tasks.rados.rados.0.smithi165.stdout:566: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-10T16:10:27.214 INFO:tasks.rados.rados.0.smithi165.stdout:566: writing smithi16534841-27 from 648803 to 1371414 tid 1 2024-09-10T16:10:27.221 INFO:tasks.rados.rados.0.smithi165.stdout:566: writing smithi16534841-27 from 1967200 to 2681947 tid 2 2024-09-10T16:10:27.228 INFO:tasks.rados.rados.0.smithi165.stdout:566: writing smithi16534841-27 from 3421983 to 3848803 tid 3 2024-09-10T16:10:27.228 INFO:tasks.rados.rados.0.smithi165.stdout:563: done (3 left) 2024-09-10T16:10:27.228 INFO:tasks.rados.rados.0.smithi165.stdout:567: snap_remove snap 27 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:567: done (3 left) 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:568: copy_from oid 5 from oid 33 current snap is 69 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:564: finishing write tid 2 to smithi16534841-12 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:565: finishing rollback tid 0 to smithi16534841-19 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:565: finishing rollback tid 1 to smithi16534841-19 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:564: finishing write tid 3 to smithi16534841-12 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:564: finishing write tid 4 to smithi16534841-12 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:565: finishing rollback tid 2 to smithi16534841-19 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 455 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:10:27.913 INFO:tasks.rados.rados.0.smithi165.stdout:564: finishing write tid 5 to smithi16534841-12 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:564: oid 12 updating version 0 to 38 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:564: oid 12 version 38 is already newer than 36 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:564: oid 12 version 38 is already newer than 37 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:564: oid 12 version 38 is already newer than 35 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 38 (ObjNum 228 snap 69 seq_num 228) dirty exists 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:564: left oid 12 (ObjNum 228 snap 69 seq_num 228) 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: finishing write tid 1 to smithi16534841-27 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: finishing write tid 2 to smithi16534841-27 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: finishing write tid 3 to smithi16534841-27 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: finishing write tid 4 to smithi16534841-27 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: finishing write tid 5 to smithi16534841-27 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: oid 27 updating version 0 to 375 2024-09-10T16:10:27.914 INFO:tasks.rados.rados.0.smithi165.stdout:566: oid 27 version 375 is already newer than 374 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:566: oid 27 version 375 is already newer than 373 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:566: oid 27 version 375 is already newer than 372 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 375 (ObjNum 229 snap 69 seq_num 229) dirty exists 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:566: left oid 27 (ObjNum 229 snap 69 seq_num 229) 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:564: done (3 left) 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:565: done (2 left) 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:566: done (1 left) 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:569: write oid 25 current snap is 69 2024-09-10T16:10:27.915 INFO:tasks.rados.rados.0.smithi165.stdout:569: seq_num 230 ranges {665610=797860,1909329=458405,2816700=410210} 2024-09-10T16:10:27.921 INFO:tasks.rados.rados.0.smithi165.stdout:569: writing smithi16534841-25 from 665610 to 1463470 tid 1 2024-09-10T16:10:27.926 INFO:tasks.rados.rados.0.smithi165.stdout:569: writing smithi16534841-25 from 1909329 to 2367734 tid 2 2024-09-10T16:10:27.932 INFO:tasks.rados.rados.0.smithi165.stdout:569: writing smithi16534841-25 from 2816700 to 3226910 tid 3 2024-09-10T16:10:27.932 INFO:tasks.rados.rados.0.smithi165.stdout:570: delete oid 31 current snap is 69 2024-09-10T16:10:27.932 INFO:tasks.rados.rados.0.smithi165.stdout:568: finishing copy_from to smithi16534841-5 2024-09-10T16:10:27.932 INFO:tasks.rados.rados.0.smithi165.stdout:568: got expected ENOENT (src dne) 2024-09-10T16:10:27.933 INFO:tasks.rados.rados.0.smithi165.stdout:568: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:10:27.947 INFO:tasks.rados.rados.0.smithi165.stdout:568: done (2 left) 2024-09-10T16:10:27.947 INFO:tasks.rados.rados.0.smithi165.stdout:570: done (1 left) 2024-09-10T16:10:27.947 INFO:tasks.rados.rados.0.smithi165.stdout:571: delete oid 2 current snap is 69 2024-09-10T16:10:27.950 INFO:tasks.rados.rados.0.smithi165.stdout:571: done (1 left) 2024-09-10T16:10:27.950 INFO:tasks.rados.rados.0.smithi165.stdout:572: delete oid 46 current snap is 69 2024-09-10T16:10:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:569: finishing write tid 1 to smithi16534841-25 2024-09-10T16:10:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:569: finishing write tid 2 to smithi16534841-25 2024-09-10T16:10:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:569: finishing write tid 3 to smithi16534841-25 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: finishing write tid 4 to smithi16534841-25 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: finishing write tid 5 to smithi16534841-25 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: oid 25 updating version 0 to 352 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: oid 25 version 352 is already newer than 345 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: oid 25 version 352 is already newer than 346 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: oid 25 version 352 is already newer than 351 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 352 (ObjNum 230 snap 69 seq_num 230) dirty exists 2024-09-10T16:10:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:569: left oid 25 (ObjNum 230 snap 69 seq_num 230) 2024-09-10T16:10:27.953 INFO:tasks.rados.rados.0.smithi165.stdout:569: done (1 left) 2024-09-10T16:10:27.953 INFO:tasks.rados.rados.0.smithi165.stdout:572: done (0 left) 2024-09-10T16:10:27.953 INFO:tasks.rados.rados.0.smithi165.stdout:573: copy_from oid 26 from oid 3 current snap is 69 2024-09-10T16:10:27.953 INFO:tasks.rados.rados.0.smithi165.stdout:574: read oid 20 snap 47 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:574: expect (ObjNum 70 snap 10 seq_num 70) 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:575: read oid 17 snap -1 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:575: expect (ObjNum 213 snap 64 seq_num 213) 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:576: delete oid 14 current snap is 69 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:573: finishing copy_from to smithi16534841-26 2024-09-10T16:10:27.954 INFO:tasks.rados.rados.0.smithi165.stdout:573: got expected ENOENT (src dne) 2024-09-10T16:10:27.955 INFO:tasks.rados.rados.0.smithi165.stdout:573: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:10:27.957 INFO:tasks.rados.rados.0.smithi165.stdout:573: done (3 left) 2024-09-10T16:10:27.957 INFO:tasks.rados.rados.0.smithi165.stdout:576: done (2 left) 2024-09-10T16:10:27.957 INFO:tasks.rados.rados.0.smithi165.stdout:577: snap_remove snap 42 2024-09-10T16:10:28.914 INFO:tasks.rados.rados.0.smithi165.stdout:577: done (2 left) 2024-09-10T16:10:28.915 INFO:tasks.rados.rados.0.smithi165.stdout:578: rollback oid 35 current snap is 69 2024-09-10T16:10:28.915 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 33 2024-09-10T16:10:28.915 INFO:tasks.rados.rados.0.smithi165.stdout:575: expect (ObjNum 213 snap 64 seq_num 213) 2024-09-10T16:10:29.013 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2817318 2024-09-10T16:10:29.034 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:29.111 INFO:tasks.rados.rados.0.smithi165.stdout:574: expect (ObjNum 70 snap 10 seq_num 70) 2024-09-10T16:10:29.215 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:29.266 INFO:tasks.rados.rados.0.smithi165.stdout:578: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:578: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 151 (ObjNum 93 snap 23 seq_num 93) dirty exists 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:574: done (2 left) 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:575: done (1 left) 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:578: done (0 left) 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:579: rollback oid 43 current snap is 69 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 48 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:580: rollback oid 49 current snap is 69 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 67 2024-09-10T16:10:29.267 INFO:tasks.rados.rados.0.smithi165.stdout:581: snap_remove snap 37 2024-09-10T16:10:29.723 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:10:29.723 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:10:29.723 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:10:29.727 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:10:29.727 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:10:29.727 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:29.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.834+0000 7f90ae8c5640 10 monclient: get_monmap_and_config 2024-09-10T16:10:29.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.834+0000 7f90ae8c5640 10 monclient: build_initial_monmap 2024-09-10T16:10:29.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.834+0000 7f90ae8c5640 10 monclient: monmap: 2024-09-10T16:10:29.836 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:29.836 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:29.835765+0000 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:29.835765+0000 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:29.837 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.835+0000 7f90ae8c5640 1 Processor -- start 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.835+0000 7f90ae8c5640 1 -- start start 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.835+0000 7f90ae8c5640 10 monclient: init 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-10T16:10:29.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f90a806d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 0x7f90a80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-a con 0x7f90a806d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90a80a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90a8135510 con 0x7f90a80a4c20 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90a809cb90 con 0x7f90a806d160 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f90a809ce70 con 0x7f90a806d540 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90ae8c5640 10 monclient(hunting): authenticate will time out at 1008.323435s 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 0x7f90a80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a7fff640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f90a806d160 0x7f90a8134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:52578/0 (socket says 172.21.15.164:52578) 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a7fff640 1 -- 172.21.15.164:0/897214362 learned_addr learned my addr 172.21.15.164:0/897214362 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:29.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 10 monclient(hunting): get_auth_request con 0x7f90a806d540 auth_method 0 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 msgr2=0x7f90a80a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 msgr2=0x7f90a80a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 0x7f90a80a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:29.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.836+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 0x7f90a80a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:29.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90ace62640 1 -- 172.21.15.164:0/897214362 >> v1:172.21.15.164:6789/0 conn(0x7f90a80a4c20 legacy=0x7f90a813f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:29.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a7fff640 1 -- 172.21.15.164:0/897214362 >> v1:172.21.15.165:6789/0 conn(0x7f90a806d160 legacy=0x7f90a8134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:29.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1123680790 0 0) 0x7f90a809cb90 con 0x7f90a806d160 2024-09-10T16:10:29.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 10 monclient(hunting): _negotiate 2024-09-10T16:10:29.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 10 monclient(hunting): my global_id is 44164 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.838+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9090003520 con 0x7f90a806d160 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 650879245 0 0) 0x7f9090003520 con 0x7f90a806d160 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 msgr2=0x7f90a80a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 --2- 172.21.15.164:0/897214362 >> v2:172.21.15.164:3300/0 conn(0x7f90a806d540 0x7f90a80a46e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 >> v1:172.21.15.164:6789/0 conn(0x7f90a80a4c20 legacy=0x7f90a813f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a809d150 con 0x7f90a806d160 2024-09-10T16:10:29.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ae8c5640 5 monclient: authenticate success, global_id 44164 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f9098001a60 con 0x7f90a806d160 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: dump: 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:29.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 >> v1:172.21.15.165:6789/0 conn(0x7f90a806d160 legacy=0x7f90a8134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090003ac0 0x7f9090003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): picked mon.b con 0x7f9090003ac0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 0x7f9090004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): picked mon.a con 0x7f9090004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9090005410 0x7f909000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): picked mon.c con 0x7f9090005410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f90a809ce70 con 0x7f9090004800 2024-09-10T16:10:29.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f909000c350 con 0x7f9090003ac0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f909000c550 con 0x7f9090005410 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ae8c5640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 0x7f9090004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): get_auth_request con 0x7f9090004800 auth_method 0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9090005410 0x7f909000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090003ac0 0x7f9090003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): get_auth_request con 0x7f9090005410 auth_method 0 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): get_auth_request con 0x7f9090003ac0 auth_method 0 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:29.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient(hunting): handle_auth_done global_id 44164 payload 293 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9090005410 msgr2=0x7f909000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9090005410 0x7f909000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090003ac0 msgr2=0x7f9090003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090003ac0 0x7f9090003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 monclient: found mon.a 2024-09-10T16:10:29.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9090003520 con 0x7f9090004800 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient: _finish_auth 0 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9090005410 0x7f909000ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.839+0000 7f90ace62640 1 -- 172.21.15.164:0/897214362 reap_dead start 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a77fe640 1 -- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 msgr2=0x7f9090004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a77fe640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 0x7f9090004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b9e0 tx=0x7f9090005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27b23f132fcda082 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c030070 con 0x7f9090004800 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:29.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: dump: 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90ae8c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f909c034400 con 0x7f9090004800 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:29.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 1 -- 172.21.15.164:0/897214362 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c00b5e0 con 0x7f9090004800 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90ae8c5640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90ae8c5640 4 set_mon_vals no callback set 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.842+0000 7f90a6ffd640 10 monclient: dump: 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:29.861 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 10 monclient: shutdown 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 -- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 msgr2=0x7f9090004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 0x7f9090004c80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f909c00b9e0 tx=0x7f9090005830 comp rx=0 tx=0).stop 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 -- 172.21.15.164:0/897214362 shutdown_connections 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/897214362 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9090004800 0x7f9090004c80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 -- 172.21.15.164:0/897214362 >> 172.21.15.164:0/897214362 conn(0x7f90a80aacb0 msgr2=0x7f90a80ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 -- 172.21.15.164:0/897214362 shutdown_connections 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 1 -- 172.21.15.164:0/897214362 wait complete. 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 10 monclient: build_initial_monmap 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.843+0000 7f90ae8c5640 10 monclient: monmap: 2024-09-10T16:10:29.862 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 Processor -- start 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 -- start start 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient: init 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:10:29.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 0x7f90a81ca090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-b con 0x7f9090004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-c con 0x7f90a81ca5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): picked mon.noname-a con 0x7f90a81cd730 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f90a8134e00 con 0x7f90a81cd730 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90a80a4950 con 0x7f9090004800 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f90a8134f80 con 0x7f90a81ca5d0 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:29.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.844+0000 7f90ae8c5640 10 monclient(hunting): authenticate will time out at 1008.331976s 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46006/0 (socket says 172.21.15.164:46006) 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43788/0 (socket says 172.21.15.164:43788) 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 -- 172.21.15.164:0/1070620823 learned_addr learned my addr 172.21.15.164:0/1070620823 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 0x7f90a81ca090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 10 monclient(hunting): get_auth_request con 0x7f90a81cd730 auth_method 0 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): get_auth_request con 0x7f90a81ca5d0 auth_method 0 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): get_auth_request con 0x7f9090004800 auth_method 0 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 msgr2=0x7f90a81cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 msgr2=0x7f90a81cdb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:29.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90ace62640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient(hunting): handle_auth_done global_id 4772 payload 824 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 msgr2=0x7f90a81ca090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 0x7f90a81ca090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 msgr2=0x7f90a81cdb50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 monclient: found mon.noname-c 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a77fe640 1 -- 172.21.15.164:0/1070620823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a8120ee0 con 0x7f90a81ca5d0 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.845+0000 7f90a7fff640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 0x7f90a81ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a77fe640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 msgr2=0x7f90a81cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a77fe640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f909000c320 tx=0x7f909c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:29.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c043020 con 0x7f90a81ca5d0 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 10 monclient: dump: 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:29.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f909c03fa90 con 0x7f90a81ca5d0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.847+0000 7f90ae8c5640 5 monclient: authenticate success, global_id 4772 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 10 monclient: _renew_subs 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90a80a2370 con 0x7f90a81ca5d0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f909c034e40 con 0x7f90a81ca5d0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:29.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a4ff9640 10 monclient: dump: 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 10 monclient: _renew_subs 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90a80a35d0 con 0x7f90a81ca5d0 2024-09-10T16:10:29.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.848+0000 7f90a67fc640 4 set_mon_vals no callback set 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f909c046020 con 0x7f90a81ca5d0 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90ae8c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90ae8c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90ae8c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90a80a3f40 con 0x7f90a81ca5d0 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90a4ff9640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 0x7f90940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(152..152 src has 1..152) ==== 4847+0+0 (secure 0 0 0) 0x7f909c073a00 con 0x7f90a81ca5d0 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90a7fff640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 0x7f90940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:29.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.849+0000 7f90a7fff640 10 monclient: get_auth_request con 0x7f909403e210 auth_method 0 2024-09-10T16:10:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.851+0000 7f90a7fff640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 0x7f90940406d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f90980014a0 tx=0x7f9098007bc0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.854+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f909c04d030 con 0x7f90a81ca5d0 2024-09-10T16:10:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.854+0000 7f90a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:29.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:29.854+0000 7f90a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:581: done (2 left) 2024-09-10T16:10:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:582: read oid 48 snap -1 2024-09-10T16:10:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:582: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:583: write oid 39 current snap is 69 2024-09-10T16:10:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:579: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:10:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:583: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-10T16:10:29.874 INFO:tasks.rados.rados.0.smithi165.stdout:583: writing smithi16534841-39 from 749645 to 1523750 tid 1 2024-09-10T16:10:29.881 INFO:tasks.rados.rados.0.smithi165.stdout:583: writing smithi16534841-39 from 2085386 to 2694787 tid 2 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:583: writing smithi16534841-39 from 3237462 to 3949645 tid 3 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:584: write (excl) oid 21 current snap is 69 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:580: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:579: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:580: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:10:29.890 INFO:tasks.rados.rados.0.smithi165.stdout:584: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2024-09-10T16:10:29.898 INFO:tasks.rados.rados.0.smithi165.stdout:584: writing smithi16534841-21 from 766452 to 1332159 tid 1 2024-09-10T16:10:29.904 INFO:tasks.rados.rados.0.smithi165.stdout:584: writing smithi16534841-21 from 1860221 to 2329633 tid 2 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:584: writing smithi16534841-21 from 2981238 to 3575433 tid 3 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:585: delete oid 12 current snap is 69 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:579: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 184 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:580: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 254 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:10:29.913 INFO:tasks.rados.rados.0.smithi165.stdout:582: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:29.942 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2285670 2024-09-10T16:10:30.079 INFO:tasks.rados.rados.0.smithi165.stdout:583: finishing write tid 1 to smithi16534841-39 2024-09-10T16:10:30.079 INFO:tasks.rados.rados.0.smithi165.stdout:583: finishing write tid 2 to smithi16534841-39 2024-09-10T16:10:30.079 INFO:tasks.rados.rados.0.smithi165.stdout:583: finishing write tid 3 to smithi16534841-39 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: finishing write tid 4 to smithi16534841-39 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: finishing write tid 5 to smithi16534841-39 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: oid 39 updating version 0 to 374 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: oid 39 version 374 is already newer than 373 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: oid 39 updating version 374 to 375 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: oid 39 version 375 is already newer than 370 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 375 (ObjNum 235 snap 69 seq_num 235) dirty exists 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:583: left oid 39 (ObjNum 235 snap 69 seq_num 235) 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:579: done (5 left) 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:580: done (4 left) 2024-09-10T16:10:30.080 INFO:tasks.rados.rados.0.smithi165.stdout:582: done (3 left) 2024-09-10T16:10:30.081 INFO:tasks.rados.rados.0.smithi165.stdout:583: done (2 left) 2024-09-10T16:10:30.081 INFO:tasks.rados.rados.0.smithi165.stdout:585: done (1 left) 2024-09-10T16:10:30.081 INFO:tasks.rados.rados.0.smithi165.stdout:586: snap_remove snap 64 2024-09-10T16:10:30.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90ae8c5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:30.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90ae8c5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90ae8c5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f90a8137ce0 con 0x7f90a81ca5d0 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90a4ff9640 1 -- 172.21.15.164:0/1070620823 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f909c03c3a0 con 0x7f90a81ca5d0 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.112+0000 7f90a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:30.114 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:30.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 msgr2=0x7f90940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 0x7f90940406d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f90980014a0 tx=0x7f9098007bc0 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 10 monclient: shutdown 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 msgr2=0x7f90a81cd1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f909000c320 tx=0x7f909c004730 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 shutdown_connections 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f909403e210 0x7f90940406d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90a81cd730 0x7f90a81cdb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90a81ca5d0 0x7f90a81cd1f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 --2- 172.21.15.164:0/1070620823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9090004800 0x7f90a81ca090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 >> 172.21.15.164:0/1070620823 conn(0x7f90a80aacb0 msgr2=0x7f90a80a5300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:30.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 shutdown_connections 2024-09-10T16:10:30.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:30.115+0000 7f90ae8c5640 1 -- 172.21.15.164:0/1070620823 wait complete. 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:586: done (1 left) 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:587: snap_create 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:584: finishing write tid 1 to smithi16534841-21 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:584: finishing write tid 2 to smithi16534841-21 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:584: finishing write tid 3 to smithi16534841-21 2024-09-10T16:10:31.412 INFO:tasks.rados.rados.0.smithi165.stdout:584: finishing write tid 4 to smithi16534841-21 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: finishing write tid 5 to smithi16534841-21 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: oid 21 updating version 0 to 365 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: oid 21 updating version 365 to 371 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: oid 21 version 371 is already newer than 364 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: oid 21 version 371 is already newer than 370 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 371 (ObjNum 236 snap 69 seq_num 236) dirty exists 2024-09-10T16:10:31.413 INFO:tasks.rados.rados.0.smithi165.stdout:584: left oid 21 (ObjNum 236 snap 69 seq_num 236) 2024-09-10T16:10:32.418 INFO:tasks.rados.rados.0.smithi165.stdout:584: done (1 left) 2024-09-10T16:10:32.419 INFO:tasks.rados.rados.0.smithi165.stdout:587: done (0 left) 2024-09-10T16:10:32.419 INFO:tasks.rados.rados.0.smithi165.stdout:588: snap_create 2024-09-10T16:10:33.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:33.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.236+0000 7f6f07dda640 10 monclient: get_monmap_and_config 2024-09-10T16:10:33.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.236+0000 7f6f07dda640 10 monclient: build_initial_monmap 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.236+0000 7f6f07dda640 10 monclient: monmap: 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:33.237074+0000 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:33.237074+0000 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:33.239 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.236+0000 7f6f07dda640 1 Processor -- start 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 1 -- start start 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient: init 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f000a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f000a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f0009ce50 con 0x7f6f000a46a0 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f00138ac0 con 0x7f6f000a4a80 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f0006db60 con 0x7f6f0006cb70 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f07dda640 10 monclient(hunting): authenticate will time out at 1011.724614s 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05375640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6f000a46a0 0x7f6f00137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36808/0 (socket says 172.21.15.164:36808) 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43850/0 (socket says 172.21.15.164:43850) 2024-09-10T16:10:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05375640 1 -- 172.21.15.164:0/100682226 learned_addr learned my addr 172.21.15.164:0/100682226 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): get_auth_request con 0x7f6f0006cb70 auth_method 0 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 10 monclient(hunting): handle_auth_done global_id 4775 payload 274 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.237+0000 7f6f05b76640 1 -- 172.21.15.164:0/100682226 >> v1:172.21.15.165:6789/0 conn(0x7f6f000a4a80 legacy=0x7f6f001383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 1 -- 172.21.15.164:0/100682226 >> v1:172.21.15.164:6789/0 conn(0x7f6f000a46a0 legacy=0x7f6f00137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 10 monclient: _finish_hunting 0 2024-09-10T16:10:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 1 monclient: found mon.noname-b 2024-09-10T16:10:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.238+0000 7f6f05b76640 1 -- 172.21.15.164:0/100682226 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0009cb70 con 0x7f6f0006cb70 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.240+0000 7f6f05b76640 1 -- 172.21.15.164:0/100682226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 msgr2=0x7f6f00134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.240+0000 7f6f05b76640 1 --2- 172.21.15.164:0/100682226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0097d0 tx=0x7f6efc029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=141f9c06fdfb0551 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 1 -- 172.21.15.164:0/100682226 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc037070 con 0x7f6f0006cb70 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: dump: 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:33.252 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: _finish_auth 0 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: _check_auth_tickets 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 1 -- 172.21.15.164:0/100682226 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6efc032440 con 0x7f6f0006cb70 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 5 monclient: authenticate success, global_id 4775 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 1 -- 172.21.15.164:0/100682226 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6efc032810 con 0x7f6f0006cb70 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 4 set_mon_vals no callback set 2024-09-10T16:10:33.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f04b74640 10 monclient: dump: 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 10 monclient: shutdown 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 -- 172.21.15.164:0/100682226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 msgr2=0x7f6f00134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 --2- 172.21.15.164:0/100682226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6efc0097d0 tx=0x7f6efc029fb0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 -- 172.21.15.164:0/100682226 shutdown_connections 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 --2- 172.21.15.164:0/100682226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f0006cb70 0x7f6f00134f70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 -- 172.21.15.164:0/100682226 >> 172.21.15.164:0/100682226 conn(0x7f6f000aa900 msgr2=0x7f6f000aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 -- 172.21.15.164:0/100682226 shutdown_connections 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.241+0000 7f6f07dda640 1 -- 172.21.15.164:0/100682226 wait complete. 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.242+0000 7f6f07dda640 10 monclient: build_initial_monmap 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.242+0000 7f6f07dda640 10 monclient: monmap: 2024-09-10T16:10:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.242+0000 7f6f07dda640 1 Processor -- start 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.242+0000 7f6f07dda640 1 -- start start 2024-09-10T16:10:33.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.242+0000 7f6f07dda640 10 monclient: init 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f0006cb70 0x7f6f001cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-b con 0x7f6f0006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-a con 0x7f6f000a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 0x7f6f00140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): picked mon.noname-c con 0x7f6f000a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0006cf50 con 0x7f6f000a46a0 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:33.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6f00141050 con 0x7f6f0006cb70 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f001411d0 con 0x7f6f000a4a80 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f07dda640 10 monclient(hunting): authenticate will time out at 1011.730129s 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 0x7f6f00140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46020/0 (socket says 172.21.15.164:46020) 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 -- 172.21.15.164:0/606473475 learned_addr learned my addr 172.21.15.164:0/606473475 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 10 monclient(hunting): get_auth_request con 0x7f6f000a46a0 auth_method 0 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:33.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): get_auth_request con 0x7f6f000a4a80 auth_method 0 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 msgr2=0x7f6f0013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 msgr2=0x7f6f0013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:33.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05375640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05b76640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f0006cb70 0x7f6f001cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05b76640 10 monclient(hunting): get_auth_request con 0x7f6f0006cb70 auth_method 0 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05b76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05b76640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f05b76640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient(hunting): handle_auth_done global_id 4778 payload 824 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f0006cb70 msgr2=0x7f6f001cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f0006cb70 0x7f6f001cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 msgr2=0x7f6f0013ffe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient: _finish_hunting 0 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 monclient: found mon.noname-c 2024-09-10T16:10:33.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.243+0000 7f6f06377640 1 -- 172.21.15.164:0/606473475 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f000a4000 con 0x7f6f000a4a80 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f06377640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 msgr2=0x7f6f00140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f06377640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 0x7f6f00140520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c780 tx=0x7f6ef800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8016020 con 0x7f6f000a4a80 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: dump: 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.261 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: _finish_auth 0 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ef800ac60 con 0x7f6f000a4a80 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f07dda640 5 monclient: authenticate success, global_id 4778 2024-09-10T16:10:33.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f07dda640 10 monclient: _renew_subs 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f07dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0006e720 con 0x7f6f000a4a80 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8012660 con 0x7f6f000a4a80 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef27fc640 10 monclient: dump: 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6f07dda640 10 monclient: _renew_subs 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.245+0000 7f6ef3fff640 4 set_mon_vals no callback set 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0013aa70 con 0x7f6f000a4a80 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6ef8019020 con 0x7f6f000a4a80 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.246+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f00134f70 con 0x7f6f000a4a80 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.247+0000 7f6ef27fc640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 0x7f6ee4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.247+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(155..155 src has 1..155) ==== 4835+0+0 (secure 0 0 0) 0x7f6ef8052c70 con 0x7f6f000a4a80 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.247+0000 7f6f05b76640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 0x7f6ee4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.247+0000 7f6f05b76640 10 monclient: get_auth_request con 0x7f6ee403e260 auth_method 0 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.249+0000 7f6f05b76640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 0x7f6ee4040720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6efc034000 tx=0x7f6efc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:33.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.251+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6ef8020030 con 0x7f6f000a4a80 2024-09-10T16:10:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.251+0000 7f6ef27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.251+0000 7f6ef27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:33.417 INFO:tasks.rados.rados.0.smithi165.stdout:588: done (0 left) 2024-09-10T16:10:33.417 INFO:tasks.rados.rados.0.smithi165.stdout:589: delete oid 37 current snap is 71 2024-09-10T16:10:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:589: done (0 left) 2024-09-10T16:10:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:590: write (excl) oid 5 current snap is 71 2024-09-10T16:10:33.421 INFO:tasks.rados.rados.0.smithi165.stdout:590: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-10T16:10:33.428 INFO:tasks.rados.rados.0.smithi165.stdout:590: writing smithi16534841-5 from 416873 to 1208327 tid 1 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:590: writing smithi16534841-5 from 1919314 to 2016873 tid 2 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:591: read oid 29 snap -1 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:591: expect (ObjNum 88 snap 20 seq_num 88) 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:592: read oid 16 snap -1 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:592: expect (ObjNum 208 snap 62 seq_num 208) 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:593: read oid 46 snap -1 2024-09-10T16:10:33.433 INFO:tasks.rados.rados.0.smithi165.stdout:593: expect deleted 2024-09-10T16:10:33.434 INFO:tasks.rados.rados.0.smithi165.stdout:594: snap_remove snap 45 2024-09-10T16:10:33.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6f07dda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:33.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6f07dda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6f07dda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6f00139270 con 0x7f6f000a4a80 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6ef27fc640 1 -- 172.21.15.164:0/606473475 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f6ef800ae10 con 0x7f6f000a4a80 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6ef27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.502+0000 7f6ef27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:33.504 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":40,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:33.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 msgr2=0x7f6ee4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:33.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 0x7f6ee4040720 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f6efc034000 tx=0x7f6efc02e000 comp rx=0 tx=0).stop 2024-09-10T16:10:33.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 10 monclient: shutdown 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 msgr2=0x7f6f00140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 0x7f6f00140520 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6ef800c780 tx=0x7f6ef800cc40 comp rx=0 tx=0).stop 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 shutdown_connections 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e260 0x7f6ee4040720 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.504+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f000a4a80 0x7f6f00140520 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.505+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f000a46a0 0x7f6f0013ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.505+0000 7f6f07dda640 1 --2- 172.21.15.164:0/606473475 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f0006cb70 0x7f6f001cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.505+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 >> 172.21.15.164:0/606473475 conn(0x7f6f000aa900 msgr2=0x7f6f00137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.505+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 shutdown_connections 2024-09-10T16:10:33.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:33.505+0000 7f6f07dda640 1 -- 172.21.15.164:0/606473475 wait complete. 2024-09-10T16:10:34.430 INFO:tasks.rados.rados.0.smithi165.stdout:594: done (4 left) 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:595: copy_from oid 27 from oid 39 current snap is 71 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: finishing write tid 1 to smithi16534841-5 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: finishing write tid 2 to smithi16534841-5 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: finishing write tid 3 to smithi16534841-5 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: finishing write tid 4 to smithi16534841-5 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: oid 5 updating version 0 to 246 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: oid 5 version 246 is already newer than 245 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: oid 5 updating version 246 to 247 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 247 (ObjNum 239 snap 71 seq_num 239) dirty exists 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:590: left oid 5 (ObjNum 239 snap 71 seq_num 239) 2024-09-10T16:10:34.431 INFO:tasks.rados.rados.0.smithi165.stdout:591: expect (ObjNum 88 snap 20 seq_num 88) 2024-09-10T16:10:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1342126 2024-09-10T16:10:34.516 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3112937 2024-09-10T16:10:34.520 INFO:tasks.rados.rados.0.smithi165.stdout:592: expect (ObjNum 208 snap 62 seq_num 208) 2024-09-10T16:10:34.520 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:34.574 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:595: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:10:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:595: finishing copy_from to smithi16534841-27 2024-09-10T16:10:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 375 (ObjNum 235 snap 69 seq_num 235) dirty exists 2024-09-10T16:10:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:590: done (4 left) 2024-09-10T16:10:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:591: done (3 left) 2024-09-10T16:10:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:592: done (2 left) 2024-09-10T16:10:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:593: done (1 left) 2024-09-10T16:10:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:595: done (0 left) 2024-09-10T16:10:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:596: snap_create 2024-09-10T16:10:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:596: done (0 left) 2024-09-10T16:10:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:597: read oid 43 snap -1 2024-09-10T16:10:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:597: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:10:34.866 INFO:tasks.rados.rados.0.smithi165.stdout:598: rollback oid 9 current snap is 72 2024-09-10T16:10:34.866 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 44 2024-09-10T16:10:34.866 INFO:tasks.rados.rados.0.smithi165.stdout:599: write (excl) oid 27 current snap is 72 2024-09-10T16:10:34.866 INFO:tasks.rados.rados.0.smithi165.stdout:599: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-10T16:10:34.870 INFO:tasks.rados.rados.0.smithi165.stdout:599: writing smithi16534841-27 from 433680 to 900383 tid 1 2024-09-10T16:10:34.877 INFO:tasks.rados.rados.0.smithi165.stdout:599: writing smithi16534841-27 from 1577796 to 2033680 tid 2 2024-09-10T16:10:34.877 INFO:tasks.rados.rados.0.smithi165.stdout:600: write oid 21 current snap is 72 2024-09-10T16:10:34.877 INFO:tasks.rados.rados.0.smithi165.stdout:598: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:10:34.877 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 0 (ObjNum 129 snap 0 seq_num 1162980328) dirty dne 2024-09-10T16:10:34.877 INFO:tasks.rados.rados.0.smithi165.stdout:600: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-10T16:10:34.882 INFO:tasks.rados.rados.0.smithi165.stdout:600: writing smithi16534841-21 from 450487 to 992439 tid 1 2024-09-10T16:10:34.888 INFO:tasks.rados.rados.0.smithi165.stdout:600: writing smithi16534841-21 from 1636278 to 2050487 tid 2 2024-09-10T16:10:34.889 INFO:tasks.rados.rados.0.smithi165.stdout:598: done (3 left) 2024-09-10T16:10:34.889 INFO:tasks.rados.rados.0.smithi165.stdout:601: rollback oid 31 current snap is 72 2024-09-10T16:10:34.889 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 67 2024-09-10T16:10:34.889 INFO:tasks.rados.rados.0.smithi165.stdout:597: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:10:35.033 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1532468 2024-09-10T16:10:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:599: finishing write tid 1 to smithi16534841-27 2024-09-10T16:10:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:599: finishing write tid 2 to smithi16534841-27 2024-09-10T16:10:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:597: done (3 left) 2024-09-10T16:10:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:602: rollback oid 16 current snap is 72 2024-09-10T16:10:35.095 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 47 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: finishing write tid 3 to smithi16534841-27 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: finishing write tid 4 to smithi16534841-27 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: oid 27 updating version 0 to 416 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: oid 27 version 416 is already newer than 414 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: oid 27 version 416 is already newer than 415 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 416 (ObjNum 240 snap 72 seq_num 240) dirty exists 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:599: left oid 27 (ObjNum 240 snap 72 seq_num 240) 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:600: finishing write tid 1 to smithi16534841-21 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:601: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 319 (ObjNum 193 snap 0 seq_num 3191136554) dirty dne 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:600: finishing write tid 2 to smithi16534841-21 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:600: finishing write tid 3 to smithi16534841-21 2024-09-10T16:10:35.096 INFO:tasks.rados.rados.0.smithi165.stdout:600: finishing write tid 4 to smithi16534841-21 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:600: oid 21 updating version 0 to 389 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:600: oid 21 updating version 389 to 391 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:600: oid 21 version 391 is already newer than 390 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 391 (ObjNum 241 snap 72 seq_num 241) dirty exists 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:600: left oid 21 (ObjNum 241 snap 72 seq_num 241) 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:599: done (3 left) 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:600: done (2 left) 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:601: done (1 left) 2024-09-10T16:10:35.097 INFO:tasks.rados.rados.0.smithi165.stdout:603: snap_remove snap 43 2024-09-10T16:10:35.875 INFO:tasks.rados.rados.0.smithi165.stdout:603: done (1 left) 2024-09-10T16:10:35.875 INFO:tasks.rados.rados.0.smithi165.stdout:604: read oid 35 snap -1 2024-09-10T16:10:35.875 INFO:tasks.rados.rados.0.smithi165.stdout:604: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-10T16:10:35.875 INFO:tasks.rados.rados.0.smithi165.stdout:605: delete oid 22 current snap is 72 2024-09-10T16:10:35.876 INFO:tasks.rados.rados.0.smithi165.stdout:602: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:10:35.876 INFO:tasks.rados.rados.0.smithi165.stdout:602: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:10:35.876 INFO:tasks.rados.rados.0.smithi165.stdout:602: finishing rollback tid 2 to smithi16534841-16 2024-09-10T16:10:35.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 519 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-10T16:10:35.890 INFO:tasks.rados.rados.0.smithi165.stdout:604: expect (ObjNum 93 snap 23 seq_num 93) 2024-09-10T16:10:35.890 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:602: done (2 left) 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:604: done (1 left) 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:605: done (0 left) 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:606: read oid 32 snap -1 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:606: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:10:36.024 INFO:tasks.rados.rados.0.smithi165.stdout:607: delete oid 46 current snap is 72 2024-09-10T16:10:36.028 INFO:tasks.rados.rados.0.smithi165.stdout:607: done (1 left) 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:608: copy_from oid 39 from oid 37 current snap is 72 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:609: read oid 14 snap -1 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:609: expect deleted 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:610: read oid 41 snap 70 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:610: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:611: write (excl) oid 12 current snap is 72 2024-09-10T16:10:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:611: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-10T16:10:36.034 INFO:tasks.rados.rados.0.smithi165.stdout:611: writing smithi16534841-12 from 500908 to 984960 tid 1 2024-09-10T16:10:36.041 INFO:tasks.rados.rados.0.smithi165.stdout:611: writing smithi16534841-12 from 1411724 to 2100908 tid 2 2024-09-10T16:10:36.041 INFO:tasks.rados.rados.0.smithi165.stdout:612: rollback oid 26 current snap is 72 2024-09-10T16:10:36.041 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 67 2024-09-10T16:10:36.041 INFO:tasks.rados.rados.0.smithi165.stdout:606: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:10:36.173 INFO:tasks.rados.rados.0.smithi165.stdout:610: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:10:36.270 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1788618 2024-09-10T16:10:36.304 INFO:tasks.rados.rados.0.smithi165.stdout:608: finishing copy_from to smithi16534841-39 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:608: got expected ENOENT (src dne) 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:608: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: finishing write tid 1 to smithi16534841-12 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:612: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 282 (ObjNum 129 snap 0 seq_num 1162570768) dirty dne 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: finishing write tid 2 to smithi16534841-12 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: finishing write tid 3 to smithi16534841-12 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: finishing write tid 4 to smithi16534841-12 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: oid 12 updating version 0 to 54 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: oid 12 version 54 is already newer than 52 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: oid 12 version 54 is already newer than 53 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 54 (ObjNum 244 snap 72 seq_num 244) dirty exists 2024-09-10T16:10:36.305 INFO:tasks.rados.rados.0.smithi165.stdout:611: left oid 12 (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:606: done (5 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:608: done (4 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:609: done (3 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:610: done (2 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:611: done (1 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:612: done (0 left) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:613: read oid 42 snap 63 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:613: expect (ObjNum 171 snap 52 seq_num 171) 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:614: rollback oid 21 current snap is 72 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 48 2024-09-10T16:10:36.306 INFO:tasks.rados.rados.0.smithi165.stdout:615: snap_create 2024-09-10T16:10:36.307 INFO:tasks.rados.rados.0.smithi165.stdout:614: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:10:36.307 INFO:tasks.rados.rados.0.smithi165.stdout:614: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:10:36.307 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 401 (ObjNum 209 snap 0 seq_num 326751611) dirty dne 2024-09-10T16:10:36.311 INFO:tasks.rados.rados.0.smithi165.stdout:613: expect (ObjNum 171 snap 52 seq_num 171) 2024-09-10T16:10:36.369 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:36.439 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:36.518 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:36.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.625+0000 7f726b305640 10 monclient: get_monmap_and_config 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.625+0000 7f726b305640 10 monclient: build_initial_monmap 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.625+0000 7f726b305640 10 monclient: monmap: 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:36.626214+0000 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:36.626214+0000 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:36.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:36.628 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:36.628 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:36.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:36.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.625+0000 7f726b305640 1 Processor -- start 2024-09-10T16:10:36.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 1 -- start start 2024-09-10T16:10:36.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient: init 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7264074af0 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72640a1090 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): picked mon.noname-c con 0x7f72640a1500 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f726409ebc0 con 0x7f72640a1090 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7264138580 con 0x7f7264074af0 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f72640698a0 con 0x7f72640a1500 2024-09-10T16:10:36.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f726b305640 10 monclient(hunting): authenticate will time out at 1015.113656s 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72688a0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f72640a1090 0x7f726413d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:37474/0 (socket says 172.21.15.164:37474) 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 learned_addr learned my addr 172.21.15.164:0/2818686787 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f72640a1500 auth_method 0 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.626+0000 7f72698a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.627+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.627+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.627+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:36.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.627+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.628+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.164:6789/0 conn(0x7f72640a1090 legacy=0x7f726413d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 legacy=0x7f7264137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 legacy=0x7f7264137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 legacy=0x7f7264137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f72690a1640 1 --1- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 0x7f7264137e70 :-1 s=OPENED pgs=50 cs=1 l=1).handle_message read tag failed 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f72690a1640 1 --1- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 0x7f7264137e70 :-1 s=OPENED pgs=50 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:10:36.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.629+0000 7f725bfff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:10:36.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:36.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f72640a1500 auth_method 0 2024-09-10T16:10:36.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:36.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:36.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:36.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:36.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:36.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:36.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:36.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f72640a1500 auth_method 0 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:37.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:37.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:37.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f72640a1500 auth_method 0 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:38.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:38.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): tick 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 msgr2=0x7f72641408c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:39.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f72640a1500 0x7f72641408c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7264074af0 legacy=0x7f7264137e70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.164:6789/0 conn(0x7f72640a1090 legacy=0x7f726413d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient: _add_conns ranks=[3,2] 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): picked mon.noname-a con 0x7f7248003690 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): picked mon.noname-b con 0x7f72480061d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7248008d90 con 0x7f7248003690 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7248008f40 con 0x7f72480061d0 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f725affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:39.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7248003690 auth_method 0 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480061d0 auth_method 0 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:39.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:39.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:39.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:39.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480061d0 auth_method 0 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7248003690 auth_method 0 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:39.828+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7248003690 auth_method 0 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480061d0 auth_method 0 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:40.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:40.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:40.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480061d0 auth_method 0 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7248003690 auth_method 0 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:41.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:41.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 10 monclient(hunting): tick 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 msgr2=0x7f7248008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.626+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72480061d0 0x7f7248008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 msgr2=0x7f7248005af0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248003690 0x7f7248005af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:10:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 reap_dead start 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72640a1090 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): picked mon.noname-c con 0x7f7248003ae0 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): picked mon.noname-b con 0x7f7248006bc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72640a1950 con 0x7f72640a1090 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7248008d90 con 0x7f7248003ae0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7248008f40 con 0x7f7248006bc0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f725affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248006bc0 auth_method 0 2024-09-10T16:10:42.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248003ae0 auth_method 0 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:42.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:42.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:42.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.627+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:42.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.629+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.164:6789/0 conn(0x7f72640a1090 legacy=0x7f724800b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:42.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:42.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248006bc0 auth_method 0 2024-09-10T16:10:42.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:42.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248003ae0 auth_method 0 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:42.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:42.828+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:43.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:43.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248006bc0 auth_method 0 2024-09-10T16:10:43.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248003ae0 auth_method 0 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:43.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:43.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:43.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:43.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:43.229+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248006bc0 auth_method 0 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:44.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248003ae0 auth_method 0 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:44.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:44.030+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): tick 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 msgr2=0x7f7248007040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7248006bc0 0x7f7248007040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 msgr2=0x7f7248006680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248003ae0 0x7f7248006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.164:6789/0 conn(0x7f72640a1090 legacy=0x7f724800b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient: _add_conns ranks=[1,3] 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f72480075b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:45.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): picked mon.noname-a con 0x7f72480081d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7248008700 con 0x7f72480075b0 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7248008d90 con 0x7f72480081d0 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f725affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480081d0 auth_method 0 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:45.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:45.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:45.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:45.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.630+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f72480075b0 legacy=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:45.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480081d0 auth_method 0 2024-09-10T16:10:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:45.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:45.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:45.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:46.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:46.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480081d0 auth_method 0 2024-09-10T16:10:46.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:46.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:46.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:46.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:46.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:47.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:47.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72480081d0 auth_method 0 2024-09-10T16:10:47.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:47.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:47.032+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:48.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): tick 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 msgr2=0x7f7248013b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f72480081d0 0x7f7248013b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f72480075b0 legacy=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 reap_dead start 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7248006680 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:48.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): picked mon.noname-c con 0x7f7248007040 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): picked mon.noname-a con 0x7f7248014a80 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7248015230 con 0x7f7248006680 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7248008d90 con 0x7f7248014a80 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 1 -- 172.21.15.164:0/2818686787 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7248008f40 con 0x7f7248007040 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f725affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:48.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248014a80 auth_method 0 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248007040 auth_method 0 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:48.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:48.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:48.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:48.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.628+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:48.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.630+0000 7f72690a1640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7248006680 legacy=0x7f7248014370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248014a80 auth_method 0 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:48.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248007040 auth_method 0 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:48.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:48.829+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248014a80 auth_method 0 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:49.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248007040 auth_method 0 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:49.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:49.230+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:10:50.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:50.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7248014a80 auth_method 0 2024-09-10T16:10:50.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:50.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:50.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7248007040 auth_method 0 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:50.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:50.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:50.031+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:10:51.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 798446111 0 0) 0x7f7248015230 con 0x7f7248006680 2024-09-10T16:10:51.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 10 monclient(hunting): _negotiate 2024-09-10T16:10:51.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:51.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 10 monclient(hunting): my global_id is 44173 2024-09-10T16:10:51.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.076+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7250003520 con 0x7f7248006680 2024-09-10T16:10:51.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 83835034 0 0) 0x7f7250003520 con 0x7f7248006680 2024-09-10T16:10:51.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 msgr2=0x7f72480079c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:51.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.165:3300/0 conn(0x7f7248007040 0x7f72480079c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 msgr2=0x7f7248012b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 --2- 172.21.15.164:0/2818686787 >> v2:172.21.15.164:3300/0 conn(0x7f7248014a80 0x7f7248012b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7248007f00 con 0x7f7248006680 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _finish_auth 0 2024-09-10T16:10:51.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f7260001ce0 con 0x7f7248006680 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f726b305640 5 monclient: authenticate success, global_id 44173 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f726b305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: dump: 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.085 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:51.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 >> v1:172.21.15.165:6789/0 conn(0x7f7248006680 legacy=0x7f7248014370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250003a20 0x7f7250003e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): picked mon.a con 0x7f7250003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7250004800 0x7f7250004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): picked mon.b con 0x7f7250004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 0x7f725000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): picked mon.c con 0x7f7250005210 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7248008d90 con 0x7f7250003a20 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7248008f40 con 0x7f7250004800 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f725000c1c0 con 0x7f7250005210 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f725bfff640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f726b305640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250003a20 0x7f7250003e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7250003a20 auth_method 0 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 0x7f725000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f7250005210 auth_method 0 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7250004800 0x7f7250004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f7250004800 auth_method 0 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72690a1640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:51.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient(hunting): handle_auth_done global_id 44173 payload 293 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7250004800 msgr2=0x7f7250004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7250004800 0x7f7250004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250003a20 msgr2=0x7f7250003e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250003a20 0x7f7250003e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient: _finish_hunting 0 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 monclient: found mon.c 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.079+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7250003520 con 0x7f7250005210 2024-09-10T16:10:51.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.080+0000 7f72698a2640 10 monclient: _finish_auth 0 2024-09-10T16:10:51.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.080+0000 7f72698a2640 10 monclient: _check_auth_tickets 2024-09-10T16:10:51.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.080+0000 7f72698a2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:51.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.080+0000 7f72690a1640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250003a20 0x7f7250003e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:51.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.080+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 reap_dead start 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.082+0000 7f72698a2640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 msgr2=0x7f725000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.082+0000 7f72698a2640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 0x7f725000b9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72640a18e0 tx=0x7f7250005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed4457ec0ec2f3bb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c03a070 con 0x7f7250005210 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: dump: 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:51.099 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c0353b0 con 0x7f7250005210 2024-09-10T16:10:51.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 1 -- 172.21.15.164:0/2818686787 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7260005be0 con 0x7f7250005210 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 4 set_mon_vals no callback set 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f725bfff640 10 monclient: dump: 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:51.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 10 monclient: shutdown 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 1 -- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 msgr2=0x7f725000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:51.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 0x7f725000b9c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f72640a18e0 tx=0x7f7250005830 comp rx=0 tx=0).stop 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 1 -- 172.21.15.164:0/2818686787 shutdown_connections 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 1 --2- 172.21.15.164:0/2818686787 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7250005210 0x7f725000b9c0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.083+0000 7f726b305640 1 -- 172.21.15.164:0/2818686787 >> 172.21.15.164:0/2818686787 conn(0x7f72640a3fb0 msgr2=0x7f72640a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.084+0000 7f726b305640 1 -- 172.21.15.164:0/2818686787 shutdown_connections 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.084+0000 7f726b305640 1 -- 172.21.15.164:0/2818686787 wait complete. 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.084+0000 7f726b305640 10 monclient: build_initial_monmap 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.084+0000 7f726b305640 10 monclient: monmap: 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:51.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 1 Processor -- start 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 1 -- start start 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 10 monclient: init 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:10:51.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 10 monclient(hunting): picked mon.noname-a con 0x7f7250005210 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72641cd510 0x7f726412c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.085+0000 7f726b305640 10 monclient(hunting): picked mon.noname-b con 0x7f72641cd510 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 0x7f726412cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): picked mon.noname-c con 0x7f72641cd970 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7264138200 con 0x7f7250005210 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7264138080 con 0x7f72641cd510 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7264138380 con 0x7f72641cd970 2024-09-10T16:10:51.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f726b305640 10 monclient(hunting): authenticate will time out at 1029.573063s 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:50024/0 (socket says 172.21.15.164:50024) 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 -- 172.21.15.164:0/1234027823 learned_addr learned my addr 172.21.15.164:0/1234027823 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 10 monclient(hunting): get_auth_request con 0x7f7250005210 auth_method 0 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 0x7f726412cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): get_auth_request con 0x7f72641cd970 auth_method 0 2024-09-10T16:10:51.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72641cd510 0x7f726412c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 msgr2=0x7f72641ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 msgr2=0x7f72641ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): get_auth_request con 0x7f72641cd510 auth_method 0 2024-09-10T16:10:51.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72690a1640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72688a0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient(hunting): handle_auth_done global_id 4781 payload 824 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72641cd510 msgr2=0x7f726412c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72641cd510 0x7f726412c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 msgr2=0x7f72641ccfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:51.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient: _finish_hunting 0 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 monclient: found mon.noname-c 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.086+0000 7f72698a2640 1 -- 172.21.15.164:0/1234027823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72640a6c00 con 0x7f72641cd970 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.088+0000 7f72698a2640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 msgr2=0x7f726412cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.088+0000 7f72698a2640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 0x7f726412cd10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f72480065b0 tx=0x7f725c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c03a040 con 0x7f72641cd970 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:51.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: dump: 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:51.110 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: _finish_auth 0 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 5 monclient: authenticate success, global_id 4781 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f725c046070 con 0x7f72641cd970 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 10 monclient: _renew_subs 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726409e6a0 con 0x7f72641cd970 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c032070 con 0x7f72641cd970 2024-09-10T16:10:51.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f7259ffb640 10 monclient: dump: 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:51.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 10 monclient: _renew_subs 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f725b7fe640 4 set_mon_vals no callback set 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.089+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f726412f880 con 0x7f72641cd970 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f725c015040 con 0x7f72641cd970 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f726b305640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f726b305640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:51.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f726b305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726412dfd0 con 0x7f72641cd970 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f7259ffb640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 0x7f7248042f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.090+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(159..159 src has 1..159) ==== 4799+0+0 (secure 0 0 0) 0x7f725c074970 con 0x7f72641cd970 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.091+0000 7f72690a1640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 0x7f7248042f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.091+0000 7f72690a1640 10 monclient: get_auth_request con 0x7f7248040aa0 auth_method 0 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.093+0000 7f72690a1640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 0x7f7248042f60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7260001900 tx=0x7f7260009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.095+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f726412dfd0 con 0x7f72641cd970 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.095+0000 7f7259ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:51.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.095+0000 7f7259ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:51.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f726b305640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:51.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f726b305640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:51.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f726b305640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:51.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f72640a2ec0 con 0x7f72641cd970 2024-09-10T16:10:51.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f7259ffb640 1 -- 172.21.15.164:0/1234027823 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f725c010040 con 0x7f72641cd970 2024-09-10T16:10:51.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f7259ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:51.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.341+0000 7f7259ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:51.344 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":50,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 msgr2=0x7f7248042f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 0x7f7248042f60 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7260001900 tx=0x7f7260009040 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 10 monclient: shutdown 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 msgr2=0x7f726412cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 0x7f726412cd10 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f72480065b0 tx=0x7f725c004a40 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 shutdown_connections 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7248040aa0 0x7f7248042f60 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72641cd970 0x7f726412cd10 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72641cd510 0x7f726412c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 --2- 172.21.15.164:0/1234027823 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7250005210 0x7f72641ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 >> 172.21.15.164:0/1234027823 conn(0x7f72640a3fb0 msgr2=0x7f72641348f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:51.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 shutdown_connections 2024-09-10T16:10:51.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:51.344+0000 7f726b305640 1 -- 172.21.15.164:0/1234027823 wait complete. 2024-09-10T16:10:52.082 INFO:tasks.rados.rados.0.smithi165.stdout:613: done (2 left) 2024-09-10T16:10:52.083 INFO:tasks.rados.rados.0.smithi165.stdout:614: done (1 left) 2024-09-10T16:10:52.083 INFO:tasks.rados.rados.0.smithi165.stdout:615: done (0 left) 2024-09-10T16:10:52.083 INFO:tasks.rados.rados.0.smithi165.stdout:616: snap_remove snap 67 2024-09-10T16:10:53.084 INFO:tasks.rados.rados.0.smithi165.stdout:616: done (0 left) 2024-09-10T16:10:53.084 INFO:tasks.rados.rados.0.smithi165.stdout:617: write oid 42 current snap is 73 2024-09-10T16:10:53.084 INFO:tasks.rados.rados.0.smithi165.stdout:617: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-10T16:10:53.090 INFO:tasks.rados.rados.0.smithi165.stdout:617: writing smithi16534841-42 from 517715 to 1077016 tid 1 2024-09-10T16:10:53.097 INFO:tasks.rados.rados.0.smithi165.stdout:617: writing smithi16534841-42 from 1870206 to 2117715 tid 2 2024-09-10T16:10:53.097 INFO:tasks.rados.rados.0.smithi165.stdout:618: write oid 19 current snap is 73 2024-09-10T16:10:53.097 INFO:tasks.rados.rados.0.smithi165.stdout:618: seq_num 246 ranges {534522=634550,1812335=322187} 2024-09-10T16:10:53.103 INFO:tasks.rados.rados.0.smithi165.stdout:618: writing smithi16534841-19 from 534522 to 1169072 tid 1 2024-09-10T16:10:53.109 INFO:tasks.rados.rados.0.smithi165.stdout:618: writing smithi16534841-19 from 1812335 to 2134522 tid 2 2024-09-10T16:10:53.109 INFO:tasks.rados.rados.0.smithi165.stdout:619: write oid 15 current snap is 73 2024-09-10T16:10:53.109 INFO:tasks.rados.rados.0.smithi165.stdout:617: finishing write tid 1 to smithi16534841-42 2024-09-10T16:10:53.109 INFO:tasks.rados.rados.0.smithi165.stdout:619: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-10T16:10:53.116 INFO:tasks.rados.rados.0.smithi165.stdout:619: writing smithi16534841-15 from 551329 to 1261128 tid 1 2024-09-10T16:10:53.121 INFO:tasks.rados.rados.0.smithi165.stdout:619: writing smithi16534841-15 from 1870817 to 2151329 tid 2 2024-09-10T16:10:53.121 INFO:tasks.rados.rados.0.smithi165.stdout:620: snap_remove snap 70 2024-09-10T16:10:54.093 INFO:tasks.rados.rados.0.smithi165.stdout:620: done (3 left) 2024-09-10T16:10:54.093 INFO:tasks.rados.rados.0.smithi165.stdout:621: rollback oid 35 current snap is 73 2024-09-10T16:10:54.093 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 68 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: finishing write tid 2 to smithi16534841-42 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: finishing write tid 3 to smithi16534841-42 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: finishing write tid 4 to smithi16534841-42 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: oid 42 updating version 0 to 241 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: oid 42 updating version 241 to 242 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: oid 42 version 242 is already newer than 240 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 242 (ObjNum 245 snap 73 seq_num 245) dirty exists 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:617: left oid 42 (ObjNum 245 snap 73 seq_num 245) 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:618: finishing write tid 1 to smithi16534841-19 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:618: finishing write tid 2 to smithi16534841-19 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:618: finishing write tid 3 to smithi16534841-19 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:618: finishing write tid 4 to smithi16534841-19 2024-09-10T16:10:54.094 INFO:tasks.rados.rados.0.smithi165.stdout:618: oid 19 updating version 0 to 540 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:618: oid 19 updating version 540 to 542 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:618: oid 19 version 542 is already newer than 541 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 542 (ObjNum 246 snap 73 seq_num 246) dirty exists 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:618: left oid 19 (ObjNum 246 snap 73 seq_num 246) 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: finishing write tid 1 to smithi16534841-15 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: finishing write tid 2 to smithi16534841-15 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: finishing write tid 3 to smithi16534841-15 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: finishing write tid 4 to smithi16534841-15 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: oid 15 updating version 0 to 268 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: oid 15 updating version 268 to 270 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: oid 15 version 270 is already newer than 269 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 270 (ObjNum 247 snap 73 seq_num 247) dirty exists 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:619: left oid 15 (ObjNum 247 snap 73 seq_num 247) 2024-09-10T16:10:54.095 INFO:tasks.rados.rados.0.smithi165.stdout:617: done (3 left) 2024-09-10T16:10:54.096 INFO:tasks.rados.rados.0.smithi165.stdout:618: done (2 left) 2024-09-10T16:10:54.096 INFO:tasks.rados.rados.0.smithi165.stdout:619: done (1 left) 2024-09-10T16:10:54.096 INFO:tasks.rados.rados.0.smithi165.stdout:622: snap_remove snap 35 2024-09-10T16:10:54.356 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.464+0000 7efc2397d640 10 monclient: get_monmap_and_config 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.464+0000 7efc2397d640 10 monclient: build_initial_monmap 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.464+0000 7efc2397d640 10 monclient: monmap: 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:54.465631+0000 2024-09-10T16:10:54.466 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:54.465631+0000 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:54.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 1 Processor -- start 2024-09-10T16:10:54.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 1 -- start start 2024-09-10T16:10:54.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 10 monclient: init 2024-09-10T16:10:54.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:54.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.465+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-a con 0x7efc1c09c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc1c09d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 0x7efc1c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-b con 0x7efc1c13a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc1c09a430 con 0x7efc1c09d160 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7efc1c0a24e0 con 0x7efc1c09c180 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efc1c0a2bf0 con 0x7efc1c13a5c0 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:54.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc2397d640 10 monclient(hunting): authenticate will time out at 1032.953053s 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 0x7efc1c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38224/0 (socket says 172.21.15.164:38224) 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 -- 172.21.15.164:0/3544025652 learned_addr learned my addr 172.21.15.164:0/3544025652 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 10 monclient(hunting): get_auth_request con 0x7efc1c09c180 auth_method 0 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): get_auth_request con 0x7efc1c13a5c0 auth_method 0 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:54.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 -- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 msgr2=0x7efc1c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 -- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 msgr2=0x7efc1c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 --2- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21719640 1 --2- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient(hunting): handle_auth_done global_id 4784 payload 274 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 -- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 msgr2=0x7efc1c09ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 --2- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 -- 172.21.15.164:0/3544025652 >> v1:172.21.15.165:6789/0 conn(0x7efc1c09d160 legacy=0x7efc1c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:54.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient: _finish_hunting 0 2024-09-10T16:10:54.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 monclient: found mon.noname-b 2024-09-10T16:10:54.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:54.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.466+0000 7efc21f1a640 1 -- 172.21.15.164:0/3544025652 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1c09c560 con 0x7efc1c13a5c0 2024-09-10T16:10:54.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc21f1a640 1 -- 172.21.15.164:0/3544025652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 msgr2=0x7efc1c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc21f1a640 1 --2- 172.21.15.164:0/3544025652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 0x7efc1c13c9b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc14009dc0 tx=0x7efc1402a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62fb9333590d5384 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 1 -- 172.21.15.164:0/3544025652 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc14007950 con 0x7efc1c13a5c0 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: dump: 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:54.479 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: _finish_auth 0 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: _check_auth_tickets 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 5 monclient: authenticate success, global_id 4784 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 1 -- 172.21.15.164:0/3544025652 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc14007af0 con 0x7efc1c13a5c0 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:54.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 1 -- 172.21.15.164:0/3544025652 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc14031400 con 0x7efc1c13a5c0 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 4 set_mon_vals no callback set 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc13fff640 10 monclient: dump: 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:54.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 10 monclient: shutdown 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 1 -- 172.21.15.164:0/3544025652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 msgr2=0x7efc1c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.469+0000 7efc2397d640 1 --2- 172.21.15.164:0/3544025652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 0x7efc1c13c9b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efc14009dc0 tx=0x7efc1402a4d0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 -- 172.21.15.164:0/3544025652 shutdown_connections 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 --2- 172.21.15.164:0/3544025652 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c13a5c0 0x7efc1c13c9b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 --2- 172.21.15.164:0/3544025652 >> v2:172.21.15.164:3300/0 conn(0x7efc1c09c180 0x7efc1c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 -- 172.21.15.164:0/3544025652 >> 172.21.15.164:0/3544025652 conn(0x7efc1c0a3110 msgr2=0x7efc1c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 -- 172.21.15.164:0/3544025652 shutdown_connections 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 1 -- 172.21.15.164:0/3544025652 wait complete. 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 10 monclient: build_initial_monmap 2024-09-10T16:10:54.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.470+0000 7efc2397d640 10 monclient: monmap: 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 Processor -- start 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 -- start start 2024-09-10T16:10:54.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient: init 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc1c09c180 0x7efc1c0995e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-b con 0x7efc1c09c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-c con 0x7efc1c09d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): picked mon.noname-a con 0x7efc1c13a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c074f90 con 0x7efc1c13a5c0 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efc1c074e10 con 0x7efc1c09c180 2024-09-10T16:10:54.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efc1c09f690 con 0x7efc1c09d160 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc2397d640 10 monclient(hunting): authenticate will time out at 1032.958594s 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38238/0 (socket says 172.21.15.164:38238) 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40058/0 (socket says 172.21.15.164:40058) 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 -- 172.21.15.164:0/3392291537 learned_addr learned my addr 172.21.15.164:0/3392291537 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 10 monclient(hunting): get_auth_request con 0x7efc1c13a5c0 auth_method 0 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:54.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): get_auth_request con 0x7efc1c09d160 auth_method 0 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 msgr2=0x7efc1c08e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 msgr2=0x7efc1c08e770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21f1a640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21719640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc1c09c180 0x7efc1c0995e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21719640 10 monclient(hunting): get_auth_request con 0x7efc1c09c180 auth_method 0 2024-09-10T16:10:54.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21719640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21719640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc21719640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 10 monclient(hunting): handle_auth_done global_id 4787 payload 824 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc1c09c180 msgr2=0x7efc1c0995e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc1c09c180 0x7efc1c0995e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 msgr2=0x7efc1c08e770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.471+0000 7efc20f18640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.472+0000 7efc20f18640 10 monclient: _finish_hunting 0 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.472+0000 7efc20f18640 1 monclient: found mon.noname-c 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.472+0000 7efc20f18640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.472+0000 7efc20f18640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.472+0000 7efc20f18640 1 -- 172.21.15.164:0/3392291537 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc1c091e40 con 0x7efc1c09d160 2024-09-10T16:10:54.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc20f18640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 msgr2=0x7efc1c08e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc20f18640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00c3e0 tx=0x7efc0c00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c0119a0 con 0x7efc1c09d160 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: dump: 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:54.488 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: _finish_auth 0 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c017070 con 0x7efc1c09d160 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 5 monclient: authenticate success, global_id 4787 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 10 monclient: _renew_subs 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc1c090620 con 0x7efc1c09d160 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c011d80 con 0x7efc1c09d160 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:54.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc11ffb640 10 monclient: dump: 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 10 monclient: _renew_subs 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc137fe640 4 set_mon_vals no callback set 2024-09-10T16:10:54.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.474+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc1c091860 con 0x7efc1c09d160 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc2397d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc2397d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc2397d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc1c09d930 con 0x7efc1c09d160 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efc0c01c460 con 0x7efc1c09d160 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc11ffb640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 0x7efc00040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.475+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(162..162 src has 1..162) ==== 4831+0+0 (secure 0 0 0) 0x7efc0c052b80 con 0x7efc1c09d160 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.476+0000 7efc21719640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 0x7efc00040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:54.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.476+0000 7efc21719640 10 monclient: get_auth_request con 0x7efc0003e270 auth_method 0 2024-09-10T16:10:54.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.478+0000 7efc21719640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 0x7efc00040730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc18006a90 tx=0x7efc18006a20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:54.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.480+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efc0c051550 con 0x7efc1c09d160 2024-09-10T16:10:54.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.480+0000 7efc11ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:54.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.480+0000 7efc11ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:54.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.724+0000 7efc2397d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:54.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.724+0000 7efc2397d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:54.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.724+0000 7efc2397d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:54.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.724+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efc1c09d5e0 con 0x7efc1c09d160 2024-09-10T16:10:54.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.725+0000 7efc11ffb640 1 -- 172.21.15.164:0/3392291537 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7efc0c012420 con 0x7efc1c09d160 2024-09-10T16:10:54.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.725+0000 7efc11ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:54.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.725+0000 7efc11ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:54.727 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":50,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 msgr2=0x7efc00040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 0x7efc00040730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efc18006a90 tx=0x7efc18006a20 comp rx=0 tx=0).stop 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 10 monclient: shutdown 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 msgr2=0x7efc1c08e230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efc0c00c3e0 tx=0x7efc0c00c8a0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 shutdown_connections 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0003e270 0x7efc00040730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc1c13a5c0 0x7efc1c08e770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc1c09d160 0x7efc1c08e230 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 --2- 172.21.15.164:0/3392291537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc1c09c180 0x7efc1c0995e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:54.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 >> 172.21.15.164:0/3392291537 conn(0x7efc1c0a3110 msgr2=0x7efc1c09db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:54.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 shutdown_connections 2024-09-10T16:10:54.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:54.727+0000 7efc2397d640 1 -- 172.21.15.164:0/3392291537 wait complete. 2024-09-10T16:10:54.867 INFO:tasks.rados.rados.0.smithi165.stdout:622: done (1 left) 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:623: read oid 39 snap -1 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:623: expect (ObjNum 235 snap 69 seq_num 235) 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:624: rollback oid 38 current snap is 73 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 47 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:621: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:621: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:621: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 188 (ObjNum 93 snap 23 seq_num 93) dirty exists 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:621: done (2 left) 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:625: write (excl) oid 14 current snap is 73 2024-09-10T16:10:54.868 INFO:tasks.rados.rados.0.smithi165.stdout:625: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-10T16:10:54.875 INFO:tasks.rados.rados.0.smithi165.stdout:625: writing smithi16534841-14 from 568136 to 1353184 tid 1 2024-09-10T16:10:54.880 INFO:tasks.rados.rados.0.smithi165.stdout:625: writing smithi16534841-14 from 1929299 to 2168136 tid 2 2024-09-10T16:10:54.880 INFO:tasks.rados.rados.0.smithi165.stdout:626: snap_create 2024-09-10T16:10:54.880 INFO:tasks.rados.rados.0.smithi165.stdout:623: expect (ObjNum 235 snap 69 seq_num 235) 2024-09-10T16:10:54.970 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2926261 2024-09-10T16:10:55.023 INFO:tasks.rados.rados.0.smithi165.stdout:624: finishing rollback tid 0 to smithi16534841-38 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:624: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:624: finishing rollback tid 2 to smithi16534841-38 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 224 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: finishing write tid 1 to smithi16534841-14 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: finishing write tid 2 to smithi16534841-14 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: finishing write tid 3 to smithi16534841-14 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: finishing write tid 4 to smithi16534841-14 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: oid 14 updating version 0 to 307 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: oid 14 version 307 is already newer than 305 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:625: oid 14 version 307 is already newer than 306 2024-09-10T16:10:55.024 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 307 (ObjNum 248 snap 73 seq_num 248) dirty exists 2024-09-10T16:10:55.025 INFO:tasks.rados.rados.0.smithi165.stdout:625: left oid 14 (ObjNum 248 snap 73 seq_num 248) 2024-09-10T16:10:55.857 INFO:tasks.rados.rados.0.smithi165.stdout:623: done (3 left) 2024-09-10T16:10:55.857 INFO:tasks.rados.rados.0.smithi165.stdout:624: done (2 left) 2024-09-10T16:10:55.857 INFO:tasks.rados.rados.0.smithi165.stdout:625: done (1 left) 2024-09-10T16:10:55.857 INFO:tasks.rados.rados.0.smithi165.stdout:626: done (0 left) 2024-09-10T16:10:55.857 INFO:tasks.rados.rados.0.smithi165.stdout:627: snap_remove snap 47 2024-09-10T16:10:57.259 INFO:tasks.rados.rados.0.smithi165.stdout:627: done (0 left) 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:628: read oid 23 snap -1 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:628: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:629: rollback oid 2 current snap is 74 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 73 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:630: write (excl) oid 24 current snap is 74 2024-09-10T16:10:57.260 INFO:tasks.rados.rados.0.smithi165.stdout:630: seq_num 249 ranges {584943=460297,1587781=597162} 2024-09-10T16:10:57.265 INFO:tasks.rados.rados.0.smithi165.stdout:630: writing smithi16534841-24 from 584943 to 1045240 tid 1 2024-09-10T16:10:57.272 INFO:tasks.rados.rados.0.smithi165.stdout:630: writing smithi16534841-24 from 1587781 to 2184943 tid 2 2024-09-10T16:10:57.272 INFO:tasks.rados.rados.0.smithi165.stdout:631: snap_create 2024-09-10T16:10:57.272 INFO:tasks.rados.rados.0.smithi165.stdout:629: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:10:57.272 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 315 (ObjNum 33 snap 0 seq_num 1162719424) dirty dne 2024-09-10T16:10:57.272 INFO:tasks.rados.rados.0.smithi165.stdout:628: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:10:57.305 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2620616 2024-09-10T16:10:57.339 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2316598 2024-09-10T16:10:57.408 INFO:tasks.rados.rados.0.smithi165.stdout:630: finishing write tid 1 to smithi16534841-24 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: finishing write tid 2 to smithi16534841-24 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: finishing write tid 3 to smithi16534841-24 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: finishing write tid 4 to smithi16534841-24 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: oid 24 updating version 0 to 446 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: oid 24 updating version 446 to 447 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: oid 24 version 447 is already newer than 445 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 447 (ObjNum 249 snap 74 seq_num 249) dirty exists 2024-09-10T16:10:57.409 INFO:tasks.rados.rados.0.smithi165.stdout:630: left oid 24 (ObjNum 249 snap 74 seq_num 249) 2024-09-10T16:10:57.739 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:10:57.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.845+0000 7f1830164640 10 monclient: get_monmap_and_config 2024-09-10T16:10:57.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.845+0000 7f1830164640 10 monclient: build_initial_monmap 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.845+0000 7f1830164640 10 monclient: monmap: 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:57.847584+0000 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:57.847584+0000 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:57.849 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:57.850 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 1 Processor -- start 2024-09-10T16:10:57.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 1 -- start start 2024-09-10T16:10:57.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient: init 2024-09-10T16:10:57.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:57.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f182809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f182809d190 0x7f182809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient(hunting): picked mon.noname-a con 0x7f182809d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 0x7f182813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.846+0000 7f1830164640 10 monclient(hunting): picked mon.noname-b con 0x7f182813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182809fbc0 con 0x7f182809c310 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f18280a2a20 con 0x7f182809d190 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f182809c6f0 con 0x7f182813a5c0 2024-09-10T16:10:57.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f1830164640 10 monclient(hunting): authenticate will time out at 1036.335026s 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f182809d190 0x7f182809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 0x7f182813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f182809d190 0x7f182809f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38250/0 (socket says 172.21.15.164:38250) 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 -- 172.21.15.164:0/3641646633 learned_addr learned my addr 172.21.15.164:0/3641646633 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): get_auth_request con 0x7f182813a5c0 auth_method 0 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): get_auth_request con 0x7f182809d190 auth_method 0 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:57.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient(hunting): handle_auth_done global_id 54099 payload 274 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 -- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 msgr2=0x7f182813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 0x7f182813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 -- 172.21.15.164:0/3641646633 >> v1:172.21.15.165:6789/0 conn(0x7f182809c310 legacy=0x7f182809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient: _finish_hunting 0 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 monclient: found mon.noname-a 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:57.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182d6ff640 1 -- 172.21.15.164:0/3641646633 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182809a4d0 con 0x7f182809d190 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.847+0000 7f182e701640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 0x7f182813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182d6ff640 1 -- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809d190 msgr2=0x7f182809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182d6ff640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809d190 0x7f182809f150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f181c009870 tx=0x7f181c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bb32bb0912746c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 1 -- 172.21.15.164:0/3641646633 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c037070 con 0x7f182809d190 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: dump: 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.864 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: _finish_auth 0 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: _check_auth_tickets 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 1 -- 172.21.15.164:0/3641646633 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181c032440 con 0x7f182809d190 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f1830164640 5 monclient: authenticate success, global_id 54099 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f1830164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:57.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f1830164640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f1830164640 4 set_mon_vals no callback set 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 1 -- 172.21.15.164:0/3641646633 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181c032810 con 0x7f182809d190 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f182cefe640 10 monclient: dump: 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.866 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.850+0000 7f1830164640 10 monclient: shutdown 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 -- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809d190 msgr2=0x7f182809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809d190 0x7f182809f150 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f181c009870 tx=0x7f181c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 -- 172.21.15.164:0/3641646633 shutdown_connections 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813a5c0 0x7f182813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 --2- 172.21.15.164:0/3641646633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809d190 0x7f182809f150 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 -- 172.21.15.164:0/3641646633 >> 172.21.15.164:0/3641646633 conn(0x7f18280a3110 msgr2=0x7f18280a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 -- 172.21.15.164:0/3641646633 shutdown_connections 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 1 -- 172.21.15.164:0/3641646633 wait complete. 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 10 monclient: build_initial_monmap 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.851+0000 7f1830164640 10 monclient: monmap: 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:57.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 Processor -- start 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 -- start start 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient: init 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): picked mon.noname-a con 0x7f182809c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 0x7f182813cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): picked mon.noname-b con 0x7f182813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 0x7f182813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): picked mon.noname-c con 0x7f182813d430 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f18280a2230 con 0x7f182809c310 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f18280a20b0 con 0x7f182813a5c0 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1828074e00 con 0x7f182813d430 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f1830164640 10 monclient(hunting): authenticate will time out at 1036.340580s 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38258/0 (socket says 172.21.15.164:38258) 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 0x7f182813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 -- 172.21.15.164:0/293699272 learned_addr learned my addr 172.21.15.164:0/293699272 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): get_auth_request con 0x7f182809c310 auth_method 0 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): get_auth_request con 0x7f182813d430 auth_method 0 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:57.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182d6ff640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 0x7f182813cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient(hunting): handle_auth_done global_id 54102 payload 824 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 -- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 msgr2=0x7f182813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 0x7f182813d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 -- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 msgr2=0x7f182813cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 0x7f182813cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient: _finish_hunting 0 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 monclient: found mon.noname-a 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:10:57.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182df00640 1 -- 172.21.15.164:0/293699272 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1828093960 con 0x7f182809c310 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182e701640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 0x7f182813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182d6ff640 10 monclient: get_auth_request con 0x7f182813a5c0 auth_method 0 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.852+0000 7f182d6ff640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 0x7f182813cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f182df00640 1 -- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 msgr2=0x7f18280993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f182df00640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f181800c430 tx=0x7f181800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18180119a0 con 0x7f182809c310 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: dump: 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: _finish_auth 0 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: _check_auth_tickets 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1818017070 con 0x7f182809c310 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 5 monclient: authenticate success, global_id 54102 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 10 monclient: _renew_subs 2024-09-10T16:10:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1828093c40 con 0x7f182809c310 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818011d80 con 0x7f182809c310 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f18267fc640 10 monclient: dump: 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 10 monclient: _renew_subs 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1827fff640 4 set_mon_vals no callback set 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.855+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1828093680 con 0x7f182809c310 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f1830164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f1830164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f1830164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182809ed30 con 0x7f182809c310 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f181801c460 con 0x7f182809c310 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f18267fc640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 0x7f18140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.856+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(165..165 src has 1..165) ==== 4815+0+0 (secure 0 0 0) 0x7f1818052bd0 con 0x7f182809c310 2024-09-10T16:10:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.857+0000 7f182d6ff640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 0x7f18140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.857+0000 7f182d6ff640 10 monclient: get_auth_request con 0x7f181403e210 auth_method 0 2024-09-10T16:10:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.859+0000 7f182d6ff640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 0x7f18140406d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f182806d4d0 tx=0x7f181c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.861+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f182809ed30 con 0x7f182809c310 2024-09-10T16:10:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.861+0000 7f18267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:57.861+0000 7f18267fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:57.878 INFO:tasks.rados.rados.0.smithi165.stdout:628: done (3 left) 2024-09-10T16:10:57.878 INFO:tasks.rados.rados.0.smithi165.stdout:629: done (2 left) 2024-09-10T16:10:57.878 INFO:tasks.rados.rados.0.smithi165.stdout:630: done (1 left) 2024-09-10T16:10:57.878 INFO:tasks.rados.rados.0.smithi165.stdout:631: done (0 left) 2024-09-10T16:10:57.878 INFO:tasks.rados.rados.0.smithi165.stdout:632: snap_create 2024-09-10T16:10:58.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f1830164640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f1830164640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f1830164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f182809f7b0 con 0x7f182809c310 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f18267fc640 1 -- 172.21.15.164:0/293699272 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f1818012470 con 0x7f182809c310 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f18267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.104+0000 7f18267fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:58.107 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:10:58.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 msgr2=0x7f18140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 0x7f18140406d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f182806d4d0 tx=0x7f181c02e000 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 10 monclient: shutdown 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 msgr2=0x7f18280993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f181800c430 tx=0x7f181800c830 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 shutdown_connections 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f181403e210 0x7f18140406d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f182813d430 0x7f182813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f182813a5c0 0x7f182813cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 --2- 172.21.15.164:0/293699272 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f182809c310 0x7f18280993e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.106+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 >> 172.21.15.164:0/293699272 conn(0x7f18280a3110 msgr2=0x7f182806e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.107+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 shutdown_connections 2024-09-10T16:10:58.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.107+0000 7f1830164640 1 -- 172.21.15.164:0/293699272 wait complete. 2024-09-10T16:10:58.120 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:10:58.120 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:10:58.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:10:58.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.216+0000 7f5723e08640 10 monclient: get_monmap_and_config 2024-09-10T16:10:58.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.216+0000 7f5723e08640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.216+0000 7f5723e08640 10 monclient: monmap: 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:58.218111+0000 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:58.218111+0000 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:58.219 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:58.220 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.216+0000 7f5723e08640 1 Processor -- start 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 -- start start 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient: init 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f571c06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f571c134f70 0x7f571c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-a con 0x7f571c134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:10:58.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f571c13d560 0x7f571c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-c con 0x7f571c13d560 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571c09ce50 con 0x7f571c06cb70 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f571c06db60 con 0x7f571c134f70 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f571c06cf50 con 0x7f571c13d560 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f5723e08640 10 monclient(hunting): authenticate will time out at 1036.705621s 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f571c134f70 0x7f571c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f571c134f70 0x7f571c13d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38272/0 (socket says 172.21.15.164:38272) 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 1 -- 172.21.15.164:0/1901730625 learned_addr learned my addr 172.21.15.164:0/1901730625 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): get_auth_request con 0x7f571c134f70 auth_method 0 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.217+0000 7f57213a3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57223a5640 1 --2- 172.21.15.164:0/1901730625 >> v2:172.21.15.165:3300/0 conn(0x7f571c13d560 0x7f571c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 10 monclient(hunting): handle_auth_done global_id 54105 payload 274 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 1 -- 172.21.15.164:0/1901730625 >> v2:172.21.15.165:3300/0 conn(0x7f571c13d560 msgr2=0x7f571c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 1 --2- 172.21.15.164:0/1901730625 >> v2:172.21.15.165:3300/0 conn(0x7f571c13d560 0x7f571c13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 1 -- 172.21.15.164:0/1901730625 >> v1:172.21.15.165:6789/0 conn(0x7f571c06cb70 legacy=0x7f571c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 1 monclient: found mon.noname-a 2024-09-10T16:10:58.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.218+0000 7f57213a3640 1 -- 172.21.15.164:0/1901730625 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c135380 con 0x7f571c134f70 2024-09-10T16:10:58.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.220+0000 7f57213a3640 1 -- 172.21.15.164:0/1901730625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c134f70 msgr2=0x7f571c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.220+0000 7f57213a3640 1 --2- 172.21.15.164:0/1901730625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c134f70 0x7f571c13d020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f570c009870 tx=0x7f570c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4713df0db6e3b1e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 1 -- 172.21.15.164:0/1901730625 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c037070 con 0x7f571c134f70 2024-09-10T16:10:58.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: dump: 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 1 -- 172.21.15.164:0/1901730625 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570c032640 con 0x7f571c134f70 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 5 monclient: authenticate success, global_id 54105 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 1 -- 172.21.15.164:0/1901730625 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c032a10 con 0x7f571c134f70 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 4 set_mon_vals no callback set 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5720ba2640 10 monclient: dump: 2024-09-10T16:10:58.228 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 10 monclient: shutdown 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 -- 172.21.15.164:0/1901730625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c134f70 msgr2=0x7f571c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 --2- 172.21.15.164:0/1901730625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c134f70 0x7f571c13d020 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f570c009870 tx=0x7f570c02a100 comp rx=0 tx=0).stop 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 -- 172.21.15.164:0/1901730625 shutdown_connections 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 --2- 172.21.15.164:0/1901730625 >> v2:172.21.15.165:3300/0 conn(0x7f571c13d560 0x7f571c13f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 --2- 172.21.15.164:0/1901730625 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c134f70 0x7f571c13d020 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 -- 172.21.15.164:0/1901730625 >> 172.21.15.164:0/1901730625 conn(0x7f571c0aa900 msgr2=0x7f571c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 -- 172.21.15.164:0/1901730625 shutdown_connections 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.223+0000 7f5723e08640 1 -- 172.21.15.164:0/1901730625 wait complete. 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.224+0000 7f5723e08640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.224+0000 7f5723e08640 10 monclient: monmap: 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:58.230 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:58.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:58.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:58.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.224+0000 7f5723e08640 1 Processor -- start 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.224+0000 7f5723e08640 1 -- start start 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.224+0000 7f5723e08640 10 monclient: init 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 0x7f571c12e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-c con 0x7f571c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c134f70 0x7f571c12ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-b con 0x7f571c134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:58.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): picked mon.noname-a con 0x7f571c13d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f571c0a4db0 con 0x7f571c13d560 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f571c09d130 con 0x7f571c134f70 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f571c13fe90 con 0x7f571c06cb70 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5723e08640 10 monclient(hunting): authenticate will time out at 1036.713205s 2024-09-10T16:10:58.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 0x7f571c12e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 0x7f571c12e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40074/0 (socket says 172.21.15.164:40074) 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 1 -- 172.21.15.164:0/1122980076 learned_addr learned my addr 172.21.15.164:0/1122980076 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): get_auth_request con 0x7f571c06cb70 auth_method 0 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38286/0 (socket says 172.21.15.164:38286) 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): get_auth_request con 0x7f571c13d560 auth_method 0 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57213a3640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c134f70 0x7f571c12ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57213a3640 10 monclient(hunting): get_auth_request con 0x7f571c134f70 auth_method 0 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57213a3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57213a3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57213a3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient(hunting): handle_auth_done global_id 54108 payload 824 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 msgr2=0x7f571c12e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 0x7f571c12e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c134f70 msgr2=0x7f571c12ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f571c134f70 0x7f571c12ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 monclient: found mon.noname-a 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f57223a5640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c138a70 con 0x7f571c13d560 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.225+0000 7f5721ba4640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f571c06cb70 0x7f571c12e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57223a5640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 msgr2=0x7f571c138db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57223a5640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f571400cca0 tx=0x7f571400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714003b90 con 0x7f571c13d560 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 10 monclient: dump: 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5714003d50 con 0x7f571c13d560 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f5723e08640 5 monclient: authenticate success, global_id 54108 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f5723e08640 10 monclient: _renew_subs 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f5723e08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.227+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571c135870 con 0x7f571c13d560 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f57127fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714004260 con 0x7f571c13d560 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f57127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f57127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f57127fc640 10 monclient: dump: 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 10 monclient: _renew_subs 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5713fff640 4 set_mon_vals no callback set 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571c13b940 con 0x7f571c13d560 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 0x7f571c1396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f571c139c10 con 0x7f571c1392f0 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.228+0000 7f5721ba4640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 0x7f571c1396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient: get_auth_request con 0x7f571c1392f0 auth_method 0 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f57140044c0 con 0x7f571c13d560 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 10 monclient(hunting): handle_auth_done global_id 54111 payload 824 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f5721ba4640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f571c0a46a0 con 0x7f571c1392f0 2024-09-10T16:10:58.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f57127fc640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f570003e210 0x7f57000406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4787+0+0 (secure 0 0 0) 0x7f5714053820 con 0x7f571c13d560 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f57213a3640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f570003e210 0x7f57000406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.229+0000 7f57213a3640 10 monclient: get_auth_request con 0x7f570003e210 auth_method 0 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f5721ba4640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 msgr2=0x7f571c1396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f5721ba4640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 0x7f571c1396d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f571800eb10 tx=0x7f5718009f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718017070 con 0x7f571c1392f0 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 10 monclient: dump: 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f571800cdf0 con 0x7f571c1392f0 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 msgr2=0x7f571c1396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 0x7f571c1396d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f571800eb10 tx=0x7f5718009f20 comp rx=0 tx=0).stop 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 msgr2=0x7f571c1396d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.231+0000 7f57127fc640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1392f0 0x7f571c1396d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.232+0000 7f57213a3640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f570003e210 0x7f57000406d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f570c031000 tx=0x7f570c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5723e08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:10:58.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5723e08640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 0x7f571c1306f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5723e08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f571c0a9d90 con 0x7f571c1302e0 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5721ba4640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 0x7f571c1306f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5721ba4640 10 monclient: get_auth_request con 0x7f571c1302e0 auth_method 0 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5721ba4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5721ba4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.237+0000 7f5721ba4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.238+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.238+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.238+0000 7f5721ba4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.238+0000 7f5721ba4640 10 monclient(hunting): handle_auth_done global_id 54114 payload 824 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.238+0000 7f5721ba4640 1 -- 172.21.15.164:0/1122980076 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f571c13bd00 con 0x7f571c1302e0 2024-09-10T16:10:58.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f5721ba4640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 msgr2=0x7f571c1306f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f5721ba4640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 0x7f571c1306f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f571800eb10 tx=0x7f57180046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718017040 con 0x7f571c1302e0 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 10 monclient: dump: 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.251 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f5718005b20 con 0x7f571c1302e0 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 msgr2=0x7f571c1306f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 0x7f571c1306f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f571800eb10 tx=0x7f57180046f0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 msgr2=0x7f571c1306f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.240+0000 7f57127fc640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c1302e0 0x7f571c1306f0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f571800eb10 tx=0x7f57180046f0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 28425, 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.253 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.254 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.255 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.256 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.257 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:58.258 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.259 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.260 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.261 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:10:58.264 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.265 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.265 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:10:58.265 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:10:58.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f570003e210 msgr2=0x7f57000406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f570003e210 0x7f57000406d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f570c031000 tx=0x7f570c02e000 comp rx=0 tx=0).stop 2024-09-10T16:10:58.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 10 monclient: shutdown 2024-09-10T16:10:58.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 msgr2=0x7f571c138db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f57223a5640 1 -- 172.21.15.164:0/1122980076 reap_dead start 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f571400cca0 tx=0x7f571400a660 comp rx=0 tx=0).stop 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 shutdown_connections 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 --2- 172.21.15.164:0/1122980076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f571c13d560 0x7f571c138db0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 >> 172.21.15.164:0/1122980076 conn(0x7f571c0aa900 msgr2=0x7f571c0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 shutdown_connections 2024-09-10T16:10:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.241+0000 7f5723e08640 1 -- 172.21.15.164:0/1122980076 wait complete. 2024-09-10T16:10:58.267 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:10:58.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:10:58.268 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.363+0000 7fc020375640 10 monclient: get_monmap_and_config 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.363+0000 7fc020375640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.363+0000 7fc020375640 10 monclient: monmap: 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:58.365473+0000 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:58.365473+0000 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.366 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:58.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 Processor -- start 2024-09-10T16:10:58.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 -- start start 2024-09-10T16:10:58.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient: init 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc01809f930 0x7fc0180a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): picked mon.noname-c con 0x7fc01809f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): picked mon.noname-b con 0x7fc0180a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc01813a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc01809fd10 con 0x7fc01813a5c0 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc01809cd20 con 0x7fc01809f930 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc01809a0d0 con 0x7fc0180a2380 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc020375640 10 monclient(hunting): authenticate will time out at 1036.852862s 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.364+0000 7fc01d910640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40084/0 (socket says 172.21.15.164:40084) 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 -- 172.21.15.164:0/2091514286 learned_addr learned my addr 172.21.15.164:0/2091514286 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): get_auth_request con 0x7fc0180a2380 auth_method 0 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01e111640 1 --2- 172.21.15.164:0/2091514286 >> v2:172.21.15.165:3300/0 conn(0x7fc01809f930 0x7fc0180a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient(hunting): handle_auth_done global_id 4799 payload 274 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 -- 172.21.15.164:0/2091514286 >> v2:172.21.15.165:3300/0 conn(0x7fc01809f930 msgr2=0x7fc0180a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 --2- 172.21.15.164:0/2091514286 >> v2:172.21.15.165:3300/0 conn(0x7fc01809f930 0x7fc0180a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 -- 172.21.15.164:0/2091514286 >> v1:172.21.15.165:6789/0 conn(0x7fc01813a5c0 legacy=0x7fc01813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 monclient: found mon.noname-b 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.365+0000 7fc01d910640 1 -- 172.21.15.164:0/2091514286 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc01809e840 con 0x7fc0180a2380 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d910640 1 -- 172.21.15.164:0/2091514286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 msgr2=0x7fc0180a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d910640 1 --2- 172.21.15.164:0/2091514286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc008009870 tx=0x7fc00802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96b95b526b52037f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 1 -- 172.21.15.164:0/2091514286 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008037070 con 0x7fc0180a2380 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: dump: 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 1 -- 172.21.15.164:0/2091514286 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc008032640 con 0x7fc0180a2380 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 5 monclient: authenticate success, global_id 4799 2024-09-10T16:10:58.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 1 -- 172.21.15.164:0/2091514286 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008032a10 con 0x7fc0180a2380 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 4 set_mon_vals no callback set 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc01d10f640 10 monclient: dump: 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.380 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 10 monclient: shutdown 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 -- 172.21.15.164:0/2091514286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 msgr2=0x7fc0180a27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 --2- 172.21.15.164:0/2091514286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc008009870 tx=0x7fc00802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 -- 172.21.15.164:0/2091514286 shutdown_connections 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 --2- 172.21.15.164:0/2091514286 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc0180a27b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 --2- 172.21.15.164:0/2091514286 >> v2:172.21.15.165:3300/0 conn(0x7fc01809f930 0x7fc0180a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 -- 172.21.15.164:0/2091514286 >> 172.21.15.164:0/2091514286 conn(0x7fc0180a3110 msgr2=0x7fc0180a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 -- 172.21.15.164:0/2091514286 shutdown_connections 2024-09-10T16:10:58.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.368+0000 7fc020375640 1 -- 172.21.15.164:0/2091514286 wait complete. 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.369+0000 7fc020375640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.369+0000 7fc020375640 10 monclient: monmap: 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.369+0000 7fc020375640 1 Processor -- start 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.369+0000 7fc020375640 1 -- start start 2024-09-10T16:10:58.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.369+0000 7fc020375640 10 monclient: init 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 0x7fc018073940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): picked mon.noname-a con 0x7fc01809f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc01809a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): picked mon.noname-c con 0x7fc0180a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc01813a5c0 0x7fc01809a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): picked mon.noname-b con 0x7fc01813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc01809d930 con 0x7fc01809f930 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc01809ee00 con 0x7fc01813a5c0 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc01809c8b0 con 0x7fc0180a2380 2024-09-10T16:10:58.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc020375640 10 monclient(hunting): authenticate will time out at 1036.858263s 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc01809a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 0x7fc018073940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc01809a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40086/0 (socket says 172.21.15.164:40086) 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 0x7fc018073940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38314/0 (socket says 172.21.15.164:38314) 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 -- 172.21.15.164:0/3354377810 learned_addr learned my addr 172.21.15.164:0/3354377810 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): get_auth_request con 0x7fc0180a2380 auth_method 0 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): get_auth_request con 0x7fc01809f930 auth_method 0 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc01813a5c0 0x7fc01809a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 10 monclient(hunting): get_auth_request con 0x7fc01813a5c0 auth_method 0 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient(hunting): handle_auth_done global_id 4802 payload 824 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc01813a5c0 msgr2=0x7fc01809a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc01813a5c0 0x7fc01809a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 msgr2=0x7fc018073940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 0x7fc018073940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 monclient: found mon.noname-c 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01d910640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc018131950 con 0x7fc0180a2380 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e912640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc01813a5c0 0x7fc01809a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.370+0000 7fc01e111640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc01809f930 0x7fc018073940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.372+0000 7fc01d910640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 msgr2=0x7fc01809a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.372+0000 7fc01d910640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc01809a430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc008009870 tx=0x7fc0080046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008041030 con 0x7fc0180a2380 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: dump: 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.387 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc00803d6f0 con 0x7fc0180a2380 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 5 monclient: authenticate success, global_id 4802 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 10 monclient: _renew_subs 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0180a1810 con 0x7fc0180a2380 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc008037040 con 0x7fc0180a2380 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 10 monclient: dump: 2024-09-10T16:10:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc01c90e640 4 set_mon_vals no callback set 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 10 monclient: _renew_subs 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc01809c140 con 0x7fc0180a2380 2024-09-10T16:10:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc008044070 con 0x7fc0180a2380 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.373+0000 7fc020375640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc020375640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 0x7fc01809aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc01802e9a0 con 0x7fc01802e5c0 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc00effd640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbffc03e210 0x7fbffc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 0x7fc01809aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient: get_auth_request con 0x7fc01802e5c0 auth_method 0 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(166..166 src has 1..166) ==== 4787+0+0 (secure 0 0 0) 0x7fc008072d90 con 0x7fc0180a2380 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2024-09-10T16:10:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e111640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0180a0760 con 0x7fc01802e5c0 2024-09-10T16:10:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e912640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbffc03e210 0x7fbffc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.374+0000 7fc01e912640 10 monclient: get_auth_request con 0x7fbffc03e210 auth_method 0 2024-09-10T16:10:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc01e111640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 msgr2=0x7fc01809aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc01e111640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 0x7fc01809aeb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc01400d590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014013640 con 0x7fc01802e5c0 2024-09-10T16:10:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.376+0000 7fc00effd640 10 monclient: dump: 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc01e912640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbffc03e210 0x7fbffc0406d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc01809c070 tx=0x7fc010009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc014013640 con 0x7fc01802e5c0 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 msgr2=0x7fc01809aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 0x7fc01809aeb0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc01400d590 comp rx=0 tx=0).stop 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 msgr2=0x7fc01809aeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.377+0000 7fc00effd640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc01802e5c0 0x7fc01809aeb0 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc01400d590 comp rx=0 tx=0).stop 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc020375640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc020375640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 0x7fc018064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc020375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc0180718e0 con 0x7fc018063d50 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 0x7fc018064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient: get_auth_request con 0x7fc018063d50 auth_method 0 2024-09-10T16:10:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.383+0000 7fc01e111640 1 -- 172.21.15.164:0/3354377810 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc018077260 con 0x7fc018063d50 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc01e111640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 msgr2=0x7fc018064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc01e111640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 0x7fc018064160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc0140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc014003610 con 0x7fc018063d50 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc00effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc00effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc00effd640 10 monclient: dump: 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.385+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fc014024070 con 0x7fc018063d50 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:10:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 msgr2=0x7fc018064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 0x7fc018064160 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc0140046c0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 msgr2=0x7fc018064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc00effd640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc018063d50 0x7fc018064160 secure :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc01400fbd0 tx=0x7fc0140046c0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:10:58.398 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 300557, 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.399 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.400 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:10:58.401 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:10:58.402 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.403 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.404 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:58.405 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.407 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.408 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.409 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.410 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:10:58.411 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:10:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbffc03e210 msgr2=0x7fbffc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbffc03e210 0x7fbffc0406d0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc01809c070 tx=0x7fc010009040 comp rx=0 tx=0).stop 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 10 monclient: shutdown 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 msgr2=0x7fc01809a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 --2- 172.21.15.164:0/3354377810 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc0180a2380 0x7fc01809a430 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc008009870 tx=0x7fc0080046c0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc01e912640 1 -- 172.21.15.164:0/3354377810 reap_dead start 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc01e912640 1 -- 172.21.15.164:0/3354377810 reap_dead start 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 shutdown_connections 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 >> 172.21.15.164:0/3354377810 conn(0x7fc0180a3110 msgr2=0x7fc018138a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 shutdown_connections 2024-09-10T16:10:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.386+0000 7fc020375640 1 -- 172.21.15.164:0/3354377810 wait complete. 2024-09-10T16:10:58.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:10:58.414 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:10:58.414 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:10:58.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 10 monclient: get_monmap_and_config 2024-09-10T16:10:58.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 10 monclient: monmap: 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:58.512053+0000 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:58.512053+0000 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:58.513 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:58.514 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 1 Processor -- start 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 1 -- start start 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.510+0000 7fdb3dbc2640 10 monclient: init 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb38134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-b con 0x7fdb3806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdb380a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:58.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdb380a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb3809ce50 con 0x7fdb380a46a0 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdb38138ac0 con 0x7fdb380a4a80 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdb3806db60 con 0x7fdb3806cb70 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb3dbc2640 10 monclient(hunting): authenticate will time out at 1036.999502s 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb36ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fdb380a46a0 0x7fdb38137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42092/0 (socket says 172.21.15.164:42092) 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb36ffd640 1 -- 172.21.15.164:0/2294809079 learned_addr learned my addr 172.21.15.164:0/2294809079 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 1 --2- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb38134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): get_auth_request con 0x7fdb3806cb70 auth_method 0 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient(hunting): handle_auth_done global_id 4811 payload 274 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 1 -- 172.21.15.164:0/2294809079 >> v1:172.21.15.165:6789/0 conn(0x7fdb380a4a80 legacy=0x7fdb381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 1 -- 172.21.15.164:0/2294809079 >> v1:172.21.15.164:6789/0 conn(0x7fdb380a46a0 legacy=0x7fdb38137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 1 monclient: found mon.noname-b 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.511+0000 7fdb377fe640 1 -- 172.21.15.164:0/2294809079 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3809cb70 con 0x7fdb3806cb70 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.514+0000 7fdb377fe640 1 -- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 msgr2=0x7fdb38134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.514+0000 7fdb377fe640 1 --2- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb38134f70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdb280097d0 tx=0x7fdb28029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fc5694413f098ac server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 1 -- 172.21.15.164:0/2294809079 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28037070 con 0x7fdb3806cb70 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: dump: 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.525 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 5 monclient: authenticate success, global_id 4811 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 1 -- 172.21.15.164:0/2294809079 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb28032440 con 0x7fdb3806cb70 2024-09-10T16:10:58.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 4 set_mon_vals no callback set 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 1 -- 172.21.15.164:0/2294809079 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28032810 con 0x7fdb3806cb70 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb367fc640 10 monclient: dump: 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 10 monclient: shutdown 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 msgr2=0x7fdb38134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb38134f70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdb280097d0 tx=0x7fdb28029fb0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/2294809079 shutdown_connections 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/2294809079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb38134f70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/2294809079 >> 172.21.15.164:0/2294809079 conn(0x7fdb380aa900 msgr2=0x7fdb380aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/2294809079 shutdown_connections 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.515+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/2294809079 wait complete. 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.516+0000 7fdb3dbc2640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.516+0000 7fdb3dbc2640 10 monclient: monmap: 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.516+0000 7fdb3dbc2640 1 Processor -- start 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.516+0000 7fdb3dbc2640 1 -- start start 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.516+0000 7fdb3dbc2640 10 monclient: init 2024-09-10T16:10:58.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb381cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-c con 0x7fdb3806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb380a46a0 0x7fdb38140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-a con 0x7fdb380a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a4a80 0x7fdb38140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): picked mon.noname-b con 0x7fdb380a4a80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdb3806cf50 con 0x7fdb380a46a0 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb38141050 con 0x7fdb380a4a80 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdb381411d0 con 0x7fdb3806cb70 2024-09-10T16:10:58.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb3dbc2640 10 monclient(hunting): authenticate will time out at 1037.005326s 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb381cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb380a46a0 0x7fdb38140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb381cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40120/0 (socket says 172.21.15.164:40120) 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 -- 172.21.15.164:0/530571257 learned_addr learned my addr 172.21.15.164:0/530571257 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): get_auth_request con 0x7fdb3806cb70 auth_method 0 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request con 0x7fdb380a46a0 auth_method 0 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb37fff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a4a80 0x7fdb38140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient(hunting): handle_auth_done global_id 4814 payload 824 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a4a80 msgr2=0x7fdb38140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a4a80 0x7fdb38140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb380a46a0 msgr2=0x7fdb38140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb380a46a0 0x7fdb38140030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 monclient: found mon.noname-c 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb377fe640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdb3813a220 con 0x7fdb3806cb70 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb37fff640 10 monclient: get_auth_request con 0x7fdb380a4a80 auth_method 0 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb37fff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a4a80 0x7fdb38140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.517+0000 7fdb36ffd640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdb380a46a0 0x7fdb38140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.519+0000 7fdb377fe640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 msgr2=0x7fdb381cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.519+0000 7fdb377fe640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb381cd590 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdb280097d0 tx=0x7fdb280046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28041070 con 0x7fdb3806cb70 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: dump: 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.534 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 5 monclient: authenticate success, global_id 4814 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdb2803c660 con 0x7fdb3806cb70 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 10 monclient: _renew_subs 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdb380a8a20 con 0x7fdb3806cb70 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb28037040 con 0x7fdb3806cb70 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb1ffff640 10 monclient: dump: 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 10 monclient: _renew_subs 2024-09-10T16:10:58.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb35ffb640 4 set_mon_vals no callback set 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdb380a4050 con 0x7fdb3806cb70 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.520+0000 7fdb3dbc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 0x7fdb3802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb380a4f10 con 0x7fdb380a2f00 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdb28044070 con 0x7fdb3806cb70 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 0x7fdb3802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb380a2f00 auth_method 0 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb1ffff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdb2403e210 0x7fdb240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(166..166 src has 1..166) ==== 4787+0+0 (secure 0 0 0) 0x7fdb280733d0 con 0x7fdb3806cb70 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb37fff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdb2403e210 0x7fdb240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb37fff640 10 monclient: get_auth_request con 0x7fdb2403e210 auth_method 0 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_done global_id 44188 payload 824 2024-09-10T16:10:58.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.521+0000 7fdb36ffd640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb38134f70 con 0x7fdb380a2f00 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.523+0000 7fdb36ffd640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 msgr2=0x7fdb3802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.523+0000 7fdb36ffd640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 0x7fdb3802d440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdb38139540 tx=0x7fdb2c007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.523+0000 7fdb37fff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdb2403e210 0x7fdb240406d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb381ce3e0 tx=0x7fdb30009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c017020 con 0x7fdb380a2f00 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 10 monclient: dump: 2024-09-10T16:10:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdb2c00fc10 con 0x7fdb380a2f00 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 msgr2=0x7fdb3802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 0x7fdb3802d440 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdb38139540 tx=0x7fdb2c007700 comp rx=0 tx=0).stop 2024-09-10T16:10:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 msgr2=0x7fdb3802d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.524+0000 7fdb1ffff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a2f00 0x7fdb3802d440 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb3dbc2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 0x7fdb3802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb3dbc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdb381392c0 con 0x7fdb380a32e0 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb36ffd640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 0x7fdb3802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb36ffd640 10 monclient: get_auth_request con 0x7fdb380a32e0 auth_method 0 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.530+0000 7fdb36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.531+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.531+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.531+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.531+0000 7fdb36ffd640 10 monclient(hunting): handle_auth_done global_id 44191 payload 824 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.531+0000 7fdb36ffd640 1 -- 172.21.15.164:0/530571257 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdb3809ed00 con 0x7fdb380a32e0 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb36ffd640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 msgr2=0x7fdb3802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb36ffd640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 0x7fdb3802d980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdb38139540 tx=0x7fdb2c004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdb2c017030 con 0x7fdb380a32e0 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 10 monclient: dump: 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fdb2c013b80 con 0x7fdb380a32e0 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 msgr2=0x7fdb3802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 0x7fdb3802d980 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fdb38139540 tx=0x7fdb2c004940 comp rx=0 tx=0).stop 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 msgr2=0x7fdb3802d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.533+0000 7fdb1ffff640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdb380a32e0 0x7fdb3802d980 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.545 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 73685, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.546 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:10:58.547 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.548 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:10:58.549 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.550 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.551 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:10:58.552 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.553 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.554 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.555 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:10:58.556 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:10:58.557 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:10:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdb2403e210 msgr2=0x7fdb240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdb2403e210 0x7fdb240406d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fdb381ce3e0 tx=0x7fdb30009040 comp rx=0 tx=0).stop 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 10 monclient: shutdown 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 msgr2=0x7fdb381cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 --2- 172.21.15.164:0/530571257 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdb3806cb70 0x7fdb381cd590 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdb280097d0 tx=0x7fdb280046c0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb37fff640 1 -- 172.21.15.164:0/530571257 reap_dead start 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb37fff640 1 -- 172.21.15.164:0/530571257 reap_dead start 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 shutdown_connections 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 >> 172.21.15.164:0/530571257 conn(0x7fdb380aa900 msgr2=0x7fdb38137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 shutdown_connections 2024-09-10T16:10:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.534+0000 7fdb3dbc2640 1 -- 172.21.15.164:0/530571257 wait complete. 2024-09-10T16:10:58.559 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:10:58.559 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:10:58.560 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.656+0000 7fd3b339f640 10 monclient: get_monmap_and_config 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.656+0000 7fd3b339f640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.656+0000 7fd3b339f640 10 monclient: monmap: 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:10:58.658506+0000 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:10:58.658506+0000 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.659 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:10:58.660 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:10:58.660 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:10:58.660 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:10:58.660 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:10:58.660 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 Processor -- start 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 -- start start 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient: init 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac135b60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3ac135f40 addr v1:172.21.15.164:6789/0 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3ac13b090 addr v1:172.21.15.165:6789/0 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3ac0a1790 con 0x7fd3ac135f40 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3ac0a1440 con 0x7fd3ac13b090 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac06fa20 con 0x7fd3ac135b60 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.657+0000 7fd3b339f640 10 monclient(hunting): authenticate will time out at 1037.145933s 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40132/0 (socket says 172.21.15.164:40132) 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 -- 172.21.15.164:0/1259632061 learned_addr learned my addr 172.21.15.164:0/1259632061 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): get_auth_request con 0x7fd3ac135b60 auth_method 0 2024-09-10T16:10:58.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b1b9c640 1 --1- 172.21.15.164:0/1259632061 >> v1:172.21.15.164:6789/0 conn(0x7fd3ac135f40 0x7fd3ac136340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42094/0 (socket says 172.21.15.164:42094) 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient(hunting): handle_auth_done global_id 4817 payload 274 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 -- 172.21.15.164:0/1259632061 >> v1:172.21.15.165:6789/0 conn(0x7fd3ac13b090 legacy=0x7fd3ac13d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 -- 172.21.15.164:0/1259632061 >> v1:172.21.15.164:6789/0 conn(0x7fd3ac135f40 legacy=0x7fd3ac136340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 monclient: found mon.noname-b 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:10:58.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.658+0000 7fd3b239d640 1 -- 172.21.15.164:0/1259632061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac0a4930 con 0x7fd3ac135b60 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b239d640 1 -- 172.21.15.164:0/1259632061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 msgr2=0x7fd3ac0a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b239d640 1 --2- 172.21.15.164:0/1259632061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098a0 tx=0x7fd3a802a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7244d2713f96a6c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 1 -- 172.21.15.164:0/1259632061 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8037070 con 0x7fd3ac135b60 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: dump: 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.670 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 5 monclient: authenticate success, global_id 4817 2024-09-10T16:10:58.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 1 -- 172.21.15.164:0/1259632061 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a8032640 con 0x7fd3ac135b60 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 1 -- 172.21.15.164:0/1259632061 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a8032a10 con 0x7fd3ac135b60 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 10 monclient: get_monmap_and_config success 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 4 set_mon_vals no callback set 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b139b640 10 monclient: dump: 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.672 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 10 monclient: shutdown 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 1 -- 172.21.15.164:0/1259632061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 msgr2=0x7fd3ac0a2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 1 --2- 172.21.15.164:0/1259632061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd3a80098a0 tx=0x7fd3a802a080 comp rx=0 tx=0).stop 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 1 -- 172.21.15.164:0/1259632061 shutdown_connections 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 1 --2- 172.21.15.164:0/1259632061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135b60 0x7fd3ac0a2090 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.661+0000 7fd3b339f640 1 -- 172.21.15.164:0/1259632061 >> 172.21.15.164:0/1259632061 conn(0x7fd3ac0a4ee0 msgr2=0x7fd3ac0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 1 -- 172.21.15.164:0/1259632061 shutdown_connections 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 1 -- 172.21.15.164:0/1259632061 wait complete. 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 10 monclient: build_initial_monmap 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 10 monclient: monmap: 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 1 Processor -- start 2024-09-10T16:10:58.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 1 -- start start 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.662+0000 7fd3b339f640 10 monclient: init 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient: _reopen_session rank -1 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 0x7fd3ac09bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-b con 0x7fd3ac135b60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-c con 0x7fd3ac135f40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 0x7fd3ac06d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): picked mon.noname-a con 0x7fd3ac13b090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac0a41e0 con 0x7fd3ac13b090 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd3ac0a4060 con 0x7fd3ac135b60 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd3ac13eba0 con 0x7fd3ac135f40 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): _renew_subs 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b339f640 10 monclient(hunting): authenticate will time out at 1037.151349s 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 0x7fd3ac06d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40142/0 (socket says 172.21.15.164:40142) 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 1 -- 172.21.15.164:0/4267995352 learned_addr learned my addr 172.21.15.164:0/4267995352 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): get_auth_request con 0x7fd3ac135f40 auth_method 0 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): get_auth_request con 0x7fd3ac13b090 auth_method 0 2024-09-10T16:10:58.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b239d640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 0x7fd3ac09bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient(hunting): handle_auth_done global_id 54123 payload 824 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 -- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 msgr2=0x7fd3ac06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 -- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 msgr2=0x7fd3ac09bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 0x7fd3ac09bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient: _finish_hunting 0 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 monclient: found mon.noname-a 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b2b9e640 1 -- 172.21.15.164:0/4267995352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3ac09f5d0 con 0x7fd3ac13b090 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b239d640 10 monclient: get_auth_request con 0x7fd3ac135b60 auth_method 0 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b1b9c640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.663+0000 7fd3b239d640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 0x7fd3ac09bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.665+0000 7fd3b2b9e640 1 -- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 msgr2=0x7fd3ac06d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.665+0000 7fd3b2b9e640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 0x7fd3ac06d0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c780 tx=0x7fd3a400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4016020 con 0x7fd3ac13b090 2024-09-10T16:10:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: dump: 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3a400ac60 con 0x7fd3ac13b090 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4012660 con 0x7fd3ac13b090 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 5 monclient: authenticate success, global_id 54123 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 10 monclient: _renew_subs 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3ac13d290 con 0x7fd3ac13b090 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b0b9a640 4 set_mon_vals no callback set 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:10:58.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 10 monclient: dump: 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 10 monclient: _renew_subs 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3ac13cfb0 con 0x7fd3ac13b090 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.666+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd3a4019020 con 0x7fd3ac13b090 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b339f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b339f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b339f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3ac134190 con 0x7fd3ac13b090 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3a2ffd640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 0x7fd390040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(166..166 src has 1..166) ==== 4787+0+0 (secure 0 0 0) 0x7fd3a4052b80 con 0x7fd3ac13b090 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b239d640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 0x7fd390040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:10:58.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.667+0000 7fd3b239d640 10 monclient: get_auth_request con 0x7fd39003e240 auth_method 0 2024-09-10T16:10:58.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.669+0000 7fd3b239d640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 0x7fd390040700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0a1bf0 tx=0x7fd3a802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:10:58.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.671+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd3ac134190 con 0x7fd3ac13b090 2024-09-10T16:10:58.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.671+0000 7fd3a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:10:58.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.671+0000 7fd3a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:10:58.877 INFO:tasks.rados.rados.0.smithi165.stdout:632: done (0 left) 2024-09-10T16:10:58.877 INFO:tasks.rados.rados.0.smithi165.stdout:633: read oid 31 snap -1 2024-09-10T16:10:58.878 INFO:tasks.rados.rados.0.smithi165.stdout:633: expect deleted 2024-09-10T16:10:58.878 INFO:tasks.rados.rados.0.smithi165.stdout:634: read oid 48 snap -1 2024-09-10T16:10:58.878 INFO:tasks.rados.rados.0.smithi165.stdout:634: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:58.878 INFO:tasks.rados.rados.0.smithi165.stdout:635: write (excl) oid 29 current snap is 76 2024-09-10T16:10:58.878 INFO:tasks.rados.rados.0.smithi165.stdout:635: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-10T16:10:58.884 INFO:tasks.rados.rados.0.smithi165.stdout:635: writing smithi16534841-29 from 601750 to 1137296 tid 1 2024-09-10T16:10:58.890 INFO:tasks.rados.rados.0.smithi165.stdout:635: writing smithi16534841-29 from 1929910 to 2201750 tid 2 2024-09-10T16:10:58.891 INFO:tasks.rados.rados.0.smithi165.stdout:636: write oid 47 current snap is 76 2024-09-10T16:10:58.891 INFO:tasks.rados.rados.0.smithi165.stdout:636: seq_num 251 ranges {618557=727148,2104745=113812} 2024-09-10T16:10:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:636: writing smithi16534841-47 from 618557 to 1345705 tid 1 2024-09-10T16:10:58.903 INFO:tasks.rados.rados.0.smithi165.stdout:636: writing smithi16534841-47 from 2104745 to 2218557 tid 2 2024-09-10T16:10:58.903 INFO:tasks.rados.rados.0.smithi165.stdout:637: delete oid 34 current snap is 76 2024-09-10T16:10:58.903 INFO:tasks.rados.rados.0.smithi165.stdout:634: expect (ObjNum 177 snap 53 seq_num 177) 2024-09-10T16:10:58.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.920+0000 7fd3b339f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:10:58.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.920+0000 7fd3b339f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:10:58.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.920+0000 7fd3b339f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:10:58.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.920+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd3ac0a2090 con 0x7fd3ac13b090 2024-09-10T16:10:58.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.921+0000 7fd3a2ffd640 1 -- 172.21.15.164:0/4267995352 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd3a4020030 con 0x7fd3ac13b090 2024-09-10T16:10:58.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.921+0000 7fd3a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:10:58.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.921+0000 7fd3a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:10:58.924 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.925 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=1424514155,config=2586702516,health=1013729810,kv=2518733882,logm=1228987440}) 2024-09-10T16:10:58.925 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=10,logm=65} crc {auth=2723859164,config=2586702516,health=1361790972,kv=3728382353,logm=3948002120}) 2024-09-10T16:10:58.925 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.925 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=12,config=2,health=11,kv=11,logm=64} crc {auth=1424514155,config=2586702516,health=1013729810,kv=2518733882,logm=1228987440}) 2024-09-10T16:10:58.925 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.922+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=150060884,config=2586702516,health=1361790972,kv=805391194,logm=2974599594}) 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276239977}) 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1929676615}) 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276239977}) 2024-09-10T16:10:58.926 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.923+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=611433100}) 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117144976}) 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=674488741}) 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1117144976}) 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188795165}) 2024-09-10T16:10:58.927 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 msgr2=0x7fd390040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.928 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469978349}) 2024-09-10T16:10:58.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 0x7fd390040700 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fd3ac0a1bf0 tx=0x7fd3a802e000 comp rx=0 tx=0).stop 2024-09-10T16:10:58.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 10 monclient: shutdown 2024-09-10T16:10:58.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:10:58.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 msgr2=0x7fd3ac06d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:10:58.929 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=905800867}) 2024-09-10T16:10:58.929 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.924+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 0x7fd3ac06d0b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd3a400c780 tx=0x7fd3a400cc40 comp rx=0 tx=0).stop 2024-09-10T16:10:58.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1469978349}) 2024-09-10T16:10:58.930 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.924+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299332411}) 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 shutdown_connections 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd39003e240 0x7fd390040700 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd3ac13b090 0x7fd3ac06d0b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd3ac135f40 0x7fd3ac06cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 --2- 172.21.15.164:0/4267995352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd3ac135b60 0x7fd3ac09bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 >> 172.21.15.164:0/4267995352 conn(0x7fd3ac0a4ee0 msgr2=0x7fd3ac0a7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:10:58.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 shutdown_connections 2024-09-10T16:10:58.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:10:58.925+0000 7fd3b339f640 1 -- 172.21.15.164:0/4267995352 wait complete. 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702024512}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=671447}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2702024512}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.925+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882580880}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219858875}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540714967}) 2024-09-10T16:10:58.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219858875}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.926+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4069988585}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388503287}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172664105}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3388503287}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506627986}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301691786}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318568400}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301691786}) 2024-09-10T16:10:58.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.927+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785044479}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286540999}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2452631596}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1286540999}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.928+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1555853130}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464184743}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565042252}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1464184743}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=679107624}) 2024-09-10T16:10:58.938 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545377218}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1480041225}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3545377218}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.929+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3464882178}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401703802}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3633218524}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401703802}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.930+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3423244624}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33} crc {logm=4090131346,mdsmap=204415920,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1550280864,mon_config_key=3907214271,monmap=1433040829,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=1505086411,osd_snap=663696334}) 2024-09-10T16:10:58.939 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=16,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30} crc {logm=2355245491,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=409629042,mon_config_key=2175818790,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1505086411,osd_snap=1702157096}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=15,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33} crc {logm=4090131346,mdsmap=204415920,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1550280864,mon_config_key=3907214271,monmap=1433040829,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=1505086411,osd_snap=663696334}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=28} crc {logm=3427667406,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=409629042,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=3297165161,osd_pg_creating=1505086411,osd_snap=3403871985}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.931+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1016581720,osdmap=316403177}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3276370959,osdmap=131557512}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1016581720,osdmap=316403177}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=2,osdmap=98} crc {osd_snap=3276370959,osdmap=479007170}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1912848194}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1256846273}) 2024-09-10T16:10:58.940 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1912848194}) 2024-09-10T16:10:58.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.932+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1604380232}) 2024-09-10T16:10:58.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=852372327}) 2024-09-10T16:10:58.941 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=56} crc {osdmap=2083352417}) 2024-09-10T16:10:58.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:10:58.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=852372327}) 2024-09-10T16:10:58.942 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:10:58.935+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=58} crc {osdmap=4283929196}) 2024-09-10T16:10:58.942 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:10:58.957 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:59.017 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:59.076 INFO:tasks.rados.rados.0.smithi165.stdout:635: finishing write tid 1 to smithi16534841-29 2024-09-10T16:10:59.076 INFO:tasks.rados.rados.0.smithi165.stdout:635: finishing write tid 2 to smithi16534841-29 2024-09-10T16:10:59.076 INFO:tasks.rados.rados.0.smithi165.stdout:635: finishing write tid 3 to smithi16534841-29 2024-09-10T16:10:59.077 INFO:tasks.rados.rados.0.smithi165.stdout:633: done (4 left) 2024-09-10T16:10:59.077 INFO:tasks.rados.rados.0.smithi165.stdout:634: done (3 left) 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:637: done (2 left) 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:638: write (excl) oid 49 current snap is 76 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:635: finishing write tid 4 to smithi16534841-29 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:635: oid 29 updating version 0 to 450 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:635: oid 29 updating version 450 to 451 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:635: oid 29 version 451 is already newer than 449 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 451 (ObjNum 250 snap 76 seq_num 250) dirty exists 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:635: left oid 29 (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:636: finishing write tid 1 to smithi16534841-47 2024-09-10T16:10:59.078 INFO:tasks.rados.rados.0.smithi165.stdout:638: seq_num 253 ranges {652171=477646,1821709=430462} 2024-09-10T16:10:59.082 INFO:tasks.rados.rados.0.smithi165.stdout:638: writing smithi16534841-49 from 652171 to 1129817 tid 1 2024-09-10T16:10:59.088 INFO:tasks.rados.rados.0.smithi165.stdout:638: writing smithi16534841-49 from 1821709 to 2252171 tid 2 2024-09-10T16:10:59.088 INFO:tasks.rados.rados.0.smithi165.stdout:635: done (2 left) 2024-09-10T16:10:59.088 INFO:tasks.rados.rados.0.smithi165.stdout:639: read oid 32 snap 63 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:639: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:640: write (excl) oid 39 current snap is 76 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: finishing write tid 2 to smithi16534841-47 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: finishing write tid 3 to smithi16534841-47 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: finishing write tid 4 to smithi16534841-47 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: oid 47 updating version 0 to 454 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: oid 47 updating version 454 to 455 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: oid 47 version 455 is already newer than 453 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 455 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:636: left oid 47 (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:10:59.089 INFO:tasks.rados.rados.0.smithi165.stdout:640: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-10T16:10:59.094 INFO:tasks.rados.rados.0.smithi165.stdout:640: writing smithi16534841-39 from 668978 to 1221873 tid 1 2024-09-10T16:10:59.101 INFO:tasks.rados.rados.0.smithi165.stdout:640: writing smithi16534841-39 from 1763838 to 2268978 tid 2 2024-09-10T16:10:59.101 INFO:tasks.rados.rados.0.smithi165.stdout:636: done (3 left) 2024-09-10T16:10:59.101 INFO:tasks.rados.rados.0.smithi165.stdout:641: write (excl) oid 30 current snap is 76 2024-09-10T16:10:59.101 INFO:tasks.rados.rados.0.smithi165.stdout:638: finishing write tid 1 to smithi16534841-49 2024-09-10T16:10:59.101 INFO:tasks.rados.rados.0.smithi165.stdout:639: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:10:59.192 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:59.241 INFO:tasks.rados.rados.0.smithi165.stdout:638: finishing write tid 2 to smithi16534841-49 2024-09-10T16:10:59.241 INFO:tasks.rados.rados.0.smithi165.stdout:638: finishing write tid 3 to smithi16534841-49 2024-09-10T16:10:59.241 INFO:tasks.rados.rados.0.smithi165.stdout:638: finishing write tid 4 to smithi16534841-49 2024-09-10T16:10:59.241 INFO:tasks.rados.rados.0.smithi165.stdout:638: oid 49 updating version 0 to 256 2024-09-10T16:10:59.241 INFO:tasks.rados.rados.0.smithi165.stdout:638: oid 49 version 256 is already newer than 255 2024-09-10T16:10:59.242 INFO:tasks.rados.rados.0.smithi165.stdout:638: oid 49 updating version 256 to 257 2024-09-10T16:10:59.242 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 257 (ObjNum 253 snap 76 seq_num 253) dirty exists 2024-09-10T16:10:59.242 INFO:tasks.rados.rados.0.smithi165.stdout:638: left oid 49 (ObjNum 253 snap 76 seq_num 253) 2024-09-10T16:10:59.242 INFO:tasks.rados.rados.0.smithi165.stdout:640: finishing write tid 1 to smithi16534841-39 2024-09-10T16:10:59.242 INFO:tasks.rados.rados.0.smithi165.stdout:641: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-10T16:10:59.248 INFO:tasks.rados.rados.0.smithi165.stdout:641: writing smithi16534841-30 from 685785 to 1313929 tid 1 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:641: writing smithi16534841-30 from 1822320 to 2285785 tid 2 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:638: done (3 left) 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:639: done (2 left) 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:642: delete oid 42 current snap is 76 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: finishing write tid 2 to smithi16534841-39 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: finishing write tid 3 to smithi16534841-39 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: finishing write tid 4 to smithi16534841-39 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: oid 39 updating version 0 to 407 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: oid 39 updating version 407 to 408 2024-09-10T16:10:59.254 INFO:tasks.rados.rados.0.smithi165.stdout:640: oid 39 updating version 408 to 409 2024-09-10T16:10:59.255 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 409 (ObjNum 254 snap 76 seq_num 254) dirty exists 2024-09-10T16:10:59.255 INFO:tasks.rados.rados.0.smithi165.stdout:640: left oid 39 (ObjNum 254 snap 76 seq_num 254) 2024-09-10T16:10:59.255 INFO:tasks.rados.rados.0.smithi165.stdout:641: finishing write tid 1 to smithi16534841-30 2024-09-10T16:10:59.256 INFO:tasks.rados.rados.0.smithi165.stdout:640: done (2 left) 2024-09-10T16:10:59.256 INFO:tasks.rados.rados.0.smithi165.stdout:642: done (1 left) 2024-09-10T16:10:59.256 INFO:tasks.rados.rados.0.smithi165.stdout:643: delete oid 34 current snap is 76 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: finishing write tid 2 to smithi16534841-30 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: finishing write tid 3 to smithi16534841-30 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: finishing write tid 4 to smithi16534841-30 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: oid 30 updating version 0 to 571 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: oid 30 updating version 571 to 573 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: oid 30 version 573 is already newer than 572 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 573 (ObjNum 255 snap 76 seq_num 255) dirty exists 2024-09-10T16:10:59.257 INFO:tasks.rados.rados.0.smithi165.stdout:641: left oid 30 (ObjNum 255 snap 76 seq_num 255) 2024-09-10T16:10:59.258 INFO:tasks.rados.rados.0.smithi165.stdout:641: done (1 left) 2024-09-10T16:10:59.258 INFO:tasks.rados.rados.0.smithi165.stdout:643: done (0 left) 2024-09-10T16:10:59.258 INFO:tasks.rados.rados.0.smithi165.stdout:644: write (excl) oid 30 current snap is 76 2024-09-10T16:10:59.258 INFO:tasks.rados.rados.0.smithi165.stdout:644: seq_num 258 ranges {736206=453891,1881413=454793} 2024-09-10T16:10:59.265 INFO:tasks.rados.rados.0.smithi165.stdout:644: writing smithi16534841-30 from 736206 to 1190097 tid 1 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:644: writing smithi16534841-30 from 1881413 to 2336206 tid 2 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:645: copy_from oid 28 from oid 46 current snap is 76 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:644: finishing write tid 1 to smithi16534841-30 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:646: rollback oid 8 current snap is 76 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 68 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:647: copy_from oid 37 from oid 19 current snap is 76 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:648: read oid 41 snap -1 2024-09-10T16:10:59.272 INFO:tasks.rados.rados.0.smithi165.stdout:648: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:10:59.273 INFO:tasks.rados.rados.0.smithi165.stdout:649: delete oid 48 current snap is 76 2024-09-10T16:10:59.273 INFO:tasks.rados.rados.0.smithi165.stdout:646: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:10:59.273 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 353 (ObjNum 33 snap 0 seq_num 1127806872) dirty dne 2024-09-10T16:10:59.276 INFO:tasks.rados.rados.0.smithi165.stdout:644: finishing write tid 2 to smithi16534841-30 2024-09-10T16:10:59.276 INFO:tasks.rados.rados.0.smithi165.stdout:644: finishing write tid 3 to smithi16534841-30 2024-09-10T16:10:59.276 INFO:tasks.rados.rados.0.smithi165.stdout:648: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:10:59.322 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:59.375 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:10:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:644: finishing write tid 4 to smithi16534841-30 2024-09-10T16:10:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:644: oid 30 updating version 0 to 576 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:644: oid 30 version 576 is already newer than 575 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:644: oid 30 version 576 is already newer than 574 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 576 (ObjNum 258 snap 76 seq_num 258) dirty exists 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:644: left oid 30 (ObjNum 258 snap 76 seq_num 258) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:645: finishing copy_from to smithi16534841-28 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:645: got expected ENOENT (src dne) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:645: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:647: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:644: done (5 left) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:645: done (4 left) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:646: done (3 left) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:648: done (2 left) 2024-09-10T16:10:59.429 INFO:tasks.rados.rados.0.smithi165.stdout:649: done (1 left) 2024-09-10T16:10:59.430 INFO:tasks.rados.rados.0.smithi165.stdout:650: snap_remove snap 72 2024-09-10T16:10:59.828 INFO:tasks.rados.rados.0.smithi165.stdout:650: done (1 left) 2024-09-10T16:10:59.828 INFO:tasks.rados.rados.0.smithi165.stdout:651: delete oid 48 current snap is 76 2024-09-10T16:10:59.828 INFO:tasks.rados.rados.0.smithi165.stdout:647: finishing copy_from to smithi16534841-37 2024-09-10T16:10:59.828 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 542 (ObjNum 246 snap 73 seq_num 246) dirty exists 2024-09-10T16:10:59.834 INFO:tasks.rados.rados.0.smithi165.stdout:647: done (1 left) 2024-09-10T16:10:59.834 INFO:tasks.rados.rados.0.smithi165.stdout:651: done (0 left) 2024-09-10T16:10:59.834 INFO:tasks.rados.rados.0.smithi165.stdout:652: rollback oid 45 current snap is 76 2024-09-10T16:10:59.834 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 75 2024-09-10T16:10:59.834 INFO:tasks.rados.rados.0.smithi165.stdout:653: snap_remove snap 68 2024-09-10T16:10:59.943 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:10:59.944 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:00.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 10 monclient: get_monmap_and_config 2024-09-10T16:11:00.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 10 monclient: monmap: 2024-09-10T16:11:00.042 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:00.041919+0000 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:00.041919+0000 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:00.043 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 1 Processor -- start 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 1 -- start start 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.040+0000 7f9d45616640 10 monclient: init 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9d4009d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9d4009ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9d4009e420 0x7f9d4013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d400a0870 con 0x7f9d4009ab50 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d40074c70 con 0x7f9d4009d930 2024-09-10T16:11:00.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9d4009d3e0 con 0x7f9d4009e420 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d45616640 10 monclient(hunting): authenticate will time out at 1038.529312s 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d37fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9d4009ab50 0x7f9d401397c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42108/0 (socket says 172.21.15.164:42108) 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d37fff640 1 -- 172.21.15.164:0/2382405211 learned_addr learned my addr 172.21.15.164:0/2382405211 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2382405211 >> v2:172.21.15.165:3300/0 conn(0x7f9d4009e420 0x7f9d4013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f9d4009e420 auth_method 0 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_done global_id 44194 payload 274 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2382405211 >> v1:172.21.15.165:6789/0 conn(0x7f9d4009d930 legacy=0x7f9d4009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2382405211 >> v1:172.21.15.164:6789/0 conn(0x7f9d4009ab50 legacy=0x7f9d401397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.041+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2382405211 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4009ecb0 con 0x7f9d4009e420 2024-09-10T16:11:00.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.044+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2382405211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 msgr2=0x7f9d4013cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.044+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2382405211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d4013cef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009d90 tx=0x7f9d3802b5a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=47c57b36cd23d178 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 1 -- 172.21.15.164:0/2382405211 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38007a30 con 0x7f9d4009e420 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: dump: 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.055 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 1 -- 172.21.15.164:0/2382405211 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d38007bd0 con 0x7f9d4009e420 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 5 monclient: authenticate success, global_id 44194 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 1 -- 172.21.15.164:0/2382405211 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38032400 con 0x7f9d4009e420 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 4 set_mon_vals no callback set 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d3e7fc640 10 monclient: dump: 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.057 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 10 monclient: shutdown 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 -- 172.21.15.164:0/2382405211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 msgr2=0x7f9d4013cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 --2- 172.21.15.164:0/2382405211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d4013cef0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9d38009d90 tx=0x7f9d3802b5a0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 -- 172.21.15.164:0/2382405211 shutdown_connections 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 --2- 172.21.15.164:0/2382405211 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d4013cef0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 -- 172.21.15.164:0/2382405211 >> 172.21.15.164:0/2382405211 conn(0x7f9d400a3110 msgr2=0x7f9d400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 -- 172.21.15.164:0/2382405211 shutdown_connections 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.045+0000 7f9d45616640 1 -- 172.21.15.164:0/2382405211 wait complete. 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 10 monclient: monmap: 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:00.058 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 1 Processor -- start 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 1 -- start start 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 10 monclient: init 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.046+0000 7f9d45616640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 0x7f9d40134940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-c con 0x7f9d4009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-a con 0x7f9d4009d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d401caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): picked mon.noname-b con 0x7f9d4009e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9d400a2940 con 0x7f9d4009d930 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4009ef30 con 0x7f9d4009e420 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9d400a2ac0 con 0x7f9d4009ab50 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d45616640 10 monclient(hunting): authenticate will time out at 1038.535062s 2024-09-10T16:11:00.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 0x7f9d40134940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 0x7f9d40134940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40148/0 (socket says 172.21.15.164:40148) 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38338/0 (socket says 172.21.15.164:38338) 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 1 -- 172.21.15.164:0/2076822781 learned_addr learned my addr 172.21.15.164:0/2076822781 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): get_auth_request con 0x7f9d4009ab50 auth_method 0 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d401caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): get_auth_request con 0x7f9d4009d930 auth_method 0 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): get_auth_request con 0x7f9d4009e420 auth_method 0 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient(hunting): handle_auth_done global_id 44197 payload 824 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 msgr2=0x7f9d40134940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 0x7f9d40134940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 msgr2=0x7f9d40134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 monclient: found mon.noname-b 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2076822781 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4009ca80 con 0x7f9d4009e420 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.047+0000 7f9d37fff640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:00.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.049+0000 7f9d3f7fe640 1 -- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 msgr2=0x7f9d401caf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.049+0000 7f9d3f7fe640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d401caf10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d4006d4d0 tx=0x7f9d38004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38042070 con 0x7f9d4009e420 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: dump: 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.064 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9d3803c440 con 0x7f9d4009e420 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 5 monclient: authenticate success, global_id 44197 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 10 monclient: _renew_subs 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4009d620 con 0x7f9d4009e420 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3803c830 con 0x7f9d4009e420 2024-09-10T16:11:00.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d377fe640 10 monclient: dump: 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d3dffb640 4 set_mon_vals no callback set 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 10 monclient: _renew_subs 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.050+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4012cbc0 con 0x7f9d4009e420 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9d38045070 con 0x7f9d4009e420 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d45616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d45616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d45616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4009dd90 con 0x7f9d4009e420 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d377fe640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 0x7f9d20040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7f9d38073ed0 con 0x7f9d4009e420 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d3effd640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 0x7f9d20040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.051+0000 7f9d3effd640 10 monclient: get_auth_request con 0x7f9d2003e290 auth_method 0 2024-09-10T16:11:00.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.054+0000 7f9d3effd640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 0x7f9d20040750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d30005020 tx=0x7f9d30007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.056+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9d3803c9d0 con 0x7f9d4009e420 2024-09-10T16:11:00.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.056+0000 7f9d377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.056+0000 7f9d377fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:00.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.301+0000 7f9d45616640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:00.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d45616640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:00.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d45616640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:11:00.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9d4009dd90 con 0x7f9d4009e420 2024-09-10T16:11:00.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d377fe640 1 -- 172.21.15.164:0/2076822781 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f9d38072890 con 0x7f9d4009e420 2024-09-10T16:11:00.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:00.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.302+0000 7f9d377fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:00.305 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 msgr2=0x7f9d20040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 0x7f9d20040750 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9d30005020 tx=0x7f9d30007ef0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 10 monclient: shutdown 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 msgr2=0x7f9d401caf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d401caf10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9d4006d4d0 tx=0x7f9d38004a40 comp rx=0 tx=0).stop 2024-09-10T16:11:00.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 shutdown_connections 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9d2003e290 0x7f9d20040750 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9d4009e420 0x7f9d401caf10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9d4009d930 0x7f9d40134e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 --2- 172.21.15.164:0/2076822781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9d4009ab50 0x7f9d40134940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 >> 172.21.15.164:0/2076822781 conn(0x7f9d400a3110 msgr2=0x7f9d4009b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 shutdown_connections 2024-09-10T16:11:00.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.304+0000 7f9d45616640 1 -- 172.21.15.164:0/2076822781 wait complete. 2024-09-10T16:11:00.317 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:11:00.317 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:11:00.318 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:11:00.318 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:11:00.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.414+0000 7fb28e394640 10 monclient: get_monmap_and_config 2024-09-10T16:11:00.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.414+0000 7fb28e394640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.415+0000 7fb28e394640 10 monclient: monmap: 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:00.416889+0000 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:00.416889+0000 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:00.418 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:00.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.415+0000 7fb28e394640 1 Processor -- start 2024-09-10T16:11:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.415+0000 7fb28e394640 1 -- start start 2024-09-10T16:11:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.415+0000 7fb28e394640 10 monclient: init 2024-09-10T16:11:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb28806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-b con 0x7fb2880a3fd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb28813d580 0x7fb28813f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-c con 0x7fb28813d580 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2880a0190 con 0x7fb28806cb70 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb28806ea00 con 0x7fb28813d580 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb2880a4960 con 0x7fb2880a3fd0 2024-09-10T16:11:00.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28e394640 10 monclient(hunting): authenticate will time out at 1038.904393s 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb287fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb28806cb70 0x7fb28806cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42112/0 (socket says 172.21.15.164:42112) 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40164/0 (socket says 172.21.15.164:40164) 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb287fff640 1 -- 172.21.15.164:0/2227063759 learned_addr learned my addr 172.21.15.164:0/2227063759 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): get_auth_request con 0x7fb2880a3fd0 auth_method 0 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28c931640 1 --2- 172.21.15.164:0/2227063759 >> v2:172.21.15.165:3300/0 conn(0x7fb28813d580 0x7fb28813f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28c931640 10 monclient(hunting): get_auth_request con 0x7fb28813d580 auth_method 0 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28c931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28c931640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb28c931640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient(hunting): handle_auth_done global_id 4826 payload 274 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 -- 172.21.15.164:0/2227063759 >> v2:172.21.15.165:3300/0 conn(0x7fb28813d580 msgr2=0x7fb28813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 --2- 172.21.15.164:0/2227063759 >> v2:172.21.15.165:3300/0 conn(0x7fb28813d580 0x7fb28813f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 -- 172.21.15.164:0/2227063759 >> v1:172.21.15.164:6789/0 conn(0x7fb28806cb70 legacy=0x7fb28806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 monclient: found mon.noname-b 2024-09-10T16:11:00.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.416+0000 7fb2877fe640 1 -- 172.21.15.164:0/2227063759 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2880a45b0 con 0x7fb2880a3fd0 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb2877fe640 1 -- 172.21.15.164:0/2227063759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 msgr2=0x7fb28806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb2877fe640 1 --2- 172.21.15.164:0/2227063759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0097a0 tx=0x7fb27c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52871af2fe0c44f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 1 -- 172.21.15.164:0/2227063759 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c037070 con 0x7fb2880a3fd0 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: dump: 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb28e394640 5 monclient: authenticate success, global_id 4826 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 1 -- 172.21.15.164:0/2227063759 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb27c032440 con 0x7fb2880a3fd0 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb28e394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb28e394640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:00.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 1 -- 172.21.15.164:0/2227063759 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27c032810 con 0x7fb2880a3fd0 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb28e394640 4 set_mon_vals no callback set 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.419+0000 7fb286ffd640 10 monclient: dump: 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.435 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 10 monclient: shutdown 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 -- 172.21.15.164:0/2227063759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 msgr2=0x7fb28806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 --2- 172.21.15.164:0/2227063759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fb27c0097a0 tx=0x7fb27c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 -- 172.21.15.164:0/2227063759 shutdown_connections 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 --2- 172.21.15.164:0/2227063759 >> v2:172.21.15.165:3300/0 conn(0x7fb28813d580 0x7fb28813f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 --2- 172.21.15.164:0/2227063759 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb2880a3fd0 0x7fb28806d660 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 -- 172.21.15.164:0/2227063759 >> 172.21.15.164:0/2227063759 conn(0x7fb2880a4ee0 msgr2=0x7fb2880a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 -- 172.21.15.164:0/2227063759 shutdown_connections 2024-09-10T16:11:00.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 1 -- 172.21.15.164:0/2227063759 wait complete. 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.420+0000 7fb28e394640 10 monclient: monmap: 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 Processor -- start 2024-09-10T16:11:00.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 -- start start 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient: init 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb28806cb70 0x7fb2881cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-b con 0x7fb28806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 0x7fb28809aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-a con 0x7fb2880a3fd0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 0x7fb28809af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): picked mon.noname-c con 0x7fb28813d580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb28806dba0 con 0x7fb2880a3fd0 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb28809be30 con 0x7fb28806cb70 2024-09-10T16:11:00.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb28806dd20 con 0x7fb28813d580 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28e394640 10 monclient(hunting): authenticate will time out at 1038.909843s 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 0x7fb28809af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 0x7fb28809aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 0x7fb28809af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40168/0 (socket says 172.21.15.164:40168) 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 0x7fb28809aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38342/0 (socket says 172.21.15.164:38342) 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 learned_addr learned my addr 172.21.15.164:0/205124729 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 10 monclient(hunting): get_auth_request con 0x7fb28813d580 auth_method 0 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb28c931640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 10 monclient(hunting): get_auth_request con 0x7fb2880a3fd0 auth_method 0 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.421+0000 7fb2877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb28806cb70 0x7fb2881cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb2877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): get_auth_request con 0x7fb28806cb70 auth_method 0 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient(hunting): handle_auth_done global_id 4829 payload 824 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb28806cb70 msgr2=0x7fb2881cd3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb28806cb70 0x7fb2881cd3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 msgr2=0x7fb28809aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 0x7fb28809aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 monclient: found mon.noname-c 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2880924c0 con 0x7fb28813d580 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.422+0000 7fb2877fe640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2880a3fd0 0x7fb28809aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 msgr2=0x7fb28809af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28c931640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 0x7fb28809af80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb280007e40 tx=0x7fb28000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280013790 con 0x7fb28813d580 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: dump: 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.442 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb280019070 con 0x7fb28813d580 2024-09-10T16:11:00.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 5 monclient: authenticate success, global_id 4829 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 10 monclient: _renew_subs 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28809f5d0 con 0x7fb28813d580 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb280013b90 con 0x7fb28813d580 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb284ff9640 10 monclient: dump: 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.444 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 10 monclient: _renew_subs 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb2867fc640 4 set_mon_vals no callback set 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.424+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb288073e70 con 0x7fb28813d580 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb28e394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb28e394640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 0x7fb2881ca220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb28809b6d0 con 0x7fb2881c9e40 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 0x7fb2881ca220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb28001e460 con 0x7fb28813d580 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient: get_auth_request con 0x7fb2881c9e40 auth_method 0 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 10 monclient(hunting): handle_auth_done global_id 54132 payload 824 2024-09-10T16:11:00.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb284ff9640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb27403e210 0x7fb2740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb287fff640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb28806a630 con 0x7fb2881c9e40 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.425+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7fb2800536e0 con 0x7fb28813d580 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.426+0000 7fb2877fe640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb27403e210 0x7fb2740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.426+0000 7fb2877fe640 10 monclient: get_auth_request con 0x7fb27403e210 auth_method 0 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb287fff640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 msgr2=0x7fb2881ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb287fff640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 0x7fb2881ca220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb27800ea70 tx=0x7fb27800ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27801a070 con 0x7fb2881c9e40 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 10 monclient: dump: 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb27800cc50 con 0x7fb2881c9e40 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 msgr2=0x7fb2881ca220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 0x7fb2881ca220 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb27800ea70 tx=0x7fb27800ef30 comp rx=0 tx=0).stop 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 msgr2=0x7fb2881ca220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb284ff9640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881c9e40 0x7fb2881ca220 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.428+0000 7fb2877fe640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb27403e210 0x7fb2740406d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb27c034000 tx=0x7fb27c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb28e394640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb28e394640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 0x7fb28809c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb28e394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb2881cdd90 con 0x7fb2881ca760 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 0x7fb28809c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient: get_auth_request con 0x7fb2881ca760 auth_method 0 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 10 monclient(hunting): handle_auth_done global_id 54135 payload 824 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.434+0000 7fb287fff640 1 -- 172.21.15.164:0/205124729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2880a92f0 con 0x7fb2881ca760 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb287fff640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 msgr2=0x7fb28809c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb287fff640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 0x7fb28809c270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb278017000 tx=0x7fb278004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb27801a050 con 0x7fb2881ca760 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 10 monclient: dump: 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.451 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fb27801b040 con 0x7fb2881ca760 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 msgr2=0x7fb28809c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 0x7fb28809c270 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb278017000 tx=0x7fb278004940 comp rx=0 tx=0).stop 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 msgr2=0x7fb28809c270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb284ff9640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb2881ca760 0x7fb28809c270 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb278017000 tx=0x7fb278004940 comp rx=0 tx=0).stop 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 30622, 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:00.453 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.454 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.455 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:00.456 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.457 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.458 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.460 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.461 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:00.462 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.464 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:00.465 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb27403e210 msgr2=0x7fb2740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb27403e210 0x7fb2740406d0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb27c034000 tx=0x7fb27c02e000 comp rx=0 tx=0).stop 2024-09-10T16:11:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 10 monclient: shutdown 2024-09-10T16:11:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 msgr2=0x7fb28809af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 --2- 172.21.15.164:0/205124729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb28813d580 0x7fb28809af80 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fb280007e40 tx=0x7fb28000e4b0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 reap_dead start 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28c931640 1 -- 172.21.15.164:0/205124729 reap_dead start 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 shutdown_connections 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 >> 172.21.15.164:0/205124729 conn(0x7fb2880a4ee0 msgr2=0x7fb28813c8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 shutdown_connections 2024-09-10T16:11:00.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.437+0000 7fb28e394640 1 -- 172.21.15.164:0/205124729 wait complete. 2024-09-10T16:11:00.467 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:00.467 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:11:00.467 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:11:00.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 10 monclient: get_monmap_and_config 2024-09-10T16:11:00.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 10 monclient: monmap: 2024-09-10T16:11:00.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:00.566905+0000 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:00.566905+0000 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:00.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 1 Processor -- start 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 1 -- start start 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.565+0000 7ff4c8c9b640 10 monclient: init 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff4c4074af0 0x7ff4c4137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c4074af0 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a1090 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4c40a2ec0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4c409ebc0 con 0x7ff4c40a2ec0 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff4c40698a0 con 0x7ff4c4074af0 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c406bfb0 con 0x7ff4c40a1090 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c8c9b640 10 monclient(hunting): authenticate will time out at 1039.054233s 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40184/0 (socket says 172.21.15.164:40184) 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 -- 172.21.15.164:0/1110325685 learned_addr learned my addr 172.21.15.164:0/1110325685 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1090 auth_method 0 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c259c640 1 --2- 172.21.15.164:0/1110325685 >> v2:172.21.15.165:3300/0 conn(0x7ff4c4074af0 0x7ff4c4137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c259c640 10 monclient(hunting): get_auth_request con 0x7ff4c4074af0 auth_method 0 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 -- 172.21.15.164:0/1110325685 >> v2:172.21.15.165:3300/0 conn(0x7ff4c4074af0 msgr2=0x7ff4c4137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/1110325685 >> v2:172.21.15.165:3300/0 conn(0x7ff4c4074af0 0x7ff4c4137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 -- 172.21.15.164:0/1110325685 >> v1:172.21.15.165:6789/0 conn(0x7ff4c40a2ec0 legacy=0x7ff4c40a3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 monclient: found mon.noname-b 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.566+0000 7ff4bbfff640 1 -- 172.21.15.164:0/1110325685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c41383b0 con 0x7ff4c40a1090 2024-09-10T16:11:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4bbfff640 1 -- 172.21.15.164:0/1110325685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 msgr2=0x7ff4c40a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/1110325685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009870 tx=0x7ff4ac02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a823ef68fe01ca9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 1 -- 172.21.15.164:0/1110325685 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac037070 con 0x7ff4c40a1090 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: dump: 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 1 -- 172.21.15.164:0/1110325685 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4ac032440 con 0x7ff4c40a1090 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c8c9b640 5 monclient: authenticate success, global_id 4832 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c8c9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c8c9b640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c8c9b640 4 set_mon_vals no callback set 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 1 -- 172.21.15.164:0/1110325685 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac032810 con 0x7ff4c40a1090 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.569+0000 7ff4c1d9b640 10 monclient: dump: 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 10 monclient: shutdown 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/1110325685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 msgr2=0x7ff4c40a14d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/1110325685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009870 tx=0x7ff4ac02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/1110325685 shutdown_connections 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/1110325685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c40a1090 0x7ff4c40a14d0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/1110325685 >> v2:172.21.15.165:3300/0 conn(0x7ff4c4074af0 0x7ff4c4137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/1110325685 >> 172.21.15.164:0/1110325685 conn(0x7ff4c40a3fb0 msgr2=0x7ff4c40a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/1110325685 shutdown_connections 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/1110325685 wait complete. 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.570+0000 7ff4c8c9b640 10 monclient: monmap: 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 Processor -- start 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 -- start start 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient: init 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 0x7ff4c41cd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-c con 0x7ff4c4074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4c40a1090 0x7ff4c41cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-a con 0x7ff4c40a1090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c40a2ec0 0x7ff4c41be100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): picked mon.noname-b con 0x7ff4c40a2ec0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c40a3c50 con 0x7ff4c40a1090 2024-09-10T16:11:00.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4c4068d70 con 0x7ff4c40a2ec0 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c41c03d0 con 0x7ff4c4074af0 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c8c9b640 10 monclient(hunting): authenticate will time out at 1039.059696s 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 0x7ff4c41cd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4c40a1090 0x7ff4c41cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 0x7ff4c41cd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40198/0 (socket says 172.21.15.164:40198) 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 1 -- 172.21.15.164:0/858548397 learned_addr learned my addr 172.21.15.164:0/858548397 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): get_auth_request con 0x7ff4c4074af0 auth_method 0 2024-09-10T16:11:00.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request con 0x7ff4c40a1090 auth_method 0 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c2d9d640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c40a2ec0 0x7ff4c41be100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request con 0x7ff4c40a2ec0 auth_method 0 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.571+0000 7ff4c2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 10 monclient(hunting): handle_auth_done global_id 4835 payload 824 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c40a2ec0 msgr2=0x7ff4c41be100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4c40a2ec0 0x7ff4c41be100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4c40a1090 msgr2=0x7ff4c41cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4c40a1090 0x7ff4c41cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 monclient: found mon.noname-c 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4c259c640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4c409e6a0 con 0x7ff4c4074af0 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.572+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4c40a1090 0x7ff4c41cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c259c640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 msgr2=0x7ff4c41cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c259c640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 0x7ff4c41cd3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400d620 tx=0x7ff4b400dae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4012800 con 0x7ff4c4074af0 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: dump: 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4019070 con 0x7ff4c4074af0 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 5 monclient: authenticate success, global_id 4835 2024-09-10T16:11:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 10 monclient: _renew_subs 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4c41c0890 con 0x7ff4c4074af0 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4012be0 con 0x7ff4c4074af0 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4baffd640 10 monclient: dump: 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c159a640 4 set_mon_vals no callback set 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 10 monclient: _renew_subs 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.574+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4c41bf930 con 0x7ff4c4074af0 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c8c9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 0x7ff4c41be640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c41bfb30 con 0x7ff4c41cde20 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff4b401e460 con 0x7ff4c4074af0 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 0x7ff4c41be640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient: get_auth_request con 0x7ff4c41cde20 auth_method 0 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_done global_id 4838 payload 824 2024-09-10T16:11:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4bbfff640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4c41c17a0 con 0x7ff4c41cde20 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4baffd640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a403e210 0x7ff4a40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7ff4b40531a0 con 0x7ff4c4074af0 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c2d9d640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a403e210 0x7ff4a40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.575+0000 7ff4c2d9d640 10 monclient: get_auth_request con 0x7ff4a403e210 auth_method 0 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.577+0000 7ff4bbfff640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 msgr2=0x7ff4c41be640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 0x7ff4c41be640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac034000 tx=0x7ff4ac0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac037040 con 0x7ff4c41cde20 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 10 monclient: dump: 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4c2d9d640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a403e210 0x7ff4a40406d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0055e0 tx=0x7ff4bc003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff4ac03c860 con 0x7ff4c41cde20 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 msgr2=0x7ff4c41be640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 0x7ff4c41be640 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac034000 tx=0x7ff4ac0046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 msgr2=0x7ff4c41be640 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.578+0000 7ff4baffd640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41cde20 0x7ff4c41be640 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4c8c9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 0x7ff4c409b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4c8c9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4c41bef60 con 0x7ff4c41beb80 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 0x7ff4c409b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient: get_auth_request con 0x7ff4c41beb80 auth_method 0 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.584+0000 7ff4bbfff640 1 -- 172.21.15.164:0/858548397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff4c4137e70 con 0x7ff4c41beb80 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4bbfff640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 msgr2=0x7ff4c409b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4bbfff640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 0x7ff4c409b3b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4ac037030 con 0x7ff4c41beb80 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 10 monclient: dump: 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7ff4ac003d20 con 0x7ff4c41beb80 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 msgr2=0x7ff4c409b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 0x7ff4c409b3b0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac008030 comp rx=0 tx=0).stop 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 msgr2=0x7ff4c409b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4baffd640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c41beb80 0x7ff4c409b3b0 secure :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac000c00 tx=0x7ff4ac008030 comp rx=0 tx=0).stop 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 302759, 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.604 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.605 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.606 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.607 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.608 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:00.609 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:00.617 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a403e210 msgr2=0x7ff4a40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a403e210 0x7ff4a40406d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0055e0 tx=0x7ff4bc003d10 comp rx=0 tx=0).stop 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 10 monclient: shutdown 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 msgr2=0x7ff4c41cd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 --2- 172.21.15.164:0/858548397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4c4074af0 0x7ff4c41cd3a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7ff4b400d620 tx=0x7ff4b400dae0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c2d9d640 1 -- 172.21.15.164:0/858548397 reap_dead start 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c2d9d640 1 -- 172.21.15.164:0/858548397 reap_dead start 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 shutdown_connections 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 >> 172.21.15.164:0/858548397 conn(0x7ff4c40a3fb0 msgr2=0x7ff4c41328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 shutdown_connections 2024-09-10T16:11:00.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.587+0000 7ff4c8c9b640 1 -- 172.21.15.164:0/858548397 wait complete. 2024-09-10T16:11:00.620 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:00.620 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:11:00.620 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:11:00.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.715+0000 7f91545da640 10 monclient: get_monmap_and_config 2024-09-10T16:11:00.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.715+0000 7f91545da640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.716+0000 7f91545da640 10 monclient: monmap: 2024-09-10T16:11:00.718 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:00.717885+0000 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:00.717885+0000 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:00.719 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.716+0000 7f91545da640 1 Processor -- start 2024-09-10T16:11:00.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.716+0000 7f91545da640 1 -- start start 2024-09-10T16:11:00.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.716+0000 7f91545da640 10 monclient: init 2024-09-10T16:11:00.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f914c09ce10 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f914c09d1f0 0x7f914c09d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): picked mon.noname-a con 0x7f914c09d1f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f914c13d230 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914c0a3620 con 0x7f914c09ce10 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914c0a1c90 con 0x7f914c13d230 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f914c073ee0 con 0x7f914c09d1f0 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f91545da640 10 monclient(hunting): authenticate will time out at 1039.205339s 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f914c09d1f0 0x7f914c09d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9152376640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f914c09ce10 0x7f914c0a2f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42118/0 (socket says 172.21.15.164:42118) 2024-09-10T16:11:00.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9152376640 1 -- 172.21.15.164:0/2529733689 learned_addr learned my addr 172.21.15.164:0/2529733689 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): get_auth_request con 0x7f914c09d1f0 auth_method 0 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient(hunting): handle_auth_done global_id 54141 payload 274 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 1 -- 172.21.15.164:0/2529733689 >> v1:172.21.15.165:6789/0 conn(0x7f914c13d230 legacy=0x7f914c13f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 1 -- 172.21.15.164:0/2529733689 >> v1:172.21.15.164:6789/0 conn(0x7f914c09ce10 legacy=0x7f914c0a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 1 monclient: found mon.noname-a 2024-09-10T16:11:00.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:00.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.717+0000 7f9151b75640 1 -- 172.21.15.164:0/2529733689 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c0a1f70 con 0x7f914c09d1f0 2024-09-10T16:11:00.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.720+0000 7f9151b75640 1 -- 172.21.15.164:0/2529733689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09d1f0 msgr2=0x7f914c09d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.720+0000 7f9151b75640 1 --2- 172.21.15.164:0/2529733689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09d1f0 0x7f914c09d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f913c0097a0 tx=0x7f913c029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c60ee606e3cba5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.720+0000 7f9151374640 1 -- 172.21.15.164:0/2529733689 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c037070 con 0x7f914c09d1f0 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.720+0000 7f9151374640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: dump: 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 1 -- 172.21.15.164:0/2529733689 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f913c032640 con 0x7f914c09d1f0 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 5 monclient: authenticate success, global_id 54141 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 1 -- 172.21.15.164:0/2529733689 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c032a10 con 0x7f914c09d1f0 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 4 set_mon_vals no callback set 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f9151374640 10 monclient: dump: 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 10 monclient: shutdown 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 -- 172.21.15.164:0/2529733689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09d1f0 msgr2=0x7f914c09d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 --2- 172.21.15.164:0/2529733689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09d1f0 0x7f914c09d600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f913c0097a0 tx=0x7f913c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 -- 172.21.15.164:0/2529733689 shutdown_connections 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 --2- 172.21.15.164:0/2529733689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09d1f0 0x7f914c09d600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 -- 172.21.15.164:0/2529733689 >> 172.21.15.164:0/2529733689 conn(0x7f914c0a7320 msgr2=0x7f914c0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 -- 172.21.15.164:0/2529733689 shutdown_connections 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.721+0000 7f91545da640 1 -- 172.21.15.164:0/2529733689 wait complete. 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient: monmap: 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:00.734 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 1 Processor -- start 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 1 -- start start 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient: init 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 0x7f914c137480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient(hunting): picked mon.noname-a con 0x7f914c09ce10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:00.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f914c09d1f0 0x7f914c1379c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.722+0000 7f91545da640 10 monclient(hunting): picked mon.noname-c con 0x7f914c09d1f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c13d230 0x7f914c11d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): picked mon.noname-b con 0x7f914c13d230 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f914c077060 con 0x7f914c09ce10 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f914c076ee0 con 0x7f914c13d230 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f914c140d40 con 0x7f914c09d1f0 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f91545da640 10 monclient(hunting): authenticate will time out at 1039.211034s 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 0x7f914c137480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 0x7f914c137480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38382/0 (socket says 172.21.15.164:38382) 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 -- 172.21.15.164:0/3887793995 learned_addr learned my addr 172.21.15.164:0/3887793995 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): get_auth_request con 0x7f914c09ce10 auth_method 0 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f914c09d1f0 0x7f914c1379c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): get_auth_request con 0x7f914c09d1f0 auth_method 0 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c13d230 0x7f914c11d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): get_auth_request con 0x7f914c13d230 auth_method 0 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152b77640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient(hunting): handle_auth_done global_id 54144 payload 824 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f914c09d1f0 msgr2=0x7f914c1379c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f914c09d1f0 0x7f914c1379c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c13d230 msgr2=0x7f914c11d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c13d230 0x7f914c11d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 monclient: found mon.noname-a 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9152376640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914c1210b0 con 0x7f914c09ce10 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.723+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f914c09d1f0 0x7f914c1379c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9152376640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 msgr2=0x7f914c137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9152376640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 0x7f914c137480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f914800c460 tx=0x7f914800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148007070 con 0x7f914c09ce10 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: dump: 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9148007230 con 0x7f914c09ce10 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 5 monclient: authenticate success, global_id 54144 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 10 monclient: _renew_subs 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914c121390 con 0x7f914c09ce10 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9148007620 con 0x7f914c09ce10 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9142ffd640 10 monclient: dump: 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f9150b73640 4 set_mon_vals no callback set 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 10 monclient: _renew_subs 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.725+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f914c11fe60 con 0x7f914c09ce10 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f914801b400 con 0x7f914c09ce10 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f91545da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f91545da640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 0x7f914c11e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f914c09d600 con 0x7f914c11dd30 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f9142ffd640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f913003e260 0x7f9130040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7f91480529b0 con 0x7f914c09ce10 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.726+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 0x7f914c11e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient: get_auth_request con 0x7f914c11dd30 auth_method 0 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9152b77640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f913003e260 0x7f9130040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9152b77640 10 monclient: get_auth_request con 0x7f913003e260 auth_method 0 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 10 monclient(hunting): handle_auth_done global_id 44215 payload 824 2024-09-10T16:11:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.727+0000 7f9151b75640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f914c11f8e0 con 0x7f914c11dd30 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9152b77640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f913003e260 0x7f9130040720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f914c073b50 tx=0x7f9144003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9151b75640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 msgr2=0x7f914c11e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 0x7f914c11e110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f914c073e70 tx=0x7f913c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c037040 con 0x7f914c11dd30 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.729+0000 7f9142ffd640 10 monclient: dump: 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f913c03ea80 con 0x7f914c11dd30 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 msgr2=0x7f914c11e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 0x7f914c11e110 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f914c073e70 tx=0x7f913c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 msgr2=0x7f914c11e110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.730+0000 7f9142ffd640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11dd30 0x7f914c11e110 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f91545da640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f91545da640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 0x7f914c09fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f91545da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f914c137f00 con 0x7f914c11e650 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 0x7f914c09fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient: get_auth_request con 0x7f914c11e650 auth_method 0 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 10 monclient(hunting): handle_auth_done global_id 44218 payload 824 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.736+0000 7f9151b75640 1 -- 172.21.15.164:0/3887793995 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f914c11ea60 con 0x7f914c11e650 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9151b75640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 msgr2=0x7f914c09fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9151b75640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 0x7f914c09fe00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a010 tx=0x7f913c008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f913c037050 con 0x7f914c11e650 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 10 monclient: dump: 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f913c051410 con 0x7f914c11e650 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 msgr2=0x7f914c09fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 0x7f914c09fe00 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f913c02a010 tx=0x7f913c008030 comp rx=0 tx=0).stop 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 msgr2=0x7f914c09fe00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9142ffd640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f914c11e650 0x7f914c09fe00 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.755 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 52, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 75891, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:11:00.756 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.757 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:00.759 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:00.760 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:00.769 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:00.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f913003e260 msgr2=0x7f9130040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f913003e260 0x7f9130040720 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f914c073b50 tx=0x7f9144003d10 comp rx=0 tx=0).stop 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 10 monclient: shutdown 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 msgr2=0x7f914c137480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 --2- 172.21.15.164:0/3887793995 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f914c09ce10 0x7f914c137480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f914800c460 tx=0x7f914800c920 comp rx=0 tx=0).stop 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9152b77640 1 -- 172.21.15.164:0/3887793995 reap_dead start 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f9152b77640 1 -- 172.21.15.164:0/3887793995 reap_dead start 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 shutdown_connections 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 >> 172.21.15.164:0/3887793995 conn(0x7f914c0a7320 msgr2=0x7f914c0a5150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 shutdown_connections 2024-09-10T16:11:00.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.739+0000 7f91545da640 1 -- 172.21.15.164:0/3887793995 wait complete. 2024-09-10T16:11:00.772 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:00.772 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:11:00.772 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:11:00.772 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:11:00.772 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:11:00.772 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:11:00.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:11:00.783 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:11:00.784 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:11:00.784 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.878+0000 7efcb266b640 10 monclient: get_monmap_and_config 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.878+0000 7efcb266b640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.878+0000 7efcb266b640 10 monclient: monmap: 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:00.880612+0000 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:00.880612+0000 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:00.882 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 1 Processor -- start 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 1 -- start start 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient: init 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efcac06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efcac0a3fd0 0x7efcac06d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-a con 0x7efcac0a3fd0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:00.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.879+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efcac13d580 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcac0a0190 con 0x7efcac13d580 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efcac0a4960 con 0x7efcac06cb70 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7efcac06ea00 con 0x7efcac0a3fd0 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb266b640 10 monclient(hunting): authenticate will time out at 1039.368032s 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcabfff640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7efcac06cb70 legacy=0x7efcac06cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efcac0a3fd0 0x7efcac06d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb0c08640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efcac13d580 0x7efcac13f970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42134/0 (socket says 172.21.15.164:42134) 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcb0c08640 1 -- 172.21.15.164:0/3672187474 learned_addr learned my addr 172.21.15.164:0/3672187474 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): get_auth_request con 0x7efcac0a3fd0 auth_method 0 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient(hunting): handle_auth_done global_id 54147 payload 274 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 1 -- 172.21.15.164:0/3672187474 >> v1:172.21.15.165:6789/0 conn(0x7efcac06cb70 legacy=0x7efcac06cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 1 -- 172.21.15.164:0/3672187474 >> v1:172.21.15.164:6789/0 conn(0x7efcac13d580 legacy=0x7efcac13f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 1 monclient: found mon.noname-a 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:00.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.880+0000 7efcab7fe640 1 -- 172.21.15.164:0/3672187474 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac0a45b0 con 0x7efcac0a3fd0 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcab7fe640 1 -- 172.21.15.164:0/3672187474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 msgr2=0x7efcac06d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcab7fe640 1 --2- 172.21.15.164:0/3672187474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac06d660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efcac0774b0 tx=0x7efca0029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d73a9c0b34d34a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 1 -- 172.21.15.164:0/3672187474 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0037070 con 0x7efcac0a3fd0 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: dump: 2024-09-10T16:11:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 1 -- 172.21.15.164:0/3672187474 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efca0032440 con 0x7efcac0a3fd0 2024-09-10T16:11:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 5 monclient: authenticate success, global_id 54147 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 1 -- 172.21.15.164:0/3672187474 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efca0032810 con 0x7efcac0a3fd0 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 4 set_mon_vals no callback set 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcaaffd640 10 monclient: dump: 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 10 monclient: shutdown 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 1 -- 172.21.15.164:0/3672187474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 msgr2=0x7efcac06d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.883+0000 7efcb266b640 1 --2- 172.21.15.164:0/3672187474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac06d660 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efcac0774b0 tx=0x7efca0029fb0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 -- 172.21.15.164:0/3672187474 shutdown_connections 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 --2- 172.21.15.164:0/3672187474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac06d660 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 -- 172.21.15.164:0/3672187474 >> 172.21.15.164:0/3672187474 conn(0x7efcac0a4ee0 msgr2=0x7efcac0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 -- 172.21.15.164:0/3672187474 shutdown_connections 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 -- 172.21.15.164:0/3672187474 wait complete. 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 10 monclient: build_initial_monmap 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 10 monclient: monmap: 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.884+0000 7efcb266b640 1 Processor -- start 2024-09-10T16:11:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 -- start start 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient: init 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 0x7efcac09a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-c con 0x7efcac06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-a con 0x7efcac0a3fd0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 0x7efcac0736d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): picked mon.noname-b con 0x7efcac13d580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efcac06dba0 con 0x7efcac0a3fd0 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efcac09be30 con 0x7efcac13d580 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efcac06dd20 con 0x7efcac06cb70 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb266b640 10 monclient(hunting): authenticate will time out at 1039.373618s 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb0c08640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 msgr2=0x7efcac0736d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38402/0 (socket says 172.21.15.164:38402) 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 1 -- 172.21.15.164:0/778785054 learned_addr learned my addr 172.21.15.164:0/778785054 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): get_auth_request con 0x7efcac0a3fd0 auth_method 0 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 0x7efcac09a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcb0c08640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 0x7efcac0736d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): get_auth_request con 0x7efcac06cb70 auth_method 0 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.885+0000 7efcabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 10 monclient(hunting): handle_auth_done global_id 4847 payload 824 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 -- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 msgr2=0x7efcac0736d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 0x7efcac0736d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 -- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 msgr2=0x7efcac073190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 monclient: found mon.noname-c 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcabfff640 1 -- 172.21.15.164:0/778785054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcac09af30 con 0x7efcac06cb70 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.886+0000 7efcab7fe640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcabfff640 1 -- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 msgr2=0x7efcac09a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcabfff640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 0x7efcac09a8a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b400 tx=0x7efc9c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c005070 con 0x7efcac06cb70 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: dump: 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:11:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9c005230 con 0x7efcac06cb70 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 5 monclient: authenticate success, global_id 4847 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 10 monclient: _renew_subs 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcac09f5d0 con 0x7efcac06cb70 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9c005620 con 0x7efcac06cb70 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efca8ff9640 10 monclient: dump: 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 10 monclient: _renew_subs 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcaa7fc640 4 set_mon_vals no callback set 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.888+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efcac093cf0 con 0x7efcac06cb70 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efcb266b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efc9c01a400 con 0x7efcac06cb70 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efcb266b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efcb266b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcac0a92f0 con 0x7efcac06cb70 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efca8ff9640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 0x7efc980406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.889+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7efc9c0528b0 con 0x7efcac06cb70 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.890+0000 7efcab7fe640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 0x7efc980406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.890+0000 7efcab7fe640 10 monclient: get_auth_request con 0x7efc9803e1f0 auth_method 0 2024-09-10T16:11:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.892+0000 7efcab7fe640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 0x7efc980406b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efca0029ff0 tx=0x7efca002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.894+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efcac0a92f0 con 0x7efcac06cb70 2024-09-10T16:11:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.894+0000 7efca8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:00.894+0000 7efca8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efcb266b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efcb266b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efcb266b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efcac06cf50 con 0x7efcac06cb70 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efca8ff9640 1 -- 172.21.15.164:0/778785054 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7efc9c051270 con 0x7efcac06cb70 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efca8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.137+0000 7efca8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:01.141 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:01.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 msgr2=0x7efc980406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:01.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 0x7efc980406b0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7efca0029ff0 tx=0x7efca002e000 comp rx=0 tx=0).stop 2024-09-10T16:11:01.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 10 monclient: shutdown 2024-09-10T16:11:01.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 msgr2=0x7efcac09a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.139+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 0x7efcac09a8a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efc9c00b400 tx=0x7efc9c00b8c0 comp rx=0 tx=0).stop 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 shutdown_connections 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc9803e1f0 0x7efc980406b0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efcac13d580 0x7efcac0736d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efcac0a3fd0 0x7efcac073190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 --2- 172.21.15.164:0/778785054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efcac06cb70 0x7efcac09a8a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 >> 172.21.15.164:0/778785054 conn(0x7efcac0a4ee0 msgr2=0x7efcac09e280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 shutdown_connections 2024-09-10T16:11:01.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:01.140+0000 7efcb266b640 1 -- 172.21.15.164:0/778785054 wait complete. 2024-09-10T16:11:04.153 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:04.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.261+0000 7f59add24640 10 monclient: get_monmap_and_config 2024-09-10T16:11:04.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.261+0000 7f59add24640 10 monclient: build_initial_monmap 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.261+0000 7f59add24640 10 monclient: monmap: 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:04.263156+0000 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:04.263156+0000 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:04.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 Processor -- start 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 -- start start 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient: init 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59a8137bb0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f59a81386a0 0x7f59a80a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): picked mon.noname-a con 0x7f59a81386a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59a813d140 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a8073820 con 0x7f59a813d140 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59a80a1c90 con 0x7f59a8137bb0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f59a80a1f70 con 0x7f59a81386a0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59add24640 10 monclient(hunting): authenticate will time out at 1042.750845s 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59a77fe640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f59a8137bb0 legacy=0x7f59a8137f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59a6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f59a81386a0 0x7f59a80a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59a7fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f59a813d140 0x7f59a813f530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42140/0 (socket says 172.21.15.164:42140) 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.262+0000 7f59a6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f59a81386a0 0x7f59a80a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38418/0 (socket says 172.21.15.164:38418) 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a7fff640 1 -- 172.21.15.164:0/2798323815 learned_addr learned my addr 172.21.15.164:0/2798323815 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): get_auth_request con 0x7f59a81386a0 auth_method 0 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient(hunting): handle_auth_done global_id 54153 payload 274 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 1 -- 172.21.15.164:0/2798323815 >> v1:172.21.15.165:6789/0 conn(0x7f59a8137bb0 legacy=0x7f59a8137f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 1 -- 172.21.15.164:0/2798323815 >> v1:172.21.15.164:6789/0 conn(0x7f59a813d140 legacy=0x7f59a813f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:11:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:11:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.263+0000 7f59a6ffd640 1 -- 172.21.15.164:0/2798323815 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a80a2250 con 0x7f59a81386a0 2024-09-10T16:11:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a6ffd640 1 -- 172.21.15.164:0/2798323815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a81386a0 msgr2=0x7f59a80a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a6ffd640 1 --2- 172.21.15.164:0/2798323815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a81386a0 0x7f59a80a2e80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a7ed0 tx=0x7f5990029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97821ebce15a0096 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 1 -- 172.21.15.164:0/2798323815 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5990037070 con 0x7f59a81386a0 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: dump: 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:04.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 1 -- 172.21.15.164:0/2798323815 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5990032440 con 0x7f59a81386a0 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 5 monclient: authenticate success, global_id 54153 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 4 set_mon_vals no callback set 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 1 -- 172.21.15.164:0/2798323815 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59900327f0 con 0x7f59a81386a0 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59a67fc640 10 monclient: dump: 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 10 monclient: shutdown 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 -- 172.21.15.164:0/2798323815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a81386a0 msgr2=0x7f59a80a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 --2- 172.21.15.164:0/2798323815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a81386a0 0x7f59a80a2e80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a7ed0 tx=0x7f5990029fe0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 -- 172.21.15.164:0/2798323815 shutdown_connections 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 --2- 172.21.15.164:0/2798323815 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a81386a0 0x7f59a80a2e80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 -- 172.21.15.164:0/2798323815 >> 172.21.15.164:0/2798323815 conn(0x7f59a80a7320 msgr2=0x7f59a80a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 -- 172.21.15.164:0/2798323815 shutdown_connections 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.266+0000 7f59add24640 1 -- 172.21.15.164:0/2798323815 wait complete. 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.267+0000 7f59add24640 10 monclient: build_initial_monmap 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.267+0000 7f59add24640 10 monclient: monmap: 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.267+0000 7f59add24640 1 Processor -- start 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 -- start start 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient: init 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 0x7f59a81ccef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): picked mon.noname-c con 0x7f59a8137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 0x7f59a81346f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): picked mon.noname-b con 0x7f59a81386a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): picked mon.noname-a con 0x7f59a813d140 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8077060 con 0x7f59a813d140 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f59a8076ee0 con 0x7f59a81386a0 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f59a8140c50 con 0x7f59a8137bb0 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59add24640 10 monclient(hunting): authenticate will time out at 1042.756541s 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 0x7f59a81ccef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 0x7f59a81ccef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:40228/0 (socket says 172.21.15.164:40228) 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38432/0 (socket says 172.21.15.164:38432) 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 1 -- 172.21.15.164:0/1228991132 learned_addr learned my addr 172.21.15.164:0/1228991132 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): get_auth_request con 0x7f59a8137bb0 auth_method 0 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): get_auth_request con 0x7f59a813d140 auth_method 0 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a6ffd640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 msgr2=0x7f59a81346f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a6ffd640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 0x7f59a81346f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient(hunting): handle_auth_done global_id 54156 payload 824 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 msgr2=0x7f59a81ccef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 0x7f59a81ccef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 msgr2=0x7f59a81346f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 0x7f59a81346f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 monclient: found mon.noname-a 2024-09-10T16:11:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.268+0000 7f59a7fff640 1 -- 172.21.15.164:0/1228991132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59a811ddb0 con 0x7f59a813d140 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59a7fff640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 msgr2=0x7f59a8139f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59a7fff640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a4d40 tx=0x7f59a000c7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a0017070 con 0x7f59a813d140 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: dump: 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: _finish_auth 0 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59a00070d0 con 0x7f59a813d140 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 5 monclient: authenticate success, global_id 54156 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 10 monclient: _renew_subs 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59a811e090 con 0x7f59a813d140 2024-09-10T16:11:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59a00074c0 con 0x7f59a813d140 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f599ffff640 10 monclient: dump: 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59a5ffb640 4 set_mon_vals no callback set 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 10 monclient: _renew_subs 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.271+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59a811dad0 con 0x7f59a813d140 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f59a001a070 con 0x7f59a813d140 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59add24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59add24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59add24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59a8140800 con 0x7f59a813d140 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f599ffff640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 0x7f5988040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7f59a00520a0 con 0x7f59a813d140 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59a77fe640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 0x7f5988040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.272+0000 7f59a77fe640 10 monclient: get_auth_request con 0x7f598803e1c0 auth_method 0 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.276+0000 7f59a77fe640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 0x7f5988040680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59980094b0 tx=0x7f5998009400 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.278+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f59a0050a60 con 0x7f59a813d140 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.278+0000 7f599ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:04.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.278+0000 7f599ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f59add24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f59add24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f59add24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f59a80a2f00 con 0x7f59a813d140 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f599ffff640 1 -- 172.21.15.164:0/1228991132 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f59a001bbb0 con 0x7f59a813d140 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f599ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.517+0000 7f599ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:04.520 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:04.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 msgr2=0x7f5988040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:04.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 0x7f5988040680 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f59980094b0 tx=0x7f5998009400 comp rx=0 tx=0).stop 2024-09-10T16:11:04.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 10 monclient: shutdown 2024-09-10T16:11:04.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 msgr2=0x7f59a8139f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f59a80a4d40 tx=0x7f59a000c7b0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 shutdown_connections 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f598803e1c0 0x7f5988040680 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59a813d140 0x7f59a8139f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59a81386a0 0x7f59a81346f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 --2- 172.21.15.164:0/1228991132 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59a8137bb0 0x7f59a81ccef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 >> 172.21.15.164:0/1228991132 conn(0x7f59a80a7320 msgr2=0x7f59a81367f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 shutdown_connections 2024-09-10T16:11:04.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:04.519+0000 7f59add24640 1 -- 172.21.15.164:0/1228991132 wait complete. 2024-09-10T16:11:07.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:07.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.641+0000 7f9be1b9f640 10 monclient: get_monmap_and_config 2024-09-10T16:11:07.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.641+0000 7f9be1b9f640 10 monclient: build_initial_monmap 2024-09-10T16:11:07.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.641+0000 7f9be1b9f640 10 monclient: monmap: 2024-09-10T16:11:07.643 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:07.643 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:07.642740+0000 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:07.642740+0000 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:07.644 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.642+0000 7f9be1b9f640 1 Processor -- start 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.642+0000 7f9be1b9f640 1 -- start start 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.642+0000 7f9be1b9f640 10 monclient: init 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 0x7f9bdc0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc09ba80 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc06d4b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc0a4810 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9bdc06ea40 con 0x7f9bdc09ba80 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc09be60 con 0x7f9bdc06cb70 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be1b9f640 10 monclient(hunting): authenticate will time out at 1046.130184s 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be0b9d640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 msgr2=0x7f9bdc0a42d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be0b9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 0x7f9bdc0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46542/0 (socket says 172.21.15.164:46542) 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:42194/0 (socket says 172.21.15.164:42194) 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 1 -- 172.21.15.164:0/78975329 learned_addr learned my addr 172.21.15.164:0/78975329 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): get_auth_request con 0x7f9bdc06d4b0 auth_method 0 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request con 0x7f9bdc06cb70 auth_method 0 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_done global_id 4853 payload 274 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 -- 172.21.15.164:0/78975329 >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 msgr2=0x7f9bdc0a42d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/78975329 >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 0x7f9bdc0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 -- 172.21.15.164:0/78975329 >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 msgr2=0x7f9bdc13e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/78975329 >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 monclient: found mon.noname-b 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9bdbfff640 1 -- 172.21.15.164:0/78975329 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc06dbf0 con 0x7f9bdc06cb70 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.643+0000 7f9be139e640 1 --2- 172.21.15.164:0/78975329 >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdbfff640 1 -- 172.21.15.164:0/78975329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 msgr2=0x7f9bdc06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/78975329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc070070 tx=0x7f9bc802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa1a339c941f4dd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 1 -- 172.21.15.164:0/78975329 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8037070 con 0x7f9bdc06cb70 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: dump: 2024-09-10T16:11:07.652 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 1 -- 172.21.15.164:0/78975329 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc8032440 con 0x7f9bdc06cb70 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9be1b9f640 5 monclient: authenticate success, global_id 4853 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9be1b9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9be1b9f640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 1 -- 172.21.15.164:0/78975329 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8032810 con 0x7f9bdc06cb70 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9be1b9f640 4 set_mon_vals no callback set 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.646+0000 7f9bdb7fe640 10 monclient: dump: 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:07.655 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 10 monclient: shutdown 2024-09-10T16:11:07.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:07.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 -- 172.21.15.164:0/78975329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 msgr2=0x7f9bdc06cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:07.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/78975329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc070070 tx=0x7f9bc802a000 comp rx=0 tx=0).stop 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 -- 172.21.15.164:0/78975329 shutdown_connections 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/78975329 >> v2:172.21.15.164:3300/0 conn(0x7f9bdc06d4b0 0x7f9bdc13e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/78975329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06cb70 0x7f9bdc06cf70 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/78975329 >> v2:172.21.15.165:3300/0 conn(0x7f9bdc09ba80 0x7f9bdc0a42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 -- 172.21.15.164:0/78975329 >> 172.21.15.164:0/78975329 conn(0x7f9bdc0a4ee0 msgr2=0x7f9bdc0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 -- 172.21.15.164:0/78975329 shutdown_connections 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 1 -- 172.21.15.164:0/78975329 wait complete. 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 10 monclient: build_initial_monmap 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.647+0000 7f9be1b9f640 10 monclient: monmap: 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:07.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 Processor -- start 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 -- start start 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient: init 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:11:07.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-a con 0x7f9bdc06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-c con 0x7f9bdc06d4b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 0x7f9bdc0943e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): picked mon.noname-b con 0x7f9bdc09ba80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc06d940 con 0x7f9bdc06cb70 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9bdc06e410 con 0x7f9bdc09ba80 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9bdc13eac0 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be1b9f640 10 monclient(hunting): authenticate will time out at 1046.135923s 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.648+0000 7f9be0b9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46556/0 (socket says 172.21.15.164:46556) 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:42210/0 (socket says 172.21.15.164:42210) 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 1 -- 172.21.15.164:0/2392416734 learned_addr learned my addr 172.21.15.164:0/2392416734 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): get_auth_request con 0x7f9bdc06cb70 auth_method 0 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:07.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be139e640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 msgr2=0x7f9bdc0943e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request con 0x7f9bdc06d4b0 auth_method 0 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be139e640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 0x7f9bdc0943e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:07.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient(hunting): handle_auth_done global_id 4856 payload 824 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 msgr2=0x7f9bdc0943e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 0x7f9bdc0943e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 msgr2=0x7f9bdc093960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 monclient: found mon.noname-c 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9bdbfff640 1 -- 172.21.15.164:0/2392416734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bdc0970d0 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.649+0000 7f9be0b9d640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:07.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bdbfff640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 msgr2=0x7f9bdc093ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bdbfff640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8031000 tx=0x7f9bc80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8041070 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: dump: 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:07.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: _finish_auth 0 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9bc803c870 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 5 monclient: authenticate success, global_id 4856 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 10 monclient: _renew_subs 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bdc0973b0 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:07.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8037040 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bd97fa640 10 monclient: dump: 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:07.670 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9bdaffd640 4 set_mon_vals no callback set 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 10 monclient: _renew_subs 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.651+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bdc09f5d0 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.652+0000 7f9be1b9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.652+0000 7f9be1b9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.652+0000 7f9be1b9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.652+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bdc0a6470 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.652+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9bc803a070 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.653+0000 7f9bd97fa640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 0x7f9bc0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.653+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(168..168 src has 1..168) ==== 4843+0+0 (secure 0 0 0) 0x7f9bc8072c20 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.653+0000 7f9be0b9d640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 0x7f9bc0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:07.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.653+0000 7f9be0b9d640 10 monclient: get_auth_request con 0x7f9bc003e1c0 auth_method 0 2024-09-10T16:11:07.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.656+0000 7f9be0b9d640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 0x7f9bc0040680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009980 tx=0x7f9bd0003b90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:07.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.657+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9bc804a070 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.657+0000 7f9bd97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:07.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.657+0000 7f9bd97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:07.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.896+0000 7f9be1b9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:07.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.896+0000 7f9be1b9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.896+0000 7f9be1b9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.896+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9bdc06cf70 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.897+0000 7f9bd97fa640 1 -- 172.21.15.164:0/2392416734 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f9bc803a360 con 0x7f9bdc06d4b0 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.897+0000 7f9bd97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.897+0000 7f9bd97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:07.899 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":52,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 msgr2=0x7f9bc0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 0x7f9bc0040680 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9bd0009980 tx=0x7f9bd0003b90 comp rx=0 tx=0).stop 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 10 monclient: shutdown 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 msgr2=0x7f9bdc093ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8031000 tx=0x7f9bc80046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 shutdown_connections 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9bc003e1c0 0x7f9bc0040680 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9bdc09ba80 0x7f9bdc0943e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9bdc06d4b0 0x7f9bdc093ea0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 --2- 172.21.15.164:0/2392416734 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9bdc06cb70 0x7f9bdc093960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:07.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 >> 172.21.15.164:0/2392416734 conn(0x7f9bdc0a4ee0 msgr2=0x7f9bdc0a1020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:07.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 shutdown_connections 2024-09-10T16:11:07.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:07.899+0000 7f9be1b9f640 1 -- 172.21.15.164:0/2392416734 wait complete. 2024-09-10T16:11:10.913 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.019+0000 7fc4017bb640 10 monclient: get_monmap_and_config 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.019+0000 7fc4017bb640 10 monclient: build_initial_monmap 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.019+0000 7fc4017bb640 10 monclient: monmap: 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:11.020835+0000 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:11.020835+0000 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:11.022 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.020+0000 7fc4017bb640 1 Processor -- start 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.020+0000 7fc4017bb640 1 -- start start 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.020+0000 7fc4017bb640 10 monclient: init 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3fc132aa0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:11.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3fc0a24d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3fc13b4d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3fc0a4a70 con 0x7fc3fc132aa0 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc3fc0a45b0 con 0x7fc3fc0a24d0 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3fc0a28d0 con 0x7fc3fc13b4d0 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc4017bb640 10 monclient(hunting): authenticate will time out at 1049.508204s 2024-09-10T16:11:11.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3faffd640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fc3fc132aa0 legacy=0x7fc3fc132e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:42226/0 (socket says 172.21.15.164:42226) 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46572/0 (socket says 172.21.15.164:46572) 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 learned_addr learned my addr 172.21.15.164:0/280443342 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3fc13b4d0 auth_method 0 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc3fc0a24d0 auth_method 0 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:11.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:11:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:11.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.021+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.222+0000 7fc3faffd640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.165:6789/0 conn(0x7fc3fc132aa0 legacy=0x7fc3fc132e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.222+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.222+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.222+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3fc13b4d0 auth_method 0 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc3fc0a24d0 auth_method 0 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:11:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:11:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.223+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:11:11.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.623+0000 7fc3faffd640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.165:6789/0 conn(0x7fc3fc132aa0 legacy=0x7fc3fc132e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:11.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc3fc0a24d0 auth_method 0 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3fc13b4d0 auth_method 0 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.624+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:11:11.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:11.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:11.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:11:11.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:11.625+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:11:12.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.424+0000 7fc3faffd640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.165:6789/0 conn(0x7fc3fc132aa0 legacy=0x7fc3fc132e80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3fc13b4d0 auth_method 0 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc3fc0a24d0 auth_method 0 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:12.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:11:12.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:12.427+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): tick 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 monclient(hunting): continuing hunt 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc13d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:14.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3fc13d8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 msgr2=0x7fc3fc13af90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.164:3300/0 conn(0x7fc3fc0a24d0 0x7fc3fc13af90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.165:6789/0 conn(0x7fc3fc132aa0 legacy=0x7fc3fc132e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient: _add_conns ranks=[0,4] 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3dc0036c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 0x7fc3dc009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fc3dc0073d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 -- 172.21.15.164:0/280443342 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3fc0a1450 con 0x7fc3dc0036c0 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 1 -- 172.21.15.164:0/280443342 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc3fc0a45b0 con 0x7fc3dc0073d0 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3f8ff9640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 msgr2=0x7fc3dc009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.021+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 0x7fc3dc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.023+0000 7fc3faffd640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.164:6789/0 conn(0x7fc3dc0036c0 legacy=0x7fc3dc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:14.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.223+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 msgr2=0x7fc3dc009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:14.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.223+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 0x7fc3dc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:11:14.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.625+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 msgr2=0x7fc3dc009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:14.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.625+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 0x7fc3dc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:11:14.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1973770807 0 0) 0x7fc3fc0a1450 con 0x7fc3dc0036c0 2024-09-10T16:11:14.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 10 monclient(hunting): _negotiate 2024-09-10T16:11:14.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:14.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:14.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 10 monclient(hunting): my global_id is 54165 2024-09-10T16:11:14.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.892+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3e8003520 con 0x7fc3dc0036c0 2024-09-10T16:11:14.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3696926705 0 0) 0x7fc3e8003520 con 0x7fc3dc0036c0 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 msgr2=0x7fc3dc009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 --2- 172.21.15.164:0/280443342 >> v2:172.21.15.165:3300/0 conn(0x7fc3dc0073d0 0x7fc3dc009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _finish_hunting 0 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc00af60 con 0x7fc3dc0036c0 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fc3f4004880 con 0x7fc3dc0036c0 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: dump: 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:14.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 >> v1:172.21.15.164:6789/0 conn(0x7fc3dc0036c0 legacy=0x7fc3dc005b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:653: done (1 left) 2024-09-10T16:11:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:654: delete oid 23 current snap is 76 2024-09-10T16:11:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:652: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:11:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:652: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:11:14.901 INFO:tasks.rados.rados.0.smithi165.stdout:652: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:11:14.901 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 214 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3e8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): picked mon.b con 0x7fc3e8003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3e8004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): picked mon.c con 0x7fc3fc13b4d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 0x7fc3e800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): picked mon.a con 0x7fc3e8004e60 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc0a45b0 con 0x7fc3e8004e60 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e80055f0 con 0x7fc3e8003a20 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e8010310 con 0x7fc3fc13b4d0 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3f9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 reap_dead start 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3faffd640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 msgr2=0x7fc3e8003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.895+0000 7fc3faffd640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3e8003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 0x7fc3e800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3e8004e60 auth_method 0 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:14.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_done global_id 54165 payload 293 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3e8004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3e8004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 msgr2=0x7fc3e8003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:14.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3e8003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 monclient: found mon.a 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc00af60 con 0x7fc3e8004e60 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.896+0000 7fc3fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:14.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.897+0000 7fc4017bb640 5 monclient: authenticate success, global_id 54165 2024-09-10T16:11:14.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.897+0000 7fc4017bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:14.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.900+0000 7fc3fb7fe640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 msgr2=0x7fc3e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:14.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.900+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 0x7fc3e800b9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0774c0 tx=0x7fc3e8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c59a89713667b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:14.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec031070 con 0x7fc3e8004e60 2024-09-10T16:11:14.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 10 monclient: dump: 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:14.908 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc4017bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ec02b770 con 0x7fc3e8004e60 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc3f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc4017bb640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.904+0000 7fc4017bb640 4 set_mon_vals no callback set 2024-09-10T16:11:14.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc3f9ffb640 1 -- 172.21.15.164:0/280443342 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ec00f040 con 0x7fc3e8004e60 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc3f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc3f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc3f9ffb640 10 monclient: dump: 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 10 monclient: shutdown 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 -- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 msgr2=0x7fc3e800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 0x7fc3e800b9c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc3fc0774c0 tx=0x7fc3e8005830 comp rx=0 tx=0).stop 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 -- 172.21.15.164:0/280443342 shutdown_connections 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3e8004e60 0x7fc3e800b9c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3fc13b4d0 0x7fc3e8004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 --2- 172.21.15.164:0/280443342 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3e8003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 -- 172.21.15.164:0/280443342 >> 172.21.15.164:0/280443342 conn(0x7fc3fc0a4ee0 msgr2=0x7fc3fc0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 -- 172.21.15.164:0/280443342 shutdown_connections 2024-09-10T16:11:14.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.905+0000 7fc4017bb640 1 -- 172.21.15.164:0/280443342 wait complete. 2024-09-10T16:11:14.912 INFO:tasks.rados.rados.0.smithi165.stdout:652: done (1 left) 2024-09-10T16:11:14.912 INFO:tasks.rados.rados.0.smithi165.stdout:654: done (0 left) 2024-09-10T16:11:14.912 INFO:tasks.rados.rados.0.smithi165.stdout:655: write oid 36 current snap is 76 2024-09-10T16:11:14.912 INFO:tasks.rados.rados.0.smithi165.stdout:655: seq_num 262 ranges {403434=471240,1315341=400756} 2024-09-10T16:11:14.918 INFO:tasks.rados.rados.0.smithi165.stdout:655: writing smithi16534841-36 from 403434 to 874674 tid 1 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.906+0000 7fc4017bb640 10 monclient: build_initial_monmap 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.907+0000 7fc4017bb640 10 monclient: monmap: 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.907+0000 7fc4017bb640 1 Processor -- start 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.907+0000 7fc4017bb640 1 -- start start 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.907+0000 7fc4017bb640 10 monclient: init 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3fc1c87b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e8003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e8004e60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 0x7fc3fc1c9230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): picked mon.noname-a con 0x7fc3fc13b4d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc132770 con 0x7fc3fc13b4d0 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3fc0a17a0 con 0x7fc3e8003a20 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3fc1328f0 con 0x7fc3e8004e60 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:14.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc4017bb640 10 monclient(hunting): authenticate will time out at 1053.395321s 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3faffd640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 msgr2=0x7fc3fc1c87b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3faffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3fc1c87b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 0x7fc3fc1c9230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49044/0 (socket says 172.21.15.164:49044) 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/4223010935 learned_addr learned my addr 172.21.15.164:0/4223010935 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc3e8004e60 auth_method 0 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3fc13b4d0 auth_method 0 2024-09-10T16:11:14.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient(hunting): handle_auth_done global_id 4859 payload 824 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 msgr2=0x7fc3fc1c87b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3fc1c87b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 msgr2=0x7fc3fc1c9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:14.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 0x7fc3fc1c9230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 monclient: found mon.noname-c 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/4223010935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3fc072740 con 0x7fc3e8004e60 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.908+0000 7fc3fb7fe640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 0x7fc3fc1c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3fa7fc640 1 -- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 msgr2=0x7fc3fc1c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3fa7fc640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4013700 tx=0x7fc3e4013bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e401c070 con 0x7fc3e8004e60 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: dump: 2024-09-10T16:11:14.923 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.924 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e4013e10 con 0x7fc3e8004e60 2024-09-10T16:11:14.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 5 monclient: authenticate success, global_id 4859 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 10 monclient: _renew_subs 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3fc0736a0 con 0x7fc3e8004e60 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40166a0 con 0x7fc3e8004e60 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f37fe640 10 monclient: dump: 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:14.926 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc3f97fa640 4 set_mon_vals no callback set 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 10 monclient: _renew_subs 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.911+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3fc1310e0 con 0x7fc3e8004e60 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc3e401f070 con 0x7fc3e8004e60 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc4017bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc4017bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:14.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc4017bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc3f37fe640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 0x7fc3dc0412f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3fc073a90 con 0x7fc3e8004e60 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(169..169 src has 1..169) ==== 4831+0+0 (secure 0 0 0) 0x7fc3e4053470 con 0x7fc3e8004e60 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc3faffd640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 0x7fc3dc0412f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.913+0000 7fc3faffd640 10 monclient: get_auth_request con 0x7fc3dc03ee30 auth_method 0 2024-09-10T16:11:14.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.916+0000 7fc3faffd640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 0x7fc3dc0412f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40041f0 tx=0x7fc3f4003e80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:14.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.919+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc3e4051e40 con 0x7fc3e8004e60 2024-09-10T16:11:14.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.919+0000 7fc3f37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:14.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:14.919+0000 7fc3f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:14.929 INFO:tasks.rados.rados.0.smithi165.stdout:655: writing smithi16534841-36 from 1315341 to 1716097 tid 2 2024-09-10T16:11:14.929 INFO:tasks.rados.rados.0.smithi165.stdout:656: snap_remove snap 33 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.166+0000 7fc4017bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.166+0000 7fc4017bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.166+0000 7fc4017bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.166+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3fc073870 con 0x7fc3e8004e60 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.167+0000 7fc3f37fe640 1 -- 172.21.15.164:0/4223010935 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc3e4016840 con 0x7fc3e8004e60 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.167+0000 7fc3f37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:15.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.167+0000 7fc3f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:15.169 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 msgr2=0x7fc3dc0412f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 0x7fc3dc0412f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc3f40041f0 tx=0x7fc3f4003e80 comp rx=0 tx=0).stop 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 10 monclient: shutdown 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 msgr2=0x7fc3fc1c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4013700 tx=0x7fc3e4013bc0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 shutdown_connections 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3dc03ee30 0x7fc3dc0412f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3fc13b4d0 0x7fc3fc1c9230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3e8004e60 0x7fc3fc1c8cf0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 --2- 172.21.15.164:0/4223010935 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3e8003a20 0x7fc3fc1c87b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 >> 172.21.15.164:0/4223010935 conn(0x7fc3fc0a4ee0 msgr2=0x7fc3fc06dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:15.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 shutdown_connections 2024-09-10T16:11:15.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.169+0000 7fc4017bb640 1 -- 172.21.15.164:0/4223010935 wait complete. 2024-09-10T16:11:15.181 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:11:15.182 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:11:15.182 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:11:15.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.279+0000 7feef332c640 10 monclient: get_monmap_and_config 2024-09-10T16:11:15.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.279+0000 7feef332c640 10 monclient: build_initial_monmap 2024-09-10T16:11:15.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.279+0000 7feef332c640 10 monclient: monmap: 2024-09-10T16:11:15.281 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:15.280165+0000 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:15.280165+0000 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.279+0000 7feef332c640 1 Processor -- start 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 -- start start 2024-09-10T16:11:15.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient: init 2024-09-10T16:11:15.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:15.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 0x7feeec09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): picked mon.noname-c con 0x7feeec09d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 0x7feeec09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): picked mon.noname-a con 0x7feeec09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): picked mon.noname-b con 0x7feeec134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7feeec09d3e0 con 0x7feeec09ab50 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7feeec09e790 con 0x7feeec09d930 2024-09-10T16:11:15.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feeec0a0870 con 0x7feeec134580 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef332c640 10 monclient(hunting): authenticate will time out at 1053.767606s 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef10c8640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 msgr2=0x7feeec09dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef10c8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 0x7feeec09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 0x7feeec09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49052/0 (socket says 172.21.15.164:49052) 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 -- 172.21.15.164:0/2881918474 learned_addr learned my addr 172.21.15.164:0/2881918474 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): get_auth_request con 0x7feeec134580 auth_method 0 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): get_auth_request con 0x7feeec09ab50 auth_method 0 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef08c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 10 monclient(hunting): handle_auth_done global_id 4862 payload 274 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 -- 172.21.15.164:0/2881918474 >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 msgr2=0x7feeec09dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 --2- 172.21.15.164:0/2881918474 >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 0x7feeec09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 -- 172.21.15.164:0/2881918474 >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 msgr2=0x7feeec09e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.280+0000 7feef18c9640 1 --2- 172.21.15.164:0/2881918474 >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 0x7feeec09e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef18c9640 10 monclient: _finish_hunting 0 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef18c9640 1 monclient: found mon.noname-b 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef18c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef18c9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef18c9640 1 -- 172.21.15.164:0/2881918474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec074c70 con 0x7feeec134580 2024-09-10T16:11:15.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.281+0000 7feef08c7640 1 --2- 172.21.15.164:0/2881918474 >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 0x7feeec09e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef18c9640 1 -- 172.21.15.164:0/2881918474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 msgr2=0x7feeec134a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef18c9640 1 --2- 172.21.15.164:0/2881918474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feeec06d500 tx=0x7feee402a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65501002415756be server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 1 -- 172.21.15.164:0/2881918474 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4036070 con 0x7feeec134580 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: dump: 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.291 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: _finish_auth 0 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 1 -- 172.21.15.164:0/2881918474 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee4007800 con 0x7feeec134580 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef332c640 5 monclient: authenticate success, global_id 4862 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef332c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 1 -- 172.21.15.164:0/2881918474 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4007bd0 con 0x7feeec134580 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef332c640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:15.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feef332c640 4 set_mon_vals no callback set 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.283+0000 7feee3fff640 10 monclient: dump: 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 10 monclient: shutdown 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 -- 172.21.15.164:0/2881918474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 msgr2=0x7feeec134a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 --2- 172.21.15.164:0/2881918474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7feeec06d500 tx=0x7feee402a590 comp rx=0 tx=0).stop 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 -- 172.21.15.164:0/2881918474 shutdown_connections 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 --2- 172.21.15.164:0/2881918474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec134a10 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 --2- 172.21.15.164:0/2881918474 >> v2:172.21.15.164:3300/0 conn(0x7feeec09ab50 0x7feeec09e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 --2- 172.21.15.164:0/2881918474 >> v2:172.21.15.165:3300/0 conn(0x7feeec09d930 0x7feeec09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 -- 172.21.15.164:0/2881918474 >> 172.21.15.164:0/2881918474 conn(0x7feeec0a3110 msgr2=0x7feeec0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 -- 172.21.15.164:0/2881918474 shutdown_connections 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.284+0000 7feef332c640 1 -- 172.21.15.164:0/2881918474 wait complete. 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient: build_initial_monmap 2024-09-10T16:11:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient: monmap: 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 Processor -- start 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 -- start start 2024-09-10T16:11:15.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient: init 2024-09-10T16:11:15.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:15.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:11:15.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 0x7feeec1c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): picked mon.noname-a con 0x7feeec09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec09d930 0x7feeec1c7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): picked mon.noname-b con 0x7feeec09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec1cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): picked mon.noname-c con 0x7feeec134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feeec09ef30 con 0x7feeec09ab50 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0a02b0 con 0x7feeec09d930 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feeec0a2940 con 0x7feeec134580 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.285+0000 7feef332c640 10 monclient(hunting): authenticate will time out at 1053.772992s 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 0x7feeec1c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec1cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 0x7feeec1c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56332/0 (socket says 172.21.15.164:56332) 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 1 -- 172.21.15.164:0/3907931743 learned_addr learned my addr 172.21.15.164:0/3907931743 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): get_auth_request con 0x7feeec09ab50 auth_method 0 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef08c7640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec09d930 msgr2=0x7feeec1c7ee0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): get_auth_request con 0x7feeec134580 auth_method 0 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef08c7640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec09d930 0x7feeec1c7ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient(hunting): handle_auth_done global_id 4865 payload 824 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec09d930 msgr2=0x7feeec1c7ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feeec09d930 0x7feeec1c7ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 msgr2=0x7feeec1c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 0x7feeec1c79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient: _finish_hunting 0 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 monclient: found mon.noname-c 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feeec1350d0 con 0x7feeec134580 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.286+0000 7feef10c8640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec09ab50 0x7feeec1c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 msgr2=0x7feeec1cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef18c9640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec1cac70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feee4030000 tx=0x7feee4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4036050 con 0x7feeec134580 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: dump: 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.306 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feee4042070 con 0x7feeec134580 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef332c640 5 monclient: authenticate success, global_id 4865 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef332c640 10 monclient: _renew_subs 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef332c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feeec09d620 con 0x7feeec134580 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:15.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee4031cf0 con 0x7feeec134580 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee1ffb640 10 monclient: dump: 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.308 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 10 monclient: _renew_subs 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feeec0732b0 con 0x7feeec134580 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.288+0000 7feee37fe640 4 set_mon_vals no callback set 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 0x7feeec1cb590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feeec0a1e40 con 0x7feeec1cb1b0 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 0x7feeec1cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient: get_auth_request con 0x7feeec1cb1b0 auth_method 0 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feee403d450 con 0x7feeec134580 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.289+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feef10c8640 10 monclient(hunting): handle_auth_done global_id 54177 payload 824 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feee1ffb640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feed003e1f0 0x7feed00406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feef10c8640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feeec06dac0 con 0x7feeec1cb1b0 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(169..169 src has 1..169) ==== 4831+0+0 (secure 0 0 0) 0x7feee40735e0 con 0x7feeec134580 2024-09-10T16:11:15.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feef08c7640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feed003e1f0 0x7feed00406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.290+0000 7feef08c7640 10 monclient: get_auth_request con 0x7feed003e1f0 auth_method 0 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feef10c8640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 msgr2=0x7feeec1cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feef10c8640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 0x7feeec1cb590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feee80053f0 tx=0x7feee80058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8016070 con 0x7feeec1cb1b0 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 10 monclient: dump: 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.311 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feee800ee20 con 0x7feeec1cb1b0 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 msgr2=0x7feeec1cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 0x7feeec1cb590 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feee80053f0 tx=0x7feee80058b0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 msgr2=0x7feeec1cb590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.292+0000 7feee1ffb640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec1cb1b0 0x7feeec1cb590 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feee80053f0 tx=0x7feee80058b0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.293+0000 7feef08c7640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feed003e1f0 0x7feed00406b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feeec074880 tx=0x7feedc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.298+0000 7feef332c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.298+0000 7feef332c640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 0x7feeec0a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.298+0000 7feef332c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.298+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feeec077260 con 0x7feeec0a4ba0 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.298+0000 7feef10c8640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 0x7feeec0a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient: get_auth_request con 0x7feeec0a4ba0 auth_method 0 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 10 monclient(hunting): handle_auth_done global_id 54180 payload 824 2024-09-10T16:11:15.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.299+0000 7feef10c8640 1 -- 172.21.15.164:0/3907931743 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feeec1c8420 con 0x7feeec0a4ba0 2024-09-10T16:11:15.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feef10c8640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 msgr2=0x7feeec0a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feef10c8640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 0x7feeec0a4fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feee8018000 tx=0x7feee80059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feee8016040 con 0x7feeec0a4ba0 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 10 monclient: dump: 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.315 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7feee8005c50 con 0x7feeec0a4ba0 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 msgr2=0x7feeec0a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 0x7feeec0a4fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7feee8018000 tx=0x7feee80059d0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 msgr2=0x7feeec0a4fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.301+0000 7feee1ffb640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feeec0a4ba0 0x7feeec0a4fb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.316 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 56, 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 45486, 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.317 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.318 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.319 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:15.320 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:15.321 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.322 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.323 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.324 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:15.325 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:15.326 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.327 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.328 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.329 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:15.330 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feed003e1f0 msgr2=0x7feed00406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feed003e1f0 0x7feed00406b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feeec074880 tx=0x7feedc009190 comp rx=0 tx=0).stop 2024-09-10T16:11:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 10 monclient: shutdown 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 msgr2=0x7feeec1cac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 --2- 172.21.15.164:0/3907931743 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feeec134580 0x7feeec1cac70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7feee4030000 tx=0x7feee4004a40 comp rx=0 tx=0).stop 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 reap_dead start 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef18c9640 1 -- 172.21.15.164:0/3907931743 reap_dead start 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 shutdown_connections 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 >> 172.21.15.164:0/3907931743 conn(0x7feeec0a3110 msgr2=0x7feeec139be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 shutdown_connections 2024-09-10T16:11:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.302+0000 7feef332c640 1 -- 172.21.15.164:0/3907931743 wait complete. 2024-09-10T16:11:15.332 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:11:15.333 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:11:15.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:11:15.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 10 monclient: get_monmap_and_config 2024-09-10T16:11:15.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 10 monclient: build_initial_monmap 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 10 monclient: monmap: 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:15.432077+0000 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:15.432077+0000 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:15.433 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 1 Processor -- start 2024-09-10T16:11:15.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 1 -- start start 2024-09-10T16:11:15.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.431+0000 7f65305a0640 10 monclient: init 2024-09-10T16:11:15.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f652809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652809d160 0x7f652809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-a con 0x7f652809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 0x7f652813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-c con 0x7f652813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652809a430 con 0x7f652809c180 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f65280a24e0 con 0x7f652809d160 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f65280a2bf0 con 0x7f652813a5c0 2024-09-10T16:11:15.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f65305a0640 10 monclient(hunting): authenticate will time out at 1053.919474s 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652809d160 0x7f652809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652e33c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f652809c180 0x7f652809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44272/0 (socket says 172.21.15.164:44272) 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652809d160 0x7f652809f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56350/0 (socket says 172.21.15.164:56350) 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652e33c640 1 -- 172.21.15.164:0/2245811721 learned_addr learned my addr 172.21.15.164:0/2245811721 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): get_auth_request con 0x7f652809d160 auth_method 0 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652eb3d640 1 -- 172.21.15.164:0/2245811721 >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 msgr2=0x7f652813c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652eb3d640 1 --2- 172.21.15.164:0/2245811721 >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 0x7f652813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient(hunting): handle_auth_done global_id 54183 payload 274 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 -- 172.21.15.164:0/2245811721 >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 msgr2=0x7f652813c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 --2- 172.21.15.164:0/2245811721 >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 0x7f652813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 -- 172.21.15.164:0/2245811721 >> v1:172.21.15.164:6789/0 conn(0x7f652809c180 legacy=0x7f652809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient: _finish_hunting 0 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 monclient: found mon.noname-a 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:15.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.432+0000 7f652db3b640 1 -- 172.21.15.164:0/2245811721 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652809c560 con 0x7f652809d160 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.435+0000 7f652db3b640 1 -- 172.21.15.164:0/2245811721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652809d160 msgr2=0x7f652809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.435+0000 7f652db3b640 1 --2- 172.21.15.164:0/2245811721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652809d160 0x7f652809f150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40edd7b87a24cc62 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 1 -- 172.21.15.164:0/2245811721 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518037070 con 0x7f652809d160 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: dump: 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: _finish_auth 0 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: _check_auth_tickets 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 5 monclient: authenticate success, global_id 54183 2024-09-10T16:11:15.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 1 -- 172.21.15.164:0/2245811721 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6518032640 con 0x7f652809d160 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 1 -- 172.21.15.164:0/2245811721 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6518032a10 con 0x7f652809d160 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 4 set_mon_vals no callback set 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f652d33a640 10 monclient: dump: 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.448 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 10 monclient: shutdown 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 -- 172.21.15.164:0/2245811721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652809d160 msgr2=0x7f652809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 --2- 172.21.15.164:0/2245811721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652809d160 0x7f652809f150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f65180097a0 tx=0x7f651802a030 comp rx=0 tx=0).stop 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 -- 172.21.15.164:0/2245811721 shutdown_connections 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 --2- 172.21.15.164:0/2245811721 >> v2:172.21.15.165:3300/0 conn(0x7f652813a5c0 0x7f652813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 --2- 172.21.15.164:0/2245811721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652809d160 0x7f652809f150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 -- 172.21.15.164:0/2245811721 >> 172.21.15.164:0/2245811721 conn(0x7f65280a3110 msgr2=0x7f65280a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 -- 172.21.15.164:0/2245811721 shutdown_connections 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.436+0000 7f65305a0640 1 -- 172.21.15.164:0/2245811721 wait complete. 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.437+0000 7f65305a0640 10 monclient: build_initial_monmap 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.437+0000 7f65305a0640 10 monclient: monmap: 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:15.450 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.437+0000 7f65305a0640 1 Processor -- start 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.437+0000 7f65305a0640 1 -- start start 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.437+0000 7f65305a0640 10 monclient: init 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652809c180 0x7f6528074210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-c con 0x7f652809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652809d160 0x7f65280933e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-b con 0x7f652809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): picked mon.noname-a con 0x7f652813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6528074f90 con 0x7f652813a5c0 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6528074e10 con 0x7f652809d160 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f652809f690 con 0x7f652809c180 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f65305a0640 10 monclient(hunting): authenticate will time out at 1053.925250s 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652809c180 0x7f6528074210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56354/0 (socket says 172.21.15.164:56354) 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 learned_addr learned my addr 172.21.15.164:0/1650923378 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652809c180 0x7f6528074210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49068/0 (socket says 172.21.15.164:49068) 2024-09-10T16:11:15.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): get_auth_request con 0x7f652813a5c0 auth_method 0 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652db3b640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652809d160 msgr2=0x7f65280933e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): get_auth_request con 0x7f652809c180 auth_method 0 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652db3b640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652809d160 0x7f65280933e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient(hunting): handle_auth_done global_id 54186 payload 824 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652809c180 msgr2=0x7f6528074210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652809c180 0x7f6528074210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652809d160 msgr2=0x7f65280933e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652809d160 0x7f65280933e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient: _finish_hunting 0 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 monclient: found mon.noname-a 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.438+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6528096150 con 0x7f652813a5c0 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 msgr2=0x7f6528093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f652eb3d640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f652000be20 tx=0x7f6520009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520016630 con 0x7f652813a5c0 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.455 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: _finish_auth 0 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6520010030 con 0x7f652813a5c0 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f65305a0640 5 monclient: authenticate success, global_id 54186 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6520016a30 con 0x7f652813a5c0 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f65305a0640 10 monclient: _renew_subs 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f65305a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65280969f0 con 0x7f652813a5c0 2024-09-10T16:11:15.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.440+0000 7f652cb39640 4 set_mon_vals no callback set 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 10 monclient: _renew_subs 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652809bba0 con 0x7f652813a5c0 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 0x7f6528093e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f652802e9a0 con 0x7f652802e5c0 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 0x7f6528093e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient: get_auth_request con 0x7f652802e5c0 auth_method 0 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f65200153e0 con 0x7f652813a5c0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.441+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f651effd640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650c03e1c0 0x7f650c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f652e33c640 10 monclient(hunting): handle_auth_done global_id 4871 payload 824 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f652e33c640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f652809ea40 con 0x7f652802e5c0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(169..169 src has 1..169) ==== 4831+0+0 (secure 0 0 0) 0x7f6520053660 con 0x7f652813a5c0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f652db3b640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650c03e1c0 0x7f650c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.442+0000 7f652db3b640 10 monclient: get_auth_request con 0x7f650c03e1c0 auth_method 0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f652e33c640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 msgr2=0x7f6528093e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f652e33c640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 0x7f6528093e60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f652400ea70 tx=0x7f652400ef30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652401a070 con 0x7f652802e5c0 2024-09-10T16:11:15.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.460 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f652400cc50 con 0x7f652802e5c0 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 msgr2=0x7f6528093e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 0x7f6528093e60 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f652400ea70 tx=0x7f652400ef30 comp rx=0 tx=0).stop 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 msgr2=0x7f6528093e60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.444+0000 7f651effd640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652802e5c0 0x7f6528093e60 secure :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f652400ea70 tx=0x7f652400ef30 comp rx=0 tx=0).stop 2024-09-10T16:11:15.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.445+0000 7f652db3b640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650c03e1c0 0x7f650c040680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f652806d4a0 tx=0x7f651802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f65305a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f65305a0640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 0x7f6528064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f65305a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f65280967d0 con 0x7f6528063d50 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 0x7f6528064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient: get_auth_request con 0x7f6528063d50 auth_method 0 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:15.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.450+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.451+0000 7f652e33c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.451+0000 7f652e33c640 10 monclient(hunting): handle_auth_done global_id 4874 payload 824 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.451+0000 7f652e33c640 1 -- 172.21.15.164:0/1650923378 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6528071ef0 con 0x7f6528063d50 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f652e33c640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 msgr2=0x7f6528064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f652e33c640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 0x7f6528064160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6524017000 tx=0x7f6524004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f652401a070 con 0x7f6528063d50 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:15.464 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f652401b040 con 0x7f6528063d50 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 msgr2=0x7f6528064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 0x7f6528064160 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6524017000 tx=0x7f6524004940 comp rx=0 tx=0).stop 2024-09-10T16:11:15.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 msgr2=0x7f6528064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.453+0000 7f651effd640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6528063d50 0x7f6528064160 secure :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f6524017000 tx=0x7f6524004940 comp rx=0 tx=0).stop 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 56, 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 317625, 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:15.466 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.467 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.468 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:15.469 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:15.470 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:15.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.472 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.473 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.474 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:15.475 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.476 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.477 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.478 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:15.479 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650c03e1c0 msgr2=0x7f650c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650c03e1c0 0x7f650c040680 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f652806d4a0 tx=0x7f651802e000 comp rx=0 tx=0).stop 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 10 monclient: shutdown 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 msgr2=0x7f6528093920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f652eb3d640 1 -- 172.21.15.164:0/1650923378 reap_dead start 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f652000be20 tx=0x7f6520009f10 comp rx=0 tx=0).stop 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 shutdown_connections 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 --2- 172.21.15.164:0/1650923378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652813a5c0 0x7f6528093920 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:15.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 >> 172.21.15.164:0/1650923378 conn(0x7f65280a3110 msgr2=0x7f6528139ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:15.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 shutdown_connections 2024-09-10T16:11:15.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:15.454+0000 7f65305a0640 1 -- 172.21.15.164:0/1650923378 wait complete. 2024-09-10T16:11:15.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:11:15.482 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:11:15.909 INFO:tasks.rados.rados.0.smithi165.stdout:656: done (1 left) 2024-09-10T16:11:15.909 INFO:tasks.rados.rados.0.smithi165.stdout:657: snap_remove snap 74 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:657: done (1 left) 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:658: rollback oid 50 current snap is 76 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 48 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: finishing write tid 1 to smithi16534841-36 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: finishing write tid 2 to smithi16534841-36 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: finishing write tid 3 to smithi16534841-36 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: finishing write tid 4 to smithi16534841-36 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: oid 36 updating version 0 to 225 2024-09-10T16:11:16.908 INFO:tasks.rados.rados.0.smithi165.stdout:655: oid 36 updating version 225 to 226 2024-09-10T16:11:16.909 INFO:tasks.rados.rados.0.smithi165.stdout:655: oid 36 updating version 226 to 227 2024-09-10T16:11:16.909 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 227 (ObjNum 262 snap 76 seq_num 262) dirty exists 2024-09-10T16:11:16.909 INFO:tasks.rados.rados.0.smithi165.stdout:655: left oid 36 (ObjNum 262 snap 76 seq_num 262) 2024-09-10T16:11:16.909 INFO:tasks.rados.rados.0.smithi165.stdout:655: done (1 left) 2024-09-10T16:11:16.909 INFO:tasks.rados.rados.0.smithi165.stdout:659: snap_remove snap 73 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:659: done (1 left) 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:660: rollback oid 27 current snap is 76 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 48 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:658: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:658: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 263 (ObjNum 33 snap 0 seq_num 1162719424) dirty dne 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:658: done (1 left) 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:661: copy_from oid 17 from oid 2 current snap is 76 2024-09-10T16:11:17.915 INFO:tasks.rados.rados.0.smithi165.stdout:662: read oid 19 snap -1 2024-09-10T16:11:17.916 INFO:tasks.rados.rados.0.smithi165.stdout:662: expect (ObjNum 246 snap 73 seq_num 246) 2024-09-10T16:11:17.916 INFO:tasks.rados.rados.0.smithi165.stdout:663: write oid 33 current snap is 76 2024-09-10T16:11:17.916 INFO:tasks.rados.rados.0.smithi165.stdout:663: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-10T16:11:17.920 INFO:tasks.rados.rados.0.smithi165.stdout:663: writing smithi16534841-33 from 420241 to 966730 tid 1 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:663: writing smithi16534841-33 from 1373823 to 1918237 tid 2 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:664: rollback oid 1 current snap is 76 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 57 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:660: finishing rollback tid 0 to smithi16534841-27 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:660: finishing rollback tid 1 to smithi16534841-27 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 491 (ObjNum 289 snap 0 seq_num 326543706) dirty dne 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:661: finishing copy_from to smithi16534841-17 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:661: got expected ENOENT (src dne) 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:661: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:11:17.927 INFO:tasks.rados.rados.0.smithi165.stdout:662: expect (ObjNum 246 snap 73 seq_num 246) 2024-09-10T16:11:17.958 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:17.988 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:18.019 INFO:tasks.rados.rados.0.smithi165.stdout:663: finishing write tid 1 to smithi16534841-33 2024-09-10T16:11:18.019 INFO:tasks.rados.rados.0.smithi165.stdout:663: finishing write tid 2 to smithi16534841-33 2024-09-10T16:11:18.019 INFO:tasks.rados.rados.0.smithi165.stdout:663: finishing write tid 3 to smithi16534841-33 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: finishing write tid 4 to smithi16534841-33 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: oid 33 updating version 0 to 437 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: oid 33 updating version 437 to 438 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: oid 33 version 438 is already newer than 436 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 438 (ObjNum 263 snap 76 seq_num 263) dirty exists 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: left oid 33 (ObjNum 263 snap 76 seq_num 263) 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:664: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:660: done (4 left) 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:661: done (3 left) 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:662: done (2 left) 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:663: done (1 left) 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:665: copy_from oid 50 from oid 43 current snap is 76 2024-09-10T16:11:18.020 INFO:tasks.rados.rados.0.smithi165.stdout:664: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:11:18.021 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 436 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-10T16:11:18.021 INFO:tasks.rados.rados.0.smithi165.stdout:664: done (1 left) 2024-09-10T16:11:18.021 INFO:tasks.rados.rados.0.smithi165.stdout:666: copy_from oid 35 from oid 8 current snap is 76 2024-09-10T16:11:18.021 INFO:tasks.rados.rados.0.smithi165.stdout:667: write oid 48 current snap is 76 2024-09-10T16:11:18.021 INFO:tasks.rados.rados.0.smithi165.stdout:667: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-10T16:11:18.024 INFO:tasks.rados.rados.0.smithi165.stdout:667: writing smithi16534841-48 from 437048 to 1058786 tid 1 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:667: writing smithi16534841-48 from 1832305 to 2236730 tid 2 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:668: copy_from oid 14 from oid 11 current snap is 76 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:666: finishing copy_from to smithi16534841-35 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:666: got expected ENOENT (src dne) 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:666: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:665: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:667: finishing write tid 1 to smithi16534841-48 2024-09-10T16:11:18.032 INFO:tasks.rados.rados.0.smithi165.stdout:666: done (3 left) 2024-09-10T16:11:18.033 INFO:tasks.rados.rados.0.smithi165.stdout:669: read oid 44 snap -1 2024-09-10T16:11:18.033 INFO:tasks.rados.rados.0.smithi165.stdout:669: expect deleted 2024-09-10T16:11:18.033 INFO:tasks.rados.rados.0.smithi165.stdout:670: write (excl) oid 1 current snap is 76 2024-09-10T16:11:18.033 INFO:tasks.rados.rados.0.smithi165.stdout:670: seq_num 265 ranges {453855=696987,1890787=548083} 2024-09-10T16:11:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:670: writing smithi16534841-1 from 453855 to 1150842 tid 1 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:670: writing smithi16534841-1 from 1890787 to 2438870 tid 2 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:671: snap_create 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:668: finishing copy_from to smithi16534841-14 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:668: got expected ENOENT (src dne) 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:668: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:665: finishing copy_from to smithi16534841-50 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 184 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:667: finishing write tid 2 to smithi16534841-48 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:667: finishing write tid 3 to smithi16534841-48 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:667: finishing write tid 4 to smithi16534841-48 2024-09-10T16:11:18.046 INFO:tasks.rados.rados.0.smithi165.stdout:667: oid 48 updating version 0 to 606 2024-09-10T16:11:18.047 INFO:tasks.rados.rados.0.smithi165.stdout:667: oid 48 updating version 606 to 607 2024-09-10T16:11:18.047 INFO:tasks.rados.rados.0.smithi165.stdout:667: oid 48 updating version 607 to 608 2024-09-10T16:11:18.047 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 608 (ObjNum 264 snap 76 seq_num 264) dirty exists 2024-09-10T16:11:18.047 INFO:tasks.rados.rados.0.smithi165.stdout:667: left oid 48 (ObjNum 264 snap 76 seq_num 264) 2024-09-10T16:11:18.047 INFO:tasks.rados.rados.0.smithi165.stdout:670: finishing write tid 1 to smithi16534841-1 2024-09-10T16:11:18.048 INFO:tasks.rados.rados.0.smithi165.stdout:670: finishing write tid 2 to smithi16534841-1 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: finishing write tid 3 to smithi16534841-1 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: finishing write tid 4 to smithi16534841-1 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: oid 1 updating version 0 to 438 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: oid 1 version 438 is already newer than 437 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: oid 1 updating version 438 to 439 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 439 (ObjNum 265 snap 76 seq_num 265) dirty exists 2024-09-10T16:11:18.050 INFO:tasks.rados.rados.0.smithi165.stdout:670: left oid 1 (ObjNum 265 snap 76 seq_num 265) 2024-09-10T16:11:18.916 INFO:tasks.rados.rados.0.smithi165.stdout:665: done (5 left) 2024-09-10T16:11:18.916 INFO:tasks.rados.rados.0.smithi165.stdout:667: done (4 left) 2024-09-10T16:11:18.916 INFO:tasks.rados.rados.0.smithi165.stdout:668: done (3 left) 2024-09-10T16:11:18.917 INFO:tasks.rados.rados.0.smithi165.stdout:669: done (2 left) 2024-09-10T16:11:18.917 INFO:tasks.rados.rados.0.smithi165.stdout:670: done (1 left) 2024-09-10T16:11:18.917 INFO:tasks.rados.rados.0.smithi165.stdout:671: done (0 left) 2024-09-10T16:11:18.917 INFO:tasks.rados.rados.0.smithi165.stdout:672: delete oid 36 current snap is 77 2024-09-10T16:11:18.921 INFO:tasks.rados.rados.0.smithi165.stdout:672: done (0 left) 2024-09-10T16:11:18.921 INFO:tasks.rados.rados.0.smithi165.stdout:673: snap_create 2024-09-10T16:11:19.830 INFO:tasks.rados.rados.0.smithi165.stdout:673: done (0 left) 2024-09-10T16:11:19.830 INFO:tasks.rados.rados.0.smithi165.stdout:674: delete oid 31 current snap is 78 2024-09-10T16:11:19.836 INFO:tasks.rados.rados.0.smithi165.stdout:674: done (0 left) 2024-09-10T16:11:19.836 INFO:tasks.rados.rados.0.smithi165.stdout:675: copy_from oid 26 from oid 12 current snap is 78 2024-09-10T16:11:19.836 INFO:tasks.rados.rados.0.smithi165.stdout:676: write oid 35 current snap is 78 2024-09-10T16:11:19.836 INFO:tasks.rados.rados.0.smithi165.stdout:676: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-10T16:11:19.842 INFO:tasks.rados.rados.0.smithi165.stdout:676: writing smithi16534841-35 from 504276 to 1143363 tid 1 2024-09-10T16:11:19.849 INFO:tasks.rados.rados.0.smithi165.stdout:676: writing smithi16534841-35 from 1666233 to 2361643 tid 2 2024-09-10T16:11:19.849 INFO:tasks.rados.rados.0.smithi165.stdout:677: rollback oid 23 current snap is 78 2024-09-10T16:11:19.849 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 57 2024-09-10T16:11:19.849 INFO:tasks.rados.rados.0.smithi165.stdout:678: delete oid 10 current snap is 78 2024-09-10T16:11:19.850 INFO:tasks.rados.rados.0.smithi165.stdout:675: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:11:19.856 INFO:tasks.rados.rados.0.smithi165.stdout:678: done (3 left) 2024-09-10T16:11:19.856 INFO:tasks.rados.rados.0.smithi165.stdout:679: delete oid 5 current snap is 78 2024-09-10T16:11:19.856 INFO:tasks.rados.rados.0.smithi165.stdout:675: finishing copy_from to smithi16534841-26 2024-09-10T16:11:19.856 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 54 (ObjNum 244 snap 72 seq_num 244) dirty exists 2024-09-10T16:11:19.856 INFO:tasks.rados.rados.0.smithi165.stdout:677: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:11:19.857 INFO:tasks.rados.rados.0.smithi165.stdout:677: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:11:19.857 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 441 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:11:19.857 INFO:tasks.rados.rados.0.smithi165.stdout:676: finishing write tid 1 to smithi16534841-35 2024-09-10T16:11:19.859 INFO:tasks.rados.rados.0.smithi165.stdout:676: finishing write tid 2 to smithi16534841-35 2024-09-10T16:11:19.860 INFO:tasks.rados.rados.0.smithi165.stdout:676: finishing write tid 3 to smithi16534841-35 2024-09-10T16:11:19.860 INFO:tasks.rados.rados.0.smithi165.stdout:676: finishing write tid 4 to smithi16534841-35 2024-09-10T16:11:19.860 INFO:tasks.rados.rados.0.smithi165.stdout:676: oid 35 updating version 0 to 213 2024-09-10T16:11:19.861 INFO:tasks.rados.rados.0.smithi165.stdout:676: oid 35 updating version 213 to 214 2024-09-10T16:11:19.861 INFO:tasks.rados.rados.0.smithi165.stdout:676: oid 35 version 214 is already newer than 212 2024-09-10T16:11:19.861 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 214 (ObjNum 268 snap 78 seq_num 268) dirty exists 2024-09-10T16:11:19.861 INFO:tasks.rados.rados.0.smithi165.stdout:676: left oid 35 (ObjNum 268 snap 78 seq_num 268) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:675: done (3 left) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:676: done (2 left) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:677: done (1 left) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:679: done (0 left) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:680: read oid 12 snap 77 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:680: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:681: read oid 11 snap -1 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:681: expect deleted 2024-09-10T16:11:19.862 INFO:tasks.rados.rados.0.smithi165.stdout:682: snap_remove snap 75 2024-09-10T16:11:20.833 INFO:tasks.rados.rados.0.smithi165.stdout:682: done (2 left) 2024-09-10T16:11:20.833 INFO:tasks.rados.rados.0.smithi165.stdout:683: copy_from oid 45 from oid 10 current snap is 78 2024-09-10T16:11:20.833 INFO:tasks.rados.rados.0.smithi165.stdout:680: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:11:20.858 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:20.880 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:683: finishing copy_from to smithi16534841-45 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:683: got expected ENOENT (src dne) 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:683: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:680: done (2 left) 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:681: done (1 left) 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:683: done (0 left) 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:684: copy_from oid 50 from oid 11 current snap is 78 2024-09-10T16:11:20.903 INFO:tasks.rados.rados.0.smithi165.stdout:685: snap_remove snap 77 2024-09-10T16:11:21.833 INFO:tasks.rados.rados.0.smithi165.stdout:685: done (1 left) 2024-09-10T16:11:21.833 INFO:tasks.rados.rados.0.smithi165.stdout:686: rollback oid 30 current snap is 78 2024-09-10T16:11:21.833 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 44 2024-09-10T16:11:21.833 INFO:tasks.rados.rados.0.smithi165.stdout:684: finishing copy_from to smithi16534841-50 2024-09-10T16:11:21.833 INFO:tasks.rados.rados.0.smithi165.stdout:684: got expected ENOENT (src dne) 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:684: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:684: done (1 left) 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:687: rollback oid 20 current snap is 78 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 57 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:688: rollback oid 19 current snap is 78 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 57 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:689: read oid 34 snap 71 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:689: expect deleted 2024-09-10T16:11:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:690: copy_from oid 1 from oid 43 current snap is 78 2024-09-10T16:11:21.835 INFO:tasks.rados.rados.0.smithi165.stdout:691: read oid 37 snap -1 2024-09-10T16:11:21.835 INFO:tasks.rados.rados.0.smithi165.stdout:691: expect (ObjNum 246 snap 73 seq_num 246) 2024-09-10T16:11:21.835 INFO:tasks.rados.rados.0.smithi165.stdout:692: write oid 35 current snap is 78 2024-09-10T16:11:21.835 INFO:tasks.rados.rados.0.smithi165.stdout:692: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-10T16:11:21.838 INFO:tasks.rados.rados.0.smithi165.stdout:692: writing smithi16534841-35 from 554697 to 1019531 tid 1 2024-09-10T16:11:21.847 INFO:tasks.rados.rados.0.smithi165.stdout:692: writing smithi16534841-35 from 1725326 to 2284416 tid 2 2024-09-10T16:11:21.847 INFO:tasks.rados.rados.0.smithi165.stdout:693: snap_create 2024-09-10T16:11:21.847 INFO:tasks.rados.rados.0.smithi165.stdout:687: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:11:21.847 INFO:tasks.rados.rados.0.smithi165.stdout:687: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:11:21.847 INFO:tasks.rados.rados.0.smithi165.stdout:687: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:11:21.848 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 442 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:11:21.848 INFO:tasks.rados.rados.0.smithi165.stdout:686: finishing rollback tid 0 to smithi16534841-30 2024-09-10T16:11:21.848 INFO:tasks.rados.rados.0.smithi165.stdout:686: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:11:21.848 INFO:tasks.rados.rados.0.smithi165.stdout:691: expect (ObjNum 246 snap 73 seq_num 246) 2024-09-10T16:11:21.848 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:21.878 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:21.908 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:686: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 616 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:688: finishing rollback tid 0 to smithi16534841-19 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:688: finishing rollback tid 1 to smithi16534841-19 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:688: finishing rollback tid 2 to smithi16534841-19 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 620 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:690: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:690: finishing copy_from to smithi16534841-1 2024-09-10T16:11:21.938 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 184 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: finishing write tid 1 to smithi16534841-35 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: finishing write tid 2 to smithi16534841-35 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: finishing write tid 3 to smithi16534841-35 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: finishing write tid 4 to smithi16534841-35 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: oid 35 updating version 0 to 217 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: oid 35 updating version 217 to 221 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: oid 35 version 221 is already newer than 220 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 221 (ObjNum 271 snap 78 seq_num 271) dirty exists 2024-09-10T16:11:21.939 INFO:tasks.rados.rados.0.smithi165.stdout:692: left oid 35 (ObjNum 271 snap 78 seq_num 271) 2024-09-10T16:11:22.833 INFO:tasks.rados.rados.0.smithi165.stdout:686: done (7 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:687: done (6 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:688: done (5 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:689: done (4 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:690: done (3 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:691: done (2 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:692: done (1 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:693: done (0 left) 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:694: rollback oid 22 current snap is 79 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 78 2024-09-10T16:11:22.834 INFO:tasks.rados.rados.0.smithi165.stdout:695: delete oid 13 current snap is 79 2024-09-10T16:11:22.841 INFO:tasks.rados.rados.0.smithi165.stdout:694: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:11:22.841 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 130 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:11:22.842 INFO:tasks.rados.rados.0.smithi165.stdout:694: done (1 left) 2024-09-10T16:11:22.842 INFO:tasks.rados.rados.0.smithi165.stdout:695: done (0 left) 2024-09-10T16:11:22.842 INFO:tasks.rados.rados.0.smithi165.stdout:696: read oid 10 snap -1 2024-09-10T16:11:22.842 INFO:tasks.rados.rados.0.smithi165.stdout:696: expect deleted 2024-09-10T16:11:22.842 INFO:tasks.rados.rados.0.smithi165.stdout:697: delete oid 36 current snap is 79 2024-09-10T16:11:22.843 INFO:tasks.rados.rados.0.smithi165.stdout:697: done (1 left) 2024-09-10T16:11:22.843 INFO:tasks.rados.rados.0.smithi165.stdout:698: delete oid 27 current snap is 79 2024-09-10T16:11:22.845 INFO:tasks.rados.rados.0.smithi165.stdout:696: done (1 left) 2024-09-10T16:11:22.845 INFO:tasks.rados.rados.0.smithi165.stdout:698: done (0 left) 2024-09-10T16:11:22.845 INFO:tasks.rados.rados.0.smithi165.stdout:699: rollback oid 10 current snap is 79 2024-09-10T16:11:22.845 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 44 2024-09-10T16:11:22.846 INFO:tasks.rados.rados.0.smithi165.stdout:700: copy_from oid 8 from oid 32 current snap is 79 2024-09-10T16:11:22.846 INFO:tasks.rados.rados.0.smithi165.stdout:701: read oid 31 snap -1 2024-09-10T16:11:22.846 INFO:tasks.rados.rados.0.smithi165.stdout:701: expect deleted 2024-09-10T16:11:22.846 INFO:tasks.rados.rados.0.smithi165.stdout:702: snap_create 2024-09-10T16:11:22.853 INFO:tasks.rados.rados.0.smithi165.stdout:699: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:11:22.853 INFO:tasks.rados.rados.0.smithi165.stdout:699: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:11:22.854 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 260 (ObjNum 106 snap 28 seq_num 106) dirty exists 2024-09-10T16:11:22.856 INFO:tasks.rados.rados.0.smithi165.stdout:700: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:11:22.860 INFO:tasks.rados.rados.0.smithi165.stdout:700: finishing copy_from to smithi16534841-8 2024-09-10T16:11:22.860 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 192 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:11:23.848 INFO:tasks.rados.rados.0.smithi165.stdout:699: done (3 left) 2024-09-10T16:11:23.848 INFO:tasks.rados.rados.0.smithi165.stdout:700: done (2 left) 2024-09-10T16:11:23.848 INFO:tasks.rados.rados.0.smithi165.stdout:701: done (1 left) 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:702: done (0 left) 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:703: rollback oid 2 current snap is 80 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 71 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:704: copy_from oid 50 from oid 11 current snap is 80 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:705: write (excl) oid 8 current snap is 80 2024-09-10T16:11:23.849 INFO:tasks.rados.rados.0.smithi165.stdout:705: seq_num 275 ranges {621925=482183,1559254=566428} 2024-09-10T16:11:23.854 INFO:tasks.rados.rados.0.smithi165.stdout:705: writing smithi16534841-8 from 621925 to 1104108 tid 1 2024-09-10T16:11:23.859 INFO:tasks.rados.rados.0.smithi165.stdout:705: writing smithi16534841-8 from 1559254 to 2125682 tid 2 2024-09-10T16:11:23.860 INFO:tasks.rados.rados.0.smithi165.stdout:706: snap_remove snap 48 2024-09-10T16:11:24.833 INFO:tasks.rados.rados.0.smithi165.stdout:706: done (3 left) 2024-09-10T16:11:24.833 INFO:tasks.rados.rados.0.smithi165.stdout:707: write oid 35 current snap is 80 2024-09-10T16:11:24.834 INFO:tasks.rados.rados.0.smithi165.stdout:703: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:11:24.834 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 315 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:11:24.834 INFO:tasks.rados.rados.0.smithi165.stdout:707: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-10T16:11:24.840 INFO:tasks.rados.rados.0.smithi165.stdout:707: writing smithi16534841-35 from 638732 to 1196164 tid 1 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:707: writing smithi16534841-35 from 1617736 to 2327822 tid 2 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:703: done (3 left) 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:708: rollback oid 25 current snap is 80 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 79 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:704: finishing copy_from to smithi16534841-50 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:704: got expected ENOENT (src dne) 2024-09-10T16:11:24.847 INFO:tasks.rados.rados.0.smithi165.stdout:704: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: finishing write tid 1 to smithi16534841-8 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: finishing write tid 2 to smithi16534841-8 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: finishing write tid 3 to smithi16534841-8 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: finishing write tid 4 to smithi16534841-8 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: oid 8 updating version 0 to 505 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: oid 8 updating version 505 to 506 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: oid 8 version 506 is already newer than 504 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 506 (ObjNum 275 snap 80 seq_num 275) dirty exists 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: left oid 8 (ObjNum 275 snap 80 seq_num 275) 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:707: finishing write tid 1 to smithi16534841-35 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:704: done (3 left) 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:705: done (2 left) 2024-09-10T16:11:24.848 INFO:tasks.rados.rados.0.smithi165.stdout:709: copy_from oid 28 from oid 40 current snap is 80 2024-09-10T16:11:24.849 INFO:tasks.rados.rados.0.smithi165.stdout:710: snap_remove snap 71 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:710: done (3 left) 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:711: delete oid 10 current snap is 80 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:707: finishing write tid 2 to smithi16534841-35 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:708: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:708: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:708: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 446 (ObjNum 230 snap 69 seq_num 230) dirty exists 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:707: finishing write tid 3 to smithi16534841-35 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:707: finishing write tid 4 to smithi16534841-35 2024-09-10T16:11:25.849 INFO:tasks.rados.rados.0.smithi165.stdout:707: oid 35 updating version 0 to 230 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:707: oid 35 updating version 230 to 232 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:707: oid 35 version 232 is already newer than 231 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 232 (ObjNum 276 snap 80 seq_num 276) dirty exists 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:707: left oid 35 (ObjNum 276 snap 80 seq_num 276) 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:709: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:709: finishing copy_from to smithi16534841-28 2024-09-10T16:11:25.850 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 300 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:11:25.855 INFO:tasks.rados.rados.0.smithi165.stdout:707: done (3 left) 2024-09-10T16:11:25.855 INFO:tasks.rados.rados.0.smithi165.stdout:708: done (2 left) 2024-09-10T16:11:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:709: done (1 left) 2024-09-10T16:11:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:711: done (0 left) 2024-09-10T16:11:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:712: snap_remove snap 57 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:712: done (0 left) 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:713: read oid 38 snap -1 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:713: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:714: rollback oid 20 current snap is 80 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 63 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:715: read oid 29 snap -1 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:715: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:716: write (excl) oid 26 current snap is 80 2024-09-10T16:11:26.854 INFO:tasks.rados.rados.0.smithi165.stdout:716: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-10T16:11:26.862 INFO:tasks.rados.rados.0.smithi165.stdout:716: writing smithi16534841-26 from 672346 to 1380276 tid 1 2024-09-10T16:11:26.870 INFO:tasks.rados.rados.0.smithi165.stdout:716: writing smithi16534841-26 from 2134700 to 2672346 tid 2 2024-09-10T16:11:26.870 INFO:tasks.rados.rados.0.smithi165.stdout:717: read oid 28 snap -1 2024-09-10T16:11:26.870 INFO:tasks.rados.rados.0.smithi165.stdout:717: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:11:26.871 INFO:tasks.rados.rados.0.smithi165.stdout:718: read oid 9 snap -1 2024-09-10T16:11:26.871 INFO:tasks.rados.rados.0.smithi165.stdout:718: expect deleted 2024-09-10T16:11:26.871 INFO:tasks.rados.rados.0.smithi165.stdout:719: delete oid 3 current snap is 80 2024-09-10T16:11:26.871 INFO:tasks.rados.rados.0.smithi165.stdout:715: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:26.880 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1472722 2024-09-10T16:11:26.897 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:26.940 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1139132 2024-09-10T16:11:26.958 INFO:tasks.rados.rados.0.smithi165.stdout:713: expect (ObjNum 37 snap 0 seq_num 37) 2024-09-10T16:11:26.975 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:27.032 INFO:tasks.rados.rados.0.smithi165.stdout:714: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:11:27.032 INFO:tasks.rados.rados.0.smithi165.stdout:714: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:11:27.032 INFO:tasks.rados.rados.0.smithi165.stdout:714: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:11:27.033 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 464 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:11:27.033 INFO:tasks.rados.rados.0.smithi165.stdout:716: finishing write tid 1 to smithi16534841-26 2024-09-10T16:11:27.033 INFO:tasks.rados.rados.0.smithi165.stdout:717: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:11:27.033 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:27.150 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: finishing write tid 2 to smithi16534841-26 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: finishing write tid 3 to smithi16534841-26 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: finishing write tid 4 to smithi16534841-26 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: oid 26 updating version 0 to 649 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: oid 26 version 649 is already newer than 644 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: oid 26 updating version 649 to 650 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 650 (ObjNum 278 snap 80 seq_num 278) dirty exists 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:716: left oid 26 (ObjNum 278 snap 80 seq_num 278) 2024-09-10T16:11:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:713: done (6 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:714: done (5 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:715: done (4 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:716: done (3 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:717: done (2 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:718: done (1 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:719: done (0 left) 2024-09-10T16:11:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:720: snap_remove snap 63 2024-09-10T16:11:27.860 INFO:tasks.rados.rados.0.smithi165.stdout:720: done (0 left) 2024-09-10T16:11:27.860 INFO:tasks.rados.rados.0.smithi165.stdout:721: snap_remove snap 76 2024-09-10T16:11:28.869 INFO:tasks.rados.rados.0.smithi165.stdout:721: done (0 left) 2024-09-10T16:11:28.947 INFO:tasks.rados.rados.0.smithi165.stdout:722: copy_from oid 34 from oid 38 current snap is 80 2024-09-10T16:11:28.947 INFO:tasks.rados.rados.0.smithi165.stdout:723: snap_remove snap 44 2024-09-10T16:11:29.837 INFO:tasks.rados.rados.0.smithi165.stdout:723: done (1 left) 2024-09-10T16:11:29.837 INFO:tasks.rados.rados.0.smithi165.stdout:724: snap_create 2024-09-10T16:11:29.837 INFO:tasks.rados.rados.0.smithi165.stdout:722: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:11:29.838 INFO:tasks.rados.rados.0.smithi165.stdout:722: finishing copy_from to smithi16534841-34 2024-09-10T16:11:29.838 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 224 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-10T16:11:31.065 INFO:tasks.rados.rados.0.smithi165.stdout:722: done (1 left) 2024-09-10T16:11:31.065 INFO:tasks.rados.rados.0.smithi165.stdout:724: done (0 left) 2024-09-10T16:11:31.065 INFO:tasks.rados.rados.0.smithi165.stdout:725: snap_remove snap 69 2024-09-10T16:11:32.068 INFO:tasks.rados.rados.0.smithi165.stdout:725: done (0 left) 2024-09-10T16:11:32.068 INFO:tasks.rados.rados.0.smithi165.stdout:726: snap_remove snap 78 2024-09-10T16:11:33.069 INFO:tasks.rados.rados.0.smithi165.stdout:726: done (0 left) 2024-09-10T16:11:33.069 INFO:tasks.rados.rados.0.smithi165.stdout:727: read oid 45 snap -1 2024-09-10T16:11:33.069 INFO:tasks.rados.rados.0.smithi165.stdout:727: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:11:33.069 INFO:tasks.rados.rados.0.smithi165.stdout:728: snap_create 2024-09-10T16:11:33.077 INFO:tasks.rados.rados.0.smithi165.stdout:727: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:11:33.085 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1058008 2024-09-10T16:11:34.073 INFO:tasks.rados.rados.0.smithi165.stdout:727: done (1 left) 2024-09-10T16:11:34.074 INFO:tasks.rados.rados.0.smithi165.stdout:728: done (0 left) 2024-09-10T16:11:34.074 INFO:tasks.rados.rados.0.smithi165.stdout:729: rollback oid 1 current snap is 82 2024-09-10T16:11:34.074 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 79 2024-09-10T16:11:34.074 INFO:tasks.rados.rados.0.smithi165.stdout:730: write oid 16 current snap is 82 2024-09-10T16:11:34.074 INFO:tasks.rados.rados.0.smithi165.stdout:730: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-10T16:11:34.080 INFO:tasks.rados.rados.0.smithi165.stdout:730: writing smithi16534841-16 from 705960 to 1164388 tid 1 2024-09-10T16:11:34.087 INFO:tasks.rados.rados.0.smithi165.stdout:730: writing smithi16534841-16 from 1735311 to 2452735 tid 2 2024-09-10T16:11:34.087 INFO:tasks.rados.rados.0.smithi165.stdout:731: delete oid 11 current snap is 82 2024-09-10T16:11:34.087 INFO:tasks.rados.rados.0.smithi165.stdout:729: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:11:34.088 INFO:tasks.rados.rados.0.smithi165.stdout:729: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:11:34.088 INFO:tasks.rados.rados.0.smithi165.stdout:729: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:11:34.088 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 483 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:11:34.088 INFO:tasks.rados.rados.0.smithi165.stdout:730: finishing write tid 1 to smithi16534841-16 2024-09-10T16:11:34.089 INFO:tasks.rados.rados.0.smithi165.stdout:729: done (2 left) 2024-09-10T16:11:34.089 INFO:tasks.rados.rados.0.smithi165.stdout:731: done (1 left) 2024-09-10T16:11:34.089 INFO:tasks.rados.rados.0.smithi165.stdout:732: delete oid 21 current snap is 82 2024-09-10T16:11:34.091 INFO:tasks.rados.rados.0.smithi165.stdout:730: finishing write tid 2 to smithi16534841-16 2024-09-10T16:11:34.094 INFO:tasks.rados.rados.0.smithi165.stdout:730: finishing write tid 3 to smithi16534841-16 2024-09-10T16:11:34.094 INFO:tasks.rados.rados.0.smithi165.stdout:730: finishing write tid 4 to smithi16534841-16 2024-09-10T16:11:34.094 INFO:tasks.rados.rados.0.smithi165.stdout:730: oid 16 updating version 0 to 685 2024-09-10T16:11:34.095 INFO:tasks.rados.rados.0.smithi165.stdout:730: oid 16 version 685 is already newer than 684 2024-09-10T16:11:34.095 INFO:tasks.rados.rados.0.smithi165.stdout:730: oid 16 updating version 685 to 686 2024-09-10T16:11:34.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 686 (ObjNum 280 snap 82 seq_num 280) dirty exists 2024-09-10T16:11:34.095 INFO:tasks.rados.rados.0.smithi165.stdout:730: left oid 16 (ObjNum 280 snap 82 seq_num 280) 2024-09-10T16:11:34.096 INFO:tasks.rados.rados.0.smithi165.stdout:730: done (1 left) 2024-09-10T16:11:34.096 INFO:tasks.rados.rados.0.smithi165.stdout:732: done (0 left) 2024-09-10T16:11:34.096 INFO:tasks.rados.rados.0.smithi165.stdout:733: write (excl) oid 34 current snap is 82 2024-09-10T16:11:34.097 INFO:tasks.rados.rados.0.smithi165.stdout:733: seq_num 283 ranges {756381=400528,1910757=464751} 2024-09-10T16:11:34.102 INFO:tasks.rados.rados.0.smithi165.stdout:733: writing smithi16534841-34 from 756381 to 1156909 tid 1 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:733: writing smithi16534841-34 from 1910757 to 2375508 tid 2 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:734: read oid 7 snap 80 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:734: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:735: rollback oid 30 current snap is 82 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 80 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:736: read oid 27 snap 80 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:736: expect deleted 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:737: rollback oid 20 current snap is 82 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 81 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:738: write oid 4 current snap is 82 2024-09-10T16:11:34.110 INFO:tasks.rados.rados.0.smithi165.stdout:738: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-10T16:11:34.117 INFO:tasks.rados.rados.0.smithi165.stdout:738: writing smithi16534841-4 from 773188 to 1248965 tid 1 2024-09-10T16:11:34.126 INFO:tasks.rados.rados.0.smithi165.stdout:738: writing smithi16534841-4 from 1969239 to 2694001 tid 2 2024-09-10T16:11:34.126 INFO:tasks.rados.rados.0.smithi165.stdout:739: read oid 1 snap -1 2024-09-10T16:11:34.126 INFO:tasks.rados.rados.0.smithi165.stdout:739: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:11:34.126 INFO:tasks.rados.rados.0.smithi165.stdout:740: rollback oid 45 current snap is 82 2024-09-10T16:11:34.127 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 79 2024-09-10T16:11:34.127 INFO:tasks.rados.rados.0.smithi165.stdout:733: finishing write tid 1 to smithi16534841-34 2024-09-10T16:11:34.127 INFO:tasks.rados.rados.0.smithi165.stdout:737: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:11:34.127 INFO:tasks.rados.rados.0.smithi165.stdout:734: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:11:34.175 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 734810 2024-09-10T16:11:34.226 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2239214 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:737: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:733: finishing write tid 2 to smithi16534841-34 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:733: finishing write tid 3 to smithi16534841-34 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:733: finishing write tid 4 to smithi16534841-34 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:733: oid 34 updating version 0 to 545 2024-09-10T16:11:34.236 INFO:tasks.rados.rados.0.smithi165.stdout:733: oid 34 version 545 is already newer than 544 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:733: oid 34 version 545 is already newer than 543 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 545 (ObjNum 283 snap 82 seq_num 283) dirty exists 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:733: left oid 34 (ObjNum 283 snap 82 seq_num 283) 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:737: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 508 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:735: finishing rollback tid 0 to smithi16534841-30 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:735: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:735: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 690 (ObjNum 118 snap 34 seq_num 118) dirty exists 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:738: finishing write tid 1 to smithi16534841-4 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:733: done (7 left) 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:734: done (6 left) 2024-09-10T16:11:34.237 INFO:tasks.rados.rados.0.smithi165.stdout:735: done (5 left) 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:736: done (4 left) 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:737: done (3 left) 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:741: write oid 50 current snap is 82 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:740: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:738: finishing write tid 2 to smithi16534841-4 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:740: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:740: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 248 (ObjNum 153 snap 44 seq_num 153) dirty exists 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:738: finishing write tid 3 to smithi16534841-4 2024-09-10T16:11:34.238 INFO:tasks.rados.rados.0.smithi165.stdout:739: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:11:34.246 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1070331 2024-09-10T16:11:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:738: finishing write tid 4 to smithi16534841-4 2024-09-10T16:11:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:738: oid 4 updating version 0 to 272 2024-09-10T16:11:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:738: oid 4 updating version 272 to 273 2024-09-10T16:11:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:738: oid 4 version 273 is already newer than 271 2024-09-10T16:11:34.449 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 273 (ObjNum 284 snap 82 seq_num 284) dirty exists 2024-09-10T16:11:34.449 INFO:tasks.rados.rados.0.smithi165.stdout:738: left oid 4 (ObjNum 284 snap 82 seq_num 284) 2024-09-10T16:11:34.449 INFO:tasks.rados.rados.0.smithi165.stdout:741: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-10T16:11:34.454 INFO:tasks.rados.rados.0.smithi165.stdout:741: writing smithi16534841-50 from 789995 to 1341021 tid 1 2024-09-10T16:11:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:741: writing smithi16534841-50 from 2027721 to 2496141 tid 2 2024-09-10T16:11:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:738: done (3 left) 2024-09-10T16:11:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:739: done (2 left) 2024-09-10T16:11:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:740: done (1 left) 2024-09-10T16:11:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:742: write oid 2 current snap is 82 2024-09-10T16:11:34.462 INFO:tasks.rados.rados.0.smithi165.stdout:742: seq_num 286 ranges {406802=626275,1686203=728431} 2024-09-10T16:11:34.466 INFO:tasks.rados.rados.0.smithi165.stdout:742: writing smithi16534841-2 from 406802 to 1033077 tid 1 2024-09-10T16:11:34.474 INFO:tasks.rados.rados.0.smithi165.stdout:742: writing smithi16534841-2 from 1686203 to 2414634 tid 2 2024-09-10T16:11:34.474 INFO:tasks.rados.rados.0.smithi165.stdout:743: snap_remove snap 80 2024-09-10T16:11:34.833 INFO:tasks.rados.rados.0.smithi165.stdout:743: done (2 left) 2024-09-10T16:11:34.833 INFO:tasks.rados.rados.0.smithi165.stdout:744: write (excl) oid 44 current snap is 82 2024-09-10T16:11:34.833 INFO:tasks.rados.rados.0.smithi165.stdout:741: finishing write tid 1 to smithi16534841-50 2024-09-10T16:11:34.834 INFO:tasks.rados.rados.0.smithi165.stdout:744: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-10T16:11:34.839 INFO:tasks.rados.rados.0.smithi165.stdout:744: writing smithi16534841-44 from 423609 to 1125133 tid 1 2024-09-10T16:11:34.845 INFO:tasks.rados.rados.0.smithi165.stdout:744: writing smithi16534841-44 from 1628332 to 2100421 tid 2 2024-09-10T16:11:34.850 INFO:tasks.rados.rados.0.smithi165.stdout:744: writing smithi16534841-44 from 2648987 to 2823609 tid 3 2024-09-10T16:11:34.850 INFO:tasks.rados.rados.0.smithi165.stdout:745: copy_from oid 15 from oid 12 current snap is 82 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: finishing write tid 2 to smithi16534841-50 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: finishing write tid 3 to smithi16534841-50 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: finishing write tid 4 to smithi16534841-50 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: oid 50 updating version 0 to 304 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: oid 50 updating version 304 to 305 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: oid 50 version 305 is already newer than 303 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 305 (ObjNum 285 snap 82 seq_num 285) dirty exists 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:741: left oid 50 (ObjNum 285 snap 82 seq_num 285) 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:742: finishing write tid 1 to smithi16534841-2 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:742: finishing write tid 2 to smithi16534841-2 2024-09-10T16:11:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:742: finishing write tid 3 to smithi16534841-2 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: finishing write tid 4 to smithi16534841-2 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: oid 2 updating version 0 to 561 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: oid 2 version 561 is already newer than 560 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: oid 2 version 561 is already newer than 559 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 561 (ObjNum 286 snap 82 seq_num 286) dirty exists 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: left oid 2 (ObjNum 286 snap 82 seq_num 286) 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:741: done (3 left) 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:742: done (2 left) 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:746: read oid 39 snap -1 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:746: expect (ObjNum 254 snap 76 seq_num 254) 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:747: snap_remove snap 79 2024-09-10T16:11:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:744: finishing write tid 1 to smithi16534841-44 2024-09-10T16:11:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:744: finishing write tid 2 to smithi16534841-44 2024-09-10T16:11:35.483 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:11:35.483 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:11:35.484 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:11:35.487 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:11:35.487 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:11:35.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:35.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.596+0000 7fcb82353640 10 monclient: get_monmap_and_config 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.596+0000 7fcb82353640 10 monclient: build_initial_monmap 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.596+0000 7fcb82353640 10 monclient: monmap: 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:35.597437+0000 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:35.597437+0000 2024-09-10T16:11:35.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:35.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 Processor -- start 2024-09-10T16:11:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 -- start start 2024-09-10T16:11:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient: init 2024-09-10T16:11:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb7c09f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7c0a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb7c0a2980 0x7fcb7c13db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c0a2980 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb7c09fd10 con 0x7fcb7c09f930 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c09cd20 con 0x7fcb7c0a2980 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcb7c09a0d0 con 0x7fcb7c0a2550 2024-09-10T16:11:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.597+0000 7fcb82353640 10 monclient(hunting): authenticate will time out at 1074.084921s 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb7c0a2980 0x7fcb7c13db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb7c0a2980 0x7fcb7c13db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48414/0 (socket says 172.21.15.164:48414) 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 -- 172.21.15.164:0/2445998788 learned_addr learned my addr 172.21.15.164:0/2445998788 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): get_auth_request con 0x7fcb7c0a2980 auth_method 0 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb7c0a2550 auth_method 0 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 msgr2=0x7fcb7c13b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 msgr2=0x7fcb7c13b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient(hunting): handle_auth_done global_id 54189 payload 274 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 -- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 msgr2=0x7fcb7c13b5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 --2- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 -- 172.21.15.164:0/2445998788 >> v1:172.21.15.165:6789/0 conn(0x7fcb7c09f930 legacy=0x7fcb7c0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient: _finish_hunting 0 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 monclient: found mon.noname-a 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.598+0000 7fcb808f0640 1 -- 172.21.15.164:0/2445998788 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c09e840 con 0x7fcb7c0a2980 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb808f0640 1 -- 172.21.15.164:0/2445998788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2980 msgr2=0x7fcb7c13db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb808f0640 1 --2- 172.21.15.164:0/2445998788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2980 0x7fcb7c13db20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c09f500 tx=0x7fcb7402a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39c5a19254cf6a68 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 1 -- 172.21.15.164:0/2445998788 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74036070 con 0x7fcb7c0a2980 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: dump: 2024-09-10T16:11:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: _finish_auth 0 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 5 monclient: authenticate success, global_id 54189 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 1 -- 172.21.15.164:0/2445998788 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb74007800 con 0x7fcb7c0a2980 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 1 -- 172.21.15.164:0/2445998788 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb74007bd0 con 0x7fcb7c0a2980 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 4 set_mon_vals no callback set 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb7affd640 10 monclient: dump: 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 10 monclient: shutdown 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 -- 172.21.15.164:0/2445998788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2980 msgr2=0x7fcb7c13db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 --2- 172.21.15.164:0/2445998788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2980 0x7fcb7c13db20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c09f500 tx=0x7fcb7402a570 comp rx=0 tx=0).stop 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 -- 172.21.15.164:0/2445998788 shutdown_connections 2024-09-10T16:11:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 --2- 172.21.15.164:0/2445998788 >> v2:172.21.15.165:3300/0 conn(0x7fcb7c0a2550 0x7fcb7c13b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 --2- 172.21.15.164:0/2445998788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2980 0x7fcb7c13db20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 -- 172.21.15.164:0/2445998788 >> 172.21.15.164:0/2445998788 conn(0x7fcb7c0a3110 msgr2=0x7fcb7c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 -- 172.21.15.164:0/2445998788 shutdown_connections 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.601+0000 7fcb82353640 1 -- 172.21.15.164:0/2445998788 wait complete. 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.602+0000 7fcb82353640 10 monclient: build_initial_monmap 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.602+0000 7fcb82353640 10 monclient: monmap: 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.602+0000 7fcb82353640 1 Processor -- start 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 -- start start 2024-09-10T16:11:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient: init 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-b con 0x7fcb7c09f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-a con 0x7fcb7c0a2550 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 0x7fcb7c09af90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): picked mon.noname-c con 0x7fcb7c0a2980 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c09d930 con 0x7fcb7c0a2550 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcb7c09eb80 con 0x7fcb7c09f930 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcb7c09c8b0 con 0x7fcb7c0a2980 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb82353640 10 monclient(hunting): authenticate will time out at 1074.090516s 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 0x7fcb7c09af90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48420/0 (socket says 172.21.15.164:48420) 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2208869066 learned_addr learned my addr 172.21.15.164:0/2208869066 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request con 0x7fcb7c0a2550 auth_method 0 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): get_auth_request con 0x7fcb7c0a2980 auth_method 0 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 10 monclient(hunting): get_auth_request con 0x7fcb7c09f930 auth_method 0 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 msgr2=0x7fcb7c09a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 msgr2=0x7fcb7c09a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:11:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient(hunting): handle_auth_done global_id 54192 payload 824 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7bfff640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 msgr2=0x7fcb7c09af90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 0x7fcb7c09af90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 msgr2=0x7fcb7c09a510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2208869066 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb7c099e50 con 0x7fcb7c0a2550 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.603+0000 7fcb808f0640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 0x7fcb7c09af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:35.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb7b7fe640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 msgr2=0x7fcb7c09aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb7b7fe640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000d7d0 tx=0x7fcb7000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000b8d0 con 0x7fcb7c0a2550 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: dump: 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:35.622 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: _finish_auth 0 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb70019070 con 0x7fcb7c0a2550 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 5 monclient: authenticate success, global_id 54192 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 10 monclient: _renew_subs 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb7c09d620 con 0x7fcb7c0a2550 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7000bcd0 con 0x7fcb7c0a2550 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:35.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb78ff9640 10 monclient: dump: 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 10 monclient: _renew_subs 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb7a7fc640 4 set_mon_vals no callback set 2024-09-10T16:11:35.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.606+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb7c096830 con 0x7fcb7c0a2550 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb82353640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb82353640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb82353640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb7c0a0760 con 0x7fcb7c0a2550 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcb7001e460 con 0x7fcb7c0a2550 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb78ff9640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 0x7fcb680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.607+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(189..189 src has 1..189) ==== 4907+0+0 (secure 0 0 0) 0x7fcb700532a0 con 0x7fcb7c0a2550 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.608+0000 7fcb7bfff640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 0x7fcb680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.608+0000 7fcb7bfff640 10 monclient: get_auth_request con 0x7fcb6803e210 auth_method 0 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.610+0000 7fcb7bfff640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 0x7fcb680406d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c007c50 tx=0x7fcb6c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.612+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcb7001dce0 con 0x7fcb7c0a2550 2024-09-10T16:11:35.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.612+0000 7fcb78ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:35.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.612+0000 7fcb78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:35.840 INFO:tasks.rados.rados.0.smithi165.stdout:747: done (3 left) 2024-09-10T16:11:35.840 INFO:tasks.rados.rados.0.smithi165.stdout:748: snap_remove snap 81 2024-09-10T16:11:35.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.858+0000 7fcb82353640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:35.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.858+0000 7fcb82353640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.859+0000 7fcb82353640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.859+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcb7c077260 con 0x7fcb7c0a2550 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.859+0000 7fcb78ff9640 1 -- 172.21.15.164:0/2208869066 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fcb70014430 con 0x7fcb7c0a2550 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.859+0000 7fcb78ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.859+0000 7fcb78ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:35.861 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 msgr2=0x7fcb680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 0x7fcb680406d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb6c007c50 tx=0x7fcb6c009040 comp rx=0 tx=0).stop 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 10 monclient: shutdown 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 msgr2=0x7fcb7c09aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb7000d7d0 tx=0x7fcb7000dc90 comp rx=0 tx=0).stop 2024-09-10T16:11:35.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 shutdown_connections 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb6803e210 0x7fcb680406d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb7c0a2980 0x7fcb7c09af90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb7c0a2550 0x7fcb7c09aa50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 --2- 172.21.15.164:0/2208869066 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb7c09f930 0x7fcb7c09a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 >> 172.21.15.164:0/2208869066 conn(0x7fcb7c0a3110 msgr2=0x7fcb7c1be530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 shutdown_connections 2024-09-10T16:11:35.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:35.861+0000 7fcb82353640 1 -- 172.21.15.164:0/2208869066 wait complete. 2024-09-10T16:11:36.844 INFO:tasks.rados.rados.0.smithi165.stdout:748: done (3 left) 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:749: read oid 7 snap -1 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:749: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:750: rollback oid 25 current snap is 82 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:750: done (4 left) 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:751: write (excl) oid 5 current snap is 82 2024-09-10T16:11:36.845 INFO:tasks.rados.rados.0.smithi165.stdout:751: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-10T16:11:36.851 INFO:tasks.rados.rados.0.smithi165.stdout:751: writing smithi16534841-5 from 440416 to 1217189 tid 1 2024-09-10T16:11:36.858 INFO:tasks.rados.rados.0.smithi165.stdout:751: writing smithi16534841-5 from 1686814 to 2418914 tid 2 2024-09-10T16:11:36.858 INFO:tasks.rados.rados.0.smithi165.stdout:752: snap_create 2024-09-10T16:11:36.858 INFO:tasks.rados.rados.0.smithi165.stdout:746: expect (ObjNum 254 snap 76 seq_num 254) 2024-09-10T16:11:36.858 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: finishing write tid 3 to smithi16534841-44 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: finishing write tid 4 to smithi16534841-44 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: finishing write tid 5 to smithi16534841-44 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: oid 44 updating version 0 to 199 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: oid 44 updating version 199 to 202 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: oid 44 version 202 is already newer than 201 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: oid 44 version 202 is already newer than 200 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 202 (ObjNum 287 snap 82 seq_num 287) dirty exists 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:744: left oid 44 (ObjNum 287 snap 82 seq_num 287) 2024-09-10T16:11:36.964 INFO:tasks.rados.rados.0.smithi165.stdout:745: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:11:36.965 INFO:tasks.rados.rados.0.smithi165.stdout:745: finishing copy_from to smithi16534841-15 2024-09-10T16:11:36.965 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 54 (ObjNum 244 snap 72 seq_num 244) dirty exists 2024-09-10T16:11:36.965 INFO:tasks.rados.rados.0.smithi165.stdout:749: expect (ObjNum 160 snap 47 seq_num 160) 2024-09-10T16:11:37.000 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:37.063 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2455967 2024-09-10T16:11:37.070 INFO:tasks.rados.rados.0.smithi165.stdout:751: finishing write tid 1 to smithi16534841-5 2024-09-10T16:11:37.070 INFO:tasks.rados.rados.0.smithi165.stdout:751: finishing write tid 2 to smithi16534841-5 2024-09-10T16:11:37.070 INFO:tasks.rados.rados.0.smithi165.stdout:751: finishing write tid 3 to smithi16534841-5 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:751: finishing write tid 4 to smithi16534841-5 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:751: oid 5 updating version 0 to 291 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:751: oid 5 version 291 is already newer than 290 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:751: oid 5 updating version 291 to 292 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 292 (ObjNum 288 snap 82 seq_num 288) dirty exists 2024-09-10T16:11:37.071 INFO:tasks.rados.rados.0.smithi165.stdout:751: left oid 5 (ObjNum 288 snap 82 seq_num 288) 2024-09-10T16:11:38.082 INFO:tasks.rados.rados.0.smithi165.stdout:744: done (5 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:745: done (4 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:746: done (3 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:749: done (2 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:751: done (1 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:752: done (0 left) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:753: rollback oid 47 current snap is 83 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 82 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:754: read oid 40 snap -1 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:754: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:11:38.083 INFO:tasks.rados.rados.0.smithi165.stdout:755: snap_create 2024-09-10T16:11:38.087 INFO:tasks.rados.rados.0.smithi165.stdout:754: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:11:38.232 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2368588 2024-09-10T16:11:38.257 INFO:tasks.rados.rados.0.smithi165.stdout:753: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:11:38.257 INFO:tasks.rados.rados.0.smithi165.stdout:753: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:11:38.258 INFO:tasks.rados.rados.0.smithi165.stdout:753: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:11:38.258 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 567 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:11:38.874 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.982+0000 7f692a32b640 10 monclient: get_monmap_and_config 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.982+0000 7f692a32b640 10 monclient: build_initial_monmap 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.982+0000 7f692a32b640 10 monclient: monmap: 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:38.984 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:38.983577+0000 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:38.983577+0000 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:38.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:38.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 1 Processor -- start 2024-09-10T16:11:38.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 1 -- start start 2024-09-10T16:11:38.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 10 monclient: init 2024-09-10T16:11:38.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:38.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.983+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-b con 0x7f692409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f692409d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f692413a5c0 0x7f692413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-a con 0x7f692413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f692409fbc0 con 0x7f692409d190 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f69240a2a20 con 0x7f692413a5c0 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f692409c6f0 con 0x7f692409c310 2024-09-10T16:11:38.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:38.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f692a32b640 10 monclient(hunting): authenticate will time out at 1077.471041s 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f692413a5c0 0x7f692413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35716/0 (socket says 172.21.15.164:35716) 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 -- 172.21.15.164:0/3475987511 learned_addr learned my addr 172.21.15.164:0/3475987511 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): get_auth_request con 0x7f692409c310 auth_method 0 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): get_auth_request con 0x7f692413a5c0 auth_method 0 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:38.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f69288c8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient(hunting): handle_auth_done global_id 4880 payload 274 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 -- 172.21.15.164:0/3475987511 >> v2:172.21.15.164:3300/0 conn(0x7f692413a5c0 msgr2=0x7f692413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 --2- 172.21.15.164:0/3475987511 >> v2:172.21.15.164:3300/0 conn(0x7f692413a5c0 0x7f692413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 -- 172.21.15.164:0/3475987511 >> v1:172.21.15.165:6789/0 conn(0x7f692409d190 legacy=0x7f692409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:38.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 monclient: found mon.noname-b 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.984+0000 7f6923fff640 1 -- 172.21.15.164:0/3475987511 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f692409a4d0 con 0x7f692409c310 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6923fff640 1 -- 172.21.15.164:0/3475987511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 msgr2=0x7f692409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6923fff640 1 --2- 172.21.15.164:0/3475987511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6914009870 tx=0x7f691402a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f769dae62fa10a0d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 1 -- 172.21.15.164:0/3475987511 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914037070 con 0x7f692409c310 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: dump: 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:38.997 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 1 -- 172.21.15.164:0/3475987511 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6914032640 con 0x7f692409c310 2024-09-10T16:11:38.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 5 monclient: authenticate success, global_id 4880 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 1 -- 172.21.15.164:0/3475987511 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6914032a10 con 0x7f692409c310 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 4 set_mon_vals no callback set 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f6922ffd640 10 monclient: dump: 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:38.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 10 monclient: shutdown 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 1 -- 172.21.15.164:0/3475987511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 msgr2=0x7f692409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 1 --2- 172.21.15.164:0/3475987511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f6914009870 tx=0x7f691402a130 comp rx=0 tx=0).stop 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.987+0000 7f692a32b640 1 -- 172.21.15.164:0/3475987511 shutdown_connections 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 --2- 172.21.15.164:0/3475987511 >> v2:172.21.15.164:3300/0 conn(0x7f692413a5c0 0x7f692413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 --2- 172.21.15.164:0/3475987511 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f692409ea40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 -- 172.21.15.164:0/3475987511 >> 172.21.15.164:0/3475987511 conn(0x7f69240a3110 msgr2=0x7f69240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 -- 172.21.15.164:0/3475987511 shutdown_connections 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 -- 172.21.15.164:0/3475987511 wait complete. 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 10 monclient: build_initial_monmap 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 10 monclient: monmap: 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:39.001 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.988+0000 7f692a32b640 1 Processor -- start 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 -- start start 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient: init 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-c con 0x7f692409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-a con 0x7f692413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6924099a10 0x7f692408ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:39.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): picked mon.noname-b con 0x7f6924099a10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f69240a2230 con 0x7f692413a5c0 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f69240a20b0 con 0x7f6924099a10 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6924074e00 con 0x7f692409c310 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f692a32b640 10 monclient(hunting): authenticate will time out at 1077.476614s 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35732/0 (socket says 172.21.15.164:35732) 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48442/0 (socket says 172.21.15.164:48442) 2024-09-10T16:11:39.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 1 -- 172.21.15.164:0/1324121439 learned_addr learned my addr 172.21.15.164:0/1324121439 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): get_auth_request con 0x7f692409c310 auth_method 0 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): get_auth_request con 0x7f692413a5c0 auth_method 0 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f6923fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:39.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69288c8640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6924099a10 0x7f692408ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69288c8640 10 monclient(hunting): get_auth_request con 0x7f6924099a10 auth_method 0 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69288c8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69288c8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69288c8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.989+0000 7f69237fe640 10 monclient(hunting): handle_auth_done global_id 54198 payload 824 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 -- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 msgr2=0x7f69241c9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 -- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6924099a10 msgr2=0x7f692408ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:39.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6924099a10 0x7f692408ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 monclient: found mon.noname-a 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f69237fe640 1 -- 172.21.15.164:0/1324121439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69240919d0 con 0x7f692413a5c0 2024-09-10T16:11:39.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.990+0000 7f6923fff640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f69237fe640 1 -- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 msgr2=0x7f69240994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f69237fe640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f691800c3e0 tx=0x7f691800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69180119a0 con 0x7f692413a5c0 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:39.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: dump: 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: _finish_auth 0 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:11:39.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 5 monclient: authenticate success, global_id 54198 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 10 monclient: _renew_subs 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6918017070 con 0x7f692413a5c0 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6924093a50 con 0x7f692413a5c0 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6918011d80 con 0x7f692413a5c0 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f6920ff9640 10 monclient: dump: 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:39.010 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 10 monclient: _renew_subs 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69240929f0 con 0x7f692413a5c0 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.992+0000 7f69227fc640 4 set_mon_vals no callback set 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.993+0000 7f692a32b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.993+0000 7f692a32b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:39.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.993+0000 7f692a32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.993+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f692409f720 con 0x7f692413a5c0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.993+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f691801c460 con 0x7f692413a5c0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.994+0000 7f6920ff9640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 0x7f69100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.994+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(192..192 src has 1..192) ==== 4755+0+0 (secure 0 0 0) 0x7f6918052ad0 con 0x7f692413a5c0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.994+0000 7f6923fff640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 0x7f69100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.994+0000 7f6923fff640 10 monclient: get_auth_request con 0x7f691003e210 auth_method 0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.996+0000 7f6923fff640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 0x7f69100406d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6914000c00 tx=0x7f691402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.998+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6918054e20 con 0x7f692413a5c0 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.998+0000 7f6920ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:39.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:38.998+0000 7f6920ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:39.085 INFO:tasks.rados.rados.0.smithi165.stdout:753: done (2 left) 2024-09-10T16:11:39.085 INFO:tasks.rados.rados.0.smithi165.stdout:754: done (1 left) 2024-09-10T16:11:39.085 INFO:tasks.rados.rados.0.smithi165.stdout:755: done (0 left) 2024-09-10T16:11:39.085 INFO:tasks.rados.rados.0.smithi165.stdout:756: write (excl) oid 25 current snap is 84 2024-09-10T16:11:39.085 INFO:tasks.rados.rados.0.smithi165.stdout:756: seq_num 289 ranges {457223=568375,1461649=475758,2420186=437037} 2024-09-10T16:11:39.090 INFO:tasks.rados.rados.0.smithi165.stdout:756: writing smithi16534841-25 from 457223 to 1025598 tid 1 2024-09-10T16:11:39.095 INFO:tasks.rados.rados.0.smithi165.stdout:756: writing smithi16534841-25 from 1461649 to 1937407 tid 2 2024-09-10T16:11:39.101 INFO:tasks.rados.rados.0.smithi165.stdout:756: writing smithi16534841-25 from 2420186 to 2857223 tid 3 2024-09-10T16:11:39.101 INFO:tasks.rados.rados.0.smithi165.stdout:756: finishing write tid 1 to smithi16534841-25 2024-09-10T16:11:39.101 INFO:tasks.rados.rados.0.smithi165.stdout:757: copy_from oid 49 from oid 18 current snap is 84 2024-09-10T16:11:39.101 INFO:tasks.rados.rados.0.smithi165.stdout:758: write oid 21 current snap is 84 2024-09-10T16:11:39.102 INFO:tasks.rados.rados.0.smithi165.stdout:758: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-10T16:11:39.107 INFO:tasks.rados.rados.0.smithi165.stdout:758: writing smithi16534841-21 from 474030 to 1117654 tid 1 2024-09-10T16:11:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:758: writing smithi16534841-21 from 1520131 to 2255900 tid 2 2024-09-10T16:11:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:759: write oid 5 current snap is 84 2024-09-10T16:11:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:756: finishing write tid 2 to smithi16534841-25 2024-09-10T16:11:39.114 INFO:tasks.rados.rados.0.smithi165.stdout:759: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-10T16:11:39.121 INFO:tasks.rados.rados.0.smithi165.stdout:759: writing smithi16534841-5 from 490837 to 1209710 tid 1 2024-09-10T16:11:39.126 INFO:tasks.rados.rados.0.smithi165.stdout:759: writing smithi16534841-5 from 1862260 to 2341687 tid 2 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:759: writing smithi16534841-5 from 2875032 to 2890837 tid 3 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:760: snap_create 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: finishing write tid 3 to smithi16534841-25 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: finishing write tid 4 to smithi16534841-25 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: finishing write tid 5 to smithi16534841-25 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: oid 25 updating version 0 to 518 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: oid 25 updating version 518 to 521 2024-09-10T16:11:39.130 INFO:tasks.rados.rados.0.smithi165.stdout:756: oid 25 version 521 is already newer than 519 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:756: oid 25 version 521 is already newer than 520 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 521 (ObjNum 289 snap 84 seq_num 289) dirty exists 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:756: left oid 25 (ObjNum 289 snap 84 seq_num 289) 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: finishing write tid 1 to smithi16534841-21 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: finishing write tid 2 to smithi16534841-21 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: finishing write tid 3 to smithi16534841-21 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: finishing write tid 4 to smithi16534841-21 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: oid 21 updating version 0 to 524 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: oid 21 version 524 is already newer than 523 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: oid 21 version 524 is already newer than 522 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 524 (ObjNum 290 snap 84 seq_num 290) dirty exists 2024-09-10T16:11:39.131 INFO:tasks.rados.rados.0.smithi165.stdout:758: left oid 21 (ObjNum 290 snap 84 seq_num 290) 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f692a32b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f692a32b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f692a32b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f692409d620 con 0x7f692413a5c0 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f6920ff9640 1 -- 172.21.15.164:0/1324121439 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f6918012420 con 0x7f692413a5c0 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f6920ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.241+0000 7f6920ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:39.243 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":56,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 msgr2=0x7f69100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 0x7f69100406d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f6914000c00 tx=0x7f691402e000 comp rx=0 tx=0).stop 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 10 monclient: shutdown 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 msgr2=0x7f69240994d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f691800c3e0 tx=0x7f691800c8a0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 shutdown_connections 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f691003e210 0x7f69100406d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6924099a10 0x7f692408ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f692413a5c0 0x7f69240994d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 --2- 172.21.15.164:0/1324121439 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f692409c310 0x7f69241c9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 >> 172.21.15.164:0/1324121439 conn(0x7f69240a3110 msgr2=0x7f69241397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 shutdown_connections 2024-09-10T16:11:39.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:39.243+0000 7f692a32b640 1 -- 172.21.15.164:0/1324121439 wait complete. 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:757: finishing copy_from to smithi16534841-49 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:757: got expected ENOENT (src dne) 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:756: done (4 left) 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:758: done (3 left) 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:760: done (2 left) 2024-09-10T16:11:39.835 INFO:tasks.rados.rados.0.smithi165.stdout:761: read oid 39 snap -1 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:761: expect (ObjNum 254 snap 76 seq_num 254) 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:762: read oid 16 snap -1 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:762: expect (ObjNum 280 snap 82 seq_num 280) 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:763: copy_from oid 24 from oid 22 current snap is 85 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:757: finishing copy_from racing read to smithi16534841-49 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:757: done (4 left) 2024-09-10T16:11:39.836 INFO:tasks.rados.rados.0.smithi165.stdout:764: snap_create 2024-09-10T16:11:39.838 INFO:tasks.rados.rados.0.smithi165.stdout:761: expect (ObjNum 254 snap 76 seq_num 254) 2024-09-10T16:11:39.915 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1184355 2024-09-10T16:11:39.940 INFO:tasks.rados.rados.0.smithi165.stdout:759: finishing write tid 1 to smithi16534841-5 2024-09-10T16:11:39.940 INFO:tasks.rados.rados.0.smithi165.stdout:762: expect (ObjNum 280 snap 82 seq_num 280) 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: finishing write tid 2 to smithi16534841-5 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: finishing write tid 3 to smithi16534841-5 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: finishing write tid 4 to smithi16534841-5 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: finishing write tid 5 to smithi16534841-5 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: oid 5 updating version 0 to 296 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: oid 5 updating version 296 to 297 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: oid 5 version 297 is already newer than 294 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:759: oid 5 version 297 is already newer than 295 2024-09-10T16:11:40.067 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 297 (ObjNum 291 snap 84 seq_num 291) dirty exists 2024-09-10T16:11:40.068 INFO:tasks.rados.rados.0.smithi165.stdout:759: left oid 5 (ObjNum 291 snap 84 seq_num 291) 2024-09-10T16:11:40.068 INFO:tasks.rados.rados.0.smithi165.stdout:763: finishing copy_from to smithi16534841-24 2024-09-10T16:11:40.068 INFO:tasks.rados.rados.0.smithi165.stdout:763: got expected ENOENT (src dne) 2024-09-10T16:11:40.068 INFO:tasks.rados.rados.0.smithi165.stdout:763: finishing copy_from racing read to smithi16534841-24 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:759: done (4 left) 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:761: done (3 left) 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:762: done (2 left) 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:763: done (1 left) 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:764: done (0 left) 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:765: read oid 8 snap 85 2024-09-10T16:11:40.840 INFO:tasks.rados.rados.0.smithi165.stdout:765: expect (ObjNum 275 snap 80 seq_num 275) 2024-09-10T16:11:40.841 INFO:tasks.rados.rados.0.smithi165.stdout:766: write (excl) oid 22 current snap is 86 2024-09-10T16:11:40.841 INFO:tasks.rados.rados.0.smithi165.stdout:766: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-10T16:11:40.847 INFO:tasks.rados.rados.0.smithi165.stdout:766: writing smithi16534841-22 from 507644 to 1301766 tid 1 2024-09-10T16:11:40.854 INFO:tasks.rados.rados.0.smithi165.stdout:766: writing smithi16534841-22 from 1920742 to 2543827 tid 2 2024-09-10T16:11:40.854 INFO:tasks.rados.rados.0.smithi165.stdout:767: delete oid 11 current snap is 86 2024-09-10T16:11:40.854 INFO:tasks.rados.rados.0.smithi165.stdout:765: expect (ObjNum 275 snap 80 seq_num 275) 2024-09-10T16:11:40.898 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:40.963 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2501357 2024-09-10T16:11:40.970 INFO:tasks.rados.rados.0.smithi165.stdout:766: finishing write tid 1 to smithi16534841-22 2024-09-10T16:11:40.970 INFO:tasks.rados.rados.0.smithi165.stdout:765: done (2 left) 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:767: done (1 left) 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:768: write oid 46 current snap is 86 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: finishing write tid 2 to smithi16534841-22 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: finishing write tid 3 to smithi16534841-22 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: finishing write tid 4 to smithi16534841-22 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: oid 22 updating version 0 to 267 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: oid 22 version 267 is already newer than 265 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: oid 22 version 267 is already newer than 266 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 267 (ObjNum 292 snap 86 seq_num 292) dirty exists 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:766: left oid 22 (ObjNum 292 snap 86 seq_num 292) 2024-09-10T16:11:40.971 INFO:tasks.rados.rados.0.smithi165.stdout:768: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-10T16:11:40.975 INFO:tasks.rados.rados.0.smithi165.stdout:768: writing smithi16534841-46 from 541258 to 1085878 tid 1 2024-09-10T16:11:40.982 INFO:tasks.rados.rados.0.smithi165.stdout:768: writing smithi16534841-46 from 1637706 to 2264460 tid 2 2024-09-10T16:11:40.986 INFO:tasks.rados.rados.0.smithi165.stdout:768: writing smithi16534841-46 from 2840948 to 2941258 tid 3 2024-09-10T16:11:40.986 INFO:tasks.rados.rados.0.smithi165.stdout:766: done (1 left) 2024-09-10T16:11:40.986 INFO:tasks.rados.rados.0.smithi165.stdout:769: read oid 6 snap -1 2024-09-10T16:11:40.986 INFO:tasks.rados.rados.0.smithi165.stdout:769: expect (ObjNum 172 snap 52 seq_num 172) 2024-09-10T16:11:40.987 INFO:tasks.rados.rados.0.smithi165.stdout:770: write (excl) oid 11 current snap is 86 2024-09-10T16:11:40.987 INFO:tasks.rados.rados.0.smithi165.stdout:768: finishing write tid 1 to smithi16534841-46 2024-09-10T16:11:40.987 INFO:tasks.rados.rados.0.smithi165.stdout:770: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2024-09-10T16:11:40.992 INFO:tasks.rados.rados.0.smithi165.stdout:770: writing smithi16534841-11 from 558065 to 1177934 tid 1 2024-09-10T16:11:40.998 INFO:tasks.rados.rados.0.smithi165.stdout:770: writing smithi16534841-11 from 1579835 to 2066600 tid 2 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:770: writing smithi16534841-11 from 2468371 to 2958065 tid 3 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:771: read oid 23 snap 85 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:771: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:772: read oid 10 snap -1 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:772: expect deleted 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:773: read oid 35 snap -1 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:773: expect (ObjNum 276 snap 80 seq_num 276) 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:774: delete oid 27 current snap is 86 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:768: finishing write tid 2 to smithi16534841-46 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:768: finishing write tid 3 to smithi16534841-46 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:768: finishing write tid 4 to smithi16534841-46 2024-09-10T16:11:41.004 INFO:tasks.rados.rados.0.smithi165.stdout:768: finishing write tid 5 to smithi16534841-46 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:768: oid 46 updating version 0 to 206 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:768: oid 46 version 206 is already newer than 204 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:768: oid 46 version 206 is already newer than 203 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:768: oid 46 version 206 is already newer than 205 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 206 (ObjNum 294 snap 86 seq_num 294) dirty exists 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:768: left oid 46 (ObjNum 294 snap 86 seq_num 294) 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:770: finishing write tid 1 to smithi16534841-11 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:769: expect (ObjNum 172 snap 52 seq_num 172) 2024-09-10T16:11:41.005 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:41.050 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2790567 2024-09-10T16:11:41.051 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: finishing write tid 2 to smithi16534841-11 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: finishing write tid 3 to smithi16534841-11 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: finishing write tid 4 to smithi16534841-11 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: finishing write tid 5 to smithi16534841-11 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: oid 11 updating version 0 to 313 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: oid 11 updating version 313 to 315 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: oid 11 version 315 is already newer than 314 2024-09-10T16:11:41.082 INFO:tasks.rados.rados.0.smithi165.stdout:770: oid 11 updating version 315 to 316 2024-09-10T16:11:41.083 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 316 (ObjNum 295 snap 86 seq_num 295) dirty exists 2024-09-10T16:11:41.083 INFO:tasks.rados.rados.0.smithi165.stdout:770: left oid 11 (ObjNum 295 snap 86 seq_num 295) 2024-09-10T16:11:41.083 INFO:tasks.rados.rados.0.smithi165.stdout:771: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:11:41.230 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2954561 2024-09-10T16:11:41.230 INFO:tasks.rados.rados.0.smithi165.stdout:773: expect (ObjNum 276 snap 80 seq_num 276) 2024-09-10T16:11:41.364 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:768: done (6 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:769: done (5 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:770: done (4 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:771: done (3 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:772: done (2 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:773: done (1 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:774: done (0 left) 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:775: write (excl) oid 31 current snap is 86 2024-09-10T16:11:41.432 INFO:tasks.rados.rados.0.smithi165.stdout:775: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-10T16:11:41.437 INFO:tasks.rados.rados.0.smithi165.stdout:775: writing smithi16534841-31 from 591679 to 1078399 tid 1 2024-09-10T16:11:41.444 INFO:tasks.rados.rados.0.smithi165.stdout:775: writing smithi16534841-31 from 1813152 to 2303586 tid 2 2024-09-10T16:11:41.444 INFO:tasks.rados.rados.0.smithi165.stdout:776: snap_create 2024-09-10T16:11:41.444 INFO:tasks.rados.rados.0.smithi165.stdout:775: finishing write tid 1 to smithi16534841-31 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: finishing write tid 2 to smithi16534841-31 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: finishing write tid 3 to smithi16534841-31 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: finishing write tid 4 to smithi16534841-31 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: oid 31 updating version 0 to 707 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: oid 31 version 707 is already newer than 706 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: oid 31 version 707 is already newer than 705 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 707 (ObjNum 297 snap 86 seq_num 297) dirty exists 2024-09-10T16:11:41.447 INFO:tasks.rados.rados.0.smithi165.stdout:775: left oid 31 (ObjNum 297 snap 86 seq_num 297) 2024-09-10T16:11:42.257 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:42.284 INFO:tasks.rados.rados.0.smithi165.stdout:775: done (1 left) 2024-09-10T16:11:42.284 INFO:tasks.rados.rados.0.smithi165.stdout:776: done (0 left) 2024-09-10T16:11:42.284 INFO:tasks.rados.rados.0.smithi165.stdout:777: read oid 1 snap -1 2024-09-10T16:11:42.284 INFO:tasks.rados.rados.0.smithi165.stdout:777: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:11:42.284 INFO:tasks.rados.rados.0.smithi165.stdout:778: rollback oid 27 current snap is 87 2024-09-10T16:11:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 83 2024-09-10T16:11:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:779: rollback oid 8 current snap is 87 2024-09-10T16:11:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 85 2024-09-10T16:11:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:780: write oid 7 current snap is 87 2024-09-10T16:11:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:780: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-10T16:11:42.291 INFO:tasks.rados.rados.0.smithi165.stdout:780: writing smithi16534841-7 from 608486 to 1170455 tid 1 2024-09-10T16:11:42.298 INFO:tasks.rados.rados.0.smithi165.stdout:780: writing smithi16534841-7 from 1871634 to 2505726 tid 2 2024-09-10T16:11:42.301 INFO:tasks.rados.rados.0.smithi165.stdout:780: writing smithi16534841-7 from 2950640 to 3008486 tid 3 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:781: write (excl) oid 42 current snap is 87 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:778: finishing rollback tid 1 to smithi16534841-27 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 0 (ObjNum 225 snap 0 seq_num 1163008928) dirty dne 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:779: finishing rollback tid 0 to smithi16534841-8 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:779: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:11:42.302 INFO:tasks.rados.rados.0.smithi165.stdout:781: seq_num 299 ranges {625293=637218,1813763=494103,2978063=47230} 2024-09-10T16:11:42.308 INFO:tasks.rados.rados.0.smithi165.stdout:781: writing smithi16534841-42 from 625293 to 1262511 tid 1 2024-09-10T16:11:42.314 INFO:tasks.rados.rados.0.smithi165.stdout:781: writing smithi16534841-42 from 1813763 to 2307866 tid 2 2024-09-10T16:11:42.318 INFO:tasks.rados.rados.0.smithi165.stdout:781: writing smithi16534841-42 from 2978063 to 3025293 tid 3 2024-09-10T16:11:42.318 INFO:tasks.rados.rados.0.smithi165.stdout:778: done (4 left) 2024-09-10T16:11:42.318 INFO:tasks.rados.rados.0.smithi165.stdout:782: snap_remove snap 83 2024-09-10T16:11:42.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.368+0000 7f87b8239640 10 monclient: get_monmap_and_config 2024-09-10T16:11:42.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.368+0000 7f87b8239640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.368+0000 7f87b8239640 10 monclient: monmap: 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:42.369083+0000 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:42.369083+0000 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:42.370 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:42.371 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:42.371 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.368+0000 7f87b8239640 1 Processor -- start 2024-09-10T16:11:42.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 -- start start 2024-09-10T16:11:42.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient: init 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87b006cb70 0x7f87b0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-a con 0x7f87b006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f87b00a46a0 0x7f87b00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-c con 0x7f87b00a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87b0137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b009ce50 con 0x7f87b0137ca0 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f87b006db60 con 0x7f87b006cb70 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f87b006cf50 con 0x7f87b00a46a0 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b8239640 10 monclient(hunting): authenticate will time out at 1080.856614s 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87b006cb70 0x7f87b0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87b006cb70 0x7f87b0134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48448/0 (socket says 172.21.15.164:48448) 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2269762146 learned_addr learned my addr 172.21.15.164:0/2269762146 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): get_auth_request con 0x7f87b006cb70 auth_method 0 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b57d4640 1 --2- 172.21.15.164:0/2269762146 >> v2:172.21.15.165:3300/0 conn(0x7f87b00a46a0 0x7f87b00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b57d4640 10 monclient(hunting): get_auth_request con 0x7f87b00a46a0 auth_method 0 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b57d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b57d4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b57d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.369+0000 7f87b5fd5640 10 monclient(hunting): handle_auth_done global_id 54201 payload 274 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2269762146 >> v2:172.21.15.165:3300/0 conn(0x7f87b00a46a0 msgr2=0x7f87b00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2269762146 >> v2:172.21.15.165:3300/0 conn(0x7f87b00a46a0 0x7f87b00a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2269762146 >> v1:172.21.15.165:6789/0 conn(0x7f87b0137ca0 legacy=0x7f87b0138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 1 monclient: found mon.noname-a 2024-09-10T16:11:42.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.370+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2269762146 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b009cb70 con 0x7f87b006cb70 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2269762146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b006cb70 msgr2=0x7f87b0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2269762146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b006cb70 0x7f87b0134f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098a0 tx=0x7f87a802a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b2e727b400bf38 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 1 -- 172.21.15.164:0/2269762146 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8037070 con 0x7f87b006cb70 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:42.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: dump: 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b8239640 5 monclient: authenticate success, global_id 54201 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 1 -- 172.21.15.164:0/2269762146 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a8032640 con 0x7f87b006cb70 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b8239640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b8239640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b8239640 4 set_mon_vals no callback set 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 1 -- 172.21.15.164:0/2269762146 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8032a10 con 0x7f87b006cb70 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.372+0000 7f87b4fd3640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b4fd3640 10 monclient: dump: 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 10 monclient: shutdown 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 -- 172.21.15.164:0/2269762146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b006cb70 msgr2=0x7f87b0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 --2- 172.21.15.164:0/2269762146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b006cb70 0x7f87b0134f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f87a80098a0 tx=0x7f87a802a110 comp rx=0 tx=0).stop 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 -- 172.21.15.164:0/2269762146 shutdown_connections 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 --2- 172.21.15.164:0/2269762146 >> v2:172.21.15.165:3300/0 conn(0x7f87b00a46a0 0x7f87b00a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 --2- 172.21.15.164:0/2269762146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b006cb70 0x7f87b0134f70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 -- 172.21.15.164:0/2269762146 >> 172.21.15.164:0/2269762146 conn(0x7f87b00aa900 msgr2=0x7f87b00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 -- 172.21.15.164:0/2269762146 shutdown_connections 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.373+0000 7f87b8239640 1 -- 172.21.15.164:0/2269762146 wait complete. 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 10 monclient: monmap: 2024-09-10T16:11:42.389 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 1 Processor -- start 2024-09-10T16:11:42.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 1 -- start start 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 10 monclient: init 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.374+0000 7f87b8239640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-b con 0x7f87b006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b00a46a0 0x7f87b012e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-c con 0x7f87b00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): picked mon.noname-a con 0x7f87b0137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87b01389c0 con 0x7f87b0137ca0 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87b0138840 con 0x7f87b006cb70 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87b0140040 con 0x7f87b00a46a0 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b8239640 10 monclient(hunting): authenticate will time out at 1080.862089s 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b00a46a0 0x7f87b012e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48460/0 (socket says 172.21.15.164:48460) 2024-09-10T16:11:42.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 -- 172.21.15.164:0/2840608711 learned_addr learned my addr 172.21.15.164:0/2840608711 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): get_auth_request con 0x7f87b00a46a0 auth_method 0 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): get_auth_request con 0x7f87b0137ca0 auth_method 0 2024-09-10T16:11:42.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b57d4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 10 monclient(hunting): get_auth_request con 0x7f87b006cb70 auth_method 0 2024-09-10T16:11:42.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 msgr2=0x7f87b012e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 msgr2=0x7f87b012e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient(hunting): handle_auth_done global_id 54204 payload 824 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b00a46a0 msgr2=0x7f87b012e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b00a46a0 0x7f87b012e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 msgr2=0x7f87b012e370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:11:42.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 monclient: found mon.noname-a 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.375+0000 7f87b67d6640 1 -- 172.21.15.164:0/2840608711 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b00a30e0 con 0x7f87b0137ca0 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b67d6640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 msgr2=0x7f87b01cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b67d6640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c780 tx=0x7f87a000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a000ac60 con 0x7f87b0137ca0 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: dump: 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a000ae00 con 0x7f87b0137ca0 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a0004ba0 con 0x7f87b0137ca0 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87ae7fc640 10 monclient: dump: 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b8239640 5 monclient: authenticate success, global_id 54204 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b8239640 10 monclient: _renew_subs 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b8239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b00a33c0 con 0x7f87b0137ca0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.377+0000 7f87affff640 4 set_mon_vals no callback set 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 10 monclient: _renew_subs 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b00a1740 con 0x7f87b0137ca0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f87a0005030 con 0x7f87b0137ca0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.378+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87b0134f70 con 0x7f87b0137ca0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.379+0000 7f87ae7fc640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 0x7f879c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.379+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 4739+0+0 (secure 0 0 0) 0x7f87a0052ef0 con 0x7f87b0137ca0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.379+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 0x7f879c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.379+0000 7f87b5fd5640 10 monclient: get_auth_request con 0x7f879c03e260 auth_method 0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.381+0000 7f87b5fd5640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 0x7f879c040720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f87a8034000 tx=0x7f87a802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.383+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f87a0020070 con 0x7f87b0137ca0 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.383+0000 7f87ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.383+0000 7f87ae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:42.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87b8239640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87b8239640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87b8239640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f87b012edf0 con 0x7f87b0137ca0 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87ae7fc640 1 -- 172.21.15.164:0/2840608711 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f87a0005320 con 0x7f87b0137ca0 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.625+0000 7f87ae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:42.627 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 msgr2=0x7f879c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 0x7f879c040720 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f87a8034000 tx=0x7f87a802e000 comp rx=0 tx=0).stop 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 10 monclient: shutdown 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 msgr2=0x7f87b01cd030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f87a000c780 tx=0x7f87a000cc40 comp rx=0 tx=0).stop 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 shutdown_connections 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f879c03e260 0x7f879c040720 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b0137ca0 0x7f87b01cd030 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b00a46a0 0x7f87b012e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 --2- 172.21.15.164:0/2840608711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b006cb70 0x7f87b012e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.627+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 >> 172.21.15.164:0/2840608711 conn(0x7f87b00aa900 msgr2=0x7f87b00a7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.628+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 shutdown_connections 2024-09-10T16:11:42.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.628+0000 7f87b8239640 1 -- 172.21.15.164:0/2840608711 wait complete. 2024-09-10T16:11:42.639 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:11:42.639 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:11:42.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.735+0000 7f938f07f640 10 monclient: get_monmap_and_config 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.735+0000 7f938f07f640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.735+0000 7f938f07f640 10 monclient: monmap: 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:42.736623+0000 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:42.736623+0000 2024-09-10T16:11:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:42.738 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 Processor -- start 2024-09-10T16:11:42.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 -- start start 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient: init 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-b con 0x7f9388074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f93880a1090 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-a con 0x7f93880a14d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f938809ebc0 con 0x7f93880a1090 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f93880698a0 con 0x7f93880a14d0 2024-09-10T16:11:42.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f938806bfb0 con 0x7f9388074af0 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.736+0000 7f938f07f640 10 monclient(hunting): authenticate will time out at 1081.223976s 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48464/0 (socket says 172.21.15.164:48464) 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 1 -- 172.21.15.164:0/281005642 learned_addr learned my addr 172.21.15.164:0/281005642 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): get_auth_request con 0x7f93880a14d0 auth_method 0 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): get_auth_request con 0x7f9388074af0 auth_method 0 2024-09-10T16:11:42.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient(hunting): handle_auth_done global_id 4889 payload 274 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 -- 172.21.15.164:0/281005642 >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 msgr2=0x7f93880a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 --2- 172.21.15.164:0/281005642 >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 -- 172.21.15.164:0/281005642 >> v1:172.21.15.165:6789/0 conn(0x7f93880a1090 legacy=0x7f93880a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 monclient: found mon.noname-b 2024-09-10T16:11:42.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:42.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938ce1b640 1 -- 172.21.15.164:0/281005642 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93880a1940 con 0x7f9388074af0 2024-09-10T16:11:42.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.737+0000 7f938d61c640 1 --2- 172.21.15.164:0/281005642 >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938ce1b640 1 -- 172.21.15.164:0/281005642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 msgr2=0x7f9388137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938ce1b640 1 --2- 172.21.15.164:0/281005642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388137e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f937c0097a0 tx=0x7f937c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b727786ca40b5f92 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 1 -- 172.21.15.164:0/281005642 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c037070 con 0x7f9388074af0 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: dump: 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.751 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 1 -- 172.21.15.164:0/281005642 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f937c032440 con 0x7f9388074af0 2024-09-10T16:11:42.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 5 monclient: authenticate success, global_id 4889 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 1 -- 172.21.15.164:0/281005642 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c032810 con 0x7f9388074af0 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 4 set_mon_vals no callback set 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f93877fe640 10 monclient: dump: 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.753 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 10 monclient: shutdown 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 -- 172.21.15.164:0/281005642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 msgr2=0x7f9388137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 --2- 172.21.15.164:0/281005642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388137e70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f937c0097a0 tx=0x7f937c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 -- 172.21.15.164:0/281005642 shutdown_connections 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 --2- 172.21.15.164:0/281005642 >> v2:172.21.15.164:3300/0 conn(0x7f93880a14d0 0x7f93880a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 --2- 172.21.15.164:0/281005642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388137e70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 -- 172.21.15.164:0/281005642 >> 172.21.15.164:0/281005642 conn(0x7f93880a3fb0 msgr2=0x7f93880a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 -- 172.21.15.164:0/281005642 shutdown_connections 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.740+0000 7f938f07f640 1 -- 172.21.15.164:0/281005642 wait complete. 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.741+0000 7f938f07f640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.741+0000 7f938f07f640 10 monclient: monmap: 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:42.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:42.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.741+0000 7f938f07f640 1 Processor -- start 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.741+0000 7f938f07f640 1 -- start start 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.741+0000 7f938f07f640 10 monclient: init 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-c con 0x7f9388074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93880a1090 0x7f9388131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-b con 0x7f93880a1090 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 0x7f938809bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): picked mon.noname-a con 0x7f93880a14d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9388069370 con 0x7f93880a14d0 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9388074ed0 con 0x7f93880a1090 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f93880a3b10 con 0x7f9388074af0 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938f07f640 10 monclient(hunting): authenticate will time out at 1081.229526s 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35752/0 (socket says 172.21.15.164:35752) 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 1 -- 172.21.15.164:0/3563961597 learned_addr learned my addr 172.21.15.164:0/3563961597 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): get_auth_request con 0x7f9388074af0 auth_method 0 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 0x7f938809bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): get_auth_request con 0x7f93880a14d0 auth_method 0 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f9387fff640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93880a1090 0x7f9388131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient(hunting): handle_auth_done global_id 54210 payload 824 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 msgr2=0x7f9388131a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388131a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93880a1090 msgr2=0x7f9388131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93880a1090 0x7f9388131f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 monclient: found mon.noname-a 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f938809ddb0 con 0x7f93880a14d0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f9387fff640 10 monclient: get_auth_request con 0x7f93880a1090 auth_method 0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f938ce1b640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9388074af0 0x7f9388131a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.742+0000 7f9387fff640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f93880a1090 0x7f9388131f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 msgr2=0x7f938809bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938d61c640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 0x7f938809bb20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9380009e50 tx=0x7f938000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938000cd90 con 0x7f93880a14d0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: dump: 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.761 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9380003990 con 0x7f93880a14d0 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 5 monclient: authenticate success, global_id 54210 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 10 monclient: _renew_subs 2024-09-10T16:11:42.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f938809dab0 con 0x7f93880a14d0 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380003d80 con 0x7f93880a14d0 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f93857fa640 10 monclient: dump: 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.763 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f9386ffd640 4 set_mon_vals no callback set 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 10 monclient: _renew_subs 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.745+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f938812d020 con 0x7f93880a14d0 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938f07f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938f07f640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 0x7f938809c060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f93881385c0 con 0x7f93881cd4e0 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9380005d10 con 0x7f93880a14d0 2024-09-10T16:11:42.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 0x7f938809c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient: get_auth_request con 0x7f93881cd4e0 auth_method 0 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f93857fa640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f936c03e210 0x7f936c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 10 monclient(hunting): handle_auth_done global_id 54213 payload 824 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 4739+0+0 (secure 0 0 0) 0x7f93800530c0 con 0x7f93880a14d0 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f938ce1b640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9388137e70 con 0x7f93881cd4e0 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f9387fff640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f936c03e210 0x7f936c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.746+0000 7f9387fff640 10 monclient: get_auth_request con 0x7f936c03e210 auth_method 0 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f938ce1b640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 msgr2=0x7f938809c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f938ce1b640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 0x7f938809c060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f938809d940 tx=0x7f937c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c043070 con 0x7f93881cd4e0 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f93857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f93857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.748+0000 7f93857fa640 10 monclient: dump: 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.766 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f937c03ca60 con 0x7f93881cd4e0 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 msgr2=0x7f938809c060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 0x7f938809c060 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f938809d940 tx=0x7f937c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 msgr2=0x7f938809c060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f93857fa640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93881cd4e0 0x7f938809c060 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.749+0000 7f9387fff640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f936c03e210 0x7f936c0406d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9388135a30 tx=0x7f9374009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938f07f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938f07f640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 0x7f93881328e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938f07f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f938809c9b0 con 0x7f938809c5a0 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 0x7f93881328e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient: get_auth_request con 0x7f938809c5a0 auth_method 0 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 10 monclient(hunting): handle_auth_done global_id 54216 payload 824 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.755+0000 7f938ce1b640 1 -- 172.21.15.164:0/3563961597 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f938812d400 con 0x7f938809c5a0 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f938ce1b640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 msgr2=0x7f93881328e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f938ce1b640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 0x7f93881328e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f938806a810 tx=0x7f937c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f937c037040 con 0x7f938809c5a0 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f93857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f93857fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.757+0000 7f93857fa640 10 monclient: dump: 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.771 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f937c003b30 con 0x7f938809c5a0 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 msgr2=0x7f93881328e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 0x7f93881328e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f938806a810 tx=0x7f937c008030 comp rx=0 tx=0).stop 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 msgr2=0x7f93881328e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:42.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f93857fa640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f938809c5a0 0x7f93881328e0 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f938806a810 tx=0x7f937c008030 comp rx=0 tx=0).stop 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 72941, 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:42.773 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.774 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:42.776 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:42.777 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.778 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.781 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:42.783 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:42.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.785 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:11:42.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.786 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.786 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:42.786 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f936c03e210 msgr2=0x7f936c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f936c03e210 0x7f936c0406d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9388135a30 tx=0x7f9374009040 comp rx=0 tx=0).stop 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 10 monclient: shutdown 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 msgr2=0x7f938809bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 --2- 172.21.15.164:0/3563961597 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f93880a14d0 0x7f938809bb20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f9380009e50 tx=0x7f938000c490 comp rx=0 tx=0).stop 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 reap_dead start 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938d61c640 1 -- 172.21.15.164:0/3563961597 reap_dead start 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 shutdown_connections 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 >> 172.21.15.164:0/3563961597 conn(0x7f93880a3fb0 msgr2=0x7f938809a360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 shutdown_connections 2024-09-10T16:11:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.758+0000 7f938f07f640 1 -- 172.21.15.164:0/3563961597 wait complete. 2024-09-10T16:11:42.788 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:42.788 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:11:42.789 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.886+0000 7f3907f52640 10 monclient: get_monmap_and_config 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.886+0000 7f3907f52640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.886+0000 7f3907f52640 10 monclient: monmap: 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:42.887560+0000 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:42.887560+0000 2024-09-10T16:11:42.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:42.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 1 Processor -- start 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 1 -- start start 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient: init 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3900074af0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 0x7f39000a1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-c con 0x7f39000a1090 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f390013d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.887+0000 7f3907f52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f390009ebc0 con 0x7f390013d190 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f3907f52640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3900138580 con 0x7f3900074af0 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f3907f52640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f39000698a0 con 0x7f39000a1090 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f3907f52640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f3907f52640 10 monclient(hunting): authenticate will time out at 1081.375028s 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39064ef640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f390013d190 0x7f390013f580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42826/0 (socket says 172.21.15.164:42826) 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39064ef640 1 -- 172.21.15.164:0/2545865059 learned_addr learned my addr 172.21.15.164:0/2545865059 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 0x7f39000a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39054ed640 10 monclient(hunting): get_auth_request con 0x7f39000a1090 auth_method 0 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39054ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39054ed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.888+0000 7f39054ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39064ef640 1 -- 172.21.15.164:0/2545865059 >> v1:172.21.15.164:6789/0 conn(0x7f390013d190 legacy=0x7f390013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040743054 0 0) 0x7f390009ebc0 con 0x7f390013d190 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): _negotiate 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): my global_id is 54219 2024-09-10T16:11:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f38ec003520 con 0x7f390013d190 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3905cee640 1 -- 172.21.15.164:0/2545865059 >> v1:172.21.15.165:6789/0 conn(0x7f3900074af0 legacy=0x7f3900137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 msgr2=0x7f39000a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 msgr2=0x7f39000a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 0x7f39000a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 0x7f39000a1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1349153164 0 0) 0x7f38ec003520 con 0x7f390013d190 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 msgr2=0x7f39000a1500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 --2- 172.21.15.164:0/2545865059 >> v2:172.21.15.165:3300/0 conn(0x7f39000a1090 0x7f39000a1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 >> v1:172.21.15.165:6789/0 conn(0x7f3900074af0 legacy=0x7f3900137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3900142cc0 con 0x7f390013d190 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f38f8001d20 con 0x7f390013d190 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3907f52640 5 monclient: authenticate success, global_id 54219 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3907f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: dump: 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 >> v1:172.21.15.164:6789/0 conn(0x7f390013d190 legacy=0x7f390013f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38ec003b10 0x7f38ec003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): picked mon.b con 0x7f38ec003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): picked mon.c con 0x7f38ec004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec0099b0 0x7f38ec00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): picked mon.a con 0x7f38ec0099b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec00c390 con 0x7f38ec0099b0 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f38ec00c540 con 0x7f38ec003b10 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f38ec00c740 con 0x7f38ec004800 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3904cec640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3907f52640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 10 monclient(hunting): get_auth_request con 0x7f38ec004800 auth_method 0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39054ed640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f39064ef640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec0099b0 0x7f38ec00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.890+0000 7f3905cee640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38ec003b10 0x7f38ec003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): get_auth_request con 0x7f38ec0099b0 auth_method 0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): get_auth_request con 0x7f38ec003b10 auth_method 0 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient(hunting): handle_auth_done global_id 54219 payload 293 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38ec003b10 msgr2=0x7f38ec003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38ec003b10 0x7f38ec003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec0099b0 msgr2=0x7f38ec00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec0099b0 0x7f38ec00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 monclient: found mon.c 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39064ef640 1 -- 172.21.15.164:0/2545865059 reap_dead start 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38ec003520 con 0x7f38ec004800 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.891+0000 7f39054ed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.893+0000 7f39054ed640 1 -- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 msgr2=0x7f38ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.893+0000 7f39054ed640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 0x7f38ec004c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b9e0 tx=0x7f38ec005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9093a5a95875ff57 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0030070 con 0x7f38ec004800 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: dump: 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.910 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38f0034400 con 0x7f38ec004800 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 1 -- 172.21.15.164:0/2545865059 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f000b5e0 con 0x7f38ec004800 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 4 set_mon_vals no callback set 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3904cec640 10 monclient: dump: 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.911 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 10 monclient: shutdown 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 -- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 msgr2=0x7f38ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 0x7f38ec004c80 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b9e0 tx=0x7f38ec005830 comp rx=0 tx=0).stop 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 -- 172.21.15.164:0/2545865059 shutdown_connections 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 --2- 172.21.15.164:0/2545865059 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38ec004800 0x7f38ec004c80 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 -- 172.21.15.164:0/2545865059 >> 172.21.15.164:0/2545865059 conn(0x7f39000a3fb0 msgr2=0x7f39000a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 -- 172.21.15.164:0/2545865059 shutdown_connections 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.894+0000 7f3907f52640 1 -- 172.21.15.164:0/2545865059 wait complete. 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.895+0000 7f3907f52640 10 monclient: build_initial_monmap 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.895+0000 7f3907f52640 10 monclient: monmap: 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:42.913 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.895+0000 7f3907f52640 1 Processor -- start 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.895+0000 7f3907f52640 1 -- start start 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.895+0000 7f3907f52640 10 monclient: init 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 0x7f390013c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-a con 0x7f38ec004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f390009a400 0x7f390009a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-c con 0x7f390009a400 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3900138f60 0x7f39001393f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): picked mon.noname-b con 0x7f3900138f60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:42.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f39000a1890 con 0x7f38ec004800 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f39000a1710 con 0x7f3900138f60 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3900068d70 con 0x7f390009a400 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3907f52640 10 monclient(hunting): authenticate will time out at 1081.383345s 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 0x7f390013c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f390009a400 0x7f390009a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 0x7f390013c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48508/0 (socket says 172.21.15.164:48508) 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 -- 172.21.15.164:0/2321616917 learned_addr learned my addr 172.21.15.164:0/2321616917 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:42.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): get_auth_request con 0x7f38ec004800 auth_method 0 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): get_auth_request con 0x7f390009a400 auth_method 0 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39064ef640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3900138f60 0x7f39001393f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39064ef640 10 monclient(hunting): get_auth_request con 0x7f3900138f60 auth_method 0 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39064ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39064ef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f39064ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient(hunting): handle_auth_done global_id 54222 payload 824 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f390009a400 msgr2=0x7f390009a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f390009a400 0x7f390009a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3900138f60 msgr2=0x7f39001393f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3900138f60 0x7f39001393f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:42.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient: _finish_hunting 0 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 monclient: found mon.noname-a 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.896+0000 7f3905cee640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f390009ded0 con 0x7f38ec004800 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.898+0000 7f3905cee640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 msgr2=0x7f390013c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.898+0000 7f3905cee640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 0x7f390013c630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38fc006f30 tx=0x7f38fc001a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc017020 con 0x7f38ec004800 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: dump: 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.920 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc00f9c0 con 0x7f38ec004800 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 5 monclient: authenticate success, global_id 54222 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 monclient: _renew_subs 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f390013a430 con 0x7f38ec004800 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00fdb0 con 0x7f38ec004800 2024-09-10T16:11:42.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f67fc640 10 monclient: dump: 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 monclient: _renew_subs 2024-09-10T16:11:42.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f38f7fff640 4 set_mon_vals no callback set 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f390013b670 con 0x7f38ec004800 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 0x7f39001338f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.899+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3900137e70 con 0x7f3900139930 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 0x7f39001338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient: get_auth_request con 0x7f3900139930 auth_method 0 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f38fc01a020 con 0x7f38ec004800 2024-09-10T16:11:42.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 10 monclient(hunting): handle_auth_done global_id 4898 payload 824 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39054ed640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3900139d10 con 0x7f3900139930 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f38f67fc640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f38e403e210 0x7f38e40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 4739+0+0 (secure 0 0 0) 0x7f38fc052d00 con 0x7f38ec004800 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39064ef640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f38e403e210 0x7f38e40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.900+0000 7f39064ef640 10 monclient: get_auth_request con 0x7f38e403e210 auth_method 0 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f39054ed640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 msgr2=0x7f39001338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f39054ed640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 0x7f39001338f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006fd0 tx=0x7f38f0004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f003d800 con 0x7f3900139930 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 10 monclient: dump: 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.927 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f38f000b6a0 con 0x7f3900139930 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 msgr2=0x7f39001338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 0x7f39001338f0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006fd0 tx=0x7f38f0004730 comp rx=0 tx=0).stop 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 msgr2=0x7f39001338f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.902+0000 7f38f67fc640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900139930 0x7f39001338f0 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f38f0006fd0 tx=0x7f38f0004730 comp rx=0 tx=0).stop 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.903+0000 7f39064ef640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f38e403e210 0x7f38e40406d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f390009ea40 tx=0x7f38f8006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f3907f52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f3907f52640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 0x7f3900134260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f3907f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f39000772c0 con 0x7f3900133e30 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 0x7f3900134260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient: get_auth_request con 0x7f3900133e30 auth_method 0 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:42.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 10 monclient(hunting): handle_auth_done global_id 4901 payload 824 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.909+0000 7f39054ed640 1 -- 172.21.15.164:0/2321616917 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39001347a0 con 0x7f3900133e30 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f39054ed640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 msgr2=0x7f3900134260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f39054ed640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 0x7f3900134260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b9e0 tx=0x7f38f0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38f0047860 con 0x7f3900133e30 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f38f67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f38f67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.911+0000 7f38f67fc640 10 monclient: dump: 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:42.930 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f38f00513f0 con 0x7f3900133e30 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 msgr2=0x7f3900134260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 0x7f3900134260 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b9e0 tx=0x7f38f0008040 comp rx=0 tx=0).stop 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 msgr2=0x7f3900134260 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f38f67fc640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3900133e30 0x7f3900134260 secure :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f38f000b9e0 tx=0x7f38f0008040 comp rx=0 tx=0).stop 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 345083, 2024-09-10T16:11:42.932 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.933 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:42.934 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.935 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:42.936 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:42.937 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.938 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.940 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:42.941 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:42.942 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:42.943 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.944 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.945 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:42.946 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f38e403e210 msgr2=0x7f38e40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f38e403e210 0x7f38e40406d0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f390009ea40 tx=0x7f38f8006040 comp rx=0 tx=0).stop 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 10 monclient: shutdown 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 msgr2=0x7f390013c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:42.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 --2- 172.21.15.164:0/2321616917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38ec004800 0x7f390013c630 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f38fc006f30 tx=0x7f38fc001a80 comp rx=0 tx=0).stop 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f39064ef640 1 -- 172.21.15.164:0/2321616917 reap_dead start 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f39064ef640 1 -- 172.21.15.164:0/2321616917 reap_dead start 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 shutdown_connections 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 >> 172.21.15.164:0/2321616917 conn(0x7f39000a3fb0 msgr2=0x7f39000a62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 shutdown_connections 2024-09-10T16:11:42.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:42.912+0000 7f3907f52640 1 -- 172.21.15.164:0/2321616917 wait complete. 2024-09-10T16:11:42.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:42.949 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:11:42.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:11:43.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.046+0000 7f50dcdcd640 10 monclient: get_monmap_and_config 2024-09-10T16:11:43.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.046+0000 7f50dcdcd640 10 monclient: build_initial_monmap 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.046+0000 7f50dcdcd640 10 monclient: monmap: 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:43.047565+0000 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:43.047565+0000 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:43.049 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 Processor -- start 2024-09-10T16:11:43.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 -- start start 2024-09-10T16:11:43.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient: init 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50d809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-a con 0x7f50d809d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50d813a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d809fbc0 con 0x7f50d813a5c0 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d80a2a20 con 0x7f50d809c310 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f50d809c6f0 con 0x7f50d809d190 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50dcdcd640 10 monclient(hunting): authenticate will time out at 1081.534874s 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50cdd9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50cdd9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48518/0 (socket says 172.21.15.164:48518) 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 1 -- 172.21.15.164:0/1244163232 learned_addr learned my addr 172.21.15.164:0/1244163232 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.047+0000 7f50d6d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f50d813a5c0 0x7f50d813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42842/0 (socket says 172.21.15.164:42842) 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request con 0x7f50d809d190 auth_method 0 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 1 -- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 msgr2=0x7f50d809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:11:43.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 1 -- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 msgr2=0x7f50d809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.048+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 1 -- 172.21.15.164:0/1244163232 >> v1:172.21.15.164:6789/0 conn(0x7f50d813a5c0 legacy=0x7f50d813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1902835956 0 0) 0x7f50d00017d0 con 0x7f50d813a5c0 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): _negotiate 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): my global_id is 54225 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50c4003520 con 0x7f50d813a5c0 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 1 -- 172.21.15.164:0/1244163232 >> v1:172.21.15.165:6789/0 conn(0x7f50d809c310 legacy=0x7f50d809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1016887138 0 0) 0x7f50c4003520 con 0x7f50d813a5c0 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 msgr2=0x7f50d809f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 --2- 172.21.15.164:0/1244163232 >> v2:172.21.15.164:3300/0 conn(0x7f50d809d190 0x7f50d809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 >> v1:172.21.15.165:6789/0 conn(0x7f50d809c310 legacy=0x7f50d809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d809a4d0 con 0x7f50d813a5c0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _finish_auth 0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f50d0001ef0 con 0x7f50d813a5c0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: dump: 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.059 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 >> v1:172.21.15.164:6789/0 conn(0x7f50d813a5c0 legacy=0x7f50d813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:11:43.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 0x7f50c4003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): picked mon.a con 0x7f50c4003b10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50c4004800 0x7f50c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): picked mon.b con 0x7f50c4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c40099b0 0x7f50c400bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): picked mon.c con 0x7f50c40099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f50d809c6f0 con 0x7f50c4003b10 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50c400c3c0 con 0x7f50c4004800 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f50c400e7d0 con 0x7f50c40099b0 2024-09-10T16:11:43.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d5d9b640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 0x7f50c4003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c40099b0 0x7f50c400bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): get_auth_request con 0x7f50c4003b10 auth_method 0 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): get_auth_request con 0x7f50c40099b0 auth_method 0 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50c4004800 0x7f50c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request con 0x7f50c4004800 auth_method 0 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50cdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.050+0000 7f50d659c640 10 monclient(hunting): handle_auth_done global_id 54225 payload 293 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 -- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c40099b0 msgr2=0x7f50c400bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c40099b0 0x7f50c400bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 -- 172.21.15.164:0/1244163232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50c4004800 msgr2=0x7f50c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:43.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50c4004800 0x7f50c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 monclient: found mon.a 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 1 -- 172.21.15.164:0/1244163232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50c4003520 con 0x7f50c4003b10 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 10 monclient: _finish_auth 0 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 10 monclient: _check_auth_tickets 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50dcdcd640 5 monclient: authenticate success, global_id 54225 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50d6d9d640 1 -- 172.21.15.164:0/1244163232 reap_dead start 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50dcdcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.051+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50c4004800 0x7f50c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:11:43.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d659c640 1 -- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 msgr2=0x7f50c4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d659c640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 0x7f50c4003f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50c8001990 tx=0x7f50c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af6d13078659d03 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8001690 con 0x7f50c4003b10 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d5d9b640 10 monclient: dump: 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.065 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50dcdcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.053+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c802e070 con 0x7f50c4003b10 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50d5d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50d5d9b640 1 -- 172.21.15.164:0/1244163232 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8007070 con 0x7f50c4003b10 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 4 set_mon_vals no callback set 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50d5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50d5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50d5d9b640 10 monclient: dump: 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 10 monclient: shutdown 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 -- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 msgr2=0x7f50c4003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 0x7f50c4003f30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f50c8001990 tx=0x7f50c4005830 comp rx=0 tx=0).stop 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 -- 172.21.15.164:0/1244163232 shutdown_connections 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/1244163232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50c4003b10 0x7f50c4003f30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 -- 172.21.15.164:0/1244163232 >> 172.21.15.164:0/1244163232 conn(0x7f50d80a3110 msgr2=0x7f50d80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:43.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 -- 172.21.15.164:0/1244163232 shutdown_connections 2024-09-10T16:11:43.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.054+0000 7f50dcdcd640 1 -- 172.21.15.164:0/1244163232 wait complete. 2024-09-10T16:11:43.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.055+0000 7f50dcdcd640 10 monclient: build_initial_monmap 2024-09-10T16:11:43.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.055+0000 7f50dcdcd640 10 monclient: monmap: 2024-09-10T16:11:43.074 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:43.074 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.055+0000 7f50dcdcd640 1 Processor -- start 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.055+0000 7f50dcdcd640 1 -- start start 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.055+0000 7f50dcdcd640 10 monclient: init 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:11:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 0x7f50d8139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-c con 0x7f50c4003b10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8099b40 0x7f50d80903d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-b con 0x7f50d8099b40 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 0x7f50d8090dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): picked mon.noname-a con 0x7f50d8090910 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f50d80a2230 con 0x7f50d8090910 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50d80a20b0 con 0x7f50d8099b40 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f50d8074e00 con 0x7f50c4003b10 2024-09-10T16:11:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50dcdcd640 10 monclient(hunting): authenticate will time out at 1081.543354s 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 0x7f50d8090dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 0x7f50d8139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 0x7f50d8090dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48534/0 (socket says 172.21.15.164:48534) 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 1 -- 172.21.15.164:0/2721641814 learned_addr learned my addr 172.21.15.164:0/2721641814 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): get_auth_request con 0x7f50d8090910 auth_method 0 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): get_auth_request con 0x7f50c4003b10 auth_method 0 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8099b40 0x7f50d80903d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request con 0x7f50d8099b40 auth_method 0 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient(hunting): handle_auth_done global_id 4904 payload 824 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8099b40 msgr2=0x7f50d80903d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8099b40 0x7f50d80903d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 msgr2=0x7f50d8090dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 0x7f50d8090dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 monclient: found mon.noname-c 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d659c640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d8093b30 con 0x7f50c4003b10 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.056+0000 7f50d6d9d640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d8090910 0x7f50d8090dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50d659c640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 msgr2=0x7f50d8139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50d659c640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 0x7f50d8139fa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f50c802f000 tx=0x7f50c8001520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8044070 con 0x7f50c4003b10 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: dump: 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c80059e0 con 0x7f50c4003b10 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 5 monclient: authenticate success, global_id 4904 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8001690 con 0x7f50c4003b10 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 10 monclient: _renew_subs 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d8093e10 con 0x7f50c4003b10 2024-09-10T16:11:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50d559a640 4 set_mon_vals no callback set 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50cf7fe640 10 monclient: dump: 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 10 monclient: _renew_subs 2024-09-10T16:11:43.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.059+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d80940f0 con 0x7f50c4003b10 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f50c8033470 con 0x7f50c4003b10 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50dcdcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 0x7f50d8063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50d802e9a0 con 0x7f50d802e5c0 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cf7fe640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803e210 0x7f50b80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(196..196 src has 1..196) ==== 4739+0+0 (secure 0 0 0) 0x7f50c8072650 con 0x7f50c4003b10 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 0x7f50d8063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50d6d9d640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803e210 0x7f50b80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient: get_auth_request con 0x7f50d802e5c0 auth_method 0 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50d6d9d640 10 monclient: get_auth_request con 0x7f50b803e210 auth_method 0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_done global_id 64102 payload 824 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.060+0000 7f50cdd9b640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50d80a5ed0 con 0x7f50d802e5c0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cdd9b640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 msgr2=0x7f50d8063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 0x7f50d8063d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f50d8064990 tx=0x7f50c0009fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50d6d9d640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803e210 0x7f50b80406d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f50d0001460 tx=0x7f50d0002020 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c001a070 con 0x7f50d802e5c0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:43.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 10 monclient: dump: 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50c0003810 con 0x7f50d802e5c0 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:43.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 msgr2=0x7f50d8063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 0x7f50d8063d50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f50d8064990 tx=0x7f50c0009fd0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 msgr2=0x7f50d8063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.063+0000 7f50cf7fe640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d802e5c0 0x7f50d8063d50 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f50d8064990 tx=0x7f50c0009fd0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.069+0000 7f50dcdcd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.069+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 0x7f50d81d37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.069+0000 7f50dcdcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.069+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50d809d930 con 0x7f50d8064290 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 0x7f50d81d37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient: get_auth_request con 0x7f50d8064290 auth_method 0 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 10 monclient(hunting): handle_auth_done global_id 64105 payload 824 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.070+0000 7f50cdd9b640 1 -- 172.21.15.164:0/2721641814 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50d809d620 con 0x7f50d8064290 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.072+0000 7f50cdd9b640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 msgr2=0x7f50d81d37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.072+0000 7f50cdd9b640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 0x7f50d81d37a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f50c001c000 tx=0x7f50c0004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c000f040 con 0x7f50d8064290 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 10 monclient: dump: 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.088 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f50c0005e30 con 0x7f50d8064290 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 msgr2=0x7f50d81d37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 0x7f50d81d37a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f50c001c000 tx=0x7f50c0004730 comp rx=0 tx=0).stop 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 msgr2=0x7f50d81d37a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:43.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50cf7fe640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d8064290 0x7f50d81d37a0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f50c001c000 tx=0x7f50c0004730 comp rx=0 tx=0).stop 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7503, 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:43.090 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:43.091 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:43.092 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:43.093 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:43.094 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:43.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:43.096 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:43.097 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:43.099 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:43.100 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:43.101 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:43.102 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803e210 msgr2=0x7f50b80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803e210 0x7f50b80406d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f50d0001460 tx=0x7f50d0002020 comp rx=0 tx=0).stop 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 10 monclient: shutdown 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 msgr2=0x7f50d8139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 --2- 172.21.15.164:0/2721641814 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50c4003b10 0x7f50d8139fa0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f50c802f000 tx=0x7f50c8001520 comp rx=0 tx=0).stop 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50d6d9d640 1 -- 172.21.15.164:0/2721641814 reap_dead start 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50d6d9d640 1 -- 172.21.15.164:0/2721641814 reap_dead start 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 shutdown_connections 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 >> 172.21.15.164:0/2721641814 conn(0x7f50d80a3110 msgr2=0x7f50d80707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 shutdown_connections 2024-09-10T16:11:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.075+0000 7f50dcdcd640 1 -- 172.21.15.164:0/2721641814 wait complete. 2024-09-10T16:11:43.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:43.105 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:11:43.105 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:11:43.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.201+0000 7fd020b24640 10 monclient: get_monmap_and_config 2024-09-10T16:11:43.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.201+0000 7fd020b24640 10 monclient: build_initial_monmap 2024-09-10T16:11:43.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.201+0000 7fd020b24640 10 monclient: monmap: 2024-09-10T16:11:43.203 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:43.202668+0000 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:43.202668+0000 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:43.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 Processor -- start 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 -- start start 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient: init 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 0x7fd01c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd01c09d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd01c13a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c09a430 con 0x7fd01c13a5c0 2024-09-10T16:11:43.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd01c0a24e0 con 0x7fd01c09d160 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd01c0a2bf0 con 0x7fd01c09c180 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.202+0000 7fd020b24640 10 monclient(hunting): authenticate will time out at 1081.689995s 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 0x7fd01c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd01c13a5c0 0x7fd01c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42852/0 (socket says 172.21.15.164:42852) 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01ad9d640 1 -- 172.21.15.164:0/851308860 learned_addr learned my addr 172.21.15.164:0/851308860 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): get_auth_request con 0x7fd01c09c180 auth_method 0 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient(hunting): handle_auth_done global_id 4907 payload 274 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 1 -- 172.21.15.164:0/851308860 >> v1:172.21.15.165:6789/0 conn(0x7fd01c09d160 legacy=0x7fd01c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 1 -- 172.21.15.164:0/851308860 >> v1:172.21.15.164:6789/0 conn(0x7fd01c13a5c0 legacy=0x7fd01c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 1 monclient: found mon.noname-b 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:43.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.203+0000 7fd01a59c640 1 -- 172.21.15.164:0/851308860 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c09c560 con 0x7fd01c09c180 2024-09-10T16:11:43.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd01a59c640 1 -- 172.21.15.164:0/851308860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 msgr2=0x7fd01c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd01a59c640 1 --2- 172.21.15.164:0/851308860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 0x7fd01c09ea40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0097a0 tx=0x7fd00c029f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e79527144440559 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 1 -- 172.21.15.164:0/851308860 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c037070 con 0x7fd01c09c180 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: dump: 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: _finish_auth 0 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 1 -- 172.21.15.164:0/851308860 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00c032440 con 0x7fd01c09c180 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 5 monclient: authenticate success, global_id 4907 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 1 -- 172.21.15.164:0/851308860 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c032810 con 0x7fd01c09c180 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 4 set_mon_vals no callback set 2024-09-10T16:11:43.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd019d9b640 10 monclient: dump: 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 10 monclient: shutdown 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 -- 172.21.15.164:0/851308860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 msgr2=0x7fd01c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 --2- 172.21.15.164:0/851308860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 0x7fd01c09ea40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd00c0097a0 tx=0x7fd00c029f80 comp rx=0 tx=0).stop 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 -- 172.21.15.164:0/851308860 shutdown_connections 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 --2- 172.21.15.164:0/851308860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c09c180 0x7fd01c09ea40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 -- 172.21.15.164:0/851308860 >> 172.21.15.164:0/851308860 conn(0x7fd01c0a3110 msgr2=0x7fd01c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 -- 172.21.15.164:0/851308860 shutdown_connections 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.206+0000 7fd020b24640 1 -- 172.21.15.164:0/851308860 wait complete. 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.207+0000 7fd020b24640 10 monclient: build_initial_monmap 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.207+0000 7fd020b24640 10 monclient: monmap: 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:43.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.207+0000 7fd020b24640 1 Processor -- start 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.207+0000 7fd020b24640 1 -- start start 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.207+0000 7fd020b24640 10 monclient: init 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 0x7fd01c073620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-a con 0x7fd01c09c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 0x7fd01c135c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-b con 0x7fd01c09d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): picked mon.noname-c con 0x7fd01c13a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c074f90 con 0x7fd01c09c180 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd01c074e10 con 0x7fd01c09d160 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd01c09f690 con 0x7fd01c13a5c0 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd020b24640 10 monclient(hunting): authenticate will time out at 1081.695415s 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 0x7fd01c073620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35814/0 (socket says 172.21.15.164:35814) 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): get_auth_request con 0x7fd01c09c180 auth_method 0 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 1 -- 172.21.15.164:0/1914982498 learned_addr learned my addr 172.21.15.164:0/1914982498 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): get_auth_request con 0x7fd01c13a5c0 auth_method 0 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 0x7fd01c135c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 10 monclient(hunting): get_auth_request con 0x7fd01c09d160 auth_method 0 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient(hunting): handle_auth_done global_id 54231 payload 824 2024-09-10T16:11:43.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 -- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 msgr2=0x7fd01c136160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 -- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 msgr2=0x7fd01c135c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 0x7fd01c135c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 monclient: found mon.noname-a 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01a59c640 1 -- 172.21.15.164:0/1914982498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd01c12ceb0 con 0x7fd01c09c180 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd01ad9d640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.208+0000 7fd013fff640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 0x7fd01c135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd01a59c640 1 -- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 msgr2=0x7fd01c073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd01a59c640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 0x7fd01c073620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd00c031000 tx=0x7fd00c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c040070 con 0x7fd01c09c180 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: dump: 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd00c03aaa0 con 0x7fd01c09c180 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 5 monclient: authenticate success, global_id 54231 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 10 monclient: _renew_subs 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01c09d620 con 0x7fd01c09c180 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd00c037040 con 0x7fd01c09c180 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:43.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd012ffd640 10 monclient: dump: 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 10 monclient: _renew_subs 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd01959a640 4 set_mon_vals no callback set 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.211+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd01c12de10 con 0x7fd01c09c180 2024-09-10T16:11:43.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd020b24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd020b24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd020b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd01c12d0f0 con 0x7fd01c09c180 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd00c04c3e0 con 0x7fd01c09c180 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd012ffd640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 0x7fcffc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(196..196 src has 1..196) ==== 4739+0+0 (secure 0 0 0) 0x7fd00c074cd0 con 0x7fd01c09c180 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd013fff640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 0x7fcffc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.212+0000 7fd013fff640 10 monclient: get_auth_request con 0x7fcffc03e210 auth_method 0 2024-09-10T16:11:43.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.215+0000 7fd013fff640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 0x7fcffc0406d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd01c12d370 tx=0x7fd004009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:43.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.217+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd00c032e10 con 0x7fd01c09c180 2024-09-10T16:11:43.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.217+0000 7fd012ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:43.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.217+0000 7fd012ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:43.291 INFO:tasks.rados.rados.0.smithi165.stdout:782: done (4 left) 2024-09-10T16:11:43.291 INFO:tasks.rados.rados.0.smithi165.stdout:783: write (excl) oid 19 current snap is 87 2024-09-10T16:11:43.291 INFO:tasks.rados.rados.0.smithi165.stdout:777: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:11:43.363 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1134037 2024-09-10T16:11:43.435 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1015301 2024-09-10T16:11:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd020b24640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:11:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd020b24640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:11:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd020b24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd01c077260 con 0x7fd01c09c180 2024-09-10T16:11:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd012ffd640 1 -- 172.21.15.164:0/1914982498 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd00c04cbf0 con 0x7fd01c09c180 2024-09-10T16:11:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd012ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:11:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.465+0000 7fd012ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 msgr2=0x7fcffc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 0x7fcffc0406d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd01c12d370 tx=0x7fd004009040 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 10 monclient: shutdown 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 msgr2=0x7fd01c073620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 0x7fd01c073620 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd00c031000 tx=0x7fd00c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 shutdown_connections 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcffc03e210 0x7fcffc0406d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd01c13a5c0 0x7fd01c136160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd01c09d160 0x7fd01c135c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 --2- 172.21.15.164:0/1914982498 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd01c09c180 0x7fd01c073620 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 >> 172.21.15.164:0/1914982498 conn(0x7fd01c0a3110 msgr2=0x7fd01c1379e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 shutdown_connections 2024-09-10T16:11:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:43.468+0000 7fd020b24640 1 -- 172.21.15.164:0/1914982498 wait complete. 2024-09-10T16:11:43.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:11:43.498 INFO:tasks.rados.rados.0.smithi165.stdout:779: finishing rollback tid 2 to smithi16534841-8 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 572 (ObjNum 275 snap 80 seq_num 275) dirty exists 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: finishing write tid 1 to smithi16534841-7 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: finishing write tid 2 to smithi16534841-7 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: finishing write tid 3 to smithi16534841-7 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: finishing write tid 4 to smithi16534841-7 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: finishing write tid 5 to smithi16534841-7 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: oid 7 updating version 0 to 709 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: oid 7 updating version 709 to 711 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: oid 7 updating version 711 to 712 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:780: oid 7 version 712 is already newer than 710 2024-09-10T16:11:43.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 712 (ObjNum 298 snap 87 seq_num 298) dirty exists 2024-09-10T16:11:43.500 INFO:tasks.rados.rados.0.smithi165.stdout:780: left oid 7 (ObjNum 298 snap 87 seq_num 298) 2024-09-10T16:11:43.500 INFO:tasks.rados.rados.0.smithi165.stdout:781: finishing write tid 1 to smithi16534841-42 2024-09-10T16:11:43.500 INFO:tasks.rados.rados.0.smithi165.stdout:781: finishing write tid 2 to smithi16534841-42 2024-09-10T16:11:43.500 INFO:tasks.rados.rados.0.smithi165.stdout:783: seq_num 300 ranges {642100=712467,1872245=637761} 2024-09-10T16:11:43.505 INFO:tasks.rados.rados.0.smithi165.stdout:783: writing smithi16534841-19 from 642100 to 1354567 tid 1 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:783: writing smithi16534841-19 from 1872245 to 2510006 tid 2 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:777: done (4 left) 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:779: done (3 left) 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:780: done (2 left) 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:784: copy_from oid 48 from oid 22 current snap is 87 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:781: finishing write tid 3 to smithi16534841-42 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:781: finishing write tid 4 to smithi16534841-42 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:781: finishing write tid 5 to smithi16534841-42 2024-09-10T16:11:43.513 INFO:tasks.rados.rados.0.smithi165.stdout:781: oid 42 updating version 0 to 318 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:781: oid 42 version 318 is already newer than 317 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:781: oid 42 updating version 318 to 319 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:781: oid 42 updating version 319 to 320 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 320 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:781: left oid 42 (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:781: done (2 left) 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:785: write oid 35 current snap is 87 2024-09-10T16:11:43.514 INFO:tasks.rados.rados.0.smithi165.stdout:785: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-10T16:11:43.520 INFO:tasks.rados.rados.0.smithi165.stdout:785: writing smithi16534841-35 from 658907 to 1446623 tid 1 2024-09-10T16:11:43.526 INFO:tasks.rados.rados.0.smithi165.stdout:785: writing smithi16534841-35 from 1930727 to 2428499 tid 2 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:785: writing smithi16534841-35 from 3032909 to 3058907 tid 3 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:786: snap_create 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:783: finishing write tid 1 to smithi16534841-19 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:783: finishing write tid 2 to smithi16534841-19 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:783: finishing write tid 3 to smithi16534841-19 2024-09-10T16:11:43.530 INFO:tasks.rados.rados.0.smithi165.stdout:783: finishing write tid 4 to smithi16534841-19 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:783: oid 19 updating version 0 to 714 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:783: oid 19 updating version 714 to 718 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:783: oid 19 version 718 is already newer than 717 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 718 (ObjNum 300 snap 87 seq_num 300) dirty exists 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:783: left oid 19 (ObjNum 300 snap 87 seq_num 300) 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:784: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:785: finishing write tid 1 to smithi16534841-35 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:784: finishing copy_from to smithi16534841-48 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 267 (ObjNum 292 snap 86 seq_num 292) dirty exists 2024-09-10T16:11:43.531 INFO:tasks.rados.rados.0.smithi165.stdout:785: finishing write tid 2 to smithi16534841-35 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: finishing write tid 3 to smithi16534841-35 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: finishing write tid 4 to smithi16534841-35 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: finishing write tid 5 to smithi16534841-35 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: oid 35 updating version 0 to 271 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: oid 35 version 271 is already newer than 269 2024-09-10T16:11:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:785: oid 35 version 271 is already newer than 270 2024-09-10T16:11:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:785: oid 35 updating version 271 to 272 2024-09-10T16:11:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 272 (ObjNum 301 snap 87 seq_num 301) dirty exists 2024-09-10T16:11:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:785: left oid 35 (ObjNum 301 snap 87 seq_num 301) 2024-09-10T16:11:44.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=12,logm=63} crc {auth=4211354438,config=2586702516,health=83974812,kv=805391194,logm=985710372}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=12,config=1,health=12,kv=12,logm=63} crc {auth=408014208,config=1747058506,health=2319137676,kv=603532201,logm=2740198110}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=10,kv=12,logm=63} crc {auth=4211354438,config=2586702516,health=83974812,kv=805391194,logm=985710372}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.056+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=3541972412,config=2586702516,health=2319137676,kv=2587009234,logm=1680664918}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172759021}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543526180}) 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4172759021}) 2024-09-10T16:11:44.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2701750234}) 2024-09-10T16:11:44.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384101361}) 2024-09-10T16:11:44.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4069647360}) 2024-09-10T16:11:44.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2384101361}) 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.057+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116244088}) 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789164515}) 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2198358240}) 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789164515}) 2024-09-10T16:11:44.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.058+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3406297913}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110085411}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=781431985}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110085411}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.110+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820337674}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352326694}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525100350}) 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352326694}) 2024-09-10T16:11:44.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.111+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402276860}) 2024-09-10T16:11:44.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402151917}) 2024-09-10T16:11:44.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2299469357}) 2024-09-10T16:11:44.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2402151917}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610925618}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789368522}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908337584}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3789368522}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.112+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745895327}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851726394}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608899950}) 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=851726394}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698292980}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379107133}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1693668096}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379107133}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.113+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1247430195}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207079622}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1015733587}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4207079622}) 2024-09-10T16:11:44.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.114+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3125964040}) 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184517990}) 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398663013}) 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184517990}) 2024-09-10T16:11:44.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1761583417}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229572236}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1013281956}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3229572236}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.115+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1766429544}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577501518}) 2024-09-10T16:11:44.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396597966}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1577501518}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.116+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856245859}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=7} crc {logm=1509435668,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216344,mgrstat=1287480378,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=2514242874,osd_pg_creating=2860698272,osd_snap=4091740109}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=41,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=14,monmap=2,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=11} crc {logm=3715703042,mdsmap=931935893,mgr=2354521040,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1287480378,mon_config_key=3432978453,monmap=765578399,nvmeofgw=3140514009,osd_metadata=3297165161,osd_pg_creating=2860698272,osd_snap=541198179}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=7} crc {logm=1509435668,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216344,mgrstat=1287480378,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=2514242874,osd_pg_creating=2860698272,osd_snap=4091740109}) 2024-09-10T16:11:44.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.117+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=3906756560,mdsmap=931935893,mgr=2880672550,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2313092001,mon_config_key=665552248,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2860698272,osd_snap=1500056296}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=1695665320,osdmap=2722789936}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=976669103,osdmap=3269691842}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=1695665320,osdmap=2722789936}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=31,osdmap=69} crc {osd_snap=2958440766,osdmap=555128058}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=284154268}) 2024-09-10T16:11:44.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3364999991}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=284154268}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.118+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4252862490}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2792478851}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2707934169}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2792478851}) 2024-09-10T16:11:44.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.119+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=963867122}) 2024-09-10T16:11:44.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.122+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:11:44.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.122+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=25} crc {osdmap=406201497}) 2024-09-10T16:11:44.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:11:44.122+0000 7f35a3174640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=23} crc {osdmap=1495049459}) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:783: done (3 left) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:784: done (2 left) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:785: done (1 left) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:786: done (0 left) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:787: read oid 29 snap -1 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:787: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:44.292 INFO:tasks.rados.rados.0.smithi165.stdout:788: copy_from oid 17 from oid 49 current snap is 88 2024-09-10T16:11:44.293 INFO:tasks.rados.rados.0.smithi165.stdout:789: write (excl) oid 25 current snap is 88 2024-09-10T16:11:44.293 INFO:tasks.rados.rados.0.smithi165.stdout:789: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-10T16:11:44.298 INFO:tasks.rados.rados.0.smithi165.stdout:789: writing smithi16534841-25 from 675714 to 1138679 tid 1 2024-09-10T16:11:44.304 INFO:tasks.rados.rados.0.smithi165.stdout:789: writing smithi16534841-25 from 1589209 to 2230639 tid 2 2024-09-10T16:11:44.309 INFO:tasks.rados.rados.0.smithi165.stdout:789: writing smithi16534841-25 from 2943979 to 3075714 tid 3 2024-09-10T16:11:44.309 INFO:tasks.rados.rados.0.smithi165.stdout:790: delete oid 44 current snap is 88 2024-09-10T16:11:44.309 INFO:tasks.rados.rados.0.smithi165.stdout:787: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:44.340 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:788: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:788: finishing copy_from to smithi16534841-17 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 257 (ObjNum 253 snap 76 seq_num 253) dirty exists 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:789: finishing write tid 1 to smithi16534841-25 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:789: finishing write tid 2 to smithi16534841-25 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:789: finishing write tid 3 to smithi16534841-25 2024-09-10T16:11:44.410 INFO:tasks.rados.rados.0.smithi165.stdout:789: finishing write tid 4 to smithi16534841-25 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: finishing write tid 5 to smithi16534841-25 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: oid 25 updating version 0 to 528 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: oid 25 updating version 528 to 530 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: oid 25 version 530 is already newer than 529 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: oid 25 updating version 530 to 531 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 531 (ObjNum 302 snap 88 seq_num 302) dirty exists 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: left oid 25 (ObjNum 302 snap 88 seq_num 302) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:787: done (3 left) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:788: done (2 left) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:789: done (1 left) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:790: done (0 left) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:791: read oid 14 snap -1 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:791: expect (ObjNum 248 snap 73 seq_num 248) 2024-09-10T16:11:44.411 INFO:tasks.rados.rados.0.smithi165.stdout:792: read oid 15 snap 87 2024-09-10T16:11:44.412 INFO:tasks.rados.rados.0.smithi165.stdout:792: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:11:44.412 INFO:tasks.rados.rados.0.smithi165.stdout:793: copy_from oid 39 from oid 41 current snap is 88 2024-09-10T16:11:44.412 INFO:tasks.rados.rados.0.smithi165.stdout:794: delete oid 17 current snap is 88 2024-09-10T16:11:44.412 INFO:tasks.rados.rados.0.smithi165.stdout:794: done (3 left) 2024-09-10T16:11:44.413 INFO:tasks.rados.rados.0.smithi165.stdout:795: write oid 26 current snap is 88 2024-09-10T16:11:44.413 INFO:tasks.rados.rados.0.smithi165.stdout:795: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-10T16:11:44.417 INFO:tasks.rados.rados.0.smithi165.stdout:795: writing smithi16534841-26 from 726135 to 1131200 tid 1 2024-09-10T16:11:44.425 INFO:tasks.rados.rados.0.smithi165.stdout:795: writing smithi16534841-26 from 1764655 to 2553412 tid 2 2024-09-10T16:11:44.428 INFO:tasks.rados.rados.0.smithi165.stdout:795: writing smithi16534841-26 from 3026248 to 3126135 tid 3 2024-09-10T16:11:44.428 INFO:tasks.rados.rados.0.smithi165.stdout:796: rollback oid 21 current snap is 88 2024-09-10T16:11:44.429 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 82 2024-09-10T16:11:44.429 INFO:tasks.rados.rados.0.smithi165.stdout:791: expect (ObjNum 248 snap 73 seq_num 248) 2024-09-10T16:11:44.480 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:11:44.480 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:11:44.486 INFO:tasks.rados.rados.0.smithi165.stdout:792: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:11:44.536 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1786051 2024-09-10T16:11:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:793: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:11:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:795: finishing write tid 1 to smithi16534841-26 2024-09-10T16:11:44.539 INFO:tasks.rados.rados.0.smithi165.stdout:793: finishing copy_from to smithi16534841-39 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 298 (ObjNum 205 snap 61 seq_num 205) dirty exists 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:791: done (4 left) 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:792: done (3 left) 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:793: done (2 left) 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:797: write (excl) oid 36 current snap is 88 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: finishing write tid 2 to smithi16534841-26 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: finishing write tid 3 to smithi16534841-26 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: finishing write tid 4 to smithi16534841-26 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: finishing write tid 5 to smithi16534841-26 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: oid 26 updating version 0 to 722 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: oid 26 updating version 722 to 724 2024-09-10T16:11:44.540 INFO:tasks.rados.rados.0.smithi165.stdout:795: oid 26 updating version 724 to 725 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:795: oid 26 version 725 is already newer than 723 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 725 (ObjNum 305 snap 88 seq_num 305) dirty exists 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:795: left oid 26 (ObjNum 305 snap 88 seq_num 305) 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:796: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:796: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 534 (ObjNum 177 snap 0 seq_num 1162502240) dirty dne 2024-09-10T16:11:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:797: seq_num 306 ranges {742942=480314,1823137=648768} 2024-09-10T16:11:44.546 INFO:tasks.rados.rados.0.smithi165.stdout:797: writing smithi16534841-36 from 742942 to 1223256 tid 1 2024-09-10T16:11:44.552 INFO:tasks.rados.rados.0.smithi165.stdout:797: writing smithi16534841-36 from 1823137 to 2471905 tid 2 2024-09-10T16:11:44.553 INFO:tasks.rados.rados.0.smithi165.stdout:795: done (2 left) 2024-09-10T16:11:44.553 INFO:tasks.rados.rados.0.smithi165.stdout:796: done (1 left) 2024-09-10T16:11:44.553 INFO:tasks.rados.rados.0.smithi165.stdout:798: snap_remove snap 86 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.579+0000 7f4b7faa2640 10 monclient: get_monmap_and_config 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.579+0000 7f4b7faa2640 10 monclient: build_initial_monmap 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.579+0000 7f4b7faa2640 10 monclient: monmap: 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:44.580553+0000 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:44.580553+0000 2024-09-10T16:11:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:44.582 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 1 Processor -- start 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 1 -- start start 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient: init 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4b7806cb70 0x7f4b78134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b7806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b780a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b780a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7809ce50 con 0x7f4b780a4ab0 2024-09-10T16:11:44.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.580+0000 7f4b7faa2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78138ac0 con 0x7f4b780a46a0 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7faa2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7faa2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4b7806db60 con 0x7f4b7806cb70 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7faa2640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7faa2640 10 monclient(hunting): authenticate will time out at 1083.068014s 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7e03f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4b780a4ab0 0x7f4b781383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51954/0 (socket says 172.21.15.164:51954) 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7e03f640 1 -- 172.21.15.164:0/3140659000 learned_addr learned my addr 172.21.15.164:0/3140659000 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 1 --2- 172.21.15.164:0/3140659000 >> v2:172.21.15.165:3300/0 conn(0x7f4b7806cb70 0x7f4b78134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): get_auth_request con 0x7f4b7806cb70 auth_method 0 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:44.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:44.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:44.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:44.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.581+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7e03f640 1 -- 172.21.15.164:0/3140659000 >> v1:172.21.15.164:6789/0 conn(0x7f4b780a4ab0 legacy=0x7f4b781383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 1 -- 172.21.15.164:0/3140659000 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1606227349 0 0) 0x7f4b700017d0 con 0x7f4b780a4ab0 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 10 monclient(hunting): _negotiate 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d03d640 1 -- 172.21.15.164:0/3140659000 >> v1:172.21.15.165:6789/0 conn(0x7f4b780a46a0 legacy=0x7f4b78137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:44.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 10 monclient(hunting): my global_id is 54234 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7c83c640 1 -- 172.21.15.164:0/3140659000 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b64003520 con 0x7f4b780a4ab0 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_done global_id 64111 payload 274 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 1 -- 172.21.15.164:0/3140659000 >> v1:172.21.15.165:6789/0 conn(0x7f4b780a46a0 legacy=0x7f4b78137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 1 -- 172.21.15.164:0/3140659000 >> v1:172.21.15.164:6789/0 conn(0x7f4b780a4ab0 legacy=0x7f4b781383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 10 monclient: _finish_hunting 0 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 1 monclient: found mon.noname-c 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.583+0000 7f4b7d83e640 1 -- 172.21.15.164:0/3140659000 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7809cb70 con 0x7f4b7806cb70 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7d83e640 1 -- 172.21.15.164:0/3140659000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 msgr2=0x7f4b78134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7d83e640 1 --2- 172.21.15.164:0/3140659000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b78134f70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4b740098d0 tx=0x7f4b7402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fe70b38d388c209 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 1 -- 172.21.15.164:0/3140659000 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74037070 con 0x7f4b7806cb70 2024-09-10T16:11:44.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: dump: 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: _finish_auth 0 2024-09-10T16:11:44.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: _check_auth_tickets 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 1 -- 172.21.15.164:0/3140659000 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b74032640 con 0x7f4b7806cb70 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7faa2640 5 monclient: authenticate success, global_id 64111 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7faa2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7faa2640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 1 -- 172.21.15.164:0/3140659000 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b74032a10 con 0x7f4b7806cb70 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7faa2640 4 set_mon_vals no callback set 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.586+0000 7f4b7c83c640 10 monclient: dump: 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:44.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 10 monclient: shutdown 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 -- 172.21.15.164:0/3140659000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 msgr2=0x7f4b78134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/3140659000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b78134f70 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4b740098d0 tx=0x7f4b7402a230 comp rx=0 tx=0).stop 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 -- 172.21.15.164:0/3140659000 shutdown_connections 2024-09-10T16:11:44.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/3140659000 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b78134f70 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 -- 172.21.15.164:0/3140659000 >> 172.21.15.164:0/3140659000 conn(0x7f4b780aa900 msgr2=0x7f4b780aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 -- 172.21.15.164:0/3140659000 shutdown_connections 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 1 -- 172.21.15.164:0/3140659000 wait complete. 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 10 monclient: build_initial_monmap 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.587+0000 7f4b7faa2640 10 monclient: monmap: 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:44.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:44.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 Processor -- start 2024-09-10T16:11:44.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 -- start start 2024-09-10T16:11:44.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient: init 2024-09-10T16:11:44.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:44.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:44.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b7812d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:44.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-c con 0x7f4b780a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b780a4ab0 0x7f4b7809e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): picked mon.noname-a con 0x7f4b780a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4b78141050 con 0x7f4b780a4ab0 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7806cf50 con 0x7f4b7806cb70 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4b781411d0 con 0x7f4b780a46a0 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7faa2640 10 monclient(hunting): authenticate will time out at 1083.075675s 2024-09-10T16:11:44.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b780a4ab0 0x7f4b7809e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55038/0 (socket says 172.21.15.164:55038) 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 1 -- 172.21.15.164:0/2814263594 learned_addr learned my addr 172.21.15.164:0/2814263594 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): get_auth_request con 0x7f4b780a46a0 auth_method 0 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): get_auth_request con 0x7f4b780a4ab0 auth_method 0 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d83e640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b7812d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:44.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d03d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7e03f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d83e640 10 monclient(hunting): get_auth_request con 0x7f4b7806cb70 auth_method 0 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d83e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d83e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.588+0000 7f4b7d83e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d83e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 10 monclient(hunting): handle_auth_done global_id 4913 payload 824 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 -- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 msgr2=0x7f4b7812d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:44.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b7812d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 -- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b780a4ab0 msgr2=0x7f4b7809e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b780a4ab0 0x7f4b7809e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 10 monclient: _finish_hunting 0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 monclient: found mon.noname-c 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.589+0000 7f4b7d03d640 1 -- 172.21.15.164:0/2814263594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b780a2c60 con 0x7f4b780a46a0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7d03d640 1 -- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 msgr2=0x7f4b7812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7d03d640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800c480 tx=0x7f4b6800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68017070 con 0x7f4b780a46a0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:44.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: dump: 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:44.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 5 monclient: authenticate success, global_id 4913 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 10 monclient: _renew_subs 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7d03d640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800c480 tx=0x7f4b6800c940 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f4b680057c0 config(0 keys) 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b780a1fc0 con 0x7f4b780a46a0 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 10 monclient: _renew_subs 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b78133f60 con 0x7f4b780a46a0 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b680057c0 con 0x7f4b780a46a0 2024-09-10T16:11:44.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.591+0000 7f4b6ffff640 4 set_mon_vals no callback set 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b7faa2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b7faa2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b7faa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b78134f70 con 0x7f4b780a46a0 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68005bb0 con 0x7f4b780a46a0 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 10 monclient: dump: 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.607 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4b6801a070 con 0x7f4b780a46a0 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b6e7fc640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 0x7f4b5c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.592+0000 7f4b7d83e640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 0x7f4b5c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.593+0000 7f4b7d83e640 10 monclient: get_auth_request con 0x7f4b5c03e1c0 auth_method 0 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.595+0000 7f4b7d83e640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 0x7f4b5c040680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b78077610 tx=0x7f4b7402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:44.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:44.597+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(198..198 src has 1..198) ==== 4755+0+0 (secure 0 0 0) 0x7f4b68052170 con 0x7f4b780a46a0 2024-09-10T16:11:44.837 INFO:tasks.rados.rados.0.smithi165.stdout:798: done (1 left) 2024-09-10T16:11:44.837 INFO:tasks.rados.rados.0.smithi165.stdout:799: write oid 50 current snap is 88 2024-09-10T16:11:44.837 INFO:tasks.rados.rados.0.smithi165.stdout:797: finishing write tid 1 to smithi16534841-36 2024-09-10T16:11:44.837 INFO:tasks.rados.rados.0.smithi165.stdout:799: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2024-09-10T16:11:44.844 INFO:tasks.rados.rados.0.smithi165.stdout:799: writing smithi16534841-50 from 759749 to 1315312 tid 1 2024-09-10T16:11:44.851 INFO:tasks.rados.rados.0.smithi165.stdout:799: writing smithi16534841-50 from 1765266 to 2557692 tid 2 2024-09-10T16:11:44.855 INFO:tasks.rados.rados.0.smithi165.stdout:799: writing smithi16534841-50 from 2964741 to 3159749 tid 3 2024-09-10T16:11:44.855 INFO:tasks.rados.rados.0.smithi165.stdout:800: write oid 4 current snap is 88 2024-09-10T16:11:44.855 INFO:tasks.rados.rados.0.smithi165.stdout:797: finishing write tid 2 to smithi16534841-36 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: finishing write tid 3 to smithi16534841-36 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: finishing write tid 4 to smithi16534841-36 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: oid 36 updating version 0 to 231 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: oid 36 version 231 is already newer than 229 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: oid 36 version 231 is already newer than 230 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 231 (ObjNum 306 snap 88 seq_num 306) dirty exists 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:797: left oid 36 (ObjNum 306 snap 88 seq_num 306) 2024-09-10T16:11:44.856 INFO:tasks.rados.rados.0.smithi165.stdout:800: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2024-09-10T16:11:44.863 INFO:tasks.rados.rados.0.smithi165.stdout:800: writing smithi16534841-4 from 776556 to 1407368 tid 1 2024-09-10T16:11:44.869 INFO:tasks.rados.rados.0.smithi165.stdout:800: writing smithi16534841-4 from 1823748 to 2476185 tid 2 2024-09-10T16:11:44.873 INFO:tasks.rados.rados.0.smithi165.stdout:800: writing smithi16534841-4 from 3108517 to 3176556 tid 3 2024-09-10T16:11:44.873 INFO:tasks.rados.rados.0.smithi165.stdout:797: done (2 left) 2024-09-10T16:11:44.873 INFO:tasks.rados.rados.0.smithi165.stdout:801: write oid 3 current snap is 88 2024-09-10T16:11:44.873 INFO:tasks.rados.rados.0.smithi165.stdout:801: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-10T16:11:44.881 INFO:tasks.rados.rados.0.smithi165.stdout:801: writing smithi16534841-3 from 793363 to 1499424 tid 1 2024-09-10T16:11:44.891 INFO:tasks.rados.rados.0.smithi165.stdout:801: writing smithi16534841-3 from 2282230 to 3078325 tid 2 2024-09-10T16:11:44.891 INFO:tasks.rados.rados.0.smithi165.stdout:802: read oid 29 snap -1 2024-09-10T16:11:44.891 INFO:tasks.rados.rados.0.smithi165.stdout:802: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:44.891 INFO:tasks.rados.rados.0.smithi165.stdout:803: snap_create 2024-09-10T16:11:44.891 INFO:tasks.rados.rados.0.smithi165.stdout:799: finishing write tid 1 to smithi16534841-50 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: finishing write tid 2 to smithi16534841-50 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: finishing write tid 3 to smithi16534841-50 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: finishing write tid 4 to smithi16534841-50 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: finishing write tid 5 to smithi16534841-50 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: oid 50 updating version 0 to 323 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: oid 50 updating version 323 to 325 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: oid 50 version 325 is already newer than 324 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: oid 50 version 325 is already newer than 322 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 325 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:799: left oid 50 (ObjNum 307 snap 88 seq_num 307) 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:800: finishing write tid 1 to smithi16534841-4 2024-09-10T16:11:44.892 INFO:tasks.rados.rados.0.smithi165.stdout:800: finishing write tid 2 to smithi16534841-4 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: finishing write tid 3 to smithi16534841-4 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: finishing write tid 4 to smithi16534841-4 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: finishing write tid 5 to smithi16534841-4 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: oid 4 updating version 0 to 275 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: oid 4 version 275 is already newer than 274 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: oid 4 updating version 275 to 277 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: oid 4 version 277 is already newer than 276 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 277 (ObjNum 308 snap 88 seq_num 308) dirty exists 2024-09-10T16:11:44.893 INFO:tasks.rados.rados.0.smithi165.stdout:800: left oid 4 (ObjNum 308 snap 88 seq_num 308) 2024-09-10T16:11:44.894 INFO:tasks.rados.rados.0.smithi165.stdout:802: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:11:44.903 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1760311 2024-09-10T16:11:44.918 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:44.952 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: finishing write tid 1 to smithi16534841-3 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: finishing write tid 2 to smithi16534841-3 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: finishing write tid 3 to smithi16534841-3 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: finishing write tid 4 to smithi16534841-3 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: oid 3 updating version 0 to 541 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: oid 3 version 541 is already newer than 535 2024-09-10T16:11:44.986 INFO:tasks.rados.rados.0.smithi165.stdout:801: oid 3 version 541 is already newer than 540 2024-09-10T16:11:44.987 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 541 (ObjNum 309 snap 88 seq_num 309) dirty exists 2024-09-10T16:11:44.987 INFO:tasks.rados.rados.0.smithi165.stdout:801: left oid 3 (ObjNum 309 snap 88 seq_num 309) 2024-09-10T16:11:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.419+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4b780a2c60 con 0x7f4b780a46a0 2024-09-10T16:11:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.419+0000 7f4b6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.419+0000 7f4b6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:45.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.660+0000 7f4b7faa2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:11:45.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.660+0000 7f4b7faa2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.660+0000 7f4b7faa2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.660+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b7812dd00 con 0x7f4b780a46a0 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.661+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/2814263594 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f4b68050b80 con 0x7f4b780a46a0 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.661+0000 7f4b6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.661+0000 7f4b6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:45.663 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":58,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:11:45.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 msgr2=0x7f4b5c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 0x7f4b5c040680 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4b78077610 tx=0x7f4b7402e000 comp rx=0 tx=0).stop 2024-09-10T16:11:45.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 10 monclient: shutdown 2024-09-10T16:11:45.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:45.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 msgr2=0x7f4b7812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f4b6800c480 tx=0x7f4b6800c940 comp rx=0 tx=0).stop 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 shutdown_connections 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5c03e1c0 0x7f4b5c040680 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b780a4ab0 0x7f4b7809e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b780a46a0 0x7f4b7812d7c0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 --2- 172.21.15.164:0/2814263594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7806cb70 0x7f4b7812d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 >> 172.21.15.164:0/2814263594 conn(0x7f4b780aa900 msgr2=0x7f4b780a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 shutdown_connections 2024-09-10T16:11:45.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.663+0000 7f4b7faa2640 1 -- 172.21.15.164:0/2814263594 wait complete. 2024-09-10T16:11:45.675 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:11:45.675 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:11:45.675 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:11:45.676 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.772+0000 7ff5ac072640 10 monclient: get_monmap_and_config 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.772+0000 7ff5ac072640 10 monclient: build_initial_monmap 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.772+0000 7ff5ac072640 10 monclient: monmap: 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:45.773688+0000 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:45.773688+0000 2024-09-10T16:11:45.774 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:45.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.773+0000 7ff5ac072640 1 Processor -- start 2024-09-10T16:11:45.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.773+0000 7ff5ac072640 1 -- start start 2024-09-10T16:11:45.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.773+0000 7ff5ac072640 10 monclient: init 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5a4137bb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff5a41386a0 0x7ff5a40a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a41386a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5a413d140 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a4073820 con 0x7ff5a4137bb0 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5a40a1c90 con 0x7ff5a413d140 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff5a40a1f70 con 0x7ff5a41386a0 2024-09-10T16:11:45.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5ac072640 10 monclient(hunting): authenticate will time out at 1084.261165s 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a9e0e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff5a4137bb0 0x7ff5a4137f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51970/0 (socket says 172.21.15.164:51970) 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/1609738054 learned_addr learned my addr 172.21.15.164:0/1609738054 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 1 --2- 172.21.15.164:0/1609738054 >> v2:172.21.15.165:3300/0 conn(0x7ff5a41386a0 0x7ff5a40a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): get_auth_request con 0x7ff5a41386a0 auth_method 0 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.774+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/1609738054 >> v1:172.21.15.164:6789/0 conn(0x7ff5a4137bb0 legacy=0x7ff5a4137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527691772 0 0) 0x7ff5a4073820 con 0x7ff5a4137bb0 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 10 monclient(hunting): _negotiate 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 10 monclient(hunting): my global_id is 54240 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff590003520 con 0x7ff5a4137bb0 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5aa60f640 1 -- 172.21.15.164:0/1609738054 >> v1:172.21.15.165:6789/0 conn(0x7ff5a413d140 legacy=0x7ff5a413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 10 monclient(hunting): handle_auth_done global_id 64120 payload 274 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 1 -- 172.21.15.164:0/1609738054 >> v1:172.21.15.165:6789/0 conn(0x7ff5a413d140 legacy=0x7ff5a413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 1 -- 172.21.15.164:0/1609738054 >> v1:172.21.15.164:6789/0 conn(0x7ff5a4137bb0 legacy=0x7ff5a4137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 10 monclient: _finish_hunting 0 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 1 monclient: found mon.noname-c 2024-09-10T16:11:45.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:45.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:11:45.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a960d640 1 -- 172.21.15.164:0/1609738054 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a40a2250 con 0x7ff5a41386a0 2024-09-10T16:11:45.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2799200648 0 0) 0x7ff590003520 con 0x7ff5a4137bb0 2024-09-10T16:11:45.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.776+0000 7ff5a8e0c640 10 monclient: discarding stray monitor message auth_reply(proto 2 0 (0) Success) 2024-09-10T16:11:45.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.779+0000 7ff5a960d640 1 -- 172.21.15.164:0/1609738054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 msgr2=0x7ff5a40a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.779+0000 7ff5a960d640 1 --2- 172.21.15.164:0/1609738054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a40a2e80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098d0 tx=0x7ff59402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2d0aafff7d63934 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594037070 con 0x7ff5a41386a0 2024-09-10T16:11:45.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: dump: 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.792 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: _finish_auth 0 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: _check_auth_tickets 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 5 monclient: authenticate success, global_id 64120 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff594032640 con 0x7ff5a41386a0 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:45.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 4 set_mon_vals no callback set 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 1 -- 172.21.15.164:0/1609738054 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594032a10 con 0x7ff5a41386a0 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5a8e0c640 10 monclient: dump: 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.794 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 10 monclient: shutdown 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 -- 172.21.15.164:0/1609738054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 msgr2=0x7ff5a40a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 --2- 172.21.15.164:0/1609738054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a40a2e80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff5940098d0 tx=0x7ff59402a230 comp rx=0 tx=0).stop 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 -- 172.21.15.164:0/1609738054 shutdown_connections 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 --2- 172.21.15.164:0/1609738054 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a40a2e80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 -- 172.21.15.164:0/1609738054 >> 172.21.15.164:0/1609738054 conn(0x7ff5a40a7320 msgr2=0x7ff5a40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 -- 172.21.15.164:0/1609738054 shutdown_connections 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.780+0000 7ff5ac072640 1 -- 172.21.15.164:0/1609738054 wait complete. 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.781+0000 7ff5ac072640 10 monclient: build_initial_monmap 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.781+0000 7ff5ac072640 10 monclient: monmap: 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:45.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.781+0000 7ff5ac072640 1 Processor -- start 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.781+0000 7ff5ac072640 1 -- start start 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.781+0000 7ff5ac072640 10 monclient: init 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 0x7ff5a41344d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-c con 0x7ff5a4137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a4139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-b con 0x7ff5a41386a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 0x7ff5a413a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): picked mon.noname-a con 0x7ff5a413d140 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a4077060 con 0x7ff5a413d140 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a4076ee0 con 0x7ff5a41386a0 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5a4140c50 con 0x7ff5a4137bb0 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5ac072640 10 monclient(hunting): authenticate will time out at 1084.269210s 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 0x7ff5a413a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 0x7ff5a41344d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 0x7ff5a413a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33360/0 (socket says 172.21.15.164:33360) 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): get_auth_request con 0x7ff5a4137bb0 auth_method 0 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 1 -- 172.21.15.164:0/2841088860 learned_addr learned my addr 172.21.15.164:0/2841088860 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): get_auth_request con 0x7ff5a413d140 auth_method 0 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a4139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): get_auth_request con 0x7ff5a41386a0 auth_method 0 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient(hunting): handle_auth_done global_id 4916 payload 824 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 msgr2=0x7ff5a4139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5a41386a0 0x7ff5a4139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 msgr2=0x7ff5a413a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 0x7ff5a413a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient: _finish_hunting 0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 monclient: found mon.noname-c 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5a411dc40 con 0x7ff5a4137bb0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.782+0000 7ff5aa60f640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413d140 0x7ff5a413a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.784+0000 7ff5a9e0e640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 msgr2=0x7ff5a41344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.784+0000 7ff5a9e0e640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 0x7ff5a41344d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40739f0 tx=0x7ff5a000c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.784+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a00047e0 con 0x7ff5a4137bb0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.784+0000 7ff59a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.784+0000 7ff59a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: dump: 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:11:45.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5a00049c0 con 0x7ff5a4137bb0 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 5 monclient: authenticate success, global_id 4916 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 monclient: _renew_subs 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5a411df20 con 0x7ff5a4137bb0 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0004db0 con 0x7ff5a4137bb0 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59a7fc640 10 monclient: dump: 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.803 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff59bfff640 4 set_mon_vals no callback set 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 monclient: _renew_subs 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5a40a5a80 con 0x7ff5a4137bb0 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:45.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 0x7ff5a413a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.785+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a40a2f00 con 0x7ff5a4134a10 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff5a001d3f0 con 0x7ff5a4137bb0 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 0x7ff5a413a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient: get_auth_request con 0x7ff5a4134a10 auth_method 0 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff59a7fc640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff58803e210 0x7ff5880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(199..199 src has 1..199) ==== 4811+0+0 (secure 0 0 0) 0x7ff5a0052750 con 0x7ff5a4137bb0 2024-09-10T16:11:45.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 10 monclient(hunting): handle_auth_done global_id 54246 payload 824 2024-09-10T16:11:45.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5a960d640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5a413c670 con 0x7ff5a4134a10 2024-09-10T16:11:45.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5aa60f640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff58803e210 0x7ff5880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.786+0000 7ff5aa60f640 10 monclient: get_auth_request con 0x7ff58803e210 auth_method 0 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff5a960d640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 msgr2=0x7ff5a413a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff5a960d640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 0x7ff5a413a720 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff594034000 tx=0x7ff5940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594043070 con 0x7ff5a4134a10 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 10 monclient: dump: 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.809 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff594004220 con 0x7ff5a4134a10 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 msgr2=0x7ff5a413a720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 0x7ff5a413a720 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff594034000 tx=0x7ff5940046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 msgr2=0x7ff5a413a720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.788+0000 7ff59a7fc640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a4134a10 0x7ff5a413a720 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff594034000 tx=0x7ff5940046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.789+0000 7ff5aa60f640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff58803e210 0x7ff5880406d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5a413bcb0 tx=0x7ff59c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.794+0000 7ff5ac072640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.794+0000 7ff5ac072640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 0x7ff5a413d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.794+0000 7ff5ac072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.794+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff5a413b040 con 0x7ff5a413ac60 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 0x7ff5a413d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient: get_auth_request con 0x7ff5a413ac60 auth_method 0 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 10 monclient(hunting): handle_auth_done global_id 54249 payload 824 2024-09-10T16:11:45.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.795+0000 7ff5a960d640 1 -- 172.21.15.164:0/2841088860 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5a406d1c0 con 0x7ff5a413ac60 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff5a960d640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 msgr2=0x7ff5a413d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff5a960d640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 0x7ff5a413d520 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff59402a270 tx=0x7ff594009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff594037040 con 0x7ff5a413ac60 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff59a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff59a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.797+0000 7ff59a7fc640 10 monclient: dump: 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.812 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7ff594004490 con 0x7ff5a413ac60 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 msgr2=0x7ff5a413d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 0x7ff5a413d520 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff59402a270 tx=0x7ff594009190 comp rx=0 tx=0).stop 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 msgr2=0x7ff5a413d520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:45.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff59a7fc640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5a413ac60 0x7ff5a413d520 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ff59402a270 tx=0x7ff594009190 comp rx=0 tx=0).stop 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 75981, 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.816 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:45.817 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.818 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.819 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:45.821 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.822 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:45.823 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:45.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.826 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:45.827 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff58803e210 msgr2=0x7ff5880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff58803e210 0x7ff5880406d0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff5a413bcb0 tx=0x7ff59c009040 comp rx=0 tx=0).stop 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 10 monclient: shutdown 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 msgr2=0x7ff5a41344d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 --2- 172.21.15.164:0/2841088860 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5a4137bb0 0x7ff5a41344d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ff5a40739f0 tx=0x7ff5a000c890 comp rx=0 tx=0).stop 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5aa60f640 1 -- 172.21.15.164:0/2841088860 reap_dead start 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5aa60f640 1 -- 172.21.15.164:0/2841088860 reap_dead start 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 shutdown_connections 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 >> 172.21.15.164:0/2841088860 conn(0x7ff5a40a7320 msgr2=0x7ff5a4136700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:45.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 shutdown_connections 2024-09-10T16:11:45.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.798+0000 7ff5ac072640 1 -- 172.21.15.164:0/2841088860 wait complete. 2024-09-10T16:11:45.830 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:45.830 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:11:45.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:11:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.927+0000 7f35f8f3a640 10 monclient: get_monmap_and_config 2024-09-10T16:11:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.927+0000 7f35f8f3a640 10 monclient: build_initial_monmap 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.927+0000 7f35f8f3a640 10 monclient: monmap: 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:45.928215+0000 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:45.928215+0000 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:45.930 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 Processor -- start 2024-09-10T16:11:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 -- start start 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient: init 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35f409c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f35f409d190 0x7f35f409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f35f409d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f35f413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f409fbc0 con 0x7f35f409c310 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35f40a2a20 con 0x7f35f413a5c0 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f35f409c6f0 con 0x7f35f409d190 2024-09-10T16:11:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f8f3a640 10 monclient(hunting): authenticate will time out at 1084.415626s 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f35f409c310 0x7f35f409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51976/0 (socket says 172.21.15.164:51976) 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35f259c640 1 -- 172.21.15.164:0/2417096973 learned_addr learned my addr 172.21.15.164:0/2417096973 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 1 --2- 172.21.15.164:0/2417096973 >> v2:172.21.15.165:3300/0 conn(0x7f35f409d190 0x7f35f409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): get_auth_request con 0x7f35f409d190 auth_method 0 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.928+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.929+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.930+0000 7f35f259c640 1 -- 172.21.15.164:0/2417096973 >> v1:172.21.15.164:6789/0 conn(0x7f35f409c310 legacy=0x7f35f409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3741307602 0 0) 0x7f35e4001450 con 0x7f35f409c310 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): _negotiate 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): my global_id is 54252 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f35e0003520 con 0x7f35f409c310 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 1 -- 172.21.15.164:0/2417096973 >> v1:172.21.15.165:6789/0 conn(0x7f35f413a5c0 legacy=0x7f35f413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 477606856 0 0) 0x7f35e0003520 con 0x7f35f409c310 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 >> v2:172.21.15.165:3300/0 conn(0x7f35f409d190 msgr2=0x7f35f409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 --2- 172.21.15.164:0/2417096973 >> v2:172.21.15.165:3300/0 conn(0x7f35f409d190 0x7f35f409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 >> v1:172.21.15.165:6789/0 conn(0x7f35f413a5c0 legacy=0x7f35f413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:11:45.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f409a4d0 con 0x7f35f409c310 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _finish_auth 0 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f8f3a640 5 monclient: authenticate success, global_id 54252 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f35e4001b70 con 0x7f35f409c310 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: dump: 2024-09-10T16:11:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 >> v1:172.21.15.164:6789/0 conn(0x7f35f409c310 legacy=0x7f35f409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35e0003b10 0x7f35e0003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): picked mon.b con 0x7f35e0003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35e0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): picked mon.c con 0x7f35e0004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35e00053c0 0x7f35e000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): picked mon.a con 0x7f35e00053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f35f409c6f0 con 0x7f35e00053c0 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f35e000c300 con 0x7f35e0003b10 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35e000c500 con 0x7f35e0004800 2024-09-10T16:11:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f1d9b640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f8f3a640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35e0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35e00053c0 0x7f35e000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): get_auth_request con 0x7f35e0004800 auth_method 0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35e0003b10 0x7f35e0003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): get_auth_request con 0x7f35e00053c0 auth_method 0 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): get_auth_request con 0x7f35e0003b10 auth_method 0 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient(hunting): handle_auth_done global_id 54252 payload 293 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 -- 172.21.15.164:0/2417096973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35e0003b10 msgr2=0x7f35e0003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35e0003b10 0x7f35e0003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 -- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35e00053c0 msgr2=0x7f35e000ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35e00053c0 0x7f35e000ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 monclient: found mon.c 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:45.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 1 -- 172.21.15.164:0/2417096973 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35e0003520 con 0x7f35e0004800 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient: _finish_auth 0 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 10 monclient: _check_auth_tickets 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35e00053c0 0x7f35e000ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.931+0000 7f35f2d9d640 1 -- 172.21.15.164:0/2417096973 reap_dead start 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35ebfff640 1 -- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 msgr2=0x7f35e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35ebfff640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35e0004c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009a90 tx=0x7f35e0005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df0edf8e96b641a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc036070 con 0x7f35e0004800 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: dump: 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.946 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f8f3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc02f510 con 0x7f35e0004800 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f8f3a640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:45.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f8f3a640 4 set_mon_vals no callback set 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 1 -- 172.21.15.164:0/2417096973 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc009690 con 0x7f35e0004800 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.934+0000 7f35f1d9b640 10 monclient: dump: 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.948 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 10 monclient: shutdown 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 -- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 msgr2=0x7f35e0004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35e0004c80 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009a90 tx=0x7f35e0005830 comp rx=0 tx=0).stop 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 -- 172.21.15.164:0/2417096973 shutdown_connections 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/2417096973 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35e0004c80 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 -- 172.21.15.164:0/2417096973 >> 172.21.15.164:0/2417096973 conn(0x7f35f40a3110 msgr2=0x7f35f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 -- 172.21.15.164:0/2417096973 shutdown_connections 2024-09-10T16:11:45.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.935+0000 7f35f8f3a640 1 -- 172.21.15.164:0/2417096973 wait complete. 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient: build_initial_monmap 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient: monmap: 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 Processor -- start 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 -- start start 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient: init 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35f41cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-c con 0x7f35e0004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 0x7f35f41ce290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-a con 0x7f35f40966f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4094340 0x7f35f40947f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): picked mon.noname-b con 0x7f35f4094340 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f35f40a2230 con 0x7f35f40966f0 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f35f40a20b0 con 0x7f35f4094340 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35f4074e00 con 0x7f35e0004800 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f8f3a640 10 monclient(hunting): authenticate will time out at 1084.423887s 2024-09-10T16:11:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35f41cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35f41cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55062/0 (socket says 172.21.15.164:55062) 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35ebfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 0x7f35f41ce290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.936+0000 7f35f259c640 1 -- 172.21.15.164:0/1655854372 learned_addr learned my addr 172.21.15.164:0/1655854372 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): get_auth_request con 0x7f35e0004800 auth_method 0 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): get_auth_request con 0x7f35f40966f0 auth_method 0 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f2d9d640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4094340 0x7f35f40947f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f2d9d640 10 monclient(hunting): get_auth_request con 0x7f35f4094340 auth_method 0 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient(hunting): handle_auth_done global_id 54255 payload 824 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 msgr2=0x7f35f41cdd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35f41cdd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4094340 msgr2=0x7f35f40947f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4094340 0x7f35f40947f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient: _finish_hunting 0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 monclient: found mon.noname-a 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35ebfff640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f409d620 con 0x7f35f40966f0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.937+0000 7f35f259c640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35e0004800 0x7f35f41cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35ebfff640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 msgr2=0x7f35f41ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35ebfff640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 0x7f35f41ce290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35e000c6d0 tx=0x7f35dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc040030 con 0x7f35f40966f0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: dump: 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: _finish_auth 0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35dc03b460 con 0x7f35f40966f0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 5 monclient: authenticate success, global_id 54255 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 10 monclient: _renew_subs 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f4097650 con 0x7f35f40966f0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35dc036040 con 0x7f35f40966f0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35eaffd640 10 monclient: dump: 2024-09-10T16:11:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f159a640 4 set_mon_vals no callback set 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 10 monclient: _renew_subs 2024-09-10T16:11:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.939+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f4095cd0 con 0x7f35f40966f0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f8f3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 0x7f35f402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35f409c930 con 0x7f35f40954f0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f35dc043070 con 0x7f35f40966f0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 0x7f35f402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient: get_auth_request con 0x7f35f40954f0 auth_method 0 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35eaffd640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35d403e210 0x7f35d40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 10 monclient(hunting): handle_auth_done global_id 4922 payload 824 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35f259c640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f35f40a2ca0 con 0x7f35f40954f0 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.940+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(199..199 src has 1..199) ==== 4811+0+0 (secure 0 0 0) 0x7f35dc0727b0 con 0x7f35f40966f0 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.941+0000 7f35f2d9d640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35d403e210 0x7f35d40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.941+0000 7f35f2d9d640 10 monclient: get_auth_request con 0x7f35d403e210 auth_method 0 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35f259c640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 msgr2=0x7f35f402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35f259c640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 0x7f35f402d440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4012620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e401b070 con 0x7f35f40954f0 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 10 monclient: dump: 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.969 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f35e4014540 con 0x7f35f40954f0 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 msgr2=0x7f35f402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 0x7f35f402d440 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4012620 comp rx=0 tx=0).stop 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 msgr2=0x7f35f402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35eaffd640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f40954f0 0x7f35f402d440 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4012620 comp rx=0 tx=0).stop 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.943+0000 7f35f2d9d640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35d403e210 0x7f35d40406d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35f4074d80 tx=0x7f35ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f8f3a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 0x7f35f402dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f8f3a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:45.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35f40a4230 con 0x7f35f402d980 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 0x7f35f402dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient: get_auth_request con 0x7f35f402d980 auth_method 0 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 10 monclient(hunting): handle_auth_done global_id 4925 payload 824 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.949+0000 7f35f259c640 1 -- 172.21.15.164:0/1655854372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f35f409d9c0 con 0x7f35f402d980 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f259c640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 msgr2=0x7f35f402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f259c640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 0x7f35f402dd90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4008cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:45.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35e401b030 con 0x7f35f402d980 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 10 monclient: dump: 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:45.972 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f35e401c040 con 0x7f35f402d980 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 msgr2=0x7f35f402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 0x7f35f402dd90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4008cc0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 msgr2=0x7f35f402dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35eaffd640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f402d980 0x7f35f402dd90 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f35e4001eb0 tx=0x7f35e4008cc0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:45.973 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 348124, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:11:45.974 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.975 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.976 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:45.977 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:45.978 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:45.979 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.980 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.981 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.982 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.983 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:45.984 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:45.985 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.986 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:45.987 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35d403e210 msgr2=0x7f35d40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35d403e210 0x7f35d40406d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f35f4074d80 tx=0x7f35ec009040 comp rx=0 tx=0).stop 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 10 monclient: shutdown 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 msgr2=0x7f35f41ce290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f2d9d640 1 -- 172.21.15.164:0/1655854372 reap_dead start 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 0x7f35f41ce290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f35e000c6d0 tx=0x7f35dc004730 comp rx=0 tx=0).stop 2024-09-10T16:11:45.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 shutdown_connections 2024-09-10T16:11:45.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 --2- 172.21.15.164:0/1655854372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35f40966f0 0x7f35f41ce290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:45.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 >> 172.21.15.164:0/1655854372 conn(0x7f35f40a3110 msgr2=0x7f35f4139c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:45.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 shutdown_connections 2024-09-10T16:11:45.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:45.952+0000 7f35f8f3a640 1 -- 172.21.15.164:0/1655854372 wait complete. 2024-09-10T16:11:45.990 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:45.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:11:45.990 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:11:46.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.087+0000 7f8547394640 10 monclient: get_monmap_and_config 2024-09-10T16:11:46.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.087+0000 7f8547394640 10 monclient: build_initial_monmap 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.087+0000 7f8547394640 10 monclient: monmap: 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:11:46.088164+0000 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:11:46.088164+0000 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:11:46.089 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:11:46.090 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.087+0000 7f8547394640 1 Processor -- start 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 -- start start 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient: init 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 0x7f854009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): picked mon.noname-c con 0x7f854009c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f854009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): picked mon.noname-b con 0x7f854009d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f854009f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854009a430 con 0x7f854009f4c0 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f85400a24e0 con 0x7f854009c180 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f85400a2bf0 con 0x7f854009d160 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8547394640 10 monclient(hunting): authenticate will time out at 1084.575528s 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f854009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8545931640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f854009f4c0 0x7f854013b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51988/0 (socket says 172.21.15.164:51988) 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8545931640 1 -- 172.21.15.164:0/1968512935 learned_addr learned my addr 172.21.15.164:0/1968512935 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): get_auth_request con 0x7f854009d160 auth_method 0 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8545130640 1 --2- 172.21.15.164:0/1968512935 >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 0x7f854009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient(hunting): handle_auth_done global_id 4928 payload 274 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 -- 172.21.15.164:0/1968512935 >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 msgr2=0x7f854009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 --2- 172.21.15.164:0/1968512935 >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 0x7f854009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 -- 172.21.15.164:0/1968512935 >> v1:172.21.15.164:6789/0 conn(0x7f854009f4c0 legacy=0x7f854013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient: _finish_hunting 0 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 monclient: found mon.noname-b 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f854492f640 1 -- 172.21.15.164:0/1968512935 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854009c560 con 0x7f854009d160 2024-09-10T16:11:46.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8545130640 10 monclient: get_auth_request con 0x7f854009c180 auth_method 0 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.088+0000 7f8545130640 1 --2- 172.21.15.164:0/1968512935 >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 0x7f854009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.091+0000 7f854492f640 1 -- 172.21.15.164:0/1968512935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 msgr2=0x7f854009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.091+0000 7f854492f640 1 --2- 172.21.15.164:0/1968512935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f854009ef80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6fa06d8d5c9ce9b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 1 -- 172.21.15.164:0/1968512935 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c037070 con 0x7f854009d160 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: dump: 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.098 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: _finish_auth 0 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: _check_auth_tickets 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 1 -- 172.21.15.164:0/1968512935 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c032640 con 0x7f854009d160 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 5 monclient: authenticate success, global_id 4928 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 10 monclient: get_monmap_and_config success 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 1 -- 172.21.15.164:0/1968512935 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c032a10 con 0x7f854009d160 2024-09-10T16:11:46.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 4 set_mon_vals no callback set 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8533fff640 10 monclient: dump: 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.100 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 10 monclient: shutdown 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 -- 172.21.15.164:0/1968512935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 msgr2=0x7f854009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 --2- 172.21.15.164:0/1968512935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f854009ef80 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f853c0097a0 tx=0x7f853c02a050 comp rx=0 tx=0).stop 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 -- 172.21.15.164:0/1968512935 shutdown_connections 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 --2- 172.21.15.164:0/1968512935 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f854009ef80 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 --2- 172.21.15.164:0/1968512935 >> v2:172.21.15.165:3300/0 conn(0x7f854009c180 0x7f854009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 -- 172.21.15.164:0/1968512935 >> 172.21.15.164:0/1968512935 conn(0x7f85400a3110 msgr2=0x7f85400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 -- 172.21.15.164:0/1968512935 shutdown_connections 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.092+0000 7f8547394640 1 -- 172.21.15.164:0/1968512935 wait complete. 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.093+0000 7f8547394640 10 monclient: build_initial_monmap 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.093+0000 7f8547394640 10 monclient: monmap: 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:11:46.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.093+0000 7f8547394640 1 Processor -- start 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.093+0000 7f8547394640 1 -- start start 2024-09-10T16:11:46.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.093+0000 7f8547394640 10 monclient: init 2024-09-10T16:11:46.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient: _reopen_session rank -1 2024-09-10T16:11:46.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:11:46.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854009c180 0x7f8540097440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): picked mon.noname-b con 0x7f854009c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:11:46.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f8540097980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): picked mon.noname-c con 0x7f854009d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854009f4c0 0x7f854008df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): picked mon.noname-a con 0x7f854009f4c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8540074f90 con 0x7f854009f4c0 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8540074e10 con 0x7f854009c180 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f854009d930 con 0x7f854009d160 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): _renew_subs 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8547394640 10 monclient(hunting): authenticate will time out at 1084.581188s 2024-09-10T16:11:46.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f8540097980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f8540097980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55088/0 (socket says 172.21.15.164:55088) 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854009f4c0 0x7f854008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 -- 172.21.15.164:0/443516034 learned_addr learned my addr 172.21.15.164:0/443516034 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): get_auth_request con 0x7f854009d160 auth_method 0 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): get_auth_request con 0x7f854009f4c0 auth_method 0 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:46.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854009c180 0x7f8540097440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 10 monclient(hunting): get_auth_request con 0x7f854009c180 auth_method 0 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient(hunting): handle_auth_done global_id 4931 payload 824 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854009c180 msgr2=0x7f8540097440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854009c180 0x7f8540097440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854009f4c0 msgr2=0x7f854008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854009f4c0 0x7f854008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient: _finish_hunting 0 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 monclient: found mon.noname-c 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f854492f640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854009d620 con 0x7f854009d160 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.094+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854009c180 0x7f8540097440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f854492f640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 msgr2=0x7f8540097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f854492f640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f8540097980 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f853c000c00 tx=0x7f853c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c037040 con 0x7f854009d160 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: dump: 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: _finish_auth 0 2024-09-10T16:11:46.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c042070 con 0x7f854009d160 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8547394640 5 monclient: authenticate success, global_id 4931 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c030d40 con 0x7f854009d160 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8547394640 10 monclient: _renew_subs 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8547394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8540090620 con 0x7f854009d160 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f8531ffb640 10 monclient: dump: 2024-09-10T16:11:46.114 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.096+0000 7f85337fe640 4 set_mon_vals no callback set 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.097+0000 7f8547394640 10 monclient: _renew_subs 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.097+0000 7f8547394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:11:46.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.097+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8540090be0 con 0x7f854009d160 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8547394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8547394640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 0x7f854008e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f854008ede0 con 0x7f854008e4c0 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f853c032500 con 0x7f854009d160 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 0x7f854008e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8531ffb640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f852403e210 0x7f85240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient: get_auth_request con 0x7f854008e4c0 auth_method 0 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(199..199 src has 1..199) ==== 4811+0+0 (secure 0 0 0) 0x7f853c0736a0 con 0x7f854009d160 2024-09-10T16:11:46.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545931640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f852403e210 0x7f85240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 10 monclient(hunting): handle_auth_done global_id 64135 payload 824 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545130640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85400a1580 con 0x7f854008e4c0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.098+0000 7f8545931640 10 monclient: get_auth_request con 0x7f852403e210 auth_method 0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.100+0000 7f8545130640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 msgr2=0x7f854008e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.100+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 0x7f854008e8a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f853400f4e0 tx=0x7f853400f9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534018070 con 0x7f854008e4c0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:46.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 10 monclient: dump: 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8545931640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f852403e210 0x7f85240406d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f85380055a0 tx=0x7f8538005530 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:46.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8534014b80 con 0x7f854008e4c0 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 msgr2=0x7f854008e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 0x7f854008e8a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f853400f4e0 tx=0x7f853400f9a0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 msgr2=0x7f854008e8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:46.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.101+0000 7f8531ffb640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854008e4c0 0x7f854008e8a0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f853400f4e0 tx=0x7f853400f9a0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.107+0000 7f8547394640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:11:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.107+0000 7f8547394640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 0x7f854006fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:11:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.107+0000 7f8547394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:11:46.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.107+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8540077260 con 0x7f854006f7e0 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 0x7f854006fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient: get_auth_request con 0x7f854006f7e0 auth_method 0 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 10 monclient(hunting): handle_auth_done global_id 64138 payload 824 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.108+0000 7f8545130640 1 -- 172.21.15.164:0/443516034 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85401c9190 con 0x7f854006f7e0 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8545130640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 msgr2=0x7f854006fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:11:46.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8545130640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 0x7f854006fbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f853401a000 tx=0x7f85340046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534029420 con 0x7f854006f7e0 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8531ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8531ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.110+0000 7f8531ffb640 10 monclient: dump: 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:11:46.123 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f8534029420 con 0x7f854006f7e0 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 msgr2=0x7f854006fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 0x7f854006fbf0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f853401a000 tx=0x7f85340046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 msgr2=0x7f854006fbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:11:46.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8531ffb640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854006f7e0 0x7f854006fbf0 secure :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f853401a000 tx=0x7f85340046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 58, 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10538, 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:11:46.125 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:46.126 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:46.127 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:11:46.128 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:11:46.129 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:46.130 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:11:46.131 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.132 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:46.133 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:11:46.134 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.135 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:11:46.136 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:11:46.137 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:11:46.137 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:11:46.137 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:11:46.137 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:11:46.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f852403e210 msgr2=0x7f85240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f852403e210 0x7f85240406d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f85380055a0 tx=0x7f8538005530 comp rx=0 tx=0).stop 2024-09-10T16:11:46.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 10 monclient: shutdown 2024-09-10T16:11:46.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:11:46.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 msgr2=0x7f8540097980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 --2- 172.21.15.164:0/443516034 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854009d160 0x7f8540097980 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f853c000c00 tx=0x7f853c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8545931640 1 -- 172.21.15.164:0/443516034 reap_dead start 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8545931640 1 -- 172.21.15.164:0/443516034 reap_dead start 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 shutdown_connections 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 >> 172.21.15.164:0/443516034 conn(0x7f85400a3110 msgr2=0x7f85401bc450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 shutdown_connections 2024-09-10T16:11:46.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:11:46.111+0000 7f8547394640 1 -- 172.21.15.164:0/443516034 wait complete. 2024-09-10T16:11:46.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:11:46.139 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:11:46.140 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-10T16:11:46.140 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:11:46.140 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:11:46.140 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:11:46.140 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:11:46.150 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:11:46.150 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-10T16:11:46.150 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-10T16:11:46.151 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:12:01.151 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-10T16:12:01.152 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-10T16:12:01.152 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:12:01.153 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.259+0000 7f7960e60640 10 monclient: get_monmap_and_config 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.259+0000 7f7960e60640 10 monclient: build_initial_monmap 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.259+0000 7f7960e60640 10 monclient: monmap: 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:12:01.261623+0000 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:12:01.261623+0000 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:12:01.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.260+0000 7f7960e60640 1 Processor -- start 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.260+0000 7f7960e60640 1 -- start start 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.260+0000 7f7960e60640 10 monclient: init 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f795c1348c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-a con 0x7f795c073780 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-b con 0x7f795c13d460 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795c073c10 con 0x7f795c1348c0 2024-09-10T16:12:01.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f795c073780 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f795c13d460 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7960e60640 10 monclient(hunting): authenticate will time out at 1099.749107s 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795a59c640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f795c1348c0 legacy=0x7f795c134ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7959d9b640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 msgr2=0x7f795c13cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f7959d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33996/0 (socket says 172.21.15.164:33996) 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 learned_addr learned my addr 172.21.15.164:0/2815364414 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f795c13d460 auth_method 0 2024-09-10T16:12:01.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:01.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.261+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:01.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f795c1348c0 legacy=0x7f795c134ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 msgr2=0x7f795c13cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f795c13d460 auth_method 0 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:01.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.462+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f795c1348c0 legacy=0x7f795c134ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 msgr2=0x7f795c13cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f795c13d460 auth_method 0 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:01.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:01.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:01.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:01.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:01.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:01.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:01.863+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:02.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f795c1348c0 legacy=0x7f795c134ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:02.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 msgr2=0x7f795c13cf20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:02.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:02.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:02.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f795c13d460 auth_method 0 2024-09-10T16:12:02.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:02.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:02.664+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:04.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 msgr2=0x7f795c13f850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f795c13d460 0x7f795c13f850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 msgr2=0x7f795c13cf20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f795c073780 0x7f795c13cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f795c1348c0 legacy=0x7f795c134ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient: _add_conns ranks=[4,1] 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:04.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f793c0036c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c006210 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795c134620 con 0x7f793c006210 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c0036c0 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0036c0 auth_method 0 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:04.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.262+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.265+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:04.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0036c0 auth_method 0 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:04.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.463+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:04.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:04.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0036c0 auth_method 0 2024-09-10T16:12:04.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:04.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:04.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:04.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:04.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:04.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:04.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:04.865+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:05.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.577+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T16:12:05.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.577+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:12:05.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.577+0000 7f7959d9b640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=45 cs=1 l=1).handle_message read tag failed 2024-09-10T16:12:05.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.577+0000 7f7959d9b640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=45 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:12:05.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.577+0000 7f795959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:12:05.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:05.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0036c0 auth_method 0 2024-09-10T16:12:05.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:05.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:05.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:05.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:05.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:05.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:05.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:05.666+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 msgr2=0x7f793c005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c0036c0 0x7f793c005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 reap_dead start 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c006210 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f793c009f10 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f793c00a480 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f795c073c10 con 0x7f793c006210 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c009f10 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f793c00a480 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00a480 auth_method 0 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.262+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.265+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00a480 auth_method 0 2024-09-10T16:12:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.463+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.464+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.464+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:07.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00a480 auth_method 0 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:07.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.864+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:07.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.865+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:07.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:07.865+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00a480 auth_method 0 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.665+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.666+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:08.666+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:10.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 msgr2=0x7f793c00a930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c00a480 0x7f793c00a930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c009f10 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient: _add_conns ranks=[4,0] 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f793c00d390 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f793c010490 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:10.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c00e250 con 0x7f793c010490 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c00d390 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.262+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:10.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:10.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:10.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.263+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:10.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.463+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:10.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:10.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:10.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.465+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:10.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.865+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:10.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:10.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:11.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.666+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:11.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.668+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:11.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.668+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:11.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.668+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.668+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.668+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.669+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.669+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.669+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:11.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:11.669+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:13.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:13.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:13.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:13.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c006210 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 reap_dead start 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f793c010490 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f793c00d390 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c014fb0 con 0x7f793c010490 2024-09-10T16:12:13.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c00e250 con 0x7f793c006210 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c00d390 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:13.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.263+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:13.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.265+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:13.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:13.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:13.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.464+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:13.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:13.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:13.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:13.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:13.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:13.865+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:14.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:14.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:14.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:14.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:14.666+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:15.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:15.578+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T16:12:15.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:15.578+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:12:15.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:15.578+0000 7f795a59c640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=58 cs=1 l=1).handle_message read tag failed 2024-09-10T16:12:15.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:15.578+0000 7f795a59c640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=58 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:12:15.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:15.578+0000 7f795959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:12:16.155 INFO:tasks.ceph.mon.c.smithi164.stderr:2024-09-10T16:12:16.153+0000 7ff4c836f640 -1 mon.c@2(probing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 101 at 2024-09-10T16:11:45.662082+0000) 2024-09-10T16:12:16.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d7a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c010490 legacy=0x7f793c012880 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient: _add_conns ranks=[3,2] 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:16.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f793c013fa0 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f793c0148c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c013fa0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f793c0148c0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 msgr2=0x7f793c014380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0148c0 auth_method 0 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:16.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:16.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:16.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:16.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:16.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:16.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.263+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:16.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 msgr2=0x7f793c014380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0148c0 auth_method 0 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:16.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.464+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.865+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 msgr2=0x7f793c014380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.865+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0148c0 auth_method 0 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:16.866+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:17.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.666+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 msgr2=0x7f793c014380 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:17.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.666+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0148c0 auth_method 0 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:17.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:17.667+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:19.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:19.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:19.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:19.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:19.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:19.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0148c0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:19.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 msgr2=0x7f793c014380 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:19.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c013fa0 0x7f793c014380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:19.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T16:12:19.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 reap_dead start 2024-09-10T16:12:19.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:19.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f793c00d390 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c006210 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f793c00b140 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c00a930 con 0x7f793c006210 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:19.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c00b140 2024-09-10T16:12:19.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:19.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f793c00d390 2024-09-10T16:12:19.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:19.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.263+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:19.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:19.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:19.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:19.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:19.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:19.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.264+0000 7f795a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:19.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.266+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:19.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.266+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:19.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.266+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:19.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.266+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:19.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.266+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:19.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.465+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:19.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.465+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.468+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:19.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.867+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:19.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.867+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:19.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:19.870+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:20.580 INFO:tasks.ceph.mon.b.smithi165.stderr:2024-09-10T16:12:20.579+0000 7f2084139640 -1 mon.b@1(probing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2024-09-10T16:11:46.111815+0000) 2024-09-10T16:12:20.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.578+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T16:12:20.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.578+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:12:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.578+0000 7f7959d9b640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=68 cs=1 l=1).handle_message read tag failed 2024-09-10T16:12:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.578+0000 7f7959d9b640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 0x7f793c0086d0 :-1 s=OPENED pgs=68 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:12:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.578+0000 7f795959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:12:20.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.668+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:20.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.668+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:20.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.671+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:20.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c00d390 auth_method 0 2024-09-10T16:12:20.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:20.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:20.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:20.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:20.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:20.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:20.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:20.672+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:21.155 INFO:tasks.ceph.mon.c.smithi164.stderr:2024-09-10T16:12:21.154+0000 7ff4c836f640 -1 mon.c@2(probing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 101 at 2024-09-10T16:11:45.662082+0000) 2024-09-10T16:12:22.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:22.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 msgr2=0x7f793c00d770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d390 0x7f793c00d770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00b140 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c006210 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient: _add_conns ranks=[0,2] 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f793c00b580 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f793c0124a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c014fb0 con 0x7f793c00b580 2024-09-10T16:12:22.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c0124a0 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c00b580 legacy=0x7f793c00b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:22.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.264+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.465+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c00b580 legacy=0x7f793c00b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.466+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.866+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c00b580 legacy=0x7f793c00b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.867+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:22.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:22.868+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:23.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.667+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c00b580 legacy=0x7f793c00b960 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:23.669+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:25.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c00b580 legacy=0x7f793c00b960 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 reap_dead start 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f793c0124a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f793c006210 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f793c00d8b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:25.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c006210 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f793c00d8b0 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f793c008ea0 con 0x7f793c0124a0 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 msgr2=0x7f793c00d370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.264+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d8b0 auth_method 0 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:25.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:12:25.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.265+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:25.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.465+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 msgr2=0x7f793c00d370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:25.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.465+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:25.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d8b0 auth_method 0 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.466+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:25.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.467+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:12:25.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.467+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.467+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.467+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:12:25.580 INFO:tasks.ceph.mon.b.smithi165.stderr:2024-09-10T16:12:25.578+0000 7f2084139640 -1 mon.b@1(probing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2024-09-10T16:11:46.111815+0000) 2024-09-10T16:12:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.866+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 msgr2=0x7f793c00d370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:25.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.866+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.867+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.868+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.868+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d8b0 auth_method 0 2024-09-10T16:12:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.868+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.868+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:25.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.868+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.869+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.869+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.869+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:25.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:25.869+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:12:26.156 INFO:tasks.ceph.mon.c.smithi164.stderr:2024-09-10T16:12:26.154+0000 7ff4c836f640 -1 mon.c@2(electing) e1 get_health_metrics reporting 4 slow ops, oldest is log(1 entries from seq 101 at 2024-09-10T16:11:45.662082+0000) 2024-09-10T16:12:26.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.668+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 msgr2=0x7f793c00d370 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:26.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.668+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0124a0 auth_method 0 2024-09-10T16:12:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.669+0000 7f795a59c640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:26.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 10 monclient(hunting): get_auth_request con 0x7f793c00d8b0 auth_method 0 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:26.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:26.671+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:12:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 msgr2=0x7f793c015180 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c0124a0 0x7f793c015180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.165:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 msgr2=0x7f793c00d370 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c006210 0x7f793c00d370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient: _add_conns ranks=[1,0] 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c0090c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f793c0094d0 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c014ca0 con 0x7f793c0094d0 2024-09-10T16:12:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c014fb0 con 0x7f793c0090c0 2024-09-10T16:12:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.265+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c0094d0 legacy=0x7f793c011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.267+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.466+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c0094d0 legacy=0x7f793c011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:28.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:28.867+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c0094d0 legacy=0x7f793c011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:29.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:29.668+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c0094d0 legacy=0x7f793c011460 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:30.580 INFO:tasks.ceph.mon.b.smithi165.stderr:2024-09-10T16:12:30.579+0000 7f2084139640 -1 mon.b@1(electing) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 4 at 2024-09-10T16:11:46.111815+0000) 2024-09-10T16:12:30.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:30.579+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-09-10T16:12:30.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:30.579+0000 7f795a59c640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:12:30.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:30.579+0000 7f795a59c640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 0x7f793c0086d0 :-1 s=OPENED pgs=83 cs=1 l=1).handle_message read tag failed 2024-09-10T16:12:30.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:30.579+0000 7f795a59c640 1 --1- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 0x7f793c0086d0 :-1 s=OPENED pgs=83 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:12:30.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:30.579+0000 7f795959a640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:12:31.156 INFO:tasks.ceph.mon.c.smithi164.stderr:2024-09-10T16:12:31.155+0000 7ff4c836f640 -1 mon.c@2(peon) e1 get_health_metrics reporting 3 slow ops, oldest is log(1 entries from seq 101 at 2024-09-10T16:11:45.662082+0000) 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): tick 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.164:6789/0 conn(0x7f793c0094d0 legacy=0x7f793c011460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 reap_dead start 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f793c0090c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f793c00d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f793c009b30 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f793c00d8b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f793c0098e0 con 0x7f793c0090c0 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f795c09ccd0 con 0x7f793c00d8b0 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c09cfb0 con 0x7f793c009b30 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f794bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f795ad9d640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f795ad9d640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f793c00d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c009b30 auth_method 0 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient(hunting): handle_auth_done global_id 4934 payload 274 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 msgr2=0x7f793c005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> v1:172.21.15.165:6789/0 conn(0x7f793c0090c0 legacy=0x7f793c0086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 1 monclient: found mon.noname-b 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.265+0000 7f7959d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.266+0000 7f7959d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.266+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f793c014ca0 con 0x7f793c009b30 2024-09-10T16:12:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7959d9b640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 msgr2=0x7f793c00d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7959d9b640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f793c00d370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7950003c20 tx=0x7f795002c4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f139e43eda8c00c9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 1 -- 172.21.15.164:0/2815364414 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950038070 con 0x7f793c009b30 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: dump: 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: _finish_auth 0 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: _check_auth_tickets 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 1 -- 172.21.15.164:0/2815364414 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7950008040 con 0x7f793c009b30 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7960e60640 5 monclient: authenticate success, global_id 4934 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7960e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 1 -- 172.21.15.164:0/2815364414 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950031a30 con 0x7f793c009b30 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7960e60640 10 monclient: get_monmap_and_config success 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f7960e60640 4 set_mon_vals no callback set 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:31.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.269+0000 7f795959a640 10 monclient: dump: 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 10 monclient: shutdown 2024-09-10T16:12:31.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 -- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 msgr2=0x7f793c00d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f793c00d370 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7950003c20 tx=0x7f795002c4f0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 -- 172.21.15.164:0/2815364414 shutdown_connections 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 --2- 172.21.15.164:0/2815364414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f793c00d370 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 --2- 172.21.15.164:0/2815364414 >> v2:172.21.15.164:3300/0 conn(0x7f793c00d8b0 0x7f793c005ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 -- 172.21.15.164:0/2815364414 >> 172.21.15.164:0/2815364414 conn(0x7f795c0a6f80 msgr2=0x7f795c0a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 -- 172.21.15.164:0/2815364414 shutdown_connections 2024-09-10T16:12:31.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.270+0000 7f7960e60640 1 -- 172.21.15.164:0/2815364414 wait complete. 2024-09-10T16:12:31.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.271+0000 7f7960e60640 10 monclient: build_initial_monmap 2024-09-10T16:12:31.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.271+0000 7f7960e60640 10 monclient: monmap: 2024-09-10T16:12:31.280 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 Processor -- start 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 -- start start 2024-09-10T16:12:31.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient: init 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f793c0090c0 0x7f795c1cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-b con 0x7f793c0090c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-c con 0x7f793c009b30 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 0x7f795c1309f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): picked mon.noname-a con 0x7f793c00d8b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f795c0731c0 con 0x7f793c00d8b0 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f795c073040 con 0x7f793c0090c0 2024-09-10T16:12:31.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f795c134ca0 con 0x7f793c009b30 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.272+0000 7f7960e60640 10 monclient(hunting): authenticate will time out at 1129.759979s 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795ad9d640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 msgr2=0x7f795c1309f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795ad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 0x7f795c1309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36506/0 (socket says 172.21.15.164:36506) 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 -- 172.21.15.164:0/3871528796 learned_addr learned my addr 172.21.15.164:0/3871528796 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): get_auth_request con 0x7f793c009b30 auth_method 0 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f793c0090c0 0x7f795c1cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): get_auth_request con 0x7f793c0090c0 auth_method 0 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f795a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient(hunting): handle_auth_done global_id 4937 payload 824 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 -- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f793c0090c0 msgr2=0x7f795c1cc9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f793c0090c0 0x7f795c1cc9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 -- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 msgr2=0x7f795c1309f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 0x7f795c1309f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 monclient: found mon.noname-c 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.273+0000 7f7959d9b640 1 -- 172.21.15.164:0/3871528796 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795c13c390 con 0x7f793c009b30 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.275+0000 7f7959d9b640 1 -- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 msgr2=0x7f795c1304b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.275+0000 7f7959d9b640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7950000c00 tx=0x7f79500073f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950043020 con 0x7f793c009b30 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: dump: 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.286 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: _finish_auth 0 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: _check_auth_tickets 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f795003e6b0 con 0x7f793c009b30 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 5 monclient: authenticate success, global_id 4937 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 10 monclient: _renew_subs 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f795c0a16e0 con 0x7f793c009b30 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7950038040 con 0x7f793c009b30 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f794affd640 10 monclient: dump: 2024-09-10T16:12:31.287 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 10 monclient: _renew_subs 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f795c13b4e0 con 0x7f793c009b30 2024-09-10T16:12:31.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.276+0000 7f7958d99640 4 set_mon_vals no callback set 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7950046020 con 0x7f793c009b30 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f7960e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f7960e60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f7960e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f795c134fd0 con 0x7f793c009b30 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f794affd640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 0x7f793c043600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.277+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(200..200 src has 1..200) ==== 4755+0+0 (secure 0 0 0) 0x7f7950075400 con 0x7f793c009b30 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.278+0000 7f795a59c640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 0x7f793c043600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.278+0000 7f795a59c640 10 monclient: get_auth_request con 0x7f793c041140 auth_method 0 2024-09-10T16:12:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.280+0000 7f795a59c640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 0x7f793c043600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f794c0095d0 tx=0x7f794c00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.282+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f795004d030 con 0x7f793c009b30 2024-09-10T16:12:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.282+0000 7f794affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:31.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.282+0000 7f794affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:12:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.530+0000 7f7960e60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:12:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.530+0000 7f7960e60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:12:31.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.530+0000 7f7960e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.530+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f795c134fd0 con 0x7f793c009b30 2024-09-10T16:12:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.531+0000 7f794affd640 1 -- 172.21.15.164:0/3871528796 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f795003f420 con 0x7f793c009b30 2024-09-10T16:12:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.531+0000 7f794affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:12:31.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.531+0000 7f794affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:12:31.533 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":60,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 msgr2=0x7f793c043600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 0x7f793c043600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f794c0095d0 tx=0x7f794c00d040 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 10 monclient: shutdown 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 msgr2=0x7f795c1304b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7950000c00 tx=0x7f79500073f0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 shutdown_connections 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f793c041140 0x7f793c043600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f793c00d8b0 0x7f795c1309f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f793c009b30 0x7f795c1304b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 --2- 172.21.15.164:0/3871528796 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f793c0090c0 0x7f795c1cc9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 >> 172.21.15.164:0/3871528796 conn(0x7f795c0a6f80 msgr2=0x7f795c0a19b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 shutdown_connections 2024-09-10T16:12:31.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.533+0000 7f7960e60640 1 -- 172.21.15.164:0/3871528796 wait complete. 2024-09-10T16:12:31.545 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:12:31.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:12:31.546 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.643+0000 7f8a39847640 10 monclient: get_monmap_and_config 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.643+0000 7f8a39847640 10 monclient: build_initial_monmap 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.643+0000 7f8a39847640 10 monclient: monmap: 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:12:31.644770+0000 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:12:31.644770+0000 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:12:31.646 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.644+0000 7f8a39847640 1 Processor -- start 2024-09-10T16:12:31.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.644+0000 7f8a39847640 1 -- start start 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.644+0000 7f8a39847640 10 monclient: init 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 0x7f8a34135000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-a con 0x7f8a3406cd00 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a340a46e0 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a340a4b30 0x7f8a34138300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-c con 0x7f8a340a4b30 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a3406df10 con 0x7f8a340a46e0 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f8a340a4f10 con 0x7f8a3406cd00 2024-09-10T16:12:31.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8a3406d0e0 con 0x7f8a340a4b30 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a39847640 10 monclient(hunting): authenticate will time out at 1130.132103s 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a32ffd640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 msgr2=0x7f8a34135000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a327fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f8a340a46e0 legacy=0x7f8a34137bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a32ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 0x7f8a34135000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a340a4b30 0x7f8a34138300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8a340a4b30 0x7f8a34138300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:56772/0 (socket says 172.21.15.164:56772) 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 -- 172.21.15.164:0/2865545695 learned_addr learned my addr 172.21.15.164:0/2865545695 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): get_auth_request con 0x7f8a340a4b30 auth_method 0 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient(hunting): handle_auth_done global_id 64144 payload 274 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 -- 172.21.15.164:0/2865545695 >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 msgr2=0x7f8a34135000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 --2- 172.21.15.164:0/2865545695 >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 0x7f8a34135000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 -- 172.21.15.164:0/2865545695 >> v1:172.21.15.164:6789/0 conn(0x7f8a340a46e0 legacy=0x7f8a34137bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient: _finish_hunting 0 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 monclient: found mon.noname-c 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:12:31.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.645+0000 7f8a337fe640 1 -- 172.21.15.164:0/2865545695 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3409cb90 con 0x7f8a340a4b30 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.648+0000 7f8a337fe640 1 -- 172.21.15.164:0/2865545695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a340a4b30 msgr2=0x7f8a34138300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.648+0000 7f8a337fe640 1 --2- 172.21.15.164:0/2865545695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a340a4b30 0x7f8a34138300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a340a5090 tx=0x7f8a2c02a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=51b8edaf4116dda8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 1 -- 172.21.15.164:0/2865545695 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c036070 con 0x7f8a340a4b30 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: dump: 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.655 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: _finish_auth 0 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 1 -- 172.21.15.164:0/2865545695 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2c007800 con 0x7f8a340a4b30 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 5 monclient: authenticate success, global_id 64144 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:12:31.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 1 -- 172.21.15.164:0/2865545695 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a2c007bd0 con 0x7f8a340a4b30 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a31ffb640 10 monclient: dump: 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.657 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 10 monclient: get_monmap_and_config success 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 4 set_mon_vals no callback set 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 10 monclient: shutdown 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 -- 172.21.15.164:0/2865545695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a340a4b30 msgr2=0x7f8a34138300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 --2- 172.21.15.164:0/2865545695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a340a4b30 0x7f8a34138300 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8a340a5090 tx=0x7f8a2c02a580 comp rx=0 tx=0).stop 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 -- 172.21.15.164:0/2865545695 shutdown_connections 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 --2- 172.21.15.164:0/2865545695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a340a4b30 0x7f8a34138300 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 --2- 172.21.15.164:0/2865545695 >> v2:172.21.15.164:3300/0 conn(0x7f8a3406cd00 0x7f8a34135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 -- 172.21.15.164:0/2865545695 >> 172.21.15.164:0/2865545695 conn(0x7f8a340aaa70 msgr2=0x7f8a340aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 -- 172.21.15.164:0/2865545695 shutdown_connections 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.649+0000 7f8a39847640 1 -- 172.21.15.164:0/2865545695 wait complete. 2024-09-10T16:12:31.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.650+0000 7f8a39847640 10 monclient: build_initial_monmap 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.650+0000 7f8a39847640 10 monclient: monmap: 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.650+0000 7f8a39847640 1 Processor -- start 2024-09-10T16:12:31.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.650+0000 7f8a39847640 1 -- start start 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.650+0000 7f8a39847640 10 monclient: init 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a3406cd00 0x7f8a34131250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-b con 0x7f8a3406cd00 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 0x7f8a34131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-c con 0x7f8a340a46e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a340a4b30 0x7f8a3413dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): picked mon.noname-a con 0x7f8a340a4b30 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34138840 con 0x7f8a340a4b30 2024-09-10T16:12:31.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8a34134b80 con 0x7f8a3406cd00 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a341389c0 con 0x7f8a340a46e0 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a39847640 10 monclient(hunting): authenticate will time out at 1130.138478s 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a337fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a340a4b30 msgr2=0x7f8a3413dc10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a337fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a340a4b30 0x7f8a3413dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a3406cd00 0x7f8a34131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 0x7f8a34131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a3406cd00 0x7f8a34131250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:56782/0 (socket says 172.21.15.164:56782) 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): get_auth_request con 0x7f8a340a46e0 auth_method 0 2024-09-10T16:12:31.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 1 -- 172.21.15.164:0/1243419224 learned_addr learned my addr 172.21.15.164:0/1243419224 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): get_auth_request con 0x7f8a3406cd00 auth_method 0 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 10 monclient(hunting): handle_auth_done global_id 4940 payload 824 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a3406cd00 msgr2=0x7f8a34131250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8a3406cd00 0x7f8a34131250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a340a4b30 msgr2=0x7f8a3413dc10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.651+0000 7f8a327fc640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8a340a4b30 0x7f8a3413dc10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.652+0000 7f8a327fc640 10 monclient: _finish_hunting 0 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.652+0000 7f8a327fc640 1 monclient: found mon.noname-c 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.652+0000 7f8a327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.652+0000 7f8a327fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.652+0000 7f8a327fc640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a3409f170 con 0x7f8a340a46e0 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a327fc640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 msgr2=0x7f8a34131790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a327fc640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 0x7f8a34131790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b400 tx=0x7f8a2800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28016070 con 0x7f8a340a46e0 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:12:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: dump: 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: _finish_auth 0 2024-09-10T16:12:31.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a28005070 con 0x7f8a340a46e0 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 5 monclient: authenticate success, global_id 4940 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 10 monclient: _renew_subs 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a3406e670 con 0x7f8a340a46e0 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a28005480 con 0x7f8a340a46e0 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:31.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a1bfff640 10 monclient: dump: 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a317fa640 4 set_mon_vals no callback set 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 10 monclient: _renew_subs 2024-09-10T16:12:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.654+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a3409ee90 con 0x7f8a340a46e0 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a39847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a39847640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 0x7f8a3413ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a34050fd0 con 0x7f8a3413e910 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8a28019070 con 0x7f8a340a46e0 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 0x7f8a3413ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient: get_auth_request con 0x7f8a3413e910 auth_method 0 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a1bfff640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a2003e1c0 0x7f8a20040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_done global_id 4943 payload 824 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(200..200 src has 1..200) ==== 4755+0+0 (secure 0 0 0) 0x7f8a28052070 con 0x7f8a340a46e0 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a32ffd640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a34135000 con 0x7f8a3413e910 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a337fe640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a2003e1c0 0x7f8a20040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.655+0000 7f8a337fe640 10 monclient: get_auth_request con 0x7f8a2003e1c0 auth_method 0 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a32ffd640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 msgr2=0x7f8a3413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a32ffd640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 0x7f8a3413ecf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a24009be0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24016070 con 0x7f8a3413e910 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.657+0000 7f8a1bfff640 10 monclient: dump: 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8a2400ebc0 con 0x7f8a3413e910 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 msgr2=0x7f8a3413ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 0x7f8a3413ecf0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a24009be0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 msgr2=0x7f8a3413ecf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a1bfff640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3413e910 0x7f8a3413ecf0 secure :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a24009be0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.658+0000 7f8a337fe640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a2003e1c0 0x7f8a20040680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a34138cf0 tx=0x7f8a2c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a39847640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a39847640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 0x7f8a3402d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a39847640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8a3413a430 con 0x7f8a3402d440 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 0x7f8a3402d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient: get_auth_request con 0x7f8a3402d440 auth_method 0 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 10 monclient(hunting): handle_auth_done global_id 4946 payload 824 2024-09-10T16:12:31.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.664+0000 7f8a32ffd640 1 -- 172.21.15.164:0/1243419224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8a34131d30 con 0x7f8a3402d440 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a32ffd640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 msgr2=0x7f8a3402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a32ffd640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 0x7f8a3402d850 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a240059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24016040 con 0x7f8a3402d440 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:31.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.666+0000 7f8a1bfff640 10 monclient: dump: 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f8a2400ee50 con 0x7f8a3402d440 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:12:31.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 msgr2=0x7f8a3402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 0x7f8a3402d850 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a240059d0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 msgr2=0x7f8a3402d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a1bfff640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a3402d440 0x7f8a3402d850 secure :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f8a24009720 tx=0x7f8a240059d0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:12:31.678 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 60, 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 393838, 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:31.679 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:31.680 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.681 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:12:31.682 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:31.683 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:31.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:12:31.685 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:12:31.686 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:31.687 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.688 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:12:31.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:12:31.691 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:31.692 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:31.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:31.692 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:12:31.692 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:12:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a2003e1c0 msgr2=0x7f8a20040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8a2003e1c0 0x7f8a20040680 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8a34138cf0 tx=0x7f8a2c009000 comp rx=0 tx=0).stop 2024-09-10T16:12:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 10 monclient: shutdown 2024-09-10T16:12:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:31.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 msgr2=0x7f8a34131790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 --2- 172.21.15.164:0/1243419224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8a340a46e0 0x7f8a34131790 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8a2800b400 tx=0x7f8a2800b8c0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a337fe640 1 -- 172.21.15.164:0/1243419224 reap_dead start 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a337fe640 1 -- 172.21.15.164:0/1243419224 reap_dead start 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 shutdown_connections 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 >> 172.21.15.164:0/1243419224 conn(0x7f8a340aaa70 msgr2=0x7f8a340a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 shutdown_connections 2024-09-10T16:12:31.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.667+0000 7f8a39847640 1 -- 172.21.15.164:0/1243419224 wait complete. 2024-09-10T16:12:31.694 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:12:31.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:12:31.695 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 10 monclient: get_monmap_and_config 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 10 monclient: build_initial_monmap 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 10 monclient: monmap: 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:12:31.793117+0000 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:12:31.793117+0000 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:31.794 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 1 Processor -- start 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 1 -- start start 2024-09-10T16:12:31.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.792+0000 7f83a193d640 10 monclient: init 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-b con 0x7f839c06d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f839c06d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f839c13d060 0x7f839c13f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-a con 0x7f839c13d060 addr v2:172.21.15.164:3300/0 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f839c09cb90 con 0x7f839c06d540 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f839c09ce70 con 0x7f839c13d060 2024-09-10T16:12:31.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f839c09d150 con 0x7f839c06d160 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f83a193d640 10 monclient(hunting): authenticate will time out at 1130.280365s 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839a7fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f839c06d540 legacy=0x7f839c0a46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839b7fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f839c13d060 msgr2=0x7f839c13f450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f839c13d060 0x7f839c13f450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36542/0 (socket says 172.21.15.164:36542) 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 learned_addr learned my addr 172.21.15.164:0/3246720979 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): get_auth_request con 0x7f839c06d160 auth_method 0 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient(hunting): handle_auth_done global_id 4949 payload 274 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 >> v2:172.21.15.164:3300/0 conn(0x7f839c13d060 msgr2=0x7f839c13f450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 --2- 172.21.15.164:0/3246720979 >> v2:172.21.15.164:3300/0 conn(0x7f839c13d060 0x7f839c13f450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 >> v1:172.21.15.164:6789/0 conn(0x7f839c06d540 legacy=0x7f839c0a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient: _finish_hunting 0 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 monclient: found mon.noname-b 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:12:31.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.793+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c0a4df0 con 0x7f839c06d160 2024-09-10T16:12:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.794+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 msgr2=0x7f839c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:12:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.794+0000 7f839affd640 1 -- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 msgr2=0x7f839c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:12:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.794+0000 7f839affd640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f8394009870 tx=0x7f839402a0d0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:12:31.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.794+0000 7f839affd640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839affd640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839affd640 10 monclient: get_auth_request con 0x7f839c06d160 auth_method 0 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839affd640 0 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839affd640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 msgr2=0x7f839c134e00 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c06d160 0x7f839c134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient: _add_conns ranks=[1,4] 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8384003500 addr v1:172.21.15.165:6789/0 2024-09-10T16:12:31.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 1 --2- 172.21.15.164:0/3246720979 >> v2:172.21.15.165:3300/0 conn(0x7f8384007490 0x7f8384009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f8384007490 addr v2:172.21.15.165:3300/0 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f838400a220 con 0x7f8384003500 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f838400b420 con 0x7f8384007490 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f8399ffb640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 1 --2- 172.21.15.164:0/3246720979 >> v2:172.21.15.165:3300/0 conn(0x7f8384007490 0x7f8384009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): get_auth_request con 0x7f8384007490 auth_method 0 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:31.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient(hunting): handle_auth_done global_id 4949 payload 293 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 1 -- 172.21.15.164:0/3246720979 >> v1:172.21.15.165:6789/0 conn(0x7f8384003500 legacy=0x7f8384005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 1 monclient: found mon.noname-c 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:12:32.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.995+0000 7f839a7fc640 1 -- 172.21.15.164:0/3246720979 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f838400b690 con 0x7f8384007490 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.997+0000 7f839a7fc640 1 -- 172.21.15.164:0/3246720979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384007490 msgr2=0x7f8384009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.997+0000 7f839a7fc640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384007490 0x7f8384009950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f839c0775f0 tx=0x7f8394009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6c7372546ec18c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390009a30 con 0x7f8384007490 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: dump: 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.007 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: _finish_auth 0 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8390015620 con 0x7f8384007490 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 1 -- 172.21.15.164:0/3246720979 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390015a10 con 0x7f8384007490 2024-09-10T16:12:32.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f8399ffb640 10 monclient: dump: 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 5 monclient: authenticate success, global_id 4949 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 10 monclient: get_monmap_and_config success 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 4 set_mon_vals no callback set 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 10 monclient: shutdown 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 1 -- 172.21.15.164:0/3246720979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384007490 msgr2=0x7f8384009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.998+0000 7f83a193d640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384007490 0x7f8384009950 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f839c0775f0 tx=0x7f8394009240 comp rx=0 tx=0).stop 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f839b7fe640 1 -- 172.21.15.164:0/3246720979 reap_dead start 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 1 -- 172.21.15.164:0/3246720979 shutdown_connections 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 1 --2- 172.21.15.164:0/3246720979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8384007490 0x7f8384009950 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 1 -- 172.21.15.164:0/3246720979 >> 172.21.15.164:0/3246720979 conn(0x7f839c0aacb0 msgr2=0x7f839c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 1 -- 172.21.15.164:0/3246720979 shutdown_connections 2024-09-10T16:12:32.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 1 -- 172.21.15.164:0/3246720979 wait complete. 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 10 monclient: build_initial_monmap 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:31.999+0000 7f83a193d640 10 monclient: monmap: 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 Processor -- start 2024-09-10T16:12:32.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 -- start start 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient: init 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient: _reopen_session rank -1 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384007490 0x7f839c1cd2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-a con 0x7f8384007490 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-c con 0x7f839c1cd830 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12dfb0 0x7f839c12e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839affd640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384007490 msgr2=0x7f839c1cd2f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): picked mon.noname-b con 0x7f839c12dfb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f839c13fb10 con 0x7f8384007490 2024-09-10T16:12:32.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384007490 0x7f839c1cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f839c13f990 con 0x7f839c12dfb0 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f839c13fc90 con 0x7f839c1cd830 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): _renew_subs 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f83a193d640 10 monclient(hunting): authenticate will time out at 1130.487754s 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36546/0 (socket says 172.21.15.164:36546) 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 1 -- 172.21.15.164:0/4039495718 learned_addr learned my addr 172.21.15.164:0/4039495718 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): get_auth_request con 0x7f839c1cd830 auth_method 0 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:32.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839b7fe640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12dfb0 0x7f839c12e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839b7fe640 10 monclient(hunting): get_auth_request con 0x7f839c12dfb0 auth_method 0 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.000+0000 7f839b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:32.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 10 monclient(hunting): handle_auth_done global_id 4952 payload 824 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12dfb0 msgr2=0x7f839c12e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12dfb0 0x7f839c12e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384007490 msgr2=0x7f839c1cd2f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8384007490 0x7f839c1cd2f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 monclient: found mon.noname-c 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.001+0000 7f839a7fc640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f839c0a13d0 con 0x7f839c1cd830 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f839a7fc640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 msgr2=0x7f839c1cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f839a7fc640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8390000f80 tx=0x7f8390005d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:32.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390025070 con 0x7f839c1cd830 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: dump: 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.016 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: _finish_auth 0 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: _check_auth_tickets 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 5 monclient: authenticate success, global_id 4952 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8390015df0 con 0x7f839c1cd830 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390020660 con 0x7f839c1cd830 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 10 monclient: _renew_subs 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f839c0a2e90 con 0x7f839c1cd830 2024-09-10T16:12:32.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f8383fff640 10 monclient: dump: 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.018 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83997fa640 4 set_mon_vals no callback set 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 10 monclient: _renew_subs 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.003+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f839c0a1f30 con 0x7f839c1cd830 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f83a193d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f83a193d640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 0x7f839c1328c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f839c09f950 con 0x7f839c12e970 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8390028070 con 0x7f839c1cd830 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 0x7f839c1328c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient: get_auth_request con 0x7f839c12e970 auth_method 0 2024-09-10T16:12:32.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f8383fff640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838803e1c0 0x7f8388040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.004+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(200..200 src has 1..200) ==== 4755+0+0 (secure 0 0 0) 0x7f8390054000 con 0x7f839c1cd830 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.005+0000 7f839affd640 10 monclient(hunting): handle_auth_done global_id 64153 payload 824 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.005+0000 7f839affd640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f839c0a4790 con 0x7f839c12e970 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.005+0000 7f839b7fe640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838803e1c0 0x7f8388040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:32.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.005+0000 7f839b7fe640 10 monclient: get_auth_request con 0x7f838803e1c0 auth_method 0 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f839affd640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 msgr2=0x7f839c1328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f839affd640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 0x7f839c1328c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f839c09d320 tx=0x7f83940046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394004020 con 0x7f839c12e970 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 10 monclient: dump: 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.023 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f839b7fe640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838803e1c0 0x7f8388040680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f839c0a83d0 tx=0x7f838c0091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8394004020 con 0x7f839c12e970 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 msgr2=0x7f839c1328c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 0x7f839c1328c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f839c09d320 tx=0x7f83940046c0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 msgr2=0x7f839c1328c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:32.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.007+0000 7f8383fff640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c12e970 0x7f839c1328c0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f839c09d320 tx=0x7f83940046c0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.013+0000 7f83a193d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.013+0000 7f83a193d640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 0x7f839c133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.013+0000 7f83a193d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.013+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f839c0a3300 con 0x7f839c132e00 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 0x7f839c133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient: get_auth_request con 0x7f839c132e00 auth_method 0 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:12:32.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 10 monclient(hunting): handle_auth_done global_id 64156 payload 824 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.014+0000 7f839affd640 1 -- 172.21.15.164:0/4039495718 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f839c134e00 con 0x7f839c132e00 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.016+0000 7f839affd640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 msgr2=0x7f839c133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.016+0000 7f839affd640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 0x7f839c133210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83940097c0 tx=0x7f8394008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.016+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8394045c70 con 0x7f839c132e00 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.016+0000 7f8383fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 10 monclient: dump: 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:12:32.026 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f8394045c70 con 0x7f839c132e00 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 msgr2=0x7f839c133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 0x7f839c133210 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83940097c0 tx=0x7f8394008030 comp rx=0 tx=0).stop 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 msgr2=0x7f839c133210 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:12:32.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f8383fff640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f839c132e00 0x7f839c133210 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83940097c0 tx=0x7f8394008030 comp rx=0 tx=0).stop 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 60, 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 56446, 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:12:32.028 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:32.029 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:32.030 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:12:32.031 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:12:32.032 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:12:32.033 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.034 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:32.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:12:32.036 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:12:32.037 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:12:32.038 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:12:32.039 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:12:32.040 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:12:32.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838803e1c0 msgr2=0x7f8388040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838803e1c0 0x7f8388040680 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f839c0a83d0 tx=0x7f838c0091c0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 10 monclient: shutdown 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 msgr2=0x7f839c1cdc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8390000f80 tx=0x7f8390005d80 comp rx=0 tx=0).stop 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f839b7fe640 1 -- 172.21.15.164:0/4039495718 reap_dead start 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 shutdown_connections 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 --2- 172.21.15.164:0/4039495718 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f839c1cd830 0x7f839c1cdc90 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 >> 172.21.15.164:0/4039495718 conn(0x7f839c0aacb0 msgr2=0x7f839c130a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 shutdown_connections 2024-09-10T16:12:32.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:12:32.017+0000 7f83a193d640 1 -- 172.21.15.164:0/4039495718 wait complete. 2024-09-10T16:12:32.043 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:12:32.043 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:12:32.043 INFO:tasks.rados.rados.0.smithi165.stdout:799: done (4 left) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:800: done (3 left) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:801: done (2 left) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:802: done (1 left) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:803: done (0 left) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:804: read oid 29 snap -1 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:804: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:805: write (excl) oid 11 current snap is 89 2024-09-10T16:12:32.044 INFO:tasks.rados.rados.0.smithi165.stdout:805: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-10T16:12:32.051 INFO:tasks.rados.rados.0.smithi165.stdout:805: writing smithi16534841-11 from 410170 to 1191480 tid 1 2024-09-10T16:12:32.058 INFO:tasks.rados.rados.0.smithi165.stdout:805: writing smithi16534841-11 from 1940712 to 2596818 tid 2 2024-09-10T16:12:32.062 INFO:tasks.rados.rados.0.smithi165.stdout:805: writing smithi16534841-11 from 3163363 to 3210170 tid 3 2024-09-10T16:12:32.062 INFO:tasks.rados.rados.0.smithi165.stdout:806: write oid 43 current snap is 89 2024-09-10T16:12:32.062 INFO:tasks.rados.rados.0.smithi165.stdout:806: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2024-09-10T16:12:32.066 INFO:tasks.rados.rados.0.smithi165.stdout:806: writing smithi16534841-43 from 426977 to 883536 tid 1 2024-09-10T16:12:32.074 INFO:tasks.rados.rados.0.smithi165.stdout:806: writing smithi16534841-43 from 1482841 to 2282605 tid 2 2024-09-10T16:12:32.080 INFO:tasks.rados.rados.0.smithi165.stdout:806: writing smithi16534841-43 from 2958080 to 3226977 tid 3 2024-09-10T16:12:32.080 INFO:tasks.rados.rados.0.smithi165.stdout:807: rollback oid 16 current snap is 89 2024-09-10T16:12:32.080 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 85 2024-09-10T16:12:32.080 INFO:tasks.rados.rados.0.smithi165.stdout:804: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:12:32.112 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:32.143 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 445874 2024-09-10T16:12:32.174 INFO:tasks.rados.rados.0.smithi165.stdout:805: finishing write tid 1 to smithi16534841-11 2024-09-10T16:12:32.174 INFO:tasks.rados.rados.0.smithi165.stdout:805: finishing write tid 2 to smithi16534841-11 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: finishing write tid 3 to smithi16534841-11 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: finishing write tid 4 to smithi16534841-11 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: finishing write tid 5 to smithi16534841-11 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: oid 11 updating version 0 to 327 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: oid 11 updating version 327 to 328 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: oid 11 updating version 328 to 329 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: oid 11 updating version 329 to 330 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 330 (ObjNum 310 snap 89 seq_num 310) dirty exists 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: left oid 11 (ObjNum 310 snap 89 seq_num 310) 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:806: finishing write tid 1 to smithi16534841-43 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:804: done (3 left) 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:805: done (2 left) 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:808: read oid 42 snap 82 2024-09-10T16:12:32.175 INFO:tasks.rados.rados.0.smithi165.stdout:808: expect deleted 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:809: snap_create 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: finishing write tid 2 to smithi16534841-43 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: finishing write tid 3 to smithi16534841-43 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: finishing write tid 4 to smithi16534841-43 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: finishing write tid 5 to smithi16534841-43 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: oid 43 updating version 0 to 258 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: oid 43 updating version 258 to 259 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: oid 43 version 259 is already newer than 257 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: oid 43 version 259 is already newer than 256 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 259 (ObjNum 311 snap 89 seq_num 311) dirty exists 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:806: left oid 43 (ObjNum 311 snap 89 seq_num 311) 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:807: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:12:32.176 INFO:tasks.rados.rados.0.smithi165.stdout:807: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:12:32.177 INFO:tasks.rados.rados.0.smithi165.stdout:807: finishing rollback tid 2 to smithi16534841-16 2024-09-10T16:12:32.177 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 737 (ObjNum 280 snap 82 seq_num 280) dirty exists 2024-09-10T16:12:33.058 INFO:tasks.rados.rados.0.smithi165.stdout:806: done (3 left) 2024-09-10T16:12:33.058 INFO:tasks.rados.rados.0.smithi165.stdout:807: done (2 left) 2024-09-10T16:12:33.058 INFO:tasks.rados.rados.0.smithi165.stdout:808: done (1 left) 2024-09-10T16:12:33.058 INFO:tasks.rados.rados.0.smithi165.stdout:809: done (0 left) 2024-09-10T16:12:33.058 INFO:tasks.rados.rados.0.smithi165.stdout:810: snap_create 2024-09-10T16:12:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:810: done (0 left) 2024-09-10T16:12:34.461 INFO:tasks.rados.rados.0.smithi165.stdout:811: rollback oid 20 current snap is 91 2024-09-10T16:12:34.462 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 89 2024-09-10T16:12:34.462 INFO:tasks.rados.rados.0.smithi165.stdout:812: snap_remove snap 82 2024-09-10T16:12:35.462 INFO:tasks.rados.rados.0.smithi165.stdout:812: done (1 left) 2024-09-10T16:12:35.462 INFO:tasks.rados.rados.0.smithi165.stdout:813: write oid 25 current snap is 91 2024-09-10T16:12:35.463 INFO:tasks.rados.rados.0.smithi165.stdout:811: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:12:35.463 INFO:tasks.rados.rados.0.smithi165.stdout:813: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-10T16:12:35.468 INFO:tasks.rados.rados.0.smithi165.stdout:813: writing smithi16534841-25 from 443784 to 1091945 tid 1 2024-09-10T16:12:35.475 INFO:tasks.rados.rados.0.smithi165.stdout:813: writing smithi16534841-25 from 1657676 to 2317451 tid 2 2024-09-10T16:12:35.481 INFO:tasks.rados.rados.0.smithi165.stdout:813: writing smithi16534841-25 from 2818209 to 3243784 tid 3 2024-09-10T16:12:35.482 INFO:tasks.rados.rados.0.smithi165.stdout:814: delete oid 31 current snap is 91 2024-09-10T16:12:35.482 INFO:tasks.rados.rados.0.smithi165.stdout:811: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:12:35.482 INFO:tasks.rados.rados.0.smithi165.stdout:811: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:12:35.482 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 545 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:12:35.482 INFO:tasks.rados.rados.0.smithi165.stdout:813: finishing write tid 1 to smithi16534841-25 2024-09-10T16:12:35.488 INFO:tasks.rados.rados.0.smithi165.stdout:813: finishing write tid 2 to smithi16534841-25 2024-09-10T16:12:35.489 INFO:tasks.rados.rados.0.smithi165.stdout:813: finishing write tid 3 to smithi16534841-25 2024-09-10T16:12:35.489 INFO:tasks.rados.rados.0.smithi165.stdout:813: finishing write tid 4 to smithi16534841-25 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: finishing write tid 5 to smithi16534841-25 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: oid 25 updating version 0 to 547 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: oid 25 updating version 547 to 554 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: oid 25 version 554 is already newer than 552 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: oid 25 version 554 is already newer than 553 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 554 (ObjNum 312 snap 91 seq_num 312) dirty exists 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: left oid 25 (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:811: done (2 left) 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:813: done (1 left) 2024-09-10T16:12:35.491 INFO:tasks.rados.rados.0.smithi165.stdout:814: done (0 left) 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:815: read oid 9 snap -1 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:815: expect deleted 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:816: read oid 12 snap -1 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:816: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:817: write oid 8 current snap is 91 2024-09-10T16:12:35.492 INFO:tasks.rados.rados.0.smithi165.stdout:817: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-10T16:12:35.497 INFO:tasks.rados.rados.0.smithi165.stdout:817: writing smithi16534841-8 from 477398 to 1276057 tid 1 2024-09-10T16:12:35.503 INFO:tasks.rados.rados.0.smithi165.stdout:817: writing smithi16534841-8 from 1774640 to 2438084 tid 2 2024-09-10T16:12:35.508 INFO:tasks.rados.rados.0.smithi165.stdout:817: writing smithi16534841-8 from 2873055 to 3277398 tid 3 2024-09-10T16:12:35.508 INFO:tasks.rados.rados.0.smithi165.stdout:818: write oid 31 current snap is 91 2024-09-10T16:12:35.508 INFO:tasks.rados.rados.0.smithi165.stdout:818: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-10T16:12:35.513 INFO:tasks.rados.rados.0.smithi165.stdout:818: writing smithi16534841-31 from 494205 to 968113 tid 1 2024-09-10T16:12:35.518 INFO:tasks.rados.rados.0.smithi165.stdout:818: writing smithi16534841-31 from 1433122 to 1840224 tid 2 2024-09-10T16:12:35.524 INFO:tasks.rados.rados.0.smithi165.stdout:818: writing smithi16534841-31 from 2384125 to 2846809 tid 3 2024-09-10T16:12:35.524 INFO:tasks.rados.rados.0.smithi165.stdout:819: rollback oid 6 current snap is 91 2024-09-10T16:12:35.524 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 85 2024-09-10T16:12:35.524 INFO:tasks.rados.rados.0.smithi165.stdout:816: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:12:35.528 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 790183 2024-09-10T16:12:35.542 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: finishing write tid 1 to smithi16534841-8 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: finishing write tid 2 to smithi16534841-8 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:815: done (4 left) 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:816: done (3 left) 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:820: delete oid 3 current snap is 91 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: finishing write tid 3 to smithi16534841-8 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: finishing write tid 4 to smithi16534841-8 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: finishing write tid 5 to smithi16534841-8 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: oid 8 updating version 0 to 583 2024-09-10T16:12:35.588 INFO:tasks.rados.rados.0.smithi165.stdout:817: oid 8 updating version 583 to 584 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:817: oid 8 version 584 is already newer than 582 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:817: oid 8 updating version 584 to 585 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 585 (ObjNum 314 snap 91 seq_num 314) dirty exists 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:817: left oid 8 (ObjNum 314 snap 91 seq_num 314) 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:818: finishing write tid 1 to smithi16534841-31 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:818: finishing write tid 2 to smithi16534841-31 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:819: finishing rollback tid 0 to smithi16534841-6 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:818: finishing write tid 3 to smithi16534841-31 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:819: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:819: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 305 (ObjNum 172 snap 52 seq_num 172) dirty exists 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:818: finishing write tid 4 to smithi16534841-31 2024-09-10T16:12:35.589 INFO:tasks.rados.rados.0.smithi165.stdout:818: finishing write tid 5 to smithi16534841-31 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:818: oid 31 updating version 0 to 752 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:818: oid 31 version 752 is already newer than 750 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:818: oid 31 updating version 752 to 753 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:818: oid 31 version 753 is already newer than 751 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 753 (ObjNum 315 snap 91 seq_num 315) dirty exists 2024-09-10T16:12:35.590 INFO:tasks.rados.rados.0.smithi165.stdout:818: left oid 31 (ObjNum 315 snap 91 seq_num 315) 2024-09-10T16:12:35.592 INFO:tasks.rados.rados.0.smithi165.stdout:817: done (3 left) 2024-09-10T16:12:35.593 INFO:tasks.rados.rados.0.smithi165.stdout:818: done (2 left) 2024-09-10T16:12:35.593 INFO:tasks.rados.rados.0.smithi165.stdout:819: done (1 left) 2024-09-10T16:12:35.593 INFO:tasks.rados.rados.0.smithi165.stdout:820: done (0 left) 2024-09-10T16:12:35.593 INFO:tasks.rados.rados.0.smithi165.stdout:821: snap_remove snap 85 2024-09-10T16:12:36.591 INFO:tasks.rados.rados.0.smithi165.stdout:821: done (0 left) 2024-09-10T16:12:36.591 INFO:tasks.rados.rados.0.smithi165.stdout:822: delete oid 3 current snap is 91 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:822: done (0 left) 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:823: read oid 23 snap -1 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:823: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:824: rollback oid 29 current snap is 91 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 88 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:825: read oid 22 snap 88 2024-09-10T16:12:36.596 INFO:tasks.rados.rados.0.smithi165.stdout:825: expect (ObjNum 292 snap 86 seq_num 292) 2024-09-10T16:12:36.597 INFO:tasks.rados.rados.0.smithi165.stdout:826: snap_remove snap 89 2024-09-10T16:12:37.596 INFO:tasks.rados.rados.0.smithi165.stdout:826: done (3 left) 2024-09-10T16:12:37.596 INFO:tasks.rados.rados.0.smithi165.stdout:827: snap_create 2024-09-10T16:12:37.597 INFO:tasks.rados.rados.0.smithi165.stdout:825: expect (ObjNum 292 snap 86 seq_num 292) 2024-09-10T16:12:37.606 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2639358 2024-09-10T16:12:37.677 INFO:tasks.rados.rados.0.smithi165.stdout:823: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:12:37.811 INFO:tasks.rados.rados.0.smithi165.stdout:824: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:12:37.811 INFO:tasks.rados.rados.0.smithi165.stdout:824: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:12:37.811 INFO:tasks.rados.rados.0.smithi165.stdout:824: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:12:37.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 589 (ObjNum 250 snap 76 seq_num 250) dirty exists 2024-09-10T16:12:38.598 INFO:tasks.rados.rados.0.smithi165.stdout:823: done (3 left) 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:824: done (2 left) 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:825: done (1 left) 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:827: done (0 left) 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:828: read oid 47 snap -1 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:828: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:829: write oid 10 current snap is 92 2024-09-10T16:12:38.599 INFO:tasks.rados.rados.0.smithi165.stdout:829: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-10T16:12:38.605 INFO:tasks.rados.rados.0.smithi165.stdout:829: writing smithi16534841-10 from 544626 to 1244281 tid 1 2024-09-10T16:12:38.612 INFO:tasks.rados.rados.0.smithi165.stdout:829: writing smithi16534841-10 from 1892215 to 2446644 tid 2 2024-09-10T16:12:38.617 INFO:tasks.rados.rados.0.smithi165.stdout:829: writing smithi16534841-10 from 3150041 to 3344626 tid 3 2024-09-10T16:12:38.617 INFO:tasks.rados.rados.0.smithi165.stdout:830: read oid 45 snap -1 2024-09-10T16:12:38.617 INFO:tasks.rados.rados.0.smithi165.stdout:830: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:831: read oid 30 snap -1 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:831: expect (ObjNum 118 snap 34 seq_num 118) 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:832: read oid 5 snap -1 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:832: expect (ObjNum 291 snap 84 seq_num 291) 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:833: write oid 50 current snap is 92 2024-09-10T16:12:38.618 INFO:tasks.rados.rados.0.smithi165.stdout:833: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2024-09-10T16:12:38.625 INFO:tasks.rados.rados.0.smithi165.stdout:833: writing smithi16534841-50 from 561433 to 1336337 tid 1 2024-09-10T16:12:38.632 INFO:tasks.rados.rados.0.smithi165.stdout:833: writing smithi16534841-50 from 1950697 to 2365137 tid 2 2024-09-10T16:12:38.638 INFO:tasks.rados.rados.0.smithi165.stdout:833: writing smithi16534841-50 from 2777464 to 3361433 tid 3 2024-09-10T16:12:38.638 INFO:tasks.rados.rados.0.smithi165.stdout:834: rollback oid 33 current snap is 92 2024-09-10T16:12:38.638 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 90 2024-09-10T16:12:38.638 INFO:tasks.rados.rados.0.smithi165.stdout:828: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:12:38.638 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:38.733 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2183144 2024-09-10T16:12:38.733 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: finishing write tid 1 to smithi16534841-10 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: finishing write tid 2 to smithi16534841-10 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: finishing write tid 3 to smithi16534841-10 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: finishing write tid 4 to smithi16534841-10 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: finishing write tid 5 to smithi16534841-10 2024-09-10T16:12:38.787 INFO:tasks.rados.rados.0.smithi165.stdout:829: oid 10 updating version 0 to 299 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:829: oid 10 version 299 is already newer than 296 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:829: oid 10 version 299 is already newer than 297 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:829: oid 10 version 299 is already newer than 298 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 299 (ObjNum 318 snap 92 seq_num 318) dirty exists 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:829: left oid 10 (ObjNum 318 snap 92 seq_num 318) 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:830: expect (ObjNum 153 snap 44 seq_num 153) 2024-09-10T16:12:38.788 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:38.846 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:38.964 INFO:tasks.rados.rados.0.smithi165.stdout:832: expect (ObjNum 291 snap 84 seq_num 291) 2024-09-10T16:12:39.009 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:39.054 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:39.099 INFO:tasks.rados.rados.0.smithi165.stdout:831: expect (ObjNum 118 snap 34 seq_num 118) 2024-09-10T16:12:39.105 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 826054 2024-09-10T16:12:39.222 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1268461 2024-09-10T16:12:39.267 INFO:tasks.rados.rados.0.smithi165.stdout:833: finishing write tid 1 to smithi16534841-50 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:834: finishing rollback tid 0 to smithi16534841-33 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:834: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: finishing write tid 2 to smithi16534841-50 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: finishing write tid 3 to smithi16534841-50 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: finishing write tid 4 to smithi16534841-50 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: finishing write tid 5 to smithi16534841-50 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: oid 50 updating version 0 to 337 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: oid 50 updating version 337 to 338 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: oid 50 version 338 is already newer than 336 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:833: oid 50 updating version 338 to 339 2024-09-10T16:12:39.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 339 (ObjNum 319 snap 92 seq_num 319) dirty exists 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:833: left oid 50 (ObjNum 319 snap 92 seq_num 319) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:834: finishing rollback tid 2 to smithi16534841-33 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 573 (ObjNum 263 snap 76 seq_num 263) dirty exists 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:828: done (6 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:829: done (5 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:830: done (4 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:831: done (3 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:832: done (2 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:833: done (1 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:834: done (0 left) 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:835: read oid 18 snap -1 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:835: expect deleted 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:836: rollback oid 50 current snap is 92 2024-09-10T16:12:39.269 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 90 2024-09-10T16:12:39.270 INFO:tasks.rados.rados.0.smithi165.stdout:837: snap_create 2024-09-10T16:12:39.270 INFO:tasks.rados.rados.0.smithi165.stdout:836: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:12:39.270 INFO:tasks.rados.rados.0.smithi165.stdout:836: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:12:39.270 INFO:tasks.rados.rados.0.smithi165.stdout:836: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:12:39.270 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 342 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:12:39.606 INFO:tasks.rados.rados.0.smithi165.stdout:835: done (2 left) 2024-09-10T16:12:39.607 INFO:tasks.rados.rados.0.smithi165.stdout:836: done (1 left) 2024-09-10T16:12:39.607 INFO:tasks.rados.rados.0.smithi165.stdout:837: done (0 left) 2024-09-10T16:12:39.607 INFO:tasks.rados.rados.0.smithi165.stdout:838: rollback oid 8 current snap is 93 2024-09-10T16:12:39.607 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 91 2024-09-10T16:12:39.607 INFO:tasks.rados.rados.0.smithi165.stdout:839: snap_remove snap 92 2024-09-10T16:12:40.612 INFO:tasks.rados.rados.0.smithi165.stdout:839: done (1 left) 2024-09-10T16:12:40.612 INFO:tasks.rados.rados.0.smithi165.stdout:840: write oid 25 current snap is 93 2024-09-10T16:12:40.612 INFO:tasks.rados.rados.0.smithi165.stdout:838: finishing rollback tid 0 to smithi16534841-8 2024-09-10T16:12:40.612 INFO:tasks.rados.rados.0.smithi165.stdout:840: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-10T16:12:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:840: writing smithi16534841-25 from 578240 to 1144746 tid 1 2024-09-10T16:12:40.624 INFO:tasks.rados.rados.0.smithi165.stdout:840: writing smithi16534841-25 from 1609179 to 2167277 tid 2 2024-09-10T16:12:40.631 INFO:tasks.rados.rados.0.smithi165.stdout:840: writing smithi16534841-25 from 2804887 to 3356637 tid 3 2024-09-10T16:12:40.631 INFO:tasks.rados.rados.0.smithi165.stdout:841: snap_remove snap 87 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:841: done (2 left) 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:842: rollback oid 22 current snap is 93 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 91 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:838: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:838: finishing rollback tid 2 to smithi16534841-8 2024-09-10T16:12:41.622 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 599 (ObjNum 314 snap 91 seq_num 314) dirty exists 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: finishing write tid 1 to smithi16534841-25 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: finishing write tid 2 to smithi16534841-25 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: finishing write tid 3 to smithi16534841-25 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: finishing write tid 4 to smithi16534841-25 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: finishing write tid 5 to smithi16534841-25 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: oid 25 updating version 0 to 578 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: oid 25 version 578 is already newer than 576 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: oid 25 version 578 is already newer than 575 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: oid 25 version 578 is already newer than 577 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 578 (ObjNum 320 snap 93 seq_num 320) dirty exists 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: left oid 25 (ObjNum 320 snap 93 seq_num 320) 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:838: done (2 left) 2024-09-10T16:12:41.623 INFO:tasks.rados.rados.0.smithi165.stdout:840: done (1 left) 2024-09-10T16:12:41.624 INFO:tasks.rados.rados.0.smithi165.stdout:843: read oid 11 snap 90 2024-09-10T16:12:41.624 INFO:tasks.rados.rados.0.smithi165.stdout:843: expect (ObjNum 310 snap 89 seq_num 310) 2024-09-10T16:12:41.624 INFO:tasks.rados.rados.0.smithi165.stdout:844: snap_remove snap 88 2024-09-10T16:12:42.626 INFO:tasks.rados.rados.0.smithi165.stdout:844: done (2 left) 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:845: snap_create 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:842: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:842: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:842: finishing rollback tid 2 to smithi16534841-22 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 282 (ObjNum 292 snap 86 seq_num 292) dirty exists 2024-09-10T16:12:42.627 INFO:tasks.rados.rados.0.smithi165.stdout:843: expect (ObjNum 310 snap 89 seq_num 310) 2024-09-10T16:12:42.666 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:42.714 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:43.632 INFO:tasks.rados.rados.0.smithi165.stdout:842: done (2 left) 2024-09-10T16:12:43.632 INFO:tasks.rados.rados.0.smithi165.stdout:843: done (1 left) 2024-09-10T16:12:43.632 INFO:tasks.rados.rados.0.smithi165.stdout:845: done (0 left) 2024-09-10T16:12:43.633 INFO:tasks.rados.rados.0.smithi165.stdout:846: rollback oid 17 current snap is 94 2024-09-10T16:12:43.633 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 84 2024-09-10T16:12:43.633 INFO:tasks.rados.rados.0.smithi165.stdout:847: write oid 7 current snap is 94 2024-09-10T16:12:43.633 INFO:tasks.rados.rados.0.smithi165.stdout:847: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-10T16:12:43.639 INFO:tasks.rados.rados.0.smithi165.stdout:847: writing smithi16534841-7 from 595047 to 1236802 tid 1 2024-09-10T16:12:43.643 INFO:tasks.rados.rados.0.smithi165.stdout:847: writing smithi16534841-7 from 1667661 to 2085770 tid 2 2024-09-10T16:12:43.650 INFO:tasks.rados.rados.0.smithi165.stdout:847: writing smithi16534841-7 from 2548663 to 3311685 tid 3 2024-09-10T16:12:43.651 INFO:tasks.rados.rados.0.smithi165.stdout:848: write (excl) oid 27 current snap is 94 2024-09-10T16:12:43.651 INFO:tasks.rados.rados.0.smithi165.stdout:846: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:12:43.651 INFO:tasks.rados.rados.0.smithi165.stdout:848: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-10T16:12:43.658 INFO:tasks.rados.rados.0.smithi165.stdout:848: writing smithi16534841-27 from 611854 to 1328858 tid 1 2024-09-10T16:12:43.665 INFO:tasks.rados.rados.0.smithi165.stdout:848: writing smithi16534841-27 from 2126143 to 2687910 tid 2 2024-09-10T16:12:43.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: writing smithi16534841-27 from 3259733 to 3411854 tid 3 2024-09-10T16:12:43.669 INFO:tasks.rados.rados.0.smithi165.stdout:849: write oid 18 current snap is 94 2024-09-10T16:12:43.669 INFO:tasks.rados.rados.0.smithi165.stdout:849: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-10T16:12:43.677 INFO:tasks.rados.rados.0.smithi165.stdout:849: writing smithi16534841-18 from 628661 to 1420914 tid 1 2024-09-10T16:12:43.683 INFO:tasks.rados.rados.0.smithi165.stdout:849: writing smithi16534841-18 from 2184625 to 2606403 tid 2 2024-09-10T16:12:43.688 INFO:tasks.rados.rados.0.smithi165.stdout:849: writing smithi16534841-18 from 3403509 to 3428661 tid 3 2024-09-10T16:12:43.688 INFO:tasks.rados.rados.0.smithi165.stdout:850: snap_remove snap 90 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:850: done (4 left) 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:851: write oid 10 current snap is 94 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:846: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 303 (ObjNum 213 snap 64 seq_num 213) dirty exists 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:847: finishing write tid 1 to smithi16534841-7 2024-09-10T16:12:44.636 INFO:tasks.rados.rados.0.smithi165.stdout:851: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-10T16:12:44.642 INFO:tasks.rados.rados.0.smithi165.stdout:851: writing smithi16534841-10 from 645468 to 1112970 tid 1 2024-09-10T16:12:44.649 INFO:tasks.rados.rados.0.smithi165.stdout:851: writing smithi16534841-10 from 1726754 to 2292190 tid 2 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:851: writing smithi16534841-10 from 2798226 to 3395064 tid 3 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:846: done (4 left) 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:852: read oid 42 snap -1 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:852: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:853: write oid 37 current snap is 94 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:847: finishing write tid 2 to smithi16534841-7 2024-09-10T16:12:44.656 INFO:tasks.rados.rados.0.smithi165.stdout:853: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2024-09-10T16:12:44.662 INFO:tasks.rados.rados.0.smithi165.stdout:853: writing smithi16534841-37 from 662275 to 1205026 tid 1 2024-09-10T16:12:44.667 INFO:tasks.rados.rados.0.smithi165.stdout:853: writing smithi16534841-37 from 1785236 to 2210683 tid 2 2024-09-10T16:12:44.674 INFO:tasks.rados.rados.0.smithi165.stdout:853: writing smithi16534841-37 from 2942002 to 3462275 tid 3 2024-09-10T16:12:44.674 INFO:tasks.rados.rados.0.smithi165.stdout:854: snap_remove snap 93 2024-09-10T16:12:45.591 INFO:tasks.rados.rados.0.smithi165.stdout:854: done (6 left) 2024-09-10T16:12:45.667 INFO:tasks.rados.rados.0.smithi165.stdout:855: rollback oid 41 current snap is 94 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 84 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: finishing write tid 3 to smithi16534841-7 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: finishing write tid 4 to smithi16534841-7 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: finishing write tid 5 to smithi16534841-7 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: oid 7 updating version 0 to 778 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: oid 7 version 778 is already newer than 777 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: oid 7 updating version 778 to 779 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: oid 7 updating version 779 to 780 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 780 (ObjNum 321 snap 94 seq_num 321) dirty exists 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:847: left oid 7 (ObjNum 321 snap 94 seq_num 321) 2024-09-10T16:12:45.668 INFO:tasks.rados.rados.0.smithi165.stdout:848: finishing write tid 1 to smithi16534841-27 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: finishing write tid 2 to smithi16534841-27 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: finishing write tid 3 to smithi16534841-27 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: finishing write tid 4 to smithi16534841-27 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: finishing write tid 5 to smithi16534841-27 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: oid 27 updating version 0 to 607 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: oid 27 updating version 607 to 608 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: oid 27 updating version 608 to 609 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: oid 27 version 609 is already newer than 606 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 609 (ObjNum 322 snap 94 seq_num 322) dirty exists 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:848: left oid 27 (ObjNum 322 snap 94 seq_num 322) 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:849: finishing write tid 1 to smithi16534841-18 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:849: finishing write tid 2 to smithi16534841-18 2024-09-10T16:12:45.669 INFO:tasks.rados.rados.0.smithi165.stdout:849: finishing write tid 3 to smithi16534841-18 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: finishing write tid 4 to smithi16534841-18 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: finishing write tid 5 to smithi16534841-18 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: oid 18 updating version 0 to 500 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: oid 18 updating version 500 to 503 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: oid 18 version 503 is already newer than 502 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: oid 18 version 503 is already newer than 501 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 503 (ObjNum 323 snap 94 seq_num 323) dirty exists 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:849: left oid 18 (ObjNum 323 snap 94 seq_num 323) 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:851: finishing write tid 1 to smithi16534841-10 2024-09-10T16:12:45.670 INFO:tasks.rados.rados.0.smithi165.stdout:852: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:12:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:851: finishing write tid 2 to smithi16534841-10 2024-09-10T16:12:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:851: finishing write tid 3 to smithi16534841-10 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: finishing write tid 4 to smithi16534841-10 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: finishing write tid 5 to smithi16534841-10 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: oid 10 updating version 0 to 308 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: oid 10 version 308 is already newer than 307 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: oid 10 version 308 is already newer than 305 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: oid 10 version 308 is already newer than 306 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 308 (ObjNum 324 snap 94 seq_num 324) dirty exists 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: left oid 10 (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:853: finishing write tid 1 to smithi16534841-37 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:847: done (6 left) 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:848: done (5 left) 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:849: done (4 left) 2024-09-10T16:12:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:851: done (3 left) 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:852: done (2 left) 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:856: copy_from oid 36 from oid 3 current snap is 94 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: finishing write tid 2 to smithi16534841-37 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: finishing write tid 3 to smithi16534841-37 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: finishing write tid 4 to smithi16534841-37 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: finishing write tid 5 to smithi16534841-37 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: oid 37 updating version 0 to 549 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: oid 37 version 549 is already newer than 547 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: oid 37 version 549 is already newer than 546 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:853: oid 37 version 549 is already newer than 548 2024-09-10T16:12:45.699 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 549 (ObjNum 325 snap 94 seq_num 325) dirty exists 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:853: left oid 37 (ObjNum 325 snap 94 seq_num 325) 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:855: finishing rollback tid 0 to smithi16534841-41 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:855: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:855: finishing rollback tid 2 to smithi16534841-41 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 507 (ObjNum 205 snap 61 seq_num 205) dirty exists 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:853: done (2 left) 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:855: done (1 left) 2024-09-10T16:12:45.700 INFO:tasks.rados.rados.0.smithi165.stdout:857: snap_remove snap 84 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:857: done (1 left) 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:858: rollback oid 25 current snap is 94 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 91 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:856: finishing copy_from to smithi16534841-36 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:856: got expected ENOENT (src dne) 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:856: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:12:46.653 INFO:tasks.rados.rados.0.smithi165.stdout:856: done (1 left) 2024-09-10T16:12:46.654 INFO:tasks.rados.rados.0.smithi165.stdout:859: read oid 26 snap -1 2024-09-10T16:12:46.654 INFO:tasks.rados.rados.0.smithi165.stdout:859: expect (ObjNum 305 snap 88 seq_num 305) 2024-09-10T16:12:46.654 INFO:tasks.rados.rados.0.smithi165.stdout:860: copy_from oid 7 from oid 50 current snap is 94 2024-09-10T16:12:46.654 INFO:tasks.rados.rados.0.smithi165.stdout:861: delete oid 33 current snap is 94 2024-09-10T16:12:46.660 INFO:tasks.rados.rados.0.smithi165.stdout:858: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:12:46.664 INFO:tasks.rados.rados.0.smithi165.stdout:859: expect (ObjNum 305 snap 88 seq_num 305) 2024-09-10T16:12:46.664 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:46.721 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:46.801 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2502536 2024-09-10T16:12:46.816 INFO:tasks.rados.rados.0.smithi165.stdout:858: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:12:46.816 INFO:tasks.rados.rados.0.smithi165.stdout:858: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:12:46.816 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 601 (ObjNum 312 snap 91 seq_num 312) dirty exists 2024-09-10T16:12:46.816 INFO:tasks.rados.rados.0.smithi165.stdout:860: finishing copy_from racing read to smithi16534841-7 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:860: finishing copy_from to smithi16534841-7 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 342 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:858: done (3 left) 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:859: done (2 left) 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:860: done (1 left) 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:861: done (0 left) 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:862: copy_from oid 27 from oid 32 current snap is 94 2024-09-10T16:12:46.817 INFO:tasks.rados.rados.0.smithi165.stdout:863: snap_create 2024-09-10T16:12:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:862: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:12:46.828 INFO:tasks.rados.rados.0.smithi165.stdout:862: finishing copy_from to smithi16534841-27 2024-09-10T16:12:46.829 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 192 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:12:47.656 INFO:tasks.rados.rados.0.smithi165.stdout:862: done (1 left) 2024-09-10T16:12:47.656 INFO:tasks.rados.rados.0.smithi165.stdout:863: done (0 left) 2024-09-10T16:12:47.656 INFO:tasks.rados.rados.0.smithi165.stdout:864: rollback oid 20 current snap is 95 2024-09-10T16:12:47.656 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 91 2024-09-10T16:12:47.656 INFO:tasks.rados.rados.0.smithi165.stdout:865: delete oid 13 current snap is 95 2024-09-10T16:12:47.660 INFO:tasks.rados.rados.0.smithi165.stdout:865: done (1 left) 2024-09-10T16:12:47.660 INFO:tasks.rados.rados.0.smithi165.stdout:866: write (excl) oid 16 current snap is 95 2024-09-10T16:12:47.661 INFO:tasks.rados.rados.0.smithi165.stdout:866: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-10T16:12:47.667 INFO:tasks.rados.rados.0.smithi165.stdout:866: writing smithi16534841-16 from 712696 to 1197547 tid 1 2024-09-10T16:12:47.675 INFO:tasks.rados.rados.0.smithi165.stdout:866: writing smithi16534841-16 from 1960682 to 2533456 tid 2 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:866: writing smithi16534841-16 from 3307918 to 3512696 tid 3 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:867: read oid 25 snap -1 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:867: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:868: snap_create 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:864: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:864: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:12:47.681 INFO:tasks.rados.rados.0.smithi165.stdout:864: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:12:47.682 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 614 (ObjNum 70 snap 10 seq_num 70) dirty exists 2024-09-10T16:12:47.682 INFO:tasks.rados.rados.0.smithi165.stdout:866: finishing write tid 1 to smithi16534841-16 2024-09-10T16:12:47.683 INFO:tasks.rados.rados.0.smithi165.stdout:866: finishing write tid 2 to smithi16534841-16 2024-09-10T16:12:47.683 INFO:tasks.rados.rados.0.smithi165.stdout:866: finishing write tid 3 to smithi16534841-16 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: finishing write tid 4 to smithi16534841-16 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: finishing write tid 5 to smithi16534841-16 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: oid 16 updating version 0 to 799 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: oid 16 updating version 799 to 800 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: oid 16 updating version 800 to 802 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: oid 16 version 802 is already newer than 801 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 802 (ObjNum 328 snap 95 seq_num 328) dirty exists 2024-09-10T16:12:47.684 INFO:tasks.rados.rados.0.smithi165.stdout:866: left oid 16 (ObjNum 328 snap 95 seq_num 328) 2024-09-10T16:12:47.689 INFO:tasks.rados.rados.0.smithi165.stdout:867: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:12:47.837 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:48.658 INFO:tasks.rados.rados.0.smithi165.stdout:864: done (3 left) 2024-09-10T16:12:48.658 INFO:tasks.rados.rados.0.smithi165.stdout:866: done (2 left) 2024-09-10T16:12:48.658 INFO:tasks.rados.rados.0.smithi165.stdout:867: done (1 left) 2024-09-10T16:12:48.658 INFO:tasks.rados.rados.0.smithi165.stdout:868: done (0 left) 2024-09-10T16:12:48.658 INFO:tasks.rados.rados.0.smithi165.stdout:869: snap_remove snap 95 2024-09-10T16:12:49.663 INFO:tasks.rados.rados.0.smithi165.stdout:869: done (0 left) 2024-09-10T16:12:49.663 INFO:tasks.rados.rados.0.smithi165.stdout:870: copy_from oid 44 from oid 27 current snap is 96 2024-09-10T16:12:49.663 INFO:tasks.rados.rados.0.smithi165.stdout:871: write oid 3 current snap is 96 2024-09-10T16:12:49.663 INFO:tasks.rados.rados.0.smithi165.stdout:871: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-10T16:12:49.670 INFO:tasks.rados.rados.0.smithi165.stdout:871: writing smithi16534841-3 from 729503 to 1289603 tid 1 2024-09-10T16:12:49.678 INFO:tasks.rados.rados.0.smithi165.stdout:871: writing smithi16534841-3 from 2019164 to 2735596 tid 2 2024-09-10T16:12:49.683 INFO:tasks.rados.rados.0.smithi165.stdout:871: writing smithi16534841-3 from 3335341 to 3529503 tid 3 2024-09-10T16:12:49.683 INFO:tasks.rados.rados.0.smithi165.stdout:871: finishing write tid 1 to smithi16534841-3 2024-09-10T16:12:49.683 INFO:tasks.rados.rados.0.smithi165.stdout:872: delete oid 49 current snap is 96 2024-09-10T16:12:49.685 INFO:tasks.rados.rados.0.smithi165.stdout:871: finishing write tid 2 to smithi16534841-3 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: finishing write tid 3 to smithi16534841-3 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: finishing write tid 4 to smithi16534841-3 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: finishing write tid 5 to smithi16534841-3 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: oid 3 updating version 0 to 616 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: oid 3 version 616 is already newer than 615 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: oid 3 updating version 616 to 617 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:871: oid 3 updating version 617 to 618 2024-09-10T16:12:49.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 618 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-09-10T16:12:49.688 INFO:tasks.rados.rados.0.smithi165.stdout:871: left oid 3 (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:12:49.694 INFO:tasks.rados.rados.0.smithi165.stdout:870: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:12:49.696 INFO:tasks.rados.rados.0.smithi165.stdout:871: done (2 left) 2024-09-10T16:12:49.696 INFO:tasks.rados.rados.0.smithi165.stdout:872: done (1 left) 2024-09-10T16:12:49.696 INFO:tasks.rados.rados.0.smithi165.stdout:873: read oid 10 snap -1 2024-09-10T16:12:49.696 INFO:tasks.rados.rados.0.smithi165.stdout:873: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:12:49.696 INFO:tasks.rados.rados.0.smithi165.stdout:874: snap_create 2024-09-10T16:12:49.698 INFO:tasks.rados.rados.0.smithi165.stdout:870: finishing copy_from to smithi16534841-44 2024-09-10T16:12:49.698 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 192 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:12:49.709 INFO:tasks.rados.rados.0.smithi165.stdout:873: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:12:49.709 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:12:50.590 INFO:tasks.rados.rados.0.smithi165.stdout:870: done (2 left) 2024-09-10T16:12:50.591 INFO:tasks.rados.rados.0.smithi165.stdout:873: done (1 left) 2024-09-10T16:12:50.591 INFO:tasks.rados.rados.0.smithi165.stdout:874: done (0 left) 2024-09-10T16:12:50.591 INFO:tasks.rados.rados.0.smithi165.stdout:875: write (excl) oid 5 current snap is 97 2024-09-10T16:12:50.591 INFO:tasks.rados.rados.0.smithi165.stdout:875: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2024-09-10T16:12:50.598 INFO:tasks.rados.rados.0.smithi165.stdout:875: writing smithi16534841-5 from 763117 to 1473715 tid 1 2024-09-10T16:12:50.606 INFO:tasks.rados.rados.0.smithi165.stdout:875: writing smithi16534841-5 from 2136128 to 2856229 tid 2 2024-09-10T16:12:50.610 INFO:tasks.rados.rados.0.smithi165.stdout:875: writing smithi16534841-5 from 3390187 to 3563117 tid 3 2024-09-10T16:12:50.610 INFO:tasks.rados.rados.0.smithi165.stdout:876: delete oid 11 current snap is 97 2024-09-10T16:12:50.617 INFO:tasks.rados.rados.0.smithi165.stdout:875: finishing write tid 1 to smithi16534841-5 2024-09-10T16:12:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:875: finishing write tid 2 to smithi16534841-5 2024-09-10T16:12:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:875: finishing write tid 3 to smithi16534841-5 2024-09-10T16:12:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:875: finishing write tid 4 to smithi16534841-5 2024-09-10T16:12:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:875: finishing write tid 5 to smithi16534841-5 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: oid 5 updating version 0 to 321 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: oid 5 version 321 is already newer than 320 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: oid 5 updating version 321 to 322 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: oid 5 version 322 is already newer than 319 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 322 (ObjNum 331 snap 97 seq_num 331) dirty exists 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: left oid 5 (ObjNum 331 snap 97 seq_num 331) 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:875: done (1 left) 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:876: done (0 left) 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:877: copy_from oid 26 from oid 23 current snap is 97 2024-09-10T16:12:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:878: delete oid 49 current snap is 97 2024-09-10T16:12:50.625 INFO:tasks.rados.rados.0.smithi165.stdout:878: done (1 left) 2024-09-10T16:12:50.625 INFO:tasks.rados.rados.0.smithi165.stdout:879: rollback oid 5 current snap is 97 2024-09-10T16:12:50.625 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 94 2024-09-10T16:12:50.625 INFO:tasks.rados.rados.0.smithi165.stdout:880: delete oid 24 current snap is 97 2024-09-10T16:12:50.630 INFO:tasks.rados.rados.0.smithi165.stdout:879: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:12:50.630 INFO:tasks.rados.rados.0.smithi165.stdout:879: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:12:50.630 INFO:tasks.rados.rados.0.smithi165.stdout:879: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:12:50.630 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 325 (ObjNum 291 snap 84 seq_num 291) dirty exists 2024-09-10T16:12:50.631 INFO:tasks.rados.rados.0.smithi165.stdout:879: done (2 left) 2024-09-10T16:12:50.631 INFO:tasks.rados.rados.0.smithi165.stdout:880: done (1 left) 2024-09-10T16:12:50.631 INFO:tasks.rados.rados.0.smithi165.stdout:881: snap_create 2024-09-10T16:12:50.633 INFO:tasks.rados.rados.0.smithi165.stdout:877: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:12:50.634 INFO:tasks.rados.rados.0.smithi165.stdout:877: finishing copy_from to smithi16534841-26 2024-09-10T16:12:50.634 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 441 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:12:51.598 INFO:tasks.rados.rados.0.smithi165.stdout:877: done (1 left) 2024-09-10T16:12:51.598 INFO:tasks.rados.rados.0.smithi165.stdout:881: done (0 left) 2024-09-10T16:12:51.598 INFO:tasks.rados.rados.0.smithi165.stdout:882: snap_remove snap 91 2024-09-10T16:12:52.043 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:12:52.043 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:12:52.044 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:12:52.046 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:12:52.046 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-10T16:12:52.047 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-10T16:12:52.047 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:13:07.048 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-10T16:13:07.048 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-10T16:13:07.049 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:13:07.049 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.156+0000 7f250dd24640 10 monclient: get_monmap_and_config 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.156+0000 7f250dd24640 10 monclient: build_initial_monmap 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.156+0000 7f250dd24640 10 monclient: monmap: 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:07.157716+0000 2024-09-10T16:13:07.158 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:07.157716+0000 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:07.159 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.157+0000 7f250dd24640 1 Processor -- start 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.157+0000 7f250dd24640 1 -- start start 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.157+0000 7f250dd24640 10 monclient: init 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f250809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f250809d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:07.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-b con 0x7f250813b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250809fbc0 con 0x7f250809d190 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25080a2a20 con 0x7f250809c310 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f250809f860 con 0x7f250813b5d0 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f250dd24640 10 monclient(hunting): authenticate will time out at 1165.645203s 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2506ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f250809d190 0x7f250809f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:37304/0 (socket says 172.21.15.164:37304) 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 learned_addr learned my addr 172.21.15.164:0/992567144 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f250813b5d0 auth_method 0 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:07.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.158+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.160+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f250809d190 legacy=0x7f250809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:07.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.160+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.165:6789/0 conn(0x7f250809c310 legacy=0x7f250809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:07.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f250813b5d0 auth_method 0 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:07.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.360+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:07.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f250813b5d0 auth_method 0 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:07.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:07.762+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f250813b5d0 auth_method 0 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:08.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:08.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:08.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:08.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:08.563+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): tick 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 msgr2=0x7f250813d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f250813b5d0 0x7f250813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.165:6789/0 conn(0x7f250809c310 legacy=0x7f250809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f250809d190 legacy=0x7f250809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:10.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-a con 0x7f24f4005a20 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-c con 0x7f24f4008570 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f250809f860 con 0x7f24f4005a20 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f24f400b0d0 con 0x7f24f4008570 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25057fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005a20 auth_method 0 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008570 auth_method 0 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:10.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.158+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:10.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005a20 auth_method 0 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008570 auth_method 0 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:10.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:10.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.359+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005a20 auth_method 0 2024-09-10T16:13:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008570 auth_method 0 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:10.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:10.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:10.760+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:11.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:11.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005a20 auth_method 0 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008570 auth_method 0 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.561+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:11.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.562+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:11.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.562+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:11.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.562+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:11.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:11.562+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:882: done (0 left) 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:883: read oid 27 snap -1 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:883: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:884: copy_from oid 40 from oid 42 current snap is 98 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:885: write (excl) oid 18 current snap is 98 2024-09-10T16:13:12.059 INFO:tasks.rados.rados.0.smithi165.stdout:885: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-10T16:13:12.066 INFO:tasks.rados.rados.0.smithi165.stdout:885: writing smithi16534841-18 from 430345 to 1158292 tid 1 2024-09-10T16:13:12.074 INFO:tasks.rados.rados.0.smithi165.stdout:885: writing smithi16534841-18 from 1570056 to 2297495 tid 2 2024-09-10T16:13:12.079 INFO:tasks.rados.rados.0.smithi165.stdout:885: writing smithi16534841-18 from 2699879 to 3169768 tid 3 2024-09-10T16:13:12.079 INFO:tasks.rados.rados.0.smithi165.stdout:886: rollback oid 5 current snap is 98 2024-09-10T16:13:12.079 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 97 2024-09-10T16:13:12.079 INFO:tasks.rados.rados.0.smithi165.stdout:883: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:12.079 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:12.168 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 793256 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:884: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:885: finishing write tid 1 to smithi16534841-18 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:885: finishing write tid 2 to smithi16534841-18 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:884: finishing copy_from to smithi16534841-40 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 320 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:885: finishing write tid 3 to smithi16534841-18 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:883: done (3 left) 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:884: done (2 left) 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:887: write oid 48 current snap is 98 2024-09-10T16:13:12.207 INFO:tasks.rados.rados.0.smithi165.stdout:885: finishing write tid 4 to smithi16534841-18 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: finishing write tid 5 to smithi16534841-18 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: oid 18 updating version 0 to 518 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: oid 18 version 518 is already newer than 517 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: oid 18 version 518 is already newer than 515 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: oid 18 version 518 is already newer than 516 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 518 (ObjNum 335 snap 98 seq_num 335) dirty exists 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:885: left oid 18 (ObjNum 335 snap 98 seq_num 335) 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:886: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:886: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:13:12.208 INFO:tasks.rados.rados.0.smithi165.stdout:887: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-10T16:13:12.211 INFO:tasks.rados.rados.0.smithi165.stdout:887: writing smithi16534841-48 from 447152 to 850348 tid 1 2024-09-10T16:13:12.218 INFO:tasks.rados.rados.0.smithi165.stdout:887: writing smithi16534841-48 from 1512185 to 2099635 tid 2 2024-09-10T16:13:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:887: writing smithi16534841-48 from 2727302 to 3408463 tid 3 2024-09-10T16:13:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:885: done (2 left) 2024-09-10T16:13:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:888: write oid 26 current snap is 98 2024-09-10T16:13:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:886: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:13:12.226 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 331 (ObjNum 291 snap 84 seq_num 291) dirty exists 2024-09-10T16:13:12.226 INFO:tasks.rados.rados.0.smithi165.stdout:888: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2024-09-10T16:13:12.230 INFO:tasks.rados.rados.0.smithi165.stdout:888: writing smithi16534841-26 from 463959 to 942404 tid 1 2024-09-10T16:13:12.238 INFO:tasks.rados.rados.0.smithi165.stdout:888: writing smithi16534841-26 from 1570667 to 2301775 tid 2 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:888: writing smithi16534841-26 from 3038372 to 3530805 tid 3 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:886: done (2 left) 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:889: read oid 39 snap -1 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:889: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:887: finishing write tid 1 to smithi16534841-48 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:887: finishing write tid 2 to smithi16534841-48 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:890: read oid 29 snap -1 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:890: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:891: read oid 25 snap -1 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:891: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:892: read oid 35 snap 96 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:892: expect (ObjNum 301 snap 87 seq_num 301) 2024-09-10T16:13:12.246 INFO:tasks.rados.rados.0.smithi165.stdout:893: read oid 31 snap -1 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:893: expect (ObjNum 315 snap 91 seq_num 315) 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:894: copy_from oid 41 from oid 33 current snap is 98 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: finishing write tid 3 to smithi16534841-48 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: finishing write tid 4 to smithi16534841-48 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: finishing write tid 5 to smithi16534841-48 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: oid 48 updating version 0 to 815 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: oid 48 version 815 is already newer than 814 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: oid 48 version 815 is already newer than 813 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: oid 48 version 815 is already newer than 812 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 815 (ObjNum 336 snap 98 seq_num 336) dirty exists 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: left oid 48 (ObjNum 336 snap 98 seq_num 336) 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:888: finishing write tid 1 to smithi16534841-26 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:887: done (7 left) 2024-09-10T16:13:12.247 INFO:tasks.rados.rados.0.smithi165.stdout:895: write (excl) oid 30 current snap is 98 2024-09-10T16:13:12.248 INFO:tasks.rados.rados.0.smithi165.stdout:895: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-10T16:13:12.251 INFO:tasks.rados.rados.0.smithi165.stdout:895: writing smithi16534841-30 from 480766 to 1034460 tid 1 2024-09-10T16:13:12.259 INFO:tasks.rados.rados.0.smithi165.stdout:895: writing smithi16534841-30 from 1629149 to 2220268 tid 2 2024-09-10T16:13:12.268 INFO:tasks.rados.rados.0.smithi165.stdout:895: writing smithi16534841-30 from 2782148 to 3202206 tid 3 2024-09-10T16:13:12.269 INFO:tasks.rados.rados.0.smithi165.stdout:896: write (excl) oid 48 current snap is 98 2024-09-10T16:13:12.269 INFO:tasks.rados.rados.0.smithi165.stdout:888: finishing write tid 2 to smithi16534841-26 2024-09-10T16:13:12.269 INFO:tasks.rados.rados.0.smithi165.stdout:896: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2024-09-10T16:13:12.271 INFO:tasks.rados.rados.0.smithi165.stdout:896: writing smithi16534841-48 from 497573 to 1126516 tid 1 2024-09-10T16:13:12.278 INFO:tasks.rados.rados.0.smithi165.stdout:896: writing smithi16534841-48 from 1687631 to 2422408 tid 2 2024-09-10T16:13:12.286 INFO:tasks.rados.rados.0.smithi165.stdout:896: writing smithi16534841-48 from 3093218 to 3697573 tid 3 2024-09-10T16:13:12.286 INFO:tasks.rados.rados.0.smithi165.stdout:891: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:13:12.439 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:12.521 INFO:tasks.rados.rados.0.smithi165.stdout:890: expect (ObjNum 250 snap 76 seq_num 250) 2024-09-10T16:13:12.522 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 246644 2024-09-10T16:13:12.562 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1717949 2024-09-10T16:13:12.576 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:12.610 INFO:tasks.rados.rados.0.smithi165.stdout:889: expect (ObjNum 205 snap 61 seq_num 205) 2024-09-10T16:13:12.610 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:12.717 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:12.771 INFO:tasks.rados.rados.0.smithi165.stdout:894: finishing copy_from to smithi16534841-41 2024-09-10T16:13:12.771 INFO:tasks.rados.rados.0.smithi165.stdout:894: got expected ENOENT (src dne) 2024-09-10T16:13:12.772 INFO:tasks.rados.rados.0.smithi165.stdout:894: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:13:12.772 INFO:tasks.rados.rados.0.smithi165.stdout:892: expect (ObjNum 301 snap 87 seq_num 301) 2024-09-10T16:13:12.777 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 613471 2024-09-10T16:13:12.927 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:13.014 INFO:tasks.rados.rados.0.smithi165.stdout:888: finishing write tid 3 to smithi16534841-26 2024-09-10T16:13:13.014 INFO:tasks.rados.rados.0.smithi165.stdout:897: read oid 14 snap 94 2024-09-10T16:13:13.014 INFO:tasks.rados.rados.0.smithi165.stdout:897: expect (ObjNum 248 snap 73 seq_num 248) 2024-09-10T16:13:13.014 INFO:tasks.rados.rados.0.smithi165.stdout:889: done (9 left) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:890: done (8 left) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:891: done (7 left) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:892: done (6 left) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:894: done (5 left) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:898: read oid 36 snap -1 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:898: expect (ObjNum 306 snap 88 seq_num 306) 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:899: write oid 45 current snap is 98 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: finishing write tid 4 to smithi16534841-26 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: finishing write tid 5 to smithi16534841-26 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: oid 26 updating version 0 to 818 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: oid 26 updating version 818 to 820 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: oid 26 version 820 is already newer than 817 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:888: oid 26 version 820 is already newer than 819 2024-09-10T16:13:13.015 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 820 (ObjNum 337 snap 98 seq_num 337) dirty exists 2024-09-10T16:13:13.016 INFO:tasks.rados.rados.0.smithi165.stdout:888: left oid 26 (ObjNum 337 snap 98 seq_num 337) 2024-09-10T16:13:13.016 INFO:tasks.rados.rados.0.smithi165.stdout:893: expect (ObjNum 315 snap 91 seq_num 315) 2024-09-10T16:13:13.088 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:13.125 INFO:tasks.rados.rados.0.smithi165.stdout:895: finishing write tid 1 to smithi16534841-30 2024-09-10T16:13:13.125 INFO:tasks.rados.rados.0.smithi165.stdout:899: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-10T16:13:13.131 INFO:tasks.rados.rados.0.smithi165.stdout:899: writing smithi16534841-45 from 514380 to 1218572 tid 1 2024-09-10T16:13:13.136 INFO:tasks.rados.rados.0.smithi165.stdout:899: writing smithi16534841-45 from 1629760 to 2108195 tid 2 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:899: writing smithi16534841-45 from 2604288 to 3046890 tid 3 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:888: done (6 left) 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:893: done (5 left) 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:900: write oid 5 current snap is 98 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:895: finishing write tid 2 to smithi16534841-30 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:895: finishing write tid 3 to smithi16534841-30 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:895: finishing write tid 4 to smithi16534841-30 2024-09-10T16:13:13.142 INFO:tasks.rados.rados.0.smithi165.stdout:900: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-10T16:13:13.148 INFO:tasks.rados.rados.0.smithi165.stdout:900: writing smithi16534841-5 from 531187 to 1310628 tid 1 2024-09-10T16:13:13.156 INFO:tasks.rados.rados.0.smithi165.stdout:900: writing smithi16534841-5 from 2088242 to 2826688 tid 2 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): tick 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 msgr2=0x7f24f400aa30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008570 0x7f24f400aa30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 msgr2=0x7f24f4007e80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:13.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005a20 0x7f24f4007e80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 reap_dead start 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-a con 0x7f24f4005e70 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-c con 0x7f24f4008f60 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-b con 0x7f24f4009880 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f250809f860 con 0x7f24f4005e70 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f24f400b0d0 con 0x7f24f4008f60 2024-09-10T16:13:13.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24f400a4c0 con 0x7f24f4009880 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25057fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005e70 auth_method 0 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f25077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008f60 auth_method 0 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f4009880 auth_method 0 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.158+0000 7f2507fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:13.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:13.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.159+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:900: writing smithi16534841-5 from 3431711 to 3731187 tid 3 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:901: read oid 15 snap -1 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:901: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:902: rollback oid 31 current snap is 98 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 94 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:895: finishing write tid 5 to smithi16534841-30 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:895: oid 30 updating version 0 to 823 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:895: oid 30 version 823 is already newer than 822 2024-09-10T16:13:13.166 INFO:tasks.rados.rados.0.smithi165.stdout:895: oid 30 updating version 823 to 825 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:895: oid 30 version 825 is already newer than 824 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 825 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:895: left oid 30 (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: finishing write tid 1 to smithi16534841-48 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: finishing write tid 2 to smithi16534841-48 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: finishing write tid 3 to smithi16534841-48 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: finishing write tid 4 to smithi16534841-48 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: finishing write tid 5 to smithi16534841-48 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: oid 48 updating version 0 to 829 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: oid 48 version 829 is already newer than 828 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: oid 48 version 829 is already newer than 826 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:896: oid 48 version 829 is already newer than 827 2024-09-10T16:13:13.167 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 829 (ObjNum 339 snap 98 seq_num 339) dirty exists 2024-09-10T16:13:13.168 INFO:tasks.rados.rados.0.smithi165.stdout:896: left oid 48 (ObjNum 339 snap 98 seq_num 339) 2024-09-10T16:13:13.168 INFO:tasks.rados.rados.0.smithi165.stdout:897: expect (ObjNum 248 snap 73 seq_num 248) 2024-09-10T16:13:13.208 INFO:tasks.rados.rados.0.smithi165.stdout:898: expect (ObjNum 306 snap 88 seq_num 306) 2024-09-10T16:13:13.208 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:13.247 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1741771 2024-09-10T16:13:13.303 INFO:tasks.rados.rados.0.smithi165.stdout:899: finishing write tid 1 to smithi16534841-45 2024-09-10T16:13:13.303 INFO:tasks.rados.rados.0.smithi165.stdout:899: finishing write tid 2 to smithi16534841-45 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: finishing write tid 3 to smithi16534841-45 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: finishing write tid 4 to smithi16534841-45 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: finishing write tid 5 to smithi16534841-45 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: oid 45 updating version 0 to 269 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: oid 45 updating version 269 to 270 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: oid 45 updating version 270 to 272 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: oid 45 version 272 is already newer than 271 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 272 (ObjNum 340 snap 98 seq_num 340) dirty exists 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:899: left oid 45 (ObjNum 340 snap 98 seq_num 340) 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:900: finishing write tid 1 to smithi16534841-5 2024-09-10T16:13:13.304 INFO:tasks.rados.rados.0.smithi165.stdout:895: done (7 left) 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:896: done (6 left) 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:897: done (5 left) 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:898: done (4 left) 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:899: done (3 left) 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:903: write oid 39 current snap is 98 2024-09-10T16:13:13.305 INFO:tasks.rados.rados.0.smithi165.stdout:903: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-10T16:13:13.308 INFO:tasks.rados.rados.0.smithi165.stdout:903: writing smithi16534841-39 from 547994 to 1002684 tid 1 2024-09-10T16:13:13.315 INFO:tasks.rados.rados.0.smithi165.stdout:903: writing smithi16534841-39 from 1746724 to 2228828 tid 2 2024-09-10T16:13:13.320 INFO:tasks.rados.rados.0.smithi165.stdout:903: writing smithi16534841-39 from 2659134 to 3124280 tid 3 2024-09-10T16:13:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:903: writing smithi16534841-39 from 3605976 to 3747994 tid 4 2024-09-10T16:13:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:904: write (excl) oid 42 current snap is 98 2024-09-10T16:13:13.323 INFO:tasks.rados.rados.0.smithi165.stdout:904: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-10T16:13:13.329 INFO:tasks.rados.rados.0.smithi165.stdout:904: writing smithi16534841-42 from 564801 to 1211093 tid 1 2024-09-10T16:13:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:904: writing smithi16534841-42 from 1921559 to 2663674 tid 2 2024-09-10T16:13:13.344 INFO:tasks.rados.rados.0.smithi165.stdout:904: writing smithi16534841-42 from 3202910 to 3764801 tid 3 2024-09-10T16:13:13.344 INFO:tasks.rados.rados.0.smithi165.stdout:905: snap_remove snap 96 2024-09-10T16:13:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f4009880 auth_method 0 2024-09-10T16:13:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005e70 auth_method 0 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008f60 auth_method 0 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.360+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.361+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005e70 auth_method 0 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.762+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f4009880 auth_method 0 2024-09-10T16:13:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f4008f60 auth_method 0 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:13.763+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:14.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:14.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:14.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f4005e70 auth_method 0 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f4009880 auth_method 0 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:14.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:14.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:14.564+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient(hunting): tick 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 msgr2=0x7f24f4009d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f4009880 0x7f24f4009d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 msgr2=0x7f24f4009340 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.165:3300/0 conn(0x7f24f4008f60 0x7f24f4009340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 msgr2=0x7f24f4008a20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 1 --2- 172.21.15.164:0/992567144 >> v2:172.21.15.164:3300/0 conn(0x7f24f4005e70 0x7f24f4008a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:16.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient: _add_conns ranks=[1,0] 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f24f400a8e0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.158+0000 7f25057fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f24f4012c70 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.159+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f250809c6f0 con 0x7f24f4012c70 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.159+0000 7f25057fa640 1 -- 172.21.15.164:0/992567144 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24f40177c0 con 0x7f24f400a8e0 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.159+0000 7f25057fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:16.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.161+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 legacy=0x7f24f4015060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:16.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:16.161+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.165:6789/0 conn(0x7f24f400a8e0 legacy=0x7f24f4011550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:17.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.140+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 legacy=0x7f24f4015060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T16:13:17.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.140+0000 7f2506ffd640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 legacy=0x7f24f4015060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:13:17.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.140+0000 7f2506ffd640 1 --1- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 0x7f24f4015060 :-1 s=OPENED pgs=21 cs=1 l=1).handle_message read tag failed 2024-09-10T16:13:17.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.140+0000 7f2506ffd640 1 --1- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 0x7f24f4015060 :-1 s=OPENED pgs=21 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:13:17.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.140+0000 7f25067fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.164:6789/0 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3750270347 0 0) 0x7f24f40177c0 con 0x7f24f400a8e0 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient(hunting): _negotiate 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient(hunting): my global_id is 64159 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24f0003520 con 0x7f24f400a8e0 2024-09-10T16:13:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 648850680 0 0) 0x7f24f0003520 con 0x7f24f400a8e0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.164:6789/0 conn(0x7f24f4012c70 legacy=0x7f24f4015060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient: _finish_hunting 0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f40189a0 con 0x7f24f400a8e0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient: _finish_auth 0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient: _check_auth_tickets 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f24f8001770 con 0x7f24f400a8e0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.603+0000 7f25067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:13:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient: dump: 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 >> v1:172.21.15.165:6789/0 conn(0x7f24f400a8e0 legacy=0x7f24f4011550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 1 -- 172.21.15.164:0/992567144 reap_dead start 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f24f0003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): picked mon.b con 0x7f24f0003a70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f0004800 0x7f24f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): picked mon.a con 0x7f24f0004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f00099b0 0x7f24f000bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): picked mon.c con 0x7f24f00099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f24f000c390 con 0x7f24f0004800 2024-09-10T16:13:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f24f000c540 con 0x7f24f0003a70 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f24f000c740 con 0x7f24f00099b0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25067fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f24f0003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f0004800 0x7f24f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f00099b0 0x7f24f000bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f0003a70 auth_method 0 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f0004800 auth_method 0 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f00099b0 auth_method 0 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:13:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient(hunting): handle_auth_done global_id 64159 payload 293 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f00099b0 msgr2=0x7f24f000bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f00099b0 0x7f24f000bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f0004800 msgr2=0x7f24f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f0004800 0x7f24f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 monclient: found mon.b 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24f0003520 con 0x7f24f0003a70 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f25077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f250dd24640 5 monclient: authenticate success, global_id 64159 2024-09-10T16:13:17.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.604+0000 7f250dd24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25077fe640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 msgr2=0x7f24f0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25077fe640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f24f0003e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f24f8001990 tx=0x7f24f0005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a8022a49ac68aa9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8033070 con 0x7f24f0003a70 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: dump: 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24f80072d0 con 0x7f24f0003a70 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f250dd24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f250dd24640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:17.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 1 -- 172.21.15.164:0/992567144 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8001b00 con 0x7f24f0003a70 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f250dd24640 4 set_mon_vals no callback set 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.608+0000 7f25067fc640 10 monclient: dump: 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.622 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 10 monclient: shutdown 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 -- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 msgr2=0x7f24f0003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f24f0003e50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f24f8001990 tx=0x7f24f0005830 comp rx=0 tx=0).stop 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 -- 172.21.15.164:0/992567144 shutdown_connections 2024-09-10T16:13:17.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f00099b0 0x7f24f000bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f0004800 0x7f24f0004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 --2- 172.21.15.164:0/992567144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f24f0003e50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 -- 172.21.15.164:0/992567144 >> 172.21.15.164:0/992567144 conn(0x7f25080a3110 msgr2=0x7f25080a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 -- 172.21.15.164:0/992567144 shutdown_connections 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.609+0000 7f250dd24640 1 -- 172.21.15.164:0/992567144 wait complete. 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.610+0000 7f250dd24640 10 monclient: build_initial_monmap 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.610+0000 7f250dd24640 10 monclient: monmap: 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:17.624 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 1 Processor -- start 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 1 -- start start 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 10 monclient: init 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:17.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-b con 0x7f24f0003a70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.611+0000 7f250dd24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 0x7f2508099670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-c con 0x7f24f0004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 0x7f2508099bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): picked mon.noname-a con 0x7f24f00099b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f25080a20b0 con 0x7f24f00099b0 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f250809a7b0 con 0x7f24f0003a70 2024-09-10T16:13:17.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f25080a2230 con 0x7f24f0004800 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f250dd24640 10 monclient(hunting): authenticate will time out at 1176.099066s 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 0x7f2508099670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 0x7f2508099bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 0x7f2508099670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56362/0 (socket says 172.21.15.164:56362) 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:55050/0 (socket says 172.21.15.164:55050) 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 1 -- 172.21.15.164:0/2332748616 learned_addr learned my addr 172.21.15.164:0/2332748616 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): get_auth_request con 0x7f24f0003a70 auth_method 0 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): get_auth_request con 0x7f24f00099b0 auth_method 0 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): get_auth_request con 0x7f24f0004800 auth_method 0 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2506ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient(hunting): handle_auth_done global_id 64162 payload 824 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 -- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 msgr2=0x7f2508099670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 0x7f2508099670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 -- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 msgr2=0x7f2508099bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 0x7f2508099bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 monclient: found mon.noname-b 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f25077fe640 1 -- 172.21.15.164:0/2332748616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25081cd370 con 0x7f24f0003a70 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.612+0000 7f2507fff640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 0x7f2508099bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.615+0000 7f25077fe640 1 -- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 msgr2=0x7f25081ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.615+0000 7f25077fe640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f24f8001110 tx=0x7f24f80014b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8033070 con 0x7f24f0003a70 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:17.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: dump: 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: _finish_auth 0 2024-09-10T16:13:17.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: _check_auth_tickets 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24f8001ae0 con 0x7f24f0003a70 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24f8001770 con 0x7f24f0003a70 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 5 monclient: authenticate success, global_id 64162 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 10 monclient: _renew_subs 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2508095090 con 0x7f24f0003a70 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f24effff640 10 monclient: dump: 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:17.633 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f2505ffb640 4 set_mon_vals no callback set 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 10 monclient: _renew_subs 2024-09-10T16:13:17.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.617+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2508095880 con 0x7f24f0003a70 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.618+0000 7f25077fe640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f24f8001110 tx=0x7f24f80014b0 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f24f8036070 mgrmap(e 5) 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.618+0000 7f250dd24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.618+0000 7f250dd24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.618+0000 7f250dd24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.618+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f250809d930 con 0x7f24f0003a70 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.619+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f24f8036070 con 0x7f24f0003a70 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.619+0000 7f24effff640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 0x7f24f4042370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.619+0000 7f2506ffd640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 0x7f24f4042370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.619+0000 7f2506ffd640 10 monclient: get_auth_request con 0x7f24f403feb0 auth_method 0 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.622+0000 7f2506ffd640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 0x7f24f4042370 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25080942a0 tx=0x7f24fc01d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:17.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.624+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(222..222 src has 184..222) ==== 4815+0+0 (secure 0 0 0) 0x7f24f8010040 con 0x7f24f0003a70 2024-09-10T16:13:17.671 INFO:tasks.rados.rados.0.smithi165.stdout:905: done (5 left) 2024-09-10T16:13:17.671 INFO:tasks.rados.rados.0.smithi165.stdout:906: copy_from oid 4 from oid 12 current snap is 98 2024-09-10T16:13:17.671 INFO:tasks.rados.rados.0.smithi165.stdout:900: finishing write tid 2 to smithi16534841-5 2024-09-10T16:13:17.671 INFO:tasks.rados.rados.0.smithi165.stdout:900: finishing write tid 3 to smithi16534841-5 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: finishing write tid 4 to smithi16534841-5 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:902: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: finishing write tid 5 to smithi16534841-5 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: oid 5 updating version 0 to 334 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: oid 5 updating version 334 to 335 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: oid 5 version 335 is already newer than 333 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:900: oid 5 version 335 is already newer than 332 2024-09-10T16:13:17.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 335 (ObjNum 341 snap 98 seq_num 341) dirty exists 2024-09-10T16:13:17.673 INFO:tasks.rados.rados.0.smithi165.stdout:900: left oid 5 (ObjNum 341 snap 98 seq_num 341) 2024-09-10T16:13:17.673 INFO:tasks.rados.rados.0.smithi165.stdout:902: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:13:17.673 INFO:tasks.rados.rados.0.smithi165.stdout:901: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:13:17.673 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 996979 2024-09-10T16:13:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.715+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f24f8072890 con 0x7f24f0003a70 2024-09-10T16:13:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.715+0000 7f24effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.715+0000 7f24effff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:17.726 INFO:tasks.rados.rados.0.smithi165.stdout:902: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:13:17.726 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 833 (ObjNum 315 snap 91 seq_num 315) dirty exists 2024-09-10T16:13:17.726 INFO:tasks.rados.rados.0.smithi165.stdout:900: done (5 left) 2024-09-10T16:13:17.726 INFO:tasks.rados.rados.0.smithi165.stdout:901: done (4 left) 2024-09-10T16:13:17.726 INFO:tasks.rados.rados.0.smithi165.stdout:902: done (3 left) 2024-09-10T16:13:17.727 INFO:tasks.rados.rados.0.smithi165.stdout:907: read oid 40 snap -1 2024-09-10T16:13:17.727 INFO:tasks.rados.rados.0.smithi165.stdout:907: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:13:17.727 INFO:tasks.rados.rados.0.smithi165.stdout:908: snap_remove snap 94 2024-09-10T16:13:17.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.960+0000 7f250dd24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.960+0000 7f250dd24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.960+0000 7f250dd24640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.960+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2508077260 con 0x7f24f0003a70 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.961+0000 7f24effff640 1 -- 172.21.15.164:0/2332748616 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f24f802ed20 con 0x7f24f0003a70 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.961+0000 7f24effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:17.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.961+0000 7f24effff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:17.963 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 msgr2=0x7f24f4042370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 0x7f24f4042370 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f25080942a0 tx=0x7f24fc01d040 comp rx=0 tx=0).stop 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 10 monclient: shutdown 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 msgr2=0x7f25081ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f24f8001110 tx=0x7f24f80014b0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 shutdown_connections 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f24f403feb0 0x7f24f4042370 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f24f00099b0 0x7f2508099bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f24f0004800 0x7f2508099670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 --2- 172.21.15.164:0/2332748616 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f24f0003a70 0x7f25081ccc50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:17.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 >> 172.21.15.164:0/2332748616 conn(0x7f25080a3110 msgr2=0x7f250813ab10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:17.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 shutdown_connections 2024-09-10T16:13:17.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:17.963+0000 7f250dd24640 1 -- 172.21.15.164:0/2332748616 wait complete. 2024-09-10T16:13:17.976 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:13:17.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:13:17.976 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:13:18.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.073+0000 7f5e021ad640 10 monclient: get_monmap_and_config 2024-09-10T16:13:18.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.073+0000 7f5e021ad640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.073+0000 7f5e021ad640 10 monclient: monmap: 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:18.074011+0000 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:18.074011+0000 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:18.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:18.076 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:18.076 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:18.076 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.073+0000 7f5e021ad640 1 Processor -- start 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 1 -- start start 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient: init 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5dfc06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5dfc134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5dfc13e740 0x7f5dfc140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-c con 0x7f5dfc13e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dfc09ce50 con 0x7f5dfc134f70 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dfc135380 con 0x7f5dfc06cb70 2024-09-10T16:13:18.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5dfc06db60 con 0x7f5dfc13e740 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5e021ad640 10 monclient(hunting): authenticate will time out at 1176.561539s 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5df2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5dfc134f70 0x7f5dfc13d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35030/0 (socket says 172.21.15.164:35030) 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5df2ffd640 1 -- 172.21.15.164:0/2007908372 learned_addr learned my addr 172.21.15.164:0/2007908372 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 1 --2- 172.21.15.164:0/2007908372 >> v2:172.21.15.165:3300/0 conn(0x7f5dfc13e740 0x7f5dfc140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): get_auth_request con 0x7f5dfc13e740 auth_method 0 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.074+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_done global_id 64165 payload 274 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 1 -- 172.21.15.164:0/2007908372 >> v1:172.21.15.165:6789/0 conn(0x7f5dfc06cb70 legacy=0x7f5dfc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 1 -- 172.21.15.164:0/2007908372 >> v1:172.21.15.164:6789/0 conn(0x7f5dfc134f70 legacy=0x7f5dfc13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 1 monclient: found mon.noname-c 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.075+0000 7f5dfbfff640 1 -- 172.21.15.164:0/2007908372 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc09cb70 con 0x7f5dfc13e740 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfbfff640 1 -- 172.21.15.164:0/2007908372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc13e740 msgr2=0x7f5dfc140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfbfff640 1 --2- 172.21.15.164:0/2007908372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc13e740 0x7f5dfc140b30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009dc0 tx=0x7f5df402a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6579f05cbb77616 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 1 -- 172.21.15.164:0/2007908372 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4007a30 con 0x7f5dfc13e740 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: dump: 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 1 -- 172.21.15.164:0/2007908372 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4007bd0 con 0x7f5dfc13e740 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5e021ad640 5 monclient: authenticate success, global_id 64165 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5e021ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 1 -- 172.21.15.164:0/2007908372 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4031400 con 0x7f5dfc13e740 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5e021ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5e021ad640 4 set_mon_vals no callback set 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.078+0000 7f5dfaffd640 10 monclient: dump: 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 10 monclient: shutdown 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 -- 172.21.15.164:0/2007908372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc13e740 msgr2=0x7f5dfc140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 --2- 172.21.15.164:0/2007908372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc13e740 0x7f5dfc140b30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009dc0 tx=0x7f5df402a4d0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 -- 172.21.15.164:0/2007908372 shutdown_connections 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 --2- 172.21.15.164:0/2007908372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc13e740 0x7f5dfc140b30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 -- 172.21.15.164:0/2007908372 >> 172.21.15.164:0/2007908372 conn(0x7f5dfc0aa900 msgr2=0x7f5dfc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 -- 172.21.15.164:0/2007908372 shutdown_connections 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 -- 172.21.15.164:0/2007908372 wait complete. 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 10 monclient: monmap: 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:18.090 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.079+0000 7f5e021ad640 1 Processor -- start 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 -- start start 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient: init 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:18.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc06cb70 0x7f5dfc1cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-b con 0x7f5dfc06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dfc134f70 0x7f5dfc1cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-c con 0x7f5dfc134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 0x7f5dfc13acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): picked mon.noname-a con 0x7f5dfc13e740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc0a4db0 con 0x7f5dfc13e740 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc06cf50 con 0x7f5dfc06cb70 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5dfc141070 con 0x7f5dfc134f70 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5e021ad640 10 monclient(hunting): authenticate will time out at 1176.567635s 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 0x7f5dfc13acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 0x7f5dfc13acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45966/0 (socket says 172.21.15.164:45966) 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dfc134f70 0x7f5dfc1cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 learned_addr learned my addr 172.21.15.164:0/1781407919 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): get_auth_request con 0x7f5dfc13e740 auth_method 0 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): get_auth_request con 0x7f5dfc134f70 auth_method 0 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5df2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfb7fe640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc06cb70 0x7f5dfc1cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfb7fe640 10 monclient(hunting): get_auth_request con 0x7f5dfc06cb70 auth_method 0 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.080+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 10 monclient(hunting): handle_auth_done global_id 74100 payload 824 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dfc134f70 msgr2=0x7f5dfc1cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dfc134f70 0x7f5dfc1cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc06cb70 msgr2=0x7f5dfc1cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5dfc06cb70 0x7f5dfc1cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 monclient: found mon.noname-a 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dfc0a4060 con 0x7f5dfc13e740 2024-09-10T16:13:18.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.081+0000 7f5df2ffd640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5dfc134f70 0x7f5dfc1cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 msgr2=0x7f5dfc13acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5dfbfff640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 0x7f5dfc13acf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009ee0 tx=0x7f5df4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4040070 con 0x7f5dfc13e740 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:18.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: dump: 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 5 monclient: authenticate success, global_id 74100 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5df4045660 con 0x7f5dfc13e740 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 10 monclient: _renew_subs 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dfc06e720 con 0x7f5dfc13e740 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5df4045a30 con 0x7f5dfc13e740 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5df8ff9640 10 monclient: dump: 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 10 monclient: _renew_subs 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5dfa7fc640 4 set_mon_vals no callback set 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.083+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dfc09dab0 con 0x7f5dfc13e740 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5df4039040 con 0x7f5dfc13e740 2024-09-10T16:13:18.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5e021ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5e021ad640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 0x7f5dfc13b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc0a4680 con 0x7f5dfc1cddf0 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5df8ff9640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5ddc03e240 0x7f5ddc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 0x7f5dfc13b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(223..223 src has 184..223) ==== 4815+0+0 (secure 0 0 0) 0x7f5df4074830 con 0x7f5dfc13e740 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient: get_auth_request con 0x7f5dfc1cddf0 auth_method 0 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5df2ffd640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5ddc03e240 0x7f5ddc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5df2ffd640 10 monclient: get_auth_request con 0x7f5ddc03e240 auth_method 0 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_done global_id 74103 payload 824 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.084+0000 7f5dfb7fe640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5dfc09d630 con 0x7f5dfc1cddf0 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5dfb7fe640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 msgr2=0x7f5dfc13b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5dfb7fe640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 0x7f5dfc13b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00e770 tx=0x7f5dec00ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec017070 con 0x7f5dfc1cddf0 2024-09-10T16:13:18.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 10 monclient: dump: 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df2ffd640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5ddc03e240 0x7f5ddc040700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5de4007c90 tx=0x7f5de4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5dec00c9b0 con 0x7f5dfc1cddf0 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 msgr2=0x7f5dfc13b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 0x7f5dfc13b230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5dec00e770 tx=0x7f5dec00ec30 comp rx=0 tx=0).stop 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 msgr2=0x7f5dfc13b230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.087+0000 7f5df8ff9640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc1cddf0 0x7f5dfc13b230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5e021ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5e021ad640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 0x7f5dfc135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5e021ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5dfc13bb80 con 0x7f5dfc13b770 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 0x7f5dfc135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient: get_auth_request con 0x7f5dfc13b770 auth_method 0 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 10 monclient(hunting): handle_auth_done global_id 74106 payload 824 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.093+0000 7f5dfb7fe640 1 -- 172.21.15.164:0/1781407919 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5dfc09de70 con 0x7f5dfc13b770 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5dfb7fe640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 msgr2=0x7f5dfc135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5dfb7fe640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 0x7f5dfc135870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5dec019000 tx=0x7f5dec0046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dec017070 con 0x7f5dfc13b770 2024-09-10T16:13:18.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 10 monclient: dump: 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f5dec0067c0 con 0x7f5dfc13b770 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 msgr2=0x7f5dfc135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 0x7f5dfc135870 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5dec019000 tx=0x7f5dec0046f0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 msgr2=0x7f5dfc135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5df8ff9640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13b770 0x7f5dfc135870 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f5dec019000 tx=0x7f5dec0046f0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:18.110 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 25963, 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.111 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:18.112 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:18.114 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.115 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:18.116 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.117 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.119 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:18.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.121 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:18.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:18.123 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:18.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5ddc03e240 msgr2=0x7f5ddc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5ddc03e240 0x7f5ddc040700 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f5de4007c90 tx=0x7f5de4007be0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 10 monclient: shutdown 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 msgr2=0x7f5dfc13acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 --2- 172.21.15.164:0/1781407919 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5dfc13e740 0x7f5dfc13acf0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5df4009ee0 tx=0x7f5df4004a40 comp rx=0 tx=0).stop 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 reap_dead start 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5dfbfff640 1 -- 172.21.15.164:0/1781407919 reap_dead start 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 shutdown_connections 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 >> 172.21.15.164:0/1781407919 conn(0x7f5dfc0aa900 msgr2=0x7f5dfc0aa0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 shutdown_connections 2024-09-10T16:13:18.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.096+0000 7f5e021ad640 1 -- 172.21.15.164:0/1781407919 wait complete. 2024-09-10T16:13:18.126 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:18.126 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:13:18.126 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:13:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 10 monclient: get_monmap_and_config 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 10 monclient: monmap: 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:18.224175+0000 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:18.224175+0000 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:18.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 1 Processor -- start 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 1 -- start start 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.223+0000 7faf091e2640 10 monclient: init 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-b con 0x7faf0406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf040a46a0 0x7faf040a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-c con 0x7faf040a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 0x7faf04138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-a con 0x7faf04137ca0 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7faf0406db60 con 0x7faf04137ca0 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faf0406cf50 con 0x7faf040a46a0 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf0409ce50 con 0x7faf0406cb70 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf091e2640 10 monclient(hunting): authenticate will time out at 1176.711441s 2024-09-10T16:13:18.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 0x7faf04138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 0x7faf04138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46002/0 (socket says 172.21.15.164:46002) 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56380/0 (socket says 172.21.15.164:56380) 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 1 -- 172.21.15.164:0/2418677054 learned_addr learned my addr 172.21.15.164:0/2418677054 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): get_auth_request con 0x7faf04137ca0 auth_method 0 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): get_auth_request con 0x7faf0406cb70 auth_method 0 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0259c640 1 --2- 172.21.15.164:0/2418677054 >> v2:172.21.15.165:3300/0 conn(0x7faf040a46a0 0x7faf040a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0259c640 10 monclient(hunting): get_auth_request con 0x7faf040a46a0 auth_method 0 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient(hunting): handle_auth_done global_id 4961 payload 274 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 -- 172.21.15.164:0/2418677054 >> v2:172.21.15.165:3300/0 conn(0x7faf040a46a0 msgr2=0x7faf040a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2418677054 >> v2:172.21.15.165:3300/0 conn(0x7faf040a46a0 0x7faf040a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 -- 172.21.15.164:0/2418677054 >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 msgr2=0x7faf04138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2418677054 >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 0x7faf04138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 monclient: found mon.noname-b 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.224+0000 7faf02d9d640 1 -- 172.21.15.164:0/2418677054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf0409cb70 con 0x7faf0406cb70 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf02d9d640 1 -- 172.21.15.164:0/2418677054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 msgr2=0x7faf04134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2418677054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faf0406d1b0 tx=0x7faef802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d66021d0291ae428 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 1 -- 172.21.15.164:0/2418677054 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8037070 con 0x7faf0406cb70 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: dump: 2024-09-10T16:13:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 1 -- 172.21.15.164:0/2418677054 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef8032440 con 0x7faf0406cb70 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf091e2640 5 monclient: authenticate success, global_id 4961 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf091e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf091e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 1 -- 172.21.15.164:0/2418677054 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8032810 con 0x7faf0406cb70 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf091e2640 4 set_mon_vals no callback set 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.227+0000 7faf01d9b640 10 monclient: dump: 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 10 monclient: shutdown 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 -- 172.21.15.164:0/2418677054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 msgr2=0x7faf04134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 --2- 172.21.15.164:0/2418677054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7faf0406d1b0 tx=0x7faef802a000 comp rx=0 tx=0).stop 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 -- 172.21.15.164:0/2418677054 shutdown_connections 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 --2- 172.21.15.164:0/2418677054 >> v2:172.21.15.164:3300/0 conn(0x7faf04137ca0 0x7faf04138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 --2- 172.21.15.164:0/2418677054 >> v2:172.21.15.165:3300/0 conn(0x7faf040a46a0 0x7faf040a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 --2- 172.21.15.164:0/2418677054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf0406cb70 0x7faf04134f70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 -- 172.21.15.164:0/2418677054 >> 172.21.15.164:0/2418677054 conn(0x7faf040aa900 msgr2=0x7faf040aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 -- 172.21.15.164:0/2418677054 shutdown_connections 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.228+0000 7faf091e2640 1 -- 172.21.15.164:0/2418677054 wait complete. 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.229+0000 7faf091e2640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.229+0000 7faf091e2640 10 monclient: monmap: 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.229+0000 7faf091e2640 1 Processor -- start 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.229+0000 7faf091e2640 1 -- start start 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.229+0000 7faf091e2640 10 monclient: init 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf0406cb70 0x7faf04131480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-b con 0x7faf0406cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 0x7faf041319c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-c con 0x7faf040a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf04137ca0 0x7faf040a27f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): picked mon.noname-a con 0x7faf04137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf041387f0 con 0x7faf04137ca0 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faf04138670 con 0x7faf0406cb70 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf04138970 con 0x7faf040a46a0 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf091e2640 10 monclient(hunting): authenticate will time out at 1176.717112s 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 0x7faf041319c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf04137ca0 0x7faf040a27f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 0x7faf041319c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56392/0 (socket says 172.21.15.164:56392) 2024-09-10T16:13:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 -- 172.21.15.164:0/2399492061 learned_addr learned my addr 172.21.15.164:0/2399492061 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): get_auth_request con 0x7faf040a46a0 auth_method 0 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): get_auth_request con 0x7faf04137ca0 auth_method 0 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf0406cb70 0x7faf04131480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): get_auth_request con 0x7faf0406cb70 auth_method 0 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient(hunting): handle_auth_done global_id 4964 payload 824 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf0406cb70 msgr2=0x7faf04131480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf0406cb70 0x7faf04131480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf04137ca0 msgr2=0x7faf040a27f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf04137ca0 0x7faf040a27f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 monclient: found mon.noname-c 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0259c640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf04131f00 con 0x7faf040a46a0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.230+0000 7faf0359e640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf04137ca0 0x7faf040a27f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf0259c640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 msgr2=0x7faf041319c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf0259c640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 0x7faf041319c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faef400d800 tx=0x7faef400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef400b8d0 con 0x7faf040a46a0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: dump: 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faef4019070 con 0x7faf040a46a0 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 5 monclient: authenticate success, global_id 4964 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 10 monclient: _renew_subs 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf04121cf0 con 0x7faf040a46a0 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef400bc90 con 0x7faf040a46a0 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faeeb7fe640 10 monclient: dump: 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 10 monclient: _renew_subs 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf04122e80 con 0x7faf040a46a0 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.232+0000 7faf0159a640 4 set_mon_vals no callback set 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.233+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faef401e460 con 0x7faf040a46a0 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.233+0000 7faf091e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.233+0000 7faf091e2640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 0x7faf040a2d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.233+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.233+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf04122920 con 0x7faf041cd210 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 0x7faf040a2d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient: get_auth_request con 0x7faf041cd210 auth_method 0 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faeeb7fe640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faef003e240 0x7faef0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(223..223 src has 184..223) ==== 4815+0+0 (secure 0 0 0) 0x7faef40531d0 con 0x7faf040a46a0 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 10 monclient(hunting): handle_auth_done global_id 4967 payload 824 2024-09-10T16:13:18.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf02d9d640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf040a4a80 con 0x7faf041cd210 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf0359e640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faef003e240 0x7faef0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.234+0000 7faf0359e640 10 monclient: get_auth_request con 0x7faef003e240 auth_method 0 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faf02d9d640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 msgr2=0x7faf040a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 0x7faf040a2d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faf040a6cc0 tx=0x7faef80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8037040 con 0x7faf041cd210 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 10 monclient: dump: 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.253 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faf0359e640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faef003e240 0x7faef0040700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faf041210d0 tx=0x7faefc00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faef803e460 con 0x7faf041cd210 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 msgr2=0x7faf040a2d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.236+0000 7faeeb7fe640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 0x7faf040a2d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faf040a6cc0 tx=0x7faef80046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.237+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 msgr2=0x7faf040a2d30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.237+0000 7faeeb7fe640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf041cd210 0x7faf040a2d30 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7faf040a6cc0 tx=0x7faef80046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.242+0000 7faf091e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:18.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf091e2640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 0x7faf04135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf091e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf040a3680 con 0x7faf040a3270 2024-09-10T16:13:18.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 0x7faf04135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient: get_auth_request con 0x7faf040a3270 auth_method 0 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 10 monclient(hunting): handle_auth_done global_id 4970 payload 824 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.243+0000 7faf02d9d640 1 -- 172.21.15.164:0/2399492061 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faf04122640 con 0x7faf040a3270 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.245+0000 7faf02d9d640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 msgr2=0x7faf04135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.245+0000 7faf02d9d640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 0x7faf04135870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faef802a040 tx=0x7faef8009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faef8037070 con 0x7faf040a3270 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 10 monclient: dump: 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7faef8053430 con 0x7faf040a3270 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 msgr2=0x7faf04135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 0x7faf04135870 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faef802a040 tx=0x7faef8009190 comp rx=0 tx=0).stop 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 msgr2=0x7faf04135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faeeb7fe640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a3270 0x7faf04135870 secure :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7faef802a040 tx=0x7faef8009190 comp rx=0 tx=0).stop 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:18.259 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 440418, 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.260 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:18.261 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:18.262 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:18.263 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.264 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.267 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.268 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.269 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.270 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.271 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.272 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:18.273 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faef003e240 msgr2=0x7faef0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faef003e240 0x7faef0040700 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7faf041210d0 tx=0x7faefc00b040 comp rx=0 tx=0).stop 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 10 monclient: shutdown 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 msgr2=0x7faf041319c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 --2- 172.21.15.164:0/2399492061 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf040a46a0 0x7faf041319c0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faef400d800 tx=0x7faef400dcc0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf0359e640 1 -- 172.21.15.164:0/2399492061 reap_dead start 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf0359e640 1 -- 172.21.15.164:0/2399492061 reap_dead start 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 shutdown_connections 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 >> 172.21.15.164:0/2399492061 conn(0x7faf040aa900 msgr2=0x7faf04132e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 shutdown_connections 2024-09-10T16:13:18.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.246+0000 7faf091e2640 1 -- 172.21.15.164:0/2399492061 wait complete. 2024-09-10T16:13:18.275 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:18.275 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:13:18.275 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.373+0000 7fbbe9bad640 10 monclient: get_monmap_and_config 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.373+0000 7fbbe9bad640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.373+0000 7fbbe9bad640 10 monclient: monmap: 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:18.374507+0000 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:18.374507+0000 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.375 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:18.376 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 Processor -- start 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 -- start start 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient: init 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:13:18.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbbe4137bb0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbbe41386a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbbe40a2e80 0x7fbbe4140870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe40a2e80 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe40a1c90 con 0x7fbbe41386a0 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbe40a1f70 con 0x7fbbe4137bb0 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbbe40a2250 con 0x7fbbe40a2e80 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.374+0000 7fbbe9bad640 10 monclient(hunting): authenticate will time out at 1176.861955s 2024-09-10T16:13:18.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbbe41386a0 0x7fbbe413d140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35038/0 (socket says 172.21.15.164:35038) 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/268340611 learned_addr learned my addr 172.21.15.164:0/268340611 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 1 --2- 172.21.15.164:0/268340611 >> v2:172.21.15.165:3300/0 conn(0x7fbbe40a2e80 0x7fbbe4140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request con 0x7fbbe40a2e80 auth_method 0 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_done global_id 64177 payload 274 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 1 -- 172.21.15.164:0/268340611 >> v1:172.21.15.165:6789/0 conn(0x7fbbe4137bb0 legacy=0x7fbbe4137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 1 -- 172.21.15.164:0/268340611 >> v1:172.21.15.164:6789/0 conn(0x7fbbe41386a0 legacy=0x7fbbe413d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 1 monclient: found mon.noname-c 2024-09-10T16:13:18.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:13:18.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.375+0000 7fbbe3fff640 1 -- 172.21.15.164:0/268340611 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe40a3310 con 0x7fbbe40a2e80 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe3fff640 1 -- 172.21.15.164:0/268340611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 msgr2=0x7fbbe4140870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe3fff640 1 --2- 172.21.15.164:0/268340611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe4140870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097d0 tx=0x7fbbdc02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7baacfd4a7bbd9ab server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 1 -- 172.21.15.164:0/268340611 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc037070 con 0x7fbbe40a2e80 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: dump: 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 1 -- 172.21.15.164:0/268340611 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbdc032440 con 0x7fbbe40a2e80 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 1 -- 172.21.15.164:0/268340611 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc032810 con 0x7fbbe40a2e80 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe9bad640 5 monclient: authenticate success, global_id 64177 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe9bad640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe9bad640 4 set_mon_vals no callback set 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.378+0000 7fbbe27fc640 10 monclient: dump: 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.390 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 10 monclient: shutdown 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 -- 172.21.15.164:0/268340611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 msgr2=0x7fbbe4140870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/268340611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe4140870 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097d0 tx=0x7fbbdc02a000 comp rx=0 tx=0).stop 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 -- 172.21.15.164:0/268340611 shutdown_connections 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/268340611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe4140870 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 -- 172.21.15.164:0/268340611 >> 172.21.15.164:0/268340611 conn(0x7fbbe40a7320 msgr2=0x7fbbe40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 -- 172.21.15.164:0/268340611 shutdown_connections 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 1 -- 172.21.15.164:0/268340611 wait complete. 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.379+0000 7fbbe9bad640 10 monclient: monmap: 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:18.392 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 Processor -- start 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 -- start start 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient: init 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe41342a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-b con 0x7fbbe40a2e80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 0x7fbbe41347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-a con 0x7fbbe4137bb0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbe41386a0 0x7fbbe413cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): picked mon.noname-c con 0x7fbbe41386a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4077060 con 0x7fbbe4137bb0 2024-09-10T16:13:18.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe4076ee0 con 0x7fbbe40a2e80 2024-09-10T16:13:18.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbbe4140db0 con 0x7fbbe41386a0 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe9bad640 10 monclient(hunting): authenticate will time out at 1176.867876s 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 0x7fbbe41347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 0x7fbbe41347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46020/0 (socket says 172.21.15.164:46020) 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/2741507393 learned_addr learned my addr 172.21.15.164:0/2741507393 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): get_auth_request con 0x7fbbe4137bb0 auth_method 0 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.380+0000 7fbbe3fff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbe41386a0 0x7fbbe413cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request con 0x7fbbe41386a0 auth_method 0 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe41342a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): get_auth_request con 0x7fbbe40a2e80 auth_method 0 2024-09-10T16:13:18.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient(hunting): handle_auth_done global_id 74115 payload 824 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbe41386a0 msgr2=0x7fbbe413cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbbe41386a0 0x7fbbe413cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 msgr2=0x7fbbe41342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe40a2e80 0x7fbbe41342a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 monclient: found mon.noname-a 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.381+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe411d830 con 0x7fbbe4137bb0 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe2ffd640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 msgr2=0x7fbbe41347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe2ffd640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 0x7fbbe41347e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800cc10 tx=0x7fbbd800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd8012400 con 0x7fbbe4137bb0 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: dump: 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.398 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 5 monclient: authenticate success, global_id 74115 2024-09-10T16:13:18.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 10 monclient: _renew_subs 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbd80125e0 con 0x7fbbe4137bb0 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbe411ddf0 con 0x7fbbe4137bb0 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd80129d0 con 0x7fbbe4137bb0 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbcbfff640 10 monclient: dump: 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.400 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 10 monclient: _renew_subs 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe1ffb640 4 set_mon_vals no callback set 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.383+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbe411db10 con 0x7fbbe4137bb0 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe9bad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 0x7fbbe4139500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe409e0f0 con 0x7fbbe4139120 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbbd8012bf0 con 0x7fbbe4137bb0 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbcbfff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbd003e210 0x7fbbd00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe37fe640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 0x7fbbe4139500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(223..223 src has 184..223) ==== 4815+0+0 (secure 0 0 0) 0x7fbbd80534b0 con 0x7fbbe4137bb0 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe37fe640 10 monclient: get_auth_request con 0x7fbbe4139120 auth_method 0 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.384+0000 7fbbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe3fff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbd003e210 0x7fbbd00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe3fff640 10 monclient: get_auth_request con 0x7fbbd003e210 auth_method 0 2024-09-10T16:13:18.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_done global_id 64183 payload 824 2024-09-10T16:13:18.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.385+0000 7fbbe37fe640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbbe4138010 con 0x7fbbe4139120 2024-09-10T16:13:18.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbe37fe640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 msgr2=0x7fbbe4139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbe37fe640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 0x7fbbe4139500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd400fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbe3fff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbd003e210 0x7fbbd00406d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd4018030 con 0x7fbbe4139120 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.387+0000 7fbbcbfff640 10 monclient: dump: 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.406 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbbd400d9b0 con 0x7fbbe4139120 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 msgr2=0x7fbbe4139500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 0x7fbbe4139500 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd400fc60 comp rx=0 tx=0).stop 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 msgr2=0x7fbbe4139500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.388+0000 7fbbcbfff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139120 0x7fbbe4139500 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd400fc60 comp rx=0 tx=0).stop 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe9bad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 0x7fbbe413d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe9bad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbbe406d1c0 con 0x7fbbe4139a40 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 0x7fbbe413d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient: get_auth_request con 0x7fbbe4139a40 auth_method 0 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 10 monclient(hunting): handle_auth_done global_id 64186 payload 824 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.394+0000 7fbbe37fe640 1 -- 172.21.15.164:0/2741507393 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbbe4139e50 con 0x7fbbe4139a40 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe37fe640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 msgr2=0x7fbbe413d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe37fe640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 0x7fbbe413d100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd4003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbd401dbe0 con 0x7fbbe4139a40 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 10 monclient: dump: 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.409 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fbbd401dbe0 con 0x7fbbe4139a40 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 msgr2=0x7fbbe413d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 0x7fbbe413d100 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd4003760 comp rx=0 tx=0).stop 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 msgr2=0x7fbbe413d100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:18.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbcbfff640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbbe4139a40 0x7fbbe413d100 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fbbd400f7a0 tx=0x7fbbd4003760 comp rx=0 tx=0).stop 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 102825, 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:18.411 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.412 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.413 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:18.414 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:18.415 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:18.416 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.417 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:18.418 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.419 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.420 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.422 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:18.423 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:18.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:18.424 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:18.424 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:18.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbd003e210 msgr2=0x7fbbd00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbd003e210 0x7fbbd00406d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc0097a0 tx=0x7fbbdc02e000 comp rx=0 tx=0).stop 2024-09-10T16:13:18.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 10 monclient: shutdown 2024-09-10T16:13:18.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 msgr2=0x7fbbe41347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 --2- 172.21.15.164:0/2741507393 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbbe4137bb0 0x7fbbe41347e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbbd800cc10 tx=0x7fbbd800a430 comp rx=0 tx=0).stop 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe3fff640 1 -- 172.21.15.164:0/2741507393 reap_dead start 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe3fff640 1 -- 172.21.15.164:0/2741507393 reap_dead start 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 shutdown_connections 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.397+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 >> 172.21.15.164:0/2741507393 conn(0x7fbbe40a7320 msgr2=0x7fbbe4136710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.398+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 shutdown_connections 2024-09-10T16:13:18.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.398+0000 7fbbe9bad640 1 -- 172.21.15.164:0/2741507393 wait complete. 2024-09-10T16:13:18.426 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:18.426 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:13:18.427 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:13:18.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.523+0000 7f703c5f3640 10 monclient: get_monmap_and_config 2024-09-10T16:13:18.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.523+0000 7f703c5f3640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.523+0000 7f703c5f3640 10 monclient: monmap: 2024-09-10T16:13:18.525 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.528 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.528 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:18.524392+0000 2024-09-10T16:13:18.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:18.524392+0000 2024-09-10T16:13:18.528 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:18.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 Processor -- start 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 -- start start 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient: init 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7034137bb0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-b con 0x7f70341386a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f703413d140 0x7f703413f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-c con 0x7f703413d140 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7034073820 con 0x7f7034137bb0 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f70340a1c90 con 0x7f703413d140 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f70340a1f70 con 0x7f70341386a0 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f703c5f3640 10 monclient(hunting): authenticate will time out at 1177.011890s 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.524+0000 7f7039b8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56434/0 (socket says 172.21.15.164:56434) 2024-09-10T16:13:18.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 -- 172.21.15.164:0/2960643213 learned_addr learned my addr 172.21.15.164:0/2960643213 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): get_auth_request con 0x7f70341386a0 auth_method 0 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 1 --2- 172.21.15.164:0/2960643213 >> v2:172.21.15.165:3300/0 conn(0x7f703413d140 0x7f703413f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): get_auth_request con 0x7f703413d140 auth_method 0 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient(hunting): handle_auth_done global_id 4976 payload 274 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 -- 172.21.15.164:0/2960643213 >> v2:172.21.15.165:3300/0 conn(0x7f703413d140 msgr2=0x7f703413f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 --2- 172.21.15.164:0/2960643213 >> v2:172.21.15.165:3300/0 conn(0x7f703413d140 0x7f703413f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 -- 172.21.15.164:0/2960643213 >> v1:172.21.15.165:6789/0 conn(0x7f7034137bb0 legacy=0x7f7034137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 monclient: found mon.noname-b 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.525+0000 7f7039b8e640 1 -- 172.21.15.164:0/2960643213 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70340a2250 con 0x7f70341386a0 2024-09-10T16:13:18.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f7039b8e640 1 -- 172.21.15.164:0/2960643213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 msgr2=0x7f70340a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f7039b8e640 1 --2- 172.21.15.164:0/2960643213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7024009870 tx=0x7f702402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78f71a6a69f3e69c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 1 -- 172.21.15.164:0/2960643213 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024037070 con 0x7f70341386a0 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: dump: 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.541 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 1 -- 172.21.15.164:0/2960643213 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7024032440 con 0x7f70341386a0 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 5 monclient: authenticate success, global_id 4976 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 1 -- 172.21.15.164:0/2960643213 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024032810 con 0x7f70341386a0 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 4 set_mon_vals no callback set 2024-09-10T16:13:18.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703938d640 10 monclient: dump: 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 10 monclient: shutdown 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 -- 172.21.15.164:0/2960643213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 msgr2=0x7f70340a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 --2- 172.21.15.164:0/2960643213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f7024009870 tx=0x7f702402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 -- 172.21.15.164:0/2960643213 shutdown_connections 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 --2- 172.21.15.164:0/2960643213 >> v2:172.21.15.165:3300/0 conn(0x7f703413d140 0x7f703413f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 --2- 172.21.15.164:0/2960643213 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f70340a2e80 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 -- 172.21.15.164:0/2960643213 >> 172.21.15.164:0/2960643213 conn(0x7f70340a7320 msgr2=0x7f70340a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 -- 172.21.15.164:0/2960643213 shutdown_connections 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.528+0000 7f703c5f3640 1 -- 172.21.15.164:0/2960643213 wait complete. 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.529+0000 7f703c5f3640 10 monclient: build_initial_monmap 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.529+0000 7f703c5f3640 10 monclient: monmap: 2024-09-10T16:13:18.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.529+0000 7f703c5f3640 1 Processor -- start 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.529+0000 7f703c5f3640 1 -- start start 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.529+0000 7f703c5f3640 10 monclient: init 2024-09-10T16:13:18.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7034137bb0 0x7f7034133ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-b con 0x7f7034137bb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-c con 0x7f70341386a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 0x7f703413af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): picked mon.noname-a con 0x7f703413d140 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f703413fa70 con 0x7f703413d140 2024-09-10T16:13:18.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7034074030 con 0x7f7034137bb0 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f703413fbf0 con 0x7f70341386a0 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703c5f3640 10 monclient(hunting): authenticate will time out at 1177.017291s 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56440/0 (socket says 172.21.15.164:56440) 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 0x7f703413af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 -- 172.21.15.164:0/1136713685 learned_addr learned my addr 172.21.15.164:0/1136713685 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): get_auth_request con 0x7f70341386a0 auth_method 0 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): get_auth_request con 0x7f703413d140 auth_method 0 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7034137bb0 0x7f7034133ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): get_auth_request con 0x7f7034137bb0 auth_method 0 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703a38f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient(hunting): handle_auth_done global_id 4979 payload 824 2024-09-10T16:13:18.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 -- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7034137bb0 msgr2=0x7f7034133ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7034137bb0 0x7f7034133ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 -- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 msgr2=0x7f703413af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 0x7f703413af00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient: _finish_hunting 0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 monclient: found mon.noname-c 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f7039b8e640 1 -- 172.21.15.164:0/1136713685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f703413cec0 con 0x7f70341386a0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.530+0000 7f703ab90640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 0x7f703413af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.532+0000 7f7039b8e640 1 -- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 msgr2=0x7f7034134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.532+0000 7f7039b8e640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f70240090b0 tx=0x7f70240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024041070 con 0x7f70341386a0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: dump: 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: _finish_auth 0 2024-09-10T16:13:18.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: _check_auth_tickets 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f702403c670 con 0x7f70341386a0 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 5 monclient: authenticate success, global_id 4979 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7024037040 con 0x7f70341386a0 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 10 monclient: _renew_subs 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f703411e920 con 0x7f70341386a0 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f702affd640 10 monclient: dump: 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:18.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f7038b8c640 4 set_mon_vals no callback set 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 10 monclient: _renew_subs 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.533+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f703409e0f0 con 0x7f70341386a0 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7024044070 con 0x7f70341386a0 2024-09-10T16:13:18.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703c5f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703c5f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70340a2e60 con 0x7f70341386a0 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f702affd640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 0x7f7018040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(223..223 src has 184..223) ==== 4815+0+0 (secure 0 0 0) 0x7f70240732d0 con 0x7f70341386a0 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703a38f640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 0x7f7018040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:18.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.534+0000 7f703a38f640 10 monclient: get_auth_request con 0x7f701803e260 auth_method 0 2024-09-10T16:13:18.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.536+0000 7f703a38f640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 0x7f7018040720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7030004f60 tx=0x7f7030007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:18.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.538+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7024071cb0 con 0x7f70341386a0 2024-09-10T16:13:18.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.538+0000 7f702affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:18.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.538+0000 7f702affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:18.674 INFO:tasks.rados.rados.0.smithi165.stdout:908: done (4 left) 2024-09-10T16:13:18.674 INFO:tasks.rados.rados.0.smithi165.stdout:909: delete oid 48 current snap is 98 2024-09-10T16:13:18.674 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 1 to smithi16534841-39 2024-09-10T16:13:18.674 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 2 to smithi16534841-39 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 3 to smithi16534841-39 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 4 to smithi16534841-39 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 5 to smithi16534841-39 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: finishing write tid 6 to smithi16534841-39 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: oid 39 updating version 0 to 521 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: oid 39 updating version 521 to 524 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: oid 39 version 524 is already newer than 523 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: oid 39 version 524 is already newer than 522 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: oid 39 version 524 is already newer than 520 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 524 (ObjNum 342 snap 98 seq_num 342) dirty exists 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:903: left oid 39 (ObjNum 342 snap 98 seq_num 342) 2024-09-10T16:13:18.675 INFO:tasks.rados.rados.0.smithi165.stdout:904: finishing write tid 1 to smithi16534841-42 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: finishing write tid 2 to smithi16534841-42 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: finishing write tid 3 to smithi16534841-42 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: finishing write tid 4 to smithi16534841-42 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: finishing write tid 5 to smithi16534841-42 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: oid 42 updating version 0 to 365 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: oid 42 version 365 is already newer than 364 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: oid 42 version 365 is already newer than 362 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: oid 42 version 365 is already newer than 363 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 365 (ObjNum 343 snap 98 seq_num 343) dirty exists 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:904: left oid 42 (ObjNum 343 snap 98 seq_num 343) 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:906: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:13:18.676 INFO:tasks.rados.rados.0.smithi165.stdout:906: finishing copy_from to smithi16534841-4 2024-09-10T16:13:18.677 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 54 (ObjNum 244 snap 72 seq_num 244) dirty exists 2024-09-10T16:13:18.677 INFO:tasks.rados.rados.0.smithi165.stdout:907: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:13:18.677 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:18.711 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:18.781 INFO:tasks.rados.rados.0.smithi165.stdout:903: done (4 left) 2024-09-10T16:13:18.781 INFO:tasks.rados.rados.0.smithi165.stdout:904: done (3 left) 2024-09-10T16:13:18.781 INFO:tasks.rados.rados.0.smithi165.stdout:906: done (2 left) 2024-09-10T16:13:18.781 INFO:tasks.rados.rados.0.smithi165.stdout:907: done (1 left) 2024-09-10T16:13:18.782 INFO:tasks.rados.rados.0.smithi165.stdout:909: done (0 left) 2024-09-10T16:13:18.782 INFO:tasks.rados.rados.0.smithi165.stdout:910: snap_remove snap 97 2024-09-10T16:13:18.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.791+0000 7f703c5f3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:13:18.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.791+0000 7f703c5f3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:13:18.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.791+0000 7f703c5f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:18.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.791+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f703406d1c0 con 0x7f70341386a0 2024-09-10T16:13:18.794 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:18.794 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=11,logm=62} crc {auth=1207370914,config=2586702516,health=2392545385,kv=1222545535,logm=992924149}) 2024-09-10T16:13:18.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=1,health=12,kv=12,logm=61} crc {auth=1207370914,config=1228700967,health=2301855438,kv=805391194,logm=1929640039}) 2024-09-10T16:13:18.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:18.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=11,logm=62} crc {auth=1207370914,config=2586702516,health=2392545385,kv=1222545535,logm=992924149}) 2024-09-10T16:13:18.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:18.793+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=13,config=2,health=12,kv=12,logm=61} crc {auth=2990036577,config=2586702516,health=2301855438,kv=805391194,logm=1939919986}) 2024-09-10T16:13:18.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.793+0000 7f702affd640 1 -- 172.21.15.164:0/1136713685 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f702403d3f0 con 0x7f70341386a0 2024-09-10T16:13:18.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.793+0000 7f702affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:13:18.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.793+0000 7f702affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:18.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 msgr2=0x7f7018040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 0x7f7018040720 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f7030004f60 tx=0x7f7030007e30 comp rx=0 tx=0).stop 2024-09-10T16:13:18.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 10 monclient: shutdown 2024-09-10T16:13:18.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 msgr2=0x7f7034134400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.795+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f70240090b0 tx=0x7f70240046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 shutdown_connections 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f701803e260 0x7f7018040720 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f703413d140 0x7f703413af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f70341386a0 0x7f7034134400 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 --2- 172.21.15.164:0/1136713685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7034137bb0 0x7f7034133ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 >> 172.21.15.164:0/1136713685 conn(0x7f70340a7320 msgr2=0x7f7034135e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 shutdown_connections 2024-09-10T16:13:18.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:18.796+0000 7f703c5f3640 1 -- 172.21.15.164:0/1136713685 wait complete. 2024-09-10T16:13:18.807 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:13:19.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004110465}) 2024-09-10T16:13:19.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=822720084}) 2024-09-10T16:13:19.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3004110465}) 2024-09-10T16:13:19.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.180+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2601590514}) 2024-09-10T16:13:19.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632722484}) 2024-09-10T16:13:19.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2224202097}) 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632722484}) 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.181+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2993012862}) 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974708992}) 2024-09-10T16:13:19.193 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3883861831}) 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974708992}) 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.182+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4204516540}) 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638323644}) 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1122943051}) 2024-09-10T16:13:19.194 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2638323644}) 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.183+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1966647880}) 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556553038}) 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3326846661}) 2024-09-10T16:13:19.195 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556553038}) 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162521373}) 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896140026}) 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3474501048}) 2024-09-10T16:13:19.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=896140026}) 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.185+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850401184}) 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448151212}) 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3851440075}) 2024-09-10T16:13:19.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448151212}) 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.186+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=888655022}) 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90091417}) 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=942916906}) 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90091417}) 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.187+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3498382846}) 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412040542}) 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3305830420}) 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=412040542}) 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.188+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2713781094}) 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270617866}) 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3166377503}) 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=270617866}) 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2434518603}) 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182585411}) 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2322858426}) 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2182585411}) 2024-09-10T16:13:19.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.190+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883047012}) 2024-09-10T16:13:19.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159351367}) 2024-09-10T16:13:19.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2764314487}) 2024-09-10T16:13:19.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1159351367}) 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.191+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1943313442}) 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970580113}) 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4105458805}) 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=970580113}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.192+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869289514}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846093509}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2294228616}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846093509}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.193+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=999770151}) 2024-09-10T16:13:19.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.194+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913070627}) 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039571950}) 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1913070627}) 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=851862235}) 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101590763}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692615203}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2101590763}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.195+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=1364331591,mdsmap=204415920}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=3} crc {logm=2167358903,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1625125579,mon_config_key=2240728274,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3665319007,osd_snap=1188194725,osdmap=1132463420}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=42,osdmap=4} crc {logm=2167358903,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1625125579,mon_config_key=1698199361,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4124331855,osd_pg_creating=3665319007,osd_snap=20438994,osdmap=2797605105}) 2024-09-10T16:13:19.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=3} crc {logm=2167358903,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1625125579,mon_config_key=2240728274,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3665319007,osd_snap=1188194725,osdmap=1132463420}) 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.197+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=3,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=42,osdmap=2} crc {logm=2167358903,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1625125579,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3665319007,osd_snap=4102658544,osdmap=506533532}) 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=2178797061}) 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=78} crc {osdmap=1372619601}) 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:13:19.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=80} crc {osdmap=2178797061}) 2024-09-10T16:13:19.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:13:19.202+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=80} crc {osdmap=1904151476}) 2024-09-10T16:13:19.808 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:13:19.808 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:19.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 10 monclient: get_monmap_and_config 2024-09-10T16:13:19.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 10 monclient: build_initial_monmap 2024-09-10T16:13:19.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 10 monclient: monmap: 2024-09-10T16:13:19.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:19.917966+0000 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:19.917966+0000 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:19.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 1 Processor -- start 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 1 -- start start 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.917+0000 7f9b41332640 10 monclient: init 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 0x7f9b3c0a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c0a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9b3c137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b3c09ce50 con 0x7f9b3c137ca0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9b3c06db60 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c06cf50 con 0x7f9b3c06cb70 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b41332640 10 monclient(hunting): authenticate will time out at 1178.405415s 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56454/0 (socket says 172.21.15.164:56454) 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 -- 172.21.15.164:0/1960925825 learned_addr learned my addr 172.21.15.164:0/1960925825 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): get_auth_request con 0x7f9b3c06cb70 auth_method 0 2024-09-10T16:13:19.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 1 --2- 172.21.15.164:0/1960925825 >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 0x7f9b3c0a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): get_auth_request con 0x7f9b3c0a46a0 auth_method 0 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:19.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient(hunting): handle_auth_done global_id 4982 payload 274 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 -- 172.21.15.164:0/1960925825 >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 msgr2=0x7f9b3c0a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 --2- 172.21.15.164:0/1960925825 >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 0x7f9b3c0a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 -- 172.21.15.164:0/1960925825 >> v1:172.21.15.165:6789/0 conn(0x7f9b3c137ca0 legacy=0x7f9b3c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient: _finish_hunting 0 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 monclient: found mon.noname-b 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:19.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:19.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3affd640 1 -- 172.21.15.164:0/1960925825 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c09cb70 con 0x7f9b3c06cb70 2024-09-10T16:13:19.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.918+0000 7f9b3a7fc640 1 --2- 172.21.15.164:0/1960925825 >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 0x7f9b3c0a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b3affd640 1 -- 172.21.15.164:0/1960925825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 msgr2=0x7f9b3c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b3affd640 1 --2- 172.21.15.164:0/1960925825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009870 tx=0x7f9b2402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=541833fdb19b5e48 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 1 -- 172.21.15.164:0/1960925825 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24037070 con 0x7f9b3c06cb70 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: dump: 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:19.932 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: _finish_auth 0 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b41332640 5 monclient: authenticate success, global_id 4982 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 1 -- 172.21.15.164:0/1960925825 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b24032440 con 0x7f9b3c06cb70 2024-09-10T16:13:19.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b41332640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 1 -- 172.21.15.164:0/1960925825 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b24032810 con 0x7f9b3c06cb70 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b41332640 4 set_mon_vals no callback set 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.921+0000 7f9b39ffb640 10 monclient: dump: 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.934 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 10 monclient: shutdown 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 -- 172.21.15.164:0/1960925825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 msgr2=0x7f9b3c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 --2- 172.21.15.164:0/1960925825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f9b24009870 tx=0x7f9b2402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 -- 172.21.15.164:0/1960925825 shutdown_connections 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 --2- 172.21.15.164:0/1960925825 >> v2:172.21.15.164:3300/0 conn(0x7f9b3c0a46a0 0x7f9b3c0a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 --2- 172.21.15.164:0/1960925825 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c134f70 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 -- 172.21.15.164:0/1960925825 >> 172.21.15.164:0/1960925825 conn(0x7f9b3c0aa900 msgr2=0x7f9b3c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 -- 172.21.15.164:0/1960925825 shutdown_connections 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 1 -- 172.21.15.164:0/1960925825 wait complete. 2024-09-10T16:13:19.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 10 monclient: build_initial_monmap 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.922+0000 7f9b41332640 10 monclient: monmap: 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 Processor -- start 2024-09-10T16:13:19.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 -- start start 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient: init 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-c con 0x7f9b3c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-a con 0x7f9b3c0a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 0x7f9b3c1cd380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): picked mon.noname-b con 0x7f9b3c137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c1389c0 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3c138840 con 0x7f9b3c137ca0 2024-09-10T16:13:19.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3c140040 con 0x7f9b3c06cb70 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.923+0000 7f9b41332640 10 monclient(hunting): authenticate will time out at 1178.410933s 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46046/0 (socket says 172.21.15.164:46046) 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56466/0 (socket says 172.21.15.164:56466) 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 -- 172.21.15.164:0/2584657552 learned_addr learned my addr 172.21.15.164:0/2584657552 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): get_auth_request con 0x7f9b3c0a46a0 auth_method 0 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:19.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): get_auth_request con 0x7f9b3c06cb70 auth_method 0 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 0x7f9b3c1cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 10 monclient(hunting): get_auth_request con 0x7f9b3c137ca0 auth_method 0 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:19.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient(hunting): handle_auth_done global_id 74124 payload 824 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 msgr2=0x7f9b3c12e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 msgr2=0x7f9b3c1cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 0x7f9b3c1cd380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 monclient: found mon.noname-a 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3a7fc640 1 -- 172.21.15.164:0/2584657552 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b3c0a3420 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:19.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.924+0000 7f9b3b7fe640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 0x7f9b3c1cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b3a7fc640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 msgr2=0x7f9b3c12e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b3a7fc640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00ebe0 tx=0x7f9b2c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c017070 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: dump: 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:19.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2c00cde0 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 5 monclient: authenticate success, global_id 74124 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 10 monclient: _renew_subs 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3c0a3700 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c013690 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:19.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b337fe640 10 monclient: dump: 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 10 monclient: _renew_subs 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:19.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3c0a13a0 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.926+0000 7f9b397fa640 4 set_mon_vals no callback set 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b41332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b41332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b41332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9b2c01a070 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b3c0aa280 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b337fe640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.927+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(224..224 src has 184..224) ==== 4799+0+0 (secure 0 0 0) 0x7f9b2c052c90 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 10 monclient: get_auth_request con 0x7f9b1c03e240 auth_method 0 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 msgr2=0x7f9b1c040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 msgr2=0x7f9b1c040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:13:19.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:19.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.928+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:19.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.932+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9b3c0aa280 con 0x7f9b3c0a46a0 2024-09-10T16:13:19.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.932+0000 7f9b337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:19.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:19.932+0000 7f9b337fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:20.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.130+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.130+0000 7f9b3affd640 10 monclient: get_auth_request con 0x7f9b1c03e240 auth_method 0 2024-09-10T16:13:20.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.132+0000 7f9b3affd640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0a6cc0 tx=0x7f9b2402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.174+0000 7f9b41332640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:20.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.174+0000 7f9b41332640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:20.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.174+0000 7f9b41332640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.174+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9b3c0a8b20 con 0x7f9b3c0a46a0 2024-09-10T16:13:20.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.175+0000 7f9b337fe640 1 -- 172.21.15.164:0/2584657552 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f9b3c0a8b20 con 0x7f9b3c0a46a0 2024-09-10T16:13:20.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.175+0000 7f9b337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:20.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.175+0000 7f9b337fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:20.177 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 msgr2=0x7f9b1c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9b3c0a6cc0 tx=0x7f9b2402e000 comp rx=0 tx=0).stop 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 10 monclient: shutdown 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 msgr2=0x7f9b3c12e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c00ebe0 tx=0x7f9b2c00c520 comp rx=0 tx=0).stop 2024-09-10T16:13:20.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 shutdown_connections 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b1c03e240 0x7f9b1c040700 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b3c137ca0 0x7f9b3c1cd380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b3c0a46a0 0x7f9b3c12e8b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 --2- 172.21.15.164:0/2584657552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b3c06cb70 0x7f9b3c12e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 >> 172.21.15.164:0/2584657552 conn(0x7f9b3c0aa900 msgr2=0x7f9b3c0a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 shutdown_connections 2024-09-10T16:13:20.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.177+0000 7f9b41332640 1 -- 172.21.15.164:0/2584657552 wait complete. 2024-09-10T16:13:20.190 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:13:20.190 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:13:20.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:13:20.190 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:13:20.267 INFO:tasks.rados.rados.0.smithi165.stdout:910: done (0 left) 2024-09-10T16:13:20.267 INFO:tasks.rados.rados.0.smithi165.stdout:911: copy_from oid 6 from oid 43 current snap is 98 2024-09-10T16:13:20.267 INFO:tasks.rados.rados.0.smithi165.stdout:912: rollback oid 14 current snap is 98 2024-09-10T16:13:20.267 INFO:tasks.rados.rados.0.smithi165.stdout:912: done (1 left) 2024-09-10T16:13:20.267 INFO:tasks.rados.rados.0.smithi165.stdout:913: rollback oid 19 current snap is 98 2024-09-10T16:13:20.268 INFO:tasks.rados.rados.0.smithi165.stdout:913: done (1 left) 2024-09-10T16:13:20.268 INFO:tasks.rados.rados.0.smithi165.stdout:914: read oid 4 snap -1 2024-09-10T16:13:20.268 INFO:tasks.rados.rados.0.smithi165.stdout:914: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:13:20.268 INFO:tasks.rados.rados.0.smithi165.stdout:915: write oid 19 current snap is 98 2024-09-10T16:13:20.268 INFO:tasks.rados.rados.0.smithi165.stdout:915: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-10T16:13:20.276 INFO:tasks.rados.rados.0.smithi165.stdout:915: writing smithi16534841-19 from 598415 to 1395205 tid 1 2024-09-10T16:13:20.283 INFO:tasks.rados.rados.0.smithi165.stdout:915: writing smithi16534841-19 from 1922170 to 2667954 tid 2 2024-09-10T16:13:20.288 INFO:tasks.rados.rados.0.smithi165.stdout:915: writing smithi16534841-19 from 3141403 to 3556718 tid 3 2024-09-10T16:13:20.288 INFO:tasks.rados.rados.0.smithi165.stdout:916: read oid 28 snap -1 2024-09-10T16:13:20.289 INFO:tasks.rados.rados.0.smithi165.stdout:916: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:13:20.289 INFO:tasks.rados.rados.0.smithi165.stdout:917: delete oid 24 current snap is 98 2024-09-10T16:13:20.289 INFO:tasks.rados.rados.0.smithi165.stdout:914: expect (ObjNum 244 snap 72 seq_num 244) 2024-09-10T16:13:20.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.300+0000 7f326d1a3640 10 monclient: get_monmap_and_config 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.300+0000 7f326d1a3640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.300+0000 7f326d1a3640 10 monclient: monmap: 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:20.301476+0000 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:20.301476+0000 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:20.302 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:20.303 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 Processor -- start 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 -- start start 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient: init 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f326806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3268134f70 0x7f326813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f3268134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f326813d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f326809ce50 con 0x7f326806cb70 2024-09-10T16:13:20.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3268135380 con 0x7f326813d560 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f326806db60 con 0x7f3268134f70 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.301+0000 7f326d1a3640 10 monclient(hunting): authenticate will time out at 1178.788922s 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f3266d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f326806cb70 0x7f32680a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35044/0 (socket says 172.21.15.164:35044) 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f3266d9d640 1 -- 172.21.15.164:0/3307347201 learned_addr learned my addr 172.21.15.164:0/3307347201 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 1 --2- 172.21.15.164:0/3307347201 >> v2:172.21.15.165:3300/0 conn(0x7f3268134f70 0x7f326813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): get_auth_request con 0x7f3268134f70 auth_method 0 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient(hunting): handle_auth_done global_id 64198 payload 274 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 1 -- 172.21.15.164:0/3307347201 >> v1:172.21.15.165:6789/0 conn(0x7f326813d560 legacy=0x7f326813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 1 -- 172.21.15.164:0/3307347201 >> v1:172.21.15.164:6789/0 conn(0x7f326806cb70 legacy=0x7f32680a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 1 monclient: found mon.noname-c 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:13:20.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.302+0000 7f325ffff640 1 -- 172.21.15.164:0/3307347201 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326809cb70 con 0x7f3268134f70 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f325ffff640 1 -- 172.21.15.164:0/3307347201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3268134f70 msgr2=0x7f326813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f325ffff640 1 --2- 172.21.15.164:0/3307347201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3268134f70 0x7f326813d020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f32500097d0 tx=0x7f325002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a757bf5dc64ce25e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 1 -- 172.21.15.164:0/3307347201 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250037070 con 0x7f3268134f70 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: dump: 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.311 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 1 -- 172.21.15.164:0/3307347201 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3250032440 con 0x7f3268134f70 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 1 -- 172.21.15.164:0/3307347201 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250032810 con 0x7f3268134f70 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326d1a3640 5 monclient: authenticate success, global_id 64198 2024-09-10T16:13:20.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326d1a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326d1a3640 4 set_mon_vals no callback set 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.305+0000 7f326659c640 10 monclient: dump: 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.313 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 10 monclient: shutdown 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 -- 172.21.15.164:0/3307347201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3268134f70 msgr2=0x7f326813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 --2- 172.21.15.164:0/3307347201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3268134f70 0x7f326813d020 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f32500097d0 tx=0x7f325002a000 comp rx=0 tx=0).stop 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 -- 172.21.15.164:0/3307347201 shutdown_connections 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 --2- 172.21.15.164:0/3307347201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3268134f70 0x7f326813d020 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 -- 172.21.15.164:0/3307347201 >> 172.21.15.164:0/3307347201 conn(0x7f32680aa900 msgr2=0x7f32680aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 -- 172.21.15.164:0/3307347201 shutdown_connections 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 1 -- 172.21.15.164:0/3307347201 wait complete. 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.306+0000 7f326d1a3640 10 monclient: monmap: 2024-09-10T16:13:20.314 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 Processor -- start 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 -- start start 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient: init 2024-09-10T16:13:20.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f326806cb70 0x7f32681cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-b con 0x7f326806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-a con 0x7f3268134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f326813d560 0x7f32680a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): picked mon.noname-c con 0x7f326813d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f32680a4db0 con 0x7f3268134f70 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f326806cf50 con 0x7f326806cb70 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3268141070 con 0x7f326813d560 2024-09-10T16:13:20.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326d1a3640 10 monclient(hunting): authenticate will time out at 1178.794751s 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f326813d560 0x7f32680a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46050/0 (socket says 172.21.15.164:46050) 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 1 -- 172.21.15.164:0/1956863721 learned_addr learned my addr 172.21.15.164:0/1956863721 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): get_auth_request con 0x7f3268134f70 auth_method 0 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f326806cb70 0x7f32681cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): get_auth_request con 0x7f326813d560 auth_method 0 2024-09-10T16:13:20.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): get_auth_request con 0x7f326806cb70 auth_method 0 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f325ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f326759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.307+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 10 monclient(hunting): handle_auth_done global_id 74127 payload 824 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f326813d560 msgr2=0x7f32680a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f326813d560 0x7f32680a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f326806cb70 msgr2=0x7f32681cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f326806cb70 0x7f32681cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 monclient: found mon.noname-a 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f325ffff640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326809dbe0 con 0x7f3268134f70 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f326806cb70 0x7f32681cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:20.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.308+0000 7f326759e640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f326813d560 0x7f32680a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:20.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325ffff640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 msgr2=0x7f32680a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325ffff640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3250009720 tx=0x7f32500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250041070 con 0x7f3268134f70 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: dump: 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 5 monclient: authenticate success, global_id 74127 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325003c670 con 0x7f3268134f70 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3250037040 con 0x7f3268134f70 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 10 monclient: _renew_subs 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f326806e720 con 0x7f3268134f70 2024-09-10T16:13:20.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f325f7fe640 10 monclient: dump: 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f3265d9b640 4 set_mon_vals no callback set 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 10 monclient: _renew_subs 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.310+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f326809e180 con 0x7f3268134f70 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f326d1a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f326d1a3640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 0x7f326813a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f32680a46a0 con 0x7f326813a060 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 0x7f326813a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient: get_auth_request con 0x7f326813a060 auth_method 0 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3250044070 con 0x7f3268134f70 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 10 monclient(hunting): handle_auth_done global_id 74130 payload 824 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f3266d9d640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f32680008d0 con 0x7f326813a060 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f325f7fe640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f324803e210 0x7f32480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.311+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(225..225 src has 184..225) ==== 4799+0+0 (secure 0 0 0) 0x7f3250073400 con 0x7f3268134f70 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.312+0000 7f326759e640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f324803e210 0x7f32480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.312+0000 7f326759e640 10 monclient: get_auth_request con 0x7f324803e210 auth_method 0 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f3266d9d640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 msgr2=0x7f326813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 0x7f326813a440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f3258009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258017070 con 0x7f326813a060 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 10 monclient: dump: 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f325800cd70 con 0x7f326813a060 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 msgr2=0x7f326813a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 0x7f326813a440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f3258009f00 comp rx=0 tx=0).stop 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 msgr2=0x7f326813a440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f325f7fe640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a060 0x7f326813a440 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.314+0000 7f326759e640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f324803e210 0x7f32480406d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f326806a520 tx=0x7f3260003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f326d1a3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f326d1a3640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 0x7f326813d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f326d1a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f326813ad90 con 0x7f326813a980 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 0x7f326813d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient: get_auth_request con 0x7f326813a980 auth_method 0 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 10 monclient(hunting): handle_auth_done global_id 74133 payload 824 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.320+0000 7f3266d9d640 1 -- 172.21.15.164:0/1956863721 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f326813ba20 con 0x7f326813a980 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.322+0000 7f3266d9d640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 msgr2=0x7f326813d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.322+0000 7f3266d9d640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 0x7f326813d940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f32580058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3258017070 con 0x7f326813a980 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 10 monclient: dump: 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f3258013680 con 0x7f326813a980 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:20.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 msgr2=0x7f326813d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 0x7f326813d940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f32580058d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 msgr2=0x7f326813d940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f325f7fe640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f326813a980 0x7f326813d940 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f325800eb10 tx=0x7f32580058d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:13:20.334 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 28190, 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.335 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.336 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:20.337 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.338 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:20.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.340 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.341 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.342 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.343 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.344 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.345 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:20.346 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f324803e210 msgr2=0x7f32480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f324803e210 0x7f32480406d0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f326806a520 tx=0x7f3260003d10 comp rx=0 tx=0).stop 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 10 monclient: shutdown 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 msgr2=0x7f32680a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326759e640 1 -- 172.21.15.164:0/1956863721 reap_dead start 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3250009720 tx=0x7f32500046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 shutdown_connections 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 --2- 172.21.15.164:0/1956863721 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3268134f70 0x7f32680a39b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 >> 172.21.15.164:0/1956863721 conn(0x7f32680aa900 msgr2=0x7f32681335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 shutdown_connections 2024-09-10T16:13:20.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.323+0000 7f326d1a3640 1 -- 172.21.15.164:0/1956863721 wait complete. 2024-09-10T16:13:20.349 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:20.349 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:13:20.349 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:13:20.350 INFO:tasks.rados.rados.0.smithi165.stdout:911: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:13:20.350 INFO:tasks.rados.rados.0.smithi165.stdout:914: done (4 left) 2024-09-10T16:13:20.350 INFO:tasks.rados.rados.0.smithi165.stdout:917: done (3 left) 2024-09-10T16:13:20.350 INFO:tasks.rados.rados.0.smithi165.stdout:918: delete oid 16 current snap is 98 2024-09-10T16:13:20.350 INFO:tasks.rados.rados.0.smithi165.stdout:916: expect (ObjNum 115 snap 31 seq_num 115) 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.448+0000 7f13fc440640 10 monclient: get_monmap_and_config 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.448+0000 7f13fc440640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.448+0000 7f13fc440640 10 monclient: monmap: 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:20.449546+0000 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:20.449546+0000 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.450 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:20.451 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:20.451 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:20.451 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:20.451 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:20.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 Processor -- start 2024-09-10T16:13:20.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 -- start start 2024-09-10T16:13:20.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient: init 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f13f409c180 0x7f13f409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-c con 0x7f13f409c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f13f409d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-b con 0x7f13f413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13f409a430 con 0x7f13f409d160 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f13f40a24e0 con 0x7f13f409c180 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13f40a2bf0 con 0x7f13f413a5c0 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.449+0000 7f13fc440640 10 monclient(hunting): authenticate will time out at 1178.936948s 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56490/0 (socket says 172.21.15.164:56490) 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 -- 172.21.15.164:0/2443970706 learned_addr learned my addr 172.21.15.164:0/2443970706 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): get_auth_request con 0x7f13f413a5c0 auth_method 0 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 1 --2- 172.21.15.164:0/2443970706 >> v2:172.21.15.165:3300/0 conn(0x7f13f409c180 0x7f13f409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request con 0x7f13f409c180 auth_method 0 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_done global_id 4991 payload 274 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 -- 172.21.15.164:0/2443970706 >> v2:172.21.15.165:3300/0 conn(0x7f13f409c180 msgr2=0x7f13f409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 --2- 172.21.15.164:0/2443970706 >> v2:172.21.15.165:3300/0 conn(0x7f13f409c180 0x7f13f409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 -- 172.21.15.164:0/2443970706 >> v1:172.21.15.165:6789/0 conn(0x7f13f409d160 legacy=0x7f13f409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 monclient: found mon.noname-b 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.450+0000 7f13fa9dd640 1 -- 172.21.15.164:0/2443970706 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f409c560 con 0x7f13f413a5c0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fa9dd640 1 -- 172.21.15.164:0/2443970706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 msgr2=0x7f13f413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fa9dd640 1 --2- 172.21.15.164:0/2443970706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0098a0 tx=0x7f13ec02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26c657ec3453d94a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 1 -- 172.21.15.164:0/2443970706 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec037070 con 0x7f13f413a5c0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: dump: 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.459 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 1 -- 172.21.15.164:0/2443970706 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec032440 con 0x7f13f413a5c0 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 5 monclient: authenticate success, global_id 4991 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:20.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 1 -- 172.21.15.164:0/2443970706 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec032810 con 0x7f13f413a5c0 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 4 set_mon_vals no callback set 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13f91da640 10 monclient: dump: 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.461 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 10 monclient: shutdown 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 1 -- 172.21.15.164:0/2443970706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 msgr2=0x7f13f413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.453+0000 7f13fc440640 1 --2- 172.21.15.164:0/2443970706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f13ec0098a0 tx=0x7f13ec02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 -- 172.21.15.164:0/2443970706 shutdown_connections 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 --2- 172.21.15.164:0/2443970706 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f413c9b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 --2- 172.21.15.164:0/2443970706 >> v2:172.21.15.165:3300/0 conn(0x7f13f409c180 0x7f13f409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 -- 172.21.15.164:0/2443970706 >> 172.21.15.164:0/2443970706 conn(0x7f13f40a3110 msgr2=0x7f13f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 -- 172.21.15.164:0/2443970706 shutdown_connections 2024-09-10T16:13:20.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 -- 172.21.15.164:0/2443970706 wait complete. 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 10 monclient: monmap: 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:20.463 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.454+0000 7f13fc440640 1 Processor -- start 2024-09-10T16:13:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 -- start start 2024-09-10T16:13:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient: init 2024-09-10T16:13:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f409c180 0x7f13f4092390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-a con 0x7f13f409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13f409d160 0x7f13f40928d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-b con 0x7f13f409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): picked mon.noname-c con 0x7f13f413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4074f90 con 0x7f13f409c180 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f13f4074e10 con 0x7f13f409d160 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13f409f690 con 0x7f13f413a5c0 2024-09-10T16:13:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fc440640 10 monclient(hunting): authenticate will time out at 1178.942505s 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f409c180 0x7f13f4092390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56492/0 (socket says 172.21.15.164:56492) 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 learned_addr learned my addr 172.21.15.164:0/351926003 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): get_auth_request con 0x7f13f413a5c0 auth_method 0 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request con 0x7f13f409c180 auth_method 0 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13f409d160 0x7f13f40928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): get_auth_request con 0x7f13f409d160 auth_method 0 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13f99db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient(hunting): handle_auth_done global_id 4994 payload 824 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13f409d160 msgr2=0x7f13f40928d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f13f409d160 0x7f13f40928d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f409c180 msgr2=0x7f13f4092390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f13f409c180 0x7f13f4092390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 monclient: found mon.noname-c 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.455+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13f4098640 con 0x7f13f413a5c0 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 msgr2=0x7f13f4092e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fa9dd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13ec035040 tx=0x7f13ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec041020 con 0x7f13f413a5c0 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: dump: 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.475 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13ec03d620 con 0x7f13f413a5c0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 5 monclient: authenticate success, global_id 4994 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 10 monclient: _renew_subs 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13f4094b80 con 0x7f13f413a5c0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13ec037040 con 0x7f13f413a5c0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f2ffd640 10 monclient: dump: 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 10 monclient: _renew_subs 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13f89d9640 4 set_mon_vals no callback set 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.458+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f13f4095710 con 0x7f13f413a5c0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f13ec044020 con 0x7f13f413a5c0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fc440640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fc440640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 0x7f13f4070bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13f409d620 con 0x7f13f40707e0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 0x7f13f4070bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient: get_auth_request con 0x7f13f40707e0 auth_method 0 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13f2ffd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e003e210 0x7f13e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(225..225 src has 184..225) ==== 4799+0+0 (secure 0 0 0) 0x7f13ec073070 con 0x7f13f413a5c0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_done global_id 4997 payload 824 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13fa1dc640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f13f409ea40 con 0x7f13f40707e0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13f99db640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e003e210 0x7f13e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.459+0000 7f13f99db640 10 monclient: get_auth_request con 0x7f13e003e210 auth_method 0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.461+0000 7f13fa1dc640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 msgr2=0x7f13f4070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.461+0000 7f13fa1dc640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 0x7f13f4070bc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e400f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e4019070 con 0x7f13f40707e0 2024-09-10T16:13:20.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 10 monclient: dump: 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f99db640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e003e210 0x7f13e00406d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13f4139a80 tx=0x7f13e8007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f13e4004030 con 0x7f13f40707e0 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 msgr2=0x7f13f4070bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 0x7f13f4070bc0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e400f8d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 msgr2=0x7f13f4070bc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.462+0000 7f13f2ffd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f40707e0 0x7f13f4070bc0 secure :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e400f8d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fc440640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fc440640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 0x7f13f41d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fc440640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f13f4077260 con 0x7f13f4071100 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 0x7f13f41d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient: get_auth_request con 0x7f13f4071100 auth_method 0 2024-09-10T16:13:20.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 10 monclient(hunting): handle_auth_done global_id 5000 payload 824 2024-09-10T16:13:20.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.468+0000 7f13fa1dc640 1 -- 172.21.15.164:0/351926003 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f13f4074400 con 0x7f13f4071100 2024-09-10T16:13:20.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.470+0000 7f13fa1dc640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 msgr2=0x7f13f41d77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.470+0000 7f13fa1dc640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 0x7f13f41d77e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13e40208b0 con 0x7f13f4071100 2024-09-10T16:13:20.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 10 monclient: dump: 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f13e401e450 con 0x7f13f4071100 2024-09-10T16:13:20.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 msgr2=0x7f13f41d77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 0x7f13f41d77e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e4004a40 comp rx=0 tx=0).stop 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 msgr2=0x7f13f41d77e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13f2ffd640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f4071100 0x7f13f41d77e0 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f13e400f410 tx=0x7f13e4004a40 comp rx=0 tx=0).stop 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:13:20.485 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 442643, 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.486 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.487 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:20.488 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.489 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:20.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:20.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.495 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:20.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:13:20.497 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:20.498 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e003e210 msgr2=0x7f13e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f13e003e210 0x7f13e00406d0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f13f4139a80 tx=0x7f13e8007e30 comp rx=0 tx=0).stop 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 10 monclient: shutdown 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 msgr2=0x7f13f4092e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fa9dd640 1 -- 172.21.15.164:0/351926003 reap_dead start 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f13ec035040 tx=0x7f13ec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 shutdown_connections 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 --2- 172.21.15.164:0/351926003 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f13f413a5c0 0x7f13f4092e10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 >> 172.21.15.164:0/351926003 conn(0x7f13f40a3110 msgr2=0x7f13f409af40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 shutdown_connections 2024-09-10T16:13:20.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.471+0000 7f13fc440640 1 -- 172.21.15.164:0/351926003 wait complete. 2024-09-10T16:13:20.501 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:20.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:13:20.501 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:911: finishing copy_from to smithi16534841-6 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 259 (ObjNum 311 snap 89 seq_num 311) dirty exists 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: finishing write tid 1 to smithi16534841-19 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: finishing write tid 2 to smithi16534841-19 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: finishing write tid 3 to smithi16534841-19 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: finishing write tid 4 to smithi16534841-19 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: finishing write tid 5 to smithi16534841-19 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: oid 19 updating version 0 to 863 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: oid 19 version 863 is already newer than 862 2024-09-10T16:13:20.515 INFO:tasks.rados.rados.0.smithi165.stdout:915: oid 19 updating version 863 to 864 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:915: oid 19 version 864 is already newer than 861 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 864 (ObjNum 345 snap 98 seq_num 345) dirty exists 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:915: left oid 19 (ObjNum 345 snap 98 seq_num 345) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:911: done (3 left) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:915: done (2 left) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:916: done (1 left) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:918: done (0 left) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:919: rollback oid 39 current snap is 98 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:919: done (0 left) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:920: read oid 3 snap -1 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:920: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:921: copy_from oid 45 from oid 42 current snap is 98 2024-09-10T16:13:20.516 INFO:tasks.rados.rados.0.smithi165.stdout:922: rollback oid 16 current snap is 98 2024-09-10T16:13:20.517 INFO:tasks.rados.rados.0.smithi165.stdout:922: done (2 left) 2024-09-10T16:13:20.517 INFO:tasks.rados.rados.0.smithi165.stdout:923: rollback oid 16 current snap is 98 2024-09-10T16:13:20.517 INFO:tasks.rados.rados.0.smithi165.stdout:923: done (2 left) 2024-09-10T16:13:20.517 INFO:tasks.rados.rados.0.smithi165.stdout:924: snap_create 2024-09-10T16:13:20.521 INFO:tasks.rados.rados.0.smithi165.stdout:920: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:13:20.525 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1889559 2024-09-10T16:13:20.550 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.600+0000 7f8e3c917640 10 monclient: get_monmap_and_config 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.600+0000 7f8e3c917640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.600+0000 7f8e3c917640 10 monclient: monmap: 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:20.601280+0000 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:20.601280+0000 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:20.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.600+0000 7f8e3c917640 1 Processor -- start 2024-09-10T16:13:20.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 -- start start 2024-09-10T16:13:20.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient: init 2024-09-10T16:13:20.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8e38137bb0 0x7f8e380a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-c con 0x7f8e38137bb0 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-b con 0x7f8e38137f90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e381346c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e38073820 con 0x7f8e381346c0 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8e380a1c90 con 0x7f8e38137bb0 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e380a1f70 con 0x7f8e38137f90 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3c917640 10 monclient(hunting): authenticate will time out at 1179.088669s 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56512/0 (socket says 172.21.15.164:56512) 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 1 -- 172.21.15.164:0/558376866 learned_addr learned my addr 172.21.15.164:0/558376866 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): get_auth_request con 0x7f8e38137f90 auth_method 0 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3659c640 1 --2- 172.21.15.164:0/558376866 >> v2:172.21.15.165:3300/0 conn(0x7f8e38137bb0 0x7f8e380a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3659c640 10 monclient(hunting): get_auth_request con 0x7f8e38137bb0 auth_method 0 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.601+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_done global_id 5003 payload 274 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 1 -- 172.21.15.164:0/558376866 >> v2:172.21.15.165:3300/0 conn(0x7f8e38137bb0 msgr2=0x7f8e380a2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/558376866 >> v2:172.21.15.165:3300/0 conn(0x7f8e38137bb0 0x7f8e380a2e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 1 -- 172.21.15.164:0/558376866 >> v1:172.21.15.165:6789/0 conn(0x7f8e381346c0 legacy=0x7f8e38134b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 1 monclient: found mon.noname-b 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.602+0000 7f8e35d9b640 1 -- 172.21.15.164:0/558376866 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e380a2250 con 0x7f8e38137f90 2024-09-10T16:13:20.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.604+0000 7f8e35d9b640 1 -- 172.21.15.164:0/558376866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 msgr2=0x7f8e381383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.604+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/558376866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009870 tx=0x7f8e2002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e7313b6f05c21d9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 1 -- 172.21.15.164:0/558376866 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20037070 con 0x7f8e38137f90 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: dump: 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 1 -- 172.21.15.164:0/558376866 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e20032640 con 0x7f8e38137f90 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 5 monclient: authenticate success, global_id 5003 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 1 -- 172.21.15.164:0/558376866 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20032a10 con 0x7f8e38137f90 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3559a640 10 monclient: dump: 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.617 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 4 set_mon_vals no callback set 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 10 monclient: shutdown 2024-09-10T16:13:20.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 -- 172.21.15.164:0/558376866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 msgr2=0x7f8e381383a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 --2- 172.21.15.164:0/558376866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009870 tx=0x7f8e2002a130 comp rx=0 tx=0).stop 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 -- 172.21.15.164:0/558376866 shutdown_connections 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 --2- 172.21.15.164:0/558376866 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137f90 0x7f8e381383a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 --2- 172.21.15.164:0/558376866 >> v2:172.21.15.165:3300/0 conn(0x7f8e38137bb0 0x7f8e380a2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 -- 172.21.15.164:0/558376866 >> 172.21.15.164:0/558376866 conn(0x7f8e380a7320 msgr2=0x7f8e380a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 -- 172.21.15.164:0/558376866 shutdown_connections 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.605+0000 7f8e3c917640 1 -- 172.21.15.164:0/558376866 wait complete. 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.606+0000 7f8e3c917640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.606+0000 7f8e3c917640 10 monclient: monmap: 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:20.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.606+0000 7f8e3c917640 1 Processor -- start 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.606+0000 7f8e3c917640 1 -- start start 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.606+0000 7f8e3c917640 10 monclient: init 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:20.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381346c0 0x7f8e381cd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-b con 0x7f8e381346c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-c con 0x7f8e38137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38137f90 0x7f8e38121650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): picked mon.noname-a con 0x7f8e38137f90 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8e38077060 con 0x7f8e38137f90 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8e38076ee0 con 0x7f8e381346c0 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e38135260 con 0x7f8e38137bb0 2024-09-10T16:13:20.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3c917640 10 monclient(hunting): authenticate will time out at 1179.094358s 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38137f90 0x7f8e38121650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56528/0 (socket says 172.21.15.164:56528) 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 -- 172.21.15.164:0/491714172 learned_addr learned my addr 172.21.15.164:0/491714172 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): get_auth_request con 0x7f8e38137bb0 auth_method 0 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): get_auth_request con 0x7f8e38137f90 auth_method 0 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381346c0 0x7f8e381cd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): get_auth_request con 0x7f8e381346c0 auth_method 0 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient(hunting): handle_auth_done global_id 5006 payload 824 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381346c0 msgr2=0x7f8e381cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381346c0 0x7f8e381cd550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38137f90 msgr2=0x7f8e38121650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38137f90 0x7f8e38121650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 monclient: found mon.noname-c 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e35d9b640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e380a5770 con 0x7f8e38137bb0 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.607+0000 7f8e36d9d640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e38137f90 0x7f8e38121650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:20.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.609+0000 7f8e35d9b640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 msgr2=0x7f8e38121110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.609+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009870 tx=0x7f8e200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20041020 con 0x7f8e38137bb0 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: dump: 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e2003d850 con 0x7f8e38137bb0 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 5 monclient: authenticate success, global_id 5006 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 monclient: _renew_subs 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e20037040 con 0x7f8e38137bb0 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e38122bd0 con 0x7f8e38137bb0 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e2effd640 10 monclient: dump: 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.627 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e34d99640 4 set_mon_vals no callback set 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 monclient: _renew_subs 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e38124f30 con 0x7f8e38137bb0 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 0x7f8e38121b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.610+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8e3809e0f0 con 0x7f8e381cda90 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e35d9b640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009870 tx=0x7f8e200046c0 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f8e20072ef0 osd_map(225..225 src has 184..225) 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8e20044020 con 0x7f8e38137bb0 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 0x7f8e38121b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient: get_auth_request con 0x7f8e381cda90 auth_method 0 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 10 monclient(hunting): handle_auth_done global_id 64216 payload 824 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e2effd640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e1803e210 0x7f8e180406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e3659c640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e38136fe0 con 0x7f8e381cda90 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e36d9d640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e1803e210 0x7f8e180406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.611+0000 7f8e36d9d640 10 monclient: get_auth_request con 0x7f8e1803e210 auth_method 0 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e3659c640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 msgr2=0x7f8e38121b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e3659c640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 0x7f8e38121b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e3000f8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e30019070 con 0x7f8e381cda90 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e2effd640 10 monclient: dump: 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.613+0000 7f8e36d9d640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e1803e210 0x7f8e180406d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e280042a0 tx=0x7f8e28007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8e30004030 con 0x7f8e381cda90 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 msgr2=0x7f8e38121b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 0x7f8e38121b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e3000f8d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 msgr2=0x7f8e38121b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.614+0000 7f8e2effd640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381cda90 0x7f8e38121b90 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e3000f8d0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.632 INFO:tasks.rados.rados.0.smithi165.stdout:921: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:13:20.632 INFO:tasks.rados.rados.0.smithi165.stdout:921: finishing copy_from to smithi16534841-45 2024-09-10T16:13:20.632 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 365 (ObjNum 343 snap 98 seq_num 343) dirty exists 2024-09-10T16:13:20.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3c917640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3c917640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 0x7f8e3811d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3c917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8e380a2e60 con 0x7f8e381220d0 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 0x7f8e3811d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient: get_auth_request con 0x7f8e381220d0 auth_method 0 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 10 monclient(hunting): handle_auth_done global_id 64219 payload 824 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.620+0000 7f8e3659c640 1 -- 172.21.15.164:0/491714172 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8e38124770 con 0x7f8e381220d0 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.622+0000 7f8e3659c640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 msgr2=0x7f8e3811d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.622+0000 7f8e3659c640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 0x7f8e3811d550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e30004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e300208b0 con 0x7f8e381220d0 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 10 monclient: dump: 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f8e3001e450 con 0x7f8e381220d0 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 msgr2=0x7f8e3811d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 0x7f8e3811d550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e30004a40 comp rx=0 tx=0).stop 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 msgr2=0x7f8e3811d550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:20.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e381220d0 0x7f8e3811d550 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8e3000f410 tx=0x7f8e30004a40 comp rx=0 tx=0).stop 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 68, 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 105051, 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.638 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:20.640 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:20.641 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.642 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:20.643 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:20.644 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:20.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:20.647 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:20.648 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:20.649 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:20.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e1803e210 msgr2=0x7f8e180406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e1803e210 0x7f8e180406d0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e280042a0 tx=0x7f8e28007db0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 10 monclient: shutdown 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 msgr2=0x7f8e38121110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 --2- 172.21.15.164:0/491714172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e38137bb0 0x7f8e38121110 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f8e20009870 tx=0x7f8e200046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e36d9d640 1 -- 172.21.15.164:0/491714172 reap_dead start 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e2effd640 1 -- 172.21.15.164:0/491714172 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(225..225 src has 184..225) ==== 4799+0+0 (unknown 0 0 0) 0x7f8e20072ef0 con 0x7f8e38137bb0 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e36d9d640 1 -- 172.21.15.164:0/491714172 reap_dead start 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 shutdown_connections 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 >> 172.21.15.164:0/491714172 conn(0x7f8e380a7320 msgr2=0x7f8e38137330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 shutdown_connections 2024-09-10T16:13:20.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.623+0000 7f8e3c917640 1 -- 172.21.15.164:0/491714172 wait complete. 2024-09-10T16:13:20.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:13:20.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:13:20.652 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:13:20.652 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:13:20.652 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:13:20.652 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:13:20.652 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:13:20.663 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:13:20.663 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:13:20.663 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:20.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.758+0000 7f40a9cb8640 10 monclient: get_monmap_and_config 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.758+0000 7f40a9cb8640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.758+0000 7f40a9cb8640 10 monclient: monmap: 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:20.759321+0000 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:20.759321+0000 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:20.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:20.761 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:20.761 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:20.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 Processor -- start 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 -- start start 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient: init 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f40a409c180 0x7f40a409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-a con 0x7f40a409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40a409d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:20.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-b con 0x7f40a413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40a409a430 con 0x7f40a409d160 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f40a40a24e0 con 0x7f40a409c180 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40a40a2bf0 con 0x7f40a413a5c0 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a9cb8640 10 monclient(hunting): authenticate will time out at 1179.246785s 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f40a409c180 0x7f40a409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f40a409d160 0x7f40a409ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35052/0 (socket says 172.21.15.164:35052) 2024-09-10T16:13:20.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a2ffd640 1 -- 172.21.15.164:0/3021666590 learned_addr learned my addr 172.21.15.164:0/3021666590 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a3fff640 10 monclient(hunting): get_auth_request con 0x7f40a413a5c0 auth_method 0 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a37fe640 10 monclient(hunting): get_auth_request con 0x7f40a409c180 auth_method 0 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.759+0000 7f40a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient(hunting): handle_auth_done global_id 74142 payload 274 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 1 -- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 msgr2=0x7f40a413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 1 -- 172.21.15.164:0/3021666590 >> v1:172.21.15.164:6789/0 conn(0x7f40a409d160 legacy=0x7f40a409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 1 monclient: found mon.noname-a 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a37fe640 1 -- 172.21.15.164:0/3021666590 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a409c560 con 0x7f40a409c180 2024-09-10T16:13:20.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.760+0000 7f40a3fff640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:20.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.762+0000 7f40a37fe640 1 -- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 msgr2=0x7f40a409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a37fe640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a409ea40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f408c0097d0 tx=0x7f408c02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fdda195ffc2d679 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 1 -- 172.21.15.164:0/3021666590 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c037070 con 0x7f40a409c180 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: dump: 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.779 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 1 -- 172.21.15.164:0/3021666590 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c032440 con 0x7f40a409c180 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 5 monclient: authenticate success, global_id 74142 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 1 -- 172.21.15.164:0/3021666590 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c032810 con 0x7f40a409c180 2024-09-10T16:13:20.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 4 set_mon_vals no callback set 2024-09-10T16:13:20.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:20.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a27fc640 10 monclient: dump: 2024-09-10T16:13:20.782 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.783 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.783 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.783 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.784 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.784 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.784 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.784 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.784 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 10 monclient: shutdown 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 -- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 msgr2=0x7f40a409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a409ea40 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f408c0097d0 tx=0x7f408c02a000 comp rx=0 tx=0).stop 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 -- 172.21.15.164:0/3021666590 shutdown_connections 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/3021666590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a409ea40 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.763+0000 7f40a9cb8640 1 -- 172.21.15.164:0/3021666590 >> 172.21.15.164:0/3021666590 conn(0x7f40a40a3110 msgr2=0x7f40a40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.764+0000 7f40a9cb8640 1 -- 172.21.15.164:0/3021666590 shutdown_connections 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.764+0000 7f40a9cb8640 1 -- 172.21.15.164:0/3021666590 wait complete. 2024-09-10T16:13:20.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.764+0000 7f40a9cb8640 10 monclient: build_initial_monmap 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.764+0000 7f40a9cb8640 10 monclient: monmap: 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:20.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.764+0000 7f40a9cb8640 1 Processor -- start 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 -- start start 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient: init 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-a con 0x7f40a409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 0x7f40a413cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-b con 0x7f40a409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): picked mon.noname-c con 0x7f40a413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:20.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4074f90 con 0x7f40a409c180 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40a4074e10 con 0x7f40a409d160 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40a409f690 con 0x7f40a413a5c0 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a9cb8640 10 monclient(hunting): authenticate will time out at 1179.252534s 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46112/0 (socket says 172.21.15.164:46112) 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 -- 172.21.15.164:0/1652027902 learned_addr learned my addr 172.21.15.164:0/1652027902 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:20.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): get_auth_request con 0x7f40a409c180 auth_method 0 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): get_auth_request con 0x7f40a413a5c0 auth_method 0 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a2ffd640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 msgr2=0x7f40a413cef0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a2ffd640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 0x7f40a413cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient(hunting): handle_auth_done global_id 74145 payload 824 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 msgr2=0x7f40a413d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 msgr2=0x7f40a413cef0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 0x7f40a413cef0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 monclient: found mon.noname-a 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a37fe640 1 -- 172.21.15.164:0/1652027902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40a4092fc0 con 0x7f40a409c180 2024-09-10T16:13:20.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.765+0000 7f40a3fff640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a37fe640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 msgr2=0x7f40a4099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a37fe640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f408c0098f0 tx=0x7f408c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c042020 con 0x7f40a409c180 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: dump: 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.791 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: _finish_auth 0 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 5 monclient: authenticate success, global_id 74145 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c03d880 con 0x7f40a409c180 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c037040 con 0x7f40a409c180 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 10 monclient: _renew_subs 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40a409eff0 con 0x7f40a409c180 2024-09-10T16:13:20.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a1ffb640 4 set_mon_vals no callback set 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f409bfff640 10 monclient: dump: 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:20.793 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 10 monclient: _renew_subs 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.768+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40a4094050 con 0x7f40a409c180 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a9cb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a9cb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f408c045020 con 0x7f40a409c180 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a9cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40a409d930 con 0x7f40a409c180 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f409bfff640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 0x7f4084040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(225..225 src has 184..225) ==== 4799+0+0 (secure 0 0 0) 0x7f408c073330 con 0x7f40a409c180 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a2ffd640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 0x7f4084040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:20.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.769+0000 7f40a2ffd640 10 monclient: get_auth_request con 0x7f408403e1c0 auth_method 0 2024-09-10T16:13:20.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.772+0000 7f40a2ffd640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 0x7f4084040680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4094007fd0 tx=0x7f4094007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:20.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.773+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f408c04c070 con 0x7f40a409c180 2024-09-10T16:13:20.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.774+0000 7f409bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:20.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:20.774+0000 7f409bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f40a9cb8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f40a9cb8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f40a9cb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f40a4071600 con 0x7f40a409c180 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f409bfff640 1 -- 172.21.15.164:0/1652027902 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f40a409eff0 con 0x7f40a409c180 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f409bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.017+0000 7f409bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:21.020 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 msgr2=0x7f4084040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 0x7f4084040680 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4094007fd0 tx=0x7f4094007b20 comp rx=0 tx=0).stop 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 10 monclient: shutdown 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 msgr2=0x7f40a4099aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f408c0098f0 tx=0x7f408c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 shutdown_connections 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e1c0 0x7f4084040680 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40a413a5c0 0x7f40a413d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f40a409d160 0x7f40a413cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:21.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 --2- 172.21.15.164:0/1652027902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40a409c180 0x7f40a4099aa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:21.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 >> 172.21.15.164:0/1652027902 conn(0x7f40a40a3110 msgr2=0x7f40a406e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:21.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 shutdown_connections 2024-09-10T16:13:21.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:21.020+0000 7f40a9cb8640 1 -- 172.21.15.164:0/1652027902 wait complete. 2024-09-10T16:13:24.033 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.141+0000 7f3771389640 10 monclient: get_monmap_and_config 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.141+0000 7f3771389640 10 monclient: build_initial_monmap 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.141+0000 7f3771389640 10 monclient: monmap: 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:24.143 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:24.142820+0000 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:24.142820+0000 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:24.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.142+0000 7f3771389640 1 Processor -- start 2024-09-10T16:13:24.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.142+0000 7f3771389640 1 -- start start 2024-09-10T16:13:24.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.142+0000 7f3771389640 10 monclient: init 2024-09-10T16:13:24.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:24.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f376c06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): picked mon.noname-b con 0x7f376c0a45f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f376c06d660 0x7f376c13fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): picked mon.noname-a con 0x7f376c06d660 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f376c0a2820 con 0x7f376c06cb70 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f376c077350 con 0x7f376c06d660 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f376c06e3b0 con 0x7f376c0a45f0 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f3771389640 10 monclient(hunting): authenticate will time out at 1182.630184s 2024-09-10T16:13:24.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376affd640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f376c06cb70 legacy=0x7f376c06cf50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f376c06d660 0x7f376c13fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56560/0 (socket says 172.21.15.164:56560) 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f376c06d660 0x7f376c13fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46124/0 (socket says 172.21.15.164:46124) 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 1 -- 172.21.15.164:0/2542784833 learned_addr learned my addr 172.21.15.164:0/2542784833 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): get_auth_request con 0x7f376c0a45f0 auth_method 0 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): get_auth_request con 0x7f376c06d660 auth_method 0 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:24.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient(hunting): handle_auth_done global_id 74148 payload 274 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 -- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 msgr2=0x7f376c13d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 -- 172.21.15.164:0/2542784833 >> v1:172.21.15.165:6789/0 conn(0x7f376c06cb70 legacy=0x7f376c06cf50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:24.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376b7fe640 1 -- 172.21.15.164:0/2542784833 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c06e690 con 0x7f376c06d660 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.143+0000 7f376a7fc640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f376b7fe640 1 -- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c06d660 msgr2=0x7f376c13fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f376b7fe640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c06d660 0x7f376c13fce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37640097a0 tx=0x7f3764029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f03870cc520d0928 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 1 -- 172.21.15.164:0/2542784833 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764037070 con 0x7f376c06d660 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: dump: 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:24.153 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: _finish_auth 0 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 1 -- 172.21.15.164:0/2542784833 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3764032440 con 0x7f376c06d660 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:24.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 1 -- 172.21.15.164:0/2542784833 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764032810 con 0x7f376c06d660 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3771389640 5 monclient: authenticate success, global_id 74148 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3771389640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3771389640 4 set_mon_vals no callback set 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.146+0000 7f3769ffb640 10 monclient: dump: 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:24.155 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 10 monclient: shutdown 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 -- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c06d660 msgr2=0x7f376c13fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c06d660 0x7f376c13fce0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f37640097a0 tx=0x7f3764029fd0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 -- 172.21.15.164:0/2542784833 shutdown_connections 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c0a45f0 0x7f376c13d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 --2- 172.21.15.164:0/2542784833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c06d660 0x7f376c13fce0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 -- 172.21.15.164:0/2542784833 >> 172.21.15.164:0/2542784833 conn(0x7f376c0a4ee0 msgr2=0x7f376c0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 -- 172.21.15.164:0/2542784833 shutdown_connections 2024-09-10T16:13:24.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 1 -- 172.21.15.164:0/2542784833 wait complete. 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 10 monclient: build_initial_monmap 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.147+0000 7f3771389640 10 monclient: monmap: 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 Processor -- start 2024-09-10T16:13:24.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 -- start start 2024-09-10T16:13:24.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient: init 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 0x7f376c138930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): picked mon.noname-b con 0x7f376c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): picked mon.noname-c con 0x7f376c06d660 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): picked mon.noname-a con 0x7f376c0a45f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0a0240 con 0x7f376c0a45f0 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f376c0a00c0 con 0x7f376c06cb70 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f376c140220 con 0x7f376c06d660 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.148+0000 7f3771389640 10 monclient(hunting): authenticate will time out at 1182.635920s 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:24.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376affd640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 msgr2=0x7f376c138930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46138/0 (socket says 172.21.15.164:46138) 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56576/0 (socket says 172.21.15.164:56576) 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 -- 172.21.15.164:0/1503131648 learned_addr learned my addr 172.21.15.164:0/1503131648 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): get_auth_request con 0x7f376c0a45f0 auth_method 0 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376affd640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 0x7f376c138930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): get_auth_request con 0x7f376c06d660 auth_method 0 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:24.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient(hunting): handle_auth_done global_id 74151 payload 824 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 -- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 msgr2=0x7f376c138e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 -- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 msgr2=0x7f376c138930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 0x7f376c138930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:13:24.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376b7fe640 1 -- 172.21.15.164:0/1503131648 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f376c09f5d0 con 0x7f376c0a45f0 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.149+0000 7f376a7fc640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f376b7fe640 1 -- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 msgr2=0x7f376c13aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f376b7fe640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37640098c0 tx=0x7f37640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764041070 con 0x7f376c0a45f0 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: dump: 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:24.166 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 5 monclient: authenticate success, global_id 74151 2024-09-10T16:13:24.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f376403c870 con 0x7f376c0a45f0 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3764037040 con 0x7f376c0a45f0 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 10 monclient: _renew_subs 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f376c13c130 con 0x7f376c0a45f0 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37537fe640 10 monclient: dump: 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:24.168 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f37697fa640 4 set_mon_vals no callback set 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 10 monclient: _renew_subs 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.151+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f376c13dbf0 con 0x7f376c0a45f0 2024-09-10T16:13:24.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.152+0000 7f3771389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.152+0000 7f3771389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.152+0000 7f3771389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.152+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f376c0a92f0 con 0x7f376c0a45f0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.152+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3764044070 con 0x7f376c0a45f0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.153+0000 7f37537fe640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 0x7f3758040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.153+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(225..225 src has 184..225) ==== 4799+0+0 (secure 0 0 0) 0x7f37640734d0 con 0x7f376c0a45f0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.153+0000 7f376affd640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 0x7f3758040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.153+0000 7f376affd640 10 monclient: get_auth_request con 0x7f375803e1c0 auth_method 0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.155+0000 7f376affd640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 0x7f3758040680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f375c003cb0 tx=0x7f375c009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.157+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f376c0a92f0 con 0x7f376c0a45f0 2024-09-10T16:13:24.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.157+0000 7f37537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:24.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.157+0000 7f37537fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:24.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f3771389640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f3771389640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f3771389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f376c13ddf0 con 0x7f376c0a45f0 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f37537fe640 1 -- 172.21.15.164:0/1503131648 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f376c13ddf0 con 0x7f376c0a45f0 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f37537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.401+0000 7f37537fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:24.403 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 msgr2=0x7f3758040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 0x7f3758040680 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f375c003cb0 tx=0x7f375c009270 comp rx=0 tx=0).stop 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 10 monclient: shutdown 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 msgr2=0x7f376c13aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:24.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f37640098c0 tx=0x7f37640046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 shutdown_connections 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f375803e1c0 0x7f3758040680 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f376c0a45f0 0x7f376c13aa20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f376c06d660 0x7f376c138e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 --2- 172.21.15.164:0/1503131648 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f376c06cb70 0x7f376c138930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 >> 172.21.15.164:0/1503131648 conn(0x7f376c0a4ee0 msgr2=0x7f376c0a1280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 shutdown_connections 2024-09-10T16:13:24.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:24.403+0000 7f3771389640 1 -- 172.21.15.164:0/1503131648 wait complete. 2024-09-10T16:13:27.417 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 10 monclient: get_monmap_and_config 2024-09-10T16:13:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 10 monclient: build_initial_monmap 2024-09-10T16:13:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 10 monclient: monmap: 2024-09-10T16:13:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:27.525909+0000 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:27.525909+0000 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:27.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:27.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 1 Processor -- start 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 1 -- start start 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.525+0000 7f6cab60c640 10 monclient: init 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ca4137bb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ca41386a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca40a2e80 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a1c90 con 0x7f6ca4137bb0 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a1f70 con 0x7f6ca41386a0 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6ca40a2250 con 0x7f6ca40a2e80 2024-09-10T16:13:27.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6cab60c640 10 monclient(hunting): authenticate will time out at 1186.013441s 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6ca9ba9640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6ca4140870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6ca8ba7640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6ca413d140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6ca93a8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6ca4137bb0 0x7f6ca4137f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36578/0 (socket says 172.21.15.164:36578) 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6ca9ba9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.526+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 learned_addr learned my addr 172.21.15.164:0/1301005672 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:27.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.528+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.164:6789/0 conn(0x7f6ca4137bb0 legacy=0x7f6ca4137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.728+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6ca4140870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:27.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.728+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6ca413d140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:27.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:27.728+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:28.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.129+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6ca413d140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:28.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.129+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6ca4140870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:28.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.129+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:28.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.931+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6ca413d140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:28.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.931+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6ca4140870 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:28.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:28.931+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): tick 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6ca4140870 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:30.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6ca4140870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6ca413d140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.164:6789/0 conn(0x7f6ca4137bb0 legacy=0x7f6ca4137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient: _add_conns ranks=[2,3] 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f6c90003770 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f6c900062a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6ca40a2250 con 0x7f6c900062a0 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6c90008e20 con 0x7f6c90003770 2024-09-10T16:13:30.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request con 0x7f6c90003770 auth_method 0 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca93a8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.526+0000 7f6ca93a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:30.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:30.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:30.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:30.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:30.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:30.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.527+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:30.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:30.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:30.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:30.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:30.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request con 0x7f6c90003770 auth_method 0 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:30.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:30.728+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:31.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:31.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:31.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:31.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:31.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:31.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request con 0x7f6c90003770 auth_method 0 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:31.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:31.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.129+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:31.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request con 0x7f6c90003770 auth_method 0 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:31.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:31.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:31.931+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): tick 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 msgr2=0x7f6c90005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c90003770 0x7f6c90005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:33.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 reap_dead start 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca40a2e80 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f6c900062a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ca41386a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a3310 con 0x7f6ca41386a0 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6ca40a2250 con 0x7f6c900062a0 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6c90008e20 con 0x7f6ca40a2e80 2024-09-10T16:13:33.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6c9000d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6c9000b290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:33.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.527+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6c9000b290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6c9000d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.728+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:33.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.729+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.729+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.729+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:33.729+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:13:34.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.129+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6c9000d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:34.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.129+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6c9000b290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:34.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.129+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:34.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:34.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:34.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:34.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.130+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:13:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.930+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6c9000b290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.930+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6c9000d820 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.930+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6c900062a0 auth_method 0 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:13:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:34.932+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:13:36.290 INFO:tasks.rados.rados.0.smithi165.stdout:920: done (2 left) 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:921: done (1 left) 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:924: done (0 left) 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:925: read oid 11 snap -1 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:925: expect deleted 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:926: write (excl) oid 41 current snap is 99 2024-09-10T16:13:36.291 INFO:tasks.rados.rados.0.smithi165.stdout:926: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-10T16:13:36.297 INFO:tasks.rados.rados.0.smithi165.stdout:926: writing smithi16534841-41 from 648836 to 1271373 tid 1 2024-09-10T16:13:36.303 INFO:tasks.rados.rados.0.smithi165.stdout:926: writing smithi16534841-41 from 1981263 to 2474374 tid 2 2024-09-10T16:13:36.311 INFO:tasks.rados.rados.0.smithi165.stdout:926: writing smithi16534841-41 from 3107319 to 3848836 tid 3 2024-09-10T16:13:36.312 INFO:tasks.rados.rados.0.smithi165.stdout:927: read oid 33 snap -1 2024-09-10T16:13:36.312 INFO:tasks.rados.rados.0.smithi165.stdout:927: expect deleted 2024-09-10T16:13:36.312 INFO:tasks.rados.rados.0.smithi165.stdout:928: delete oid 22 current snap is 99 2024-09-10T16:13:36.312 INFO:tasks.rados.rados.0.smithi165.stdout:926: finishing write tid 1 to smithi16534841-41 2024-09-10T16:13:36.312 INFO:tasks.rados.rados.0.smithi165.stdout:926: finishing write tid 2 to smithi16534841-41 2024-09-10T16:13:36.316 INFO:tasks.rados.rados.0.smithi165.stdout:926: finishing write tid 3 to smithi16534841-41 2024-09-10T16:13:36.317 INFO:tasks.rados.rados.0.smithi165.stdout:925: done (3 left) 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:927: done (2 left) 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:928: done (1 left) 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:929: snap_create 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:926: finishing write tid 4 to smithi16534841-41 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:926: finishing write tid 5 to smithi16534841-41 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:926: oid 41 updating version 0 to 538 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:926: oid 41 updating version 538 to 541 2024-09-10T16:13:36.318 INFO:tasks.rados.rados.0.smithi165.stdout:926: oid 41 version 541 is already newer than 540 2024-09-10T16:13:36.319 INFO:tasks.rados.rados.0.smithi165.stdout:926: oid 41 version 541 is already newer than 539 2024-09-10T16:13:36.319 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 541 (ObjNum 348 snap 99 seq_num 348) dirty exists 2024-09-10T16:13:36.319 INFO:tasks.rados.rados.0.smithi165.stdout:926: left oid 41 (ObjNum 348 snap 99 seq_num 348) 2024-09-10T16:13:36.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): tick 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 msgr2=0x7f6c9000b290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.165:3300/0 conn(0x7f6ca40a2e80 0x7f6c9000b290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 msgr2=0x7f6c90008760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> v2:172.21.15.164:3300/0 conn(0x7f6c900062a0 0x7f6c90008760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.165:6789/0 conn(0x7f6ca41386a0 legacy=0x7f6c9000d820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient: _add_conns ranks=[2,0] 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 0x7f6c9000f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9000efd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:36.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c90005d10 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca40a1f70 con 0x7f6c90005d10 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 1 -- 172.21.15.164:0/1301005672 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca40a2250 con 0x7f6c9000efd0 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 0x7f6c9000f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request con 0x7f6c9000efd0 auth_method 0 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.527+0000 7f6ca93a8640 10 monclient(hunting): handle_auth_done global_id 5021 payload 274 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> v1:172.21.15.164:6789/0 conn(0x7f6c90005d10 legacy=0x7f6c90011fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 10 monclient: _finish_hunting 0 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 1 monclient: found mon.noname-b 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.528+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca40a3310 con 0x7f6c9000efd0 2024-09-10T16:13:36.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.530+0000 7f6ca93a8640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 msgr2=0x7f6c9000f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.530+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 0x7f6c9000f3b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6c94001410 tx=0x7f6c9402b5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=feef490e23936e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 1 -- 172.21.15.164:0/1301005672 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c94039070 con 0x7f6c9000efd0 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: dump: 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.534 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: _finish_auth 0 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 1 -- 172.21.15.164:0/1301005672 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c94034410 con 0x7f6c9000efd0 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 1 -- 172.21.15.164:0/1301005672 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c940345e0 con 0x7f6c9000efd0 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6ca3fff640 10 monclient: dump: 2024-09-10T16:13:36.535 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6cab60c640 5 monclient: authenticate success, global_id 5021 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6cab60c640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.531+0000 7f6cab60c640 4 set_mon_vals no callback set 2024-09-10T16:13:36.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 10 monclient: shutdown 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 -- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 msgr2=0x7f6c9000f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 --2- 172.21.15.164:0/1301005672 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6c9000efd0 0x7f6c9000f3b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f6c94001410 tx=0x7f6c9402b5b0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/1301005672 reap_dead start 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 -- 172.21.15.164:0/1301005672 shutdown_connections 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 -- 172.21.15.164:0/1301005672 >> 172.21.15.164:0/1301005672 conn(0x7f6ca40a7320 msgr2=0x7f6ca40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 -- 172.21.15.164:0/1301005672 shutdown_connections 2024-09-10T16:13:36.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.532+0000 7f6cab60c640 1 -- 172.21.15.164:0/1301005672 wait complete. 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.533+0000 7f6cab60c640 10 monclient: build_initial_monmap 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.533+0000 7f6cab60c640 10 monclient: monmap: 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:36.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.533+0000 7f6cab60c640 1 Processor -- start 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.533+0000 7f6cab60c640 1 -- start start 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.533+0000 7f6cab60c640 10 monclient: init 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 0x7f6ca41cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-b con 0x7f6c9000efd0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:36.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-a con 0x7f6ca4124840 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 0x7f6ca4122690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): picked mon.noname-c con 0x7f6ca4122220 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca4077060 con 0x7f6ca4124840 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca4076ee0 con 0x7f6c9000efd0 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca4137f90 con 0x7f6ca4122220 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6cab60c640 10 monclient(hunting): authenticate will time out at 1195.021272s 2024-09-10T16:13:36.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca93a8640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 msgr2=0x7f6ca41cd4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 0x7f6ca4122690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:50060/0 (socket says 172.21.15.164:50060) 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 1 -- 172.21.15.164:0/580382015 learned_addr learned my addr 172.21.15.164:0/580382015 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): get_auth_request con 0x7f6ca4122220 auth_method 0 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 0x7f6ca41cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request con 0x7f6ca4124840 auth_method 0 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient(hunting): handle_auth_done global_id 5024 payload 824 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 msgr2=0x7f6ca41cd4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 0x7f6ca41cd4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 msgr2=0x7f6ca41cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient: _finish_hunting 0 2024-09-10T16:13:36.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 monclient: found mon.noname-c 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/580382015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca4123ce0 con 0x7f6ca4122220 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.534+0000 7f6ca8ba7640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca9ba9640 1 -- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 msgr2=0x7f6ca4122690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca9ba9640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 0x7f6ca4122690 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6ca409dfb0 tx=0x7f6c9c00b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c006230 con 0x7f6ca4122220 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: dump: 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9c003e50 con 0x7f6ca4122220 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 5 monclient: authenticate success, global_id 5024 2024-09-10T16:13:36.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 10 monclient: _renew_subs 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca4126040 con 0x7f6ca4122220 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c9c006630 con 0x7f6ca4122220 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca1ffb640 10 monclient: dump: 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.549 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 10 monclient: _renew_subs 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6ca37fe640 4 set_mon_vals no callback set 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.537+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca4122bd0 con 0x7f6ca4122220 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.538+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6c9c010070 con 0x7f6ca4122220 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.538+0000 7f6ca1ffb640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 0x7f6c90041d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.538+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(226..226 src has 184..226) ==== 4771+0+0 (secure 0 0 0) 0x7f6c9c0520d0 con 0x7f6ca4122220 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6cab60c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6cab60c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:36.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6cab60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ca41382e0 con 0x7f6ca4122220 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 0x7f6c90041d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.539+0000 7f6ca93a8640 10 monclient: get_auth_request con 0x7f6c9003f8d0 auth_method 0 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.541+0000 7f6ca93a8640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 0x7f6c90041d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c94033000 tx=0x7f6c9402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.544+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6c9c013c20 con 0x7f6ca4122220 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.544+0000 7f6ca1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:36.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.544+0000 7f6ca1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:36.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.789+0000 7f6cab60c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.789+0000 7f6cab60c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.789+0000 7f6cab60c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.789+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ca406d1c0 con 0x7f6ca4122220 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.790+0000 7f6ca1ffb640 1 -- 172.21.15.164:0/580382015 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f6c9c0113b0 con 0x7f6ca4122220 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.790+0000 7f6ca1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.790+0000 7f6ca1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:36.792 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 msgr2=0x7f6c90041d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 0x7f6c90041d90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6c94033000 tx=0x7f6c9402f000 comp rx=0 tx=0).stop 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 10 monclient: shutdown 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 msgr2=0x7f6ca4122690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 0x7f6ca4122690 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f6ca409dfb0 tx=0x7f6c9c00b870 comp rx=0 tx=0).stop 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 shutdown_connections 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6c9003f8d0 0x7f6c90041d90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ca4122220 0x7f6ca4122690 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ca4124840 0x7f6ca41cda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 --2- 172.21.15.164:0/580382015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6c9000efd0 0x7f6ca41cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 >> 172.21.15.164:0/580382015 conn(0x7f6ca40a7320 msgr2=0x7f6ca40a0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:36.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 shutdown_connections 2024-09-10T16:13:36.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.792+0000 7f6cab60c640 1 -- 172.21.15.164:0/580382015 wait complete. 2024-09-10T16:13:36.804 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:13:36.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:13:36.805 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:13:36.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.902+0000 7f64501a0640 10 monclient: get_monmap_and_config 2024-09-10T16:13:36.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.902+0000 7f64501a0640 10 monclient: build_initial_monmap 2024-09-10T16:13:36.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.902+0000 7f64501a0640 10 monclient: monmap: 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:36.903133+0000 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:36.903133+0000 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:36.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.902+0000 7f64501a0640 1 Processor -- start 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 -- start start 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient: init 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f644806cb70 0x7f6448134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-a con 0x7f644806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64480a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:13:36.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 0x7f64481383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-c con 0x7f64480a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f644809ce50 con 0x7f64480a46a0 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f644806db60 con 0x7f644806cb70 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f644806cf50 con 0x7f64480a4ab0 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f64501a0640 10 monclient(hunting): authenticate will time out at 1195.390522s 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f644806cb70 0x7f6448134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644d73b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f64480a46a0 0x7f6448137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:56712/0 (socket says 172.21.15.164:56712) 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644d73b640 1 -- 172.21.15.164:0/2627507944 learned_addr learned my addr 172.21.15.164:0/2627507944 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:13:36.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644e73d640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 msgr2=0x7f64481383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): get_auth_request con 0x7f644806cb70 auth_method 0 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644e73d640 1 --2- 172.21.15.164:0/2627507944 >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 0x7f64481383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient(hunting): handle_auth_done global_id 74157 payload 274 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 -- 172.21.15.164:0/2627507944 >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 msgr2=0x7f64481383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:36.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 --2- 172.21.15.164:0/2627507944 >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 0x7f64481383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 -- 172.21.15.164:0/2627507944 >> v1:172.21.15.164:6789/0 conn(0x7f64480a46a0 legacy=0x7f6448137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient: _finish_hunting 0 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 monclient: found mon.noname-a 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:36.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.903+0000 7f644df3c640 1 -- 172.21.15.164:0/2627507944 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644809cb70 con 0x7f644806cb70 2024-09-10T16:13:36.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644df3c640 1 -- 172.21.15.164:0/2627507944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 msgr2=0x7f6448134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644df3c640 1 --2- 172.21.15.164:0/2627507944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f6448134f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f64400097d0 tx=0x7f644002a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ec4ac03a8e036c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 1 -- 172.21.15.164:0/2627507944 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440037070 con 0x7f644806cb70 2024-09-10T16:13:36.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: dump: 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: _finish_auth 0 2024-09-10T16:13:36.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: _check_auth_tickets 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 1 -- 172.21.15.164:0/2627507944 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6440032640 con 0x7f644806cb70 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f64501a0640 5 monclient: authenticate success, global_id 74157 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f64501a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 1 -- 172.21.15.164:0/2627507944 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644002ae60 con 0x7f644806cb70 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f64501a0640 4 set_mon_vals no callback set 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.906+0000 7f644cf3a640 10 monclient: dump: 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.919 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 10 monclient: shutdown 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 -- 172.21.15.164:0/2627507944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 msgr2=0x7f6448134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 --2- 172.21.15.164:0/2627507944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f6448134f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f64400097d0 tx=0x7f644002a060 comp rx=0 tx=0).stop 2024-09-10T16:13:36.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 -- 172.21.15.164:0/2627507944 shutdown_connections 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 --2- 172.21.15.164:0/2627507944 >> v2:172.21.15.165:3300/0 conn(0x7f64480a4ab0 0x7f64481383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 --2- 172.21.15.164:0/2627507944 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f6448134f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 -- 172.21.15.164:0/2627507944 >> 172.21.15.164:0/2627507944 conn(0x7f64480aa900 msgr2=0x7f64480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 -- 172.21.15.164:0/2627507944 shutdown_connections 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.907+0000 7f64501a0640 1 -- 172.21.15.164:0/2627507944 wait complete. 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient: build_initial_monmap 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient: monmap: 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 Processor -- start 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 -- start start 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient: init 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f64480a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-a con 0x7f644806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64480a46a0 0x7f644812e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-c con 0x7f64480a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:36.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64480a4ab0 0x7f644812e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): picked mon.noname-b con 0x7f64480a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6448138a70 con 0x7f644806cb70 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f64481388f0 con 0x7f64480a4ab0 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6448140040 con 0x7f64480a46a0 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f64501a0640 10 monclient(hunting): authenticate will time out at 1195.395879s 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f64480a3d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f64480a3d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:50096/0 (socket says 172.21.15.164:50096) 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 1 -- 172.21.15.164:0/707195051 learned_addr learned my addr 172.21.15.164:0/707195051 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:36.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644d73b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64480a46a0 0x7f644812e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 10 monclient(hunting): get_auth_request con 0x7f644806cb70 auth_method 0 2024-09-10T16:13:36.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.908+0000 7f644df3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): get_auth_request con 0x7f64480a46a0 auth_method 0 2024-09-10T16:13:36.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644e73d640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64480a4ab0 msgr2=0x7f644812e9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644e73d640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64480a4ab0 0x7f644812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient(hunting): handle_auth_done global_id 74160 payload 824 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64480a46a0 msgr2=0x7f644812e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64480a46a0 0x7f644812e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64480a4ab0 msgr2=0x7f644812e9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:13:36.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64480a4ab0 0x7f644812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient: _finish_hunting 0 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 monclient: found mon.noname-a 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.909+0000 7f644df3c640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6448133fd0 con 0x7f644806cb70 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f644df3c640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 msgr2=0x7f64480a3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f644df3c640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f64480a3d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6440009720 tx=0x7f64400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6440037040 con 0x7f644806cb70 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: dump: 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.929 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.930 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: _finish_auth 0 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: _check_auth_tickets 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6440004030 con 0x7f644806cb70 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 5 monclient: authenticate success, global_id 74160 2024-09-10T16:13:36.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 10 monclient: _renew_subs 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644806e720 con 0x7f644806cb70 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644002ae60 con 0x7f644806cb70 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64467fc640 10 monclient: dump: 2024-09-10T16:13:36.932 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.933 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.934 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 10 monclient: _renew_subs 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64480a43d0 con 0x7f644806cb70 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.911+0000 7f6447fff640 4 set_mon_vals no callback set 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64501a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:36.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64501a0640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 0x7f64481cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6448134f70 con 0x7f64481cd2d0 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f644003a070 con 0x7f644806cb70 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 0x7f64481cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient: get_auth_request con 0x7f64481cd2d0 auth_method 0 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64467fc640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f643403e210 0x7f64340406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 10 monclient(hunting): handle_auth_done global_id 74163 payload 824 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(226..226 src has 184..226) ==== 4771+0+0 (secure 0 0 0) 0x7f6440073200 con 0x7f644806cb70 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.912+0000 7f644d73b640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f64480a2dd0 con 0x7f64481cd2d0 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.913+0000 7f644e73d640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f643403e210 0x7f64340406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.913+0000 7f644e73d640 10 monclient: get_auth_request con 0x7f643403e210 auth_method 0 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f644d73b640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 msgr2=0x7f64481cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f644d73b640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 0x7f64481cd6b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6448134690 tx=0x7f643c00f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c017030 con 0x7f64481cd2d0 2024-09-10T16:13:36.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 10 monclient: dump: 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f643c006020 con 0x7f64481cd2d0 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 msgr2=0x7f64481cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 0x7f64481cd6b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6448134690 tx=0x7f643c00f5a0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 msgr2=0x7f64481cd6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:36.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f64467fc640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cd2d0 0x7f64481cd6b0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6448134690 tx=0x7f643c00f5a0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.915+0000 7f644e73d640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f643403e210 0x7f64340406d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6438003930 tx=0x7f6438006c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f64501a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f64501a0640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 0x7f6448130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f64501a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f64480a30b0 con 0x7f64481cdbf0 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 0x7f6448130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient: get_auth_request con 0x7f64481cdbf0 auth_method 0 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:36.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 10 monclient(hunting): handle_auth_done global_id 74166 payload 824 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.921+0000 7f644d73b640 1 -- 172.21.15.164:0/707195051 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f64480770f0 con 0x7f64481cdbf0 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f644d73b640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 msgr2=0x7f6448130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f644d73b640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 0x7f6448130250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f643c017000 tx=0x7f643c004cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c01ed80 con 0x7f64481cdbf0 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 10 monclient: dump: 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:36.944 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f643c01ed80 con 0x7f64481cdbf0 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 msgr2=0x7f6448130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 0x7f6448130250 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f643c017000 tx=0x7f643c004cc0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 msgr2=0x7f6448130250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:36.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64467fc640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64481cdbf0 0x7f6448130250 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f643c017000 tx=0x7f643c004cc0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 72, 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 44791, 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:13:36.946 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:36.947 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:36.948 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:36.949 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:36.950 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:36.951 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:36.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:36.953 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:36.954 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:36.955 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.956 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:36.957 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f643403e210 msgr2=0x7f64340406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f643403e210 0x7f64340406d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6438003930 tx=0x7f6438006c70 comp rx=0 tx=0).stop 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 10 monclient: shutdown 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 msgr2=0x7f64480a3d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 --2- 172.21.15.164:0/707195051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f644806cb70 0x7f64480a3d20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6440009720 tx=0x7f64400046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f644e73d640 1 -- 172.21.15.164:0/707195051 reap_dead start 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f644e73d640 1 -- 172.21.15.164:0/707195051 reap_dead start 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 shutdown_connections 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 >> 172.21.15.164:0/707195051 conn(0x7f64480aa900 msgr2=0x7f64480a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 shutdown_connections 2024-09-10T16:13:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:36.924+0000 7f64501a0640 1 -- 172.21.15.164:0/707195051 wait complete. 2024-09-10T16:13:36.960 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:13:36.960 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:13:36.960 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.058+0000 7f7e2014b640 10 monclient: get_monmap_and_config 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.058+0000 7f7e2014b640 10 monclient: build_initial_monmap 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.058+0000 7f7e2014b640 10 monclient: monmap: 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:37.059660+0000 2024-09-10T16:13:37.060 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:37.059660+0000 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:37.061 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.059+0000 7f7e2014b640 1 Processor -- start 2024-09-10T16:13:37.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.059+0000 7f7e2014b640 1 -- start start 2024-09-10T16:13:37.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.059+0000 7f7e2014b640 10 monclient: init 2024-09-10T16:13:37.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 0x7f7e18134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1806d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7e180a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e1809cb90 con 0x7f7e180a4c20 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7e1809ce70 con 0x7f7e1806d160 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1809d150 con 0x7f7e1806d540 2024-09-10T16:13:37.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e2014b640 10 monclient(hunting): authenticate will time out at 1195.547177s 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1e6e8640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f7e180a4c20 legacy=0x7f7e18137ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1dee7640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 msgr2=0x7f7e18134e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1dee7640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 0x7f7e18134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35622/0 (socket says 172.21.15.164:35622) 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/3207141827 learned_addr learned my addr 172.21.15.164:0/3207141827 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): get_auth_request con 0x7f7e1806d540 auth_method 0 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:37.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_done global_id 5030 payload 274 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/3207141827 >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 msgr2=0x7f7e18134e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/3207141827 >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 0x7f7e18134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/3207141827 >> v1:172.21.15.165:6789/0 conn(0x7f7e180a4c20 legacy=0x7f7e18137ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient: _finish_hunting 0 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 monclient: found mon.noname-b 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:13:37.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.060+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/3207141827 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e18135340 con 0x7f7e1806d540 2024-09-10T16:13:37.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/3207141827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 msgr2=0x7f7e180a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/3207141827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009870 tx=0x7f7e0c02a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=da8392bffe241c21 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 1 -- 172.21.15.164:0/3207141827 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c037070 con 0x7f7e1806d540 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: dump: 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: _finish_auth 0 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: _check_auth_tickets 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 1 -- 172.21.15.164:0/3207141827 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e0c032640 con 0x7f7e1806d540 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e2014b640 5 monclient: authenticate success, global_id 5030 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e2014b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e2014b640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 1 -- 172.21.15.164:0/3207141827 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c032a10 con 0x7f7e1806d540 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e2014b640 4 set_mon_vals no callback set 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.063+0000 7f7e1cee5640 10 monclient: dump: 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.076 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 10 monclient: shutdown 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 -- 172.21.15.164:0/3207141827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 msgr2=0x7f7e180a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 --2- 172.21.15.164:0/3207141827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009870 tx=0x7f7e0c02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 -- 172.21.15.164:0/3207141827 shutdown_connections 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 --2- 172.21.15.164:0/3207141827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d540 0x7f7e180a46e0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 --2- 172.21.15.164:0/3207141827 >> v2:172.21.15.165:3300/0 conn(0x7f7e1806d160 0x7f7e18134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 -- 172.21.15.164:0/3207141827 >> 172.21.15.164:0/3207141827 conn(0x7f7e180aacb0 msgr2=0x7f7e180ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 -- 172.21.15.164:0/3207141827 shutdown_connections 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 1 -- 172.21.15.164:0/3207141827 wait complete. 2024-09-10T16:13:37.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 10 monclient: build_initial_monmap 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.064+0000 7f7e2014b640 10 monclient: monmap: 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 Processor -- start 2024-09-10T16:13:37.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 -- start start 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient: init 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e1806d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e1806d540 0x7f7e1813e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e1806d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e180a4c20 0x7f7e1813eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): picked mon.noname-a con 0x7f7e180a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7e18138870 con 0x7f7e180a4c20 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7e181386f0 con 0x7f7e1806d540 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e181389f0 con 0x7f7e1806d160 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e2014b640 10 monclient(hunting): authenticate will time out at 1195.552815s 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35638/0 (socket says 172.21.15.164:35638) 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 1 -- 172.21.15.164:0/2537085904 learned_addr learned my addr 172.21.15.164:0/2537085904 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): get_auth_request con 0x7f7e1806d160 auth_method 0 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1e6e8640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e180a4c20 0x7f7e1813eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e1806d540 msgr2=0x7f7e1813e660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1e6e8640 10 monclient(hunting): get_auth_request con 0x7f7e180a4c20 auth_method 0 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1e6e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1e6e8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1e6e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e1806d540 0x7f7e1813e660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.065+0000 7f7e1dee7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1e6e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1e6e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1e6e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 10 monclient(hunting): handle_auth_done global_id 5033 payload 824 2024-09-10T16:13:37.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e1806d540 msgr2=0x7f7e1813e660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7e1806d540 0x7f7e1813e660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e180a4c20 msgr2=0x7f7e1813eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7e180a4c20 0x7f7e1813eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 10 monclient: _finish_hunting 0 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 monclient: found mon.noname-c 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.066+0000 7f7e1dee7640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e1809f4d0 con 0x7f7e1806d160 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e1dee7640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 msgr2=0x7f7e1813e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e1dee7640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800b580 tx=0x7f7e0800ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080050f0 con 0x7f7e1806d160 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: dump: 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: _finish_auth 0 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: _check_auth_tickets 2024-09-10T16:13:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e08015070 con 0x7f7e1806d160 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 5 monclient: authenticate success, global_id 5033 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 10 monclient: _renew_subs 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e1809f7b0 con 0x7f7e1806d160 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e080054f0 con 0x7f7e1806d160 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e167fc640 10 monclient: dump: 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 10 monclient: _renew_subs 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e17fff640 4 set_mon_vals no callback set 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.068+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e1809f1f0 con 0x7f7e1806d160 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e2014b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e2014b640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 0x7f7e181cd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e1809d6b0 con 0x7f7e181cd250 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7e08010480 con 0x7f7e1806d160 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 0x7f7e181cd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient: get_auth_request con 0x7f7e181cd250 auth_method 0 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e167fc640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e0403e1c0 0x7f7e04040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_done global_id 5036 payload 824 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e180a7c60 con 0x7f7e181cd250 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.069+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(226..226 src has 184..226) ==== 4771+0+0 (secure 0 0 0) 0x7f7e08052b70 con 0x7f7e1806d160 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.070+0000 7f7e1e6e8640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e0403e1c0 0x7f7e04040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.070+0000 7f7e1e6e8640 10 monclient: get_auth_request con 0x7f7e0403e1c0 auth_method 0 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 msgr2=0x7f7e181cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 0x7f7e181cd630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c034000 tx=0x7f7e0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c037040 con 0x7f7e181cd250 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 10 monclient: dump: 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e0c009490 con 0x7f7e181cd250 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 msgr2=0x7f7e181cd630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 0x7f7e181cd630 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c034000 tx=0x7f7e0c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 msgr2=0x7f7e181cd630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e167fc640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cd250 0x7f7e181cd630 secure :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c034000 tx=0x7f7e0c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.072+0000 7f7e1e6e8640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e0403e1c0 0x7f7e04040680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e180a4890 tx=0x7f7e1000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e2014b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e2014b640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 0x7f7e18138cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e2014b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7e18134e00 con 0x7f7e181cdb70 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 0x7f7e18138cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient: get_auth_request con 0x7f7e181cdb70 auth_method 0 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 10 monclient(hunting): handle_auth_done global_id 5039 payload 824 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.078+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/2537085904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e180aa1c0 con 0x7f7e181cdb70 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e1d6e6640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 msgr2=0x7f7e18138cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e1d6e6640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 0x7f7e18138cd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009870 tx=0x7f7e0c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e0c0482e0 con 0x7f7e181cdb70 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 10 monclient: dump: 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f7e0c03e670 con 0x7f7e181cdb70 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 msgr2=0x7f7e18138cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 0x7f7e18138cd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009870 tx=0x7f7e0c009240 comp rx=0 tx=0).stop 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 msgr2=0x7f7e18138cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:13:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e167fc640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e181cdb70 0x7f7e18138cd0 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f7e0c009870 tx=0x7f7e0c009240 comp rx=0 tx=0).stop 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 72, 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 459253, 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:13:37.094 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:37.095 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:37.096 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:37.097 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:13:37.098 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:13:37.099 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.101 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:37.102 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.103 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.104 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:37.105 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:13:37.106 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:13:37.107 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:13:37.107 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:13:37.107 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:13:37.107 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:13:37.107 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:13:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e0403e1c0 msgr2=0x7f7e04040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7e0403e1c0 0x7f7e04040680 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f7e180a4890 tx=0x7f7e1000d040 comp rx=0 tx=0).stop 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 10 monclient: shutdown 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 msgr2=0x7f7e1813e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f7e0800b580 tx=0x7f7e0800ba40 comp rx=0 tx=0).stop 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e1e6e8640 1 -- 172.21.15.164:0/2537085904 reap_dead start 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 shutdown_connections 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 --2- 172.21.15.164:0/2537085904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7e1806d160 0x7f7e1813e120 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 >> 172.21.15.164:0/2537085904 conn(0x7f7e180aacb0 msgr2=0x7f7e180a7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 shutdown_connections 2024-09-10T16:13:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:37.081+0000 7f7e2014b640 1 -- 172.21.15.164:0/2537085904 wait complete. 2024-09-10T16:13:37.110 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:13:37.110 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:926: done (1 left) 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:929: done (0 left) 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:930: read oid 11 snap -1 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:930: expect deleted 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:931: write (excl) oid 22 current snap is 100 2024-09-10T16:13:37.146 INFO:tasks.rados.rados.0.smithi165.stdout:931: seq_num 350 ranges {682450=489388,1814580=496780,2878518=788028} 2024-09-10T16:13:37.152 INFO:tasks.rados.rados.0.smithi165.stdout:931: writing smithi16534841-22 from 682450 to 1171838 tid 1 2024-09-10T16:13:37.158 INFO:tasks.rados.rados.0.smithi165.stdout:931: writing smithi16534841-22 from 1814580 to 2311360 tid 2 2024-09-10T16:13:37.166 INFO:tasks.rados.rados.0.smithi165.stdout:931: writing smithi16534841-22 from 2878518 to 3666546 tid 3 2024-09-10T16:13:37.166 INFO:tasks.rados.rados.0.smithi165.stdout:932: read oid 23 snap -1 2024-09-10T16:13:37.166 INFO:tasks.rados.rados.0.smithi165.stdout:932: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:13:37.167 INFO:tasks.rados.rados.0.smithi165.stdout:933: write oid 20 current snap is 100 2024-09-10T16:13:37.167 INFO:tasks.rados.rados.0.smithi165.stdout:933: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2024-09-10T16:13:37.173 INFO:tasks.rados.rados.0.smithi165.stdout:933: writing smithi16534841-20 from 699257 to 1263894 tid 1 2024-09-10T16:13:37.181 INFO:tasks.rados.rados.0.smithi165.stdout:933: writing smithi16534841-20 from 1873062 to 2513500 tid 2 2024-09-10T16:13:37.189 INFO:tasks.rados.rados.0.smithi165.stdout:933: writing smithi16534841-20 from 3305941 to 3899257 tid 3 2024-09-10T16:13:37.189 INFO:tasks.rados.rados.0.smithi165.stdout:934: snap_create 2024-09-10T16:13:37.189 INFO:tasks.rados.rados.0.smithi165.stdout:931: finishing write tid 1 to smithi16534841-22 2024-09-10T16:13:37.189 INFO:tasks.rados.rados.0.smithi165.stdout:931: finishing write tid 2 to smithi16534841-22 2024-09-10T16:13:37.189 INFO:tasks.rados.rados.0.smithi165.stdout:932: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:13:37.266 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2269915 2024-09-10T16:13:37.285 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:37.340 INFO:tasks.rados.rados.0.smithi165.stdout:931: finishing write tid 3 to smithi16534841-22 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: finishing write tid 4 to smithi16534841-22 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: finishing write tid 5 to smithi16534841-22 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: oid 22 updating version 0 to 296 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: oid 22 updating version 296 to 297 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: oid 22 version 297 is already newer than 295 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: oid 22 version 297 is already newer than 294 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 297 (ObjNum 350 snap 100 seq_num 350) dirty exists 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:931: left oid 22 (ObjNum 350 snap 100 seq_num 350) 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:933: finishing write tid 1 to smithi16534841-20 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:933: finishing write tid 2 to smithi16534841-20 2024-09-10T16:13:37.341 INFO:tasks.rados.rados.0.smithi165.stdout:933: finishing write tid 3 to smithi16534841-20 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: finishing write tid 4 to smithi16534841-20 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: finishing write tid 5 to smithi16534841-20 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: oid 20 updating version 0 to 629 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: oid 20 updating version 629 to 630 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: oid 20 updating version 630 to 631 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: oid 20 version 631 is already newer than 628 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 631 (ObjNum 351 snap 100 seq_num 351) dirty exists 2024-09-10T16:13:37.342 INFO:tasks.rados.rados.0.smithi165.stdout:933: left oid 20 (ObjNum 351 snap 100 seq_num 351) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:930: done (4 left) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:931: done (3 left) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:932: done (2 left) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:933: done (1 left) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:934: done (0 left) 2024-09-10T16:13:38.149 INFO:tasks.rados.rados.0.smithi165.stdout:935: rollback oid 49 current snap is 101 2024-09-10T16:13:38.150 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 98 2024-09-10T16:13:38.150 INFO:tasks.rados.rados.0.smithi165.stdout:936: write oid 14 current snap is 101 2024-09-10T16:13:38.150 INFO:tasks.rados.rados.0.smithi165.stdout:936: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-10T16:13:38.155 INFO:tasks.rados.rados.0.smithi165.stdout:936: writing smithi16534841-14 from 716064 to 1355950 tid 1 2024-09-10T16:13:38.161 INFO:tasks.rados.rados.0.smithi165.stdout:936: writing smithi16534841-14 from 1815191 to 2315640 tid 2 2024-09-10T16:13:38.167 INFO:tasks.rados.rados.0.smithi165.stdout:936: writing smithi16534841-14 from 2817011 to 3343936 tid 3 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:937: write oid 19 current snap is 101 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:935: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 0 (ObjNum 257 snap 0 seq_num 1162718512) dirty dne 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:936: finishing write tid 1 to smithi16534841-14 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:936: finishing write tid 2 to smithi16534841-14 2024-09-10T16:13:38.168 INFO:tasks.rados.rados.0.smithi165.stdout:937: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-10T16:13:38.174 INFO:tasks.rados.rados.0.smithi165.stdout:937: writing smithi16534841-19 from 732871 to 1448006 tid 1 2024-09-10T16:13:38.180 INFO:tasks.rados.rados.0.smithi165.stdout:937: writing smithi16534841-19 from 1873673 to 2517780 tid 2 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:937: writing smithi16534841-19 from 3244434 to 3932871 tid 3 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:935: done (2 left) 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:938: rollback oid 1 current snap is 101 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 98 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:936: finishing write tid 3 to smithi16534841-14 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:936: finishing write tid 4 to smithi16534841-14 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:936: finishing write tid 5 to smithi16534841-14 2024-09-10T16:13:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:936: oid 14 updating version 0 to 322 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:936: oid 14 updating version 322 to 324 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:936: oid 14 version 324 is already newer than 323 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:936: oid 14 version 324 is already newer than 321 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 324 (ObjNum 352 snap 101 seq_num 352) dirty exists 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:936: left oid 14 (ObjNum 352 snap 101 seq_num 352) 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:937: finishing write tid 1 to smithi16534841-19 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:937: finishing write tid 2 to smithi16534841-19 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:936: done (2 left) 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:939: read oid 24 snap -1 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:939: expect deleted 2024-09-10T16:13:38.190 INFO:tasks.rados.rados.0.smithi165.stdout:940: snap_create 2024-09-10T16:13:38.192 INFO:tasks.rados.rados.0.smithi165.stdout:938: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:13:38.192 INFO:tasks.rados.rados.0.smithi165.stdout:938: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:13:38.193 INFO:tasks.rados.rados.0.smithi165.stdout:938: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:13:38.193 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 545 (ObjNum 114 snap 31 seq_num 114) dirty exists 2024-09-10T16:13:38.193 INFO:tasks.rados.rados.0.smithi165.stdout:937: finishing write tid 3 to smithi16534841-19 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: finishing write tid 4 to smithi16534841-19 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: finishing write tid 5 to smithi16534841-19 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: oid 19 updating version 0 to 870 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: oid 19 version 870 is already newer than 867 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: oid 19 version 870 is already newer than 869 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: oid 19 version 870 is already newer than 868 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 870 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-10T16:13:38.194 INFO:tasks.rados.rados.0.smithi165.stdout:937: left oid 19 (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:13:39.150 INFO:tasks.rados.rados.0.smithi165.stdout:937: done (3 left) 2024-09-10T16:13:39.150 INFO:tasks.rados.rados.0.smithi165.stdout:938: done (2 left) 2024-09-10T16:13:39.150 INFO:tasks.rados.rados.0.smithi165.stdout:939: done (1 left) 2024-09-10T16:13:39.151 INFO:tasks.rados.rados.0.smithi165.stdout:940: done (0 left) 2024-09-10T16:13:39.151 INFO:tasks.rados.rados.0.smithi165.stdout:941: delete oid 5 current snap is 102 2024-09-10T16:13:39.156 INFO:tasks.rados.rados.0.smithi165.stdout:941: done (0 left) 2024-09-10T16:13:39.156 INFO:tasks.rados.rados.0.smithi165.stdout:942: snap_create 2024-09-10T16:13:40.162 INFO:tasks.rados.rados.0.smithi165.stdout:942: done (0 left) 2024-09-10T16:13:40.162 INFO:tasks.rados.rados.0.smithi165.stdout:943: read oid 16 snap -1 2024-09-10T16:13:40.162 INFO:tasks.rados.rados.0.smithi165.stdout:943: expect deleted 2024-09-10T16:13:40.163 INFO:tasks.rados.rados.0.smithi165.stdout:944: read oid 2 snap -1 2024-09-10T16:13:40.163 INFO:tasks.rados.rados.0.smithi165.stdout:944: expect (ObjNum 286 snap 82 seq_num 286) 2024-09-10T16:13:40.163 INFO:tasks.rados.rados.0.smithi165.stdout:945: write (excl) oid 12 current snap is 103 2024-09-10T16:13:40.163 INFO:tasks.rados.rados.0.smithi165.stdout:945: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-10T16:13:40.168 INFO:tasks.rados.rados.0.smithi165.stdout:945: writing smithi16534841-12 from 766485 to 1232118 tid 1 2024-09-10T16:13:40.176 INFO:tasks.rados.rados.0.smithi165.stdout:945: writing smithi16534841-12 from 1990637 to 2638413 tid 2 2024-09-10T16:13:40.184 INFO:tasks.rados.rados.0.smithi165.stdout:945: writing smithi16534841-12 from 3299280 to 3966485 tid 3 2024-09-10T16:13:40.184 INFO:tasks.rados.rados.0.smithi165.stdout:946: copy_from oid 22 from oid 3 current snap is 103 2024-09-10T16:13:40.184 INFO:tasks.rados.rados.0.smithi165.stdout:944: expect (ObjNum 286 snap 82 seq_num 286) 2024-09-10T16:13:40.234 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:40.265 INFO:tasks.rados.rados.0.smithi165.stdout:945: finishing write tid 1 to smithi16534841-12 2024-09-10T16:13:40.265 INFO:tasks.rados.rados.0.smithi165.stdout:945: finishing write tid 2 to smithi16534841-12 2024-09-10T16:13:40.265 INFO:tasks.rados.rados.0.smithi165.stdout:945: finishing write tid 3 to smithi16534841-12 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: finishing write tid 4 to smithi16534841-12 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: finishing write tid 5 to smithi16534841-12 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: oid 12 updating version 0 to 76 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: oid 12 version 76 is already newer than 75 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: oid 12 updating version 76 to 77 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: oid 12 version 77 is already newer than 74 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 77 (ObjNum 355 snap 103 seq_num 355) dirty exists 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:945: left oid 12 (ObjNum 355 snap 103 seq_num 355) 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:946: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:946: finishing copy_from to smithi16534841-22 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 618 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:943: done (3 left) 2024-09-10T16:13:40.266 INFO:tasks.rados.rados.0.smithi165.stdout:944: done (2 left) 2024-09-10T16:13:40.267 INFO:tasks.rados.rados.0.smithi165.stdout:945: done (1 left) 2024-09-10T16:13:40.267 INFO:tasks.rados.rados.0.smithi165.stdout:946: done (0 left) 2024-09-10T16:13:40.267 INFO:tasks.rados.rados.0.smithi165.stdout:947: snap_remove snap 102 2024-09-10T16:13:41.163 INFO:tasks.rados.rados.0.smithi165.stdout:947: done (0 left) 2024-09-10T16:13:41.163 INFO:tasks.rados.rados.0.smithi165.stdout:948: rollback oid 28 current snap is 103 2024-09-10T16:13:41.163 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 28 to 98 2024-09-10T16:13:41.163 INFO:tasks.rados.rados.0.smithi165.stdout:949: delete oid 14 current snap is 103 2024-09-10T16:13:41.175 INFO:tasks.rados.rados.0.smithi165.stdout:949: done (1 left) 2024-09-10T16:13:41.175 INFO:tasks.rados.rados.0.smithi165.stdout:950: delete oid 30 current snap is 103 2024-09-10T16:13:41.176 INFO:tasks.rados.rados.0.smithi165.stdout:948: finishing rollback tid 0 to smithi16534841-28 2024-09-10T16:13:41.176 INFO:tasks.rados.rados.0.smithi165.stdout:948: finishing rollback tid 1 to smithi16534841-28 2024-09-10T16:13:41.176 INFO:tasks.rados.rados.0.smithi165.stdout:948: finishing rollback tid 2 to smithi16534841-28 2024-09-10T16:13:41.176 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 303 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:13:41.180 INFO:tasks.rados.rados.0.smithi165.stdout:948: done (1 left) 2024-09-10T16:13:41.181 INFO:tasks.rados.rados.0.smithi165.stdout:950: done (0 left) 2024-09-10T16:13:41.181 INFO:tasks.rados.rados.0.smithi165.stdout:951: delete oid 36 current snap is 103 2024-09-10T16:13:41.183 INFO:tasks.rados.rados.0.smithi165.stdout:951: done (0 left) 2024-09-10T16:13:41.183 INFO:tasks.rados.rados.0.smithi165.stdout:952: snap_remove snap 98 2024-09-10T16:13:42.155 INFO:tasks.rados.rados.0.smithi165.stdout:952: done (0 left) 2024-09-10T16:13:42.155 INFO:tasks.rados.rados.0.smithi165.stdout:953: read oid 27 snap -1 2024-09-10T16:13:42.155 INFO:tasks.rados.rados.0.smithi165.stdout:953: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:42.155 INFO:tasks.rados.rados.0.smithi165.stdout:954: delete oid 50 current snap is 103 2024-09-10T16:13:42.164 INFO:tasks.rados.rados.0.smithi165.stdout:954: done (1 left) 2024-09-10T16:13:42.164 INFO:tasks.rados.rados.0.smithi165.stdout:955: snap_remove snap 100 2024-09-10T16:13:43.160 INFO:tasks.rados.rados.0.smithi165.stdout:955: done (1 left) 2024-09-10T16:13:43.160 INFO:tasks.rados.rados.0.smithi165.stdout:956: copy_from oid 46 from oid 3 current snap is 103 2024-09-10T16:13:43.160 INFO:tasks.rados.rados.0.smithi165.stdout:953: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:43.160 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:43.272 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1824021 2024-09-10T16:13:43.297 INFO:tasks.rados.rados.0.smithi165.stdout:956: finishing copy_from racing read to smithi16534841-46 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:956: finishing copy_from to smithi16534841-46 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 618 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:953: done (1 left) 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:956: done (0 left) 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:957: write oid 41 current snap is 103 2024-09-10T16:13:43.298 INFO:tasks.rados.rados.0.smithi165.stdout:957: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-10T16:13:43.302 INFO:tasks.rados.rados.0.smithi165.stdout:957: writing smithi16534841-41 from 450520 to 1008751 tid 1 2024-09-10T16:13:43.308 INFO:tasks.rados.rados.0.smithi165.stdout:957: writing smithi16534841-41 from 1483047 to 1998172 tid 2 2024-09-10T16:13:43.308 INFO:tasks.rados.rados.0.smithi165.stdout:958: delete oid 45 current snap is 103 2024-09-10T16:13:43.308 INFO:tasks.rados.rados.0.smithi165.stdout:957: finishing write tid 1 to smithi16534841-41 2024-09-10T16:13:43.313 INFO:tasks.rados.rados.0.smithi165.stdout:958: done (1 left) 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:959: rollback oid 17 current snap is 103 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 101 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:960: read oid 23 snap -1 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:960: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:961: read oid 45 snap -1 2024-09-10T16:13:43.314 INFO:tasks.rados.rados.0.smithi165.stdout:961: expect deleted 2024-09-10T16:13:43.315 INFO:tasks.rados.rados.0.smithi165.stdout:962: snap_create 2024-09-10T16:13:43.315 INFO:tasks.rados.rados.0.smithi165.stdout:957: finishing write tid 2 to smithi16534841-41 2024-09-10T16:13:43.316 INFO:tasks.rados.rados.0.smithi165.stdout:960: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:13:43.316 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: finishing write tid 3 to smithi16534841-41 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: finishing write tid 4 to smithi16534841-41 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: oid 41 updating version 0 to 554 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: oid 41 updating version 554 to 555 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: oid 41 version 555 is already newer than 553 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 555 (ObjNum 360 snap 103 seq_num 360) dirty exists 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:957: left oid 41 (ObjNum 360 snap 103 seq_num 360) 2024-09-10T16:13:43.483 INFO:tasks.rados.rados.0.smithi165.stdout:959: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:13:43.484 INFO:tasks.rados.rados.0.smithi165.stdout:959: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:13:43.484 INFO:tasks.rados.rados.0.smithi165.stdout:959: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:13:43.484 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 323 (ObjNum 213 snap 64 seq_num 213) dirty exists 2024-09-10T16:13:44.164 INFO:tasks.rados.rados.0.smithi165.stdout:957: done (4 left) 2024-09-10T16:13:44.164 INFO:tasks.rados.rados.0.smithi165.stdout:959: done (3 left) 2024-09-10T16:13:44.164 INFO:tasks.rados.rados.0.smithi165.stdout:960: done (2 left) 2024-09-10T16:13:44.165 INFO:tasks.rados.rados.0.smithi165.stdout:961: done (1 left) 2024-09-10T16:13:44.165 INFO:tasks.rados.rados.0.smithi165.stdout:962: done (0 left) 2024-09-10T16:13:44.165 INFO:tasks.rados.rados.0.smithi165.stdout:963: write (excl) oid 50 current snap is 104 2024-09-10T16:13:44.165 INFO:tasks.rados.rados.0.smithi165.stdout:963: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-10T16:13:44.170 INFO:tasks.rados.rados.0.smithi165.stdout:963: writing smithi16534841-50 from 484134 to 1192863 tid 1 2024-09-10T16:13:44.176 INFO:tasks.rados.rados.0.smithi165.stdout:963: writing smithi16534841-50 from 1883658 to 2084134 tid 2 2024-09-10T16:13:44.176 INFO:tasks.rados.rados.0.smithi165.stdout:964: write (excl) oid 32 current snap is 104 2024-09-10T16:13:44.176 INFO:tasks.rados.rados.0.smithi165.stdout:964: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-10T16:13:44.183 INFO:tasks.rados.rados.0.smithi165.stdout:964: writing smithi16534841-32 from 500941 to 1284919 tid 1 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:964: writing smithi16534841-32 from 1942140 to 2100941 tid 2 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:965: delete oid 43 current snap is 104 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: finishing write tid 1 to smithi16534841-50 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: finishing write tid 2 to smithi16534841-50 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: finishing write tid 3 to smithi16534841-50 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: finishing write tid 4 to smithi16534841-50 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: oid 50 updating version 0 to 381 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: oid 50 version 381 is already newer than 380 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: oid 50 updating version 381 to 382 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 382 (ObjNum 362 snap 104 seq_num 362) dirty exists 2024-09-10T16:13:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:963: left oid 50 (ObjNum 362 snap 104 seq_num 362) 2024-09-10T16:13:44.192 INFO:tasks.rados.rados.0.smithi165.stdout:964: finishing write tid 1 to smithi16534841-32 2024-09-10T16:13:44.195 INFO:tasks.rados.rados.0.smithi165.stdout:964: finishing write tid 2 to smithi16534841-32 2024-09-10T16:13:44.198 INFO:tasks.rados.rados.0.smithi165.stdout:964: finishing write tid 3 to smithi16534841-32 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:963: done (2 left) 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:965: done (1 left) 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:966: read oid 50 snap -1 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:966: expect (ObjNum 362 snap 104 seq_num 362) 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:967: copy_from oid 8 from oid 28 current snap is 104 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:968: read oid 45 snap -1 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:968: expect deleted 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:969: snap_create 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:964: finishing write tid 4 to smithi16534841-32 2024-09-10T16:13:44.199 INFO:tasks.rados.rados.0.smithi165.stdout:964: oid 32 updating version 0 to 351 2024-09-10T16:13:44.200 INFO:tasks.rados.rados.0.smithi165.stdout:964: oid 32 version 351 is already newer than 350 2024-09-10T16:13:44.200 INFO:tasks.rados.rados.0.smithi165.stdout:964: oid 32 updating version 351 to 352 2024-09-10T16:13:44.200 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 352 (ObjNum 363 snap 104 seq_num 363) dirty exists 2024-09-10T16:13:44.200 INFO:tasks.rados.rados.0.smithi165.stdout:964: left oid 32 (ObjNum 363 snap 104 seq_num 363) 2024-09-10T16:13:44.201 INFO:tasks.rados.rados.0.smithi165.stdout:966: expect (ObjNum 362 snap 104 seq_num 362) 2024-09-10T16:13:44.201 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:44.261 INFO:tasks.rados.rados.0.smithi165.stdout:967: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:13:44.261 INFO:tasks.rados.rados.0.smithi165.stdout:967: finishing copy_from to smithi16534841-8 2024-09-10T16:13:44.261 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 303 (ObjNum 115 snap 31 seq_num 115) dirty exists 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:964: done (4 left) 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:966: done (3 left) 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:967: done (2 left) 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:968: done (1 left) 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:969: done (0 left) 2024-09-10T16:13:45.167 INFO:tasks.rados.rados.0.smithi165.stdout:970: snap_remove snap 103 2024-09-10T16:13:46.173 INFO:tasks.rados.rados.0.smithi165.stdout:970: done (0 left) 2024-09-10T16:13:46.174 INFO:tasks.rados.rados.0.smithi165.stdout:971: snap_remove snap 99 2024-09-10T16:13:47.149 INFO:tasks.rados.rados.0.smithi165.stdout:971: done (0 left) 2024-09-10T16:13:47.149 INFO:tasks.rados.rados.0.smithi165.stdout:972: snap_create 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:972: done (0 left) 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:973: copy_from oid 32 from oid 29 current snap is 106 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:974: read oid 16 snap -1 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:974: expect deleted 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:975: rollback oid 31 current snap is 106 2024-09-10T16:13:48.151 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 101 2024-09-10T16:13:48.152 INFO:tasks.rados.rados.0.smithi165.stdout:976: write oid 8 current snap is 106 2024-09-10T16:13:48.152 INFO:tasks.rados.rados.0.smithi165.stdout:976: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-10T16:13:48.157 INFO:tasks.rados.rados.0.smithi165.stdout:976: writing smithi16534841-8 from 534555 to 1185384 tid 1 2024-09-10T16:13:48.162 INFO:tasks.rados.rados.0.smithi165.stdout:976: writing smithi16534841-8 from 1659104 to 2134555 tid 2 2024-09-10T16:13:48.163 INFO:tasks.rados.rados.0.smithi165.stdout:977: snap_remove snap 104 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:977: done (4 left) 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:978: snap_create 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:975: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:975: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:975: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:13:49.152 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 888 (ObjNum 315 snap 91 seq_num 315) dirty exists 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:973: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: finishing write tid 1 to smithi16534841-8 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:973: finishing copy_from to smithi16534841-32 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 589 (ObjNum 250 snap 76 seq_num 250) dirty exists 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: finishing write tid 2 to smithi16534841-8 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: finishing write tid 3 to smithi16534841-8 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: finishing write tid 4 to smithi16534841-8 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: oid 8 updating version 0 to 639 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: oid 8 version 639 is already newer than 638 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:976: oid 8 version 639 is already newer than 637 2024-09-10T16:13:49.153 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 639 (ObjNum 365 snap 106 seq_num 365) dirty exists 2024-09-10T16:13:49.154 INFO:tasks.rados.rados.0.smithi165.stdout:976: left oid 8 (ObjNum 365 snap 106 seq_num 365) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:973: done (4 left) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:974: done (3 left) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:975: done (2 left) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:976: done (1 left) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:978: done (0 left) 2024-09-10T16:13:50.158 INFO:tasks.rados.rados.0.smithi165.stdout:979: snap_create 2024-09-10T16:13:51.162 INFO:tasks.rados.rados.0.smithi165.stdout:979: done (0 left) 2024-09-10T16:13:51.162 INFO:tasks.rados.rados.0.smithi165.stdout:980: read oid 10 snap 106 2024-09-10T16:13:51.162 INFO:tasks.rados.rados.0.smithi165.stdout:980: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:13:51.162 INFO:tasks.rados.rados.0.smithi165.stdout:981: rollback oid 30 current snap is 108 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 101 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:982: rollback oid 9 current snap is 108 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 101 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:983: read oid 16 snap -1 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:983: expect deleted 2024-09-10T16:13:51.163 INFO:tasks.rados.rados.0.smithi165.stdout:984: delete oid 28 current snap is 108 2024-09-10T16:13:51.165 INFO:tasks.rados.rados.0.smithi165.stdout:982: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:13:51.165 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 0 (ObjNum 513 snap 0 seq_num 1162534776) dirty dne 2024-09-10T16:13:51.381 INFO:tasks.rados.rados.0.smithi165.stdout:982: done (4 left) 2024-09-10T16:13:51.381 INFO:tasks.rados.rados.0.smithi165.stdout:984: done (3 left) 2024-09-10T16:13:51.381 INFO:tasks.rados.rados.0.smithi165.stdout:985: snap_remove snap 107 2024-09-10T16:13:52.150 INFO:tasks.rados.rados.0.smithi165.stdout:985: done (3 left) 2024-09-10T16:13:52.151 INFO:tasks.rados.rados.0.smithi165.stdout:986: delete oid 49 current snap is 108 2024-09-10T16:13:52.151 INFO:tasks.rados.rados.0.smithi165.stdout:980: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:13:52.194 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:52.276 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3067422 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:980: done (3 left) 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:983: done (2 left) 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:986: done (1 left) 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:987: write (excl) oid 21 current snap is 108 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:981: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:981: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 892 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-10T16:13:52.286 INFO:tasks.rados.rados.0.smithi165.stdout:987: seq_num 368 ranges {584976=476576,1834550=350426} 2024-09-10T16:13:52.291 INFO:tasks.rados.rados.0.smithi165.stdout:987: writing smithi16534841-21 from 584976 to 1061552 tid 1 2024-09-10T16:13:52.298 INFO:tasks.rados.rados.0.smithi165.stdout:987: writing smithi16534841-21 from 1834550 to 2184976 tid 2 2024-09-10T16:13:52.298 INFO:tasks.rados.rados.0.smithi165.stdout:981: done (1 left) 2024-09-10T16:13:52.298 INFO:tasks.rados.rados.0.smithi165.stdout:988: read oid 24 snap 105 2024-09-10T16:13:52.298 INFO:tasks.rados.rados.0.smithi165.stdout:988: expect deleted 2024-09-10T16:13:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:989: read oid 25 snap -1 2024-09-10T16:13:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:989: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:13:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:990: write (excl) oid 17 current snap is 108 2024-09-10T16:13:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:987: finishing write tid 1 to smithi16534841-21 2024-09-10T16:13:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:990: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-10T16:13:52.305 INFO:tasks.rados.rados.0.smithi165.stdout:990: writing smithi16534841-17 from 601783 to 1153608 tid 1 2024-09-10T16:13:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:990: writing smithi16534841-17 from 1776679 to 2201783 tid 2 2024-09-10T16:13:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:991: delete oid 20 current snap is 108 2024-09-10T16:13:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:989: expect (ObjNum 312 snap 91 seq_num 312) 2024-09-10T16:13:52.345 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2064892 2024-09-10T16:13:52.528 INFO:tasks.rados.rados.0.smithi165.stdout:987: finishing write tid 2 to smithi16534841-21 2024-09-10T16:13:52.528 INFO:tasks.rados.rados.0.smithi165.stdout:987: finishing write tid 3 to smithi16534841-21 2024-09-10T16:13:52.528 INFO:tasks.rados.rados.0.smithi165.stdout:987: finishing write tid 4 to smithi16534841-21 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:987: oid 21 updating version 0 to 637 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:987: oid 21 updating version 637 to 638 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:987: oid 21 version 638 is already newer than 636 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 638 (ObjNum 368 snap 108 seq_num 368) dirty exists 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:987: left oid 21 (ObjNum 368 snap 108 seq_num 368) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:990: finishing write tid 1 to smithi16534841-17 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:987: done (4 left) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:988: done (3 left) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:989: done (2 left) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:991: done (1 left) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:992: read oid 26 snap -1 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:992: expect (ObjNum 337 snap 98 seq_num 337) 2024-09-10T16:13:52.529 INFO:tasks.rados.rados.0.smithi165.stdout:993: snap_remove snap 106 2024-09-10T16:13:53.168 INFO:tasks.rados.rados.0.smithi165.stdout:993: done (2 left) 2024-09-10T16:13:53.168 INFO:tasks.rados.rados.0.smithi165.stdout:994: delete oid 16 current snap is 108 2024-09-10T16:13:53.168 INFO:tasks.rados.rados.0.smithi165.stdout:990: finishing write tid 2 to smithi16534841-17 2024-09-10T16:13:53.168 INFO:tasks.rados.rados.0.smithi165.stdout:990: finishing write tid 3 to smithi16534841-17 2024-09-10T16:13:53.168 INFO:tasks.rados.rados.0.smithi165.stdout:990: finishing write tid 4 to smithi16534841-17 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:990: oid 17 updating version 0 to 330 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:990: oid 17 version 330 is already newer than 328 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:990: oid 17 version 330 is already newer than 329 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 330 (ObjNum 369 snap 108 seq_num 369) dirty exists 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:990: left oid 17 (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:13:53.169 INFO:tasks.rados.rados.0.smithi165.stdout:992: expect (ObjNum 337 snap 98 seq_num 337) 2024-09-10T16:13:53.185 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1429262 2024-09-10T16:13:53.241 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:53.321 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:53.402 INFO:tasks.rados.rados.0.smithi165.stdout:990: done (2 left) 2024-09-10T16:13:53.402 INFO:tasks.rados.rados.0.smithi165.stdout:992: done (1 left) 2024-09-10T16:13:53.402 INFO:tasks.rados.rados.0.smithi165.stdout:994: done (0 left) 2024-09-10T16:13:53.402 INFO:tasks.rados.rados.0.smithi165.stdout:995: copy_from oid 42 from oid 7 current snap is 108 2024-09-10T16:13:53.403 INFO:tasks.rados.rados.0.smithi165.stdout:996: write oid 49 current snap is 108 2024-09-10T16:13:53.403 INFO:tasks.rados.rados.0.smithi165.stdout:996: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-10T16:13:53.410 INFO:tasks.rados.rados.0.smithi165.stdout:996: writing smithi16534841-49 from 652204 to 1429776 tid 1 2024-09-10T16:13:53.415 INFO:tasks.rados.rados.0.smithi165.stdout:996: writing smithi16534841-49 from 1952125 to 2252204 tid 2 2024-09-10T16:13:53.415 INFO:tasks.rados.rados.0.smithi165.stdout:997: snap_create 2024-09-10T16:13:53.417 INFO:tasks.rados.rados.0.smithi165.stdout:995: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:13:53.418 INFO:tasks.rados.rados.0.smithi165.stdout:996: finishing write tid 1 to smithi16534841-49 2024-09-10T16:13:53.418 INFO:tasks.rados.rados.0.smithi165.stdout:996: finishing write tid 2 to smithi16534841-49 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: finishing write tid 3 to smithi16534841-49 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: finishing write tid 4 to smithi16534841-49 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: oid 49 updating version 0 to 335 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: oid 49 version 335 is already newer than 333 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: oid 49 version 335 is already newer than 334 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 335 (ObjNum 372 snap 108 seq_num 372) dirty exists 2024-09-10T16:13:53.419 INFO:tasks.rados.rados.0.smithi165.stdout:996: left oid 49 (ObjNum 372 snap 108 seq_num 372) 2024-09-10T16:13:53.421 INFO:tasks.rados.rados.0.smithi165.stdout:995: finishing copy_from to smithi16534841-42 2024-09-10T16:13:53.421 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 342 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:13:54.171 INFO:tasks.rados.rados.0.smithi165.stdout:995: done (2 left) 2024-09-10T16:13:54.171 INFO:tasks.rados.rados.0.smithi165.stdout:996: done (1 left) 2024-09-10T16:13:54.171 INFO:tasks.rados.rados.0.smithi165.stdout:997: done (0 left) 2024-09-10T16:13:54.172 INFO:tasks.rados.rados.0.smithi165.stdout:998: write (excl) oid 28 current snap is 109 2024-09-10T16:13:54.172 INFO:tasks.rados.rados.0.smithi165.stdout:998: seq_num 373 ranges {669011=569174,2010607=258404} 2024-09-10T16:13:54.177 INFO:tasks.rados.rados.0.smithi165.stdout:998: writing smithi16534841-28 from 669011 to 1238185 tid 1 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:998: writing smithi16534841-28 from 2010607 to 2269011 tid 2 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:999: copy_from oid 29 from oid 42 current snap is 109 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:1000: read oid 44 snap -1 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:1000: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:1001: write (excl) oid 4 current snap is 109 2024-09-10T16:13:54.183 INFO:tasks.rados.rados.0.smithi165.stdout:1001: seq_num 374 ranges {685818=644423,2069089=216729} 2024-09-10T16:13:54.190 INFO:tasks.rados.rados.0.smithi165.stdout:1001: writing smithi16534841-4 from 685818 to 1330241 tid 1 2024-09-10T16:13:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:1001: writing smithi16534841-4 from 2069089 to 2285818 tid 2 2024-09-10T16:13:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:1002: write (excl) oid 15 current snap is 109 2024-09-10T16:13:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:998: finishing write tid 1 to smithi16534841-28 2024-09-10T16:13:54.196 INFO:tasks.rados.rados.0.smithi165.stdout:1002: seq_num 375 ranges {702625=719672,2127571=175054} 2024-09-10T16:13:54.203 INFO:tasks.rados.rados.0.smithi165.stdout:1002: writing smithi16534841-15 from 702625 to 1422297 tid 1 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:1002: writing smithi16534841-15 from 2127571 to 2302625 tid 2 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:1003: read oid 10 snap -1 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:1003: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:1004: snap_create 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:998: finishing write tid 2 to smithi16534841-28 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:998: finishing write tid 3 to smithi16534841-28 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:998: finishing write tid 4 to smithi16534841-28 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:998: oid 28 updating version 0 to 307 2024-09-10T16:13:54.209 INFO:tasks.rados.rados.0.smithi165.stdout:998: oid 28 version 307 is already newer than 305 2024-09-10T16:13:54.210 INFO:tasks.rados.rados.0.smithi165.stdout:998: oid 28 version 307 is already newer than 306 2024-09-10T16:13:54.210 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 307 (ObjNum 373 snap 109 seq_num 373) dirty exists 2024-09-10T16:13:54.210 INFO:tasks.rados.rados.0.smithi165.stdout:998: left oid 28 (ObjNum 373 snap 109 seq_num 373) 2024-09-10T16:13:54.210 INFO:tasks.rados.rados.0.smithi165.stdout:1000: expect (ObjNum 190 snap 58 seq_num 190) 2024-09-10T16:13:54.348 INFO:tasks.rados.rados.0.smithi165.stdout:1001: finishing write tid 1 to smithi16534841-4 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:999: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: finishing write tid 2 to smithi16534841-4 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: finishing write tid 3 to smithi16534841-4 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: finishing write tid 4 to smithi16534841-4 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: oid 4 updating version 0 to 278 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: oid 4 updating version 278 to 280 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: oid 4 version 280 is already newer than 279 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 280 (ObjNum 374 snap 109 seq_num 374) dirty exists 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:1001: left oid 4 (ObjNum 374 snap 109 seq_num 374) 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:999: finishing copy_from to smithi16534841-29 2024-09-10T16:13:54.349 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 342 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:13:54.350 INFO:tasks.rados.rados.0.smithi165.stdout:1002: finishing write tid 1 to smithi16534841-15 2024-09-10T16:13:54.350 INFO:tasks.rados.rados.0.smithi165.stdout:1002: finishing write tid 2 to smithi16534841-15 2024-09-10T16:13:54.350 INFO:tasks.rados.rados.0.smithi165.stdout:1003: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:13:54.350 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:54.418 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2175895 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: finishing write tid 3 to smithi16534841-15 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: finishing write tid 4 to smithi16534841-15 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: oid 15 updating version 0 to 282 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: oid 15 updating version 282 to 283 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: oid 15 version 283 is already newer than 281 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 283 (ObjNum 375 snap 109 seq_num 375) dirty exists 2024-09-10T16:13:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:1002: left oid 15 (ObjNum 375 snap 109 seq_num 375) 2024-09-10T16:13:55.175 INFO:tasks.rados.rados.0.smithi165.stdout:998: done (6 left) 2024-09-10T16:13:55.175 INFO:tasks.rados.rados.0.smithi165.stdout:999: done (5 left) 2024-09-10T16:13:55.175 INFO:tasks.rados.rados.0.smithi165.stdout:1000: done (4 left) 2024-09-10T16:13:55.175 INFO:tasks.rados.rados.0.smithi165.stdout:1001: done (3 left) 2024-09-10T16:13:55.175 INFO:tasks.rados.rados.0.smithi165.stdout:1002: done (2 left) 2024-09-10T16:13:55.176 INFO:tasks.rados.rados.0.smithi165.stdout:1003: done (1 left) 2024-09-10T16:13:55.176 INFO:tasks.rados.rados.0.smithi165.stdout:1004: done (0 left) 2024-09-10T16:13:55.176 INFO:tasks.rados.rados.0.smithi165.stdout:1005: copy_from oid 21 from oid 47 current snap is 110 2024-09-10T16:13:55.176 INFO:tasks.rados.rados.0.smithi165.stdout:1006: snap_create 2024-09-10T16:13:55.184 INFO:tasks.rados.rados.0.smithi165.stdout:1005: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:13:55.187 INFO:tasks.rados.rados.0.smithi165.stdout:1005: finishing copy_from to smithi16534841-21 2024-09-10T16:13:55.187 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 567 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:13:56.180 INFO:tasks.rados.rados.0.smithi165.stdout:1005: done (1 left) 2024-09-10T16:13:56.181 INFO:tasks.rados.rados.0.smithi165.stdout:1006: done (0 left) 2024-09-10T16:13:56.181 INFO:tasks.rados.rados.0.smithi165.stdout:1007: write oid 41 current snap is 111 2024-09-10T16:13:56.181 INFO:tasks.rados.rados.0.smithi165.stdout:1007: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-10T16:13:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:1007: writing smithi16534841-41 from 719432 to 1514353 tid 1 2024-09-10T16:13:56.193 INFO:tasks.rados.rados.0.smithi165.stdout:1007: writing smithi16534841-41 from 2186053 to 2319432 tid 2 2024-09-10T16:13:56.193 INFO:tasks.rados.rados.0.smithi165.stdout:1008: snap_create 2024-09-10T16:13:56.195 INFO:tasks.rados.rados.0.smithi165.stdout:1007: finishing write tid 1 to smithi16534841-41 2024-09-10T16:13:56.195 INFO:tasks.rados.rados.0.smithi165.stdout:1007: finishing write tid 2 to smithi16534841-41 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: finishing write tid 3 to smithi16534841-41 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: finishing write tid 4 to smithi16534841-41 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: oid 41 updating version 0 to 563 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: oid 41 version 563 is already newer than 562 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: oid 41 version 563 is already newer than 561 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 563 (ObjNum 376 snap 111 seq_num 376) dirty exists 2024-09-10T16:13:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:1007: left oid 41 (ObjNum 376 snap 111 seq_num 376) 2024-09-10T16:13:57.110 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:13:57.110 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:13:57.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:13:57.114 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:13:57.114 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:13:57.114 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:13:57.151 INFO:tasks.rados.rados.0.smithi165.stdout:1007: done (1 left) 2024-09-10T16:13:57.151 INFO:tasks.rados.rados.0.smithi165.stdout:1008: done (0 left) 2024-09-10T16:13:57.151 INFO:tasks.rados.rados.0.smithi165.stdout:1009: snap_remove snap 109 2024-09-10T16:13:57.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.224+0000 7f63cec9b640 10 monclient: get_monmap_and_config 2024-09-10T16:13:57.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.225+0000 7f63cec9b640 10 monclient: build_initial_monmap 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.225+0000 7f63cec9b640 10 monclient: monmap: 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:13:57.226897+0000 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:13:57.226897+0000 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:13:57.393 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:13:57.394 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.225+0000 7f63cec9b640 1 Processor -- start 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.225+0000 7f63cec9b640 1 -- start start 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.225+0000 7f63cec9b640 10 monclient: init 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-c con 0x7f63c809c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63c809d160 0x7f63c809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-a con 0x7f63c809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f63c809f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63c809a430 con 0x7f63c809f4c0 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:57.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f63c80a24e0 con 0x7f63c809d160 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f63c80a2bf0 con 0x7f63c809c180 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cec9b640 10 monclient(hunting): authenticate will time out at 1215.714369s 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63c809d160 0x7f63c809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63c809d160 0x7f63c809ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33492/0 (socket says 172.21.15.164:33492) 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 -- 172.21.15.164:0/3098862146 learned_addr learned my addr 172.21.15.164:0/3098862146 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): get_auth_request con 0x7f63c809d160 auth_method 0 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 1 --2- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 10 monclient(hunting): get_auth_request con 0x7f63c809c180 auth_method 0 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:57.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 1 -- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 msgr2=0x7f63c809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 1 -- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 msgr2=0x7f63c809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 1 --2- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63cca37640 1 --2- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient(hunting): handle_auth_done global_id 74172 payload 274 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 -- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 msgr2=0x7f63c809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 --2- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 -- 172.21.15.164:0/3098862146 >> v1:172.21.15.165:6789/0 conn(0x7f63c809f4c0 legacy=0x7f63c813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 monclient: found mon.noname-a 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.226+0000 7f63c7fff640 1 -- 172.21.15.164:0/3098862146 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c809c560 con 0x7f63c809d160 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c7fff640 1 -- 172.21.15.164:0/3098862146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809d160 msgr2=0x7f63c809ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c7fff640 1 --2- 172.21.15.164:0/3098862146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809d160 0x7f63c809ef80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d150 tx=0x7f63b802c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3d5c0ee3d874aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 1 -- 172.21.15.164:0/3098862146 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8009800 con 0x7f63c809d160 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: dump: 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.408 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: _finish_auth 0 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 1 -- 172.21.15.164:0/3098862146 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b80099a0 con 0x7f63c809d160 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63cec9b640 5 monclient: authenticate success, global_id 74172 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63cec9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:13:57.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63cec9b640 10 monclient: get_monmap_and_config success 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 1 -- 172.21.15.164:0/3098862146 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8009d70 con 0x7f63c809d160 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63cec9b640 4 set_mon_vals no callback set 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.229+0000 7f63c77fe640 10 monclient: dump: 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.410 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 10 monclient: shutdown 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 -- 172.21.15.164:0/3098862146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809d160 msgr2=0x7f63c809ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 --2- 172.21.15.164:0/3098862146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809d160 0x7f63c809ef80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f63b800d150 tx=0x7f63b802c7c0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 -- 172.21.15.164:0/3098862146 shutdown_connections 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 --2- 172.21.15.164:0/3098862146 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809d160 0x7f63c809ef80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 --2- 172.21.15.164:0/3098862146 >> v2:172.21.15.165:3300/0 conn(0x7f63c809c180 0x7f63c809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 -- 172.21.15.164:0/3098862146 >> 172.21.15.164:0/3098862146 conn(0x7f63c80a3110 msgr2=0x7f63c80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 -- 172.21.15.164:0/3098862146 shutdown_connections 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.230+0000 7f63cec9b640 1 -- 172.21.15.164:0/3098862146 wait complete. 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.231+0000 7f63cec9b640 10 monclient: build_initial_monmap 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.231+0000 7f63cec9b640 10 monclient: monmap: 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:13:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.231+0000 7f63cec9b640 1 Processor -- start 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.231+0000 7f63cec9b640 1 -- start start 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.231+0000 7f63cec9b640 10 monclient: init 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-a con 0x7f63c809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c809d160 0x7f63c80997b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-c con 0x7f63c809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:13:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 0x7f63c81c9ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): picked mon.noname-b con 0x7f63c809f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8074f90 con 0x7f63c809c180 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63c8074e10 con 0x7f63c809f4c0 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63c809d930 con 0x7f63c809d160 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cec9b640 10 monclient(hunting): authenticate will time out at 1215.720266s 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33496/0 (socket says 172.21.15.164:33496) 2024-09-10T16:13:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c809d160 0x7f63c80997b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 -- 172.21.15.164:0/2146423084 learned_addr learned my addr 172.21.15.164:0/2146423084 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): get_auth_request con 0x7f63c809c180 auth_method 0 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): get_auth_request con 0x7f63c809d160 auth_method 0 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63c7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient(hunting): handle_auth_done global_id 74175 payload 824 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cd238640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 0x7f63c81c9ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 -- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c809d160 msgr2=0x7f63c80997b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c809d160 0x7f63c80997b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 -- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 msgr2=0x7f63c81c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 0x7f63c81c9ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient: _finish_hunting 0 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 monclient: found mon.noname-a 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cca37640 1 -- 172.21.15.164:0/2146423084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63c8095df0 con 0x7f63c809c180 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cd238640 10 monclient: get_auth_request con 0x7f63c809f4c0 auth_method 0 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.232+0000 7f63cd238640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 0x7f63c81c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.234+0000 7f63cca37640 1 -- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 msgr2=0x7f63c81ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.234+0000 7f63cca37640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63b000d7d0 tx=0x7f63b000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b000b8d0 con 0x7f63c809c180 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: dump: 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: _finish_auth 0 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 5 monclient: authenticate success, global_id 74175 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b0019070 con 0x7f63c809c180 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _renew_subs 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63c8097290 con 0x7f63c809c180 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b000bcd0 con 0x7f63c809c180 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c57fa640 10 monclient: dump: 2024-09-10T16:13:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _renew_subs 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63c6ffd640 4 set_mon_vals no callback set 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63c8094f30 con 0x7f63c809c180 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.235+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63c80a5d20 con 0x7f63c809c180 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.236+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f63b001e460 con 0x7f63c809c180 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.236+0000 7f63c57fa640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 0x7f63a80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.236+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(247..247 src has 184..247) ==== 4799+0+0 (secure 0 0 0) 0x7f63b0053200 con 0x7f63c809c180 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.236+0000 7f63c7fff640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 0x7f63a80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.236+0000 7f63c7fff640 10 monclient: get_auth_request con 0x7f63a803e210 auth_method 0 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.239+0000 7f63c7fff640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 0x7f63a80406d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f63c809a720 tx=0x7f63b8030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:13:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.240+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f63c80a5d20 con 0x7f63c809c180 2024-09-10T16:13:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.240+0000 7f63c57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:13:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.240+0000 7f63c57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.487+0000 7f63cec9b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.487+0000 7f63cec9b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.487+0000 7f63cec9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.487+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f63c8077260 con 0x7f63c809c180 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.488+0000 7f63c57fa640 1 -- 172.21.15.164:0/2146423084 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f63b001e930 con 0x7f63c809c180 2024-09-10T16:13:57.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.488+0000 7f63c57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:13:57.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.488+0000 7f63c57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:13:57.491 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:13:57.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 msgr2=0x7f63a80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:57.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 0x7f63a80406d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f63c809a720 tx=0x7f63b8030000 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 10 monclient: shutdown 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 msgr2=0x7f63c81ccff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63b000d7d0 tx=0x7f63b000dc90 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 shutdown_connections 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63a803e210 0x7f63a80406d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63c809f4c0 0x7f63c81c9ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63c809d160 0x7f63c80997b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 --2- 172.21.15.164:0/2146423084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63c809c180 0x7f63c81ccff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 >> 172.21.15.164:0/2146423084 conn(0x7f63c80a3110 msgr2=0x7f63c81bc3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 shutdown_connections 2024-09-10T16:13:57.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:13:57.490+0000 7f63cec9b640 1 -- 172.21.15.164:0/2146423084 wait complete. 2024-09-10T16:13:58.152 INFO:tasks.rados.rados.0.smithi165.stdout:1009: done (0 left) 2024-09-10T16:13:58.152 INFO:tasks.rados.rados.0.smithi165.stdout:1010: delete oid 4 current snap is 112 2024-09-10T16:13:58.160 INFO:tasks.rados.rados.0.smithi165.stdout:1010: done (0 left) 2024-09-10T16:13:58.160 INFO:tasks.rados.rados.0.smithi165.stdout:1011: copy_from oid 14 from oid 40 current snap is 112 2024-09-10T16:13:58.160 INFO:tasks.rados.rados.0.smithi165.stdout:1012: read oid 2 snap -1 2024-09-10T16:13:58.160 INFO:tasks.rados.rados.0.smithi165.stdout:1012: expect (ObjNum 286 snap 82 seq_num 286) 2024-09-10T16:13:58.160 INFO:tasks.rados.rados.0.smithi165.stdout:1013: snap_create 2024-09-10T16:13:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:1012: expect (ObjNum 286 snap 82 seq_num 286) 2024-09-10T16:13:58.191 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:13:58.247 INFO:tasks.rados.rados.0.smithi165.stdout:1011: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:13:58.248 INFO:tasks.rados.rados.0.smithi165.stdout:1011: finishing copy_from to smithi16534841-14 2024-09-10T16:13:58.248 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 320 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:13:59.154 INFO:tasks.rados.rados.0.smithi165.stdout:1011: done (2 left) 2024-09-10T16:13:59.154 INFO:tasks.rados.rados.0.smithi165.stdout:1012: done (1 left) 2024-09-10T16:13:59.154 INFO:tasks.rados.rados.0.smithi165.stdout:1013: done (0 left) 2024-09-10T16:13:59.154 INFO:tasks.rados.rados.0.smithi165.stdout:1014: read oid 23 snap -1 2024-09-10T16:13:59.154 INFO:tasks.rados.rados.0.smithi165.stdout:1014: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:13:59.155 INFO:tasks.rados.rados.0.smithi165.stdout:1015: snap_remove snap 101 2024-09-10T16:14:00.158 INFO:tasks.rados.rados.0.smithi165.stdout:1015: done (1 left) 2024-09-10T16:14:00.158 INFO:tasks.rados.rados.0.smithi165.stdout:1016: snap_create 2024-09-10T16:14:00.158 INFO:tasks.rados.rados.0.smithi165.stdout:1014: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:00.161 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 581764 2024-09-10T16:14:00.504 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:00.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 10 monclient: get_monmap_and_config 2024-09-10T16:14:00.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 10 monclient: build_initial_monmap 2024-09-10T16:14:00.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 10 monclient: monmap: 2024-09-10T16:14:00.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:00.613932+0000 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:00.613932+0000 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:00.615 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:00.616 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:00.616 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:00.616 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 1 Processor -- start 2024-09-10T16:14:00.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 1 -- start start 2024-09-10T16:14:00.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.612+0000 7f68559b5640 10 monclient: init 2024-09-10T16:14:00.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:00.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-10T16:14:00.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f685006cb70 0x7f6850134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:00.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-a con 0x7f685006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:00.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f68500a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f68500a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f685009ce50 con 0x7f68500a4ab0 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6850138ac0 con 0x7f68500a46a0 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f685006db60 con 0x7f685006cb70 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f68559b5640 10 monclient(hunting): authenticate will time out at 1219.101250s 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f685006cb70 0x7f6850134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:00.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f685006cb70 0x7f6850134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33510/0 (socket says 172.21.15.164:33510) 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f68500a4ab0 0x7f68501383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41802/0 (socket says 172.21.15.164:41802) 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 -- 172.21.15.164:0/3225798140 learned_addr learned my addr 172.21.15.164:0/3225798140 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): get_auth_request con 0x7f685006cb70 auth_method 0 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:00.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient(hunting): handle_auth_done global_id 74178 payload 274 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 -- 172.21.15.164:0/3225798140 >> v1:172.21.15.165:6789/0 conn(0x7f68500a46a0 legacy=0x7f6850137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 -- 172.21.15.164:0/3225798140 >> v1:172.21.15.164:6789/0 conn(0x7f68500a4ab0 legacy=0x7f68501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 monclient: found mon.noname-a 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.613+0000 7f684effd640 1 -- 172.21.15.164:0/3225798140 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685009cb70 con 0x7f685006cb70 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684effd640 1 -- 172.21.15.164:0/3225798140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f685006cb70 msgr2=0x7f6850134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:00.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684effd640 1 --2- 172.21.15.164:0/3225798140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f685006cb70 0x7f6850134f70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68400098a0 tx=0x7f684002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d104c1e3dc151d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 1 -- 172.21.15.164:0/3225798140 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840037070 con 0x7f685006cb70 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: dump: 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:00.625 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 1 -- 172.21.15.164:0/3225798140 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6840032640 con 0x7f685006cb70 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f68559b5640 5 monclient: authenticate success, global_id 74178 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f68559b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f68559b5640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 1 -- 172.21.15.164:0/3225798140 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6840032a10 con 0x7f685006cb70 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f68559b5640 4 set_mon_vals no callback set 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.616+0000 7f684dffb640 10 monclient: dump: 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:00.627 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 10 monclient: shutdown 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 -- 172.21.15.164:0/3225798140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f685006cb70 msgr2=0x7f6850134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 --2- 172.21.15.164:0/3225798140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f685006cb70 0x7f6850134f70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f68400098a0 tx=0x7f684002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 -- 172.21.15.164:0/3225798140 shutdown_connections 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 --2- 172.21.15.164:0/3225798140 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f685006cb70 0x7f6850134f70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 -- 172.21.15.164:0/3225798140 >> 172.21.15.164:0/3225798140 conn(0x7f68500aa900 msgr2=0x7f68500aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:00.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 -- 172.21.15.164:0/3225798140 shutdown_connections 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 1 -- 172.21.15.164:0/3225798140 wait complete. 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 10 monclient: build_initial_monmap 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.617+0000 7f68559b5640 10 monclient: monmap: 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:00.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:00.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:00.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 Processor -- start 2024-09-10T16:14:00.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 -- start start 2024-09-10T16:14:00.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient: init 2024-09-10T16:14:00.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:00.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:00.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 0x7f68500a3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:00.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-c con 0x7f685006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:00.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:00.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-b con 0x7f68500a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:00.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:00.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): picked mon.noname-a con 0x7f68500a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:00.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:00.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6850141050 con 0x7f68500a4ab0 2024-09-10T16:14:00.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:00.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f685006cf50 con 0x7f68500a46a0 2024-09-10T16:14:00.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:00.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f68501411d0 con 0x7f685006cb70 2024-09-10T16:14:00.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:00.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f68559b5640 10 monclient(hunting): authenticate will time out at 1219.106884s 2024-09-10T16:14:00.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f684f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:00.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.618+0000 7f684effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 0x7f68500a3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33526/0 (socket says 172.21.15.164:33526) 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 0x7f68500a3af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56782/0 (socket says 172.21.15.164:56782) 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 -- 172.21.15.164:0/713747109 learned_addr learned my addr 172.21.15.164:0/713747109 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): get_auth_request con 0x7f68500a4ab0 auth_method 0 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:00.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): get_auth_request con 0x7f685006cb70 auth_method 0 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:00.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 10 monclient(hunting): get_auth_request con 0x7f68500a46a0 auth_method 0 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:00.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:00.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:00.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:00.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:00.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 msgr2=0x7f68500a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:14:00.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 msgr2=0x7f68500a4030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:00.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:00.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684e7fc640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:00.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient(hunting): handle_auth_done global_id 74181 payload 824 2024-09-10T16:14:00.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 msgr2=0x7f68500a3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:00.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 0x7f68500a3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 msgr2=0x7f68500a4030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.619+0000 7f684f7fe640 1 -- 172.21.15.164:0/713747109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f685006e720 con 0x7f68500a4ab0 2024-09-10T16:14:00.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f684f7fe640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 msgr2=0x7f685009e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f684f7fe640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f684800c750 tx=0x7f684800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6848017070 con 0x7f68500a4ab0 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: dump: 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:00.647 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: _finish_auth 0 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f684800a860 con 0x7f68500a4ab0 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 5 monclient: authenticate success, global_id 74181 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 10 monclient: _renew_subs 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68500aa120 con 0x7f68500a4ab0 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f684800ce60 con 0x7f68500a4ab0 2024-09-10T16:14:00.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f6847fff640 10 monclient: dump: 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:00.649 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 10 monclient: _renew_subs 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68500a25c0 con 0x7f68500a4ab0 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.621+0000 7f684d7fa640 4 set_mon_vals no callback set 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f684801a070 con 0x7f68500a4ab0 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f68559b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f68559b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f68559b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6850134f70 con 0x7f68500a4ab0 2024-09-10T16:14:00.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f6847fff640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 0x7f68300406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.622+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(250..250 src has 184..250) ==== 4799+0+0 (secure 0 0 0) 0x7f6848052460 con 0x7f68500a4ab0 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.623+0000 7f684effd640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 0x7f68300406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.623+0000 7f684effd640 10 monclient: get_auth_request con 0x7f683003e210 auth_method 0 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.625+0000 7f684effd640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 0x7f68300406d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6850077610 tx=0x7f684002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.627+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6848050e50 con 0x7f68500a4ab0 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.627+0000 7f6847fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:00.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.627+0000 7f6847fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f68559b5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f68559b5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f68559b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f68500770f0 con 0x7f68500a4ab0 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f6847fff640 1 -- 172.21.15.164:0/713747109 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f6848052140 con 0x7f68500a4ab0 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f6847fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.867+0000 7f6847fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:00.870 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":72,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 msgr2=0x7f68300406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 0x7f68300406d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6850077610 tx=0x7f684002e000 comp rx=0 tx=0).stop 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 10 monclient: shutdown 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 msgr2=0x7f685009e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f684800c750 tx=0x7f684800cc10 comp rx=0 tx=0).stop 2024-09-10T16:14:00.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 shutdown_connections 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f683003e210 0x7f68300406d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f68500a4ab0 0x7f685009e960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f68500a46a0 0x7f68500a4030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 --2- 172.21.15.164:0/713747109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f685006cb70 0x7f68500a3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.869+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 >> 172.21.15.164:0/713747109 conn(0x7f68500aa900 msgr2=0x7f68500a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.870+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 shutdown_connections 2024-09-10T16:14:00.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:00.870+0000 7f68559b5640 1 -- 172.21.15.164:0/713747109 wait complete. 2024-09-10T16:14:01.160 INFO:tasks.rados.rados.0.smithi165.stdout:1014: done (1 left) 2024-09-10T16:14:01.160 INFO:tasks.rados.rados.0.smithi165.stdout:1016: done (0 left) 2024-09-10T16:14:01.160 INFO:tasks.rados.rados.0.smithi165.stdout:1017: snap_create 2024-09-10T16:14:02.153 INFO:tasks.rados.rados.0.smithi165.stdout:1017: done (0 left) 2024-09-10T16:14:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1018: rollback oid 4 current snap is 115 2024-09-10T16:14:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 112 2024-09-10T16:14:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1019: snap_remove snap 113 2024-09-10T16:14:03.159 INFO:tasks.rados.rados.0.smithi165.stdout:1019: done (1 left) 2024-09-10T16:14:03.160 INFO:tasks.rados.rados.0.smithi165.stdout:1020: snap_create 2024-09-10T16:14:03.160 INFO:tasks.rados.rados.0.smithi165.stdout:1018: finishing rollback tid 1 to smithi16534841-4 2024-09-10T16:14:03.160 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 284 (ObjNum 257 snap 0 seq_num 1162947968) dirty dne 2024-09-10T16:14:03.883 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:03.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: get_monmap_and_config 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: build_initial_monmap 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: monmap: 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:03.993223+0000 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:03.993223+0000 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:03.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:03.995 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:03.995 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:03.995 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 Processor -- start 2024-09-10T16:14:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 -- start start 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: init 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f536009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f536009ab50 0x7f53601397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-a con 0x7f536009ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 0x7f536013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-c con 0x7f536009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53600a0870 con 0x7f536009d930 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f536009d3e0 con 0x7f536009ab50 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5360074c70 con 0x7f536009e420 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f5365d52640 10 monclient(hunting): authenticate will time out at 1222.480655s 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f536009ab50 0x7f53601397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f536009d930 0x7f536009dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41806/0 (socket says 172.21.15.164:41806) 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f536009ab50 0x7f53601397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33542/0 (socket says 172.21.15.164:33542) 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535f7fe640 1 -- 172.21.15.164:0/431948148 learned_addr learned my addr 172.21.15.164:0/431948148 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f536009ab50 auth_method 0 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535ffff640 1 --2- 172.21.15.164:0/431948148 >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 0x7f536013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535ffff640 10 monclient(hunting): get_auth_request con 0x7f536009e420 auth_method 0 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.992+0000 7f535ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 10 monclient(hunting): handle_auth_done global_id 74184 payload 274 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 1 -- 172.21.15.164:0/431948148 >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 msgr2=0x7f536013bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 1 --2- 172.21.15.164:0/431948148 >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 0x7f536013bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 1 -- 172.21.15.164:0/431948148 >> v1:172.21.15.164:6789/0 conn(0x7f536009d930 legacy=0x7f536009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 1 monclient: found mon.noname-a 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535effd640 1 -- 172.21.15.164:0/431948148 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536009ecb0 con 0x7f536009ab50 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.993+0000 7f535ffff640 1 --2- 172.21.15.164:0/431948148 >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 0x7f536013bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.995+0000 7f535effd640 1 -- 172.21.15.164:0/431948148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 msgr2=0x7f53601397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.995+0000 7f535effd640 1 --2- 172.21.15.164:0/431948148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f53601397c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f536009af60 tx=0x7f534802b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f0a0d9e3de183d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 1 -- 172.21.15.164:0/431948148 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5348038070 con 0x7f536009ab50 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: dump: 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 5 monclient: authenticate success, global_id 74184 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 1 -- 172.21.15.164:0/431948148 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53480306b0 con 0x7f536009ab50 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 1 -- 172.21.15.164:0/431948148 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f534800b040 con 0x7f536009ab50 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 4 set_mon_vals no callback set 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f535e7fc640 10 monclient: dump: 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 10 monclient: shutdown 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 -- 172.21.15.164:0/431948148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 msgr2=0x7f53601397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 --2- 172.21.15.164:0/431948148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f53601397c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f536009af60 tx=0x7f534802b3a0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 -- 172.21.15.164:0/431948148 shutdown_connections 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 --2- 172.21.15.164:0/431948148 >> v2:172.21.15.165:3300/0 conn(0x7f536009e420 0x7f536013bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 --2- 172.21.15.164:0/431948148 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f53601397c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 -- 172.21.15.164:0/431948148 >> 172.21.15.164:0/431948148 conn(0x7f53600a3110 msgr2=0x7f53600a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 -- 172.21.15.164:0/431948148 shutdown_connections 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.996+0000 7f5365d52640 1 -- 172.21.15.164:0/431948148 wait complete. 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.997+0000 7f5365d52640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.997+0000 7f5365d52640 10 monclient: monmap: 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.997+0000 7f5365d52640 1 Processor -- start 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.997+0000 7f5365d52640 1 -- start start 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.997+0000 7f5365d52640 10 monclient: init 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-a con 0x7f536009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536009d930 0x7f53600735c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-c con 0x7f536009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 0x7f5360073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): picked mon.noname-b con 0x7f536009e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53600a2940 con 0x7f536009ab50 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f536009ef30 con 0x7f536009e420 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53600a2ac0 con 0x7f536009d930 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f5365d52640 10 monclient(hunting): authenticate will time out at 1222.486332s 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536009d930 0x7f53600735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33554/0 (socket says 172.21.15.164:33554) 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 -- 172.21.15.164:0/3272237341 learned_addr learned my addr 172.21.15.164:0/3272237341 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): get_auth_request con 0x7f536009ab50 auth_method 0 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): get_auth_request con 0x7f536009d930 auth_method 0 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535ffff640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 0x7f5360073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient(hunting): handle_auth_done global_id 74187 payload 824 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 -- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536009d930 msgr2=0x7f53600735c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536009d930 0x7f53600735c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 -- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 msgr2=0x7f5360073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 0x7f5360073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535f7fe640 1 -- 172.21.15.164:0/3272237341 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f536012f3d0 con 0x7f536009ab50 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535ffff640 10 monclient: get_auth_request con 0x7f536009e420 auth_method 0 2024-09-10T16:14:04.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:03.998+0000 7f535ffff640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 0x7f5360073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:14:04.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.000+0000 7f535f7fe640 1 -- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 msgr2=0x7f5360073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.000+0000 7f535f7fe640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f535800cc10 tx=0x7f535800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5358012400 con 0x7f536009ab50 2024-09-10T16:14:04.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: dump: 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.019 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53580125e0 con 0x7f536009ab50 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 5 monclient: authenticate success, global_id 74187 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 10 monclient: _renew_subs 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f536012f6b0 con 0x7f536009ab50 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53580129d0 con 0x7f536009ab50 2024-09-10T16:14:04.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5357fff640 10 monclient: dump: 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f535dffb640 4 set_mon_vals no callback set 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 10 monclient: _renew_subs 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.001+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f536009d620 con 0x7f536009ab50 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5358012bf0 con 0x7f536009ab50 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5365d52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5365d52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5365d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f536009dd10 con 0x7f536009ab50 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5357fff640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 0x7f5340040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(253..253 src has 184..253) ==== 4799+0+0 (secure 0 0 0) 0x7f5358053480 con 0x7f536009ab50 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f535effd640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 0x7f5340040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.002+0000 7f535effd640 10 monclient: get_auth_request con 0x7f534003e260 auth_method 0 2024-09-10T16:14:04.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.005+0000 7f535effd640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 0x7f5340040720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f536006d500 tx=0x7f534802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.006+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5358051e70 con 0x7f536009ab50 2024-09-10T16:14:04.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.006+0000 7f5357fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.006+0000 7f5357fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5365d52640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5365d52640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5365d52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f536006daa0 con 0x7f536009ab50 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5357fff640 1 -- 172.21.15.164:0/3272237341 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f5358053160 con 0x7f536009ab50 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5357fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.251+0000 7f5357fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:04.255 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:04.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 msgr2=0x7f5340040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 0x7f5340040720 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f536006d500 tx=0x7f534802f000 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 10 monclient: shutdown 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 msgr2=0x7f5360073080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f535800cc10 tx=0x7f535800a430 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 shutdown_connections 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f534003e260 0x7f5340040720 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f536009e420 0x7f5360073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f536009d930 0x7f53600735c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 --2- 172.21.15.164:0/3272237341 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f536009ab50 0x7f5360073080 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 >> 172.21.15.164:0/3272237341 conn(0x7f53600a3110 msgr2=0x7f53600994d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 shutdown_connections 2024-09-10T16:14:04.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.254+0000 7f5365d52640 1 -- 172.21.15.164:0/3272237341 wait complete. 2024-09-10T16:14:04.267 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:14:04.267 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:14:04.267 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: get_monmap_and_config 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: monmap: 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:04.365550+0000 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:04.365550+0000 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 Processor -- start 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 -- start start 2024-09-10T16:14:04.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: init 2024-09-10T16:14:04.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:14:04.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb3b409d930 0x7fb3b409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b409d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3b409ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb3b409e250 0x7fb3b4134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b409e250 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3b40a0870 con 0x7fb3b409ab50 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b409d3e0 con 0x7fb3b409d930 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb3b4074c70 con 0x7fb3b409e250 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.364+0000 7fb3bc382640 10 monclient(hunting): authenticate will time out at 1222.852927s 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb3b409d930 0x7fb3b409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb3b409d930 0x7fb3b409dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60134/0 (socket says 172.21.15.164:60134) 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 -- 172.21.15.164:0/4071886723 learned_addr learned my addr 172.21.15.164:0/4071886723 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3b991d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb3b409ab50 0x7fb3b4134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55144/0 (socket says 172.21.15.164:55144) 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): get_auth_request con 0x7fb3b409d930 auth_method 0 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba91f640 1 --2- 172.21.15.164:0/4071886723 >> v2:172.21.15.165:3300/0 conn(0x7fb3b409e250 0x7fb3b4134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba91f640 10 monclient(hunting): get_auth_request con 0x7fb3b409e250 auth_method 0 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba91f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba91f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba91f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_done global_id 74190 payload 274 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 -- 172.21.15.164:0/4071886723 >> v2:172.21.15.165:3300/0 conn(0x7fb3b409e250 msgr2=0x7fb3b4134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 --2- 172.21.15.164:0/4071886723 >> v2:172.21.15.165:3300/0 conn(0x7fb3b409e250 0x7fb3b4134c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 -- 172.21.15.164:0/4071886723 >> v1:172.21.15.164:6789/0 conn(0x7fb3b409ab50 legacy=0x7fb3b4134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 monclient: found mon.noname-a 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.365+0000 7fb3ba11e640 1 -- 172.21.15.164:0/4071886723 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b409ecb0 con 0x7fb3b409d930 2024-09-10T16:14:04.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3ba11e640 1 -- 172.21.15.164:0/4071886723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409d930 msgr2=0x7fb3b409dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3ba11e640 1 --2- 172.21.15.164:0/4071886723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409d930 0x7fb3b409dd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3b409a5a0 tx=0x7fb3b002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9cdfe52feb2b85c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 1 -- 172.21.15.164:0/4071886723 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0037070 con 0x7fb3b409d930 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: dump: 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.380 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 1 -- 172.21.15.164:0/4071886723 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b0032640 con 0x7fb3b409d930 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 5 monclient: authenticate success, global_id 74190 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 1 -- 172.21.15.164:0/4071886723 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0032a10 con 0x7fb3b409d930 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 4 set_mon_vals no callback set 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3b911c640 10 monclient: dump: 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.382 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 10 monclient: shutdown 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 -- 172.21.15.164:0/4071886723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409d930 msgr2=0x7fb3b409dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 --2- 172.21.15.164:0/4071886723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409d930 0x7fb3b409dd10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb3b409a5a0 tx=0x7fb3b002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 -- 172.21.15.164:0/4071886723 shutdown_connections 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 --2- 172.21.15.164:0/4071886723 >> v2:172.21.15.165:3300/0 conn(0x7fb3b409e250 0x7fb3b4134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 --2- 172.21.15.164:0/4071886723 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409d930 0x7fb3b409dd10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 -- 172.21.15.164:0/4071886723 >> 172.21.15.164:0/4071886723 conn(0x7fb3b40a3110 msgr2=0x7fb3b40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 -- 172.21.15.164:0/4071886723 shutdown_connections 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.368+0000 7fb3bc382640 1 -- 172.21.15.164:0/4071886723 wait complete. 2024-09-10T16:14:04.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.369+0000 7fb3bc382640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.369+0000 7fb3bc382640 10 monclient: monmap: 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.369+0000 7fb3bc382640 1 Processor -- start 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.369+0000 7fb3bc382640 1 -- start start 2024-09-10T16:14:04.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.369+0000 7fb3bc382640 10 monclient: init 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 0x7fb3b4072530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-a con 0x7fb3b409ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b409d930 0x7fb3b4072a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-c con 0x7fb3b409d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3b409e250 0x7fb3b4072fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): picked mon.noname-b con 0x7fb3b409e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b409e660 con 0x7fb3b409ab50 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b409ef30 con 0x7fb3b409e250 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb3b40a2940 con 0x7fb3b409d930 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3bc382640 10 monclient(hunting): authenticate will time out at 1222.858213s 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 0x7fb3b4072530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 0x7fb3b4072530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60140/0 (socket says 172.21.15.164:60140) 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b409d930 0x7fb3b4072a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 -- 172.21.15.164:0/2163110999 learned_addr learned my addr 172.21.15.164:0/2163110999 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): get_auth_request con 0x7fb3b409ab50 auth_method 0 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): get_auth_request con 0x7fb3b409d930 auth_method 0 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba91f640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3b409e250 0x7fb3b4072fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba91f640 10 monclient(hunting): get_auth_request con 0x7fb3b409e250 auth_method 0 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba91f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba91f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba91f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient(hunting): handle_auth_done global_id 74193 payload 824 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b409d930 msgr2=0x7fb3b4072a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b409d930 0x7fb3b4072a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3b409e250 msgr2=0x7fb3b4072fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3b409e250 0x7fb3b4072fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 monclient: found mon.noname-a 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3ba11e640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3b4090400 con 0x7fb3b409ab50 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.370+0000 7fb3b991d640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb3b409d930 0x7fb3b4072a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3ba11e640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 msgr2=0x7fb3b4072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3ba11e640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 0x7fb3b4072530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0041070 con 0x7fb3b409ab50 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 10 monclient: dump: 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3b003c880 con 0x7fb3b409ab50 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3bc382640 5 monclient: authenticate success, global_id 74193 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3bc382640 10 monclient: _renew_subs 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3bc382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.372+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3b408eed0 con 0x7fb3b409ab50 2024-09-10T16:14:04.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3b0037040 con 0x7fb3b409ab50 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 10 monclient: dump: 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3b891b640 4 set_mon_vals no callback set 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 10 monclient: _renew_subs 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3b4093bd0 con 0x7fb3b409ab50 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb3b0044070 con 0x7fb3b409ab50 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 0x7fb3b4097ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.373+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b4050fd0 con 0x7fb3b40976e0 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 0x7fb3b4097ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3aaffd640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39803e210 0x7fb3980406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient: get_auth_request con 0x7fb3b40976e0 auth_method 0 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(253..253 src has 184..253) ==== 4799+0+0 (secure 0 0 0) 0x7fb3b0073590 con 0x7fb3b409ab50 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 10 monclient(hunting): handle_auth_done global_id 74196 payload 824 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3b991d640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3b409dd10 con 0x7fb3b40976e0 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3ba91f640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39803e210 0x7fb3980406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.374+0000 7fb3ba91f640 10 monclient: get_auth_request con 0x7fb39803e210 auth_method 0 2024-09-10T16:14:04.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3b991d640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 msgr2=0x7fb3b4097ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3b991d640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 0x7fb3b4097ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3b408e7d0 tx=0x7fb3a400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4017070 con 0x7fb3b40976e0 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 10 monclient: dump: 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.394 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3ba91f640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39803e210 0x7fb3980406d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009f80 tx=0x7fb3ac00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb3a400cd30 con 0x7fb3b40976e0 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 msgr2=0x7fb3b4097ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 0x7fb3b4097ac0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3b408e7d0 tx=0x7fb3a400ef70 comp rx=0 tx=0).stop 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 msgr2=0x7fb3b4097ac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.376+0000 7fb3aaffd640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b40976e0 0x7fb3b4097ac0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3bc382640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3bc382640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 0x7fb3b41d56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3bc382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb3b408ffb0 con 0x7fb3b4098000 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 0x7fb3b41d56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient: get_auth_request con 0x7fb3b4098000 auth_method 0 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 10 monclient(hunting): handle_auth_done global_id 74199 payload 824 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.383+0000 7fb3b991d640 1 -- 172.21.15.164:0/2163110999 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3b4077260 con 0x7fb3b4098000 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3b991d640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 msgr2=0x7fb3b41d56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3b991d640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 0x7fb3b41d56e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4016000 tx=0x7fb3a40058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3a4017070 con 0x7fb3b4098000 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 10 monclient: dump: 2024-09-10T16:14:04.398 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fb3a4013680 con 0x7fb3b4098000 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 msgr2=0x7fb3b41d56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 0x7fb3b41d56e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4016000 tx=0x7fb3a40058c0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 msgr2=0x7fb3b41d56e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.385+0000 7fb3aaffd640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b4098000 0x7fb3b41d56e0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4016000 tx=0x7fb3a40058c0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 72252, 2024-09-10T16:14:04.400 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.401 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.402 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:04.403 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:04.404 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.405 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:04.407 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.410 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:04.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:04.412 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39803e210 msgr2=0x7fb3980406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39803e210 0x7fb3980406d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009f80 tx=0x7fb3ac00b040 comp rx=0 tx=0).stop 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 10 monclient: shutdown 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 msgr2=0x7fb3b4072530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 --2- 172.21.15.164:0/2163110999 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3b409ab50 0x7fb3b4072530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b00046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3ba91f640 1 -- 172.21.15.164:0/2163110999 reap_dead start 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3ba91f640 1 -- 172.21.15.164:0/2163110999 reap_dead start 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 shutdown_connections 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 >> 172.21.15.164:0/2163110999 conn(0x7fb3b40a3110 msgr2=0x7fb3b40a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 shutdown_connections 2024-09-10T16:14:04.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.386+0000 7fb3bc382640 1 -- 172.21.15.164:0/2163110999 wait complete. 2024-09-10T16:14:04.415 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:04.415 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:14:04.415 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:14:04.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 10 monclient: get_monmap_and_config 2024-09-10T16:14:04.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 10 monclient: monmap: 2024-09-10T16:14:04.515 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:04.514876+0000 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:04.514876+0000 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 1 Processor -- start 2024-09-10T16:14:04.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 1 -- start start 2024-09-10T16:14:04.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.513+0000 7f7b472c6640 10 monclient: init 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b4009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7b4009ab50 0x7f7b401397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4009ab50 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4009e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b400a0870 con 0x7f7b4009d930 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7b4009d3e0 con 0x7f7b4009ab50 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b40074c70 con 0x7f7b4009e420 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b472c6640 10 monclient(hunting): authenticate will time out at 1223.002217s 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45062640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7b4009d930 0x7f7b4009dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55154/0 (socket says 172.21.15.164:55154) 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57968/0 (socket says 172.21.15.164:57968) 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45062640 1 -- 172.21.15.164:0/3031355907 learned_addr learned my addr 172.21.15.164:0/3031355907 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): get_auth_request con 0x7f7b4009e420 auth_method 0 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b44861640 1 --2- 172.21.15.164:0/3031355907 >> v2:172.21.15.165:3300/0 conn(0x7f7b4009ab50 0x7f7b401397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b44861640 10 monclient(hunting): get_auth_request con 0x7f7b4009ab50 auth_method 0 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b44861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b44861640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b44861640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient(hunting): handle_auth_done global_id 5054 payload 274 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 -- 172.21.15.164:0/3031355907 >> v2:172.21.15.165:3300/0 conn(0x7f7b4009ab50 msgr2=0x7f7b401397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 --2- 172.21.15.164:0/3031355907 >> v2:172.21.15.165:3300/0 conn(0x7f7b4009ab50 0x7f7b401397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 -- 172.21.15.164:0/3031355907 >> v1:172.21.15.164:6789/0 conn(0x7f7b4009d930 legacy=0x7f7b4009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 monclient: found mon.noname-b 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:04.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.514+0000 7f7b45863640 1 -- 172.21.15.164:0/3031355907 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b4009ecb0 con 0x7f7b4009e420 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b45863640 1 -- 172.21.15.164:0/3031355907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 msgr2=0x7f7b4013bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b45863640 1 --2- 172.21.15.164:0/3031355907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009d90 tx=0x7f7b3002a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9e026994e8f3542 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 1 -- 172.21.15.164:0/3031355907 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30007950 con 0x7f7b4009e420 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:04.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: dump: 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 1 -- 172.21.15.164:0/3031355907 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b30007af0 con 0x7f7b4009e420 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b472c6640 5 monclient: authenticate success, global_id 5054 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b472c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b472c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 1 -- 172.21.15.164:0/3031355907 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b30031400 con 0x7f7b4009e420 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b472c6640 4 set_mon_vals no callback set 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.517+0000 7f7b3ffff640 10 monclient: dump: 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.535 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 10 monclient: shutdown 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 -- 172.21.15.164:0/3031355907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 msgr2=0x7f7b4013bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 --2- 172.21.15.164:0/3031355907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009d90 tx=0x7f7b3002a490 comp rx=0 tx=0).stop 2024-09-10T16:14:04.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 -- 172.21.15.164:0/3031355907 shutdown_connections 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 --2- 172.21.15.164:0/3031355907 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009e420 0x7f7b4013bd10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 --2- 172.21.15.164:0/3031355907 >> v2:172.21.15.165:3300/0 conn(0x7f7b4009ab50 0x7f7b401397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 -- 172.21.15.164:0/3031355907 >> 172.21.15.164:0/3031355907 conn(0x7f7b400a3110 msgr2=0x7f7b400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 -- 172.21.15.164:0/3031355907 shutdown_connections 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 1 -- 172.21.15.164:0/3031355907 wait complete. 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.518+0000 7f7b472c6640 10 monclient: monmap: 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.537 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 Processor -- start 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 -- start start 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient: init 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 0x7f7b401347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-a con 0x7f7b4009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009d930 0x7f7b40134d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4009e420 0x7f7b40096010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4009e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7b400a2940 con 0x7f7b4009ab50 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7b4009ef30 con 0x7f7b4009e420 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b400a2ac0 con 0x7f7b4009d930 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b472c6640 10 monclient(hunting): authenticate will time out at 1223.007643s 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 0x7f7b401347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009d930 0x7f7b40134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 0x7f7b401347f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60162/0 (socket says 172.21.15.164:60162) 2024-09-10T16:14:04.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 1 -- 172.21.15.164:0/2712100586 learned_addr learned my addr 172.21.15.164:0/2712100586 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): get_auth_request con 0x7f7b4009ab50 auth_method 0 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): get_auth_request con 0x7f7b4009d930 auth_method 0 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45062640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45863640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4009e420 0x7f7b40096010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45863640 10 monclient(hunting): get_auth_request con 0x7f7b4009e420 auth_method 0 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45863640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45863640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.519+0000 7f7b45863640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 10 monclient(hunting): handle_auth_done global_id 74202 payload 824 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009d930 msgr2=0x7f7b40134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009d930 0x7f7b40134d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4009e420 msgr2=0x7f7b40096010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4009e420 0x7f7b40096010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 monclient: found mon.noname-a 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45062640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b40099960 con 0x7f7b4009ab50 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b45863640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b4009e420 0x7f7b40096010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.520+0000 7f7b44861640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b4009d930 0x7f7b40134d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b45062640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 msgr2=0x7f7b401347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b45062640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 0x7f7b401347f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c460 tx=0x7f7b3800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38017070 con 0x7f7b4009ab50 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: dump: 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b380070d0 con 0x7f7b4009ab50 2024-09-10T16:14:04.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 5 monclient: authenticate success, global_id 74202 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 10 monclient: _renew_subs 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b40099100 con 0x7f7b4009ab50 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b380074c0 con 0x7f7b4009ab50 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3dffb640 10 monclient: dump: 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 10 monclient: _renew_subs 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b3f7fe640 4 set_mon_vals no callback set 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.522+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b40099680 con 0x7f7b4009ab50 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b472c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b472c6640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 0x7f7b40096930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b4009c710 con 0x7f7b40096550 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 0x7f7b40096930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient: get_auth_request con 0x7f7b40096550 auth_method 0 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7b3801a070 con 0x7f7b4009ab50 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 10 monclient(hunting): handle_auth_done global_id 5060 payload 824 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b44861640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b4009e9d0 con 0x7f7b40096550 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.523+0000 7f7b3dffb640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b2c03e240 0x7f7b2c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.524+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(254..254 src has 184..254) ==== 4771+0+0 (secure 0 0 0) 0x7f7b38052230 con 0x7f7b4009ab50 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.524+0000 7f7b45863640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b2c03e240 0x7f7b2c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.524+0000 7f7b45863640 10 monclient: get_auth_request con 0x7f7b2c03e240 auth_method 0 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b44861640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 msgr2=0x7f7b40096930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b44861640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 0x7f7b40096930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b4009a080 tx=0x7f7b3400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34018020 con 0x7f7b40096550 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 10 monclient: dump: 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b34010e70 con 0x7f7b40096550 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 msgr2=0x7f7b40096930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 0x7f7b40096930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b4009a080 tx=0x7f7b3400e5a0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 msgr2=0x7f7b40096930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b3dffb640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096550 0x7f7b40096930 secure :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f7b4009a080 tx=0x7f7b3400e5a0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.526+0000 7f7b45863640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b2c03e240 0x7f7b2c040700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b4006d500 tx=0x7f7b3002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b472c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b472c6640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 0x7f7b4009b000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b472c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b4009ca30 con 0x7f7b40096e70 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 0x7f7b4009b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient: get_auth_request con 0x7f7b40096e70 auth_method 0 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 10 monclient(hunting): handle_auth_done global_id 5063 payload 824 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.533+0000 7f7b44861640 1 -- 172.21.15.164:0/2712100586 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b4006daa0 con 0x7f7b40096e70 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b44861640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 msgr2=0x7f7b4009b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b44861640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 0x7f7b4009b000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7b34019000 tx=0x7f7b340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34018030 con 0x7f7b40096e70 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 10 monclient: dump: 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f7b34003820 con 0x7f7b40096e70 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 msgr2=0x7f7b4009b000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 0x7f7b4009b000 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7b34019000 tx=0x7f7b340046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 msgr2=0x7f7b4009b000 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b3dffb640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b40096e70 0x7f7b4009b000 secure :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f7b34019000 tx=0x7f7b340046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 486707, 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:04.554 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.555 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.556 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:04.557 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.558 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:04.559 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:04.561 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.562 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.564 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:04.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.566 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:04.567 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b2c03e240 msgr2=0x7f7b2c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b2c03e240 0x7f7b2c040700 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7b4006d500 tx=0x7f7b3002f000 comp rx=0 tx=0).stop 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 10 monclient: shutdown 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 msgr2=0x7f7b401347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 --2- 172.21.15.164:0/2712100586 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b4009ab50 0x7f7b401347f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800c460 tx=0x7f7b3800c920 comp rx=0 tx=0).stop 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b45863640 1 -- 172.21.15.164:0/2712100586 reap_dead start 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b45863640 1 -- 172.21.15.164:0/2712100586 reap_dead start 2024-09-10T16:14:04.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 shutdown_connections 2024-09-10T16:14:04.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.535+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 >> 172.21.15.164:0/2712100586 conn(0x7f7b400a3110 msgr2=0x7f7b40093590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.536+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 shutdown_connections 2024-09-10T16:14:04.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.536+0000 7f7b472c6640 1 -- 172.21.15.164:0/2712100586 wait complete. 2024-09-10T16:14:04.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:04.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:14:04.570 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:14:04.595 INFO:tasks.rados.rados.0.smithi165.stdout:1018: done (1 left) 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1020: done (0 left) 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1021: read oid 22 snap -1 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1021: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1022: rollback oid 36 current snap is 116 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 114 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1023: read oid 45 snap -1 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1023: expect deleted 2024-09-10T16:14:04.596 INFO:tasks.rados.rados.0.smithi165.stdout:1024: snap_create 2024-09-10T16:14:04.597 INFO:tasks.rados.rados.0.smithi165.stdout:1022: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:14:04.597 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 0 (ObjNum 129 snap 0 seq_num 1162983128) dirty dne 2024-09-10T16:14:04.606 INFO:tasks.rados.rados.0.smithi165.stdout:1021: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:14:04.646 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:04.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 10 monclient: get_monmap_and_config 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 10 monclient: monmap: 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:04.669995+0000 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:04.669995+0000 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.671 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 1 Processor -- start 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 1 -- start start 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.668+0000 7f42e3eb8640 10 monclient: init 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42dc09c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42dc09d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 0x7f42dc13d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc13b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42dc09fbc0 con 0x7f42dc09d190 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42dc0a2a20 con 0x7f42dc09c310 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42dc09f860 con 0x7f42dc13b5d0 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e3eb8640 10 monclient(hunting): authenticate will time out at 1223.157431s 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 0x7f42dc13d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e1453640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f42dc09d190 0x7f42dc09f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55160/0 (socket says 172.21.15.164:55160) 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e1453640 1 -- 172.21.15.164:0/2675669606 learned_addr learned my addr 172.21.15.164:0/2675669606 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): get_auth_request con 0x7f42dc13b5d0 auth_method 0 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient(hunting): handle_auth_done global_id 5066 payload 274 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 1 -- 172.21.15.164:0/2675669606 >> v1:172.21.15.165:6789/0 conn(0x7f42dc09c310 legacy=0x7f42dc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 1 -- 172.21.15.164:0/2675669606 >> v1:172.21.15.164:6789/0 conn(0x7f42dc09d190 legacy=0x7f42dc09f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 1 monclient: found mon.noname-b 2024-09-10T16:14:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.669+0000 7f42e2455640 1 -- 172.21.15.164:0/2675669606 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42dc09c6f0 con 0x7f42dc13b5d0 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e2455640 1 -- 172.21.15.164:0/2675669606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 msgr2=0x7f42dc13d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e2455640 1 --2- 172.21.15.164:0/2675669606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 0x7f42dc13d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009dc0 tx=0x7f42cc02a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362226569f7e1354 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 1 -- 172.21.15.164:0/2675669606 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc007a30 con 0x7f42dc13b5d0 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: dump: 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 1 -- 172.21.15.164:0/2675669606 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42cc007bd0 con 0x7f42dc13b5d0 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e3eb8640 5 monclient: authenticate success, global_id 5066 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e3eb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e3eb8640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:04.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 1 -- 172.21.15.164:0/2675669606 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc031400 con 0x7f42dc13b5d0 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e3eb8640 4 set_mon_vals no callback set 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.672+0000 7f42e0c52640 10 monclient: dump: 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 10 monclient: shutdown 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2675669606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 msgr2=0x7f42dc13d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2675669606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 0x7f42dc13d9c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f42cc009dc0 tx=0x7f42cc02a480 comp rx=0 tx=0).stop 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2675669606 shutdown_connections 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2675669606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc13b5d0 0x7f42dc13d9c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2675669606 >> 172.21.15.164:0/2675669606 conn(0x7f42dc0a3110 msgr2=0x7f42dc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2675669606 shutdown_connections 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2675669606 wait complete. 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 10 monclient: build_initial_monmap 2024-09-10T16:14:04.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.673+0000 7f42e3eb8640 10 monclient: monmap: 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 Processor -- start 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 -- start start 2024-09-10T16:14:04.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient: init 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09c310 0x7f42dc099720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-b con 0x7f42dc09c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 0x7f42dc08ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-a con 0x7f42dc13b5d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:04.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 0x7f42dc0908d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): picked mon.noname-c con 0x7f42dc090480 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc0a20b0 con 0x7f42dc13b5d0 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc0a2230 con 0x7f42dc09c310 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42dc074e00 con 0x7f42dc090480 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e3eb8640 10 monclient(hunting): authenticate will time out at 1223.162755s 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 0x7f42dc08ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 0x7f42dc0908d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 0x7f42dc08ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60168/0 (socket says 172.21.15.164:60168) 2024-09-10T16:14:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 1 -- 172.21.15.164:0/2519072177 learned_addr learned my addr 172.21.15.164:0/2519072177 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): get_auth_request con 0x7f42dc090480 auth_method 0 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): get_auth_request con 0x7f42dc13b5d0 auth_method 0 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09c310 0x7f42dc099720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1c54640 10 monclient(hunting): get_auth_request con 0x7f42dc09c310 auth_method 0 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e2455640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.674+0000 7f42e1453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e1453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 10 monclient(hunting): handle_auth_done global_id 5069 payload 824 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09c310 msgr2=0x7f42dc099720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09c310 0x7f42dc099720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 msgr2=0x7f42dc08ff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 0x7f42dc08ff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 10 monclient: _finish_hunting 0 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 monclient: found mon.noname-c 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42dc13d940 con 0x7f42dc090480 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.675+0000 7f42e1453640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42dc13b5d0 0x7f42dc08ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 msgr2=0x7f42dc0908d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e2455640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 0x7f42dc0908d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f42cc031000 tx=0x7f42cc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc040070 con 0x7f42dc090480 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: dump: 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42cc045460 con 0x7f42dc090480 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 5 monclient: authenticate success, global_id 5069 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 10 monclient: _renew_subs 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42dc093a80 con 0x7f42dc090480 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42cc007a30 con 0x7f42dc090480 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42da7fc640 10 monclient: dump: 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.695 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 10 monclient: _renew_subs 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42dbfff640 4 set_mon_vals no callback set 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:04.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.677+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42dc092560 con 0x7f42dc090480 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e3eb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc09a580 con 0x7f42dc09d190 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f42cc039040 con 0x7f42dc090480 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient: get_auth_request con 0x7f42dc09d190 auth_method 0 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42da7fc640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42c803e210 0x7f42c80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(254..254 src has 184..254) ==== 4771+0+0 (secure 0 0 0) 0x7f42cc0747e0 con 0x7f42dc090480 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 10 monclient(hunting): handle_auth_done global_id 84118 payload 824 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42dc0936e0 con 0x7f42dc09d190 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1453640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42c803e210 0x7f42c80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.678+0000 7f42e1453640 10 monclient: get_auth_request con 0x7f42c803e210 auth_method 0 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.679+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 msgr2=0x7f42dc0729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.679+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 msgr2=0x7f42dc0729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.679+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f42d400f770 tx=0x7f42d400fc30 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:04.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.679+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:04.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.681+0000 7f42e1453640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42c803e210 0x7f42c80406d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f42d0007d20 tx=0x7f42d0007c70 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.699 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2063712 2024-09-10T16:14:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: get_auth_request con 0x7f42dc09d190 auth_method 0 2024-09-10T16:14:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: _init_auth method 2 2024-09-10T16:14:04.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: _init_auth already have auth, reseting 2024-09-10T16:14:04.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T16:14:04.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T16:14:04.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.880+0000 7f42e1c54640 10 monclient: handle_auth_done global_id 84118 payload 293 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 msgr2=0x7f42dc0729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42d4003b60 tx=0x7f42d4007e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d401a040 con 0x7f42dc09d190 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 10 monclient: dump: 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.885 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42d4023070 con 0x7f42dc09d190 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:04.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 msgr2=0x7f42dc0729f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42d4003b60 tx=0x7f42d4007e30 comp rx=0 tx=0).stop 2024-09-10T16:14:04.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 msgr2=0x7f42dc0729f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.882+0000 7f42da7fc640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc09d190 0x7f42dc0729f0 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f42d4003b60 tx=0x7f42d4007e30 comp rx=0 tx=0).stop 2024-09-10T16:14:04.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e3eb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:04.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 0x7f42dc073340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:04.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e3eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:04.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42dc09ea40 con 0x7f42dc072f30 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 0x7f42dc073340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient: get_auth_request con 0x7f42dc072f30 auth_method 0 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:04.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:04.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:04.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:04.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 10 monclient(hunting): handle_auth_done global_id 84121 payload 824 2024-09-10T16:14:04.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.889+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42dc077260 con 0x7f42dc072f30 2024-09-10T16:14:04.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e1c54640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 msgr2=0x7f42dc073340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:04.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e1c54640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 0x7f42dc073340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f42d401f000 tx=0x7f42d4003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:04.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4007460 con 0x7f42dc072f30 2024-09-10T16:14:04.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:04.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 10 monclient: dump: 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:04.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f42d4007460 con 0x7f42dc072f30 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 msgr2=0x7f42dc073340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 0x7f42dc073340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f42d401f000 tx=0x7f42d4003760 comp rx=0 tx=0).stop 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 msgr2=0x7f42dc073340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42da7fc640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42dc072f30 0x7f42dc073340 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f42d401f000 tx=0x7f42d4003760 comp rx=0 tx=0).stop 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:04.899 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7688, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:14:04.900 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.901 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.902 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:04.903 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:04.904 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:04.905 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.906 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.907 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.908 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.909 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:04.910 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:04.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.912 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:04.913 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:04.914 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:04.914 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:04.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:04.914 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:04.914 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:04.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42c803e210 msgr2=0x7f42c80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42c803e210 0x7f42c80406d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f42d0007d20 tx=0x7f42d0007c70 comp rx=0 tx=0).stop 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 10 monclient: shutdown 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 msgr2=0x7f42dc0908d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 --2- 172.21.15.164:0/2519072177 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42dc090480 0x7f42dc0908d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f42cc031000 tx=0x7f42cc004a40 comp rx=0 tx=0).stop 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 reap_dead start 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e2455640 1 -- 172.21.15.164:0/2519072177 reap_dead start 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 shutdown_connections 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 >> 172.21.15.164:0/2519072177 conn(0x7f42dc0a3110 msgr2=0x7f42dc09e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 shutdown_connections 2024-09-10T16:14:04.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:04.893+0000 7f42e3eb8640 1 -- 172.21.15.164:0/2519072177 wait complete. 2024-09-10T16:14:04.917 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:04.917 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:14:04.917 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: get_monmap_and_config 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: build_initial_monmap 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: monmap: 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:05.017379+0000 2024-09-10T16:14:05.018 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:05.017379+0000 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 Processor -- start 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 -- start start 2024-09-10T16:14:05.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: init 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc2fc06cb70 0x7fc2fc134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): picked mon.noname-a con 0x7fc2fc06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 0x7fc2fc0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): picked mon.noname-b con 0x7fc2fc0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2fc137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2fc09ce50 con 0x7fc2fc137ca0 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc2fc06db60 con 0x7fc2fc06cb70 2024-09-10T16:14:05.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc2fc06cf50 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc301352640 10 monclient(hunting): authenticate will time out at 1223.504658s 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc2fc06cb70 0x7fc2fc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc2fc06cb70 0x7fc2fc134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60174/0 (socket says 172.21.15.164:60174) 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 1 -- 172.21.15.164:0/2460683774 learned_addr learned my addr 172.21.15.164:0/2460683774 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): get_auth_request con 0x7fc2fc06cb70 auth_method 0 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 0x7fc2fc0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc2fc0a46a0 auth_method 0 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.016+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 10 monclient(hunting): handle_auth_done global_id 74208 payload 274 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 1 -- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 msgr2=0x7fc2fc0a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 0x7fc2fc0a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 1 -- 172.21.15.164:0/2460683774 >> v1:172.21.15.165:6789/0 conn(0x7fc2fc137ca0 legacy=0x7fc2fc138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 1 monclient: found mon.noname-a 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2faffd640 1 -- 172.21.15.164:0/2460683774 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc09cb70 con 0x7fc2fc06cb70 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.017+0000 7fc2fa7fc640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 0x7fc2fc0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2faffd640 1 -- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc06cb70 msgr2=0x7fc2fc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc134f70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009870 tx=0x7fc2ec02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0465f90376412b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2f9ffb640 1 -- 172.21.15.164:0/2460683774 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec037070 con 0x7fc2fc06cb70 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2f9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:05.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.019+0000 7fc2f9ffb640 10 monclient: dump: 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:05.030 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 1 -- 172.21.15.164:0/2460683774 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ec032440 con 0x7fc2fc06cb70 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 5 monclient: authenticate success, global_id 74208 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 1 -- 172.21.15.164:0/2460683774 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ec032810 con 0x7fc2fc06cb70 2024-09-10T16:14:05.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 4 set_mon_vals no callback set 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc2f9ffb640 10 monclient: dump: 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:05.032 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 10 monclient: shutdown 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 -- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc06cb70 msgr2=0x7fc2fc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc134f70 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2ec009870 tx=0x7fc2ec02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 -- 172.21.15.164:0/2460683774 shutdown_connections 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc0a46a0 0x7fc2fc0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 --2- 172.21.15.164:0/2460683774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc134f70 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 -- 172.21.15.164:0/2460683774 >> 172.21.15.164:0/2460683774 conn(0x7fc2fc0aa900 msgr2=0x7fc2fc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 -- 172.21.15.164:0/2460683774 shutdown_connections 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.020+0000 7fc301352640 1 -- 172.21.15.164:0/2460683774 wait complete. 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.021+0000 7fc301352640 10 monclient: build_initial_monmap 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.021+0000 7fc301352640 10 monclient: monmap: 2024-09-10T16:14:05.033 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.021+0000 7fc301352640 1 Processor -- start 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.021+0000 7fc301352640 1 -- start start 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.021+0000 7fc301352640 10 monclient: init 2024-09-10T16:14:05.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc12e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): picked mon.noname-b con 0x7fc2fc06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): picked mon.noname-a con 0x7fc2fc0a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc137ca0 0x7fc2fc1cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): picked mon.noname-c con 0x7fc2fc137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc1389c0 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:05.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc2fc138840 con 0x7fc2fc06cb70 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc2fc140040 con 0x7fc2fc137ca0 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc301352640 10 monclient(hunting): authenticate will time out at 1223.510139s 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc137ca0 0x7fc2fc1cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60178/0 (socket says 172.21.15.164:60178) 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 -- 172.21.15.164:0/2877044829 learned_addr learned my addr 172.21.15.164:0/2877044829 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): get_auth_request con 0x7fc2fc0a46a0 auth_method 0 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:05.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): get_auth_request con 0x7fc2fc137ca0 auth_method 0 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc12e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): get_auth_request con 0x7fc2fc06cb70 auth_method 0 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:05.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient(hunting): handle_auth_done global_id 74211 payload 824 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 -- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc137ca0 msgr2=0x7fc2fc1cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc137ca0 0x7fc2fc1cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 -- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 msgr2=0x7fc2fc12e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc12e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:14:05.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 monclient: found mon.noname-a 2024-09-10T16:14:05.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:05.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2fa7fc640 1 -- 172.21.15.164:0/2877044829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2fc0a3270 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.022+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc12e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:05.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2fa7fc640 1 -- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 msgr2=0x7fc2fc12e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:05.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2fa7fc640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ebe0 tx=0x7fc2f000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0017070 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: dump: 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:05.042 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2f000ce40 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 5 monclient: authenticate success, global_id 74211 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 10 monclient: _renew_subs 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2fc0a3550 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2f0013690 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:05.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2e37fe640 10 monclient: dump: 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 10 monclient: _renew_subs 2024-09-10T16:14:05.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2fc06e720 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.024+0000 7fc2f97fa640 4 set_mon_vals no callback set 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc2f001a070 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc301352640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc301352640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc301352640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2fc134f70 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc2e37fe640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 0x7fc2e8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(254..254 src has 184..254) ==== 4771+0+0 (secure 0 0 0) 0x7fc2f0052c80 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 0x7fc2e8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.026+0000 7fc2faffd640 10 monclient: get_auth_request con 0x7fc2e803e240 auth_method 0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.029+0000 7fc2faffd640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 0x7fc2e8040700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc074d10 tx=0x7fc2ec02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:05.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.030+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc2f0051680 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.030+0000 7fc2e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:05.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.030+0000 7fc2e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:05.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.273+0000 7fc301352640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:14:05.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.273+0000 7fc301352640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:14:05.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.273+0000 7fc301352640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:05.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.273+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc2fc12edf0 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.274+0000 7fc2e37fe640 1 -- 172.21.15.164:0/2877044829 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc2f0023d00 con 0x7fc2fc0a46a0 2024-09-10T16:14:05.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.274+0000 7fc2e37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:14:05.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.274+0000 7fc2e37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:05.276 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.277 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3527735661,config=2586702516,health=3550891638,kv=805391194,logm=3953794799}) 2024-09-10T16:14:05.277 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=650558605,config=2586702516,health=1824611349,kv=805391194,logm=3990307618}) 2024-09-10T16:14:05.277 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.277 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=2,health=11,kv=12,logm=62} crc {auth=3527735661,config=2586702516,health=3550891638,kv=805391194,logm=3953794799}) 2024-09-10T16:14:05.277 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.275+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=11,kv=12,logm=60} crc {auth=2065507860,config=2586702516,health=3820421734,kv=805391194,logm=1913636523}) 2024-09-10T16:14:05.278 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.278 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490334753}) 2024-09-10T16:14:05.278 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1136823880}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490334753}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.276+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=561006049}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211622276}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4083796671}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1211622276}) 2024-09-10T16:14:05.279 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1523975830}) 2024-09-10T16:14:05.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 msgr2=0x7fc2e8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:05.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 0x7fc2e8040700 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc2fc074d10 tx=0x7fc2ec02e000 comp rx=0 tx=0).stop 2024-09-10T16:14:05.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 10 monclient: shutdown 2024-09-10T16:14:05.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:05.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 msgr2=0x7fc2fc12e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fc2f000ebe0 tx=0x7fc2f000c520 comp rx=0 tx=0).stop 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 shutdown_connections 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc2e803e240 0x7fc2e8040700 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc2fc137ca0 0x7fc2fc1cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc2fc0a46a0 0x7fc2fc12e8b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 --2- 172.21.15.164:0/2877044829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc2fc06cb70 0x7fc2fc12e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 >> 172.21.15.164:0/2877044829 conn(0x7fc2fc0aa900 msgr2=0x7fc2fc0a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 shutdown_connections 2024-09-10T16:14:05.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:05.277+0000 7fc301352640 1 -- 172.21.15.164:0/2877044829 wait complete. 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001228865}) 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779203249}) 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1001228865}) 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.278+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1443393945}) 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131593809}) 2024-09-10T16:14:05.285 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3569871802}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=131593809}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2714804703}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968943480}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043984452}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968943480}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.279+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1007479963}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347168529}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2532367707}) 2024-09-10T16:14:05.286 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1347168529}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.280+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1136205199}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690067823}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1358650592}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=690067823}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.281+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4025460778}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975003770}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3792757845}) 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.287 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975003770}) 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.282+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3514649361}) 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966536148}) 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1831699215}) 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2966536148}) 2024-09-10T16:14:05.288 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.283+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411588896}) 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613637346}) 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021850241}) 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2613637346}) 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.284+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1594478235}) 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.293 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830712246}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1504573341}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830712246}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.285+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2913468458}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604801399}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057747466}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2604801399}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.286+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3610547240}) 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.294 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637075899}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2927285062}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=637075899}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.287+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1263038085}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274815184}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=783651918}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1274815184}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.288+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1144003673}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783709252}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349071034}) 2024-09-10T16:14:05.295 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1783709252}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.289+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1526790055}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643444444}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=952905173}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643444444}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2390948750}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.290+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184253073}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750414420}) 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.296 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=184253073}) 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=400202221}) 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350474228}) 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2788094598}) 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1350474228}) 2024-09-10T16:14:05.297 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.291+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4176753032}) 2024-09-10T16:14:05.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.292+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.292+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=925440310,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2884506610,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2743528979,osd_snap=43778211}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.292+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=11} crc {logm=3632756556,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1055854461,mon_config_key=2240728274,monmap=1433040829,nvmeofgw=598337687,osd_metadata=4005186995,osd_pg_creating=2743528979,osd_snap=3787618720}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.292+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=925440310,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2884506610,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2743528979,osd_snap=43778211}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=40,mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=10} crc {logm=3750278283,mdsmap=1228700967,mgr=3154768931,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2884506610,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2743528979,osd_snap=43778211}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=969193934,osdmap=546199920}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=38,osdmap=62} crc {osd_snap=2689113468,osdmap=3198321071}) 2024-09-10T16:14:05.300 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=39,osdmap=61} crc {osd_snap=969193934,osdmap=546199920}) 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.293+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=38,osdmap=62} crc {osd_snap=1248069277,osdmap=1120796264}) 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=2421160214}) 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=59} crc {osdmap=1647688631}) 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=2421160214}) 2024-09-10T16:14:05.301 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:05.298+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=61} crc {osdmap=1691728141}) 2024-09-10T16:14:05.488 INFO:tasks.rados.rados.0.smithi165.stdout:1021: done (3 left) 2024-09-10T16:14:05.488 INFO:tasks.rados.rados.0.smithi165.stdout:1022: done (2 left) 2024-09-10T16:14:05.488 INFO:tasks.rados.rados.0.smithi165.stdout:1023: done (1 left) 2024-09-10T16:14:05.488 INFO:tasks.rados.rados.0.smithi165.stdout:1024: done (0 left) 2024-09-10T16:14:05.489 INFO:tasks.rados.rados.0.smithi165.stdout:1025: delete oid 25 current snap is 117 2024-09-10T16:14:05.495 INFO:tasks.rados.rados.0.smithi165.stdout:1025: done (0 left) 2024-09-10T16:14:05.495 INFO:tasks.rados.rados.0.smithi165.stdout:1026: rollback oid 40 current snap is 117 2024-09-10T16:14:05.495 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 114 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1027: rollback oid 28 current snap is 117 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 28 to 115 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1028: read oid 35 snap -1 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1028: expect (ObjNum 301 snap 87 seq_num 301) 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1029: copy_from oid 26 from oid 36 current snap is 117 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1030: copy_from oid 24 from oid 10 current snap is 117 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1031: write (excl) oid 3 current snap is 117 2024-09-10T16:14:05.496 INFO:tasks.rados.rados.0.smithi165.stdout:1031: seq_num 379 ranges {769853=620668,1845146=524707} 2024-09-10T16:14:05.503 INFO:tasks.rados.rados.0.smithi165.stdout:1031: writing smithi16534841-3 from 769853 to 1390521 tid 1 2024-09-10T16:14:05.509 INFO:tasks.rados.rados.0.smithi165.stdout:1031: writing smithi16534841-3 from 1845146 to 2369853 tid 2 2024-09-10T16:14:05.509 INFO:tasks.rados.rados.0.smithi165.stdout:1032: snap_create 2024-09-10T16:14:05.509 INFO:tasks.rados.rados.0.smithi165.stdout:1029: finishing copy_from to smithi16534841-26 2024-09-10T16:14:05.509 INFO:tasks.rados.rados.0.smithi165.stdout:1029: got expected ENOENT (src dne) 2024-09-10T16:14:05.510 INFO:tasks.rados.rados.0.smithi165.stdout:1029: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:14:05.510 INFO:tasks.rados.rados.0.smithi165.stdout:1026: finishing rollback tid 0 to smithi16534841-40 2024-09-10T16:14:05.510 INFO:tasks.rados.rados.0.smithi165.stdout:1026: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:14:05.510 INFO:tasks.rados.rados.0.smithi165.stdout:1028: expect (ObjNum 301 snap 87 seq_num 301) 2024-09-10T16:14:05.687 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 987212 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1027: finishing rollback tid 0 to smithi16534841-28 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1027: finishing rollback tid 1 to smithi16534841-28 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1026: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 328 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1027: finishing rollback tid 2 to smithi16534841-28 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 310 (ObjNum 373 snap 109 seq_num 373) dirty exists 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1031: finishing write tid 1 to smithi16534841-3 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1030: finishing copy_from racing read to smithi16534841-24 2024-09-10T16:14:05.752 INFO:tasks.rados.rados.0.smithi165.stdout:1030: finishing copy_from to smithi16534841-24 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 308 (ObjNum 324 snap 94 seq_num 324) dirty exists 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: finishing write tid 2 to smithi16534841-3 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: finishing write tid 3 to smithi16534841-3 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: finishing write tid 4 to smithi16534841-3 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: oid 3 updating version 0 to 654 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: oid 3 version 654 is already newer than 653 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: oid 3 version 654 is already newer than 652 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 654 (ObjNum 379 snap 117 seq_num 379) dirty exists 2024-09-10T16:14:05.753 INFO:tasks.rados.rados.0.smithi165.stdout:1031: left oid 3 (ObjNum 379 snap 117 seq_num 379) 2024-09-10T16:14:06.299 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:14:06.299 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.402+0000 7fc7b0ba0640 10 monclient: get_monmap_and_config 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.402+0000 7fc7b0ba0640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.402+0000 7fc7b0ba0640 10 monclient: monmap: 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:06.403492+0000 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:06.403492+0000 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:06.404 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:06.405 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:06.405 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:06.405 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:06.405 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 Processor -- start 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 -- start start 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient: init 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc7ac06cd00 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc7ac135000 0x7fc7ac13cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ac135000 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc7ac13d4b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac135450 con 0x7fc7ac13d4b0 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7ac06df10 con 0x7fc7ac06cd00 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc7ac06d0e0 con 0x7fc7ac135000 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.403+0000 7fc7b0ba0640 10 monclient(hunting): authenticate will time out at 1224.890907s 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc7ac135000 0x7fc7ac13cf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7aad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc7ac13d4b0 0x7fc7ac13f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55162/0 (socket says 172.21.15.164:55162) 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc7ac135000 0x7fc7ac13cf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60188/0 (socket says 172.21.15.164:60188) 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7aad9d640 1 -- 172.21.15.164:0/4276278909 learned_addr learned my addr 172.21.15.164:0/4276278909 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): get_auth_request con 0x7fc7ac135000 auth_method 0 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_done global_id 74214 payload 274 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/4276278909 >> v1:172.21.15.165:6789/0 conn(0x7fc7ac06cd00 legacy=0x7fc7ac0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/4276278909 >> v1:172.21.15.164:6789/0 conn(0x7fc7ac13d4b0 legacy=0x7fc7ac13f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 monclient: found mon.noname-a 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:06.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.404+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/4276278909 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac09cb90 con 0x7fc7ac135000 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.406+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/4276278909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac135000 msgr2=0x7fc7ac13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a9d9b640 1 --2- 172.21.15.164:0/4276278909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac135000 0x7fc7ac13cf70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a0029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bad3314713466059 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 1 -- 172.21.15.164:0/4276278909 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0037070 con 0x7fc7ac135000 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: dump: 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.415 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 5 monclient: authenticate success, global_id 74214 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 1 -- 172.21.15.164:0/4276278909 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a0032640 con 0x7fc7ac135000 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 1 -- 172.21.15.164:0/4276278909 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0032a10 con 0x7fc7ac135000 2024-09-10T16:14:06.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 4 set_mon_vals no callback set 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7a959a640 10 monclient: dump: 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.417 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 10 monclient: shutdown 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/4276278909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac135000 msgr2=0x7fc7ac13cf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/4276278909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac135000 0x7fc7ac13cf70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/4276278909 shutdown_connections 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/4276278909 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac135000 0x7fc7ac13cf70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/4276278909 >> 172.21.15.164:0/4276278909 conn(0x7fc7ac0aaa70 msgr2=0x7fc7ac0aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/4276278909 shutdown_connections 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.407+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/4276278909 wait complete. 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.408+0000 7fc7b0ba0640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.408+0000 7fc7b0ba0640 10 monclient: monmap: 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.408+0000 7fc7b0ba0640 1 Processor -- start 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 -- start start 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient: init 2024-09-10T16:14:06.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7ac06cd00 0x7fc7ac1cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-b con 0x7fc7ac06cd00 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-c con 0x7fc7ac135000 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ac13d4b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac0a4df0 con 0x7fc7ac13d4b0 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc7ac140fc0 con 0x7fc7ac06cd00 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc7ac141140 con 0x7fc7ac135000 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7b0ba0640 10 monclient(hunting): authenticate will time out at 1224.896484s 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58046/0 (socket says 172.21.15.164:58046) 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/3358566092 learned_addr learned my addr 172.21.15.164:0/3358566092 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60190/0 (socket says 172.21.15.164:60190) 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): get_auth_request con 0x7fc7ac135000 auth_method 0 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): get_auth_request con 0x7fc7ac13d4b0 auth_method 0 2024-09-10T16:14:06.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aa59c640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7ac06cd00 0x7fc7ac1cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aa59c640 10 monclient(hunting): get_auth_request con 0x7fc7ac06cd00 auth_method 0 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient(hunting): handle_auth_done global_id 5078 payload 824 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7ac06cd00 msgr2=0x7fc7ac1cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7ac06cd00 0x7fc7ac1cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 msgr2=0x7fc7ac1ce220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 monclient: found mon.noname-c 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/3358566092 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7ac09e240 con 0x7fc7ac135000 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.409+0000 7fc7aad9d640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7a9d9b640 1 -- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 msgr2=0x7fc7ac1cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7a9d9b640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0041070 con 0x7fc7ac135000 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: dump: 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7a003c880 con 0x7fc7ac135000 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 5 monclient: authenticate success, global_id 5078 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 10 monclient: _renew_subs 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7ac12ed70 con 0x7fc7ac135000 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7a0037040 con 0x7fc7ac135000 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc79affd640 10 monclient: dump: 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7a8d99640 4 set_mon_vals no callback set 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 10 monclient: _renew_subs 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.412+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7ac06e670 con 0x7fc7ac135000 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc7a0044070 con 0x7fc7ac135000 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7b0ba0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7b0ba0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7b0ba0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7ac09e910 con 0x7fc7ac135000 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc79affd640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 0x7fc78c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(255..255 src has 184..255) ==== 4771+0+0 (secure 0 0 0) 0x7fc7a0073730 con 0x7fc7ac135000 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7aa59c640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 0x7fc78c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.413+0000 7fc7aa59c640 10 monclient: get_auth_request con 0x7fc78c03e210 auth_method 0 2024-09-10T16:14:06.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.416+0000 7fc7aa59c640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 0x7fc78c0406d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc79c006fd0 tx=0x7fc79c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.418+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc7a0072130 con 0x7fc7ac135000 2024-09-10T16:14:06.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.418+0000 7fc79affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.418+0000 7fc79affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:06.493 INFO:tasks.rados.rados.0.smithi165.stdout:1026: done (6 left) 2024-09-10T16:14:06.493 INFO:tasks.rados.rados.0.smithi165.stdout:1027: done (5 left) 2024-09-10T16:14:06.493 INFO:tasks.rados.rados.0.smithi165.stdout:1028: done (4 left) 2024-09-10T16:14:06.494 INFO:tasks.rados.rados.0.smithi165.stdout:1029: done (3 left) 2024-09-10T16:14:06.494 INFO:tasks.rados.rados.0.smithi165.stdout:1030: done (2 left) 2024-09-10T16:14:06.494 INFO:tasks.rados.rados.0.smithi165.stdout:1031: done (1 left) 2024-09-10T16:14:06.494 INFO:tasks.rados.rados.0.smithi165.stdout:1032: done (0 left) 2024-09-10T16:14:06.494 INFO:tasks.rados.rados.0.smithi165.stdout:1033: snap_remove snap 111 2024-09-10T16:14:06.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc7b0ba0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc7b0ba0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc7b0ba0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc7ac13cd90 con 0x7fc7ac135000 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc79affd640 1 -- 172.21.15.164:0/3358566092 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc7a004ed60 con 0x7fc7ac135000 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc79affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.665+0000 7fc79affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:06.667 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 msgr2=0x7fc78c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 0x7fc78c0406d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fc79c006fd0 tx=0x7fc79c008040 comp rx=0 tx=0).stop 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 10 monclient: shutdown 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 msgr2=0x7fc7ac1cdce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00097a0 tx=0x7fc7a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 shutdown_connections 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc78c03e210 0x7fc78c0406d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc7ac13d4b0 0x7fc7ac1ce220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc7ac135000 0x7fc7ac1cdce0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 --2- 172.21.15.164:0/3358566092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc7ac06cd00 0x7fc7ac1cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 >> 172.21.15.164:0/3358566092 conn(0x7fc7ac0aaa70 msgr2=0x7fc7ac133400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 shutdown_connections 2024-09-10T16:14:06.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.667+0000 7fc7b0ba0640 1 -- 172.21.15.164:0/3358566092 wait complete. 2024-09-10T16:14:06.679 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:14:06.679 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:14:06.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:14:06.680 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.777+0000 7fb477b0e640 10 monclient: get_monmap_and_config 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.777+0000 7fb477b0e640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.777+0000 7fb477b0e640 10 monclient: monmap: 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:06.778672+0000 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:06.778672+0000 2024-09-10T16:14:06.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:06.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.778+0000 7fb477b0e640 1 Processor -- start 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.778+0000 7fb477b0e640 1 -- start start 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.778+0000 7fb477b0e640 10 monclient: init 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb4700a4540 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 0x7fb47013cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb4701385c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb4700a60b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:06.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4700a71c0 con 0x7fb4700a4540 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb47009c9f0 con 0x7fb4700a60b0 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb47009ccd0 con 0x7fb4701385c0 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb477b0e640 10 monclient(hunting): authenticate will time out at 1225.266137s 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4758aa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb4700a4540 0x7fb4700a4920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55164/0 (socket says 172.21.15.164:55164) 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 0x7fb47013cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4758aa640 1 -- 172.21.15.164:0/3567554549 learned_addr learned my addr 172.21.15.164:0/3567554549 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): get_auth_request con 0x7fb4701385c0 auth_method 0 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient(hunting): handle_auth_done global_id 5081 payload 274 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 1 -- 172.21.15.164:0/3567554549 >> v1:172.21.15.165:6789/0 conn(0x7fb4700a60b0 legacy=0x7fb47013f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 1 -- 172.21.15.164:0/3567554549 >> v1:172.21.15.164:6789/0 conn(0x7fb4700a4540 legacy=0x7fb4700a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 1 monclient: found mon.noname-b 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.779+0000 7fb4750a9640 1 -- 172.21.15.164:0/3567554549 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47009cfb0 con 0x7fb4701385c0 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4750a9640 1 -- 172.21.15.164:0/3567554549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 msgr2=0x7fb47013cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4750a9640 1 --2- 172.21.15.164:0/3567554549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 0x7fb47013cee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb4600097d0 tx=0x7fb460029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a5e2489ac148f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 1 -- 172.21.15.164:0/3567554549 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460037070 con 0x7fb4701385c0 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: dump: 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 1 -- 172.21.15.164:0/3567554549 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb460032440 con 0x7fb4701385c0 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 5 monclient: authenticate success, global_id 5081 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 1 -- 172.21.15.164:0/3567554549 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460032810 con 0x7fb4701385c0 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 4 set_mon_vals no callback set 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb4748a8640 10 monclient: dump: 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 10 monclient: shutdown 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 1 -- 172.21.15.164:0/3567554549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 msgr2=0x7fb47013cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.782+0000 7fb477b0e640 1 --2- 172.21.15.164:0/3567554549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 0x7fb47013cee0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fb4600097d0 tx=0x7fb460029fb0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 -- 172.21.15.164:0/3567554549 shutdown_connections 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 --2- 172.21.15.164:0/3567554549 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4701385c0 0x7fb47013cee0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 -- 172.21.15.164:0/3567554549 >> 172.21.15.164:0/3567554549 conn(0x7fb4700aaa00 msgr2=0x7fb4700aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 -- 172.21.15.164:0/3567554549 shutdown_connections 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 -- 172.21.15.164:0/3567554549 wait complete. 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 10 monclient: monmap: 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:06.799 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.783+0000 7fb477b0e640 1 Processor -- start 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 -- start start 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient: init 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 0x7fb470131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fb4700a4540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:06.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4700a60b0 0x7fb470131f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fb4700a60b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a2150 0x7fb4700a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fb4700a2150 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb470140cd0 con 0x7fb4700a2150 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb470140b50 con 0x7fb4700a60b0 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb470140e50 con 0x7fb4700a4540 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb477b0e640 10 monclient(hunting): authenticate will time out at 1225.271535s 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 0x7fb470131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 0x7fb470131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58068/0 (socket says 172.21.15.164:58068) 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 -- 172.21.15.164:0/764109695 learned_addr learned my addr 172.21.15.164:0/764109695 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): get_auth_request con 0x7fb4700a4540 auth_method 0 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a2150 0x7fb4700a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): get_auth_request con 0x7fb4700a2150 auth_method 0 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4700a60b0 0x7fb470131f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): get_auth_request con 0x7fb4700a60b0 auth_method 0 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient(hunting): handle_auth_done global_id 5084 payload 824 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4700a60b0 msgr2=0x7fb470131f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb4700a60b0 0x7fb470131f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a2150 msgr2=0x7fb4700a2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a2150 0x7fb4700a2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 monclient: found mon.noname-c 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4758aa640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb47006e7d0 con 0x7fb4700a4540 2024-09-10T16:14:06.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.784+0000 7fb4760ab640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a2150 0x7fb4700a2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4758aa640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 msgr2=0x7fb470131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4758aa640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 0x7fb470131a40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c460 tx=0x7fb46c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c007070 con 0x7fb4700a4540 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: dump: 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.805 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb46c007230 con 0x7fb4700a4540 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 5 monclient: authenticate success, global_id 5084 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 10 monclient: _renew_subs 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4700a3c10 con 0x7fb4700a4540 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb46c007620 con 0x7fb4700a4540 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb4667fc640 10 monclient: dump: 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.807 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb467fff640 4 set_mon_vals no callback set 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 10 monclient: _renew_subs 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.787+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb47013a710 con 0x7fb4700a4540 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb477b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb477b0e640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 0x7fb47002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb4700a4920 con 0x7fb4701385c0 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 0x7fb47002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient: get_auth_request con 0x7fb4701385c0 auth_method 0 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb46c01b400 con 0x7fb4700a4540 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 10 monclient(hunting): handle_auth_done global_id 74223 payload 824 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4750a9640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb47009ebe0 con 0x7fb4701385c0 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4667fc640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb45403e210 0x7fb4540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.788+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7fb46c0529a0 con 0x7fb4700a4540 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.789+0000 7fb4760ab640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb45403e210 0x7fb4540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.789+0000 7fb4760ab640 10 monclient: get_auth_request con 0x7fb45403e210 auth_method 0 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4750a9640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 msgr2=0x7fb47002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4750a9640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 0x7fb47002d440 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4700a5bc0 tx=0x7fb4600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460037040 con 0x7fb4701385c0 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.790+0000 7fb4667fc640 10 monclient: dump: 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.810 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb46003c640 con 0x7fb4701385c0 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 msgr2=0x7fb47002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 0x7fb47002d440 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4700a5bc0 tx=0x7fb4600046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 msgr2=0x7fb47002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4667fc640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4701385c0 0x7fb47002d440 secure :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb4700a5bc0 tx=0x7fb4600046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.791+0000 7fb4760ab640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb45403e210 0x7fb4540406d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb46800f1d0 tx=0x7fb468003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb477b0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:06.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb477b0e640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 0x7fb47002d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb477b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb470139f50 con 0x7fb4700a3300 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 0x7fb47002d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient: get_auth_request con 0x7fb4700a3300 auth_method 0 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 10 monclient(hunting): handle_auth_done global_id 74226 payload 824 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.797+0000 7fb4750a9640 1 -- 172.21.15.164:0/764109695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb470138f10 con 0x7fb4700a3300 2024-09-10T16:14:06.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.799+0000 7fb4750a9640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 msgr2=0x7fb47002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.799+0000 7fb4750a9640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 0x7fb47002d980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb460000c00 tx=0x7fb460008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb460037040 con 0x7fb4700a3300 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 10 monclient: dump: 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.815 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fb46003b620 con 0x7fb4700a3300 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 msgr2=0x7fb47002d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 0x7fb47002d980 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb460000c00 tx=0x7fb460008030 comp rx=0 tx=0).stop 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 msgr2=0x7fb47002d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:06.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4667fc640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb4700a3300 0x7fb47002d980 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb460000c00 tx=0x7fb460008030 comp rx=0 tx=0).stop 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 74667, 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.817 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.818 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.819 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:06.820 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.821 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:06.825 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:06.826 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.827 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.828 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:06.829 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:06.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb45403e210 msgr2=0x7fb4540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb45403e210 0x7fb4540406d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb46800f1d0 tx=0x7fb468003d10 comp rx=0 tx=0).stop 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 10 monclient: shutdown 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 msgr2=0x7fb470131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 --2- 172.21.15.164:0/764109695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb4700a4540 0x7fb470131a40 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00c460 tx=0x7fb46c00c920 comp rx=0 tx=0).stop 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4760ab640 1 -- 172.21.15.164:0/764109695 reap_dead start 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb4760ab640 1 -- 172.21.15.164:0/764109695 reap_dead start 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 shutdown_connections 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 >> 172.21.15.164:0/764109695 conn(0x7fb4700aaa00 msgr2=0x7fb4701334b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 shutdown_connections 2024-09-10T16:14:06.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.800+0000 7fb477b0e640 1 -- 172.21.15.164:0/764109695 wait complete. 2024-09-10T16:14:06.832 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:06.832 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:14:06.832 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.928+0000 7f5624c08640 10 monclient: get_monmap_and_config 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.928+0000 7f5624c08640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.928+0000 7f5624c08640 10 monclient: monmap: 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:06.929465+0000 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:06.929465+0000 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:06.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:06.931 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:06.931 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:06.931 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:06.931 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 Processor -- start 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 -- start start 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient: init 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f562009d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 0x7f56201397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-b con 0x7f562009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f562009e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56200a0870 con 0x7f562009e420 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5620074c70 con 0x7f562009d930 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f562009d3e0 con 0x7f562009ab50 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f5624c08640 10 monclient(hunting): authenticate will time out at 1225.416782s 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 0x7f56201397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561ed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f562009e420 0x7f562013bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55170/0 (socket says 172.21.15.164:55170) 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561ed9d640 1 -- 172.21.15.164:0/4270518689 learned_addr learned my addr 172.21.15.164:0/4270518689 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561dd9b640 10 monclient(hunting): get_auth_request con 0x7f562009ab50 auth_method 0 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.929+0000 7f561dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient(hunting): handle_auth_done global_id 5087 payload 274 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 1 -- 172.21.15.164:0/4270518689 >> v1:172.21.15.165:6789/0 conn(0x7f562009d930 legacy=0x7f562009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 1 -- 172.21.15.164:0/4270518689 >> v1:172.21.15.164:6789/0 conn(0x7f562009e420 legacy=0x7f562013bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 1 monclient: found mon.noname-b 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:06.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.930+0000 7f561dd9b640 1 -- 172.21.15.164:0/4270518689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f562009ecb0 con 0x7f562009ab50 2024-09-10T16:14:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.932+0000 7f561dd9b640 1 -- 172.21.15.164:0/4270518689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 msgr2=0x7f56201397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.932+0000 7f561dd9b640 1 --2- 172.21.15.164:0/4270518689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 0x7f56201397c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f56140097d0 tx=0x7f561402a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1089365afeae255 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 1 -- 172.21.15.164:0/4270518689 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614037070 con 0x7f562009ab50 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: dump: 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 1 -- 172.21.15.164:0/4270518689 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5614032640 con 0x7f562009ab50 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 5 monclient: authenticate success, global_id 5087 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 1 -- 172.21.15.164:0/4270518689 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614032a10 con 0x7f562009ab50 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 4 set_mon_vals no callback set 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f561d59a640 10 monclient: dump: 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.944 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 10 monclient: shutdown 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 -- 172.21.15.164:0/4270518689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 msgr2=0x7f56201397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 --2- 172.21.15.164:0/4270518689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 0x7f56201397c0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f56140097d0 tx=0x7f561402a030 comp rx=0 tx=0).stop 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 -- 172.21.15.164:0/4270518689 shutdown_connections 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 --2- 172.21.15.164:0/4270518689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009ab50 0x7f56201397c0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 -- 172.21.15.164:0/4270518689 >> 172.21.15.164:0/4270518689 conn(0x7f56200a3110 msgr2=0x7f56200a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 -- 172.21.15.164:0/4270518689 shutdown_connections 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.933+0000 7f5624c08640 1 -- 172.21.15.164:0/4270518689 wait complete. 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.934+0000 7f5624c08640 10 monclient: build_initial_monmap 2024-09-10T16:14:06.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.934+0000 7f5624c08640 10 monclient: monmap: 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.934+0000 7f5624c08640 1 Processor -- start 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.934+0000 7f5624c08640 1 -- start start 2024-09-10T16:14:06.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.934+0000 7f5624c08640 10 monclient: init 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f562009ab50 0x7f56201cd8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-b con 0x7f562009ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 0x7f56201cddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-a con 0x7f562009d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009e420 0x7f562013f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): picked mon.noname-c con 0x7f562009e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f56200a2940 con 0x7f562009d930 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f562009ef30 con 0x7f562009ab50 2024-09-10T16:14:06.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f56200a2ac0 con 0x7f562009e420 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f5624c08640 10 monclient(hunting): authenticate will time out at 1225.422297s 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 0x7f56201cddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009e420 0x7f562013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 0x7f56201cddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60232/0 (socket says 172.21.15.164:60232) 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 -- 172.21.15.164:0/3558953657 learned_addr learned my addr 172.21.15.164:0/3558953657 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): get_auth_request con 0x7f562009d930 auth_method 0 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): get_auth_request con 0x7f562009e420 auth_method 0 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f562009ab50 0x7f56201cd8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 10 monclient(hunting): get_auth_request con 0x7f562009ab50 auth_method 0 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient(hunting): handle_auth_done global_id 74229 payload 824 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009e420 msgr2=0x7f562013f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009e420 0x7f562013f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f562009ab50 msgr2=0x7f56201cd8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f562009ab50 0x7f56201cd8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 monclient: found mon.noname-a 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561dd9b640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56201394d0 con 0x7f562009d930 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561ed9d640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562009e420 0x7f562013f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.935+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f562009ab50 0x7f56201cd8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.937+0000 7f561dd9b640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 msgr2=0x7f56201cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.937+0000 7f561dd9b640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 0x7f56201cddf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5614009720 tx=0x7f56140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614041070 con 0x7f562009d930 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: dump: 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: _finish_auth 0 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f561403c880 con 0x7f562009d930 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 5 monclient: authenticate success, global_id 74229 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5614037040 con 0x7f562009d930 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 monclient: _renew_subs 2024-09-10T16:14:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f562009cdd0 con 0x7f562009d930 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f561cd99640 4 set_mon_vals no callback set 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f560effd640 10 monclient: dump: 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 monclient: _renew_subs 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5620137cd0 con 0x7f562009d930 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 0x7f562002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.938+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f562013e460 con 0x7f562013dec0 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 0x7f562002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient: get_auth_request con 0x7f562013dec0 auth_method 0 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5614044070 con 0x7f562009d930 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 10 monclient(hunting): handle_auth_done global_id 5093 payload 824 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561e59c640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5620138ff0 con 0x7f562013dec0 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f560effd640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f560003e210 0x7f56000406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7f56140736a0 con 0x7f562009d930 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561ed9d640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f560003e210 0x7f56000406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.939+0000 7f561ed9d640 10 monclient: get_auth_request con 0x7f560003e210 auth_method 0 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f561e59c640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 msgr2=0x7f562002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 0x7f562002d440 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f561000e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610013800 con 0x7f562013dec0 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 10 monclient: dump: 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5610013800 con 0x7f562013dec0 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 msgr2=0x7f562002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 0x7f562002d440 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f561000e9c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 msgr2=0x7f562002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.941+0000 7f560effd640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562013dec0 0x7f562002d440 secure :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f561000e9c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.942+0000 7f561ed9d640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f560003e210 0x7f56000406d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5618005c00 tx=0x7f5618009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.947+0000 7f5624c08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.947+0000 7f5624c08640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 0x7f562002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.947+0000 7f5624c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.947+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f56200a4650 con 0x7f562002d980 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 0x7f562002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient: get_auth_request con 0x7f562002d980 auth_method 0 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 10 monclient(hunting): handle_auth_done global_id 5096 payload 824 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.948+0000 7f561e59c640 1 -- 172.21.15.164:0/3558953657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f562006dac0 con 0x7f562002d980 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f561e59c640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 msgr2=0x7f562002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f561e59c640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 0x7f562002dd90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f5610003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5610005220 con 0x7f562002d980 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 10 monclient: dump: 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f5610005220 con 0x7f562002d980 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 msgr2=0x7f562002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 0x7f562002dd90 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f5610003760 comp rx=0 tx=0).stop 2024-09-10T16:14:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 msgr2=0x7f562002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.950+0000 7f560effd640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f562002d980 0x7f562002dd90 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f561000e500 tx=0x7f5610003760 comp rx=0 tx=0).stop 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 489122, 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:06.963 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.964 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.965 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:06.966 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:06.967 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:06.968 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:06.969 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.970 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.972 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.974 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:14:06.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:06.976 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:06.977 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:06.977 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:06.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:06.977 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:06.977 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:06.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f560003e210 msgr2=0x7f56000406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f560003e210 0x7f56000406d0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5618005c00 tx=0x7f5618009040 comp rx=0 tx=0).stop 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 10 monclient: shutdown 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 msgr2=0x7f56201cddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 --2- 172.21.15.164:0/3558953657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f562009d930 0x7f56201cddf0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5614009720 tx=0x7f56140046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f561ed9d640 1 -- 172.21.15.164:0/3558953657 reap_dead start 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f561ed9d640 1 -- 172.21.15.164:0/3558953657 reap_dead start 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 shutdown_connections 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 >> 172.21.15.164:0/3558953657 conn(0x7f56200a3110 msgr2=0x7f56200707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 shutdown_connections 2024-09-10T16:14:06.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:06.951+0000 7f5624c08640 1 -- 172.21.15.164:0/3558953657 wait complete. 2024-09-10T16:14:06.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:06.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:14:06.980 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:14:07.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 10 monclient: get_monmap_and_config 2024-09-10T16:14:07.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 10 monclient: build_initial_monmap 2024-09-10T16:14:07.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 10 monclient: monmap: 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:07.078901+0000 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:07.078901+0000 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:07.080 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 1 Processor -- start 2024-09-10T16:14:07.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 1 -- start start 2024-09-10T16:14:07.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.078+0000 7f1a1e247640 10 monclient: init 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a180a3e60 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a1806d280 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1a1813e600 0x7f1a181409f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-a con 0x7f1a1813e600 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a180a4600 con 0x7f1a1806d280 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a180a5d70 con 0x7f1a180a3e60 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1a180a49b0 con 0x7f1a1813e600 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:07.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a1e247640 10 monclient(hunting): authenticate will time out at 1225.566290s 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a16ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1a1806d280 0x7f1a1806d660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55184/0 (socket says 172.21.15.164:55184) 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2850012869 learned_addr learned my addr 172.21.15.164:0/2850012869 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1a1813e600 0x7f1a181409f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): get_auth_request con 0x7f1a1813e600 auth_method 0 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient(hunting): handle_auth_done global_id 74232 payload 274 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 1 -- 172.21.15.164:0/2850012869 >> v1:172.21.15.165:6789/0 conn(0x7f1a180a3e60 legacy=0x7f1a1806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 1 -- 172.21.15.164:0/2850012869 >> v1:172.21.15.164:6789/0 conn(0x7f1a1806d280 legacy=0x7f1a1806d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:07.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient: _finish_hunting 0 2024-09-10T16:14:07.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 1 monclient: found mon.noname-a 2024-09-10T16:14:07.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:07.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:07.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.079+0000 7f1a17fff640 1 -- 172.21.15.164:0/2850012869 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a1806e340 con 0x7f1a1813e600 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a17fff640 1 -- 172.21.15.164:0/2850012869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a1813e600 msgr2=0x7f1a181409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a17fff640 1 --2- 172.21.15.164:0/2850012869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a1813e600 0x7f1a181409f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1a180702e0 tx=0x7f1a10029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7959807be0a1219 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 1 -- 172.21.15.164:0/2850012869 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10037070 con 0x7f1a1813e600 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: dump: 2024-09-10T16:14:07.091 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 1 -- 172.21.15.164:0/2850012869 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a10032440 con 0x7f1a1813e600 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a1e247640 5 monclient: authenticate success, global_id 74232 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a1e247640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a1e247640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 1 -- 172.21.15.164:0/2850012869 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a10032810 con 0x7f1a1813e600 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a1e247640 4 set_mon_vals no callback set 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.082+0000 7f1a167fc640 10 monclient: dump: 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.093 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.094 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 10 monclient: shutdown 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 -- 172.21.15.164:0/2850012869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a1813e600 msgr2=0x7f1a181409f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2850012869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a1813e600 0x7f1a181409f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1a180702e0 tx=0x7f1a10029fb0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 -- 172.21.15.164:0/2850012869 shutdown_connections 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2850012869 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a1813e600 0x7f1a181409f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 -- 172.21.15.164:0/2850012869 >> 172.21.15.164:0/2850012869 conn(0x7f1a180a4f50 msgr2=0x7f1a180a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 -- 172.21.15.164:0/2850012869 shutdown_connections 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 1 -- 172.21.15.164:0/2850012869 wait complete. 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 10 monclient: build_initial_monmap 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.083+0000 7f1a1e247640 10 monclient: monmap: 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:07.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 Processor -- start 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 -- start start 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient: init 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1806d280 0x7f1a1809adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-b con 0x7f1a1806d280 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 0x7f1a18092390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-a con 0x7f1a180a3e60 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a1813e600 0x7f1a180928d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): picked mon.noname-c con 0x7f1a1813e600 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1a18140f30 con 0x7f1a180a3e60 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1a180a4240 con 0x7f1a1806d280 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1a181410b0 con 0x7f1a1813e600 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a1e247640 10 monclient(hunting): authenticate will time out at 1225.571744s 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 0x7f1a18092390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 0x7f1a18092390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60250/0 (socket says 172.21.15.164:60250) 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a1813e600 0x7f1a180928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2031134975 learned_addr learned my addr 172.21.15.164:0/2031134975 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:07.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): get_auth_request con 0x7f1a180a3e60 auth_method 0 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): get_auth_request con 0x7f1a1813e600 auth_method 0 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a16ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a17fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.084+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1806d280 0x7f1a1809adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a177fe640 10 monclient(hunting): get_auth_request con 0x7f1a1806d280 auth_method 0 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 10 monclient(hunting): handle_auth_done global_id 74235 payload 824 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a1813e600 msgr2=0x7f1a180928d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a1813e600 0x7f1a180928d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1806d280 msgr2=0x7f1a1809adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1806d280 0x7f1a1809adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 monclient: found mon.noname-a 2024-09-10T16:14:07.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a18094f80 con 0x7f1a180a3e60 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1806d280 0x7f1a1809adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.085+0000 7f1a17fff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a1813e600 0x7f1a180928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a16ffd640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 msgr2=0x7f1a18092390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a16ffd640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 0x7f1a18092390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00b410 tx=0x7f1a0c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c015020 con 0x7f1a180a3e60 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: dump: 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: _finish_auth 0 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a0c011a60 con 0x7f1a180a3e60 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 5 monclient: authenticate success, global_id 74235 2024-09-10T16:14:07.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 10 monclient: _renew_subs 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a18095260 con 0x7f1a180a3e60 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0c011e50 con 0x7f1a180a3e60 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f19fffff640 10 monclient: dump: 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 10 monclient: _renew_subs 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a15ffb640 4 set_mon_vals no callback set 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.087+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a1809c140 con 0x7f1a180a3e60 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a1e247640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 0x7f1a1813e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1a18094b70 con 0x7f1a18092e10 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1a0c018020 con 0x7f1a180a3e60 2024-09-10T16:14:07.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 0x7f1a1813e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient: get_auth_request con 0x7f1a18092e10 auth_method 0 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f19fffff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a0403e3b0 0x7f1a04040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7f1a0c052c10 con 0x7f1a180a3e60 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a17fff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a0403e3b0 0x7f1a04040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 10 monclient(hunting): handle_auth_done global_id 84139 payload 824 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a177fe640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a180931f0 con 0x7f1a18092e10 2024-09-10T16:14:07.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.088+0000 7f1a17fff640 10 monclient: get_auth_request con 0x7f1a0403e3b0 auth_method 0 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f1a177fe640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 msgr2=0x7f1a1813e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 0x7f1a1813e9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800e890 tx=0x7f1a0800ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f1a17fff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a0403e3b0 0x7f1a04040870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a180a2b10 tx=0x7f1a1002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a0800cb50 con 0x7f1a18092e10 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 10 monclient: dump: 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.108 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1a0801a070 con 0x7f1a18092e10 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 msgr2=0x7f1a1813e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 0x7f1a1813e9e0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800e890 tx=0x7f1a0800ed50 comp rx=0 tx=0).stop 2024-09-10T16:14:07.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 msgr2=0x7f1a1813e9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.091+0000 7f19fffff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a18092e10 0x7f1a1813e9e0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1a0800e890 tx=0x7f1a0800ed50 comp rx=0 tx=0).stop 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a1e247640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 0x7f1a1813f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a1e247640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1806d660 con 0x7f1a1813ef20 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 0x7f1a1813f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient: get_auth_request con 0x7f1a1813ef20 auth_method 0 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 10 monclient(hunting): handle_auth_done global_id 84142 payload 824 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.098+0000 7f1a177fe640 1 -- 172.21.15.164:0/2031134975 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1a18094040 con 0x7f1a1813ef20 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f1a177fe640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 msgr2=0x7f1a1813f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f1a177fe640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 0x7f1a1813f330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a08019000 tx=0x7f1a08003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a08025070 con 0x7f1a1813ef20 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 10 monclient: dump: 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f1a08015430 con 0x7f1a1813ef20 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 msgr2=0x7f1a1813f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 0x7f1a1813f330 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a08019000 tx=0x7f1a08003ae0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 msgr2=0x7f1a1813f330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:07.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.101+0000 7f19fffff640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a1813ef20 0x7f1a1813f330 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1a08019000 tx=0x7f1a08003ae0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 74, 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 9896, 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:07.113 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:07.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:07.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:07.116 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:07.117 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:07.118 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:07.121 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:07.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:07.123 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:07.124 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:07.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:07.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:07.126 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:07.126 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:07.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a0403e3b0 msgr2=0x7f1a04040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a0403e3b0 0x7f1a04040870 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f1a180a2b10 tx=0x7f1a1002e000 comp rx=0 tx=0).stop 2024-09-10T16:14:07.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 10 monclient: shutdown 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 msgr2=0x7f1a18092390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 --2- 172.21.15.164:0/2031134975 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a180a3e60 0x7f1a18092390 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1a0c00b410 tx=0x7f1a0c00b8d0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a17fff640 1 -- 172.21.15.164:0/2031134975 reap_dead start 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a17fff640 1 -- 172.21.15.164:0/2031134975 reap_dead start 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 shutdown_connections 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 >> 172.21.15.164:0/2031134975 conn(0x7f1a180a4f50 msgr2=0x7f1a180a64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 shutdown_connections 2024-09-10T16:14:07.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.102+0000 7f1a1e247640 1 -- 172.21.15.164:0/2031134975 wait complete. 2024-09-10T16:14:07.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:07.129 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:14:07.129 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:14:07.129 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:14:07.129 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:14:07.129 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:14:07.129 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:14:07.140 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:14:07.141 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:14:07.141 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:07.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 10 monclient: get_monmap_and_config 2024-09-10T16:14:07.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 10 monclient: build_initial_monmap 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 10 monclient: monmap: 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:07.236221+0000 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:07.236221+0000 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:07.237 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 1 Processor -- start 2024-09-10T16:14:07.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 1 -- start start 2024-09-10T16:14:07.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.235+0000 7fa858394640 10 monclient: init 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa85006cd00 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa850135000 0x7fa85013cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): picked mon.noname-c con 0x7fa850135000 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa85013d4b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa850135450 con 0x7fa85013d4b0 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85006df10 con 0x7fa85006cd00 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa85006d0e0 con 0x7fa850135000 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa858394640 10 monclient(hunting): authenticate will time out at 1225.723634s 2024-09-10T16:14:07.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa856130640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fa85006cd00 legacy=0x7fa8500a46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:14:07.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa85592f640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fa850135000 msgr2=0x7fa85013cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:07.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa856931640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa85013d4b0 0x7fa85013f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55198/0 (socket says 172.21.15.164:55198) 2024-09-10T16:14:07.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa856931640 1 -- 172.21.15.164:0/236397327 learned_addr learned my addr 172.21.15.164:0/236397327 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:07.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.236+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> v2:172.21.15.165:3300/0 conn(0x7fa850135000 0x7fa85013cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.238+0000 7fa856931640 1 -- 172.21.15.164:0/236397327 >> v1:172.21.15.164:6789/0 conn(0x7fa85013d4b0 legacy=0x7fa85013f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 108047129 0 0) 0x7fa850135450 con 0x7fa85013d4b0 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): _negotiate 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): my global_id is 74238 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa83c003520 con 0x7fa85013d4b0 2024-09-10T16:14:07.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2808881079 0 0) 0x7fa83c003520 con 0x7fa85013d4b0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 >> v2:172.21.15.165:3300/0 conn(0x7fa850135000 msgr2=0x7fa85013cf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 --2- 172.21.15.164:0/236397327 >> v2:172.21.15.165:3300/0 conn(0x7fa850135000 0x7fa85013cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 >> v1:172.21.15.165:6789/0 conn(0x7fa85006cd00 legacy=0x7fa8500a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85009cb90 con 0x7fa85013d4b0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _finish_auth 0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _check_auth_tickets 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fa848001d20 con 0x7fa85013d4b0 2024-09-10T16:14:07.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa858394640 5 monclient: authenticate success, global_id 74238 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa858394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: dump: 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.253 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:14:07.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:07.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 >> v1:172.21.15.164:6789/0 conn(0x7fa85013d4b0 legacy=0x7fa85013f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa83c003a20 0x7fa83c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): picked mon.c con 0x7fa83c003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): picked mon.a con 0x7fa83c004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c0099b0 0x7fa83c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): picked mon.b con 0x7fa83c0099b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa85006d0e0 con 0x7fa83c004800 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c00c390 con 0x7fa83c0099b0 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa83c00c590 con 0x7fa83c003a20 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85512e640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa858394640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa83c003a20 0x7fa83c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): get_auth_request con 0x7fa83c004800 auth_method 0 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): get_auth_request con 0x7fa83c003a20 auth_method 0 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856931640 1 -- 172.21.15.164:0/236397327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c0099b0 msgr2=0x7fa83c00bda0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856931640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c0099b0 0x7fa83c00bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856130640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:14:07.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient(hunting): handle_auth_done global_id 74238 payload 293 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 -- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa83c003a20 msgr2=0x7fa83c003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa83c003a20 0x7fa83c003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 -- 172.21.15.164:0/236397327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c0099b0 msgr2=0x7fa83c00bda0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c0099b0 0x7fa83c00bda0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient: _finish_hunting 0 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 monclient: found mon.a 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa856931640 1 -- 172.21.15.164:0/236397327 reap_dead start 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 1 -- 172.21.15.164:0/236397327 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c003520 con 0x7fa83c004800 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient: _finish_auth 0 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 10 monclient: _check_auth_tickets 2024-09-10T16:14:07.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.239+0000 7fa85592f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85592f640 1 -- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 msgr2=0x7fa83c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa844009900 tx=0x7fa83c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf58c57edcbc78e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844009500 con 0x7fa83c004800 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85592f640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa844009900 tx=0x7fa83c005830 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7fa84402a910 mon_map magic: 0 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 10 monclient: dump: 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.259 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84402e070 con 0x7fa83c004800 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa858394640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa85512e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa858394640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.242+0000 7fa858394640 4 set_mon_vals no callback set 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 10 monclient: shutdown 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 -- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 msgr2=0x7fa83c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fa844009900 tx=0x7fa83c005830 comp rx=0 tx=0).stop 2024-09-10T16:14:07.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa85512e640 1 -- 172.21.15.164:0/236397327 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7fa84402a910 con 0x7fa83c004800 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa85512e640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 -- 172.21.15.164:0/236397327 shutdown_connections 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 --2- 172.21.15.164:0/236397327 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa83c004800 0x7fa83c004c80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 -- 172.21.15.164:0/236397327 >> 172.21.15.164:0/236397327 conn(0x7fa8500aaa70 msgr2=0x7fa8500aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 -- 172.21.15.164:0/236397327 shutdown_connections 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.243+0000 7fa858394640 1 -- 172.21.15.164:0/236397327 wait complete. 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient: build_initial_monmap 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient: monmap: 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:07.261 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 Processor -- start 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 -- start start 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient: init 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 0x7fa85013b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): picked mon.noname-b con 0x7fa83c004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:07.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): picked mon.noname-c con 0x7fa85013b8d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 0x7fa8501d3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): picked mon.noname-a con 0x7fa85013c270 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8500a46e0 con 0x7fa85013c270 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa85009cec0 con 0x7fa83c004800 2024-09-10T16:14:07.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8500a4860 con 0x7fa85013b8d0 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.244+0000 7fa858394640 10 monclient(hunting): authenticate will time out at 1225.731925s 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 0x7fa8501d3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58140/0 (socket says 172.21.15.164:58140) 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 -- 172.21.15.164:0/3555917285 learned_addr learned my addr 172.21.15.164:0/3555917285 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): get_auth_request con 0x7fa85013b8d0 auth_method 0 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): get_auth_request con 0x7fa85013c270 auth_method 0 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856130640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 msgr2=0x7fa85013b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856130640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 0x7fa85013b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient(hunting): handle_auth_done global_id 5102 payload 824 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 msgr2=0x7fa85013b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 0x7fa85013b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 msgr2=0x7fa8501d3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 0x7fa8501d3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient: _finish_hunting 0 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 monclient: found mon.noname-c 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:07.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa85592f640 1 -- 172.21.15.164:0/3555917285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85006e670 con 0x7fa85013b8d0 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.245+0000 7fa856931640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 0x7fa8501d3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa85592f640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 msgr2=0x7fa85013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa85592f640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa8440099f0 tx=0x7fa844005a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844005480 con 0x7fa85013b8d0 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: dump: 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.267 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: _finish_auth 0 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa844042070 con 0x7fa85013b8d0 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 5 monclient: authenticate success, global_id 5102 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 10 monclient: _renew_subs 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8500a5830 con 0x7fa85013b8d0 2024-09-10T16:14:07.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844005820 con 0x7fa85013b8d0 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa842ffd640 10 monclient: dump: 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:07.269 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa85492d640 4 set_mon_vals no callback set 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 10 monclient: _renew_subs 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.247+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa85009f9c0 con 0x7fa85013b8d0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa844033470 con 0x7fa85013b8d0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa858394640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa858394640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa858394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8500a4c90 con 0x7fa85013b8d0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.248+0000 7fa842ffd640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 0x7fa834040850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.249+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7fa844072b30 con 0x7fa85013b8d0 2024-09-10T16:14:07.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.249+0000 7fa856130640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 0x7fa834040850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.249+0000 7fa856130640 10 monclient: get_auth_request con 0x7fa83403e390 auth_method 0 2024-09-10T16:14:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.251+0000 7fa856130640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 0x7fa834040850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa850075e70 tx=0x7fa84c0090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.253+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa85006e670 con 0x7fa85013b8d0 2024-09-10T16:14:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.253+0000 7fa842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:07.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.253+0000 7fa842ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.499+0000 7fa858394640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.499+0000 7fa858394640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.499+0000 7fa858394640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.499+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8500770a0 con 0x7fa85013b8d0 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.500+0000 7fa842ffd640 1 -- 172.21.15.164:0/3555917285 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fa844071530 con 0x7fa85013b8d0 2024-09-10T16:14:07.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.500+0000 7fa842ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:07.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.500+0000 7fa842ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:07.502 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:07.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 msgr2=0x7fa834040850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 0x7fa834040850 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa850075e70 tx=0x7fa84c0090e0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 10 monclient: shutdown 2024-09-10T16:14:07.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 msgr2=0x7fa85013bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa8440099f0 tx=0x7fa844005a10 comp rx=0 tx=0).stop 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 shutdown_connections 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83403e390 0x7fa834040850 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85013c270 0x7fa8501d3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85013b8d0 0x7fa85013bd30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 --2- 172.21.15.164:0/3555917285 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa83c004800 0x7fa85013b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 >> 172.21.15.164:0/3555917285 conn(0x7fa8500aaa70 msgr2=0x7fa8501313f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 shutdown_connections 2024-09-10T16:14:07.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:07.502+0000 7fa858394640 1 -- 172.21.15.164:0/3555917285 wait complete. 2024-09-10T16:14:10.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.625+0000 7faafc6b6640 10 monclient: get_monmap_and_config 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.625+0000 7faafc6b6640 10 monclient: build_initial_monmap 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.625+0000 7faafc6b6640 10 monclient: monmap: 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:10.626326+0000 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:10.626326+0000 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:10.627 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:10.628 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:10.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 Processor -- start 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 -- start start 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient: init 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-b con 0x7faaf409f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:10.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 0x7faaf40a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-c con 0x7faaf40a2380 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaf413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaf409fd10 con 0x7faaf413a5c0 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faaf40a2cc0 con 0x7faaf40a2380 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faaf409cd70 con 0x7faaf409f930 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafc6b6640 10 monclient(hunting): authenticate will time out at 1229.113905s 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.626+0000 7faafa452640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faaf9c51640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 msgr2=0x7faaf40a2780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58148/0 (socket says 172.21.15.164:58148) 2024-09-10T16:14:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 -- 172.21.15.164:0/3133891978 learned_addr learned my addr 172.21.15.164:0/3133891978 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): get_auth_request con 0x7faaf409f930 auth_method 0 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faaf9c51640 1 --2- 172.21.15.164:0/3133891978 >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 0x7faaf40a2780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafac53640 1 -- 172.21.15.164:0/3133891978 >> v1:172.21.15.165:6789/0 conn(0x7faaf413a5c0 legacy=0x7faaf413c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient(hunting): handle_auth_done global_id 5105 payload 274 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 -- 172.21.15.164:0/3133891978 >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 msgr2=0x7faaf40a2780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 --2- 172.21.15.164:0/3133891978 >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 0x7faaf40a2780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 -- 172.21.15.164:0/3133891978 >> v1:172.21.15.165:6789/0 conn(0x7faaf413a5c0 legacy=0x7faaf413c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient: _finish_hunting 0 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 monclient: found mon.noname-b 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.627+0000 7faafa452640 1 -- 172.21.15.164:0/3133891978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf409a120 con 0x7faaf409f930 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafa452640 1 -- 172.21.15.164:0/3133891978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 msgr2=0x7faaf40a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafa452640 1 --2- 172.21.15.164:0/3133891978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009870 tx=0x7faaf002a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a84a4ff8ac6a0125 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 1 -- 172.21.15.164:0/3133891978 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0037070 con 0x7faaf409f930 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: dump: 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: _finish_auth 0 2024-09-10T16:14:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: _check_auth_tickets 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 1 -- 172.21.15.164:0/3133891978 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faaf0032640 con 0x7faaf409f930 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 5 monclient: authenticate success, global_id 5105 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 4 set_mon_vals no callback set 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 1 -- 172.21.15.164:0/3133891978 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf0032a10 con 0x7faaf409f930 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faaf9450640 10 monclient: dump: 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 10 monclient: shutdown 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 -- 172.21.15.164:0/3133891978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 msgr2=0x7faaf40a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 --2- 172.21.15.164:0/3133891978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7faaf0009870 tx=0x7faaf002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 -- 172.21.15.164:0/3133891978 shutdown_connections 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 --2- 172.21.15.164:0/3133891978 >> v2:172.21.15.165:3300/0 conn(0x7faaf40a2380 0x7faaf40a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 --2- 172.21.15.164:0/3133891978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf409f930 0x7faaf40a1e40 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 -- 172.21.15.164:0/3133891978 >> 172.21.15.164:0/3133891978 conn(0x7faaf40a3110 msgr2=0x7faaf40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 -- 172.21.15.164:0/3133891978 shutdown_connections 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.630+0000 7faafc6b6640 1 -- 172.21.15.164:0/3133891978 wait complete. 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.631+0000 7faafc6b6640 10 monclient: build_initial_monmap 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.631+0000 7faafc6b6640 10 monclient: monmap: 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.631+0000 7faafc6b6640 1 Processor -- start 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.631+0000 7faafc6b6640 1 -- start start 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.631+0000 7faafc6b6640 10 monclient: init 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 0x7faaf412c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-b con 0x7faaf409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-c con 0x7faaf40a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaf413a5c0 0x7faaf412cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): picked mon.noname-a con 0x7faaf413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faaf409d930 con 0x7faaf413a5c0 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faaf409ee00 con 0x7faaf409f930 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faaf409c8b0 con 0x7faaf40a2380 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafc6b6640 10 monclient(hunting): authenticate will time out at 1229.119356s 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafa452640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 msgr2=0x7faaf412c540 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafa452640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 0x7faaf412c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaf413a5c0 0x7faaf412cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58162/0 (socket says 172.21.15.164:58162) 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 -- 172.21.15.164:0/440377527 learned_addr learned my addr 172.21.15.164:0/440377527 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): get_auth_request con 0x7faaf40a2380 auth_method 0 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): get_auth_request con 0x7faaf413a5c0 auth_method 0 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faafac53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient(hunting): handle_auth_done global_id 5108 payload 824 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 -- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 msgr2=0x7faaf412c540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 0x7faaf412c540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 -- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaf413a5c0 msgr2=0x7faaf412cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaf413a5c0 0x7faaf412cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient: _finish_hunting 0 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 monclient: found mon.noname-c 2024-09-10T16:14:10.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.632+0000 7faaf9c51640 1 -- 172.21.15.164:0/440377527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaf409d620 con 0x7faaf40a2380 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaf9c51640 1 -- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 msgr2=0x7faaf412ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaf9c51640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faae400b550 tx=0x7faae400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae4013020 con 0x7faaf40a2380 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: dump: 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.647 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: _finish_auth 0 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faae4005070 con 0x7faaf40a2380 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae4005480 con 0x7faaf40a2380 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 5 monclient: authenticate success, global_id 5108 2024-09-10T16:14:10.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaeaffd640 10 monclient: dump: 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:10.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 10 monclient: _renew_subs 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaf4136310 con 0x7faaf40a2380 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faaf8c4f640 4 set_mon_vals no callback set 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 10 monclient: _renew_subs 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.635+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faaf412f370 con 0x7faaf40a2380 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faae4016020 con 0x7faaf40a2380 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafc6b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafc6b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafc6b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faaf40a0760 con 0x7faaf40a2380 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faaeaffd640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 0x7faad8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:10.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7faae4052aa0 con 0x7faaf40a2380 2024-09-10T16:14:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafa452640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 0x7faad8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.636+0000 7faafa452640 10 monclient: get_auth_request con 0x7faad803e1c0 auth_method 0 2024-09-10T16:14:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.639+0000 7faafa452640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 0x7faad8040680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faaf409f5c0 tx=0x7faaf002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:10.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:10.641+0000 7faaf9c51640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faae400b550 tx=0x7faae400ba10 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7faae401d020 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:14:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.469+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faae401d020 con 0x7faaf40a2380 2024-09-10T16:14:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.469+0000 7faaeaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.469+0000 7faaeaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.715+0000 7faafc6b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.715+0000 7faafc6b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.715+0000 7faafc6b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.715+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faaf4071a60 con 0x7faaf40a2380 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.715+0000 7faaeaffd640 1 -- 172.21.15.164:0/440377527 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7faae40514a0 con 0x7faaf40a2380 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.716+0000 7faaeaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.716+0000 7faaeaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:11.717 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:11.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 msgr2=0x7faad8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:11.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 0x7faad8040680 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7faaf409f5c0 tx=0x7faaf002e000 comp rx=0 tx=0).stop 2024-09-10T16:14:11.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 10 monclient: shutdown 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 msgr2=0x7faaf412ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faae400b550 tx=0x7faae400ba10 comp rx=0 tx=0).stop 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 shutdown_connections 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad803e1c0 0x7faad8040680 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaf413a5c0 0x7faaf412cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaf40a2380 0x7faaf412ca80 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 --2- 172.21.15.164:0/440377527 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaf409f930 0x7faaf412c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 >> 172.21.15.164:0/440377527 conn(0x7faaf40a3110 msgr2=0x7faaf4138d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 shutdown_connections 2024-09-10T16:14:11.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:11.718+0000 7faafc6b6640 1 -- 172.21.15.164:0/440377527 wait complete. 2024-09-10T16:14:14.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.836+0000 7f067feed640 10 monclient: get_monmap_and_config 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.836+0000 7f067feed640 10 monclient: build_initial_monmap 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.836+0000 7f067feed640 10 monclient: monmap: 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:14.837765+0000 2024-09-10T16:14:14.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:14.837765+0000 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:14.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.837+0000 7f067feed640 1 Processor -- start 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.837+0000 7f067feed640 1 -- start start 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.837+0000 7f067feed640 10 monclient: init 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f067809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:14.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f067809d160 0x7f067809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): picked mon.noname-a con 0x7f067809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 0x7f067813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): picked mon.noname-c con 0x7f067813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f067809a430 con 0x7f067809c180 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f06780a24e0 con 0x7f067809d160 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f06780a2bf0 con 0x7f067813a5c0 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067feed640 10 monclient(hunting): authenticate will time out at 1233.325321s 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067dc89640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f067809c180 0x7f067809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52930/0 (socket says 172.21.15.164:52930) 2024-09-10T16:14:14.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067dc89640 1 -- 172.21.15.164:0/3506440431 learned_addr learned my addr 172.21.15.164:0/3506440431 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 --2- 172.21.15.164:0/3506440431 >> v2:172.21.15.164:3300/0 conn(0x7f067809d160 0x7f067809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067e48a640 1 -- 172.21.15.164:0/3506440431 >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 msgr2=0x7f067813c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): get_auth_request con 0x7f067809d160 auth_method 0 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067e48a640 1 --2- 172.21.15.164:0/3506440431 >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 0x7f067813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:14.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient(hunting): handle_auth_done global_id 74247 payload 274 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 -- 172.21.15.164:0/3506440431 >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 msgr2=0x7f067813c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 --2- 172.21.15.164:0/3506440431 >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 0x7f067813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 -- 172.21.15.164:0/3506440431 >> v1:172.21.15.164:6789/0 conn(0x7f067809c180 legacy=0x7f067809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient: _finish_hunting 0 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 monclient: found mon.noname-a 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:14.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.838+0000 7f067d488640 1 -- 172.21.15.164:0/3506440431 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f067809c560 con 0x7f067809d160 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067d488640 1 -- 172.21.15.164:0/3506440431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809d160 msgr2=0x7f067809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067d488640 1 --2- 172.21.15.164:0/3506440431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809d160 0x7f067809f150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02a030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a39b8f1e65366dae server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 1 -- 172.21.15.164:0/3506440431 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c037070 con 0x7f067809d160 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: dump: 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:14.851 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: _finish_auth 0 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: _check_auth_tickets 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:14.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067feed640 5 monclient: authenticate success, global_id 74247 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 1 -- 172.21.15.164:0/3506440431 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f066c032640 con 0x7f067809d160 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067feed640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 1 -- 172.21.15.164:0/3506440431 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066c032a10 con 0x7f067809d160 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067feed640 4 set_mon_vals no callback set 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.841+0000 7f067cc87640 10 monclient: dump: 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:14.853 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 10 monclient: shutdown 2024-09-10T16:14:14.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 -- 172.21.15.164:0/3506440431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809d160 msgr2=0x7f067809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 --2- 172.21.15.164:0/3506440431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809d160 0x7f067809f150 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f066c0097a0 tx=0x7f066c02a030 comp rx=0 tx=0).stop 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 -- 172.21.15.164:0/3506440431 shutdown_connections 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 --2- 172.21.15.164:0/3506440431 >> v2:172.21.15.165:3300/0 conn(0x7f067813a5c0 0x7f067813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 --2- 172.21.15.164:0/3506440431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809d160 0x7f067809f150 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 -- 172.21.15.164:0/3506440431 >> 172.21.15.164:0/3506440431 conn(0x7f06780a3110 msgr2=0x7f06780a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 -- 172.21.15.164:0/3506440431 shutdown_connections 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.842+0000 7f067feed640 1 -- 172.21.15.164:0/3506440431 wait complete. 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient: build_initial_monmap 2024-09-10T16:14:14.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient: monmap: 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 Processor -- start 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 -- start start 2024-09-10T16:14:14.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient: init 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): picked mon.noname-a con 0x7f067809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 0x7f06781c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): picked mon.noname-c con 0x7f067809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 0x7f067808fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): picked mon.noname-b con 0x7f067813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0678074f90 con 0x7f067809c180 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0678074e10 con 0x7f067813a5c0 2024-09-10T16:14:14.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f067809f690 con 0x7f067809d160 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.843+0000 7f067feed640 10 monclient(hunting): authenticate will time out at 1233.330955s 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47508/0 (socket says 172.21.15.164:47508) 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 -- 172.21.15.164:0/3293726712 learned_addr learned my addr 172.21.15.164:0/3293726712 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): get_auth_request con 0x7f067809c180 auth_method 0 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 0x7f06781c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): get_auth_request con 0x7f067809d160 auth_method 0 2024-09-10T16:14:14.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067e48a640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 msgr2=0x7f067808fae0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067e48a640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 0x7f067808fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient(hunting): handle_auth_done global_id 74250 payload 824 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 msgr2=0x7f06781c9f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:14.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 0x7f06781c9f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 msgr2=0x7f067808fae0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 0x7f067808fae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient: _finish_hunting 0 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 monclient: found mon.noname-a 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067dc89640 1 -- 172.21.15.164:0/3293726712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0678091410 con 0x7f067809c180 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.844+0000 7f067d488640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 0x7f06781c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067dc89640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 msgr2=0x7f06781c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:14.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067dc89640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f066800cc10 tx=0x7f066800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:14.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0668012400 con 0x7f067809c180 2024-09-10T16:14:14.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:14.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:14.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: dump: 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:14.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06680125e0 con 0x7f067809c180 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 5 monclient: authenticate success, global_id 74250 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 10 monclient: _renew_subs 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f067809d620 con 0x7f067809c180 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06680129d0 con 0x7f067809c180 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f06767fc640 10 monclient: dump: 2024-09-10T16:14:14.863 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f0677fff640 4 set_mon_vals no callback set 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 10 monclient: _renew_subs 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:14.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.846+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0678092ed0 con 0x7f067809c180 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.847+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0668012bf0 con 0x7f067809c180 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.847+0000 7f067feed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.847+0000 7f067feed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.847+0000 7f067feed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.847+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f067809ed50 con 0x7f067809c180 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.848+0000 7f06767fc640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 0x7f06640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.848+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(256..256 src has 184..256) ==== 4771+0+0 (secure 0 0 0) 0x7f0668053410 con 0x7f067809c180 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.848+0000 7f067d488640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 0x7f06640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:14.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.848+0000 7f067d488640 10 monclient: get_auth_request con 0x7f066403e210 auth_method 0 2024-09-10T16:14:14.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.850+0000 7f067d488640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 0x7f06640406d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f067806d4a0 tx=0x7f066c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:14.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.850+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:14:14.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.852+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f067809ed50 con 0x7f067809c180 2024-09-10T16:14:14.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.852+0000 7f06767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:14.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:14.852+0000 7f06767fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:15.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f067feed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f067feed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f067feed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f067809d930 con 0x7f067809c180 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f06767fc640 1 -- 172.21.15.164:0/3293726712 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f0668051e10 con 0x7f067809c180 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f06767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.095+0000 7f06767fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:15.097 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 msgr2=0x7f06640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 0x7f06640406d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f067806d4a0 tx=0x7f066c02e000 comp rx=0 tx=0).stop 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 10 monclient: shutdown 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 msgr2=0x7f06781c99c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:15.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f066800cc10 tx=0x7f066800a430 comp rx=0 tx=0).stop 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 shutdown_connections 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f066403e210 0x7f06640406d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f067813a5c0 0x7f067808fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f067809d160 0x7f06781c9f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 --2- 172.21.15.164:0/3293726712 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f067809c180 0x7f06781c99c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 >> 172.21.15.164:0/3293726712 conn(0x7f06780a3110 msgr2=0x7f0678139dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 shutdown_connections 2024-09-10T16:14:15.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:15.097+0000 7f067feed640 1 -- 172.21.15.164:0/3293726712 wait complete. 2024-09-10T16:14:18.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.213+0000 7fad47271640 10 monclient: get_monmap_and_config 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.213+0000 7fad47271640 10 monclient: build_initial_monmap 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.213+0000 7fad47271640 10 monclient: monmap: 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:18.214572+0000 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:18.214572+0000 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:18.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:18.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 Processor -- start 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 -- start start 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient: init 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad4009c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): picked mon.noname-c con 0x7fad4009d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): picked mon.noname-a con 0x7fad4013a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4009a430 con 0x7fad4009c180 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fad400a24e0 con 0x7fad4013a5c0 2024-09-10T16:14:18.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fad400a2bf0 con 0x7fad4009d160 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.214+0000 7fad47271640 10 monclient(hunting): authenticate will time out at 1236.701957s 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4480c640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 msgr2=0x7fad4009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4500d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fad4009c180 0x7fad4009ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52938/0 (socket says 172.21.15.164:52938) 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4500d640 1 -- 172.21.15.164:0/2499897970 learned_addr learned my addr 172.21.15.164:0/2499897970 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47510/0 (socket says 172.21.15.164:47510) 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 10 monclient(hunting): get_auth_request con 0x7fad4013a5c0 auth_method 0 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:18.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:14:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.215+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.217+0000 7fad4500d640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.164:6789/0 conn(0x7fad4009c180 legacy=0x7fad4009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:18.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 msgr2=0x7fad4009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:18.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:14:18.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:18.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 10 monclient(hunting): get_auth_request con 0x7fad4013a5c0 auth_method 0 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:18.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.416+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 msgr2=0x7fad4009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 10 monclient(hunting): get_auth_request con 0x7fad4013a5c0 auth_method 0 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:18.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:14:18.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:18.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:18.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:18.817+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 msgr2=0x7fad4009f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 10 monclient(hunting): get_auth_request con 0x7fad4013a5c0 auth_method 0 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:19.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:14:19.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:19.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:19.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:19.618+0000 7fad4580e640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:14:21.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): tick 2024-09-10T16:14:21.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:14:21.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 monclient(hunting): continuing hunt 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 msgr2=0x7fad4009f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.165:3300/0 conn(0x7fad4009d160 0x7fad4009f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 -- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 msgr2=0x7fad4013c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 --2- 172.21.15.164:0/2499897970 >> v2:172.21.15.164:3300/0 conn(0x7fad4013a5c0 0x7fad4013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.164:6789/0 conn(0x7fad4009c180 legacy=0x7fad4009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient: _add_conns ranks=[1,2] 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fad2c003740 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): picked mon.noname-b con 0x7fad2c007430 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 -- 172.21.15.164:0/2499897970 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4009c560 con 0x7fad2c003740 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:21.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 1 -- 172.21.15.164:0/2499897970 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad400a24e0 con 0x7fad2c007430 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad3effd640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4500d640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.165:6789/0 conn(0x7fad2c003740 legacy=0x7fad2c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 10 monclient(hunting): get_auth_request con 0x7fad2c007430 auth_method 0 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:21.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.215+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:21.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.416+0000 7fad4500d640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.165:6789/0 conn(0x7fad2c003740 legacy=0x7fad2c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 10 monclient(hunting): get_auth_request con 0x7fad2c007430 auth_method 0 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:21.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.417+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1033: done (0 left) 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1034: read oid 50 snap -1 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1034: expect (ObjNum 362 snap 104 seq_num 362) 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1035: rollback oid 29 current snap is 118 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 116 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1036: read oid 47 snap -1 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1036: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:21.507 INFO:tasks.rados.rados.0.smithi165.stdout:1037: copy_from oid 27 from oid 14 current snap is 118 2024-09-10T16:14:21.508 INFO:tasks.rados.rados.0.smithi165.stdout:1038: rollback oid 44 current snap is 118 2024-09-10T16:14:21.508 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 105 2024-09-10T16:14:21.508 INFO:tasks.rados.rados.0.smithi165.stdout:1039: snap_remove snap 108 2024-09-10T16:14:21.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.817+0000 7fad4500d640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.165:6789/0 conn(0x7fad2c003740 legacy=0x7fad2c005bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:14:21.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): get_auth_request con 0x7fad2c007430 auth_method 0 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient(hunting): handle_auth_done global_id 5114 payload 274 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> v1:172.21.15.165:6789/0 conn(0x7fad2c003740 legacy=0x7fad2c005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient: _finish_hunting 0 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 1 monclient: found mon.noname-b 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:21.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:21.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.819+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c00b0b0 con 0x7fad2c007430 2024-09-10T16:14:21.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad4480c640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:21.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad4480c640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fad34009d60 tx=0x7fad3402b0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c4c375b2e5e3ee server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 1 -- 172.21.15.164:0/2499897970 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad34038070 con 0x7fad2c007430 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: dump: 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:21.825 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: _finish_auth 0 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 1 -- 172.21.15.164:0/2499897970 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad34033640 con 0x7fad2c007430 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad47271640 5 monclient: authenticate success, global_id 5114 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad47271640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad47271640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad47271640 4 set_mon_vals no callback set 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 1 -- 172.21.15.164:0/2499897970 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad34033a10 con 0x7fad2c007430 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:21.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.822+0000 7fad3ffff640 10 monclient: dump: 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 10 monclient: shutdown 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:21.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 -- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad2c0098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 --2- 172.21.15.164:0/2499897970 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad2c0098f0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fad34009d60 tx=0x7fad3402b0e0 comp rx=0 tx=0).stop 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad4580e640 1 -- 172.21.15.164:0/2499897970 reap_dead start 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 -- 172.21.15.164:0/2499897970 shutdown_connections 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 -- 172.21.15.164:0/2499897970 >> 172.21.15.164:0/2499897970 conn(0x7fad400a3110 msgr2=0x7fad400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 -- 172.21.15.164:0/2499897970 shutdown_connections 2024-09-10T16:14:21.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.823+0000 7fad47271640 1 -- 172.21.15.164:0/2499897970 wait complete. 2024-09-10T16:14:21.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.824+0000 7fad47271640 10 monclient: build_initial_monmap 2024-09-10T16:14:21.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.824+0000 7fad47271640 10 monclient: monmap: 2024-09-10T16:14:21.832 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.824+0000 7fad47271640 1 Processor -- start 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.824+0000 7fad47271640 1 -- start start 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient: init 2024-09-10T16:14:21.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): picked mon.noname-c con 0x7fad2c007430 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 0x7fad4006ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): picked mon.noname-b con 0x7fad4006e7d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 0x7fad401d37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): picked mon.noname-a con 0x7fad4006f150 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fad40074f90 con 0x7fad4006f150 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad40074e10 con 0x7fad4006e7d0 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:21.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad4009f690 con 0x7fad2c007430 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad47271640 10 monclient(hunting): authenticate will time out at 1240.312513s 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4480c640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 msgr2=0x7fad4006ec10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 0x7fad401d37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53602/0 (socket says 172.21.15.164:53602) 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 1 -- 172.21.15.164:0/3790095967 learned_addr learned my addr 172.21.15.164:0/3790095967 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): get_auth_request con 0x7fad2c007430 auth_method 0 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4480c640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 0x7fad4006ec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:21.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): get_auth_request con 0x7fad4006f150 auth_method 0 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4580e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 10 monclient(hunting): handle_auth_done global_id 5117 payload 824 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 1 -- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 msgr2=0x7fad4006ec10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.825+0000 7fad4500d640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 0x7fad4006ec10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:21.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 1 -- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 msgr2=0x7fad401d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 0x7fad401d37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 10 monclient: _finish_hunting 0 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 1 monclient: found mon.noname-c 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4500d640 1 -- 172.21.15.164:0/3790095967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad40098260 con 0x7fad2c007430 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.826+0000 7fad4580e640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 0x7fad401d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad4500d640 1 -- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad40139f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad4500d640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad3000ce70 tx=0x7fad3000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30017070 con 0x7fad2c007430 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:21.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: dump: 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:21.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad47271640 5 monclient: authenticate success, global_id 5117 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad30012400 con 0x7fad2c007430 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad47271640 10 monclient: _renew_subs 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad47271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad400a5a20 con 0x7fad2c007430 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad300127b0 con 0x7fad2c007430 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3dffb640 10 monclient: dump: 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:21.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad47271640 10 monclient: _renew_subs 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.829+0000 7fad47271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.828+0000 7fad3effd640 4 set_mon_vals no callback set 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.829+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fad400979c0 con 0x7fad2c007430 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad47271640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:21.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fad3001a070 con 0x7fad2c007430 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad47271640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad47271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4009ea40 con 0x7fad2c007430 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad3dffb640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 0x7fad2c041680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(257..257 src has 184..257) ==== 4827+0+0 (secure 0 0 0) 0x7fad30052b80 con 0x7fad2c007430 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad4480c640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 0x7fad2c041680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:21.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.830+0000 7fad4480c640 10 monclient: get_auth_request con 0x7fad2c03f1c0 auth_method 0 2024-09-10T16:14:21.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.833+0000 7fad4480c640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 0x7fad2c041680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0036f0 tx=0x7fad3402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:21.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.835+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fad30051550 con 0x7fad2c007430 2024-09-10T16:14:21.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.835+0000 7fad3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:21.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:21.835+0000 7fad3dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:22.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad47271640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:22.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad47271640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad47271640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad400008d0 con 0x7fad2c007430 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad3dffb640 1 -- 172.21.15.164:0/3790095967 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fad300078e0 con 0x7fad2c007430 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.084+0000 7fad3dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:22.086 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:22.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 msgr2=0x7fad2c041680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 0x7fad2c041680 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fad2c0036f0 tx=0x7fad3402f000 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 10 monclient: shutdown 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 msgr2=0x7fad40139f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fad3000ce70 tx=0x7fad3000a430 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 shutdown_connections 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad2c03f1c0 0x7fad2c041680 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad4006f150 0x7fad401d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad4006e7d0 0x7fad4006ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 --2- 172.21.15.164:0/3790095967 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad2c007430 0x7fad40139f80 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 >> 172.21.15.164:0/3790095967 conn(0x7fad400a3110 msgr2=0x7fad4009d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 shutdown_connections 2024-09-10T16:14:22.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.087+0000 7fad47271640 1 -- 172.21.15.164:0/3790095967 wait complete. 2024-09-10T16:14:22.099 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:14:22.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:14:22.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.197+0000 7f6e31cef640 10 monclient: get_monmap_and_config 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.197+0000 7f6e31cef640 10 monclient: build_initial_monmap 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.197+0000 7f6e31cef640 10 monclient: monmap: 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:22.198596+0000 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:22.198596+0000 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:22.200 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:22.200 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:22.200 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:22.200 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:22.200 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 Processor -- start 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 -- start start 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient: init 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6e2c09d930 0x7f6e2c09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c09d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 0x7f6e2c134a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c134580 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c09d3e0 con 0x7f6e2c09d930 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6e2c09e790 con 0x7f6e2c134580 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2c0a0870 con 0x7f6e2c09ab50 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.198+0000 7f6e31cef640 10 monclient(hunting): authenticate will time out at 1240.685984s 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6e2c09d930 0x7f6e2c09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2bfff640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 msgr2=0x7f6e2c134a30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53610/0 (socket says 172.21.15.164:53610) 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6e2c09d930 0x7f6e2c09dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47568/0 (socket says 172.21.15.164:47568) 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 1 -- 172.21.15.164:0/801898922 learned_addr learned my addr 172.21.15.164:0/801898922 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request con 0x7f6e2c09ab50 auth_method 0 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2bfff640 1 --2- 172.21.15.164:0/801898922 >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 0x7f6e2c134a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e2c09d930 auth_method 0 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_done global_id 74256 payload 274 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 msgr2=0x7f6e2c09e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/801898922 >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 msgr2=0x7f6e2c134a30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/801898922 >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 0x7f6e2c134a30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/801898922 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c074c70 con 0x7f6e2c09d930 2024-09-10T16:14:22.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.199+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:22.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 msgr2=0x7f6e2c09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c09dd10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c09a180 tx=0x7f6e1c02b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb08ac7480c44437 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 1 -- 172.21.15.164:0/801898922 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c00b040 con 0x7f6e2c09d930 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: dump: 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: _finish_auth 0 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 1 -- 172.21.15.164:0/801898922 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c030920 con 0x7f6e2c09d930 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 5 monclient: authenticate success, global_id 74256 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 1 -- 172.21.15.164:0/801898922 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c030cf0 con 0x7f6e2c09d930 2024-09-10T16:14:22.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 4 set_mon_vals no callback set 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e2affd640 10 monclient: dump: 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 10 monclient: shutdown 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 -- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 msgr2=0x7f6e2c09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c09dd10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c09a180 tx=0x7f6e1c02b480 comp rx=0 tx=0).stop 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 -- 172.21.15.164:0/801898922 shutdown_connections 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 --2- 172.21.15.164:0/801898922 >> v2:172.21.15.165:3300/0 conn(0x7f6e2c134580 0x7f6e2c134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c09e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 --2- 172.21.15.164:0/801898922 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c09dd10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 -- 172.21.15.164:0/801898922 >> 172.21.15.164:0/801898922 conn(0x7f6e2c0a3110 msgr2=0x7f6e2c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 -- 172.21.15.164:0/801898922 shutdown_connections 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.202+0000 7f6e31cef640 1 -- 172.21.15.164:0/801898922 wait complete. 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.203+0000 7f6e31cef640 10 monclient: build_initial_monmap 2024-09-10T16:14:22.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.203+0000 7f6e31cef640 10 monclient: monmap: 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.203+0000 7f6e31cef640 1 Processor -- start 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 -- start start 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient: init 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c1337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-c con 0x7f6e2c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c133d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-a con 0x7f6e2c09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2c134580 0x7f6e2c072b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): picked mon.noname-b con 0x7f6e2c134580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c09ef30 con 0x7f6e2c09d930 2024-09-10T16:14:22.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c0a02b0 con 0x7f6e2c134580 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e2c0a2940 con 0x7f6e2c09ab50 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e31cef640 10 monclient(hunting): authenticate will time out at 1240.691461s 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c133d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c1337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c133d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47572/0 (socket says 172.21.15.164:47572) 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): get_auth_request con 0x7f6e2c09ab50 auth_method 0 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 1 -- 172.21.15.164:0/2212046014 learned_addr learned my addr 172.21.15.164:0/2212046014 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request con 0x7f6e2c09d930 auth_method 0 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2bfff640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2c134580 msgr2=0x7f6e2c072b20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2bfff640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2c134580 0x7f6e2c072b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient(hunting): handle_auth_done global_id 5123 payload 824 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2c134580 msgr2=0x7f6e2c072b20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e2c134580 0x7f6e2c072b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 msgr2=0x7f6e2c133d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c133d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 monclient: found mon.noname-c 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e2c12d270 con 0x7f6e2c09ab50 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.204+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c09d930 0x7f6e2c133d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e2b7fe640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 msgr2=0x7f6e2c1337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e2b7fe640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c1337f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c032000 tx=0x7f6e1c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c040070 con 0x7f6e2c09ab50 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: dump: 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: _finish_auth 0 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e1c033b60 con 0x7f6e2c09ab50 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 5 monclient: authenticate success, global_id 5123 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e1c03b430 con 0x7f6e2c09ab50 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 10 monclient: _renew_subs 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e2c1352c0 con 0x7f6e2c09ab50 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e28ff9640 10 monclient: dump: 2024-09-10T16:14:22.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e2a7fc640 4 set_mon_vals no callback set 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 10 monclient: _renew_subs 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.207+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e2c074350 con 0x7f6e2c09ab50 2024-09-10T16:14:22.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e31cef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e31cef640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 0x7f6e2c073060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c09d680 con 0x7f6e2c1c8fa0 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 0x7f6e2c073060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6e1c043070 con 0x7f6e2c09ab50 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient: get_auth_request con 0x7f6e2c1c8fa0 auth_method 0 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_done global_id 74262 payload 824 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e28ff9640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e0c03e1c0 0x7f6e0c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e22ffd640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e2c0a1e40 con 0x7f6e2c1c8fa0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(257..257 src has 184..257) ==== 4827+0+0 (secure 0 0 0) 0x7f6e1c073850 con 0x7f6e2c09ab50 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e2bfff640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e0c03e1c0 0x7f6e0c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.208+0000 7f6e2bfff640 10 monclient: get_auth_request con 0x7f6e0c03e1c0 auth_method 0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e22ffd640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 msgr2=0x7f6e2c073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 0x7f6e2c073060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e1400f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14019070 con 0x7f6e2c1c8fa0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.210+0000 7f6e28ff9640 10 monclient: dump: 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.227 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e14004450 con 0x7f6e2c1c8fa0 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 msgr2=0x7f6e2c073060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 0x7f6e2c073060 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e1400f610 comp rx=0 tx=0).stop 2024-09-10T16:14:22.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 msgr2=0x7f6e2c073060 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:22.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e28ff9640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c1c8fa0 0x7f6e2c073060 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e1400f610 comp rx=0 tx=0).stop 2024-09-10T16:14:22.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.211+0000 7f6e2bfff640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e0c03e1c0 0x7f6e0c040680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c12caa0 tx=0x7f6e24009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e31cef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e31cef640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 0x7f6e2c06e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e31cef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e2c0739b0 con 0x7f6e2c0735a0 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 0x7f6e2c06e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient: get_auth_request con 0x7f6e2c0735a0 auth_method 0 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 10 monclient(hunting): handle_auth_done global_id 74265 payload 824 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.217+0000 7f6e22ffd640 1 -- 172.21.15.164:0/2212046014 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e2c06dac0 con 0x7f6e2c0735a0 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e22ffd640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 msgr2=0x7f6e2c06e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e22ffd640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 0x7f6e2c06e7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e14019040 con 0x7f6e2c0735a0 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e28ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e28ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e28ff9640 10 monclient: dump: 2024-09-10T16:14:22.232 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.219+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4543 (secure 0 0 0) 0x7f6e1401e870 con 0x7f6e2c0735a0 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:22.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 msgr2=0x7f6e2c06e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 0x7f6e2c06e7b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e140046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 msgr2=0x7f6e2c06e7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e28ff9640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e2c0735a0 0x7f6e2c06e7b0 secure :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c074110 tx=0x7f6e140046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 78, 2024-09-10T16:14:22.234 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 90086, 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.235 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.236 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:22.237 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:22.238 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:22.239 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.240 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.241 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.242 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.243 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:22.244 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.245 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:22.246 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e0c03e1c0 msgr2=0x7f6e0c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e0c03e1c0 0x7f6e0c040680 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f6e2c12caa0 tx=0x7f6e24009270 comp rx=0 tx=0).stop 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 10 monclient: shutdown 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 msgr2=0x7f6e2c1337f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 --2- 172.21.15.164:0/2212046014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e2c09ab50 0x7f6e2c1337f0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6e1c032000 tx=0x7f6e1c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e2bfff640 1 -- 172.21.15.164:0/2212046014 reap_dead start 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e2bfff640 1 -- 172.21.15.164:0/2212046014 reap_dead start 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 shutdown_connections 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 >> 172.21.15.164:0/2212046014 conn(0x7f6e2c0a3110 msgr2=0x7f6e2c1bc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:22.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 shutdown_connections 2024-09-10T16:14:22.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.220+0000 7f6e31cef640 1 -- 172.21.15.164:0/2212046014 wait complete. 2024-09-10T16:14:22.249 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:14:22.249 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:14:22.249 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:14:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:1039: done (5 left) 2024-09-10T16:14:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:1040: rollback oid 46 current snap is 118 2024-09-10T16:14:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 116 2024-09-10T16:14:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:1034: expect (ObjNum 362 snap 104 seq_num 362) 2024-09-10T16:14:22.314 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:22.343 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 992055 2024-09-10T16:14:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.350+0000 7f158c47c640 10 monclient: get_monmap_and_config 2024-09-10T16:14:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.350+0000 7f158c47c640 10 monclient: build_initial_monmap 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.350+0000 7f158c47c640 10 monclient: monmap: 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:22.351322+0000 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:22.351322+0000 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:22.352 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:22.353 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:22.353 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:22.353 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 Processor -- start 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 -- start start 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient: init 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 0x7f158409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-b con 0x7f158409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 0x7f158409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-a con 0x7f158409d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:22.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f158409f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f158409fbc0 con 0x7f158409f4c0 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f15840a2a20 con 0x7f158409d190 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f158409c6f0 con 0x7f158409c310 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158c47c640 10 monclient(hunting): authenticate will time out at 1240.838728s 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 0x7f158409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158aa19640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f158409f4c0 0x7f158413b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52954/0 (socket says 172.21.15.164:52954) 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158aa19640 1 -- 172.21.15.164:0/1366400179 learned_addr learned my addr 172.21.15.164:0/1366400179 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 1 --2- 172.21.15.164:0/1366400179 >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 0x7f158409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): get_auth_request con 0x7f158409c310 auth_method 0 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): get_auth_request con 0x7f158409d190 auth_method 0 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.351+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 10 monclient(hunting): handle_auth_done global_id 5126 payload 274 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 1 -- 172.21.15.164:0/1366400179 >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 msgr2=0x7f158409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 1 --2- 172.21.15.164:0/1366400179 >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 0x7f158409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 1 -- 172.21.15.164:0/1366400179 >> v1:172.21.15.164:6789/0 conn(0x7f158409f4c0 legacy=0x7f158413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 10 monclient: _finish_hunting 0 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 1 monclient: found mon.noname-b 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f158a218640 1 -- 172.21.15.164:0/1366400179 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158409a4d0 con 0x7f158409c310 2024-09-10T16:14:22.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.352+0000 7f1589a17640 1 --2- 172.21.15.164:0/1366400179 >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 0x7f158409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.354+0000 7f158a218640 1 -- 172.21.15.164:0/1366400179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 msgr2=0x7f158409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.354+0000 7f158a218640 1 --2- 172.21.15.164:0/1366400179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 0x7f158409ea40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f15740097a0 tx=0x7f1574029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4a8ca62b85e422f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 1 -- 172.21.15.164:0/1366400179 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574037070 con 0x7f158409c310 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: dump: 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.362 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: _finish_auth 0 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: _check_auth_tickets 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 5 monclient: authenticate success, global_id 5126 2024-09-10T16:14:22.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 1 -- 172.21.15.164:0/1366400179 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1574032440 con 0x7f158409c310 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 4 set_mon_vals no callback set 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 1 -- 172.21.15.164:0/1366400179 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574032810 con 0x7f158409c310 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f1589216640 10 monclient: dump: 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.364 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 10 monclient: shutdown 2024-09-10T16:14:22.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 -- 172.21.15.164:0/1366400179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 msgr2=0x7f158409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 --2- 172.21.15.164:0/1366400179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 0x7f158409ea40 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f15740097a0 tx=0x7f1574029fd0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 -- 172.21.15.164:0/1366400179 shutdown_connections 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 --2- 172.21.15.164:0/1366400179 >> v2:172.21.15.164:3300/0 conn(0x7f158409d190 0x7f158409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 --2- 172.21.15.164:0/1366400179 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409c310 0x7f158409ea40 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 -- 172.21.15.164:0/1366400179 >> 172.21.15.164:0/1366400179 conn(0x7f15840a3110 msgr2=0x7f15840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 -- 172.21.15.164:0/1366400179 shutdown_connections 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.355+0000 7f158c47c640 1 -- 172.21.15.164:0/1366400179 wait complete. 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.356+0000 7f158c47c640 10 monclient: build_initial_monmap 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.356+0000 7f158c47c640 10 monclient: monmap: 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:22.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.356+0000 7f158c47c640 1 Processor -- start 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.356+0000 7f158c47c640 1 -- start start 2024-09-10T16:14:22.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.356+0000 7f158c47c640 10 monclient: init 2024-09-10T16:14:22.368 INFO:tasks.rados.rados.0.smithi165.stdout:1036: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 0x7f15841cd180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-a con 0x7f158409c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409f4c0 0x7f1584073b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-c con 0x7f158409f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1584074040 0x7f158413ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): picked mon.noname-b con 0x7f1584074040 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15840a2230 con 0x7f158409c310 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f15840a20b0 con 0x7f1584074040 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1584074e00 con 0x7f158409f4c0 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158c47c640 10 monclient(hunting): authenticate will time out at 1240.844101s 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 0x7f15841cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 0x7f15841cd180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47612/0 (socket says 172.21.15.164:47612) 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 -- 172.21.15.164:0/4127832007 learned_addr learned my addr 172.21.15.164:0/4127832007 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): get_auth_request con 0x7f158409c310 auth_method 0 2024-09-10T16:14:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158aa19640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1584074040 msgr2=0x7f158413ba80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409f4c0 0x7f1584073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158aa19640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1584074040 0x7f158413ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): get_auth_request con 0x7f158409f4c0 auth_method 0 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient(hunting): handle_auth_done global_id 74271 payload 824 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409f4c0 msgr2=0x7f1584073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409f4c0 0x7f1584073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1584074040 msgr2=0x7f158413ba80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1584074040 0x7f158413ba80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient: _finish_hunting 0 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 monclient: found mon.noname-a 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f158a218640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158413e870 con 0x7f158409c310 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.357+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409f4c0 0x7f1584073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158a218640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 msgr2=0x7f15841cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158a218640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 0x7f15841cd180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1574000c00 tx=0x7f15740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574037040 con 0x7f158409c310 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: dump: 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.383 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: _finish_auth 0 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:22.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1574042070 con 0x7f158409c310 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 5 monclient: authenticate success, global_id 74271 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 10 monclient: _renew_subs 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158409d620 con 0x7f158409c310 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157403bb00 con 0x7f158409c310 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f157affd640 10 monclient: dump: 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.385 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.386 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.386 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.386 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.387 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.387 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.387 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f1588a15640 4 set_mon_vals no callback set 2024-09-10T16:14:22.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 10 monclient: _renew_subs 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.359+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15840744d0 con 0x7f158409c310 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f158c47c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f158c47c640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 0x7f158402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f158409d930 con 0x7f158409d190 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1574047460 con 0x7f158409c310 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 0x7f158402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient: get_auth_request con 0x7f158409d190 auth_method 0 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.360+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f157affd640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f156803e1c0 0x7f1568040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f1589a17640 10 monclient(hunting): handle_auth_done global_id 5132 payload 824 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(258..258 src has 184..258) ==== 4799+0+0 (secure 0 0 0) 0x7f15740734e0 con 0x7f158409c310 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f1589a17640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f158413fc70 con 0x7f158409d190 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f158aa19640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f156803e1c0 0x7f1568040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.361+0000 7f158aa19640 10 monclient: get_auth_request con 0x7f156803e1c0 auth_method 0 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f1589a17640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 msgr2=0x7f158402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 0x7f158402d440 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f158000f610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580019070 con 0x7f158409d190 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 10 monclient: dump: 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1580004240 con 0x7f158409d190 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 msgr2=0x7f158402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 0x7f158402d440 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f158000f610 comp rx=0 tx=0).stop 2024-09-10T16:14:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 msgr2=0x7f158402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f157affd640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158409d190 0x7f158402d440 secure :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f158000f610 comp rx=0 tx=0).stop 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.363+0000 7f158aa19640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f156803e1c0 0x7f1568040680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f157c004b90 tx=0x7f157c005080 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f158c47c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f158c47c640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 0x7f158402dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f158c47c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f158409a040 con 0x7f158402d980 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 0x7f158402dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient: get_auth_request con 0x7f158402d980 auth_method 0 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.369+0000 7f1589a17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.370+0000 7f1589a17640 10 monclient(hunting): handle_auth_done global_id 5135 payload 824 2024-09-10T16:14:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.370+0000 7f1589a17640 1 -- 172.21.15.164:0/4127832007 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f158413e3b0 con 0x7f158402d980 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f1589a17640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 msgr2=0x7f158402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f1589a17640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 0x7f158402dd90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f15800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580019040 con 0x7f158402d980 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 10 monclient: dump: 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f15800044d0 con 0x7f158402d980 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 msgr2=0x7f158402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 0x7f158402dd90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f15800046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.372+0000 7f157affd640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 msgr2=0x7f158402dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f157affd640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158402d980 0x7f158402dd90 secure :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f15840a4f50 tx=0x7f15800046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 78, 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 504544, 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:22.397 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.398 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.399 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:22.400 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:22.401 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.402 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.403 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:22.405 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.406 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.407 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.408 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.408 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.408 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:22.409 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.410 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.411 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:22.412 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:22.413 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:22.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f156803e1c0 msgr2=0x7f1568040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f156803e1c0 0x7f1568040680 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f157c004b90 tx=0x7f157c005080 comp rx=0 tx=0).stop 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 10 monclient: shutdown 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 msgr2=0x7f15841cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 --2- 172.21.15.164:0/4127832007 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158409c310 0x7f15841cd180 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f1574000c00 tx=0x7f15740046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158aa19640 1 -- 172.21.15.164:0/4127832007 reap_dead start 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158aa19640 1 -- 172.21.15.164:0/4127832007 reap_dead start 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 shutdown_connections 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 >> 172.21.15.164:0/4127832007 conn(0x7f15840a3110 msgr2=0x7f1584097210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 shutdown_connections 2024-09-10T16:14:22.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:22.373+0000 7f158c47c640 1 -- 172.21.15.164:0/4127832007 wait complete. 2024-09-10T16:14:22.416 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:14:22.416 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:14:22.456 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 951190 2024-09-10T16:14:22.500 INFO:tasks.rados.rados.0.smithi165.stdout:1035: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:14:22.500 INFO:tasks.rados.rados.0.smithi165.stdout:1035: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:14:22.500 INFO:tasks.rados.rados.0.smithi165.stdout:1035: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:14:22.500 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 652 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1038: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1038: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1038: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 621 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1037: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1037: finishing copy_from to smithi16534841-27 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 320 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1034: done (5 left) 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1035: done (4 left) 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1036: done (3 left) 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1037: done (2 left) 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1038: done (1 left) 2024-09-10T16:14:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:1041: snap_create 2024-09-10T16:14:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:1040: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:14:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:1040: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:14:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:1040: finishing rollback tid 2 to smithi16534841-46 2024-09-10T16:14:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 624 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1040: done (1 left) 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1041: done (0 left) 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1042: read oid 14 snap -1 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1042: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1043: read oid 9 snap -1 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1043: expect deleted 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1044: read oid 28 snap -1 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1044: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-10T16:14:23.516 INFO:tasks.rados.rados.0.smithi165.stdout:1045: snap_create 2024-09-10T16:14:23.520 INFO:tasks.rados.rados.0.smithi165.stdout:1042: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:14:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1887003 2024-09-10T16:14:23.543 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:23.580 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1777208 2024-09-10T16:14:23.600 INFO:tasks.rados.rados.0.smithi165.stdout:1044: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-10T16:14:23.639 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1042: done (3 left) 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1043: done (2 left) 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1044: done (1 left) 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1045: done (0 left) 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1046: read oid 5 snap -1 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1046: expect deleted 2024-09-10T16:14:24.529 INFO:tasks.rados.rados.0.smithi165.stdout:1047: snap_remove snap 119 2024-09-10T16:14:25.537 INFO:tasks.rados.rados.0.smithi165.stdout:1047: done (1 left) 2024-09-10T16:14:25.538 INFO:tasks.rados.rados.0.smithi165.stdout:1048: read oid 23 snap -1 2024-09-10T16:14:25.538 INFO:tasks.rados.rados.0.smithi165.stdout:1048: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:25.538 INFO:tasks.rados.rados.0.smithi165.stdout:1049: snap_create 2024-09-10T16:14:25.543 INFO:tasks.rados.rados.0.smithi165.stdout:1048: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:25.656 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 473635 2024-09-10T16:14:26.539 INFO:tasks.rados.rados.0.smithi165.stdout:1046: done (2 left) 2024-09-10T16:14:26.540 INFO:tasks.rados.rados.0.smithi165.stdout:1048: done (1 left) 2024-09-10T16:14:26.540 INFO:tasks.rados.rados.0.smithi165.stdout:1049: done (0 left) 2024-09-10T16:14:26.540 INFO:tasks.rados.rados.0.smithi165.stdout:1050: snap_create 2024-09-10T16:14:27.156 INFO:tasks.rados.rados.0.smithi165.stdout:1050: done (0 left) 2024-09-10T16:14:27.156 INFO:tasks.rados.rados.0.smithi165.stdout:1051: snap_remove snap 114 2024-09-10T16:14:28.165 INFO:tasks.rados.rados.0.smithi165.stdout:1051: done (0 left) 2024-09-10T16:14:28.165 INFO:tasks.rados.rados.0.smithi165.stdout:1052: read oid 40 snap -1 2024-09-10T16:14:28.165 INFO:tasks.rados.rados.0.smithi165.stdout:1052: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:14:28.165 INFO:tasks.rados.rados.0.smithi165.stdout:1053: snap_create 2024-09-10T16:14:28.176 INFO:tasks.rados.rados.0.smithi165.stdout:1052: expect (ObjNum 299 snap 87 seq_num 299) 2024-09-10T16:14:28.180 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1742429 2024-09-10T16:14:28.201 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:28.234 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1052: done (1 left) 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1053: done (0 left) 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1054: copy_from oid 43 from oid 39 current snap is 123 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1055: read oid 4 snap -1 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1055: expect deleted 2024-09-10T16:14:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:1056: snap_remove snap 120 2024-09-10T16:14:30.177 INFO:tasks.rados.rados.0.smithi165.stdout:1056: done (2 left) 2024-09-10T16:14:30.178 INFO:tasks.rados.rados.0.smithi165.stdout:1057: write (excl) oid 8 current snap is 123 2024-09-10T16:14:30.178 INFO:tasks.rados.rados.0.smithi165.stdout:1057: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-10T16:14:30.185 INFO:tasks.rados.rados.0.smithi165.stdout:1057: writing smithi16534841-8 from 786660 to 1482577 tid 1 2024-09-10T16:14:30.190 INFO:tasks.rados.rados.0.smithi165.stdout:1057: writing smithi16534841-8 from 1903628 to 2339090 tid 2 2024-09-10T16:14:30.190 INFO:tasks.rados.rados.0.smithi165.stdout:1058: read oid 45 snap -1 2024-09-10T16:14:30.190 INFO:tasks.rados.rados.0.smithi165.stdout:1058: expect deleted 2024-09-10T16:14:30.190 INFO:tasks.rados.rados.0.smithi165.stdout:1059: snap_remove snap 117 2024-09-10T16:14:31.174 INFO:tasks.rados.rados.0.smithi165.stdout:1059: done (4 left) 2024-09-10T16:14:31.174 INFO:tasks.rados.rados.0.smithi165.stdout:1060: write oid 37 current snap is 123 2024-09-10T16:14:31.175 INFO:tasks.rados.rados.0.smithi165.stdout:1060: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-10T16:14:31.179 INFO:tasks.rados.rados.0.smithi165.stdout:1060: writing smithi16534841-37 from 403467 to 890986 tid 1 2024-09-10T16:14:31.186 INFO:tasks.rados.rados.0.smithi165.stdout:1060: writing smithi16534841-37 from 1562110 to 2141230 tid 2 2024-09-10T16:14:31.186 INFO:tasks.rados.rados.0.smithi165.stdout:1061: snap_remove snap 115 2024-09-10T16:14:32.159 INFO:tasks.rados.rados.0.smithi165.stdout:1061: done (5 left) 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1062: copy_from oid 29 from oid 47 current snap is 123 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1054: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1054: finishing copy_from to smithi16534841-43 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 524 (ObjNum 342 snap 98 seq_num 342) dirty exists 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: finishing write tid 1 to smithi16534841-8 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: finishing write tid 2 to smithi16534841-8 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: finishing write tid 3 to smithi16534841-8 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: finishing write tid 4 to smithi16534841-8 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: oid 8 updating version 0 to 666 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: oid 8 version 666 is already newer than 665 2024-09-10T16:14:32.160 INFO:tasks.rados.rados.0.smithi165.stdout:1057: oid 8 version 666 is already newer than 664 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 666 (ObjNum 380 snap 123 seq_num 380) dirty exists 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1057: left oid 8 (ObjNum 380 snap 123 seq_num 380) 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: finishing write tid 1 to smithi16534841-37 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: finishing write tid 2 to smithi16534841-37 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: finishing write tid 3 to smithi16534841-37 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: finishing write tid 4 to smithi16534841-37 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: oid 37 updating version 0 to 551 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: oid 37 version 551 is already newer than 550 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: oid 37 updating version 551 to 552 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 552 (ObjNum 381 snap 123 seq_num 381) dirty exists 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1060: left oid 37 (ObjNum 381 snap 123 seq_num 381) 2024-09-10T16:14:32.161 INFO:tasks.rados.rados.0.smithi165.stdout:1054: done (5 left) 2024-09-10T16:14:32.162 INFO:tasks.rados.rados.0.smithi165.stdout:1055: done (4 left) 2024-09-10T16:14:32.162 INFO:tasks.rados.rados.0.smithi165.stdout:1057: done (3 left) 2024-09-10T16:14:32.162 INFO:tasks.rados.rados.0.smithi165.stdout:1058: done (2 left) 2024-09-10T16:14:32.162 INFO:tasks.rados.rados.0.smithi165.stdout:1060: done (1 left) 2024-09-10T16:14:32.162 INFO:tasks.rados.rados.0.smithi165.stdout:1063: snap_remove snap 118 2024-09-10T16:14:33.165 INFO:tasks.rados.rados.0.smithi165.stdout:1063: done (1 left) 2024-09-10T16:14:33.166 INFO:tasks.rados.rados.0.smithi165.stdout:1064: write oid 32 current snap is 123 2024-09-10T16:14:33.166 INFO:tasks.rados.rados.0.smithi165.stdout:1062: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:14:33.166 INFO:tasks.rados.rados.0.smithi165.stdout:1064: seq_num 382 ranges {420274=562768,1620592=439131} 2024-09-10T16:14:33.170 INFO:tasks.rados.rados.0.smithi165.stdout:1064: writing smithi16534841-32 from 420274 to 983042 tid 1 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:1064: writing smithi16534841-32 from 1620592 to 2059723 tid 2 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:1065: write oid 33 current snap is 123 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:1062: finishing copy_from to smithi16534841-29 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 567 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:1064: finishing write tid 1 to smithi16534841-32 2024-09-10T16:14:33.177 INFO:tasks.rados.rados.0.smithi165.stdout:1065: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-10T16:14:33.182 INFO:tasks.rados.rados.0.smithi165.stdout:1065: writing smithi16534841-33 from 437081 to 1075098 tid 1 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1065: writing smithi16534841-33 from 1679074 to 2261863 tid 2 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1062: done (2 left) 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1066: snap_create 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: finishing write tid 2 to smithi16534841-32 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: finishing write tid 3 to smithi16534841-32 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: finishing write tid 4 to smithi16534841-32 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: oid 32 updating version 0 to 592 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: oid 32 version 592 is already newer than 590 2024-09-10T16:14:33.191 INFO:tasks.rados.rados.0.smithi165.stdout:1064: oid 32 version 592 is already newer than 591 2024-09-10T16:14:33.192 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 592 (ObjNum 382 snap 123 seq_num 382) dirty exists 2024-09-10T16:14:33.192 INFO:tasks.rados.rados.0.smithi165.stdout:1064: left oid 32 (ObjNum 382 snap 123 seq_num 382) 2024-09-10T16:14:33.192 INFO:tasks.rados.rados.0.smithi165.stdout:1065: finishing write tid 1 to smithi16534841-33 2024-09-10T16:14:33.193 INFO:tasks.rados.rados.0.smithi165.stdout:1065: finishing write tid 2 to smithi16534841-33 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: finishing write tid 3 to smithi16534841-33 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: finishing write tid 4 to smithi16534841-33 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: oid 33 updating version 0 to 674 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: oid 33 version 674 is already newer than 673 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: oid 33 updating version 674 to 675 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 675 (ObjNum 383 snap 123 seq_num 383) dirty exists 2024-09-10T16:14:33.194 INFO:tasks.rados.rados.0.smithi165.stdout:1065: left oid 33 (ObjNum 383 snap 123 seq_num 383) 2024-09-10T16:14:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:1064: done (2 left) 2024-09-10T16:14:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:1065: done (1 left) 2024-09-10T16:14:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:1066: done (0 left) 2024-09-10T16:14:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:1067: copy_from oid 15 from oid 37 current snap is 124 2024-09-10T16:14:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:1068: write (excl) oid 32 current snap is 124 2024-09-10T16:14:34.665 INFO:tasks.rados.rados.0.smithi165.stdout:1068: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-10T16:14:34.666 INFO:tasks.rados.rados.0.smithi165.stdout:1068: writing smithi16534841-32 from 453888 to 1167154 tid 1 2024-09-10T16:14:34.672 INFO:tasks.rados.rados.0.smithi165.stdout:1068: writing smithi16534841-32 from 1737556 to 2180356 tid 2 2024-09-10T16:14:34.672 INFO:tasks.rados.rados.0.smithi165.stdout:1069: snap_remove snap 123 2024-09-10T16:14:35.177 INFO:tasks.rados.rados.0.smithi165.stdout:1069: done (2 left) 2024-09-10T16:14:35.177 INFO:tasks.rados.rados.0.smithi165.stdout:1070: write oid 5 current snap is 124 2024-09-10T16:14:35.177 INFO:tasks.rados.rados.0.smithi165.stdout:1068: finishing write tid 1 to smithi16534841-32 2024-09-10T16:14:35.177 INFO:tasks.rados.rados.0.smithi165.stdout:1070: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-10T16:14:35.183 INFO:tasks.rados.rados.0.smithi165.stdout:1070: writing smithi16534841-5 from 470695 to 1259210 tid 1 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1070: writing smithi16534841-5 from 1679685 to 2266143 tid 2 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1071: copy_from oid 11 from oid 47 current snap is 124 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1067: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: finishing write tid 2 to smithi16534841-32 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: finishing write tid 3 to smithi16534841-32 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: finishing write tid 4 to smithi16534841-32 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: oid 32 updating version 0 to 595 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: oid 32 version 595 is already newer than 593 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: oid 32 version 595 is already newer than 594 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 595 (ObjNum 384 snap 124 seq_num 384) dirty exists 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1068: left oid 32 (ObjNum 384 snap 124 seq_num 384) 2024-09-10T16:14:35.190 INFO:tasks.rados.rados.0.smithi165.stdout:1067: finishing copy_from to smithi16534841-15 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 552 (ObjNum 381 snap 123 seq_num 381) dirty exists 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:1067: done (3 left) 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:1068: done (2 left) 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:1072: read oid 20 snap -1 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:1072: expect deleted 2024-09-10T16:14:35.191 INFO:tasks.rados.rados.0.smithi165.stdout:1073: snap_remove snap 122 2024-09-10T16:14:36.186 INFO:tasks.rados.rados.0.smithi165.stdout:1073: done (3 left) 2024-09-10T16:14:36.186 INFO:tasks.rados.rados.0.smithi165.stdout:1074: snap_create 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: finishing write tid 1 to smithi16534841-5 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: finishing write tid 2 to smithi16534841-5 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: finishing write tid 3 to smithi16534841-5 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: finishing write tid 4 to smithi16534841-5 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: oid 5 updating version 0 to 596 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: oid 5 updating version 596 to 597 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: oid 5 updating version 597 to 598 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 598 (ObjNum 385 snap 124 seq_num 385) dirty exists 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1070: left oid 5 (ObjNum 385 snap 124 seq_num 385) 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1071: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:1071: finishing copy_from to smithi16534841-11 2024-09-10T16:14:36.187 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 567 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:14:37.161 INFO:tasks.rados.rados.0.smithi165.stdout:1070: done (3 left) 2024-09-10T16:14:37.161 INFO:tasks.rados.rados.0.smithi165.stdout:1071: done (2 left) 2024-09-10T16:14:37.161 INFO:tasks.rados.rados.0.smithi165.stdout:1072: done (1 left) 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:1074: done (0 left) 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:1075: rollback oid 50 current snap is 125 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 105 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:1076: read oid 17 snap -1 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:1076: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:14:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:1077: delete oid 6 current snap is 125 2024-09-10T16:14:37.167 INFO:tasks.rados.rados.0.smithi165.stdout:1075: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:14:37.168 INFO:tasks.rados.rados.0.smithi165.stdout:1077: done (2 left) 2024-09-10T16:14:37.168 INFO:tasks.rados.rados.0.smithi165.stdout:1078: delete oid 14 current snap is 125 2024-09-10T16:14:37.170 INFO:tasks.rados.rados.0.smithi165.stdout:1075: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:14:37.170 INFO:tasks.rados.rados.0.smithi165.stdout:1075: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:14:37.170 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 570 (ObjNum 362 snap 104 seq_num 362) dirty exists 2024-09-10T16:14:37.170 INFO:tasks.rados.rados.0.smithi165.stdout:1076: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:14:37.261 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 261574 2024-09-10T16:14:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:1075: done (2 left) 2024-09-10T16:14:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:1076: done (1 left) 2024-09-10T16:14:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:1078: done (0 left) 2024-09-10T16:14:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:1079: write (excl) oid 26 current snap is 125 2024-09-10T16:14:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:1079: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-10T16:14:37.311 INFO:tasks.rados.rados.0.smithi165.stdout:1079: writing smithi16534841-26 from 521116 to 1251731 tid 1 2024-09-10T16:14:37.318 INFO:tasks.rados.rados.0.smithi165.stdout:1079: writing smithi16534841-26 from 1971484 to 2521116 tid 2 2024-09-10T16:14:37.319 INFO:tasks.rados.rados.0.smithi165.stdout:1080: snap_create 2024-09-10T16:14:37.319 INFO:tasks.rados.rados.0.smithi165.stdout:1079: finishing write tid 1 to smithi16534841-26 2024-09-10T16:14:37.324 INFO:tasks.rados.rados.0.smithi165.stdout:1079: finishing write tid 2 to smithi16534841-26 2024-09-10T16:14:37.324 INFO:tasks.rados.rados.0.smithi165.stdout:1079: finishing write tid 3 to smithi16534841-26 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:1079: finishing write tid 4 to smithi16534841-26 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:1079: oid 26 updating version 0 to 900 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:1079: oid 26 updating version 900 to 901 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:1079: oid 26 version 901 is already newer than 899 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 901 (ObjNum 388 snap 125 seq_num 388) dirty exists 2024-09-10T16:14:37.325 INFO:tasks.rados.rados.0.smithi165.stdout:1079: left oid 26 (ObjNum 388 snap 125 seq_num 388) 2024-09-10T16:14:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:1079: done (1 left) 2024-09-10T16:14:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:1080: done (0 left) 2024-09-10T16:14:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:1081: copy_from oid 35 from oid 48 current snap is 126 2024-09-10T16:14:38.189 INFO:tasks.rados.rados.0.smithi165.stdout:1082: snap_create 2024-09-10T16:14:38.192 INFO:tasks.rados.rados.0.smithi165.stdout:1081: finishing copy_from to smithi16534841-35 2024-09-10T16:14:38.192 INFO:tasks.rados.rados.0.smithi165.stdout:1081: got expected ENOENT (src dne) 2024-09-10T16:14:38.192 INFO:tasks.rados.rados.0.smithi165.stdout:1081: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:14:39.193 INFO:tasks.rados.rados.0.smithi165.stdout:1081: done (1 left) 2024-09-10T16:14:39.193 INFO:tasks.rados.rados.0.smithi165.stdout:1082: done (0 left) 2024-09-10T16:14:39.193 INFO:tasks.rados.rados.0.smithi165.stdout:1083: read oid 23 snap 121 2024-09-10T16:14:39.193 INFO:tasks.rados.rados.0.smithi165.stdout:1083: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:39.193 INFO:tasks.rados.rados.0.smithi165.stdout:1084: snap_remove snap 124 2024-09-10T16:14:40.199 INFO:tasks.rados.rados.0.smithi165.stdout:1084: done (1 left) 2024-09-10T16:14:40.199 INFO:tasks.rados.rados.0.smithi165.stdout:1085: delete oid 22 current snap is 127 2024-09-10T16:14:40.199 INFO:tasks.rados.rados.0.smithi165.stdout:1083: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:40.199 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 257440 2024-09-10T16:14:40.254 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:40.311 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:40.367 INFO:tasks.rados.rados.0.smithi165.stdout:1083: done (1 left) 2024-09-10T16:14:40.368 INFO:tasks.rados.rados.0.smithi165.stdout:1085: done (0 left) 2024-09-10T16:14:40.368 INFO:tasks.rados.rados.0.smithi165.stdout:1086: read oid 1 snap -1 2024-09-10T16:14:40.368 INFO:tasks.rados.rados.0.smithi165.stdout:1086: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:14:40.368 INFO:tasks.rados.rados.0.smithi165.stdout:1087: snap_remove snap 126 2024-09-10T16:14:41.210 INFO:tasks.rados.rados.0.smithi165.stdout:1087: done (1 left) 2024-09-10T16:14:41.211 INFO:tasks.rados.rados.0.smithi165.stdout:1088: read oid 30 snap 125 2024-09-10T16:14:41.211 INFO:tasks.rados.rados.0.smithi165.stdout:1088: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:14:41.211 INFO:tasks.rados.rados.0.smithi165.stdout:1089: write oid 13 current snap is 127 2024-09-10T16:14:41.211 INFO:tasks.rados.rados.0.smithi165.stdout:1089: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-10T16:14:41.216 INFO:tasks.rados.rados.0.smithi165.stdout:1089: writing smithi16534841-13 from 554730 to 1035843 tid 1 2024-09-10T16:14:41.224 INFO:tasks.rados.rados.0.smithi165.stdout:1089: writing smithi16534841-13 from 1572095 to 2309549 tid 2 2024-09-10T16:14:41.224 INFO:tasks.rados.rados.0.smithi165.stdout:1090: copy_from oid 42 from oid 45 current snap is 127 2024-09-10T16:14:41.224 INFO:tasks.rados.rados.0.smithi165.stdout:1086: expect (ObjNum 114 snap 31 seq_num 114) 2024-09-10T16:14:41.224 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:41.303 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:41.459 INFO:tasks.rados.rados.0.smithi165.stdout:1088: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: finishing write tid 1 to smithi16534841-13 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1090: finishing copy_from to smithi16534841-42 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1090: got expected ENOENT (src dne) 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1090: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: finishing write tid 2 to smithi16534841-13 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: finishing write tid 3 to smithi16534841-13 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: finishing write tid 4 to smithi16534841-13 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: oid 13 updating version 0 to 620 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: oid 13 updating version 620 to 621 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:1089: oid 13 updating version 621 to 622 2024-09-10T16:14:41.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 622 (ObjNum 390 snap 127 seq_num 390) dirty exists 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1089: left oid 13 (ObjNum 390 snap 127 seq_num 390) 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1086: done (3 left) 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1088: done (2 left) 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1089: done (1 left) 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1090: done (0 left) 2024-09-10T16:14:41.668 INFO:tasks.rados.rados.0.smithi165.stdout:1091: snap_create 2024-09-10T16:14:42.165 INFO:tasks.rados.rados.0.smithi165.stdout:1091: done (0 left) 2024-09-10T16:14:42.165 INFO:tasks.rados.rados.0.smithi165.stdout:1092: write oid 38 current snap is 128 2024-09-10T16:14:42.165 INFO:tasks.rados.rados.0.smithi165.stdout:1092: seq_num 391 ranges {571537=556362,1630577=597465} 2024-09-10T16:14:42.170 INFO:tasks.rados.rados.0.smithi165.stdout:1092: writing smithi16534841-38 from 571537 to 1127899 tid 1 2024-09-10T16:14:42.176 INFO:tasks.rados.rados.0.smithi165.stdout:1092: writing smithi16534841-38 from 1630577 to 2228042 tid 2 2024-09-10T16:14:42.177 INFO:tasks.rados.rados.0.smithi165.stdout:1093: snap_create 2024-09-10T16:14:42.181 INFO:tasks.rados.rados.0.smithi165.stdout:1092: finishing write tid 1 to smithi16534841-38 2024-09-10T16:14:42.189 INFO:tasks.rados.rados.0.smithi165.stdout:1092: finishing write tid 2 to smithi16534841-38 2024-09-10T16:14:42.189 INFO:tasks.rados.rados.0.smithi165.stdout:1092: finishing write tid 3 to smithi16534841-38 2024-09-10T16:14:42.189 INFO:tasks.rados.rados.0.smithi165.stdout:1092: finishing write tid 4 to smithi16534841-38 2024-09-10T16:14:42.189 INFO:tasks.rados.rados.0.smithi165.stdout:1092: oid 38 updating version 0 to 233 2024-09-10T16:14:42.190 INFO:tasks.rados.rados.0.smithi165.stdout:1092: oid 38 updating version 233 to 234 2024-09-10T16:14:42.190 INFO:tasks.rados.rados.0.smithi165.stdout:1092: oid 38 updating version 234 to 235 2024-09-10T16:14:42.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 235 (ObjNum 391 snap 128 seq_num 391) dirty exists 2024-09-10T16:14:42.190 INFO:tasks.rados.rados.0.smithi165.stdout:1092: left oid 38 (ObjNum 391 snap 128 seq_num 391) 2024-09-10T16:14:42.416 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:14:42.417 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:14:42.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:14:42.421 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:14:42.421 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:14:42.421 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.525+0000 7fa7a7789640 10 monclient: get_monmap_and_config 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.525+0000 7fa7a7789640 10 monclient: build_initial_monmap 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.525+0000 7fa7a7789640 10 monclient: monmap: 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:42.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:42.526734+0000 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:42.526734+0000 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:42.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.526+0000 7fa7a7789640 1 Processor -- start 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.526+0000 7fa7a7789640 1 -- start start 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.526+0000 7fa7a7789640 10 monclient: init 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a009c180 0x7fa7a009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a009c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 0x7fa7a013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a009a430 con 0x7fa7a009d160 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa7a00a24e0 con 0x7fa7a009c180 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a00a2bf0 con 0x7fa7a013a5c0 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:42.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a7789640 10 monclient(hunting): authenticate will time out at 1261.014212s 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a009c180 0x7fa7a009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 0x7fa7a013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a009c180 0x7fa7a009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36514/0 (socket says 172.21.15.164:36514) 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 -- 172.21.15.164:0/2239564896 learned_addr learned my addr 172.21.15.164:0/2239564896 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): get_auth_request con 0x7fa7a009c180 auth_method 0 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): get_auth_request con 0x7fa7a013a5c0 auth_method 0 2024-09-10T16:14:42.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient(hunting): handle_auth_done global_id 74274 payload 274 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 -- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 msgr2=0x7fa7a013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 0x7fa7a013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 -- 172.21.15.164:0/2239564896 >> v1:172.21.15.165:6789/0 conn(0x7fa7a009d160 legacy=0x7fa7a009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient: _finish_hunting 0 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 monclient: found mon.noname-a 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5525640 1 -- 172.21.15.164:0/2239564896 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a009c560 con 0x7fa7a009c180 2024-09-10T16:14:42.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.527+0000 7fa7a5d26640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 0x7fa7a013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a5525640 1 -- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a009c180 msgr2=0x7fa7a009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a5525640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a009c180 0x7fa7a009ea40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0098a0 tx=0x7fa79c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f71d520b6fba8d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 1 -- 172.21.15.164:0/2239564896 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c037070 con 0x7fa7a009c180 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: dump: 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:42.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:42.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:42.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:42.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: _finish_auth 0 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 1 -- 172.21.15.164:0/2239564896 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa79c032440 con 0x7fa7a009c180 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a7789640 5 monclient: authenticate success, global_id 74274 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a7789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:42.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 1 -- 172.21.15.164:0/2239564896 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c032810 con 0x7fa7a009c180 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a7789640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa7a7789640 4 set_mon_vals no callback set 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.530+0000 7fa797fff640 10 monclient: dump: 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 10 monclient: shutdown 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 -- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a009c180 msgr2=0x7fa7a009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a009c180 0x7fa7a009ea40 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa79c0098a0 tx=0x7fa79c02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 -- 172.21.15.164:0/2239564896 shutdown_connections 2024-09-10T16:14:42.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a013a5c0 0x7fa7a013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 --2- 172.21.15.164:0/2239564896 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a009c180 0x7fa7a009ea40 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 -- 172.21.15.164:0/2239564896 >> 172.21.15.164:0/2239564896 conn(0x7fa7a00a3110 msgr2=0x7fa7a00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 -- 172.21.15.164:0/2239564896 shutdown_connections 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 1 -- 172.21.15.164:0/2239564896 wait complete. 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 10 monclient: build_initial_monmap 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.531+0000 7fa7a7789640 10 monclient: monmap: 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:42.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 Processor -- start 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 -- start start 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient: init 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a009c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:42.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a009d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 0x7fa7a012d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a013a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0074f90 con 0x7fa7a013a5c0 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a0074e10 con 0x7fa7a009c180 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a009f690 con 0x7fa7a009d160 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a7789640 10 monclient(hunting): authenticate will time out at 1261.019668s 2024-09-10T16:14:42.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 0x7fa7a012d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58484/0 (socket says 172.21.15.164:58484) 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 1 -- 172.21.15.164:0/1185303859 learned_addr learned my addr 172.21.15.164:0/1185303859 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): get_auth_request con 0x7fa7a009d160 auth_method 0 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): get_auth_request con 0x7fa7a013a5c0 auth_method 0 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:42.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a4d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5525640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5525640 10 monclient(hunting): get_auth_request con 0x7fa7a009c180 auth_method 0 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.532+0000 7fa7a5525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a5525640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 msgr2=0x7fa7a012ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a5525640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 msgr2=0x7fa7a012ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a5525640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a5525640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 10 monclient(hunting): handle_auth_done global_id 5141 payload 824 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 msgr2=0x7fa7a012ce40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 msgr2=0x7fa7a012d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 0x7fa7a012d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 10 monclient: _finish_hunting 0 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 monclient: found mon.noname-c 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a4d24640 1 -- 172.21.15.164:0/1185303859 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a00a5d70 con 0x7fa7a009d160 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.533+0000 7fa7a5d26640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 0x7fa7a012d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a4d24640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 msgr2=0x7fa7a012d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a4d24640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c400 tx=0x7fa79000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900119a0 con 0x7fa7a009d160 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: dump: 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:42.553 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:42.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa790017070 con 0x7fa7a009d160 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 5 monclient: authenticate success, global_id 5141 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 10 monclient: _renew_subs 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a012ed80 con 0x7fa7a009d160 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790011d80 con 0x7fa7a009d160 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa795ffb640 10 monclient: dump: 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:42.555 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 10 monclient: _renew_subs 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7977fe640 4 set_mon_vals no callback set 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.535+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a012f1d0 con 0x7fa7a009d160 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa79001c460 con 0x7fa7a009d160 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa7a7789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:42.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa7a7789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa7a7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a009ea40 con 0x7fa7a009d160 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa795ffb640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 0x7fa7840406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.536+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(278..278 src has 184..278) ==== 4815+0+0 (secure 0 0 0) 0x7fa790052bc0 con 0x7fa7a009d160 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.537+0000 7fa7a5525640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 0x7fa7840406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.537+0000 7fa7a5525640 10 monclient: get_auth_request con 0x7fa78403e210 auth_method 0 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.539+0000 7fa7a5525640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 0x7fa7840406d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa79c034000 tx=0x7fa79c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:42.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.541+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa7a00a5d70 con 0x7fa7a009d160 2024-09-10T16:14:42.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.541+0000 7fa795ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:42.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.541+0000 7fa795ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:42.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa7a7789640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:42.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa7a7789640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:42.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa7a7789640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:42.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa7a00720f0 con 0x7fa7a009d160 2024-09-10T16:14:42.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa795ffb640 1 -- 172.21.15.164:0/1185303859 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fa79001ac60 con 0x7fa7a009d160 2024-09-10T16:14:42.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa795ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:42.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.782+0000 7fa795ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:42.785 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 msgr2=0x7fa7840406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 0x7fa7840406d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fa79c034000 tx=0x7fa79c02e000 comp rx=0 tx=0).stop 2024-09-10T16:14:42.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 10 monclient: shutdown 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 msgr2=0x7fa7a012d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa79000c400 tx=0x7fa79000c800 comp rx=0 tx=0).stop 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 shutdown_connections 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78403e210 0x7fa7840406d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a013a5c0 0x7fa7a012d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a009d160 0x7fa7a012d380 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 --2- 172.21.15.164:0/1185303859 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a009c180 0x7fa7a012ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:42.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 >> 172.21.15.164:0/1185303859 conn(0x7fa7a00a3110 msgr2=0x7fa7a0139ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:42.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 shutdown_connections 2024-09-10T16:14:42.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:42.784+0000 7fa7a7789640 1 -- 172.21.15.164:0/1185303859 wait complete. 2024-09-10T16:14:43.171 INFO:tasks.rados.rados.0.smithi165.stdout:1092: done (1 left) 2024-09-10T16:14:43.171 INFO:tasks.rados.rados.0.smithi165.stdout:1093: done (0 left) 2024-09-10T16:14:43.171 INFO:tasks.rados.rados.0.smithi165.stdout:1094: delete oid 25 current snap is 129 2024-09-10T16:14:43.174 INFO:tasks.rados.rados.0.smithi165.stdout:1094: done (0 left) 2024-09-10T16:14:43.175 INFO:tasks.rados.rados.0.smithi165.stdout:1095: write (excl) oid 5 current snap is 129 2024-09-10T16:14:43.175 INFO:tasks.rados.rados.0.smithi165.stdout:1095: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-10T16:14:43.181 INFO:tasks.rados.rados.0.smithi165.stdout:1095: writing smithi16534841-5 from 605151 to 1312011 tid 1 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1095: writing smithi16534841-5 from 2031188 to 2605151 tid 2 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1096: read oid 23 snap 116 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1096: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1097: rollback oid 44 current snap is 129 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 110 2024-09-10T16:14:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1098: delete oid 16 current snap is 129 2024-09-10T16:14:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1095: finishing write tid 1 to smithi16534841-5 2024-09-10T16:14:43.192 INFO:tasks.rados.rados.0.smithi165.stdout:1098: done (3 left) 2024-09-10T16:14:43.192 INFO:tasks.rados.rados.0.smithi165.stdout:1099: read oid 29 snap -1 2024-09-10T16:14:43.192 INFO:tasks.rados.rados.0.smithi165.stdout:1099: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:43.192 INFO:tasks.rados.rados.0.smithi165.stdout:1100: write (excl) oid 25 current snap is 129 2024-09-10T16:14:43.192 INFO:tasks.rados.rados.0.smithi165.stdout:1100: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-10T16:14:43.200 INFO:tasks.rados.rados.0.smithi165.stdout:1100: writing smithi16534841-25 from 638765 to 1096123 tid 1 2024-09-10T16:14:43.205 INFO:tasks.rados.rados.0.smithi165.stdout:1100: writing smithi16534841-25 from 1748152 to 2352955 tid 2 2024-09-10T16:14:43.205 INFO:tasks.rados.rados.0.smithi165.stdout:1101: snap_create 2024-09-10T16:14:43.205 INFO:tasks.rados.rados.0.smithi165.stdout:1096: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:14:43.334 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2388912 2024-09-10T16:14:43.351 INFO:tasks.rados.rados.0.smithi165.stdout:1097: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: finishing write tid 2 to smithi16534841-5 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1097: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: finishing write tid 3 to smithi16534841-5 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: finishing write tid 4 to smithi16534841-5 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: oid 5 updating version 0 to 602 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: oid 5 version 602 is already newer than 600 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: oid 5 version 602 is already newer than 601 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 602 (ObjNum 393 snap 129 seq_num 393) dirty exists 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1095: left oid 5 (ObjNum 393 snap 129 seq_num 393) 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1097: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 627 (ObjNum 190 snap 58 seq_num 190) dirty exists 2024-09-10T16:14:43.352 INFO:tasks.rados.rados.0.smithi165.stdout:1099: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 796285 2024-09-10T16:14:43.507 INFO:tasks.rados.rados.0.smithi165.stdout:1100: finishing write tid 1 to smithi16534841-25 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: finishing write tid 2 to smithi16534841-25 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: finishing write tid 3 to smithi16534841-25 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: finishing write tid 4 to smithi16534841-25 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: oid 25 updating version 0 to 679 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: oid 25 version 679 is already newer than 678 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: oid 25 version 679 is already newer than 677 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 679 (ObjNum 395 snap 129 seq_num 395) dirty exists 2024-09-10T16:14:43.508 INFO:tasks.rados.rados.0.smithi165.stdout:1100: left oid 25 (ObjNum 395 snap 129 seq_num 395) 2024-09-10T16:14:44.172 INFO:tasks.rados.rados.0.smithi165.stdout:1095: done (5 left) 2024-09-10T16:14:44.172 INFO:tasks.rados.rados.0.smithi165.stdout:1096: done (4 left) 2024-09-10T16:14:44.172 INFO:tasks.rados.rados.0.smithi165.stdout:1097: done (3 left) 2024-09-10T16:14:44.173 INFO:tasks.rados.rados.0.smithi165.stdout:1099: done (2 left) 2024-09-10T16:14:44.173 INFO:tasks.rados.rados.0.smithi165.stdout:1100: done (1 left) 2024-09-10T16:14:44.173 INFO:tasks.rados.rados.0.smithi165.stdout:1101: done (0 left) 2024-09-10T16:14:44.173 INFO:tasks.rados.rados.0.smithi165.stdout:1102: delete oid 45 current snap is 130 2024-09-10T16:14:44.174 INFO:tasks.rados.rados.0.smithi165.stdout:1102: done (0 left) 2024-09-10T16:14:44.174 INFO:tasks.rados.rados.0.smithi165.stdout:1103: write oid 18 current snap is 130 2024-09-10T16:14:44.174 INFO:tasks.rados.rados.0.smithi165.stdout:1103: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-10T16:14:44.182 INFO:tasks.rados.rados.0.smithi165.stdout:1103: writing smithi16534841-18 from 672379 to 1396588 tid 1 2024-09-10T16:14:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:1103: writing smithi16534841-18 from 1865116 to 2473588 tid 2 2024-09-10T16:14:44.188 INFO:tasks.rados.rados.0.smithi165.stdout:1104: read oid 24 snap 127 2024-09-10T16:14:44.189 INFO:tasks.rados.rados.0.smithi165.stdout:1104: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:14:44.189 INFO:tasks.rados.rados.0.smithi165.stdout:1105: write oid 25 current snap is 130 2024-09-10T16:14:44.189 INFO:tasks.rados.rados.0.smithi165.stdout:1105: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-10T16:14:44.196 INFO:tasks.rados.rados.0.smithi165.stdout:1105: writing smithi16534841-25 from 689186 to 1488644 tid 1 2024-09-10T16:14:44.203 INFO:tasks.rados.rados.0.smithi165.stdout:1105: writing smithi16534841-25 from 1923598 to 2675728 tid 2 2024-09-10T16:14:44.203 INFO:tasks.rados.rados.0.smithi165.stdout:1106: read oid 43 snap 112 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1106: expect deleted 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1107: write (excl) oid 33 current snap is 130 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1103: finishing write tid 1 to smithi16534841-18 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1103: finishing write tid 2 to smithi16534841-18 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1103: finishing write tid 3 to smithi16534841-18 2024-09-10T16:14:44.204 INFO:tasks.rados.rados.0.smithi165.stdout:1107: seq_num 399 ranges {705993=474707,1582080=612141} 2024-09-10T16:14:44.209 INFO:tasks.rados.rados.0.smithi165.stdout:1107: writing smithi16534841-33 from 705993 to 1180700 tid 1 2024-09-10T16:14:44.215 INFO:tasks.rados.rados.0.smithi165.stdout:1107: writing smithi16534841-33 from 1582080 to 2194221 tid 2 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1108: delete oid 37 current snap is 130 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1103: finishing write tid 4 to smithi16534841-18 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1103: oid 18 updating version 0 to 571 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1103: oid 18 updating version 571 to 573 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1103: oid 18 version 573 is already newer than 572 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1103: left oid 18 (ObjNum 397 snap 130 seq_num 397) 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:1104: expect (ObjNum 324 snap 94 seq_num 324) 2024-09-10T16:14:44.216 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:44.308 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:44.362 INFO:tasks.rados.rados.0.smithi165.stdout:1105: finishing write tid 1 to smithi16534841-25 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1105: finishing write tid 2 to smithi16534841-25 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1105: finishing write tid 3 to smithi16534841-25 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1103: done (5 left) 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1104: done (4 left) 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1106: done (3 left) 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1108: done (2 left) 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1109: read oid 34 snap -1 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1109: expect (ObjNum 283 snap 82 seq_num 283) 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1110: copy_from oid 41 from oid 13 current snap is 130 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1105: finishing write tid 4 to smithi16534841-25 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1105: oid 25 updating version 0 to 681 2024-09-10T16:14:44.363 INFO:tasks.rados.rados.0.smithi165.stdout:1105: oid 25 updating version 681 to 683 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1105: oid 25 version 683 is already newer than 682 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 683 (ObjNum 398 snap 130 seq_num 398) dirty exists 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1105: left oid 25 (ObjNum 398 snap 130 seq_num 398) 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: finishing write tid 1 to smithi16534841-33 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: finishing write tid 2 to smithi16534841-33 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: finishing write tid 3 to smithi16534841-33 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: finishing write tid 4 to smithi16534841-33 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: oid 33 updating version 0 to 686 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: oid 33 updating version 686 to 687 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: oid 33 version 687 is already newer than 685 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 687 (ObjNum 399 snap 130 seq_num 399) dirty exists 2024-09-10T16:14:44.364 INFO:tasks.rados.rados.0.smithi165.stdout:1107: left oid 33 (ObjNum 399 snap 130 seq_num 399) 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1105: done (3 left) 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1107: done (2 left) 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1111: delete oid 44 current snap is 130 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1111: done (2 left) 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1112: write (excl) oid 1 current snap is 130 2024-09-10T16:14:44.365 INFO:tasks.rados.rados.0.smithi165.stdout:1112: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-10T16:14:44.373 INFO:tasks.rados.rados.0.smithi165.stdout:1112: writing smithi16534841-1 from 756414 to 1456868 tid 1 2024-09-10T16:14:44.380 INFO:tasks.rados.rados.0.smithi165.stdout:1112: writing smithi16534841-1 from 2041173 to 2756414 tid 2 2024-09-10T16:14:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:1113: copy_from oid 28 from oid 7 current snap is 130 2024-09-10T16:14:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:1109: expect (ObjNum 283 snap 82 seq_num 283) 2024-09-10T16:14:44.514 INFO:tasks.rados.rados.0.smithi165.stdout:1110: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:14:44.514 INFO:tasks.rados.rados.0.smithi165.stdout:1110: finishing copy_from to smithi16534841-41 2024-09-10T16:14:44.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 622 (ObjNum 390 snap 127 seq_num 390) dirty exists 2024-09-10T16:14:44.515 INFO:tasks.rados.rados.0.smithi165.stdout:1112: finishing write tid 1 to smithi16534841-1 2024-09-10T16:14:44.515 INFO:tasks.rados.rados.0.smithi165.stdout:1112: finishing write tid 2 to smithi16534841-1 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:1112: finishing write tid 3 to smithi16534841-1 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:1112: finishing write tid 4 to smithi16534841-1 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:1112: oid 1 updating version 0 to 624 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:1112: oid 1 updating version 624 to 625 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:1112: oid 1 version 625 is already newer than 623 2024-09-10T16:14:44.516 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 625 (ObjNum 402 snap 130 seq_num 402) dirty exists 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1112: left oid 1 (ObjNum 402 snap 130 seq_num 402) 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1113: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1109: done (3 left) 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1110: done (2 left) 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1112: done (1 left) 2024-09-10T16:14:44.517 INFO:tasks.rados.rados.0.smithi165.stdout:1114: read oid 34 snap -1 2024-09-10T16:14:44.518 INFO:tasks.rados.rados.0.smithi165.stdout:1114: expect (ObjNum 283 snap 82 seq_num 283) 2024-09-10T16:14:44.518 INFO:tasks.rados.rados.0.smithi165.stdout:1115: write (excl) oid 25 current snap is 130 2024-09-10T16:14:44.518 INFO:tasks.rados.rados.0.smithi165.stdout:1113: finishing copy_from to smithi16534841-28 2024-09-10T16:14:44.518 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 342 (ObjNum 307 snap 88 seq_num 307) dirty exists 2024-09-10T16:14:44.518 INFO:tasks.rados.rados.0.smithi165.stdout:1115: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-10T16:14:44.520 INFO:tasks.rados.rados.0.smithi165.stdout:1115: writing smithi16534841-25 from 773221 to 1265277 tid 1 2024-09-10T16:14:44.526 INFO:tasks.rados.rados.0.smithi165.stdout:1115: writing smithi16534841-25 from 1816008 to 2319134 tid 2 2024-09-10T16:14:44.526 INFO:tasks.rados.rados.0.smithi165.stdout:1113: done (2 left) 2024-09-10T16:14:44.527 INFO:tasks.rados.rados.0.smithi165.stdout:1116: write (excl) oid 6 current snap is 130 2024-09-10T16:14:44.527 INFO:tasks.rados.rados.0.smithi165.stdout:1116: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-10T16:14:44.533 INFO:tasks.rados.rados.0.smithi165.stdout:1116: writing smithi16534841-6 from 790028 to 1357333 tid 1 2024-09-10T16:14:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:1116: writing smithi16534841-6 from 1874490 to 2637627 tid 2 2024-09-10T16:14:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:1117: delete oid 39 current snap is 130 2024-09-10T16:14:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:1114: expect (ObjNum 283 snap 82 seq_num 283) 2024-09-10T16:14:44.674 INFO:tasks.rados.rados.0.smithi165.stdout:1115: finishing write tid 1 to smithi16534841-25 2024-09-10T16:14:44.674 INFO:tasks.rados.rados.0.smithi165.stdout:1115: finishing write tid 2 to smithi16534841-25 2024-09-10T16:14:44.674 INFO:tasks.rados.rados.0.smithi165.stdout:1115: finishing write tid 3 to smithi16534841-25 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: finishing write tid 4 to smithi16534841-25 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: oid 25 updating version 0 to 690 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: oid 25 version 690 is already newer than 688 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: oid 25 version 690 is already newer than 689 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 690 (ObjNum 403 snap 130 seq_num 403) dirty exists 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: left oid 25 (ObjNum 403 snap 130 seq_num 403) 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1116: finishing write tid 1 to smithi16534841-6 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1114: done (3 left) 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1115: done (2 left) 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1117: done (1 left) 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1118: read oid 4 snap -1 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1118: expect deleted 2024-09-10T16:14:44.675 INFO:tasks.rados.rados.0.smithi165.stdout:1119: copy_from oid 26 from oid 33 current snap is 130 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: finishing write tid 2 to smithi16534841-6 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: finishing write tid 3 to smithi16534841-6 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: finishing write tid 4 to smithi16534841-6 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: oid 6 updating version 0 to 603 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: oid 6 updating version 603 to 604 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: oid 6 updating version 604 to 605 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 605 (ObjNum 404 snap 130 seq_num 404) dirty exists 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: left oid 6 (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1116: done (2 left) 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1120: copy_from oid 43 from oid 19 current snap is 130 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1121: rollback oid 21 current snap is 130 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 110 2024-09-10T16:14:44.676 INFO:tasks.rados.rados.0.smithi165.stdout:1122: snap_create 2024-09-10T16:14:44.685 INFO:tasks.rados.rados.0.smithi165.stdout:1121: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:14:44.685 INFO:tasks.rados.rados.0.smithi165.stdout:1121: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:14:44.685 INFO:tasks.rados.rados.0.smithi165.stdout:1121: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:14:44.685 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 694 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:14:44.689 INFO:tasks.rados.rados.0.smithi165.stdout:1120: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:14:44.693 INFO:tasks.rados.rados.0.smithi165.stdout:1119: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:14:44.696 INFO:tasks.rados.rados.0.smithi165.stdout:1119: finishing copy_from to smithi16534841-26 2024-09-10T16:14:44.696 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 687 (ObjNum 399 snap 130 seq_num 399) dirty exists 2024-09-10T16:14:44.704 INFO:tasks.rados.rados.0.smithi165.stdout:1120: finishing copy_from to smithi16534841-43 2024-09-10T16:14:44.704 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 870 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1118: done (4 left) 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1119: done (3 left) 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1120: done (2 left) 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1121: done (1 left) 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1122: done (0 left) 2024-09-10T16:14:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:1123: delete oid 42 current snap is 131 2024-09-10T16:14:45.228 INFO:tasks.rados.rados.0.smithi165.stdout:1123: done (0 left) 2024-09-10T16:14:45.229 INFO:tasks.rados.rados.0.smithi165.stdout:1124: snap_create 2024-09-10T16:14:45.798 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.905+0000 7ffb94ff8640 10 monclient: get_monmap_and_config 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.905+0000 7ffb94ff8640 10 monclient: build_initial_monmap 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.905+0000 7ffb94ff8640 10 monclient: monmap: 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:45.906640+0000 2024-09-10T16:14:45.907 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:45.906640+0000 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:45.908 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.906+0000 7ffb94ff8640 1 Processor -- start 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.906+0000 7ffb94ff8640 1 -- start start 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.906+0000 7ffb94ff8640 10 monclient: init 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffb9009c180 0x7ffb9009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb9009c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb9009d160 0x7ffb9009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb9009d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb9009f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:45.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb9009a430 con 0x7ffb9009f4c0 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ffb900a24e0 con 0x7ffb9009d160 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ffb900a2bf0 con 0x7ffb9009c180 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb94ff8640 10 monclient(hunting): authenticate will time out at 1264.394253s 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb9009d160 0x7ffb9009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb9009d160 0x7ffb9009ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48982/0 (socket says 172.21.15.164:48982) 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2742705576 learned_addr learned my addr 172.21.15.164:0/2742705576 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:45.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): get_auth_request con 0x7ffb9009d160 auth_method 0 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8e59c640 1 --2- 172.21.15.164:0/2742705576 >> v2:172.21.15.165:3300/0 conn(0x7ffb9009c180 0x7ffb9009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8e59c640 10 monclient(hunting): get_auth_request con 0x7ffb9009c180 auth_method 0 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:45.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_done global_id 74280 payload 274 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2742705576 >> v2:172.21.15.165:3300/0 conn(0x7ffb9009c180 msgr2=0x7ffb9009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 --2- 172.21.15.164:0/2742705576 >> v2:172.21.15.165:3300/0 conn(0x7ffb9009c180 0x7ffb9009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2742705576 >> v1:172.21.15.165:6789/0 conn(0x7ffb9009f4c0 legacy=0x7ffb9013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 monclient: found mon.noname-a 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:45.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.907+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2742705576 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9009c560 con 0x7ffb9009d160 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2742705576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 msgr2=0x7ffb9009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8dd9b640 1 --2- 172.21.15.164:0/2742705576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb9009ef80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b449804bb9e8c017 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 1 -- 172.21.15.164:0/2742705576 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84037070 con 0x7ffb9009d160 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: dump: 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: _finish_auth 0 2024-09-10T16:14:45.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 1 -- 172.21.15.164:0/2742705576 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb84032640 con 0x7ffb9009d160 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb94ff8640 5 monclient: authenticate success, global_id 74280 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb94ff8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb94ff8640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 1 -- 172.21.15.164:0/2742705576 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84032a10 con 0x7ffb9009d160 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb94ff8640 4 set_mon_vals no callback set 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:45.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.910+0000 7ffb8d59a640 10 monclient: dump: 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 10 monclient: shutdown 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2742705576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 msgr2=0x7ffb9009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2742705576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb9009ef80 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8402a100 comp rx=0 tx=0).stop 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2742705576 shutdown_connections 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2742705576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb9009ef80 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2742705576 >> v2:172.21.15.165:3300/0 conn(0x7ffb9009c180 0x7ffb9009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2742705576 >> 172.21.15.164:0/2742705576 conn(0x7ffb900a3110 msgr2=0x7ffb900a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2742705576 shutdown_connections 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.911+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2742705576 wait complete. 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 10 monclient: build_initial_monmap 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 10 monclient: monmap: 2024-09-10T16:14:45.925 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 1 Processor -- start 2024-09-10T16:14:45.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 1 -- start start 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 10 monclient: init 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.912+0000 7ffb94ff8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb9009c180 0x7ffb9009d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-c con 0x7ffb9009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-a con 0x7ffb9009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb9009f4c0 0x7ffb901cb6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): picked mon.noname-b con 0x7ffb9009f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffb90074f90 con 0x7ffb9009d160 2024-09-10T16:14:45.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffb90074e10 con 0x7ffb9009f4c0 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffb9013cce0 con 0x7ffb9009c180 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb94ff8640 10 monclient(hunting): authenticate will time out at 1264.400059s 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb9009c180 0x7ffb9009d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48990/0 (socket says 172.21.15.164:48990) 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2518522063 learned_addr learned my addr 172.21.15.164:0/2518522063 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:45.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): get_auth_request con 0x7ffb9009d160 auth_method 0 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): get_auth_request con 0x7ffb9009c180 auth_method 0 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:45.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8ed9d640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb9009f4c0 0x7ffb901cb6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8ed9d640 10 monclient(hunting): get_auth_request con 0x7ffb9009f4c0 auth_method 0 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient(hunting): handle_auth_done global_id 74283 payload 824 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb9009c180 msgr2=0x7ffb9009d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb9009c180 0x7ffb9009d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb9009f4c0 msgr2=0x7ffb901cb6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:45.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb9009f4c0 0x7ffb901cb6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 monclient: found mon.noname-a 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.913+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2518522063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb901cbc20 con 0x7ffb9009d160 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb8dd9b640 1 -- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 msgr2=0x7ffb901cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb8dd9b640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb840090b0 tx=0x7ffb840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84041020 con 0x7ffb9009d160 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:45.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: dump: 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: _finish_auth 0 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb8403d8f0 con 0x7ffb9009d160 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 5 monclient: authenticate success, global_id 74283 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 10 monclient: _renew_subs 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb900a5490 con 0x7ffb9009d160 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84037040 con 0x7ffb9009d160 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:45.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb7effd640 10 monclient: dump: 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:45.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb8cd99640 4 set_mon_vals no callback set 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 10 monclient: _renew_subs 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.915+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb9009e1d0 con 0x7ffb9009d160 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.916+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ffb84044020 con 0x7ffb9009d160 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.916+0000 7ffb7effd640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 0x7ffb70040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb94ff8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb94ff8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb94ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(281..281 src has 184..281) ==== 4771+0+0 (secure 0 0 0) 0x7ffb84073070 con 0x7ffb9009d160 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb9009d620 con 0x7ffb9009d160 2024-09-10T16:14:45.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb8e59c640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 0x7ffb70040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.917+0000 7ffb8e59c640 10 monclient: get_auth_request con 0x7ffb7003e290 auth_method 0 2024-09-10T16:14:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.919+0000 7ffb8e59c640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 0x7ffb70040750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb80003b80 tx=0x7ffb800098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.921+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ffb8404b070 con 0x7ffb9009d160 2024-09-10T16:14:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.921+0000 7ffb7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:45.921+0000 7ffb7effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:46.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb94ff8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb94ff8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb94ff8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffb900a4f80 con 0x7ffb9009d160 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb7effd640 1 -- 172.21.15.164:0/2518522063 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7ffb900a5490 con 0x7ffb9009d160 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.162+0000 7ffb7effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:46.164 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":78,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:46.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 msgr2=0x7ffb70040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:46.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 0x7ffb70040750 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb80003b80 tx=0x7ffb800098d0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 10 monclient: shutdown 2024-09-10T16:14:46.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:46.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 msgr2=0x7ffb901cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffb840090b0 tx=0x7ffb840046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 shutdown_connections 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb7003e290 0x7ffb70040750 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb9009f4c0 0x7ffb901cb6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb9009d160 0x7ffb901cb1a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 --2- 172.21.15.164:0/2518522063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb9009c180 0x7ffb9009d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 >> 172.21.15.164:0/2518522063 conn(0x7ffb900a3110 msgr2=0x7ffb9013a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 shutdown_connections 2024-09-10T16:14:46.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:46.164+0000 7ffb94ff8640 1 -- 172.21.15.164:0/2518522063 wait complete. 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1124: done (0 left) 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1125: read oid 29 snap -1 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1125: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1126: copy_from oid 15 from oid 25 current snap is 132 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1127: write (excl) oid 32 current snap is 132 2024-09-10T16:14:46.230 INFO:tasks.rados.rados.0.smithi165.stdout:1127: seq_num 407 ranges {440449=793052,1933583=510464} 2024-09-10T16:14:46.237 INFO:tasks.rados.rados.0.smithi165.stdout:1127: writing smithi16534841-32 from 440449 to 1233501 tid 1 2024-09-10T16:14:46.244 INFO:tasks.rados.rados.0.smithi165.stdout:1127: writing smithi16534841-32 from 1933583 to 2444047 tid 2 2024-09-10T16:14:46.244 INFO:tasks.rados.rados.0.smithi165.stdout:1128: write (excl) oid 5 current snap is 132 2024-09-10T16:14:46.244 INFO:tasks.rados.rados.0.smithi165.stdout:1128: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2024-09-10T16:14:46.249 INFO:tasks.rados.rados.0.smithi165.stdout:1128: writing smithi16534841-5 from 457256 to 925557 tid 1 2024-09-10T16:14:46.256 INFO:tasks.rados.rados.0.smithi165.stdout:1128: writing smithi16534841-5 from 1592065 to 2362540 tid 2 2024-09-10T16:14:46.259 INFO:tasks.rados.rados.0.smithi165.stdout:1128: writing smithi16534841-5 from 2854581 to 2857256 tid 3 2024-09-10T16:14:46.259 INFO:tasks.rados.rados.0.smithi165.stdout:1129: write (excl) oid 22 current snap is 132 2024-09-10T16:14:46.259 INFO:tasks.rados.rados.0.smithi165.stdout:1129: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-10T16:14:46.264 INFO:tasks.rados.rados.0.smithi165.stdout:1129: writing smithi16534841-22 from 474063 to 1017613 tid 1 2024-09-10T16:14:46.270 INFO:tasks.rados.rados.0.smithi165.stdout:1129: writing smithi16534841-22 from 1650547 to 2164680 tid 2 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1129: writing smithi16534841-22 from 2765651 to 2874063 tid 3 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1130: read oid 20 snap -1 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1130: expect deleted 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1131: read oid 36 snap -1 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1131: expect deleted 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1132: snap_create 2024-09-10T16:14:46.275 INFO:tasks.rados.rados.0.smithi165.stdout:1125: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: finishing write tid 1 to smithi16534841-32 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1126: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: finishing write tid 2 to smithi16534841-32 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: finishing write tid 3 to smithi16534841-32 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: finishing write tid 4 to smithi16534841-32 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: oid 32 updating version 0 to 606 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: oid 32 updating version 606 to 608 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: oid 32 version 608 is already newer than 607 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 608 (ObjNum 407 snap 132 seq_num 407) dirty exists 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1127: left oid 32 (ObjNum 407 snap 132 seq_num 407) 2024-09-10T16:14:46.430 INFO:tasks.rados.rados.0.smithi165.stdout:1126: finishing copy_from to smithi16534841-15 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 690 (ObjNum 403 snap 130 seq_num 403) dirty exists 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: finishing write tid 1 to smithi16534841-5 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: finishing write tid 2 to smithi16534841-5 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: finishing write tid 3 to smithi16534841-5 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: finishing write tid 4 to smithi16534841-5 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: finishing write tid 5 to smithi16534841-5 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: oid 5 updating version 0 to 609 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: oid 5 updating version 609 to 611 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: oid 5 updating version 611 to 612 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: oid 5 version 612 is already newer than 610 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 612 (ObjNum 408 snap 132 seq_num 408) dirty exists 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1128: left oid 5 (ObjNum 408 snap 132 seq_num 408) 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1129: finishing write tid 1 to smithi16534841-22 2024-09-10T16:14:46.431 INFO:tasks.rados.rados.0.smithi165.stdout:1129: finishing write tid 2 to smithi16534841-22 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: finishing write tid 3 to smithi16534841-22 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: finishing write tid 4 to smithi16534841-22 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: finishing write tid 5 to smithi16534841-22 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: oid 22 updating version 0 to 624 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: oid 22 updating version 624 to 626 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: oid 22 version 626 is already newer than 623 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: oid 22 version 626 is already newer than 625 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 626 (ObjNum 409 snap 132 seq_num 409) dirty exists 2024-09-10T16:14:46.432 INFO:tasks.rados.rados.0.smithi165.stdout:1129: left oid 22 (ObjNum 409 snap 132 seq_num 409) 2024-09-10T16:14:47.162 INFO:tasks.rados.rados.0.smithi165.stdout:1125: done (7 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1126: done (6 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1127: done (5 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1128: done (4 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1129: done (3 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1130: done (2 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1131: done (1 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1132: done (0 left) 2024-09-10T16:14:47.163 INFO:tasks.rados.rados.0.smithi165.stdout:1133: snap_remove snap 105 2024-09-10T16:14:48.604 INFO:tasks.rados.rados.0.smithi165.stdout:1133: done (0 left) 2024-09-10T16:14:48.604 INFO:tasks.rados.rados.0.smithi165.stdout:1134: snap_create 2024-09-10T16:14:49.178 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.286+0000 7f0c60672640 10 monclient: get_monmap_and_config 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.286+0000 7f0c60672640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.286+0000 7f0c60672640 10 monclient: monmap: 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:49.287668+0000 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:49.287668+0000 2024-09-10T16:14:49.288 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:49.289 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:49.290 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.287+0000 7f0c60672640 1 Processor -- start 2024-09-10T16:14:49.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.287+0000 7f0c60672640 1 -- start start 2024-09-10T16:14:49.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.287+0000 7f0c60672640 10 monclient: init 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c5809c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c5809d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 0x7f0c5813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5813b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5809a430 con 0x7f0c5809d160 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c5809f860 con 0x7f0c5809c180 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0c580a24e0 con 0x7f0c5813b5d0 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c60672640 10 monclient(hunting): authenticate will time out at 1267.775214s 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 0x7f0c5813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5dc0d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0c5809d160 0x7f0c5809f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41476/0 (socket says 172.21.15.164:41476) 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5dc0d640 1 -- 172.21.15.164:0/3020298574 learned_addr learned my addr 172.21.15.164:0/3020298574 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): get_auth_request con 0x7f0c5813b5d0 auth_method 0 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient(hunting): handle_auth_done global_id 5147 payload 274 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/3020298574 >> v1:172.21.15.165:6789/0 conn(0x7f0c5809c180 legacy=0x7f0c5809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:49.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/3020298574 >> v1:172.21.15.164:6789/0 conn(0x7f0c5809d160 legacy=0x7f0c5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 1 monclient: found mon.noname-b 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.288+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/3020298574 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c580a2bf0 con 0x7f0c5813b5d0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/3020298574 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 msgr2=0x7f0c5813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5ec0f640 1 --2- 172.21.15.164:0/3020298574 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 0x7f0c5813d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009800 tx=0x7f0c50029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57cca2d7187c77ce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 1 -- 172.21.15.164:0/3020298574 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c50037070 con 0x7f0c5813b5d0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: dump: 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.305 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c60672640 5 monclient: authenticate success, global_id 5147 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c60672640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 1 -- 172.21.15.164:0/3020298574 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c50032440 con 0x7f0c5813b5d0 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c60672640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c60672640 4 set_mon_vals no callback set 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 1 -- 172.21.15.164:0/3020298574 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c500327f0 con 0x7f0c5813b5d0 2024-09-10T16:14:49.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.291+0000 7f0c5d40c640 10 monclient: dump: 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.307 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 10 monclient: shutdown 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 -- 172.21.15.164:0/3020298574 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 msgr2=0x7f0c5813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 --2- 172.21.15.164:0/3020298574 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 0x7f0c5813d9c0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f0c50009800 tx=0x7f0c50029fe0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 -- 172.21.15.164:0/3020298574 shutdown_connections 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 --2- 172.21.15.164:0/3020298574 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5813b5d0 0x7f0c5813d9c0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 -- 172.21.15.164:0/3020298574 >> 172.21.15.164:0/3020298574 conn(0x7f0c580a3110 msgr2=0x7f0c580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 -- 172.21.15.164:0/3020298574 shutdown_connections 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.292+0000 7f0c60672640 1 -- 172.21.15.164:0/3020298574 wait complete. 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient: monmap: 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:49.309 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 Processor -- start 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 -- start start 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient: init 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:49.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-a con 0x7f0c5809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 0x7f0c581cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-c con 0x7f0c5809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): picked mon.noname-b con 0x7f0c5813b5d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580a2230 con 0x7f0c5809c180 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c580a20b0 con 0x7f0c5813b5d0 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0c58074e10 con 0x7f0c5809d160 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.293+0000 7f0c60672640 10 monclient(hunting): authenticate will time out at 1267.780956s 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 0x7f0c581cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49014/0 (socket says 172.21.15.164:49014) 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 1 -- 172.21.15.164:0/140027839 learned_addr learned my addr 172.21.15.164:0/140027839 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): get_auth_request con 0x7f0c5809c180 auth_method 0 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): get_auth_request con 0x7f0c5809d160 auth_method 0 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 10 monclient(hunting): get_auth_request con 0x7f0c5813b5d0 auth_method 0 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 msgr2=0x7f0c58093f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 msgr2=0x7f0c58093f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:49.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient(hunting): handle_auth_done global_id 5150 payload 824 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5ec0f640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 msgr2=0x7f0c58093f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 msgr2=0x7f0c581cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 monclient: found mon.noname-c 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5dc0d640 1 -- 172.21.15.164:0/140027839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c58097af0 con 0x7f0c5809d160 2024-09-10T16:14:49.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.294+0000 7f0c5e40e640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c5dc0d640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 msgr2=0x7f0c581cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c5dc0d640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 0x7f0c581cdeb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0c5809d5b0 tx=0x7f0c4c00c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c017070 con 0x7f0c5809d160 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: dump: 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c4c0070d0 con 0x7f0c5809d160 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 5 monclient: authenticate success, global_id 5150 2024-09-10T16:14:49.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c0074c0 con 0x7f0c5809d160 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c56ffd640 10 monclient: dump: 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 10 monclient: _renew_subs 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c5cc0b640 4 set_mon_vals no callback set 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c580a4cf0 con 0x7f0c5809d160 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 10 monclient: _renew_subs 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.296+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c58096f20 con 0x7f0c5809d160 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.297+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0c4c01a070 con 0x7f0c5809d160 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.297+0000 7f0c60672640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.297+0000 7f0c60672640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.297+0000 7f0c60672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.297+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c5809ea40 con 0x7f0c5809d160 2024-09-10T16:14:49.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.298+0000 7f0c56ffd640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 0x7f0c440406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.298+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(284..284 src has 184..284) ==== 4767+0+0 (secure 0 0 0) 0x7f0c4c0521c0 con 0x7f0c5809d160 2024-09-10T16:14:49.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.298+0000 7f0c5e40e640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 0x7f0c440406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.298+0000 7f0c5e40e640 10 monclient: get_auth_request con 0x7f0c4403e210 auth_method 0 2024-09-10T16:14:49.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.300+0000 7f0c5e40e640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 0x7f0c440406d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0c48007fd0 tx=0x7f0c48003e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.302+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0c4c0076e0 con 0x7f0c5809d160 2024-09-10T16:14:49.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.302+0000 7f0c56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.302+0000 7f0c56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c60672640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c60672640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c60672640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c5813a600 con 0x7f0c5809d160 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c56ffd640 1 -- 172.21.15.164:0/140027839 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f0c4c050bd0 con 0x7f0c5809d160 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.547+0000 7f0c56ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:49.549 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 msgr2=0x7f0c440406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 0x7f0c440406d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0c48007fd0 tx=0x7f0c48003e60 comp rx=0 tx=0).stop 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 10 monclient: shutdown 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 msgr2=0x7f0c581cdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 0x7f0c581cdeb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f0c5809d5b0 tx=0x7f0c4c00c870 comp rx=0 tx=0).stop 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 shutdown_connections 2024-09-10T16:14:49.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c4403e210 0x7f0c440406d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c5813b5d0 0x7f0c58093f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c5809d160 0x7f0c581cdeb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 --2- 172.21.15.164:0/140027839 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c5809c180 0x7f0c581cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 >> 172.21.15.164:0/140027839 conn(0x7f0c580a3110 msgr2=0x7f0c5809e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 shutdown_connections 2024-09-10T16:14:49.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.549+0000 7f0c60672640 1 -- 172.21.15.164:0/140027839 wait complete. 2024-09-10T16:14:49.561 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:14:49.562 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:14:49.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:14:49.607 INFO:tasks.rados.rados.0.smithi165.stdout:1134: done (0 left) 2024-09-10T16:14:49.608 INFO:tasks.rados.rados.0.smithi165.stdout:1135: snap_create 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.660+0000 7f17fb0dd640 10 monclient: get_monmap_and_config 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.660+0000 7f17fb0dd640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.660+0000 7f17fb0dd640 10 monclient: monmap: 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:49.661584+0000 2024-09-10T16:14:49.662 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:49.661584+0000 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:49.663 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 Processor -- start 2024-09-10T16:14:49.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 -- start start 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient: init 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f17f409f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-c con 0x7f17f40a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f17f40a2980 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17f409fd10 con 0x7f17f409f930 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f17f409cd20 con 0x7f17f40a2550 2024-09-10T16:14:49.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17f409a0d0 con 0x7f17f40a2980 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.661+0000 7f17fb0dd640 10 monclient(hunting): authenticate will time out at 1268.148981s 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38880/0 (socket says 172.21.15.164:38880) 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 -- 172.21.15.164:0/1169231514 learned_addr learned my addr 172.21.15.164:0/1169231514 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): get_auth_request con 0x7f17f40a2980 auth_method 0 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 1 --2- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 10 monclient(hunting): get_auth_request con 0x7f17f40a2550 auth_method 0 2024-09-10T16:14:49.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 1 -- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 msgr2=0x7f17f413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 1 -- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 msgr2=0x7f17f413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 1 --2- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f3fff640 1 --2- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient(hunting): handle_auth_done global_id 5153 payload 274 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 -- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 msgr2=0x7f17f413b5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 --2- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 -- 172.21.15.164:0/1169231514 >> v1:172.21.15.165:6789/0 conn(0x7f17f409f930 legacy=0x7f17f40a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 monclient: found mon.noname-b 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.662+0000 7f17f967a640 1 -- 172.21.15.164:0/1169231514 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f409e840 con 0x7f17f40a2980 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f967a640 1 -- 172.21.15.164:0/1169231514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 msgr2=0x7f17f413db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f967a640 1 --2- 172.21.15.164:0/1169231514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009e50 tx=0x7f17ec02a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9728cdb493113a45 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 1 -- 172.21.15.164:0/1169231514 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec036070 con 0x7f17f40a2980 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: dump: 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.683 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 1 -- 172.21.15.164:0/1169231514 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17ec007800 con 0x7f17f40a2980 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 5 monclient: authenticate success, global_id 5153 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 1 -- 172.21.15.164:0/1169231514 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17ec007bd0 con 0x7f17f40a2980 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 4 set_mon_vals no callback set 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17f37fe640 10 monclient: dump: 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.685 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 10 monclient: shutdown 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 -- 172.21.15.164:0/1169231514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 msgr2=0x7f17f413db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/1169231514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f17ec009e50 tx=0x7f17ec02a560 comp rx=0 tx=0).stop 2024-09-10T16:14:49.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 -- 172.21.15.164:0/1169231514 shutdown_connections 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/1169231514 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2980 0x7f17f413db20 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/1169231514 >> v2:172.21.15.165:3300/0 conn(0x7f17f40a2550 0x7f17f413b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 -- 172.21.15.164:0/1169231514 >> 172.21.15.164:0/1169231514 conn(0x7f17f40a3110 msgr2=0x7f17f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.665+0000 7f17fb0dd640 1 -- 172.21.15.164:0/1169231514 shutdown_connections 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 1 -- 172.21.15.164:0/1169231514 wait complete. 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 10 monclient: monmap: 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:49.687 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 1 Processor -- start 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 1 -- start start 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.666+0000 7f17fb0dd640 10 monclient: init 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409f930 0x7f17f409a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-a con 0x7f17f409f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:49.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-c con 0x7f17f40a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f40a2980 0x7f17f409aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): picked mon.noname-b con 0x7f17f40a2980 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17f409d930 con 0x7f17f409f930 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f17f409ebe0 con 0x7f17f40a2980 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17f409c8b0 con 0x7f17f40a2550 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17fb0dd640 10 monclient(hunting): authenticate will time out at 1268.154525s 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409f930 0x7f17f409a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38896/0 (socket says 172.21.15.164:38896) 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 -- 172.21.15.164:0/2364274660 learned_addr learned my addr 172.21.15.164:0/2364274660 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): get_auth_request con 0x7f17f40a2550 auth_method 0 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): get_auth_request con 0x7f17f409f930 auth_method 0 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f967a640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f40a2980 0x7f17f409aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f967a640 10 monclient(hunting): get_auth_request con 0x7f17f40a2980 auth_method 0 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f967a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f967a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f967a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient(hunting): handle_auth_done global_id 5156 payload 824 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f40a2980 msgr2=0x7f17f409aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17f40a2980 0x7f17f409aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409f930 msgr2=0x7f17f409a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409f930 0x7f17f409a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 monclient: found mon.noname-c 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.667+0000 7f17f3fff640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f4096050 con 0x7f17f40a2550 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f3fff640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 msgr2=0x7f17f409a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f3fff640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f17e000d800 tx=0x7f17e000dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e000b8d0 con 0x7f17f40a2550 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: dump: 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17e0019070 con 0x7f17f40a2550 2024-09-10T16:14:49.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 5 monclient: authenticate success, global_id 5156 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 10 monclient: _renew_subs 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f4096330 con 0x7f17f40a2550 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e000bcd0 con 0x7f17f40a2550 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f17fa640 10 monclient: dump: 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.694 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17f2ffd640 4 set_mon_vals no callback set 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 10 monclient: _renew_subs 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.670+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f406ec90 con 0x7f17f40a2550 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17fb0dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 0x7f17f40723c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17f40a0760 con 0x7f17f409b420 2024-09-10T16:14:49.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f17e001e460 con 0x7f17f40a2550 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 0x7f17f40723c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient: get_auth_request con 0x7f17f409b420 auth_method 0 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 10 monclient(hunting): handle_auth_done global_id 74292 payload 824 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f8e79640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17f409b800 con 0x7f17f409b420 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f17fa640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17d803e240 0x7f17d8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(285..285 src has 184..285) ==== 4739+0+0 (secure 0 0 0) 0x7f17e0053190 con 0x7f17f40a2550 2024-09-10T16:14:49.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f967a640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17d803e240 0x7f17d8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.671+0000 7f17f967a640 10 monclient: get_auth_request con 0x7f17d803e240 auth_method 0 2024-09-10T16:14:49.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f8e79640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 msgr2=0x7f17f40723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f8e79640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 0x7f17f40723c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f7a0 tx=0x7f17e800fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8018030 con 0x7f17f409b420 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.673+0000 7f17f17fa640 10 monclient: dump: 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.698 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f17e800d9b0 con 0x7f17f409b420 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 msgr2=0x7f17f40723c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 0x7f17f40723c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f7a0 tx=0x7f17e800fc60 comp rx=0 tx=0).stop 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 msgr2=0x7f17f40723c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f17fa640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f409b420 0x7f17f40723c0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f7a0 tx=0x7f17e800fc60 comp rx=0 tx=0).stop 2024-09-10T16:14:49.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.674+0000 7f17f967a640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17d803e240 0x7f17d8040700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17ec033000 tx=0x7f17ec009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17fb0dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:49.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 0x7f17f4072d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17fb0dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17f4095e30 con 0x7f17f4072900 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 0x7f17f4072d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient: get_auth_request con 0x7f17f4072900 auth_method 0 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 10 monclient(hunting): handle_auth_done global_id 74295 payload 824 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.680+0000 7f17f8e79640 1 -- 172.21.15.164:0/2364274660 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17f4077260 con 0x7f17f4072900 2024-09-10T16:14:49.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f8e79640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 msgr2=0x7f17f4072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f8e79640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 0x7f17f4072d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f7a0 tx=0x7f17e8003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e801dbe0 con 0x7f17f4072900 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.682+0000 7f17f17fa640 10 monclient: dump: 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.703 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f17e801dbe0 con 0x7f17f4072900 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 msgr2=0x7f17f4072d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 0x7f17f4072d10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f17e800f7a0 tx=0x7f17e8003760 comp rx=0 tx=0).stop 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 msgr2=0x7f17f4072d10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:49.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f17fa640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17f4072900 0x7f17f4072d10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 117549, 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:49.705 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.706 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.707 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:49.708 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:49.710 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.711 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.712 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.713 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.714 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.714 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:49.714 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:49.715 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.715 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.716 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.719 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:49.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.721 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:49.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.722 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.722 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.724 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:49.724 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:49.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17d803e240 msgr2=0x7f17d8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17d803e240 0x7f17d8040700 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f17ec033000 tx=0x7f17ec009000 comp rx=0 tx=0).stop 2024-09-10T16:14:49.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 10 monclient: shutdown 2024-09-10T16:14:49.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 msgr2=0x7f17f409a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f17e000d800 tx=0x7f17e000dcc0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17f967a640 1 -- 172.21.15.164:0/2364274660 reap_dead start 2024-09-10T16:14:49.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 shutdown_connections 2024-09-10T16:14:49.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 --2- 172.21.15.164:0/2364274660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17f40a2550 0x7f17f409a9a0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 >> 172.21.15.164:0/2364274660 conn(0x7f17f40a3110 msgr2=0x7f17f41bd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 shutdown_connections 2024-09-10T16:14:49.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.683+0000 7f17fb0dd640 1 -- 172.21.15.164:0/2364274660 wait complete. 2024-09-10T16:14:49.729 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:49.730 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:14:49.730 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:14:49.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.828+0000 7f87b9c23640 10 monclient: get_monmap_and_config 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.828+0000 7f87b9c23640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.828+0000 7f87b9c23640 10 monclient: monmap: 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:49.829404+0000 2024-09-10T16:14:49.830 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:49.829404+0000 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:49.831 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 Processor -- start 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 -- start start 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient: init 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 0x7f87b4132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-b con 0x7f87b41329f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87b40a2f10 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:49.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f87b409d520 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:49.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b40a0730 con 0x7f87b409d520 2024-09-10T16:14:49.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87b40a21a0 con 0x7f87b40a2f10 2024-09-10T16:14:49.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87b40a2480 con 0x7f87b41329f0 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.829+0000 7f87b9c23640 10 monclient(hunting): authenticate will time out at 1268.316929s 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 0x7f87b4132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b3fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f87b409d520 0x7f87b413dd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41478/0 (socket says 172.21.15.164:41478) 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b3fff640 1 -- 172.21.15.164:0/1192540543 learned_addr learned my addr 172.21.15.164:0/1192540543 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:49.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): get_auth_request con 0x7f87b41329f0 auth_method 0 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient(hunting): handle_auth_done global_id 5159 payload 274 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 1 -- 172.21.15.164:0/1192540543 >> v1:172.21.15.165:6789/0 conn(0x7f87b40a2f10 legacy=0x7f87b409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 1 -- 172.21.15.164:0/1192540543 >> v1:172.21.15.164:6789/0 conn(0x7f87b409d520 legacy=0x7f87b413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 1 monclient: found mon.noname-b 2024-09-10T16:14:49.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.830+0000 7f87b37fe640 1 -- 172.21.15.164:0/1192540543 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b40a1060 con 0x7f87b41329f0 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b37fe640 1 -- 172.21.15.164:0/1192540543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 msgr2=0x7f87b4132dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b37fe640 1 --2- 172.21.15.164:0/1192540543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 0x7f87b4132dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87a40097d0 tx=0x7f87a4029fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11669610867225c5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 1 -- 172.21.15.164:0/1192540543 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4037070 con 0x7f87b41329f0 2024-09-10T16:14:49.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:49.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: dump: 2024-09-10T16:14:49.844 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.844 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.844 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 1 -- 172.21.15.164:0/1192540543 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a4032440 con 0x7f87b41329f0 2024-09-10T16:14:49.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 5 monclient: authenticate success, global_id 5159 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 1 -- 172.21.15.164:0/1192540543 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a40327f0 con 0x7f87b41329f0 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 4 set_mon_vals no callback set 2024-09-10T16:14:49.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b27fc640 10 monclient: dump: 2024-09-10T16:14:49.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 10 monclient: shutdown 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 -- 172.21.15.164:0/1192540543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 msgr2=0x7f87b4132dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 --2- 172.21.15.164:0/1192540543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 0x7f87b4132dd0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f87a40097d0 tx=0x7f87a4029fa0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 -- 172.21.15.164:0/1192540543 shutdown_connections 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 --2- 172.21.15.164:0/1192540543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41329f0 0x7f87b4132dd0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 -- 172.21.15.164:0/1192540543 >> 172.21.15.164:0/1192540543 conn(0x7f87b40a7320 msgr2=0x7f87b40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 -- 172.21.15.164:0/1192540543 shutdown_connections 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.833+0000 7f87b9c23640 1 -- 172.21.15.164:0/1192540543 wait complete. 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.834+0000 7f87b9c23640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.834+0000 7f87b9c23640 10 monclient: monmap: 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.834+0000 7f87b9c23640 1 Processor -- start 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.834+0000 7f87b9c23640 1 -- start start 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.834+0000 7f87b9c23640 10 monclient: init 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:49.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 0x7f87b41cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-b con 0x7f87b409d520 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 0x7f87b4131940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-c con 0x7f87b40a2f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b41329f0 0x7f87b4131e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): picked mon.noname-a con 0x7f87b41329f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87b4133490 con 0x7f87b41329f0 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87b4133310 con 0x7f87b409d520 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87b409d990 con 0x7f87b40a2f10 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b9c23640 10 monclient(hunting): authenticate will time out at 1268.322453s 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 0x7f87b4131940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b41329f0 0x7f87b4131e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 0x7f87b4131940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38902/0 (socket says 172.21.15.164:38902) 2024-09-10T16:14:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 -- 172.21.15.164:0/3286903522 learned_addr learned my addr 172.21.15.164:0/3286903522 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:49.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): get_auth_request con 0x7f87b40a2f10 auth_method 0 2024-09-10T16:14:49.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): get_auth_request con 0x7f87b41329f0 auth_method 0 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 0x7f87b41cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 10 monclient(hunting): get_auth_request con 0x7f87b409d520 auth_method 0 2024-09-10T16:14:49.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient(hunting): handle_auth_done global_id 5162 payload 824 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 msgr2=0x7f87b41cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 msgr2=0x7f87b41cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:49.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 0x7f87b41cacc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 msgr2=0x7f87b41cacc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 0x7f87b41cacc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87b409d520 0x7f87b41cacc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b41329f0 msgr2=0x7f87b4131e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b41329f0 0x7f87b4131e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 monclient: found mon.noname-c 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b2ffd640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87b4132440 con 0x7f87b40a2f10 2024-09-10T16:14:49.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.835+0000 7f87b3fff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87b41329f0 0x7f87b4131e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:49.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b2ffd640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 msgr2=0x7f87b4131940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b2ffd640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 0x7f87b4131940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c430 tx=0x7f87a800c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a80119a0 con 0x7f87b40a2f10 2024-09-10T16:14:49.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: dump: 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.865 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: _finish_auth 0 2024-09-10T16:14:49.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 5 monclient: authenticate success, global_id 5162 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87a8017070 con 0x7f87b40a2f10 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 10 monclient: _renew_subs 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87b40a6a00 con 0x7f87b40a2f10 2024-09-10T16:14:49.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a8011d80 con 0x7f87b40a2f10 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f879bfff640 10 monclient: dump: 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 10 monclient: _renew_subs 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b17fa640 4 set_mon_vals no callback set 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.838+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87b41210e0 con 0x7f87b40a2f10 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f87a801c460 con 0x7f87b40a2f10 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b9c23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b9c23640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 0x7f87b411fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87b41235a0 con 0x7f87b41231c0 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 0x7f87b411fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient: get_auth_request con 0x7f87b41231c0 auth_method 0 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f879bfff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87a003e210 0x7f87a00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(285..285 src has 184..285) ==== 4739+0+0 (secure 0 0 0) 0x7f87a8052b80 con 0x7f87b40a2f10 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 10 monclient(hunting): handle_auth_done global_id 5165 payload 824 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87b4132dd0 con 0x7f87b41231c0 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b3fff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87a003e210 0x7f87a00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.839+0000 7f87b3fff640 10 monclient: get_auth_request con 0x7f87a003e210 auth_method 0 2024-09-10T16:14:49.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 msgr2=0x7f87b411fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 0x7f87b411fde0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87a4034000 tx=0x7f87a40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4037040 con 0x7f87b41231c0 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f879bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f879bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.841+0000 7f879bfff640 10 monclient: dump: 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.872 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f87b3fff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87a003e210 0x7f87a00406d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005c00 tx=0x7f87ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87a4032e20 con 0x7f87b41231c0 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:49.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:49.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 msgr2=0x7f87b411fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 0x7f87b411fde0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87a4034000 tx=0x7f87a40046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 msgr2=0x7f87b411fde0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:49.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.842+0000 7f879bfff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b41231c0 0x7f87b411fde0 secure :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f87a4034000 tx=0x7f87a40046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b9c23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b9c23640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 0x7f87b402d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b9c23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87b4072a10 con 0x7f87b402d440 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 0x7f87b402d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient: get_auth_request con 0x7f87b402d440 auth_method 0 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:49.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 10 monclient(hunting): handle_auth_done global_id 5168 payload 824 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.848+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87b4120320 con 0x7f87b402d440 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.850+0000 7f87b37fe640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 msgr2=0x7f87b402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.850+0000 7f87b37fe640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 0x7f87b402d850 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f87a4000c00 tx=0x7f87a4008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:49.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87a4037040 con 0x7f87b402d440 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 10 monclient: dump: 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:49.877 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:49.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f87a4049c60 con 0x7f87b402d440 2024-09-10T16:14:49.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:49.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:49.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 msgr2=0x7f87b402d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 0x7f87b402d850 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f87a4000c00 tx=0x7f87a4008030 comp rx=0 tx=0).stop 2024-09-10T16:14:49.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 msgr2=0x7f87b402d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:49.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f879bfff640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b402d440 0x7f87b402d850 secure :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f87a4000c00 tx=0x7f87a4008030 comp rx=0 tx=0).stop 2024-09-10T16:14:49.880 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:49.880 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 532023, 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:49.881 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.882 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.883 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.884 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:49.885 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:49.886 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:49.887 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.888 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.890 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.891 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.891 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.891 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.892 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:49.892 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:49.892 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.892 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.892 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:49.893 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.894 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.895 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.896 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:49.897 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87a003e210 msgr2=0x7f87a00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87a003e210 0x7f87a00406d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f87ac005c00 tx=0x7f87ac009040 comp rx=0 tx=0).stop 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 10 monclient: shutdown 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 msgr2=0x7f87b4131940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 --2- 172.21.15.164:0/3286903522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87b40a2f10 0x7f87b4131940 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f87a800c430 tx=0x7f87a800c8f0 comp rx=0 tx=0).stop 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b3fff640 1 -- 172.21.15.164:0/3286903522 reap_dead start 2024-09-10T16:14:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b3fff640 1 -- 172.21.15.164:0/3286903522 reap_dead start 2024-09-10T16:14:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 shutdown_connections 2024-09-10T16:14:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 >> 172.21.15.164:0/3286903522 conn(0x7f87b40a7320 msgr2=0x7f87b4133e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 shutdown_connections 2024-09-10T16:14:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.851+0000 7f87b9c23640 1 -- 172.21.15.164:0/3286903522 wait complete. 2024-09-10T16:14:49.900 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:49.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:14:49.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:14:49.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.997+0000 7ff56dd9f640 10 monclient: get_monmap_and_config 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.997+0000 7ff56dd9f640 10 monclient: build_initial_monmap 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.997+0000 7ff56dd9f640 10 monclient: monmap: 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:49.998227+0000 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:49.998227+0000 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:50.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 Processor -- start 2024-09-10T16:14:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 -- start start 2024-09-10T16:14:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient: init 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5681354c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff5680a5ff0 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff56813d530 0x7ff56813f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff56813d530 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff56806e180 con 0x7ff5681354c0 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff5680a63d0 con 0x7ff56813d530 2024-09-10T16:14:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff568135090 con 0x7ff5680a5ff0 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56dd9f640 10 monclient(hunting): authenticate will time out at 1268.485592s 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff56813d530 0x7ff56813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff56813d530 0x7ff56813f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49072/0 (socket says 172.21.15.164:49072) 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 1 -- 172.21.15.164:0/3202860654 learned_addr learned my addr 172.21.15.164:0/3202860654 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): get_auth_request con 0x7ff56813d530 auth_method 0 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:50.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.998+0000 7ff567fff640 1 --2- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 10 monclient(hunting): get_auth_request con 0x7ff5680a5ff0 auth_method 0 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 1 -- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 msgr2=0x7ff56813cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 1 -- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 msgr2=0x7ff56813cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 1 --2- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:49.999+0000 7ff567fff640 1 --2- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.000+0000 7ff56cd9d640 1 -- 172.21.15.164:0/3202860654 >> v1:172.21.15.164:6789/0 conn(0x7ff5681354c0 legacy=0x7ff56809d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.000+0000 7ff56d59e640 10 monclient(hunting): handle_auth_done global_id 74301 payload 274 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 1 -- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 msgr2=0x7ff56813cff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:14:50.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 1 --2- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 1 -- 172.21.15.164:0/3202860654 >> v1:172.21.15.164:6789/0 conn(0x7ff5681354c0 legacy=0x7ff56809d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 1 monclient: found mon.noname-a 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:50.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.001+0000 7ff56d59e640 1 -- 172.21.15.164:0/3202860654 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56806e460 con 0x7ff56813d530 2024-09-10T16:14:50.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.003+0000 7ff56d59e640 1 -- 172.21.15.164:0/3202860654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 msgr2=0x7ff56813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:50.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.003+0000 7ff56d59e640 1 --2- 172.21.15.164:0/3202860654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56813f920 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff5680a4ca0 tx=0x7ff56002a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3457992874b2b832 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:50.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 1 -- 172.21.15.164:0/3202860654 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560036070 con 0x7ff56813d530 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: dump: 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:50.013 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 5 monclient: authenticate success, global_id 74301 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 1 -- 172.21.15.164:0/3202860654 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff560007800 con 0x7ff56813d530 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 4 set_mon_vals no callback set 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 1 -- 172.21.15.164:0/3202860654 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560007bd0 con 0x7ff56813d530 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:50.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff5677fe640 10 monclient: dump: 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 10 monclient: shutdown 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 -- 172.21.15.164:0/3202860654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 msgr2=0x7ff56813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:50.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/3202860654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56813f920 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff5680a4ca0 tx=0x7ff56002a6d0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 -- 172.21.15.164:0/3202860654 shutdown_connections 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/3202860654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56813f920 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/3202860654 >> v2:172.21.15.165:3300/0 conn(0x7ff5680a5ff0 0x7ff56813cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 -- 172.21.15.164:0/3202860654 >> 172.21.15.164:0/3202860654 conn(0x7ff5680a70e0 msgr2=0x7ff5680a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 -- 172.21.15.164:0/3202860654 shutdown_connections 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.004+0000 7ff56dd9f640 1 -- 172.21.15.164:0/3202860654 wait complete. 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.005+0000 7ff56dd9f640 10 monclient: build_initial_monmap 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.005+0000 7ff56dd9f640 10 monclient: monmap: 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:50.016 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.005+0000 7ff56dd9f640 1 Processor -- start 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.005+0000 7ff56dd9f640 1 -- start start 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.005+0000 7ff56dd9f640 10 monclient: init 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 0x7ff568097360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-c con 0x7ff5680a5ff0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 0x7ff5680978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-b con 0x7ff5681354c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): picked mon.noname-a con 0x7ff56813d530 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff568134d80 con 0x7ff56813d530 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff56809dba0 con 0x7ff5681354c0 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff56809dec0 con 0x7ff5680a5ff0 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56dd9f640 10 monclient(hunting): authenticate will time out at 1268.493231s 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 0x7ff568097360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49088/0 (socket says 172.21.15.164:49088) 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 0x7ff568097360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38930/0 (socket says 172.21.15.164:38930) 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 learned_addr learned my addr 172.21.15.164:0/1788833120 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): get_auth_request con 0x7ff56813d530 auth_method 0 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request con 0x7ff5680a5ff0 auth_method 0 2024-09-10T16:14:50.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 0x7ff5680978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:50.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 10 monclient(hunting): get_auth_request con 0x7ff5681354c0 auth_method 0 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 msgr2=0x7ff5680978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 msgr2=0x7ff5680978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 0x7ff5680978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff567fff640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 0x7ff5680978a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient(hunting): handle_auth_done global_id 74304 payload 824 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 msgr2=0x7ff568097360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 0x7ff568097360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 msgr2=0x7ff5680978a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:50.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5681354c0 0x7ff5680978a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 monclient: found mon.noname-a 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff568138540 con 0x7ff56813d530 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.006+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5680a5ff0 0x7ff568097360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.008+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 msgr2=0x7ff56809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.008+0000 7ff56d59e640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff56002a2b0 tx=0x7ff560004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560036050 con 0x7ff56813d530 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: dump: 2024-09-10T16:14:50.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: _finish_auth 0 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: _check_auth_tickets 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff560042070 con 0x7ff56813d530 2024-09-10T16:14:50.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 5 monclient: authenticate success, global_id 74304 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 10 monclient: _renew_subs 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560031d90 con 0x7ff56813d530 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5680a1930 con 0x7ff56813d530 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff5657fa640 10 monclient: dump: 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:50.024 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff566ffd640 4 set_mon_vals no callback set 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 10 monclient: _renew_subs 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.009+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff568097de0 con 0x7ff56813d530 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56dd9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff560047460 con 0x7ff56813d530 2024-09-10T16:14:50.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff5680a12d0 con 0x7ff56809fcd0 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff5657fa640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff54c03e210 0x7ff54c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(285..285 src has 184..285) ==== 4739+0+0 (secure 0 0 0) 0x7ff560073cf0 con 0x7ff56813d530 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809fcd0 auth_method 0 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff567fff640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff54c03e210 0x7ff54c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:50.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:50.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff567fff640 10 monclient: get_auth_request con 0x7ff54c03e210 auth_method 0 2024-09-10T16:14:50.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.010+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:50.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.012+0000 7ff567fff640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff54c03e210 0x7ff54c0406d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff56813fda0 tx=0x7ff554008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809fcd0 auth_method 0 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.212+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:14:50.610 INFO:tasks.rados.rados.0.smithi165.stdout:1135: done (0 left) 2024-09-10T16:14:50.610 INFO:tasks.rados.rados.0.smithi165.stdout:1136: snap_remove snap 128 2024-09-10T16:14:50.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809fcd0 auth_method 0 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:50.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:50.614+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:14:51.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809fcd0 auth_method 0 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:14:51.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:14:51.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:14:51.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:51.415+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:14:51.620 INFO:tasks.rados.rados.0.smithi165.stdout:1136: done (0 left) 2024-09-10T16:14:51.620 INFO:tasks.rados.rados.0.smithi165.stdout:1137: copy_from oid 38 from oid 35 current snap is 135 2024-09-10T16:14:51.620 INFO:tasks.rados.rados.0.smithi165.stdout:1138: delete oid 33 current snap is 135 2024-09-10T16:14:51.629 INFO:tasks.rados.rados.0.smithi165.stdout:1138: done (1 left) 2024-09-10T16:14:51.629 INFO:tasks.rados.rados.0.smithi165.stdout:1139: snap_create 2024-09-10T16:14:51.638 INFO:tasks.rados.rados.0.smithi165.stdout:1137: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:14:51.642 INFO:tasks.rados.rados.0.smithi165.stdout:1137: finishing copy_from to smithi16534841-38 2024-09-10T16:14:51.642 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 272 (ObjNum 301 snap 87 seq_num 301) dirty exists 2024-09-10T16:14:52.163 INFO:tasks.rados.rados.0.smithi165.stdout:1137: done (1 left) 2024-09-10T16:14:52.163 INFO:tasks.rados.rados.0.smithi165.stdout:1139: done (0 left) 2024-09-10T16:14:52.163 INFO:tasks.rados.rados.0.smithi165.stdout:1140: delete oid 13 current snap is 136 2024-09-10T16:14:52.168 INFO:tasks.rados.rados.0.smithi165.stdout:1140: done (0 left) 2024-09-10T16:14:52.168 INFO:tasks.rados.rados.0.smithi165.stdout:1141: snap_remove snap 134 2024-09-10T16:14:53.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.006+0000 7ff553fff640 10 monclient: tick 2024-09-10T16:14:53.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.006+0000 7ff553fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:53.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.006+0000 7ff553fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:53.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.006+0000 7ff553fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:53.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809fcd0 auth_method 0 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_done global_id 94099 payload 824 2024-09-10T16:14:53.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.018+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff56809d450 con 0x7ff56809fcd0 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.020+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c013700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c0072e0 con 0x7ff56809fcd0 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 10 monclient: dump: 2024-09-10T16:14:53.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.023 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff55c01a030 con 0x7ff56809fcd0 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c013700 comp rx=0 tx=0).stop 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 msgr2=0x7ff56809ea20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:53.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.021+0000 7ff5657fa640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809fcd0 0x7ff56809ea20 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c013700 comp rx=0 tx=0).stop 2024-09-10T16:14:53.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.028+0000 7ff56dd9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:53.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.028+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 0x7ff568099240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.028+0000 7ff56dd9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:53.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.028+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff56809f370 con 0x7ff56809ef60 2024-09-10T16:14:53.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 0x7ff568099240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient: get_auth_request con 0x7ff56809ef60 auth_method 0 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 10 monclient(hunting): handle_auth_done global_id 94102 payload 824 2024-09-10T16:14:53.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.029+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff568076eb0 con 0x7ff56809ef60 2024-09-10T16:14:53.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.031+0000 7ff56cd9d640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 msgr2=0x7ff568099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:53.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.031+0000 7ff56cd9d640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 0x7ff568099240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:53.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff55c01f420 con 0x7ff56809ef60 2024-09-10T16:14:53.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 10 monclient: dump: 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7ff55c016420 con 0x7ff56809ef60 2024-09-10T16:14:53.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 msgr2=0x7ff568099240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 0x7ff568099240 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c000ed0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 msgr2=0x7ff568099240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff5657fa640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff56809ef60 0x7ff568099240 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff55c015bf0 tx=0x7ff55c000ed0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:53.036 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10532, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:53.037 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:53.038 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.039 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:53.040 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:53.041 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.042 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.043 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:53.044 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:53.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:53.046 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.048 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:53.049 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:53.050 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:53.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff54c03e210 msgr2=0x7ff54c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff54c03e210 0x7ff54c0406d0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff56813fda0 tx=0x7ff554008040 comp rx=0 tx=0).stop 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 10 monclient: shutdown 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 msgr2=0x7ff56809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff56002a2b0 tx=0x7ff560004a40 comp rx=0 tx=0).stop 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56d59e640 1 -- 172.21.15.164:0/1788833120 reap_dead start 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 shutdown_connections 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 --2- 172.21.15.164:0/1788833120 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff56813d530 0x7ff56809e4e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.032+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 >> 172.21.15.164:0/1788833120 conn(0x7ff5680a70e0 msgr2=0x7ff56809b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.033+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 shutdown_connections 2024-09-10T16:14:53.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.033+0000 7ff56dd9f640 1 -- 172.21.15.164:0/1788833120 wait complete. 2024-09-10T16:14:53.053 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:53.053 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:14:53.053 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:14:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.158+0000 7f16a1e52640 10 monclient: get_monmap_and_config 2024-09-10T16:14:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.158+0000 7f16a1e52640 10 monclient: build_initial_monmap 2024-09-10T16:14:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.158+0000 7f16a1e52640 10 monclient: monmap: 2024-09-10T16:14:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:53.159177+0000 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:53.159177+0000 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:53.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 Processor -- start 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 -- start start 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient: init 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f169c06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 0x7f169c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-b con 0x7f169c134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f169c13d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c09ce50 con 0x7f169c13d560 2024-09-10T16:14:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f169c135380 con 0x7f169c06cb70 2024-09-10T16:14:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f169c06db60 con 0x7f169c134f70 2024-09-10T16:14:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f16a1e52640 10 monclient(hunting): authenticate will time out at 1271.646758s 2024-09-10T16:14:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 0x7f169c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f169c13d560 0x7f169c13f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41488/0 (socket says 172.21.15.164:41488) 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169bfff640 1 -- 172.21.15.164:0/4289721562 learned_addr learned my addr 172.21.15.164:0/4289721562 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): get_auth_request con 0x7f169c134f70 auth_method 0 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.159+0000 7f169affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 10 monclient(hunting): handle_auth_done global_id 5174 payload 274 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 1 -- 172.21.15.164:0/4289721562 >> v1:172.21.15.165:6789/0 conn(0x7f169c06cb70 legacy=0x7f169c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 1 -- 172.21.15.164:0/4289721562 >> v1:172.21.15.164:6789/0 conn(0x7f169c13d560 legacy=0x7f169c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 1 monclient: found mon.noname-b 2024-09-10T16:14:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:14:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.160+0000 7f169affd640 1 -- 172.21.15.164:0/4289721562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c09cb70 con 0x7f169c134f70 2024-09-10T16:14:53.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169affd640 1 -- 172.21.15.164:0/4289721562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 msgr2=0x7f169c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:53.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169affd640 1 --2- 172.21.15.164:0/4289721562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 0x7f169c13d020 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f16900097d0 tx=0x7f1690029fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=67bed2ada2f96150 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 1 -- 172.21.15.164:0/4289721562 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1690037070 con 0x7f169c134f70 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: dump: 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.173 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 5 monclient: authenticate success, global_id 5174 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 1 -- 172.21.15.164:0/4289721562 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1690032440 con 0x7f169c134f70 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 1 -- 172.21.15.164:0/4289721562 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900327f0 con 0x7f169c134f70 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 4 set_mon_vals no callback set 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f169a7fc640 10 monclient: dump: 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 10 monclient: shutdown 2024-09-10T16:14:53.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 1 -- 172.21.15.164:0/4289721562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 msgr2=0x7f169c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 1 --2- 172.21.15.164:0/4289721562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 0x7f169c13d020 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f16900097d0 tx=0x7f1690029fa0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 1 -- 172.21.15.164:0/4289721562 shutdown_connections 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.163+0000 7f16a1e52640 1 --2- 172.21.15.164:0/4289721562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c134f70 0x7f169c13d020 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 1 -- 172.21.15.164:0/4289721562 >> 172.21.15.164:0/4289721562 conn(0x7f169c0aa900 msgr2=0x7f169c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 1 -- 172.21.15.164:0/4289721562 shutdown_connections 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 1 -- 172.21.15.164:0/4289721562 wait complete. 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 10 monclient: build_initial_monmap 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 10 monclient: monmap: 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:53.176 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.164+0000 7f16a1e52640 1 Processor -- start 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 -- start start 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient: init 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-c con 0x7f169c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 0x7f169c0a3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-b con 0x7f169c134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 0x7f169c0a4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): picked mon.noname-a con 0x7f169c13d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f169c0a4db0 con 0x7f169c13d560 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f169c06cf50 con 0x7f169c134f70 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f169c141070 con 0x7f169c06cb70 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f16a1e52640 10 monclient(hunting): authenticate will time out at 1271.652719s 2024-09-10T16:14:53.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 0x7f169c0a4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38946/0 (socket says 172.21.15.164:38946) 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 1 -- 172.21.15.164:0/1487677778 learned_addr learned my addr 172.21.15.164:0/1487677778 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): get_auth_request con 0x7f169c06cb70 auth_method 0 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): get_auth_request con 0x7f169c13d560 auth_method 0 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:53.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.165+0000 7f169bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169affd640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 0x7f169c0a3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 10 monclient(hunting): handle_auth_done global_id 74307 payload 824 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 -- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 msgr2=0x7f169c1cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 -- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 msgr2=0x7f169c0a3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 0x7f169c0a3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 monclient: found mon.noname-a 2024-09-10T16:14:53.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169bfff640 1 -- 172.21.15.164:0/1487677778 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f169c140a00 con 0x7f169c13d560 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169affd640 10 monclient: get_auth_request con 0x7f169c134f70 auth_method 0 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169b7fe640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.166+0000 7f169affd640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 0x7f169c0a3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f169bfff640 1 -- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 msgr2=0x7f169c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f169bfff640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 0x7f169c0a4050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f169400cac0 tx=0x7f169400cec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169400ad80 con 0x7f169c13d560 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: dump: 2024-09-10T16:14:53.181 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: _finish_auth 0 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1694012400 con 0x7f169c13d560 2024-09-10T16:14:53.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 5 monclient: authenticate success, global_id 74307 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16940127f0 con 0x7f169c13d560 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 10 monclient: _renew_subs 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f169c06e720 con 0x7f169c13d560 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1683fff640 10 monclient: dump: 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:53.183 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f1699ffb640 4 set_mon_vals no callback set 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 10 monclient: _renew_subs 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.168+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f169c13c3e0 con 0x7f169c13d560 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1694012a10 con 0x7f169c13d560 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f16a1e52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f16a1e52640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:53.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f16a1e52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:53.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f169c0a46a0 con 0x7f169c13d560 2024-09-10T16:14:53.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f1683fff640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 0x7f1688040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:53.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.169+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(288..288 src has 184..288) ==== 4755+0+0 (secure 0 0 0) 0x7f16940532c0 con 0x7f169c13d560 2024-09-10T16:14:53.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.170+0000 7f169b7fe640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 0x7f1688040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:53.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.170+0000 7f169b7fe640 10 monclient: get_auth_request con 0x7f168803e240 auth_method 0 2024-09-10T16:14:53.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.172+0000 7f169b7fe640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 0x7f1688040700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f168c009d80 tx=0x7f168c009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:53.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.174+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1694051cd0 con 0x7f169c13d560 2024-09-10T16:14:53.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.174+0000 7f1683fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:53.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.174+0000 7f1683fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:53.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.417+0000 7f16a1e52640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:14:53.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.417+0000 7f16a1e52640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:14:53.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.417+0000 7f16a1e52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:53.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.417+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f169c09e040 con 0x7f169c13d560 2024-09-10T16:14:53.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.418+0000 7f1683fff640 1 -- 172.21.15.164:0/1487677778 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1694053080 con 0x7f169c13d560 2024-09-10T16:14:53.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.418+0000 7f1683fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:14:53.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.418+0000 7f1683fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 msgr2=0x7f1688040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 0x7f1688040700 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f168c009d80 tx=0x7f168c009110 comp rx=0 tx=0).stop 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 10 monclient: shutdown 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 msgr2=0x7f169c0a4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 0x7f169c0a4050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f169400cac0 tx=0x7f169400cec0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 shutdown_connections 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f168803e240 0x7f1688040700 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f169c13d560 0x7f169c0a4050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f169c134f70 0x7f169c0a3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 --2- 172.21.15.164:0/1487677778 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f169c06cb70 0x7f169c1cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:53.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 >> 172.21.15.164:0/1487677778 conn(0x7f169c0aa900 msgr2=0x7f169c1335a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:53.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 shutdown_connections 2024-09-10T16:14:53.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:53.420+0000 7f16a1e52640 1 -- 172.21.15.164:0/1487677778 wait complete. 2024-09-10T16:14:53.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=2317207689,config=2586702516,health=2533334119,kv=3273972735,logm=3309617684}) 2024-09-10T16:14:53.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=16,config=2,health=11,kv=12,logm=59} crc {auth=4184414259,config=2586702516,health=2533334119,kv=805391194,logm=1070720326}) 2024-09-10T16:14:53.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=11,kv=10,logm=61} crc {auth=2317207689,config=2586702516,health=2533334119,kv=3273972735,logm=3309617684}) 2024-09-10T16:14:53.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.421+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=1,health=12,kv=9,logm=61} crc {auth=1141046314,config=1228700967,health=418911152,kv=2295417942,logm=2606769894}) 2024-09-10T16:14:53.426 INFO:tasks.rados.rados.0.smithi165.stdout:1141: done (0 left) 2024-09-10T16:14:53.426 INFO:tasks.rados.rados.0.smithi165.stdout:1142: write (excl) oid 24 current snap is 136 2024-09-10T16:14:53.426 INFO:tasks.rados.rados.0.smithi165.stdout:1142: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-10T16:14:53.431 INFO:tasks.rados.rados.0.smithi165.stdout:1142: writing smithi16534841-24 from 524484 to 1010134 tid 1 2024-09-10T16:14:53.435 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:14:53.437 INFO:tasks.rados.rados.0.smithi165.stdout:1142: writing smithi16534841-24 from 1425993 to 2087453 tid 2 2024-09-10T16:14:53.444 INFO:tasks.rados.rados.0.smithi165.stdout:1142: writing smithi16534841-24 from 2847920 to 2924484 tid 3 2024-09-10T16:14:53.444 INFO:tasks.rados.rados.0.smithi165.stdout:1143: rollback oid 46 current snap is 136 2024-09-10T16:14:53.444 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 121 2024-09-10T16:14:53.444 INFO:tasks.rados.rados.0.smithi165.stdout:1142: finishing write tid 1 to smithi16534841-24 2024-09-10T16:14:53.445 INFO:tasks.rados.rados.0.smithi165.stdout:1142: finishing write tid 2 to smithi16534841-24 2024-09-10T16:14:53.445 INFO:tasks.rados.rados.0.smithi165.stdout:1144: delete oid 44 current snap is 136 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: finishing write tid 3 to smithi16534841-24 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: finishing write tid 4 to smithi16534841-24 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: finishing write tid 5 to smithi16534841-24 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: oid 24 updating version 0 to 690 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: oid 24 updating version 690 to 691 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: oid 24 updating version 691 to 693 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: oid 24 version 693 is already newer than 692 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 693 (ObjNum 412 snap 136 seq_num 412) dirty exists 2024-09-10T16:14:53.446 INFO:tasks.rados.rados.0.smithi165.stdout:1142: left oid 24 (ObjNum 412 snap 136 seq_num 412) 2024-09-10T16:14:53.449 INFO:tasks.rados.rados.0.smithi165.stdout:1143: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:14:53.451 INFO:tasks.rados.rados.0.smithi165.stdout:1143: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:14:53.451 INFO:tasks.rados.rados.0.smithi165.stdout:1143: finishing rollback tid 2 to smithi16534841-46 2024-09-10T16:14:53.451 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 631 (ObjNum 329 snap 96 seq_num 329) dirty exists 2024-09-10T16:14:53.452 INFO:tasks.rados.rados.0.smithi165.stdout:1142: done (2 left) 2024-09-10T16:14:53.452 INFO:tasks.rados.rados.0.smithi165.stdout:1143: done (1 left) 2024-09-10T16:14:53.452 INFO:tasks.rados.rados.0.smithi165.stdout:1144: done (0 left) 2024-09-10T16:14:53.452 INFO:tasks.rados.rados.0.smithi165.stdout:1145: write (excl) oid 21 current snap is 136 2024-09-10T16:14:53.452 INFO:tasks.rados.rados.0.smithi165.stdout:1145: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-10T16:14:53.458 INFO:tasks.rados.rados.0.smithi165.stdout:1145: writing smithi16534841-21 from 558098 to 1194246 tid 1 2024-09-10T16:14:53.469 INFO:tasks.rados.rados.0.smithi165.stdout:1145: writing smithi16534841-21 from 1826604 to 2491733 tid 2 2024-09-10T16:14:53.470 INFO:tasks.rados.rados.0.smithi165.stdout:1146: write (excl) oid 45 current snap is 136 2024-09-10T16:14:53.470 INFO:tasks.rados.rados.0.smithi165.stdout:1145: finishing write tid 1 to smithi16534841-21 2024-09-10T16:14:53.470 INFO:tasks.rados.rados.0.smithi165.stdout:1146: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-10T16:14:53.473 INFO:tasks.rados.rados.0.smithi165.stdout:1146: writing smithi16534841-45 from 574905 to 1286302 tid 1 2024-09-10T16:14:53.479 INFO:tasks.rados.rados.0.smithi165.stdout:1146: writing smithi16534841-45 from 1885086 to 2410226 tid 2 2024-09-10T16:14:53.482 INFO:tasks.rados.rados.0.smithi165.stdout:1146: writing smithi16534841-45 from 2930189 to 2974905 tid 3 2024-09-10T16:14:53.482 INFO:tasks.rados.rados.0.smithi165.stdout:1147: copy_from oid 26 from oid 49 current snap is 136 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: finishing write tid 2 to smithi16534841-21 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: finishing write tid 3 to smithi16534841-21 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: finishing write tid 4 to smithi16534841-21 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: oid 21 updating version 0 to 712 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: oid 21 version 712 is already newer than 711 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: oid 21 version 712 is already newer than 710 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 712 (ObjNum 414 snap 136 seq_num 414) dirty exists 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: left oid 21 (ObjNum 414 snap 136 seq_num 414) 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1146: finishing write tid 1 to smithi16534841-45 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1145: done (2 left) 2024-09-10T16:14:53.483 INFO:tasks.rados.rados.0.smithi165.stdout:1148: write oid 34 current snap is 136 2024-09-10T16:14:53.484 INFO:tasks.rados.rados.0.smithi165.stdout:1148: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-10T16:14:53.490 INFO:tasks.rados.rados.0.smithi165.stdout:1148: writing smithi16534841-34 from 591712 to 1378358 tid 1 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1148: writing smithi16534841-34 from 1943568 to 2612366 tid 2 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1149: read oid 5 snap 135 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1149: expect (ObjNum 408 snap 132 seq_num 408) 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1150: snap_create 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1146: finishing write tid 2 to smithi16534841-45 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1146: finishing write tid 3 to smithi16534841-45 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1146: finishing write tid 4 to smithi16534841-45 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1146: finishing write tid 5 to smithi16534841-45 2024-09-10T16:14:53.498 INFO:tasks.rados.rados.0.smithi165.stdout:1146: oid 45 updating version 0 to 872 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1146: oid 45 updating version 872 to 873 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1146: oid 45 version 873 is already newer than 871 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1146: oid 45 updating version 873 to 874 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 874 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1146: left oid 45 (ObjNum 415 snap 136 seq_num 415) 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1147: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1147: finishing copy_from to smithi16534841-26 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 335 (ObjNum 372 snap 108 seq_num 372) dirty exists 2024-09-10T16:14:53.499 INFO:tasks.rados.rados.0.smithi165.stdout:1148: finishing write tid 1 to smithi16534841-34 2024-09-10T16:14:53.501 INFO:tasks.rados.rados.0.smithi165.stdout:1148: finishing write tid 2 to smithi16534841-34 2024-09-10T16:14:53.501 INFO:tasks.rados.rados.0.smithi165.stdout:1148: finishing write tid 3 to smithi16534841-34 2024-09-10T16:14:53.501 INFO:tasks.rados.rados.0.smithi165.stdout:1148: finishing write tid 4 to smithi16534841-34 2024-09-10T16:14:53.502 INFO:tasks.rados.rados.0.smithi165.stdout:1148: oid 34 updating version 0 to 554 2024-09-10T16:14:53.502 INFO:tasks.rados.rados.0.smithi165.stdout:1148: oid 34 updating version 554 to 555 2024-09-10T16:14:53.502 INFO:tasks.rados.rados.0.smithi165.stdout:1148: oid 34 updating version 555 to 556 2024-09-10T16:14:53.502 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 556 (ObjNum 416 snap 136 seq_num 416) dirty exists 2024-09-10T16:14:53.502 INFO:tasks.rados.rados.0.smithi165.stdout:1148: left oid 34 (ObjNum 416 snap 136 seq_num 416) 2024-09-10T16:14:53.508 INFO:tasks.rados.rados.0.smithi165.stdout:1149: expect (ObjNum 408 snap 132 seq_num 408) 2024-09-10T16:14:53.511 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 512928 2024-09-10T16:14:53.552 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 592482 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=675328791}) 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3657682954}) 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=675328791}) 2024-09-10T16:14:53.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.572+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=461776210}) 2024-09-10T16:14:53.588 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:14:53.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362013803}) 2024-09-10T16:14:53.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1081980813}) 2024-09-10T16:14:53.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362013803}) 2024-09-10T16:14:53.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.615+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4140637869}) 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279302714}) 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4169473384}) 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3279302714}) 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.620+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2786426883}) 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.633 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1953318708}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520457657}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1953318708}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=554589255}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.621+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084268173}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=599975041}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2084268173}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674757041}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=480188114}) 2024-09-10T16:14:53.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1941950729}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=480188114}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.622+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870425926}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067119272}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001996493}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067119272}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=12253857}) 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711651327}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022203190}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.623+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3711651327}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1670157100}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255020259}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692688561}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2255020259}) 2024-09-10T16:14:53.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.624+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078267246}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809145503}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456793677}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809145503}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287082369}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523814715}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665713874}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2523814715}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.625+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682367546}) 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655178865}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1032146643}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2655178865}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.626+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3810380799}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354743386}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3982449144}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=354743386}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4066769737}) 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872295622}) 2024-09-10T16:14:53.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=924100113}) 2024-09-10T16:14:53.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3872295622}) 2024-09-10T16:14:53.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.627+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=866238870}) 2024-09-10T16:14:53.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144833817}) 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3979557948}) 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2144833817}) 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.628+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198675607}) 2024-09-10T16:14:53.641 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.642 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807821455}) 2024-09-10T16:14:53.642 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2233404402}) 2024-09-10T16:14:53.642 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:53.642 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2807821455}) 2024-09-10T16:14:53.642 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:53.629+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548669037}) 2024-09-10T16:14:54.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2334505257}) 2024-09-10T16:14:54.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2233271851}) 2024-09-10T16:14:54.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2334505257}) 2024-09-10T16:14:54.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.419+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1388692391}) 2024-09-10T16:14:54.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043487615}) 2024-09-10T16:14:54.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3596829650}) 2024-09-10T16:14:54.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3043487615}) 2024-09-10T16:14:54.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.424+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=835803338}) 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682902431}) 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4228528441}) 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682902431}) 2024-09-10T16:14:54.431 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.429+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3654020683}) 2024-09-10T16:14:54.435 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:14:54.436 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1146: done (4 left) 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1147: done (3 left) 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1148: done (2 left) 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1149: done (1 left) 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1150: done (0 left) 2024-09-10T16:14:54.438 INFO:tasks.rados.rados.0.smithi165.stdout:1151: read oid 28 snap -1 2024-09-10T16:14:54.439 INFO:tasks.rados.rados.0.smithi165.stdout:1151: expect (ObjNum 307 snap 88 seq_num 307) 2024-09-10T16:14:54.439 INFO:tasks.rados.rados.0.smithi165.stdout:1152: rollback oid 11 current snap is 137 2024-09-10T16:14:54.439 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 116 2024-09-10T16:14:54.439 INFO:tasks.rados.rados.0.smithi165.stdout:1153: snap_remove snap 130 2024-09-10T16:14:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.542+0000 7f63493a1640 10 monclient: get_monmap_and_config 2024-09-10T16:14:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.542+0000 7f63493a1640 10 monclient: build_initial_monmap 2024-09-10T16:14:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.542+0000 7f63493a1640 10 monclient: monmap: 2024-09-10T16:14:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:54.542947+0000 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:54.542947+0000 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:54.544 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 Processor -- start 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 -- start start 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient: init 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T16:14:54.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 0x7f634409da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-c con 0x7f634409d680 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f634406d1f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f634406d5d0 0x7f6344138290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-a con 0x7f634406d5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f634406e110 con 0x7f634406d1f0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f63440ab520 con 0x7f634406d5d0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f634406e3f0 con 0x7f634409d680 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f63493a1640 10 monclient(hunting): authenticate will time out at 1273.030752s 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f634406d5d0 0x7f6344138290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f634406d5d0 0x7f6344138290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35872/0 (socket says 172.21.15.164:35872) 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 1 -- 172.21.15.164:0/3310239711 learned_addr learned my addr 172.21.15.164:0/3310239711 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): get_auth_request con 0x7f634406d5d0 auth_method 0 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.543+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 10 monclient(hunting): handle_auth_done global_id 74310 payload 274 2024-09-10T16:14:54.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 1 -- 172.21.15.164:0/3310239711 >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 msgr2=0x7f634409da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6343fff640 1 --2- 172.21.15.164:0/3310239711 >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 0x7f634409da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 1 --2- 172.21.15.164:0/3310239711 >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 0x7f634409da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 1 -- 172.21.15.164:0/3310239711 >> v1:172.21.15.165:6789/0 conn(0x7f634406d1f0 legacy=0x7f6344137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 10 monclient: _finish_hunting 0 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 1 monclient: found mon.noname-a 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6348ba0640 1 -- 172.21.15.164:0/3310239711 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634409dfa0 con 0x7f634406d5d0 2024-09-10T16:14:54.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6343fff640 10 monclient: get_auth_request con 0x7f634409d680 auth_method 0 2024-09-10T16:14:54.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.544+0000 7f6343fff640 1 --2- 172.21.15.164:0/3310239711 >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 0x7f634409da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:14:54.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.546+0000 7f6348ba0640 1 -- 172.21.15.164:0/3310239711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634406d5d0 msgr2=0x7f6344138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f6348ba0640 1 --2- 172.21.15.164:0/3310239711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634406d5d0 0x7f6344138290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63440a8080 tx=0x7f633c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd6231523da2ecec server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 1 -- 172.21.15.164:0/3310239711 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c036070 con 0x7f634406d5d0 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: dump: 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 1 -- 172.21.15.164:0/3310239711 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633c007800 con 0x7f634406d5d0 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 5 monclient: authenticate success, global_id 74310 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 1 -- 172.21.15.164:0/3310239711 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c007bd0 con 0x7f634406d5d0 2024-09-10T16:14:54.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 4 set_mon_vals no callback set 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63437fe640 10 monclient: dump: 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 10 monclient: shutdown 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 1 -- 172.21.15.164:0/3310239711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634406d5d0 msgr2=0x7f6344138290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.547+0000 7f63493a1640 1 --2- 172.21.15.164:0/3310239711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634406d5d0 0x7f6344138290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f63440a8080 tx=0x7f633c02a570 comp rx=0 tx=0).stop 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 -- 172.21.15.164:0/3310239711 shutdown_connections 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 --2- 172.21.15.164:0/3310239711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634406d5d0 0x7f6344138290 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 --2- 172.21.15.164:0/3310239711 >> v2:172.21.15.165:3300/0 conn(0x7f634409d680 0x7f634409da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 -- 172.21.15.164:0/3310239711 >> 172.21.15.164:0/3310239711 conn(0x7f63440a70e0 msgr2=0x7f63440a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 -- 172.21.15.164:0/3310239711 shutdown_connections 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 -- 172.21.15.164:0/3310239711 wait complete. 2024-09-10T16:14:54.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 10 monclient: build_initial_monmap 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 10 monclient: monmap: 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.548+0000 7f63493a1640 1 Processor -- start 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 -- start start 2024-09-10T16:14:54.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient: init 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 0x7f634409a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-c con 0x7f634406d1f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f634406d5d0 0x7f634409a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-b con 0x7f634406d5d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): picked mon.noname-a con 0x7f634409d680 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6344138950 con 0x7f634409d680 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63441387d0 con 0x7f634406d5d0 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f634413fec0 con 0x7f634406d1f0 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f63493a1640 10 monclient(hunting): authenticate will time out at 1273.036495s 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 0x7f634409a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35876/0 (socket says 172.21.15.164:35876) 2024-09-10T16:14:54.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 -- 172.21.15.164:0/227918271 learned_addr learned my addr 172.21.15.164:0/227918271 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): get_auth_request con 0x7f634409d680 auth_method 0 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): get_auth_request con 0x7f634406d1f0 auth_method 0 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f634406d5d0 0x7f634409a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): get_auth_request con 0x7f634406d5d0 auth_method 0 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f633bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient(hunting): handle_auth_done global_id 74313 payload 824 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 -- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 msgr2=0x7f634409a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 0x7f634409a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 -- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f634406d5d0 msgr2=0x7f634409a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f634406d5d0 0x7f634409a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient: _finish_hunting 0 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 monclient: found mon.noname-a 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6348ba0640 1 -- 172.21.15.164:0/227918271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63440a8570 con 0x7f634409d680 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.549+0000 7f6343fff640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 0x7f634409a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f6348ba0640 1 -- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 msgr2=0x7f634412f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f6348ba0640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f633c0093e0 tx=0x7f633c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c036050 con 0x7f634409d680 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: dump: 2024-09-10T16:14:54.568 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: _finish_auth 0 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: _check_auth_tickets 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f633c042070 con 0x7f634409d680 2024-09-10T16:14:54.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 5 monclient: authenticate success, global_id 74313 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 10 monclient: _renew_subs 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6344133c80 con 0x7f634409d680 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633c031d90 con 0x7f634409d680 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63417fa640 10 monclient: dump: 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f6342ffd640 4 set_mon_vals no callback set 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 10 monclient: _renew_subs 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.552+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6344132a00 con 0x7f634409d680 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f633c047460 con 0x7f634409d680 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63493a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63493a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63493a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f634409da60 con 0x7f634409d680 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63417fa640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 0x7f6324040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7f633c073ee0 con 0x7f634409d680 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f6343fff640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 0x7f6324040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.553+0000 7f6343fff640 10 monclient: get_auth_request con 0x7f632403e240 auth_method 0 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.556+0000 7f6343fff640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 0x7f6324040700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f633400af30 tx=0x7f6334008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.558+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f63440a8570 con 0x7f634409d680 2024-09-10T16:14:54.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.558+0000 7f63417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:54.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.558+0000 7f63417fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024671699}) 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165429000}) 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2024671699}) 2024-09-10T16:14:54.573 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.571+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3429957294}) 2024-09-10T16:14:54.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3453363937,mdsmap=204415920}) 2024-09-10T16:14:54.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=4015531273,mdsmap=931935893}) 2024-09-10T16:14:54.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=3453363937,mdsmap=204415920}) 2024-09-10T16:14:54.597 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=728387159,mdsmap=2674069449}) 2024-09-10T16:14:54.597 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.597 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {mdsmap=1228700967,mgr=3377243026,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1474175699,mon_config_key=605496218,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1940671545,osd_snap=3782908443}) 2024-09-10T16:14:54.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=53} crc {mdsmap=1228700967,mgr=2354521040,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2591046986,mon_config_key=555717673,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1940671545,osd_snap=984655497}) 2024-09-10T16:14:54.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {mdsmap=1228700967,mgr=3377243026,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1474175699,mon_config_key=605496218,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1940671545,osd_snap=3782908443}) 2024-09-10T16:14:54.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgrstat=9,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=53} crc {mdsmap=1228700967,mgr=1154067016,mgr_command_descs=3192186153,mgrstat=2591046986,mon_config_key=665552247,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3014334576,osd_pg_creating=1940671545,osd_snap=3862072510}) 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=2969701442,osdmap=3536477841}) 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=2597111416,osdmap=232028593}) 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=5,osdmap=95} crc {osd_snap=2969701442,osdmap=3536477841}) 2024-09-10T16:14:54.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=3175106087,osdmap=3492166065}) 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=2058350917}) 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=76} crc {osdmap=1295664898}) 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=2058350917}) 2024-09-10T16:14:54.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:14:54.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=74} crc {osdmap=113462042}) 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63493a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63493a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63493a1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6344132e10 con 0x7f634409d680 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63417fa640 1 -- 172.21.15.164:0/227918271 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f633c0728e0 con 0x7f634409d680 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:54.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.803+0000 7f63417fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:54.806 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:54.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 msgr2=0x7f6324040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 0x7f6324040700 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f633400af30 tx=0x7f6334008040 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 10 monclient: shutdown 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 msgr2=0x7f634412f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f633c0093e0 tx=0x7f633c004a40 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 shutdown_connections 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f632403e240 0x7f6324040700 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f634409d680 0x7f634412f6a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f634406d5d0 0x7f634409a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 --2- 172.21.15.164:0/227918271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f634406d1f0 0x7f634409a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 >> 172.21.15.164:0/227918271 conn(0x7f63440a70e0 msgr2=0x7f634409c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:54.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 shutdown_connections 2024-09-10T16:14:54.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.806+0000 7f63493a1640 1 -- 172.21.15.164:0/227918271 wait complete. 2024-09-10T16:14:54.818 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:14:54.818 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:14:54.818 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:14:54.819 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 10 monclient: get_monmap_and_config 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 10 monclient: build_initial_monmap 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 10 monclient: monmap: 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:54.917233+0000 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:54.917233+0000 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 1 Processor -- start 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 1 -- start start 2024-09-10T16:14:54.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.916+0000 7fcb947f3640 10 monclient: init 2024-09-10T16:14:54.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:54.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:14:54.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb8c0a3b60 0x7fcb8c137c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8c0a3b60 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 0x7fcb8c0a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8c0a0530 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb8c0a5860 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb8c09cfe0 con 0x7fcb8c0a5860 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fcb8c0a3f40 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcb8c074520 con 0x7fcb8c0a0530 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb947f3640 10 monclient(hunting): authenticate will time out at 1273.404625s 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb8c0a3b60 0x7fcb8c137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb8c0a3b60 0x7fcb8c137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35886/0 (socket says 172.21.15.164:35886) 2024-09-10T16:14:54.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 1 -- 172.21.15.164:0/4033598349 learned_addr learned my addr 172.21.15.164:0/4033598349 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): get_auth_request con 0x7fcb8c0a3b60 auth_method 0 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb91d8e640 1 --2- 172.21.15.164:0/4033598349 >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 0x7fcb8c0a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb91d8e640 10 monclient(hunting): get_auth_request con 0x7fcb8c0a0530 auth_method 0 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb91d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb91d8e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.917+0000 7fcb91d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 10 monclient(hunting): handle_auth_done global_id 74316 payload 274 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 1 -- 172.21.15.164:0/4033598349 >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 msgr2=0x7fcb8c0a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 1 --2- 172.21.15.164:0/4033598349 >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 0x7fcb8c0a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 1 -- 172.21.15.164:0/4033598349 >> v1:172.21.15.165:6789/0 conn(0x7fcb8c0a5860 legacy=0x7fcb8c0a5cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 10 monclient: _finish_hunting 0 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 1 monclient: found mon.noname-a 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb9258f640 1 -- 172.21.15.164:0/4033598349 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8c073930 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.918+0000 7fcb91d8e640 1 --2- 172.21.15.164:0/4033598349 >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 0x7fcb8c0a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9258f640 1 -- 172.21.15.164:0/4033598349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a3b60 msgr2=0x7fcb8c137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9258f640 1 --2- 172.21.15.164:0/4033598349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a3b60 0x7fcb8c137c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c074450 tx=0x7fcb7c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf04f98e236105a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 1 -- 172.21.15.164:0/4033598349 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c037070 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:54.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: dump: 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.933 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: _finish_auth 0 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: _check_auth_tickets 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 1 -- 172.21.15.164:0/4033598349 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb7c032640 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb947f3640 5 monclient: authenticate success, global_id 74316 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb947f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:54.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 1 -- 172.21.15.164:0/4033598349 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c032a10 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb9158d640 10 monclient: dump: 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.935 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb947f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.920+0000 7fcb947f3640 4 set_mon_vals no callback set 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 10 monclient: shutdown 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 -- 172.21.15.164:0/4033598349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a3b60 msgr2=0x7fcb8c137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 --2- 172.21.15.164:0/4033598349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a3b60 0x7fcb8c137c80 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c074450 tx=0x7fcb7c02a0b0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 -- 172.21.15.164:0/4033598349 shutdown_connections 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 --2- 172.21.15.164:0/4033598349 >> v2:172.21.15.165:3300/0 conn(0x7fcb8c0a0530 0x7fcb8c0a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 --2- 172.21.15.164:0/4033598349 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a3b60 0x7fcb8c137c80 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 -- 172.21.15.164:0/4033598349 >> 172.21.15.164:0/4033598349 conn(0x7fcb8c0a6f80 msgr2=0x7fcb8c0a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 -- 172.21.15.164:0/4033598349 shutdown_connections 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.921+0000 7fcb947f3640 1 -- 172.21.15.164:0/4033598349 wait complete. 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient: build_initial_monmap 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient: monmap: 2024-09-10T16:14:54.937 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 Processor -- start 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 -- start start 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient: init 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:54.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb8c0a0530 0x7fcb8c1314c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-b con 0x7fcb8c0a0530 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c0a3b60 0x7fcb8c131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-c con 0x7fcb8c0a3b60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): picked mon.noname-a con 0x7fcb8c0a5860 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c0a6400 con 0x7fcb8c0a5860 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c0a6580 con 0x7fcb8c0a0530 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcb8c13ff40 con 0x7fcb8c0a3b60 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.922+0000 7fcb947f3640 10 monclient(hunting): authenticate will time out at 1273.409918s 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c0a3b60 0x7fcb8c131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35890/0 (socket says 172.21.15.164:35890) 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 learned_addr learned my addr 172.21.15.164:0/73519566 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): get_auth_request con 0x7fcb8c0a5860 auth_method 0 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): get_auth_request con 0x7fcb8c0a3b60 auth_method 0 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb8c0a0530 0x7fcb8c1314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 10 monclient(hunting): get_auth_request con 0x7fcb8c0a0530 auth_method 0 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient(hunting): handle_auth_done global_id 74319 payload 824 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c0a3b60 msgr2=0x7fcb8c131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c0a3b60 0x7fcb8c131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb8c0a0530 msgr2=0x7fcb8c1314c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb8c0a0530 0x7fcb8c1314c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient: _finish_hunting 0 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 monclient: found mon.noname-a 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb8c13ebc0 con 0x7fcb8c0a5860 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb8c0a0530 0x7fcb8c1314c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.923+0000 7fcb91d8e640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb8c0a3b60 0x7fcb8c131a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 msgr2=0x7fcb8c1cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb92d90640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c410 tx=0x7fcb8400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb840119a0 con 0x7fcb8c0a5860 2024-09-10T16:14:54.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: dump: 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: _finish_auth 0 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: _check_auth_tickets 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 5 monclient: authenticate success, global_id 74319 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb84017070 con 0x7fcb8c0a5860 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 10 monclient: _renew_subs 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb8c0a14c0 con 0x7fcb8c0a5860 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb84011d80 con 0x7fcb8c0a5860 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb8affd640 10 monclient: dump: 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 10 monclient: _renew_subs 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb8c13c860 con 0x7fcb8c0a5860 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.925+0000 7fcb90d8c640 4 set_mon_vals no callback set 2024-09-10T16:14:54.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb947f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb947f3640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 0x7fcb8c1cd970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c1383d0 con 0x7fcb8c1cd590 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 0x7fcb8c1cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcb8401c460 con 0x7fcb8c0a5860 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient: get_auth_request con 0x7fcb8c1cd590 auth_method 0 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb8affd640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb7803e240 0x7fcb78040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 10 monclient(hunting): handle_auth_done global_id 74322 payload 824 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb9258f640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb8c1cdeb0 con 0x7fcb8c1cd590 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.926+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7fcb840528c0 con 0x7fcb8c0a5860 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.927+0000 7fcb91d8e640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb7803e240 0x7fcb78040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.927+0000 7fcb91d8e640 10 monclient: get_auth_request con 0x7fcb7803e240 auth_method 0 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb9258f640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 msgr2=0x7fcb8c1cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 0x7fcb8c1cd970 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c034000 tx=0x7fcb7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c042070 con 0x7fcb8c1cd590 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 10 monclient: dump: 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.947 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb7c03e660 con 0x7fcb8c1cd590 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 msgr2=0x7fcb8c1cd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 0x7fcb8c1cd970 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c034000 tx=0x7fcb7c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 msgr2=0x7fcb8c1cd970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:54.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb8affd640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c1cd590 0x7fcb8c1cd970 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c034000 tx=0x7fcb7c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.929+0000 7fcb91d8e640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb7803e240 0x7fcb78040700 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c13ff10 tx=0x7fcb80009870 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb947f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb947f3640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 0x7fcb8c132930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb947f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb8c131f40 con 0x7fcb8c132520 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 0x7fcb8c132930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient: get_auth_request con 0x7fcb8c132520 auth_method 0 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 10 monclient(hunting): handle_auth_done global_id 74325 payload 824 2024-09-10T16:14:54.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.935+0000 7fcb9258f640 1 -- 172.21.15.164:0/73519566 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb8c06cb70 con 0x7fcb8c132520 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb9258f640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 msgr2=0x7fcb8c132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb9258f640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 0x7fcb8c132930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0090b0 tx=0x7fcb7c008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb7c037040 con 0x7fcb8c132520 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 10 monclient: dump: 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:54.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fcb7c003a20 con 0x7fcb8c132520 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 msgr2=0x7fcb8c132930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 0x7fcb8c132930 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcb7c0090b0 tx=0x7fcb7c008030 comp rx=0 tx=0).stop 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 msgr2=0x7fcb8c132930 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb8affd640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c132520 0x7fcb8c132930 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.953 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 122804, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:54.954 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:54.955 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:54.956 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:54.957 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:54.958 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:54.959 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.960 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:54.961 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:54.962 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:54.963 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:54.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:54.966 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:14:54.967 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:54.968 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb7803e240 msgr2=0x7fcb78040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb7803e240 0x7fcb78040700 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fcb8c13ff10 tx=0x7fcb80009870 comp rx=0 tx=0).stop 2024-09-10T16:14:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 10 monclient: shutdown 2024-09-10T16:14:54.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 msgr2=0x7fcb8c1cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb92d90640 1 -- 172.21.15.164:0/73519566 reap_dead start 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcb8400c410 tx=0x7fcb8400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 shutdown_connections 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 --2- 172.21.15.164:0/73519566 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb8c0a5860 0x7fcb8c1cd050 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 >> 172.21.15.164:0/73519566 conn(0x7fcb8c0a6f80 msgr2=0x7fcb8c09f510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 shutdown_connections 2024-09-10T16:14:54.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:54.938+0000 7fcb947f3640 1 -- 172.21.15.164:0/73519566 wait complete. 2024-09-10T16:14:54.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:54.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:14:54.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:14:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.069+0000 7fa811733640 10 monclient: get_monmap_and_config 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.069+0000 7fa811733640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.069+0000 7fa811733640 10 monclient: monmap: 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:55.070393+0000 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:55.070393+0000 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:55.071 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:55.072 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:55.072 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:55.072 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 Processor -- start 2024-09-10T16:14:55.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 -- start start 2024-09-10T16:14:55.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient: init 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa80c09d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa80c09ab50 0x7fa80c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): picked mon.noname-a con 0x7fa80c09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa80c09e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa80c0a0870 con 0x7fa80c09e420 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa80c074c70 con 0x7fa80c09d930 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa80c09d3e0 con 0x7fa80c09ab50 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa811733640 10 monclient(hunting): authenticate will time out at 1273.557792s 2024-09-10T16:14:55.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa80c09ab50 0x7fa80c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa80c09ab50 0x7fa80c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35912/0 (socket says 172.21.15.164:35912) 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 1 -- 172.21.15.164:0/486245394 learned_addr learned my addr 172.21.15.164:0/486245394 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa80c09e420 0x7fa80c13bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57082/0 (socket says 172.21.15.164:57082) 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request con 0x7fa80c09ab50 auth_method 0 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.070+0000 7fa80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80b7fe640 1 -- 172.21.15.164:0/486245394 >> v1:172.21.15.164:6789/0 conn(0x7fa80c09e420 legacy=0x7fa80c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80affd640 1 -- 172.21.15.164:0/486245394 >> v1:172.21.15.165:6789/0 conn(0x7fa80c09d930 legacy=0x7fa80c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_done global_id 74328 payload 274 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 1 -- 172.21.15.164:0/486245394 >> v1:172.21.15.165:6789/0 conn(0x7fa80c09d930 legacy=0x7fa80c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 1 -- 172.21.15.164:0/486245394 >> v1:172.21.15.164:6789/0 conn(0x7fa80c09e420 legacy=0x7fa80c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.073+0000 7fa80a7fc640 1 -- 172.21.15.164:0/486245394 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c09ecb0 con 0x7fa80c09ab50 2024-09-10T16:14:55.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa80a7fc640 1 -- 172.21.15.164:0/486245394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 msgr2=0x7fa80c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/486245394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1397c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa80000ab10 tx=0x7fa80002b6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a637e544a6a0b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 1 -- 172.21.15.164:0/486245394 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80000b040 con 0x7fa80c09ab50 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: dump: 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 1 -- 172.21.15.164:0/486245394 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa800030a00 con 0x7fa80c09ab50 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 5 monclient: authenticate success, global_id 74328 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 1 -- 172.21.15.164:0/486245394 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800030dd0 con 0x7fa80c09ab50 2024-09-10T16:14:55.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 4 set_mon_vals no callback set 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa809ffb640 10 monclient: dump: 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 10 monclient: shutdown 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 -- 172.21.15.164:0/486245394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 msgr2=0x7fa80c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 --2- 172.21.15.164:0/486245394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1397c0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fa80000ab10 tx=0x7fa80002b6a0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 -- 172.21.15.164:0/486245394 shutdown_connections 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 --2- 172.21.15.164:0/486245394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1397c0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 -- 172.21.15.164:0/486245394 >> 172.21.15.164:0/486245394 conn(0x7fa80c0a3110 msgr2=0x7fa80c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 -- 172.21.15.164:0/486245394 shutdown_connections 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.076+0000 7fa811733640 1 -- 172.21.15.164:0/486245394 wait complete. 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.077+0000 7fa811733640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.077+0000 7fa811733640 10 monclient: monmap: 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.077+0000 7fa811733640 1 Processor -- start 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.077+0000 7fa811733640 1 -- start start 2024-09-10T16:14:55.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.077+0000 7fa811733640 10 monclient: init 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): picked mon.noname-a con 0x7fa80c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa80c09d930 0x7fa80c1c91a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): picked mon.noname-b con 0x7fa80c09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c09e420 0x7fa80c12c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): picked mon.noname-c con 0x7fa80c09e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c0a2940 con 0x7fa80c09ab50 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa80c09ef30 con 0x7fa80c09d930 2024-09-10T16:14:55.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c0a2ac0 con 0x7fa80c09e420 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa811733640 10 monclient(hunting): authenticate will time out at 1273.565474s 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35922/0 (socket says 172.21.15.164:35922) 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c09e420 0x7fa80c12c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 -- 172.21.15.164:0/3814686208 learned_addr learned my addr 172.21.15.164:0/3814686208 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): get_auth_request con 0x7fa80c09ab50 auth_method 0 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): get_auth_request con 0x7fa80c09e420 auth_method 0 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa80c09d930 0x7fa80c1c91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request con 0x7fa80c09d930 auth_method 0 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient(hunting): handle_auth_done global_id 74334 payload 824 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c09e420 msgr2=0x7fa80c12c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c09e420 0x7fa80c12c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa80c09d930 msgr2=0x7fa80c1c91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa80c09d930 0x7fa80c1c91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80affd640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa80c1305a0 con 0x7fa80c09ab50 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.078+0000 7fa80b7fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c09e420 0x7fa80c12c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa80affd640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 msgr2=0x7fa80c1c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa80affd640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00c490 tx=0x7fa7fc00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc017070 con 0x7fa80c09ab50 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: dump: 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.097 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7fc0057c0 con 0x7fa80c09ab50 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 5 monclient: authenticate success, global_id 74334 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 10 monclient: _renew_subs 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa80c09cab0 con 0x7fa80c09ab50 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc005bb0 con 0x7fa80c09ab50 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa7f37fe640 10 monclient: dump: 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 10 monclient: _renew_subs 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa80c135230 con 0x7fa80c09ab50 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.081+0000 7fa8097fa640 4 set_mon_vals no callback set 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa811733640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa811733640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 0x7fa80c02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c12fe70 con 0x7fa80c02d440 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 0x7fa80c02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa7fc01a070 con 0x7fa80c09ab50 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient: get_auth_request con 0x7fa80c02d440 auth_method 0 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_done global_id 5189 payload 824 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa7f37fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7f803e240 0x7fa7f8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80a7fc640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa80c09dd90 con 0x7fa80c02d440 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7fa7fc051fc0 con 0x7fa80c09ab50 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80b7fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7f803e240 0x7fa7f8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.082+0000 7fa80b7fe640 10 monclient: get_auth_request con 0x7fa7f803e240 auth_method 0 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa80a7fc640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 msgr2=0x7fa80c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 0x7fa80c02d820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa800035000 tx=0x7fa8000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa800032df0 con 0x7fa80c02d440 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa7f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.084+0000 7fa7f37fe640 10 monclient: dump: 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa80000a630 con 0x7fa80c02d440 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 msgr2=0x7fa80c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 0x7fa80c02d820 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa800035000 tx=0x7fa8000046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 msgr2=0x7fa80c02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa7f37fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02d440 0x7fa80c02d820 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.085+0000 7fa80b7fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7f803e240 0x7fa7f8040700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa804002230 tx=0x7fa8040018b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa811733640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:55.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa811733640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 0x7fa80c02e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa811733640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa80c06dac0 con 0x7fa80c02dd60 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 0x7fa80c02e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient: get_auth_request con 0x7fa80c02dd60 auth_method 0 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 10 monclient(hunting): handle_auth_done global_id 5192 payload 824 2024-09-10T16:14:55.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.091+0000 7fa80a7fc640 1 -- 172.21.15.164:0/3814686208 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa80c072f00 con 0x7fa80c02dd60 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa80a7fc640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 msgr2=0x7fa80c02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa80a7fc640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 0x7fa80c02e170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa80c12fff0 tx=0x7fa800031000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa80003c620 con 0x7fa80c02dd60 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 10 monclient: dump: 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fa80003c620 con 0x7fa80c02dd60 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 msgr2=0x7fa80c02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.093+0000 7fa7f37fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 0x7fa80c02e170 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa80c12fff0 tx=0x7fa800031000 comp rx=0 tx=0).stop 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa7f37fe640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 msgr2=0x7fa80c02e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa7f37fe640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa80c02dd60 0x7fa80c02e170 secure :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa80c12fff0 tx=0x7fa800031000 comp rx=0 tx=0).stop 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 537265, 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:55.112 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.114 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:55.115 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.117 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:55.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.120 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:55.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:55.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:55.125 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7f803e240 msgr2=0x7fa7f8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7f803e240 0x7fa7f8040700 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fa804002230 tx=0x7fa8040018b0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 10 monclient: shutdown 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 msgr2=0x7fa80c1c8c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00c490 tx=0x7fa7fc00c950 comp rx=0 tx=0).stop 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa80b7fe640 1 -- 172.21.15.164:0/3814686208 reap_dead start 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 shutdown_connections 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 --2- 172.21.15.164:0/3814686208 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa80c09ab50 0x7fa80c1c8c60 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 >> 172.21.15.164:0/3814686208 conn(0x7fa80c0a3110 msgr2=0x7fa80c099460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 shutdown_connections 2024-09-10T16:14:55.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.094+0000 7fa811733640 1 -- 172.21.15.164:0/3814686208 wait complete. 2024-09-10T16:14:55.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:55.128 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:14:55.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.224+0000 7f6051d0e640 10 monclient: get_monmap_and_config 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.224+0000 7f6051d0e640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.224+0000 7f6051d0e640 10 monclient: monmap: 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:55.225514+0000 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:55.225514+0000 2024-09-10T16:14:55.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:55.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 Processor -- start 2024-09-10T16:14:55.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 -- start start 2024-09-10T16:14:55.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient: init 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f604c09c180 0x7f604c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f604c09c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f604c09d160 0x7f604c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f604c09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c13b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f604c09f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f604c09a430 con 0x7f604c09c180 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f604c0a24e0 con 0x7f604c09d160 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f604c0a2bf0 con 0x7f604c09f4c0 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.225+0000 7f6051d0e640 10 monclient(hunting): authenticate will time out at 1273.712940s 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f604c09c180 0x7f604c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f604c09c180 0x7f604c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35926/0 (socket says 172.21.15.164:35926) 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 -- 172.21.15.164:0/3909515067 learned_addr learned my addr 172.21.15.164:0/3909515067 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): get_auth_request con 0x7f604c09c180 auth_method 0 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c13b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): get_auth_request con 0x7f604c09f4c0 auth_method 0 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604affd640 1 --2- 172.21.15.164:0/3909515067 >> v2:172.21.15.165:3300/0 conn(0x7f604c09d160 0x7f604c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604affd640 10 monclient(hunting): get_auth_request con 0x7f604c09d160 auth_method 0 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient(hunting): handle_auth_done global_id 74337 payload 274 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 -- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 msgr2=0x7f604c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c13b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 -- 172.21.15.164:0/3909515067 >> v2:172.21.15.165:3300/0 conn(0x7f604c09d160 msgr2=0x7f604c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 --2- 172.21.15.164:0/3909515067 >> v2:172.21.15.165:3300/0 conn(0x7f604c09d160 0x7f604c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.226+0000 7f604b7fe640 1 -- 172.21.15.164:0/3909515067 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604c09c560 con 0x7f604c09c180 2024-09-10T16:14:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.229+0000 7f604b7fe640 1 -- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09c180 msgr2=0x7f604c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.229+0000 7f604b7fe640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09c180 0x7f604c09ea40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79ac5d63df9eeff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 1 -- 172.21.15.164:0/3909515067 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c037070 con 0x7f604c09c180 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: dump: 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.251 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 5 monclient: authenticate success, global_id 74337 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 1 -- 172.21.15.164:0/3909515067 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f603c032640 con 0x7f604c09c180 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 4 set_mon_vals no callback set 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 1 -- 172.21.15.164:0/3909515067 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c032a10 con 0x7f604c09c180 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f604a7fc640 10 monclient: dump: 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 10 monclient: shutdown 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 -- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09c180 msgr2=0x7f604c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09c180 0x7f604c09ea40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f603c009870 tx=0x7f603c02a100 comp rx=0 tx=0).stop 2024-09-10T16:14:55.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 -- 172.21.15.164:0/3909515067 shutdown_connections 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c13b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 --2- 172.21.15.164:0/3909515067 >> v2:172.21.15.165:3300/0 conn(0x7f604c09d160 0x7f604c09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 --2- 172.21.15.164:0/3909515067 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09c180 0x7f604c09ea40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 -- 172.21.15.164:0/3909515067 >> 172.21.15.164:0/3909515067 conn(0x7f604c0a3110 msgr2=0x7f604c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 -- 172.21.15.164:0/3909515067 shutdown_connections 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.231+0000 7f6051d0e640 1 -- 172.21.15.164:0/3909515067 wait complete. 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.232+0000 7f6051d0e640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.232+0000 7f6051d0e640 10 monclient: monmap: 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:55.255 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.232+0000 7f6051d0e640 1 Processor -- start 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.232+0000 7f6051d0e640 1 -- start start 2024-09-10T16:14:55.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.232+0000 7f6051d0e640 10 monclient: init 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c09c180 0x7f604c073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-b con 0x7f604c09c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 0x7f604c0965b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-a con 0x7f604c09d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c1ccfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): picked mon.noname-c con 0x7f604c09f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f604c074f90 con 0x7f604c09d160 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f604c074e10 con 0x7f604c09c180 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f604c13bb00 con 0x7f604c09f4c0 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f6051d0e640 10 monclient(hunting): authenticate will time out at 1273.720386s 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 0x7f604c0965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 0x7f604c0965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35942/0 (socket says 172.21.15.164:35942) 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 -- 172.21.15.164:0/1679395223 learned_addr learned my addr 172.21.15.164:0/1679395223 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): get_auth_request con 0x7f604c09d160 auth_method 0 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c1ccfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): get_auth_request con 0x7f604c09f4c0 auth_method 0 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c09c180 0x7f604c073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): get_auth_request con 0x7f604c09c180 auth_method 0 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient(hunting): handle_auth_done global_id 74340 payload 824 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 msgr2=0x7f604c1ccfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c1ccfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c09c180 msgr2=0x7f604c073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c09c180 0x7f604c073a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604affd640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f604c093080 con 0x7f604c09d160 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.233+0000 7f604bfff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f604c09f4c0 0x7f604c1ccfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f604affd640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 msgr2=0x7f604c0965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f604affd640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 0x7f604c0965b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f604000d7d0 tx=0x7f604000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6040004020 con 0x7f604c09d160 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: dump: 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60400041e0 con 0x7f604c09d160 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 5 monclient: authenticate success, global_id 74340 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 10 monclient: _renew_subs 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f604c09d620 con 0x7f604c09d160 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60400043d0 con 0x7f604c09d160 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6033fff640 10 monclient: dump: 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 10 monclient: _renew_subs 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f60497fa640 4 set_mon_vals no callback set 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f604c092da0 con 0x7f604c09d160 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.236+0000 7f6051d0e640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 0x7f604c1cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f604c09d930 con 0x7f604c13c140 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f604001b020 con 0x7f604c09d160 2024-09-10T16:14:55.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 0x7f604c1cd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient: get_auth_request con 0x7f604c13c140 auth_method 0 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f6033fff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f603803e240 0x7f6038040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7f6040052b40 con 0x7f604c09d160 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 10 monclient(hunting): handle_auth_done global_id 94126 payload 824 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604b7fe640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f604c0008d0 con 0x7f604c13c140 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604bfff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f603803e240 0x7f6038040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.237+0000 7f604bfff640 10 monclient: get_auth_request con 0x7f603803e240 auth_method 0 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.239+0000 7f604b7fe640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 msgr2=0x7f604c1cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.239+0000 7f604b7fe640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 0x7f604c1cd500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f603c034000 tx=0x7f603c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c043070 con 0x7f604c13c140 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 10 monclient: dump: 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.267 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f604bfff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f603803e240 0x7f6038040700 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f604400e740 tx=0x7f604400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f603c009490 con 0x7f604c13c140 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 msgr2=0x7f604c1cd500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 0x7f604c1cd500 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f603c034000 tx=0x7f603c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 msgr2=0x7f604c1cd500 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.240+0000 7f6033fff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c13c140 0x7f604c1cd500 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f603c034000 tx=0x7f603c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f6051d0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f6051d0e640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 0x7f604c1cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f6051d0e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f604c1d8350 con 0x7f604c096af0 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 0x7f604c1cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient: get_auth_request con 0x7f604c096af0 auth_method 0 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.246+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.247+0000 7f604b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.247+0000 7f604b7fe640 10 monclient(hunting): handle_auth_done global_id 94129 payload 824 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.247+0000 7f604b7fe640 1 -- 172.21.15.164:0/1679395223 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f604c1d8500 con 0x7f604c096af0 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f604b7fe640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 msgr2=0x7f604c1cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f604b7fe640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 0x7f604c1cda40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f603c042000 tx=0x7f603c03c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f603c037040 con 0x7f604c096af0 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f6033fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f6033fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.249+0000 7f6033fff640 10 monclient: dump: 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.270 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f603c048b70 con 0x7f604c096af0 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 msgr2=0x7f604c1cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 0x7f604c1cda40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f603c042000 tx=0x7f603c03c000 comp rx=0 tx=0).stop 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 msgr2=0x7f604c1cda40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6033fff640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f604c096af0 0x7f604c1cda40 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f603c042000 tx=0x7f603c03c000 comp rx=0 tx=0).stop 2024-09-10T16:14:55.271 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 80, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12749, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:14:55.272 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.273 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.274 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:14:55.275 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:14:55.276 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:14:55.277 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.278 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.279 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.280 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:14:55.281 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:14:55.282 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.283 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:14:55.284 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:14:55.285 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f603803e240 msgr2=0x7f6038040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f603803e240 0x7f6038040700 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f604400e740 tx=0x7f604400b040 comp rx=0 tx=0).stop 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 10 monclient: shutdown 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 msgr2=0x7f604c0965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 --2- 172.21.15.164:0/1679395223 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f604c09d160 0x7f604c0965b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f604000d7d0 tx=0x7f604000dc90 comp rx=0 tx=0).stop 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f604bfff640 1 -- 172.21.15.164:0/1679395223 reap_dead start 2024-09-10T16:14:55.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f604bfff640 1 -- 172.21.15.164:0/1679395223 reap_dead start 2024-09-10T16:14:55.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 shutdown_connections 2024-09-10T16:14:55.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 >> 172.21.15.164:0/1679395223 conn(0x7f604c0a3110 msgr2=0x7f604c09e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 shutdown_connections 2024-09-10T16:14:55.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.250+0000 7f6051d0e640 1 -- 172.21.15.164:0/1679395223 wait complete. 2024-09-10T16:14:55.287 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:14:55.287 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:14:55.287 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:14:55.287 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:14:55.288 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:14:55.288 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:14:55.288 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:14:55.302 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:14:55.302 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:14:55.303 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:55.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 10 monclient: get_monmap_and_config 2024-09-10T16:14:55.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 10 monclient: monmap: 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:55.439171+0000 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:55.439171+0000 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:55.440 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 1 Processor -- start 2024-09-10T16:14:55.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 1 -- start start 2024-09-10T16:14:55.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.438+0000 7fecd1164640 10 monclient: init 2024-09-10T16:14:55.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feccc06cb70 0x7feccc134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-a con 0x7feccc06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feccc0a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feccc0a4ab0 0x7feccc1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-c con 0x7feccc0a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feccc09ce50 con 0x7feccc0a46a0 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7feccc06db60 con 0x7feccc06cb70 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7feccc06cf50 con 0x7feccc0a4ab0 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7fecd1164640 10 monclient(hunting): authenticate will time out at 1273.926471s 2024-09-10T16:14:55.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feccc06cb70 0x7feccc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feccc06cb70 0x7feccc134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35970/0 (socket says 172.21.15.164:35970) 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 -- 172.21.15.164:0/1328236008 learned_addr learned my addr 172.21.15.164:0/1328236008 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): get_auth_request con 0x7feccc06cb70 auth_method 0 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient(hunting): handle_auth_done global_id 74343 payload 274 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 -- 172.21.15.164:0/1328236008 >> v2:172.21.15.165:3300/0 conn(0x7feccc0a4ab0 msgr2=0x7feccc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 --2- 172.21.15.164:0/1328236008 >> v2:172.21.15.165:3300/0 conn(0x7feccc0a4ab0 0x7feccc1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 -- 172.21.15.164:0/1328236008 >> v1:172.21.15.165:6789/0 conn(0x7feccc0a46a0 legacy=0x7feccc137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.439+0000 7feccad9d640 1 -- 172.21.15.164:0/1328236008 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc09cb70 con 0x7feccc06cb70 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7feccad9d640 1 -- 172.21.15.164:0/1328236008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc06cb70 msgr2=0x7feccc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7feccad9d640 1 --2- 172.21.15.164:0/1328236008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc06cb70 0x7feccc134f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0098a0 tx=0x7fecbc02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc9114eedfa9bc69 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 1 -- 172.21.15.164:0/1328236008 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc037070 con 0x7feccc06cb70 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: dump: 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.451 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 1 -- 172.21.15.164:0/1328236008 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecbc032640 con 0x7feccc06cb70 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 1 -- 172.21.15.164:0/1328236008 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc032a10 con 0x7feccc06cb70 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecd1164640 5 monclient: authenticate success, global_id 74343 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecd1164640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecd1164640 4 set_mon_vals no callback set 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.442+0000 7fecc9d9b640 10 monclient: dump: 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.453 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 10 monclient: shutdown 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 -- 172.21.15.164:0/1328236008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc06cb70 msgr2=0x7feccc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 --2- 172.21.15.164:0/1328236008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc06cb70 0x7feccc134f70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0098a0 tx=0x7fecbc02a110 comp rx=0 tx=0).stop 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 -- 172.21.15.164:0/1328236008 shutdown_connections 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 --2- 172.21.15.164:0/1328236008 >> v2:172.21.15.165:3300/0 conn(0x7feccc0a4ab0 0x7feccc1383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 --2- 172.21.15.164:0/1328236008 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc06cb70 0x7feccc134f70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 -- 172.21.15.164:0/1328236008 >> 172.21.15.164:0/1328236008 conn(0x7feccc0aa900 msgr2=0x7feccc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 -- 172.21.15.164:0/1328236008 shutdown_connections 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.443+0000 7fecd1164640 1 -- 172.21.15.164:0/1328236008 wait complete. 2024-09-10T16:14:55.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 10 monclient: build_initial_monmap 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 10 monclient: monmap: 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:55.455 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 1 Processor -- start 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 1 -- start start 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 10 monclient: init 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.444+0000 7fecd1164640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feccc06cb70 0x7feccc09eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-b con 0x7feccc06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 0x7feccc09f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-c con 0x7feccc0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): picked mon.noname-a con 0x7feccc0a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feccc138a70 con 0x7feccc0a4ab0 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feccc1388f0 con 0x7feccc06cb70 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feccc140040 con 0x7feccc0a46a0 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecd1164640 10 monclient(hunting): authenticate will time out at 1273.932079s 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecca59c640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 msgr2=0x7feccc09f000 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7fecca59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 0x7feccc09f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35972/0 (socket says 172.21.15.164:35972) 2024-09-10T16:14:55.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 -- 172.21.15.164:0/1476510578 learned_addr learned my addr 172.21.15.164:0/1476510578 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): get_auth_request con 0x7feccc0a4ab0 auth_method 0 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feccc06cb70 0x7feccc09eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): get_auth_request con 0x7feccc06cb70 auth_method 0 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:55.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient(hunting): handle_auth_done global_id 74346 payload 824 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 -- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 msgr2=0x7feccc09f000 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 0x7feccc09f000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 -- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feccc06cb70 msgr2=0x7feccc09eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feccc06cb70 0x7feccc09eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient: _finish_hunting 0 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 monclient: found mon.noname-a 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.445+0000 7feccb59e640 1 -- 172.21.15.164:0/1476510578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feccc0a11e0 con 0x7feccc0a4ab0 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7feccb59e640 1 -- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 msgr2=0x7feccc09f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:55.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7feccb59e640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecc400b970 tx=0x7fecc400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4019070 con 0x7feccc0a4ab0 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: dump: 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.460 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: _finish_auth 0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 5 monclient: authenticate success, global_id 74346 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecc400cd70 con 0x7feccc0a4ab0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 10 monclient: _renew_subs 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feccc06e720 con 0x7feccc0a4ab0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecc4014680 con 0x7feccc0a4ab0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:55.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc37fe640 10 monclient: dump: 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 10 monclient: _renew_subs 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecc959a640 4 set_mon_vals no callback set 2024-09-10T16:14:55.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.447+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feccc0a2570 con 0x7feccc0a4ab0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.448+0000 7fecd1164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.448+0000 7fecd1164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.448+0000 7fecd1164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.448+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fecc401c070 con 0x7feccc0a4ab0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.448+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feccc0a9f80 con 0x7feccc0a4ab0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.449+0000 7fecc37fe640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 0x7fecac0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.449+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7fecc4052a50 con 0x7feccc0a4ab0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.449+0000 7feccad9d640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 0x7fecac0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:55.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.449+0000 7feccad9d640 10 monclient: get_auth_request con 0x7fecac03e1f0 auth_method 0 2024-09-10T16:14:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.452+0000 7feccad9d640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 0x7fecac0406b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feccc077610 tx=0x7fecbc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.453+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7feccc0a9f80 con 0x7feccc0a4ab0 2024-09-10T16:14:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.453+0000 7fecc37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.453+0000 7fecc37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:55.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.695+0000 7fecd1164640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:55.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.695+0000 7fecd1164640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.695+0000 7fecd1164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.695+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feccc0770f0 con 0x7feccc0a4ab0 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.696+0000 7fecc37fe640 1 -- 172.21.15.164:0/1476510578 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fecc4051450 con 0x7feccc0a4ab0 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.696+0000 7fecc37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.696+0000 7fecc37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:55.698 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:55.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 msgr2=0x7fecac0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 0x7fecac0406b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7feccc077610 tx=0x7fecbc02e000 comp rx=0 tx=0).stop 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 10 monclient: shutdown 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 msgr2=0x7feccc09f540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fecc400b970 tx=0x7fecc400be30 comp rx=0 tx=0).stop 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 shutdown_connections 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecac03e1f0 0x7fecac0406b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feccc0a4ab0 0x7feccc09f540 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feccc0a46a0 0x7feccc09f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 --2- 172.21.15.164:0/1476510578 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feccc06cb70 0x7feccc09eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:55.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 >> 172.21.15.164:0/1476510578 conn(0x7feccc0aa900 msgr2=0x7feccc0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:55.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 shutdown_connections 2024-09-10T16:14:55.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:55.698+0000 7fecd1164640 1 -- 172.21.15.164:0/1476510578 wait complete. 2024-09-10T16:14:58.712 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.818+0000 7fce5d447640 10 monclient: get_monmap_and_config 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.818+0000 7fce5d447640 10 monclient: build_initial_monmap 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.818+0000 7fce5d447640 10 monclient: monmap: 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:14:58.820698+0000 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:14:58.820698+0000 2024-09-10T16:14:58.821 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:14:58.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 1 Processor -- start 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 1 -- start start 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 10 monclient: init 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.819+0000 7fce5d447640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fce5809d930 0x7fce5809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-c con 0x7fce5809d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:14:58.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce5809ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:14:58.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fce5809e250 0x7fce58134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-a con 0x7fce5809e250 addr v2:172.21.15.164:3300/0 2024-09-10T16:14:58.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce580a0870 con 0x7fce5809ab50 2024-09-10T16:14:58.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fce5809d3e0 con 0x7fce5809e250 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fce58074c70 con 0x7fce5809d930 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce5d447640 10 monclient(hunting): authenticate will time out at 1277.308064s 2024-09-10T16:14:58.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fce5809e250 0x7fce58134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fce5809e250 0x7fce58134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35976/0 (socket says 172.21.15.164:35976) 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 -- 172.21.15.164:0/805701804 learned_addr learned my addr 172.21.15.164:0/805701804 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): get_auth_request con 0x7fce5809e250 auth_method 0 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:58.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 1 --2- 172.21.15.164:0/805701804 >> v2:172.21.15.165:3300/0 conn(0x7fce5809d930 0x7fce5809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): get_auth_request con 0x7fce5809d930 auth_method 0 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:58.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:58.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:58.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:58.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient(hunting): handle_auth_done global_id 74349 payload 274 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 -- 172.21.15.164:0/805701804 >> v2:172.21.15.165:3300/0 conn(0x7fce5809d930 msgr2=0x7fce5809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 --2- 172.21.15.164:0/805701804 >> v2:172.21.15.165:3300/0 conn(0x7fce5809d930 0x7fce5809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 -- 172.21.15.164:0/805701804 >> v1:172.21.15.165:6789/0 conn(0x7fce5809ab50 legacy=0x7fce58134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient: _finish_hunting 0 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 monclient: found mon.noname-a 2024-09-10T16:14:58.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:58.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:58.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.820+0000 7fce577fe640 1 -- 172.21.15.164:0/805701804 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5809ecb0 con 0x7fce5809e250 2024-09-10T16:14:58.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce577fe640 1 -- 172.21.15.164:0/805701804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809e250 msgr2=0x7fce58134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce577fe640 1 --2- 172.21.15.164:0/805701804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809e250 0x7fce58134c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fce5806d500 tx=0x7fce5002ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15181376eaebdcb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 1 -- 172.21.15.164:0/805701804 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50009800 con 0x7fce5809e250 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: dump: 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:58.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: _finish_auth 0 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: _check_auth_tickets 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 1 -- 172.21.15.164:0/805701804 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce500099a0 con 0x7fce5809e250 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce5d447640 5 monclient: authenticate success, global_id 74349 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce5d447640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce5d447640 10 monclient: get_monmap_and_config success 2024-09-10T16:14:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 1 -- 172.21.15.164:0/805701804 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce50009d70 con 0x7fce5809e250 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce5d447640 4 set_mon_vals no callback set 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.823+0000 7fce567fc640 10 monclient: dump: 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:58.843 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 10 monclient: shutdown 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 -- 172.21.15.164:0/805701804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809e250 msgr2=0x7fce58134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 --2- 172.21.15.164:0/805701804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809e250 0x7fce58134c90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fce5806d500 tx=0x7fce5002ed00 comp rx=0 tx=0).stop 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 -- 172.21.15.164:0/805701804 shutdown_connections 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 --2- 172.21.15.164:0/805701804 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809e250 0x7fce58134c90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 --2- 172.21.15.164:0/805701804 >> v2:172.21.15.165:3300/0 conn(0x7fce5809d930 0x7fce5809dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 -- 172.21.15.164:0/805701804 >> 172.21.15.164:0/805701804 conn(0x7fce580a3110 msgr2=0x7fce580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 -- 172.21.15.164:0/805701804 shutdown_connections 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 1 -- 172.21.15.164:0/805701804 wait complete. 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 10 monclient: build_initial_monmap 2024-09-10T16:14:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.824+0000 7fce5d447640 10 monclient: monmap: 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 Processor -- start 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 -- start start 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient: init 2024-09-10T16:14:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient: _reopen_session rank -1 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-a con 0x7fce5809ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 0x7fce58134d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-c con 0x7fce5809d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce5809e250 0x7fce5812b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): picked mon.noname-b con 0x7fce5809e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fce5809ef30 con 0x7fce5809ab50 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fce5809e6e0 con 0x7fce5809e250 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fce580a2940 con 0x7fce5809d930 2024-09-10T16:14:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): _renew_subs 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce5d447640 10 monclient(hunting): authenticate will time out at 1277.313764s 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce4e7fc640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 msgr2=0x7fce58134d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce4e7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 0x7fce58134d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35986/0 (socket says 172.21.15.164:35986) 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 1 -- 172.21.15.164:0/964078838 learned_addr learned my addr 172.21.15.164:0/964078838 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): get_auth_request con 0x7fce5809ab50 auth_method 0 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:14:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.825+0000 7fce56ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 10 monclient(hunting): handle_auth_done global_id 74352 payload 824 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 -- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 msgr2=0x7fce58134d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 0x7fce58134d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 -- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce5809e250 msgr2=0x7fce5812b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce5809e250 0x7fce5812b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 monclient: found mon.noname-a 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.826+0000 7fce56ffd640 1 -- 172.21.15.164:0/964078838 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5812df80 con 0x7fce5809ab50 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce56ffd640 1 -- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 msgr2=0x7fce581347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce56ffd640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fce4800e9e0 tx=0x7fce4800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48017070 con 0x7fce5809ab50 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: dump: 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:58.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: _finish_auth 0 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 5 monclient: authenticate success, global_id 74352 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4800cb60 con 0x7fce5809ab50 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 10 monclient: _renew_subs 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce5812ca60 con 0x7fce5809ab50 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48013470 con 0x7fce5809ab50 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce4ffff640 10 monclient: dump: 2024-09-10T16:14:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 10 monclient: _renew_subs 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce5812d9c0 con 0x7fce5809ab50 2024-09-10T16:14:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.828+0000 7fce55ffb640 4 set_mon_vals no callback set 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce5d447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce5d447640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce5d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fce4801a070 con 0x7fce5809ab50 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce5809dd10 con 0x7fce5809ab50 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.829+0000 7fce4ffff640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 0x7fce38040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.830+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7fce48052cf0 con 0x7fce5809ab50 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.830+0000 7fce4e7fc640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 0x7fce38040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.830+0000 7fce4e7fc640 10 monclient: get_auth_request con 0x7fce3803e1c0 auth_method 0 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.832+0000 7fce4e7fc640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 0x7fce38040680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fce5809a080 tx=0x7fce40007a10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.834+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fce48058020 con 0x7fce5809ab50 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.834+0000 7fce4ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:14:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:58.834+0000 7fce4ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.074+0000 7fce5d447640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.074+0000 7fce5d447640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.074+0000 7fce5d447640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.074+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fce5806dac0 con 0x7fce5809ab50 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.075+0000 7fce4ffff640 1 -- 172.21.15.164:0/964078838 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fce480516f0 con 0x7fce5809ab50 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.075+0000 7fce4ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:14:59.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.075+0000 7fce4ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:14:59.078 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 msgr2=0x7fce38040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 0x7fce38040680 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fce5809a080 tx=0x7fce40007a10 comp rx=0 tx=0).stop 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 10 monclient: shutdown 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 msgr2=0x7fce581347e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fce4800e9e0 tx=0x7fce4800eea0 comp rx=0 tx=0).stop 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 shutdown_connections 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fce3803e1c0 0x7fce38040680 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:59.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fce5809e250 0x7fce5812b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:59.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fce5809d930 0x7fce58134d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:59.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 --2- 172.21.15.164:0/964078838 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fce5809ab50 0x7fce581347e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:14:59.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 >> 172.21.15.164:0/964078838 conn(0x7fce580a3110 msgr2=0x7fce581bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:14:59.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 shutdown_connections 2024-09-10T16:14:59.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:14:59.077+0000 7fce5d447640 1 -- 172.21.15.164:0/964078838 wait complete. 2024-09-10T16:15:02.091 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.199+0000 7fa859700640 10 monclient: get_monmap_and_config 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.199+0000 7fa859700640 10 monclient: build_initial_monmap 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.199+0000 7fa859700640 10 monclient: monmap: 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:02.201275+0000 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:02.201275+0000 2024-09-10T16:15:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 Processor -- start 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 -- start start 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient: init 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-10T16:15:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa85406cb70 0x7fa854134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): picked mon.noname-c con 0x7fa85406cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8540a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8540a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85409ce50 con 0x7fa8540a4ab0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa854138ac0 con 0x7fa8540a46a0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa85406db60 con 0x7fa85406cb70 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa859700640 10 monclient(hunting): authenticate will time out at 1280.688691s 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa8537fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa8540a4ab0 0x7fa8541383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57090/0 (socket says 172.21.15.164:57090) 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa8537fe640 1 -- 172.21.15.164:0/1966323711 learned_addr learned my addr 172.21.15.164:0/1966323711 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.200+0000 7fa852ffd640 1 --2- 172.21.15.164:0/1966323711 >> v2:172.21.15.165:3300/0 conn(0x7fa85406cb70 0x7fa854134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): get_auth_request con 0x7fa85406cb70 auth_method 0 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.201+0000 7fa852ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.202+0000 7fa8537fe640 1 -- 172.21.15.164:0/1966323711 >> v1:172.21.15.164:6789/0 conn(0x7fa8540a4ab0 legacy=0x7fa8541383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa8527fc640 1 -- 172.21.15.164:0/1966323711 >> v1:172.21.15.165:6789/0 conn(0x7fa8540a46a0 legacy=0x7fa854137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 10 monclient(hunting): handle_auth_done global_id 94138 payload 274 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 1 -- 172.21.15.164:0/1966323711 >> v1:172.21.15.165:6789/0 conn(0x7fa8540a46a0 legacy=0x7fa854137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 1 -- 172.21.15.164:0/1966323711 >> v1:172.21.15.164:6789/0 conn(0x7fa8540a4ab0 legacy=0x7fa8541383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 1 monclient: found mon.noname-c 2024-09-10T16:15:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:02.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:02.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.203+0000 7fa852ffd640 1 -- 172.21.15.164:0/1966323711 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85409cb70 con 0x7fa85406cb70 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.206+0000 7fa852ffd640 1 -- 172.21.15.164:0/1966323711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa85406cb70 msgr2=0x7fa854134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.206+0000 7fa852ffd640 1 --2- 172.21.15.164:0/1966323711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa85406cb70 0x7fa854134f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa844009ec0 tx=0x7fa84402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8af621a19e5db744 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 1 -- 172.21.15.164:0/1966323711 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844007a30 con 0x7fa85406cb70 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:02.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: dump: 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: _finish_auth 0 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:15:02.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 1 -- 172.21.15.164:0/1966323711 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa844007bd0 con 0x7fa85406cb70 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 5 monclient: authenticate success, global_id 94138 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 4 set_mon_vals no callback set 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 1 -- 172.21.15.164:0/1966323711 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa844031400 con 0x7fa85406cb70 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa851ffb640 10 monclient: dump: 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 10 monclient: shutdown 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 -- 172.21.15.164:0/1966323711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa85406cb70 msgr2=0x7fa854134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 --2- 172.21.15.164:0/1966323711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa85406cb70 0x7fa854134f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa844009ec0 tx=0x7fa84402a710 comp rx=0 tx=0).stop 2024-09-10T16:15:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 -- 172.21.15.164:0/1966323711 shutdown_connections 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 --2- 172.21.15.164:0/1966323711 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa85406cb70 0x7fa854134f70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 -- 172.21.15.164:0/1966323711 >> 172.21.15.164:0/1966323711 conn(0x7fa8540aa900 msgr2=0x7fa8540aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 -- 172.21.15.164:0/1966323711 shutdown_connections 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.207+0000 7fa859700640 1 -- 172.21.15.164:0/1966323711 wait complete. 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.208+0000 7fa859700640 10 monclient: build_initial_monmap 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.208+0000 7fa859700640 10 monclient: monmap: 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.208+0000 7fa859700640 1 Processor -- start 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.208+0000 7fa859700640 1 -- start start 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.208+0000 7fa859700640 10 monclient: init 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 0x7fa8541318b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): picked mon.noname-c con 0x7fa85406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8540a46a0 0x7fa8540a2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): picked mon.noname-b con 0x7fa8540a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): picked mon.noname-a con 0x7fa8540a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa854141050 con 0x7fa8540a4ab0 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa85406cf50 con 0x7fa8540a46a0 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8541411d0 con 0x7fa85406cb70 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa852ffd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 msgr2=0x7fa8541318b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa859700640 10 monclient(hunting): authenticate will time out at 1280.697355s 2024-09-10T16:15:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa852ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 0x7fa8541318b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35998/0 (socket says 172.21.15.164:35998) 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 -- 172.21.15.164:0/1088935300 learned_addr learned my addr 172.21.15.164:0/1088935300 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): get_auth_request con 0x7fa8540a4ab0 auth_method 0 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8527fc640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8540a46a0 0x7fa8540a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8527fc640 10 monclient(hunting): get_auth_request con 0x7fa8540a46a0 auth_method 0 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8527fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8527fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient(hunting): handle_auth_done global_id 74355 payload 824 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 -- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 msgr2=0x7fa8541318b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 0x7fa8541318b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 -- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8540a46a0 msgr2=0x7fa8540a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8540a46a0 0x7fa8540a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.209+0000 7fa8537fe640 1 -- 172.21.15.164:0/1088935300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85413ad20 con 0x7fa8540a4ab0 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.211+0000 7fa8537fe640 1 -- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 msgr2=0x7fa8540a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.211+0000 7fa8537fe640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c410 tx=0x7fa84c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c017070 con 0x7fa8540a4ab0 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: dump: 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa84c0070d0 con 0x7fa8540a4ab0 2024-09-10T16:15:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 5 monclient: authenticate success, global_id 74355 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 10 monclient: _renew_subs 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa854139520 con 0x7fa8540a4ab0 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa84c007480 con 0x7fa8540a4ab0 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa83b7fe640 10 monclient: dump: 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa8517fa640 4 set_mon_vals no callback set 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 10 monclient: _renew_subs 2024-09-10T16:15:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.212+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa854140b90 con 0x7fa8540a4ab0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa859700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa859700640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa859700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa84c01a070 con 0x7fa8540a4ab0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8541382d0 con 0x7fa8540a4ab0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa83b7fe640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 0x7fa8400406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(290..290 src has 184..290) ==== 4771+0+0 (secure 0 0 0) 0x7fa84c052050 con 0x7fa8540a4ab0 2024-09-10T16:15:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa852ffd640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 0x7fa8400406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.213+0000 7fa852ffd640 10 monclient: get_auth_request con 0x7fa84003e1f0 auth_method 0 2024-09-10T16:15:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.216+0000 7fa852ffd640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 0x7fa8400406b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa844031000 tx=0x7fa84402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.217+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa8541382d0 con 0x7fa8540a4ab0 2024-09-10T16:15:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.217+0000 7fa83b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.217+0000 7fa83b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:02.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.460+0000 7fa859700640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:02.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.460+0000 7fa859700640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:02.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.460+0000 7fa859700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:02.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.460+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa85413b6d0 con 0x7fa8540a4ab0 2024-09-10T16:15:02.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.461+0000 7fa83b7fe640 1 -- 172.21.15.164:0/1088935300 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fa85409ce50 con 0x7fa8540a4ab0 2024-09-10T16:15:02.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.461+0000 7fa83b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:02.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.461+0000 7fa83b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:02.464 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 msgr2=0x7fa8400406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 0x7fa8400406b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa844031000 tx=0x7fa84402f000 comp rx=0 tx=0).stop 2024-09-10T16:15:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 10 monclient: shutdown 2024-09-10T16:15:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 msgr2=0x7fa8540a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c410 tx=0x7fa84c00c8d0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 shutdown_connections 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa84003e1f0 0x7fa8400406b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8540a4ab0 0x7fa8540a2dc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8540a46a0 0x7fa8540a2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 --2- 172.21.15.164:0/1088935300 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85406cb70 0x7fa8541318b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 >> 172.21.15.164:0/1088935300 conn(0x7fa8540aa900 msgr2=0x7fa854137bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 shutdown_connections 2024-09-10T16:15:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:02.463+0000 7fa859700640 1 -- 172.21.15.164:0/1088935300 wait complete. 2024-09-10T16:15:05.477 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:05.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 10 monclient: get_monmap_and_config 2024-09-10T16:15:05.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 10 monclient: build_initial_monmap 2024-09-10T16:15:05.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 10 monclient: monmap: 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:05.589094+0000 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:05.589094+0000 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:05.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 1 Processor -- start 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 1 -- start start 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.588+0000 7f9dd8924640 10 monclient: init 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9dd406cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9dd4134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-a con 0x7f9dd413e740 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dd409ce50 con 0x7f9dd4134f70 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dd4135380 con 0x7f9dd406cb70 2024-09-10T16:15:05.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9dd406db60 con 0x7f9dd413e740 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd8924640 10 monclient(hunting): authenticate will time out at 1284.076623s 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd1d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9dd4134f70 0x7f9dd413d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42436/0 (socket says 172.21.15.164:42436) 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 learned_addr learned my addr 172.21.15.164:0/1863022836 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f9dd413e740 auth_method 0 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:15:05.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:05.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:05.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.589+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:05.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.591+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v1:172.21.15.164:6789/0 conn(0x7f9dd4134f70 legacy=0x7f9dd413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:05.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.592+0000 7f9dd259c640 1 -- 172.21.15.164:0/1863022836 >> v1:172.21.15.165:6789/0 conn(0x7f9dd406cb70 legacy=0x7f9dd40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f9dd413e740 auth_method 0 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:05.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:05.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:05.790+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f9dd413e740 auth_method 0 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.191+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f9dd413e740 auth_method 0 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:15:06.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:06.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:06.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:06.992+0000 7f9dd2d9d640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:15:08.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): tick 2024-09-10T16:15:08.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:15:08.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 msgr2=0x7f9dd4140b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.164:3300/0 conn(0x7f9dd413e740 0x7f9dd4140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 -- 172.21.15.164:0/1863022836 >> v1:172.21.15.165:6789/0 conn(0x7f9dd406cb70 legacy=0x7f9dd40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 -- 172.21.15.164:0/1863022836 >> v1:172.21.15.164:6789/0 conn(0x7f9dd4134f70 legacy=0x7f9dd413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient: _add_conns ranks=[2,4] 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): picked mon.noname-b con 0x7f9db4003860 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): picked mon.noname-c con 0x7f9db40063c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:08.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 -- 172.21.15.164:0/1863022836 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9db4008f10 con 0x7f9db40063c0 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 1 -- 172.21.15.164:0/1863022836 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9db40090c0 con 0x7f9db4003860 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dcbfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dd259c640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 msgr2=0x7f9db4005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.589+0000 7f9dd259c640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f9db40063c0 auth_method 0 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:08.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:08.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.590+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:08.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.790+0000 7f9dd259c640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 msgr2=0x7f9db4005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:08.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.790+0000 7f9dd259c640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f9db40063c0 auth_method 0 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:08.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:08.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:08.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:08.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:08.792+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:09.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.191+0000 7f9dd259c640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 msgr2=0x7f9db4005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:09.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.191+0000 7f9dd259c640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:09.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f9db40063c0 auth_method 0 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 msgr2=0x7f9db4008880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:09.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.194+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:09.654 INFO:tasks.rados.rados.0.smithi165.stdout:1153: done (2 left) 2024-09-10T16:15:09.655 INFO:tasks.rados.rados.0.smithi165.stdout:1154: copy_from oid 33 from oid 20 current snap is 137 2024-09-10T16:15:09.655 INFO:tasks.rados.rados.0.smithi165.stdout:1152: finishing rollback tid 0 to smithi16534841-11 2024-09-10T16:15:09.655 INFO:tasks.rados.rados.0.smithi165.stdout:1152: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:15:09.655 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 573 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:15:09.655 INFO:tasks.rados.rados.0.smithi165.stdout:1151: expect (ObjNum 307 snap 88 seq_num 307) 2024-09-10T16:15:09.882 INFO:tasks.rados.rados.0.smithi165.stdout:1154: finishing copy_from to smithi16534841-33 2024-09-10T16:15:09.882 INFO:tasks.rados.rados.0.smithi165.stdout:1154: got expected ENOENT (src dne) 2024-09-10T16:15:09.882 INFO:tasks.rados.rados.0.smithi165.stdout:1154: finishing copy_from racing read to smithi16534841-33 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1151: done (2 left) 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1152: done (1 left) 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1154: done (0 left) 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1155: read oid 23 snap -1 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1155: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1156: write (excl) oid 50 current snap is 137 2024-09-10T16:15:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:1156: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-10T16:15:09.887 INFO:tasks.rados.rados.0.smithi165.stdout:1156: writing smithi16534841-50 from 608519 to 1070414 tid 1 2024-09-10T16:15:09.893 INFO:tasks.rados.rados.0.smithi165.stdout:1156: writing smithi16534841-50 from 1602050 to 2130859 tid 2 2024-09-10T16:15:09.898 INFO:tasks.rados.rados.0.smithi165.stdout:1156: writing smithi16534841-50 from 2585035 to 3008519 tid 3 2024-09-10T16:15:09.898 INFO:tasks.rados.rados.0.smithi165.stdout:1157: delete oid 26 current snap is 137 2024-09-10T16:15:09.898 INFO:tasks.rados.rados.0.smithi165.stdout:1155: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:15:09.899 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:09.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.992+0000 7f9dd259c640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 msgr2=0x7f9db4005c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:09.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.993+0000 7f9dd259c640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> v2:172.21.15.165:3300/0 conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request con 0x7f9db40063c0 auth_method 0 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient(hunting): handle_auth_done global_id 94144 payload 274 2024-09-10T16:15:09.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 msgr2=0x7f9db4005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9db4003860 0x7f9db4005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 1 monclient: found mon.noname-c 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:09.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.995+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd409cb70 con 0x7f9db40063c0 2024-09-10T16:15:10.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.998+0000 7f9dd1d9b640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 msgr2=0x7f9db4008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.998+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9db4008880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4007fd0 tx=0x7f9dc402a940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5aca76bf594bb595 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 1 -- 172.21.15.164:0/1863022836 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031640 con 0x7f9db40063c0 2024-09-10T16:15:10.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: dump: 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.001 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 1 -- 172.21.15.164:0/1863022836 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dc4037070 con 0x7f9db40063c0 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 1 -- 172.21.15.164:0/1863022836 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dc4031a20 con 0x7f9db40063c0 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd8924640 5 monclient: authenticate success, global_id 94144 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd8924640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd8924640 4 set_mon_vals no callback set 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:09.999+0000 7f9dd159a640 10 monclient: dump: 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.002 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.003 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 10 monclient: shutdown 2024-09-10T16:15:10.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 -- 172.21.15.164:0/1863022836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 msgr2=0x7f9db4008880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9db4008880 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4007fd0 tx=0x7f9dc402a940 comp rx=0 tx=0).stop 2024-09-10T16:15:10.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd2d9d640 1 -- 172.21.15.164:0/1863022836 reap_dead start 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 -- 172.21.15.164:0/1863022836 shutdown_connections 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 --2- 172.21.15.164:0/1863022836 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9db4008880 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 -- 172.21.15.164:0/1863022836 >> 172.21.15.164:0/1863022836 conn(0x7f9dd40aa900 msgr2=0x7f9dd40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 -- 172.21.15.164:0/1863022836 shutdown_connections 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.000+0000 7f9dd8924640 1 -- 172.21.15.164:0/1863022836 wait complete. 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.001+0000 7f9dd8924640 10 monclient: build_initial_monmap 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.001+0000 7f9dd8924640 10 monclient: monmap: 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:10.008 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.001+0000 7f9dd8924640 1 Processor -- start 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 -- start start 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient: init 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-b con 0x7f9db40063c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:10.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 0x7f9dd413acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-c con 0x7f9dd4131800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 0x7f9dd413b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): picked mon.noname-a con 0x7f9dd4131c40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd40a4db0 con 0x7f9dd4131c40 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9dd406cf50 con 0x7f9db40063c0 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9dd4141070 con 0x7f9dd4131800 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd8924640 10 monclient(hunting): authenticate will time out at 1288.489716s 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd1d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 msgr2=0x7f9dd413acb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:10.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd1d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 0x7f9dd413acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 0x7f9dd413b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:48536/0 (socket says 172.21.15.164:48536) 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 1 -- 172.21.15.164:0/2671109809 learned_addr learned my addr 172.21.15.164:0/2671109809 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 0x7f9dd413b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57292/0 (socket says 172.21.15.164:57292) 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request con 0x7f9dd4131c40 auth_method 0 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 10 monclient(hunting): get_auth_request con 0x7f9db40063c0 auth_method 0 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.002+0000 7f9dd259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient(hunting): handle_auth_done global_id 94147 payload 824 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 -- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 msgr2=0x7f9dd413acb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:10.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 0x7f9dd413acb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 -- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 msgr2=0x7f9dd413b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 0x7f9dd413b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient: _finish_hunting 0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 monclient: found mon.noname-b 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.003+0000 7f9dd259c640 1 -- 172.21.15.164:0/2671109809 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dd409dbb0 con 0x7f9db40063c0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd259c640 1 -- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 msgr2=0x7f9dd41312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd259c640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c520 tx=0x7f9dbc00c9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc017070 con 0x7f9db40063c0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:15:10.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: dump: 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dbc0057c0 con 0x7f9db40063c0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 5 monclient: authenticate success, global_id 94147 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 10 monclient: _renew_subs 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9dd413cec0 con 0x7f9db40063c0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dbc005b90 con 0x7f9db40063c0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dcaffd640 10 monclient: dump: 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd0d99640 4 set_mon_vals no callback set 2024-09-10T16:15:10.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 10 monclient: _renew_subs 2024-09-10T16:15:10.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:15:10.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.005+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9dd409de90 con 0x7f9db40063c0 2024-09-10T16:15:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.006+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9dbc01a070 con 0x7f9db40063c0 2024-09-10T16:15:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.006+0000 7f9dd8924640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dd8924640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dd8924640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:15:10.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9dd40a46a0 con 0x7f9db40063c0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dcaffd640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 0x7f9db4041730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(291..291 src has 184..291) ==== 4815+0+0 (secure 0 0 0) 0x7f9dbc052180 con 0x7f9db40063c0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 0x7f9db4041730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.007+0000 7f9dd1d9b640 10 monclient: get_auth_request con 0x7f9db403f270 auth_method 0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.009+0000 7f9dd1d9b640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 0x7f9db4041730 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4033000 tx=0x7f9dc4010040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.016+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9dbc050b60 con 0x7f9db40063c0 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.016+0000 7f9dcaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.016+0000 7f9dcaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:10.062 INFO:tasks.rados.rados.0.smithi165.stdout:1156: finishing write tid 1 to smithi16534841-50 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: finishing write tid 2 to smithi16534841-50 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: finishing write tid 3 to smithi16534841-50 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: finishing write tid 4 to smithi16534841-50 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: finishing write tid 5 to smithi16534841-50 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: oid 50 updating version 0 to 577 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: oid 50 version 577 is already newer than 575 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: oid 50 version 577 is already newer than 574 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: oid 50 version 577 is already newer than 576 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 577 (ObjNum 417 snap 137 seq_num 417) dirty exists 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: left oid 50 (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1155: done (2 left) 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1156: done (1 left) 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1157: done (0 left) 2024-09-10T16:15:10.063 INFO:tasks.rados.rados.0.smithi165.stdout:1158: snap_create 2024-09-10T16:15:10.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.264+0000 7f9dd8924640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:10.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.264+0000 7f9dd8924640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:10.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.264+0000 7f9dd8924640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:15:10.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.264+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9dd40770f0 con 0x7f9db40063c0 2024-09-10T16:15:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.265+0000 7f9dcaffd640 1 -- 172.21.15.164:0/2671109809 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f9dbc005d30 con 0x7f9db40063c0 2024-09-10T16:15:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.265+0000 7f9dcaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:10.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.265+0000 7f9dcaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:10.267 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 msgr2=0x7f9db4041730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 0x7f9db4041730 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9dc4033000 tx=0x7f9dc4010040 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 10 monclient: shutdown 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 msgr2=0x7f9dd41312c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f9dbc00c520 tx=0x7f9dbc00c9e0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 shutdown_connections 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9db403f270 0x7f9db4041730 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9dd4131c40 0x7f9dd413b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9dd4131800 0x7f9dd413acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 --2- 172.21.15.164:0/2671109809 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9db40063c0 0x7f9dd41312c0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 >> 172.21.15.164:0/2671109809 conn(0x7f9dd40aa900 msgr2=0x7f9dd4133990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 shutdown_connections 2024-09-10T16:15:10.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.267+0000 7f9dd8924640 1 -- 172.21.15.164:0/2671109809 wait complete. 2024-09-10T16:15:10.281 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:15:10.281 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:15:10.281 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.378+0000 7f4dc8cbd640 10 monclient: get_monmap_and_config 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.378+0000 7f4dc8cbd640 10 monclient: build_initial_monmap 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.378+0000 7f4dc8cbd640 10 monclient: monmap: 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:10.379448+0000 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:10.379448+0000 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.383 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 Processor -- start 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 -- start start 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient: init 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dc409d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:10.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4dc409ab50 0x7f4dc41397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4dc409e420 0x7f4dc413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc409e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc40a0870 con 0x7f4dc409d930 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc409d3e0 con 0x7f4dc409ab50 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc4074c70 con 0x7f4dc409e420 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc8cbd640 10 monclient(hunting): authenticate will time out at 1288.866797s 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc1d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4dc409ab50 0x7f4dc41397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4dc409d930 0x7f4dc409dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42448/0 (socket says 172.21.15.164:42448) 2024-09-10T16:15:10.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc259c640 1 -- 172.21.15.164:0/171102741 learned_addr learned my addr 172.21.15.164:0/171102741 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc1d9b640 10 monclient(hunting): get_auth_request con 0x7f4dc409ab50 auth_method 0 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.379+0000 7f4dc1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc2d9d640 1 --2- 172.21.15.164:0/171102741 >> v2:172.21.15.165:3300/0 conn(0x7f4dc409e420 0x7f4dc413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc2d9d640 10 monclient(hunting): get_auth_request con 0x7f4dc409e420 auth_method 0 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient(hunting): handle_auth_done global_id 74361 payload 274 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 1 -- 172.21.15.164:0/171102741 >> v2:172.21.15.165:3300/0 conn(0x7f4dc409e420 msgr2=0x7f4dc413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 1 --2- 172.21.15.164:0/171102741 >> v2:172.21.15.165:3300/0 conn(0x7f4dc409e420 0x7f4dc413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 1 -- 172.21.15.164:0/171102741 >> v1:172.21.15.164:6789/0 conn(0x7f4dc409d930 legacy=0x7f4dc409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 1 monclient: found mon.noname-a 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.380+0000 7f4dc1d9b640 1 -- 172.21.15.164:0/171102741 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc409ecb0 con 0x7f4dc409ab50 2024-09-10T16:15:10.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.382+0000 7f4dc1d9b640 1 -- 172.21.15.164:0/171102741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409ab50 msgr2=0x7f4dc41397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.382+0000 7f4dc1d9b640 1 --2- 172.21.15.164:0/171102741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409ab50 0x7f4dc41397c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4dc409af60 tx=0x7f4db802c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abd8454558133a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 1 -- 172.21.15.164:0/171102741 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8031920 con 0x7f4dc409ab50 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: dump: 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.395 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 5 monclient: authenticate success, global_id 74361 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 1 -- 172.21.15.164:0/171102741 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db8031ac0 con 0x7f4dc409ab50 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 4 set_mon_vals no callback set 2024-09-10T16:15:10.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 1 -- 172.21.15.164:0/171102741 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db8033400 con 0x7f4dc409ab50 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc159a640 10 monclient: dump: 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.397 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 10 monclient: shutdown 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/171102741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409ab50 msgr2=0x7f4dc41397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/171102741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409ab50 0x7f4dc41397c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4dc409af60 tx=0x7f4db802c6a0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/171102741 shutdown_connections 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/171102741 >> v2:172.21.15.165:3300/0 conn(0x7f4dc409e420 0x7f4dc413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/171102741 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409ab50 0x7f4dc41397c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/171102741 >> 172.21.15.164:0/171102741 conn(0x7f4dc40a3110 msgr2=0x7f4dc40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/171102741 shutdown_connections 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.383+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/171102741 wait complete. 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.384+0000 7f4dc8cbd640 10 monclient: build_initial_monmap 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.384+0000 7f4dc8cbd640 10 monclient: monmap: 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:10.398 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.384+0000 7f4dc8cbd640 1 Processor -- start 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.384+0000 7f4dc8cbd640 1 -- start start 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.384+0000 7f4dc8cbd640 10 monclient: init 2024-09-10T16:15:10.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dc409ab50 0x7f4dc4073a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-b con 0x7f4dc409ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc409d930 0x7f4dc41ca060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-c con 0x7f4dc409d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): picked mon.noname-a con 0x7f4dc409e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc40a2940 con 0x7f4dc409e420 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc409ef30 con 0x7f4dc409ab50 2024-09-10T16:15:10.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc40a2ac0 con 0x7f4dc409d930 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc8cbd640 10 monclient(hunting): authenticate will time out at 1288.872259s 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc1d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc409d930 msgr2=0x7f4dc41ca060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc1d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc409d930 0x7f4dc41ca060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57312/0 (socket says 172.21.15.164:57312) 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 learned_addr learned my addr 172.21.15.164:0/3628178760 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): get_auth_request con 0x7f4dc409e420 auth_method 0 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dc409ab50 0x7f4dc4073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): get_auth_request con 0x7f4dc409ab50 auth_method 0 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient(hunting): handle_auth_done global_id 74364 payload 824 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc409d930 msgr2=0x7f4dc41ca060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4dc409d930 0x7f4dc41ca060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dc409ab50 msgr2=0x7f4dc4073a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4dc409ab50 0x7f4dc4073a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.385+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc409cc90 con 0x7f4dc409e420 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 msgr2=0x7f4dc41ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc2d9d640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc007e40 tx=0x7f4dbc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc013790 con 0x7f4dc409e420 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: dump: 2024-09-10T16:15:10.403 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc8cbd640 5 monclient: authenticate success, global_id 74364 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dbc019070 con 0x7f4dc409e420 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc8cbd640 10 monclient: _renew_subs 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc8cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dc4131dd0 con 0x7f4dc409e420 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc013b90 con 0x7f4dc409e420 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4db2ffd640 10 monclient: dump: 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.406 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.387+0000 7f4dc0d99640 4 set_mon_vals no callback set 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 10 monclient: _renew_subs 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc4074330 con 0x7f4dc409e420 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 0x7f4dc412fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc409dd10 con 0x7f4dc412f900 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4dbc01e460 con 0x7f4dc409e420 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 0x7f4dc412fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient: get_auth_request con 0x7f4dc412f900 auth_method 0 2024-09-10T16:15:10.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.388+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4db2ffd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4da403e1c0 0x7f4da4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4dc259c640 10 monclient(hunting): handle_auth_done global_id 74367 payload 824 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(291..291 src has 184..291) ==== 4815+0+0 (secure 0 0 0) 0x7f4dbc0535e0 con 0x7f4dc409e420 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4dc259c640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4dc406daa0 con 0x7f4dc412f900 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4dc1d9b640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4da403e1c0 0x7f4da4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.389+0000 7f4dc1d9b640 10 monclient: get_auth_request con 0x7f4da403e1c0 auth_method 0 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4dc259c640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 msgr2=0x7f4dc412fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4dc259c640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 0x7f4dc412fce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4db400ea70 tx=0x7f4db400ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db401a070 con 0x7f4dc412f900 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 10 monclient: dump: 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.409 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4db400cc50 con 0x7f4dc412f900 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 msgr2=0x7f4dc412fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4dc1d9b640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4da403e1c0 0x7f4da4040680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406d500 tx=0x7f4db8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 0x7f4dc412fce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f4db400ea70 tx=0x7f4db400ef30 comp rx=0 tx=0).stop 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 msgr2=0x7f4dc412fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:10.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.391+0000 7f4db2ffd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc412f900 0x7f4dc412fce0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc8cbd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 0x7f4dc406f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc8cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc4077260 con 0x7f4dc4130220 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc259c640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 0x7f4dc406f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc259c640 10 monclient: get_auth_request con 0x7f4dc4130220 auth_method 0 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.397+0000 7f4dc259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.398+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.398+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.398+0000 7f4dc259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.398+0000 7f4dc259c640 10 monclient(hunting): handle_auth_done global_id 74370 payload 824 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.398+0000 7f4dc259c640 1 -- 172.21.15.164:0/3628178760 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4dc4131090 con 0x7f4dc4130220 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4dc259c640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 msgr2=0x7f4dc406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4dc259c640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 0x7f4dc406f7e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4db4017000 tx=0x7f4db4004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db401a050 con 0x7f4dc4130220 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 10 monclient: dump: 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f4db401b040 con 0x7f4dc4130220 2024-09-10T16:15:10.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 msgr2=0x7f4dc406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 0x7f4dc406f7e0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4db4017000 tx=0x7f4db4004940 comp rx=0 tx=0).stop 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 msgr2=0x7f4dc406f7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.400+0000 7f4db2ffd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc4130220 0x7f4dc406f7e0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f4db4017000 tx=0x7f4db4004940 comp rx=0 tx=0).stop 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:15:10.415 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 84, 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 138267, 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.416 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.417 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:10.418 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:10.419 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.420 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:10.421 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:10.423 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:10.424 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.426 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.427 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:10.428 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:10.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4da403e1c0 msgr2=0x7f4da4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4da403e1c0 0x7f4da4040680 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4dc406d500 tx=0x7f4db8008040 comp rx=0 tx=0).stop 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 10 monclient: shutdown 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 msgr2=0x7f4dc41ca5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc2d9d640 1 -- 172.21.15.164:0/3628178760 reap_dead start 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc007e40 tx=0x7f4dbc00e4b0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 shutdown_connections 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 --2- 172.21.15.164:0/3628178760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4dc409e420 0x7f4dc41ca5a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 >> 172.21.15.164:0/3628178760 conn(0x7f4dc40a3110 msgr2=0x7f4dc41bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 shutdown_connections 2024-09-10T16:15:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.401+0000 7f4dc8cbd640 1 -- 172.21.15.164:0/3628178760 wait complete. 2024-09-10T16:15:10.430 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:15:10.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:15:10.431 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.528+0000 7f2bde1c6640 10 monclient: get_monmap_and_config 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.528+0000 7f2bde1c6640 10 monclient: build_initial_monmap 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.528+0000 7f2bde1c6640 10 monclient: monmap: 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:10.529704+0000 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:10.529704+0000 2024-09-10T16:15:10.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:10.531 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.529+0000 7f2bde1c6640 1 Processor -- start 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.529+0000 7f2bde1c6640 1 -- start start 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.529+0000 7f2bde1c6640 10 monclient: init 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 0x7f2bd80a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f2bd80a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2bd80a2380 0x7f2bd80a27e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f2bd80a2380 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2bd813a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2bd80a0210 con 0x7f2bd813a5c0 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2bd809d3f0 con 0x7f2bd80a2380 2024-09-10T16:15:10.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd809ed90 con 0x7f2bd80a0410 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bde1c6640 10 monclient(hunting): authenticate will time out at 1289.017120s 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd77fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 msgr2=0x7f2bd80a1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 0x7f2bd80a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2bd80a2380 0x7f2bd80a27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2bd80a2380 0x7f2bd80a27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57334/0 (socket says 172.21.15.164:57334) 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/2319388806 learned_addr learned my addr 172.21.15.164:0/2319388806 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): get_auth_request con 0x7f2bd80a2380 auth_method 0 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_done global_id 74373 payload 274 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 msgr2=0x7f2bd80a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 --2- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 0x7f2bd80a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/2319388806 >> v1:172.21.15.165:6789/0 conn(0x7f2bd813a5c0 legacy=0x7f2bd813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.530+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/2319388806 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd809d930 con 0x7f2bd80a2380 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 msgr2=0x7f2bd80a27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd6ffd640 1 --2- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd80a27e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2bd806dad0 tx=0x7f2bcc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81e948b5eda635c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 1 -- 172.21.15.164:0/2319388806 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc037070 con 0x7f2bd80a2380 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: dump: 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 5 monclient: authenticate success, global_id 74373 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 1 -- 172.21.15.164:0/2319388806 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bcc032640 con 0x7f2bd80a2380 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 4 set_mon_vals no callback set 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 1 -- 172.21.15.164:0/2319388806 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc032a10 con 0x7f2bd80a2380 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bd67fc640 10 monclient: dump: 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 10 monclient: shutdown 2024-09-10T16:15:10.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 -- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 msgr2=0x7f2bd80a27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd80a27e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f2bd806dad0 tx=0x7f2bcc02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 -- 172.21.15.164:0/2319388806 shutdown_connections 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd80a27e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/2319388806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd80a0410 0x7f2bd80a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.533+0000 7f2bde1c6640 1 -- 172.21.15.164:0/2319388806 >> 172.21.15.164:0/2319388806 conn(0x7f2bd80a3110 msgr2=0x7f2bd80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.534+0000 7f2bde1c6640 1 -- 172.21.15.164:0/2319388806 shutdown_connections 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.534+0000 7f2bde1c6640 1 -- 172.21.15.164:0/2319388806 wait complete. 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.534+0000 7f2bde1c6640 10 monclient: build_initial_monmap 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.534+0000 7f2bde1c6640 10 monclient: monmap: 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:10.548 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:10.549 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:10.549 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:10.549 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.549 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.534+0000 7f2bde1c6640 1 Processor -- start 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 -- start start 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient: init 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:10.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd80a0410 0x7f2bd8090700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-b con 0x7f2bd80a0410 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-a con 0x7f2bd80a2380 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd813a5c0 0x7f2bd8091180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): picked mon.noname-c con 0x7f2bd813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd809c910 con 0x7f2bd80a2380 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd809e890 con 0x7f2bd80a0410 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2bd80a07f0 con 0x7f2bd813a5c0 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bde1c6640 10 monclient(hunting): authenticate will time out at 1289.022608s 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd7fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd813a5c0 msgr2=0x7f2bd8091180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd813a5c0 0x7f2bd8091180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:57340/0 (socket says 172.21.15.164:57340) 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/3263211974 learned_addr learned my addr 172.21.15.164:0/3263211974 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): get_auth_request con 0x7f2bd80a2380 auth_method 0 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd80a0410 0x7f2bd8090700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): get_auth_request con 0x7f2bd80a0410 auth_method 0 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient(hunting): handle_auth_done global_id 74376 payload 824 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd813a5c0 msgr2=0x7f2bd8091180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2bd813a5c0 0x7f2bd8091180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd80a0410 msgr2=0x7f2bd8090700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd80a0410 0x7f2bd8090700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.535+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd8092ee0 con 0x7f2bd80a2380 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bd6ffd640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 msgr2=0x7f2bd8090c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bd6ffd640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0090b0 tx=0x7f2bcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc041070 con 0x7f2bd80a2380 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: dump: 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: _finish_auth 0 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: _check_auth_tickets 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 5 monclient: authenticate success, global_id 74376 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bcc03c880 con 0x7f2bd80a2380 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bcc037040 con 0x7f2bd80a2380 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 10 monclient: _renew_subs 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bd8094130 con 0x7f2bd80a2380 2024-09-10T16:15:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bbffff640 10 monclient: dump: 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bd5ffb640 4 set_mon_vals no callback set 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 10 monclient: _renew_subs 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.538+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bd8093e50 con 0x7f2bd80a2380 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bde1c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 0x7f2bd80916c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd809d620 con 0x7f2bd802e5c0 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2bcc044070 con 0x7f2bd80a2380 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 0x7f2bd80916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient: get_auth_request con 0x7f2bd802e5c0 auth_method 0 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bbffff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bc403e1c0 0x7f2bc4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_done global_id 94159 payload 824 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd77fe640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bd802e9a0 con 0x7f2bd802e5c0 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(291..291 src has 184..291) ==== 4815+0+0 (secure 0 0 0) 0x7f2bcc0730c0 con 0x7f2bd80a2380 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd7fff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bc403e1c0 0x7f2bc4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.539+0000 7f2bd7fff640 10 monclient: get_auth_request con 0x7f2bc403e1c0 auth_method 0 2024-09-10T16:15:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.541+0000 7f2bd77fe640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 msgr2=0x7f2bd80916c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.541+0000 7f2bd77fe640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 0x7f2bd80916c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8007660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc801a070 con 0x7f2bd802e5c0 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 10 monclient: dump: 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bd7fff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bc403e1c0 0x7f2bc4040680 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80948e0 tx=0x7f2bd0007990 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2bc80056f0 con 0x7f2bd802e5c0 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 msgr2=0x7f2bd80916c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 0x7f2bd80916c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8007660 comp rx=0 tx=0).stop 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 msgr2=0x7f2bd80916c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:10.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.542+0000 7f2bbffff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd802e5c0 0x7f2bd80916c0 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8007660 comp rx=0 tx=0).stop 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bde1c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 0x7f2bd8064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bde1c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2bd809fb80 con 0x7f2bd8063d50 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 0x7f2bd8064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient: get_auth_request con 0x7f2bd8063d50 auth_method 0 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 10 monclient(hunting): handle_auth_done global_id 94162 payload 824 2024-09-10T16:15:10.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.548+0000 7f2bd77fe640 1 -- 172.21.15.164:0/3263211974 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2bd80a1e80 con 0x7f2bd8063d50 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bd77fe640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 msgr2=0x7f2bd8064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bd77fe640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 0x7f2bd8064160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc801a030 con 0x7f2bd8063d50 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 10 monclient: dump: 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:10.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7f2bc801b040 con 0x7f2bd8063d50 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 msgr2=0x7f2bd8064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 0x7f2bd8064160 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8005d50 comp rx=0 tx=0).stop 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 msgr2=0x7f2bd8064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:10.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bbffff640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2bd8063d50 0x7f2bd8064160 secure :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2bc8009ed0 tx=0x7f2bc8005d50 comp rx=0 tx=0).stop 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 84, 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 28051, 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:15:10.572 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.573 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.574 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:10.575 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:10.576 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:10.577 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:10.580 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:10.581 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.582 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:10.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:10.584 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:15:10.585 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:10.585 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:10.585 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:10.585 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:10.585 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bc403e1c0 msgr2=0x7f2bc4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bc403e1c0 0x7f2bc4040680 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2bd80948e0 tx=0x7f2bd0007990 comp rx=0 tx=0).stop 2024-09-10T16:15:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 10 monclient: shutdown 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 msgr2=0x7f2bd8090c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc0090b0 tx=0x7f2bcc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bd7fff640 1 -- 172.21.15.164:0/3263211974 reap_dead start 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 shutdown_connections 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 --2- 172.21.15.164:0/3263211974 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2bd80a2380 0x7f2bd8090c40 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 >> 172.21.15.164:0/3263211974 conn(0x7f2bd80a3110 msgr2=0x7f2bd81389f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 shutdown_connections 2024-09-10T16:15:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:10.551+0000 7f2bde1c6640 1 -- 172.21.15.164:0/3263211974 wait complete. 2024-09-10T16:15:10.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:15:10.588 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:15:10.658 INFO:tasks.rados.rados.0.smithi165.stdout:1158: done (0 left) 2024-09-10T16:15:10.658 INFO:tasks.rados.rados.0.smithi165.stdout:1159: read oid 36 snap -1 2024-09-10T16:15:10.658 INFO:tasks.rados.rados.0.smithi165.stdout:1159: expect deleted 2024-09-10T16:15:10.658 INFO:tasks.rados.rados.0.smithi165.stdout:1160: write oid 21 current snap is 138 2024-09-10T16:15:10.658 INFO:tasks.rados.rados.0.smithi165.stdout:1160: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-10T16:15:10.665 INFO:tasks.rados.rados.0.smithi165.stdout:1160: writing smithi16534841-21 from 642133 to 1370879 tid 1 2024-09-10T16:15:10.673 INFO:tasks.rados.rados.0.smithi165.stdout:1160: writing smithi16534841-21 from 2119014 to 2651492 tid 2 2024-09-10T16:15:10.674 INFO:tasks.rados.rados.0.smithi165.stdout:1161: delete oid 44 current snap is 138 2024-09-10T16:15:10.674 INFO:tasks.rados.rados.0.smithi165.stdout:1160: finishing write tid 1 to smithi16534841-21 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1159: done (2 left) 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1161: done (1 left) 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1162: read oid 44 snap -1 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1162: expect deleted 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1163: write (excl) oid 47 current snap is 138 2024-09-10T16:15:10.675 INFO:tasks.rados.rados.0.smithi165.stdout:1163: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-10T16:15:10.681 INFO:tasks.rados.rados.0.smithi165.stdout:1163: writing smithi16534841-47 from 675747 to 1154991 tid 1 2024-09-10T16:15:10.688 INFO:tasks.rados.rados.0.smithi165.stdout:1163: writing smithi16534841-47 from 1835978 to 2372125 tid 2 2024-09-10T16:15:10.689 INFO:tasks.rados.rados.0.smithi165.stdout:1164: snap_remove snap 135 2024-09-10T16:15:11.666 INFO:tasks.rados.rados.0.smithi165.stdout:1164: done (3 left) 2024-09-10T16:15:11.666 INFO:tasks.rados.rados.0.smithi165.stdout:1165: read oid 5 snap -1 2024-09-10T16:15:11.666 INFO:tasks.rados.rados.0.smithi165.stdout:1165: expect (ObjNum 408 snap 132 seq_num 408) 2024-09-10T16:15:11.666 INFO:tasks.rados.rados.0.smithi165.stdout:1166: delete oid 3 current snap is 138 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: finishing write tid 2 to smithi16534841-21 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: finishing write tid 3 to smithi16534841-21 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: finishing write tid 4 to smithi16534841-21 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: oid 21 updating version 0 to 719 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: oid 21 updating version 719 to 720 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: oid 21 version 720 is already newer than 718 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 720 (ObjNum 419 snap 138 seq_num 419) dirty exists 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1160: left oid 21 (ObjNum 419 snap 138 seq_num 419) 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1163: finishing write tid 1 to smithi16534841-47 2024-09-10T16:15:11.667 INFO:tasks.rados.rados.0.smithi165.stdout:1163: finishing write tid 2 to smithi16534841-47 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: finishing write tid 3 to smithi16534841-47 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: finishing write tid 4 to smithi16534841-47 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: oid 47 updating version 0 to 698 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: oid 47 version 698 is already newer than 697 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: oid 47 updating version 698 to 699 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 699 (ObjNum 421 snap 138 seq_num 421) dirty exists 2024-09-10T16:15:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:1163: left oid 47 (ObjNum 421 snap 138 seq_num 421) 2024-09-10T16:15:11.671 INFO:tasks.rados.rados.0.smithi165.stdout:1165: expect (ObjNum 408 snap 132 seq_num 408) 2024-09-10T16:15:11.679 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1430174 2024-09-10T16:15:11.766 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2201607 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1160: done (4 left) 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1162: done (3 left) 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1163: done (2 left) 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1165: done (1 left) 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1166: done (0 left) 2024-09-10T16:15:11.782 INFO:tasks.rados.rados.0.smithi165.stdout:1167: read oid 30 snap -1 2024-09-10T16:15:11.783 INFO:tasks.rados.rados.0.smithi165.stdout:1167: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:15:11.783 INFO:tasks.rados.rados.0.smithi165.stdout:1168: write (excl) oid 20 current snap is 138 2024-09-10T16:15:11.783 INFO:tasks.rados.rados.0.smithi165.stdout:1168: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-10T16:15:11.789 INFO:tasks.rados.rados.0.smithi165.stdout:1168: writing smithi16534841-20 from 709361 to 1339103 tid 1 2024-09-10T16:15:11.796 INFO:tasks.rados.rados.0.smithi165.stdout:1168: writing smithi16534841-20 from 1836589 to 2376405 tid 2 2024-09-10T16:15:11.799 INFO:tasks.rados.rados.0.smithi165.stdout:1168: writing smithi16534841-20 from 3033220 to 3109361 tid 3 2024-09-10T16:15:11.799 INFO:tasks.rados.rados.0.smithi165.stdout:1169: write (excl) oid 7 current snap is 138 2024-09-10T16:15:11.799 INFO:tasks.rados.rados.0.smithi165.stdout:1169: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-10T16:15:11.807 INFO:tasks.rados.rados.0.smithi165.stdout:1169: writing smithi16534841-7 from 726168 to 1431159 tid 1 2024-09-10T16:15:11.813 INFO:tasks.rados.rados.0.smithi165.stdout:1169: writing smithi16534841-7 from 1895071 to 2578545 tid 2 2024-09-10T16:15:11.814 INFO:tasks.rados.rados.0.smithi165.stdout:1170: write oid 8 current snap is 138 2024-09-10T16:15:11.814 INFO:tasks.rados.rados.0.smithi165.stdout:1170: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-10T16:15:11.822 INFO:tasks.rados.rados.0.smithi165.stdout:1170: writing smithi16534841-8 from 742975 to 1523215 tid 1 2024-09-10T16:15:11.826 INFO:tasks.rados.rados.0.smithi165.stdout:1170: writing smithi16534841-8 from 1953553 to 2380685 tid 2 2024-09-10T16:15:11.831 INFO:tasks.rados.rados.0.smithi165.stdout:1170: writing smithi16534841-8 from 2971713 to 3142975 tid 3 2024-09-10T16:15:11.831 INFO:tasks.rados.rados.0.smithi165.stdout:1171: snap_remove snap 137 2024-09-10T16:15:12.166 INFO:tasks.rados.rados.0.smithi165.stdout:1171: done (4 left) 2024-09-10T16:15:12.166 INFO:tasks.rados.rados.0.smithi165.stdout:1172: snap_create 2024-09-10T16:15:12.166 INFO:tasks.rados.rados.0.smithi165.stdout:1167: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:15:12.166 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:12.349 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3350282 2024-09-10T16:15:12.372 INFO:tasks.rados.rados.0.smithi165.stdout:1168: finishing write tid 1 to smithi16534841-20 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: finishing write tid 2 to smithi16534841-20 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: finishing write tid 3 to smithi16534841-20 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: finishing write tid 4 to smithi16534841-20 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: finishing write tid 5 to smithi16534841-20 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: oid 20 updating version 0 to 727 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: oid 20 version 727 is already newer than 726 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: oid 20 version 727 is already newer than 725 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: oid 20 updating version 727 to 728 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 728 (ObjNum 423 snap 138 seq_num 423) dirty exists 2024-09-10T16:15:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:1168: left oid 20 (ObjNum 423 snap 138 seq_num 423) 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: finishing write tid 1 to smithi16534841-7 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: finishing write tid 2 to smithi16534841-7 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: finishing write tid 3 to smithi16534841-7 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: finishing write tid 4 to smithi16534841-7 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: oid 7 updating version 0 to 922 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: oid 7 updating version 922 to 923 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: oid 7 updating version 923 to 924 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 924 (ObjNum 424 snap 138 seq_num 424) dirty exists 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1169: left oid 7 (ObjNum 424 snap 138 seq_num 424) 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1170: finishing write tid 1 to smithi16534841-8 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1170: finishing write tid 2 to smithi16534841-8 2024-09-10T16:15:12.374 INFO:tasks.rados.rados.0.smithi165.stdout:1170: finishing write tid 3 to smithi16534841-8 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: finishing write tid 4 to smithi16534841-8 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: finishing write tid 5 to smithi16534841-8 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: oid 8 updating version 0 to 705 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: oid 8 updating version 705 to 708 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: oid 8 version 708 is already newer than 707 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: oid 8 version 708 is already newer than 706 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 708 (ObjNum 425 snap 138 seq_num 425) dirty exists 2024-09-10T16:15:12.375 INFO:tasks.rados.rados.0.smithi165.stdout:1170: left oid 8 (ObjNum 425 snap 138 seq_num 425) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1167: done (4 left) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1168: done (3 left) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1169: done (2 left) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1170: done (1 left) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1172: done (0 left) 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:1173: rollback oid 30 current snap is 139 2024-09-10T16:15:13.170 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 131 2024-09-10T16:15:13.171 INFO:tasks.rados.rados.0.smithi165.stdout:1174: read oid 32 snap -1 2024-09-10T16:15:13.171 INFO:tasks.rados.rados.0.smithi165.stdout:1174: expect (ObjNum 407 snap 132 seq_num 407) 2024-09-10T16:15:13.171 INFO:tasks.rados.rados.0.smithi165.stdout:1175: write oid 9 current snap is 139 2024-09-10T16:15:13.171 INFO:tasks.rados.rados.0.smithi165.stdout:1175: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-10T16:15:13.177 INFO:tasks.rados.rados.0.smithi165.stdout:1175: writing smithi16534841-9 from 759782 to 1331624 tid 1 2024-09-10T16:15:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:1175: writing smithi16534841-9 from 2012035 to 2699178 tid 2 2024-09-10T16:15:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:1176: snap_create 2024-09-10T16:15:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:1173: finishing rollback tid 0 to smithi16534841-30 2024-09-10T16:15:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:1173: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:15:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:1175: finishing write tid 1 to smithi16534841-9 2024-09-10T16:15:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:1173: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:15:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 930 (ObjNum 338 snap 98 seq_num 338) dirty exists 2024-09-10T16:15:13.188 INFO:tasks.rados.rados.0.smithi165.stdout:1175: finishing write tid 2 to smithi16534841-9 2024-09-10T16:15:13.189 INFO:tasks.rados.rados.0.smithi165.stdout:1174: expect (ObjNum 407 snap 132 seq_num 407) 2024-09-10T16:15:13.251 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:13.313 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 248697 2024-09-10T16:15:13.373 INFO:tasks.rados.rados.0.smithi165.stdout:1175: finishing write tid 3 to smithi16534841-9 2024-09-10T16:15:13.373 INFO:tasks.rados.rados.0.smithi165.stdout:1175: finishing write tid 4 to smithi16534841-9 2024-09-10T16:15:13.373 INFO:tasks.rados.rados.0.smithi165.stdout:1175: oid 9 updating version 0 to 330 2024-09-10T16:15:13.373 INFO:tasks.rados.rados.0.smithi165.stdout:1175: oid 9 updating version 330 to 332 2024-09-10T16:15:13.374 INFO:tasks.rados.rados.0.smithi165.stdout:1175: oid 9 version 332 is already newer than 331 2024-09-10T16:15:13.374 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 332 (ObjNum 426 snap 139 seq_num 426) dirty exists 2024-09-10T16:15:13.374 INFO:tasks.rados.rados.0.smithi165.stdout:1175: left oid 9 (ObjNum 426 snap 139 seq_num 426) 2024-09-10T16:15:14.178 INFO:tasks.rados.rados.0.smithi165.stdout:1173: done (3 left) 2024-09-10T16:15:14.178 INFO:tasks.rados.rados.0.smithi165.stdout:1174: done (2 left) 2024-09-10T16:15:14.178 INFO:tasks.rados.rados.0.smithi165.stdout:1175: done (1 left) 2024-09-10T16:15:14.178 INFO:tasks.rados.rados.0.smithi165.stdout:1176: done (0 left) 2024-09-10T16:15:14.178 INFO:tasks.rados.rados.0.smithi165.stdout:1177: snap_remove snap 125 2024-09-10T16:15:15.175 INFO:tasks.rados.rados.0.smithi165.stdout:1177: done (0 left) 2024-09-10T16:15:15.175 INFO:tasks.rados.rados.0.smithi165.stdout:1178: read oid 22 snap -1 2024-09-10T16:15:15.175 INFO:tasks.rados.rados.0.smithi165.stdout:1178: expect (ObjNum 409 snap 132 seq_num 409) 2024-09-10T16:15:15.175 INFO:tasks.rados.rados.0.smithi165.stdout:1179: snap_remove snap 110 2024-09-10T16:15:16.183 INFO:tasks.rados.rados.0.smithi165.stdout:1179: done (1 left) 2024-09-10T16:15:16.183 INFO:tasks.rados.rados.0.smithi165.stdout:1180: write (excl) oid 27 current snap is 140 2024-09-10T16:15:16.183 INFO:tasks.rados.rados.0.smithi165.stdout:1180: seq_num 427 ranges {776589=647091,2070517=430801,3026559=150030} 2024-09-10T16:15:16.191 INFO:tasks.rados.rados.0.smithi165.stdout:1180: writing smithi16534841-27 from 776589 to 1423680 tid 1 2024-09-10T16:15:16.197 INFO:tasks.rados.rados.0.smithi165.stdout:1180: writing smithi16534841-27 from 2070517 to 2501318 tid 2 2024-09-10T16:15:16.201 INFO:tasks.rados.rados.0.smithi165.stdout:1180: writing smithi16534841-27 from 3026559 to 3176589 tid 3 2024-09-10T16:15:16.201 INFO:tasks.rados.rados.0.smithi165.stdout:1181: read oid 4 snap -1 2024-09-10T16:15:16.202 INFO:tasks.rados.rados.0.smithi165.stdout:1181: expect deleted 2024-09-10T16:15:16.202 INFO:tasks.rados.rados.0.smithi165.stdout:1182: snap_create 2024-09-10T16:15:16.202 INFO:tasks.rados.rados.0.smithi165.stdout:1178: expect (ObjNum 409 snap 132 seq_num 409) 2024-09-10T16:15:16.202 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 70726 2024-09-10T16:15:16.251 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2872411 2024-09-10T16:15:16.258 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2517335 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: finishing write tid 1 to smithi16534841-27 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: finishing write tid 2 to smithi16534841-27 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: finishing write tid 3 to smithi16534841-27 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: finishing write tid 4 to smithi16534841-27 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: finishing write tid 5 to smithi16534841-27 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: oid 27 updating version 0 to 720 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: oid 27 updating version 720 to 727 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: oid 27 version 727 is already newer than 726 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:1180: oid 27 version 727 is already newer than 725 2024-09-10T16:15:16.267 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 727 (ObjNum 427 snap 140 seq_num 427) dirty exists 2024-09-10T16:15:16.268 INFO:tasks.rados.rados.0.smithi165.stdout:1180: left oid 27 (ObjNum 427 snap 140 seq_num 427) 2024-09-10T16:15:17.167 INFO:tasks.rados.rados.0.smithi165.stdout:1178: done (3 left) 2024-09-10T16:15:17.168 INFO:tasks.rados.rados.0.smithi165.stdout:1180: done (2 left) 2024-09-10T16:15:17.168 INFO:tasks.rados.rados.0.smithi165.stdout:1181: done (1 left) 2024-09-10T16:15:17.168 INFO:tasks.rados.rados.0.smithi165.stdout:1182: done (0 left) 2024-09-10T16:15:17.168 INFO:tasks.rados.rados.0.smithi165.stdout:1183: delete oid 2 current snap is 141 2024-09-10T16:15:17.171 INFO:tasks.rados.rados.0.smithi165.stdout:1183: done (0 left) 2024-09-10T16:15:17.171 INFO:tasks.rados.rados.0.smithi165.stdout:1184: snap_remove snap 116 2024-09-10T16:15:18.173 INFO:tasks.rados.rados.0.smithi165.stdout:1184: done (0 left) 2024-09-10T16:15:18.173 INFO:tasks.rados.rados.0.smithi165.stdout:1185: snap_remove snap 129 2024-09-10T16:15:19.175 INFO:tasks.rados.rados.0.smithi165.stdout:1185: done (0 left) 2024-09-10T16:15:19.175 INFO:tasks.rados.rados.0.smithi165.stdout:1186: write oid 34 current snap is 141 2024-09-10T16:15:19.175 INFO:tasks.rados.rados.0.smithi165.stdout:1186: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-10T16:15:19.181 INFO:tasks.rados.rados.0.smithi165.stdout:1186: writing smithi16534841-34 from 410203 to 1207792 tid 1 2024-09-10T16:15:19.187 INFO:tasks.rados.rados.0.smithi165.stdout:1186: writing smithi16534841-34 from 1787481 to 2221951 tid 2 2024-09-10T16:15:19.193 INFO:tasks.rados.rados.0.smithi165.stdout:1186: writing smithi16534841-34 from 2681405 to 3210203 tid 3 2024-09-10T16:15:19.193 INFO:tasks.rados.rados.0.smithi165.stdout:1187: write oid 35 current snap is 141 2024-09-10T16:15:19.193 INFO:tasks.rados.rados.0.smithi165.stdout:1187: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2024-09-10T16:15:19.198 INFO:tasks.rados.rados.0.smithi165.stdout:1187: writing smithi16534841-35 from 427010 to 899848 tid 1 2024-09-10T16:15:19.204 INFO:tasks.rados.rados.0.smithi165.stdout:1187: writing smithi16534841-35 from 1329610 to 2024091 tid 2 2024-09-10T16:15:19.212 INFO:tasks.rados.rados.0.smithi165.stdout:1187: writing smithi16534841-35 from 2708828 to 3227010 tid 3 2024-09-10T16:15:19.212 INFO:tasks.rados.rados.0.smithi165.stdout:1188: copy_from oid 25 from oid 45 current snap is 141 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: finishing write tid 1 to smithi16534841-34 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: finishing write tid 2 to smithi16534841-34 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: finishing write tid 3 to smithi16534841-34 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: finishing write tid 4 to smithi16534841-34 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: finishing write tid 5 to smithi16534841-34 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: oid 34 updating version 0 to 558 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: oid 34 version 558 is already newer than 557 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: oid 34 updating version 558 to 560 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: oid 34 version 560 is already newer than 559 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 560 (ObjNum 429 snap 141 seq_num 429) dirty exists 2024-09-10T16:15:19.213 INFO:tasks.rados.rados.0.smithi165.stdout:1186: left oid 34 (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:15:19.214 INFO:tasks.rados.rados.0.smithi165.stdout:1187: finishing write tid 1 to smithi16534841-35 2024-09-10T16:15:19.214 INFO:tasks.rados.rados.0.smithi165.stdout:1187: finishing write tid 2 to smithi16534841-35 2024-09-10T16:15:19.214 INFO:tasks.rados.rados.0.smithi165.stdout:1186: done (2 left) 2024-09-10T16:15:19.214 INFO:tasks.rados.rados.0.smithi165.stdout:1189: delete oid 40 current snap is 141 2024-09-10T16:15:19.216 INFO:tasks.rados.rados.0.smithi165.stdout:1187: finishing write tid 3 to smithi16534841-35 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: finishing write tid 4 to smithi16534841-35 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: finishing write tid 5 to smithi16534841-35 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: oid 35 updating version 0 to 630 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: oid 35 version 630 is already newer than 628 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: oid 35 version 630 is already newer than 629 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: oid 35 updating version 630 to 631 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 631 (ObjNum 430 snap 141 seq_num 430) dirty exists 2024-09-10T16:15:19.218 INFO:tasks.rados.rados.0.smithi165.stdout:1187: left oid 35 (ObjNum 430 snap 141 seq_num 430) 2024-09-10T16:15:19.219 INFO:tasks.rados.rados.0.smithi165.stdout:1187: done (2 left) 2024-09-10T16:15:19.219 INFO:tasks.rados.rados.0.smithi165.stdout:1189: done (1 left) 2024-09-10T16:15:19.219 INFO:tasks.rados.rados.0.smithi165.stdout:1190: write oid 21 current snap is 141 2024-09-10T16:15:19.219 INFO:tasks.rados.rados.0.smithi165.stdout:1190: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2024-09-10T16:15:19.226 INFO:tasks.rados.rados.0.smithi165.stdout:1190: writing smithi16534841-21 from 460624 to 1083960 tid 1 2024-09-10T16:15:19.234 INFO:tasks.rados.rados.0.smithi165.stdout:1190: writing smithi16534841-21 from 1846574 to 2544724 tid 2 2024-09-10T16:15:19.238 INFO:tasks.rados.rados.0.smithi165.stdout:1190: writing smithi16534841-21 from 3163674 to 3260624 tid 3 2024-09-10T16:15:19.238 INFO:tasks.rados.rados.0.smithi165.stdout:1191: read oid 44 snap -1 2024-09-10T16:15:19.238 INFO:tasks.rados.rados.0.smithi165.stdout:1191: expect deleted 2024-09-10T16:15:19.238 INFO:tasks.rados.rados.0.smithi165.stdout:1192: write oid 28 current snap is 141 2024-09-10T16:15:19.238 INFO:tasks.rados.rados.0.smithi165.stdout:1192: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-10T16:15:19.245 INFO:tasks.rados.rados.0.smithi165.stdout:1192: writing smithi16534841-28 from 477431 to 1176016 tid 1 2024-09-10T16:15:19.252 INFO:tasks.rados.rados.0.smithi165.stdout:1192: writing smithi16534841-28 from 1905056 to 2346864 tid 2 2024-09-10T16:15:19.252 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:15:19.251+0000 7f1d6e738640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 79 after trimming was started 2024-09-10T16:15:19.257 INFO:tasks.rados.rados.0.smithi165.stdout:1192: writing smithi16534841-28 from 3074744 to 3277431 tid 3 2024-09-10T16:15:19.258 INFO:tasks.rados.rados.0.smithi165.stdout:1193: snap_remove snap 133 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:1193: done (4 left) 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:1194: copy_from oid 27 from oid 5 current snap is 141 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:1188: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:1188: finishing copy_from to smithi16534841-25 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 874 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:15:20.182 INFO:tasks.rados.rados.0.smithi165.stdout:1190: finishing write tid 1 to smithi16534841-21 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: finishing write tid 2 to smithi16534841-21 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: finishing write tid 3 to smithi16534841-21 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: finishing write tid 4 to smithi16534841-21 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: finishing write tid 5 to smithi16534841-21 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: oid 21 updating version 0 to 878 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: oid 21 version 878 is already newer than 876 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: oid 21 version 878 is already newer than 877 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: oid 21 version 878 is already newer than 875 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 878 (ObjNum 432 snap 141 seq_num 432) dirty exists 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1190: left oid 21 (ObjNum 432 snap 141 seq_num 432) 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1192: finishing write tid 1 to smithi16534841-28 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1192: finishing write tid 2 to smithi16534841-28 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1192: finishing write tid 3 to smithi16534841-28 2024-09-10T16:15:20.183 INFO:tasks.rados.rados.0.smithi165.stdout:1192: finishing write tid 4 to smithi16534841-28 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: finishing write tid 5 to smithi16534841-28 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: oid 28 updating version 0 to 344 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: oid 28 version 344 is already newer than 343 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: oid 28 updating version 344 to 346 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: oid 28 version 346 is already newer than 345 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 346 (ObjNum 433 snap 141 seq_num 433) dirty exists 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: left oid 28 (ObjNum 433 snap 141 seq_num 433) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1188: done (4 left) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1190: done (3 left) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1191: done (2 left) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1192: done (1 left) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1195: read oid 22 snap -1 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1195: expect (ObjNum 409 snap 132 seq_num 409) 2024-09-10T16:15:20.184 INFO:tasks.rados.rados.0.smithi165.stdout:1196: copy_from oid 21 from oid 48 current snap is 141 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1197: read oid 15 snap 136 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1197: expect (ObjNum 403 snap 130 seq_num 403) 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1198: rollback oid 14 current snap is 141 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 112 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1199: copy_from oid 9 from oid 33 current snap is 141 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1200: write oid 16 current snap is 141 2024-09-10T16:15:20.185 INFO:tasks.rados.rados.0.smithi165.stdout:1200: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-10T16:15:20.188 INFO:tasks.rados.rados.0.smithi165.stdout:1200: writing smithi16534841-16 from 494238 to 984425 tid 1 2024-09-10T16:15:20.197 INFO:tasks.rados.rados.0.smithi165.stdout:1200: writing smithi16534841-16 from 1563538 to 2265357 tid 2 2024-09-10T16:15:20.203 INFO:tasks.rados.rados.0.smithi165.stdout:1200: writing smithi16534841-16 from 2818520 to 3294238 tid 3 2024-09-10T16:15:20.203 INFO:tasks.rados.rados.0.smithi165.stdout:1201: write oid 26 current snap is 141 2024-09-10T16:15:20.203 INFO:tasks.rados.rados.0.smithi165.stdout:1196: finishing copy_from to smithi16534841-21 2024-09-10T16:15:20.203 INFO:tasks.rados.rados.0.smithi165.stdout:1196: got expected ENOENT (src dne) 2024-09-10T16:15:20.203 INFO:tasks.rados.rados.0.smithi165.stdout:1201: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-10T16:15:20.209 INFO:tasks.rados.rados.0.smithi165.stdout:1201: writing smithi16534841-26 from 511045 to 1076481 tid 1 2024-09-10T16:15:20.214 INFO:tasks.rados.rados.0.smithi165.stdout:1201: writing smithi16534841-26 from 1622020 to 2067497 tid 2 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1201: writing smithi16534841-26 from 2729590 to 3286445 tid 3 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1202: read oid 25 snap -1 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1202: expect (ObjNum 415 snap 136 seq_num 415) 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1203: write oid 41 current snap is 141 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1196: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1199: finishing copy_from to smithi16534841-9 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1199: got expected ENOENT (src dne) 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1199: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1198: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:1198: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:15:20.222 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 335 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:15:20.223 INFO:tasks.rados.rados.0.smithi165.stdout:1195: expect (ObjNum 409 snap 132 seq_num 409) 2024-09-10T16:15:20.224 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1299012 2024-09-10T16:15:20.247 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:20.305 INFO:tasks.rados.rados.0.smithi165.stdout:1197: expect (ObjNum 403 snap 130 seq_num 403) 2024-09-10T16:15:20.305 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:20.373 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1043149 2024-09-10T16:15:20.410 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1200: finishing write tid 1 to smithi16534841-16 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1194: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1200: finishing write tid 2 to smithi16534841-16 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1200: finishing write tid 3 to smithi16534841-16 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1194: finishing copy_from to smithi16534841-27 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 612 (ObjNum 408 snap 132 seq_num 408) dirty exists 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1200: finishing write tid 4 to smithi16534841-16 2024-09-10T16:15:20.467 INFO:tasks.rados.rados.0.smithi165.stdout:1203: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-10T16:15:20.473 INFO:tasks.rados.rados.0.smithi165.stdout:1203: writing smithi16534841-41 from 527852 to 1168537 tid 1 2024-09-10T16:15:20.479 INFO:tasks.rados.rados.0.smithi165.stdout:1203: writing smithi16534841-41 from 1680502 to 2269637 tid 2 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1203: writing smithi16534841-41 from 2757013 to 3327852 tid 3 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1194: done (9 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1195: done (8 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1196: done (7 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1197: done (6 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1198: done (5 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1199: done (4 left) 2024-09-10T16:15:20.485 INFO:tasks.rados.rados.0.smithi165.stdout:1204: snap_remove snap 139 2024-09-10T16:15:21.185 INFO:tasks.rados.rados.0.smithi165.stdout:1204: done (4 left) 2024-09-10T16:15:21.185 INFO:tasks.rados.rados.0.smithi165.stdout:1205: read oid 19 snap -1 2024-09-10T16:15:21.185 INFO:tasks.rados.rados.0.smithi165.stdout:1205: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:15:21.185 INFO:tasks.rados.rados.0.smithi165.stdout:1206: write oid 40 current snap is 141 2024-09-10T16:15:21.185 INFO:tasks.rados.rados.0.smithi165.stdout:1206: seq_num 437 ranges {544659=715934,1738984=449146,2784436=560223} 2024-09-10T16:15:21.191 INFO:tasks.rados.rados.0.smithi165.stdout:1206: writing smithi16534841-40 from 544659 to 1260593 tid 1 2024-09-10T16:15:21.197 INFO:tasks.rados.rados.0.smithi165.stdout:1206: writing smithi16534841-40 from 1738984 to 2188130 tid 2 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1206: writing smithi16534841-40 from 2784436 to 3344659 tid 3 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1207: rollback oid 13 current snap is 141 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 136 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: finishing write tid 5 to smithi16534841-16 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: oid 16 updating version 0 to 963 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: oid 16 version 963 is already newer than 962 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: oid 16 version 963 is already newer than 961 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: oid 16 updating version 963 to 964 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 964 (ObjNum 434 snap 141 seq_num 434) dirty exists 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1200: left oid 16 (ObjNum 434 snap 141 seq_num 434) 2024-09-10T16:15:21.204 INFO:tasks.rados.rados.0.smithi165.stdout:1201: finishing write tid 1 to smithi16534841-26 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: finishing write tid 2 to smithi16534841-26 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: finishing write tid 3 to smithi16534841-26 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: finishing write tid 4 to smithi16534841-26 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: finishing write tid 5 to smithi16534841-26 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: oid 26 updating version 0 to 967 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: oid 26 version 967 is already newer than 966 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: oid 26 updating version 967 to 968 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: oid 26 version 968 is already newer than 965 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 968 (ObjNum 435 snap 141 seq_num 435) dirty exists 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1201: left oid 26 (ObjNum 435 snap 141 seq_num 435) 2024-09-10T16:15:21.205 INFO:tasks.rados.rados.0.smithi165.stdout:1202: expect (ObjNum 415 snap 136 seq_num 415) 2024-09-10T16:15:21.269 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1906014 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: finishing write tid 1 to smithi16534841-41 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: finishing write tid 2 to smithi16534841-41 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: finishing write tid 3 to smithi16534841-41 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: finishing write tid 4 to smithi16534841-41 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: finishing write tid 5 to smithi16534841-41 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: oid 41 updating version 0 to 633 2024-09-10T16:15:21.281 INFO:tasks.rados.rados.0.smithi165.stdout:1203: oid 41 version 633 is already newer than 632 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:1203: oid 41 version 633 is already newer than 631 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:1203: oid 41 updating version 633 to 634 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 634 (ObjNum 436 snap 141 seq_num 436) dirty exists 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:1203: left oid 41 (ObjNum 436 snap 141 seq_num 436) 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:1205: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:15:21.282 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:21.433 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3633055 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: finishing write tid 1 to smithi16534841-40 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: finishing write tid 2 to smithi16534841-40 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1207: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 627 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: finishing write tid 3 to smithi16534841-40 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: finishing write tid 4 to smithi16534841-40 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: finishing write tid 5 to smithi16534841-40 2024-09-10T16:15:21.521 INFO:tasks.rados.rados.0.smithi165.stdout:1206: oid 40 updating version 0 to 336 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1206: oid 40 updating version 336 to 339 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1206: oid 40 version 339 is already newer than 337 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1206: oid 40 version 339 is already newer than 338 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 339 (ObjNum 437 snap 141 seq_num 437) dirty exists 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1206: left oid 40 (ObjNum 437 snap 141 seq_num 437) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1200: done (6 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1201: done (5 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1202: done (4 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1203: done (3 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1205: done (2 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1206: done (1 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1207: done (0 left) 2024-09-10T16:15:21.522 INFO:tasks.rados.rados.0.smithi165.stdout:1208: snap_remove snap 136 2024-09-10T16:15:22.169 INFO:tasks.rados.rados.0.smithi165.stdout:1208: done (0 left) 2024-09-10T16:15:22.169 INFO:tasks.rados.rados.0.smithi165.stdout:1209: snap_create 2024-09-10T16:15:23.168 INFO:tasks.rados.rados.0.smithi165.stdout:1209: done (0 left) 2024-09-10T16:15:23.168 INFO:tasks.rados.rados.0.smithi165.stdout:1210: write oid 47 current snap is 142 2024-09-10T16:15:23.168 INFO:tasks.rados.rados.0.smithi165.stdout:1210: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-10T16:15:23.175 INFO:tasks.rados.rados.0.smithi165.stdout:1210: writing smithi16534841-47 from 561466 to 1352649 tid 1 2024-09-10T16:15:23.183 INFO:tasks.rados.rados.0.smithi165.stdout:1210: writing smithi16534841-47 from 2081113 to 2673917 tid 2 2024-09-10T16:15:23.187 INFO:tasks.rados.rados.0.smithi165.stdout:1210: writing smithi16534841-47 from 3095506 to 3361466 tid 3 2024-09-10T16:15:23.188 INFO:tasks.rados.rados.0.smithi165.stdout:1211: write (excl) oid 41 current snap is 142 2024-09-10T16:15:23.188 INFO:tasks.rados.rados.0.smithi165.stdout:1211: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-10T16:15:23.193 INFO:tasks.rados.rados.0.smithi165.stdout:1211: writing smithi16534841-41 from 578273 to 1044705 tid 1 2024-09-10T16:15:23.199 INFO:tasks.rados.rados.0.smithi165.stdout:1211: writing smithi16534841-41 from 1739595 to 2192410 tid 2 2024-09-10T16:15:23.206 INFO:tasks.rados.rados.0.smithi165.stdout:1211: writing smithi16534841-41 from 2722929 to 3378273 tid 3 2024-09-10T16:15:23.206 INFO:tasks.rados.rados.0.smithi165.stdout:1212: copy_from oid 1 from oid 45 current snap is 142 2024-09-10T16:15:23.206 INFO:tasks.rados.rados.0.smithi165.stdout:1210: finishing write tid 1 to smithi16534841-47 2024-09-10T16:15:23.207 INFO:tasks.rados.rados.0.smithi165.stdout:1213: write (excl) oid 38 current snap is 142 2024-09-10T16:15:23.207 INFO:tasks.rados.rados.0.smithi165.stdout:1213: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2024-09-10T16:15:23.212 INFO:tasks.rados.rados.0.smithi165.stdout:1213: writing smithi16534841-38 from 595080 to 1136761 tid 1 2024-09-10T16:15:23.221 INFO:tasks.rados.rados.0.smithi165.stdout:1213: writing smithi16534841-38 from 1798077 to 2394550 tid 2 2024-09-10T16:15:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:1213: writing smithi16534841-38 from 3150352 to 3395080 tid 3 2024-09-10T16:15:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:1214: snap_remove snap 127 2024-09-10T16:15:24.173 INFO:tasks.rados.rados.0.smithi165.stdout:1214: done (4 left) 2024-09-10T16:15:24.173 INFO:tasks.rados.rados.0.smithi165.stdout:1215: copy_from oid 32 from oid 11 current snap is 142 2024-09-10T16:15:24.173 INFO:tasks.rados.rados.0.smithi165.stdout:1210: finishing write tid 2 to smithi16534841-47 2024-09-10T16:15:24.173 INFO:tasks.rados.rados.0.smithi165.stdout:1210: finishing write tid 3 to smithi16534841-47 2024-09-10T16:15:24.173 INFO:tasks.rados.rados.0.smithi165.stdout:1210: finishing write tid 4 to smithi16534841-47 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: finishing write tid 5 to smithi16534841-47 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: oid 47 updating version 0 to 775 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: oid 47 updating version 775 to 776 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: oid 47 updating version 776 to 777 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: oid 47 updating version 777 to 778 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 778 (ObjNum 438 snap 142 seq_num 438) dirty exists 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1210: left oid 47 (ObjNum 438 snap 142 seq_num 438) 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1211: finishing write tid 1 to smithi16534841-41 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1211: finishing write tid 2 to smithi16534841-41 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1211: finishing write tid 3 to smithi16534841-41 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1211: finishing write tid 4 to smithi16534841-41 2024-09-10T16:15:24.174 INFO:tasks.rados.rados.0.smithi165.stdout:1211: finishing write tid 5 to smithi16534841-41 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1211: oid 41 updating version 0 to 643 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1211: oid 41 version 643 is already newer than 640 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1211: oid 41 version 643 is already newer than 641 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1211: oid 41 version 643 is already newer than 642 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 643 (ObjNum 439 snap 142 seq_num 439) dirty exists 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1211: left oid 41 (ObjNum 439 snap 142 seq_num 439) 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1213: finishing write tid 1 to smithi16534841-38 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1212: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1212: finishing copy_from to smithi16534841-1 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 874 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1213: finishing write tid 2 to smithi16534841-38 2024-09-10T16:15:24.175 INFO:tasks.rados.rados.0.smithi165.stdout:1213: finishing write tid 3 to smithi16534841-38 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: finishing write tid 4 to smithi16534841-38 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: finishing write tid 5 to smithi16534841-38 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: oid 38 updating version 0 to 275 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: oid 38 version 275 is already newer than 273 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: oid 38 version 275 is already newer than 274 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: oid 38 updating version 275 to 276 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 276 (ObjNum 440 snap 142 seq_num 440) dirty exists 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: left oid 38 (ObjNum 440 snap 142 seq_num 440) 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1210: done (4 left) 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1211: done (3 left) 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1212: done (2 left) 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1213: done (1 left) 2024-09-10T16:15:24.176 INFO:tasks.rados.rados.0.smithi165.stdout:1216: read oid 49 snap 132 2024-09-10T16:15:24.177 INFO:tasks.rados.rados.0.smithi165.stdout:1216: expect (ObjNum 372 snap 108 seq_num 372) 2024-09-10T16:15:24.177 INFO:tasks.rados.rados.0.smithi165.stdout:1217: rollback oid 28 current snap is 142 2024-09-10T16:15:24.177 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 28 to 121 2024-09-10T16:15:24.177 INFO:tasks.rados.rados.0.smithi165.stdout:1218: write (excl) oid 10 current snap is 142 2024-09-10T16:15:24.177 INFO:tasks.rados.rados.0.smithi165.stdout:1218: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-10T16:15:24.180 INFO:tasks.rados.rados.0.smithi165.stdout:1218: writing smithi16534841-10 from 611887 to 1345170 tid 1 2024-09-10T16:15:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1218: writing smithi16534841-10 from 1972912 to 2429396 tid 2 2024-09-10T16:15:24.192 INFO:tasks.rados.rados.0.smithi165.stdout:1218: writing smithi16534841-10 from 2894128 to 3411887 tid 3 2024-09-10T16:15:24.192 INFO:tasks.rados.rados.0.smithi165.stdout:1219: delete oid 1 current snap is 142 2024-09-10T16:15:24.192 INFO:tasks.rados.rados.0.smithi165.stdout:1216: expect (ObjNum 372 snap 108 seq_num 372) 2024-09-10T16:15:24.210 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:24.260 INFO:tasks.rados.rados.0.smithi165.stdout:1215: finishing copy_from to smithi16534841-32 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1215: got expected ENOENT (src dne) 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1215: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1217: finishing rollback tid 0 to smithi16534841-28 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1217: finishing rollback tid 1 to smithi16534841-28 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1217: finishing rollback tid 2 to smithi16534841-28 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 349 (ObjNum 373 snap 109 seq_num 373) dirty exists 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: finishing write tid 1 to smithi16534841-10 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: finishing write tid 2 to smithi16534841-10 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: finishing write tid 3 to smithi16534841-10 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: finishing write tid 4 to smithi16534841-10 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: finishing write tid 5 to smithi16534841-10 2024-09-10T16:15:24.261 INFO:tasks.rados.rados.0.smithi165.stdout:1218: oid 10 updating version 0 to 342 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1218: oid 10 version 342 is already newer than 341 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1218: oid 10 updating version 342 to 344 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1218: oid 10 version 344 is already newer than 343 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 344 (ObjNum 441 snap 142 seq_num 441) dirty exists 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1218: left oid 10 (ObjNum 441 snap 142 seq_num 441) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1215: done (4 left) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1216: done (3 left) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1217: done (2 left) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1218: done (1 left) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1219: done (0 left) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1220: read oid 28 snap -1 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1220: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-10T16:15:24.262 INFO:tasks.rados.rados.0.smithi165.stdout:1221: write oid 26 current snap is 142 2024-09-10T16:15:24.263 INFO:tasks.rados.rados.0.smithi165.stdout:1221: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-10T16:15:24.266 INFO:tasks.rados.rados.0.smithi165.stdout:1221: writing smithi16534841-26 from 645501 to 1129282 tid 1 2024-09-10T16:15:24.271 INFO:tasks.rados.rados.0.smithi165.stdout:1221: writing smithi16534841-26 from 1573523 to 2033676 tid 2 2024-09-10T16:15:24.278 INFO:tasks.rados.rados.0.smithi165.stdout:1221: writing smithi16534841-26 from 2548974 to 3312358 tid 3 2024-09-10T16:15:24.278 INFO:tasks.rados.rados.0.smithi165.stdout:1222: snap_create 2024-09-10T16:15:24.279 INFO:tasks.rados.rados.0.smithi165.stdout:1220: expect (ObjNum 373 snap 109 seq_num 373) 2024-09-10T16:15:24.324 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 657360 2024-09-10T16:15:24.344 INFO:tasks.rados.rados.0.smithi165.stdout:1221: finishing write tid 1 to smithi16534841-26 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: finishing write tid 2 to smithi16534841-26 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: finishing write tid 3 to smithi16534841-26 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: finishing write tid 4 to smithi16534841-26 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: finishing write tid 5 to smithi16534841-26 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: oid 26 updating version 0 to 982 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: oid 26 updating version 982 to 983 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: oid 26 updating version 983 to 984 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: oid 26 updating version 984 to 985 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 985 (ObjNum 443 snap 142 seq_num 443) dirty exists 2024-09-10T16:15:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:1221: left oid 26 (ObjNum 443 snap 142 seq_num 443) 2024-09-10T16:15:25.174 INFO:tasks.rados.rados.0.smithi165.stdout:1220: done (2 left) 2024-09-10T16:15:25.174 INFO:tasks.rados.rados.0.smithi165.stdout:1221: done (1 left) 2024-09-10T16:15:25.174 INFO:tasks.rados.rados.0.smithi165.stdout:1222: done (0 left) 2024-09-10T16:15:25.174 INFO:tasks.rados.rados.0.smithi165.stdout:1223: snap_remove snap 132 2024-09-10T16:15:26.175 INFO:tasks.rados.rados.0.smithi165.stdout:1223: done (0 left) 2024-09-10T16:15:26.175 INFO:tasks.rados.rados.0.smithi165.stdout:1224: read oid 46 snap -1 2024-09-10T16:15:26.175 INFO:tasks.rados.rados.0.smithi165.stdout:1224: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:15:26.175 INFO:tasks.rados.rados.0.smithi165.stdout:1225: write oid 10 current snap is 143 2024-09-10T16:15:26.175 INFO:tasks.rados.rados.0.smithi165.stdout:1225: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-10T16:15:26.182 INFO:tasks.rados.rados.0.smithi165.stdout:1225: writing smithi16534841-10 from 662308 to 1221338 tid 1 2024-09-10T16:15:26.188 INFO:tasks.rados.rados.0.smithi165.stdout:1225: writing smithi16534841-10 from 1632005 to 2235816 tid 2 2024-09-10T16:15:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1225: writing smithi16534841-10 from 2860044 to 3462308 tid 3 2024-09-10T16:15:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1226: read oid 37 snap -1 2024-09-10T16:15:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1226: expect deleted 2024-09-10T16:15:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1227: write oid 48 current snap is 143 2024-09-10T16:15:26.196 INFO:tasks.rados.rados.0.smithi165.stdout:1227: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-10T16:15:26.203 INFO:tasks.rados.rados.0.smithi165.stdout:1227: writing smithi16534841-48 from 679115 to 1313394 tid 1 2024-09-10T16:15:26.209 INFO:tasks.rados.rados.0.smithi165.stdout:1227: writing smithi16534841-48 from 2090487 to 2554309 tid 2 2024-09-10T16:15:26.215 INFO:tasks.rados.rados.0.smithi165.stdout:1227: writing smithi16534841-48 from 3003820 to 3479115 tid 3 2024-09-10T16:15:26.215 INFO:tasks.rados.rados.0.smithi165.stdout:1228: copy_from oid 2 from oid 18 current snap is 143 2024-09-10T16:15:26.215 INFO:tasks.rados.rados.0.smithi165.stdout:1224: expect (ObjNum 329 snap 96 seq_num 329) 2024-09-10T16:15:26.215 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:26.289 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: finishing write tid 1 to smithi16534841-10 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: finishing write tid 2 to smithi16534841-10 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: finishing write tid 3 to smithi16534841-10 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: finishing write tid 4 to smithi16534841-10 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: finishing write tid 5 to smithi16534841-10 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: oid 10 updating version 0 to 351 2024-09-10T16:15:26.329 INFO:tasks.rados.rados.0.smithi165.stdout:1225: oid 10 version 351 is already newer than 350 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1225: oid 10 version 351 is already newer than 349 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1225: oid 10 version 351 is already newer than 348 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 351 (ObjNum 444 snap 143 seq_num 444) dirty exists 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1225: left oid 10 (ObjNum 444 snap 143 seq_num 444) 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1227: finishing write tid 1 to smithi16534841-48 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1227: finishing write tid 2 to smithi16534841-48 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1224: done (4 left) 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1225: done (3 left) 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1226: done (2 left) 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1229: write (excl) oid 3 current snap is 143 2024-09-10T16:15:26.330 INFO:tasks.rados.rados.0.smithi165.stdout:1229: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-10T16:15:26.336 INFO:tasks.rados.rados.0.smithi165.stdout:1229: writing smithi16534841-3 from 695922 to 1405450 tid 1 2024-09-10T16:15:26.342 INFO:tasks.rados.rados.0.smithi165.stdout:1229: writing smithi16534841-3 from 2032616 to 2640096 tid 2 2024-09-10T16:15:26.348 INFO:tasks.rados.rados.0.smithi165.stdout:1229: writing smithi16534841-3 from 3198537 to 3495922 tid 3 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1230: rollback oid 15 current snap is 143 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 142 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: finishing write tid 3 to smithi16534841-48 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: finishing write tid 4 to smithi16534841-48 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: finishing write tid 5 to smithi16534841-48 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: oid 48 updating version 0 to 995 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: oid 48 version 995 is already newer than 993 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: oid 48 version 995 is already newer than 994 2024-09-10T16:15:26.349 INFO:tasks.rados.rados.0.smithi165.stdout:1227: oid 48 version 995 is already newer than 992 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 995 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1227: left oid 48 (ObjNum 445 snap 143 seq_num 445) 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1228: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1228: finishing copy_from to smithi16534841-2 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1227: done (3 left) 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1228: done (2 left) 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1231: copy_from oid 35 from oid 20 current snap is 143 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1229: finishing write tid 1 to smithi16534841-3 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1229: finishing write tid 2 to smithi16534841-3 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1232: copy_from oid 31 from oid 33 current snap is 143 2024-09-10T16:15:26.350 INFO:tasks.rados.rados.0.smithi165.stdout:1233: snap_remove snap 121 2024-09-10T16:15:27.171 INFO:tasks.rados.rados.0.smithi165.stdout:1233: done (4 left) 2024-09-10T16:15:27.171 INFO:tasks.rados.rados.0.smithi165.stdout:1234: write (excl) oid 1 current snap is 143 2024-09-10T16:15:27.171 INFO:tasks.rados.rados.0.smithi165.stdout:1229: finishing write tid 3 to smithi16534841-3 2024-09-10T16:15:27.171 INFO:tasks.rados.rados.0.smithi165.stdout:1234: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-10T16:15:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1234: writing smithi16534841-1 from 712729 to 1497506 tid 1 2024-09-10T16:15:27.185 INFO:tasks.rados.rados.0.smithi165.stdout:1234: writing smithi16534841-1 from 2091098 to 2558589 tid 2 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1234: writing smithi16534841-1 from 3342313 to 3512729 tid 3 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1235: write (excl) oid 32 current snap is 143 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1232: finishing copy_from to smithi16534841-31 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1232: got expected ENOENT (src dne) 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1232: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1230: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:15:27.191 INFO:tasks.rados.rados.0.smithi165.stdout:1235: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-10T16:15:27.197 INFO:tasks.rados.rados.0.smithi165.stdout:1235: writing smithi16534841-32 from 729536 to 1189562 tid 1 2024-09-10T16:15:27.204 INFO:tasks.rados.rados.0.smithi165.stdout:1235: writing smithi16534841-32 from 1749580 to 2360729 tid 2 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1235: writing smithi16534841-32 from 2853383 to 3305833 tid 3 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1232: done (5 left) 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1236: delete oid 23 current snap is 143 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1230: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1230: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 693 (ObjNum 403 snap 130 seq_num 403) dirty exists 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1229: finishing write tid 4 to smithi16534841-3 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1229: finishing write tid 5 to smithi16534841-3 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1229: oid 3 updating version 0 to 881 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1229: oid 3 version 881 is already newer than 879 2024-09-10T16:15:27.210 INFO:tasks.rados.rados.0.smithi165.stdout:1229: oid 3 version 881 is already newer than 880 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1229: oid 3 updating version 881 to 882 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 882 (ObjNum 446 snap 143 seq_num 446) dirty exists 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1229: left oid 3 (ObjNum 446 snap 143 seq_num 446) 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1231: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1231: finishing copy_from to smithi16534841-35 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 728 (ObjNum 423 snap 138 seq_num 423) dirty exists 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: finishing write tid 1 to smithi16534841-1 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: finishing write tid 2 to smithi16534841-1 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: finishing write tid 3 to smithi16534841-1 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: finishing write tid 4 to smithi16534841-1 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: finishing write tid 5 to smithi16534841-1 2024-09-10T16:15:27.211 INFO:tasks.rados.rados.0.smithi165.stdout:1234: oid 1 updating version 0 to 878 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1234: oid 1 version 878 is already newer than 876 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1234: oid 1 updating version 878 to 879 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1234: oid 1 version 879 is already newer than 877 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 879 (ObjNum 447 snap 143 seq_num 447) dirty exists 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1234: left oid 1 (ObjNum 447 snap 143 seq_num 447) 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1235: finishing write tid 1 to smithi16534841-32 2024-09-10T16:15:27.212 INFO:tasks.rados.rados.0.smithi165.stdout:1235: finishing write tid 2 to smithi16534841-32 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1235: finishing write tid 3 to smithi16534841-32 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1229: done (5 left) 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1230: done (4 left) 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1231: done (3 left) 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1234: done (2 left) 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1236: done (1 left) 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1237: write oid 7 current snap is 143 2024-09-10T16:15:27.213 INFO:tasks.rados.rados.0.smithi165.stdout:1237: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-10T16:15:27.220 INFO:tasks.rados.rados.0.smithi165.stdout:1237: writing smithi16534841-7 from 763150 to 1490027 tid 1 2024-09-10T16:15:27.226 INFO:tasks.rados.rados.0.smithi165.stdout:1237: writing smithi16534841-7 from 1982897 to 2597715 tid 2 2024-09-10T16:15:27.232 INFO:tasks.rados.rados.0.smithi165.stdout:1237: writing smithi16534841-7 from 3024582 to 3499576 tid 3 2024-09-10T16:15:27.232 INFO:tasks.rados.rados.0.smithi165.stdout:1238: write (excl) oid 23 current snap is 143 2024-09-10T16:15:27.232 INFO:tasks.rados.rados.0.smithi165.stdout:1235: finishing write tid 4 to smithi16534841-32 2024-09-10T16:15:27.232 INFO:tasks.rados.rados.0.smithi165.stdout:1238: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2024-09-10T16:15:27.238 INFO:tasks.rados.rados.0.smithi165.stdout:1238: writing smithi16534841-23 from 779957 to 1182083 tid 1 2024-09-10T16:15:27.247 INFO:tasks.rados.rados.0.smithi165.stdout:1238: writing smithi16534841-23 from 1925026 to 2683502 tid 2 2024-09-10T16:15:27.252 INFO:tasks.rados.rados.0.smithi165.stdout:1238: writing smithi16534841-23 from 3335652 to 3579957 tid 3 2024-09-10T16:15:27.252 INFO:tasks.rados.rados.0.smithi165.stdout:1239: read oid 48 snap -1 2024-09-10T16:15:27.252 INFO:tasks.rados.rados.0.smithi165.stdout:1239: expect (ObjNum 445 snap 143 seq_num 445) 2024-09-10T16:15:27.252 INFO:tasks.rados.rados.0.smithi165.stdout:1240: delete oid 18 current snap is 143 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: finishing write tid 5 to smithi16534841-32 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: oid 32 updating version 0 to 616 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: oid 32 version 616 is already newer than 614 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: oid 32 version 616 is already newer than 615 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: oid 32 version 616 is already newer than 613 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 616 (ObjNum 448 snap 143 seq_num 448) dirty exists 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1235: left oid 32 (ObjNum 448 snap 143 seq_num 448) 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: finishing write tid 1 to smithi16534841-7 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: finishing write tid 2 to smithi16534841-7 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: finishing write tid 3 to smithi16534841-7 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: finishing write tid 4 to smithi16534841-7 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: finishing write tid 5 to smithi16534841-7 2024-09-10T16:15:27.253 INFO:tasks.rados.rados.0.smithi165.stdout:1237: oid 7 updating version 0 to 1006 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1237: oid 7 version 1006 is already newer than 1003 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1237: oid 7 version 1006 is already newer than 1004 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1237: oid 7 version 1006 is already newer than 1005 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1006 (ObjNum 450 snap 143 seq_num 450) dirty exists 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1237: left oid 7 (ObjNum 450 snap 143 seq_num 450) 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1238: finishing write tid 1 to smithi16534841-23 2024-09-10T16:15:27.254 INFO:tasks.rados.rados.0.smithi165.stdout:1238: finishing write tid 2 to smithi16534841-23 2024-09-10T16:15:27.256 INFO:tasks.rados.rados.0.smithi165.stdout:1238: finishing write tid 3 to smithi16534841-23 2024-09-10T16:15:27.256 INFO:tasks.rados.rados.0.smithi165.stdout:1238: finishing write tid 4 to smithi16534841-23 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: finishing write tid 5 to smithi16534841-23 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: oid 23 updating version 0 to 881 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: oid 23 updating version 881 to 883 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: oid 23 version 883 is already newer than 882 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: oid 23 updating version 883 to 884 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 884 (ObjNum 451 snap 143 seq_num 451) dirty exists 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: left oid 23 (ObjNum 451 snap 143 seq_num 451) 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1235: done (4 left) 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1237: done (3 left) 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1238: done (2 left) 2024-09-10T16:15:27.257 INFO:tasks.rados.rados.0.smithi165.stdout:1240: done (1 left) 2024-09-10T16:15:27.258 INFO:tasks.rados.rados.0.smithi165.stdout:1241: snap_remove snap 131 2024-09-10T16:15:28.644 INFO:tasks.rados.rados.0.smithi165.stdout:1241: done (1 left) 2024-09-10T16:15:28.644 INFO:tasks.rados.rados.0.smithi165.stdout:1242: write oid 12 current snap is 143 2024-09-10T16:15:28.644 INFO:tasks.rados.rados.0.smithi165.stdout:1242: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-10T16:15:28.649 INFO:tasks.rados.rados.0.smithi165.stdout:1242: writing smithi16534841-12 from 413571 to 966195 tid 1 2024-09-10T16:15:28.657 INFO:tasks.rados.rados.0.smithi165.stdout:1242: writing smithi16534841-12 from 1641990 to 2404135 tid 2 2024-09-10T16:15:28.663 INFO:tasks.rados.rados.0.smithi165.stdout:1242: writing smithi16534841-12 from 2990498 to 3415661 tid 3 2024-09-10T16:15:28.663 INFO:tasks.rados.rados.0.smithi165.stdout:1243: copy_from oid 19 from oid 27 current snap is 143 2024-09-10T16:15:28.663 INFO:tasks.rados.rados.0.smithi165.stdout:1239: expect (ObjNum 445 snap 143 seq_num 445) 2024-09-10T16:15:28.707 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 721919 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: finishing write tid 1 to smithi16534841-12 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: finishing write tid 2 to smithi16534841-12 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: finishing write tid 3 to smithi16534841-12 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: finishing write tid 4 to smithi16534841-12 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: finishing write tid 5 to smithi16534841-12 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: oid 12 updating version 0 to 90 2024-09-10T16:15:28.759 INFO:tasks.rados.rados.0.smithi165.stdout:1242: oid 12 updating version 90 to 92 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1242: oid 12 version 92 is already newer than 91 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1242: oid 12 version 92 is already newer than 89 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 92 (ObjNum 453 snap 143 seq_num 453) dirty exists 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1242: left oid 12 (ObjNum 453 snap 143 seq_num 453) 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1243: finishing copy_from racing read to smithi16534841-19 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1243: finishing copy_from to smithi16534841-19 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 612 (ObjNum 408 snap 132 seq_num 408) dirty exists 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1239: done (2 left) 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1242: done (1 left) 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1243: done (0 left) 2024-09-10T16:15:28.760 INFO:tasks.rados.rados.0.smithi165.stdout:1244: snap_remove snap 141 2024-09-10T16:15:29.587 INFO:tasks.rados.rados.0.smithi165.stdout:1244: done (0 left) 2024-09-10T16:15:29.587 INFO:tasks.rados.rados.0.smithi165.stdout:1245: snap_remove snap 138 2024-09-10T16:15:30.588 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:15:30.589 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:15:30.589 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:15:30.590 INFO:tasks.rados.rados.0.smithi165.stdout:1245: done (0 left) 2024-09-10T16:15:30.590 INFO:tasks.rados.rados.0.smithi165.stdout:1246: read oid 30 snap -1 2024-09-10T16:15:30.590 INFO:tasks.rados.rados.0.smithi165.stdout:1246: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:15:30.591 INFO:tasks.rados.rados.0.smithi165.stdout:1247: snap_remove snap 112 2024-09-10T16:15:30.593 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:15:30.593 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:15:30.593 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:30.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.701+0000 7f52d4cbd640 10 monclient: get_monmap_and_config 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.701+0000 7f52d4cbd640 10 monclient: build_initial_monmap 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.701+0000 7f52d4cbd640 10 monclient: monmap: 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:30.702549+0000 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:30.702549+0000 2024-09-10T16:15:30.704 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:30.705 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 Processor -- start 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 -- start start 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient: init 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-10T16:15:30.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-b con 0x7f52d006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f52d006d540 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52d013d060 0x7f52d013f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-a con 0x7f52d013d060 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52d009cb90 con 0x7f52d006d540 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f52d009ce70 con 0x7f52d013d060 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f52d009d150 con 0x7f52d006d160 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.702+0000 7f52d4cbd640 10 monclient(hunting): authenticate will time out at 1309.189953s 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52d013d060 0x7f52d013f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52c7fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f52d006d540 0x7f52d00a46e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47530/0 (socket says 172.21.15.164:47530) 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52d013d060 0x7f52d013f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:40646/0 (socket says 172.21.15.164:40646) 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52c7fff640 1 -- 172.21.15.164:0/1097474015 learned_addr learned my addr 172.21.15.164:0/1097474015 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55290/0 (socket says 172.21.15.164:55290) 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 10 monclient(hunting): get_auth_request con 0x7f52d006d160 auth_method 0 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): get_auth_request con 0x7f52d013d060 auth_method 0 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:30.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 -- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 msgr2=0x7f52d0134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 -- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 msgr2=0x7f52d0134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ce59c640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient(hunting): handle_auth_done global_id 74379 payload 274 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 -- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 msgr2=0x7f52d0134e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 -- 172.21.15.164:0/1097474015 >> v1:172.21.15.164:6789/0 conn(0x7f52d006d540 legacy=0x7f52d00a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:30.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.703+0000 7f52ced9d640 1 -- 172.21.15.164:0/1097474015 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d00a4df0 con 0x7f52d013d060 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52ced9d640 1 -- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 msgr2=0x7f52d013f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52ced9d640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d013f450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009d90 tx=0x7f52c802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c2afd1723f29b07 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 1 -- 172.21.15.164:0/1097474015 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8007950 con 0x7f52d013d060 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: dump: 2024-09-10T16:15:30.713 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: _finish_auth 0 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:30.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 1 -- 172.21.15.164:0/1097474015 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c8007af0 con 0x7f52d013d060 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 1 -- 172.21.15.164:0/1097474015 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8031400 con 0x7f52d013d060 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52cdd9b640 10 monclient: dump: 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:30.715 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52d4cbd640 5 monclient: authenticate success, global_id 74379 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52d4cbd640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.706+0000 7f52d4cbd640 4 set_mon_vals no callback set 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 10 monclient: shutdown 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 msgr2=0x7f52d013f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d013f450 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009d90 tx=0x7f52c802a490 comp rx=0 tx=0).stop 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1097474015 shutdown_connections 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d013f450 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1097474015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d160 0x7f52d0134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1097474015 >> 172.21.15.164:0/1097474015 conn(0x7f52d00aacb0 msgr2=0x7f52d00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:30.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1097474015 shutdown_connections 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1097474015 wait complete. 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 10 monclient: build_initial_monmap 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.707+0000 7f52d4cbd640 10 monclient: monmap: 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 Processor -- start 2024-09-10T16:15:30.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 -- start start 2024-09-10T16:15:30.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient: init 2024-09-10T16:15:30.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52d006d160 0x7f52d0135d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-b con 0x7f52d006d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-c con 0x7f52d006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): picked mon.noname-a con 0x7f52d013d060 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f52d013fb10 con 0x7f52d013d060 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f52d013f990 con 0x7f52d006d160 2024-09-10T16:15:30.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f52d013fc90 con 0x7f52d006d540 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52d4cbd640 10 monclient(hunting): authenticate will time out at 1309.195740s 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:40648/0 (socket says 172.21.15.164:40648) 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 1 -- 172.21.15.164:0/1990127042 learned_addr learned my addr 172.21.15.164:0/1990127042 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): get_auth_request con 0x7f52d013d060 auth_method 0 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 10 monclient(hunting): get_auth_request con 0x7f52d006d540 auth_method 0 2024-09-10T16:15:30.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 msgr2=0x7f52d0136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 msgr2=0x7f52d0136290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.708+0000 7f52c7fff640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ce59c640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52d006d160 0x7f52d0135d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ce59c640 10 monclient(hunting): get_auth_request con 0x7f52d006d160 auth_method 0 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ce59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:30.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ce59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 10 monclient(hunting): handle_auth_done global_id 74382 payload 824 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 msgr2=0x7f52d0136290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52d006d160 msgr2=0x7f52d0135d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52d006d160 0x7f52d0135d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.709+0000 7f52ced9d640 1 -- 172.21.15.164:0/1990127042 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52d013c6e0 con 0x7f52d013d060 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52ced9d640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 msgr2=0x7f52d01cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:30.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52ced9d640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009eb0 tx=0x7f52c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c8040070 con 0x7f52d013d060 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: dump: 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:30.726 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 5 monclient: authenticate success, global_id 74382 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52c803c620 con 0x7f52d013d060 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 10 monclient: _renew_subs 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52d01377d0 con 0x7f52d013d060 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52c80314c0 con 0x7f52d013d060 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:30.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52c6ffd640 10 monclient: dump: 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 10 monclient: _renew_subs 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52d01388f0 con 0x7f52d013d060 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.711+0000 7f52cd59a640 4 set_mon_vals no callback set 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52d4cbd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52d4cbd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52d4cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f52c8039040 con 0x7f52d013d060 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52d0134e00 con 0x7f52d013d060 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52c6ffd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 0x7f52b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.712+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(312..312 src has 184..312) ==== 4831+0+0 (secure 0 0 0) 0x7f52c80743a0 con 0x7f52d013d060 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.713+0000 7f52ce59c640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 0x7f52b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.713+0000 7f52ce59c640 10 monclient: get_auth_request con 0x7f52b003e210 auth_method 0 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.715+0000 7f52ce59c640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 0x7f52b00406d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52d0136c00 tx=0x7f52c0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.717+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f52c8072d70 con 0x7f52d013d060 2024-09-10T16:15:30.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.717+0000 7f52c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:30.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.717+0000 7f52c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52d4cbd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52d4cbd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52d4cbd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52d00aa1c0 con 0x7f52d013d060 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52c6ffd640 1 -- 172.21.15.164:0/1990127042 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f52c80740d0 con 0x7f52d013d060 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.958+0000 7f52c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:30.960 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 msgr2=0x7f52b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 0x7f52b00406d0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f52d0136c00 tx=0x7f52c0008040 comp rx=0 tx=0).stop 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 10 monclient: shutdown 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 msgr2=0x7f52d01cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f52c8009eb0 tx=0x7f52c8004a40 comp rx=0 tx=0).stop 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 shutdown_connections 2024-09-10T16:15:30.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f52b003e210 0x7f52b00406d0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52d013d060 0x7f52d01cd2b0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52d006d540 0x7f52d0136290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 --2- 172.21.15.164:0/1990127042 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f52d006d160 0x7f52d0135d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 >> 172.21.15.164:0/1990127042 conn(0x7f52d00aacb0 msgr2=0x7f52d0132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 shutdown_connections 2024-09-10T16:15:30.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:30.960+0000 7f52d4cbd640 1 -- 172.21.15.164:0/1990127042 wait complete. 2024-09-10T16:15:31.594 INFO:tasks.rados.rados.0.smithi165.stdout:1247: done (1 left) 2024-09-10T16:15:31.595 INFO:tasks.rados.rados.0.smithi165.stdout:1248: rollback oid 39 current snap is 143 2024-09-10T16:15:31.595 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 140 2024-09-10T16:15:31.595 INFO:tasks.rados.rados.0.smithi165.stdout:1246: expect (ObjNum 338 snap 98 seq_num 338) 2024-09-10T16:15:31.737 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:31.811 INFO:tasks.rados.rados.0.smithi165.stdout:1248: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:15:31.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 0 (ObjNum 289 snap 0 seq_num 326749518) dirty dne 2024-09-10T16:15:31.812 INFO:tasks.rados.rados.0.smithi165.stdout:1246: done (1 left) 2024-09-10T16:15:31.812 INFO:tasks.rados.rados.0.smithi165.stdout:1248: done (0 left) 2024-09-10T16:15:31.812 INFO:tasks.rados.rados.0.smithi165.stdout:1249: write (excl) oid 4 current snap is 143 2024-09-10T16:15:31.812 INFO:tasks.rados.rados.0.smithi165.stdout:1249: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-10T16:15:31.817 INFO:tasks.rados.rados.0.smithi165.stdout:1249: writing smithi16534841-4 from 430378 to 1058251 tid 1 2024-09-10T16:15:31.824 INFO:tasks.rados.rados.0.smithi165.stdout:1249: writing smithi16534841-4 from 1700472 to 2322628 tid 2 2024-09-10T16:15:31.831 INFO:tasks.rados.rados.0.smithi165.stdout:1249: writing smithi16534841-4 from 3017921 to 3630378 tid 3 2024-09-10T16:15:31.831 INFO:tasks.rados.rados.0.smithi165.stdout:1250: delete oid 32 current snap is 143 2024-09-10T16:15:31.831 INFO:tasks.rados.rados.0.smithi165.stdout:1249: finishing write tid 1 to smithi16534841-4 2024-09-10T16:15:31.831 INFO:tasks.rados.rados.0.smithi165.stdout:1249: finishing write tid 2 to smithi16534841-4 2024-09-10T16:15:31.834 INFO:tasks.rados.rados.0.smithi165.stdout:1249: finishing write tid 3 to smithi16534841-4 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1250: done (1 left) 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1251: delete oid 39 current snap is 143 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: finishing write tid 4 to smithi16534841-4 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: finishing write tid 5 to smithi16534841-4 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: oid 4 updating version 0 to 695 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: oid 4 updating version 695 to 697 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: oid 4 version 697 is already newer than 696 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: oid 4 version 697 is already newer than 694 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 697 (ObjNum 454 snap 143 seq_num 454) dirty exists 2024-09-10T16:15:31.835 INFO:tasks.rados.rados.0.smithi165.stdout:1249: left oid 4 (ObjNum 454 snap 143 seq_num 454) 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1249: done (1 left) 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1251: done (0 left) 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1252: rollback oid 23 current snap is 143 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 140 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1253: rollback oid 16 current snap is 143 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 142 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1254: rollback oid 25 current snap is 143 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 142 2024-09-10T16:15:31.837 INFO:tasks.rados.rados.0.smithi165.stdout:1255: rollback oid 5 current snap is 143 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 140 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:1256: copy_from oid 36 from oid 6 current snap is 143 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:1257: read oid 29 snap -1 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:1257: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:1258: write (excl) oid 28 current snap is 143 2024-09-10T16:15:31.838 INFO:tasks.rados.rados.0.smithi165.stdout:1258: seq_num 457 ranges {480799=569973,1475918=769483,2700190=586604} 2024-09-10T16:15:31.842 INFO:tasks.rados.rados.0.smithi165.stdout:1258: writing smithi16534841-28 from 480799 to 1050772 tid 1 2024-09-10T16:15:31.849 INFO:tasks.rados.rados.0.smithi165.stdout:1258: writing smithi16534841-28 from 1475918 to 2245401 tid 2 2024-09-10T16:15:31.855 INFO:tasks.rados.rados.0.smithi165.stdout:1258: writing smithi16534841-28 from 2700190 to 3286794 tid 3 2024-09-10T16:15:31.855 INFO:tasks.rados.rados.0.smithi165.stdout:1259: write oid 49 current snap is 143 2024-09-10T16:15:31.855 INFO:tasks.rados.rados.0.smithi165.stdout:1259: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-10T16:15:31.861 INFO:tasks.rados.rados.0.smithi165.stdout:1259: writing smithi16534841-49 from 497606 to 1142828 tid 1 2024-09-10T16:15:31.868 INFO:tasks.rados.rados.0.smithi165.stdout:1259: writing smithi16534841-49 from 1934400 to 2563894 tid 2 2024-09-10T16:15:31.875 INFO:tasks.rados.rados.0.smithi165.stdout:1259: writing smithi16534841-49 from 3243966 to 3697606 tid 3 2024-09-10T16:15:31.875 INFO:tasks.rados.rados.0.smithi165.stdout:1260: copy_from oid 2 from oid 37 current snap is 143 2024-09-10T16:15:31.875 INFO:tasks.rados.rados.0.smithi165.stdout:1252: finishing rollback tid 0 to smithi16534841-23 2024-09-10T16:15:31.875 INFO:tasks.rados.rados.0.smithi165.stdout:1257: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:15:31.929 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:31.982 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1252: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1253: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1254: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1254: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1253: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:1252: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:15:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 888 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1253: finishing rollback tid 2 to smithi16534841-16 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1038 (ObjNum 434 snap 141 seq_num 434) dirty exists 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1254: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 885 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1255: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1255: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1255: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 620 (ObjNum 408 snap 132 seq_num 408) dirty exists 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1258: finishing write tid 1 to smithi16534841-28 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1256: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1256: finishing copy_from to smithi16534841-36 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 605 (ObjNum 404 snap 130 seq_num 404) dirty exists 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1258: finishing write tid 2 to smithi16534841-28 2024-09-10T16:15:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:1258: finishing write tid 3 to smithi16534841-28 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: finishing write tid 4 to smithi16534841-28 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: finishing write tid 5 to smithi16534841-28 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: oid 28 updating version 0 to 352 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: oid 28 version 352 is already newer than 350 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: oid 28 updating version 352 to 353 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: oid 28 version 353 is already newer than 351 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 353 (ObjNum 457 snap 143 seq_num 457) dirty exists 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1258: left oid 28 (ObjNum 457 snap 143 seq_num 457) 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1259: finishing write tid 1 to smithi16534841-49 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1259: finishing write tid 2 to smithi16534841-49 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1252: done (8 left) 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1253: done (7 left) 2024-09-10T16:15:32.039 INFO:tasks.rados.rados.0.smithi165.stdout:1254: done (6 left) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1255: done (5 left) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1256: done (4 left) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1257: done (3 left) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1258: done (2 left) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1261: read oid 34 snap -1 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1261: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1262: snap_create 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1260: finishing copy_from to smithi16534841-2 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1260: got expected ENOENT (src dne) 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1260: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1259: finishing write tid 3 to smithi16534841-49 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1259: finishing write tid 4 to smithi16534841-49 2024-09-10T16:15:32.040 INFO:tasks.rados.rados.0.smithi165.stdout:1259: finishing write tid 5 to smithi16534841-49 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:1259: oid 49 updating version 0 to 365 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:1259: oid 49 updating version 365 to 366 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:1259: oid 49 version 366 is already newer than 363 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:1259: oid 49 version 366 is already newer than 364 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 366 (ObjNum 458 snap 143 seq_num 458) dirty exists 2024-09-10T16:15:32.041 INFO:tasks.rados.rados.0.smithi165.stdout:1259: left oid 49 (ObjNum 458 snap 143 seq_num 458) 2024-09-10T16:15:32.047 INFO:tasks.rados.rados.0.smithi165.stdout:1261: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:15:32.047 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:32.213 INFO:tasks.rados.rados.0.smithi165.stdout:1259: done (3 left) 2024-09-10T16:15:32.213 INFO:tasks.rados.rados.0.smithi165.stdout:1260: done (2 left) 2024-09-10T16:15:32.213 INFO:tasks.rados.rados.0.smithi165.stdout:1261: done (1 left) 2024-09-10T16:15:32.213 INFO:tasks.rados.rados.0.smithi165.stdout:1262: done (0 left) 2024-09-10T16:15:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:1263: rollback oid 39 current snap is 144 2024-09-10T16:15:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 140 2024-09-10T16:15:32.214 INFO:tasks.rados.rados.0.smithi165.stdout:1264: delete oid 27 current snap is 144 2024-09-10T16:15:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:1263: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:15:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 0 (ObjNum 129 snap 0 seq_num 1162772832) dirty dne 2024-09-10T16:15:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:1263: done (1 left) 2024-09-10T16:15:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:1264: done (0 left) 2024-09-10T16:15:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:1265: snap_remove snap 143 2024-09-10T16:15:33.171 INFO:tasks.rados.rados.0.smithi165.stdout:1265: done (0 left) 2024-09-10T16:15:33.171 INFO:tasks.rados.rados.0.smithi165.stdout:1266: rollback oid 45 current snap is 144 2024-09-10T16:15:33.171 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 140 2024-09-10T16:15:33.171 INFO:tasks.rados.rados.0.smithi165.stdout:1267: snap_remove snap 142 2024-09-10T16:15:33.974 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 10 monclient: get_monmap_and_config 2024-09-10T16:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 10 monclient: build_initial_monmap 2024-09-10T16:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 10 monclient: monmap: 2024-09-10T16:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:34.085 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:34.084908+0000 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:34.084908+0000 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:34.086 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 1 Processor -- start 2024-09-10T16:15:34.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 1 -- start start 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.084+0000 7fd8817f3640 10 monclient: init 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd87c0a40d0 0x7fd87c1360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-c con 0x7fd87c0a40d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd87c136600 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd87c0a4ac0 con 0x7fd87c136600 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:34.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd87c0a1400 con 0x7fd87c0a40d0 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd87c0a1120 con 0x7fd87c135ce0 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd8817f3640 10 monclient(hunting): authenticate will time out at 1312.572474s 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd87c136600 0x7fd87c09b0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47546/0 (socket says 172.21.15.164:47546) 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87b7fe640 1 -- 172.21.15.164:0/3624676222 learned_addr learned my addr 172.21.15.164:0/3624676222 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 10 monclient(hunting): get_auth_request con 0x7fd87c135ce0 auth_method 0 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:34.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 1 -- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 msgr2=0x7fd87c0a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 1 -- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 msgr2=0x7fd87c0a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd87affd640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 1 --2- 172.21.15.164:0/3624676222 >> v2:172.21.15.165:3300/0 conn(0x7fd87c0a40d0 0x7fd87c1360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): get_auth_request con 0x7fd87c0a40d0 auth_method 0 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.085+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:34.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 10 monclient(hunting): handle_auth_done global_id 94168 payload 274 2024-09-10T16:15:34.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 1 -- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 msgr2=0x7fd87c0a23a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:34.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 1 -- 172.21.15.164:0/3624676222 >> v1:172.21.15.164:6789/0 conn(0x7fd87c136600 legacy=0x7fd87c09b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:34.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:34.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 1 monclient: found mon.noname-c 2024-09-10T16:15:34.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:34.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:34.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.086+0000 7fd873fff640 1 -- 172.21.15.164:0/3624676222 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87c0a28e0 con 0x7fd87c0a40d0 2024-09-10T16:15:34.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd873fff640 1 -- 172.21.15.164:0/3624676222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c0a40d0 msgr2=0x7fd87c1360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd873fff640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c0a40d0 0x7fd87c1360c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8640097a0 tx=0x7fd864029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44599cfa9d3cdc9e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 1 -- 172.21.15.164:0/3624676222 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864037070 con 0x7fd87c0a40d0 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: dump: 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:34.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 1 -- 172.21.15.164:0/3624676222 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd864032440 con 0x7fd87c0a40d0 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 1 -- 172.21.15.164:0/3624676222 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd864032810 con 0x7fd87c0a40d0 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd87a7fc640 10 monclient: dump: 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 5 monclient: authenticate success, global_id 94168 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 4 set_mon_vals no callback set 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 10 monclient: shutdown 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 1 -- 172.21.15.164:0/3624676222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c0a40d0 msgr2=0x7fd87c1360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.089+0000 7fd8817f3640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c0a40d0 0x7fd87c1360c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd8640097a0 tx=0x7fd864029fd0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 -- 172.21.15.164:0/3624676222 shutdown_connections 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c0a40d0 0x7fd87c1360c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 --2- 172.21.15.164:0/3624676222 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c135ce0 0x7fd87c0a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 -- 172.21.15.164:0/3624676222 >> 172.21.15.164:0/3624676222 conn(0x7fd87c0a4f50 msgr2=0x7fd87c0a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 -- 172.21.15.164:0/3624676222 shutdown_connections 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 -- 172.21.15.164:0/3624676222 wait complete. 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 10 monclient: build_initial_monmap 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 10 monclient: monmap: 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.106 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.090+0000 7fd8817f3640 1 Processor -- start 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 -- start start 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient: init 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-a con 0x7fd87c0a40d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c135ce0 0x7fd87c09d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-b con 0x7fd87c135ce0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:34.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): picked mon.noname-c con 0x7fd87c136600 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c09b970 con 0x7fd87c0a40d0 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd87c09b7f0 con 0x7fd87c135ce0 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd87c09baf0 con 0x7fd87c136600 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd8817f3640 10 monclient(hunting): authenticate will time out at 1312.578550s 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:40680/0 (socket says 172.21.15.164:40680) 2024-09-10T16:15:34.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 -- 172.21.15.164:0/2144286643 learned_addr learned my addr 172.21.15.164:0/2144286643 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): get_auth_request con 0x7fd87c0a40d0 auth_method 0 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 10 monclient(hunting): get_auth_request con 0x7fd87c136600 auth_method 0 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:34.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c135ce0 0x7fd87c09d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 msgr2=0x7fd87c09d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 msgr2=0x7fd87c09d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): get_auth_request con 0x7fd87c135ce0 auth_method 0 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87b7fe640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:34.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient(hunting): handle_auth_done global_id 74385 payload 824 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 msgr2=0x7fd87c09d8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c135ce0 msgr2=0x7fd87c09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c135ce0 0x7fd87c09d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 monclient: found mon.noname-a 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.091+0000 7fd87affd640 1 -- 172.21.15.164:0/2144286643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd87c09eb30 con 0x7fd87c0a40d0 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd87affd640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 msgr2=0x7fd87c09aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd87affd640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00d800 tx=0x7fd86c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00b8d0 con 0x7fd87c0a40d0 2024-09-10T16:15:34.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: dump: 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:34.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: _finish_auth 0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: _check_auth_tickets 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 5 monclient: authenticate success, global_id 74385 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd86c019070 con 0x7fd87c0a40d0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 10 monclient: _renew_subs 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd87c06e500 con 0x7fd87c0a40d0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd86c00bcd0 con 0x7fd87c0a40d0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:34.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8737fe640 10 monclient: dump: 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd879ffb640 4 set_mon_vals no callback set 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 10 monclient: _renew_subs 2024-09-10T16:15:34.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:34.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.094+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd87c09af80 con 0x7fd87c0a40d0 2024-09-10T16:15:34.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd86c01e460 con 0x7fd87c0a40d0 2024-09-10T16:15:34.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8737fe640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 0x7fd85c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:34.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8817f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8817f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8817f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd87c09bf00 con 0x7fd87c0a40d0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(315..315 src has 184..315) ==== 4831+0+0 (secure 0 0 0) 0x7fd86c0531e0 con 0x7fd87c0a40d0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd873fff640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 0x7fd85c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.095+0000 7fd873fff640 10 monclient: get_auth_request con 0x7fd85c03e210 auth_method 0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.098+0000 7fd873fff640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 0x7fd85c0406d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd864031000 tx=0x7fd86402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.100+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd86c014430 con 0x7fd87c0a40d0 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.100+0000 7fd8737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:34.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.100+0000 7fd8737fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:1267: done (1 left) 2024-09-10T16:15:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:1268: read oid 15 snap -1 2024-09-10T16:15:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:1268: expect (ObjNum 403 snap 130 seq_num 403) 2024-09-10T16:15:34.179 INFO:tasks.rados.rados.0.smithi165.stdout:1269: rollback oid 49 current snap is 144 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 140 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1266: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1266: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1266: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 877 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1266: done (2 left) 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1270: write oid 37 current snap is 144 2024-09-10T16:15:34.180 INFO:tasks.rados.rados.0.smithi165.stdout:1270: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2024-09-10T16:15:34.186 INFO:tasks.rados.rados.0.smithi165.stdout:1270: writing smithi16534841-37 from 531220 to 1326940 tid 1 2024-09-10T16:15:34.193 INFO:tasks.rados.rados.0.smithi165.stdout:1270: writing smithi16534841-37 from 1935011 to 2451821 tid 2 2024-09-10T16:15:34.198 INFO:tasks.rados.rados.0.smithi165.stdout:1270: writing smithi16534841-37 from 3066106 to 3486526 tid 3 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:1271: copy_from oid 5 from oid 2 current snap is 144 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:1269: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:1269: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:1269: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 373 (ObjNum 372 snap 108 seq_num 372) dirty exists 2024-09-10T16:15:34.199 INFO:tasks.rados.rados.0.smithi165.stdout:1268: expect (ObjNum 403 snap 130 seq_num 403) 2024-09-10T16:15:34.247 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:34.303 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8817f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8817f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8817f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd87c074fe0 con 0x7fd87c0a40d0 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8737fe640 1 -- 172.21.15.164:0/2144286643 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fd86c01eb80 con 0x7fd87c0a40d0 2024-09-10T16:15:34.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:34.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.355+0000 7fd8737fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:34.358 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":84,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:34.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.357+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 msgr2=0x7fd85c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:34.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 0x7fd85c0406d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd864031000 tx=0x7fd86402e000 comp rx=0 tx=0).stop 2024-09-10T16:15:34.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 10 monclient: shutdown 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 msgr2=0x7fd87c09aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd86c00d800 tx=0x7fd86c00dcc0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 shutdown_connections 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd85c03e210 0x7fd85c0406d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd87c136600 0x7fd87c09d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd87c135ce0 0x7fd87c09d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 --2- 172.21.15.164:0/2144286643 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd87c0a40d0 0x7fd87c09aa40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 >> 172.21.15.164:0/2144286643 conn(0x7fd87c0a4f50 msgr2=0x7fd87c0a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 shutdown_connections 2024-09-10T16:15:34.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:34.358+0000 7fd8817f3640 1 -- 172.21.15.164:0/2144286643 wait complete. 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: finishing write tid 1 to smithi16534841-37 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: finishing write tid 2 to smithi16534841-37 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: finishing write tid 3 to smithi16534841-37 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: finishing write tid 4 to smithi16534841-37 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: finishing write tid 5 to smithi16534841-37 2024-09-10T16:15:34.361 INFO:tasks.rados.rados.0.smithi165.stdout:1270: oid 37 updating version 0 to 562 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1270: oid 37 updating version 562 to 564 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1270: oid 37 version 564 is already newer than 563 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1270: oid 37 version 564 is already newer than 561 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 564 (ObjNum 460 snap 144 seq_num 460) dirty exists 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1270: left oid 37 (ObjNum 460 snap 144 seq_num 460) 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1271: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1271: finishing copy_from to smithi16534841-5 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1268: done (3 left) 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1269: done (2 left) 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1270: done (1 left) 2024-09-10T16:15:34.362 INFO:tasks.rados.rados.0.smithi165.stdout:1271: done (0 left) 2024-09-10T16:15:34.363 INFO:tasks.rados.rados.0.smithi165.stdout:1272: copy_from oid 10 from oid 48 current snap is 144 2024-09-10T16:15:34.363 INFO:tasks.rados.rados.0.smithi165.stdout:1273: snap_remove snap 140 2024-09-10T16:15:35.182 INFO:tasks.rados.rados.0.smithi165.stdout:1273: done (1 left) 2024-09-10T16:15:35.183 INFO:tasks.rados.rados.0.smithi165.stdout:1274: delete oid 46 current snap is 144 2024-09-10T16:15:35.183 INFO:tasks.rados.rados.0.smithi165.stdout:1272: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:15:35.183 INFO:tasks.rados.rados.0.smithi165.stdout:1272: finishing copy_from to smithi16534841-10 2024-09-10T16:15:35.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 995 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:15:35.188 INFO:tasks.rados.rados.0.smithi165.stdout:1272: done (1 left) 2024-09-10T16:15:35.188 INFO:tasks.rados.rados.0.smithi165.stdout:1274: done (0 left) 2024-09-10T16:15:35.188 INFO:tasks.rados.rados.0.smithi165.stdout:1275: write oid 26 current snap is 144 2024-09-10T16:15:35.188 INFO:tasks.rados.rados.0.smithi165.stdout:1275: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-10T16:15:35.194 INFO:tasks.rados.rados.0.smithi165.stdout:1275: writing smithi16534841-26 from 564834 to 1111052 tid 1 2024-09-10T16:15:35.201 INFO:tasks.rados.rados.0.smithi165.stdout:1275: writing smithi16534841-26 from 1651975 to 2172454 tid 2 2024-09-10T16:15:35.207 INFO:tasks.rados.rados.0.smithi165.stdout:1275: writing smithi16534841-26 from 2720952 to 3280269 tid 3 2024-09-10T16:15:35.207 INFO:tasks.rados.rados.0.smithi165.stdout:1276: snap_create 2024-09-10T16:15:35.210 INFO:tasks.rados.rados.0.smithi165.stdout:1275: finishing write tid 1 to smithi16534841-26 2024-09-10T16:15:35.212 INFO:tasks.rados.rados.0.smithi165.stdout:1275: finishing write tid 2 to smithi16534841-26 2024-09-10T16:15:35.212 INFO:tasks.rados.rados.0.smithi165.stdout:1275: finishing write tid 3 to smithi16534841-26 2024-09-10T16:15:35.212 INFO:tasks.rados.rados.0.smithi165.stdout:1275: finishing write tid 4 to smithi16534841-26 2024-09-10T16:15:35.212 INFO:tasks.rados.rados.0.smithi165.stdout:1275: finishing write tid 5 to smithi16534841-26 2024-09-10T16:15:35.212 INFO:tasks.rados.rados.0.smithi165.stdout:1275: oid 26 updating version 0 to 1051 2024-09-10T16:15:35.213 INFO:tasks.rados.rados.0.smithi165.stdout:1275: oid 26 version 1051 is already newer than 1049 2024-09-10T16:15:35.213 INFO:tasks.rados.rados.0.smithi165.stdout:1275: oid 26 updating version 1051 to 1052 2024-09-10T16:15:35.213 INFO:tasks.rados.rados.0.smithi165.stdout:1275: oid 26 version 1052 is already newer than 1050 2024-09-10T16:15:35.213 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1052 (ObjNum 462 snap 144 seq_num 462) dirty exists 2024-09-10T16:15:35.213 INFO:tasks.rados.rados.0.smithi165.stdout:1275: left oid 26 (ObjNum 462 snap 144 seq_num 462) 2024-09-10T16:15:36.189 INFO:tasks.rados.rados.0.smithi165.stdout:1275: done (1 left) 2024-09-10T16:15:36.190 INFO:tasks.rados.rados.0.smithi165.stdout:1276: done (0 left) 2024-09-10T16:15:36.190 INFO:tasks.rados.rados.0.smithi165.stdout:1277: snap_create 2024-09-10T16:15:37.171 INFO:tasks.rados.rados.0.smithi165.stdout:1277: done (0 left) 2024-09-10T16:15:37.171 INFO:tasks.rados.rados.0.smithi165.stdout:1278: delete oid 21 current snap is 146 2024-09-10T16:15:37.175 INFO:tasks.rados.rados.0.smithi165.stdout:1278: done (0 left) 2024-09-10T16:15:37.175 INFO:tasks.rados.rados.0.smithi165.stdout:1279: snap_remove snap 144 2024-09-10T16:15:37.374 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.482+0000 7fc1f0465640 10 monclient: get_monmap_and_config 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.482+0000 7fc1f0465640 10 monclient: build_initial_monmap 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.482+0000 7fc1f0465640 10 monclient: monmap: 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:37.483761+0000 2024-09-10T16:15:37.484 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:37.483761+0000 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:37.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.483+0000 7fc1f0465640 1 Processor -- start 2024-09-10T16:15:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 1 -- start start 2024-09-10T16:15:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient: init 2024-09-10T16:15:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:37.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc1e809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc1e809d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1e813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e813b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e809fbc0 con 0x7fc1e809d190 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1e80a2a20 con 0x7fc1e809c310 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e809f860 con 0x7fc1e813b5d0 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1f0465640 10 monclient(hunting): authenticate will time out at 1315.971777s 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1eda00640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc1e809d190 0x7fc1e809f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54114/0 (socket says 172.21.15.164:54114) 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.484+0000 7fc1eda00640 1 -- 172.21.15.164:0/2085879141 learned_addr learned my addr 172.21.15.164:0/2085879141 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:37.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1e813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 10 monclient(hunting): get_auth_request con 0x7fc1e813b5d0 auth_method 0 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 msgr2=0x7fc1e813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 msgr2=0x7fc1e813d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1e813d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:37.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.485+0000 7fc1eea02640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1e813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 1 -- 172.21.15.164:0/2085879141 >> v1:172.21.15.164:6789/0 conn(0x7fc1e809d190 legacy=0x7fc1e809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 1 -- 172.21.15.164:0/2085879141 >> v1:172.21.15.165:6789/0 conn(0x7fc1e809c310 legacy=0x7fc1e809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2206764708 0 0) 0x7fc1d8001450 con 0x7fc1e809d190 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): _negotiate 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): my global_id is 74388 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d4003520 con 0x7fc1e809d190 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2526732940 0 0) 0x7fc1d4003520 con 0x7fc1e809d190 2024-09-10T16:15:37.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 msgr2=0x7fc1e813d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1e813d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 >> v1:172.21.15.165:6789/0 conn(0x7fc1e809c310 legacy=0x7fc1e809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e809c6f0 con 0x7fc1e809d190 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _finish_auth 0 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _check_auth_tickets 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fc1d8001b70 con 0x7fc1e809d190 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1f0465640 5 monclient: authenticate success, global_id 74388 2024-09-10T16:15:37.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1f0465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: dump: 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.499 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 >> v1:172.21.15.164:6789/0 conn(0x7fc1e809d190 legacy=0x7fc1e809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1d4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): picked mon.a con 0x7fc1d4003aa0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): picked mon.c con 0x7fc1e813b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d4004e60 0x7fc1d400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): picked mon.b con 0x7fc1d4004e60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e809f860 con 0x7fc1d4003aa0 2024-09-10T16:15:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1d4005650 con 0x7fc1d4004e60 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc1d400bf50 con 0x7fc1e813b5d0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ed1ff640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1f0465640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1d4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 10 monclient(hunting): get_auth_request con 0x7fc1d4003aa0 auth_method 0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1ee201640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 10 monclient(hunting): get_auth_request con 0x7fc1e813b5d0 auth_method 0 2024-09-10T16:15:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.487+0000 7fc1eda00640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d4004e60 0x7fc1d400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eda00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eda00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eda00640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): get_auth_request con 0x7fc1d4004e60 auth_method 0 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eea02640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient(hunting): handle_auth_done global_id 74388 payload 293 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 msgr2=0x7fc1d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d4004e60 msgr2=0x7fc1d400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1d4004e60 0x7fc1d400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient: _finish_hunting 0 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 monclient: found mon.a 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 1 -- 172.21.15.164:0/2085879141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1d4003520 con 0x7fc1d4003aa0 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient: _finish_auth 0 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 10 monclient: _check_auth_tickets 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1ee201640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.488+0000 7fc1eda00640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e813b5d0 0x7fc1d4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ee201640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 msgr2=0x7fc1d4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ee201640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1d4003e80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4001a40 tx=0x7fc1d4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9475c4642d8c54bb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4001740 con 0x7fc1d4003aa0 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: dump: 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.504 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e402e070 con 0x7fc1d4003aa0 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 1 -- 172.21.15.164:0/2085879141 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e40072b0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 4 set_mon_vals no callback set 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1ed1ff640 10 monclient: dump: 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.506 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 10 monclient: shutdown 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 1 -- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 msgr2=0x7fc1d4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 1 --2- 172.21.15.164:0/2085879141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1d4003e80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4001a40 tx=0x7fc1d4005830 comp rx=0 tx=0).stop 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1eea02640 1 -- 172.21.15.164:0/2085879141 reap_dead start 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.491+0000 7fc1f0465640 1 -- 172.21.15.164:0/2085879141 shutdown_connections 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.492+0000 7fc1f0465640 1 -- 172.21.15.164:0/2085879141 >> 172.21.15.164:0/2085879141 conn(0x7fc1e80a3110 msgr2=0x7fc1e80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.492+0000 7fc1f0465640 1 -- 172.21.15.164:0/2085879141 shutdown_connections 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.492+0000 7fc1f0465640 1 -- 172.21.15.164:0/2085879141 wait complete. 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.492+0000 7fc1f0465640 10 monclient: build_initial_monmap 2024-09-10T16:15:37.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.492+0000 7fc1f0465640 10 monclient: monmap: 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 Processor -- start 2024-09-10T16:15:37.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 -- start start 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient: init 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-a con 0x7fc1d4003aa0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 0x7fc1e80936e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-b con 0x7fc1e80932a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): picked mon.noname-c con 0x7fc1e8093c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e80a20b0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc1e809a7b0 con 0x7fc1e80932a0 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc1e80a2230 con 0x7fc1e8093c20 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1f0465640 10 monclient(hunting): authenticate will time out at 1315.980681s 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47438/0 (socket says 172.21.15.164:47438) 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 1 -- 172.21.15.164:0/3577276088 learned_addr learned my addr 172.21.15.164:0/3577276088 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): get_auth_request con 0x7fc1d4003aa0 auth_method 0 2024-09-10T16:15:37.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 10 monclient(hunting): get_auth_request con 0x7fc1e8093c20 auth_method 0 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 msgr2=0x7fc1e81d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 msgr2=0x7fc1e81d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1ee201640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:37.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eea02640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.493+0000 7fc1eda00640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 0x7fc1e80936e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 10 monclient(hunting): handle_auth_done global_id 74391 payload 824 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 msgr2=0x7fc1e81d17a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 msgr2=0x7fc1e80936e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 0x7fc1e80936e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 10 monclient: _finish_hunting 0 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 monclient: found mon.noname-a 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1ee201640 1 -- 172.21.15.164:0/3577276088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1e8095f50 con 0x7fc1d4003aa0 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1eda00640 10 monclient: get_auth_request con 0x7fc1e80932a0 auth_method 0 2024-09-10T16:15:37.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.494+0000 7fc1eda00640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 0x7fc1e80936e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1ee201640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 msgr2=0x7fc1e8099a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1ee201640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4001110 tx=0x7fc1e402c070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4001740 con 0x7fc1d4003aa0 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: dump: 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.515 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1e4005de0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 5 monclient: authenticate success, global_id 74391 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 10 monclient: _renew_subs 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1e4033690 con 0x7fc1d4003aa0 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1e809d620 con 0x7fc1d4003aa0 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1deffd640 10 monclient: dump: 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1ec9fe640 4 set_mon_vals no callback set 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 10 monclient: _renew_subs 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.496+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1e8095c70 con 0x7fc1d4003aa0 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc1e4046060 con 0x7fc1d4003aa0 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1f0465640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:37.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1f0465640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1f0465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1e809ea40 con 0x7fc1d4003aa0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1deffd640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 0x7fc1cc040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.497+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(319..319 src has 184..319) ==== 4831+0+0 (secure 0 0 0) 0x7fc1e40726e0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.498+0000 7fc1eda00640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 0x7fc1cc040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.498+0000 7fc1eda00640 10 monclient: get_auth_request con 0x7fc1cc03e290 auth_method 0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.500+0000 7fc1eda00640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 0x7fc1cc040750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80010e0 tx=0x7fc1d8006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.502+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc1e40710b0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.502+0000 7fc1deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:37.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.502+0000 7fc1deffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:37.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1f0465640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1f0465640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1f0465640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc1e80746a0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1deffd640 1 -- 172.21.15.164:0/3577276088 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fc1e40723a0 con 0x7fc1d4003aa0 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.746+0000 7fc1deffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:37.748 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:37.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 msgr2=0x7fc1cc040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 0x7fc1cc040750 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc1d80010e0 tx=0x7fc1d8006040 comp rx=0 tx=0).stop 2024-09-10T16:15:37.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 10 monclient: shutdown 2024-09-10T16:15:37.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:37.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 msgr2=0x7fc1e8099a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fc1e4001110 tx=0x7fc1e402c070 comp rx=0 tx=0).stop 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 shutdown_connections 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc1cc03e290 0x7fc1cc040750 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.748+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc1e8093c20 0x7fc1e81d17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.749+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc1e80932a0 0x7fc1e80936e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.749+0000 7fc1f0465640 1 --2- 172.21.15.164:0/3577276088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1d4003aa0 0x7fc1e8099a90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.749+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 >> 172.21.15.164:0/3577276088 conn(0x7fc1e80a3110 msgr2=0x7fc1e809af20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.749+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 shutdown_connections 2024-09-10T16:15:37.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.749+0000 7fc1f0465640 1 -- 172.21.15.164:0/3577276088 wait complete. 2024-09-10T16:15:37.761 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:15:37.761 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:15:37.761 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:15:37.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.858+0000 7f43de3b5640 10 monclient: get_monmap_and_config 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.858+0000 7f43de3b5640 10 monclient: build_initial_monmap 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.858+0000 7f43de3b5640 10 monclient: monmap: 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:37.859297+0000 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:37.859297+0000 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.860 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:37.861 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:37.861 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:37.861 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:37.861 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:37.861 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 Processor -- start 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 -- start start 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient: init 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43d809c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f43d809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:37.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f43d813a5c0 0x7f43d813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f43d813a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43d809a430 con 0x7f43d809c180 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f43d80a24e0 con 0x7f43d813a5c0 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43d80a2bf0 con 0x7f43d809d160 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43de3b5640 10 monclient(hunting): authenticate will time out at 1316.346696s 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f43d813a5c0 0x7f43d813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34028/0 (socket says 172.21.15.164:34028) 2024-09-10T16:15:37.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f43d813a5c0 0x7f43d813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47440/0 (socket says 172.21.15.164:47440) 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 -- 172.21.15.164:0/224628669 learned_addr learned my addr 172.21.15.164:0/224628669 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 10 monclient(hunting): get_auth_request con 0x7f43d809d160 auth_method 0 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): get_auth_request con 0x7f43d813a5c0 auth_method 0 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 -- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 msgr2=0x7f43d809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 -- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 msgr2=0x7f43d809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:37.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.859+0000 7f43d77fe640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 10 monclient(hunting): handle_auth_done global_id 74394 payload 274 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 1 -- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 msgr2=0x7f43d809f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 1 -- 172.21.15.164:0/224628669 >> v1:172.21.15.165:6789/0 conn(0x7f43d809c180 legacy=0x7f43d809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 10 monclient: _finish_hunting 0 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 1 monclient: found mon.noname-a 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.860+0000 7f43dc952640 1 -- 172.21.15.164:0/224628669 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d809c560 con 0x7f43d813a5c0 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.862+0000 7f43dc952640 1 -- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 msgr2=0x7f43d813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.862+0000 7f43dc952640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d813c9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f43d809abb0 tx=0x7f43d002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e018a98165e1ba4a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 1 -- 172.21.15.164:0/224628669 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0007950 con 0x7f43d813a5c0 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: dump: 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.869 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 1 -- 172.21.15.164:0/224628669 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d0007af0 con 0x7f43d813a5c0 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 5 monclient: authenticate success, global_id 74394 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 1 -- 172.21.15.164:0/224628669 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0031400 con 0x7f43d813a5c0 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 4 set_mon_vals no callback set 2024-09-10T16:15:37.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43d6ffd640 10 monclient: dump: 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 10 monclient: shutdown 2024-09-10T16:15:37.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 -- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 msgr2=0x7f43d813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d813c9b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f43d809abb0 tx=0x7f43d002a480 comp rx=0 tx=0).stop 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 -- 172.21.15.164:0/224628669 shutdown_connections 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809d160 0x7f43d809f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 --2- 172.21.15.164:0/224628669 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d813c9b0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 -- 172.21.15.164:0/224628669 >> 172.21.15.164:0/224628669 conn(0x7f43d80a3110 msgr2=0x7f43d80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 -- 172.21.15.164:0/224628669 shutdown_connections 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.863+0000 7f43de3b5640 1 -- 172.21.15.164:0/224628669 wait complete. 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.864+0000 7f43de3b5640 10 monclient: build_initial_monmap 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.864+0000 7f43de3b5640 10 monclient: monmap: 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:37.872 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:37.873 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.864+0000 7f43de3b5640 1 Processor -- start 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.864+0000 7f43de3b5640 1 -- start start 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.864+0000 7f43de3b5640 10 monclient: init 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f43d809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d809d160 0x7f43d8092590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f43d809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f43d813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:37.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8074f90 con 0x7f43d813a5c0 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8074e10 con 0x7f43d809d160 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f43d809f690 con 0x7f43d809c180 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43de3b5640 10 monclient(hunting): authenticate will time out at 1316.352182s 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34038/0 (socket says 172.21.15.164:34038) 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47454/0 (socket says 172.21.15.164:47454) 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 learned_addr learned my addr 172.21.15.164:0/2176557524 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 10 monclient(hunting): get_auth_request con 0x7f43d809c180 auth_method 0 2024-09-10T16:15:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): get_auth_request con 0x7f43d813a5c0 auth_method 0 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 msgr2=0x7f43d8094a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 msgr2=0x7f43d8094a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d77fe640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d809d160 0x7f43d8092590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d77fe640 10 monclient(hunting): get_auth_request con 0x7f43d809d160 auth_method 0 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient(hunting): handle_auth_done global_id 74397 payload 824 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 msgr2=0x7f43d8094a70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f43d809c180 0x7f43d8094a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d809d160 msgr2=0x7f43d8092590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f43d809d160 0x7f43d8092590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient: _finish_hunting 0 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 monclient: found mon.noname-a 2024-09-10T16:15:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.865+0000 7f43dc952640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43d8096270 con 0x7f43d813a5c0 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43dc952640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 msgr2=0x7f43d8092ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43dc952640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f43d0031000 tx=0x7f43d0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d0040070 con 0x7f43d813a5c0 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43d4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.867+0000 7f43d4ff9640 10 monclient: dump: 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d0045660 con 0x7f43d813a5c0 2024-09-10T16:15:37.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 5 monclient: authenticate success, global_id 74397 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 monclient: _renew_subs 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43d8095400 con 0x7f43d813a5c0 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d00314c0 con 0x7f43d813a5c0 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 10 monclient: dump: 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.886 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 monclient: _renew_subs 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43d8093f50 con 0x7f43d813a5c0 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d67fc640 4 set_mon_vals no callback set 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 0x7f43d808df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43d809ea40 con 0x7f43d8093010 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f43d0039040 con 0x7f43d813a5c0 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 0x7f43d808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d7fff640 10 monclient: get_auth_request con 0x7f43d8093010 auth_method 0 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.868+0000 7f43d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d4ff9640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f43c403e210 0x7f43c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(319..319 src has 184..319) ==== 4831+0+0 (secure 0 0 0) 0x7f43d00746e0 con 0x7f43d813a5c0 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d7fff640 10 monclient(hunting): handle_auth_done global_id 74400 payload 824 2024-09-10T16:15:37.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43d8095ef0 con 0x7f43d8093010 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d77fe640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f43c403e210 0x7f43c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.869+0000 7f43d77fe640 10 monclient: get_auth_request con 0x7f43c403e210 auth_method 0 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 msgr2=0x7f43d808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 0x7f43d808df80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43c800e4f0 tx=0x7f43c800e9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c801a070 con 0x7f43d8093010 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 10 monclient: dump: 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.889 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f43c8007450 con 0x7f43d8093010 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 msgr2=0x7f43d808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 0x7f43d808df80 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43c800e4f0 tx=0x7f43c800e9b0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 msgr2=0x7f43d808df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.871+0000 7f43d4ff9640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d8093010 0x7f43d808df80 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f43c800e4f0 tx=0x7f43c800e9b0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.872+0000 7f43d77fe640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f43c403e210 0x7f43c40406d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43d8094a00 tx=0x7f43cc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43de3b5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43de3b5640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 0x7f43d808e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43de3b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f43d8077260 con 0x7f43d80933f0 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 0x7f43d808e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.877+0000 7f43d7fff640 10 monclient: get_auth_request con 0x7f43d80933f0 auth_method 0 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:37.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:37.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:37.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:37.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 10 monclient(hunting): handle_auth_done global_id 74403 payload 824 2024-09-10T16:15:37.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.878+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43d81d6940 con 0x7f43d80933f0 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d7fff640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 msgr2=0x7f43d808e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d7fff640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 0x7f43d808e4c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43c8000c00 tx=0x7f43c8003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43c801a030 con 0x7f43d80933f0 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 10 monclient: dump: 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f43c801b040 con 0x7f43d80933f0 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 msgr2=0x7f43d808e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 0x7f43d808e4c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43c8000c00 tx=0x7f43c8003760 comp rx=0 tx=0).stop 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 msgr2=0x7f43d808e4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:37.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.880+0000 7f43d4ff9640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d80933f0 0x7f43d808e4c0 secure :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f43c8000c00 tx=0x7f43c8003760 comp rx=0 tx=0).stop 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 165747, 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:37.895 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:37.896 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:37.897 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:37.898 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:37.899 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:37.900 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.901 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.902 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.903 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.904 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:37.905 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.906 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.907 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:37.908 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:37.909 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f43c403e210 msgr2=0x7f43c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f43c403e210 0x7f43c40406d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f43d8094a00 tx=0x7f43cc008040 comp rx=0 tx=0).stop 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 10 monclient: shutdown 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 msgr2=0x7f43d8092ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43dc952640 1 -- 172.21.15.164:0/2176557524 reap_dead start 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f43d0031000 tx=0x7f43d0004a40 comp rx=0 tx=0).stop 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 shutdown_connections 2024-09-10T16:15:37.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 --2- 172.21.15.164:0/2176557524 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f43d813a5c0 0x7f43d8092ad0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:37.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 >> 172.21.15.164:0/2176557524 conn(0x7f43d80a3110 msgr2=0x7f43d80729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:37.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 shutdown_connections 2024-09-10T16:15:37.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:37.881+0000 7f43de3b5640 1 -- 172.21.15.164:0/2176557524 wait complete. 2024-09-10T16:15:37.911 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:37.911 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:15:37.911 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.008+0000 7f0529ab8640 10 monclient: get_monmap_and_config 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.008+0000 7f0529ab8640 10 monclient: build_initial_monmap 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.008+0000 7f0529ab8640 10 monclient: monmap: 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:38.009431+0000 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:38.009431+0000 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:38.010 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:38.011 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 Processor -- start 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 -- start start 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient: init 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f052409c310 0x7f052409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-a con 0x7f052409c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f052409d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:38.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f052409f4c0 0x7f052413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-c con 0x7f052409f4c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f052409fbc0 con 0x7f052409c310 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f05240a2a20 con 0x7f052409f4c0 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f052409c6f0 con 0x7f052409d190 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0529ab8640 10 monclient(hunting): authenticate will time out at 1316.496862s 2024-09-10T16:15:38.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f052409c310 0x7f052409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f052409c310 0x7f052409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47480/0 (socket says 172.21.15.164:47480) 2024-09-10T16:15:38.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 1 -- 172.21.15.164:0/3154295865 learned_addr learned my addr 172.21.15.164:0/3154295865 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:38.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 10 monclient(hunting): get_auth_request con 0x7f052409c310 auth_method 0 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f05237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.009+0000 7f0522ffd640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 10 monclient(hunting): get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 1 --2- 172.21.15.164:0/3154295865 >> v2:172.21.15.165:3300/0 conn(0x7f052409f4c0 0x7f052413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 1 -- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f052409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): get_auth_request con 0x7f052409f4c0 auth_method 0 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 1 -- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f052409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0522ffd640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f0523fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient(hunting): handle_auth_done global_id 74406 payload 274 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 -- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f052409ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 -- 172.21.15.164:0/3154295865 >> v2:172.21.15.165:3300/0 conn(0x7f052409f4c0 msgr2=0x7f052413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:38.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 --2- 172.21.15.164:0/3154295865 >> v2:172.21.15.165:3300/0 conn(0x7f052409f4c0 0x7f052413b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.010+0000 7f05237fe640 1 -- 172.21.15.164:0/3154295865 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f052409a4d0 con 0x7f052409c310 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05237fe640 1 -- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 msgr2=0x7f052409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05237fe640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052409ea40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0524074d80 tx=0x7f051402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52b0bec7d8b8d93b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 1 -- 172.21.15.164:0/3154295865 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514037070 con 0x7f052409c310 2024-09-10T16:15:38.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:38.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:38.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: dump: 2024-09-10T16:15:38.029 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:38.029 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:38.034 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 1 -- 172.21.15.164:0/3154295865 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0514032640 con 0x7f052409c310 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 5 monclient: authenticate success, global_id 74406 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 1 -- 172.21.15.164:0/3154295865 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514032a10 con 0x7f052409c310 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 4 set_mon_vals no callback set 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:38.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f05227fc640 10 monclient: dump: 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 10 monclient: shutdown 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:38.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 -- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 msgr2=0x7f052409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052409ea40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f0524074d80 tx=0x7f051402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 -- 172.21.15.164:0/3154295865 shutdown_connections 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 --2- 172.21.15.164:0/3154295865 >> v2:172.21.15.165:3300/0 conn(0x7f052409f4c0 0x7f052413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f052409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 --2- 172.21.15.164:0/3154295865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052409ea40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 -- 172.21.15.164:0/3154295865 >> 172.21.15.164:0/3154295865 conn(0x7f05240a3110 msgr2=0x7f05240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 -- 172.21.15.164:0/3154295865 shutdown_connections 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.013+0000 7f0529ab8640 1 -- 172.21.15.164:0/3154295865 wait complete. 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.014+0000 7f0529ab8640 10 monclient: build_initial_monmap 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.014+0000 7f0529ab8640 10 monclient: monmap: 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:38.037 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.014+0000 7f0529ab8640 1 Processor -- start 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.014+0000 7f0529ab8640 1 -- start start 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.014+0000 7f0529ab8640 10 monclient: init 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:38.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052408f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-a con 0x7f052409c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f052409f4c0 0x7f052408f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f052409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 0x7f05241c89c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): picked mon.noname-c con 0x7f052408fcb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05240a2230 con 0x7f052409c310 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05240a20b0 con 0x7f052409f4c0 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0524074e00 con 0x7f052408fcb0 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0529ab8640 10 monclient(hunting): authenticate will time out at 1316.502359s 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052408f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 0x7f05241c89c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052408f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47492/0 (socket says 172.21.15.164:47492) 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 -- 172.21.15.164:0/1703623534 learned_addr learned my addr 172.21.15.164:0/1703623534 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:38.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): get_auth_request con 0x7f052409c310 auth_method 0 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 10 monclient(hunting): get_auth_request con 0x7f052408fcb0 auth_method 0 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 msgr2=0x7f05241c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 msgr2=0x7f05241c89c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:38.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 0x7f05241c89c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0523fff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 0x7f05241c89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f052409f4c0 0x7f052408f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 10 monclient(hunting): get_auth_request con 0x7f052409f4c0 auth_method 0 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient(hunting): handle_auth_done global_id 74409 payload 824 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 msgr2=0x7f05241c89c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052408fcb0 0x7f05241c89c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f052409f4c0 msgr2=0x7f052408f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f052409f4c0 0x7f052408f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:38.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:38.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:38.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:38.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f05237fe640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0524090d60 con 0x7f052409c310 2024-09-10T16:15:38.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.015+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f052409f4c0 0x7f052408f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f05237fe640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 msgr2=0x7f052408f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f05237fe640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052408f230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f05140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514041020 con 0x7f052409c310 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: dump: 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:38.044 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: _finish_auth 0 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f051403d830 con 0x7f052409c310 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 5 monclient: authenticate success, global_id 74409 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 monclient: _renew_subs 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05240913e0 con 0x7f052409c310 2024-09-10T16:15:38.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0514037040 con 0x7f052409c310 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f050bfff640 10 monclient: dump: 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:38.046 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 monclient: _renew_subs 2024-09-10T16:15:38.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0524092340 con 0x7f052409c310 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0521ffb640 4 set_mon_vals no callback set 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.018+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f052409d930 con 0x7f052409d190 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0514044020 con 0x7f052409c310 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:38.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f050bfff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f051003e210 0x7f05100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(319..319 src has 184..319) ==== 4831+0+0 (secure 0 0 0) 0x7f05140732e0 con 0x7f052409c310 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0523fff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f051003e210 0x7f05100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.019+0000 7f0523fff640 10 monclient: get_auth_request con 0x7f051003e210 auth_method 0 2024-09-10T16:15:38.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.022+0000 7f0523fff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f051003e210 0x7f05100406d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f051c00bd60 tx=0x7f051c00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:38.171 INFO:tasks.rados.rados.0.smithi165.stdout:1279: done (0 left) 2024-09-10T16:15:38.171 INFO:tasks.rados.rados.0.smithi165.stdout:1280: snap_remove snap 145 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:38.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:38.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.220+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:38.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:38.621+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:1280: done (0 left) 2024-09-10T16:15:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:1281: rollback oid 6 current snap is 146 2024-09-10T16:15:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:1281: done (0 left) 2024-09-10T16:15:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:1282: read oid 3 snap -1 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1282: expect (ObjNum 446 snap 143 seq_num 446) 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1283: read oid 35 snap -1 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1283: expect (ObjNum 423 snap 138 seq_num 423) 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1284: copy_from oid 30 from oid 29 current snap is 146 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1285: read oid 17 snap -1 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1285: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1286: write (excl) oid 33 current snap is 146 2024-09-10T16:15:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:1286: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-10T16:15:39.180 INFO:tasks.rados.rados.0.smithi165.stdout:1286: writing smithi16534841-33 from 598448 to 1011517 tid 1 2024-09-10T16:15:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:1286: writing smithi16534841-33 from 1768939 to 2293087 tid 2 2024-09-10T16:15:39.194 INFO:tasks.rados.rados.0.smithi165.stdout:1286: writing smithi16534841-33 from 2775798 to 3357659 tid 3 2024-09-10T16:15:39.197 INFO:tasks.rados.rados.0.smithi165.stdout:1286: writing smithi16534841-33 from 3781488 to 3798448 tid 4 2024-09-10T16:15:39.197 INFO:tasks.rados.rados.0.smithi165.stdout:1287: read oid 31 snap -1 2024-09-10T16:15:39.197 INFO:tasks.rados.rados.0.smithi165.stdout:1287: expect (ObjNum 315 snap 91 seq_num 315) 2024-09-10T16:15:39.197 INFO:tasks.rados.rados.0.smithi165.stdout:1288: copy_from oid 13 from oid 38 current snap is 146 2024-09-10T16:15:39.197 INFO:tasks.rados.rados.0.smithi165.stdout:1285: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:15:39.202 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 786523 2024-09-10T16:15:39.239 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 438482 2024-09-10T16:15:39.305 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1719962 2024-09-10T16:15:39.315 INFO:tasks.rados.rados.0.smithi165.stdout:1284: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:15:39.315 INFO:tasks.rados.rados.0.smithi165.stdout:1282: expect (ObjNum 446 snap 143 seq_num 446) 2024-09-10T16:15:39.316 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:39.378 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:1284: finishing copy_from to smithi16534841-30 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 567 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 1 to smithi16534841-33 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:1282: done (6 left) 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:1284: done (5 left) 2024-09-10T16:15:39.417 INFO:tasks.rados.rados.0.smithi165.stdout:1285: done (4 left) 2024-09-10T16:15:39.418 INFO:tasks.rados.rados.0.smithi165.stdout:1289: write oid 15 current snap is 146 2024-09-10T16:15:39.418 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 2 to smithi16534841-33 2024-09-10T16:15:39.418 INFO:tasks.rados.rados.0.smithi165.stdout:1289: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-10T16:15:39.422 INFO:tasks.rados.rados.0.smithi165.stdout:1289: writing smithi16534841-15 from 615255 to 1103573 tid 1 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:39.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:39.422+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:15:39.431 INFO:tasks.rados.rados.0.smithi165.stdout:1289: writing smithi16534841-15 from 1827421 to 2611580 tid 2 2024-09-10T16:15:39.438 INFO:tasks.rados.rados.0.smithi165.stdout:1289: writing smithi16534841-15 from 3203221 to 3815255 tid 3 2024-09-10T16:15:39.438 INFO:tasks.rados.rados.0.smithi165.stdout:1290: read oid 32 snap -1 2024-09-10T16:15:39.438 INFO:tasks.rados.rados.0.smithi165.stdout:1290: expect deleted 2024-09-10T16:15:39.438 INFO:tasks.rados.rados.0.smithi165.stdout:1291: write (excl) oid 47 current snap is 146 2024-09-10T16:15:39.439 INFO:tasks.rados.rados.0.smithi165.stdout:1291: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2024-09-10T16:15:39.444 INFO:tasks.rados.rados.0.smithi165.stdout:1291: writing smithi16534841-47 from 632062 to 1195629 tid 1 2024-09-10T16:15:39.452 INFO:tasks.rados.rados.0.smithi165.stdout:1291: writing smithi16534841-47 from 1885903 to 2413720 tid 2 2024-09-10T16:15:39.457 INFO:tasks.rados.rados.0.smithi165.stdout:1291: writing smithi16534841-47 from 2830644 to 3551402 tid 3 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1292: read oid 42 snap -1 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1292: expect deleted 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1293: delete oid 39 current snap is 146 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 3 to smithi16534841-33 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 4 to smithi16534841-33 2024-09-10T16:15:39.458 INFO:tasks.rados.rados.0.smithi165.stdout:1287: expect (ObjNum 315 snap 91 seq_num 315) 2024-09-10T16:15:39.551 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 5 to smithi16534841-33 2024-09-10T16:15:39.551 INFO:tasks.rados.rados.0.smithi165.stdout:1286: finishing write tid 6 to smithi16534841-33 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: oid 33 updating version 0 to 891 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: oid 33 version 891 is already newer than 888 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: oid 33 version 891 is already newer than 890 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: oid 33 version 891 is already newer than 889 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: oid 33 version 891 is already newer than 887 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 891 (ObjNum 464 snap 146 seq_num 464) dirty exists 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: left oid 33 (ObjNum 464 snap 146 seq_num 464) 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1286: done (8 left) 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1287: done (7 left) 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1293: done (6 left) 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1294: write oid 37 current snap is 146 2024-09-10T16:15:39.552 INFO:tasks.rados.rados.0.smithi165.stdout:1294: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-10T16:15:39.557 INFO:tasks.rados.rados.0.smithi165.stdout:1294: writing smithi16534841-37 from 665676 to 1379741 tid 1 2024-09-10T16:15:39.562 INFO:tasks.rados.rados.0.smithi165.stdout:1294: writing smithi16534841-37 from 1886514 to 2418000 tid 2 2024-09-10T16:15:39.570 INFO:tasks.rados.rados.0.smithi165.stdout:1294: writing smithi16534841-37 from 3169137 to 3865676 tid 3 2024-09-10T16:15:39.570 INFO:tasks.rados.rados.0.smithi165.stdout:1295: write (excl) oid 22 current snap is 146 2024-09-10T16:15:39.570 INFO:tasks.rados.rados.0.smithi165.stdout:1295: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-10T16:15:39.578 INFO:tasks.rados.rados.0.smithi165.stdout:1295: writing smithi16534841-22 from 682483 to 1471797 tid 1 2024-09-10T16:15:39.585 INFO:tasks.rados.rados.0.smithi165.stdout:1295: writing smithi16534841-22 from 1944996 to 2736493 tid 2 2024-09-10T16:15:39.591 INFO:tasks.rados.rados.0.smithi165.stdout:1295: writing smithi16534841-22 from 3196560 to 3751134 tid 3 2024-09-10T16:15:39.591 INFO:tasks.rados.rados.0.smithi165.stdout:1296: snap_create 2024-09-10T16:15:39.591 INFO:tasks.rados.rados.0.smithi165.stdout:1283: expect (ObjNum 423 snap 138 seq_num 423) 2024-09-10T16:15:39.618 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2730108 2024-09-10T16:15:39.649 INFO:tasks.rados.rados.0.smithi165.stdout:1289: finishing write tid 1 to smithi16534841-15 2024-09-10T16:15:39.649 INFO:tasks.rados.rados.0.smithi165.stdout:1288: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:15:39.649 INFO:tasks.rados.rados.0.smithi165.stdout:1288: finishing copy_from to smithi16534841-13 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 276 (ObjNum 440 snap 142 seq_num 440) dirty exists 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: finishing write tid 2 to smithi16534841-15 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: finishing write tid 3 to smithi16534841-15 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: finishing write tid 4 to smithi16534841-15 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: finishing write tid 5 to smithi16534841-15 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: oid 15 updating version 0 to 700 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: oid 15 version 700 is already newer than 698 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: oid 15 updating version 700 to 701 2024-09-10T16:15:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:1289: oid 15 version 701 is already newer than 699 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 701 (ObjNum 465 snap 146 seq_num 465) dirty exists 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1289: left oid 15 (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: finishing write tid 1 to smithi16534841-47 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: finishing write tid 2 to smithi16534841-47 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: finishing write tid 3 to smithi16534841-47 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: finishing write tid 4 to smithi16534841-47 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: finishing write tid 5 to smithi16534841-47 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: oid 47 updating version 0 to 850 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: oid 47 updating version 850 to 853 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: oid 47 version 853 is already newer than 852 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: oid 47 version 853 is already newer than 851 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 853 (ObjNum 466 snap 146 seq_num 466) dirty exists 2024-09-10T16:15:39.651 INFO:tasks.rados.rados.0.smithi165.stdout:1291: left oid 47 (ObjNum 466 snap 146 seq_num 466) 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: finishing write tid 1 to smithi16534841-37 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: finishing write tid 2 to smithi16534841-37 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: finishing write tid 3 to smithi16534841-37 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: finishing write tid 4 to smithi16534841-37 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: finishing write tid 5 to smithi16534841-37 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: oid 37 updating version 0 to 567 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: oid 37 version 567 is already newer than 566 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: oid 37 version 567 is already newer than 565 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: oid 37 updating version 567 to 568 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 568 (ObjNum 468 snap 146 seq_num 468) dirty exists 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1294: left oid 37 (ObjNum 468 snap 146 seq_num 468) 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1295: finishing write tid 1 to smithi16534841-22 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1295: finishing write tid 2 to smithi16534841-22 2024-09-10T16:15:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1295: finishing write tid 3 to smithi16534841-22 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: finishing write tid 4 to smithi16534841-22 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: finishing write tid 5 to smithi16534841-22 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: oid 22 updating version 0 to 731 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: oid 22 version 731 is already newer than 730 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: oid 22 updating version 731 to 732 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: oid 22 version 732 is already newer than 729 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 732 (ObjNum 469 snap 146 seq_num 469) dirty exists 2024-09-10T16:15:39.653 INFO:tasks.rados.rados.0.smithi165.stdout:1295: left oid 22 (ObjNum 469 snap 146 seq_num 469) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1283: done (8 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1288: done (7 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1289: done (6 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1290: done (5 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1291: done (4 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1292: done (3 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1294: done (2 left) 2024-09-10T16:15:40.177 INFO:tasks.rados.rados.0.smithi165.stdout:1295: done (1 left) 2024-09-10T16:15:40.178 INFO:tasks.rados.rados.0.smithi165.stdout:1296: done (0 left) 2024-09-10T16:15:40.178 INFO:tasks.rados.rados.0.smithi165.stdout:1297: rollback oid 10 current snap is 147 2024-09-10T16:15:40.178 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 146 2024-09-10T16:15:40.178 INFO:tasks.rados.rados.0.smithi165.stdout:1298: copy_from oid 20 from oid 36 current snap is 147 2024-09-10T16:15:40.178 INFO:tasks.rados.rados.0.smithi165.stdout:1299: snap_create 2024-09-10T16:15:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:1297: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:15:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:1297: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:15:40.183 INFO:tasks.rados.rados.0.smithi165.stdout:1297: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:15:40.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 998 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:15:40.196 INFO:tasks.rados.rados.0.smithi165.stdout:1298: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:15:40.197 INFO:tasks.rados.rados.0.smithi165.stdout:1298: finishing copy_from to smithi16534841-20 2024-09-10T16:15:40.197 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 605 (ObjNum 404 snap 130 seq_num 404) dirty exists 2024-09-10T16:15:41.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.015+0000 7f050affd640 10 monclient: tick 2024-09-10T16:15:41.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.015+0000 7f050affd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:41.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.015+0000 7f050affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:41.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.015+0000 7f050affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f052409d190 auth_method 0 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 10 monclient(hunting): handle_auth_done global_id 104099 payload 824 2024-09-10T16:15:41.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.023+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0524092680 con 0x7f052409d190 2024-09-10T16:15:41.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.025+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.025+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f0518017e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518013ba0 con 0x7f052409d190 2024-09-10T16:15:41.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 10 monclient: dump: 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.030 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0518013ba0 con 0x7f052409d190 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f0518017e10 comp rx=0 tx=0).stop 2024-09-10T16:15:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 msgr2=0x7f05241c8f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:41.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.028+0000 7f050bfff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f052409d190 0x7f05241c8f00 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f0518017e10 comp rx=0 tx=0).stop 2024-09-10T16:15:41.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0529ab8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0529ab8640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 0x7f05240717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0529ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0524092890 con 0x7f05241c9440 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 0x7f05240717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient: get_auth_request con 0x7f05241c9440 auth_method 0 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.035+0000 7f0522ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.036+0000 7f0522ffd640 10 monclient(hunting): handle_auth_done global_id 104102 payload 824 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.036+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05241c9850 con 0x7f05241c9440 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f0522ffd640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 msgr2=0x7f05240717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f0522ffd640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 0x7f05240717f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f05180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0518027af0 con 0x7f05241c9440 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 10 monclient: dump: 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.045 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f0518027af0 con 0x7f05241c9440 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:41.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:41.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 msgr2=0x7f05240717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 0x7f05240717f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f05180046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 msgr2=0x7f05240717f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:41.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.038+0000 7f050bfff640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05241c9440 0x7f05240717f0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0518006fd0 tx=0x7f05180046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10355, 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:41.048 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:41.049 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:41.049 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:41.049 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.049 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:15:41.049 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.050 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.051 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:41.052 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:41.053 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:41.053 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:41.053 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:41.053 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.054 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:41.055 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:41.058 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.060 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:41.061 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:41.062 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:41.062 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.062 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:41.063 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f051003e210 msgr2=0x7f05100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f051003e210 0x7f05100406d0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f051c00bd60 tx=0x7f051c00d040 comp rx=0 tx=0).stop 2024-09-10T16:15:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 10 monclient: shutdown 2024-09-10T16:15:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:41.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 msgr2=0x7f052408f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 --2- 172.21.15.164:0/1703623534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f052409c310 0x7f052408f230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f0514000c00 tx=0x7f05140046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0523fff640 1 -- 172.21.15.164:0/1703623534 reap_dead start 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0523fff640 1 -- 172.21.15.164:0/1703623534 reap_dead start 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 shutdown_connections 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 >> 172.21.15.164:0/1703623534 conn(0x7f05240a3110 msgr2=0x7f05241bc370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 shutdown_connections 2024-09-10T16:15:41.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.039+0000 7f0529ab8640 1 -- 172.21.15.164:0/1703623534 wait complete. 2024-09-10T16:15:41.067 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:41.067 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:15:41.067 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:15:41.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.173+0000 7fa8766b1640 10 monclient: get_monmap_and_config 2024-09-10T16:15:41.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.173+0000 7fa8766b1640 10 monclient: build_initial_monmap 2024-09-10T16:15:41.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.173+0000 7fa8766b1640 10 monclient: monmap: 2024-09-10T16:15:41.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:41.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:41.174763+0000 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:41.174763+0000 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:41.176 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.174+0000 7fa8766b1640 1 Processor -- start 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.174+0000 7fa8766b1640 1 -- start start 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.174+0000 7fa8766b1640 10 monclient: init 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa87006cb70 0x7fa870134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-c con 0x7fa87006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 0x7fa8700a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-b con 0x7fa8700a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa870137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa87009ce50 con 0x7fa870137ca0 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa87006db60 con 0x7fa87006cb70 2024-09-10T16:15:41.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa87006cf50 con 0x7fa8700a46a0 2024-09-10T16:15:41.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:41.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa8766b1640 10 monclient(hunting): authenticate will time out at 1319.662532s 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa87006cb70 0x7fa870134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa87006cb70 0x7fa870134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:50416/0 (socket says 172.21.15.164:50416) 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 1 -- 172.21.15.164:0/2254718588 learned_addr learned my addr 172.21.15.164:0/2254718588 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): get_auth_request con 0x7fa87006cb70 auth_method 0 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 0x7fa8700a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request con 0x7fa8700a46a0 auth_method 0 2024-09-10T16:15:41.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.175+0000 7fa86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 10 monclient(hunting): handle_auth_done global_id 94186 payload 274 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 1 -- 172.21.15.164:0/2254718588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 msgr2=0x7fa8700a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 0x7fa8700a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 1 -- 172.21.15.164:0/2254718588 >> v1:172.21.15.164:6789/0 conn(0x7fa870137ca0 legacy=0x7fa870138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 1 monclient: found mon.noname-c 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:41.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:41.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86ffff640 1 -- 172.21.15.164:0/2254718588 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87009cb70 con 0x7fa87006cb70 2024-09-10T16:15:41.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.176+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 0x7fa8700a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:41.183 INFO:tasks.rados.rados.0.smithi165.stdout:1297: done (2 left) 2024-09-10T16:15:41.183 INFO:tasks.rados.rados.0.smithi165.stdout:1298: done (1 left) 2024-09-10T16:15:41.183 INFO:tasks.rados.rados.0.smithi165.stdout:1299: done (0 left) 2024-09-10T16:15:41.184 INFO:tasks.rados.rados.0.smithi165.stdout:1300: delete oid 31 current snap is 148 2024-09-10T16:15:41.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.179+0000 7fa86ffff640 1 -- 172.21.15.164:0/2254718588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87006cb70 msgr2=0x7fa870134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.179+0000 7fa86ffff640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87006cb70 0x7fa870134f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa86002a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb767bb9b19373c8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 1 -- 172.21.15.164:0/2254718588 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860037070 con 0x7fa87006cb70 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: dump: 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: _finish_auth 0 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 1 -- 172.21.15.164:0/2254718588 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa860032440 con 0x7fa87006cb70 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa8766b1640 5 monclient: authenticate success, global_id 94186 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa8766b1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa8766b1640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 1 -- 172.21.15.164:0/2254718588 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860032810 con 0x7fa87006cb70 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa8766b1640 4 set_mon_vals no callback set 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:41.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.180+0000 7fa86effd640 10 monclient: dump: 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 10 monclient: shutdown 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:41.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 -- 172.21.15.164:0/2254718588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87006cb70 msgr2=0x7fa870134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87006cb70 0x7fa870134f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa86002a0a0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 -- 172.21.15.164:0/2254718588 shutdown_connections 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8700a46a0 0x7fa8700a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 --2- 172.21.15.164:0/2254718588 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87006cb70 0x7fa870134f70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 -- 172.21.15.164:0/2254718588 >> 172.21.15.164:0/2254718588 conn(0x7fa8700aa900 msgr2=0x7fa8700aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 -- 172.21.15.164:0/2254718588 shutdown_connections 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.181+0000 7fa8766b1640 1 -- 172.21.15.164:0/2254718588 wait complete. 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.182+0000 7fa8766b1640 10 monclient: build_initial_monmap 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.182+0000 7fa8766b1640 10 monclient: monmap: 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:41.191 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.182+0000 7fa8766b1640 1 Processor -- start 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 -- start start 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient: init 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:15:41.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 0x7fa8701cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-c con 0x7fa87006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 0x7fa8701cd670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-a con 0x7fa8700a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870137ca0 0x7fa8701cdbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): picked mon.noname-b con 0x7fa870137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8701389c0 con 0x7fa8700a46a0 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa870138840 con 0x7fa870137ca0 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa870140040 con 0x7fa87006cb70 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa8766b1640 10 monclient(hunting): authenticate will time out at 1319.670520s 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 0x7fa8701cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 0x7fa8701cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 0x7fa8701cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34150/0 (socket says 172.21.15.164:34150) 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 0x7fa8701cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47514/0 (socket says 172.21.15.164:47514) 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 1 -- 172.21.15.164:0/3591291131 learned_addr learned my addr 172.21.15.164:0/3591291131 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): get_auth_request con 0x7fa87006cb70 auth_method 0 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request con 0x7fa8700a46a0 auth_method 0 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870137ca0 0x7fa8701cdbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): get_auth_request con 0x7fa870137ca0 auth_method 0 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa86ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.183+0000 7fa874c4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 10 monclient(hunting): handle_auth_done global_id 104108 payload 824 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870137ca0 msgr2=0x7fa8701cdbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa870137ca0 0x7fa8701cdbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 msgr2=0x7fa8701cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 0x7fa8701cd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 monclient: found mon.noname-c 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86ffff640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa87013ebc0 con 0x7fa87006cb70 2024-09-10T16:15:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.184+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8700a46a0 0x7fa8701cd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86ffff640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 msgr2=0x7fa8701cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86ffff640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 0x7fa8701cd130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa8600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1300: done (0 left) 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1301: rollback oid 48 current snap is 148 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 146 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1302: read oid 38 snap -1 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1302: expect (ObjNum 440 snap 142 seq_num 440) 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1303: snap_create 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1301: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1301: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:15:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:1301: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:15:41.200 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1059 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:15:41.200 INFO:tasks.rados.rados.0.smithi165.stdout:1302: expect (ObjNum 440 snap 142 seq_num 440) 2024-09-10T16:15:41.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860041020 con 0x7fa87006cb70 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: dump: 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 5 monclient: authenticate success, global_id 104108 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 10 monclient: _renew_subs 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8700a95e0 con 0x7fa87006cb70 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa86003d630 con 0x7fa87006cb70 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:41.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa860037040 con 0x7fa87006cb70 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86cff9640 10 monclient: dump: 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.206 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa86e7fc640 4 set_mon_vals no callback set 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 10 monclient: _renew_subs 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.186+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa87013ae80 con 0x7fa87006cb70 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.187+0000 7fa8766b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.187+0000 7fa8766b1640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 0x7fa87012e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa870134f70 con 0x7fa87012e020 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa860044020 con 0x7fa87006cb70 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86cff9640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e210 0x7fa85c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 0x7fa87012e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient: get_auth_request con 0x7fa87012e020 auth_method 0 2024-09-10T16:15:41.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(323..323 src has 184..323) ==== 4787+0+0 (secure 0 0 0) 0x7fa860073020 con 0x7fa87006cb70 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.188+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.189+0000 7fa874c4e640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e210 0x7fa85c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.189+0000 7fa874c4e640 10 monclient: get_auth_request con 0x7fa85c03e210 auth_method 0 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.189+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_done global_id 94192 payload 824 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.189+0000 7fa86f7fe640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8700008d0 con 0x7fa87012e020 2024-09-10T16:15:41.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86f7fe640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 msgr2=0x7fa87012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 0x7fa87012e400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa87013d350 tx=0x7fa86400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa874c4e640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e210 0x7fa85c0406d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8680055e0 tx=0x7fa868003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864019070 con 0x7fa87012e020 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 10 monclient: dump: 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.209 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa864004240 con 0x7fa87012e020 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 msgr2=0x7fa87012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 0x7fa87012e400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa87013d350 tx=0x7fa86400f610 comp rx=0 tx=0).stop 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 msgr2=0x7fa87012e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.192+0000 7fa86cff9640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e020 0x7fa87012e400 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.198+0000 7fa8766b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa8766b1640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 0x7fa870130370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa8766b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa87013e320 con 0x7fa87012e940 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 0x7fa870130370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient: get_auth_request con 0x7fa87012e940 auth_method 0 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 10 monclient(hunting): handle_auth_done global_id 94195 payload 824 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.199+0000 7fa86f7fe640 1 -- 172.21.15.164:0/3591291131 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa87012ed50 con 0x7fa87012e940 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.201+0000 7fa86f7fe640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 msgr2=0x7fa870130370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.201+0000 7fa86f7fe640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 0x7fa870130370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa86401b000 tx=0x7fa8640046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa864019040 con 0x7fa87012e940 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 10 monclient: dump: 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.212 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7fa86401e950 con 0x7fa87012e940 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 msgr2=0x7fa870130370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 0x7fa870130370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa86401b000 tx=0x7fa8640046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 msgr2=0x7fa870130370 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa86cff9640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa87012e940 0x7fa870130370 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fa86401b000 tx=0x7fa8640046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.213 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 58702, 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:41.214 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.215 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.216 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:41.217 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:41.218 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:41.219 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.220 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.221 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.222 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:41.223 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:41.224 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:41.225 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.226 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:41.227 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:41.228 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:41.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e210 msgr2=0x7fa85c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa85c03e210 0x7fa85c0406d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fa8680055e0 tx=0x7fa868003d10 comp rx=0 tx=0).stop 2024-09-10T16:15:41.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 10 monclient: shutdown 2024-09-10T16:15:41.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:41.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 msgr2=0x7fa8701cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 --2- 172.21.15.164:0/3591291131 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa87006cb70 0x7fa8701cd130 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa860009870 tx=0x7fa8600046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa874c4e640 1 -- 172.21.15.164:0/3591291131 reap_dead start 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa874c4e640 1 -- 172.21.15.164:0/3591291131 reap_dead start 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 shutdown_connections 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 >> 172.21.15.164:0/3591291131 conn(0x7fa8700aa900 msgr2=0x7fa8700a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 shutdown_connections 2024-09-10T16:15:41.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.202+0000 7fa8766b1640 1 -- 172.21.15.164:0/3591291131 wait complete. 2024-09-10T16:15:41.230 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:41.231 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:15:41.231 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:15:41.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 10 monclient: get_monmap_and_config 2024-09-10T16:15:41.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 10 monclient: build_initial_monmap 2024-09-10T16:15:41.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 10 monclient: monmap: 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:41.328953+0000 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:41.328953+0000 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:41.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 1 Processor -- start 2024-09-10T16:15:41.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 1 -- start start 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.328+0000 7fcd02552640 10 monclient: init 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fccfc06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fccfc134f70 0x7fccfc13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-c con 0x7fccfc134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccfc13d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccfc09ce50 con 0x7fccfc06cb70 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccfc135380 con 0x7fccfc13d560 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fccfc06db60 con 0x7fccfc134f70 2024-09-10T16:15:41.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fcd02552640 10 monclient(hunting): authenticate will time out at 1319.816415s 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfbfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fccfc06cb70 0x7fccfc0a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54122/0 (socket says 172.21.15.164:54122) 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfbfff640 1 -- 172.21.15.164:0/3439138792 learned_addr learned my addr 172.21.15.164:0/3439138792 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 1 --2- 172.21.15.164:0/3439138792 >> v2:172.21.15.165:3300/0 conn(0x7fccfc134f70 0x7fccfc13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): get_auth_request con 0x7fccfc134f70 auth_method 0 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_done global_id 94198 payload 274 2024-09-10T16:15:41.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 1 -- 172.21.15.164:0/3439138792 >> v1:172.21.15.165:6789/0 conn(0x7fccfc13d560 legacy=0x7fccfc13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 1 -- 172.21.15.164:0/3439138792 >> v1:172.21.15.164:6789/0 conn(0x7fccfc06cb70 legacy=0x7fccfc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 1 monclient: found mon.noname-c 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:41.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.329+0000 7fccfb7fe640 1 -- 172.21.15.164:0/3439138792 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc09cb70 con 0x7fccfc134f70 2024-09-10T16:15:41.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.332+0000 7fccfb7fe640 1 -- 172.21.15.164:0/3439138792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc134f70 msgr2=0x7fccfc13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.332+0000 7fccfb7fe640 1 --2- 172.21.15.164:0/3439138792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc134f70 0x7fccfc13d020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fccf00097d0 tx=0x7fccf002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=196e23cdb6291f49 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 1 -- 172.21.15.164:0/3439138792 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0037070 con 0x7fccfc134f70 2024-09-10T16:15:41.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: dump: 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.342 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 1 -- 172.21.15.164:0/3439138792 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf0032440 con 0x7fccfc134f70 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 5 monclient: authenticate success, global_id 94198 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 4 set_mon_vals no callback set 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 1 -- 172.21.15.164:0/3439138792 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0032810 con 0x7fccfc134f70 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:41.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fccfaffd640 10 monclient: dump: 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 10 monclient: shutdown 2024-09-10T16:15:41.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 -- 172.21.15.164:0/3439138792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc134f70 msgr2=0x7fccfc13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 --2- 172.21.15.164:0/3439138792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc134f70 0x7fccfc13d020 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fccf00097d0 tx=0x7fccf002a000 comp rx=0 tx=0).stop 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 -- 172.21.15.164:0/3439138792 shutdown_connections 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 --2- 172.21.15.164:0/3439138792 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc134f70 0x7fccfc13d020 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 -- 172.21.15.164:0/3439138792 >> 172.21.15.164:0/3439138792 conn(0x7fccfc0aa900 msgr2=0x7fccfc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 -- 172.21.15.164:0/3439138792 shutdown_connections 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.333+0000 7fcd02552640 1 -- 172.21.15.164:0/3439138792 wait complete. 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.334+0000 7fcd02552640 10 monclient: build_initial_monmap 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.334+0000 7fcd02552640 10 monclient: monmap: 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:41.345 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.334+0000 7fcd02552640 1 Processor -- start 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.334+0000 7fcd02552640 1 -- start start 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.334+0000 7fcd02552640 10 monclient: init 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:15:41.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 0x7fccfc1cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-c con 0x7fccfc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-a con 0x7fccfc134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc13d560 0x7fccfc0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): picked mon.noname-b con 0x7fccfc13d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc0a4db0 con 0x7fccfc134f70 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fccfc06cf50 con 0x7fccfc13d560 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fccfc141070 con 0x7fccfc06cb70 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:41.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd02552640 10 monclient(hunting): authenticate will time out at 1319.822251s 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47516/0 (socket says 172.21.15.164:47516) 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 0x7fccfc1cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 -- 172.21.15.164:0/1055244949 learned_addr learned my addr 172.21.15.164:0/1055244949 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): get_auth_request con 0x7fccfc134f70 auth_method 0 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): get_auth_request con 0x7fccfc06cb70 auth_method 0 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc13d560 0x7fccfc0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): get_auth_request con 0x7fccfc13d560 auth_method 0 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:41.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fcd00aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient(hunting): handle_auth_done global_id 74415 payload 824 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 -- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 msgr2=0x7fccfc1cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 0x7fccfc1cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 -- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc13d560 msgr2=0x7fccfc0a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc13d560 0x7fccfc0a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfb7fe640 1 -- 172.21.15.164:0/1055244949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccfc09dbe0 con 0x7fccfc134f70 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.335+0000 7fccfbfff640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 0x7fccfc1cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:41.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.337+0000 7fccfb7fe640 1 -- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 msgr2=0x7fccfc0a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.337+0000 7fccfb7fe640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009720 tx=0x7fccf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0041070 con 0x7fccfc134f70 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: dump: 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.351 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 5 monclient: authenticate success, global_id 74415 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccf003c670 con 0x7fccfc134f70 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 10 monclient: _renew_subs 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccfc06e720 con 0x7fccfc134f70 2024-09-10T16:15:41.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccf0037040 con 0x7fccfc134f70 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccf8ff9640 10 monclient: dump: 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:41.353 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 10 monclient: _renew_subs 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccfc09e180 con 0x7fccfc134f70 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.338+0000 7fccfa7fc640 4 set_mon_vals no callback set 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fcd02552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fcd02552640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fcd02552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:41.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccfc0a46a0 con 0x7fccfc134f70 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fccf0044070 con 0x7fccfc134f70 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fccf8ff9640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 0x7fcce80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(323..323 src has 184..323) ==== 4787+0+0 (secure 0 0 0) 0x7fccf0073400 con 0x7fccfc134f70 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fccfbfff640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 0x7fcce80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.339+0000 7fccfbfff640 10 monclient: get_auth_request con 0x7fcce803e210 auth_method 0 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.342+0000 7fccfbfff640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 0x7fcce80406d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccec009de0 tx=0x7fccec0098a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.343+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fccfc06e720 con 0x7fccfc134f70 2024-09-10T16:15:41.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.343+0000 7fccf8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:41.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.343+0000 7fccf8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:41.408 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.586+0000 7fcd02552640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.586+0000 7fcd02552640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.586+0000 7fcd02552640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.586+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fccfc0770f0 con 0x7fccfc134f70 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fccf8ff9640 1 -- 172.21.15.164:0/1055244949 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fccfc09ce50 con 0x7fccfc134f70 2024-09-10T16:15:41.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fccf8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:15:41.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fccf8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=1074637249,config=2586702516,health=514704760,kv=805391194,logm=2651863198}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=12,logm=59} crc {auth=1504210231,config=2586702516,health=3528681917,kv=805391194,logm=3338067773}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=1074637249,config=2586702516,health=514704760,kv=805391194,logm=2651863198}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.587+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=12,kv=12,logm=56} crc {auth=1746384699,config=2586702516,health=3286790080,kv=603532201,logm=2189379477}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075056742}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1757127999}) 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.590 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1075056742}) 2024-09-10T16:15:41.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.588+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322597521}) 2024-09-10T16:15:41.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224332007}) 2024-09-10T16:15:41.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153269436}) 2024-09-10T16:15:41.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1224332007}) 2024-09-10T16:15:41.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3628283102}) 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 msgr2=0x7fcce80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 0x7fcce80406d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fccec009de0 tx=0x7fccec0098a0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 10 monclient: shutdown 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 msgr2=0x7fccfc0a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fccf0009720 tx=0x7fccf00046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 shutdown_connections 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcce803e210 0x7fcce80406d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccfc13d560 0x7fccfc0a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccfc134f70 0x7fccfc0a39b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 --2- 172.21.15.164:0/1055244949 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccfc06cb70 0x7fccfc1cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:41.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 >> 172.21.15.164:0/1055244949 conn(0x7fccfc0aa900 msgr2=0x7fccfc1335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:41.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.589+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 shutdown_connections 2024-09-10T16:15:41.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fcd02552640 1 -- 172.21.15.164:0/1055244949 wait complete. 2024-09-10T16:15:41.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421837751}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133933208}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421837751}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.590+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3697337070}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434427360}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=229609210}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434427360}) 2024-09-10T16:15:41.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2927627020}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817187887}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312615996}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1817187887}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.591+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2184999502}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820504788}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597377653}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820504788}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.592+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024115356}) 2024-09-10T16:15:41.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225510300}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=67879514}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3225510300}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1599185545}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288230110}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=606521484}) 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.601 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4288230110}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.593+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1915537258}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929821060}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3994570251}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1929821060}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.594+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=587887391}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394651678}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3873497124}) 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.602 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=394651678}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1816740289}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721550931}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779991031}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2721550931}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.595+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1039928430}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294179632}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2337901456}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2294179632}) 2024-09-10T16:15:41.603 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.596+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380302393}) 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39573104}) 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733801163}) 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39573104}) 2024-09-10T16:15:41.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.597+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1717667013}) 2024-09-10T16:15:41.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036810962}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1164767013}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3036810962}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=918164246}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405920692}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3756858526}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=405920692}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.598+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=644377755}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402185205}) 2024-09-10T16:15:41.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487650872}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1402185205}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.599+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3102097880}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066299571}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=857592084}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066299571}) 2024-09-10T16:15:41.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1998441026}) 2024-09-10T16:15:41.608 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:15:41.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379688885}) 2024-09-10T16:15:41.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689709015}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=379688885}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.600+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=206304999}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772407199}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903021171}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2772407199}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.601+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1065670189}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696778765}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4043206174}) 2024-09-10T16:15:41.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696778765}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4068891513}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426933527}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804196443}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.602+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2426933527}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3236979929}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770741311}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=951434908}) 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2770741311}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.603+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1350070595}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910738623}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3755293004}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=910738623}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.604+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1216145151}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13} crc {logm=3213955614,mdsmap=204415920,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2093056682}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=13} crc {logm=2052009527,mdsmap=3140514009,mgr=2907349437,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1559662368}) 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13} crc {logm=3213955614,mdsmap=204415920,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2093056682}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=1} crc {logm=376554441,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3941140310,mon_config_key=1867747704}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=63,osdmap=8} crc {mon_config_key=3774134534,monmap=1433040829,nvmeofgw=598337686,osd_metadata=1338542320,osd_pg_creating=1566267513,osd_snap=2355054587,osdmap=1348499572}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=56,osdmap=14} crc {mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1566267513,osd_snap=1073069359,osdmap=3547185110}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.605+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=63,osdmap=8} crc {mon_config_key=3774134534,monmap=1433040829,nvmeofgw=598337686,osd_metadata=1338542320,osd_pg_creating=1566267513,osd_snap=2355054587,osdmap=1348499572}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60,osdmap=13} crc {mon_config_key=4131094683,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1566267513,osd_snap=1752943944,osdmap=3234670695}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3050922346}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4087418157}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3050922346}) 2024-09-10T16:15:41.618 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.606+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2913073611}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.607+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.607+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1314542092}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.607+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1336413249}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.607+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.607+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1314542092}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.608+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1561710643}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.610+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.610+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=5} crc {osdmap=2211247912}) 2024-09-10T16:15:41.619 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:15:41.610+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=4} crc {osdmap=934455232}) 2024-09-10T16:15:42.171 INFO:tasks.rados.rados.0.smithi165.stdout:1301: done (2 left) 2024-09-10T16:15:42.171 INFO:tasks.rados.rados.0.smithi165.stdout:1302: done (1 left) 2024-09-10T16:15:42.171 INFO:tasks.rados.rados.0.smithi165.stdout:1303: done (0 left) 2024-09-10T16:15:42.171 INFO:tasks.rados.rados.0.smithi165.stdout:1304: copy_from oid 2 from oid 28 current snap is 149 2024-09-10T16:15:42.171 INFO:tasks.rados.rados.0.smithi165.stdout:1305: read oid 4 snap -1 2024-09-10T16:15:42.172 INFO:tasks.rados.rados.0.smithi165.stdout:1305: expect (ObjNum 454 snap 143 seq_num 454) 2024-09-10T16:15:42.172 INFO:tasks.rados.rados.0.smithi165.stdout:1306: snap_create 2024-09-10T16:15:42.187 INFO:tasks.rados.rados.0.smithi165.stdout:1305: expect (ObjNum 454 snap 143 seq_num 454) 2024-09-10T16:15:42.247 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 193604 2024-09-10T16:15:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:1304: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:15:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:1304: finishing copy_from to smithi16534841-2 2024-09-10T16:15:42.285 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 353 (ObjNum 457 snap 143 seq_num 457) dirty exists 2024-09-10T16:15:42.608 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:15:42.609 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.708+0000 7f7161a60640 10 monclient: get_monmap_and_config 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.708+0000 7f7161a60640 10 monclient: build_initial_monmap 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.708+0000 7f7161a60640 10 monclient: monmap: 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:42.709390+0000 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:42.709390+0000 2024-09-10T16:15:42.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:42.711 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 Processor -- start 2024-09-10T16:15:42.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 -- start start 2024-09-10T16:15:42.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient: init 2024-09-10T16:15:42.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f715c0a9870 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f715c134b90 0x7f715c13cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-c con 0x7f715c134b90 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-b con 0x7f715c13d510 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f715c135190 con 0x7f715c0a9870 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f715c0a9630 con 0x7f715c134b90 2024-09-10T16:15:42.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f715c0a9c50 con 0x7f715c13d510 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f7161a60640 10 monclient(hunting): authenticate will time out at 1321.196822s 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f715b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f715b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34166/0 (socket says 172.21.15.164:34166) 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.709+0000 7f715b7fe640 1 -- 172.21.15.164:0/2047227897 learned_addr learned my addr 172.21.15.164:0/2047227897 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): get_auth_request con 0x7f715c13d510 auth_method 0 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:42.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f7153fff640 1 --2- 172.21.15.164:0/2047227897 >> v2:172.21.15.165:3300/0 conn(0x7f715c134b90 0x7f715c13cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f7153fff640 10 monclient(hunting): get_auth_request con 0x7f715c134b90 auth_method 0 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f7153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f7153fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f7153fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient(hunting): handle_auth_done global_id 104114 payload 274 2024-09-10T16:15:42.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 1 -- 172.21.15.164:0/2047227897 >> v2:172.21.15.165:3300/0 conn(0x7f715c134b90 msgr2=0x7f715c13cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 1 --2- 172.21.15.164:0/2047227897 >> v2:172.21.15.165:3300/0 conn(0x7f715c134b90 0x7f715c13cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 1 -- 172.21.15.164:0/2047227897 >> v1:172.21.15.165:6789/0 conn(0x7f715c0a9870 legacy=0x7f715c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 1 monclient: found mon.noname-b 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.710+0000 7f715b7fe640 1 -- 172.21.15.164:0/2047227897 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c0a91b0 con 0x7f715c13d510 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715b7fe640 1 -- 172.21.15.164:0/2047227897 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 msgr2=0x7f715c13f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715b7fe640 1 --2- 172.21.15.164:0/2047227897 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7154009e50 tx=0x7f715402a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5da95093b72e6d74 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:42.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 1 -- 172.21.15.164:0/2047227897 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154036070 con 0x7f715c13d510 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: dump: 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:42.723 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 1 -- 172.21.15.164:0/2047227897 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7154007800 con 0x7f715c13d510 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 5 monclient: authenticate success, global_id 104114 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 4 set_mon_vals no callback set 2024-09-10T16:15:42.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 1 -- 172.21.15.164:0/2047227897 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7154007bd0 con 0x7f715c13d510 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f715a7fc640 10 monclient: dump: 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:42.725 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 10 monclient: shutdown 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 -- 172.21.15.164:0/2047227897 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 msgr2=0x7f715c13f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 --2- 172.21.15.164:0/2047227897 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7154009e50 tx=0x7f715402a560 comp rx=0 tx=0).stop 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 -- 172.21.15.164:0/2047227897 shutdown_connections 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 --2- 172.21.15.164:0/2047227897 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c13d510 0x7f715c13f900 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 --2- 172.21.15.164:0/2047227897 >> v2:172.21.15.165:3300/0 conn(0x7f715c134b90 0x7f715c13cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 -- 172.21.15.164:0/2047227897 >> 172.21.15.164:0/2047227897 conn(0x7f715c0aa890 msgr2=0x7f715c0aaca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 -- 172.21.15.164:0/2047227897 shutdown_connections 2024-09-10T16:15:42.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.713+0000 7f7161a60640 1 -- 172.21.15.164:0/2047227897 wait complete. 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.714+0000 7f7161a60640 10 monclient: build_initial_monmap 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.714+0000 7f7161a60640 10 monclient: monmap: 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:42.727 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.714+0000 7f7161a60640 1 Processor -- start 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.714+0000 7f7161a60640 1 -- start start 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.714+0000 7f7161a60640 10 monclient: init 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 0x7f715c0a7a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-a con 0x7f715c0a9870 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-c con 0x7f715c134b90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 0x7f715c1cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): picked mon.noname-b con 0x7f715c13d510 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f715c0a4df0 con 0x7f715c0a9870 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f715c09ce70 con 0x7f715c13d510 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f715c13fe40 con 0x7f715c134b90 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7161a60640 10 monclient(hunting): authenticate will time out at 1321.202253s 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 0x7f715c0a7a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34174/0 (socket says 172.21.15.164:34174) 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 -- 172.21.15.164:0/2069615844 learned_addr learned my addr 172.21.15.164:0/2069615844 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): get_auth_request con 0x7f715c134b90 auth_method 0 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): get_auth_request con 0x7f715c0a9870 auth_method 0 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:42.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715b7fe640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 0x7f715c1cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient(hunting): handle_auth_done global_id 104117 payload 824 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 -- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 msgr2=0x7f715c1cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 0x7f715c1cdb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 -- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 msgr2=0x7f715c0a7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 0x7f715c0a7a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 monclient: found mon.noname-c 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f7153fff640 1 -- 172.21.15.164:0/2069615844 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f715c0a04c0 con 0x7f715c134b90 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715b7fe640 10 monclient: get_auth_request con 0x7f715c13d510 auth_method 0 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715b7fe640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 0x7f715c1cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:15:42.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.715+0000 7f715affd640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 0x7f715c0a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:42.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7153fff640 1 -- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 msgr2=0x7f715c1cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:42.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7153fff640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f714400d800 tx=0x7f714400dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714400b8d0 con 0x7f715c134b90 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7158ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.717+0000 7f7158ff9640 10 monclient: dump: 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.735 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7144019070 con 0x7f715c134b90 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 5 monclient: authenticate success, global_id 104117 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714400bc90 con 0x7f715c134b90 2024-09-10T16:15:42.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _renew_subs 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f715c0a07a0 con 0x7f715c134b90 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 10 monclient: dump: 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:42.737 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7159ffb640 4 set_mon_vals no callback set 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _renew_subs 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f715c06e690 con 0x7f715c134b90 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f714401e460 con 0x7f715c134b90 2024-09-10T16:15:42.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.718+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f715c0a46e0 con 0x7f715c134b90 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.719+0000 7f7158ff9640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 0x7f713c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.719+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(324..324 src has 184..324) ==== 4815+0+0 (secure 0 0 0) 0x7f7144053240 con 0x7f715c134b90 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.719+0000 7f715affd640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 0x7f713c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.719+0000 7f715affd640 10 monclient: get_auth_request con 0x7f713c03e210 auth_method 0 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.722+0000 7f715affd640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 0x7f713c0406d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f714c006fd0 tx=0x7f714c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.723+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7144051c20 con 0x7f715c134b90 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.723+0000 7f7158ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:42.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.723+0000 7f7158ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:42.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.961+0000 7f7161a60640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:42.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7161a60640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7161a60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f715c0a8110 con 0x7f715c134b90 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7158ff9640 1 -- 172.21.15.164:0/2069615844 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7144014430 con 0x7f715c134b90 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7158ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.962+0000 7f7158ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:42.964 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 msgr2=0x7f713c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 0x7f713c0406d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f714c006fd0 tx=0x7f714c009060 comp rx=0 tx=0).stop 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 10 monclient: shutdown 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 msgr2=0x7f715c1cd640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f714400d800 tx=0x7f714400dcc0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 shutdown_connections 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f713c03e210 0x7f713c0406d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f715c13d510 0x7f715c1cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f715c134b90 0x7f715c1cd640 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 --2- 172.21.15.164:0/2069615844 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f715c0a9870 0x7f715c0a7a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 >> 172.21.15.164:0/2069615844 conn(0x7f715c0aa890 msgr2=0x7f715c133460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 shutdown_connections 2024-09-10T16:15:42.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:42.964+0000 7f7161a60640 1 -- 172.21.15.164:0/2069615844 wait complete. 2024-09-10T16:15:42.976 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:15:42.976 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:15:42.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:15:42.977 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.073+0000 7f80cd52b640 10 monclient: get_monmap_and_config 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.073+0000 7f80cd52b640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.073+0000 7f80cd52b640 10 monclient: monmap: 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:43.074619+0000 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:43.074619+0000 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.075 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:43.076 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 Processor -- start 2024-09-10T16:15:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 -- start start 2024-09-10T16:15:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient: init 2024-09-10T16:15:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80c80a4540 0x7f80c80a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-a con 0x7f80c80a4540 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f80c81385c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80c80a60b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80c80a71c0 con 0x7f80c81385c0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80c8135400 con 0x7f80c80a60b0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f80c809c9f0 con 0x7f80c80a4540 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80cd52b640 10 monclient(hunting): authenticate will time out at 1321.561841s 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80c6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80c80a4540 0x7f80c80a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80c6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80c80a4540 0x7f80c80a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47542/0 (socket says 172.21.15.164:47542) 2024-09-10T16:15:43.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.074+0000 7f80c67fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f80c81385c0 0x7f80c81345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54128/0 (socket says 172.21.15.164:54128) 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 1 -- 172.21.15.164:0/3287815567 learned_addr learned my addr 172.21.15.164:0/3287815567 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request con 0x7f80c80a4540 auth_method 0 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_done global_id 74421 payload 274 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 1 -- 172.21.15.164:0/3287815567 >> v1:172.21.15.165:6789/0 conn(0x7f80c80a60b0 legacy=0x7f80c8134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 1 -- 172.21.15.164:0/3287815567 >> v1:172.21.15.164:6789/0 conn(0x7f80c81385c0 legacy=0x7f80c81345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:15:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.075+0000 7f80c6ffd640 1 -- 172.21.15.164:0/3287815567 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c809ccd0 con 0x7f80c80a4540 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c6ffd640 1 -- 172.21.15.164:0/3287815567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 msgr2=0x7f80c80a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/3287815567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a4920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80b80098a0 tx=0x7f80b802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3b253949254e21f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 1 -- 172.21.15.164:0/3287815567 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8037070 con 0x7f80c80a4540 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: dump: 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.092 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 1 -- 172.21.15.164:0/3287815567 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80b8032640 con 0x7f80c80a4540 2024-09-10T16:15:43.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 5 monclient: authenticate success, global_id 74421 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 1 -- 172.21.15.164:0/3287815567 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8032a10 con 0x7f80c80a4540 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 4 set_mon_vals no callback set 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80c5ffb640 10 monclient: dump: 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 10 monclient: shutdown 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 -- 172.21.15.164:0/3287815567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 msgr2=0x7f80c80a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 --2- 172.21.15.164:0/3287815567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a4920 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f80b80098a0 tx=0x7f80b802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 -- 172.21.15.164:0/3287815567 shutdown_connections 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 --2- 172.21.15.164:0/3287815567 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a4920 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 -- 172.21.15.164:0/3287815567 >> 172.21.15.164:0/3287815567 conn(0x7f80c80aaa00 msgr2=0x7f80c80aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 -- 172.21.15.164:0/3287815567 shutdown_connections 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.078+0000 7f80cd52b640 1 -- 172.21.15.164:0/3287815567 wait complete. 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.079+0000 7f80cd52b640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.079+0000 7f80cd52b640 10 monclient: monmap: 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:43.096 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.079+0000 7f80cd52b640 1 Processor -- start 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 -- start start 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient: init 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-a con 0x7f80c80a4540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-c con 0x7f80c80a60b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80c81cd440 0x7f80c809e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): picked mon.noname-b con 0x7f80c81cd440 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f80c8137dd0 con 0x7f80c80a4540 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f80c80a4e60 con 0x7f80c81cd440 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f80c8137f50 con 0x7f80c80a60b0 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80cd52b640 10 monclient(hunting): authenticate will time out at 1321.567506s 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47546/0 (socket says 172.21.15.164:47546) 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34186/0 (socket says 172.21.15.164:34186) 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 1 -- 172.21.15.164:0/1093546590 learned_addr learned my addr 172.21.15.164:0/1093546590 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request con 0x7f80c80a4540 auth_method 0 2024-09-10T16:15:43.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): get_auth_request con 0x7f80c80a60b0 auth_method 0 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c77fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80c81cd440 0x7f80c809e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c77fe640 10 monclient(hunting): get_auth_request con 0x7f80c81cd440 auth_method 0 2024-09-10T16:15:43.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient(hunting): handle_auth_done global_id 104120 payload 824 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80c81cd440 msgr2=0x7f80c809e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80c81cd440 0x7f80c809e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 msgr2=0x7f80c80a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 monclient: found mon.noname-c 2024-09-10T16:15:43.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c67fc640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80c80a1fc0 con 0x7f80c80a60b0 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.080+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c80a4540 0x7f80c80a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80c67fc640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 msgr2=0x7f80c80a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80c67fc640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f80c81356a0 tx=0x7f80bc00c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc017070 con 0x7f80c80a60b0 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: dump: 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 5 monclient: authenticate success, global_id 104120 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80bc005810 con 0x7f80c80a60b0 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 10 monclient: _renew_subs 2024-09-10T16:15:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80c80a4250 con 0x7f80c80a60b0 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80bc005c00 con 0x7f80c80a60b0 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80af7fe640 10 monclient: dump: 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 10 monclient: _renew_subs 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f80c80aa550 con 0x7f80c80a60b0 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.083+0000 7f80c57fa640 4 set_mon_vals no callback set 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80cd52b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80cd52b640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 0x7f80c809ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f80c80a4920 con 0x7f80c81385c0 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f80bc01a070 con 0x7f80c80a60b0 2024-09-10T16:15:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 0x7f80c809ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient: get_auth_request con 0x7f80c81385c0 auth_method 0 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_done global_id 74427 payload 824 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80af7fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80b403e210 0x7f80b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c6ffd640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80c80a0620 con 0x7f80c81385c0 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(324..324 src has 184..324) ==== 4815+0+0 (secure 0 0 0) 0x7f80bc052080 con 0x7f80c80a60b0 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c77fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80b403e210 0x7f80b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.084+0000 7f80c77fe640 10 monclient: get_auth_request con 0x7f80b403e210 auth_method 0 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80c6ffd640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 msgr2=0x7f80c809ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 0x7f80c809ec20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f80b8035040 tx=0x7f80b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8037040 con 0x7f80c81385c0 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.086+0000 7f80af7fe640 10 monclient: dump: 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.107 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80b803ca30 con 0x7f80c81385c0 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 msgr2=0x7f80c809ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 0x7f80c809ec20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f80b8035040 tx=0x7f80b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 msgr2=0x7f80c809ec20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80af7fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c81385c0 0x7f80c809ec20 secure :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f80b8035040 tx=0x7f80b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.087+0000 7f80c77fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80b403e210 0x7f80b40406d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80c81cd8d0 tx=0x7f80c0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80cd52b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:43.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80cd52b640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 0x7f80c812f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80cd52b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f80c809f570 con 0x7f80c809f160 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 0x7f80c812f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient: get_auth_request con 0x7f80c809f160 auth_method 0 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 10 monclient(hunting): handle_auth_done global_id 74430 payload 824 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.093+0000 7f80c6ffd640 1 -- 172.21.15.164:0/1093546590 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80c80a0aa0 con 0x7f80c809f160 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80c6ffd640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 msgr2=0x7f80c812f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80c6ffd640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 0x7f80c812f340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80b802a110 tx=0x7f80b8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80b8037040 con 0x7f80c809f160 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80af7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.095+0000 7f80af7fe640 10 monclient: dump: 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f80b8003610 con 0x7f80c809f160 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 msgr2=0x7f80c812f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 0x7f80c812f340 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80b802a110 tx=0x7f80b8008030 comp rx=0 tx=0).stop 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 msgr2=0x7f80c812f340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80af7fe640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80c809f160 0x7f80c812f340 secure :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f80b802a110 tx=0x7f80b8008030 comp rx=0 tx=0).stop 2024-09-10T16:15:43.113 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 170962, 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:43.114 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.116 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.117 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.119 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:43.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:43.122 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.124 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.125 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.126 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.127 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:43.128 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80b403e210 msgr2=0x7f80b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f80b403e210 0x7f80b40406d0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f80c81cd8d0 tx=0x7f80c0009040 comp rx=0 tx=0).stop 2024-09-10T16:15:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 10 monclient: shutdown 2024-09-10T16:15:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 msgr2=0x7f80c80a3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80c77fe640 1 -- 172.21.15.164:0/1093546590 reap_dead start 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f80c81356a0 tx=0x7f80bc00c870 comp rx=0 tx=0).stop 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 shutdown_connections 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 --2- 172.21.15.164:0/1093546590 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80c80a60b0 0x7f80c80a3d10 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 >> 172.21.15.164:0/1093546590 conn(0x7f80c80aaa00 msgr2=0x7f80c80a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 shutdown_connections 2024-09-10T16:15:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.096+0000 7f80cd52b640 1 -- 172.21.15.164:0/1093546590 wait complete. 2024-09-10T16:15:43.132 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:43.132 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:15:43.132 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:15:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1304: done (2 left) 2024-09-10T16:15:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1305: done (1 left) 2024-09-10T16:15:43.189 INFO:tasks.rados.rados.0.smithi165.stdout:1306: done (0 left) 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1307: read oid 29 snap -1 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1307: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1308: read oid 22 snap -1 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1308: expect (ObjNum 469 snap 146 seq_num 469) 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1309: write (excl) oid 35 current snap is 150 2024-09-10T16:15:43.190 INFO:tasks.rados.rados.0.smithi165.stdout:1309: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-10T16:15:43.195 INFO:tasks.rados.rados.0.smithi165.stdout:1309: writing smithi16534841-35 from 716097 to 1255909 tid 1 2024-09-10T16:15:43.207 INFO:tasks.rados.rados.0.smithi165.stdout:1309: writing smithi16534841-35 from 1945607 to 2740773 tid 2 2024-09-10T16:15:43.214 INFO:tasks.rados.rados.0.smithi165.stdout:1309: writing smithi16534841-35 from 3535053 to 3916097 tid 3 2024-09-10T16:15:43.214 INFO:tasks.rados.rados.0.smithi165.stdout:1310: copy_from oid 44 from oid 37 current snap is 150 2024-09-10T16:15:43.214 INFO:tasks.rados.rados.0.smithi165.stdout:1307: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:15:43.237 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1939658 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.238+0000 7f10c5edd640 10 monclient: get_monmap_and_config 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.238+0000 7f10c5edd640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.238+0000 7f10c5edd640 10 monclient: monmap: 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:43.239006+0000 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:43.239006+0000 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:43.243 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.238+0000 7f10c5edd640 1 Processor -- start 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 1 -- start start 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient: init 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10c006cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10c0134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f10c013e740 0x7f10c0140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-c con 0x7f10c013e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10c009ce50 con 0x7f10c006cb70 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10c0135380 con 0x7f10c0134f70 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f10c006db60 con 0x7f10c013e740 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10c5edd640 10 monclient(hunting): authenticate will time out at 1321.726565s 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bf7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f10c006cb70 0x7f10c00a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54144/0 (socket says 172.21.15.164:54144) 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3166098861 learned_addr learned my addr 172.21.15.164:0/3166098861 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 1 --2- 172.21.15.164:0/3166098861 >> v2:172.21.15.165:3300/0 conn(0x7f10c013e740 0x7f10c0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): get_auth_request con 0x7f10c013e740 auth_method 0 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.239+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 10 monclient(hunting): handle_auth_done global_id 94210 payload 274 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 1 -- 172.21.15.164:0/3166098861 >> v1:172.21.15.165:6789/0 conn(0x7f10c0134f70 legacy=0x7f10c013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 1 -- 172.21.15.164:0/3166098861 >> v1:172.21.15.164:6789/0 conn(0x7f10c006cb70 legacy=0x7f10c00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 1 monclient: found mon.noname-c 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:43.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.240+0000 7f10bffff640 1 -- 172.21.15.164:0/3166098861 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c009cb70 con 0x7f10c013e740 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10bffff640 1 -- 172.21.15.164:0/3166098861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c013e740 msgr2=0x7f10c0140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10bffff640 1 --2- 172.21.15.164:0/3166098861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c013e740 0x7f10c0140b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009dc0 tx=0x7f10b802a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24c79421427e5d90 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 1 -- 172.21.15.164:0/3166098861 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8007a30 con 0x7f10c013e740 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: dump: 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 1 -- 172.21.15.164:0/3166098861 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b8007bd0 con 0x7f10c013e740 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 5 monclient: authenticate success, global_id 94210 2024-09-10T16:15:43.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 1 -- 172.21.15.164:0/3166098861 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b8031400 con 0x7f10c013e740 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 4 set_mon_vals no callback set 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10beffd640 10 monclient: dump: 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 10 monclient: shutdown 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 -- 172.21.15.164:0/3166098861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c013e740 msgr2=0x7f10c0140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3166098861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c013e740 0x7f10c0140b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f10b8009dc0 tx=0x7f10b802a4d0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 -- 172.21.15.164:0/3166098861 shutdown_connections 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3166098861 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c013e740 0x7f10c0140b30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 -- 172.21.15.164:0/3166098861 >> 172.21.15.164:0/3166098861 conn(0x7f10c00aa900 msgr2=0x7f10c00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 -- 172.21.15.164:0/3166098861 shutdown_connections 2024-09-10T16:15:43.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.243+0000 7f10c5edd640 1 -- 172.21.15.164:0/3166098861 wait complete. 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.244+0000 7f10c5edd640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.244+0000 7f10c5edd640 10 monclient: monmap: 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.244+0000 7f10c5edd640 1 Processor -- start 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 -- start start 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient: init 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 0x7f10c013b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-a con 0x7f10c006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c0134f70 0x7f10c013b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-b con 0x7f10c0134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c013e740 0x7f10c01cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): picked mon.noname-c con 0x7f10c013e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f10c00a4db0 con 0x7f10c006cb70 2024-09-10T16:15:43.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f10c006cf50 con 0x7f10c0134f70 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f10c0141070 con 0x7f10c013e740 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10c5edd640 10 monclient(hunting): authenticate will time out at 1321.732451s 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 0x7f10c013b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 0x7f10c013b010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47574/0 (socket says 172.21.15.164:47574) 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3030484903 learned_addr learned my addr 172.21.15.164:0/3030484903 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): get_auth_request con 0x7f10c006cb70 auth_method 0 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c013e740 0x7f10c01cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): get_auth_request con 0x7f10c013e740 auth_method 0 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c0134f70 0x7f10c013b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): get_auth_request con 0x7f10c0134f70 auth_method 0 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient(hunting): handle_auth_done global_id 74433 payload 824 2024-09-10T16:15:43.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c013e740 msgr2=0x7f10c01cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c013e740 0x7f10c01cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c0134f70 msgr2=0x7f10c013b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10c0134f70 0x7f10c013b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10c00a4080 con 0x7f10c006cb70 2024-09-10T16:15:43.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.245+0000 7f10bffff640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c013e740 0x7f10c01cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bf7fe640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 msgr2=0x7f10c013b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bf7fe640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 0x7f10c013b010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f10b000b430 tx=0x7f10b000b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0015020 con 0x7f10c006cb70 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: dump: 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.265 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 5 monclient: authenticate success, global_id 74433 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 10 monclient: _renew_subs 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10c013c860 con 0x7f10c006cb70 2024-09-10T16:15:43.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b0011a60 con 0x7f10c006cb70 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0011e50 con 0x7f10c006cb70 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bcff9640 10 monclient: dump: 2024-09-10T16:15:43.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.268 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.269 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.269 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 10 monclient: _renew_subs 2024-09-10T16:15:43.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10c009db60 con 0x7f10c006cb70 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.248+0000 7f10bdffb640 4 set_mon_vals no callback set 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10c5edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 0x7f10c002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f10c013d420 con 0x7f10c01cde00 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f10b0018020 con 0x7f10c006cb70 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 0x7f10c002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient: get_auth_request con 0x7f10c01cde00 auth_method 0 2024-09-10T16:15:43.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_done global_id 104126 payload 824 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10bcff9640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f10a003e3e0 0x7f10a00408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10b6ffd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10c00a46a0 con 0x7f10c01cde00 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7f10b0052c80 con 0x7f10c006cb70 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10bffff640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f10a003e3e0 0x7f10a00408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.249+0000 7f10bffff640 10 monclient: get_auth_request con 0x7f10a003e3e0 auth_method 0 2024-09-10T16:15:43.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10b6ffd640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 msgr2=0x7f10c002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10b6ffd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 0x7f10c002d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f10a800f7a0 tx=0x7f10a800fc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a8018030 con 0x7f10c01cde00 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.251+0000 7f10bcff9640 10 monclient: dump: 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f10a800d9b0 con 0x7f10c01cde00 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 msgr2=0x7f10c002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 0x7f10c002d440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f10a800f7a0 tx=0x7f10a800fc60 comp rx=0 tx=0).stop 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 msgr2=0x7f10c002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bcff9640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c01cde00 0x7f10c002d440 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.252+0000 7f10bffff640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f10a003e3e0 0x7f10a00408a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f10c006e110 tx=0x7f10b802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10c5edd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 0x7f10c002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10c5edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f10c0136f80 con 0x7f10c002d980 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 0x7f10c002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient: get_auth_request con 0x7f10c002d980 auth_method 0 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 10 monclient(hunting): handle_auth_done global_id 104129 payload 824 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.258+0000 7f10b6ffd640 1 -- 172.21.15.164:0/3030484903 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10c009df40 con 0x7f10c002d980 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10b6ffd640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 msgr2=0x7f10c002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10b6ffd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 0x7f10c002dd90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10a800f7a0 tx=0x7f10a8003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10a801dd70 con 0x7f10c002d980 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10bcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10bcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.260+0000 7f10bcff9640 10 monclient: dump: 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.277 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f10a801dd70 con 0x7f10c002d980 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 msgr2=0x7f10c002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 0x7f10c002dd90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10a800f7a0 tx=0x7f10a8003760 comp rx=0 tx=0).stop 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 msgr2=0x7f10c002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bcff9640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10c002d980 0x7f10c002dd90 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f10a800f7a0 tx=0x7f10a8003760 comp rx=0 tx=0).stop 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12577, 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:43.279 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.280 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.281 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.282 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.283 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:43.284 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:43.285 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.285 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:43.289 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:43.290 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.293 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.294 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:43.295 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:43.296 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2074661 2024-09-10T16:15:43.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f10a003e3e0 msgr2=0x7f10a00408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f10a003e3e0 0x7f10a00408a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f10c006e110 tx=0x7f10b802f000 comp rx=0 tx=0).stop 2024-09-10T16:15:43.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 10 monclient: shutdown 2024-09-10T16:15:43.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 msgr2=0x7f10c013b010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 --2- 172.21.15.164:0/3030484903 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10c006cb70 0x7f10c013b010 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f10b000b430 tx=0x7f10b000b8f0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bffff640 1 -- 172.21.15.164:0/3030484903 reap_dead start 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10bffff640 1 -- 172.21.15.164:0/3030484903 reap_dead start 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 shutdown_connections 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 >> 172.21.15.164:0/3030484903 conn(0x7f10c00aa900 msgr2=0x7f10c01339a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 shutdown_connections 2024-09-10T16:15:43.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.261+0000 7f10c5edd640 1 -- 172.21.15.164:0/3030484903 wait complete. 2024-09-10T16:15:43.298 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:43.298 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:15:43.299 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:15:43.354 INFO:tasks.rados.rados.0.smithi165.stdout:1308: expect (ObjNum 469 snap 146 seq_num 469) 2024-09-10T16:15:43.354 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:43.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 10 monclient: get_monmap_and_config 2024-09-10T16:15:43.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 10 monclient: monmap: 2024-09-10T16:15:43.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:43.397037+0000 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:43.397037+0000 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:43.398 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 1 Processor -- start 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 1 -- start start 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.396+0000 7f45c39b5640 10 monclient: init 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 0x7f45bc134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-b con 0x7f45bc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45bc0a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45bc0a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45bc09ce50 con 0x7f45bc0a46a0 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45bc138ac0 con 0x7f45bc0a4a80 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45bc06db60 con 0x7f45bc06cb70 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c39b5640 10 monclient(hunting): authenticate will time out at 1321.884487s 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c0f50640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f45bc0a46a0 0x7f45bc137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54154/0 (socket says 172.21.15.164:54154) 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c0f50640 1 -- 172.21.15.164:0/1012606012 learned_addr learned my addr 172.21.15.164:0/1012606012 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 1 --2- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 0x7f45bc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): get_auth_request con 0x7f45bc06cb70 auth_method 0 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient(hunting): handle_auth_done global_id 104132 payload 274 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 1 -- 172.21.15.164:0/1012606012 >> v1:172.21.15.165:6789/0 conn(0x7f45bc0a4a80 legacy=0x7f45bc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 1 -- 172.21.15.164:0/1012606012 >> v1:172.21.15.164:6789/0 conn(0x7f45bc0a46a0 legacy=0x7f45bc137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 1 monclient: found mon.noname-b 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:15:43.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.397+0000 7f45c1751640 1 -- 172.21.15.164:0/1012606012 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45bc09cb70 con 0x7f45bc06cb70 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c1751640 1 -- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 msgr2=0x7f45bc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c1751640 1 --2- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 0x7f45bc134f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45b80097d0 tx=0x7f45b8029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37527461af648660 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 1 -- 172.21.15.164:0/1012606012 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8037070 con 0x7f45bc06cb70 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: dump: 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.406 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 1 -- 172.21.15.164:0/1012606012 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b8032440 con 0x7f45bc06cb70 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c39b5640 5 monclient: authenticate success, global_id 104132 2024-09-10T16:15:43.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c39b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c39b5640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 1 -- 172.21.15.164:0/1012606012 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8032810 con 0x7f45bc06cb70 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45c39b5640 4 set_mon_vals no callback set 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.400+0000 7f45b3fff640 10 monclient: dump: 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.408 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 10 monclient: shutdown 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 -- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 msgr2=0x7f45bc134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 --2- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 0x7f45bc134f70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f45b80097d0 tx=0x7f45b8029fb0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 -- 172.21.15.164:0/1012606012 shutdown_connections 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 --2- 172.21.15.164:0/1012606012 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc06cb70 0x7f45bc134f70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 -- 172.21.15.164:0/1012606012 >> 172.21.15.164:0/1012606012 conn(0x7f45bc0aa900 msgr2=0x7f45bc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 -- 172.21.15.164:0/1012606012 shutdown_connections 2024-09-10T16:15:43.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.401+0000 7f45c39b5640 1 -- 172.21.15.164:0/1012606012 wait complete. 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient: monmap: 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 Processor -- start 2024-09-10T16:15:43.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 -- start start 2024-09-10T16:15:43.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient: init 2024-09-10T16:15:43.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 0x7f45bc1cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-a con 0x7f45bc06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc0a46a0 0x7f45bc140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-c con 0x7f45bc0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a4a80 0x7f45bc140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): picked mon.noname-b con 0x7f45bc0a4a80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc06cf50 con 0x7f45bc06cb70 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc141050 con 0x7f45bc0a4a80 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45bc1411d0 con 0x7f45bc0a46a0 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c39b5640 10 monclient(hunting): authenticate will time out at 1321.889863s 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c0f50640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc0a46a0 0x7f45bc140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 0x7f45bc1cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 0x7f45bc1cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47582/0 (socket says 172.21.15.164:47582) 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 1 -- 172.21.15.164:0/3049976004 learned_addr learned my addr 172.21.15.164:0/3049976004 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 10 monclient(hunting): get_auth_request con 0x7f45bc06cb70 auth_method 0 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c1751640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c0f50640 10 monclient(hunting): get_auth_request con 0x7f45bc0a46a0 auth_method 0 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c0f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c0f50640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.402+0000 7f45c0f50640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a4a80 0x7f45bc140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): get_auth_request con 0x7f45bc0a4a80 auth_method 0 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1f52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient(hunting): handle_auth_done global_id 74436 payload 824 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc0a46a0 msgr2=0x7f45bc140030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc0a46a0 0x7f45bc140030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a4a80 msgr2=0x7f45bc140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a4a80 0x7f45bc140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 monclient: found mon.noname-a 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c1751640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45bc13a220 con 0x7f45bc06cb70 2024-09-10T16:15:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.403+0000 7f45c0f50640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45bc0a46a0 0x7f45bc140030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:15:43.417 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:15:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c1751640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 msgr2=0x7f45bc1cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c1751640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 0x7f45bc1cd590 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f45b80097d0 tx=0x7f45b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8041070 con 0x7f45bc06cb70 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: dump: 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b803c660 con 0x7f45bc06cb70 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 5 monclient: authenticate success, global_id 74436 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45b8037040 con 0x7f45bc06cb70 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 10 monclient: _renew_subs 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45bc0a8a20 con 0x7f45bc06cb70 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b37fe640 4 set_mon_vals no callback set 2024-09-10T16:15:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45b1ffb640 10 monclient: dump: 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 10 monclient: _renew_subs 2024-09-10T16:15:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.405+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45bc0a4050 con 0x7f45bc06cb70 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c39b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c39b5640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 0x7f45bc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc0a4f10 con 0x7f45bc0a2f00 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f45b8044070 con 0x7f45bc06cb70 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 0x7f45bc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient: get_auth_request con 0x7f45bc0a2f00 auth_method 0 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45b1ffb640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f45a003e210 0x7f45a00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.406+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7f45b80733a0 con 0x7f45bc06cb70 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.407+0000 7f45c0f50640 10 monclient(hunting): handle_auth_done global_id 94219 payload 824 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.407+0000 7f45c0f50640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45bc134f70 con 0x7f45bc0a2f00 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.407+0000 7f45c1f52640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f45a003e210 0x7f45a00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.407+0000 7f45c1f52640 10 monclient: get_auth_request con 0x7f45a003e210 auth_method 0 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45c0f50640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 msgr2=0x7f45bc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45c0f50640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 0x7f45bc02d440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac017020 con 0x7f45bc0a2f00 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 10 monclient: dump: 2024-09-10T16:15:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45c1f52640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f45a003e210 0x7f45a00406d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f45bc1ce3e0 tx=0x7f45b4003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f45ac00fc10 con 0x7f45bc0a2f00 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 msgr2=0x7f45bc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 0x7f45bc02d440 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac007700 comp rx=0 tx=0).stop 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 msgr2=0x7f45bc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.409+0000 7f45b1ffb640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a2f00 0x7f45bc02d440 secure :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac007700 comp rx=0 tx=0).stop 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.415+0000 7f45c39b5640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.415+0000 7f45c39b5640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 0x7f45bc02d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.415+0000 7f45c39b5640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.415+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f45bc1392c0 con 0x7f45bc0a32e0 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.415+0000 7f45c0f50640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 0x7f45bc02d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient: get_auth_request con 0x7f45bc0a32e0 auth_method 0 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 10 monclient(hunting): handle_auth_done global_id 94222 payload 824 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.416+0000 7f45c0f50640 1 -- 172.21.15.164:0/3049976004 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45bc09ed00 con 0x7f45bc0a32e0 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45c0f50640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 msgr2=0x7f45bc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45c0f50640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 0x7f45bc02d980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45ac01e2e0 con 0x7f45bc0a32e0 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 10 monclient: dump: 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7f45ac013880 con 0x7f45bc0a32e0 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 msgr2=0x7f45bc02d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 0x7f45bc02d980 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac004940 comp rx=0 tx=0).stop 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 msgr2=0x7f45bc02d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.418+0000 7f45b1ffb640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45bc0a32e0 0x7f45bc02d980 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f45bc139540 tx=0x7f45ac004940 comp rx=0 tx=0).stop 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:43.431 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 86, 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 60918, 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.432 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.433 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:43.435 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:43.436 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:43.436 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:43.436 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:43.436 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:43.437 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.439 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:43.441 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.442 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:43.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:43.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:43.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:43.446 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f45a003e210 msgr2=0x7f45a00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f45a003e210 0x7f45a00406d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f45bc1ce3e0 tx=0x7f45b4003d10 comp rx=0 tx=0).stop 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 10 monclient: shutdown 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 msgr2=0x7f45bc1cd590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 --2- 172.21.15.164:0/3049976004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45bc06cb70 0x7f45bc1cd590 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f45b80097d0 tx=0x7f45b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c1f52640 1 -- 172.21.15.164:0/3049976004 reap_dead start 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c1f52640 1 -- 172.21.15.164:0/3049976004 reap_dead start 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 shutdown_connections 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 >> 172.21.15.164:0/3049976004 conn(0x7f45bc0aa900 msgr2=0x7f45bc137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 shutdown_connections 2024-09-10T16:15:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.419+0000 7f45c39b5640 1 -- 172.21.15.164:0/3049976004 wait complete. 2024-09-10T16:15:43.449 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:15:43.449 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:15:43.449 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:15:43.449 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:15:43.449 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:15:43.449 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:15:43.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:15:43.461 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:15:43.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:15:43.463 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: finishing write tid 1 to smithi16534841-35 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: finishing write tid 2 to smithi16534841-35 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: finishing write tid 3 to smithi16534841-35 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: finishing write tid 4 to smithi16534841-35 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: finishing write tid 5 to smithi16534841-35 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: oid 35 updating version 0 to 733 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: oid 35 updating version 733 to 734 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: oid 35 updating version 734 to 736 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: oid 35 version 736 is already newer than 735 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 736 (ObjNum 471 snap 150 seq_num 471) dirty exists 2024-09-10T16:15:43.533 INFO:tasks.rados.rados.0.smithi165.stdout:1309: left oid 35 (ObjNum 471 snap 150 seq_num 471) 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1310: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1310: finishing copy_from to smithi16534841-44 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 568 (ObjNum 468 snap 146 seq_num 468) dirty exists 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1307: done (3 left) 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1308: done (2 left) 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1309: done (1 left) 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1310: done (0 left) 2024-09-10T16:15:43.534 INFO:tasks.rados.rados.0.smithi165.stdout:1311: snap_create 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.597+0000 7f8cb4840640 10 monclient: get_monmap_and_config 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.597+0000 7f8cb4840640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.597+0000 7f8cb4840640 10 monclient: monmap: 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:43.598527+0000 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:43.598527+0000 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:43.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:43.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:43.600 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:43.600 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:43.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 Processor -- start 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 -- start start 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient: init 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cb00a3b60 0x7f8cb0137c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb00a3b60 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 0x7f8cb00a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb00a0530 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 0x7f8cb00a5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb00a5860 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f8cb00a3f40 con 0x7f8cb00a3b60 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8cb009cfe0 con 0x7f8cb00a0530 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb0074520 con 0x7f8cb00a5860 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8cb4840640 10 monclient(hunting): authenticate will time out at 1322.085914s 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8caed9d640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 msgr2=0x7f8cb00a5cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.598+0000 7f8caed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 0x7f8cb00a5cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cb00a3b60 0x7f8cb0137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8cb00a3b60 0x7f8cb0137c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47598/0 (socket says 172.21.15.164:47598) 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 -- 172.21.15.164:0/1430252407 learned_addr learned my addr 172.21.15.164:0/1430252407 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): get_auth_request con 0x7f8cb00a3b60 auth_method 0 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 1 --2- 172.21.15.164:0/1430252407 >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 0x7f8cb00a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 10 monclient(hunting): get_auth_request con 0x7f8cb00a0530 auth_method 0 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient(hunting): handle_auth_done global_id 74439 payload 274 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 -- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 msgr2=0x7f8cb00a5cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 --2- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 0x7f8cb00a5cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 -- 172.21.15.164:0/1430252407 >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 msgr2=0x7f8cb00a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 --2- 172.21.15.164:0/1430252407 >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 0x7f8cb00a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 monclient: found mon.noname-a 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8cae59c640 1 -- 172.21.15.164:0/1430252407 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb0073930 con 0x7f8cb00a3b60 2024-09-10T16:15:43.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.599+0000 7f8ca7fff640 1 --2- 172.21.15.164:0/1430252407 >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 0x7f8cb00a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:15:43.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cae59c640 1 -- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a3b60 msgr2=0x7f8cb0137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cae59c640 1 --2- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a3b60 0x7f8cb0137c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0074450 tx=0x7f8ca002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7e78569424c409 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 1 -- 172.21.15.164:0/1430252407 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0037070 con 0x7f8cb00a3b60 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: dump: 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.612 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 1 -- 172.21.15.164:0/1430252407 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca0032640 con 0x7f8cb00a3b60 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 5 monclient: authenticate success, global_id 74439 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:43.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 1 -- 172.21.15.164:0/1430252407 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca0032a10 con 0x7f8cb00a3b60 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 4 set_mon_vals no callback set 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cadd9b640 10 monclient: dump: 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.614 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 10 monclient: shutdown 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 -- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a3b60 msgr2=0x7f8cb0137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 --2- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a3b60 0x7f8cb0137c80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8cb0074450 tx=0x7f8ca002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 -- 172.21.15.164:0/1430252407 shutdown_connections 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 --2- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a5860 0x7f8cb00a5cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 --2- 172.21.15.164:0/1430252407 >> v2:172.21.15.165:3300/0 conn(0x7f8cb00a0530 0x7f8cb00a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 --2- 172.21.15.164:0/1430252407 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a3b60 0x7f8cb0137c80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 -- 172.21.15.164:0/1430252407 >> 172.21.15.164:0/1430252407 conn(0x7f8cb00a6f80 msgr2=0x7f8cb00a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 -- 172.21.15.164:0/1430252407 shutdown_connections 2024-09-10T16:15:43.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.602+0000 7f8cb4840640 1 -- 172.21.15.164:0/1430252407 wait complete. 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.603+0000 7f8cb4840640 10 monclient: build_initial_monmap 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.603+0000 7f8cb4840640 10 monclient: monmap: 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:43.616 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.603+0000 7f8cb4840640 1 Processor -- start 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.603+0000 7f8cb4840640 1 -- start start 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.603+0000 7f8cb4840640 10 monclient: init 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb00a0530 0x7f8cb01310c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-b con 0x7f8cb00a0530 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 0x7f8cb0133470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-c con 0x7f8cb00a3b60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): picked mon.noname-a con 0x7f8cb00a5860 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:43.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb00a6230 con 0x7f8cb00a5860 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8cb00a63b0 con 0x7f8cb00a0530 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8cb00a6530 con 0x7f8cb00a3b60 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cb4840640 10 monclient(hunting): authenticate will time out at 1322.091432s 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8ca7fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 msgr2=0x7f8cb0133470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8ca7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 0x7f8cb0133470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47612/0 (socket says 172.21.15.164:47612) 2024-09-10T16:15:43.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 -- 172.21.15.164:0/2792257363 learned_addr learned my addr 172.21.15.164:0/2792257363 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): get_auth_request con 0x7f8cb00a5860 auth_method 0 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8cae59c640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb00a0530 0x7f8cb01310c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient(hunting): handle_auth_done global_id 74442 payload 824 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 -- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 msgr2=0x7f8cb0133470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 0x7f8cb0133470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 -- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb00a0530 msgr2=0x7f8cb01310c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb00a0530 0x7f8cb01310c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:43.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.604+0000 7f8caed9d640 1 -- 172.21.15.164:0/2792257363 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8cb011eae0 con 0x7f8cb00a5860 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8caed9d640 1 -- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 msgr2=0x7f8cb01339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8caed9d640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800be60 tx=0x7f8ca800bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8caed9d640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800be60 tx=0x7f8ca800bf50 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f8ca8016630 mon_map magic: 0 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8016630 con 0x7f8cb00a5860 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: dump: 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ca8010030 con 0x7f8cb00a5860 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 5 monclient: authenticate success, global_id 74442 2024-09-10T16:15:43.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 10 monclient: _renew_subs 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8cb00a1bf0 con 0x7f8cb00a5860 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8ca6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 10 monclient: _renew_subs 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cad59a640 4 set_mon_vals no callback set 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.607+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8cb0134f30 con 0x7f8cb00a5860 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ca8016a10 con 0x7f8cb00a5860 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 10 monclient: dump: 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:43.622 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8ca80153e0 con 0x7f8cb00a5860 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cb4840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cb4840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cb4840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:43.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8cb0137c80 con 0x7f8cb00a5860 2024-09-10T16:15:43.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8ca6ffd640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 0x7f8c90040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:43.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cae59c640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 0x7f8c90040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:43.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.608+0000 7f8cae59c640 10 monclient: get_auth_request con 0x7f8c9003e1c0 auth_method 0 2024-09-10T16:15:43.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.611+0000 7f8cae59c640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 0x7f8c90040680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00a7970 tx=0x7f8ca002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:43.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:43.612+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7f8ca8053850 con 0x7f8cb00a5860 2024-09-10T16:15:43.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-10T16:15:44.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.231+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8ca8052270 con 0x7f8cb00a5860 2024-09-10T16:15:44.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.231+0000 7f8ca6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:44.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.231+0000 7f8ca6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:44.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8cb4840640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8cb4840640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8cb4840640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8cb011e8c0 con 0x7f8cb00a5860 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8ca6ffd640 1 -- 172.21.15.164:0/2792257363 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f8ca8013cf0 con 0x7f8cb00a5860 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8ca6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.472+0000 7f8ca6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:44.474 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:44.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 msgr2=0x7f8c90040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:44.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 0x7f8c90040680 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8cb00a7970 tx=0x7f8ca002e000 comp rx=0 tx=0).stop 2024-09-10T16:15:44.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 10 monclient: shutdown 2024-09-10T16:15:44.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 msgr2=0x7f8cb01339b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f8ca800be60 tx=0x7f8ca800bf50 comp rx=0 tx=0).stop 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 shutdown_connections 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8c9003e1c0 0x7f8c90040680 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8cb00a5860 0x7f8cb01339b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8cb00a3b60 0x7f8cb0133470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 --2- 172.21.15.164:0/2792257363 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8cb00a0530 0x7f8cb01310c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 >> 172.21.15.164:0/2792257363 conn(0x7f8cb00a6f80 msgr2=0x7f8cb01bfca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 shutdown_connections 2024-09-10T16:15:44.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:44.474+0000 7f8cb4840640 1 -- 172.21.15.164:0/2792257363 wait complete. 2024-09-10T16:15:47.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:47.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 10 monclient: get_monmap_and_config 2024-09-10T16:15:47.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 10 monclient: build_initial_monmap 2024-09-10T16:15:47.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 10 monclient: monmap: 2024-09-10T16:15:47.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:47.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:47.596944+0000 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:47.596944+0000 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:47.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 1 Processor -- start 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 1 -- start start 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.596+0000 7fe869f5d640 10 monclient: init 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 0x7fe864134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe86406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe8640a46a0 0x7fe8640a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8640a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe864137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe86409ce50 con 0x7fe864137ca0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe86406db60 con 0x7fe8640a46a0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe86406cf50 con 0x7fe86406cb70 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe869f5d640 10 monclient(hunting): authenticate will time out at 1326.084501s 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe8637fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 msgr2=0x7fe864134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 0x7fe864134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe8640a46a0 0x7fe8640a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe863fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe864137ca0 0x7fe864138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42488/0 (socket says 172.21.15.164:42488) 2024-09-10T16:15:47.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe8640a46a0 0x7fe8640a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59564/0 (socket says 172.21.15.164:59564) 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe863fff640 1 -- 172.21.15.164:0/1139480314 learned_addr learned my addr 172.21.15.164:0/1139480314 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): get_auth_request con 0x7fe8640a46a0 auth_method 0 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient(hunting): handle_auth_done global_id 74445 payload 274 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 -- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 msgr2=0x7fe864134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 --2- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 0x7fe864134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 -- 172.21.15.164:0/1139480314 >> v1:172.21.15.164:6789/0 conn(0x7fe864137ca0 legacy=0x7fe864138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:47.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:47.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 monclient: found mon.noname-a 2024-09-10T16:15:47.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:47.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.597+0000 7fe862ffd640 1 -- 172.21.15.164:0/1139480314 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86409cb70 con 0x7fe8640a46a0 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.600+0000 7fe862ffd640 1 -- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 msgr2=0x7fe8640a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.600+0000 7fe862ffd640 1 --2- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8580097a0 tx=0x7fe858029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30f706538fa54429 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.600+0000 7fe8627fc640 1 -- 172.21.15.164:0/1139480314 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858037070 con 0x7fe8640a46a0 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.600+0000 7fe8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: dump: 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:47.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 1 -- 172.21.15.164:0/1139480314 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858032640 con 0x7fe8640a46a0 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 5 monclient: authenticate success, global_id 74445 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 1 -- 172.21.15.164:0/1139480314 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858032a10 con 0x7fe8640a46a0 2024-09-10T16:15:47.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 4 set_mon_vals no callback set 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe8627fc640 10 monclient: dump: 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:47.612 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 10 monclient: shutdown 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 -- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 msgr2=0x7fe8640a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 --2- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a4a80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fe8580097a0 tx=0x7fe858029fe0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 -- 172.21.15.164:0/1139480314 shutdown_connections 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 --2- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a4a80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 --2- 172.21.15.164:0/1139480314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe86406cb70 0x7fe864134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 -- 172.21.15.164:0/1139480314 >> 172.21.15.164:0/1139480314 conn(0x7fe8640aa900 msgr2=0x7fe8640aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 -- 172.21.15.164:0/1139480314 shutdown_connections 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.601+0000 7fe869f5d640 1 -- 172.21.15.164:0/1139480314 wait complete. 2024-09-10T16:15:47.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.602+0000 7fe869f5d640 10 monclient: build_initial_monmap 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.602+0000 7fe869f5d640 10 monclient: monmap: 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.602+0000 7fe869f5d640 1 Processor -- start 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.602+0000 7fe869f5d640 1 -- start start 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.602+0000 7fe869f5d640 10 monclient: init 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe86406cb70 0x7fe864131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fe86406cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8640a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 0x7fe8641c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fe864137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe8641389c0 con 0x7fe8640a46a0 2024-09-10T16:15:47.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe864138840 con 0x7fe86406cb70 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe864140040 con 0x7fe864137ca0 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe869f5d640 10 monclient(hunting): authenticate will time out at 1326.090275s 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe863fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 msgr2=0x7fe8641c9f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe863fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 0x7fe8641c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59572/0 (socket says 172.21.15.164:59572) 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 -- 172.21.15.164:0/3281465968 learned_addr learned my addr 172.21.15.164:0/3281465968 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:47.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): get_auth_request con 0x7fe8640a46a0 auth_method 0 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe8637fe640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe86406cb70 0x7fe864131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe8637fe640 10 monclient(hunting): get_auth_request con 0x7fe86406cb70 auth_method 0 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe8637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe8637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient(hunting): handle_auth_done global_id 74448 payload 824 2024-09-10T16:15:47.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 -- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 msgr2=0x7fe8641c9f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 0x7fe8641c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 -- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe86406cb70 msgr2=0x7fe864131780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe86406cb70 0x7fe864131780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 monclient: found mon.noname-a 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.603+0000 7fe862ffd640 1 -- 172.21.15.164:0/3281465968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86406e720 con 0x7fe8640a46a0 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.605+0000 7fe862ffd640 1 -- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 msgr2=0x7fe8640a3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.605+0000 7fe862ffd640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe8580098c0 tx=0x7fe8580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858042020 con 0x7fe8640a46a0 2024-09-10T16:15:47.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: dump: 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: _finish_auth 0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe858030880 con 0x7fe8640a46a0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 5 monclient: authenticate success, global_id 74448 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 10 monclient: _renew_subs 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86409ff40 con 0x7fe8640a46a0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe858037040 con 0x7fe8640a46a0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:47.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe84bfff640 10 monclient: dump: 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe861ffb640 4 set_mon_vals no callback set 2024-09-10T16:15:47.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 10 monclient: _renew_subs 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.606+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86409f200 con 0x7fe8640a46a0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe869f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe869f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe869f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe858045020 con 0x7fe8640a46a0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe864134f70 con 0x7fe8640a46a0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe84bfff640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 0x7fe850040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7fe858073690 con 0x7fe8640a46a0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe8637fe640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 0x7fe850040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:47.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.607+0000 7fe8637fe640 10 monclient: get_auth_request con 0x7fe85003e1c0 auth_method 0 2024-09-10T16:15:47.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.610+0000 7fe8637fe640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 0x7fe850040680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe854003d00 tx=0x7fe8540094d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:47.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.611+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe85804c030 con 0x7fe8640a46a0 2024-09-10T16:15:47.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.611+0000 7fe84bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:47.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.611+0000 7fe84bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:47.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.851+0000 7fe869f5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:47.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.851+0000 7fe869f5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:47.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.851+0000 7fe869f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:47.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.851+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe8640770f0 con 0x7fe8640a46a0 2024-09-10T16:15:47.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.852+0000 7fe84bfff640 1 -- 172.21.15.164:0/3281465968 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fe85802ae20 con 0x7fe8640a46a0 2024-09-10T16:15:47.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.852+0000 7fe84bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:47.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.852+0000 7fe84bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:47.854 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:47.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 msgr2=0x7fe850040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:47.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 0x7fe850040680 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fe854003d00 tx=0x7fe8540094d0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 10 monclient: shutdown 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 msgr2=0x7fe8640a3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fe8580098c0 tx=0x7fe8580046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 shutdown_connections 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe85003e1c0 0x7fe850040680 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe864137ca0 0x7fe8641c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe8640a46a0 0x7fe8640a3d30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 --2- 172.21.15.164:0/3281465968 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe86406cb70 0x7fe864131780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 >> 172.21.15.164:0/3281465968 conn(0x7fe8640aa900 msgr2=0x7fe8640a9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 shutdown_connections 2024-09-10T16:15:47.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:47.854+0000 7fe869f5d640 1 -- 172.21.15.164:0/3281465968 wait complete. 2024-09-10T16:15:49.570 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-10T16:15:50.867 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.974+0000 7fbbafac6640 10 monclient: get_monmap_and_config 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.974+0000 7fbbafac6640 10 monclient: build_initial_monmap 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.974+0000 7fbbafac6640 10 monclient: monmap: 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:50.975743+0000 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:50.975743+0000 2024-09-10T16:15:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:50.977 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:50.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.975+0000 7fbbafac6640 1 Processor -- start 2024-09-10T16:15:50.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.975+0000 7fbbafac6640 1 -- start start 2024-09-10T16:15:50.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.975+0000 7fbbafac6640 10 monclient: init 2024-09-10T16:15:50.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbba80a0410 0x7fbba80a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-c con 0x7fbba80a0410 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbba80a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbba80a2810 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba80a0210 con 0x7fbba80a2380 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbba809d930 con 0x7fbba80a2810 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbba809d3f0 con 0x7fbba80a0410 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbafac6640 10 monclient(hunting): authenticate will time out at 1329.463223s 2024-09-10T16:15:50.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad061640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbba80a2380 0x7fbba813a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42502/0 (socket says 172.21.15.164:42502) 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad061640 1 -- 172.21.15.164:0/110180233 learned_addr learned my addr 172.21.15.164:0/110180233 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 1 --2- 172.21.15.164:0/110180233 >> v2:172.21.15.165:3300/0 conn(0x7fbba80a0410 0x7fbba80a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): get_auth_request con 0x7fbba80a0410 auth_method 0 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient(hunting): handle_auth_done global_id 94231 payload 274 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 1 -- 172.21.15.164:0/110180233 >> v1:172.21.15.165:6789/0 conn(0x7fbba80a2810 legacy=0x7fbba813dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:50.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 1 -- 172.21.15.164:0/110180233 >> v1:172.21.15.164:6789/0 conn(0x7fbba80a2380 legacy=0x7fbba813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:50.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient: _finish_hunting 0 2024-09-10T16:15:50.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 1 monclient: found mon.noname-c 2024-09-10T16:15:50.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:50.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:15:50.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.976+0000 7fbbad862640 1 -- 172.21.15.164:0/110180233 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba809e7d0 con 0x7fbba80a0410 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbad862640 1 -- 172.21.15.164:0/110180233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a0410 msgr2=0x7fbba80a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbad862640 1 --2- 172.21.15.164:0/110180233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a0410 0x7fbba80a1e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbba40097a0 tx=0x7fbba4029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ab1b7aec85f19a0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 1 -- 172.21.15.164:0/110180233 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4037070 con 0x7fbba80a0410 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: dump: 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: _finish_auth 0 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: _check_auth_tickets 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 1 -- 172.21.15.164:0/110180233 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba4032640 con 0x7fbba80a0410 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:50.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 1 -- 172.21.15.164:0/110180233 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba4032a10 con 0x7fbba80a0410 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbac860640 10 monclient: dump: 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:50.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 5 monclient: authenticate success, global_id 94231 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 4 set_mon_vals no callback set 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 10 monclient: shutdown 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 1 -- 172.21.15.164:0/110180233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a0410 msgr2=0x7fbba80a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.980+0000 7fbbafac6640 1 --2- 172.21.15.164:0/110180233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a0410 0x7fbba80a1e40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fbba40097a0 tx=0x7fbba4029fd0 comp rx=0 tx=0).stop 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 -- 172.21.15.164:0/110180233 shutdown_connections 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 --2- 172.21.15.164:0/110180233 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a0410 0x7fbba80a1e40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 -- 172.21.15.164:0/110180233 >> 172.21.15.164:0/110180233 conn(0x7fbba80a3110 msgr2=0x7fbba80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 -- 172.21.15.164:0/110180233 shutdown_connections 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 -- 172.21.15.164:0/110180233 wait complete. 2024-09-10T16:15:50.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 10 monclient: build_initial_monmap 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 10 monclient: monmap: 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:50.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.981+0000 7fbbafac6640 1 Processor -- start 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 -- start start 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient: init 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 0x7fbba81caf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-c con 0x7fbba80a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a2380 0x7fbba81ba6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-b con 0x7fbba80a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): picked mon.noname-a con 0x7fbba80a2810 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbba809c910 con 0x7fbba80a2810 2024-09-10T16:15:50.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbba80a2ca0 con 0x7fbba80a2380 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbba80a07f0 con 0x7fbba80a0410 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad862640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 msgr2=0x7fbba81caf90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbafac6640 10 monclient(hunting): authenticate will time out at 1329.469515s 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad862640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 0x7fbba81caf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59580/0 (socket says 172.21.15.164:59580) 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 -- 172.21.15.164:0/1454839052 learned_addr learned my addr 172.21.15.164:0/1454839052 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): get_auth_request con 0x7fbba80a2810 auth_method 0 2024-09-10T16:15:50.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a2380 0x7fbba81ba6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): get_auth_request con 0x7fbba80a2380 auth_method 0 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:50.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbad061640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient(hunting): handle_auth_done global_id 74451 payload 824 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 -- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 msgr2=0x7fbba81caf90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 0x7fbba81caf90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 -- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a2380 msgr2=0x7fbba81ba6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a2380 0x7fbba81ba6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient: _finish_hunting 0 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 monclient: found mon.noname-a 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.982+0000 7fbbae063640 1 -- 172.21.15.164:0/1454839052 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbba809d620 con 0x7fbba80a2810 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbae063640 1 -- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 msgr2=0x7fbba81bac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:50.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbae063640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c780 tx=0x7fbba000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba000ac00 con 0x7fbba80a2810 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: dump: 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:50.997 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbba000ada0 con 0x7fbba80a2810 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 5 monclient: authenticate success, global_id 74451 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 10 monclient: _renew_subs 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:50.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbba81b3e70 con 0x7fbba80a2810 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbba0004ba0 con 0x7fbba80a2810 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9e7fc640 10 monclient: dump: 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:50.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbb9ffff640 4 set_mon_vals no callback set 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 10 monclient: _renew_subs 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.985+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbba81b3b90 con 0x7fbba80a2810 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbbafac6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbbafac6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbbafac6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbba0005030 con 0x7fbba80a2810 2024-09-10T16:15:51.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbba809faa0 con 0x7fbba80a2810 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbb9e7fc640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 0x7fbb8c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7fbba0053200 con 0x7fbba80a2810 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.986+0000 7fbbad862640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 0x7fbb8c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.987+0000 7fbbad862640 10 monclient: get_auth_request con 0x7fbb8c03e1c0 auth_method 0 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.989+0000 7fbbad862640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 0x7fbb8c040680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbba8074de0 tx=0x7fbba402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:51.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.991+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbba809d620 con 0x7fbba80a2810 2024-09-10T16:15:51.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.991+0000 7fbb9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:51.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:50.991+0000 7fbb9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:51.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.229+0000 7fbbafac6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:51.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbbafac6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:51.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbbafac6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:51.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbba806eab0 con 0x7fbba80a2810 2024-09-10T16:15:51.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbb9e7fc640 1 -- 172.21.15.164:0/1454839052 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fbba0011380 con 0x7fbba80a2810 2024-09-10T16:15:51.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbb9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:51.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.230+0000 7fbb9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:51.232 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":86,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 msgr2=0x7fbb8c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 0x7fbb8c040680 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbba8074de0 tx=0x7fbba402e000 comp rx=0 tx=0).stop 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 10 monclient: shutdown 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 msgr2=0x7fbba81bac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbba000c780 tx=0x7fbba000cc40 comp rx=0 tx=0).stop 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 shutdown_connections 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb8c03e1c0 0x7fbb8c040680 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:51.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbba80a2810 0x7fbba81bac10 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:51.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbba80a2380 0x7fbba81ba6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:51.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 --2- 172.21.15.164:0/1454839052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbba80a0410 0x7fbba81caf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:51.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 >> 172.21.15.164:0/1454839052 conn(0x7fbba80a3110 msgr2=0x7fbba81bd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:51.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 shutdown_connections 2024-09-10T16:15:51.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:51.232+0000 7fbbafac6640 1 -- 172.21.15.164:0/1454839052 wait complete. 2024-09-10T16:15:54.246 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:15:54.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 10 monclient: get_monmap_and_config 2024-09-10T16:15:54.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 10 monclient: build_initial_monmap 2024-09-10T16:15:54.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 10 monclient: monmap: 2024-09-10T16:15:54.355 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:54.355 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:54.355080+0000 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:54.355080+0000 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:54.356 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 1 Processor -- start 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 1 -- start start 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.354+0000 7f589d085640 10 monclient: init 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:54.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): picked mon.noname-b con 0x7f589806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58980a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58980a4a80 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589809ce50 con 0x7f58980a4a80 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5898138ac0 con 0x7f58980a46a0 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f589806db60 con 0x7f589806cb70 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589d085640 10 monclient(hunting): authenticate will time out at 1332.842381s 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f5896d9d640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 msgr2=0x7f5898134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589759e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f58980a4a80 0x7f58981383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41834/0 (socket says 172.21.15.164:41834) 2024-09-10T16:15:54.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.355+0000 7f589759e640 1 -- 172.21.15.164:0/4042969632 learned_addr learned my addr 172.21.15.164:0/4042969632 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:54.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.357+0000 7f589759e640 1 -- 172.21.15.164:0/4042969632 >> v1:172.21.15.164:6789/0 conn(0x7f58980a4a80 legacy=0x7f58981383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:54.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.358+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v1:172.21.15.165:6789/0 conn(0x7f58980a46a0 legacy=0x7f5898137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:54.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.556+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 msgr2=0x7f5898134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:54.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.556+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:54.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.956+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 msgr2=0x7f5898134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:54.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:54.956+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:55.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:55.757+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 msgr2=0x7f5898134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:55.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:55.757+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): tick 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 monclient(hunting): continuing hunt 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 msgr2=0x7f5898134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:15:57.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589806cb70 0x7f5898134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 -- 172.21.15.164:0/4042969632 >> v1:172.21.15.165:6789/0 conn(0x7f58980a46a0 legacy=0x7f5898137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 -- 172.21.15.164:0/4042969632 >> v1:172.21.15.164:6789/0 conn(0x7f58980a4a80 legacy=0x7f58981383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient: _add_conns ranks=[4,3] 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): picked mon.noname-c con 0x7f5878003810 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): picked mon.noname-a con 0x7f5878006380 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 -- 172.21.15.164:0/4042969632 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5878008eb0 con 0x7f5878006380 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 1 -- 172.21.15.164:0/4042969632 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5878009060 con 0x7f5878003810 2024-09-10T16:15:57.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5894d99640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 10 monclient(hunting): get_auth_request con 0x7f5878006380 auth_method 0 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 10 monclient(hunting): get_auth_request con 0x7f5878003810 auth_method 0 2024-09-10T16:15:57.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.355+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:57.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 10 monclient(hunting): get_auth_request con 0x7f5878006380 auth_method 0 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 10 monclient(hunting): get_auth_request con 0x7f5878003810 auth_method 0 2024-09-10T16:15:57.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.556+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:15:57.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 10 monclient(hunting): get_auth_request con 0x7f5878006380 auth_method 0 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 10 monclient(hunting): get_auth_request con 0x7f5878003810 auth_method 0 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:15:57.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 msgr2=0x7f5878008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:15:57.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:57.957+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.164:3300/0 conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): get_auth_request con 0x7f5878006380 auth_method 0 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:58.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f5896d9d640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f5896d9d640 10 monclient(hunting): get_auth_request con 0x7f5878003810 auth_method 0 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f5896d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f5896d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f5896d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient(hunting): handle_auth_done global_id 74454 payload 274 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 msgr2=0x7f5878005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> v2:172.21.15.165:3300/0 conn(0x7f5878003810 0x7f5878005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient: _finish_hunting 0 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 1 monclient: found mon.noname-a 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:58.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.759+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589809cb70 con 0x7f5878006380 2024-09-10T16:15:58.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589659c640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 msgr2=0x7f5878008840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:58.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 0x7f5878008840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58780037a0 tx=0x7f588002a960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42315dba653c9042 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:58.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 1 -- 172.21.15.164:0/4042969632 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58800072e0 con 0x7f5878006380 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589659c640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 0x7f5878008840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58780037a0 tx=0x7f588002a960 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f5880035070 config(0 keys) 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: dump: 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:58.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: _finish_auth 0 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 1 -- 172.21.15.164:0/4042969632 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5880035070 con 0x7f5878006380 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589d085640 5 monclient: authenticate success, global_id 74454 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f5895d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589d085640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:58.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.762+0000 7f589d085640 4 set_mon_vals no callback set 2024-09-10T16:15:58.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 10 monclient: shutdown 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 -- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 msgr2=0x7f5878008840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 0x7f5878008840 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f58780037a0 tx=0x7f588002a960 comp rx=0 tx=0).stop 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589759e640 1 -- 172.21.15.164:0/4042969632 reap_dead start 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f5895d9b640 1 -- 172.21.15.164:0/4042969632 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f58800057c0 con 0x7f5878006380 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f5895d9b640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 -- 172.21.15.164:0/4042969632 shutdown_connections 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 --2- 172.21.15.164:0/4042969632 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5878006380 0x7f5878008840 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 -- 172.21.15.164:0/4042969632 >> 172.21.15.164:0/4042969632 conn(0x7f58980aa900 msgr2=0x7f58980aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 -- 172.21.15.164:0/4042969632 shutdown_connections 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.763+0000 7f589d085640 1 -- 172.21.15.164:0/4042969632 wait complete. 2024-09-10T16:15:58.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.764+0000 7f589d085640 10 monclient: build_initial_monmap 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.764+0000 7f589d085640 10 monclient: monmap: 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 Processor -- start 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 -- start start 2024-09-10T16:15:58.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient: init 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): picked mon.noname-a con 0x7f589812d280 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 0x7f589809e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): picked mon.noname-b con 0x7f589812dba0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 0x7f589809ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): picked mon.noname-c con 0x7f589809e9c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f589806cf50 con 0x7f589812d280 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:58.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5898141050 con 0x7f589812dba0 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f58981411d0 con 0x7f589809e9c0 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589d085640 10 monclient(hunting): authenticate will time out at 1337.253828s 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589759e640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 msgr2=0x7f589809ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f589759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 0x7f589809ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33354/0 (socket says 172.21.15.164:33354) 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 1 -- 172.21.15.164:0/2043434516 learned_addr learned my addr 172.21.15.164:0/2043434516 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:58.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 10 monclient(hunting): get_auth_request con 0x7f589812d280 auth_method 0 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.765+0000 7f5896d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 0x7f589809e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 10 monclient(hunting): get_auth_request con 0x7f589812dba0 auth_method 0 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient(hunting): handle_auth_done global_id 74457 payload 824 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 -- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 msgr2=0x7f589809ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:58.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 0x7f589809ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 -- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 msgr2=0x7f589809e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 0x7f589809e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f5896d9d640 1 -- 172.21.15.164:0/2043434516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589806e720 con 0x7f589812d280 2024-09-10T16:15:58.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.766+0000 7f589659c640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 0x7f589809e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:15:58.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f5896d9d640 1 -- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 msgr2=0x7f589812d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:58.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f5896d9d640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f588800db70 tx=0x7f588800a3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:58.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800b650 con 0x7f589812d280 2024-09-10T16:15:58.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:58.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: dump: 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:58.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588800b7f0 con 0x7f589812d280 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 5 monclient: authenticate success, global_id 74457 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 10 monclient: _renew_subs 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589809fe30 con 0x7f589812d280 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800bbe0 con 0x7f589812d280 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:58.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f588f7fe640 10 monclient: dump: 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 10 monclient: _renew_subs 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:58.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589559a640 4 set_mon_vals no callback set 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.768+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58980a0990 con 0x7f589812d280 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5888017440 con 0x7f589812d280 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589d085640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589d085640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589d085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5898134f70 con 0x7f589812d280 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f588f7fe640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 0x7f5878041690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7f5888054140 con 0x7f589812d280 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589659c640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 0x7f5878041690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.770+0000 7f589659c640 10 monclient: get_auth_request con 0x7f587803f1d0 auth_method 0 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.773+0000 7f589659c640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 0x7f5878041690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5880034000 tx=0x7f5880007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.775+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5888012050 con 0x7f589812d280 2024-09-10T16:15:58.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.775+0000 7f588f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:58.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:58.775+0000 7f588f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f589d085640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f589d085640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f589d085640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f58980770f0 con 0x7f589812d280 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f588f7fe640 1 -- 172.21.15.164:0/2043434516 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f5888052b60 con 0x7f589812d280 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f588f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:15:59.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.021+0000 7f588f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:59.025 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":90,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:15:59.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 msgr2=0x7f5878041690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 0x7f5878041690 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5880034000 tx=0x7f5880007000 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 10 monclient: shutdown 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 msgr2=0x7f589812d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f588800db70 tx=0x7f588800a3e0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 shutdown_connections 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587803f1d0 0x7f5878041690 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589809e9c0 0x7f589809ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589812dba0 0x7f589809e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 --2- 172.21.15.164:0/2043434516 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f589812d280 0x7f589812d660 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 >> 172.21.15.164:0/2043434516 conn(0x7f58980aa900 msgr2=0x7f58980a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:59.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 shutdown_connections 2024-09-10T16:15:59.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.024+0000 7f589d085640 1 -- 172.21.15.164:0/2043434516 wait complete. 2024-09-10T16:15:59.037 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:15:59.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:15:59.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:15:59.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.135+0000 7f9a750bd640 10 monclient: get_monmap_and_config 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.135+0000 7f9a750bd640 10 monclient: build_initial_monmap 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.135+0000 7f9a750bd640 10 monclient: monmap: 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:59.137386+0000 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:59.137386+0000 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:59.138 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:59.139 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:59.139 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 Processor -- start 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 -- start start 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient: init 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a700a3e60 0x7f9a7006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-a con 0x7f9a700a3e60 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9a7006d0b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9a7013d5f0 0x7f9a7013f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9a7013d5f0 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:59.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a700a4600 con 0x7f9a7006d0b0 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9a700a49b0 con 0x7f9a700a3e60 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9a700a4240 con 0x7f9a7013d5f0 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a750bd640 10 monclient(hunting): authenticate will time out at 1337.624594s 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a700a3e60 0x7f9a7006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6e59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9a7006d0b0 0x7f9a7006d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:41842/0 (socket says 172.21.15.164:41842) 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6e59c640 1 -- 172.21.15.164:0/415312912 learned_addr learned my addr 172.21.15.164:0/415312912 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): get_auth_request con 0x7f9a700a3e60 auth_method 0 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_done global_id 74460 payload 274 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/415312912 >> v2:172.21.15.165:3300/0 conn(0x7f9a7013d5f0 msgr2=0x7f9a7013f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6f59e640 1 --2- 172.21.15.164:0/415312912 >> v2:172.21.15.165:3300/0 conn(0x7f9a7013d5f0 0x7f9a7013f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 1 --2- 172.21.15.164:0/415312912 >> v2:172.21.15.165:3300/0 conn(0x7f9a7013d5f0 0x7f9a7013f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.136+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/415312912 >> v1:172.21.15.164:6789/0 conn(0x7f9a7006d0b0 legacy=0x7f9a7006d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.137+0000 7f9a6ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.137+0000 7f9a6ed9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.137+0000 7f9a6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.137+0000 7f9a6ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.137+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/415312912 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a7006dba0 con 0x7f9a700a3e60 2024-09-10T16:15:59.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/415312912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a700a3e60 msgr2=0x7f9a7006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6ed9d640 1 --2- 172.21.15.164:0/415312912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a700a3e60 0x7f9a7006cb70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a700702e0 tx=0x7f9a60029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25254d7cd36fc4e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 1 -- 172.21.15.164:0/415312912 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60037070 con 0x7f9a700a3e60 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: dump: 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.152 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: _finish_auth 0 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a750bd640 5 monclient: authenticate success, global_id 74460 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 1 -- 172.21.15.164:0/415312912 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a60032440 con 0x7f9a700a3e60 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a750bd640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a750bd640 4 set_mon_vals no callback set 2024-09-10T16:15:59.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 1 -- 172.21.15.164:0/415312912 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60032810 con 0x7f9a700a3e60 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.139+0000 7f9a6dd9b640 10 monclient: dump: 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.154 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 10 monclient: shutdown 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 -- 172.21.15.164:0/415312912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a700a3e60 msgr2=0x7f9a7006cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 --2- 172.21.15.164:0/415312912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a700a3e60 0x7f9a7006cb70 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f9a700702e0 tx=0x7f9a60029fb0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 -- 172.21.15.164:0/415312912 shutdown_connections 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 --2- 172.21.15.164:0/415312912 >> v2:172.21.15.165:3300/0 conn(0x7f9a7013d5f0 0x7f9a7013f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 --2- 172.21.15.164:0/415312912 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a700a3e60 0x7f9a7006cb70 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 -- 172.21.15.164:0/415312912 >> 172.21.15.164:0/415312912 conn(0x7f9a700a4f50 msgr2=0x7f9a700a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 -- 172.21.15.164:0/415312912 shutdown_connections 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.140+0000 7f9a750bd640 1 -- 172.21.15.164:0/415312912 wait complete. 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.141+0000 7f9a750bd640 10 monclient: build_initial_monmap 2024-09-10T16:15:59.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.141+0000 7f9a750bd640 10 monclient: monmap: 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.141+0000 7f9a750bd640 1 Processor -- start 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.141+0000 7f9a750bd640 1 -- start start 2024-09-10T16:15:59.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.141+0000 7f9a750bd640 10 monclient: init 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 0x7f9a701cd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-a con 0x7f9a7006d0b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700a3e60 0x7f9a7009a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-c con 0x7f9a700a3e60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7013d5f0 0x7f9a700713b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): picked mon.noname-b con 0x7f9a7013d5f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7009bd30 con 0x7f9a7006d0b0 2024-09-10T16:15:59.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7006e340 con 0x7f9a7013d5f0 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9a7013ff20 con 0x7f9a700a3e60 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a750bd640 10 monclient(hunting): authenticate will time out at 1337.630167s 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6e59c640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700a3e60 msgr2=0x7f9a7009a9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700a3e60 0x7f9a7009a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 0x7f9a701cd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 0x7f9a701cd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33380/0 (socket says 172.21.15.164:33380) 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/370026128 learned_addr learned my addr 172.21.15.164:0/370026128 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): get_auth_request con 0x7f9a7006d0b0 auth_method 0 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6f59e640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7013d5f0 0x7f9a700713b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6f59e640 10 monclient(hunting): get_auth_request con 0x7f9a7013d5f0 auth_method 0 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient(hunting): handle_auth_done global_id 74463 payload 824 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700a3e60 msgr2=0x7f9a7009a9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:59.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a700a3e60 0x7f9a7009a9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7013d5f0 msgr2=0x7f9a700713b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a7013d5f0 0x7f9a700713b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 monclient: found mon.noname-a 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.142+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a701ce160 con 0x7f9a7006d0b0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a6ed9d640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 msgr2=0x7f9a701cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a6ed9d640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 0x7f9a701cd490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a60000c00 tx=0x7f9a600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a60037040 con 0x7f9a7006d0b0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:59.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: dump: 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: _finish_auth 0 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: _check_auth_tickets 2024-09-10T16:15:59.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a60042070 con 0x7f9a7006d0b0 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a750bd640 5 monclient: authenticate success, global_id 74463 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a750bd640 10 monclient: _renew_subs 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a750bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a7013f9f0 con 0x7f9a7006d0b0 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.144+0000 7f9a577fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a6003bb20 con 0x7f9a7006d0b0 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a577fe640 10 monclient: dump: 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.162 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 10 monclient: _renew_subs 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a70073d20 con 0x7f9a7006d0b0 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6d59a640 4 set_mon_vals no callback set 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 0x7f9a700718f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9a7006cb70 con 0x7f9a7009aef0 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6e59c640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 0x7f9a700718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9a6003bcc0 con 0x7f9a7006d0b0 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6e59c640 10 monclient: get_auth_request con 0x7f9a7009aef0 auth_method 0 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.145+0000 7f9a6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_done global_id 74466 payload 824 2024-09-10T16:15:59.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a577fe640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a5c03e1f0 0x7f9a5c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6e59c640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a70074e40 con 0x7f9a7009aef0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(325..325 src has 184..325) ==== 4739+0+0 (secure 0 0 0) 0x7f9a60073500 con 0x7f9a7006d0b0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6f59e640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a5c03e1f0 0x7f9a5c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.146+0000 7f9a6f59e640 10 monclient: get_auth_request con 0x7f9a5c03e1f0 auth_method 0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a6e59c640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 msgr2=0x7f9a700718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a6e59c640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 0x7f9a700718f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a70074440 tx=0x7f9a6400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64017070 con 0x7f9a7009aef0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 10 monclient: dump: 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.165 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a640070d0 con 0x7f9a7009aef0 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 msgr2=0x7f9a700718f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 0x7f9a700718f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a70074440 tx=0x7f9a6400c7a0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 msgr2=0x7f9a700718f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:59.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a577fe640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7009aef0 0x7f9a700718f0 secure :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a70074440 tx=0x7f9a6400c7a0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.148+0000 7f9a6f59e640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a5c03e1f0 0x7f9a5c0406b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9a68007f20 tx=0x7f9a680074b0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.154+0000 7f9a750bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.154+0000 7f9a750bd640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 0x7f9a7009c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.154+0000 7f9a750bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.154+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9a70072240 con 0x7f9a70071e30 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.154+0000 7f9a6e59c640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 0x7f9a7009c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient: get_auth_request con 0x7f9a70071e30 auth_method 0 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 10 monclient(hunting): handle_auth_done global_id 74469 payload 824 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.155+0000 7f9a6e59c640 1 -- 172.21.15.164:0/370026128 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a700770e0 con 0x7f9a70071e30 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a6e59c640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 msgr2=0x7f9a7009c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a6e59c640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 0x7f9a7009c330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a64014000 tx=0x7f9a64004930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a64017040 con 0x7f9a70071e30 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 10 monclient: dump: 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.170 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f9a64012400 con 0x7f9a70071e30 2024-09-10T16:15:59.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 msgr2=0x7f9a7009c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 0x7f9a7009c330 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a64014000 tx=0x7f9a64004930 comp rx=0 tx=0).stop 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 msgr2=0x7f9a7009c330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.157+0000 7f9a577fe640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a70071e30 0x7f9a7009c330 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f9a64014000 tx=0x7f9a64004930 comp rx=0 tx=0).stop 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:59.172 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 90, 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 187024, 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:59.173 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.174 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.175 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:59.176 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:59.177 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.178 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:59.182 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.183 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.184 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:59.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.186 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:59.187 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:59.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a5c03e1f0 msgr2=0x7f9a5c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a5c03e1f0 0x7f9a5c0406b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9a68007f20 tx=0x7f9a680074b0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 10 monclient: shutdown 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 msgr2=0x7f9a701cd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 --2- 172.21.15.164:0/370026128 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a7006d0b0 0x7f9a701cd490 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f9a60000c00 tx=0x7f9a600046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a6f59e640 1 -- 172.21.15.164:0/370026128 reap_dead start 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a6f59e640 1 -- 172.21.15.164:0/370026128 reap_dead start 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 shutdown_connections 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 >> 172.21.15.164:0/370026128 conn(0x7f9a700a4f50 msgr2=0x7f9a7013c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 shutdown_connections 2024-09-10T16:15:59.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.158+0000 7f9a750bd640 1 -- 172.21.15.164:0/370026128 wait complete. 2024-09-10T16:15:59.190 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:15:59.190 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:15:59.190 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:15:59.222 INFO:tasks.rados.rados.0.smithi165.stdout:1311: done (0 left) 2024-09-10T16:15:59.222 INFO:tasks.rados.rados.0.smithi165.stdout:1312: snap_create 2024-09-10T16:15:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.287+0000 7fd71e524640 10 monclient: get_monmap_and_config 2024-09-10T16:15:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.287+0000 7fd71e524640 10 monclient: build_initial_monmap 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.287+0000 7fd71e524640 10 monclient: monmap: 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:15:59.289171+0000 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:15:59.289171+0000 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:15:59.290 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:15:59.291 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 Processor -- start 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 -- start start 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient: init 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd71809f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd7180a2550 0x7fd71813b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-a con 0x7fd7180a2550 addr v2:172.21.15.164:3300/0 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd7180a2980 0x7fd71813db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-c con 0x7fd7180a2980 addr v2:172.21.15.165:3300/0 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd71809fd10 con 0x7fd71809f930 2024-09-10T16:15:59.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd71809cd20 con 0x7fd7180a2550 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd71809a0d0 con 0x7fd7180a2980 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71e524640 10 monclient(hunting): authenticate will time out at 1337.776598s 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd7180a2550 0x7fd71813b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd7180a2550 0x7fd71813b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33412/0 (socket says 172.21.15.164:33412) 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 -- 172.21.15.164:0/365048361 learned_addr learned my addr 172.21.15.164:0/365048361 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): get_auth_request con 0x7fd7180a2550 auth_method 0 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd71cac1640 1 --2- 172.21.15.164:0/365048361 >> v2:172.21.15.165:3300/0 conn(0x7fd7180a2980 0x7fd71813db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient(hunting): handle_auth_done global_id 74472 payload 274 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 -- 172.21.15.164:0/365048361 >> v2:172.21.15.165:3300/0 conn(0x7fd7180a2980 msgr2=0x7fd71813db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 --2- 172.21.15.164:0/365048361 >> v2:172.21.15.165:3300/0 conn(0x7fd7180a2980 0x7fd71813db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 -- 172.21.15.164:0/365048361 >> v1:172.21.15.165:6789/0 conn(0x7fd71809f930 legacy=0x7fd7180a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.288+0000 7fd7177fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.289+0000 7fd7177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.289+0000 7fd7177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.289+0000 7fd7177fe640 1 -- 172.21.15.164:0/365048361 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd71809e840 con 0x7fd7180a2550 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd7177fe640 1 -- 172.21.15.164:0/365048361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 msgr2=0x7fd71813b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd7177fe640 1 --2- 172.21.15.164:0/365048361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd71813b5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd71809f620 tx=0x7fd70002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e89b0dee94ff07f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 1 -- 172.21.15.164:0/365048361 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700037070 con 0x7fd7180a2550 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: dump: 2024-09-10T16:15:59.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: _finish_auth 0 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:59.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 1 -- 172.21.15.164:0/365048361 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd700032640 con 0x7fd7180a2550 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd71e524640 5 monclient: authenticate success, global_id 74472 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd71e524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd71e524640 10 monclient: get_monmap_and_config success 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd71e524640 4 set_mon_vals no callback set 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 1 -- 172.21.15.164:0/365048361 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700032a10 con 0x7fd7180a2550 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.291+0000 7fd716ffd640 10 monclient: dump: 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.304 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 10 monclient: shutdown 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 -- 172.21.15.164:0/365048361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 msgr2=0x7fd71813b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 --2- 172.21.15.164:0/365048361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd71813b5d0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd71809f620 tx=0x7fd70002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 -- 172.21.15.164:0/365048361 shutdown_connections 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 --2- 172.21.15.164:0/365048361 >> v2:172.21.15.165:3300/0 conn(0x7fd7180a2980 0x7fd71813db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 --2- 172.21.15.164:0/365048361 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd71813b5d0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 -- 172.21.15.164:0/365048361 >> 172.21.15.164:0/365048361 conn(0x7fd7180a3110 msgr2=0x7fd7180a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 -- 172.21.15.164:0/365048361 shutdown_connections 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.292+0000 7fd71e524640 1 -- 172.21.15.164:0/365048361 wait complete. 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.293+0000 7fd71e524640 10 monclient: build_initial_monmap 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.293+0000 7fd71e524640 10 monclient: monmap: 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.306 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.293+0000 7fd71e524640 1 Processor -- start 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.293+0000 7fd71e524640 1 -- start start 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.293+0000 7fd71e524640 10 monclient: init 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient: _reopen_session rank -1 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd71809f930 0x7fd718091370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-b con 0x7fd71809f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd7180918b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-a con 0x7fd7180a2550 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180a2980 0x7fd718091df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): picked mon.noname-c con 0x7fd7180a2980 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd71809d930 con 0x7fd7180a2550 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd71809eb80 con 0x7fd71809f930 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd71809c8b0 con 0x7fd7180a2980 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): _renew_subs 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71e524640 10 monclient(hunting): authenticate will time out at 1337.782161s 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71cac1640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180a2980 msgr2=0x7fd718091df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:15:59.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd71cac1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180a2980 0x7fd718091df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd7180918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd7180918b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:33426/0 (socket says 172.21.15.164:33426) 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 -- 172.21.15.164:0/1877159657 learned_addr learned my addr 172.21.15.164:0/1877159657 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): get_auth_request con 0x7fd7180a2550 auth_method 0 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd71809f930 0x7fd718091370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 10 monclient(hunting): get_auth_request con 0x7fd71809f930 auth_method 0 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient(hunting): handle_auth_done global_id 74475 payload 824 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180a2980 msgr2=0x7fd718091df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd7180a2980 0x7fd718091df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd71809f930 msgr2=0x7fd718091370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd71809f930 0x7fd718091370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient: _finish_hunting 0 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 monclient: found mon.noname-a 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:15:59.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd7177fe640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd718094df0 con 0x7fd7180a2550 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.294+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd71809f930 0x7fd718091370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd7177fe640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 msgr2=0x7fd7180918b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd7177fe640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd7180918b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd7000090b0 tx=0x7fd7000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700041020 con 0x7fd7180a2550 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 10 monclient: dump: 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.311 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 10 monclient: _finish_auth 0 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd70003d910 con 0x7fd7180a2550 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.296+0000 7fd71e524640 5 monclient: authenticate success, global_id 74475 2024-09-10T16:15:59.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 monclient: _renew_subs 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd71806ee50 con 0x7fd7180a2550 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd700037040 con 0x7fd7180a2550 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 10 monclient: dump: 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.313 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd7167fc640 4 set_mon_vals no callback set 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 monclient: _renew_subs 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd718093450 con 0x7fd7180a2550 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd700044020 con 0x7fd7180a2550 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:59.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 0x7fd7181cb270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.297+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd718094590 con 0x7fd7181cae90 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd714ff9640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd6f803e1c0 0x7fd6f8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(326..326 src has 184..326) ==== 4739+0+0 (secure 0 0 0) 0x7fd700073100 con 0x7fd7180a2550 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 0x7fd7181cb270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient: get_auth_request con 0x7fd7181cae90 auth_method 0 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd71cac1640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd6f803e1c0 0x7fd6f8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd71cac1640 10 monclient: get_auth_request con 0x7fd6f803e1c0 auth_method 0 2024-09-10T16:15:59.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 10 monclient(hunting): handle_auth_done global_id 94249 payload 824 2024-09-10T16:15:59.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.298+0000 7fd717fff640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd7180a0760 con 0x7fd7181cae90 2024-09-10T16:15:59.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd717fff640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 msgr2=0x7fd7181cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 0x7fd7181cb270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd70800e9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd71cac1640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd6f803e1c0 0x7fd6f8040680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd718093370 tx=0x7fd710008300 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708013800 con 0x7fd7181cae90 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd714ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd714ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.300+0000 7fd714ff9640 10 monclient: dump: 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd708013800 con 0x7fd7181cae90 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 msgr2=0x7fd7181cb270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 0x7fd7181cb270 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd70800e9c0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 msgr2=0x7fd7181cb270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:59.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.301+0000 7fd714ff9640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cae90 0x7fd7181cb270 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd71e524640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd71e524640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 0x7fd71808df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd71e524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd71809d5a0 con 0x7fd7181cb7b0 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 0x7fd71808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient: get_auth_request con 0x7fd7181cb7b0 auth_method 0 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:15:59.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.307+0000 7fd717fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:15:59.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.308+0000 7fd717fff640 10 monclient(hunting): handle_auth_done global_id 94252 payload 824 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.308+0000 7fd717fff640 1 -- 172.21.15.164:0/1877159657 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd71806e890 con 0x7fd7181cb7b0 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd717fff640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 msgr2=0x7fd71808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd717fff640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 0x7fd71808df80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd708003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd708006df0 con 0x7fd7181cb7b0 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 10 monclient: dump: 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.324 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:15:59.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fd708006df0 con 0x7fd7181cb7b0 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 msgr2=0x7fd71808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 0x7fd71808df80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd708003760 comp rx=0 tx=0).stop 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 msgr2=0x7fd71808df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.310+0000 7fd714ff9640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd7181cb7b0 0x7fd71808df80 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fd70800e500 tx=0x7fd708003760 comp rx=0 tx=0).stop 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:59.326 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 90, 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 76811, 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:15:59.327 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:15:59.328 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.329 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:15:59.330 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:15:59.331 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:15:59.332 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.333 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:15:59.334 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.336 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:15:59.339 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:15:59.340 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:15:59.341 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:15:59.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd6f803e1c0 msgr2=0x7fd6f8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd6f803e1c0 0x7fd6f8040680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fd718093370 tx=0x7fd710008300 comp rx=0 tx=0).stop 2024-09-10T16:15:59.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 10 monclient: shutdown 2024-09-10T16:15:59.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:15:59.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 msgr2=0x7fd7180918b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 --2- 172.21.15.164:0/1877159657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd7180a2550 0x7fd7180918b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd7000090b0 tx=0x7fd7000046c0 comp rx=0 tx=0).stop 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71cac1640 1 -- 172.21.15.164:0/1877159657 reap_dead start 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71cac1640 1 -- 172.21.15.164:0/1877159657 reap_dead start 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 shutdown_connections 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 >> 172.21.15.164:0/1877159657 conn(0x7fd7180a3110 msgr2=0x7fd7181be5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 shutdown_connections 2024-09-10T16:15:59.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:15:59.311+0000 7fd71e524640 1 -- 172.21.15.164:0/1877159657 wait complete. 2024-09-10T16:15:59.343 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:15:59.343 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:16:00.225 INFO:tasks.rados.rados.0.smithi165.stdout:1312: done (0 left) 2024-09-10T16:16:00.225 INFO:tasks.rados.rados.0.smithi165.stdout:1313: read oid 29 snap -1 2024-09-10T16:16:00.225 INFO:tasks.rados.rados.0.smithi165.stdout:1313: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:16:00.225 INFO:tasks.rados.rados.0.smithi165.stdout:1314: write (excl) oid 49 current snap is 152 2024-09-10T16:16:00.225 INFO:tasks.rados.rados.0.smithi165.stdout:1314: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-10T16:16:00.232 INFO:tasks.rados.rados.0.smithi165.stdout:1314: writing smithi16534841-49 from 732904 to 1464318 tid 1 2024-09-10T16:16:00.239 INFO:tasks.rados.rados.0.smithi165.stdout:1314: writing smithi16534841-49 from 2120442 to 2659266 tid 2 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:1314: writing smithi16534841-49 from 3278829 to 3783572 tid 3 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:1315: read oid 11 snap -1 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:1315: expect deleted 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:1316: rollback oid 14 current snap is 152 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 146 2024-09-10T16:16:00.245 INFO:tasks.rados.rados.0.smithi165.stdout:1313: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:16:00.355 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:00.409 INFO:tasks.rados.rados.0.smithi165.stdout:1314: finishing write tid 1 to smithi16534841-49 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: finishing write tid 2 to smithi16534841-49 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: finishing write tid 3 to smithi16534841-49 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: finishing write tid 4 to smithi16534841-49 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: finishing write tid 5 to smithi16534841-49 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: oid 49 updating version 0 to 1000 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: oid 49 updating version 1000 to 1002 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: oid 49 version 1002 is already newer than 1001 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: oid 49 version 1002 is already newer than 999 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1002 (ObjNum 472 snap 152 seq_num 472) dirty exists 2024-09-10T16:16:00.410 INFO:tasks.rados.rados.0.smithi165.stdout:1314: left oid 49 (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1316: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1313: done (3 left) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1314: done (2 left) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1315: done (1 left) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1317: read oid 34 snap -1 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1317: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1318: read oid 9 snap -1 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1318: expect (ObjNum 426 snap 139 seq_num 426) 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1319: rollback oid 47 current snap is 152 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 147 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1316: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:1316: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:16:00.411 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 342 (ObjNum 299 snap 87 seq_num 299) dirty exists 2024-09-10T16:16:00.412 INFO:tasks.rados.rados.0.smithi165.stdout:1316: done (3 left) 2024-09-10T16:16:00.412 INFO:tasks.rados.rados.0.smithi165.stdout:1320: snap_remove snap 151 2024-09-10T16:16:01.234 INFO:tasks.rados.rados.0.smithi165.stdout:1320: done (3 left) 2024-09-10T16:16:01.234 INFO:tasks.rados.rados.0.smithi165.stdout:1321: copy_from oid 16 from oid 43 current snap is 152 2024-09-10T16:16:01.234 INFO:tasks.rados.rados.0.smithi165.stdout:1318: expect (ObjNum 426 snap 139 seq_num 426) 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1319: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1319: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1319: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 859 (ObjNum 466 snap 146 seq_num 466) dirty exists 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1318: done (3 left) 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1319: done (2 left) 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1322: rollback oid 50 current snap is 152 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 146 2024-09-10T16:16:01.312 INFO:tasks.rados.rados.0.smithi165.stdout:1317: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:16:01.313 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:01.372 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 147802 2024-09-10T16:16:01.489 INFO:tasks.rados.rados.0.smithi165.stdout:1321: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:16:01.489 INFO:tasks.rados.rados.0.smithi165.stdout:1321: finishing copy_from to smithi16534841-16 2024-09-10T16:16:01.489 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 870 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-10T16:16:01.489 INFO:tasks.rados.rados.0.smithi165.stdout:1322: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1322: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1322: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 580 (ObjNum 417 snap 137 seq_num 417) dirty exists 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1317: done (2 left) 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1321: done (1 left) 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1322: done (0 left) 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1323: read oid 20 snap -1 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1323: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1324: copy_from oid 47 from oid 40 current snap is 152 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1325: rollback oid 29 current snap is 152 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 148 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1326: read oid 7 snap 147 2024-09-10T16:16:01.490 INFO:tasks.rados.rados.0.smithi165.stdout:1326: expect (ObjNum 450 snap 143 seq_num 450) 2024-09-10T16:16:01.491 INFO:tasks.rados.rados.0.smithi165.stdout:1327: delete oid 12 current snap is 152 2024-09-10T16:16:01.493 INFO:tasks.rados.rados.0.smithi165.stdout:1327: done (4 left) 2024-09-10T16:16:01.493 INFO:tasks.rados.rados.0.smithi165.stdout:1328: snap_remove snap 149 2024-09-10T16:16:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:1328: done (4 left) 2024-09-10T16:16:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:1329: delete oid 11 current snap is 152 2024-09-10T16:16:02.173 INFO:tasks.rados.rados.0.smithi165.stdout:1323: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:02.251 INFO:tasks.rados.rados.0.smithi165.stdout:1324: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1323: done (4 left) 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1329: done (3 left) 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1330: read oid 34 snap -1 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1330: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1331: delete oid 20 current snap is 152 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1325: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1325: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1325: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 865 (ObjNum 251 snap 76 seq_num 251) dirty exists 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:1326: expect (ObjNum 450 snap 143 seq_num 450) 2024-09-10T16:16:02.252 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:02.381 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 823206 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1324: finishing copy_from to smithi16534841-47 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 339 (ObjNum 437 snap 141 seq_num 437) dirty exists 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1324: done (4 left) 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1325: done (3 left) 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1326: done (2 left) 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1331: done (1 left) 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1332: write oid 44 current snap is 152 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:1330: expect (ObjNum 429 snap 141 seq_num 429) 2024-09-10T16:16:02.577 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:02.693 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:02.752 INFO:tasks.rados.rados.0.smithi165.stdout:1332: seq_num 476 ranges {400132=632410,1438017=546162} 2024-09-10T16:16:02.757 INFO:tasks.rados.rados.0.smithi165.stdout:1332: writing smithi16534841-44 from 400132 to 1032542 tid 1 2024-09-10T16:16:02.765 INFO:tasks.rados.rados.0.smithi165.stdout:1332: writing smithi16534841-44 from 1438017 to 1984179 tid 2 2024-09-10T16:16:02.765 INFO:tasks.rados.rados.0.smithi165.stdout:1330: done (1 left) 2024-09-10T16:16:02.765 INFO:tasks.rados.rados.0.smithi165.stdout:1333: snap_create 2024-09-10T16:16:02.766 INFO:tasks.rados.rados.0.smithi165.stdout:1332: finishing write tid 1 to smithi16534841-44 2024-09-10T16:16:02.767 INFO:tasks.rados.rados.0.smithi165.stdout:1332: finishing write tid 2 to smithi16534841-44 2024-09-10T16:16:02.768 INFO:tasks.rados.rados.0.smithi165.stdout:1332: finishing write tid 3 to smithi16534841-44 2024-09-10T16:16:02.768 INFO:tasks.rados.rados.0.smithi165.stdout:1332: finishing write tid 4 to smithi16534841-44 2024-09-10T16:16:02.768 INFO:tasks.rados.rados.0.smithi165.stdout:1332: oid 44 updating version 0 to 633 2024-09-10T16:16:02.768 INFO:tasks.rados.rados.0.smithi165.stdout:1332: oid 44 updating version 633 to 635 2024-09-10T16:16:02.768 INFO:tasks.rados.rados.0.smithi165.stdout:1332: oid 44 version 635 is already newer than 634 2024-09-10T16:16:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 635 (ObjNum 476 snap 152 seq_num 476) dirty exists 2024-09-10T16:16:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:1332: left oid 44 (ObjNum 476 snap 152 seq_num 476) 2024-09-10T16:16:03.240 INFO:tasks.rados.rados.0.smithi165.stdout:1332: done (1 left) 2024-09-10T16:16:03.240 INFO:tasks.rados.rados.0.smithi165.stdout:1333: done (0 left) 2024-09-10T16:16:03.240 INFO:tasks.rados.rados.0.smithi165.stdout:1334: copy_from oid 26 from oid 12 current snap is 153 2024-09-10T16:16:03.241 INFO:tasks.rados.rados.0.smithi165.stdout:1335: delete oid 39 current snap is 153 2024-09-10T16:16:03.243 INFO:tasks.rados.rados.0.smithi165.stdout:1335: done (1 left) 2024-09-10T16:16:03.243 INFO:tasks.rados.rados.0.smithi165.stdout:1336: write (excl) oid 29 current snap is 153 2024-09-10T16:16:03.243 INFO:tasks.rados.rados.0.smithi165.stdout:1336: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-10T16:16:03.249 INFO:tasks.rados.rados.0.smithi165.stdout:1336: writing smithi16534841-29 from 433746 to 1216654 tid 1 2024-09-10T16:16:03.254 INFO:tasks.rados.rados.0.smithi165.stdout:1336: writing smithi16534841-29 from 1954981 to 2033746 tid 2 2024-09-10T16:16:03.254 INFO:tasks.rados.rados.0.smithi165.stdout:1337: rollback oid 44 current snap is 153 2024-09-10T16:16:03.254 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 152 2024-09-10T16:16:03.254 INFO:tasks.rados.rados.0.smithi165.stdout:1334: finishing copy_from to smithi16534841-26 2024-09-10T16:16:03.254 INFO:tasks.rados.rados.0.smithi165.stdout:1334: got expected ENOENT (src dne) 2024-09-10T16:16:03.255 INFO:tasks.rados.rados.0.smithi165.stdout:1334: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:16:03.255 INFO:tasks.rados.rados.0.smithi165.stdout:1334: done (2 left) 2024-09-10T16:16:03.255 INFO:tasks.rados.rados.0.smithi165.stdout:1338: delete oid 41 current snap is 153 2024-09-10T16:16:03.258 INFO:tasks.rados.rados.0.smithi165.stdout:1337: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:16:03.258 INFO:tasks.rados.rados.0.smithi165.stdout:1336: finishing write tid 1 to smithi16534841-29 2024-09-10T16:16:03.258 INFO:tasks.rados.rados.0.smithi165.stdout:1337: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:16:03.259 INFO:tasks.rados.rados.0.smithi165.stdout:1338: done (2 left) 2024-09-10T16:16:03.259 INFO:tasks.rados.rados.0.smithi165.stdout:1339: read oid 26 snap -1 2024-09-10T16:16:03.259 INFO:tasks.rados.rados.0.smithi165.stdout:1339: expect (ObjNum 462 snap 144 seq_num 462) 2024-09-10T16:16:03.259 INFO:tasks.rados.rados.0.smithi165.stdout:1340: snap_remove snap 147 2024-09-10T16:16:04.246 INFO:tasks.rados.rados.0.smithi165.stdout:1340: done (3 left) 2024-09-10T16:16:04.246 INFO:tasks.rados.rados.0.smithi165.stdout:1341: rollback oid 13 current snap is 153 2024-09-10T16:16:04.246 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 146 2024-09-10T16:16:04.246 INFO:tasks.rados.rados.0.smithi165.stdout:1337: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:16:04.246 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 638 (ObjNum 476 snap 152 seq_num 476) dirty exists 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: finishing write tid 2 to smithi16534841-29 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: finishing write tid 3 to smithi16534841-29 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: finishing write tid 4 to smithi16534841-29 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: oid 29 updating version 0 to 874 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: oid 29 version 874 is already newer than 872 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: oid 29 version 874 is already newer than 873 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 874 (ObjNum 478 snap 153 seq_num 478) dirty exists 2024-09-10T16:16:04.247 INFO:tasks.rados.rados.0.smithi165.stdout:1336: left oid 29 (ObjNum 478 snap 153 seq_num 478) 2024-09-10T16:16:04.248 INFO:tasks.rados.rados.0.smithi165.stdout:1339: expect (ObjNum 462 snap 144 seq_num 462) 2024-09-10T16:16:04.389 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:04.464 INFO:tasks.rados.rados.0.smithi165.stdout:1341: finishing rollback tid 0 to smithi16534841-13 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1341: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1341: finishing rollback tid 2 to smithi16534841-13 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 739 (ObjNum 440 snap 142 seq_num 440) dirty exists 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1336: done (3 left) 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1337: done (2 left) 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1339: done (1 left) 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1341: done (0 left) 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1342: write oid 37 current snap is 153 2024-09-10T16:16:04.465 INFO:tasks.rados.rados.0.smithi165.stdout:1342: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-10T16:16:04.470 INFO:tasks.rados.rados.0.smithi165.stdout:1342: writing smithi16534841-37 from 467360 to 1117119 tid 1 2024-09-10T16:16:04.475 INFO:tasks.rados.rados.0.smithi165.stdout:1342: writing smithi16534841-37 from 1671945 to 2067360 tid 2 2024-09-10T16:16:04.476 INFO:tasks.rados.rados.0.smithi165.stdout:1343: write oid 34 current snap is 153 2024-09-10T16:16:04.476 INFO:tasks.rados.rados.0.smithi165.stdout:1343: seq_num 481 ranges {484167=725008,1730427=353740} 2024-09-10T16:16:04.481 INFO:tasks.rados.rados.0.smithi165.stdout:1343: writing smithi16534841-34 from 484167 to 1209175 tid 1 2024-09-10T16:16:04.487 INFO:tasks.rados.rados.0.smithi165.stdout:1343: writing smithi16534841-34 from 1730427 to 2084167 tid 2 2024-09-10T16:16:04.487 INFO:tasks.rados.rados.0.smithi165.stdout:1344: write (excl) oid 14 current snap is 153 2024-09-10T16:16:04.487 INFO:tasks.rados.rados.0.smithi165.stdout:1344: seq_num 482 ranges {500974=400257,1388909=557169} 2024-09-10T16:16:04.491 INFO:tasks.rados.rados.0.smithi165.stdout:1344: writing smithi16534841-14 from 500974 to 901231 tid 1 2024-09-10T16:16:04.497 INFO:tasks.rados.rados.0.smithi165.stdout:1344: writing smithi16534841-14 from 1388909 to 1946078 tid 2 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1345: rollback oid 39 current snap is 153 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 148 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: finishing write tid 1 to smithi16534841-37 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: finishing write tid 2 to smithi16534841-37 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: finishing write tid 3 to smithi16534841-37 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: finishing write tid 4 to smithi16534841-37 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: oid 37 updating version 0 to 569 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: oid 37 updating version 569 to 571 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:1342: oid 37 version 571 is already newer than 570 2024-09-10T16:16:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 571 (ObjNum 480 snap 153 seq_num 480) dirty exists 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1342: left oid 37 (ObjNum 480 snap 153 seq_num 480) 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: finishing write tid 1 to smithi16534841-34 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: finishing write tid 2 to smithi16534841-34 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: finishing write tid 3 to smithi16534841-34 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: finishing write tid 4 to smithi16534841-34 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: oid 34 updating version 0 to 573 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: oid 34 updating version 573 to 574 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: oid 34 version 574 is already newer than 572 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 574 (ObjNum 481 snap 153 seq_num 481) dirty exists 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: left oid 34 (ObjNum 481 snap 153 seq_num 481) 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1344: finishing write tid 1 to smithi16534841-14 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1342: done (3 left) 2024-09-10T16:16:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:1343: done (2 left) 2024-09-10T16:16:04.500 INFO:tasks.rados.rados.0.smithi165.stdout:1346: snap_remove snap 152 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1346: done (2 left) 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1347: copy_from oid 35 from oid 41 current snap is 153 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1345: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 0 (ObjNum 257 snap 0 seq_num 1162620416) dirty dne 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1344: finishing write tid 2 to smithi16534841-14 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1344: finishing write tid 3 to smithi16534841-14 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1344: finishing write tid 4 to smithi16534841-14 2024-09-10T16:16:05.256 INFO:tasks.rados.rados.0.smithi165.stdout:1344: oid 14 updating version 0 to 344 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1344: oid 14 version 344 is already newer than 343 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1344: oid 14 updating version 344 to 345 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 345 (ObjNum 482 snap 153 seq_num 482) dirty exists 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1344: left oid 14 (ObjNum 482 snap 153 seq_num 482) 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1344: done (2 left) 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1345: done (1 left) 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1348: read oid 24 snap -1 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1348: expect (ObjNum 412 snap 136 seq_num 412) 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1349: write (excl) oid 2 current snap is 153 2024-09-10T16:16:05.257 INFO:tasks.rados.rados.0.smithi165.stdout:1349: seq_num 483 ranges {517781=475506,1731038=386743} 2024-09-10T16:16:05.261 INFO:tasks.rados.rados.0.smithi165.stdout:1349: writing smithi16534841-2 from 517781 to 993287 tid 1 2024-09-10T16:16:05.269 INFO:tasks.rados.rados.0.smithi165.stdout:1349: writing smithi16534841-2 from 1731038 to 2117781 tid 2 2024-09-10T16:16:05.269 INFO:tasks.rados.rados.0.smithi165.stdout:1350: rollback oid 4 current snap is 153 2024-09-10T16:16:05.269 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 146 2024-09-10T16:16:05.269 INFO:tasks.rados.rados.0.smithi165.stdout:1348: expect (ObjNum 412 snap 136 seq_num 412) 2024-09-10T16:16:05.383 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1347: finishing copy_from to smithi16534841-35 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1347: got expected ENOENT (src dne) 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1347: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1349: finishing write tid 1 to smithi16534841-2 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1349: finishing write tid 2 to smithi16534841-2 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1349: finishing write tid 3 to smithi16534841-2 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1349: finishing write tid 4 to smithi16534841-2 2024-09-10T16:16:05.442 INFO:tasks.rados.rados.0.smithi165.stdout:1349: oid 2 updating version 0 to 885 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1349: oid 2 updating version 885 to 886 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1349: oid 2 version 886 is already newer than 884 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 886 (ObjNum 483 snap 153 seq_num 483) dirty exists 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1349: left oid 2 (ObjNum 483 snap 153 seq_num 483) 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1350: finishing rollback tid 0 to smithi16534841-4 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1350: finishing rollback tid 1 to smithi16534841-4 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1350: finishing rollback tid 2 to smithi16534841-4 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 704 (ObjNum 454 snap 143 seq_num 454) dirty exists 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1347: done (3 left) 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1348: done (2 left) 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1349: done (1 left) 2024-09-10T16:16:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:1350: done (0 left) 2024-09-10T16:16:05.444 INFO:tasks.rados.rados.0.smithi165.stdout:1351: write (excl) oid 3 current snap is 153 2024-09-10T16:16:05.444 INFO:tasks.rados.rados.0.smithi165.stdout:1351: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-10T16:16:05.447 INFO:tasks.rados.rados.0.smithi165.stdout:1351: writing smithi16534841-3 from 534588 to 1085343 tid 1 2024-09-10T16:16:05.453 INFO:tasks.rados.rados.0.smithi165.stdout:1351: writing smithi16534841-3 from 1789520 to 2134588 tid 2 2024-09-10T16:16:05.453 INFO:tasks.rados.rados.0.smithi165.stdout:1352: delete oid 17 current snap is 153 2024-09-10T16:16:05.453 INFO:tasks.rados.rados.0.smithi165.stdout:1351: finishing write tid 1 to smithi16534841-3 2024-09-10T16:16:05.456 INFO:tasks.rados.rados.0.smithi165.stdout:1351: finishing write tid 2 to smithi16534841-3 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1352: done (1 left) 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1353: rollback oid 46 current snap is 153 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 148 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1354: read oid 8 snap -1 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1354: expect (ObjNum 425 snap 138 seq_num 425) 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1355: write (excl) oid 10 current snap is 153 2024-09-10T16:16:05.457 INFO:tasks.rados.rados.0.smithi165.stdout:1355: seq_num 486 ranges {568202=701253,1906484=261718} 2024-09-10T16:16:05.463 INFO:tasks.rados.rados.0.smithi165.stdout:1355: writing smithi16534841-10 from 568202 to 1269455 tid 1 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1355: writing smithi16534841-10 from 1906484 to 2168202 tid 2 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1356: read oid 49 snap -1 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1356: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1357: delete oid 47 current snap is 153 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1351: finishing write tid 3 to smithi16534841-3 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1351: finishing write tid 4 to smithi16534841-3 2024-09-10T16:16:05.469 INFO:tasks.rados.rados.0.smithi165.stdout:1351: oid 3 updating version 0 to 895 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:1351: oid 3 version 895 is already newer than 893 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:1351: oid 3 version 895 is already newer than 894 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 895 (ObjNum 484 snap 153 seq_num 484) dirty exists 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:1351: left oid 3 (ObjNum 484 snap 153 seq_num 484) 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:1353: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:16:05.470 INFO:tasks.rados.rados.0.smithi165.stdout:1354: expect (ObjNum 425 snap 138 seq_num 425) 2024-09-10T16:16:05.568 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2200568 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: finishing write tid 1 to smithi16534841-10 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: finishing write tid 2 to smithi16534841-10 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: finishing write tid 3 to smithi16534841-10 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: finishing write tid 4 to smithi16534841-10 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: oid 10 updating version 0 to 1006 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: oid 10 version 1006 is already newer than 1004 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: oid 10 version 1006 is already newer than 1005 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1006 (ObjNum 486 snap 153 seq_num 486) dirty exists 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1355: left oid 10 (ObjNum 486 snap 153 seq_num 486) 2024-09-10T16:16:05.651 INFO:tasks.rados.rados.0.smithi165.stdout:1356: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:05.801 INFO:tasks.rados.rados.0.smithi165.stdout:1351: done (5 left) 2024-09-10T16:16:05.801 INFO:tasks.rados.rados.0.smithi165.stdout:1353: done (4 left) 2024-09-10T16:16:05.801 INFO:tasks.rados.rados.0.smithi165.stdout:1354: done (3 left) 2024-09-10T16:16:05.801 INFO:tasks.rados.rados.0.smithi165.stdout:1355: done (2 left) 2024-09-10T16:16:05.801 INFO:tasks.rados.rados.0.smithi165.stdout:1356: done (1 left) 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1357: done (0 left) 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1358: rollback oid 45 current snap is 153 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 146 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1359: read oid 23 snap -1 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1359: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1360: rollback oid 17 current snap is 153 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 146 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1361: copy_from oid 41 from oid 15 current snap is 153 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1362: write oid 11 current snap is 153 2024-09-10T16:16:05.802 INFO:tasks.rados.rados.0.smithi165.stdout:1362: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-10T16:16:05.807 INFO:tasks.rados.rados.0.smithi165.stdout:1362: writing smithi16534841-11 from 601816 to 1169920 tid 1 2024-09-10T16:16:05.813 INFO:tasks.rados.rados.0.smithi165.stdout:1362: writing smithi16534841-11 from 1623448 to 2075271 tid 2 2024-09-10T16:16:05.813 INFO:tasks.rados.rados.0.smithi165.stdout:1363: write oid 27 current snap is 153 2024-09-10T16:16:05.813 INFO:tasks.rados.rados.0.smithi165.stdout:1360: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:16:05.813 INFO:tasks.rados.rados.0.smithi165.stdout:1363: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-10T16:16:05.819 INFO:tasks.rados.rados.0.smithi165.stdout:1363: writing smithi16534841-27 from 618623 to 1261976 tid 1 2024-09-10T16:16:05.825 INFO:tasks.rados.rados.0.smithi165.stdout:1363: writing smithi16534841-27 from 1681930 to 2218623 tid 2 2024-09-10T16:16:05.825 INFO:tasks.rados.rados.0.smithi165.stdout:1364: snap_remove snap 148 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:1364: done (6 left) 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:1365: snap_create 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:1360: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1008 (ObjNum 369 snap 108 seq_num 369) dirty exists 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:1358: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:16:06.263 INFO:tasks.rados.rados.0.smithi165.stdout:1358: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:16:06.264 INFO:tasks.rados.rados.0.smithi165.stdout:1358: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:16:06.264 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 880 (ObjNum 415 snap 136 seq_num 415) dirty exists 2024-09-10T16:16:06.264 INFO:tasks.rados.rados.0.smithi165.stdout:1359: expect (ObjNum 179 snap 54 seq_num 179) 2024-09-10T16:16:06.432 INFO:tasks.rados.rados.0.smithi165.stdout:1362: finishing write tid 1 to smithi16534841-11 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: finishing write tid 2 to smithi16534841-11 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: finishing write tid 3 to smithi16534841-11 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: finishing write tid 4 to smithi16534841-11 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: oid 11 updating version 0 to 582 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: oid 11 version 582 is already newer than 581 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: oid 11 updating version 582 to 583 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 583 (ObjNum 488 snap 153 seq_num 488) dirty exists 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1362: left oid 11 (ObjNum 488 snap 153 seq_num 488) 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1361: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:1361: finishing copy_from to smithi16534841-41 2024-09-10T16:16:06.433 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 701 (ObjNum 465 snap 146 seq_num 465) dirty exists 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: finishing write tid 1 to smithi16534841-27 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: finishing write tid 2 to smithi16534841-27 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: finishing write tid 3 to smithi16534841-27 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: finishing write tid 4 to smithi16534841-27 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: oid 27 updating version 0 to 889 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: oid 27 updating version 889 to 890 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: oid 27 version 890 is already newer than 888 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 890 (ObjNum 489 snap 153 seq_num 489) dirty exists 2024-09-10T16:16:06.434 INFO:tasks.rados.rados.0.smithi165.stdout:1363: left oid 27 (ObjNum 489 snap 153 seq_num 489) 2024-09-10T16:16:07.176 INFO:tasks.rados.rados.0.smithi165.stdout:1358: done (6 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1359: done (5 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1360: done (4 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1361: done (3 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1362: done (2 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1363: done (1 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1365: done (0 left) 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1366: copy_from oid 23 from oid 32 current snap is 154 2024-09-10T16:16:07.177 INFO:tasks.rados.rados.0.smithi165.stdout:1367: delete oid 38 current snap is 154 2024-09-10T16:16:07.181 INFO:tasks.rados.rados.0.smithi165.stdout:1366: finishing copy_from to smithi16534841-23 2024-09-10T16:16:07.181 INFO:tasks.rados.rados.0.smithi165.stdout:1366: got expected ENOENT (src dne) 2024-09-10T16:16:07.181 INFO:tasks.rados.rados.0.smithi165.stdout:1366: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:16:07.184 INFO:tasks.rados.rados.0.smithi165.stdout:1366: done (1 left) 2024-09-10T16:16:07.184 INFO:tasks.rados.rados.0.smithi165.stdout:1367: done (0 left) 2024-09-10T16:16:07.185 INFO:tasks.rados.rados.0.smithi165.stdout:1368: snap_create 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1368: done (0 left) 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1369: copy_from oid 45 from oid 20 current snap is 155 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1370: read oid 6 snap -1 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1370: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1371: write oid 13 current snap is 155 2024-09-10T16:16:08.184 INFO:tasks.rados.rados.0.smithi165.stdout:1371: seq_num 491 ranges {652237=793851,2198894=53343} 2024-09-10T16:16:08.192 INFO:tasks.rados.rados.0.smithi165.stdout:1371: writing smithi16534841-13 from 652237 to 1446088 tid 1 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1371: writing smithi16534841-13 from 2198894 to 2252237 tid 2 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1372: read oid 18 snap -1 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1372: expect deleted 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1373: write (excl) oid 42 current snap is 155 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1369: finishing copy_from to smithi16534841-45 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1369: got expected ENOENT (src dne) 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1369: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:1370: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:08.197 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:08.233 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 964271 2024-09-10T16:16:08.282 INFO:tasks.rados.rados.0.smithi165.stdout:1371: finishing write tid 1 to smithi16534841-13 2024-09-10T16:16:08.282 INFO:tasks.rados.rados.0.smithi165.stdout:1371: finishing write tid 2 to smithi16534841-13 2024-09-10T16:16:08.282 INFO:tasks.rados.rados.0.smithi165.stdout:1371: finishing write tid 3 to smithi16534841-13 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1371: finishing write tid 4 to smithi16534841-13 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1371: oid 13 updating version 0 to 741 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1371: oid 13 updating version 741 to 742 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1371: oid 13 version 742 is already newer than 740 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 742 (ObjNum 491 snap 155 seq_num 491) dirty exists 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1371: left oid 13 (ObjNum 491 snap 155 seq_num 491) 2024-09-10T16:16:08.283 INFO:tasks.rados.rados.0.smithi165.stdout:1373: seq_num 492 ranges {669044=469100,1741023=459161} 2024-09-10T16:16:08.288 INFO:tasks.rados.rados.0.smithi165.stdout:1373: writing smithi16534841-42 from 669044 to 1138144 tid 1 2024-09-10T16:16:08.294 INFO:tasks.rados.rados.0.smithi165.stdout:1373: writing smithi16534841-42 from 1741023 to 2200184 tid 2 2024-09-10T16:16:08.294 INFO:tasks.rados.rados.0.smithi165.stdout:1369: done (4 left) 2024-09-10T16:16:08.294 INFO:tasks.rados.rados.0.smithi165.stdout:1370: done (3 left) 2024-09-10T16:16:08.295 INFO:tasks.rados.rados.0.smithi165.stdout:1371: done (2 left) 2024-09-10T16:16:08.295 INFO:tasks.rados.rados.0.smithi165.stdout:1372: done (1 left) 2024-09-10T16:16:08.295 INFO:tasks.rados.rados.0.smithi165.stdout:1374: copy_from oid 12 from oid 5 current snap is 155 2024-09-10T16:16:08.295 INFO:tasks.rados.rados.0.smithi165.stdout:1373: finishing write tid 1 to smithi16534841-42 2024-09-10T16:16:08.295 INFO:tasks.rados.rados.0.smithi165.stdout:1375: delete oid 23 current snap is 155 2024-09-10T16:16:08.301 INFO:tasks.rados.rados.0.smithi165.stdout:1373: finishing write tid 2 to smithi16534841-42 2024-09-10T16:16:08.301 INFO:tasks.rados.rados.0.smithi165.stdout:1373: finishing write tid 3 to smithi16534841-42 2024-09-10T16:16:08.301 INFO:tasks.rados.rados.0.smithi165.stdout:1373: finishing write tid 4 to smithi16534841-42 2024-09-10T16:16:08.301 INFO:tasks.rados.rados.0.smithi165.stdout:1373: oid 42 updating version 0 to 584 2024-09-10T16:16:08.301 INFO:tasks.rados.rados.0.smithi165.stdout:1373: oid 42 updating version 584 to 586 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1373: oid 42 version 586 is already newer than 585 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 586 (ObjNum 492 snap 155 seq_num 492) dirty exists 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1373: left oid 42 (ObjNum 492 snap 155 seq_num 492) 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1373: done (2 left) 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1375: done (1 left) 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1376: copy_from oid 43 from oid 23 current snap is 155 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1377: copy_from oid 41 from oid 6 current snap is 155 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1378: rollback oid 37 current snap is 155 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 150 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1379: write (excl) oid 39 current snap is 155 2024-09-10T16:16:08.302 INFO:tasks.rados.rados.0.smithi165.stdout:1379: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-10T16:16:08.307 INFO:tasks.rados.rados.0.smithi165.stdout:1379: writing smithi16534841-39 from 702658 to 1322256 tid 1 2024-09-10T16:16:08.313 INFO:tasks.rados.rados.0.smithi165.stdout:1379: writing smithi16534841-39 from 1857987 to 2302658 tid 2 2024-09-10T16:16:08.313 INFO:tasks.rados.rados.0.smithi165.stdout:1380: copy_from oid 27 from oid 48 current snap is 155 2024-09-10T16:16:08.313 INFO:tasks.rados.rados.0.smithi165.stdout:1376: finishing copy_from to smithi16534841-43 2024-09-10T16:16:08.313 INFO:tasks.rados.rados.0.smithi165.stdout:1376: got expected ENOENT (src dne) 2024-09-10T16:16:08.313 INFO:tasks.rados.rados.0.smithi165.stdout:1376: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1378: finishing rollback tid 0 to smithi16534841-37 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1378: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1378: finishing rollback tid 2 to smithi16534841-37 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 577 (ObjNum 468 snap 146 seq_num 468) dirty exists 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1374: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1377: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1376: done (5 left) 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1378: done (4 left) 2024-09-10T16:16:08.314 INFO:tasks.rados.rados.0.smithi165.stdout:1381: snap_remove snap 150 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1381: done (4 left) 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1382: read oid 30 snap 153 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1382: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1383: delete oid 14 current snap is 155 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1374: finishing copy_from to smithi16534841-12 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1377: finishing copy_from to smithi16534841-41 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 605 (ObjNum 404 snap 130 seq_num 404) dirty exists 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1379: finishing write tid 1 to smithi16534841-39 2024-09-10T16:16:09.268 INFO:tasks.rados.rados.0.smithi165.stdout:1379: finishing write tid 2 to smithi16534841-39 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: finishing write tid 3 to smithi16534841-39 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: finishing write tid 4 to smithi16534841-39 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: oid 39 updating version 0 to 891 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: oid 39 updating version 891 to 893 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: oid 39 version 893 is already newer than 892 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 893 (ObjNum 494 snap 155 seq_num 494) dirty exists 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1379: left oid 39 (ObjNum 494 snap 155 seq_num 494) 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1380: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:1380: finishing copy_from to smithi16534841-27 2024-09-10T16:16:09.269 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1059 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1374: done (5 left) 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1377: done (4 left) 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1379: done (3 left) 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1380: done (2 left) 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1383: done (1 left) 2024-09-10T16:16:09.275 INFO:tasks.rados.rados.0.smithi165.stdout:1384: snap_remove snap 146 2024-09-10T16:16:10.273 INFO:tasks.rados.rados.0.smithi165.stdout:1384: done (1 left) 2024-09-10T16:16:10.274 INFO:tasks.rados.rados.0.smithi165.stdout:1385: read oid 15 snap -1 2024-09-10T16:16:10.274 INFO:tasks.rados.rados.0.smithi165.stdout:1385: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:10.274 INFO:tasks.rados.rados.0.smithi165.stdout:1386: copy_from oid 19 from oid 34 current snap is 155 2024-09-10T16:16:10.274 INFO:tasks.rados.rados.0.smithi165.stdout:1382: expect (ObjNum 251 snap 76 seq_num 251) 2024-09-10T16:16:10.274 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:10.383 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:10.437 INFO:tasks.rados.rados.0.smithi165.stdout:1385: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:10.473 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2188445 2024-09-10T16:16:10.648 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3278868 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1386: finishing copy_from racing read to smithi16534841-19 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1386: finishing copy_from to smithi16534841-19 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 574 (ObjNum 481 snap 153 seq_num 481) dirty exists 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1382: done (2 left) 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1385: done (1 left) 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1386: done (0 left) 2024-09-10T16:16:10.653 INFO:tasks.rados.rados.0.smithi165.stdout:1387: copy_from oid 38 from oid 50 current snap is 155 2024-09-10T16:16:10.654 INFO:tasks.rados.rados.0.smithi165.stdout:1388: write oid 28 current snap is 155 2024-09-10T16:16:10.654 INFO:tasks.rados.rados.0.smithi165.stdout:1388: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-10T16:16:10.659 INFO:tasks.rados.rados.0.smithi165.stdout:1388: writing smithi16534841-28 from 736272 to 1222721 tid 1 2024-09-10T16:16:10.666 INFO:tasks.rados.rados.0.smithi165.stdout:1388: writing smithi16534841-28 from 1974951 to 2336272 tid 2 2024-09-10T16:16:10.666 INFO:tasks.rados.rados.0.smithi165.stdout:1389: snap_create 2024-09-10T16:16:10.666 INFO:tasks.rados.rados.0.smithi165.stdout:1387: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:16:10.669 INFO:tasks.rados.rados.0.smithi165.stdout:1387: finishing copy_from to smithi16534841-38 2024-09-10T16:16:10.669 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 580 (ObjNum 417 snap 137 seq_num 417) dirty exists 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: finishing write tid 1 to smithi16534841-28 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: finishing write tid 2 to smithi16534841-28 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: finishing write tid 3 to smithi16534841-28 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: finishing write tid 4 to smithi16534841-28 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: oid 28 updating version 0 to 355 2024-09-10T16:16:10.671 INFO:tasks.rados.rados.0.smithi165.stdout:1388: oid 28 version 355 is already newer than 354 2024-09-10T16:16:10.672 INFO:tasks.rados.rados.0.smithi165.stdout:1388: oid 28 updating version 355 to 356 2024-09-10T16:16:10.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 356 (ObjNum 496 snap 155 seq_num 496) dirty exists 2024-09-10T16:16:10.672 INFO:tasks.rados.rados.0.smithi165.stdout:1388: left oid 28 (ObjNum 496 snap 155 seq_num 496) 2024-09-10T16:16:11.277 INFO:tasks.rados.rados.0.smithi165.stdout:1387: done (2 left) 2024-09-10T16:16:11.278 INFO:tasks.rados.rados.0.smithi165.stdout:1388: done (1 left) 2024-09-10T16:16:11.278 INFO:tasks.rados.rados.0.smithi165.stdout:1389: done (0 left) 2024-09-10T16:16:11.278 INFO:tasks.rados.rados.0.smithi165.stdout:1390: write (excl) oid 34 current snap is 156 2024-09-10T16:16:11.278 INFO:tasks.rados.rados.0.smithi165.stdout:1390: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-10T16:16:11.284 INFO:tasks.rados.rados.0.smithi165.stdout:1390: writing smithi16534841-34 from 753079 to 1314777 tid 1 2024-09-10T16:16:11.290 INFO:tasks.rados.rados.0.smithi165.stdout:1390: writing smithi16534841-34 from 2033433 to 2353079 tid 2 2024-09-10T16:16:11.290 INFO:tasks.rados.rados.0.smithi165.stdout:1391: rollback oid 20 current snap is 156 2024-09-10T16:16:11.290 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 155 2024-09-10T16:16:11.290 INFO:tasks.rados.rados.0.smithi165.stdout:1392: snap_create 2024-09-10T16:16:11.292 INFO:tasks.rados.rados.0.smithi165.stdout:1390: finishing write tid 1 to smithi16534841-34 2024-09-10T16:16:11.292 INFO:tasks.rados.rados.0.smithi165.stdout:1391: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:16:11.292 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 0 (ObjNum 369 snap 0 seq_num 4088875728) dirty dne 2024-09-10T16:16:11.296 INFO:tasks.rados.rados.0.smithi165.stdout:1390: finishing write tid 2 to smithi16534841-34 2024-09-10T16:16:11.296 INFO:tasks.rados.rados.0.smithi165.stdout:1390: finishing write tid 3 to smithi16534841-34 2024-09-10T16:16:11.296 INFO:tasks.rados.rados.0.smithi165.stdout:1390: finishing write tid 4 to smithi16534841-34 2024-09-10T16:16:11.296 INFO:tasks.rados.rados.0.smithi165.stdout:1390: oid 34 updating version 0 to 579 2024-09-10T16:16:11.297 INFO:tasks.rados.rados.0.smithi165.stdout:1390: oid 34 updating version 579 to 580 2024-09-10T16:16:11.297 INFO:tasks.rados.rados.0.smithi165.stdout:1390: oid 34 version 580 is already newer than 578 2024-09-10T16:16:11.297 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 580 (ObjNum 497 snap 156 seq_num 497) dirty exists 2024-09-10T16:16:11.297 INFO:tasks.rados.rados.0.smithi165.stdout:1390: left oid 34 (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:16:12.180 INFO:tasks.rados.rados.0.smithi165.stdout:1390: done (2 left) 2024-09-10T16:16:12.180 INFO:tasks.rados.rados.0.smithi165.stdout:1391: done (1 left) 2024-09-10T16:16:12.180 INFO:tasks.rados.rados.0.smithi165.stdout:1392: done (0 left) 2024-09-10T16:16:12.181 INFO:tasks.rados.rados.0.smithi165.stdout:1393: delete oid 27 current snap is 157 2024-09-10T16:16:12.184 INFO:tasks.rados.rados.0.smithi165.stdout:1393: done (0 left) 2024-09-10T16:16:12.185 INFO:tasks.rados.rados.0.smithi165.stdout:1394: read oid 24 snap -1 2024-09-10T16:16:12.185 INFO:tasks.rados.rados.0.smithi165.stdout:1394: expect (ObjNum 412 snap 136 seq_num 412) 2024-09-10T16:16:12.185 INFO:tasks.rados.rados.0.smithi165.stdout:1395: write (excl) oid 47 current snap is 157 2024-09-10T16:16:12.185 INFO:tasks.rados.rados.0.smithi165.stdout:1395: seq_num 499 ranges {786693=712196,2150397=236296} 2024-09-10T16:16:12.192 INFO:tasks.rados.rados.0.smithi165.stdout:1395: writing smithi16534841-47 from 786693 to 1498889 tid 1 2024-09-10T16:16:12.197 INFO:tasks.rados.rados.0.smithi165.stdout:1395: writing smithi16534841-47 from 2150397 to 2386693 tid 2 2024-09-10T16:16:12.197 INFO:tasks.rados.rados.0.smithi165.stdout:1396: snap_remove snap 156 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:1396: done (2 left) 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:1397: read oid 38 snap 155 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:1397: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:1398: rollback oid 23 current snap is 157 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 153 2024-09-10T16:16:13.179 INFO:tasks.rados.rados.0.smithi165.stdout:1394: expect (ObjNum 412 snap 136 seq_num 412) 2024-09-10T16:16:13.242 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1165438 2024-09-10T16:16:13.279 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: finishing write tid 1 to smithi16534841-47 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: finishing write tid 2 to smithi16534841-47 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: finishing write tid 3 to smithi16534841-47 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: finishing write tid 4 to smithi16534841-47 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: oid 47 updating version 0 to 1063 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: oid 47 version 1063 is already newer than 1062 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: oid 47 version 1063 is already newer than 1061 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1063 (ObjNum 499 snap 157 seq_num 499) dirty exists 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1395: left oid 47 (ObjNum 499 snap 157 seq_num 499) 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1398: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:16:13.337 INFO:tasks.rados.rados.0.smithi165.stdout:1394: done (3 left) 2024-09-10T16:16:13.338 INFO:tasks.rados.rados.0.smithi165.stdout:1395: done (2 left) 2024-09-10T16:16:13.338 INFO:tasks.rados.rados.0.smithi165.stdout:1399: snap_create 2024-09-10T16:16:13.338 INFO:tasks.rados.rados.0.smithi165.stdout:1398: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:16:13.338 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 895 (ObjNum 179 snap 54 seq_num 179) dirty exists 2024-09-10T16:16:13.338 INFO:tasks.rados.rados.0.smithi165.stdout:1397: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:13.412 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:14.181 INFO:tasks.rados.rados.0.smithi165.stdout:1397: done (2 left) 2024-09-10T16:16:14.181 INFO:tasks.rados.rados.0.smithi165.stdout:1398: done (1 left) 2024-09-10T16:16:14.181 INFO:tasks.rados.rados.0.smithi165.stdout:1399: done (0 left) 2024-09-10T16:16:14.182 INFO:tasks.rados.rados.0.smithi165.stdout:1400: read oid 26 snap -1 2024-09-10T16:16:14.182 INFO:tasks.rados.rados.0.smithi165.stdout:1400: expect (ObjNum 462 snap 144 seq_num 462) 2024-09-10T16:16:14.182 INFO:tasks.rados.rados.0.smithi165.stdout:1401: snap_remove snap 153 2024-09-10T16:16:15.188 INFO:tasks.rados.rados.0.smithi165.stdout:1401: done (1 left) 2024-09-10T16:16:15.188 INFO:tasks.rados.rados.0.smithi165.stdout:1402: write (excl) oid 23 current snap is 158 2024-09-10T16:16:15.188 INFO:tasks.rados.rados.0.smithi165.stdout:1402: seq_num 500 ranges {403500=787445,1692526=473837} 2024-09-10T16:16:15.194 INFO:tasks.rados.rados.0.smithi165.stdout:1402: writing smithi16534841-23 from 403500 to 1190945 tid 1 2024-09-10T16:16:15.200 INFO:tasks.rados.rados.0.smithi165.stdout:1402: writing smithi16534841-23 from 1692526 to 2166363 tid 2 2024-09-10T16:16:15.200 INFO:tasks.rados.rados.0.smithi165.stdout:1403: write oid 20 current snap is 158 2024-09-10T16:16:15.200 INFO:tasks.rados.rados.0.smithi165.stdout:1403: seq_num 501 ranges {420307=462694,1351008=617495} 2024-09-10T16:16:15.205 INFO:tasks.rados.rados.0.smithi165.stdout:1403: writing smithi16534841-20 from 420307 to 883001 tid 1 2024-09-10T16:16:15.211 INFO:tasks.rados.rados.0.smithi165.stdout:1403: writing smithi16534841-20 from 1351008 to 1968503 tid 2 2024-09-10T16:16:15.211 INFO:tasks.rados.rados.0.smithi165.stdout:1404: copy_from oid 38 from oid 2 current snap is 158 2024-09-10T16:16:15.211 INFO:tasks.rados.rados.0.smithi165.stdout:1400: expect (ObjNum 462 snap 144 seq_num 462) 2024-09-10T16:16:15.211 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:15.288 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:15.441 INFO:tasks.rados.rados.0.smithi165.stdout:1402: finishing write tid 1 to smithi16534841-23 2024-09-10T16:16:15.441 INFO:tasks.rados.rados.0.smithi165.stdout:1402: finishing write tid 2 to smithi16534841-23 2024-09-10T16:16:15.441 INFO:tasks.rados.rados.0.smithi165.stdout:1402: finishing write tid 3 to smithi16534841-23 2024-09-10T16:16:15.442 INFO:tasks.rados.rados.0.smithi165.stdout:1402: finishing write tid 4 to smithi16534841-23 2024-09-10T16:16:15.442 INFO:tasks.rados.rados.0.smithi165.stdout:1402: oid 23 updating version 0 to 896 2024-09-10T16:16:15.442 INFO:tasks.rados.rados.0.smithi165.stdout:1402: oid 23 updating version 896 to 897 2024-09-10T16:16:15.442 INFO:tasks.rados.rados.0.smithi165.stdout:1402: oid 23 updating version 897 to 898 2024-09-10T16:16:15.442 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 898 (ObjNum 500 snap 158 seq_num 500) dirty exists 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1402: left oid 23 (ObjNum 500 snap 158 seq_num 500) 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1403: finishing write tid 1 to smithi16534841-20 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1403: finishing write tid 2 to smithi16534841-20 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1403: finishing write tid 3 to smithi16534841-20 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1403: finishing write tid 4 to smithi16534841-20 2024-09-10T16:16:15.443 INFO:tasks.rados.rados.0.smithi165.stdout:1403: oid 20 updating version 0 to 898 2024-09-10T16:16:15.444 INFO:tasks.rados.rados.0.smithi165.stdout:1403: oid 20 version 898 is already newer than 897 2024-09-10T16:16:15.444 INFO:tasks.rados.rados.0.smithi165.stdout:1403: oid 20 version 898 is already newer than 896 2024-09-10T16:16:15.444 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 898 (ObjNum 501 snap 158 seq_num 501) dirty exists 2024-09-10T16:16:15.444 INFO:tasks.rados.rados.0.smithi165.stdout:1403: left oid 20 (ObjNum 501 snap 158 seq_num 501) 2024-09-10T16:16:15.444 INFO:tasks.rados.rados.0.smithi165.stdout:1404: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1404: finishing copy_from to smithi16534841-38 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 886 (ObjNum 483 snap 153 seq_num 483) dirty exists 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1400: done (3 left) 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1402: done (2 left) 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1403: done (1 left) 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1404: done (0 left) 2024-09-10T16:16:15.445 INFO:tasks.rados.rados.0.smithi165.stdout:1405: read oid 38 snap 155 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:1405: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:1406: copy_from oid 23 from oid 48 current snap is 158 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:1407: delete oid 30 current snap is 158 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:1407: done (2 left) 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:1408: rollback oid 36 current snap is 158 2024-09-10T16:16:15.446 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 157 2024-09-10T16:16:15.447 INFO:tasks.rados.rados.0.smithi165.stdout:1409: snap_create 2024-09-10T16:16:15.448 INFO:tasks.rados.rados.0.smithi165.stdout:1405: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:15.474 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2715941 2024-09-10T16:16:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:1408: finishing rollback tid 0 to smithi16534841-36 2024-09-10T16:16:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:1408: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:16:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:1408: finishing rollback tid 2 to smithi16534841-36 2024-09-10T16:16:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 889 (ObjNum 404 snap 130 seq_num 404) dirty exists 2024-09-10T16:16:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:1406: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:16:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:1406: finishing copy_from to smithi16534841-23 2024-09-10T16:16:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1059 (ObjNum 445 snap 143 seq_num 445) dirty exists 2024-09-10T16:16:16.192 INFO:tasks.rados.rados.0.smithi165.stdout:1405: done (3 left) 2024-09-10T16:16:16.193 INFO:tasks.rados.rados.0.smithi165.stdout:1406: done (2 left) 2024-09-10T16:16:16.193 INFO:tasks.rados.rados.0.smithi165.stdout:1408: done (1 left) 2024-09-10T16:16:16.193 INFO:tasks.rados.rados.0.smithi165.stdout:1409: done (0 left) 2024-09-10T16:16:16.193 INFO:tasks.rados.rados.0.smithi165.stdout:1410: delete oid 35 current snap is 159 2024-09-10T16:16:16.197 INFO:tasks.rados.rados.0.smithi165.stdout:1410: done (0 left) 2024-09-10T16:16:16.197 INFO:tasks.rados.rados.0.smithi165.stdout:1411: write (excl) oid 28 current snap is 159 2024-09-10T16:16:16.197 INFO:tasks.rados.rados.0.smithi165.stdout:1411: seq_num 504 ranges {470728=404794,1526454=481175} 2024-09-10T16:16:16.201 INFO:tasks.rados.rados.0.smithi165.stdout:1411: writing smithi16534841-28 from 470728 to 875522 tid 1 2024-09-10T16:16:16.207 INFO:tasks.rados.rados.0.smithi165.stdout:1411: writing smithi16534841-28 from 1526454 to 2007629 tid 2 2024-09-10T16:16:16.208 INFO:tasks.rados.rados.0.smithi165.stdout:1412: snap_remove snap 158 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1412: done (1 left) 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1413: rollback oid 42 current snap is 159 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 154 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: finishing write tid 1 to smithi16534841-28 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: finishing write tid 2 to smithi16534841-28 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: finishing write tid 3 to smithi16534841-28 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: finishing write tid 4 to smithi16534841-28 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: oid 28 updating version 0 to 357 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: oid 28 updating version 357 to 358 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:1411: oid 28 updating version 358 to 359 2024-09-10T16:16:17.179 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 359 (ObjNum 504 snap 159 seq_num 504) dirty exists 2024-09-10T16:16:17.180 INFO:tasks.rados.rados.0.smithi165.stdout:1411: left oid 28 (ObjNum 504 snap 159 seq_num 504) 2024-09-10T16:16:17.180 INFO:tasks.rados.rados.0.smithi165.stdout:1411: done (1 left) 2024-09-10T16:16:17.180 INFO:tasks.rados.rados.0.smithi165.stdout:1414: delete oid 30 current snap is 159 2024-09-10T16:16:17.181 INFO:tasks.rados.rados.0.smithi165.stdout:1414: done (1 left) 2024-09-10T16:16:17.181 INFO:tasks.rados.rados.0.smithi165.stdout:1415: read oid 18 snap -1 2024-09-10T16:16:17.182 INFO:tasks.rados.rados.0.smithi165.stdout:1415: expect deleted 2024-09-10T16:16:17.182 INFO:tasks.rados.rados.0.smithi165.stdout:1416: read oid 31 snap -1 2024-09-10T16:16:17.182 INFO:tasks.rados.rados.0.smithi165.stdout:1416: expect deleted 2024-09-10T16:16:17.182 INFO:tasks.rados.rados.0.smithi165.stdout:1417: delete oid 43 current snap is 159 2024-09-10T16:16:17.187 INFO:tasks.rados.rados.0.smithi165.stdout:1413: finishing rollback tid 0 to smithi16534841-42 2024-09-10T16:16:17.189 INFO:tasks.rados.rados.0.smithi165.stdout:1413: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:16:17.189 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 588 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:16:17.195 INFO:tasks.rados.rados.0.smithi165.stdout:1413: done (3 left) 2024-09-10T16:16:17.196 INFO:tasks.rados.rados.0.smithi165.stdout:1415: done (2 left) 2024-09-10T16:16:17.196 INFO:tasks.rados.rados.0.smithi165.stdout:1416: done (1 left) 2024-09-10T16:16:17.196 INFO:tasks.rados.rados.0.smithi165.stdout:1417: done (0 left) 2024-09-10T16:16:17.196 INFO:tasks.rados.rados.0.smithi165.stdout:1418: delete oid 7 current snap is 159 2024-09-10T16:16:17.200 INFO:tasks.rados.rados.0.smithi165.stdout:1418: done (0 left) 2024-09-10T16:16:17.200 INFO:tasks.rados.rados.0.smithi165.stdout:1419: write (excl) oid 31 current snap is 159 2024-09-10T16:16:17.200 INFO:tasks.rados.rados.0.smithi165.stdout:1419: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-10T16:16:17.207 INFO:tasks.rados.rados.0.smithi165.stdout:1419: writing smithi16534841-31 from 537956 to 1243746 tid 1 2024-09-10T16:16:17.214 INFO:tasks.rados.rados.0.smithi165.stdout:1419: writing smithi16534841-31 from 1644029 to 2416189 tid 2 2024-09-10T16:16:17.214 INFO:tasks.rados.rados.0.smithi165.stdout:1420: write (excl) oid 8 current snap is 159 2024-09-10T16:16:17.214 INFO:tasks.rados.rados.0.smithi165.stdout:1420: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-10T16:16:17.221 INFO:tasks.rados.rados.0.smithi165.stdout:1420: writing smithi16534841-8 from 554763 to 1335802 tid 1 2024-09-10T16:16:17.228 INFO:tasks.rados.rados.0.smithi165.stdout:1420: writing smithi16534841-8 from 2102511 to 2554763 tid 2 2024-09-10T16:16:17.228 INFO:tasks.rados.rados.0.smithi165.stdout:1421: read oid 36 snap -1 2024-09-10T16:16:17.228 INFO:tasks.rados.rados.0.smithi165.stdout:1421: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:17.228 INFO:tasks.rados.rados.0.smithi165.stdout:1422: write (excl) oid 14 current snap is 159 2024-09-10T16:16:17.228 INFO:tasks.rados.rados.0.smithi165.stdout:1419: finishing write tid 1 to smithi16534841-31 2024-09-10T16:16:17.229 INFO:tasks.rados.rados.0.smithi165.stdout:1422: seq_num 510 ranges {571570=572641,1877346=694224} 2024-09-10T16:16:17.234 INFO:tasks.rados.rados.0.smithi165.stdout:1422: writing smithi16534841-14 from 571570 to 1144211 tid 1 2024-09-10T16:16:17.242 INFO:tasks.rados.rados.0.smithi165.stdout:1422: writing smithi16534841-14 from 1877346 to 2571570 tid 2 2024-09-10T16:16:17.242 INFO:tasks.rados.rados.0.smithi165.stdout:1423: write oid 11 current snap is 159 2024-09-10T16:16:17.243 INFO:tasks.rados.rados.0.smithi165.stdout:1419: finishing write tid 2 to smithi16534841-31 2024-09-10T16:16:17.243 INFO:tasks.rados.rados.0.smithi165.stdout:1419: finishing write tid 3 to smithi16534841-31 2024-09-10T16:16:17.243 INFO:tasks.rados.rados.0.smithi165.stdout:1423: seq_num 511 ranges {588377=647890,1935828=635840} 2024-09-10T16:16:17.249 INFO:tasks.rados.rados.0.smithi165.stdout:1423: writing smithi16534841-11 from 588377 to 1236267 tid 1 2024-09-10T16:16:17.257 INFO:tasks.rados.rados.0.smithi165.stdout:1423: writing smithi16534841-11 from 1935828 to 2571668 tid 2 2024-09-10T16:16:17.257 INFO:tasks.rados.rados.0.smithi165.stdout:1424: read oid 17 snap -1 2024-09-10T16:16:17.257 INFO:tasks.rados.rados.0.smithi165.stdout:1424: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:16:17.257 INFO:tasks.rados.rados.0.smithi165.stdout:1425: snap_remove snap 155 2024-09-10T16:16:18.304 INFO:tasks.rados.rados.0.smithi165.stdout:1425: done (6 left) 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1426: write oid 9 current snap is 159 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1419: finishing write tid 4 to smithi16534841-31 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1419: oid 31 updating version 0 to 1090 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1419: oid 31 updating version 1090 to 1091 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1419: oid 31 version 1091 is already newer than 1089 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1091 (ObjNum 508 snap 159 seq_num 508) dirty exists 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1419: left oid 31 (ObjNum 508 snap 159 seq_num 508) 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1420: finishing write tid 1 to smithi16534841-8 2024-09-10T16:16:18.305 INFO:tasks.rados.rados.0.smithi165.stdout:1426: seq_num 512 ranges {605184=723139,1877957=727227} 2024-09-10T16:16:18.311 INFO:tasks.rados.rados.0.smithi165.stdout:1426: writing smithi16534841-9 from 605184 to 1328323 tid 1 2024-09-10T16:16:18.319 INFO:tasks.rados.rados.0.smithi165.stdout:1426: writing smithi16534841-9 from 1877957 to 2605184 tid 2 2024-09-10T16:16:18.319 INFO:tasks.rados.rados.0.smithi165.stdout:1419: done (6 left) 2024-09-10T16:16:18.319 INFO:tasks.rados.rados.0.smithi165.stdout:1427: write (excl) oid 22 current snap is 159 2024-09-10T16:16:18.319 INFO:tasks.rados.rados.0.smithi165.stdout:1427: seq_num 513 ranges {621991=798388,1936439=639509} 2024-09-10T16:16:18.327 INFO:tasks.rados.rados.0.smithi165.stdout:1427: writing smithi16534841-22 from 621991 to 1420379 tid 1 2024-09-10T16:16:18.333 INFO:tasks.rados.rados.0.smithi165.stdout:1427: writing smithi16534841-22 from 1936439 to 2575948 tid 2 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1428: write oid 21 current snap is 159 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: finishing write tid 2 to smithi16534841-8 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: finishing write tid 3 to smithi16534841-8 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: finishing write tid 4 to smithi16534841-8 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: oid 8 updating version 0 to 1065 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: oid 8 updating version 1065 to 1066 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: oid 8 version 1066 is already newer than 1064 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1066 (ObjNum 509 snap 159 seq_num 509) dirty exists 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1420: left oid 8 (ObjNum 509 snap 159 seq_num 509) 2024-09-10T16:16:18.334 INFO:tasks.rados.rados.0.smithi165.stdout:1428: seq_num 514 ranges {638798=473637,1594921=783167} 2024-09-10T16:16:18.339 INFO:tasks.rados.rados.0.smithi165.stdout:1428: writing smithi16534841-21 from 638798 to 1112435 tid 1 2024-09-10T16:16:18.347 INFO:tasks.rados.rados.0.smithi165.stdout:1428: writing smithi16534841-21 from 1594921 to 2378088 tid 2 2024-09-10T16:16:18.347 INFO:tasks.rados.rados.0.smithi165.stdout:1420: done (7 left) 2024-09-10T16:16:18.347 INFO:tasks.rados.rados.0.smithi165.stdout:1429: delete oid 48 current snap is 159 2024-09-10T16:16:18.347 INFO:tasks.rados.rados.0.smithi165.stdout:1421: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:18.378 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:18.413 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1398923 2024-09-10T16:16:18.432 INFO:tasks.rados.rados.0.smithi165.stdout:1422: finishing write tid 1 to smithi16534841-14 2024-09-10T16:16:18.432 INFO:tasks.rados.rados.0.smithi165.stdout:1422: finishing write tid 2 to smithi16534841-14 2024-09-10T16:16:18.432 INFO:tasks.rados.rados.0.smithi165.stdout:1422: finishing write tid 3 to smithi16534841-14 2024-09-10T16:16:18.432 INFO:tasks.rados.rados.0.smithi165.stdout:1422: finishing write tid 4 to smithi16534841-14 2024-09-10T16:16:18.432 INFO:tasks.rados.rados.0.smithi165.stdout:1422: oid 14 updating version 0 to 349 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:1422: oid 14 version 349 is already newer than 348 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:1422: oid 14 version 349 is already newer than 347 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 349 (ObjNum 510 snap 159 seq_num 510) dirty exists 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:1422: left oid 14 (ObjNum 510 snap 159 seq_num 510) 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:1423: finishing write tid 1 to smithi16534841-11 2024-09-10T16:16:18.433 INFO:tasks.rados.rados.0.smithi165.stdout:1423: finishing write tid 2 to smithi16534841-11 2024-09-10T16:16:18.434 INFO:tasks.rados.rados.0.smithi165.stdout:1423: finishing write tid 3 to smithi16534841-11 2024-09-10T16:16:18.434 INFO:tasks.rados.rados.0.smithi165.stdout:1423: finishing write tid 4 to smithi16534841-11 2024-09-10T16:16:18.434 INFO:tasks.rados.rados.0.smithi165.stdout:1423: oid 11 updating version 0 to 590 2024-09-10T16:16:18.434 INFO:tasks.rados.rados.0.smithi165.stdout:1423: oid 11 updating version 590 to 591 2024-09-10T16:16:18.434 INFO:tasks.rados.rados.0.smithi165.stdout:1423: oid 11 version 591 is already newer than 589 2024-09-10T16:16:18.435 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 591 (ObjNum 511 snap 159 seq_num 511) dirty exists 2024-09-10T16:16:18.435 INFO:tasks.rados.rados.0.smithi165.stdout:1423: left oid 11 (ObjNum 511 snap 159 seq_num 511) 2024-09-10T16:16:18.435 INFO:tasks.rados.rados.0.smithi165.stdout:1424: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:16:18.558 INFO:tasks.rados.rados.0.smithi165.stdout:1426: finishing write tid 1 to smithi16534841-9 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: finishing write tid 2 to smithi16534841-9 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: finishing write tid 3 to smithi16534841-9 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: finishing write tid 4 to smithi16534841-9 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: oid 9 updating version 0 to 350 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: oid 9 updating version 350 to 352 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: oid 9 version 352 is already newer than 351 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 352 (ObjNum 512 snap 159 seq_num 512) dirty exists 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1426: left oid 9 (ObjNum 512 snap 159 seq_num 512) 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1427: finishing write tid 1 to smithi16534841-22 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1427: finishing write tid 2 to smithi16534841-22 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1427: finishing write tid 3 to smithi16534841-22 2024-09-10T16:16:18.559 INFO:tasks.rados.rados.0.smithi165.stdout:1427: finishing write tid 4 to smithi16534841-22 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1427: oid 22 updating version 0 to 744 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1427: oid 22 updating version 744 to 746 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1427: oid 22 version 746 is already newer than 745 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 746 (ObjNum 513 snap 159 seq_num 513) dirty exists 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1427: left oid 22 (ObjNum 513 snap 159 seq_num 513) 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1428: finishing write tid 1 to smithi16534841-21 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1428: finishing write tid 2 to smithi16534841-21 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1428: finishing write tid 3 to smithi16534841-21 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1421: done (7 left) 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1422: done (6 left) 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1423: done (5 left) 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1424: done (4 left) 2024-09-10T16:16:18.560 INFO:tasks.rados.rados.0.smithi165.stdout:1426: done (3 left) 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1427: done (2 left) 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1429: done (1 left) 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1430: write (excl) oid 39 current snap is 159 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1428: finishing write tid 4 to smithi16534841-21 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1428: oid 21 updating version 0 to 900 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1428: oid 21 updating version 900 to 901 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1428: oid 21 version 901 is already newer than 899 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 901 (ObjNum 514 snap 159 seq_num 514) dirty exists 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1428: left oid 21 (ObjNum 514 snap 159 seq_num 514) 2024-09-10T16:16:18.561 INFO:tasks.rados.rados.0.smithi165.stdout:1430: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-10T16:16:18.562 INFO:tasks.rados.rados.0.smithi165.stdout:1430: writing smithi16534841-39 from 672412 to 1296547 tid 1 2024-09-10T16:16:18.568 INFO:tasks.rados.rados.0.smithi165.stdout:1430: writing smithi16534841-39 from 1995532 to 2672412 tid 2 2024-09-10T16:16:18.568 INFO:tasks.rados.rados.0.smithi165.stdout:1428: done (1 left) 2024-09-10T16:16:18.568 INFO:tasks.rados.rados.0.smithi165.stdout:1431: snap_remove snap 157 2024-09-10T16:16:19.344 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:16:19.344 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:16:19.345 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:16:19.347 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:16:19.347 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:16:19.348 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.451+0000 7feab9ebd640 10 monclient: get_monmap_and_config 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.451+0000 7feab9ebd640 10 monclient: build_initial_monmap 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.451+0000 7feab9ebd640 10 monclient: monmap: 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:19.452194+0000 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:19.452194+0000 2024-09-10T16:16:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:19.454 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 Processor -- start 2024-09-10T16:16:19.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 -- start start 2024-09-10T16:16:19.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient: init 2024-09-10T16:16:19.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-b con 0x7feab406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7feab406d540 0x7feab40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-c con 0x7feab406d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feab40a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab409cb90 con 0x7feab40a4c20 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:19.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7feab409ce70 con 0x7feab406d540 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feab409d150 con 0x7feab406d160 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.452+0000 7feab9ebd640 10 monclient(hunting): authenticate will time out at 1357.939900s 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39548/0 (socket says 172.21.15.164:39548) 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 -- 172.21.15.164:0/1959015816 learned_addr learned my addr 172.21.15.164:0/1959015816 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:19.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 10 monclient(hunting): get_auth_request con 0x7feab406d160 auth_method 0 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 -- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 msgr2=0x7feab4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 -- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 msgr2=0x7feab4134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feab37fe640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 --2- 172.21.15.164:0/1959015816 >> v2:172.21.15.165:3300/0 conn(0x7feab406d540 0x7feab40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): get_auth_request con 0x7feab406d540 auth_method 0 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:19.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient(hunting): handle_auth_done global_id 94255 payload 274 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 -- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 msgr2=0x7feab4134e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 -- 172.21.15.164:0/1959015816 >> v1:172.21.15.164:6789/0 conn(0x7feab40a4c20 legacy=0x7feab4137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient: _finish_hunting 0 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 monclient: found mon.noname-c 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:16:19.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.453+0000 7feaabfff640 1 -- 172.21.15.164:0/1959015816 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab4135340 con 0x7feab406d540 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.456+0000 7feaabfff640 1 -- 172.21.15.164:0/1959015816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 msgr2=0x7feab40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.456+0000 7feaabfff640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab40a46e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0097a0 tx=0x7fea9c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=528c9bb3d053794e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 1 -- 172.21.15.164:0/1959015816 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c037070 con 0x7feab406d540 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:19.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: dump: 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:19.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 1 -- 172.21.15.164:0/1959015816 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea9c032440 con 0x7feab406d540 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 5 monclient: authenticate success, global_id 94255 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 1 -- 172.21.15.164:0/1959015816 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea9c032810 con 0x7feab406d540 2024-09-10T16:16:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 4 set_mon_vals no callback set 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab2ffd640 10 monclient: dump: 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 10 monclient: shutdown 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 -- 172.21.15.164:0/1959015816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 msgr2=0x7feab40a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab40a46e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fea9c0097a0 tx=0x7fea9c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 -- 172.21.15.164:0/1959015816 shutdown_connections 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab40a46e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 --2- 172.21.15.164:0/1959015816 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab406d160 0x7feab4134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 -- 172.21.15.164:0/1959015816 >> 172.21.15.164:0/1959015816 conn(0x7feab40aacb0 msgr2=0x7feab40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 -- 172.21.15.164:0/1959015816 shutdown_connections 2024-09-10T16:16:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.457+0000 7feab9ebd640 1 -- 172.21.15.164:0/1959015816 wait complete. 2024-09-10T16:16:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.458+0000 7feab9ebd640 10 monclient: build_initial_monmap 2024-09-10T16:16:19.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.458+0000 7feab9ebd640 10 monclient: monmap: 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:19.474 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.458+0000 7feab9ebd640 1 Processor -- start 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.458+0000 7feab9ebd640 1 -- start start 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.458+0000 7feab9ebd640 10 monclient: init 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-a con 0x7feab406d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab409e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-b con 0x7feab406d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): picked mon.noname-c con 0x7feab40a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feab4138830 con 0x7feab406d160 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feab41386b0 con 0x7feab406d540 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feab41389b0 con 0x7feab40a4c20 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab9ebd640 10 monclient(hunting): authenticate will time out at 1357.946201s 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44394/0 (socket says 172.21.15.164:44394) 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 -- 172.21.15.164:0/3672410774 learned_addr learned my addr 172.21.15.164:0/3672410774 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): get_auth_request con 0x7feab406d160 auth_method 0 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 10 monclient(hunting): get_auth_request con 0x7feab40a4c20 auth_method 0 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:19.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 msgr2=0x7feab41cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 msgr2=0x7feab41cd110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab3fff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feaabfff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab409e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feaabfff640 10 monclient(hunting): get_auth_request con 0x7feab406d540 auth_method 0 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feaabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:19.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient(hunting): handle_auth_done global_id 74478 payload 824 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 msgr2=0x7feab41cd110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 msgr2=0x7feab409e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab409e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 monclient: found mon.noname-a 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:19.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.459+0000 7feab37fe640 1 -- 172.21.15.164:0/3672410774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab40a0100 con 0x7feab406d160 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab37fe640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 msgr2=0x7feab409db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab37fe640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feaa400e810 tx=0x7feaa400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4017070 con 0x7feab406d160 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:19.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: dump: 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:19.481 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa4004020 con 0x7feab406d160 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab9ebd640 5 monclient: authenticate success, global_id 74478 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa400f040 con 0x7feab406d160 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab0ff9640 10 monclient: dump: 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:19.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab9ebd640 10 monclient: _renew_subs 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab9ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab409ebe0 con 0x7feab406d160 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.461+0000 7feab27fc640 4 set_mon_vals no callback set 2024-09-10T16:16:19.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab9ebd640 10 monclient: _renew_subs 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab9ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab40a4900 con 0x7feab406d160 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.462+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feaa400c8e0 con 0x7feab406d160 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab9ebd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab9ebd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab9ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab4134e00 con 0x7feab406d160 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab0ff9640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 0x7fea94040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(346..346 src has 184..346) ==== 4783+0+0 (secure 0 0 0) 0x7feaa4052a30 con 0x7feab406d160 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feaabfff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 0x7fea94040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.463+0000 7feaabfff640 10 monclient: get_auth_request con 0x7fea9403e240 auth_method 0 2024-09-10T16:16:19.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.466+0000 7feaabfff640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 0x7fea94040700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fea9c031000 tx=0x7fea9c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:19.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.467+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7feaa4004450 con 0x7feab406d160 2024-09-10T16:16:19.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.467+0000 7feab0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:19.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.467+0000 7feab0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:19.587 INFO:tasks.rados.rados.0.smithi165.stdout:1431: done (1 left) 2024-09-10T16:16:19.587 INFO:tasks.rados.rados.0.smithi165.stdout:1432: snap_remove snap 154 2024-09-10T16:16:19.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.709+0000 7feab9ebd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:19.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.709+0000 7feab9ebd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:19.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.709+0000 7feab9ebd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:19.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.709+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feab409fb40 con 0x7feab406d160 2024-09-10T16:16:19.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.710+0000 7feab0ff9640 1 -- 172.21.15.164:0/3672410774 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7feaa4051430 con 0x7feab406d160 2024-09-10T16:16:19.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.710+0000 7feab0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:19.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.710+0000 7feab0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:19.712 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":90,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 msgr2=0x7fea94040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 0x7fea94040700 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fea9c031000 tx=0x7fea9c02e000 comp rx=0 tx=0).stop 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 10 monclient: shutdown 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 msgr2=0x7feab409db80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7feaa400e810 tx=0x7feaa400ecd0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 shutdown_connections 2024-09-10T16:16:19.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fea9403e240 0x7fea94040700 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feab40a4c20 0x7feab41cd110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feab406d540 0x7feab409e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 --2- 172.21.15.164:0/3672410774 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feab406d160 0x7feab409db80 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 >> 172.21.15.164:0/3672410774 conn(0x7feab40aacb0 msgr2=0x7feab4132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 shutdown_connections 2024-09-10T16:16:19.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:19.712+0000 7feab9ebd640 1 -- 172.21.15.164:0/3672410774 wait complete. 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1432: done (1 left) 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1433: delete oid 31 current snap is 159 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1430: finishing write tid 1 to smithi16534841-39 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1430: finishing write tid 2 to smithi16534841-39 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1430: finishing write tid 3 to smithi16534841-39 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1430: finishing write tid 4 to smithi16534841-39 2024-09-10T16:16:20.314 INFO:tasks.rados.rados.0.smithi165.stdout:1430: oid 39 updating version 0 to 1062 2024-09-10T16:16:20.315 INFO:tasks.rados.rados.0.smithi165.stdout:1430: oid 39 version 1062 is already newer than 1061 2024-09-10T16:16:20.315 INFO:tasks.rados.rados.0.smithi165.stdout:1430: oid 39 version 1062 is already newer than 1060 2024-09-10T16:16:20.315 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1062 (ObjNum 516 snap 159 seq_num 516) dirty exists 2024-09-10T16:16:20.315 INFO:tasks.rados.rados.0.smithi165.stdout:1430: left oid 39 (ObjNum 516 snap 159 seq_num 516) 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1430: done (1 left) 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1433: done (0 left) 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1434: read oid 21 snap -1 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1434: expect (ObjNum 514 snap 159 seq_num 514) 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1435: write oid 39 current snap is 159 2024-09-10T16:16:20.323 INFO:tasks.rados.rados.0.smithi165.stdout:1435: seq_num 518 ranges {706026=490986,1828849=790505} 2024-09-10T16:16:20.329 INFO:tasks.rados.rados.0.smithi165.stdout:1435: writing smithi16534841-39 from 706026 to 1197012 tid 1 2024-09-10T16:16:20.338 INFO:tasks.rados.rados.0.smithi165.stdout:1435: writing smithi16534841-39 from 1828849 to 2619354 tid 2 2024-09-10T16:16:20.338 INFO:tasks.rados.rados.0.smithi165.stdout:1436: read oid 6 snap -1 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1436: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1437: read oid 15 snap -1 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1437: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1438: read oid 49 snap -1 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1438: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1439: write oid 2 current snap is 159 2024-09-10T16:16:20.339 INFO:tasks.rados.rados.0.smithi165.stdout:1439: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-10T16:16:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1439: writing smithi16534841-2 from 722833 to 1289068 tid 1 2024-09-10T16:16:20.354 INFO:tasks.rados.rados.0.smithi165.stdout:1439: writing smithi16534841-2 from 1887331 to 2537847 tid 2 2024-09-10T16:16:20.354 INFO:tasks.rados.rados.0.smithi165.stdout:1440: copy_from oid 33 from oid 13 current snap is 159 2024-09-10T16:16:20.354 INFO:tasks.rados.rados.0.smithi165.stdout:1434: expect (ObjNum 514 snap 159 seq_num 514) 2024-09-10T16:16:20.384 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:20.414 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 567663 2024-09-10T16:16:20.439 INFO:tasks.rados.rados.0.smithi165.stdout:1435: finishing write tid 1 to smithi16534841-39 2024-09-10T16:16:20.439 INFO:tasks.rados.rados.0.smithi165.stdout:1436: expect (ObjNum 404 snap 130 seq_num 404) 2024-09-10T16:16:20.452 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2487878 2024-09-10T16:16:20.509 INFO:tasks.rados.rados.0.smithi165.stdout:1435: finishing write tid 2 to smithi16534841-39 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: finishing write tid 3 to smithi16534841-39 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: finishing write tid 4 to smithi16534841-39 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: oid 39 updating version 0 to 1064 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: oid 39 updating version 1064 to 1065 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: oid 39 version 1065 is already newer than 1063 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1065 (ObjNum 518 snap 159 seq_num 518) dirty exists 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1435: left oid 39 (ObjNum 518 snap 159 seq_num 518) 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:1437: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:20.510 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:20.727 INFO:tasks.rados.rados.0.smithi165.stdout:1439: finishing write tid 1 to smithi16534841-2 2024-09-10T16:16:20.727 INFO:tasks.rados.rados.0.smithi165.stdout:1438: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:20.754 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3243229 2024-09-10T16:16:20.777 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 806083 2024-09-10T16:16:20.825 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:20.889 INFO:tasks.rados.rados.0.smithi165.stdout:1439: finishing write tid 2 to smithi16534841-2 2024-09-10T16:16:20.889 INFO:tasks.rados.rados.0.smithi165.stdout:1439: finishing write tid 3 to smithi16534841-2 2024-09-10T16:16:20.889 INFO:tasks.rados.rados.0.smithi165.stdout:1439: finishing write tid 4 to smithi16534841-2 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1439: oid 2 updating version 0 to 1068 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1439: oid 2 version 1068 is already newer than 1067 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1439: oid 2 updating version 1068 to 1069 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1069 (ObjNum 519 snap 159 seq_num 519) dirty exists 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1439: left oid 2 (ObjNum 519 snap 159 seq_num 519) 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1440: finishing copy_from racing read to smithi16534841-33 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1440: finishing copy_from to smithi16534841-33 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 742 (ObjNum 491 snap 155 seq_num 491) dirty exists 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1434: done (6 left) 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1435: done (5 left) 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1436: done (4 left) 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1437: done (3 left) 2024-09-10T16:16:20.890 INFO:tasks.rados.rados.0.smithi165.stdout:1438: done (2 left) 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1439: done (1 left) 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1440: done (0 left) 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1441: read oid 16 snap -1 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1441: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1442: read oid 27 snap -1 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1442: expect deleted 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1443: write oid 6 current snap is 159 2024-09-10T16:16:20.891 INFO:tasks.rados.rados.0.smithi165.stdout:1443: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-10T16:16:20.897 INFO:tasks.rados.rados.0.smithi165.stdout:1443: writing smithi16534841-6 from 739640 to 1381124 tid 1 2024-09-10T16:16:20.904 INFO:tasks.rados.rados.0.smithi165.stdout:1443: writing smithi16534841-6 from 1829460 to 2623634 tid 2 2024-09-10T16:16:20.904 INFO:tasks.rados.rados.0.smithi165.stdout:1444: rollback oid 5 current snap is 159 2024-09-10T16:16:20.904 INFO:tasks.rados.rados.0.smithi165.stdout:1444: done (3 left) 2024-09-10T16:16:20.904 INFO:tasks.rados.rados.0.smithi165.stdout:1445: delete oid 11 current snap is 159 2024-09-10T16:16:20.905 INFO:tasks.rados.rados.0.smithi165.stdout:1441: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1443: finishing write tid 1 to smithi16534841-6 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1441: done (3 left) 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1442: done (2 left) 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1445: done (1 left) 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1446: snap_create 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1443: finishing write tid 2 to smithi16534841-6 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1443: finishing write tid 3 to smithi16534841-6 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1443: finishing write tid 4 to smithi16534841-6 2024-09-10T16:16:21.158 INFO:tasks.rados.rados.0.smithi165.stdout:1443: oid 6 updating version 0 to 621 2024-09-10T16:16:21.159 INFO:tasks.rados.rados.0.smithi165.stdout:1443: oid 6 updating version 621 to 622 2024-09-10T16:16:21.159 INFO:tasks.rados.rados.0.smithi165.stdout:1443: oid 6 updating version 622 to 623 2024-09-10T16:16:21.159 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 623 (ObjNum 520 snap 159 seq_num 520) dirty exists 2024-09-10T16:16:21.159 INFO:tasks.rados.rados.0.smithi165.stdout:1443: left oid 6 (ObjNum 520 snap 159 seq_num 520) 2024-09-10T16:16:21.315 INFO:tasks.rados.rados.0.smithi165.stdout:1443: done (1 left) 2024-09-10T16:16:21.315 INFO:tasks.rados.rados.0.smithi165.stdout:1446: done (0 left) 2024-09-10T16:16:21.315 INFO:tasks.rados.rados.0.smithi165.stdout:1447: snap_remove snap 159 2024-09-10T16:16:22.179 INFO:tasks.rados.rados.0.smithi165.stdout:1447: done (0 left) 2024-09-10T16:16:22.179 INFO:tasks.rados.rados.0.smithi165.stdout:1448: snap_create 2024-09-10T16:16:22.725 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:22.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 10 monclient: get_monmap_and_config 2024-09-10T16:16:22.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 10 monclient: build_initial_monmap 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 10 monclient: monmap: 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:22.833071+0000 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:22.833071+0000 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:22.834 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:22.835 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 1 Processor -- start 2024-09-10T16:16:22.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 1 -- start start 2024-09-10T16:16:22.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.832+0000 7f35b4fec640 10 monclient: init 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f35b009c310 0x7f35b009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-a con 0x7f35b009c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f35b009d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-b con 0x7f35b013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:22.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35b009fbc0 con 0x7f35b009d190 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f35b00a2a20 con 0x7f35b009c310 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35b009c6f0 con 0x7f35b013a5c0 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35b4fec640 10 monclient(hunting): authenticate will time out at 1361.320634s 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f35b009c310 0x7f35b009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f35b009c310 0x7f35b009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44438/0 (socket says 172.21.15.164:44438) 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39580/0 (socket says 172.21.15.164:39580) 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 1 -- 172.21.15.164:0/2626000468 learned_addr learned my addr 172.21.15.164:0/2626000468 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): get_auth_request con 0x7f35b009c310 auth_method 0 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:22.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35add9b640 1 --1- 172.21.15.164:0/2626000468 >> v1:172.21.15.164:6789/0 conn(0x7f35b009d190 0x7f35b009ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:48868/0 (socket says 172.21.15.164:48868) 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 10 monclient(hunting): get_auth_request con 0x7f35b013a5c0 auth_method 0 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:22.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 -- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 msgr2=0x7f35b013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 -- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 msgr2=0x7f35b013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.833+0000 7f35aed9d640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 10 monclient(hunting): handle_auth_done global_id 74481 payload 274 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 1 -- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 msgr2=0x7f35b013c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 1 -- 172.21.15.164:0/2626000468 >> v1:172.21.15.164:6789/0 conn(0x7f35b009d190 legacy=0x7f35b009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:22.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 10 monclient: _finish_hunting 0 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 1 monclient: found mon.noname-a 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.834+0000 7f35ae59c640 1 -- 172.21.15.164:0/2626000468 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b009a4d0 con 0x7f35b009c310 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.836+0000 7f35ae59c640 1 -- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b009c310 msgr2=0x7f35b009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.836+0000 7f35ae59c640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b009c310 0x7f35b009ea40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f359800b920 tx=0x7f359802a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f009ca736571683e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 1 -- 172.21.15.164:0/2626000468 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f359802f920 con 0x7f35b009c310 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:22.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: dump: 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.853 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:22.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:22.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:22.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: _finish_auth 0 2024-09-10T16:16:22.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: _check_auth_tickets 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 1 -- 172.21.15.164:0/2626000468 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f359802fac0 con 0x7f35b009c310 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 5 monclient: authenticate success, global_id 74481 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 1 -- 172.21.15.164:0/2626000468 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3598031400 con 0x7f35b009c310 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:22.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 4 set_mon_vals no callback set 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35ad59a640 10 monclient: dump: 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:22.858 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 10 monclient: shutdown 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 -- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b009c310 msgr2=0x7f35b009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b009c310 0x7f35b009ea40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f359800b920 tx=0x7f359802a540 comp rx=0 tx=0).stop 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 -- 172.21.15.164:0/2626000468 shutdown_connections 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b013a5c0 0x7f35b013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 --2- 172.21.15.164:0/2626000468 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b009c310 0x7f35b009ea40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 -- 172.21.15.164:0/2626000468 >> 172.21.15.164:0/2626000468 conn(0x7f35b00a3110 msgr2=0x7f35b00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:22.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 -- 172.21.15.164:0/2626000468 shutdown_connections 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.837+0000 7f35b4fec640 1 -- 172.21.15.164:0/2626000468 wait complete. 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.838+0000 7f35b4fec640 10 monclient: build_initial_monmap 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.838+0000 7f35b4fec640 10 monclient: monmap: 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:22.860 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.838+0000 7f35b4fec640 1 Processor -- start 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.838+0000 7f35b4fec640 1 -- start start 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.838+0000 7f35b4fec640 10 monclient: init 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35b009c310 0x7f35b0099510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-b con 0x7f35b009c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-a con 0x7f35b013a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): picked mon.noname-c con 0x7f35b01c8da0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:22.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f35b00a2230 con 0x7f35b013a5c0 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f35b00a20b0 con 0x7f35b009c310 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35b0074e00 con 0x7f35b01c8da0 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35b4fec640 10 monclient(hunting): authenticate will time out at 1361.326330s 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44450/0 (socket says 172.21.15.164:44450) 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 -- 172.21.15.164:0/946250182 learned_addr learned my addr 172.21.15.164:0/946250182 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:22.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): get_auth_request con 0x7f35b013a5c0 auth_method 0 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 10 monclient(hunting): get_auth_request con 0x7f35b01c8da0 auth_method 0 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:22.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 msgr2=0x7f35b008e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 msgr2=0x7f35b008e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35aed9d640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:22.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35ae59c640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35b009c310 0x7f35b0099510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35ae59c640 10 monclient(hunting): get_auth_request con 0x7f35b009c310 auth_method 0 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient(hunting): handle_auth_done global_id 74484 payload 824 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 msgr2=0x7f35b008e3c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:22.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35b009c310 msgr2=0x7f35b0099510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35b009c310 0x7f35b0099510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient: _finish_hunting 0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 monclient: found mon.noname-a 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.839+0000 7f35add9b640 1 -- 172.21.15.164:0/946250182 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35b009ea20 con 0x7f35b013a5c0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35add9b640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 msgr2=0x7f35b0099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35add9b640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c480 tx=0x7f35a000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0017070 con 0x7f35b013a5c0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: dump: 2024-09-10T16:16:22.866 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35a0005810 con 0x7f35b013a5c0 2024-09-10T16:16:22.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 5 monclient: authenticate success, global_id 74484 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 10 monclient: _renew_subs 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35b0091ca0 con 0x7f35b013a5c0 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35a0007690 con 0x7f35b013a5c0 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35a6ffd640 10 monclient: dump: 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:22.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35acd99640 4 set_mon_vals no callback set 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 10 monclient: _renew_subs 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.842+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35b00919c0 con 0x7f35b013a5c0 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f35a001a070 con 0x7f35b013a5c0 2024-09-10T16:16:22.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35a6ffd640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 0x7f3590040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35b4fec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35b4fec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35b4fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(349..349 src has 184..349) ==== 4827+0+0 (secure 0 0 0) 0x7f35a00521c0 con 0x7f35b013a5c0 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35b009f400 con 0x7f35b013a5c0 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35ae59c640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 0x7f3590040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:22.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.843+0000 7f35ae59c640 10 monclient: get_auth_request con 0x7f359003e240 auth_method 0 2024-09-10T16:16:22.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.846+0000 7f35ae59c640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 0x7f3590040700 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35b00a0280 tx=0x7f359802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:22.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.848+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f35b009f400 con 0x7f35b013a5c0 2024-09-10T16:16:22.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.848+0000 7f35a6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:22.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:22.848+0000 7f35a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:23.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35b4fec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:23.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35b4fec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:23.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35b4fec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:23.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f35b009f7b0 con 0x7f35b013a5c0 2024-09-10T16:16:23.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35a6ffd640 1 -- 172.21.15.164:0/946250182 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f35b009ea20 con 0x7f35b013a5c0 2024-09-10T16:16:23.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35a6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:23.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.092+0000 7f35a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:23.095 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":90,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:23.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 msgr2=0x7f3590040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:23.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 0x7f3590040700 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f35b00a0280 tx=0x7f359802f000 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 10 monclient: shutdown 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 msgr2=0x7f35b0099a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f35a000c480 tx=0x7f35a000c940 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 shutdown_connections 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f359003e240 0x7f3590040700 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35b01c8da0 0x7f35b008e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f35b013a5c0 0x7f35b0099a50 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 --2- 172.21.15.164:0/946250182 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35b009c310 0x7f35b0099510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 >> 172.21.15.164:0/946250182 conn(0x7f35b00a3110 msgr2=0x7f35b009d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:23.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 shutdown_connections 2024-09-10T16:16:23.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:23.095+0000 7f35b4fec640 1 -- 172.21.15.164:0/946250182 wait complete. 2024-09-10T16:16:23.181 INFO:tasks.rados.rados.0.smithi165.stdout:1448: done (0 left) 2024-09-10T16:16:23.181 INFO:tasks.rados.rados.0.smithi165.stdout:1449: snap_remove snap 160 2024-09-10T16:16:24.185 INFO:tasks.rados.rados.0.smithi165.stdout:1449: done (0 left) 2024-09-10T16:16:24.185 INFO:tasks.rados.rados.0.smithi165.stdout:1450: read oid 27 snap -1 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1450: expect deleted 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1451: rollback oid 24 current snap is 161 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1451: done (1 left) 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1452: read oid 15 snap -1 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1452: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1453: rollback oid 25 current snap is 161 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1453: done (2 left) 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1454: rollback oid 16 current snap is 161 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1454: done (2 left) 2024-09-10T16:16:24.186 INFO:tasks.rados.rados.0.smithi165.stdout:1455: read oid 28 snap -1 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1455: expect (ObjNum 504 snap 159 seq_num 504) 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1456: read oid 50 snap -1 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1456: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1457: copy_from oid 37 from oid 5 current snap is 161 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1458: write oid 25 current snap is 161 2024-09-10T16:16:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:1458: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-10T16:16:24.194 INFO:tasks.rados.rados.0.smithi165.stdout:1458: writing smithi16534841-25 from 773254 to 1565236 tid 1 2024-09-10T16:16:24.201 INFO:tasks.rados.rados.0.smithi165.stdout:1458: writing smithi16534841-25 from 2346424 to 2773254 tid 2 2024-09-10T16:16:24.201 INFO:tasks.rados.rados.0.smithi165.stdout:1459: write (excl) oid 7 current snap is 161 2024-09-10T16:16:24.201 INFO:tasks.rados.rados.0.smithi165.stdout:1459: seq_num 523 ranges {790061=467231,2004906=541501} 2024-09-10T16:16:24.208 INFO:tasks.rados.rados.0.smithi165.stdout:1459: writing smithi16534841-7 from 790061 to 1257292 tid 1 2024-09-10T16:16:24.215 INFO:tasks.rados.rados.0.smithi165.stdout:1459: writing smithi16534841-7 from 2004906 to 2546407 tid 2 2024-09-10T16:16:24.216 INFO:tasks.rados.rados.0.smithi165.stdout:1460: write (excl) oid 40 current snap is 161 2024-09-10T16:16:24.216 INFO:tasks.rados.rados.0.smithi165.stdout:1460: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-10T16:16:24.221 INFO:tasks.rados.rados.0.smithi165.stdout:1460: writing smithi16534841-40 from 406868 to 949348 tid 1 2024-09-10T16:16:24.226 INFO:tasks.rados.rados.0.smithi165.stdout:1460: writing smithi16534841-40 from 1547035 to 1948547 tid 2 2024-09-10T16:16:24.231 INFO:tasks.rados.rados.0.smithi165.stdout:1460: writing smithi16534841-40 from 2690354 to 2806868 tid 3 2024-09-10T16:16:24.231 INFO:tasks.rados.rados.0.smithi165.stdout:1461: write oid 1 current snap is 161 2024-09-10T16:16:24.231 INFO:tasks.rados.rados.0.smithi165.stdout:1461: seq_num 525 ranges {423675=734082,1721870=545170,2717777=105898} 2024-09-10T16:16:24.237 INFO:tasks.rados.rados.0.smithi165.stdout:1461: writing smithi16534841-1 from 423675 to 1157757 tid 1 2024-09-10T16:16:24.243 INFO:tasks.rados.rados.0.smithi165.stdout:1461: writing smithi16534841-1 from 1721870 to 2267040 tid 2 2024-09-10T16:16:24.247 INFO:tasks.rados.rados.0.smithi165.stdout:1461: writing smithi16534841-1 from 2717777 to 2823675 tid 3 2024-09-10T16:16:24.247 INFO:tasks.rados.rados.0.smithi165.stdout:1462: rollback oid 35 current snap is 161 2024-09-10T16:16:24.247 INFO:tasks.rados.rados.0.smithi165.stdout:1462: done (9 left) 2024-09-10T16:16:24.247 INFO:tasks.rados.rados.0.smithi165.stdout:1463: snap_create 2024-09-10T16:16:24.247 INFO:tasks.rados.rados.0.smithi165.stdout:1456: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:16:24.279 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:24.323 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 227807 2024-09-10T16:16:24.363 INFO:tasks.rados.rados.0.smithi165.stdout:1452: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:16:24.422 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3207102 2024-09-10T16:16:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:1457: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:16:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:1455: expect (ObjNum 504 snap 159 seq_num 504) 2024-09-10T16:16:24.592 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1167737 2024-09-10T16:16:24.656 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1627064 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1457: finishing copy_from to smithi16534841-37 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: finishing write tid 1 to smithi16534841-25 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: finishing write tid 2 to smithi16534841-25 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: finishing write tid 3 to smithi16534841-25 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: finishing write tid 4 to smithi16534841-25 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: oid 25 updating version 0 to 903 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: oid 25 updating version 903 to 904 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:1458: oid 25 version 904 is already newer than 902 2024-09-10T16:16:24.724 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 904 (ObjNum 522 snap 161 seq_num 522) dirty exists 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1458: left oid 25 (ObjNum 522 snap 161 seq_num 522) 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: finishing write tid 1 to smithi16534841-7 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: finishing write tid 2 to smithi16534841-7 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: finishing write tid 3 to smithi16534841-7 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: finishing write tid 4 to smithi16534841-7 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: oid 7 updating version 0 to 1113 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: oid 7 updating version 1113 to 1114 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: oid 7 updating version 1114 to 1115 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1115 (ObjNum 523 snap 161 seq_num 523) dirty exists 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1459: left oid 7 (ObjNum 523 snap 161 seq_num 523) 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1460: finishing write tid 1 to smithi16534841-40 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1460: finishing write tid 2 to smithi16534841-40 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1460: finishing write tid 3 to smithi16534841-40 2024-09-10T16:16:24.725 INFO:tasks.rados.rados.0.smithi165.stdout:1460: finishing write tid 4 to smithi16534841-40 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: finishing write tid 5 to smithi16534841-40 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: oid 40 updating version 0 to 355 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: oid 40 updating version 355 to 356 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: oid 40 version 356 is already newer than 354 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: oid 40 version 356 is already newer than 353 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 356 (ObjNum 524 snap 161 seq_num 524) dirty exists 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1460: left oid 40 (ObjNum 524 snap 161 seq_num 524) 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: finishing write tid 1 to smithi16534841-1 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: finishing write tid 2 to smithi16534841-1 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: finishing write tid 3 to smithi16534841-1 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: finishing write tid 4 to smithi16534841-1 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: finishing write tid 5 to smithi16534841-1 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: oid 1 updating version 0 to 1067 2024-09-10T16:16:24.726 INFO:tasks.rados.rados.0.smithi165.stdout:1461: oid 1 version 1067 is already newer than 1066 2024-09-10T16:16:24.727 INFO:tasks.rados.rados.0.smithi165.stdout:1461: oid 1 updating version 1067 to 1069 2024-09-10T16:16:24.727 INFO:tasks.rados.rados.0.smithi165.stdout:1461: oid 1 version 1069 is already newer than 1068 2024-09-10T16:16:24.727 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1069 (ObjNum 525 snap 161 seq_num 525) dirty exists 2024-09-10T16:16:24.727 INFO:tasks.rados.rados.0.smithi165.stdout:1461: left oid 1 (ObjNum 525 snap 161 seq_num 525) 2024-09-10T16:16:25.183 INFO:tasks.rados.rados.0.smithi165.stdout:1450: done (9 left) 2024-09-10T16:16:25.183 INFO:tasks.rados.rados.0.smithi165.stdout:1452: done (8 left) 2024-09-10T16:16:25.183 INFO:tasks.rados.rados.0.smithi165.stdout:1455: done (7 left) 2024-09-10T16:16:25.183 INFO:tasks.rados.rados.0.smithi165.stdout:1456: done (6 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1457: done (5 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1458: done (4 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1459: done (3 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1460: done (2 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1461: done (1 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1463: done (0 left) 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1464: read oid 30 snap -1 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1464: expect deleted 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1465: write oid 38 current snap is 162 2024-09-10T16:16:25.184 INFO:tasks.rados.rados.0.smithi165.stdout:1465: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-10T16:16:25.187 INFO:tasks.rados.rados.0.smithi165.stdout:1465: writing smithi16534841-38 from 440482 to 849813 tid 1 2024-09-10T16:16:25.192 INFO:tasks.rados.rados.0.smithi165.stdout:1465: writing smithi16534841-38 from 1380352 to 1785533 tid 2 2024-09-10T16:16:25.198 INFO:tasks.rados.rados.0.smithi165.stdout:1465: writing smithi16534841-38 from 2461553 to 2840482 tid 3 2024-09-10T16:16:25.199 INFO:tasks.rados.rados.0.smithi165.stdout:1466: snap_create 2024-09-10T16:16:25.199 INFO:tasks.rados.rados.0.smithi165.stdout:1465: finishing write tid 1 to smithi16534841-38 2024-09-10T16:16:25.199 INFO:tasks.rados.rados.0.smithi165.stdout:1465: finishing write tid 2 to smithi16534841-38 2024-09-10T16:16:25.201 INFO:tasks.rados.rados.0.smithi165.stdout:1465: finishing write tid 3 to smithi16534841-38 2024-09-10T16:16:25.202 INFO:tasks.rados.rados.0.smithi165.stdout:1465: finishing write tid 4 to smithi16534841-38 2024-09-10T16:16:25.202 INFO:tasks.rados.rados.0.smithi165.stdout:1465: finishing write tid 5 to smithi16534841-38 2024-09-10T16:16:25.202 INFO:tasks.rados.rados.0.smithi165.stdout:1465: oid 38 updating version 0 to 892 2024-09-10T16:16:25.203 INFO:tasks.rados.rados.0.smithi165.stdout:1465: oid 38 version 892 is already newer than 890 2024-09-10T16:16:25.203 INFO:tasks.rados.rados.0.smithi165.stdout:1465: oid 38 updating version 892 to 893 2024-09-10T16:16:25.203 INFO:tasks.rados.rados.0.smithi165.stdout:1465: oid 38 version 893 is already newer than 891 2024-09-10T16:16:25.203 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 893 (ObjNum 526 snap 162 seq_num 526) dirty exists 2024-09-10T16:16:25.203 INFO:tasks.rados.rados.0.smithi165.stdout:1465: left oid 38 (ObjNum 526 snap 162 seq_num 526) 2024-09-10T16:16:26.107 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:26.192 INFO:tasks.rados.rados.0.smithi165.stdout:1464: done (2 left) 2024-09-10T16:16:26.192 INFO:tasks.rados.rados.0.smithi165.stdout:1465: done (1 left) 2024-09-10T16:16:26.192 INFO:tasks.rados.rados.0.smithi165.stdout:1466: done (0 left) 2024-09-10T16:16:26.192 INFO:tasks.rados.rados.0.smithi165.stdout:1467: delete oid 41 current snap is 163 2024-09-10T16:16:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1467: done (0 left) 2024-09-10T16:16:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1468: write (excl) oid 29 current snap is 163 2024-09-10T16:16:26.195 INFO:tasks.rados.rados.0.smithi165.stdout:1468: seq_num 528 ranges {474096=559829,1780963=408850,2800046=74050} 2024-09-10T16:16:26.200 INFO:tasks.rados.rados.0.smithi165.stdout:1468: writing smithi16534841-29 from 474096 to 1033925 tid 1 2024-09-10T16:16:26.207 INFO:tasks.rados.rados.0.smithi165.stdout:1468: writing smithi16534841-29 from 1780963 to 2189813 tid 2 2024-09-10T16:16:26.211 INFO:tasks.rados.rados.0.smithi165.stdout:1468: writing smithi16534841-29 from 2800046 to 2874096 tid 3 2024-09-10T16:16:26.211 INFO:tasks.rados.rados.0.smithi165.stdout:1469: copy_from oid 18 from oid 40 current snap is 163 2024-09-10T16:16:26.212 INFO:tasks.rados.rados.0.smithi165.stdout:1468: finishing write tid 1 to smithi16534841-29 2024-09-10T16:16:26.212 INFO:tasks.rados.rados.0.smithi165.stdout:1470: write oid 2 current snap is 163 2024-09-10T16:16:26.212 INFO:tasks.rados.rados.0.smithi165.stdout:1470: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-10T16:16:26.217 INFO:tasks.rados.rados.0.smithi165.stdout:1470: writing smithi16534841-2 from 490903 to 1125981 tid 1 2024-09-10T16:16:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.219+0000 7f714fbf8640 10 monclient: get_monmap_and_config 2024-09-10T16:16:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.219+0000 7f714fbf8640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.219+0000 7f714fbf8640 10 monclient: monmap: 2024-09-10T16:16:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.220 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:26.220202+0000 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:26.220202+0000 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:26.221 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.219+0000 7f714fbf8640 1 Processor -- start 2024-09-10T16:16:26.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.220+0000 7f714fbf8640 1 -- start start 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.220+0000 7f714fbf8640 10 monclient: init 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f714806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7148134f70 0x7f714813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-a con 0x7f7148134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f714813d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f714809ce50 con 0x7f714806cb70 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7148135380 con 0x7f714813d560 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f714806db60 con 0x7f7148134f70 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714fbf8640 10 monclient(hunting): authenticate will time out at 1364.708205s 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7148134f70 0x7f714813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d994640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f714806cb70 0x7f71480a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52498/0 (socket says 172.21.15.164:52498) 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7148134f70 0x7f714813d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35370/0 (socket says 172.21.15.164:35370) 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d994640 1 -- 172.21.15.164:0/929562865 learned_addr learned my addr 172.21.15.164:0/929562865 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): get_auth_request con 0x7f7148134f70 auth_method 0 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient(hunting): handle_auth_done global_id 74487 payload 274 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 -- 172.21.15.164:0/929562865 >> v1:172.21.15.165:6789/0 conn(0x7f714813d560 legacy=0x7f714813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 -- 172.21.15.164:0/929562865 >> v1:172.21.15.164:6789/0 conn(0x7f714806cb70 legacy=0x7f71480a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 monclient: found mon.noname-a 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:26.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.221+0000 7f714d193640 1 -- 172.21.15.164:0/929562865 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714809cb70 con 0x7f7148134f70 2024-09-10T16:16:26.230 INFO:tasks.rados.rados.0.smithi165.stdout:1470: writing smithi16534841-2 from 1839445 to 2391953 tid 2 2024-09-10T16:16:26.230 INFO:tasks.rados.rados.0.smithi165.stdout:1471: read oid 16 snap 162 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1471: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1472: rollback oid 26 current snap is 163 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 162 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: finishing write tid 2 to smithi16534841-29 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: finishing write tid 3 to smithi16534841-29 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: finishing write tid 4 to smithi16534841-29 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: finishing write tid 5 to smithi16534841-29 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: oid 29 updating version 0 to 1073 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: oid 29 version 1073 is already newer than 1070 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: oid 29 version 1073 is already newer than 1071 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:1468: oid 29 version 1073 is already newer than 1072 2024-09-10T16:16:26.231 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1073 (ObjNum 528 snap 163 seq_num 528) dirty exists 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1468: left oid 29 (ObjNum 528 snap 163 seq_num 528) 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1469: finishing copy_from racing read to smithi16534841-18 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1469: finishing copy_from to smithi16534841-18 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 356 (ObjNum 524 snap 161 seq_num 524) dirty exists 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1468: done (4 left) 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1469: done (3 left) 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1473: read oid 13 snap -1 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1473: expect (ObjNum 491 snap 155 seq_num 491) 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1474: rollback oid 10 current snap is 163 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 162 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1475: delete oid 12 current snap is 163 2024-09-10T16:16:26.232 INFO:tasks.rados.rados.0.smithi165.stdout:1475: done (5 left) 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1476: read oid 30 snap -1 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1476: expect deleted 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1477: read oid 35 snap 161 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1477: expect deleted 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1478: snap_create 2024-09-10T16:16:26.234 INFO:tasks.rados.rados.0.smithi165.stdout:1470: finishing write tid 1 to smithi16534841-2 2024-09-10T16:16:26.237 INFO:tasks.rados.rados.0.smithi165.stdout:1470: finishing write tid 2 to smithi16534841-2 2024-09-10T16:16:26.237 INFO:tasks.rados.rados.0.smithi165.stdout:1470: finishing write tid 3 to smithi16534841-2 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1470: finishing write tid 4 to smithi16534841-2 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1470: oid 2 updating version 0 to 1075 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1470: oid 2 updating version 1075 to 1076 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1470: oid 2 version 1076 is already newer than 1074 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1076 (ObjNum 529 snap 163 seq_num 529) dirty exists 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1470: left oid 2 (ObjNum 529 snap 163 seq_num 529) 2024-09-10T16:16:26.238 INFO:tasks.rados.rados.0.smithi165.stdout:1473: expect (ObjNum 491 snap 155 seq_num 491) 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.224+0000 7f714d193640 1 -- 172.21.15.164:0/929562865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 msgr2=0x7f714813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.224+0000 7f714d193640 1 --2- 172.21.15.164:0/929562865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f714813d020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f71380097d0 tx=0x7f713802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0fcf688fb49cf47 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 1 -- 172.21.15.164:0/929562865 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138037070 con 0x7f7148134f70 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: dump: 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.246 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 1 -- 172.21.15.164:0/929562865 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7138032440 con 0x7f7148134f70 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 1 -- 172.21.15.164:0/929562865 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7138032810 con 0x7f7148134f70 2024-09-10T16:16:26.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.225+0000 7f714c992640 10 monclient: dump: 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.227+0000 7f714fbf8640 5 monclient: authenticate success, global_id 74487 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.227+0000 7f714fbf8640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.227+0000 7f714fbf8640 4 set_mon_vals no callback set 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 10 monclient: shutdown 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 -- 172.21.15.164:0/929562865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 msgr2=0x7f714813d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 --2- 172.21.15.164:0/929562865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f714813d020 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f71380097d0 tx=0x7f713802a000 comp rx=0 tx=0).stop 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 -- 172.21.15.164:0/929562865 shutdown_connections 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 --2- 172.21.15.164:0/929562865 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f714813d020 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 -- 172.21.15.164:0/929562865 >> 172.21.15.164:0/929562865 conn(0x7f71480aa900 msgr2=0x7f71480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.228+0000 7f714fbf8640 1 -- 172.21.15.164:0/929562865 shutdown_connections 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.229+0000 7f714fbf8640 1 -- 172.21.15.164:0/929562865 wait complete. 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.229+0000 7f714fbf8640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.229+0000 7f714fbf8640 10 monclient: monmap: 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.230+0000 7f714fbf8640 1 Processor -- start 2024-09-10T16:16:26.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.230+0000 7f714fbf8640 1 -- start start 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.230+0000 7f714fbf8640 10 monclient: init 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f714806cb70 0x7f71481cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-b con 0x7f714806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f71480a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-a con 0x7f7148134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): picked mon.noname-c con 0x7f714813d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71480a4db0 con 0x7f7148134f70 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f714806cf50 con 0x7f714806cb70 2024-09-10T16:16:26.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7148141070 con 0x7f714813d560 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714fbf8640 10 monclient(hunting): authenticate will time out at 1364.718680s 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f71480a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60402/0 (socket says 172.21.15.164:60402) 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 1 -- 172.21.15.164:0/3430932686 learned_addr learned my addr 172.21.15.164:0/3430932686 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): get_auth_request con 0x7f714813d560 auth_method 0 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): get_auth_request con 0x7f7148134f70 auth_method 0 2024-09-10T16:16:26.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714e195640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.231+0000 7f714d193640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 10 monclient(hunting): handle_auth_done global_id 114101 payload 824 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 -- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f714806cb70 msgr2=0x7f71481cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f714806cb70 0x7f71481cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 -- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 msgr2=0x7f71480a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f71480a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 monclient: found mon.noname-c 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714e195640 1 -- 172.21.15.164:0/3430932686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f714809dbe0 con 0x7f714813d560 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.232+0000 7f714d193640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f71480a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714e195640 1 -- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 msgr2=0x7f71480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714e195640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f714000c700 tx=0x7f714000cbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140016020 con 0x7f714813d560 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: dump: 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.255 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 5 monclient: authenticate success, global_id 114101 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f714000ac60 con 0x7f714813d560 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 10 monclient: _renew_subs 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7140012660 con 0x7f714813d560 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f714806e720 con 0x7f714813d560 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713e7fc640 10 monclient: dump: 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f713ffff640 4 set_mon_vals no callback set 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 10 monclient: _renew_subs 2024-09-10T16:16:26.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.234+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f714809e180 con 0x7f714813d560 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7140019020 con 0x7f714813d560 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f714fbf8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f714fbf8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f714fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71480a46a0 con 0x7f714813d560 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f713e7fc640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 0x7f712c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.235+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(353..353 src has 184..353) ==== 4755+0+0 (secure 0 0 0) 0x7f7140052f30 con 0x7f714813d560 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.236+0000 7f714d994640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 0x7f712c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.236+0000 7f714d994640 10 monclient: get_auth_request con 0x7f712c03e210 auth_method 0 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.238+0000 7f714d994640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 0x7f712c0406d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7144009690 tx=0x7f7144006bd0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.240+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7140020020 con 0x7f714813d560 2024-09-10T16:16:26.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.240+0000 7f713e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.240+0000 7f713e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:26.259 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 858375 2024-09-10T16:16:26.295 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:26.359 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 486670 2024-09-10T16:16:26.419 INFO:tasks.rados.rados.0.smithi165.stdout:1471: expect (ObjNum 353 snap 101 seq_num 353) 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f714fbf8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f714fbf8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f714fbf8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f714809e520 con 0x7f714813d560 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f713e7fc640 1 -- 172.21.15.164:0/3430932686 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f714806e720 con 0x7f714813d560 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f713e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.484+0000 7f713e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:26.486 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:26.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 msgr2=0x7f712c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 0x7f712c0406d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f7144009690 tx=0x7f7144006bd0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 10 monclient: shutdown 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 msgr2=0x7f71480a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f714000c700 tx=0x7f714000cbc0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 shutdown_connections 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f712c03e210 0x7f712c0406d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f714813d560 0x7f71480a3ef0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7148134f70 0x7f71480a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 --2- 172.21.15.164:0/3430932686 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f714806cb70 0x7f71481cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 >> 172.21.15.164:0/3430932686 conn(0x7f71480aa900 msgr2=0x7f71481335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 shutdown_connections 2024-09-10T16:16:26.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.487+0000 7f714fbf8640 1 -- 172.21.15.164:0/3430932686 wait complete. 2024-09-10T16:16:26.499 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:16:26.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:16:26.499 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.596+0000 7fb397ab6640 10 monclient: get_monmap_and_config 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.596+0000 7fb397ab6640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.596+0000 7fb397ab6640 10 monclient: monmap: 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:26.597494+0000 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:26.597494+0000 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:26.599 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:26.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:26.599 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:26.599 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:26.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 Processor -- start 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 -- start start 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient: init 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb39006cb70 0x7fb390134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-a con 0x7fb39006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb3900a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb3900a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb39009ce50 con 0x7fb3900a4ab0 2024-09-10T16:16:26.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb390138ac0 con 0x7fb3900a46a0 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb39006db60 con 0x7fb39006cb70 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.597+0000 7fb397ab6640 10 monclient(hunting): authenticate will time out at 1365.084903s 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb39006cb70 0x7fb390134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb396053640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb3900a4ab0 0x7fb3901383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52510/0 (socket says 172.21.15.164:52510) 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb396053640 1 -- 172.21.15.164:0/957761841 learned_addr learned my addr 172.21.15.164:0/957761841 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb39006cb70 0x7fb390134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35398/0 (socket says 172.21.15.164:35398) 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): get_auth_request con 0x7fb39006cb70 auth_method 0 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient(hunting): handle_auth_done global_id 74493 payload 274 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 -- 172.21.15.164:0/957761841 >> v1:172.21.15.165:6789/0 conn(0x7fb3900a46a0 legacy=0x7fb390137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 -- 172.21.15.164:0/957761841 >> v1:172.21.15.164:6789/0 conn(0x7fb3900a4ab0 legacy=0x7fb3901383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 monclient: found mon.noname-a 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.598+0000 7fb395852640 1 -- 172.21.15.164:0/957761841 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb39009cb70 con 0x7fb39006cb70 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb395852640 1 -- 172.21.15.164:0/957761841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39006cb70 msgr2=0x7fb390134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb395852640 1 --2- 172.21.15.164:0/957761841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39006cb70 0x7fb390134f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb3800097d0 tx=0x7fb38002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5634f964071cc41 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 1 -- 172.21.15.164:0/957761841 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380037070 con 0x7fb39006cb70 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: dump: 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.613 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 1 -- 172.21.15.164:0/957761841 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb380032440 con 0x7fb39006cb70 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 5 monclient: authenticate success, global_id 74493 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 1 -- 172.21.15.164:0/957761841 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380032810 con 0x7fb39006cb70 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 4 set_mon_vals no callback set 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb394850640 10 monclient: dump: 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 10 monclient: shutdown 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 1 -- 172.21.15.164:0/957761841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39006cb70 msgr2=0x7fb390134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 1 --2- 172.21.15.164:0/957761841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39006cb70 0x7fb390134f70 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fb3800097d0 tx=0x7fb38002a000 comp rx=0 tx=0).stop 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 1 -- 172.21.15.164:0/957761841 shutdown_connections 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 1 --2- 172.21.15.164:0/957761841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39006cb70 0x7fb390134f70 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.601+0000 7fb397ab6640 1 -- 172.21.15.164:0/957761841 >> 172.21.15.164:0/957761841 conn(0x7fb3900aa900 msgr2=0x7fb3900aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 1 -- 172.21.15.164:0/957761841 shutdown_connections 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 1 -- 172.21.15.164:0/957761841 wait complete. 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 10 monclient: monmap: 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 1 Processor -- start 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 1 -- start start 2024-09-10T16:16:26.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.602+0000 7fb397ab6640 10 monclient: init 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-c con 0x7fb39006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb39012d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb39009e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3900a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb390141050 con 0x7fb3900a4ab0 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb39006cf50 con 0x7fb3900a46a0 2024-09-10T16:16:26.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb3901411d0 con 0x7fb39006cb70 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb397ab6640 10 monclient(hunting): authenticate will time out at 1365.090402s 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60408/0 (socket says 172.21.15.164:60408) 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 learned_addr learned my addr 172.21.15.164:0/293893252 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb39009e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): get_auth_request con 0x7fb39006cb70 auth_method 0 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): get_auth_request con 0x7fb3900a4ab0 auth_method 0 2024-09-10T16:16:26.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb39012d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): get_auth_request con 0x7fb3900a46a0 auth_method 0 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient(hunting): handle_auth_done global_id 114104 payload 824 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 msgr2=0x7fb39012d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb39012d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 msgr2=0x7fb39009e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb39009e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 monclient: found mon.noname-c 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3900a1e80 con 0x7fb39006cb70 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.603+0000 7fb396053640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb39009e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 msgr2=0x7fb39012d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb38002a5f0 tx=0x7fb3800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 msgr2=0x7fb39012d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 msgr2=0x7fb39012d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb38002a5f0 tx=0x7fb3800046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb38002a5f0 tx=0x7fb3800046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 msgr2=0x7fb39012d280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb39006cb70 0x7fb39012d280 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb3740054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): picked mon.noname-a con 0x7fb3900a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 0x7fb374008330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): picked mon.noname-c con 0x7fb374005ea0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb37400aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): picked mon.noname-b con 0x7fb3900a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb37400b900 con 0x7fb3900a4ab0 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb37400bab0 con 0x7fb3900a46a0 2024-09-10T16:16:26.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb37400bcb0 con 0x7fb374005ea0 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb3867fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb3740054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): get_auth_request con 0x7fb3900a4ab0 auth_method 0 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 0x7fb374008330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): get_auth_request con 0x7fb374005ea0 auth_method 0 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:26.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb37400aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): get_auth_request con 0x7fb3900a46a0 auth_method 0 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb396053640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient(hunting): handle_auth_done global_id 114104 payload 293 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 msgr2=0x7fb37400aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb3900a46a0 0x7fb37400aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 msgr2=0x7fb3740054b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb3740054b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 monclient: found mon.noname-c 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395051640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37400bf70 con 0x7fb374005ea0 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.606+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb3900a4ab0 0x7fb3740054b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.608+0000 7fb395051640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 msgr2=0x7fb374008330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.608+0000 7fb395051640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 0x7fb374008330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb3900a4640 tx=0x7fb38002e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c018030 con 0x7fb374005ea0 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: dump: 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb38c00f9c0 con 0x7fb374005ea0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 5 monclient: authenticate success, global_id 114104 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 10 monclient: _renew_subs 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3900a2160 con 0x7fb374005ea0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c003bb0 con 0x7fb374005ea0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb3867fc640 10 monclient: dump: 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 10 monclient: _renew_subs 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb387fff640 4 set_mon_vals no callback set 2024-09-10T16:16:26.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.609+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3900a1ba0 con 0x7fb374005ea0 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb397ab6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb397ab6640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 0x7fb39012d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb390134010 con 0x7fb39012d280 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 0x7fb39012d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient: get_auth_request con 0x7fb39012d280 auth_method 0 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb38c01b070 con 0x7fb374005ea0 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 10 monclient(hunting): handle_auth_done global_id 74499 payload 824 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb390134f70 con 0x7fb39012d280 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39006cb70 0x7fb374048a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(353..353 src has 184..353) ==== 4755+0+0 (secure 0 0 0) 0x7fb38c051bc0 con 0x7fb374005ea0 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb396053640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39006cb70 0x7fb374048a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.610+0000 7fb396053640 10 monclient: get_auth_request con 0x7fb39006cb70 auth_method 0 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 msgr2=0x7fb39012d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 0x7fb39012d660 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb380043000 tx=0x7fb380008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380058070 con 0x7fb39012d280 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 10 monclient: dump: 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.634 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb380032440 con 0x7fb39012d280 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 msgr2=0x7fb39012d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 0x7fb39012d660 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb380043000 tx=0x7fb380008030 comp rx=0 tx=0).stop 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 msgr2=0x7fb39012d660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.612+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012d280 0x7fb39012d660 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.613+0000 7fb396053640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39006cb70 0x7fb374048a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3901352c0 tx=0x7fb388009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.618+0000 7fb397ab6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:26.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.618+0000 7fb397ab6640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 0x7fb39012f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.618+0000 7fb397ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.618+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb3900770f0 con 0x7fb39012dba0 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 0x7fb39012f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient: get_auth_request con 0x7fb39012dba0 auth_method 0 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 10 monclient(hunting): handle_auth_done global_id 74502 payload 824 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.619+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb39009f5c0 con 0x7fb39012dba0 2024-09-10T16:16:26.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb395852640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 msgr2=0x7fb39012f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb395852640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 0x7fb39012f240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb390134190 tx=0x7fb380009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb380058040 con 0x7fb39012dba0 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 10 monclient: dump: 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.637 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7fb380047ae0 con 0x7fb39012dba0 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 msgr2=0x7fb39012f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 0x7fb39012f240 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fb390134190 tx=0x7fb380009190 comp rx=0 tx=0).stop 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 msgr2=0x7fb39012f240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.621+0000 7fb3867fc640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb39012dba0 0x7fb39012f240 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.638 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 214488, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:26.639 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.640 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.641 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:26.642 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:26.643 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:26.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.646 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.647 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.648 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:26.649 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:26.650 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:26.651 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:26.652 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39006cb70 msgr2=0x7fb374048a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb39006cb70 0x7fb374048a40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fb3901352c0 tx=0x7fb388009040 comp rx=0 tx=0).stop 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 10 monclient: shutdown 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 msgr2=0x7fb374008330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 --2- 172.21.15.164:0/293893252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb374005ea0 0x7fb374008330 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb3900a4640 tx=0x7fb38002e000 comp rx=0 tx=0).stop 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb396053640 1 -- 172.21.15.164:0/293893252 reap_dead start 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb396053640 1 -- 172.21.15.164:0/293893252 reap_dead start 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 shutdown_connections 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 >> 172.21.15.164:0/293893252 conn(0x7fb3900aa900 msgr2=0x7fb3900a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 shutdown_connections 2024-09-10T16:16:26.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.622+0000 7fb397ab6640 1 -- 172.21.15.164:0/293893252 wait complete. 2024-09-10T16:16:26.655 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:26.655 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:16:26.655 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:16:26.666 INFO:tasks.rados.rados.0.smithi165.stdout:1474: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:1474: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:1472: finishing rollback tid 0 to smithi16534841-26 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:1472: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:1474: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1011 (ObjNum 486 snap 153 seq_num 486) dirty exists 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:1472: finishing rollback tid 2 to smithi16534841-26 2024-09-10T16:16:26.667 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1119 (ObjNum 462 snap 144 seq_num 462) dirty exists 2024-09-10T16:16:26.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.752+0000 7ff08f1ec640 10 monclient: get_monmap_and_config 2024-09-10T16:16:26.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.752+0000 7ff08f1ec640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.752+0000 7ff08f1ec640 10 monclient: monmap: 2024-09-10T16:16:26.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:26.753193+0000 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:26.753193+0000 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 Processor -- start 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 -- start start 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient: init 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-a con 0x7ff08809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 0x7ff08809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-b con 0x7ff08809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 0x7ff08813b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-c con 0x7ff08809f4c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:26.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff08809a430 con 0x7ff08809c180 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff0880a24e0 con 0x7ff08809f4c0 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff0880a2bf0 con 0x7ff08809d160 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08f1ec640 10 monclient(hunting): authenticate will time out at 1365.240596s 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 0x7ff08809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35454/0 (socket says 172.21.15.164:35454) 2024-09-10T16:16:26.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 1 -- 172.21.15.164:0/3373459188 learned_addr learned my addr 172.21.15.164:0/3373459188 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): get_auth_request con 0x7ff08809c180 auth_method 0 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): get_auth_request con 0x7ff08809d160 auth_method 0 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 10 monclient(hunting): handle_auth_done global_id 114107 payload 274 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.753+0000 7ff08d789640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 0x7ff08813b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 -- 172.21.15.164:0/3373459188 >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 msgr2=0x7ff08813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 0x7ff08813b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 -- 172.21.15.164:0/3373459188 >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 msgr2=0x7ff08809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 monclient: found mon.noname-b 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff087fff640 1 -- 172.21.15.164:0/3373459188 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08809c560 con 0x7ff08809d160 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff08d789640 10 monclient: get_auth_request con 0x7ff08809f4c0 auth_method 0 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff08cf88640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.754+0000 7ff08d789640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 0x7ff08813b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff087fff640 1 -- 172.21.15.164:0/3373459188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 msgr2=0x7ff08809ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff087fff640 1 --2- 172.21.15.164:0/3373459188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 0x7ff08809ef80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff08809d5b0 tx=0x7ff07402a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2bd4420ac92726ca server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 1 -- 172.21.15.164:0/3373459188 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074037070 con 0x7ff08809d160 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: dump: 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.773 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 1 -- 172.21.15.164:0/3373459188 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff074032440 con 0x7ff08809d160 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff08f1ec640 5 monclient: authenticate success, global_id 114107 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff08f1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff0877fe640 1 -- 172.21.15.164:0/3373459188 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0740327f0 con 0x7ff08809d160 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.756+0000 7ff08f1ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 4 set_mon_vals no callback set 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff0877fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff0877fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff0877fe640 10 monclient: dump: 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.775 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 10 monclient: shutdown 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 -- 172.21.15.164:0/3373459188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 msgr2=0x7ff08809ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/3373459188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 0x7ff08809ef80 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff08809d5b0 tx=0x7ff07402a080 comp rx=0 tx=0).stop 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 -- 172.21.15.164:0/3373459188 shutdown_connections 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.165:3300/0 conn(0x7ff08809f4c0 0x7ff08813b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/3373459188 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809d160 0x7ff08809ef80 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/3373459188 >> v2:172.21.15.164:3300/0 conn(0x7ff08809c180 0x7ff08809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 -- 172.21.15.164:0/3373459188 >> 172.21.15.164:0/3373459188 conn(0x7ff0880a3110 msgr2=0x7ff0880a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 -- 172.21.15.164:0/3373459188 shutdown_connections 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.757+0000 7ff08f1ec640 1 -- 172.21.15.164:0/3373459188 wait complete. 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.758+0000 7ff08f1ec640 10 monclient: build_initial_monmap 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.758+0000 7ff08f1ec640 10 monclient: monmap: 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.758+0000 7ff08f1ec640 1 Processor -- start 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.758+0000 7ff08f1ec640 1 -- start start 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.758+0000 7ff08f1ec640 10 monclient: init 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff08809c180 0x7ff0881cacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-b con 0x7ff08809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:26.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 0x7ff0881cb1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-a con 0x7ff08809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): picked mon.noname-c con 0x7ff08809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff088074f90 con 0x7ff08809d160 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff088074e10 con 0x7ff08809c180 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff08813bb00 con 0x7ff08809f4c0 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08f1ec640 10 monclient(hunting): authenticate will time out at 1365.246199s 2024-09-10T16:16:26.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 0x7ff0881cb1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 0x7ff0881cb1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35470/0 (socket says 172.21.15.164:35470) 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 1 -- 172.21.15.164:0/1265262594 learned_addr learned my addr 172.21.15.164:0/1265262594 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): get_auth_request con 0x7ff08809d160 auth_method 0 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): get_auth_request con 0x7ff08809f4c0 auth_method 0 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff08809c180 0x7ff0881cacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): get_auth_request con 0x7ff08809c180 auth_method 0 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient(hunting): handle_auth_done global_id 114110 payload 824 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff08809c180 msgr2=0x7ff0881cacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff08809c180 0x7ff0881cacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 msgr2=0x7ff0881cb1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 0x7ff0881cb1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient: _finish_hunting 0 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 monclient: found mon.noname-c 2024-09-10T16:16:26.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff08d789640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08812e0a0 con 0x7ff08809f4c0 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.759+0000 7ff087fff640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff08809d160 0x7ff0881cb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff08d789640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 msgr2=0x7ff0881cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff08d789640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff08000cdf0 tx=0x7ff08000a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08001a070 con 0x7ff08809f4c0 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: dump: 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.783 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: _finish_auth 0 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: _check_auth_tickets 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:26.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff08000da50 con 0x7ff08809f4c0 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff08f1ec640 5 monclient: authenticate success, global_id 114110 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 monclient: _renew_subs 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.761+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff08000de40 con 0x7ff08809f4c0 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff08812e380 con 0x7ff08809f4c0 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff0857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff0857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff0857fa640 10 monclient: dump: 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff086ffd640 4 set_mon_vals no callback set 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 monclient: _renew_subs 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:26.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff08812d820 con 0x7ff08809f4c0 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08d789640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff08000cdf0 tx=0x7ff08000a710 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7ff080053480 osd_map(353..353 src has 184..353) 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff08812dd00 con 0x7ff0880736a0 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.762+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff08001d070 con 0x7ff08809f4c0 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient: get_auth_request con 0x7ff0880736a0 auth_method 0 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff0857fa640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff06c03e210 0x7ff06c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 10 monclient(hunting): handle_auth_done global_id 114113 payload 824 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff08809ea40 con 0x7ff0880736a0 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff087fff640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff06c03e210 0x7ff06c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff087fff640 10 monclient: get_auth_request con 0x7ff06c03e210 auth_method 0 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 msgr2=0x7ff088099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:16:26.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 msgr2=0x7ff088099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:26.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7ff07c0118f0 tx=0x7ff07c011db0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:26.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.763+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:26.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.765+0000 7ff087fff640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff06c03e210 0x7ff06c0406d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff08809e040 tx=0x7ff07402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.964+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: get_auth_request con 0x7ff0880736a0 auth_method 0 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: _init_auth method 2 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: _init_auth already have auth, reseting 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:26.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.965+0000 7ff08cf88640 10 monclient: handle_auth_done global_id 114113 payload 293 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 msgr2=0x7ff088099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff07c011df0 tx=0x7ff07c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c023030 con 0x7ff0880736a0 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 10 monclient: dump: 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff07c014650 con 0x7ff0880736a0 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 msgr2=0x7ff088099030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff07c011df0 tx=0x7ff07c009240 comp rx=0 tx=0).stop 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 msgr2=0x7ff088099030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:26.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.967+0000 7ff0857fa640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff0880736a0 0x7ff088099030 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff07c011df0 tx=0x7ff07c009240 comp rx=0 tx=0).stop 2024-09-10T16:16:26.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.973+0000 7ff08f1ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:26.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.973+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 0x7ff088099980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:26.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.973+0000 7ff08f1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.973+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff088099ec0 con 0x7ff088099570 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.973+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 0x7ff088099980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient: get_auth_request con 0x7ff088099570 auth_method 0 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 10 monclient(hunting): handle_auth_done global_id 114116 payload 824 2024-09-10T16:16:26.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.974+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff08809dc30 con 0x7ff088099570 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff08cf88640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 msgr2=0x7ff088099980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff08cf88640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 0x7ff088099980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff07c006ca0 tx=0x7ff07c004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07c00fa20 con 0x7ff088099570 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 10 monclient: dump: 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:26.978 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7ff07c00f980 con 0x7ff088099570 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 msgr2=0x7ff088099980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 0x7ff088099980 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff07c006ca0 tx=0x7ff07c004730 comp rx=0 tx=0).stop 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 msgr2=0x7ff088099980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.976+0000 7ff0857fa640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff088099570 0x7ff088099980 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff07c006ca0 tx=0x7ff07c004730 comp rx=0 tx=0).stop 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7544, 2024-09-10T16:16:26.980 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:26.981 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.982 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.983 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:26.984 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:26.985 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.986 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.988 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:26.989 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:26.990 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:26.991 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.992 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:26.993 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:26.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff06c03e210 msgr2=0x7ff06c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff06c03e210 0x7ff06c0406d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff08809e040 tx=0x7ff07402e000 comp rx=0 tx=0).stop 2024-09-10T16:16:26.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 10 monclient: shutdown 2024-09-10T16:16:26.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:26.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 msgr2=0x7ff0881cb730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08d789640 1 -- 172.21.15.164:0/1265262594 reap_dead start 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff08000cdf0 tx=0x7ff08000a710 comp rx=0 tx=0).stop 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff0857fa640 1 -- 172.21.15.164:0/1265262594 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(353..353 src has 184..353) ==== 4755+0+0 (unknown 0 0 0) 0x7ff080053480 con 0x7ff08809f4c0 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 shutdown_connections 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 --2- 172.21.15.164:0/1265262594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff08809f4c0 0x7ff0881cb730 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 >> 172.21.15.164:0/1265262594 conn(0x7ff0880a3110 msgr2=0x7ff0881386e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 shutdown_connections 2024-09-10T16:16:26.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:26.977+0000 7ff08f1ec640 1 -- 172.21.15.164:0/1265262594 wait complete. 2024-09-10T16:16:26.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:26.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:16:26.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:16:27.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.093+0000 7fac1fe95640 10 monclient: get_monmap_and_config 2024-09-10T16:16:27.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.093+0000 7fac1fe95640 10 monclient: build_initial_monmap 2024-09-10T16:16:27.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.093+0000 7fac1fe95640 10 monclient: monmap: 2024-09-10T16:16:27.095 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:27.094477+0000 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:27.094477+0000 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:27.096 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 Processor -- start 2024-09-10T16:16:27.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 -- start start 2024-09-10T16:16:27.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient: init 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac1809c180 0x7fac1809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-a con 0x7fac1809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac1809d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fac1813a5c0 0x7fac1813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-c con 0x7fac1813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac1809a430 con 0x7fac1809d160 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fac180a24e0 con 0x7fac1809c180 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fac180a2bf0 con 0x7fac1813a5c0 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1fe95640 10 monclient(hunting): authenticate will time out at 1365.581860s 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.094+0000 7fac1dc31640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac1809c180 0x7fac1809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fac1809c180 0x7fac1809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35482/0 (socket says 172.21.15.164:35482) 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 -- 172.21.15.164:0/741201487 learned_addr learned my addr 172.21.15.164:0/741201487 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): get_auth_request con 0x7fac1809c180 auth_method 0 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 1 --2- 172.21.15.164:0/741201487 >> v2:172.21.15.165:3300/0 conn(0x7fac1813a5c0 0x7fac1813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): get_auth_request con 0x7fac1813a5c0 auth_method 0 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient(hunting): handle_auth_done global_id 74511 payload 274 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 -- 172.21.15.164:0/741201487 >> v2:172.21.15.165:3300/0 conn(0x7fac1813a5c0 msgr2=0x7fac1813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 --2- 172.21.15.164:0/741201487 >> v2:172.21.15.165:3300/0 conn(0x7fac1813a5c0 0x7fac1813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 -- 172.21.15.164:0/741201487 >> v1:172.21.15.165:6789/0 conn(0x7fac1809d160 legacy=0x7fac1809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient: _finish_hunting 0 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 monclient: found mon.noname-a 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.095+0000 7fac1dc31640 1 -- 172.21.15.164:0/741201487 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac1809c560 con 0x7fac1809c180 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1dc31640 1 -- 172.21.15.164:0/741201487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809c180 msgr2=0x7fac1809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1dc31640 1 --2- 172.21.15.164:0/741201487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809c180 0x7fac1809ea40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fac14009870 tx=0x7fac1402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c75c7659f2193e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 1 -- 172.21.15.164:0/741201487 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14037070 con 0x7fac1809c180 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: dump: 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: _finish_auth 0 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: _check_auth_tickets 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 1 -- 172.21.15.164:0/741201487 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac14032640 con 0x7fac1809c180 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:27.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 1 -- 172.21.15.164:0/741201487 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14032a10 con 0x7fac1809c180 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 5 monclient: authenticate success, global_id 74511 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1cc2f640 10 monclient: dump: 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 4 set_mon_vals no callback set 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 10 monclient: shutdown 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 -- 172.21.15.164:0/741201487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809c180 msgr2=0x7fac1809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 --2- 172.21.15.164:0/741201487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809c180 0x7fac1809ea40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fac14009870 tx=0x7fac1402a100 comp rx=0 tx=0).stop 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 -- 172.21.15.164:0/741201487 shutdown_connections 2024-09-10T16:16:27.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 --2- 172.21.15.164:0/741201487 >> v2:172.21.15.165:3300/0 conn(0x7fac1813a5c0 0x7fac1813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 --2- 172.21.15.164:0/741201487 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809c180 0x7fac1809ea40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 -- 172.21.15.164:0/741201487 >> 172.21.15.164:0/741201487 conn(0x7fac180a3110 msgr2=0x7fac180a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 -- 172.21.15.164:0/741201487 shutdown_connections 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.098+0000 7fac1fe95640 1 -- 172.21.15.164:0/741201487 wait complete. 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.099+0000 7fac1fe95640 10 monclient: build_initial_monmap 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.099+0000 7fac1fe95640 10 monclient: monmap: 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.114 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.099+0000 7fac1fe95640 1 Processor -- start 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.099+0000 7fac1fe95640 1 -- start start 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.099+0000 7fac1fe95640 10 monclient: init 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1809c180 0x7fac18099380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-b con 0x7fac1809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809d160 0x7fac180998c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-a con 0x7fac1809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 0x7fac1808df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): picked mon.noname-c con 0x7fac1813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fac18074f90 con 0x7fac1809d160 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac18074e10 con 0x7fac1809c180 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fac1809f690 con 0x7fac1813a5c0 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1fe95640 10 monclient(hunting): authenticate will time out at 1365.587173s 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 0x7fac1808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809d160 0x7fac180998c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 0x7fac1808df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60480/0 (socket says 172.21.15.164:60480) 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): get_auth_request con 0x7fac1809d160 auth_method 0 2024-09-10T16:16:27.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 learned_addr learned my addr 172.21.15.164:0/3474727601 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): get_auth_request con 0x7fac1813a5c0 auth_method 0 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1809c180 0x7fac18099380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): get_auth_request con 0x7fac1809c180 auth_method 0 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient(hunting): handle_auth_done global_id 114119 payload 824 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1809c180 msgr2=0x7fac18099380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1809c180 0x7fac18099380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809d160 msgr2=0x7fac180998c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809d160 0x7fac180998c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient: _finish_hunting 0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 monclient: found mon.noname-c 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac18074170 con 0x7fac1813a5c0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.100+0000 7fac1d430640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fac1809d160 0x7fac180998c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 msgr2=0x7fac1808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac1e432640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 0x7fac1808df80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e30 tx=0x7fac1000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10018020 con 0x7fac1813a5c0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 10 monclient: dump: 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.119 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.102+0000 7fac0e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 5 monclient: authenticate success, global_id 114119 2024-09-10T16:16:27.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 monclient: _renew_subs 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac100146a0 con 0x7fac1813a5c0 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac1809efd0 con 0x7fac1813a5c0 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac10014a90 con 0x7fac1813a5c0 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 10 monclient: dump: 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.121 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 monclient: _renew_subs 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0ffff640 4 set_mon_vals no callback set 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fac18090330 con 0x7fac1813a5c0 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 0x7fac1808e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac1808ede0 con 0x7fac1808e4c0 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.103+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fac10005d10 con 0x7fac1813a5c0 2024-09-10T16:16:27.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 0x7fac1808e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient: get_auth_request con 0x7fac1808e4c0 auth_method 0 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac0e7fc640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fabfc03e210 0x7fabfc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(353..353 src has 184..353) ==== 4755+0+0 (secure 0 0 0) 0x7fac10053c50 con 0x7fac1813a5c0 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 10 monclient(hunting): handle_auth_done global_id 94276 payload 824 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1dc31640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac1809d930 con 0x7fac1808e4c0 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1d430640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fabfc03e210 0x7fabfc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.104+0000 7fac1d430640 10 monclient: get_auth_request con 0x7fabfc03e210 auth_method 0 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac1dc31640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 msgr2=0x7fac1808e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac1dc31640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 0x7fac1808e8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fac14034000 tx=0x7fac140046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14037040 con 0x7fac1808e4c0 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac0e7fc640 10 monclient: dump: 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.126 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.106+0000 7fac1d430640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fabfc03e210 0x7fabfc0406d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac1809a010 tx=0x7fac08007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fac1403c610 con 0x7fac1808e4c0 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:27.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 msgr2=0x7fac1808e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 0x7fac1808e8a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fac14034000 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 msgr2=0x7fac1808e8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.107+0000 7fac0e7fc640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1808e4c0 0x7fac1808e8a0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fac14034000 tx=0x7fac140046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1fe95640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1fe95640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 0x7fac1806fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1fe95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fac1808f3d0 con 0x7fac1806f7e0 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 0x7fac1806fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient: get_auth_request con 0x7fac1806f7e0 auth_method 0 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 10 monclient(hunting): handle_auth_done global_id 94279 payload 824 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.113+0000 7fac1dc31640 1 -- 172.21.15.164:0/3474727601 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fac18077260 con 0x7fac1806f7e0 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1dc31640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 msgr2=0x7fac1806fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1dc31640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 0x7fac1806fbf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fac1402a140 tx=0x7fac14008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac14037040 con 0x7fac1806f7e0 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 10 monclient: dump: 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.129 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fac1403ce00 con 0x7fac1806f7e0 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 msgr2=0x7fac1806fbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 0x7fac1806fbf0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fac1402a140 tx=0x7fac14008030 comp rx=0 tx=0).stop 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 msgr2=0x7fac1806fbf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:27.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac0e7fc640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fac1806f7e0 0x7fac1806fbf0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fac1402a140 tx=0x7fac14008030 comp rx=0 tx=0).stop 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 104616, 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:27.131 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:27.132 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:27.133 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:27.134 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:27.135 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:27.136 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.137 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:27.138 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:27.139 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:27.141 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:27.142 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:27.143 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fabfc03e210 msgr2=0x7fabfc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fabfc03e210 0x7fabfc0406d0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fac1809a010 tx=0x7fac08007e30 comp rx=0 tx=0).stop 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 10 monclient: shutdown 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 msgr2=0x7fac1808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 --2- 172.21.15.164:0/3474727601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fac1813a5c0 0x7fac1808df80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fac10009e30 tx=0x7fac1000c490 comp rx=0 tx=0).stop 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 reap_dead start 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1e432640 1 -- 172.21.15.164:0/3474727601 reap_dead start 2024-09-10T16:16:27.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 shutdown_connections 2024-09-10T16:16:27.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.116+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 >> 172.21.15.164:0/3474727601 conn(0x7fac180a3110 msgr2=0x7fac1813c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:27.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.117+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 shutdown_connections 2024-09-10T16:16:27.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.117+0000 7fac1fe95640 1 -- 172.21.15.164:0/3474727601 wait complete. 2024-09-10T16:16:27.146 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:27.146 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:16:27.146 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1470: done (7 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1471: done (6 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1472: done (5 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1473: done (4 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1474: done (3 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1476: done (2 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1477: done (1 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1478: done (0 left) 2024-09-10T16:16:27.179 INFO:tasks.rados.rados.0.smithi165.stdout:1479: delete oid 20 current snap is 164 2024-09-10T16:16:27.186 INFO:tasks.rados.rados.0.smithi165.stdout:1479: done (0 left) 2024-09-10T16:16:27.186 INFO:tasks.rados.rados.0.smithi165.stdout:1480: snap_create 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.246+0000 7fc87a28c640 10 monclient: get_monmap_and_config 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.246+0000 7fc87a28c640 10 monclient: build_initial_monmap 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.246+0000 7fc87a28c640 10 monclient: monmap: 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:27.247487+0000 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:27.247487+0000 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:27.249 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:27.249 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:27.249 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:27.249 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:27.249 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 1 Processor -- start 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 1 -- start start 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 10 monclient: init 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc87409d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc87409ab50 0x7fc8741397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-a con 0x7fc87409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.247+0000 7fc87a28c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc87409e420 0x7fc87413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-c con 0x7fc87409e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8740a0870 con 0x7fc87409d930 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc87409d3e0 con 0x7fc87409ab50 2024-09-10T16:16:27.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc874074c70 con 0x7fc87409e420 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc87a28c640 10 monclient(hunting): authenticate will time out at 1365.735043s 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc87409ab50 0x7fc8741397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc87409ab50 0x7fc8741397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35512/0 (socket says 172.21.15.164:35512) 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 -- 172.21.15.164:0/767265698 learned_addr learned my addr 172.21.15.164:0/767265698 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): get_auth_request con 0x7fc87409ab50 auth_method 0 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 1 --2- 172.21.15.164:0/767265698 >> v2:172.21.15.165:3300/0 conn(0x7fc87409e420 0x7fc87413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): get_auth_request con 0x7fc87409e420 auth_method 0 2024-09-10T16:16:27.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc878829640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient(hunting): handle_auth_done global_id 74517 payload 274 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 -- 172.21.15.164:0/767265698 >> v2:172.21.15.165:3300/0 conn(0x7fc87409e420 msgr2=0x7fc87413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 --2- 172.21.15.164:0/767265698 >> v2:172.21.15.165:3300/0 conn(0x7fc87409e420 0x7fc87413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 -- 172.21.15.164:0/767265698 >> v1:172.21.15.165:6789/0 conn(0x7fc87409d930 legacy=0x7fc87409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:27.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:16:27.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 monclient: found mon.noname-a 2024-09-10T16:16:27.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:27.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:27.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.248+0000 7fc86bfff640 1 -- 172.21.15.164:0/767265698 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc87409ecb0 con 0x7fc87409ab50 2024-09-10T16:16:27.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc86bfff640 1 -- 172.21.15.164:0/767265698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409ab50 msgr2=0x7fc8741397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc86bfff640 1 --2- 172.21.15.164:0/767265698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409ab50 0x7fc8741397c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc87409af60 tx=0x7fc85c02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa0e3db2f1b32a5e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 1 -- 172.21.15.164:0/767265698 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c037070 con 0x7fc87409ab50 2024-09-10T16:16:27.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: dump: 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.261 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: _finish_auth 0 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: _check_auth_tickets 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 1 -- 172.21.15.164:0/767265698 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c032640 con 0x7fc87409ab50 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 5 monclient: authenticate success, global_id 74517 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 1 -- 172.21.15.164:0/767265698 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c032a10 con 0x7fc87409ab50 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 4 set_mon_vals no callback set 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:27.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc8737fe640 10 monclient: dump: 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 10 monclient: shutdown 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 -- 172.21.15.164:0/767265698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409ab50 msgr2=0x7fc8741397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 --2- 172.21.15.164:0/767265698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409ab50 0x7fc8741397c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fc87409af60 tx=0x7fc85c02a0b0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 -- 172.21.15.164:0/767265698 shutdown_connections 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 --2- 172.21.15.164:0/767265698 >> v2:172.21.15.165:3300/0 conn(0x7fc87409e420 0x7fc87413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 --2- 172.21.15.164:0/767265698 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409ab50 0x7fc8741397c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 -- 172.21.15.164:0/767265698 >> 172.21.15.164:0/767265698 conn(0x7fc8740a3110 msgr2=0x7fc8740a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 -- 172.21.15.164:0/767265698 shutdown_connections 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.251+0000 7fc87a28c640 1 -- 172.21.15.164:0/767265698 wait complete. 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.252+0000 7fc87a28c640 10 monclient: build_initial_monmap 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.252+0000 7fc87a28c640 10 monclient: monmap: 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:27.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.252+0000 7fc87a28c640 1 Processor -- start 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.252+0000 7fc87a28c640 1 -- start start 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.252+0000 7fc87a28c640 10 monclient: init 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 0x7fc874137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-c con 0x7fc87409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-a con 0x7fc87409d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 0x7fc874137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): picked mon.noname-b con 0x7fc87409e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc8740a2940 con 0x7fc87409d930 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc87409ef30 con 0x7fc87409e420 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc8740a2ac0 con 0x7fc87409ab50 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc87a28c640 10 monclient(hunting): authenticate will time out at 1365.740378s 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35526/0 (socket says 172.21.15.164:35526) 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 0x7fc874137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 1 -- 172.21.15.164:0/361766076 learned_addr learned my addr 172.21.15.164:0/361766076 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): get_auth_request con 0x7fc87409d930 auth_method 0 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): get_auth_request con 0x7fc87409ab50 auth_method 0 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:27.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc878829640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 0x7fc874137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient(hunting): handle_auth_done global_id 114122 payload 824 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 -- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 msgr2=0x7fc874137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 0x7fc874137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 -- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 msgr2=0x7fc8741377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient: _finish_hunting 0 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 monclient: found mon.noname-c 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc873fff640 1 -- 172.21.15.164:0/361766076 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8741393a0 con 0x7fc87409ab50 2024-09-10T16:16:27.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc878829640 10 monclient: get_auth_request con 0x7fc87409e420 auth_method 0 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc86bfff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.253+0000 7fc878829640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 0x7fc874137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.255+0000 7fc873fff640 1 -- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 msgr2=0x7fc874137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.255+0000 7fc873fff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 0x7fc874137260 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c410 tx=0x7fc86400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc864017070 con 0x7fc87409ab50 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: dump: 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.269 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: _finish_auth 0 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: _check_auth_tickets 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc864005230 con 0x7fc87409ab50 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 5 monclient: authenticate success, global_id 114122 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8640117a0 con 0x7fc87409ab50 2024-09-10T16:16:27.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 10 monclient: dump: 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:27.271 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 10 monclient: _renew_subs 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc872ffd640 4 set_mon_vals no callback set 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc87409b1b0 con 0x7fc87409ab50 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 10 monclient: _renew_subs 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc87413acd0 con 0x7fc87409ab50 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.256+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc86401a070 con 0x7fc87409ab50 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc87a28c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc87a28c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:27.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc87a28c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc87409dd10 con 0x7fc87409ab50 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc8717fa640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 0x7fc8540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(354..354 src has 184..354) ==== 4783+0+0 (secure 0 0 0) 0x7fc864051fc0 con 0x7fc87409ab50 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc86bfff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 0x7fc8540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.257+0000 7fc86bfff640 10 monclient: get_auth_request con 0x7fc85403e210 auth_method 0 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.259+0000 7fc86bfff640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 0x7fc8540406d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc85c02a0f0 tx=0x7fc85c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:27.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.261+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc8640509c0 con 0x7fc87409ab50 2024-09-10T16:16:27.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.261+0000 7fc8717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:27.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.261+0000 7fc8717fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.503+0000 7fc87a28c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:16:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.503+0000 7fc87a28c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:16:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.503+0000 7fc87a28c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.503+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc87406dac0 con 0x7fc87409ab50 2024-09-10T16:16:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.504+0000 7fc8717fa640 1 -- 172.21.15.164:0/361766076 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc864004b00 con 0x7fc87409ab50 2024-09-10T16:16:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.504+0000 7fc8717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:16:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.504+0000 7fc8717fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:27.506 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=4041075812,config=2586702516,health=555194467,kv=805391194,logm=959562735}) 2024-09-10T16:16:27.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=11,kv=12,logm=57} crc {auth=3008573522,config=2586702516,health=555194467,kv=805391194,logm=198349453}) 2024-09-10T16:16:27.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=4041075812,config=2586702516,health=555194467,kv=805391194,logm=959562735}) 2024-09-10T16:16:27.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.505+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=18,config=2,health=11,kv=12,logm=57} crc {auth=4009361680,config=2586702516,health=2267228605,kv=805391194,logm=1664452771}) 2024-09-10T16:16:27.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704818020}) 2024-09-10T16:16:27.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1577754808}) 2024-09-10T16:16:27.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704818020}) 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.506+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=468935152}) 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172569698}) 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3430392100}) 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172569698}) 2024-09-10T16:16:27.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.507+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3931509684}) 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1564791864}) 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2346712708}) 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1564791864}) 2024-09-10T16:16:27.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2989267574}) 2024-09-10T16:16:27.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 msgr2=0x7fc8540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 0x7fc8540406d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc85c02a0f0 tx=0x7fc85c02e000 comp rx=0 tx=0).stop 2024-09-10T16:16:27.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 10 monclient: shutdown 2024-09-10T16:16:27.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:27.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 msgr2=0x7fc874137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 0x7fc874137260 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc86400c410 tx=0x7fc86400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 shutdown_connections 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc85403e210 0x7fc8540406d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc87409e420 0x7fc874137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc87409d930 0x7fc8741377a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 --2- 172.21.15.164:0/361766076 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc87409ab50 0x7fc874137260 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 >> 172.21.15.164:0/361766076 conn(0x7fc8740a3110 msgr2=0x7fc8740707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 shutdown_connections 2024-09-10T16:16:27.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fc87a28c640 1 -- 172.21.15.164:0/361766076 wait complete. 2024-09-10T16:16:27.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309705469}) 2024-09-10T16:16:27.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=72639280}) 2024-09-10T16:16:27.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.508+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2309705469}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780450377}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331062967}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639569798}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331062967}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.509+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1107197749}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=262693615}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1177032581}) 2024-09-10T16:16:27.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=262693615}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.510+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995891190}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967114272}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3971537572}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1967114272}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4040235849}) 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198281081}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1391471881}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198281081}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.511+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115139697}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728526089}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2693607595}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728526089}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.512+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3089901120}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415465297}) 2024-09-10T16:16:27.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=534694783}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3415465297}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=610123919}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053469732}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1258220102}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1053469732}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.513+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2336637233}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502472831}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2818684525}) 2024-09-10T16:16:27.521 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502472831}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038181646}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580090725}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2076961444}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.514+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=580090725}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218350198}) 2024-09-10T16:16:27.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159711622}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3462490626}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2159711622}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.515+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3688427374}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694602941}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=872316489}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3694602941}) 2024-09-10T16:16:27.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=305606176}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812093190}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2383933922}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=812093190}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.516+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1680170666}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361656687}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=620165973}) 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.526 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361656687}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.517+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1089600491}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712344322}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1603055791}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712344322}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.518+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2538629169}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919015479}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2986735146}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=919015479}) 2024-09-10T16:16:27.527 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035267027}) 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907101522}) 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4069641106}) 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1907101522}) 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.519+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1200805618}) 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.528 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041809659}) 2024-09-10T16:16:27.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1339740665}) 2024-09-10T16:16:27.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1041809659}) 2024-09-10T16:16:27.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.520+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442032897}) 2024-09-10T16:16:27.530 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502597004}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1290359909}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3502597004}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.521+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2205328770}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865551908}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=44014137}) 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865551908}) 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.522+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=390845517}) 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004007368}) 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4054026298}) 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004007368}) 2024-09-10T16:16:27.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.529+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3885795987}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821026538}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3494500002}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1821026538}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.530+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1298633000}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384568293}) 2024-09-10T16:16:27.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503067022}) 2024-09-10T16:16:27.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=384568293}) 2024-09-10T16:16:27.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.531+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=957903776}) 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=4113733026,mdsmap=931935893,mgr=1154067016}) 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=1743967758,mdsmap=931935893,mgr=2354521040}) 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=89,mdsmap=3,mgr=8} crc {logm=4113733026,mdsmap=931935893,mgr=1154067016}) 2024-09-10T16:16:27.588 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:27.586+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=91,mdsmap=3,mgr=6} crc {logm=3019049533,mdsmap=931935893,mgr=3606505178}) 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1839124478,mon_config_key=1189190047,monmap=1433040829,nvmeofgw=598337687,osd_metadata=3297165161,osd_pg_creating=4184209182,osd_snap=3051568709}) 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=57} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=311829010,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4124331855,osd_pg_creating=4184209182,osd_snap=622979638}) 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1839124478,mon_config_key=1189190047,monmap=1433040829,nvmeofgw=598337687,osd_metadata=3297165161,osd_pg_creating=4184209182,osd_snap=3051568709}) 2024-09-10T16:16:28.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.179+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=58} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4071078268,mon_config_key=2240728274,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=4184209182,osd_snap=2061445346}) 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1949410797,osdmap=776355591}) 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1949410798,osdmap=1560119052}) 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1949410797,osdmap=776355591}) 2024-09-10T16:16:28.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.184+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=1,osdmap=99} crc {osd_snap=1949410797,osdmap=1586256545}) 2024-09-10T16:16:28.186 INFO:tasks.rados.rados.0.smithi165.stdout:1480: done (0 left) 2024-09-10T16:16:28.186 INFO:tasks.rados.rados.0.smithi165.stdout:1481: delete oid 36 current snap is 165 2024-09-10T16:16:28.190 INFO:tasks.rados.rados.0.smithi165.stdout:1481: done (0 left) 2024-09-10T16:16:28.190 INFO:tasks.rados.rados.0.smithi165.stdout:1482: snap_create 2024-09-10T16:16:28.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=925045777}) 2024-09-10T16:16:28.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3043655519}) 2024-09-10T16:16:28.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=925045777}) 2024-09-10T16:16:28.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.189+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1477123925}) 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=71} crc {osdmap=3335619991}) 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=69} crc {osdmap=3194382524}) 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=71} crc {osdmap=3335619991}) 2024-09-10T16:16:28.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:16:28.196+0000 7fa3d3697640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=758740939}) 2024-09-10T16:16:28.530 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:16:28.531 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:28.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 10 monclient: get_monmap_and_config 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 10 monclient: build_initial_monmap 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 10 monclient: monmap: 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:28.640162+0000 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:28.640162+0000 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:28.641 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:28.642 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:28.642 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:28.642 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 1 Processor -- start 2024-09-10T16:16:28.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 1 -- start start 2024-09-10T16:16:28.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.639+0000 7f71cb59e640 10 monclient: init 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f71cc09d680 0x7f71cc09da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc09d680 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f71cc06d1f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:28.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71cc06d5d0 0x7f71cc1380c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc06d5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc0ab520 con 0x7f71cc06d5d0 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc06e110 con 0x7f71cc09d680 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc06e3f0 con 0x7f71cc06d1f0 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cb59e640 10 monclient(hunting): authenticate will time out at 1367.127592s 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71cc06d5d0 0x7f71cc1380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60496/0 (socket says 172.21.15.164:60496) 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f71cc06d5d0 0x7f71cc1380c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35532/0 (socket says 172.21.15.164:35532) 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 1 -- 172.21.15.164:0/2163571392 learned_addr learned my addr 172.21.15.164:0/2163571392 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): get_auth_request con 0x7f71cc06d1f0 auth_method 0 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): get_auth_request con 0x7f71cc06d5d0 auth_method 0 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71ca59c640 1 --2- 172.21.15.164:0/2163571392 >> v2:172.21.15.165:3300/0 conn(0x7f71cc09d680 0x7f71cc09da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71ca59c640 10 monclient(hunting): get_auth_request con 0x7f71cc09d680 auth_method 0 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.640+0000 7f71ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 10 monclient(hunting): handle_auth_done global_id 74523 payload 274 2024-09-10T16:16:28.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 -- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 msgr2=0x7f71cc137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 -- 172.21.15.164:0/2163571392 >> v2:172.21.15.165:3300/0 conn(0x7f71cc09d680 msgr2=0x7f71cc09da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 --2- 172.21.15.164:0/2163571392 >> v2:172.21.15.165:3300/0 conn(0x7f71cc09d680 0x7f71cc09da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 monclient: found mon.noname-a 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71cad9d640 1 -- 172.21.15.164:0/2163571392 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc09dfa0 con 0x7f71cc06d5d0 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.641+0000 7f71c9d9b640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.643+0000 7f71cad9d640 1 -- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc06d5d0 msgr2=0x7f71cc1380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.643+0000 7f71cad9d640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc06d5d0 0x7f71cc1380c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a8080 tx=0x7f71c402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=146ced2624d75596 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 1 -- 172.21.15.164:0/2163571392 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4007950 con 0x7f71cc06d5d0 2024-09-10T16:16:28.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: dump: 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:28.658 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: _finish_auth 0 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: _check_auth_tickets 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 5 monclient: authenticate success, global_id 74523 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 1 -- 172.21.15.164:0/2163571392 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4007af0 con 0x7f71cc06d5d0 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:28.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:28.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 4 set_mon_vals no callback set 2024-09-10T16:16:28.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 1 -- 172.21.15.164:0/2163571392 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4031400 con 0x7f71cc06d5d0 2024-09-10T16:16:28.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71c959a640 10 monclient: dump: 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 10 monclient: shutdown 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 -- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc06d5d0 msgr2=0x7f71cc1380c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc06d5d0 0x7f71cc1380c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a8080 tx=0x7f71c402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 -- 172.21.15.164:0/2163571392 shutdown_connections 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc06d5d0 0x7f71cc1380c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 --2- 172.21.15.164:0/2163571392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d1f0 0x7f71cc137b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 --2- 172.21.15.164:0/2163571392 >> v2:172.21.15.165:3300/0 conn(0x7f71cc09d680 0x7f71cc09da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 -- 172.21.15.164:0/2163571392 >> 172.21.15.164:0/2163571392 conn(0x7f71cc0a70e0 msgr2=0x7f71cc0a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 -- 172.21.15.164:0/2163571392 shutdown_connections 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.644+0000 7f71cb59e640 1 -- 172.21.15.164:0/2163571392 wait complete. 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.645+0000 7f71cb59e640 10 monclient: build_initial_monmap 2024-09-10T16:16:28.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.645+0000 7f71cb59e640 10 monclient: monmap: 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.663 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.645+0000 7f71cb59e640 1 Processor -- start 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.645+0000 7f71cb59e640 1 -- start start 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.645+0000 7f71cb59e640 10 monclient: init 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71cc06d1f0 0x7f71cc1cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-b con 0x7f71cc06d1f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 0x7f71cc1349f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc06d5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 0x7f71cc134f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc09d680 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc138780 con 0x7f71cc09d680 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc138600 con 0x7f71cc06d1f0 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc138900 con 0x7f71cc06d5d0 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cb59e640 10 monclient(hunting): authenticate will time out at 1367.133428s 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 0x7f71cc1349f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 0x7f71cc134f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 0x7f71cc1349f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60510/0 (socket says 172.21.15.164:60510) 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 1 -- 172.21.15.164:0/3124329467 learned_addr learned my addr 172.21.15.164:0/3124329467 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): get_auth_request con 0x7f71cc09d680 auth_method 0 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): get_auth_request con 0x7f71cc06d5d0 auth_method 0 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71cc06d1f0 0x7f71cc1cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): get_auth_request con 0x7f71cc06d1f0 auth_method 0 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient(hunting): handle_auth_done global_id 74526 payload 824 2024-09-10T16:16:28.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 -- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 msgr2=0x7f71cc1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 0x7f71cc1349f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 -- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71cc06d1f0 msgr2=0x7f71cc1cd0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71cc06d1f0 0x7f71cc1cd0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 monclient: found mon.noname-a 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.646+0000 7f71cad9d640 1 -- 172.21.15.164:0/3124329467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc1c1690 con 0x7f71cc09d680 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cad9d640 1 -- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 msgr2=0x7f71cc134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cad9d640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 0x7f71cc134f30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f71c4031000 tx=0x7f71c4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4040070 con 0x7f71cc09d680 2024-09-10T16:16:28.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: dump: 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: _finish_auth 0 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c4045460 con 0x7f71cc09d680 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 5 monclient: authenticate success, global_id 74526 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 10 monclient: _renew_subs 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc0a8cc0 con 0x7f71cc09d680 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c4007950 con 0x7f71cc09d680 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71baffd640 10 monclient: dump: 2024-09-10T16:16:28.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 10 monclient: _renew_subs 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71cc0a17d0 con 0x7f71cc09d680 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.649+0000 7f71c8d99640 4 set_mon_vals no callback set 2024-09-10T16:16:28.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f71c4039040 con 0x7f71cc09d680 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71cb59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71cb59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71cb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71cc09da60 con 0x7f71cc09d680 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71baffd640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 0x7f71ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7f71c4074560 con 0x7f71cc09d680 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71ca59c640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 0x7f71ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:28.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.650+0000 7f71ca59c640 10 monclient: get_auth_request con 0x7f71ac03e210 auth_method 0 2024-09-10T16:16:28.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.653+0000 7f71ca59c640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 0x7f71ac0406d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009ac0 tx=0x7f71bc009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:28.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.655+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f71cc0a8cc0 con 0x7f71cc09d680 2024-09-10T16:16:28.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.655+0000 7f71baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:28.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.655+0000 7f71baffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:28.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71cb59e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:28.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71cb59e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71cb59e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71cc1bfe90 con 0x7f71cc09d680 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71baffd640 1 -- 172.21.15.164:0/3124329467 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f71c4072f80 con 0x7f71cc09d680 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.897+0000 7f71baffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:28.899 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 msgr2=0x7f71ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 0x7f71ac0406d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f71bc009ac0 tx=0x7f71bc009190 comp rx=0 tx=0).stop 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 10 monclient: shutdown 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 msgr2=0x7f71cc134f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 0x7f71cc134f30 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f71c4031000 tx=0x7f71c4004a40 comp rx=0 tx=0).stop 2024-09-10T16:16:28.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 shutdown_connections 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71ac03e210 0x7f71ac0406d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f71cc09d680 0x7f71cc134f30 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f71cc06d5d0 0x7f71cc1349f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 --2- 172.21.15.164:0/3124329467 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f71cc06d1f0 0x7f71cc1cd0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 >> 172.21.15.164:0/3124329467 conn(0x7f71cc0a70e0 msgr2=0x7f71cc09c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.899+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 shutdown_connections 2024-09-10T16:16:28.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:28.900+0000 7f71cb59e640 1 -- 172.21.15.164:0/3124329467 wait complete. 2024-09-10T16:16:28.912 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:16:28.912 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:16:28.912 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:16:28.912 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:16:29.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.009+0000 7f81cc994640 10 monclient: get_monmap_and_config 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.009+0000 7f81cc994640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.009+0000 7f81cc994640 10 monclient: monmap: 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:29.010450+0000 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:29.010450+0000 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:29.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:29.012 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:29.012 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:29.012 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 Processor -- start 2024-09-10T16:16:29.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 -- start start 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient: init 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81c809d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f81c809ab50 0x7f81c81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-c con 0x7f81c809ab50 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81c809e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c80a0870 con 0x7f81c809e420 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81c8074c70 con 0x7f81c809d930 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f81c809d3e0 con 0x7f81c809ab50 2024-09-10T16:16:29.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81cc994640 10 monclient(hunting): authenticate will time out at 1367.497810s 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81c6d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f81c809e420 0x7f81c813bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52518/0 (socket says 172.21.15.164:52518) 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.010+0000 7f81c6d9d640 1 -- 172.21.15.164:0/4158450746 learned_addr learned my addr 172.21.15.164:0/4158450746 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 1 --2- 172.21.15.164:0/4158450746 >> v2:172.21.15.165:3300/0 conn(0x7f81c809ab50 0x7f81c81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): get_auth_request con 0x7f81c809ab50 auth_method 0 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.011+0000 7f81c5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c6d9d640 1 -- 172.21.15.164:0/4158450746 >> v1:172.21.15.164:6789/0 conn(0x7f81c809e420 legacy=0x7f81c813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c659c640 1 -- 172.21.15.164:0/4158450746 >> v1:172.21.15.165:6789/0 conn(0x7f81c809d930 legacy=0x7f81c809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 1 -- 172.21.15.164:0/4158450746 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3767249622 0 0) 0x7f81c80a0870 con 0x7f81c809e420 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 10 monclient(hunting): _negotiate 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 10 monclient(hunting): my global_id is 74529 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c559a640 1 -- 172.21.15.164:0/4158450746 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81b4003520 con 0x7f81c809e420 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 10 monclient(hunting): handle_auth_done global_id 94291 payload 274 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 1 -- 172.21.15.164:0/4158450746 >> v1:172.21.15.165:6789/0 conn(0x7f81c809d930 legacy=0x7f81c809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 1 -- 172.21.15.164:0/4158450746 >> v1:172.21.15.164:6789/0 conn(0x7f81c809e420 legacy=0x7f81c813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 1 monclient: found mon.noname-c 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:16:29.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.013+0000 7f81c5d9b640 1 -- 172.21.15.164:0/4158450746 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c809ecb0 con 0x7f81c809ab50 2024-09-10T16:16:29.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c5d9b640 1 -- 172.21.15.164:0/4158450746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809ab50 msgr2=0x7f81c81397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c5d9b640 1 --2- 172.21.15.164:0/4158450746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809ab50 0x7f81c81397c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81b00098a0 tx=0x7f81b002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65d237f48167fadd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 1 -- 172.21.15.164:0/4158450746 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0037070 con 0x7f81c809ab50 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: dump: 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.025 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 5 monclient: authenticate success, global_id 94291 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 1 -- 172.21.15.164:0/4158450746 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81b0032640 con 0x7f81c809ab50 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 4 set_mon_vals no callback set 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 1 -- 172.21.15.164:0/4158450746 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b0032a10 con 0x7f81c809ab50 2024-09-10T16:16:29.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81c559a640 10 monclient: dump: 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 10 monclient: shutdown 2024-09-10T16:16:29.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 1 -- 172.21.15.164:0/4158450746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809ab50 msgr2=0x7f81c81397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.016+0000 7f81cc994640 1 --2- 172.21.15.164:0/4158450746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809ab50 0x7f81c81397c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f81b00098a0 tx=0x7f81b002a210 comp rx=0 tx=0).stop 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 1 -- 172.21.15.164:0/4158450746 shutdown_connections 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 1 --2- 172.21.15.164:0/4158450746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809ab50 0x7f81c81397c0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 1 -- 172.21.15.164:0/4158450746 >> 172.21.15.164:0/4158450746 conn(0x7f81c80a3110 msgr2=0x7f81c80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 1 -- 172.21.15.164:0/4158450746 shutdown_connections 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 1 -- 172.21.15.164:0/4158450746 wait complete. 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.017+0000 7f81cc994640 10 monclient: monmap: 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:29.028 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 Processor -- start 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 -- start start 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient: init 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:16:29.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 0x7f81c81ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-a con 0x7f81c809ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809d930 0x7f81c81caa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-b con 0x7f81c809d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 0x7f81c81373d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): picked mon.noname-c con 0x7f81c809e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81c80a2940 con 0x7f81c809ab50 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f81c809ef30 con 0x7f81c809d930 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81c80a2ac0 con 0x7f81c809e420 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81cc994640 10 monclient(hunting): authenticate will time out at 1367.505581s 2024-09-10T16:16:29.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 0x7f81c81ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 0x7f81c81ca530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35554/0 (socket says 172.21.15.164:35554) 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 1 -- 172.21.15.164:0/3165968627 learned_addr learned my addr 172.21.15.164:0/3165968627 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): get_auth_request con 0x7f81c809ab50 auth_method 0 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 0x7f81c81373d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): get_auth_request con 0x7f81c809e420 auth_method 0 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c5d9b640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809d930 0x7f81c81caa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c5d9b640 10 monclient(hunting): get_auth_request con 0x7f81c809d930 auth_method 0 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 10 monclient(hunting): handle_auth_done global_id 114131 payload 824 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809d930 msgr2=0x7f81c81caa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.018+0000 7f81c6d9d640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c809d930 0x7f81c81caa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 msgr2=0x7f81c81ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 0x7f81c81ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 1 monclient: found mon.noname-c 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c6d9d640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c813b0e0 con 0x7f81c809e420 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.019+0000 7f81c659c640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c809ab50 0x7f81c81ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81c6d9d640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 msgr2=0x7f81c81373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81c6d9d640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 0x7f81c81373d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81c000cdc0 tx=0x7f81c000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c000ad40 con 0x7f81c809e420 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:29.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: dump: 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.034 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81c0012400 con 0x7f81c809e420 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 5 monclient: authenticate success, global_id 114131 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 10 monclient: _renew_subs 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c0012830 con 0x7f81c809e420 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c80a5fd0 con 0x7f81c809e420 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81beffd640 10 monclient: dump: 2024-09-10T16:16:29.035 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81c4d99640 4 set_mon_vals no callback set 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 10 monclient: _renew_subs 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.021+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c81cafb0 con 0x7f81c809e420 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81cc994640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81cc994640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 0x7f81c81384b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81c809c950 con 0x7f81c81380d0 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f81c0012a90 con 0x7f81c809e420 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 0x7f81c81384b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient: get_auth_request con 0x7f81c81380d0 auth_method 0 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81beffd640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81a803e240 0x7f81a8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 10 monclient(hunting): handle_auth_done global_id 74535 payload 824 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c659c640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f81c809dd90 con 0x7f81c81380d0 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7f81c0053280 con 0x7f81c809e420 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c5d9b640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81a803e240 0x7f81a8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.022+0000 7f81c5d9b640 10 monclient: get_auth_request con 0x7f81a803e240 auth_method 0 2024-09-10T16:16:29.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.024+0000 7f81c659c640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 msgr2=0x7f81c81384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.024+0000 7f81c659c640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 0x7f81c81384b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81b800fa10 tx=0x7f81b800fed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b80074f0 con 0x7f81c81380d0 2024-09-10T16:16:29.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 10 monclient: dump: 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f81b8018070 con 0x7f81c81380d0 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 msgr2=0x7f81c81384b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 0x7f81c81384b0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f81b800fa10 tx=0x7f81b800fed0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 msgr2=0x7f81c81384b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81beffd640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c81380d0 0x7f81c81384b0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.025+0000 7f81c5d9b640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81a803e240 0x7f81a8040700 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f81c809af60 tx=0x7f81b002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81cc994640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81cc994640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 0x7f81c802d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81cc994640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81c809d620 con 0x7f81c802d440 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 0x7f81c802d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient: get_auth_request con 0x7f81c802d440 auth_method 0 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 10 monclient(hunting): handle_auth_done global_id 74538 payload 824 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.031+0000 7f81c659c640 1 -- 172.21.15.164:0/3165968627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f81c806dac0 con 0x7f81c802d440 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81c659c640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 msgr2=0x7f81c802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81c659c640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 0x7f81c802d850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f81b800fa10 tx=0x7f81b8000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8005d80 con 0x7f81c802d440 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 10 monclient: dump: 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4889 (secure 0 0 0) 0x7f81b8005d80 con 0x7f81c802d440 2024-09-10T16:16:29.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 msgr2=0x7f81c802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 0x7f81c802d850 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f81b800fa10 tx=0x7f81b8000ed0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 msgr2=0x7f81c802d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81beffd640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c802d440 0x7f81c802d850 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f81b800fa10 tx=0x7f81b8000ed0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:16:29.045 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 216901, 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.046 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.047 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:29.048 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:29.049 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.050 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:29.054 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.056 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.057 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:29.058 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81a803e240 msgr2=0x7f81a8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81a803e240 0x7f81a8040700 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f81c809af60 tx=0x7f81b002e000 comp rx=0 tx=0).stop 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 10 monclient: shutdown 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 msgr2=0x7f81c81373d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 0x7f81c81373d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f81c000cdc0 tx=0x7f81c000a430 comp rx=0 tx=0).stop 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81c6d9d640 1 -- 172.21.15.164:0/3165968627 reap_dead start 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 shutdown_connections 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 --2- 172.21.15.164:0/3165968627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81c809e420 0x7f81c81373d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 >> 172.21.15.164:0/3165968627 conn(0x7f81c80a3110 msgr2=0x7f81c8074020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 shutdown_connections 2024-09-10T16:16:29.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.034+0000 7f81cc994640 1 -- 172.21.15.164:0/3165968627 wait complete. 2024-09-10T16:16:29.061 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:29.061 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:16:29.062 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:16:29.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 10 monclient: get_monmap_and_config 2024-09-10T16:16:29.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 10 monclient: monmap: 2024-09-10T16:16:29.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.159 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:29.159031+0000 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:29.159031+0000 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:29.160 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 1 Processor -- start 2024-09-10T16:16:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 1 -- start start 2024-09-10T16:16:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.158+0000 7f53408ef640 10 monclient: init 2024-09-10T16:16:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f533c06d160 0x7f533c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-c con 0x7f533c06d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f533c06d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f533c13d060 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f533c09ce70 con 0x7f533c13d060 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f533c09d150 con 0x7f533c06d540 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f533c09cc20 con 0x7f533c06d160 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f53408ef640 10 monclient(hunting): authenticate will time out at 1367.646566s 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f533c13d060 0x7f533c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52524/0 (socket says 172.21.15.164:52524) 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533ad9d640 1 -- 172.21.15.164:0/3373918501 learned_addr learned my addr 172.21.15.164:0/3373918501 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:16:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 1 --2- 172.21.15.164:0/3373918501 >> v2:172.21.15.165:3300/0 conn(0x7f533c06d160 0x7f533c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): get_auth_request con 0x7f533c06d160 auth_method 0 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.159+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.161+0000 7f533ad9d640 1 -- 172.21.15.164:0/3373918501 >> v1:172.21.15.164:6789/0 conn(0x7f533c13d060 legacy=0x7f533c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 1 -- 172.21.15.164:0/3373918501 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2947018936 0 0) 0x7f533c09ce70 con 0x7f533c13d060 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 10 monclient(hunting): _negotiate 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 10 monclient(hunting): my global_id is 74541 2024-09-10T16:16:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533959a640 1 -- 172.21.15.164:0/3373918501 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5324003520 con 0x7f533c13d060 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f5339d9b640 1 -- 172.21.15.164:0/3373918501 >> v1:172.21.15.165:6789/0 conn(0x7f533c06d540 legacy=0x7f533c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 10 monclient(hunting): handle_auth_done global_id 94300 payload 274 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 1 -- 172.21.15.164:0/3373918501 >> v1:172.21.15.165:6789/0 conn(0x7f533c06d540 legacy=0x7f533c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 1 -- 172.21.15.164:0/3373918501 >> v1:172.21.15.164:6789/0 conn(0x7f533c13d060 legacy=0x7f533c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 1 monclient: found mon.noname-c 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:16:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.162+0000 7f533a59c640 1 -- 172.21.15.164:0/3373918501 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533c0a4df0 con 0x7f533c06d160 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533a59c640 1 -- 172.21.15.164:0/3373918501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 msgr2=0x7f533c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533a59c640 1 --2- 172.21.15.164:0/3373918501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c134e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f532c009e90 tx=0x7f532c02a6d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d842e0453c1bd2e9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 1 -- 172.21.15.164:0/3373918501 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c007950 con 0x7f533c06d160 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: dump: 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 1 -- 172.21.15.164:0/3373918501 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f532c007af0 con 0x7f533c06d160 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 1 -- 172.21.15.164:0/3373918501 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c031400 con 0x7f533c06d160 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 5 monclient: authenticate success, global_id 94300 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 4 set_mon_vals no callback set 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f533959a640 10 monclient: dump: 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 10 monclient: shutdown 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 1 -- 172.21.15.164:0/3373918501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 msgr2=0x7f533c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.165+0000 7f53408ef640 1 --2- 172.21.15.164:0/3373918501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c134e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f532c009e90 tx=0x7f532c02a6d0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 1 -- 172.21.15.164:0/3373918501 shutdown_connections 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 1 --2- 172.21.15.164:0/3373918501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c134e00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 1 -- 172.21.15.164:0/3373918501 >> 172.21.15.164:0/3373918501 conn(0x7f533c0aacb0 msgr2=0x7f533c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 1 -- 172.21.15.164:0/3373918501 shutdown_connections 2024-09-10T16:16:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 1 -- 172.21.15.164:0/3373918501 wait complete. 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.166+0000 7f53408ef640 10 monclient: monmap: 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:29.175 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 Processor -- start 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 -- start start 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient: init 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c12e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-b con 0x7f533c06d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 0x7f533c12e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-a con 0x7f533c06d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 0x7f533c09e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): picked mon.noname-c con 0x7f533c13d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f533c140cf0 con 0x7f533c06d540 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f533c140b70 con 0x7f533c06d160 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f533c140e70 con 0x7f533c13d060 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f53408ef640 10 monclient(hunting): authenticate will time out at 1367.654657s 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 0x7f533c12e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 0x7f533c09e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 0x7f533c12e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35584/0 (socket says 172.21.15.164:35584) 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 1 -- 172.21.15.164:0/2233530181 learned_addr learned my addr 172.21.15.164:0/2233530181 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): get_auth_request con 0x7f533c06d540 auth_method 0 2024-09-10T16:16:29.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): get_auth_request con 0x7f533c13d060 auth_method 0 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f5339d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.167+0000 7f533a59c640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c12e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533a59c640 10 monclient(hunting): get_auth_request con 0x7f533c06d160 auth_method 0 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 10 monclient(hunting): handle_auth_done global_id 114134 payload 824 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 msgr2=0x7f533c12e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f533c06d160 0x7f533c12e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 msgr2=0x7f533c12e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 0x7f533c12e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 monclient: found mon.noname-c 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f533c0a2c60 con 0x7f533c13d060 2024-09-10T16:16:29.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.168+0000 7f5339d9b640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f533c06d540 0x7f533c12e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 msgr2=0x7f533c09e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f533ad9d640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 0x7f533c09e560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53340014d0 tx=0x7f533400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334017070 con 0x7f533c13d060 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: dump: 2024-09-10T16:16:29.184 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 5 monclient: authenticate success, global_id 114134 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53340070d0 con 0x7f533c13d060 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 10 monclient: _renew_subs 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f533c0a1fa0 con 0x7f533c13d060 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5334007500 con 0x7f533c13d060 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f532affd640 10 monclient: dump: 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.186 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 10 monclient: _renew_subs 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f5338d99640 4 set_mon_vals no callback set 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.170+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f533c0a0a80 con 0x7f533c13d060 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f53408ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f53408ef640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 0x7f533c09ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f533c140570 con 0x7f533c09eaa0 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 0x7f533c09ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f533401a070 con 0x7f533c13d060 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient: get_auth_request con 0x7f533c09eaa0 auth_method 0 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 10 monclient(hunting): handle_auth_done global_id 114137 payload 824 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f533a59c640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f533c134e00 con 0x7f533c09eaa0 2024-09-10T16:16:29.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f532affd640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f531c03e210 0x7f531c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.171+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7f5334052000 con 0x7f533c13d060 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.172+0000 7f5339d9b640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f531c03e210 0x7f531c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.172+0000 7f5339d9b640 10 monclient: get_auth_request con 0x7f531c03e210 auth_method 0 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.173+0000 7f533a59c640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 msgr2=0x7f533c09ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.173+0000 7f533a59c640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 0x7f533c09ee80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f532c034000 tx=0x7f532c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c031a40 con 0x7f533c09eaa0 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 10 monclient: dump: 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.189 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f532c031400 con 0x7f533c09eaa0 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 msgr2=0x7f533c09ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 0x7f533c09ee80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f532c034000 tx=0x7f532c004a40 comp rx=0 tx=0).stop 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 msgr2=0x7f533c09ee80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f532affd640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09eaa0 0x7f533c09ee80 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f532c034000 tx=0x7f532c004a40 comp rx=0 tx=0).stop 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.174+0000 7f5339d9b640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f531c03e210 0x7f531c0406d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f533c132630 tx=0x7f5330009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f53408ef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f53408ef640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 0x7f533c13d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f53408ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f533c0770b0 con 0x7f533c09f3c0 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 0x7f533c13d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient: get_auth_request con 0x7f533c09f3c0 auth_method 0 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 10 monclient(hunting): handle_auth_done global_id 114140 payload 824 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.180+0000 7f533a59c640 1 -- 172.21.15.164:0/2233530181 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f533c0a4650 con 0x7f533c09f3c0 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.182+0000 7f533a59c640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 msgr2=0x7f533c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.182+0000 7f533a59c640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 0x7f533c13d440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f532c02a710 tx=0x7f532c030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f532c04f070 con 0x7f533c09f3c0 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 10 monclient: dump: 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.192 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f532c04b2d0 con 0x7f533c09f3c0 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 msgr2=0x7f533c13d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 0x7f533c13d440 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f532c02a710 tx=0x7f532c030000 comp rx=0 tx=0).stop 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 msgr2=0x7f533c13d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f532affd640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c09f3c0 0x7f533c13d440 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f532c02a710 tx=0x7f532c030000 comp rx=0 tx=0).stop 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 9751, 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:29.194 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.195 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.196 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.197 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.198 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.199 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.200 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.201 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.202 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.203 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:29.204 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.205 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.206 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.206 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:29.206 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f531c03e210 msgr2=0x7f531c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f531c03e210 0x7f531c0406d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f533c132630 tx=0x7f5330009040 comp rx=0 tx=0).stop 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 10 monclient: shutdown 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 msgr2=0x7f533c09e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 --2- 172.21.15.164:0/2233530181 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f533c13d060 0x7f533c09e560 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f53340014d0 tx=0x7f533400c820 comp rx=0 tx=0).stop 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 reap_dead start 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f533ad9d640 1 -- 172.21.15.164:0/2233530181 reap_dead start 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 shutdown_connections 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 >> 172.21.15.164:0/2233530181 conn(0x7f533c0aacb0 msgr2=0x7f533c12f420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 shutdown_connections 2024-09-10T16:16:29.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.183+0000 7f53408ef640 1 -- 172.21.15.164:0/2233530181 wait complete. 2024-09-10T16:16:29.208 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:29.208 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:16:29.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.306+0000 7fc855fa0640 10 monclient: get_monmap_and_config 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.306+0000 7fc855fa0640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.306+0000 7fc855fa0640 10 monclient: monmap: 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:29.307551+0000 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:29.307551+0000 2024-09-10T16:16:29.308 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:29.309 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 Processor -- start 2024-09-10T16:16:29.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 -- start start 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient: init 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc85009f930 0x7fc8500a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fc85009f930 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8500a2380 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc8500a27b0 0x7fc85013dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fc8500a27b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc85009fd10 con 0x7fc8500a2380 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc85009cd40 con 0x7fc85009f930 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc8500a2c40 con 0x7fc8500a27b0 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.307+0000 7fc855fa0640 10 monclient(hunting): authenticate will time out at 1367.794966s 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc85009f930 0x7fc8500a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc85009f930 0x7fc8500a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35598/0 (socket says 172.21.15.164:35598) 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 -- 172.21.15.164:0/3622740849 learned_addr learned my addr 172.21.15.164:0/3622740849 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): get_auth_request con 0x7fc85009f930 auth_method 0 2024-09-10T16:16:29.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_done global_id 74547 payload 274 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84ffff640 1 --2- 172.21.15.164:0/3622740849 >> v2:172.21.15.165:3300/0 conn(0x7fc8500a27b0 0x7fc85013dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 -- 172.21.15.164:0/3622740849 >> v2:172.21.15.165:3300/0 conn(0x7fc8500a27b0 msgr2=0x7fc85013dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/3622740849 >> v2:172.21.15.165:3300/0 conn(0x7fc8500a27b0 0x7fc85013dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 -- 172.21.15.164:0/3622740849 >> v1:172.21.15.165:6789/0 conn(0x7fc8500a2380 legacy=0x7fc85013a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:16:29.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:29.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.308+0000 7fc84f7fe640 1 -- 172.21.15.164:0/3622740849 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85009a0d0 con 0x7fc85009f930 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84f7fe640 1 -- 172.21.15.164:0/3622740849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc85009f930 msgr2=0x7fc8500a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/3622740849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc85009f930 0x7fc8500a1e40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc85006d470 tx=0x7fc84002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=327054fd8ff700d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 1 -- 172.21.15.164:0/3622740849 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840037070 con 0x7fc85009f930 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: dump: 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.322 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 1 -- 172.21.15.164:0/3622740849 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc840032640 con 0x7fc85009f930 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 5 monclient: authenticate success, global_id 74547 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 1 -- 172.21.15.164:0/3622740849 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840032a10 con 0x7fc85009f930 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 4 set_mon_vals no callback set 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc84effd640 10 monclient: dump: 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.324 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 10 monclient: shutdown 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 -- 172.21.15.164:0/3622740849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc85009f930 msgr2=0x7fc8500a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 --2- 172.21.15.164:0/3622740849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc85009f930 0x7fc8500a1e40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fc85006d470 tx=0x7fc84002a0a0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 -- 172.21.15.164:0/3622740849 shutdown_connections 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 --2- 172.21.15.164:0/3622740849 >> v2:172.21.15.165:3300/0 conn(0x7fc8500a27b0 0x7fc85013dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 --2- 172.21.15.164:0/3622740849 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc85009f930 0x7fc8500a1e40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 -- 172.21.15.164:0/3622740849 >> 172.21.15.164:0/3622740849 conn(0x7fc8500a3110 msgr2=0x7fc8500a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 -- 172.21.15.164:0/3622740849 shutdown_connections 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.311+0000 7fc855fa0640 1 -- 172.21.15.164:0/3622740849 wait complete. 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.312+0000 7fc855fa0640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.312+0000 7fc855fa0640 10 monclient: monmap: 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.312+0000 7fc855fa0640 1 Processor -- start 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.312+0000 7fc855fa0640 1 -- start start 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.312+0000 7fc855fa0640 10 monclient: init 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc85009f930 0x7fc8501cadb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-b con 0x7fc85009f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8500a2380 0x7fc8501cb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-c con 0x7fc8500a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 0x7fc8501b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): picked mon.noname-a con 0x7fc8500a27b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc85009ee00 con 0x7fc8500a27b0 2024-09-10T16:16:29.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc85009e8d0 con 0x7fc85009f930 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc85009d930 con 0x7fc8500a2380 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc855fa0640 10 monclient(hunting): authenticate will time out at 1367.800335s 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 0x7fc8501b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8500a2380 0x7fc8501cb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 0x7fc8501b0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35600/0 (socket says 172.21.15.164:35600) 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 learned_addr learned my addr 172.21.15.164:0/2514568612 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): get_auth_request con 0x7fc8500a27b0 auth_method 0 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): get_auth_request con 0x7fc8500a2380 auth_method 0 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc85009f930 0x7fc8501cadb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient(hunting): handle_auth_done global_id 74550 payload 824 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8500a2380 msgr2=0x7fc8501cb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8500a2380 0x7fc8501cb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc85009f930 msgr2=0x7fc8501cadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc85009f930 0x7fc8501cadb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 monclient: found mon.noname-a 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8501b1a10 con 0x7fc8500a27b0 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84f7fe640 10 monclient: get_auth_request con 0x7fc85009f930 auth_method 0 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc846ffd640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc8500a2380 0x7fc8501cb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.313+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc85009f930 0x7fc8501cadb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:16:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.315+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 msgr2=0x7fc8501b0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.315+0000 7fc84ffff640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 0x7fc8501b0200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc84800cd70 tx=0x7fc84800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848003b90 con 0x7fc8500a27b0 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: dump: 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc848003d30 con 0x7fc8500a27b0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 5 monclient: authenticate success, global_id 74550 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 monclient: _renew_subs 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8501b34d0 con 0x7fc8500a27b0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc848004240 con 0x7fc8500a27b0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 10 monclient: dump: 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 monclient: _renew_subs 2024-09-10T16:16:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84e7fc640 4 set_mon_vals no callback set 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8501b47a0 con 0x7fc8500a27b0 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 0x7fc8501b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501b3c20 con 0x7fc8501b0740 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.316+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc848004460 con 0x7fc8500a27b0 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 0x7fc8501b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient: get_auth_request con 0x7fc8501b0740 auth_method 0 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84cff9640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc83003e240 0x7fc830040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7fc8480538e0 con 0x7fc8500a27b0 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc846ffd640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc83003e240 0x7fc830040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc846ffd640 10 monclient: get_auth_request con 0x7fc83003e240 auth_method 0 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_done global_id 94309 payload 824 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.317+0000 7fc84f7fe640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc8500a0760 con 0x7fc8501b0740 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84f7fe640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 msgr2=0x7fc8501b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 0x7fc8501b0b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc85009f500 tx=0x7fc8400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc846ffd640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc83003e240 0x7fc830040700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8501b3330 tx=0x7fc838009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840037040 con 0x7fc8501b0740 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.319+0000 7fc84cff9640 10 monclient: dump: 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc84003c870 con 0x7fc8501b0740 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 msgr2=0x7fc8501b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 0x7fc8501b0b20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc85009f500 tx=0x7fc8400046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 msgr2=0x7fc8501b0b20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.320+0000 7fc84cff9640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b0740 0x7fc8501b0b20 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc85009f500 tx=0x7fc8400046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc855fa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc855fa0640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 0x7fc850070520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc855fa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc8501b2660 con 0x7fc8501b1060 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 0x7fc850070520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient: get_auth_request con 0x7fc8501b1060 auth_method 0 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.326+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.327+0000 7fc84f7fe640 10 monclient(hunting): handle_auth_done global_id 94312 payload 824 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.327+0000 7fc84f7fe640 1 -- 172.21.15.164:0/2514568612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc8501cb830 con 0x7fc8501b1060 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84f7fe640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 msgr2=0x7fc850070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84f7fe640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 0x7fc850070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc8500694c0 tx=0x7fc840008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc840037040 con 0x7fc8501b1060 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 10 monclient: dump: 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fc84004b070 con 0x7fc8501b1060 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 msgr2=0x7fc850070520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 0x7fc850070520 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc8500694c0 tx=0x7fc840008030 comp rx=0 tx=0).stop 2024-09-10T16:16:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 msgr2=0x7fc850070520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.329+0000 7fc84cff9640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8501b1060 0x7fc850070520 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fc8500694c0 tx=0x7fc840008030 comp rx=0 tx=0).stop 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 92, 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 106829, 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:29.342 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.343 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.344 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:29.345 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:29.346 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:29.347 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.349 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.350 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:29.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.352 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.353 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:29.354 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc83003e240 msgr2=0x7fc830040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc83003e240 0x7fc830040700 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fc8501b3330 tx=0x7fc838009040 comp rx=0 tx=0).stop 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 10 monclient: shutdown 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 msgr2=0x7fc8501b0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 --2- 172.21.15.164:0/2514568612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc8500a27b0 0x7fc8501b0200 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fc84800cd70 tx=0x7fc84800a660 comp rx=0 tx=0).stop 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 reap_dead start 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc84ffff640 1 -- 172.21.15.164:0/2514568612 reap_dead start 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 shutdown_connections 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 >> 172.21.15.164:0/2514568612 conn(0x7fc8500a3110 msgr2=0x7fc8500a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 shutdown_connections 2024-09-10T16:16:29.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.330+0000 7fc855fa0640 1 -- 172.21.15.164:0/2514568612 wait complete. 2024-09-10T16:16:29.357 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:16:29.357 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:16:29.357 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:16:29.357 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:16:29.357 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:16:29.357 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:16:29.357 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:16:29.370 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:16:29.370 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:16:29.370 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:29.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.505+0000 7f30a4a79640 10 monclient: get_monmap_and_config 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.505+0000 7f30a4a79640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.505+0000 7f30a4a79640 10 monclient: monmap: 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:29.506606+0000 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:29.506606+0000 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 Processor -- start 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 -- start start 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient: init 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-b con 0x7f30a006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f30a00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f30a00a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30a009ce50 con 0x7f30a00a46a0 2024-09-10T16:16:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30a0138ac0 con 0x7f30a00a4a80 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f30a006db60 con 0x7f30a006cb70 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f30a4a79640 10 monclient(hunting): authenticate will time out at 1367.993967s 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.506+0000 7f309dd9b640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f30a00a46a0 legacy=0x7f30a0137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60604/0 (socket says 172.21.15.164:60604) 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 -- 172.21.15.164:0/3348335543 learned_addr learned my addr 172.21.15.164:0/3348335543 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): get_auth_request con 0x7f30a006cb70 auth_method 0 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient(hunting): handle_auth_done global_id 114146 payload 274 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 -- 172.21.15.164:0/3348335543 >> v1:172.21.15.165:6789/0 conn(0x7f30a00a4a80 legacy=0x7f30a01383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 -- 172.21.15.164:0/3348335543 >> v1:172.21.15.164:6789/0 conn(0x7f30a00a46a0 legacy=0x7f30a0137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 monclient: found mon.noname-b 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.507+0000 7f309e59c640 1 -- 172.21.15.164:0/3348335543 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a009cb70 con 0x7f30a006cb70 2024-09-10T16:16:29.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309e59c640 1 -- 172.21.15.164:0/3348335543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 msgr2=0x7f30a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309e59c640 1 --2- 172.21.15.164:0/3348335543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30900098a0 tx=0x7f309002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6281dcbeed80f7da server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 1 -- 172.21.15.164:0/3348335543 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090037070 con 0x7f30a006cb70 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: dump: 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 1 -- 172.21.15.164:0/3348335543 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3090032640 con 0x7f30a006cb70 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 5 monclient: authenticate success, global_id 114146 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 1 -- 172.21.15.164:0/3348335543 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3090032a10 con 0x7f30a006cb70 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:29.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 4 set_mon_vals no callback set 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f309d59a640 10 monclient: dump: 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.533 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 10 monclient: shutdown 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 -- 172.21.15.164:0/3348335543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 msgr2=0x7f30a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 --2- 172.21.15.164:0/3348335543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f30900098a0 tx=0x7f309002a080 comp rx=0 tx=0).stop 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 -- 172.21.15.164:0/3348335543 shutdown_connections 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 --2- 172.21.15.164:0/3348335543 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a006cb70 0x7f30a0134f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 -- 172.21.15.164:0/3348335543 >> 172.21.15.164:0/3348335543 conn(0x7f30a00aa900 msgr2=0x7f30a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 -- 172.21.15.164:0/3348335543 shutdown_connections 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.510+0000 7f30a4a79640 1 -- 172.21.15.164:0/3348335543 wait complete. 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.511+0000 7f30a4a79640 10 monclient: build_initial_monmap 2024-09-10T16:16:29.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.511+0000 7f30a4a79640 10 monclient: monmap: 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.511+0000 7f30a4a79640 1 Processor -- start 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.511+0000 7f30a4a79640 1 -- start start 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.511+0000 7f30a4a79640 10 monclient: init 2024-09-10T16:16:29.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a006cb70 0x7f30a009eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-b con 0x7f30a006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 0x7f30a009f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-a con 0x7f30a00a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): picked mon.noname-c con 0x7f30a00a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f30a006cf50 con 0x7f30a00a46a0 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f30a0141050 con 0x7f30a006cb70 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:29.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f30a01411d0 con 0x7f30a00a4a80 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f30a4a79640 10 monclient(hunting): authenticate will time out at 1367.999313s 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309dd9b640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 msgr2=0x7f30a009f070 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 0x7f30a009f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60620/0 (socket says 172.21.15.164:60620) 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 -- 172.21.15.164:0/2818885673 learned_addr learned my addr 172.21.15.164:0/2818885673 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): get_auth_request con 0x7f30a00a4a80 auth_method 0 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a006cb70 0x7f30a009eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): get_auth_request con 0x7f30a006cb70 auth_method 0 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient(hunting): handle_auth_done global_id 114149 payload 824 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a006cb70 msgr2=0x7f30a009eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a006cb70 0x7f30a009eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 msgr2=0x7f30a009f070 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:29.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 0x7f30a009f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 monclient: found mon.noname-c 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.512+0000 7f309ed9d640 1 -- 172.21.15.164:0/2818885673 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30a00a3580 con 0x7f30a00a4a80 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.514+0000 7f309ed9d640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 msgr2=0x7f30a009f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.514+0000 7f309ed9d640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f309800c780 tx=0x7f309800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f309800ac60 con 0x7f30a00a4a80 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: dump: 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.539 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: _finish_auth 0 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f309800ae00 con 0x7f30a00a4a80 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 5 monclient: authenticate success, global_id 114149 2024-09-10T16:16:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 10 monclient: _renew_subs 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30a00a2670 con 0x7f30a00a4a80 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3098004ba0 con 0x7f30a00a4a80 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 10 monclient: dump: 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 10 monclient: _renew_subs 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f309cd99640 4 set_mon_vals no callback set 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f30a00a8a20 con 0x7f30a00a4a80 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.515+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3098005030 con 0x7f30a00a4a80 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f30a4a79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f30a4a79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f30a4a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30a006a450 con 0x7f30a00a4a80 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f308effd640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7f3098052fe0 con 0x7f30a00a4a80 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 10 monclient: get_auth_request con 0x7f308003e1c0 auth_method 0 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 msgr2=0x7f3080040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 msgr2=0x7f3080040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.516+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.521+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f30a006a450 con 0x7f30a00a4a80 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.521+0000 7f308effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.521+0000 7f308effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.718+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:29.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.718+0000 7f309e59c640 10 monclient: get_auth_request con 0x7f308003e1c0 auth_method 0 2024-09-10T16:16:29.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.720+0000 7f309e59c640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3090034000 tx=0x7f309002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:29.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.761+0000 7f30a4a79640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:29.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.761+0000 7f30a4a79640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:29.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.762+0000 7f30a4a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:29.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.762+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30a01322f0 con 0x7f30a00a4a80 2024-09-10T16:16:29.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.762+0000 7f308effd640 1 -- 172.21.15.164:0/2818885673 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f30a01322f0 con 0x7f30a00a4a80 2024-09-10T16:16:29.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.762+0000 7f308effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:29.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.762+0000 7f308effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:29.764 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 msgr2=0x7f3080040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f3090034000 tx=0x7f309002e000 comp rx=0 tx=0).stop 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 10 monclient: shutdown 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 msgr2=0x7f30a009f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f309800c780 tx=0x7f309800cc40 comp rx=0 tx=0).stop 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 shutdown_connections 2024-09-10T16:16:29.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f308003e1c0 0x7f3080040680 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f30a00a4a80 0x7f30a009f5b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f30a00a46a0 0x7f30a009f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 --2- 172.21.15.164:0/2818885673 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f30a006cb70 0x7f30a009eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 >> 172.21.15.164:0/2818885673 conn(0x7f30a00aa900 msgr2=0x7f30a0132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 shutdown_connections 2024-09-10T16:16:29.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:29.764+0000 7f30a4a79640 1 -- 172.21.15.164:0/2818885673 wait complete. 2024-09-10T16:16:32.778 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:32.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.886+0000 7f75b5ea9640 10 monclient: get_monmap_and_config 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.886+0000 7f75b5ea9640 10 monclient: build_initial_monmap 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.886+0000 7f75b5ea9640 10 monclient: monmap: 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:32.887417+0000 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:32.887417+0000 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:32.888 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:32.889 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:32.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 Processor -- start 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 -- start start 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient: init 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-10T16:16:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f75b0137bb0 0x7f75b00a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-c con 0x7f75b0137bb0 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f75b0137f90 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 0x7f75b013f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-a con 0x7f75b013d140 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75b0073820 con 0x7f75b0137f90 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f75b00a1c90 con 0x7f75b013d140 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f75b00a1f70 con 0x7f75b0137bb0 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75b5ea9640 10 monclient(hunting): authenticate will time out at 1371.374899s 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75aeffd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f75b0137f90 legacy=0x7f75b01383a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75affff640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 msgr2=0x7f75b013f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.887+0000 7f75affff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 0x7f75b013f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f75b0137bb0 0x7f75b00a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f75b0137bb0 0x7f75b00a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:42730/0 (socket says 172.21.15.164:42730) 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 -- 172.21.15.164:0/1020943200 learned_addr learned my addr 172.21.15.164:0/1020943200 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): get_auth_request con 0x7f75b0137bb0 auth_method 0 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient(hunting): handle_auth_done global_id 94318 payload 274 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 -- 172.21.15.164:0/1020943200 >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 msgr2=0x7f75b013f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:32.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1020943200 >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 0x7f75b013f530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 -- 172.21.15.164:0/1020943200 >> v1:172.21.15.164:6789/0 conn(0x7f75b0137f90 legacy=0x7f75b01383a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 monclient: found mon.noname-c 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:16:32.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.888+0000 7f75af7fe640 1 -- 172.21.15.164:0/1020943200 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b00a2250 con 0x7f75b0137bb0 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75af7fe640 1 -- 172.21.15.164:0/1020943200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b0137bb0 msgr2=0x7f75b00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1020943200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b0137bb0 0x7f75b00a2e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75b00a7ed0 tx=0x7f75a002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8758180db3c832c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 1 -- 172.21.15.164:0/1020943200 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0037070 con 0x7f75b0137bb0 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: dump: 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:32.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: _finish_auth 0 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 1 -- 172.21.15.164:0/1020943200 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a0032640 con 0x7f75b0137bb0 2024-09-10T16:16:32.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75b5ea9640 5 monclient: authenticate success, global_id 94318 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75b5ea9640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75b5ea9640 4 set_mon_vals no callback set 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 1 -- 172.21.15.164:0/1020943200 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0032a10 con 0x7f75b0137bb0 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.891+0000 7f75ae7fc640 10 monclient: dump: 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 10 monclient: shutdown 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1020943200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b0137bb0 msgr2=0x7f75b00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1020943200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b0137bb0 0x7f75b00a2e80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f75b00a7ed0 tx=0x7f75a002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1020943200 shutdown_connections 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1020943200 >> v2:172.21.15.164:3300/0 conn(0x7f75b013d140 0x7f75b013f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1020943200 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b0137bb0 0x7f75b00a2e80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1020943200 >> 172.21.15.164:0/1020943200 conn(0x7f75b00a7320 msgr2=0x7f75b00a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1020943200 shutdown_connections 2024-09-10T16:16:32.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1020943200 wait complete. 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 10 monclient: build_initial_monmap 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.892+0000 7f75b5ea9640 10 monclient: monmap: 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 Processor -- start 2024-09-10T16:16:32.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 -- start start 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient: init 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-c con 0x7f75b0137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 0x7f75b0134860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-a con 0x7f75b0137f90 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b013d140 0x7f75b009f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): picked mon.noname-b con 0x7f75b013d140 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0077060 con 0x7f75b0137f90 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f75b0076ee0 con 0x7f75b013d140 2024-09-10T16:16:32.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f75b013fa70 con 0x7f75b0137bb0 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75b5ea9640 10 monclient(hunting): authenticate will time out at 1371.380855s 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75aeffd640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 msgr2=0x7f75b0134860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75aeffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 0x7f75b0134860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60626/0 (socket says 172.21.15.164:60626) 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 1 -- 172.21.15.164:0/1913552660 learned_addr learned my addr 172.21.15.164:0/1913552660 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 10 monclient(hunting): get_auth_request con 0x7f75b0137bb0 auth_method 0 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.893+0000 7f75af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:32.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b013d140 0x7f75b009f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): get_auth_request con 0x7f75b013d140 auth_method 0 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient(hunting): handle_auth_done global_id 114152 payload 824 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 -- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b013d140 msgr2=0x7f75b009f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b013d140 0x7f75b009f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 -- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 msgr2=0x7f75b0134860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 0x7f75b0134860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 monclient: found mon.noname-c 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.894+0000 7f75af7fe640 1 -- 172.21.15.164:0/1913552660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75b011f210 con 0x7f75b0137bb0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75af7fe640 1 -- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 msgr2=0x7f75b01cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75a00090b0 tx=0x7f75a00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0041070 con 0x7f75b0137bb0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: dump: 2024-09-10T16:16:32.908 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: _finish_auth 0 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: _check_auth_tickets 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 5 monclient: authenticate success, global_id 114152 2024-09-10T16:16:32.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75a003c880 con 0x7f75b0137bb0 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75a0037040 con 0x7f75b0137bb0 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 10 monclient: _renew_subs 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75b00a0cf0 con 0x7f75b0137bb0 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f7597fff640 10 monclient: dump: 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:32.910 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75adffb640 4 set_mon_vals no callback set 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 10 monclient: _renew_subs 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.896+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f75b011dde0 con 0x7f75b0137bb0 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.897+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f75a0044070 con 0x7f75b0137bb0 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.897+0000 7f75b5ea9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.897+0000 7f75b5ea9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.897+0000 7f75b5ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:32.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.897+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75b00a2f00 con 0x7f75b0137bb0 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.898+0000 7f7597fff640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 0x7f759c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.898+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(355..355 src has 184..355) ==== 4739+0+0 (secure 0 0 0) 0x7f75a0073540 con 0x7f75b0137bb0 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.898+0000 7f75aeffd640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 0x7f759c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.898+0000 7f75aeffd640 10 monclient: get_auth_request con 0x7f759c03e1f0 auth_method 0 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.900+0000 7f75aeffd640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 0x7f759c0406b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f75a4006fd0 tx=0x7f75a4006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:32.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:32.902+0000 7f75af7fe640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75a00090b0 tx=0x7f75a00046c0 comp rx=0 tx=0).queue_received will delay after 1.72598e+09 on 0x7f75a003cc50 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:16:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.730+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f75a003cc50 con 0x7f75b0137bb0 2024-09-10T16:16:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.730+0000 7f7597fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:33.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.730+0000 7f7597fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:33.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f75b5ea9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f75b5ea9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f75b5ea9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75b011e5e0 con 0x7f75b0137bb0 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f7597fff640 1 -- 172.21.15.164:0/1913552660 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f75a0071f60 con 0x7f75b0137bb0 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f7597fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:33.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.975+0000 7f7597fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:33.978 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":92,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:33.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 msgr2=0x7f759c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 0x7f759c0406b0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f75a4006fd0 tx=0x7f75a4006d80 comp rx=0 tx=0).stop 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 10 monclient: shutdown 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 msgr2=0x7f75b01cce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f75a00090b0 tx=0x7f75a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:33.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 shutdown_connections 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f759c03e1f0 0x7f759c0406b0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75b013d140 0x7f75b009f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75b0137f90 0x7f75b0134860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 --2- 172.21.15.164:0/1913552660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75b0137bb0 0x7f75b01cce70 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 >> 172.21.15.164:0/1913552660 conn(0x7f75b00a7320 msgr2=0x7f75b0136bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 shutdown_connections 2024-09-10T16:16:33.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:33.977+0000 7f75b5ea9640 1 -- 172.21.15.164:0/1913552660 wait complete. 2024-09-10T16:16:36.991 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:16:37.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.098+0000 7f02019a3640 10 monclient: get_monmap_and_config 2024-09-10T16:16:37.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.099+0000 7f02019a3640 10 monclient: build_initial_monmap 2024-09-10T16:16:37.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.099+0000 7f02019a3640 10 monclient: monmap: 2024-09-10T16:16:37.100 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:37.099899+0000 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:37.099899+0000 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:37.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.099+0000 7f02019a3640 1 Processor -- start 2024-09-10T16:16:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.099+0000 7f02019a3640 1 -- start start 2024-09-10T16:16:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.099+0000 7f02019a3640 10 monclient: init 2024-09-10T16:16:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01fc09d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-a con 0x7f01fc09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01fc09e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01fc0a0870 con 0x7f01fc09e420 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01fc074c70 con 0x7f01fc09d930 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f01fc09d3e0 con 0x7f01fc09ab50 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f02019a3640 10 monclient(hunting): authenticate will time out at 1375.587156s 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f01fa7fc640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 msgr2=0x7f01fc1397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f01fa7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f01fb7fe640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f01fc09e420 legacy=0x7f01fc13bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f01faffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f01fc09d930 0x7f01fc09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:35868/0 (socket says 172.21.15.164:35868) 2024-09-10T16:16:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.100+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 learned_addr learned my addr 172.21.15.164:0/3150731746 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:16:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.102+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.165:6789/0 conn(0x7f01fc09d930 legacy=0x7f01fc09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:37.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.301+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 msgr2=0x7f01fc1397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:37.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.301+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:16:37.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.301+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01fc09e420 legacy=0x7f01fc13bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:37.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.702+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 msgr2=0x7f01fc1397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:37.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.702+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01fc09e420 legacy=0x7f01fc13bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:37.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:37.702+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:16:38.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:38.503+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 msgr2=0x7f01fc1397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:38.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:38.503+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01fc09e420 legacy=0x7f01fc13bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:38.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:38.503+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:16:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): tick 2024-09-10T16:16:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 monclient(hunting): continuing hunt 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 msgr2=0x7f01fc1397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.164:3300/0 conn(0x7f01fc09ab50 0x7f01fc1397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.165:6789/0 conn(0x7f01fc09d930 legacy=0x7f01fc09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01fc09e420 legacy=0x7f01fc13bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient: _add_conns ranks=[2,4] 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f01e8003740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f01e8006270 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f01fc09d3e0 con 0x7f01e8006270 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01e8008df0 con 0x7f01e8003740 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01f8ff9640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 10 monclient(hunting): get_auth_request con 0x7f01e8003740 auth_method 0 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01e8006270 auth_method 0 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:40.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:16:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.100+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 10 monclient(hunting): get_auth_request con 0x7f01e8003740 auth_method 0 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:40.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01e8006270 auth_method 0 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.301+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 10 monclient(hunting): get_auth_request con 0x7f01e8003740 auth_method 0 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01e8006270 auth_method 0 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:40.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:40.702+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:16:41.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:41.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 10 monclient(hunting): get_auth_request con 0x7f01e8003740 auth_method 0 2024-09-10T16:16:41.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:41.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:41.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01e8006270 auth_method 0 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:41.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:41.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:41.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:16:41.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:41.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:41.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:41.503+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): tick 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 monclient(hunting): continuing hunt 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 msgr2=0x7f01e8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e8003740 0x7f01e8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 msgr2=0x7f01e8008730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8006270 0x7f01e8008730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 reap_dead start 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 0x7f01e8006720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f01e8003b80 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01e8006c60 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01e80087f0 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01fc09ecb0 con 0x7f01e80087f0 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01fc0a0870 con 0x7f01e8006c60 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 1 -- 172.21.15.164:0/3150731746 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f01fc09d3e0 con 0x7f01e8003b80 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01f8ff9640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01e80087f0 legacy=0x7f01e800d360 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:43.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 0x7f01e8006720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 10 monclient(hunting): get_auth_request con 0x7f01e8003b80 auth_method 0 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 msgr2=0x7f01e8006720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 msgr2=0x7f01e8006720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 0x7f01e8006720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.101+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 0x7f01e8006720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.103+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.165:6789/0 conn(0x7f01e8006c60 legacy=0x7f01e8007070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4060413654 0 0) 0x7f01fc0a0870 con 0x7f01e8006c60 2024-09-10T16:16:43.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 10 monclient(hunting): _negotiate 2024-09-10T16:16:43.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 10 monclient(hunting): my global_id is 94324 2024-09-10T16:16:43.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.292+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01e4005730 con 0x7f01e8006c60 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1773401751 0 0) 0x7f01e4005730 con 0x7f01e8006c60 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 msgr2=0x7f01e8006720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 --2- 172.21.15.164:0/3150731746 >> v2:172.21.15.165:3300/0 conn(0x7f01e8003b80 0x7f01e8006720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.164:6789/0 conn(0x7f01e80087f0 legacy=0x7f01e800d360 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e8010cd0 con 0x7f01e8006c60 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f01f00086e0 con 0x7f01e8006c60 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f02019a3640 5 monclient: authenticate success, global_id 94324 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f02019a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: dump: 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.300 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.301 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.301 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.301 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 >> v1:172.21.15.165:6789/0 conn(0x7f01e8006c60 legacy=0x7f01e8007070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4005c30 0x7f01e4006030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): picked mon.a con 0x7f01e4005c30 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 0x7f01e4006e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): picked mon.c con 0x7f01e4006a10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01e40074f0 0x7f01e400dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): picked mon.b con 0x7f01e40074f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc09d3e0 con 0x7f01e4005c30 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01e400e3c0 con 0x7f01e40074f0 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01e400e5c0 con 0x7f01e4006a10 2024-09-10T16:16:43.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01f9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f02019a3640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01faffd640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4005c30 msgr2=0x7f01e4006030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01faffd640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4005c30 0x7f01e4006030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 0x7f01e4006e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01e4006a10 auth_method 0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01e40074f0 0x7f01e400dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): get_auth_request con 0x7f01e40074f0 auth_method 0 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_done global_id 94324 payload 293 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01e40074f0 msgr2=0x7f01e400dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01e40074f0 0x7f01e400dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4005c30 msgr2=0x7f01e4006030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4005c30 0x7f01e4006030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 monclient: found mon.c 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01e4005730 con 0x7f01e4006a10 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3150731746 reap_dead start 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.295+0000 7f01fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.298+0000 7f01fa7fc640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 msgr2=0x7f01e4006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.298+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 0x7f01e4006e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01f0003cb0 tx=0x7f01e4007a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ebaa83a660af195 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f0031070 con 0x7f01e4006a10 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: dump: 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.313 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f0006e50 con 0x7f01e4006a10 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f02019a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 1 -- 172.21.15.164:0/3150731746 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f000bdd0 con 0x7f01e4006a10 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f02019a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:43.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f02019a3640 4 set_mon_vals no callback set 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.299+0000 7f01f9ffb640 10 monclient: dump: 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.315 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 10 monclient: shutdown 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 -- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 msgr2=0x7f01e4006e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 0x7f01e4006e90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f01f0003cb0 tx=0x7f01e4007a40 comp rx=0 tx=0).stop 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 -- 172.21.15.164:0/3150731746 shutdown_connections 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 --2- 172.21.15.164:0/3150731746 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01e4006a10 0x7f01e4006e90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 -- 172.21.15.164:0/3150731746 >> 172.21.15.164:0/3150731746 conn(0x7f01fc0a3110 msgr2=0x7f01fc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 -- 172.21.15.164:0/3150731746 shutdown_connections 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 1 -- 172.21.15.164:0/3150731746 wait complete. 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 10 monclient: build_initial_monmap 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.300+0000 7f02019a3640 10 monclient: monmap: 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:43.316 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.301+0000 7f02019a3640 1 Processor -- start 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.301+0000 7f02019a3640 1 -- start start 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.301+0000 7f02019a3640 10 monclient: init 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:16:43.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 0x7f01fc12bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-a con 0x7f01e4006a10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01fc12c4a0 0x7f01fc1c8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-b con 0x7f01fc12c4a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): picked mon.noname-c con 0x7f01fc1c9110 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc09ef30 con 0x7f01e4006a10 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc0a0030 con 0x7f01fc12c4a0 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01fc0a2940 con 0x7f01fc1c9110 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f02019a3640 10 monclient(hunting): authenticate will time out at 1381.789310s 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01faffd640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 msgr2=0x7f01fc12bf60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01faffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 0x7f01fc12bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34658/0 (socket says 172.21.15.164:34658) 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3907633471 learned_addr learned my addr 172.21.15.164:0/3907633471 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:43.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): get_auth_request con 0x7f01fc1c9110 auth_method 0 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01fc12c4a0 0x7f01fc1c8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request con 0x7f01fc12c4a0 auth_method 0 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fa7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient(hunting): handle_auth_done global_id 114155 payload 824 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01fc12c4a0 msgr2=0x7f01fc1c8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01fc12c4a0 0x7f01fc1c8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 msgr2=0x7f01fc12bf60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 0x7f01fc12bf60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 monclient: found mon.noname-c 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.302+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3907633471 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc09c9e0 con 0x7f01fc1c9110 2024-09-10T16:16:43.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01fb7fe640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 msgr2=0x7f01fc1c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01fb7fe640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f01fc09f930 tx=0x7f01f400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f400ce70 con 0x7f01fc1c9110 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: dump: 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.322 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01f4013070 con 0x7f01fc1c9110 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4017670 con 0x7f01fc1c9110 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01e3fff640 10 monclient: dump: 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.323 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 5 monclient: authenticate success, global_id 114155 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 10 monclient: _renew_subs 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc0745c0 con 0x7f01fc1c9110 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f01f97fa640 4 set_mon_vals no callback set 2024-09-10T16:16:43.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 10 monclient: _renew_subs 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.305+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc12e550 con 0x7f01fc1c9110 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f01f401e020 con 0x7f01fc1c9110 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f02019a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f02019a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f02019a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f01e3fff640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01fc09dda0 con 0x7f01fc1c9110 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(356..356 src has 184..356) ==== 4739+0+0 (secure 0 0 0) 0x7f01f4053070 con 0x7f01fc1c9110 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f01faffd640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.306+0000 7f01faffd640 10 monclient: get_auth_request con 0x7f01e803f7b0 auth_method 0 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.307+0000 7f01faffd640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 msgr2=0x7f01e8041c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:16:43.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.307+0000 7f01faffd640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 msgr2=0x7f01e8041c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:16:43.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.307+0000 7f01faffd640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:16:43.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.307+0000 7f01faffd640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.311+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f01f4051a90 con 0x7f01fc1c9110 2024-09-10T16:16:43.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.311+0000 7f01e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.311+0000 7f01e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:43.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.508+0000 7f01faffd640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.508+0000 7f01faffd640 10 monclient: get_auth_request con 0x7f01e803f7b0 auth_method 0 2024-09-10T16:16:43.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.511+0000 7f01faffd640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f01ec007fd0 tx=0x7f01ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.555+0000 7f02019a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:16:43.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.555+0000 7f02019a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.555+0000 7f02019a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.555+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f01fc06dac0 con 0x7f01fc1c9110 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.556+0000 7f01e3fff640 1 -- 172.21.15.164:0/3907633471 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f01f400f410 con 0x7f01fc1c9110 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.556+0000 7f01e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.556+0000 7f01e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:43.559 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":94,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:16:43.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 msgr2=0x7f01e8041c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f01ec007fd0 tx=0x7f01ec009040 comp rx=0 tx=0).stop 2024-09-10T16:16:43.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 10 monclient: shutdown 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 msgr2=0x7f01fc1c9590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f01fc09f930 tx=0x7f01f400ed70 comp rx=0 tx=0).stop 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 shutdown_connections 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01e803f7b0 0x7f01e8041c70 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01fc1c9110 0x7f01fc1c9590 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01fc12c4a0 0x7f01fc1c8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 --2- 172.21.15.164:0/3907633471 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01e4006a10 0x7f01fc12bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 >> 172.21.15.164:0/3907633471 conn(0x7f01fc0a3110 msgr2=0x7f01fc1bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 shutdown_connections 2024-09-10T16:16:43.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.558+0000 7f02019a3640 1 -- 172.21.15.164:0/3907633471 wait complete. 2024-09-10T16:16:43.571 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:16:43.571 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:16:43.571 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.668+0000 7fe4ca5e2640 10 monclient: get_monmap_and_config 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.668+0000 7fe4ca5e2640 10 monclient: build_initial_monmap 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.668+0000 7fe4ca5e2640 10 monclient: monmap: 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:43.669646+0000 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:43.669646+0000 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:43.670 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:43.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.669+0000 7fe4ca5e2640 1 Processor -- start 2024-09-10T16:16:43.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.669+0000 7fe4ca5e2640 1 -- start start 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.669+0000 7fe4ca5e2640 10 monclient: init 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe4c409c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 0x7fe4c409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c409d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c409a430 con 0x7fe4c409c180 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe4c40a24e0 con 0x7fe4c409d160 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c40a2bf0 con 0x7fe4c413a5c0 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c3fff640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fe4c409c180 legacy=0x7fe4c409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c37fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 msgr2=0x7fe4c409f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4ca5e2640 10 monclient(hunting): authenticate will time out at 1382.157160s 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 0x7fe4c409f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34664/0 (socket says 172.21.15.164:34664) 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/1688904909 learned_addr learned my addr 172.21.15.164:0/1688904909 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): get_auth_request con 0x7fe4c413a5c0 auth_method 0 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient(hunting): handle_auth_done global_id 114158 payload 274 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/1688904909 >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 msgr2=0x7fe4c409f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 --2- 172.21.15.164:0/1688904909 >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 0x7fe4c409f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/1688904909 >> v1:172.21.15.164:6789/0 conn(0x7fe4c409c180 legacy=0x7fe4c409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 monclient: found mon.noname-b 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.670+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/1688904909 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c409c560 con 0x7fe4c413a5c0 2024-09-10T16:16:43.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/1688904909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 msgr2=0x7fe4c413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c8b7f640 1 --2- 172.21.15.164:0/1688904909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009e80 tx=0x7fe4bc02a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d21d1f5da394b4c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 1 -- 172.21.15.164:0/1688904909 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc036070 con 0x7fe4c413a5c0 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: dump: 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.684 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 1 -- 172.21.15.164:0/1688904909 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc007800 con 0x7fe4c413a5c0 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 5 monclient: authenticate success, global_id 114158 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 1 -- 172.21.15.164:0/1688904909 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc007bd0 con 0x7fe4c413a5c0 2024-09-10T16:16:43.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 4 set_mon_vals no callback set 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4c2ffd640 10 monclient: dump: 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.686 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 10 monclient: shutdown 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/1688904909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 msgr2=0x7fe4c413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/1688904909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009e80 tx=0x7fe4bc02a5b0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/1688904909 shutdown_connections 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/1688904909 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c413a5c0 0x7fe4c413c9b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.673+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/1688904909 >> v2:172.21.15.164:3300/0 conn(0x7fe4c409d160 0x7fe4c409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/1688904909 >> 172.21.15.164:0/1688904909 conn(0x7fe4c40a3110 msgr2=0x7fe4c40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/1688904909 shutdown_connections 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/1688904909 wait complete. 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 10 monclient: build_initial_monmap 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 10 monclient: monmap: 2024-09-10T16:16:43.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.674+0000 7fe4ca5e2640 1 Processor -- start 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 -- start start 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient: init 2024-09-10T16:16:43.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c409c180 0x7fe4c406f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe4c409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe4c409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4c413a5c0 0x7fe4c406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe4c413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4074f90 con 0x7fe4c409c180 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c4074e10 con 0x7fe4c413a5c0 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c409f690 con 0x7fe4c409d160 2024-09-10T16:16:43.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4ca5e2640 10 monclient(hunting): authenticate will time out at 1382.162543s 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c3fff640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c409c180 msgr2=0x7fe4c406f230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c409c180 0x7fe4c406f230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34666/0 (socket says 172.21.15.164:34666) 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 -- 172.21.15.164:0/881558430 learned_addr learned my addr 172.21.15.164:0/881558430 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request con 0x7fe4c409d160 auth_method 0 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient(hunting): handle_auth_done global_id 114161 payload 824 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4c413a5c0 msgr2=0x7fe4c406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe4c413a5c0 0x7fe4c406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c409c180 msgr2=0x7fe4c406f230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe4c409c180 0x7fe4c406f230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 monclient: found mon.noname-c 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.675+0000 7fe4c37fe640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c40977e0 con 0x7fe4c409d160 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c37fe640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 msgr2=0x7fe4c406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c37fe640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4b800b4d0 tx=0x7fe4b800b990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8005070 con 0x7fe4c409d160 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: dump: 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b8015070 con 0x7fe4c409d160 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 5 monclient: authenticate success, global_id 114161 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 10 monclient: _renew_subs 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c40a4c50 con 0x7fe4c409d160 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b8005410 con 0x7fe4c409d160 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c0ff9640 10 monclient: dump: 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 10 monclient: _renew_subs 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4c27fc640 4 set_mon_vals no callback set 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.678+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c4095d30 con 0x7fe4c409d160 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4ca5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 0x7fe4c408df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c4095fe0 con 0x7fe4c409e820 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 0x7fe4c408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient: get_auth_request con 0x7fe4c409e820 auth_method 0 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe4b800f070 con 0x7fe4c409d160 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c0ff9640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe4b003e210 0x7fe4b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_done global_id 114164 payload 824 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(356..356 src has 184..356) ==== 4739+0+0 (secure 0 0 0) 0x7fe4b80525e0 con 0x7fe4c409d160 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c3fff640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe4c40008d0 con 0x7fe4c409e820 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.679+0000 7fe4c8b7f640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe4b003e210 0x7fe4b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.680+0000 7fe4c8b7f640 10 monclient: get_auth_request con 0x7fe4b003e210 auth_method 0 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.681+0000 7fe4c3fff640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 msgr2=0x7fe4c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.681+0000 7fe4c3fff640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 0x7fe4c408df80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4005270 con 0x7fe4c409e820 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 10 monclient: dump: 2024-09-10T16:16:43.696 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe4b4015070 con 0x7fe4c409e820 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:43.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 msgr2=0x7fe4c408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 0x7fe4c408df80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b400b910 comp rx=0 tx=0).stop 2024-09-10T16:16:43.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 msgr2=0x7fe4c408df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:43.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c0ff9640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409e820 0x7fe4c408df80 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b400b910 comp rx=0 tx=0).stop 2024-09-10T16:16:43.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.682+0000 7fe4c8b7f640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe4b003e210 0x7fe4b00406d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe4c406d4a0 tx=0x7fe4bc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4ca5e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 0x7fe4c408e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4ca5e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c408ee10 con 0x7fe4c408e4c0 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 0x7fe4c408e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient: get_auth_request con 0x7fe4c408e4c0 auth_method 0 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 10 monclient(hunting): handle_auth_done global_id 114167 payload 824 2024-09-10T16:16:43.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.688+0000 7fe4c3fff640 1 -- 172.21.15.164:0/881558430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe4c40701f0 con 0x7fe4c408e4c0 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.690+0000 7fe4c3fff640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 msgr2=0x7fe4c408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.690+0000 7fe4c3fff640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 0x7fe4c408e8d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b40059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4024470 con 0x7fe4c408e4c0 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 10 monclient: dump: 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.701 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4708 (secure 0 0 0) 0x7fe4b4010400 con 0x7fe4c408e4c0 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 msgr2=0x7fe4c408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 0x7fe4c408e8d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b40059d0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 msgr2=0x7fe4c408e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c0ff9640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c408e4c0 0x7fe4c408e8d0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe4b400b450 tx=0x7fe4b40059d0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.702 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 94, 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 24259, 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:16:43.703 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.704 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.705 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:43.706 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.707 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.708 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.708 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.708 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.708 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.708 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.709 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:43.710 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:43.711 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:43.712 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.713 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.714 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.715 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.716 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.717 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:43.718 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.719 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.719 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:43.719 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:43.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe4b003e210 msgr2=0x7fe4b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe4b003e210 0x7fe4b00406d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe4c406d4a0 tx=0x7fe4bc02e000 comp rx=0 tx=0).stop 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 10 monclient: shutdown 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 msgr2=0x7fe4c406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe4b800b4d0 tx=0x7fe4b800b990 comp rx=0 tx=0).stop 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4c8b7f640 1 -- 172.21.15.164:0/881558430 reap_dead start 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 shutdown_connections 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 --2- 172.21.15.164:0/881558430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe4c409d160 0x7fe4c406f770 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 >> 172.21.15.164:0/881558430 conn(0x7fe4c40a3110 msgr2=0x7fe4c4139c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 shutdown_connections 2024-09-10T16:16:43.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.691+0000 7fe4ca5e2640 1 -- 172.21.15.164:0/881558430 wait complete. 2024-09-10T16:16:43.722 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:16:43.722 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:16:43.722 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.819+0000 7f6106b73640 10 monclient: get_monmap_and_config 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.819+0000 7f6106b73640 10 monclient: build_initial_monmap 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.819+0000 7f6106b73640 10 monclient: monmap: 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:16:43.820486+0000 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:16:43.820486+0000 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:16:43.821 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:16:43.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:16:43.822 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:16:43.822 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:16:43.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 Processor -- start 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 -- start start 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient: init 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f610009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f610009ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f610009e420 0x7f610013cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-c con 0x7f610009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:16:43.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61000a0870 con 0x7f610009d930 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6100074c70 con 0x7f610009ab50 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f610009d3e0 con 0x7f610009e420 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f6106b73640 10 monclient(hunting): authenticate will time out at 1382.307849s 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.820+0000 7f610490f640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f610009d930 legacy=0x7f610009dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f610009e420 0x7f610013cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f610009e420 0x7f610013cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:46848/0 (socket says 172.21.15.164:46848) 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 1 -- 172.21.15.164:0/2338949702 learned_addr learned my addr 172.21.15.164:0/2338949702 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): get_auth_request con 0x7f610009e420 auth_method 0 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.821+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f60fffff640 1 -- 172.21.15.164:0/2338949702 >> v1:172.21.15.165:6789/0 conn(0x7f610009ab50 legacy=0x7f61001397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 10 monclient(hunting): handle_auth_done global_id 94330 payload 274 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 1 -- 172.21.15.164:0/2338949702 >> v1:172.21.15.165:6789/0 conn(0x7f610009ab50 legacy=0x7f61001397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 1 -- 172.21.15.164:0/2338949702 >> v1:172.21.15.164:6789/0 conn(0x7f610009d930 legacy=0x7f610009dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 1 monclient: found mon.noname-c 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:16:43.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.823+0000 7f6105110640 1 -- 172.21.15.164:0/2338949702 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f610009ecb0 con 0x7f610009e420 2024-09-10T16:16:43.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.826+0000 7f6105110640 1 -- 172.21.15.164:0/2338949702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009e420 msgr2=0x7f610013cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.826+0000 7f6105110640 1 --2- 172.21.15.164:0/2338949702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009e420 0x7f610013cef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60f800bb80 tx=0x7f60f802c640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb502368274d4ea server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 1 -- 172.21.15.164:0/2338949702 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f803a070 con 0x7f610009e420 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: dump: 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 1 -- 172.21.15.164:0/2338949702 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f8032770 con 0x7f610009e420 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 1 -- 172.21.15.164:0/2338949702 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f802ce60 con 0x7f610009e420 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 5 monclient: authenticate success, global_id 94330 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 10 monclient: get_monmap_and_config success 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 4 set_mon_vals no callback set 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f60ff7fe640 10 monclient: dump: 2024-09-10T16:16:43.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 10 monclient: shutdown 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 -- 172.21.15.164:0/2338949702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009e420 msgr2=0x7f610013cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 --2- 172.21.15.164:0/2338949702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009e420 0x7f610013cef0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f60f800bb80 tx=0x7f60f802c640 comp rx=0 tx=0).stop 2024-09-10T16:16:43.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 -- 172.21.15.164:0/2338949702 shutdown_connections 2024-09-10T16:16:43.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 --2- 172.21.15.164:0/2338949702 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009e420 0x7f610013cef0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 -- 172.21.15.164:0/2338949702 >> 172.21.15.164:0/2338949702 conn(0x7f61000a3110 msgr2=0x7f61000a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 -- 172.21.15.164:0/2338949702 shutdown_connections 2024-09-10T16:16:43.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.827+0000 7f6106b73640 1 -- 172.21.15.164:0/2338949702 wait complete. 2024-09-10T16:16:43.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.828+0000 7f6106b73640 10 monclient: build_initial_monmap 2024-09-10T16:16:43.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.828+0000 7f6106b73640 10 monclient: monmap: 2024-09-10T16:16:43.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:16:43.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.828+0000 7f6106b73640 1 Processor -- start 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.828+0000 7f6106b73640 1 -- start start 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.828+0000 7f6106b73640 10 monclient: init 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient: _reopen_session rank -1 2024-09-10T16:16:43.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f610009ab50 0x7f61001c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-a con 0x7f610009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009d930 0x7f6100134b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-b con 0x7f610009d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): picked mon.noname-c con 0x7f610009e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f61000a2940 con 0x7f610009ab50 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f610009ef30 con 0x7f610009d930 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f61000a2ac0 con 0x7f610009e420 2024-09-10T16:16:43.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): _renew_subs 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6106b73640 10 monclient(hunting): authenticate will time out at 1382.316124s 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f610490f640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f610009ab50 msgr2=0x7f61001c8b60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f610490f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f610009ab50 0x7f61001c8b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34702/0 (socket says 172.21.15.164:34702) 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 learned_addr learned my addr 172.21.15.164:0/4007265500 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): get_auth_request con 0x7f610009e420 auth_method 0 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009d930 0x7f6100134b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): get_auth_request con 0x7f610009d930 auth_method 0 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f60fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient(hunting): handle_auth_done global_id 114170 payload 824 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009d930 msgr2=0x7f6100134b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610009d930 0x7f6100134b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f610009ab50 msgr2=0x7f61001c8b60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f610009ab50 0x7f61001c8b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient: _finish_hunting 0 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 monclient: found mon.noname-c 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.829+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6100131630 con 0x7f610009e420 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 msgr2=0x7f61001c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f6105110640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60f800bb80 tx=0x7f60f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f8045070 con 0x7f610009e420 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: dump: 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60f8041880 con 0x7f610009e420 2024-09-10T16:16:43.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f803a040 con 0x7f610009e420 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60fd7fa640 10 monclient: dump: 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 5 monclient: authenticate success, global_id 114170 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 monclient: _renew_subs 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6100130000 con 0x7f610009e420 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.831+0000 7f60feffd640 4 set_mon_vals no callback set 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 monclient: _renew_subs 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f610012f2d0 con 0x7f610009e420 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 0x7f610002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f610009d620 con 0x7f61001350a0 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.832+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f60f8048070 con 0x7f610009e420 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f60fd7fa640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60ec03e1c0 0x7f60ec040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 0x7f610002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient: get_auth_request con 0x7f61001350a0 auth_method 0 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(356..356 src has 184..356) ==== 4739+0+0 (secure 0 0 0) 0x7f60f8075850 con 0x7f610009e420 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f60fffff640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60ec03e1c0 0x7f60ec040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f60fffff640 10 monclient: get_auth_request con 0x7f60ec03e1c0 auth_method 0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 10 monclient(hunting): handle_auth_done global_id 94339 payload 824 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.833+0000 7f610490f640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f610009dd90 con 0x7f61001350a0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f610490f640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 msgr2=0x7f610002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f610490f640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 0x7f610002d440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f000b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f60fffff640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60ec03e1c0 0x7f60ec040680 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60f40012e0 tx=0x7f60f4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0005270 con 0x7f61001350a0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f60fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f60fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.835+0000 7f60fd7fa640 10 monclient: dump: 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60f0015070 con 0x7f61001350a0 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 msgr2=0x7f610002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 0x7f610002d440 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f000b910 comp rx=0 tx=0).stop 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 msgr2=0x7f610002d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:43.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.836+0000 7f60fd7fa640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f61001350a0 0x7f610002d440 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f000b910 comp rx=0 tx=0).stop 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f6106b73640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f6106b73640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 0x7f610002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f6106b73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6100073610 con 0x7f610002d980 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 0x7f610002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient: get_auth_request con 0x7f610002d980 auth_method 0 2024-09-10T16:16:43.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.842+0000 7f610490f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.843+0000 7f610490f640 10 monclient(hunting): handle_auth_done global_id 94342 payload 824 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.843+0000 7f610490f640 1 -- 172.21.15.164:0/4007265500 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f610006daa0 con 0x7f610002d980 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f610490f640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 msgr2=0x7f610002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f610490f640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 0x7f610002dd90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60f0023410 con 0x7f610002d980 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f60fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f60fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:16:43.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.845+0000 7f60fd7fa640 10 monclient: dump: 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f60f0023410 con 0x7f610002d980 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:16:43.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 msgr2=0x7f610002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 0x7f610002dd90 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f00059d0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 msgr2=0x7f610002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f60fd7fa640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f610002d980 0x7f610002dd90 secure :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f60f000b450 tx=0x7f60f00059d0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 94, 2024-09-10T16:16:43.854 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 121345, 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.855 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.856 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:16:43.857 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:16:43.858 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.859 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.860 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:16:43.861 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.862 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:16:43.863 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:16:43.864 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.865 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:16:43.866 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:16:43.867 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:16:43.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60ec03e1c0 msgr2=0x7f60ec040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60ec03e1c0 0x7f60ec040680 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f60f40012e0 tx=0x7f60f4009040 comp rx=0 tx=0).stop 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 10 monclient: shutdown 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 msgr2=0x7f61001c5640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f60f800bb80 tx=0x7f60f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6105110640 1 -- 172.21.15.164:0/4007265500 reap_dead start 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 shutdown_connections 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 --2- 172.21.15.164:0/4007265500 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f610009e420 0x7f61001c5640 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 >> 172.21.15.164:0/4007265500 conn(0x7f61000a3110 msgr2=0x7f61000707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 shutdown_connections 2024-09-10T16:16:43.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:16:43.846+0000 7f6106b73640 1 -- 172.21.15.164:0/4007265500 wait complete. 2024-09-10T16:16:43.869 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:16:43.870 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:16:44.296 INFO:tasks.rados.rados.0.smithi165.stdout:1482: done (0 left) 2024-09-10T16:16:44.296 INFO:tasks.rados.rados.0.smithi165.stdout:1483: rollback oid 3 current snap is 166 2024-09-10T16:16:44.296 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 165 2024-09-10T16:16:44.296 INFO:tasks.rados.rados.0.smithi165.stdout:1484: write (excl) oid 2 current snap is 166 2024-09-10T16:16:44.296 INFO:tasks.rados.rados.0.smithi165.stdout:1484: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2024-09-10T16:16:44.302 INFO:tasks.rados.rados.0.smithi165.stdout:1484: writing smithi16534841-2 from 558131 to 1210558 tid 1 2024-09-10T16:16:44.308 INFO:tasks.rados.rados.0.smithi165.stdout:1484: writing smithi16534841-2 from 1673373 to 2233219 tid 2 2024-09-10T16:16:44.312 INFO:tasks.rados.rados.0.smithi165.stdout:1484: writing smithi16534841-2 from 2820808 to 2958131 tid 3 2024-09-10T16:16:44.312 INFO:tasks.rados.rados.0.smithi165.stdout:1485: read oid 5 snap 161 2024-09-10T16:16:44.313 INFO:tasks.rados.rados.0.smithi165.stdout:1485: expect (ObjNum 397 snap 130 seq_num 397) 2024-09-10T16:16:44.313 INFO:tasks.rados.rados.0.smithi165.stdout:1486: snap_remove snap 162 2024-09-10T16:16:45.298 INFO:tasks.rados.rados.0.smithi165.stdout:1486: done (3 left) 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1487: snap_create 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1483: finishing rollback tid 0 to smithi16534841-3 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1483: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1483: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 908 (ObjNum 484 snap 153 seq_num 484) dirty exists 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1484: finishing write tid 1 to smithi16534841-2 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1484: finishing write tid 2 to smithi16534841-2 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:1485: expect (ObjNum 397 snap 130 seq_num 397) 2024-09-10T16:16:45.299 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:45.349 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:45.398 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:45.447 INFO:tasks.rados.rados.0.smithi165.stdout:1484: finishing write tid 3 to smithi16534841-2 2024-09-10T16:16:45.447 INFO:tasks.rados.rados.0.smithi165.stdout:1484: finishing write tid 4 to smithi16534841-2 2024-09-10T16:16:45.447 INFO:tasks.rados.rados.0.smithi165.stdout:1484: finishing write tid 5 to smithi16534841-2 2024-09-10T16:16:45.447 INFO:tasks.rados.rados.0.smithi165.stdout:1484: oid 2 updating version 0 to 1078 2024-09-10T16:16:45.447 INFO:tasks.rados.rados.0.smithi165.stdout:1484: oid 2 version 1078 is already newer than 1077 2024-09-10T16:16:45.448 INFO:tasks.rados.rados.0.smithi165.stdout:1484: oid 2 updating version 1078 to 1080 2024-09-10T16:16:45.448 INFO:tasks.rados.rados.0.smithi165.stdout:1484: oid 2 version 1080 is already newer than 1079 2024-09-10T16:16:45.448 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1080 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:16:45.448 INFO:tasks.rados.rados.0.smithi165.stdout:1484: left oid 2 (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:46.303 INFO:tasks.rados.rados.0.smithi165.stdout:1483: done (3 left) 2024-09-10T16:16:46.303 INFO:tasks.rados.rados.0.smithi165.stdout:1484: done (2 left) 2024-09-10T16:16:46.303 INFO:tasks.rados.rados.0.smithi165.stdout:1485: done (1 left) 2024-09-10T16:16:46.303 INFO:tasks.rados.rados.0.smithi165.stdout:1487: done (0 left) 2024-09-10T16:16:46.304 INFO:tasks.rados.rados.0.smithi165.stdout:1488: write oid 25 current snap is 167 2024-09-10T16:16:46.304 INFO:tasks.rados.rados.0.smithi165.stdout:1488: seq_num 534 ranges {574938=727676,1731855=703504,2848231=126707} 2024-09-10T16:16:46.310 INFO:tasks.rados.rados.0.smithi165.stdout:1488: writing smithi16534841-25 from 574938 to 1302614 tid 1 2024-09-10T16:16:46.316 INFO:tasks.rados.rados.0.smithi165.stdout:1488: writing smithi16534841-25 from 1731855 to 2435359 tid 2 2024-09-10T16:16:46.319 INFO:tasks.rados.rados.0.smithi165.stdout:1488: writing smithi16534841-25 from 2848231 to 2974938 tid 3 2024-09-10T16:16:46.320 INFO:tasks.rados.rados.0.smithi165.stdout:1489: copy_from oid 3 from oid 16 current snap is 167 2024-09-10T16:16:46.320 INFO:tasks.rados.rados.0.smithi165.stdout:1490: snap_remove snap 164 2024-09-10T16:16:47.308 INFO:tasks.rados.rados.0.smithi165.stdout:1490: done (2 left) 2024-09-10T16:16:47.308 INFO:tasks.rados.rados.0.smithi165.stdout:1491: snap_create 2024-09-10T16:16:47.308 INFO:tasks.rados.rados.0.smithi165.stdout:1488: finishing write tid 1 to smithi16534841-25 2024-09-10T16:16:47.308 INFO:tasks.rados.rados.0.smithi165.stdout:1488: finishing write tid 2 to smithi16534841-25 2024-09-10T16:16:47.309 INFO:tasks.rados.rados.0.smithi165.stdout:1488: finishing write tid 3 to smithi16534841-25 2024-09-10T16:16:47.309 INFO:tasks.rados.rados.0.smithi165.stdout:1488: finishing write tid 4 to smithi16534841-25 2024-09-10T16:16:47.309 INFO:tasks.rados.rados.0.smithi165.stdout:1488: finishing write tid 5 to smithi16534841-25 2024-09-10T16:16:47.309 INFO:tasks.rados.rados.0.smithi165.stdout:1488: oid 25 updating version 0 to 911 2024-09-10T16:16:47.309 INFO:tasks.rados.rados.0.smithi165.stdout:1488: oid 25 updating version 911 to 912 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:1488: oid 25 version 912 is already newer than 910 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:1488: oid 25 version 912 is already newer than 909 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 912 (ObjNum 534 snap 167 seq_num 534) dirty exists 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:1488: left oid 25 (ObjNum 534 snap 167 seq_num 534) 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:1489: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:16:47.310 INFO:tasks.rados.rados.0.smithi165.stdout:1489: finishing copy_from to smithi16534841-3 2024-09-10T16:16:47.311 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 870 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-10T16:16:47.520 INFO:tasks.rados.rados.0.smithi165.stdout:1488: done (2 left) 2024-09-10T16:16:47.520 INFO:tasks.rados.rados.0.smithi165.stdout:1489: done (1 left) 2024-09-10T16:16:47.521 INFO:tasks.rados.rados.0.smithi165.stdout:1491: done (0 left) 2024-09-10T16:16:47.521 INFO:tasks.rados.rados.0.smithi165.stdout:1492: delete oid 45 current snap is 168 2024-09-10T16:16:47.525 INFO:tasks.rados.rados.0.smithi165.stdout:1492: done (0 left) 2024-09-10T16:16:47.525 INFO:tasks.rados.rados.0.smithi165.stdout:1493: snap_create 2024-09-10T16:16:49.244 INFO:tasks.rados.rados.0.smithi165.stdout:1493: done (0 left) 2024-09-10T16:16:49.244 INFO:tasks.rados.rados.0.smithi165.stdout:1494: copy_from oid 17 from oid 27 current snap is 169 2024-09-10T16:16:49.244 INFO:tasks.rados.rados.0.smithi165.stdout:1495: read oid 10 snap -1 2024-09-10T16:16:49.244 INFO:tasks.rados.rados.0.smithi165.stdout:1495: expect (ObjNum 486 snap 153 seq_num 486) 2024-09-10T16:16:49.244 INFO:tasks.rados.rados.0.smithi165.stdout:1496: write oid 1 current snap is 169 2024-09-10T16:16:49.245 INFO:tasks.rados.rados.0.smithi165.stdout:1496: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-10T16:16:49.250 INFO:tasks.rados.rados.0.smithi165.stdout:1496: writing smithi16534841-1 from 608552 to 1086726 tid 1 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1496: writing smithi16534841-1 from 1848819 to 2555992 tid 2 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1497: delete oid 26 current snap is 169 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1494: finishing copy_from to smithi16534841-17 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1494: got expected ENOENT (src dne) 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1494: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:1495: expect (ObjNum 486 snap 153 seq_num 486) 2024-09-10T16:16:49.259 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1496: finishing write tid 1 to smithi16534841-1 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1494: done (3 left) 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1495: done (2 left) 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1497: done (1 left) 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1498: write (excl) oid 7 current snap is 169 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1496: finishing write tid 2 to smithi16534841-1 2024-09-10T16:16:49.346 INFO:tasks.rados.rados.0.smithi165.stdout:1498: seq_num 538 ranges {642166=628672,1849430=710842} 2024-09-10T16:16:49.352 INFO:tasks.rados.rados.0.smithi165.stdout:1498: writing smithi16534841-7 from 642166 to 1270838 tid 1 2024-09-10T16:16:49.360 INFO:tasks.rados.rados.0.smithi165.stdout:1498: writing smithi16534841-7 from 1849430 to 2560272 tid 2 2024-09-10T16:16:49.360 INFO:tasks.rados.rados.0.smithi165.stdout:1499: write oid 21 current snap is 169 2024-09-10T16:16:49.360 INFO:tasks.rados.rados.0.smithi165.stdout:1499: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-10T16:16:49.367 INFO:tasks.rados.rados.0.smithi165.stdout:1499: writing smithi16534841-21 from 658973 to 1362894 tid 1 2024-09-10T16:16:49.373 INFO:tasks.rados.rados.0.smithi165.stdout:1499: writing smithi16534841-21 from 1907912 to 2478765 tid 2 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1499: writing smithi16534841-21 from 2985346 to 3058973 tid 3 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1500: read oid 2 snap -1 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1500: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1501: read oid 11 snap -1 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1501: expect deleted 2024-09-10T16:16:49.377 INFO:tasks.rados.rados.0.smithi165.stdout:1502: snap_create 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: finishing write tid 3 to smithi16534841-1 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: finishing write tid 4 to smithi16534841-1 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: oid 1 updating version 0 to 1073 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: oid 1 version 1073 is already newer than 1071 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: oid 1 version 1073 is already newer than 1072 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1073 (ObjNum 536 snap 169 seq_num 536) dirty exists 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1496: left oid 1 (ObjNum 536 snap 169 seq_num 536) 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: finishing write tid 1 to smithi16534841-7 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: finishing write tid 2 to smithi16534841-7 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: finishing write tid 3 to smithi16534841-7 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: finishing write tid 4 to smithi16534841-7 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: oid 7 updating version 0 to 1127 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: oid 7 version 1127 is already newer than 1125 2024-09-10T16:16:49.378 INFO:tasks.rados.rados.0.smithi165.stdout:1498: oid 7 version 1127 is already newer than 1126 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1127 (ObjNum 538 snap 169 seq_num 538) dirty exists 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:1498: left oid 7 (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:1499: finishing write tid 1 to smithi16534841-21 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:1499: finishing write tid 2 to smithi16534841-21 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:1499: finishing write tid 3 to smithi16534841-21 2024-09-10T16:16:49.379 INFO:tasks.rados.rados.0.smithi165.stdout:1499: finishing write tid 4 to smithi16534841-21 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: finishing write tid 5 to smithi16534841-21 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: oid 21 updating version 0 to 913 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: oid 21 updating version 913 to 915 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: oid 21 updating version 915 to 916 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: oid 21 version 916 is already newer than 914 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 916 (ObjNum 539 snap 169 seq_num 539) dirty exists 2024-09-10T16:16:49.380 INFO:tasks.rados.rados.0.smithi165.stdout:1499: left oid 21 (ObjNum 539 snap 169 seq_num 539) 2024-09-10T16:16:49.383 INFO:tasks.rados.rados.0.smithi165.stdout:1500: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:49.409 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1700714 2024-09-10T16:16:50.252 INFO:tasks.rados.rados.0.smithi165.stdout:1496: done (5 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1498: done (4 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1499: done (3 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1500: done (2 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1501: done (1 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1502: done (0 left) 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1503: copy_from oid 8 from oid 31 current snap is 170 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1504: write (excl) oid 35 current snap is 170 2024-09-10T16:16:50.253 INFO:tasks.rados.rados.0.smithi165.stdout:1504: seq_num 540 ranges {675780=495523,1682747=714511,3012769=63011} 2024-09-10T16:16:50.258 INFO:tasks.rados.rados.0.smithi165.stdout:1504: writing smithi16534841-35 from 675780 to 1171303 tid 1 2024-09-10T16:16:50.266 INFO:tasks.rados.rados.0.smithi165.stdout:1504: writing smithi16534841-35 from 1682747 to 2397258 tid 2 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1504: writing smithi16534841-35 from 3012769 to 3075780 tid 3 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1505: copy_from oid 28 from oid 17 current snap is 170 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1503: finishing copy_from to smithi16534841-8 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1503: got expected ENOENT (src dne) 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1503: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1504: finishing write tid 1 to smithi16534841-35 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1504: finishing write tid 2 to smithi16534841-35 2024-09-10T16:16:50.270 INFO:tasks.rados.rados.0.smithi165.stdout:1503: done (2 left) 2024-09-10T16:16:50.271 INFO:tasks.rados.rados.0.smithi165.stdout:1506: delete oid 44 current snap is 170 2024-09-10T16:16:50.271 INFO:tasks.rados.rados.0.smithi165.stdout:1504: finishing write tid 3 to smithi16534841-35 2024-09-10T16:16:50.272 INFO:tasks.rados.rados.0.smithi165.stdout:1504: finishing write tid 4 to smithi16534841-35 2024-09-10T16:16:50.272 INFO:tasks.rados.rados.0.smithi165.stdout:1504: finishing write tid 5 to smithi16534841-35 2024-09-10T16:16:50.272 INFO:tasks.rados.rados.0.smithi165.stdout:1504: oid 35 updating version 0 to 749 2024-09-10T16:16:50.272 INFO:tasks.rados.rados.0.smithi165.stdout:1504: oid 35 version 749 is already newer than 748 2024-09-10T16:16:50.273 INFO:tasks.rados.rados.0.smithi165.stdout:1504: oid 35 updating version 749 to 750 2024-09-10T16:16:50.273 INFO:tasks.rados.rados.0.smithi165.stdout:1504: oid 35 version 750 is already newer than 747 2024-09-10T16:16:50.273 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 750 (ObjNum 540 snap 170 seq_num 540) dirty exists 2024-09-10T16:16:50.273 INFO:tasks.rados.rados.0.smithi165.stdout:1504: left oid 35 (ObjNum 540 snap 170 seq_num 540) 2024-09-10T16:16:50.274 INFO:tasks.rados.rados.0.smithi165.stdout:1504: done (2 left) 2024-09-10T16:16:50.274 INFO:tasks.rados.rados.0.smithi165.stdout:1506: done (1 left) 2024-09-10T16:16:50.274 INFO:tasks.rados.rados.0.smithi165.stdout:1507: snap_create 2024-09-10T16:16:50.278 INFO:tasks.rados.rados.0.smithi165.stdout:1505: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:16:50.281 INFO:tasks.rados.rados.0.smithi165.stdout:1505: finishing copy_from to smithi16534841-28 2024-09-10T16:16:50.281 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1008 (ObjNum 369 snap 108 seq_num 369) dirty exists 2024-09-10T16:16:51.315 INFO:tasks.rados.rados.0.smithi165.stdout:1505: done (1 left) 2024-09-10T16:16:51.315 INFO:tasks.rados.rados.0.smithi165.stdout:1507: done (0 left) 2024-09-10T16:16:51.316 INFO:tasks.rados.rados.0.smithi165.stdout:1508: copy_from oid 20 from oid 4 current snap is 171 2024-09-10T16:16:51.316 INFO:tasks.rados.rados.0.smithi165.stdout:1509: read oid 29 snap -1 2024-09-10T16:16:51.316 INFO:tasks.rados.rados.0.smithi165.stdout:1509: expect (ObjNum 528 snap 163 seq_num 528) 2024-09-10T16:16:51.316 INFO:tasks.rados.rados.0.smithi165.stdout:1510: delete oid 22 current snap is 171 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1510: done (2 left) 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1511: read oid 34 snap -1 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1511: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1512: read oid 2 snap -1 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1512: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1513: copy_from oid 40 from oid 37 current snap is 171 2024-09-10T16:16:51.321 INFO:tasks.rados.rados.0.smithi165.stdout:1514: delete oid 19 current snap is 171 2024-09-10T16:16:51.324 INFO:tasks.rados.rados.0.smithi165.stdout:1509: expect (ObjNum 528 snap 163 seq_num 528) 2024-09-10T16:16:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 654065 2024-09-10T16:16:51.502 INFO:tasks.rados.rados.0.smithi165.stdout:1512: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:51.503 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:51.656 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 609430 2024-09-10T16:16:51.722 INFO:tasks.rados.rados.0.smithi165.stdout:1511: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:16:51.768 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1575253 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1508: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1508: finishing copy_from to smithi16534841-20 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 704 (ObjNum 454 snap 143 seq_num 454) dirty exists 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1513: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1513: finishing copy_from to smithi16534841-40 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1508: done (5 left) 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1509: done (4 left) 2024-09-10T16:16:51.827 INFO:tasks.rados.rados.0.smithi165.stdout:1511: done (3 left) 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1512: done (2 left) 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1513: done (1 left) 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1514: done (0 left) 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1515: copy_from oid 10 from oid 35 current snap is 171 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1516: write oid 24 current snap is 171 2024-09-10T16:16:51.828 INFO:tasks.rados.rados.0.smithi165.stdout:1516: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-10T16:16:51.834 INFO:tasks.rados.rados.0.smithi165.stdout:1516: writing smithi16534841-24 from 743008 to 1539527 tid 1 2024-09-10T16:16:51.842 INFO:tasks.rados.rados.0.smithi165.stdout:1516: writing smithi16534841-24 from 2200322 to 2922171 tid 2 2024-09-10T16:16:51.842 INFO:tasks.rados.rados.0.smithi165.stdout:1517: snap_remove snap 161 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:1517: done (2 left) 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:1518: delete oid 16 current snap is 171 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:1515: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:1515: finishing copy_from to smithi16534841-10 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 750 (ObjNum 540 snap 170 seq_num 540) dirty exists 2024-09-10T16:16:52.268 INFO:tasks.rados.rados.0.smithi165.stdout:1516: finishing write tid 1 to smithi16534841-24 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: finishing write tid 2 to smithi16534841-24 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: finishing write tid 3 to smithi16534841-24 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: finishing write tid 4 to smithi16534841-24 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: oid 24 updating version 0 to 1083 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: oid 24 version 1083 is already newer than 1082 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: oid 24 version 1083 is already newer than 1081 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1083 (ObjNum 544 snap 171 seq_num 544) dirty exists 2024-09-10T16:16:52.269 INFO:tasks.rados.rados.0.smithi165.stdout:1516: left oid 24 (ObjNum 544 snap 171 seq_num 544) 2024-09-10T16:16:52.279 INFO:tasks.rados.rados.0.smithi165.stdout:1515: done (2 left) 2024-09-10T16:16:52.279 INFO:tasks.rados.rados.0.smithi165.stdout:1516: done (1 left) 2024-09-10T16:16:52.279 INFO:tasks.rados.rados.0.smithi165.stdout:1518: done (0 left) 2024-09-10T16:16:52.280 INFO:tasks.rados.rados.0.smithi165.stdout:1519: read oid 32 snap -1 2024-09-10T16:16:52.280 INFO:tasks.rados.rados.0.smithi165.stdout:1519: expect deleted 2024-09-10T16:16:52.280 INFO:tasks.rados.rados.0.smithi165.stdout:1520: write oid 38 current snap is 171 2024-09-10T16:16:52.280 INFO:tasks.rados.rados.0.smithi165.stdout:1520: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-10T16:16:52.287 INFO:tasks.rados.rados.0.smithi165.stdout:1520: writing smithi16534841-38 from 776622 to 1323639 tid 1 2024-09-10T16:16:52.293 INFO:tasks.rados.rados.0.smithi165.stdout:1520: writing smithi16534841-38 from 1800933 to 2526451 tid 2 2024-09-10T16:16:52.297 INFO:tasks.rados.rados.0.smithi165.stdout:1520: writing smithi16534841-38 from 2944601 to 3176622 tid 3 2024-09-10T16:16:52.297 INFO:tasks.rados.rados.0.smithi165.stdout:1521: delete oid 3 current snap is 171 2024-09-10T16:16:52.297 INFO:tasks.rados.rados.0.smithi165.stdout:1520: finishing write tid 1 to smithi16534841-38 2024-09-10T16:16:52.297 INFO:tasks.rados.rados.0.smithi165.stdout:1520: finishing write tid 2 to smithi16534841-38 2024-09-10T16:16:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:1520: finishing write tid 3 to smithi16534841-38 2024-09-10T16:16:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:1519: done (2 left) 2024-09-10T16:16:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:1521: done (1 left) 2024-09-10T16:16:52.299 INFO:tasks.rados.rados.0.smithi165.stdout:1522: write (excl) oid 50 current snap is 171 2024-09-10T16:16:52.300 INFO:tasks.rados.rados.0.smithi165.stdout:1522: seq_num 548 ranges {410236=413868,1234250=729187,2715800=423063} 2024-09-10T16:16:52.304 INFO:tasks.rados.rados.0.smithi165.stdout:1522: writing smithi16534841-50 from 410236 to 824104 tid 1 2024-09-10T16:16:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:1522: writing smithi16534841-50 from 1234250 to 1963437 tid 2 2024-09-10T16:16:52.317 INFO:tasks.rados.rados.0.smithi165.stdout:1522: writing smithi16534841-50 from 2715800 to 3138863 tid 3 2024-09-10T16:16:52.317 INFO:tasks.rados.rados.0.smithi165.stdout:1523: snap_remove snap 165 2024-09-10T16:16:52.521 INFO:tasks.rados.rados.0.smithi165.stdout:1523: done (2 left) 2024-09-10T16:16:52.521 INFO:tasks.rados.rados.0.smithi165.stdout:1524: read oid 2 snap -1 2024-09-10T16:16:52.522 INFO:tasks.rados.rados.0.smithi165.stdout:1524: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:52.522 INFO:tasks.rados.rados.0.smithi165.stdout:1525: snap_remove snap 168 2024-09-10T16:16:53.524 INFO:tasks.rados.rados.0.smithi165.stdout:1525: done (3 left) 2024-09-10T16:16:53.524 INFO:tasks.rados.rados.0.smithi165.stdout:1526: rollback oid 10 current snap is 171 2024-09-10T16:16:53.524 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 170 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: finishing write tid 4 to smithi16534841-38 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: finishing write tid 5 to smithi16534841-38 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: oid 38 updating version 0 to 897 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: oid 38 version 897 is already newer than 896 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: oid 38 version 897 is already newer than 895 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: oid 38 updating version 897 to 898 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 898 (ObjNum 546 snap 171 seq_num 546) dirty exists 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1520: left oid 38 (ObjNum 546 snap 171 seq_num 546) 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: finishing write tid 1 to smithi16534841-50 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: finishing write tid 2 to smithi16534841-50 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: finishing write tid 3 to smithi16534841-50 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: finishing write tid 4 to smithi16534841-50 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: finishing write tid 5 to smithi16534841-50 2024-09-10T16:16:53.525 INFO:tasks.rados.rados.0.smithi165.stdout:1522: oid 50 updating version 0 to 596 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:1522: oid 50 version 596 is already newer than 594 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:1522: oid 50 version 596 is already newer than 593 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:1522: oid 50 version 596 is already newer than 595 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 596 (ObjNum 548 snap 171 seq_num 548) dirty exists 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:1522: left oid 50 (ObjNum 548 snap 171 seq_num 548) 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:1524: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:16:53.526 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:53.658 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2633185 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1526: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1526: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1526: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1014 (ObjNum 486 snap 153 seq_num 486) dirty exists 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1520: done (3 left) 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1522: done (2 left) 2024-09-10T16:16:53.733 INFO:tasks.rados.rados.0.smithi165.stdout:1524: done (1 left) 2024-09-10T16:16:53.734 INFO:tasks.rados.rados.0.smithi165.stdout:1526: done (0 left) 2024-09-10T16:16:53.734 INFO:tasks.rados.rados.0.smithi165.stdout:1527: write oid 28 current snap is 171 2024-09-10T16:16:53.734 INFO:tasks.rados.rados.0.smithi165.stdout:1527: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2024-09-10T16:16:53.737 INFO:tasks.rados.rados.0.smithi165.stdout:1527: writing smithi16534841-28 from 427043 to 916160 tid 1 2024-09-10T16:16:53.743 INFO:tasks.rados.rados.0.smithi165.stdout:1527: writing smithi16534841-28 from 1576379 to 2049224 tid 2 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1527: writing smithi16534841-28 from 2626870 to 3227043 tid 3 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1528: read oid 25 snap -1 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1528: expect (ObjNum 534 snap 167 seq_num 534) 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1529: read oid 8 snap -1 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1529: expect (ObjNum 509 snap 159 seq_num 509) 2024-09-10T16:16:53.750 INFO:tasks.rados.rados.0.smithi165.stdout:1530: snap_remove snap 166 2024-09-10T16:16:54.524 INFO:tasks.rados.rados.0.smithi165.stdout:1530: done (3 left) 2024-09-10T16:16:54.524 INFO:tasks.rados.rados.0.smithi165.stdout:1531: snap_create 2024-09-10T16:16:54.525 INFO:tasks.rados.rados.0.smithi165.stdout:1527: finishing write tid 1 to smithi16534841-28 2024-09-10T16:16:54.525 INFO:tasks.rados.rados.0.smithi165.stdout:1527: finishing write tid 2 to smithi16534841-28 2024-09-10T16:16:54.525 INFO:tasks.rados.rados.0.smithi165.stdout:1529: expect (ObjNum 509 snap 159 seq_num 509) 2024-09-10T16:16:54.525 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:54.590 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:54.701 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2310603 2024-09-10T16:16:54.703 INFO:tasks.rados.rados.0.smithi165.stdout:1527: finishing write tid 3 to smithi16534841-28 2024-09-10T16:16:54.703 INFO:tasks.rados.rados.0.smithi165.stdout:1527: finishing write tid 4 to smithi16534841-28 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: finishing write tid 5 to smithi16534841-28 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: oid 28 updating version 0 to 1012 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: oid 28 version 1012 is already newer than 1011 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: oid 28 version 1012 is already newer than 1009 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: oid 28 version 1012 is already newer than 1010 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1012 (ObjNum 549 snap 171 seq_num 549) dirty exists 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1527: left oid 28 (ObjNum 549 snap 171 seq_num 549) 2024-09-10T16:16:54.704 INFO:tasks.rados.rados.0.smithi165.stdout:1528: expect (ObjNum 534 snap 167 seq_num 534) 2024-09-10T16:16:54.724 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2150893 2024-09-10T16:16:54.754 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1542280 2024-09-10T16:16:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:1527: done (3 left) 2024-09-10T16:16:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:1528: done (2 left) 2024-09-10T16:16:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:1529: done (1 left) 2024-09-10T16:16:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:1531: done (0 left) 2024-09-10T16:16:55.529 INFO:tasks.rados.rados.0.smithi165.stdout:1532: delete oid 38 current snap is 172 2024-09-10T16:16:55.533 INFO:tasks.rados.rados.0.smithi165.stdout:1532: done (0 left) 2024-09-10T16:16:55.533 INFO:tasks.rados.rados.0.smithi165.stdout:1533: snap_remove snap 163 2024-09-10T16:16:56.535 INFO:tasks.rados.rados.0.smithi165.stdout:1533: done (0 left) 2024-09-10T16:16:56.535 INFO:tasks.rados.rados.0.smithi165.stdout:1534: write oid 23 current snap is 172 2024-09-10T16:16:56.535 INFO:tasks.rados.rados.0.smithi165.stdout:1534: seq_num 551 ranges {460657=639615,1693343=476514,2681716=578941} 2024-09-10T16:16:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:1534: writing smithi16534841-23 from 460657 to 1100272 tid 1 2024-09-10T16:16:56.547 INFO:tasks.rados.rados.0.smithi165.stdout:1534: writing smithi16534841-23 from 1693343 to 2169857 tid 2 2024-09-10T16:16:56.554 INFO:tasks.rados.rados.0.smithi165.stdout:1534: writing smithi16534841-23 from 2681716 to 3260657 tid 3 2024-09-10T16:16:56.554 INFO:tasks.rados.rados.0.smithi165.stdout:1535: rollback oid 33 current snap is 172 2024-09-10T16:16:56.554 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 167 2024-09-10T16:16:56.555 INFO:tasks.rados.rados.0.smithi165.stdout:1534: finishing write tid 1 to smithi16534841-23 2024-09-10T16:16:56.555 INFO:tasks.rados.rados.0.smithi165.stdout:1534: finishing write tid 2 to smithi16534841-23 2024-09-10T16:16:56.555 INFO:tasks.rados.rados.0.smithi165.stdout:1536: snap_create 2024-09-10T16:16:56.557 INFO:tasks.rados.rados.0.smithi165.stdout:1534: finishing write tid 3 to smithi16534841-23 2024-09-10T16:16:56.559 INFO:tasks.rados.rados.0.smithi165.stdout:1535: finishing rollback tid 0 to smithi16534841-33 2024-09-10T16:16:56.559 INFO:tasks.rados.rados.0.smithi165.stdout:1535: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:16:56.559 INFO:tasks.rados.rados.0.smithi165.stdout:1534: finishing write tid 4 to smithi16534841-23 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: finishing write tid 5 to smithi16534841-23 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: oid 23 updating version 0 to 1074 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: oid 23 updating version 1074 to 1077 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: oid 23 version 1077 is already newer than 1076 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: oid 23 version 1077 is already newer than 1075 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1077 (ObjNum 551 snap 172 seq_num 551) dirty exists 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1534: left oid 23 (ObjNum 551 snap 172 seq_num 551) 2024-09-10T16:16:56.560 INFO:tasks.rados.rados.0.smithi165.stdout:1535: finishing rollback tid 2 to smithi16534841-33 2024-09-10T16:16:56.561 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 937 (ObjNum 491 snap 155 seq_num 491) dirty exists 2024-09-10T16:16:57.524 INFO:tasks.rados.rados.0.smithi165.stdout:1534: done (2 left) 2024-09-10T16:16:57.524 INFO:tasks.rados.rados.0.smithi165.stdout:1535: done (1 left) 2024-09-10T16:16:57.524 INFO:tasks.rados.rados.0.smithi165.stdout:1536: done (0 left) 2024-09-10T16:16:57.524 INFO:tasks.rados.rados.0.smithi165.stdout:1537: write oid 25 current snap is 173 2024-09-10T16:16:57.524 INFO:tasks.rados.rados.0.smithi165.stdout:1537: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-10T16:16:57.530 INFO:tasks.rados.rados.0.smithi165.stdout:1537: writing smithi16534841-25 from 477464 to 1192328 tid 1 2024-09-10T16:16:57.538 INFO:tasks.rados.rados.0.smithi165.stdout:1537: writing smithi16534841-25 from 1751825 to 2488350 tid 2 2024-09-10T16:16:57.542 INFO:tasks.rados.rados.0.smithi165.stdout:1537: writing smithi16534841-25 from 3225492 to 3277464 tid 3 2024-09-10T16:16:57.543 INFO:tasks.rados.rados.0.smithi165.stdout:1538: snap_create 2024-09-10T16:16:57.543 INFO:tasks.rados.rados.0.smithi165.stdout:1537: finishing write tid 1 to smithi16534841-25 2024-09-10T16:16:57.545 INFO:tasks.rados.rados.0.smithi165.stdout:1537: finishing write tid 2 to smithi16534841-25 2024-09-10T16:16:57.550 INFO:tasks.rados.rados.0.smithi165.stdout:1537: finishing write tid 3 to smithi16534841-25 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: finishing write tid 4 to smithi16534841-25 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: finishing write tid 5 to smithi16534841-25 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: oid 25 updating version 0 to 942 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: oid 25 version 942 is already newer than 940 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: oid 25 version 942 is already newer than 941 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: oid 25 version 942 is already newer than 939 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 942 (ObjNum 552 snap 173 seq_num 552) dirty exists 2024-09-10T16:16:57.551 INFO:tasks.rados.rados.0.smithi165.stdout:1537: left oid 25 (ObjNum 552 snap 173 seq_num 552) 2024-09-10T16:16:58.535 INFO:tasks.rados.rados.0.smithi165.stdout:1537: done (1 left) 2024-09-10T16:16:58.535 INFO:tasks.rados.rados.0.smithi165.stdout:1538: done (0 left) 2024-09-10T16:16:58.535 INFO:tasks.rados.rados.0.smithi165.stdout:1539: write (excl) oid 4 current snap is 174 2024-09-10T16:16:58.535 INFO:tasks.rados.rados.0.smithi165.stdout:1539: seq_num 553 ranges {494271=790113,1693954=480183,2620209=512129} 2024-09-10T16:16:58.541 INFO:tasks.rados.rados.0.smithi165.stdout:1539: writing smithi16534841-4 from 494271 to 1284384 tid 1 2024-09-10T16:16:58.546 INFO:tasks.rados.rados.0.smithi165.stdout:1539: writing smithi16534841-4 from 1693954 to 2174137 tid 2 2024-09-10T16:16:58.552 INFO:tasks.rados.rados.0.smithi165.stdout:1539: writing smithi16534841-4 from 2620209 to 3132338 tid 3 2024-09-10T16:16:58.552 INFO:tasks.rados.rados.0.smithi165.stdout:1540: write oid 22 current snap is 174 2024-09-10T16:16:58.552 INFO:tasks.rados.rados.0.smithi165.stdout:1539: finishing write tid 1 to smithi16534841-4 2024-09-10T16:16:58.552 INFO:tasks.rados.rados.0.smithi165.stdout:1540: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2024-09-10T16:16:58.557 INFO:tasks.rados.rados.0.smithi165.stdout:1540: writing smithi16534841-22 from 511078 to 976440 tid 1 2024-09-10T16:16:58.565 INFO:tasks.rados.rados.0.smithi165.stdout:1540: writing smithi16534841-22 from 1752436 to 2492630 tid 2 2024-09-10T16:16:58.570 INFO:tasks.rados.rados.0.smithi165.stdout:1540: writing smithi16534841-22 from 3163985 to 3311078 tid 3 2024-09-10T16:16:58.570 INFO:tasks.rados.rados.0.smithi165.stdout:1541: write (excl) oid 28 current snap is 174 2024-09-10T16:16:58.570 INFO:tasks.rados.rados.0.smithi165.stdout:1539: finishing write tid 2 to smithi16534841-4 2024-09-10T16:16:58.570 INFO:tasks.rados.rados.0.smithi165.stdout:1539: finishing write tid 3 to smithi16534841-4 2024-09-10T16:16:58.571 INFO:tasks.rados.rados.0.smithi165.stdout:1539: finishing write tid 4 to smithi16534841-4 2024-09-10T16:16:58.571 INFO:tasks.rados.rados.0.smithi165.stdout:1541: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-10T16:16:58.576 INFO:tasks.rados.rados.0.smithi165.stdout:1541: writing smithi16534841-28 from 527885 to 1184849 tid 1 2024-09-10T16:16:58.583 INFO:tasks.rados.rados.0.smithi165.stdout:1541: writing smithi16534841-28 from 1927271 to 2411123 tid 2 2024-09-10T16:16:58.588 INFO:tasks.rados.rados.0.smithi165.stdout:1541: writing smithi16534841-28 from 3191408 to 3327885 tid 3 2024-09-10T16:16:58.588 INFO:tasks.rados.rados.0.smithi165.stdout:1542: snap_remove snap 173 2024-09-10T16:16:59.542 INFO:tasks.rados.rados.0.smithi165.stdout:1542: done (3 left) 2024-09-10T16:16:59.542 INFO:tasks.rados.rados.0.smithi165.stdout:1543: read oid 12 snap -1 2024-09-10T16:16:59.542 INFO:tasks.rados.rados.0.smithi165.stdout:1543: expect deleted 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1544: copy_from oid 48 from oid 46 current snap is 174 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: finishing write tid 5 to smithi16534841-4 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: oid 4 updating version 0 to 705 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: oid 4 updating version 705 to 708 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: oid 4 version 708 is already newer than 706 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: oid 4 version 708 is already newer than 707 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 708 (ObjNum 553 snap 174 seq_num 553) dirty exists 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1539: left oid 4 (ObjNum 553 snap 174 seq_num 553) 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1540: finishing write tid 1 to smithi16534841-22 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1540: finishing write tid 2 to smithi16534841-22 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1540: finishing write tid 3 to smithi16534841-22 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1540: finishing write tid 4 to smithi16534841-22 2024-09-10T16:16:59.543 INFO:tasks.rados.rados.0.smithi165.stdout:1540: finishing write tid 5 to smithi16534841-22 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1540: oid 22 updating version 0 to 752 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1540: oid 22 updating version 752 to 753 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1540: oid 22 updating version 753 to 754 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1540: oid 22 updating version 754 to 755 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 755 (ObjNum 554 snap 174 seq_num 554) dirty exists 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1540: left oid 22 (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: finishing write tid 1 to smithi16534841-28 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: finishing write tid 2 to smithi16534841-28 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: finishing write tid 3 to smithi16534841-28 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: finishing write tid 4 to smithi16534841-28 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: finishing write tid 5 to smithi16534841-28 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: oid 28 updating version 0 to 1014 2024-09-10T16:16:59.544 INFO:tasks.rados.rados.0.smithi165.stdout:1541: oid 28 updating version 1014 to 1016 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1541: oid 28 version 1016 is already newer than 1013 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1541: oid 28 version 1016 is already newer than 1015 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1016 (ObjNum 555 snap 174 seq_num 555) dirty exists 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1541: left oid 28 (ObjNum 555 snap 174 seq_num 555) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1539: done (4 left) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1540: done (3 left) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1541: done (2 left) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1545: read oid 35 snap -1 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1545: expect (ObjNum 540 snap 170 seq_num 540) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1546: read oid 24 snap -1 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1546: expect (ObjNum 544 snap 171 seq_num 544) 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1547: write (excl) oid 18 current snap is 174 2024-09-10T16:16:59.545 INFO:tasks.rados.rados.0.smithi165.stdout:1547: seq_num 556 ranges {544692=732213,1985753=743863,3335184=9508} 2024-09-10T16:16:59.549 INFO:tasks.rados.rados.0.smithi165.stdout:1547: writing smithi16534841-18 from 544692 to 1276905 tid 1 2024-09-10T16:16:59.557 INFO:tasks.rados.rados.0.smithi165.stdout:1547: writing smithi16534841-18 from 1985753 to 2729616 tid 2 2024-09-10T16:16:59.561 INFO:tasks.rados.rados.0.smithi165.stdout:1547: writing smithi16534841-18 from 3335184 to 3344692 tid 3 2024-09-10T16:16:59.561 INFO:tasks.rados.rados.0.smithi165.stdout:1548: write (excl) oid 38 current snap is 174 2024-09-10T16:16:59.561 INFO:tasks.rados.rados.0.smithi165.stdout:1548: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-10T16:16:59.566 INFO:tasks.rados.rados.0.smithi165.stdout:1548: writing smithi16534841-38 from 561499 to 968961 tid 1 2024-09-10T16:16:59.572 INFO:tasks.rados.rados.0.smithi165.stdout:1548: writing smithi16534841-38 from 1527882 to 2015403 tid 2 2024-09-10T16:16:59.579 INFO:tasks.rados.rados.0.smithi165.stdout:1548: writing smithi16534841-38 from 2729901 to 3287118 tid 3 2024-09-10T16:16:59.579 INFO:tasks.rados.rados.0.smithi165.stdout:1549: read oid 49 snap 169 2024-09-10T16:16:59.579 INFO:tasks.rados.rados.0.smithi165.stdout:1549: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:59.580 INFO:tasks.rados.rados.0.smithi165.stdout:1550: snap_create 2024-09-10T16:16:59.580 INFO:tasks.rados.rados.0.smithi165.stdout:1544: finishing copy_from to smithi16534841-48 2024-09-10T16:16:59.580 INFO:tasks.rados.rados.0.smithi165.stdout:1544: got expected ENOENT (src dne) 2024-09-10T16:16:59.580 INFO:tasks.rados.rados.0.smithi165.stdout:1544: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:16:59.580 INFO:tasks.rados.rados.0.smithi165.stdout:1546: expect (ObjNum 544 snap 171 seq_num 544) 2024-09-10T16:16:59.752 INFO:tasks.rados.rados.0.smithi165.stdout:1545: expect (ObjNum 540 snap 170 seq_num 540) 2024-09-10T16:16:59.763 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2124342 2024-09-10T16:16:59.773 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:59.807 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: finishing write tid 1 to smithi16534841-18 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: finishing write tid 2 to smithi16534841-18 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: finishing write tid 3 to smithi16534841-18 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: finishing write tid 4 to smithi16534841-18 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: finishing write tid 5 to smithi16534841-18 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: oid 18 updating version 0 to 1080 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: oid 18 version 1080 is already newer than 1078 2024-09-10T16:16:59.842 INFO:tasks.rados.rados.0.smithi165.stdout:1547: oid 18 updating version 1080 to 1081 2024-09-10T16:16:59.843 INFO:tasks.rados.rados.0.smithi165.stdout:1547: oid 18 version 1081 is already newer than 1079 2024-09-10T16:16:59.843 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1081 (ObjNum 556 snap 174 seq_num 556) dirty exists 2024-09-10T16:16:59.843 INFO:tasks.rados.rados.0.smithi165.stdout:1547: left oid 18 (ObjNum 556 snap 174 seq_num 556) 2024-09-10T16:16:59.843 INFO:tasks.rados.rados.0.smithi165.stdout:1548: finishing write tid 1 to smithi16534841-38 2024-09-10T16:16:59.843 INFO:tasks.rados.rados.0.smithi165.stdout:1548: finishing write tid 2 to smithi16534841-38 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: finishing write tid 3 to smithi16534841-38 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: finishing write tid 4 to smithi16534841-38 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: finishing write tid 5 to smithi16534841-38 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: oid 38 updating version 0 to 902 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: oid 38 version 902 is already newer than 901 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: oid 38 updating version 902 to 903 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:1548: oid 38 version 903 is already newer than 900 2024-09-10T16:16:59.844 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 903 (ObjNum 557 snap 174 seq_num 557) dirty exists 2024-09-10T16:16:59.845 INFO:tasks.rados.rados.0.smithi165.stdout:1548: left oid 38 (ObjNum 557 snap 174 seq_num 557) 2024-09-10T16:16:59.845 INFO:tasks.rados.rados.0.smithi165.stdout:1549: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:16:59.898 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:00.541 INFO:tasks.rados.rados.0.smithi165.stdout:1543: done (7 left) 2024-09-10T16:17:00.541 INFO:tasks.rados.rados.0.smithi165.stdout:1544: done (6 left) 2024-09-10T16:17:00.541 INFO:tasks.rados.rados.0.smithi165.stdout:1545: done (5 left) 2024-09-10T16:17:00.541 INFO:tasks.rados.rados.0.smithi165.stdout:1546: done (4 left) 2024-09-10T16:17:00.541 INFO:tasks.rados.rados.0.smithi165.stdout:1547: done (3 left) 2024-09-10T16:17:00.542 INFO:tasks.rados.rados.0.smithi165.stdout:1548: done (2 left) 2024-09-10T16:17:00.542 INFO:tasks.rados.rados.0.smithi165.stdout:1549: done (1 left) 2024-09-10T16:17:00.542 INFO:tasks.rados.rados.0.smithi165.stdout:1550: done (0 left) 2024-09-10T16:17:00.542 INFO:tasks.rados.rados.0.smithi165.stdout:1551: write (excl) oid 6 current snap is 175 2024-09-10T16:17:00.542 INFO:tasks.rados.rados.0.smithi165.stdout:1551: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-09-10T16:17:00.547 INFO:tasks.rados.rados.0.smithi165.stdout:1551: writing smithi16534841-6 from 578306 to 1061017 tid 1 2024-09-10T16:17:00.553 INFO:tasks.rados.rados.0.smithi165.stdout:1551: writing smithi16534841-6 from 1586364 to 2217543 tid 2 2024-09-10T16:17:00.558 INFO:tasks.rados.rados.0.smithi165.stdout:1551: writing smithi16534841-6 from 2757324 to 3242166 tid 3 2024-09-10T16:17:00.559 INFO:tasks.rados.rados.0.smithi165.stdout:1552: copy_from oid 39 from oid 12 current snap is 175 2024-09-10T16:17:00.559 INFO:tasks.rados.rados.0.smithi165.stdout:1551: finishing write tid 1 to smithi16534841-6 2024-09-10T16:17:00.559 INFO:tasks.rados.rados.0.smithi165.stdout:1551: finishing write tid 2 to smithi16534841-6 2024-09-10T16:17:00.559 INFO:tasks.rados.rados.0.smithi165.stdout:1553: snap_create 2024-09-10T16:17:00.561 INFO:tasks.rados.rados.0.smithi165.stdout:1551: finishing write tid 3 to smithi16534841-6 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: finishing write tid 4 to smithi16534841-6 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: finishing write tid 5 to smithi16534841-6 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: oid 6 updating version 0 to 626 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: oid 6 version 626 is already newer than 625 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: oid 6 version 626 is already newer than 624 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: oid 6 updating version 626 to 627 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 627 (ObjNum 558 snap 175 seq_num 558) dirty exists 2024-09-10T16:17:00.562 INFO:tasks.rados.rados.0.smithi165.stdout:1551: left oid 6 (ObjNum 558 snap 175 seq_num 558) 2024-09-10T16:17:00.563 INFO:tasks.rados.rados.0.smithi165.stdout:1552: finishing copy_from to smithi16534841-39 2024-09-10T16:17:00.563 INFO:tasks.rados.rados.0.smithi165.stdout:1552: got expected ENOENT (src dne) 2024-09-10T16:17:00.563 INFO:tasks.rados.rados.0.smithi165.stdout:1552: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:17:01.550 INFO:tasks.rados.rados.0.smithi165.stdout:1551: done (2 left) 2024-09-10T16:17:01.550 INFO:tasks.rados.rados.0.smithi165.stdout:1552: done (1 left) 2024-09-10T16:17:01.550 INFO:tasks.rados.rados.0.smithi165.stdout:1553: done (0 left) 2024-09-10T16:17:01.550 INFO:tasks.rados.rados.0.smithi165.stdout:1554: snap_remove snap 174 2024-09-10T16:17:02.524 INFO:tasks.rados.rados.0.smithi165.stdout:1554: done (0 left) 2024-09-10T16:17:02.524 INFO:tasks.rados.rados.0.smithi165.stdout:1555: copy_from oid 3 from oid 7 current snap is 176 2024-09-10T16:17:02.524 INFO:tasks.rados.rados.0.smithi165.stdout:1556: snap_remove snap 171 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1556: done (1 left) 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1557: read oid 43 snap 175 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1557: expect deleted 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1558: snap_create 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1555: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:17:03.526 INFO:tasks.rados.rados.0.smithi165.stdout:1555: finishing copy_from to smithi16534841-3 2024-09-10T16:17:03.527 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1127 (ObjNum 538 snap 169 seq_num 538) dirty exists 2024-09-10T16:17:03.871 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:17:03.871 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:17:03.871 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:17:03.874 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:17:03.874 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:17:03.874 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:17:03.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.983+0000 7fbc0fd94640 10 monclient: get_monmap_and_config 2024-09-10T16:17:03.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.983+0000 7fbc0fd94640 10 monclient: build_initial_monmap 2024-09-10T16:17:03.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.983+0000 7fbc0fd94640 10 monclient: monmap: 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:03.984157+0000 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:03.984157+0000 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:03.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.983+0000 7fbc0fd94640 1 Processor -- start 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 -- start start 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient: init 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 0x7fbc08134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-b con 0x7fbc0806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-a con 0x7fbc080a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc08137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc0809ce50 con 0x7fbc08137ca0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbc0806db60 con 0x7fbc080a46a0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbc0806cf50 con 0x7fbc0806cb70 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0fd94640 10 monclient(hunting): authenticate will time out at 1402.471758s 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 0x7fbc08134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56348/0 (socket says 172.21.15.164:56348) 2024-09-10T16:17:03.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 -- 172.21.15.164:0/2822241893 learned_addr learned my addr 172.21.15.164:0/2822241893 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 10 monclient(hunting): get_auth_request con 0x7fbc080a46a0 auth_method 0 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): get_auth_request con 0x7fbc0806cb70 auth_method 0 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 -- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 msgr2=0x7fbc080a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 -- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 msgr2=0x7fbc080a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0db30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:03.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.984+0000 7fbc0d32f640 1 --2- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0d32f640 1 --2- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 10 monclient(hunting): handle_auth_done global_id 114173 payload 274 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 1 -- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 msgr2=0x7fbc080a4a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 1 --2- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 1 -- 172.21.15.164:0/2822241893 >> v1:172.21.15.165:6789/0 conn(0x7fbc08137ca0 legacy=0x7fbc08138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 10 monclient: _finish_hunting 0 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 1 monclient: found mon.noname-b 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:03.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.985+0000 7fbc0db30640 1 -- 172.21.15.164:0/2822241893 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0809cb70 con 0x7fbc0806cb70 2024-09-10T16:17:03.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.987+0000 7fbc0db30640 1 -- 172.21.15.164:0/2822241893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 msgr2=0x7fbc08134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:03.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.987+0000 7fbc0db30640 1 --2- 172.21.15.164:0/2822241893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 0x7fbc08134f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc040097a0 tx=0x7fbc04029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fcc5b3863c0fc29 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:03.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 1 -- 172.21.15.164:0/2822241893 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04037070 con 0x7fbc0806cb70 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: dump: 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:03.998 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: _finish_auth 0 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: _check_auth_tickets 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 1 -- 172.21.15.164:0/2822241893 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc04032440 con 0x7fbc0806cb70 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 5 monclient: authenticate success, global_id 114173 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 1 -- 172.21.15.164:0/2822241893 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc04032810 con 0x7fbc0806cb70 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:03.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0cb2e640 10 monclient: dump: 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 4 set_mon_vals no callback set 2024-09-10T16:17:04.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 10 monclient: shutdown 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 -- 172.21.15.164:0/2822241893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 msgr2=0x7fbc08134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/2822241893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 0x7fbc08134f70 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbc040097a0 tx=0x7fbc04029fd0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 -- 172.21.15.164:0/2822241893 shutdown_connections 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/2822241893 >> v2:172.21.15.164:3300/0 conn(0x7fbc080a46a0 0x7fbc080a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/2822241893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc0806cb70 0x7fbc08134f70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 -- 172.21.15.164:0/2822241893 >> 172.21.15.164:0/2822241893 conn(0x7fbc080aa900 msgr2=0x7fbc080aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 -- 172.21.15.164:0/2822241893 shutdown_connections 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.988+0000 7fbc0fd94640 1 -- 172.21.15.164:0/2822241893 wait complete. 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.989+0000 7fbc0fd94640 10 monclient: build_initial_monmap 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.989+0000 7fbc0fd94640 10 monclient: monmap: 2024-09-10T16:17:04.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:04.002 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.989+0000 7fbc0fd94640 1 Processor -- start 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.989+0000 7fbc0fd94640 1 -- start start 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.989+0000 7fbc0fd94640 10 monclient: init 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-a con 0x7fbc0806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc080a46a0 0x7fbc080a3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-b con 0x7fbc080a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 0x7fbc080a40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): picked mon.noname-c con 0x7fbc08137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbc081389c0 con 0x7fbc0806cb70 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbc08138840 con 0x7fbc080a46a0 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbc08140040 con 0x7fbc08137ca0 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0fd94640 10 monclient(hunting): authenticate will time out at 1402.477271s 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56362/0 (socket says 172.21.15.164:56362) 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 -- 172.21.15.164:0/3303915751 learned_addr learned my addr 172.21.15.164:0/3303915751 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 10 monclient(hunting): get_auth_request con 0x7fbc0806cb70 auth_method 0 2024-09-10T16:17:04.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 0x7fbc080a40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): get_auth_request con 0x7fbc08137ca0 auth_method 0 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 msgr2=0x7fbc08131790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 msgr2=0x7fbc08131790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0db30640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:04.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc080a46a0 0x7fbc080a3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): get_auth_request con 0x7fbc080a46a0 auth_method 0 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0d32f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient(hunting): handle_auth_done global_id 114176 payload 824 2024-09-10T16:17:04.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc080a46a0 msgr2=0x7fbc080a3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc080a46a0 0x7fbc080a3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 msgr2=0x7fbc08131790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient: _finish_hunting 0 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 monclient: found mon.noname-c 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.990+0000 7fbc0e331640 1 -- 172.21.15.164:0/3303915751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc0806e720 con 0x7fbc08137ca0 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.992+0000 7fbc0e331640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 msgr2=0x7fbc080a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.992+0000 7fbc0e331640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 0x7fbc080a40e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c780 tx=0x7fbc0000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:04.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00016020 con 0x7fbc08137ca0 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: dump: 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:04.008 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: _finish_auth 0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc0000ac60 con 0x7fbc08137ca0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 5 monclient: authenticate success, global_id 114176 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc00012660 con 0x7fbc08137ca0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 10 monclient: _renew_subs 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc0809dc50 con 0x7fbc08137ca0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:04.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfffff640 4 set_mon_vals no callback set 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbbfe7fc640 10 monclient: dump: 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:04.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 10 monclient: _renew_subs 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.993+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc081cd3b0 con 0x7fbc08137ca0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbc00019020 con 0x7fbc08137ca0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0fd94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0fd94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0fd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc08134f70 con 0x7fbc08137ca0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbbfe7fc640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 0x7fbbec040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(377..377 src has 184..377) ==== 4783+0+0 (secure 0 0 0) 0x7fbc00052be0 con 0x7fbc08137ca0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0db30640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 0x7fbbec040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:04.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.994+0000 7fbc0db30640 10 monclient: get_auth_request con 0x7fbbec03e260 auth_method 0 2024-09-10T16:17:04.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.997+0000 7fbc0db30640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 0x7fbbec040720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009560 tx=0x7fbc0402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.999+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbc00020030 con 0x7fbc08137ca0 2024-09-10T16:17:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.999+0000 7fbbfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:04.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:03.999+0000 7fbbfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:04.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbc0fd94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:17:04.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbc0fd94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:04.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbc0fd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:04.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc08132e90 con 0x7fbc08137ca0 2024-09-10T16:17:04.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbbfe7fc640 1 -- 172.21.15.164:0/3303915751 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fbc0000ae10 con 0x7fbc08137ca0 2024-09-10T16:17:04.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbbfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:04.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.243+0000 7fbbfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:04.245 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":94,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 msgr2=0x7fbbec040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 0x7fbbec040720 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fbc04009560 tx=0x7fbc0402e000 comp rx=0 tx=0).stop 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 10 monclient: shutdown 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 msgr2=0x7fbc080a40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 0x7fbc080a40e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbc0000c780 tx=0x7fbc0000cc40 comp rx=0 tx=0).stop 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 shutdown_connections 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbbec03e260 0x7fbbec040720 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc08137ca0 0x7fbc080a40e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc080a46a0 0x7fbc080a3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 --2- 172.21.15.164:0/3303915751 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc0806cb70 0x7fbc08131790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:04.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 >> 172.21.15.164:0/3303915751 conn(0x7fbc080aa900 msgr2=0x7fbc080a91e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:04.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 shutdown_connections 2024-09-10T16:17:04.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:04.245+0000 7fbc0fd94640 1 -- 172.21.15.164:0/3303915751 wait complete. 2024-09-10T16:17:04.532 INFO:tasks.rados.rados.0.smithi165.stdout:1555: done (2 left) 2024-09-10T16:17:04.532 INFO:tasks.rados.rados.0.smithi165.stdout:1557: done (1 left) 2024-09-10T16:17:04.532 INFO:tasks.rados.rados.0.smithi165.stdout:1558: done (0 left) 2024-09-10T16:17:04.532 INFO:tasks.rados.rados.0.smithi165.stdout:1559: delete oid 21 current snap is 177 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:1559: done (0 left) 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:1560: read oid 39 snap -1 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:1560: expect (ObjNum 518 snap 159 seq_num 518) 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:1561: rollback oid 8 current snap is 177 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 175 2024-09-10T16:17:04.537 INFO:tasks.rados.rados.0.smithi165.stdout:1562: read oid 17 snap -1 2024-09-10T16:17:04.538 INFO:tasks.rados.rados.0.smithi165.stdout:1562: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:17:04.538 INFO:tasks.rados.rados.0.smithi165.stdout:1563: rollback oid 43 current snap is 177 2024-09-10T16:17:04.538 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 172 2024-09-10T16:17:04.538 INFO:tasks.rados.rados.0.smithi165.stdout:1564: write oid 42 current snap is 177 2024-09-10T16:17:04.538 INFO:tasks.rados.rados.0.smithi165.stdout:1564: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-10T16:17:04.544 INFO:tasks.rados.rados.0.smithi165.stdout:1564: writing smithi16534841-42 from 611920 to 1245129 tid 1 2024-09-10T16:17:04.550 INFO:tasks.rados.rados.0.smithi165.stdout:1564: writing smithi16534841-42 from 1703328 to 2338176 tid 2 2024-09-10T16:17:04.556 INFO:tasks.rados.rados.0.smithi165.stdout:1564: writing smithi16534841-42 from 2812170 to 3319556 tid 3 2024-09-10T16:17:04.556 INFO:tasks.rados.rados.0.smithi165.stdout:1565: snap_create 2024-09-10T16:17:04.556 INFO:tasks.rados.rados.0.smithi165.stdout:1563: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:17:04.556 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:17:04.557 INFO:tasks.rados.rados.0.smithi165.stdout:1562: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:17:04.672 INFO:tasks.rados.rados.0.smithi165.stdout:1560: expect (ObjNum 518 snap 159 seq_num 518) 2024-09-10T16:17:04.716 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 697486 2024-09-10T16:17:04.801 INFO:tasks.rados.rados.0.smithi165.stdout:1561: finishing rollback tid 0 to smithi16534841-8 2024-09-10T16:17:04.801 INFO:tasks.rados.rados.0.smithi165.stdout:1561: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:17:04.801 INFO:tasks.rados.rados.0.smithi165.stdout:1561: finishing rollback tid 2 to smithi16534841-8 2024-09-10T16:17:04.801 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1086 (ObjNum 509 snap 159 seq_num 509) dirty exists 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: finishing write tid 1 to smithi16534841-42 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: finishing write tid 2 to smithi16534841-42 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: finishing write tid 3 to smithi16534841-42 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: finishing write tid 4 to smithi16534841-42 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: finishing write tid 5 to smithi16534841-42 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: oid 42 updating version 0 to 598 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: oid 42 version 598 is already newer than 597 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: oid 42 updating version 598 to 600 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: oid 42 version 600 is already newer than 599 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 600 (ObjNum 560 snap 177 seq_num 560) dirty exists 2024-09-10T16:17:04.802 INFO:tasks.rados.rados.0.smithi165.stdout:1564: left oid 42 (ObjNum 560 snap 177 seq_num 560) 2024-09-10T16:17:07.258 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:17:07.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 10 monclient: get_monmap_and_config 2024-09-10T16:17:07.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 10 monclient: build_initial_monmap 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 10 monclient: monmap: 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:07.368093+0000 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:07.368093+0000 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:07.369 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:07.370 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:07.370 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:07.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 1 Processor -- start 2024-09-10T16:17:07.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 1 -- start start 2024-09-10T16:17:07.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.367+0000 7fae56eb1640 10 monclient: init 2024-09-10T16:17:07.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-a con 0x7fae5009c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-b con 0x7fae5009d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:07.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fae5009f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae5009fbc0 con 0x7fae5009f4c0 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fae500a2a20 con 0x7fae5009c310 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae5009c6f0 con 0x7fae5009d190 2024-09-10T16:17:07.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae56eb1640 10 monclient(hunting): authenticate will time out at 1405.855547s 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37672/0 (socket says 172.21.15.164:37672) 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55568/0 (socket says 172.21.15.164:55568) 2024-09-10T16:17:07.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 learned_addr learned my addr 172.21.15.164:0/1614399398 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:07.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae5009c310 auth_method 0 2024-09-10T16:17:07.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:07.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae5009d190 auth_method 0 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:07.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:07.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.368+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:07.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.370+0000 7fae5544e640 1 -- 172.21.15.164:0/1614399398 >> v1:172.21.15.165:6789/0 conn(0x7fae5009f4c0 legacy=0x7fae5013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae5009c310 auth_method 0 2024-09-10T16:17:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae5009d190 auth_method 0 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:17:07.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.569+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae5009d190 auth_method 0 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:07.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae5009c310 auth_method 0 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:17:07.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:07.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:07.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:07.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:07.970+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:17:08.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:08.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae5009d190 auth_method 0 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae5009c310 auth_method 0 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:17:08.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:08.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:08.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:08.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:08.771+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:17:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3244574254 0 0) 0x7fae48001820 con 0x7fae5009f4c0 2024-09-10T16:17:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 10 monclient(hunting): _negotiate 2024-09-10T16:17:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:09.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:09.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 10 monclient(hunting): my global_id is 94348 2024-09-10T16:17:09.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:09.601+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae40003520 con 0x7fae5009f4c0 2024-09-10T16:17:10.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3635292072 0 0) 0x7fae40003520 con 0x7fae5009f4c0 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 msgr2=0x7fae5009ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5009d190 0x7fae5009ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 msgr2=0x7fae5009ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 --2- 172.21.15.164:0/1614399398 >> v2:172.21.15.164:3300/0 conn(0x7fae5009c310 0x7fae5009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:17:10.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5009a4d0 con 0x7fae5009f4c0 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fae48001f40 con 0x7fae5009f4c0 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: dump: 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 >> v1:172.21.15.165:6789/0 conn(0x7fae5009f4c0 legacy=0x7fae5013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae40003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): picked mon.c con 0x7fae40003a70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae40004800 0x7fae40004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): picked mon.b con 0x7fae40004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae400051c0 0x7fae4000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): picked mon.a con 0x7fae400051c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:10.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fae500a2a20 con 0x7fae400051c0 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fae5009c6f0 con 0x7fae40004800 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae4000c110 con 0x7fae40003a70 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae40003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae400051c0 0x7fae4000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae40003a70 auth_method 0 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae40004800 0x7fae40004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): get_auth_request con 0x7fae400051c0 auth_method 0 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae40004800 auth_method 0 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae4ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.208+0000 7fae5544e640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 10 monclient(hunting): handle_auth_done global_id 94348 payload 293 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae40004800 msgr2=0x7fae40004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae40004800 0x7fae40004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae400051c0 msgr2=0x7fae4000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae400051c0 0x7fae4000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 monclient: found mon.c 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae40003520 con 0x7fae40003a70 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae54c4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae56eb1640 5 monclient: authenticate success, global_id 94348 2024-09-10T16:17:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae56eb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.209+0000 7fae5544e640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae400051c0 0x7fae4000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae54c4d640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 msgr2=0x7fae40003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae40003e50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae440129f0 tx=0x7fae40005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adccda3f90e85452 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44031070 con 0x7fae40003a70 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: dump: 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae4402b920 con 0x7fae40003a70 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae56eb1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae56eb1640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 1 -- 172.21.15.164:0/1614399398 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4400f040 con 0x7fae40003a70 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae56eb1640 4 set_mon_vals no callback set 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.212+0000 7fae4f7fe640 10 monclient: dump: 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 10 monclient: shutdown 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 -- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 msgr2=0x7fae40003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1614399398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae40003e50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fae440129f0 tx=0x7fae40005830 comp rx=0 tx=0).stop 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae5544e640 1 -- 172.21.15.164:0/1614399398 reap_dead start 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 -- 172.21.15.164:0/1614399398 shutdown_connections 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 -- 172.21.15.164:0/1614399398 >> 172.21.15.164:0/1614399398 conn(0x7fae500a3110 msgr2=0x7fae500a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 -- 172.21.15.164:0/1614399398 shutdown_connections 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.213+0000 7fae56eb1640 1 -- 172.21.15.164:0/1614399398 wait complete. 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.214+0000 7fae56eb1640 10 monclient: build_initial_monmap 2024-09-10T16:17:10.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.214+0000 7fae56eb1640 10 monclient: monmap: 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:10.229 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.214+0000 7fae56eb1640 1 Processor -- start 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.214+0000 7fae56eb1640 1 -- start start 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.214+0000 7fae56eb1640 10 monclient: init 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-c con 0x7fae40003a70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae501c8970 0x7fae50099950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-b con 0x7fae501c8970 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:10.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): picked mon.noname-a con 0x7fae50090200 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fae500a2230 con 0x7fae50090200 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fae500a20b0 con 0x7fae501c8970 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae50074e00 con 0x7fae40003a70 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae56eb1640 10 monclient(hunting): authenticate will time out at 1408.702388s 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55610/0 (socket says 172.21.15.164:55610) 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 -- 172.21.15.164:0/1918965078 learned_addr learned my addr 172.21.15.164:0/1918965078 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): get_auth_request con 0x7fae40003a70 auth_method 0 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 10 monclient(hunting): get_auth_request con 0x7fae50090200 auth_method 0 2024-09-10T16:17:10.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 msgr2=0x7fae50090690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 msgr2=0x7fae50090690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae5544e640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae501c8970 0x7fae50099950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae4ffff640 10 monclient(hunting): get_auth_request con 0x7fae501c8970 auth_method 0 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient(hunting): handle_auth_done global_id 114179 payload 824 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae501c8970 msgr2=0x7fae50099950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae501c8970 0x7fae50099950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 msgr2=0x7fae50090690 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 monclient: found mon.noname-c 2024-09-10T16:17:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.215+0000 7fae54c4d640 1 -- 172.21.15.164:0/1918965078 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5009d620 con 0x7fae40003a70 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae54c4d640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 msgr2=0x7fae50099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae54c4d640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae44037040 tx=0x7fae440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae44044020 con 0x7fae40003a70 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: dump: 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae44040a90 con 0x7fae40003a70 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 5 monclient: authenticate success, global_id 114179 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 10 monclient: _renew_subs 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae50099e90 con 0x7fae40003a70 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae4403e440 con 0x7fae40003a70 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4d7fa640 10 monclient: dump: 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 10 monclient: _renew_subs 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae4effd640 4 set_mon_vals no callback set 2024-09-10T16:17:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.218+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fae50093600 con 0x7fae40003a70 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fae44047020 con 0x7fae40003a70 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae56eb1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae56eb1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae56eb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae500a2c60 con 0x7fae40003a70 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.219+0000 7fae4d7fa640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 0x7fae34040780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.220+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(378..378 src has 184..378) ==== 4755+0+0 (secure 0 0 0) 0x7fae44074100 con 0x7fae40003a70 2024-09-10T16:17:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.220+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 0x7fae34040780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.220+0000 7fae4ffff640 10 monclient: get_auth_request con 0x7fae3403e2c0 auth_method 0 2024-09-10T16:17:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.222+0000 7fae4ffff640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 0x7fae34040780 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fae501c8d50 tx=0x7fae3c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.224+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fae500a2c60 con 0x7fae40003a70 2024-09-10T16:17:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.224+0000 7fae4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.224+0000 7fae4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae56eb1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae56eb1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae56eb1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fae5009d190 con 0x7fae40003a70 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae4d7fa640 1 -- 172.21.15.164:0/1918965078 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fae4404e030 con 0x7fae40003a70 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:10.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.472+0000 7fae4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:10.475 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:17:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 msgr2=0x7fae34040780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 0x7fae34040780 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fae501c8d50 tx=0x7fae3c008040 comp rx=0 tx=0).stop 2024-09-10T16:17:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 10 monclient: shutdown 2024-09-10T16:17:10.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 msgr2=0x7fae50099410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fae44037040 tx=0x7fae440046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 shutdown_connections 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae3403e2c0 0x7fae34040780 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae50090200 0x7fae50090690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae501c8970 0x7fae50099950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 --2- 172.21.15.164:0/1918965078 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae40003a70 0x7fae50099410 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 >> 172.21.15.164:0/1918965078 conn(0x7fae500a3110 msgr2=0x7fae500707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 shutdown_connections 2024-09-10T16:17:10.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.475+0000 7fae56eb1640 1 -- 172.21.15.164:0/1918965078 wait complete. 2024-09-10T16:17:10.487 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:17:10.487 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:17:10.488 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.586+0000 7fbb10b89640 10 monclient: get_monmap_and_config 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.586+0000 7fbb10b89640 10 monclient: build_initial_monmap 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.586+0000 7fbb10b89640 10 monclient: monmap: 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:10.587542+0000 2024-09-10T16:17:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:10.587542+0000 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:10.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 Processor -- start 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 -- start start 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient: init 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:17:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb0c09c310 0x7fbb0c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0c09c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbb0c09d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbb0c13a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb0c09fbc0 con 0x7fbb0c13a5c0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb0c0a2a20 con 0x7fbb0c09d190 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbb0c09c6f0 con 0x7fbb0c09c310 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb10b89640 10 monclient(hunting): authenticate will time out at 1409.074722s 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb0c09c310 0x7fbb0c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbb0c13a5c0 0x7fbb0c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:33956/0 (socket says 172.21.15.164:33956) 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0ad9d640 1 -- 172.21.15.164:0/3344725222 learned_addr learned my addr 172.21.15.164:0/3344725222 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): get_auth_request con 0x7fbb0c09c310 auth_method 0 2024-09-10T16:17:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.587+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_done global_id 124098 payload 274 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 1 -- 172.21.15.164:0/3344725222 >> v1:172.21.15.165:6789/0 conn(0x7fbb0c09d190 legacy=0x7fbb0c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 1 -- 172.21.15.164:0/3344725222 >> v1:172.21.15.164:6789/0 conn(0x7fbb0c13a5c0 legacy=0x7fbb0c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 1 monclient: found mon.noname-a 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.588+0000 7fbb0a59c640 1 -- 172.21.15.164:0/3344725222 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c09a4d0 con 0x7fbb0c09c310 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.590+0000 7fbb0a59c640 1 -- 172.21.15.164:0/3344725222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 msgr2=0x7fbb0c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.590+0000 7fbb0a59c640 1 --2- 172.21.15.164:0/3344725222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c09ea40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0097a0 tx=0x7fbafc029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d53b652f0987e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 1 -- 172.21.15.164:0/3344725222 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc037070 con 0x7fbb0c09c310 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: dump: 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.596 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 1 -- 172.21.15.164:0/3344725222 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbafc032640 con 0x7fbb0c09c310 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 5 monclient: authenticate success, global_id 124098 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 1 -- 172.21.15.164:0/3344725222 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc032a10 con 0x7fbb0c09c310 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:10.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 4 set_mon_vals no callback set 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb09d9b640 10 monclient: dump: 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 10 monclient: shutdown 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 -- 172.21.15.164:0/3344725222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 msgr2=0x7fbb0c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 --2- 172.21.15.164:0/3344725222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c09ea40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0097a0 tx=0x7fbafc029fd0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 -- 172.21.15.164:0/3344725222 shutdown_connections 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 --2- 172.21.15.164:0/3344725222 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c09ea40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 -- 172.21.15.164:0/3344725222 >> 172.21.15.164:0/3344725222 conn(0x7fbb0c0a3110 msgr2=0x7fbb0c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 -- 172.21.15.164:0/3344725222 shutdown_connections 2024-09-10T16:17:10.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.591+0000 7fbb10b89640 1 -- 172.21.15.164:0/3344725222 wait complete. 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.592+0000 7fbb10b89640 10 monclient: build_initial_monmap 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.592+0000 7fbb10b89640 10 monclient: monmap: 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:10.605 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 Processor -- start 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 -- start start 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient: init 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c1395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-a con 0x7fbb0c09c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb0c13a5c0 0x7fbb0c139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-b con 0x7fbb0c13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb0c097f00 0x7fbb0c13a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): picked mon.noname-c con 0x7fbb0c097f00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c0a2230 con 0x7fbb0c09c310 2024-09-10T16:17:10.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c0a20b0 con 0x7fbb0c13a5c0 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbb0c074e00 con 0x7fbb0c097f00 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb10b89640 10 monclient(hunting): authenticate will time out at 1409.080660s 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c1395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c1395d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37742/0 (socket says 172.21.15.164:37742) 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 1 -- 172.21.15.164:0/2052779989 learned_addr learned my addr 172.21.15.164:0/2052779989 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb0c097f00 0x7fbb0c13a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): get_auth_request con 0x7fbb0c09c310 auth_method 0 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): get_auth_request con 0x7fbb0c097f00 auth_method 0 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.593+0000 7fbb0a59c640 10 monclient(hunting): handle_auth_done global_id 124101 payload 824 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb0c097f00 msgr2=0x7fbb0c13a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb0c097f00 0x7fbb0c13a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb0c13a5c0 msgr2=0x7fbb0c139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb0c13a5c0 0x7fbb0c139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 monclient: found mon.noname-a 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.594+0000 7fbb0a59c640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c095430 con 0x7fbb0c09c310 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb0a59c640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 msgr2=0x7fbb0c1395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb0a59c640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c1395d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098c0 tx=0x7fbafc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc042020 con 0x7fbb0c09c310 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: dump: 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbafc030880 con 0x7fbb0c09c310 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 5 monclient: authenticate success, global_id 124101 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 10 monclient: _renew_subs 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c0974b0 con 0x7fbb0c09c310 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbafc037040 con 0x7fbb0c09c310 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb037fe640 10 monclient: dump: 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.611 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 10 monclient: _renew_subs 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb0959a640 4 set_mon_vals no callback set 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.596+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c0971d0 con 0x7fbb0c09c310 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb10b89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb10b89640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 0x7fbb0c094130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c09ba10 con 0x7fbb0c09d190 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 0x7fbb0c094130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient: get_auth_request con 0x7fbb0c09d190 auth_method 0 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbafc045020 con 0x7fbb0c09c310 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_done global_id 124104 payload 824 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbb0c09d620 con 0x7fbb0c09d190 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb037fe640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbaec03e270 0x7fbaec040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.597+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(378..378 src has 184..378) ==== 4755+0+0 (secure 0 0 0) 0x7fbafc073450 con 0x7fbb0c09c310 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.598+0000 7fbb0ad9d640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbaec03e270 0x7fbaec040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.598+0000 7fbb0ad9d640 10 monclient: get_auth_request con 0x7fbaec03e270 auth_method 0 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.599+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 msgr2=0x7fbb0c094130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.599+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 0x7fbb0c094130 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c06e780 tx=0x7fbaf400c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.599+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4007220 con 0x7fbb0c09d190 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.599+0000 7fbb037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:10.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 10 monclient: dump: 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbaf4007220 con 0x7fbb0c09d190 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:10.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 msgr2=0x7fbb0c094130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 0x7fbb0c094130 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c06e780 tx=0x7fbaf400c7a0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 msgr2=0x7fbb0c094130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb037fe640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09d190 0x7fbb0c094130 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c06e780 tx=0x7fbaf400c7a0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.600+0000 7fbb0ad9d640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbaec03e270 0x7fbaec040730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c098390 tx=0x7fbb04009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb10b89640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb10b89640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb10b89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c0a5cf0 con 0x7fbb0c02d440 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 msgr2=0x7fbb0c02d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:17:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 msgr2=0x7fbb0c02d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:10.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.606+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient: get_auth_request con 0x7fbb0c02d440 auth_method 0 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 10 monclient(hunting): handle_auth_done global_id 124107 payload 824 2024-09-10T16:17:10.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.808+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbb0c09f800 con 0x7fbb0c02d440 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb01d9b640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 msgr2=0x7fbb0c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb01d9b640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c06e780 tx=0x7fbaf4001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbaf4019870 con 0x7fbb0c02d440 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 10 monclient: dump: 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.812 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fbaf4019870 con 0x7fbb0c02d440 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 msgr2=0x7fbb0c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.810+0000 7fbb037fe640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c06e780 tx=0x7fbaf4001020 comp rx=0 tx=0).stop 2024-09-10T16:17:10.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb037fe640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 msgr2=0x7fbb0c02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb037fe640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c02d440 0x7fbb0c02d820 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 6844, 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:10.814 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.815 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.816 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:10.817 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.818 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:10.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:10.820 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:10.822 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.824 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:10.825 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:17:10.826 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.826 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.826 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:10.826 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbaec03e270 msgr2=0x7fbaec040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbaec03e270 0x7fbaec040730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c098390 tx=0x7fbb04009040 comp rx=0 tx=0).stop 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 10 monclient: shutdown 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 msgr2=0x7fbb0c1395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 --2- 172.21.15.164:0/2052779989 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb0c09c310 0x7fbb0c1395d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098c0 tx=0x7fbafc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb0ad9d640 1 -- 172.21.15.164:0/2052779989 reap_dead start 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb0ad9d640 1 -- 172.21.15.164:0/2052779989 reap_dead start 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 shutdown_connections 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 >> 172.21.15.164:0/2052779989 conn(0x7fbb0c0a3110 msgr2=0x7fbb0c0997a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 shutdown_connections 2024-09-10T16:17:10.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.811+0000 7fbb10b89640 1 -- 172.21.15.164:0/2052779989 wait complete. 2024-09-10T16:17:10.828 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:10.828 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:17:10.828 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:17:10.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.927+0000 7f3d5e470640 10 monclient: get_monmap_and_config 2024-09-10T16:17:10.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.927+0000 7f3d5e470640 10 monclient: build_initial_monmap 2024-09-10T16:17:10.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.927+0000 7f3d5e470640 10 monclient: monmap: 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:10.928109+0000 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:10.928109+0000 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:10.929 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 Processor -- start 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 -- start start 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient: init 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d5806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d58134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d5813e740 0x7f3d58140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5813e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d5809ce50 con 0x7f3d5806cb70 2024-09-10T16:17:10.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d58135380 con 0x7f3d58134f70 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3d5806db60 con 0x7f3d5813e740 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5e470640 10 monclient(hunting): authenticate will time out at 1409.415689s 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d57fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3d5806cb70 0x7f3d580a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:33972/0 (socket says 172.21.15.164:33972) 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d57fff640 1 -- 172.21.15.164:0/2272523424 learned_addr learned my addr 172.21.15.164:0/2272523424 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5ca0d640 1 --2- 172.21.15.164:0/2272523424 >> v2:172.21.15.165:3300/0 conn(0x7f3d5813e740 0x7f3d58140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5ca0d640 10 monclient(hunting): get_auth_request con 0x7f3d5813e740 auth_method 0 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5ca0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5ca0d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.928+0000 7f3d5ca0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_done global_id 94354 payload 274 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/2272523424 >> v1:172.21.15.165:6789/0 conn(0x7f3d58134f70 legacy=0x7f3d5813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/2272523424 >> v1:172.21.15.164:6789/0 conn(0x7f3d5806cb70 legacy=0x7f3d580a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 1 monclient: found mon.noname-c 2024-09-10T16:17:10.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:17:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.929+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/2272523424 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5809cb70 con 0x7f3d5813e740 2024-09-10T16:17:10.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/2272523424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d5813e740 msgr2=0x7f3d58140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5ca0d640 1 --2- 172.21.15.164:0/2272523424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d5813e740 0x7f3d58140b30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009dc0 tx=0x7f3d5002a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d255eca4c4fc80e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 1 -- 172.21.15.164:0/2272523424 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50007a30 con 0x7f3d5813e740 2024-09-10T16:17:10.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:10.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: dump: 2024-09-10T16:17:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.955 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 5 monclient: authenticate success, global_id 94354 2024-09-10T16:17:10.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:10.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 1 -- 172.21.15.164:0/2272523424 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d50007bd0 con 0x7f3d5813e740 2024-09-10T16:17:10.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:10.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 4 set_mon_vals no callback set 2024-09-10T16:17:10.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 1 -- 172.21.15.164:0/2272523424 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50031400 con 0x7f3d5813e740 2024-09-10T16:17:10.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:17:10.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d577fe640 10 monclient: dump: 2024-09-10T16:17:10.962 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.963 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 10 monclient: shutdown 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:10.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 1 -- 172.21.15.164:0/2272523424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d5813e740 msgr2=0x7f3d58140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.932+0000 7f3d5e470640 1 --2- 172.21.15.164:0/2272523424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d5813e740 0x7f3d58140b30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009dc0 tx=0x7f3d5002a4d0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 -- 172.21.15.164:0/2272523424 shutdown_connections 2024-09-10T16:17:10.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 --2- 172.21.15.164:0/2272523424 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d5813e740 0x7f3d58140b30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 -- 172.21.15.164:0/2272523424 >> 172.21.15.164:0/2272523424 conn(0x7f3d580aa900 msgr2=0x7f3d580aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:10.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 -- 172.21.15.164:0/2272523424 shutdown_connections 2024-09-10T16:17:10.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 -- 172.21.15.164:0/2272523424 wait complete. 2024-09-10T16:17:10.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 10 monclient: build_initial_monmap 2024-09-10T16:17:10.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 10 monclient: monmap: 2024-09-10T16:17:10.966 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:10.966 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 Processor -- start 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 1 -- start start 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.933+0000 7f3d5e470640 10 monclient: init 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 0x7f3d581cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-c con 0x7f3d5806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d58134f70 0x7f3d580a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-b con 0x7f3d58134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): picked mon.noname-a con 0x7f3d5813e740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:10.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d580a4db0 con 0x7f3d5813e740 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5806cf50 con 0x7f3d58134f70 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d58141070 con 0x7f3d5806cb70 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5e470640 10 monclient(hunting): authenticate will time out at 1409.421473s 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 0x7f3d581cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 0x7f3d581cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55624/0 (socket says 172.21.15.164:55624) 2024-09-10T16:17:10.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 1 -- 172.21.15.164:0/1547184963 learned_addr learned my addr 172.21.15.164:0/1547184963 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37778/0 (socket says 172.21.15.164:37778) 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): get_auth_request con 0x7f3d5806cb70 auth_method 0 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): get_auth_request con 0x7f3d5813e740 auth_method 0 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d58134f70 0x7f3d580a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): get_auth_request con 0x7f3d58134f70 auth_method 0 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient(hunting): handle_auth_done global_id 124110 payload 824 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 msgr2=0x7f3d581cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 0x7f3d581cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d58134f70 msgr2=0x7f3d580a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d58134f70 0x7f3d580a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient: _finish_hunting 0 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 monclient: found mon.noname-a 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d58140bd0 con 0x7f3d5813e740 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d57fff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5806cb70 0x7f3d581cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.934+0000 7f3d4ffff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d58134f70 0x7f3d580a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 msgr2=0x7f3d580a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5ca0d640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3d50031000 tx=0x7f3d50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50040070 con 0x7f3d5813e740 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: dump: 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.974 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: _finish_auth 0 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: _check_auth_tickets 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d50045460 con 0x7f3d5813e740 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 5 monclient: authenticate success, global_id 124110 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 10 monclient: _renew_subs 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5813c690 con 0x7f3d5813e740 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:10.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d50007a30 con 0x7f3d5813e740 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d557fa640 10 monclient: dump: 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.976 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d56ffd640 4 set_mon_vals no callback set 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 10 monclient: _renew_subs 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.937+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5809d980 con 0x7f3d5813e740 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d5e470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d5e470640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 0x7f3d5813aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d580a46a0 con 0x7f3d5813ac10 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 0x7f3d5813aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient: get_auth_request con 0x7f3d5813ac10 auth_method 0 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3d50039040 con 0x7f3d5813e740 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 10 monclient(hunting): handle_auth_done global_id 114188 payload 824 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d57fff640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d5813dd20 con 0x7f3d5813ac10 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d557fa640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d3803e210 0x7f3d380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.938+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(378..378 src has 184..378) ==== 4755+0+0 (secure 0 0 0) 0x7f3d50074680 con 0x7f3d5813e740 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.939+0000 7f3d4ffff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d3803e210 0x7f3d380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.939+0000 7f3d4ffff640 10 monclient: get_auth_request con 0x7f3d3803e210 auth_method 0 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.940+0000 7f3d57fff640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 msgr2=0x7f3d5813aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.940+0000 7f3d57fff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 0x7f3d5813aff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800eae0 tx=0x7f3d4800c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48017070 con 0x7f3d5813ac10 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 10 monclient: dump: 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.979 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d4800ce10 con 0x7f3d5813ac10 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:10.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 msgr2=0x7f3d5813aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 0x7f3d5813aff0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800eae0 tx=0x7f3d4800c150 comp rx=0 tx=0).stop 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 msgr2=0x7f3d5813aff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d557fa640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813ac10 0x7f3d5813aff0 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800eae0 tx=0x7f3d4800c150 comp rx=0 tx=0).stop 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.941+0000 7f3d4ffff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d3803e210 0x7f3d380406d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3d580a9a90 tx=0x7f3d40007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d5e470640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d5e470640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 0x7f3d58136880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d5e470640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d5809dd40 con 0x7f3d5813b530 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 0x7f3d58136880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient: get_auth_request con 0x7f3d5813b530 auth_method 0 2024-09-10T16:17:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 10 monclient(hunting): handle_auth_done global_id 114191 payload 824 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.947+0000 7f3d57fff640 1 -- 172.21.15.164:0/1547184963 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d5813d1f0 con 0x7f3d5813b530 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.949+0000 7f3d57fff640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 msgr2=0x7f3d58136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.949+0000 7f3d57fff640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 0x7f3d58136880 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d48019000 tx=0x7f3d480058d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48017070 con 0x7f3d5813b530 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 10 monclient: dump: 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f3d48013470 con 0x7f3d5813b530 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 msgr2=0x7f3d58136880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 0x7f3d58136880 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3d48019000 tx=0x7f3d480058d0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 msgr2=0x7f3d58136880 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:10.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d557fa640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d5813b530 0x7f3d58136880 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:10.986 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:10.986 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:10.986 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:10.986 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:17:10.986 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 51519, 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:17:10.987 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.988 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:10.989 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:10.991 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:10.992 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.993 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.994 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.995 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.996 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:10.997 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:10.998 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:10.999 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:11.000 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d3803e210 msgr2=0x7f3d380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d3803e210 0x7f3d380406d0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f3d580a9a90 tx=0x7f3d40007e30 comp rx=0 tx=0).stop 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 10 monclient: shutdown 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 msgr2=0x7f3d580a3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3d50031000 tx=0x7f3d50004a40 comp rx=0 tx=0).stop 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5ca0d640 1 -- 172.21.15.164:0/1547184963 reap_dead start 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 shutdown_connections 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 --2- 172.21.15.164:0/1547184963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d5813e740 0x7f3d580a3cd0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 >> 172.21.15.164:0/1547184963 conn(0x7f3d580aa900 msgr2=0x7f3d580a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 shutdown_connections 2024-09-10T16:17:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:10.950+0000 7f3d5e470640 1 -- 172.21.15.164:0/1547184963 wait complete. 2024-09-10T16:17:11.003 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:11.003 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:17:11.003 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:17:11.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.101+0000 7f96a66cf640 10 monclient: get_monmap_and_config 2024-09-10T16:17:11.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.101+0000 7f96a66cf640 10 monclient: build_initial_monmap 2024-09-10T16:17:11.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.101+0000 7f96a66cf640 10 monclient: monmap: 2024-09-10T16:17:11.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:11.102039+0000 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:11.102039+0000 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:11.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.101+0000 7f96a66cf640 1 Processor -- start 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 -- start start 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient: init 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 0x7f96a0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-c con 0x7f96a006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96a00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 0x7f96a01383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-b con 0x7f96a00a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96a009ce50 con 0x7f96a00a46a0 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f96a006db60 con 0x7f96a006cb70 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96a006cf50 con 0x7f96a00a4ab0 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a66cf640 10 monclient(hunting): authenticate will time out at 1409.589604s 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 0x7f96a01383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f96a00a46a0 0x7f96a0137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:33980/0 (socket says 172.21.15.164:33980) 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969f7fe640 1 -- 172.21.15.164:0/1779900079 learned_addr learned my addr 172.21.15.164:0/1779900079 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): get_auth_request con 0x7f96a00a4ab0 auth_method 0 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969ffff640 1 --2- 172.21.15.164:0/1779900079 >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 0x7f96a0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969ffff640 10 monclient(hunting): get_auth_request con 0x7f96a006cb70 auth_method 0 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f969ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 10 monclient(hunting): handle_auth_done global_id 114194 payload 274 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 1 -- 172.21.15.164:0/1779900079 >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 msgr2=0x7f96a0134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.102+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/1779900079 >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 0x7f96a0134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 1 -- 172.21.15.164:0/1779900079 >> v1:172.21.15.164:6789/0 conn(0x7f96a00a46a0 legacy=0x7f96a0137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 10 monclient: _finish_hunting 0 2024-09-10T16:17:11.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 1 monclient: found mon.noname-b 2024-09-10T16:17:11.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:11.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f96a4c6c640 1 -- 172.21.15.164:0/1779900079 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a009cb70 con 0x7f96a00a4ab0 2024-09-10T16:17:11.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.103+0000 7f969ffff640 1 --2- 172.21.15.164:0/1779900079 >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 0x7f96a0134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:17:11.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f96a4c6c640 1 -- 172.21.15.164:0/1779900079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 msgr2=0x7f96a01383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/1779900079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 0x7f96a01383b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9698009dc0 tx=0x7f969802a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2fb009ecde2a1ce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 1 -- 172.21.15.164:0/1779900079 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698007950 con 0x7f96a00a4ab0 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: dump: 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.116 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: _finish_auth 0 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: _check_auth_tickets 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 1 -- 172.21.15.164:0/1779900079 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9698007af0 con 0x7f96a00a4ab0 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 1 -- 172.21.15.164:0/1779900079 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9698031400 con 0x7f96a00a4ab0 2024-09-10T16:17:11.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f96a66cf640 5 monclient: authenticate success, global_id 114194 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f96a66cf640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f96a66cf640 4 set_mon_vals no callback set 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.105+0000 7f969effd640 10 monclient: dump: 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.118 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 10 monclient: shutdown 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 -- 172.21.15.164:0/1779900079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 msgr2=0x7f96a01383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 --2- 172.21.15.164:0/1779900079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 0x7f96a01383b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9698009dc0 tx=0x7f969802a4d0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 -- 172.21.15.164:0/1779900079 shutdown_connections 2024-09-10T16:17:11.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 --2- 172.21.15.164:0/1779900079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a4ab0 0x7f96a01383b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 --2- 172.21.15.164:0/1779900079 >> v2:172.21.15.165:3300/0 conn(0x7f96a006cb70 0x7f96a0134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 -- 172.21.15.164:0/1779900079 >> 172.21.15.164:0/1779900079 conn(0x7f96a00aa900 msgr2=0x7f96a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 -- 172.21.15.164:0/1779900079 shutdown_connections 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.106+0000 7f96a66cf640 1 -- 172.21.15.164:0/1779900079 wait complete. 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.107+0000 7f96a66cf640 10 monclient: build_initial_monmap 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.107+0000 7f96a66cf640 10 monclient: monmap: 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.120 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.107+0000 7f96a66cf640 1 Processor -- start 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.107+0000 7f96a66cf640 1 -- start start 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.107+0000 7f96a66cf640 10 monclient: init 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 0x7f96a0131680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-a con 0x7f96a006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 0x7f96a013dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-c con 0x7f96a00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:11.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00a4ab0 0x7f96a013e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): picked mon.noname-b con 0x7f96a00a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:11.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96a0138a70 con 0x7f96a006cb70 2024-09-10T16:17:11.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96a01388f0 con 0x7f96a00a4ab0 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96a0140040 con 0x7f96a00a46a0 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a66cf640 10 monclient(hunting): authenticate will time out at 1409.595134s 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 0x7f96a0131680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 0x7f96a013dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 0x7f96a0131680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37780/0 (socket says 172.21.15.164:37780) 2024-09-10T16:17:11.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 0x7f96a013dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55660/0 (socket says 172.21.15.164:55660) 2024-09-10T16:17:11.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 learned_addr learned my addr 172.21.15.164:0/812871612 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): get_auth_request con 0x7f96a006cb70 auth_method 0 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): get_auth_request con 0x7f96a00a46a0 auth_method 0 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00a4ab0 0x7f96a013e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 10 monclient(hunting): get_auth_request con 0x7f96a00a4ab0 auth_method 0 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient(hunting): handle_auth_done global_id 114197 payload 824 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00a4ab0 msgr2=0x7f96a013e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00a4ab0 0x7f96a013e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 msgr2=0x7f96a0131680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 0x7f96a0131680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969f7fe640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96a009f2f0 con 0x7f96a00a46a0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96a006cb70 0x7f96a0131680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.108+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a00a4ab0 0x7f96a013e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969f7fe640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 msgr2=0x7f96a013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969f7fe640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 0x7f96a013dcb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f969400c410 tx=0x7f969400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9694017070 con 0x7f96a00a46a0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: dump: 2024-09-10T16:17:11.127 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: _finish_auth 0 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:11.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96940070d0 con 0x7f96a00a46a0 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 5 monclient: authenticate success, global_id 114197 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 10 monclient: _renew_subs 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a006e720 con 0x7f96a00a46a0 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96940074c0 con 0x7f96a00a46a0 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969cff9640 10 monclient: dump: 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.129 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 10 monclient: _renew_subs 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96a00a4250 con 0x7f96a00a46a0 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.110+0000 7f969e7fc640 4 set_mon_vals no callback set 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f96a66cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f96a66cf640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96a0134f70 con 0x7f96a0131bc0 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f969401a070 con 0x7f96a00a46a0 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.111+0000 7f969cff9640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f968c03e210 0x7f968c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 10 monclient: get_auth_request con 0x7f96a0131bc0 auth_method 0 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(378..378 src has 184..378) ==== 4755+0+0 (secure 0 0 0) 0x7f9694052120 con 0x7f96a00a46a0 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 msgr2=0x7f96a013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 msgr2=0x7f96a013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f968c03e210 0x7f968c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.112+0000 7f96a4c6c640 10 monclient: get_auth_request con 0x7f968c03e210 auth_method 0 2024-09-10T16:17:11.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.114+0000 7f96a4c6c640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f968c03e210 0x7f968c0406d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9698009d90 tx=0x7f969802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.210 INFO:tasks.rados.rados.0.smithi165.stdout:1560: done (5 left) 2024-09-10T16:17:11.210 INFO:tasks.rados.rados.0.smithi165.stdout:1561: done (4 left) 2024-09-10T16:17:11.210 INFO:tasks.rados.rados.0.smithi165.stdout:1562: done (3 left) 2024-09-10T16:17:11.211 INFO:tasks.rados.rados.0.smithi165.stdout:1563: done (2 left) 2024-09-10T16:17:11.211 INFO:tasks.rados.rados.0.smithi165.stdout:1564: done (1 left) 2024-09-10T16:17:11.211 INFO:tasks.rados.rados.0.smithi165.stdout:1565: done (0 left) 2024-09-10T16:17:11.211 INFO:tasks.rados.rados.0.smithi165.stdout:1566: write (excl) oid 21 current snap is 178 2024-09-10T16:17:11.211 INFO:tasks.rados.rados.0.smithi165.stdout:1566: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-10T16:17:11.217 INFO:tasks.rados.rados.0.smithi165.stdout:1566: writing smithi16534841-21 from 628727 to 1337185 tid 1 2024-09-10T16:17:11.223 INFO:tasks.rados.rados.0.smithi165.stdout:1566: writing smithi16534841-21 from 2045457 to 2540316 tid 2 2024-09-10T16:17:11.228 INFO:tasks.rados.rados.0.smithi165.stdout:1566: writing smithi16534841-21 from 3123240 to 3428727 tid 3 2024-09-10T16:17:11.228 INFO:tasks.rados.rados.0.smithi165.stdout:1567: write (excl) oid 24 current snap is 178 2024-09-10T16:17:11.228 INFO:tasks.rados.rados.0.smithi165.stdout:1567: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2024-09-10T16:17:11.235 INFO:tasks.rados.rados.0.smithi165.stdout:1567: writing smithi16534841-24 from 645534 to 1429241 tid 1 2024-09-10T16:17:11.243 INFO:tasks.rados.rados.0.smithi165.stdout:1567: writing smithi16534841-24 from 2103939 to 2742456 tid 2 2024-09-10T16:17:11.248 INFO:tasks.rados.rados.0.smithi165.stdout:1567: writing smithi16534841-24 from 3150663 to 3445534 tid 3 2024-09-10T16:17:11.248 INFO:tasks.rados.rados.0.smithi165.stdout:1568: read oid 14 snap -1 2024-09-10T16:17:11.248 INFO:tasks.rados.rados.0.smithi165.stdout:1568: expect (ObjNum 510 snap 159 seq_num 510) 2024-09-10T16:17:11.248 INFO:tasks.rados.rados.0.smithi165.stdout:1569: copy_from oid 45 from oid 19 current snap is 178 2024-09-10T16:17:11.248 INFO:tasks.rados.rados.0.smithi165.stdout:1567: finishing write tid 1 to smithi16534841-24 2024-09-10T16:17:11.249 INFO:tasks.rados.rados.0.smithi165.stdout:1570: write (excl) oid 30 current snap is 178 2024-09-10T16:17:11.249 INFO:tasks.rados.rados.0.smithi165.stdout:1567: finishing write tid 2 to smithi16534841-24 2024-09-10T16:17:11.249 INFO:tasks.rados.rados.0.smithi165.stdout:1570: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-10T16:17:11.254 INFO:tasks.rados.rados.0.smithi165.stdout:1570: writing smithi16534841-30 from 662341 to 1237650 tid 1 2024-09-10T16:17:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:1570: writing smithi16534841-30 from 1878774 to 2377302 tid 2 2024-09-10T16:17:11.268 INFO:tasks.rados.rados.0.smithi165.stdout:1570: writing smithi16534841-30 from 2894439 to 3351994 tid 3 2024-09-10T16:17:11.268 INFO:tasks.rados.rados.0.smithi165.stdout:1571: rollback oid 38 current snap is 178 2024-09-10T16:17:11.268 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 170 2024-09-10T16:17:11.268 INFO:tasks.rados.rados.0.smithi165.stdout:1567: finishing write tid 3 to smithi16534841-24 2024-09-10T16:17:11.268 INFO:tasks.rados.rados.0.smithi165.stdout:1567: finishing write tid 4 to smithi16534841-24 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:1567: finishing write tid 5 to smithi16534841-24 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:1567: oid 24 updating version 0 to 1090 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:1567: oid 24 version 1090 is already newer than 1088 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:1567: oid 24 version 1090 is already newer than 1089 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:1567: oid 24 version 1090 is already newer than 1087 2024-09-10T16:17:11.269 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1090 (ObjNum 562 snap 178 seq_num 562) dirty exists 2024-09-10T16:17:11.270 INFO:tasks.rados.rados.0.smithi165.stdout:1567: left oid 24 (ObjNum 562 snap 178 seq_num 562) 2024-09-10T16:17:11.270 INFO:tasks.rados.rados.0.smithi165.stdout:1570: finishing write tid 1 to smithi16534841-30 2024-09-10T16:17:11.270 INFO:tasks.rados.rados.0.smithi165.stdout:1570: finishing write tid 2 to smithi16534841-30 2024-09-10T16:17:11.270 INFO:tasks.rados.rados.0.smithi165.stdout:1567: done (5 left) 2024-09-10T16:17:11.270 INFO:tasks.rados.rados.0.smithi165.stdout:1572: snap_remove snap 176 2024-09-10T16:17:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.313+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.313+0000 7f969ffff640 10 monclient: get_auth_request con 0x7f96a0131bc0 auth_method 0 2024-09-10T16:17:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.313+0000 7f969ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.313+0000 7f969ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.313+0000 7f969ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.314+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.314+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.314+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.314+0000 7f969ffff640 10 monclient(hunting): handle_auth_done global_id 94369 payload 824 2024-09-10T16:17:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.314+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96a00008d0 con 0x7f96a0131bc0 2024-09-10T16:17:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.316+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 msgr2=0x7f96a013e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.316+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9690006d40 tx=0x7f9690010620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f969001b070 con 0x7f96a0131bc0 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 10 monclient: dump: 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9690015660 con 0x7f96a0131bc0 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 msgr2=0x7f96a013e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f9690006d40 tx=0x7f9690010620 comp rx=0 tx=0).stop 2024-09-10T16:17:11.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 msgr2=0x7f96a013e730 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:11.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.317+0000 7f969cff9640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a0131bc0 0x7f96a013e730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f96a66cf640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f96a66cf640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 0x7f96a0138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f96a66cf640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96a009dc40 con 0x7f96a013ec70 2024-09-10T16:17:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 0x7f96a0138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f969ffff640 10 monclient: get_auth_request con 0x7f96a013ec70 auth_method 0 2024-09-10T16:17:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f969ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f969ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.323+0000 7f969ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.324+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.324+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.324+0000 7f969ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.324+0000 7f969ffff640 10 monclient(hunting): handle_auth_done global_id 94372 payload 824 2024-09-10T16:17:11.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.324+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96a013f080 con 0x7f96a013ec70 2024-09-10T16:17:11.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969ffff640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 msgr2=0x7f96a0138db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969ffff640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 0x7f96a0138db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f969001d000 tx=0x7f9690004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9690015e00 con 0x7f96a013ec70 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 10 monclient: dump: 2024-09-10T16:17:11.332 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.333 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f9690015e00 con 0x7f96a013ec70 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 msgr2=0x7f96a0138db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 0x7f96a0138db0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f969001d000 tx=0x7f9690004730 comp rx=0 tx=0).stop 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 msgr2=0x7f96a0138db0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:11.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.326+0000 7f969cff9640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96a013ec70 0x7f96a0138db0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.335 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:17:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 148826, 2024-09-10T16:17:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:11.342 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:11.342 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:11.342 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.342 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:11.343 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:11.344 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:11.344 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:11.344 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:11.344 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:11.345 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:11.346 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:11.347 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.349 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:11.349 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:11.350 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.350 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.350 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.350 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:11.351 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:11.351 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:11.352 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:11.352 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:11.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:11.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:11.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:11.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:11.356 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:11.358 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:11.359 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:11.359 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:11.359 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:11.359 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f968c03e210 msgr2=0x7f968c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f968c03e210 0x7f968c0406d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f9698009d90 tx=0x7f969802f000 comp rx=0 tx=0).stop 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 10 monclient: shutdown 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 msgr2=0x7f96a013dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 --2- 172.21.15.164:0/812871612 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96a00a46a0 0x7f96a013dcb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f969400c410 tx=0x7f969400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a4c6c640 1 -- 172.21.15.164:0/812871612 reap_dead start 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a4c6c640 1 -- 172.21.15.164:0/812871612 reap_dead start 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 shutdown_connections 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 >> 172.21.15.164:0/812871612 conn(0x7f96a00aa900 msgr2=0x7f96a01334a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 shutdown_connections 2024-09-10T16:17:11.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.327+0000 7f96a66cf640 1 -- 172.21.15.164:0/812871612 wait complete. 2024-09-10T16:17:11.362 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:11.362 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:17:11.362 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:17:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 10 monclient: get_monmap_and_config 2024-09-10T16:17:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 10 monclient: build_initial_monmap 2024-09-10T16:17:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 10 monclient: monmap: 2024-09-10T16:17:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:11.460903+0000 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:11.460903+0000 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:11.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 1 Processor -- start 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 1 -- start start 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.460+0000 7f1a41608640 10 monclient: init 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a3c0a4540 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3c1385c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1a3c0a60b0 0x7f1a3c13f430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-c con 0x7f1a3c0a60b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a3c0a71c0 con 0x7f1a3c0a4540 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1a3c09c9f0 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3c09ccd0 con 0x7f1a3c1385c0 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a41608640 10 monclient(hunting): authenticate will time out at 1409.948241s 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55672/0 (socket says 172.21.15.164:55672) 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1850601522 learned_addr learned my addr 172.21.15.164:0/1850601522 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): get_auth_request con 0x7f1a3c1385c0 auth_method 0 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3b7fe640 1 --2- 172.21.15.164:0/1850601522 >> v2:172.21.15.165:3300/0 conn(0x7f1a3c0a60b0 0x7f1a3c13f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3b7fe640 10 monclient(hunting): get_auth_request con 0x7f1a3c0a60b0 auth_method 0 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_done global_id 114200 payload 274 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1850601522 >> v2:172.21.15.165:3300/0 conn(0x7f1a3c0a60b0 msgr2=0x7f1a3c13f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 --2- 172.21.15.164:0/1850601522 >> v2:172.21.15.165:3300/0 conn(0x7f1a3c0a60b0 0x7f1a3c13f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1850601522 >> v1:172.21.15.165:6789/0 conn(0x7f1a3c0a4540 legacy=0x7f1a3c0a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:17:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 monclient: found mon.noname-b 2024-09-10T16:17:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.461+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1850601522 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c09cfb0 con 0x7f1a3c1385c0 2024-09-10T16:17:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1850601522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 msgr2=0x7f1a3c13cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a3a7fc640 1 --2- 172.21.15.164:0/1850601522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009870 tx=0x7f1a2402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45b3e515343b452a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 1 -- 172.21.15.164:0/1850601522 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24037070 con 0x7f1a3c1385c0 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: dump: 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.474 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: _finish_auth 0 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 1 -- 172.21.15.164:0/1850601522 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a24032440 con 0x7f1a3c1385c0 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a41608640 5 monclient: authenticate success, global_id 114200 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a41608640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 1 -- 172.21.15.164:0/1850601522 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24032810 con 0x7f1a3c1385c0 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a41608640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a41608640 4 set_mon_vals no callback set 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:11.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.464+0000 7f1a39ffb640 10 monclient: dump: 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 10 monclient: shutdown 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:11.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 -- 172.21.15.164:0/1850601522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 msgr2=0x7f1a3c13cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 --2- 172.21.15.164:0/1850601522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1a24009870 tx=0x7f1a2402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 -- 172.21.15.164:0/1850601522 shutdown_connections 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 --2- 172.21.15.164:0/1850601522 >> v2:172.21.15.165:3300/0 conn(0x7f1a3c0a60b0 0x7f1a3c13f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 --2- 172.21.15.164:0/1850601522 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c1385c0 0x7f1a3c13cee0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 -- 172.21.15.164:0/1850601522 >> 172.21.15.164:0/1850601522 conn(0x7f1a3c0aaa00 msgr2=0x7f1a3c0aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 -- 172.21.15.164:0/1850601522 shutdown_connections 2024-09-10T16:17:11.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 1 -- 172.21.15.164:0/1850601522 wait complete. 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 10 monclient: build_initial_monmap 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.465+0000 7f1a41608640 10 monclient: monmap: 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:11.478 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 Processor -- start 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 -- start start 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient: init 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a3c0a4540 0x7f1a3c13d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-b con 0x7f1a3c0a4540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-c con 0x7f1a3c0a60b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 0x7f1a3c1cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): picked mon.noname-a con 0x7f1a3c13e460 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c13faf0 con 0x7f1a3c13e460 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c13f970 con 0x7f1a3c0a4540 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3c13fc70 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a41608640 10 monclient(hunting): authenticate will time out at 1409.953728s 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55686/0 (socket says 172.21.15.164:55686) 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 0x7f1a3c1cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1566322292 learned_addr learned my addr 172.21.15.164:0/1566322292 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): get_auth_request con 0x7f1a3c0a60b0 auth_method 0 2024-09-10T16:17:11.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): get_auth_request con 0x7f1a3c13e460 auth_method 0 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3affd640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a3c0a4540 0x7f1a3c13d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3affd640 10 monclient(hunting): get_auth_request con 0x7f1a3c0a4540 auth_method 0 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.466+0000 7f1a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 10 monclient(hunting): handle_auth_done global_id 114203 payload 824 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a3c0a4540 msgr2=0x7f1a3c13d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a3c0a4540 0x7f1a3c13d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 msgr2=0x7f1a3c1cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 0x7f1a3c1cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 monclient: found mon.noname-c 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1566322292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c09f2a0 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.467+0000 7f1a3b7fe640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 0x7f1a3c1cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a3a7fc640 1 -- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 msgr2=0x7f1a3c13df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a3a7fc640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a240090b0 tx=0x7f1a240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24041020 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:11.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: dump: 2024-09-10T16:17:11.484 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.484 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a2403d630 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 5 monclient: authenticate success, global_id 114203 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 10 monclient: _renew_subs 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a3c132d40 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a24037040 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a337fe640 10 monclient: dump: 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:11.486 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 10 monclient: _renew_subs 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a397fa640 4 set_mon_vals no callback set 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.469+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a3c09efc0 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1a24044020 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a41608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a41608640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a41608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a3c0a4920 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a337fe640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 0x7f1a1c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.470+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(379..379 src has 184..379) ==== 4755+0+0 (secure 0 0 0) 0x7f1a24072d80 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.471+0000 7f1a3affd640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 0x7f1a1c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.471+0000 7f1a3affd640 10 monclient: get_auth_request con 0x7f1a1c03e210 auth_method 0 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.473+0000 7f1a3affd640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 0x7f1a1c0406d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c09e930 tx=0x7f1a34003d90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:11.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.475+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1a2404b070 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.475+0000 7f1a337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:11.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.475+0000 7f1a337fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.723+0000 7f1a41608640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:17:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.723+0000 7f1a41608640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:17:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.723+0000 7f1a41608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:11.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.723+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1a3c0a4270 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f1a337fe640 1 -- 172.21.15.164:0/1566322292 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1a2403b3f0 con 0x7f1a3c0a60b0 2024-09-10T16:17:11.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f1a337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:17:11.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f1a337fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3118567747,config=2586702516,health=3946847016,kv=805391194,logm=1267137774}) 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=19,config=2,health=12,kv=11,logm=56} crc {auth=1628891706,config=575665002,health=2974672015,kv=2165536312,logm=1241305396}) 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=19,config=2,health=11,kv=12,logm=56} crc {auth=3118567747,config=2586702516,health=3946847016,kv=805391194,logm=1267137774}) 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.724+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=2767957836,config=2586702516,health=2974672015,kv=805391194,logm=2062467251}) 2024-09-10T16:17:11.727 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151921192}) 2024-09-10T16:17:11.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098768711}) 2024-09-10T16:17:11.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1151921192}) 2024-09-10T16:17:11.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.725+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963752090}) 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605122976}) 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2900578692}) 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605122976}) 2024-09-10T16:17:11.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1450399482}) 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 msgr2=0x7f1a1c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 0x7f1a1c0406d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1a3c09e930 tx=0x7f1a34003d90 comp rx=0 tx=0).stop 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 10 monclient: shutdown 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 msgr2=0x7f1a3c13df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1a240090b0 tx=0x7f1a240046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 shutdown_connections 2024-09-10T16:17:11.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1a1c03e210 0x7f1a1c0406d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1a3c13e460 0x7f1a3c1cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1a3c0a60b0 0x7f1a3c13df20 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.726+0000 7f1a41608640 1 --2- 172.21.15.164:0/1566322292 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1a3c0a4540 0x7f1a3c13d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 >> 172.21.15.164:0/1566322292 conn(0x7f1a3c0aaa00 msgr2=0x7f1a3c0a8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 shutdown_connections 2024-09-10T16:17:11.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f1a41608640 1 -- 172.21.15.164:0/1566322292 wait complete. 2024-09-10T16:17:11.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770083531}) 2024-09-10T16:17:11.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=773917612}) 2024-09-10T16:17:11.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3770083531}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.727+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=110178011}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693815888}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757011146}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693815888}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.728+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946049011}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063308575}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3863160191}) 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4063308575}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2977303555}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719454559}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4160853350}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2719454559}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.729+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=913870820}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369688007}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1197666124}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3369688007}) 2024-09-10T16:17:11.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.730+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=464090274}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367540327}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=35336872}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367540327}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.731+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544306071}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269378949}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2688276072}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269378949}) 2024-09-10T16:17:11.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3161935315}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784859713}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3952830040}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784859713}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.732+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862002926}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.733+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.733+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226671013}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.733+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1160267683}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226671013}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3395681087}) 2024-09-10T16:17:11.739 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403801661}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2562086639}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.734+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1403801661}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4049393956}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981167527}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1517123376}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=981167527}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.735+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4028975457}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045305955}) 2024-09-10T16:17:11.740 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742301044}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045305955}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.736+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2214867434}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899138433}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3025659617}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1899138433}) 2024-09-10T16:17:11.741 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.737+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=594949996}) 2024-09-10T16:17:11.742 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.742 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682494280}) 2024-09-10T16:17:11.742 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3281866389}) 2024-09-10T16:17:11.742 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1682494280}) 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.738+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=347542356}) 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248213569}) 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4010011324}) 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248213569}) 2024-09-10T16:17:11.743 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.739+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525624458}) 2024-09-10T16:17:11.744 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:17:11.748 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.748 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773609419}) 2024-09-10T16:17:11.748 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512384489}) 2024-09-10T16:17:11.748 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.748 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2773609419}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3217190863}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676082990}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2309210746}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3676082990}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.740+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1946850757}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780872840}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291115525}) 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.749 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1780872840}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.741+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876865499}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961160310}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2506695023}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961160310}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.742+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3543635174}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885389690}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645545481}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2885389690}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1100412728}) 2024-09-10T16:17:11.750 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646419812}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3216079115}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646419812}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.743+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775829541}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476338123}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2013610473}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476338123}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.744+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=991826765}) 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.751 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216832658}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136955837}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216832658}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.745+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=33682346}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425598532}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1018370955}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425598532}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3007039579}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470913413}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=725149840}) 2024-09-10T16:17:11.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1470913413}) 2024-09-10T16:17:11.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.746+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1935186257}) 2024-09-10T16:17:11.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020236822}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=483591501}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3020236822}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.748+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3587159996}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901927949}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=719905036}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1901927949}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.749+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=480593497}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=968732846,mdsmap=931935893,mgr=2907349437,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1081436939}) 2024-09-10T16:17:11.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=3794347948,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2614661302}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=83,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=968732846,mdsmap=931935893,mgr=2907349437,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1081436939}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.750+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=80,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7} crc {logm=2003977324,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2226140193}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgrstat=967409518,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2308925921,osd_snap=3780120800}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=7,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgrstat=967409518,mon_config_key=1307060516,monmap=1433040829,nvmeofgw=598337686,osd_metadata=633904022,osd_pg_creating=2308925921,osd_snap=2554058072}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=7,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgrstat=967409518,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2308925921,osd_snap=3780120800}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.751+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=7,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=66} crc {mgrstat=967409518,mon_config_key=1614167405,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2308925921,osd_snap=3176713297}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=2476088531,osdmap=1390735896}) 2024-09-10T16:17:11.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=6,osdmap=94} crc {osd_snap=3755604208,osdmap=2774959419}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=2476088531,osdmap=1390735896}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.752+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=2476088531,osdmap=1455430352}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2211307219}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2350215323}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2211307219}) 2024-09-10T16:17:11.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.753+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=900692381}) 2024-09-10T16:17:11.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2310097854}) 2024-09-10T16:17:11.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=99} crc {osdmap=2502195781}) 2024-09-10T16:17:11.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2310097854}) 2024-09-10T16:17:11.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.758+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=774788085}) 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=2} crc {osdmap=4251672972}) 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=3} crc {osdmap=2967390722}) 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=2} crc {osdmap=4251672972}) 2024-09-10T16:17:11.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:17:11.763+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=3} crc {osdmap=2967390722}) 2024-09-10T16:17:12.470 INFO:tasks.rados.rados.0.smithi165.stdout:1572: done (5 left) 2024-09-10T16:17:12.470 INFO:tasks.rados.rados.0.smithi165.stdout:1573: snap_create 2024-09-10T16:17:12.470 INFO:tasks.rados.rados.0.smithi165.stdout:1571: finishing rollback tid 0 to smithi16534841-38 2024-09-10T16:17:12.470 INFO:tasks.rados.rados.0.smithi165.stdout:1571: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:17:12.470 INFO:tasks.rados.rados.0.smithi165.stdout:1571: finishing rollback tid 2 to smithi16534841-38 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 906 (ObjNum 526 snap 162 seq_num 526) dirty exists 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:1570: finishing write tid 3 to smithi16534841-30 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:1570: finishing write tid 4 to smithi16534841-30 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:1570: finishing write tid 5 to smithi16534841-30 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:1570: oid 30 updating version 0 to 1170 2024-09-10T16:17:12.471 INFO:tasks.rados.rados.0.smithi165.stdout:1570: oid 30 updating version 1170 to 1172 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1570: oid 30 version 1172 is already newer than 1171 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1570: oid 30 updating version 1172 to 1173 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1173 (ObjNum 563 snap 178 seq_num 563) dirty exists 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1570: left oid 30 (ObjNum 563 snap 178 seq_num 563) 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1569: finishing copy_from to smithi16534841-45 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1569: got expected ENOENT (src dne) 2024-09-10T16:17:12.472 INFO:tasks.rados.rados.0.smithi165.stdout:1569: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:17:12.473 INFO:tasks.rados.rados.0.smithi165.stdout:1566: finishing write tid 1 to smithi16534841-21 2024-09-10T16:17:12.473 INFO:tasks.rados.rados.0.smithi165.stdout:1566: finishing write tid 2 to smithi16534841-21 2024-09-10T16:17:12.473 INFO:tasks.rados.rados.0.smithi165.stdout:1568: expect (ObjNum 510 snap 159 seq_num 510) 2024-09-10T16:17:12.474 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 847373 2024-09-10T16:17:12.493 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:12.521 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:12.548 INFO:tasks.rados.rados.0.smithi165.stdout:1566: finishing write tid 3 to smithi16534841-21 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: finishing write tid 4 to smithi16534841-21 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: finishing write tid 5 to smithi16534841-21 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: oid 21 updating version 0 to 1129 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: oid 21 updating version 1129 to 1131 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: oid 21 updating version 1131 to 1132 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:1566: oid 21 version 1132 is already newer than 1130 2024-09-10T16:17:12.549 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1132 (ObjNum 561 snap 178 seq_num 561) dirty exists 2024-09-10T16:17:12.550 INFO:tasks.rados.rados.0.smithi165.stdout:1566: left oid 21 (ObjNum 561 snap 178 seq_num 561) 2024-09-10T16:17:12.745 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:17:12.746 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:17:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.842+0000 7f63d2aed640 10 monclient: get_monmap_and_config 2024-09-10T16:17:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.842+0000 7f63d2aed640 10 monclient: build_initial_monmap 2024-09-10T16:17:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.842+0000 7f63d2aed640 10 monclient: monmap: 2024-09-10T16:17:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:12.843836+0000 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:12.843836+0000 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:12.846 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.843+0000 7f63d2aed640 1 Processor -- start 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.843+0000 7f63d2aed640 1 -- start start 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.843+0000 7f63d2aed640 10 monclient: init 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-b con 0x7f63cc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63cc09d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 0x7f63cc13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-a con 0x7f63cc13a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63cc09a430 con 0x7f63cc09d160 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f63cc0a24e0 con 0x7f63cc13a5c0 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63cc0a2bf0 con 0x7f63cc09c180 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d2aed640 10 monclient(hunting): authenticate will time out at 1411.331146s 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 0x7f63cc13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55690/0 (socket says 172.21.15.164:55690) 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 -- 172.21.15.164:0/1572350453 learned_addr learned my addr 172.21.15.164:0/1572350453 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): get_auth_request con 0x7f63cc13a5c0 auth_method 0 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): get_auth_request con 0x7f63cc09c180 auth_method 0 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient(hunting): handle_auth_done global_id 114206 payload 274 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 -- 172.21.15.164:0/1572350453 >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 msgr2=0x7f63cc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 --2- 172.21.15.164:0/1572350453 >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 0x7f63cc13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 -- 172.21.15.164:0/1572350453 >> v1:172.21.15.164:6789/0 conn(0x7f63cc09d160 legacy=0x7f63cc09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient: _finish_hunting 0 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 monclient: found mon.noname-b 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d0889640 1 -- 172.21.15.164:0/1572350453 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc09c560 con 0x7f63cc09c180 2024-09-10T16:17:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.844+0000 7f63d108a640 1 --2- 172.21.15.164:0/1572350453 >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 0x7f63cc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d0889640 1 -- 172.21.15.164:0/1572350453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d0889640 1 --2- 172.21.15.164:0/1572350453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63cc09abb0 tx=0x7f63c002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57cb75dd7e21cb2e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 1 -- 172.21.15.164:0/1572350453 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0037070 con 0x7f63cc09c180 2024-09-10T16:17:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:12.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: dump: 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:12.865 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 5 monclient: authenticate success, global_id 114206 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 1 -- 172.21.15.164:0/1572350453 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c0032440 con 0x7f63cc09c180 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:12.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 4 set_mon_vals no callback set 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 1 -- 172.21.15.164:0/1572350453 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c00327f0 con 0x7f63cc09c180 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:12.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63cb7fe640 10 monclient: dump: 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:12.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:12.869 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:12.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 10 monclient: shutdown 2024-09-10T16:17:12.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:12.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 1 -- 172.21.15.164:0/1572350453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.847+0000 7f63d2aed640 1 --2- 172.21.15.164:0/1572350453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f63cc09abb0 tx=0x7f63c002a080 comp rx=0 tx=0).stop 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 -- 172.21.15.164:0/1572350453 shutdown_connections 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 --2- 172.21.15.164:0/1572350453 >> v2:172.21.15.164:3300/0 conn(0x7f63cc13a5c0 0x7f63cc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 --2- 172.21.15.164:0/1572350453 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc09ea40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 -- 172.21.15.164:0/1572350453 >> 172.21.15.164:0/1572350453 conn(0x7f63cc0a3110 msgr2=0x7f63cc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 -- 172.21.15.164:0/1572350453 shutdown_connections 2024-09-10T16:17:12.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 -- 172.21.15.164:0/1572350453 wait complete. 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 10 monclient: build_initial_monmap 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 10 monclient: monmap: 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:12.871 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 Processor -- start 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 1 -- start start 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.848+0000 7f63d2aed640 10 monclient: init 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-c con 0x7f63cc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63cc13cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-a con 0x7f63cc09d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:12.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63cc13d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): picked mon.noname-b con 0x7f63cc13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc074f90 con 0x7f63cc09d160 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63cc074e10 con 0x7f63cc13a5c0 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63cc09f690 con 0x7f63cc09c180 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:12.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d2aed640 10 monclient(hunting): authenticate will time out at 1411.336578s 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55698/0 (socket says 172.21.15.164:55698) 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63cc13cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 learned_addr learned my addr 172.21.15.164:0/2429578833 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): get_auth_request con 0x7f63cc09c180 auth_method 0 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:12.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): get_auth_request con 0x7f63cc09d160 auth_method 0 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient(hunting): handle_auth_done global_id 114209 payload 824 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 msgr2=0x7f63cc13d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:12.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63cc13d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 msgr2=0x7f63cc13cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63cc13cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient: _finish_hunting 0 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 monclient: found mon.noname-c 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:12.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63cc099c90 con 0x7f63cc09c180 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.849+0000 7f63cbfff640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63cc13cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc0994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63c00090b0 tx=0x7f63c00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0041020 con 0x7f63cc09c180 2024-09-10T16:17:12.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: dump: 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: _finish_auth 0 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63c003d830 con 0x7f63cc09c180 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 5 monclient: authenticate success, global_id 114209 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 10 monclient: _renew_subs 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63c0037040 con 0x7f63cc09c180 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63cc093a40 con 0x7f63cc09c180 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:12.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: dump: 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63caffd640 4 set_mon_vals no callback set 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 10 monclient: _renew_subs 2024-09-10T16:17:12.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f63cc0a42e0 con 0x7f63cc09c180 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc0994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc0994c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63c00090b0 tx=0x7f63c00046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f63c00090b0 tx=0x7f63c00046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 msgr2=0x7f63cc0994c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.852+0000 7f63c97fa640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63cc09c180 0x7f63cc0994c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63b0005870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): picked mon.b con 0x7f63cc13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63b0008270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): picked mon.a con 0x7f63cc09d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 0x7f63b000b110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): picked mon.c con 0x7f63b0008c80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f63b000bbc0 con 0x7f63cc09d160 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63b000bd40 con 0x7f63cc13a5c0 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:12.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63b000bef0 con 0x7f63b0008c80 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63c97fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63b0008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): get_auth_request con 0x7f63cc09d160 auth_method 0 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 0x7f63b000b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): get_auth_request con 0x7f63b0008c80 auth_method 0 2024-09-10T16:17:12.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63b0005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d0889640 10 monclient(hunting): get_auth_request con 0x7f63cc13a5c0 auth_method 0 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d0889640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d0889640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d0889640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient(hunting): handle_auth_done global_id 114209 payload 293 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 msgr2=0x7f63b000b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 0x7f63b000b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 msgr2=0x7f63b0005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63b0005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:12.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d2aed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 monclient: found mon.a 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f63b000c210 con 0x7f63cc09d160 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient: _finish_auth 0 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d108a640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 0x7f63b000b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d2aed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d2aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:12.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.853+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63cc09d620 con 0x7f63cc09d160 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63cbfff640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 msgr2=0x7f63b0008270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63cbfff640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63b0008270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f63cc093090 tx=0x7f63c002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b8003f80 con 0x7f63cc09d160 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 10 monclient: dump: 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:12.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63b8019070 con 0x7f63cc09d160 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63c97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.855+0000 7f63ca7fc640 4 set_mon_vals no callback set 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.856+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f63b8007030 con 0x7f63cc09d160 2024-09-10T16:17:12.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.856+0000 7f63c97fa640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 0x7f63b0048b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.856+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 4 ==== osd_map(380..380 src has 184..380) ==== 4799+0+0 (secure 0 0 0) 0x7f63b8051210 con 0x7f63cc09d160 2024-09-10T16:17:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.857+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 0x7f63b0048b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.857+0000 7f63d0889640 10 monclient: get_auth_request con 0x7f63cc09c180 auth_method 0 2024-09-10T16:17:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.859+0000 7f63d0889640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 0x7f63b0048b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63c00090b0 tx=0x7f63c003b000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.861+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f63b80133d0 con 0x7f63cc09d160 2024-09-10T16:17:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.861+0000 7f63c97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:12.861+0000 7f63c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63d2aed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63d2aed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63d2aed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f63cc077260 con 0x7f63cc09d160 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63c97fa640 1 -- 172.21.15.164:0/2429578833 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f63b800fe20 con 0x7f63cc09d160 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63c97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.103+0000 7f63c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:13.105 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":102,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:17:13.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 msgr2=0x7f63b0048b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 0x7f63b0048b00 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f63c00090b0 tx=0x7f63c003b000 comp rx=0 tx=0).stop 2024-09-10T16:17:13.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 10 monclient: shutdown 2024-09-10T16:17:13.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 msgr2=0x7f63b0008270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.105+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63b0008270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f63cc093090 tx=0x7f63c002e000 comp rx=0 tx=0).stop 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 shutdown_connections 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63cc09c180 0x7f63b0048b00 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63b0008c80 0x7f63b000b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63cc09d160 0x7f63b0008270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 --2- 172.21.15.164:0/2429578833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63cc13a5c0 0x7f63b0005870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 >> 172.21.15.164:0/2429578833 conn(0x7f63cc0a3110 msgr2=0x7f63cc06e7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 shutdown_connections 2024-09-10T16:17:13.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.106+0000 7f63d2aed640 1 -- 172.21.15.164:0/2429578833 wait complete. 2024-09-10T16:17:13.117 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:17:13.118 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:17:13.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:17:13.118 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:17:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.215+0000 7f1843a79640 10 monclient: get_monmap_and_config 2024-09-10T16:17:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.215+0000 7f1843a79640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.215+0000 7f1843a79640 10 monclient: monmap: 2024-09-10T16:17:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:13.216193+0000 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:13.216193+0000 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.215+0000 7f1843a79640 1 Processor -- start 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 -- start start 2024-09-10T16:17:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient: init 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-b con 0x7f183c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f183c0a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f183c0a4a80 0x7f183c1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-c con 0x7f183c0a4a80 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:13.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f183c09ce50 con 0x7f183c0a46a0 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f183c06db60 con 0x7f183c0a4a80 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f183c0a4f10 con 0x7f183c06cb70 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1843a79640 10 monclient(hunting): authenticate will time out at 1411.703626s 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55720/0 (socket says 172.21.15.164:55720) 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 1 -- 172.21.15.164:0/2127983490 learned_addr learned my addr 172.21.15.164:0/2127983490 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): get_auth_request con 0x7f183c06cb70 auth_method 0 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1842016640 1 --2- 172.21.15.164:0/2127983490 >> v2:172.21.15.165:3300/0 conn(0x7f183c0a4a80 0x7f183c1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1842016640 10 monclient(hunting): get_auth_request con 0x7f183c0a4a80 auth_method 0 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1842016640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1842016640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.216+0000 7f1842016640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 10 monclient(hunting): handle_auth_done global_id 114212 payload 274 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 1 -- 172.21.15.164:0/2127983490 >> v2:172.21.15.165:3300/0 conn(0x7f183c0a4a80 msgr2=0x7f183c1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 1 --2- 172.21.15.164:0/2127983490 >> v2:172.21.15.165:3300/0 conn(0x7f183c0a4a80 0x7f183c1383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 1 -- 172.21.15.164:0/2127983490 >> v1:172.21.15.165:6789/0 conn(0x7f183c0a46a0 legacy=0x7f183c137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 1 monclient: found mon.noname-b 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.217+0000 7f1841815640 1 -- 172.21.15.164:0/2127983490 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c09cb70 con 0x7f183c06cb70 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1841815640 1 -- 172.21.15.164:0/2127983490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 msgr2=0x7f183c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1841815640 1 --2- 172.21.15.164:0/2127983490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f182c009870 tx=0x7f182c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8e362288cf8882b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 1 -- 172.21.15.164:0/2127983490 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c037070 con 0x7f183c06cb70 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: dump: 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.234 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 1 -- 172.21.15.164:0/2127983490 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182c032440 con 0x7f183c06cb70 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1843a79640 5 monclient: authenticate success, global_id 114212 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1843a79640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1843a79640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 1 -- 172.21.15.164:0/2127983490 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c032810 con 0x7f183c06cb70 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1843a79640 4 set_mon_vals no callback set 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:13.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.219+0000 7f1840813640 10 monclient: dump: 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.236 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 10 monclient: shutdown 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 -- 172.21.15.164:0/2127983490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 msgr2=0x7f183c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 --2- 172.21.15.164:0/2127983490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f182c009870 tx=0x7f182c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 -- 172.21.15.164:0/2127983490 shutdown_connections 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 --2- 172.21.15.164:0/2127983490 >> v2:172.21.15.165:3300/0 conn(0x7f183c0a4a80 0x7f183c1383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 --2- 172.21.15.164:0/2127983490 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c06cb70 0x7f183c134f70 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 -- 172.21.15.164:0/2127983490 >> 172.21.15.164:0/2127983490 conn(0x7f183c0aa900 msgr2=0x7f183c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 -- 172.21.15.164:0/2127983490 shutdown_connections 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.220+0000 7f1843a79640 1 -- 172.21.15.164:0/2127983490 wait complete. 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient: monmap: 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:13.238 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 1 Processor -- start 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 1 -- start start 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient: init 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 0x7f183c09fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-a con 0x7f183c06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.221+0000 7f1843a79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 0x7f183c0a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-c con 0x7f183c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183c0a4a80 0x7f183c0a05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): picked mon.noname-b con 0x7f183c0a4a80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f183c1388f0 con 0x7f183c06cb70 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f183c06cf50 con 0x7f183c0a4a80 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f183c138a70 con 0x7f183c0a46a0 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1843a79640 10 monclient(hunting): authenticate will time out at 1411.709046s 2024-09-10T16:17:13.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 0x7f183c09fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 0x7f183c0a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 0x7f183c09fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37834/0 (socket says 172.21.15.164:37834) 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 1 -- 172.21.15.164:0/18654681 learned_addr learned my addr 172.21.15.164:0/18654681 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): get_auth_request con 0x7f183c06cb70 auth_method 0 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): get_auth_request con 0x7f183c0a46a0 auth_method 0 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1842016640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183c0a4a80 0x7f183c0a05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1842016640 10 monclient(hunting): get_auth_request con 0x7f183c0a4a80 auth_method 0 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1842016640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1842016640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1842016640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient(hunting): handle_auth_done global_id 114215 payload 824 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183c0a4a80 msgr2=0x7f183c0a05c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f183c0a4a80 0x7f183c0a05c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 msgr2=0x7f183c09fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 0x7f183c09fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 monclient: found mon.noname-c 2024-09-10T16:17:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841014640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f183c0a3150 con 0x7f183c0a46a0 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.222+0000 7f1841815640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c06cb70 0x7f183c09fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1841014640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 msgr2=0x7f183c0a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1841014640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 0x7f183c0a0080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f183000c440 tx=0x7f183000c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18300119a0 con 0x7f183c0a46a0 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: dump: 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.244 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1830017070 con 0x7f183c0a46a0 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 5 monclient: authenticate success, global_id 114215 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 10 monclient: _renew_subs 2024-09-10T16:17:13.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f183c0a3430 con 0x7f183c0a46a0 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1830011d80 con 0x7f183c0a46a0 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183a7fc640 10 monclient: dump: 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.246 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 10 monclient: _renew_subs 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f183bfff640 4 set_mon_vals no callback set 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.224+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f183c0a1f10 con 0x7f183c0a46a0 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1843a79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1843a79640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 0x7f183c12e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f183c135210 con 0x7f183c0a3e80 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 0x7f183c12e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient: get_auth_request con 0x7f183c0a3e80 auth_method 0 2024-09-10T16:17:13.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841014640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 0x7f183c0a0080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f183000c440 tx=0x7f183000c900 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f1830052ac0 osd_map(380..380 src has 184..380) 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f183001c460 con 0x7f183c0a46a0 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 10 monclient(hunting): handle_auth_done global_id 124128 payload 824 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.225+0000 7f1841815640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f183c134f70 con 0x7f183c0a3e80 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.226+0000 7f183a7fc640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f182803e210 0x7f18280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.226+0000 7f1842016640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f182803e210 0x7f18280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.226+0000 7f1842016640 10 monclient: get_auth_request con 0x7f182803e210 auth_method 0 2024-09-10T16:17:13.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f1841815640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 msgr2=0x7f183c12e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f1841815640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 0x7f183c12e0b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f182c035040 tx=0x7f182c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c043070 con 0x7f183c0a3e80 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 10 monclient: dump: 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.249 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f182c03c860 con 0x7f183c0a3e80 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 msgr2=0x7f183c12e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 0x7f183c12e0b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f182c035040 tx=0x7f182c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 msgr2=0x7f183c12e0b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f183a7fc640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c0a3e80 0x7f183c12e0b0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f182c035040 tx=0x7f182c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.228+0000 7f1842016640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f182803e210 0x7f18280406d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1834009f80 tx=0x7f183400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1843a79640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1843a79640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 0x7f183c12e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1843a79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f183c12ef10 con 0x7f183c12e5f0 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 0x7f183c12e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient: get_auth_request con 0x7f183c12e5f0 auth_method 0 2024-09-10T16:17:13.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.234+0000 7f1841815640 10 monclient(hunting): handle_auth_done global_id 124131 payload 824 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.235+0000 7f1841815640 1 -- 172.21.15.164:0/18654681 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f183c0770f0 con 0x7f183c12e5f0 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1841815640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 msgr2=0x7f183c12e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1841815640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 0x7f183c12e9d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f182c02a0e0 tx=0x7f182c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f182c037040 con 0x7f183c12e5f0 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 10 monclient: dump: 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f182c0044d0 con 0x7f183c12e5f0 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:13.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 msgr2=0x7f183c12e9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 0x7f183c12e9d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f182c02a0e0 tx=0x7f182c009240 comp rx=0 tx=0).stop 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 msgr2=0x7f183c12e9d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f183c12e5f0 0x7f183c12e9d0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f182c02a0e0 tx=0x7f182c009240 comp rx=0 tx=0).stop 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:17:13.255 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 9271, 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.256 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.257 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:13.258 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:13.259 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:13.260 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.261 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:13.262 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.263 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.264 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.265 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:13.266 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:13.267 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f182803e210 msgr2=0x7f18280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f182803e210 0x7f18280406d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f1834009f80 tx=0x7f183400b040 comp rx=0 tx=0).stop 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 10 monclient: shutdown 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 msgr2=0x7f183c0a0080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 --2- 172.21.15.164:0/18654681 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f183c0a46a0 0x7f183c0a0080 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f183000c440 tx=0x7f183000c900 comp rx=0 tx=0).stop 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1842016640 1 -- 172.21.15.164:0/18654681 reap_dead start 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f183a7fc640 1 -- 172.21.15.164:0/18654681 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(380..380 src has 184..380) ==== 4799+0+0 (unknown 0 0 0) 0x7f1830052ac0 con 0x7f183c0a46a0 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1842016640 1 -- 172.21.15.164:0/18654681 reap_dead start 2024-09-10T16:17:13.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 shutdown_connections 2024-09-10T16:17:13.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.237+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 >> 172.21.15.164:0/18654681 conn(0x7f183c0aa900 msgr2=0x7f183c132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.238+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 shutdown_connections 2024-09-10T16:17:13.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.238+0000 7f1843a79640 1 -- 172.21.15.164:0/18654681 wait complete. 2024-09-10T16:17:13.270 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:13.270 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:17:13.270 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.366+0000 7f07c4840640 10 monclient: get_monmap_and_config 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.366+0000 7f07c4840640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.366+0000 7f07c4840640 10 monclient: monmap: 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:13.367652+0000 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:13.367652+0000 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:13.369 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 1 Processor -- start 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 1 -- start start 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 10 monclient: init 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07c009c180 0x7f07c009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.367+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-a con 0x7f07c009c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07c009d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f07c013a5c0 0x7f07c013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-c con 0x7f07c013a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07c009a430 con 0x7f07c009d160 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f07c00a24e0 con 0x7f07c009c180 2024-09-10T16:17:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f07c00a2bf0 con 0x7f07c013a5c0 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07c4840640 10 monclient(hunting): authenticate will time out at 1411.855058s 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f07c009c180 0x7f07c009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07b7fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f07c009d160 0x7f07c009ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34002/0 (socket says 172.21.15.164:34002) 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07b7fff640 1 -- 172.21.15.164:0/2107793888 learned_addr learned my addr 172.21.15.164:0/2107793888 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): get_auth_request con 0x7f07c009c180 auth_method 0 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient(hunting): handle_auth_done global_id 124134 payload 274 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 -- 172.21.15.164:0/2107793888 >> v2:172.21.15.165:3300/0 conn(0x7f07c013a5c0 msgr2=0x7f07c013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 --2- 172.21.15.164:0/2107793888 >> v2:172.21.15.165:3300/0 conn(0x7f07c013a5c0 0x7f07c013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 -- 172.21.15.164:0/2107793888 >> v1:172.21.15.164:6789/0 conn(0x7f07c009d160 legacy=0x7f07c009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 monclient: found mon.noname-a 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.368+0000 7f07be59c640 1 -- 172.21.15.164:0/2107793888 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c009c560 con 0x7f07c009c180 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07be59c640 1 -- 172.21.15.164:0/2107793888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c009c180 msgr2=0x7f07c009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07be59c640 1 --2- 172.21.15.164:0/2107793888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c009c180 0x7f07c009ea40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8d4db1b19a995b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 1 -- 172.21.15.164:0/2107793888 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0037070 con 0x7f07c009c180 2024-09-10T16:17:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: dump: 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 5 monclient: authenticate success, global_id 124134 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 1 -- 172.21.15.164:0/2107793888 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b0032640 con 0x7f07c009c180 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 1 -- 172.21.15.164:0/2107793888 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0032a10 con 0x7f07c009c180 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 4 set_mon_vals no callback set 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07bdd9b640 10 monclient: dump: 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 10 monclient: shutdown 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 1 -- 172.21.15.164:0/2107793888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c009c180 msgr2=0x7f07c009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.371+0000 7f07c4840640 1 --2- 172.21.15.164:0/2107793888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c009c180 0x7f07c009ea40 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b002a010 comp rx=0 tx=0).stop 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 -- 172.21.15.164:0/2107793888 shutdown_connections 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 --2- 172.21.15.164:0/2107793888 >> v2:172.21.15.165:3300/0 conn(0x7f07c013a5c0 0x7f07c013c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 --2- 172.21.15.164:0/2107793888 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c009c180 0x7f07c009ea40 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 -- 172.21.15.164:0/2107793888 >> 172.21.15.164:0/2107793888 conn(0x7f07c00a3110 msgr2=0x7f07c00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 -- 172.21.15.164:0/2107793888 shutdown_connections 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 1 -- 172.21.15.164:0/2107793888 wait complete. 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.372+0000 7f07c4840640 10 monclient: monmap: 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:13.380 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:13.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:13.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 Processor -- start 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 -- start start 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient: init 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 0x7f07c0139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-c con 0x7f07c009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07c009d160 0x7f07c013df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-b con 0x7f07c009d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:13.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c013a5c0 0x7f07c013e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): picked mon.noname-a con 0x7f07c013a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0074f90 con 0x7f07c013a5c0 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07c0074e10 con 0x7f07c009d160 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07c009f690 con 0x7f07c009c180 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07c4840640 10 monclient(hunting): authenticate will time out at 1411.860667s 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 0x7f07c0139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c013a5c0 0x7f07c013e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 0x7f07c0139dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55746/0 (socket says 172.21.15.164:55746) 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 1 -- 172.21.15.164:0/610931765 learned_addr learned my addr 172.21.15.164:0/610931765 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): get_auth_request con 0x7f07c009c180 auth_method 0 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): get_auth_request con 0x7f07c013a5c0 auth_method 0 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.373+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07c009d160 0x7f07c013df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 10 monclient(hunting): handle_auth_done global_id 114218 payload 824 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07c009d160 msgr2=0x7f07c013df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07c009d160 0x7f07c013df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c013a5c0 msgr2=0x7f07c013e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c013a5c0 0x7f07c013e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 monclient: found mon.noname-c 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07be59c640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07c009d620 con 0x7f07c009c180 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.374+0000 7f07bed9d640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f07c013a5c0 0x7f07c013e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07be59c640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 msgr2=0x7f07c0139dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07be59c640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 0x7f07c0139dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0041070 con 0x7f07c009c180 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: dump: 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.390 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07b003c880 con 0x7f07c009c180 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 5 monclient: authenticate success, global_id 114218 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07b0037040 con 0x7f07c009c180 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 10 monclient: _renew_subs 2024-09-10T16:17:13.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07c0092a80 con 0x7f07c009c180 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07b6ffd640 10 monclient: dump: 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.392 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07bd59a640 4 set_mon_vals no callback set 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 10 monclient: _renew_subs 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.376+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f07c0099be0 con 0x7f07c009c180 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07c4840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07c4840640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 0x7f07c006f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07c013ed60 con 0x7f07c013e980 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f07b0044070 con 0x7f07c009c180 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 0x7f07c006f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient: get_auth_request con 0x7f07c013e980 auth_method 0 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 10 monclient(hunting): handle_auth_done global_id 114221 payload 824 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b6ffd640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f07a003e210 0x7f07a00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07c00a42c0 con 0x7f07c013e980 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(380..380 src has 184..380) ==== 4799+0+0 (secure 0 0 0) 0x7f07b00737e0 con 0x7f07c009c180 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07bed9d640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f07a003e210 0x7f07a00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.377+0000 7f07bed9d640 10 monclient: get_auth_request con 0x7f07a003e210 auth_method 0 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.379+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 msgr2=0x7f07c006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.379+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 0x7f07c006f7e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d360 tx=0x7f07a800d820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.379+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8018070 con 0x7f07c013e980 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.379+0000 7f07b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 10 monclient: dump: 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f07a80070d0 con 0x7f07c013e980 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 msgr2=0x7f07c006f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 0x7f07c006f7e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d360 tx=0x7f07a800d820 comp rx=0 tx=0).stop 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 msgr2=0x7f07c006f7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07b6ffd640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c013e980 0x7f07c006f7e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.380+0000 7f07bed9d640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f07a003e210 0x7f07a00406d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f07c009a7c0 tx=0x7f07b800d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07c4840640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:13.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07c4840640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07c4840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f07c009d930 con 0x7f07c006fd20 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 msgr2=0x7f07c0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 msgr2=0x7f07c0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:13.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.386+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1566: done (5 left) 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1568: done (4 left) 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1569: done (3 left) 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1570: done (2 left) 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1571: done (1 left) 2024-09-10T16:17:13.468 INFO:tasks.rados.rados.0.smithi165.stdout:1573: done (0 left) 2024-09-10T16:17:13.469 INFO:tasks.rados.rados.0.smithi165.stdout:1574: snap_create 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.587+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient: get_auth_request con 0x7f07c006fd20 auth_method 0 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 10 monclient(hunting): handle_auth_done global_id 114224 payload 824 2024-09-10T16:17:13.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.588+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07c013f2e0 con 0x7f07c006fd20 2024-09-10T16:17:13.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b7fff640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 msgr2=0x7f07c0070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b7fff640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d360 tx=0x7f07a8004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8018040 con 0x7f07c006fd20 2024-09-10T16:17:13.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 10 monclient: dump: 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f07a801aa90 con 0x7f07c006fd20 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:13.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:13.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 msgr2=0x7f07c0070130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d360 tx=0x7f07a8004940 comp rx=0 tx=0).stop 2024-09-10T16:17:13.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 msgr2=0x7f07c0070130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.590+0000 7f07b6ffd640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c006fd20 0x7f07c0070130 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d360 tx=0x7f07a8004940 comp rx=0 tx=0).stop 2024-09-10T16:17:13.596 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 54159, 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:13.597 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.598 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.599 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:13.600 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.601 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:13.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.604 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.605 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.605 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.605 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.606 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:13.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:13.612 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.613 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.614 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.615 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.615 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:17:13.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.615 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.616 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:13.616 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:13.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f07a003e210 msgr2=0x7f07a00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f07a003e210 0x7f07a00406d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f07c009a7c0 tx=0x7f07b800d040 comp rx=0 tx=0).stop 2024-09-10T16:17:13.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 10 monclient: shutdown 2024-09-10T16:17:13.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 msgr2=0x7f07c0139dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 --2- 172.21.15.164:0/610931765 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07c009c180 0x7f07c0139dc0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07b00097a0 tx=0x7f07b00046c0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07bed9d640 1 -- 172.21.15.164:0/610931765 reap_dead start 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07bed9d640 1 -- 172.21.15.164:0/610931765 reap_dead start 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 shutdown_connections 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 >> 172.21.15.164:0/610931765 conn(0x7f07c00a3110 msgr2=0x7f07c013c9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 shutdown_connections 2024-09-10T16:17:13.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.591+0000 7f07c4840640 1 -- 172.21.15.164:0/610931765 wait complete. 2024-09-10T16:17:13.625 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:13.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:17:13.626 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:17:13.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 10 monclient: get_monmap_and_config 2024-09-10T16:17:13.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 10 monclient: monmap: 2024-09-10T16:17:13.725 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.725 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.726 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:13.724910+0000 2024-09-10T16:17:13.726 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:13.724910+0000 2024-09-10T16:17:13.726 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.726 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.726 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:13.727 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:13.727 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:13.727 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:13.727 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:13.727 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 1 Processor -- start 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 1 -- start start 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.724+0000 7fb5af524640 10 monclient: init 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb5a806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb5a809bcc0 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb5a80a40d0 0x7fb5a806d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a80a40d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a806e900 con 0x7fb5a806cb70 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a806e340 con 0x7fb5a809bcc0 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a80a4600 con 0x7fb5a80a40d0 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5af524640 10 monclient(hunting): authenticate will time out at 1412.212333s 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5ad2c0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb5a806cb70 0x7fb5a806cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34014/0 (socket says 172.21.15.164:34014) 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb5a80a40d0 0x7fb5a806d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5ad2c0640 1 -- 172.21.15.164:0/57263230 learned_addr learned my addr 172.21.15.164:0/57263230 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): get_auth_request con 0x7fb5a80a40d0 auth_method 0 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient(hunting): handle_auth_done global_id 124140 payload 274 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 1 -- 172.21.15.164:0/57263230 >> v1:172.21.15.165:6789/0 conn(0x7fb5a809bcc0 legacy=0x7fb5a813d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 1 -- 172.21.15.164:0/57263230 >> v1:172.21.15.164:6789/0 conn(0x7fb5a806cb70 legacy=0x7fb5a806cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 1 monclient: found mon.noname-a 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.725+0000 7fb5adac1640 1 -- 172.21.15.164:0/57263230 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a80a01e0 con 0x7fb5a80a40d0 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5adac1640 1 -- 172.21.15.164:0/57263230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a80a40d0 msgr2=0x7fb5a806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5adac1640 1 --2- 172.21.15.164:0/57263230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a80a40d0 0x7fb5a806d660 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb598009d90 tx=0x7fb59802a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1865eee4b6b697e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 1 -- 172.21.15.164:0/57263230 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598007a30 con 0x7fb5a80a40d0 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: dump: 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.737 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 1 -- 172.21.15.164:0/57263230 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb598007bd0 con 0x7fb5a80a40d0 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5af524640 5 monclient: authenticate success, global_id 124140 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5af524640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5af524640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 1 -- 172.21.15.164:0/57263230 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598031400 con 0x7fb5a80a40d0 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5af524640 4 set_mon_vals no callback set 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.728+0000 7fb5a7fff640 10 monclient: dump: 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 10 monclient: shutdown 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 -- 172.21.15.164:0/57263230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a80a40d0 msgr2=0x7fb5a806d660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 --2- 172.21.15.164:0/57263230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a80a40d0 0x7fb5a806d660 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb598009d90 tx=0x7fb59802a530 comp rx=0 tx=0).stop 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 -- 172.21.15.164:0/57263230 shutdown_connections 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 --2- 172.21.15.164:0/57263230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a80a40d0 0x7fb5a806d660 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 -- 172.21.15.164:0/57263230 >> 172.21.15.164:0/57263230 conn(0x7fb5a80a4f50 msgr2=0x7fb5a80a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 -- 172.21.15.164:0/57263230 shutdown_connections 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.729+0000 7fb5af524640 1 -- 172.21.15.164:0/57263230 wait complete. 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient: build_initial_monmap 2024-09-10T16:17:13.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient: monmap: 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:13.742 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 Processor -- start 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 -- start start 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient: init 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 0x7fb5a813fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-a con 0x7fb5a806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 0x7fb5a8093320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-c con 0x7fb5a80a40d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:13.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a8140330 0x7fb5a8093860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): picked mon.noname-b con 0x7fb5a8140330 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a81410b0 con 0x7fb5a806cb70 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8140f30 con 0x7fb5a8140330 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a8141230 con 0x7fb5a80a40d0 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.730+0000 7fb5af524640 10 monclient(hunting): authenticate will time out at 1412.217931s 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 0x7fb5a813fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 0x7fb5a8093320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 0x7fb5a8093320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55768/0 (socket says 172.21.15.164:55768) 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): get_auth_request con 0x7fb5a806cb70 auth_method 0 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 1 -- 172.21.15.164:0/3861229452 learned_addr learned my addr 172.21.15.164:0/3861229452 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): get_auth_request con 0x7fb5a80a40d0 auth_method 0 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a8140330 0x7fb5a8093860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): get_auth_request con 0x7fb5a8140330 auth_method 0 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5adac1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient(hunting): handle_auth_done global_id 124143 payload 824 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 msgr2=0x7fb5a8093320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 0x7fb5a8093320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a8140330 msgr2=0x7fb5a8093860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a8140330 0x7fb5a8093860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient: _finish_hunting 0 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 monclient: found mon.noname-a 2024-09-10T16:17:13.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:13.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5ad2c0640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a8094de0 con 0x7fb5a806cb70 2024-09-10T16:17:13.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.731+0000 7fb5acabf640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb5a80a40d0 0x7fb5a8093320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:13.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5ad2c0640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 msgr2=0x7fb5a813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5ad2c0640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 0x7fb5a813fdf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c3e0 tx=0x7fb5a000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a0017070 con 0x7fb5a806cb70 2024-09-10T16:17:13.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: dump: 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5a00070d0 con 0x7fb5a806cb70 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5af524640 5 monclient: authenticate success, global_id 124143 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5af524640 10 monclient: _renew_subs 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5a00074c0 con 0x7fb5a806cb70 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5af524640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a80a6550 con 0x7fb5a806cb70 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a5ffb640 10 monclient: dump: 2024-09-10T16:17:13.755 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.733+0000 7fb5a77fe640 4 set_mon_vals no callback set 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.734+0000 7fb5af524640 10 monclient: _renew_subs 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.734+0000 7fb5af524640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:13.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.734+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a8097140 con 0x7fb5a806cb70 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.734+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb5a001a070 con 0x7fb5a806cb70 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.734+0000 7fb5af524640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5af524640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 0x7fb5a802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a8096dc0 con 0x7fb5a809bcc0 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5a5ffb640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb59403e290 0x7fb594040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(381..381 src has 184..381) ==== 4771+0+0 (secure 0 0 0) 0x7fb5a0052180 con 0x7fb5a806cb70 2024-09-10T16:17:13.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 0x7fb5a802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient: get_auth_request con 0x7fb5a809bcc0 auth_method 0 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5adac1640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb59403e290 0x7fb594040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5adac1640 10 monclient: get_auth_request con 0x7fb59403e290 auth_method 0 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 10 monclient(hunting): handle_auth_done global_id 94390 payload 824 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.735+0000 7fb5acabf640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb5a806a650 con 0x7fb5a809bcc0 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5acabf640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 msgr2=0x7fb5a802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5acabf640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 0x7fb5a802d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00fb10 tx=0x7fb59c004580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5adac1640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb59403e290 0x7fb594040750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8070290 tx=0x7fb59802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c018020 con 0x7fb5a809bcc0 2024-09-10T16:17:13.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:17:13.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 10 monclient: dump: 2024-09-10T16:17:13.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.761 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.761 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.761 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb59c00de50 con 0x7fb5a809bcc0 2024-09-10T16:17:13.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:13.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:13.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 msgr2=0x7fb5a802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 0x7fb5a802d440 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fb59c00fb10 tx=0x7fb59c004580 comp rx=0 tx=0).stop 2024-09-10T16:17:13.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 msgr2=0x7fb5a802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.738+0000 7fb5a5ffb640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a809bcc0 0x7fb5a802d440 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5af524640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5af524640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 0x7fb5a802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5af524640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a806cf50 con 0x7fb5a802d980 2024-09-10T16:17:13.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 0x7fb5a802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient: get_auth_request con 0x7fb5a802d980 auth_method 0 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:13.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.744+0000 7fb5acabf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.745+0000 7fb5acabf640 10 monclient(hunting): handle_auth_done global_id 94393 payload 824 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.745+0000 7fb5acabf640 1 -- 172.21.15.164:0/3861229452 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb5a8072860 con 0x7fb5a802d980 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5acabf640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 msgr2=0x7fb5a802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5acabf640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 0x7fb5a802dd90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb59c019000 tx=0x7fb59c0036f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c018030 con 0x7fb5a802d980 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5a5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5a5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.747+0000 7fb5a5ffb640 10 monclient: dump: 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:13.766 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:13.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fb59c01d420 con 0x7fb5a802d980 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 msgr2=0x7fb5a802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 0x7fb5a802dd90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb59c019000 tx=0x7fb59c0036f0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 msgr2=0x7fb5a802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:13.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5a5ffb640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb5a802d980 0x7fb5a802dd90 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fb59c019000 tx=0x7fb59c0036f0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.769 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:13.769 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 102, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 151247, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:13.770 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:13.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:13.773 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:13.774 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:13.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:13.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.778 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:13.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.780 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.781 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.782 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:13.783 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb59403e290 msgr2=0x7fb594040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb59403e290 0x7fb594040750 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb5a8070290 tx=0x7fb59802f000 comp rx=0 tx=0).stop 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 10 monclient: shutdown 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 msgr2=0x7fb5a813fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:13.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5adac1640 1 -- 172.21.15.164:0/3861229452 reap_dead start 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 0x7fb5a813fdf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5a000c3e0 tx=0x7fb5a000c8a0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 shutdown_connections 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 --2- 172.21.15.164:0/3861229452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb5a806cb70 0x7fb5a813fdf0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 >> 172.21.15.164:0/3861229452 conn(0x7fb5a80a4f50 msgr2=0x7fb5a80a0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 shutdown_connections 2024-09-10T16:17:13.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:13.748+0000 7fb5af524640 1 -- 172.21.15.164:0/3861229452 wait complete. 2024-09-10T16:17:13.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:17:13.786 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:17:13.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-10T16:17:13.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:17:13.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:17:13.787 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:17:13.787 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:17:13.803 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:17:13.803 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:17:13.804 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:17:13.804 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:17:28.805 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:17:28.807 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:17:28.807 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:17:28.807 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:17:28.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 10 monclient: get_monmap_and_config 2024-09-10T16:17:28.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 10 monclient: build_initial_monmap 2024-09-10T16:17:28.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 10 monclient: monmap: 2024-09-10T16:17:28.917 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:28.917 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:28.917 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:28.915996+0000 2024-09-10T16:17:28.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:28.915996+0000 2024-09-10T16:17:28.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:28.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:28.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:28.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:28.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:28.919 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:28.919 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:28.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:28.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 1 Processor -- start 2024-09-10T16:17:28.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 1 -- start start 2024-09-10T16:17:28.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.915+0000 7f41a7edd640 10 monclient: init 2024-09-10T16:17:28.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:28.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-10T16:17:28.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:28.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-c con 0x7f41a006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:28.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-b con 0x7f41a00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f41a0137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a009ce50 con 0x7f41a0137ca0 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f41a006db60 con 0x7f41a006cb70 2024-09-10T16:17:28.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:28.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f41a006cf50 con 0x7f41a00a46a0 2024-09-10T16:17:28.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:28.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a7edd640 10 monclient(hunting): authenticate will time out at 1427.403503s 2024-09-10T16:17:28.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:28.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5c79640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 msgr2=0x7f41a0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:28.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a647a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f41a0137ca0 0x7f41a0138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44644/0 (socket says 172.21.15.164:44644) 2024-09-10T16:17:28.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a647a640 1 -- 172.21.15.164:0/2527033277 learned_addr learned my addr 172.21.15.164:0/2527033277 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:28.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:28.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 10 monclient(hunting): get_auth_request con 0x7f41a00a46a0 auth_method 0 2024-09-10T16:17:28.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:28.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:28.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:28.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:28.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:28.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:28.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.916+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:28.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:28.918+0000 7f41a647a640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.164:6789/0 conn(0x7f41a0137ca0 legacy=0x7f41a0138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:29.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 msgr2=0x7f41a0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 10 monclient(hunting): get_auth_request con 0x7f41a00a46a0 auth_method 0 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:29.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.117+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:17:29.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 msgr2=0x7f41a0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:29.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:17:29.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 10 monclient(hunting): get_auth_request con 0x7f41a00a46a0 auth_method 0 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:29.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:29.518+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 10 monclient(hunting): get_auth_request con 0x7f41a00a46a0 auth_method 0 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:17:30.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 msgr2=0x7f41a0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:30.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:30.319+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:17:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): tick 2024-09-10T16:17:31.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41a00a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 msgr2=0x7f41a0134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.165:3300/0 conn(0x7f41a006cb70 0x7f41a0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.164:6789/0 conn(0x7f41a0137ca0 legacy=0x7f41a0138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4184003760 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f41840062b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41a009cb70 con 0x7f41840062b0 2024-09-10T16:17:31.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f41a006cf50 con 0x7f4184003760 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41977fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f41840062b0 legacy=0x7f4184008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5c79640 10 monclient(hunting): get_auth_request con 0x7f4184003760 auth_method 0 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5c79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.916+0000 7f41a5c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.917+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.917+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.917+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:31.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:31.917+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:32.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.117+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f41840062b0 legacy=0x7f4184008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 10 monclient(hunting): get_auth_request con 0x7f4184003760 auth_method 0 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:32.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.118+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:17:32.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.518+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f41840062b0 legacy=0x7f4184008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 10 monclient(hunting): get_auth_request con 0x7f4184003760 auth_method 0 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:32.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:32.519+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:17:33.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.319+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f41840062b0 legacy=0x7f4184008770 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 10 monclient(hunting): get_auth_request con 0x7f4184003760 auth_method 0 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:17:33.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:17:33.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:33.320+0000 7f41a5c79640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:1574: done (0 left) 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:1575: rollback oid 3 current snap is 180 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 169 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:1576: rollback oid 12 current snap is 180 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 175 2024-09-10T16:17:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:1577: snap_create 2024-09-10T16:17:34.824 INFO:tasks.rados.rados.0.smithi165.stdout:1576: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:17:34.825 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 0 (ObjNum 33 snap 0 seq_num 1162764464) dirty dne 2024-09-10T16:17:34.828 INFO:tasks.rados.rados.0.smithi165.stdout:1575: finishing rollback tid 0 to smithi16534841-3 2024-09-10T16:17:34.829 INFO:tasks.rados.rados.0.smithi165.stdout:1575: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:17:34.832 INFO:tasks.rados.rados.0.smithi165.stdout:1575: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:17:34.832 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1135 (ObjNum 353 snap 101 seq_num 353) dirty exists 2024-09-10T16:17:34.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): tick 2024-09-10T16:17:34.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:17:34.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:17:34.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f4184003760 0x7f4184005bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f41840062b0 legacy=0x7f4184008770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 1 -- 172.21.15.164:0/2527033277 reap_dead start 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f418400a010 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41840086f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): picked mon.noname-b con 0x7f41a00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 0x7f4184005bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): picked mon.noname-a con 0x7f418400ac00 addr v2:172.21.15.164:3300/0 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f418400b070 con 0x7f418400a010 2024-09-10T16:17:34.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f41a006cf50 con 0x7f418400ac00 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 1 -- 172.21.15.164:0/2527033277 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f418400e4a0 con 0x7f41a00a46a0 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41977fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5c79640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f418400a010 legacy=0x7f418400a3f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 0x7f4184005bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41840086f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): get_auth_request con 0x7f418400ac00 auth_method 0 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:34.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): get_auth_request con 0x7f41a00a46a0 auth_method 0 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient(hunting): handle_auth_done global_id 114230 payload 274 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 msgr2=0x7f4184005bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 0x7f4184005bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> v1:172.21.15.165:6789/0 conn(0x7f418400a010 legacy=0x7f418400a3f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient: _finish_hunting 0 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 monclient: found mon.noname-b 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a009cb70 con 0x7f41a00a46a0 2024-09-10T16:17:34.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.917+0000 7f41a647a640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 0x7f4184005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.920+0000 7f41a5478640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41840086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.920+0000 7f41a5478640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41840086f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4184009f50 tx=0x7f419002abf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ab1ba80a9d39ed server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 1 -- 172.21.15.164:0/2527033277 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4190031070 con 0x7f41a00a46a0 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: dump: 2024-09-10T16:17:34.927 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: _finish_auth 0 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: _check_auth_tickets 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:34.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 1 -- 172.21.15.164:0/2527033277 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419002b960 con 0x7f41a00a46a0 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 1 -- 172.21.15.164:0/2527033277 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419002bd50 con 0x7f41a00a46a0 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 5 monclient: authenticate success, global_id 114230 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 4 set_mon_vals no callback set 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a4c77640 10 monclient: dump: 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:34.929 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 10 monclient: shutdown 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 1 -- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 msgr2=0x7f41840086f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.921+0000 7f41a7edd640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41840086f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4184009f50 tx=0x7f419002abf0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 -- 172.21.15.164:0/2527033277 shutdown_connections 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 --2- 172.21.15.164:0/2527033277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a00a46a0 0x7f41840086f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 --2- 172.21.15.164:0/2527033277 >> v2:172.21.15.164:3300/0 conn(0x7f418400ac00 0x7f4184005bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 -- 172.21.15.164:0/2527033277 >> 172.21.15.164:0/2527033277 conn(0x7f41a00aa900 msgr2=0x7f41a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:34.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 -- 172.21.15.164:0/2527033277 shutdown_connections 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.922+0000 7f41a7edd640 1 -- 172.21.15.164:0/2527033277 wait complete. 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.923+0000 7f41a7edd640 10 monclient: build_initial_monmap 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.923+0000 7f41a7edd640 10 monclient: monmap: 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:34.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.923+0000 7f41a7edd640 1 Processor -- start 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.923+0000 7f41a7edd640 1 -- start start 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.923+0000 7f41a7edd640 10 monclient: init 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-a con 0x7f418400a010 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 0x7f41a00a3d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-b con 0x7f41a00a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 0x7f41a009e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): picked mon.noname-c con 0x7f41a01cd110 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f41a00a4c30 con 0x7f418400a010 2024-09-10T16:17:34.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f41a00a4ab0 con 0x7f41a00a46a0 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f41a00a4db0 con 0x7f41a01cd110 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a7edd640 10 monclient(hunting): authenticate will time out at 1433.411147s 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48784/0 (socket says 172.21.15.164:48784) 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 -- 172.21.15.164:0/4051616899 learned_addr learned my addr 172.21.15.164:0/4051616899 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): get_auth_request con 0x7f418400a010 auth_method 0 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:34.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 0x7f41a009e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5478640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a3d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): get_auth_request con 0x7f41a01cd110 auth_method 0 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5478640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 0x7f41a00a3d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient(hunting): handle_auth_done global_id 124149 payload 824 2024-09-10T16:17:34.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 msgr2=0x7f41a009e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 0x7f41a009e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 msgr2=0x7f41a00a3d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 0x7f41a00a3d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient: _finish_hunting 0 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 monclient: found mon.noname-a 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a5c79640 1 -- 172.21.15.164:0/4051616899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41a006e720 con 0x7f418400a010 2024-09-10T16:17:34.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.924+0000 7f41a647a640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 0x7f41a009e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.926+0000 7f41a5c79640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 msgr2=0x7f41a00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.926+0000 7f41a5c79640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f419c00a010 tx=0x7f419c00bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c01b020 con 0x7f418400a010 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: dump: 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.938 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: _finish_auth 0 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: _check_auth_tickets 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f419c0173f0 con 0x7f418400a010 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 5 monclient: authenticate success, global_id 124149 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419c0177a0 con 0x7f418400a010 2024-09-10T16:17:34.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 10 monclient: _renew_subs 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41a00a22b0 con 0x7f418400a010 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41967fc640 10 monclient: dump: 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:34.940 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f4197fff640 4 set_mon_vals no callback set 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 10 monclient: _renew_subs 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.927+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41a00a0ab0 con 0x7f418400a010 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f419c01e020 con 0x7f418400a010 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a7edd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a7edd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a7edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41967fc640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 0x7f4184041bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41a0135050 con 0x7f418400a010 2024-09-10T16:17:34.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(383..383 src has 184..383) ==== 4771+0+0 (secure 0 0 0) 0x7f419c053ac0 con 0x7f418400a010 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a5478640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 0x7f4184041bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.928+0000 7f41a5478640 10 monclient: get_auth_request con 0x7f418403f730 auth_method 0 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.931+0000 7f41a5478640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 0x7f4184041bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4190033000 tx=0x7f419000f040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.933+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f419c025020 con 0x7f418400a010 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.933+0000 7f41967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:34.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:34.933+0000 7f41967fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41a7edd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41a7edd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41a7edd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f41a0134420 con 0x7f418400a010 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41967fc640 1 -- 172.21.15.164:0/4051616899 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f41a006e720 con 0x7f418400a010 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.177+0000 7f41967fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:35.179 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":106,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:17:35.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 msgr2=0x7f4184041bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 0x7f4184041bf0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f4190033000 tx=0x7f419000f040 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 10 monclient: shutdown 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 msgr2=0x7f41a00a37e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f419c00a010 tx=0x7f419c00bf20 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.179+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 shutdown_connections 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f418403f730 0x7f4184041bf0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f41a01cd110 0x7f41a009e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f41a00a46a0 0x7f41a00a3d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 --2- 172.21.15.164:0/4051616899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f418400a010 0x7f41a00a37e0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 >> 172.21.15.164:0/4051616899 conn(0x7f41a00aa900 msgr2=0x7f41a0132da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 shutdown_connections 2024-09-10T16:17:35.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.180+0000 7f41a7edd640 1 -- 172.21.15.164:0/4051616899 wait complete. 2024-09-10T16:17:35.192 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:17:35.192 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:17:35.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.288+0000 7f4ae30fe640 10 monclient: get_monmap_and_config 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.288+0000 7f4ae30fe640 10 monclient: build_initial_monmap 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.288+0000 7f4ae30fe640 10 monclient: monmap: 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:35.289720+0000 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:35.289720+0000 2024-09-10T16:17:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:35.291 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.289+0000 7f4ae30fe640 1 Processor -- start 2024-09-10T16:17:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.289+0000 7f4ae30fe640 1 -- start start 2024-09-10T16:17:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.289+0000 7f4ae30fe640 10 monclient: init 2024-09-10T16:17:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4adc09d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4adc09e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 0x7f4adc13bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc1397c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4adc074c70 con 0x7f4adc09d930 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4adc09a540 con 0x7f4adc1397c0 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4adc0a0060 con 0x7f4adc09e420 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae30fe640 10 monclient(hunting): authenticate will time out at 1433.777204s 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae169b640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 msgr2=0x7f4adc13bbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae0e9a640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f4adc09d930 legacy=0x7f4adc09dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4ae169b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 0x7f4adc13bbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37018/0 (socket says 172.21.15.164:37018) 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 -- 172.21.15.164:0/1709154615 learned_addr learned my addr 172.21.15.164:0/1709154615 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): get_auth_request con 0x7f4adc09e420 auth_method 0 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient(hunting): handle_auth_done global_id 114236 payload 274 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 -- 172.21.15.164:0/1709154615 >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 msgr2=0x7f4adc13bbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 --2- 172.21.15.164:0/1709154615 >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 0x7f4adc13bbb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 -- 172.21.15.164:0/1709154615 >> v1:172.21.15.165:6789/0 conn(0x7f4adc09d930 legacy=0x7f4adc09dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 monclient: found mon.noname-b 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:17:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.290+0000 7f4adbfff640 1 -- 172.21.15.164:0/1709154615 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4adc09ed40 con 0x7f4adc09e420 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adbfff640 1 -- 172.21.15.164:0/1709154615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 msgr2=0x7f4adc09ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adbfff640 1 --2- 172.21.15.164:0/1709154615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00d150 tx=0x7f4acc02c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dace1c5bb81ab786 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 1 -- 172.21.15.164:0/1709154615 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc032070 con 0x7f4adc09e420 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: dump: 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.302 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 1 -- 172.21.15.164:0/1709154615 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4acc032910 con 0x7f4adc09e420 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 1 -- 172.21.15.164:0/1709154615 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4acc032ce0 con 0x7f4adc09e420 2024-09-10T16:17:35.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4ae30fe640 5 monclient: authenticate success, global_id 114236 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4ae30fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4ae30fe640 4 set_mon_vals no callback set 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.293+0000 7f4adb7fe640 10 monclient: dump: 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.304 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 10 monclient: shutdown 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 -- 172.21.15.164:0/1709154615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 msgr2=0x7f4adc09ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/1709154615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4acc00d150 tx=0x7f4acc02c780 comp rx=0 tx=0).stop 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 -- 172.21.15.164:0/1709154615 shutdown_connections 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/1709154615 >> v2:172.21.15.165:3300/0 conn(0x7f4adc1397c0 0x7f4adc13bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/1709154615 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc09e420 0x7f4adc09ab20 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 -- 172.21.15.164:0/1709154615 >> 172.21.15.164:0/1709154615 conn(0x7f4adc0a3110 msgr2=0x7f4adc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 -- 172.21.15.164:0/1709154615 shutdown_connections 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 1 -- 172.21.15.164:0/1709154615 wait complete. 2024-09-10T16:17:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 10 monclient: build_initial_monmap 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.294+0000 7f4ae30fe640 10 monclient: monmap: 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 Processor -- start 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 -- start start 2024-09-10T16:17:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient: init 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4adc09d930 0x7f4adc1c8e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-b con 0x7f4adc09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 0x7f4adc134a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-a con 0x7f4adc09e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 0x7f4adc072cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): picked mon.noname-c con 0x7f4adc1397c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc0a2ac0 con 0x7f4adc09e420 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc0a2940 con 0x7f4adc09d930 2024-09-10T16:17:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4adc0a1e40 con 0x7f4adc1397c0 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae30fe640 10 monclient(hunting): authenticate will time out at 1433.782801s 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae0e9a640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4adc09d930 msgr2=0x7f4adc1c8e40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 0x7f4adc134a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 0x7f4adc072cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae0e9a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4adc09d930 0x7f4adc1c8e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 0x7f4adc134a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48786/0 (socket says 172.21.15.164:48786) 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 1 -- 172.21.15.164:0/2030462915 learned_addr learned my addr 172.21.15.164:0/2030462915 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 10 monclient(hunting): get_auth_request con 0x7f4adc09e420 auth_method 0 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 0x7f4adc072cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37032/0 (socket says 172.21.15.164:37032) 2024-09-10T16:17:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 10 monclient(hunting): get_auth_request con 0x7f4adc1397c0 auth_method 0 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.295+0000 7f4ae169b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4adbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient(hunting): handle_auth_done global_id 114239 payload 824 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4adc09d930 msgr2=0x7f4adc1c8e40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4adc09d930 0x7f4adc1c8e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 msgr2=0x7f4adc134a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 0x7f4adc134a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient: _finish_hunting 0 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 monclient: found mon.noname-c 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4adc0a5f90 con 0x7f4adc1397c0 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.296+0000 7f4adbfff640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc09e420 0x7f4adc134a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 msgr2=0x7f4adc072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae169b640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 0x7f4adc072cb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400b3e0 tx=0x7f4ad400b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad4017070 con 0x7f4adc1397c0 2024-09-10T16:17:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:17:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: dump: 2024-09-10T16:17:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: _finish_auth 0 2024-09-10T16:17:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 5 monclient: authenticate success, global_id 114239 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ad40060d0 con 0x7f4adc1397c0 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 10 monclient: _renew_subs 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ad40064c0 con 0x7f4adc1397c0 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4adc09d620 con 0x7f4adc1397c0 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ad97fa640 10 monclient: dump: 2024-09-10T16:17:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4adaffd640 4 set_mon_vals no callback set 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 10 monclient: _renew_subs 2024-09-10T16:17:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.298+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4adc12d2c0 con 0x7f4adc1397c0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae30fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 0x7f4adc02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc0a2290 con 0x7f4adc134fc0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 0x7f4adc02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient: get_auth_request con 0x7f4adc134fc0 auth_method 0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4ad401a070 con 0x7f4adc1397c0 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_done global_id 124155 payload 824 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ae0e9a640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4adc06fee0 con 0x7f4adc134fc0 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ad97fa640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4abc03e240 0x7f4abc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(383..383 src has 184..383) ==== 4771+0+0 (secure 0 0 0) 0x7f4ad4051f60 con 0x7f4adc1397c0 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.299+0000 7f4adbfff640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4abc03e240 0x7f4abc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.300+0000 7f4adbfff640 10 monclient: get_auth_request con 0x7f4abc03e240 auth_method 0 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ae0e9a640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 msgr2=0x7f4adc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ae0e9a640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 0x7f4adc02d440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400b450 tx=0x7f4ac400b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4005270 con 0x7f4adc134fc0 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 10 monclient: dump: 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4ac4015070 con 0x7f4adc134fc0 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 msgr2=0x7f4adc02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 0x7f4adc02d440 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400b450 tx=0x7f4ac400b910 comp rx=0 tx=0).stop 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 msgr2=0x7f4adc02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4ad97fa640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc134fc0 0x7f4adc02d440 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.302+0000 7f4adbfff640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4abc03e240 0x7f4abc040700 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4acc038040 tx=0x7f4acc032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae30fe640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 0x7f4adc02dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae30fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4adc1c9380 con 0x7f4adc02d980 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 0x7f4adc02dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient: get_auth_request con 0x7f4adc02d980 auth_method 0 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 10 monclient(hunting): handle_auth_done global_id 124158 payload 824 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.308+0000 7f4ae0e9a640 1 -- 172.21.15.164:0/2030462915 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4adc077260 con 0x7f4adc02d980 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae0e9a640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 msgr2=0x7f4adc02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae0e9a640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 0x7f4adc02dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400b450 tx=0x7f4ac4005990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ac4010610 con 0x7f4adc02d980 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 10 monclient: dump: 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f4ac4010610 con 0x7f4adc02d980 2024-09-10T16:17:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 msgr2=0x7f4adc02dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 0x7f4adc02dd90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400b450 tx=0x7f4ac4005990 comp rx=0 tx=0).stop 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 msgr2=0x7f4adc02dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ad97fa640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4adc02d980 0x7f4adc02dd90 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f4ac400b450 tx=0x7f4ac4005990 comp rx=0 tx=0).stop 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:35.324 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 106, 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 31345, 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:35.325 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:35.326 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.327 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:35.328 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.329 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:35.330 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.331 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:17:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:35.338 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:35.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4abc03e240 msgr2=0x7f4abc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4abc03e240 0x7f4abc040700 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f4acc038040 tx=0x7f4acc032000 comp rx=0 tx=0).stop 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 10 monclient: shutdown 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 msgr2=0x7f4adc072cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 --2- 172.21.15.164:0/2030462915 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4adc1397c0 0x7f4adc072cb0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4ad400b3e0 tx=0x7f4ad400b8a0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 reap_dead start 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae169b640 1 -- 172.21.15.164:0/2030462915 reap_dead start 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 shutdown_connections 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 >> 172.21.15.164:0/2030462915 conn(0x7f4adc0a3110 msgr2=0x7f4adc138ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 shutdown_connections 2024-09-10T16:17:35.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.311+0000 7f4ae30fe640 1 -- 172.21.15.164:0/2030462915 wait complete. 2024-09-10T16:17:35.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:17:35.341 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:17:35.341 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.438+0000 7f105a2bd640 10 monclient: get_monmap_and_config 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.438+0000 7f105a2bd640 10 monclient: build_initial_monmap 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.438+0000 7f105a2bd640 10 monclient: monmap: 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:17:35.439712+0000 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:17:35.439712+0000 2024-09-10T16:17:35.440 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:17:35.441 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.439+0000 7f105a2bd640 1 Processor -- start 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.439+0000 7f105a2bd640 1 -- start start 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.439+0000 7f105a2bd640 10 monclient: init 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f105406cd00 addr v1:172.21.15.165:6789/0 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1054135000 0x7f105413cf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-c con 0x7f1054135000 addr v2:172.21.15.165:3300/0 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f105413d4b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:17:35.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1054135450 con 0x7f105413d4b0 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f105406df10 con 0x7f105406cd00 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f105406d0e0 con 0x7f1054135000 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105a2bd640 10 monclient(hunting): authenticate will time out at 1433.927165s 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f1053fff640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f105406cd00 legacy=0x7f10540a46e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f10537fe640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f1054135000 msgr2=0x7f105413cf70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:35.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f10537fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1054135000 0x7f105413cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:35.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105885a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f105413d4b0 0x7f105413f8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54882/0 (socket says 172.21.15.164:54882) 2024-09-10T16:17:35.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.440+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 learned_addr learned my addr 172.21.15.164:0/2714840348 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 >> v1:172.21.15.164:6789/0 conn(0x7f105413d4b0 legacy=0x7f105413f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982762294 0 0) 0x7f1054135450 con 0x7f105413d4b0 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): _negotiate 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): my global_id is 124161 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f103c003520 con 0x7f105413d4b0 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1588846875 0 0) 0x7f103c003520 con 0x7f105413d4b0 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 >> v2:172.21.15.165:3300/0 conn(0x7f1054135000 msgr2=0x7f105413cf70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:35.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 --2- 172.21.15.164:0/2714840348 >> v2:172.21.15.165:3300/0 conn(0x7f1054135000 0x7f105413cf70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 >> v1:172.21.15.165:6789/0 conn(0x7f105406cd00 legacy=0x7f10540a46e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105409cb90 con 0x7f105413d4b0 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _finish_auth 0 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f104c001d20 con 0x7f105413d4b0 2024-09-10T16:17:35.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f105a2bd640 5 monclient: authenticate success, global_id 124161 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f105a2bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: dump: 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.457 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 >> v1:172.21.15.164:6789/0 conn(0x7f105413d4b0 legacy=0x7f105413f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f103c003a70 0x7f103c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): picked mon.b con 0x7f103c003a70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c004800 0x7f103c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): picked mon.c con 0x7f103c004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 0x7f103c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): picked mon.a con 0x7f103c0099b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:35.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f105406d0e0 con 0x7f103c0099b0 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f103c00c3c0 con 0x7f103c003a70 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f103c00c5c0 con 0x7f103c004800 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1052ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f105a2bd640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.442+0000 7f1053fff640 1 -- 172.21.15.164:0/2714840348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f103c003a70 msgr2=0x7f103c003e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f1053fff640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f103c003a70 0x7f103c003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 0x7f103c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c004800 0x7f103c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): get_auth_request con 0x7f103c0099b0 auth_method 0 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): get_auth_request con 0x7f103c004800 auth_method 0 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient(hunting): handle_auth_done global_id 124161 payload 293 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c004800 msgr2=0x7f103c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c004800 0x7f103c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f103c003a70 msgr2=0x7f103c003e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f103c003a70 0x7f103c003e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient: _finish_hunting 0 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 monclient: found mon.a 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f103c003520 con 0x7f103c0099b0 2024-09-10T16:17:35.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient: _finish_auth 0 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 10 monclient: _check_auth_tickets 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f10537fe640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c004800 0x7f103c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.443+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 reap_dead start 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105885a640 1 -- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 msgr2=0x7f103c00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105885a640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 0x7f103c00bda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f104c001850 tx=0x7f103c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaffd5d95865e08f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1048009430 con 0x7f103c0099b0 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:35.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: dump: 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c02f070 con 0x7f103c0099b0 2024-09-10T16:17:35.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 1 -- 172.21.15.164:0/2714840348 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00a690 con 0x7f103c0099b0 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 10 monclient: get_monmap_and_config success 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 4 set_mon_vals no callback set 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f1052ffd640 10 monclient: dump: 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.464 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 10 monclient: shutdown 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 -- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 msgr2=0x7f103c00bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 0x7f103c00bda0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f104c001850 tx=0x7f103c005830 comp rx=0 tx=0).stop 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 -- 172.21.15.164:0/2714840348 shutdown_connections 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 --2- 172.21.15.164:0/2714840348 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f103c0099b0 0x7f103c00bda0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 -- 172.21.15.164:0/2714840348 >> 172.21.15.164:0/2714840348 conn(0x7f10540aaa70 msgr2=0x7f10540aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 -- 172.21.15.164:0/2714840348 shutdown_connections 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.446+0000 7f105a2bd640 1 -- 172.21.15.164:0/2714840348 wait complete. 2024-09-10T16:17:35.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.447+0000 7f105a2bd640 10 monclient: build_initial_monmap 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.447+0000 7f105a2bd640 10 monclient: monmap: 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.447+0000 7f105a2bd640 1 Processor -- start 2024-09-10T16:17:35.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.447+0000 7f105a2bd640 1 -- start start 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.447+0000 7f105a2bd640 10 monclient: init 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 0x7f10541ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-c con 0x7f103c0099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10541cd220 0x7f10541cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-b con 0x7f10541cd220 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 0x7f10541c9c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): picked mon.noname-a con 0x7f10541c9780 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f10540a46e0 con 0x7f10541c9780 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1054134b80 con 0x7f10541cd220 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f10540a4860 con 0x7f103c0099b0 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105a2bd640 10 monclient(hunting): authenticate will time out at 1433.935320s 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 0x7f10541ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 0x7f10541c9c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 0x7f10541ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37052/0 (socket says 172.21.15.164:37052) 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 1 -- 172.21.15.164:0/1174582474 learned_addr learned my addr 172.21.15.164:0/1174582474 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f10537fe640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10541cd220 msgr2=0x7f10541cd680 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): get_auth_request con 0x7f103c0099b0 auth_method 0 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f10537fe640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10541cd220 0x7f10541cd680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): get_auth_request con 0x7f10541c9780 auth_method 0 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient(hunting): handle_auth_done global_id 124164 payload 824 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 msgr2=0x7f10541ccce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 0x7f10541ccce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10541cd220 msgr2=0x7f10541cd680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f10541cd220 0x7f10541cd680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient: _finish_hunting 0 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 monclient: found mon.noname-a 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f105409e530 con 0x7f10541c9780 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.448+0000 7f1053fff640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f103c0099b0 0x7f10541ccce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.450+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 msgr2=0x7f10541c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.450+0000 7f105885a640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 0x7f10541c9c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1054134b50 tx=0x7f105406d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c00a690 con 0x7f10541c9780 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: dump: 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.471 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: _finish_auth 0 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 5 monclient: authenticate success, global_id 124164 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f104c02bb30 con 0x7f10541c9780 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f104c034470 con 0x7f10541c9780 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 monclient: _renew_subs 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f105413c9b0 con 0x7f10541c9780 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f1050ff9640 10 monclient: dump: 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.473 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f10527fc640 4 set_mon_vals no callback set 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 monclient: _renew_subs 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10541ca140 con 0x7f10541c9780 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 0x7f105402e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.451+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f105413cc80 con 0x7f10541ca310 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 0x7f105402e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient: get_auth_request con 0x7f10541ca310 auth_method 0 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f104c034610 con 0x7f10541c9780 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 10 monclient(hunting): handle_auth_done global_id 114245 payload 824 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1053fff640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10540a4c40 con 0x7f10541ca310 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1050ff9640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f104003e1f0 0x7f10400406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(383..383 src has 184..383) ==== 4771+0+0 (secure 0 0 0) 0x7f104c072d90 con 0x7f10541c9780 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f10537fe640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f104003e1f0 0x7f10400406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.452+0000 7f10537fe640 10 monclient: get_auth_request con 0x7f104003e1f0 auth_method 0 2024-09-10T16:17:35.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1053fff640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 msgr2=0x7f105402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1053fff640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 0x7f105402e5c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f10440053f0 tx=0x7f10440058b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044016070 con 0x7f10541ca310 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 10 monclient: dump: 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.478 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f104400ee20 con 0x7f10541ca310 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 msgr2=0x7f105402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 0x7f105402e5c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f10440053f0 tx=0x7f10440058b0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 msgr2=0x7f105402e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.454+0000 7f1050ff9640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f10541ca310 0x7f105402e5c0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f10440053f0 tx=0x7f10440058b0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.455+0000 7f10537fe640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f104003e1f0 0x7f10400406b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f105413cea0 tx=0x7f1048006f50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.460+0000 7f105a2bd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:17:35.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.460+0000 7f105a2bd640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 0x7f1054064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.460+0000 7f105a2bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.460+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f105413cf80 con 0x7f1054063d50 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 0x7f1054064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient: get_auth_request con 0x7f1054063d50 auth_method 0 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 10 monclient(hunting): handle_auth_done global_id 114248 payload 824 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.461+0000 7f1053fff640 1 -- 172.21.15.164:0/1174582474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f105406e670 con 0x7f1054063d50 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1053fff640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 msgr2=0x7f1054064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:17:35.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1053fff640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 0x7f1054064160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1044018000 tx=0x7f10440059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1044016040 con 0x7f1054063d50 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 10 monclient: dump: 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:17:35.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f1044005c50 con 0x7f1054063d50 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 msgr2=0x7f1054064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 0x7f1054064160 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1044018000 tx=0x7f10440059d0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 msgr2=0x7f1054064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:17:35.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.463+0000 7f1050ff9640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1054063d50 0x7f1054064160 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1044018000 tx=0x7f10440059d0 comp rx=0 tx=0).stop 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 106, 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 76031, 2024-09-10T16:17:35.484 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.485 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.486 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:35.487 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:17:35.488 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:17:35.489 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.490 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.491 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.492 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.493 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.495 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:17:35.496 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:17:35.497 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:17:35.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f104003e1f0 msgr2=0x7f10400406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f104003e1f0 0x7f10400406b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f105413cea0 tx=0x7f1048006f50 comp rx=0 tx=0).stop 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 10 monclient: shutdown 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 msgr2=0x7f10541c9c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 --2- 172.21.15.164:0/1174582474 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f10541c9780 0x7f10541c9c00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1054134b50 tx=0x7f105406d710 comp rx=0 tx=0).stop 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 reap_dead start 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105885a640 1 -- 172.21.15.164:0/1174582474 reap_dead start 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 shutdown_connections 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 >> 172.21.15.164:0/1174582474 conn(0x7f10540aaa70 msgr2=0x7f10540a7860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 shutdown_connections 2024-09-10T16:17:35.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:17:35.464+0000 7f105a2bd640 1 -- 172.21.15.164:0/1174582474 wait complete. 2024-09-10T16:17:35.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:17:35.500 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:17:36.251 INFO:tasks.rados.rados.0.smithi165.stdout:1575: done (2 left) 2024-09-10T16:17:36.251 INFO:tasks.rados.rados.0.smithi165.stdout:1576: done (1 left) 2024-09-10T16:17:36.251 INFO:tasks.rados.rados.0.smithi165.stdout:1577: done (0 left) 2024-09-10T16:17:36.251 INFO:tasks.rados.rados.0.smithi165.stdout:1578: snap_remove snap 177 2024-09-10T16:17:37.261 INFO:tasks.rados.rados.0.smithi165.stdout:1578: done (0 left) 2024-09-10T16:17:37.261 INFO:tasks.rados.rados.0.smithi165.stdout:1579: delete oid 11 current snap is 181 2024-09-10T16:17:37.263 INFO:tasks.rados.rados.0.smithi165.stdout:1579: done (0 left) 2024-09-10T16:17:37.263 INFO:tasks.rados.rados.0.smithi165.stdout:1580: rollback oid 6 current snap is 181 2024-09-10T16:17:37.263 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 179 2024-09-10T16:17:37.264 INFO:tasks.rados.rados.0.smithi165.stdout:1581: copy_from oid 1 from oid 49 current snap is 181 2024-09-10T16:17:37.264 INFO:tasks.rados.rados.0.smithi165.stdout:1582: snap_remove snap 172 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1582: done (2 left) 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1583: delete oid 10 current snap is 181 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1580: finishing rollback tid 0 to smithi16534841-6 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1580: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1580: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 630 (ObjNum 558 snap 175 seq_num 558) dirty exists 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1581: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:1581: finishing copy_from to smithi16534841-1 2024-09-10T16:17:38.277 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1002 (ObjNum 472 snap 152 seq_num 472) dirty exists 2024-09-10T16:17:38.280 INFO:tasks.rados.rados.0.smithi165.stdout:1580: done (2 left) 2024-09-10T16:17:38.280 INFO:tasks.rados.rados.0.smithi165.stdout:1581: done (1 left) 2024-09-10T16:17:38.280 INFO:tasks.rados.rados.0.smithi165.stdout:1583: done (0 left) 2024-09-10T16:17:38.281 INFO:tasks.rados.rados.0.smithi165.stdout:1584: copy_from oid 23 from oid 2 current snap is 181 2024-09-10T16:17:38.281 INFO:tasks.rados.rados.0.smithi165.stdout:1585: write oid 37 current snap is 181 2024-09-10T16:17:38.281 INFO:tasks.rados.rados.0.smithi165.stdout:1585: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-10T16:17:38.286 INFO:tasks.rados.rados.0.smithi165.stdout:1585: writing smithi16534841-37 from 712762 to 1113818 tid 1 2024-09-10T16:17:38.292 INFO:tasks.rados.rados.0.smithi165.stdout:1585: writing smithi16534841-37 from 1537867 to 2183722 tid 2 2024-09-10T16:17:38.300 INFO:tasks.rados.rados.0.smithi165.stdout:1585: writing smithi16534841-37 from 2860355 to 3512762 tid 3 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:1586: snap_create 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:1584: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:1585: finishing write tid 1 to smithi16534841-37 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:1585: finishing write tid 2 to smithi16534841-37 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:1584: finishing copy_from to smithi16534841-23 2024-09-10T16:17:38.301 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1080 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:17:38.305 INFO:tasks.rados.rados.0.smithi165.stdout:1585: finishing write tid 3 to smithi16534841-37 2024-09-10T16:17:38.305 INFO:tasks.rados.rados.0.smithi165.stdout:1585: finishing write tid 4 to smithi16534841-37 2024-09-10T16:17:38.305 INFO:tasks.rados.rados.0.smithi165.stdout:1585: finishing write tid 5 to smithi16534841-37 2024-09-10T16:17:38.305 INFO:tasks.rados.rados.0.smithi165.stdout:1585: oid 37 updating version 0 to 583 2024-09-10T16:17:38.306 INFO:tasks.rados.rados.0.smithi165.stdout:1585: oid 37 version 583 is already newer than 582 2024-09-10T16:17:38.306 INFO:tasks.rados.rados.0.smithi165.stdout:1585: oid 37 version 583 is already newer than 581 2024-09-10T16:17:38.306 INFO:tasks.rados.rados.0.smithi165.stdout:1585: oid 37 updating version 583 to 584 2024-09-10T16:17:38.306 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 584 (ObjNum 566 snap 181 seq_num 566) dirty exists 2024-09-10T16:17:38.306 INFO:tasks.rados.rados.0.smithi165.stdout:1585: left oid 37 (ObjNum 566 snap 181 seq_num 566) 2024-09-10T16:17:38.814 INFO:tasks.rados.rados.0.smithi165.stdout:1584: done (2 left) 2024-09-10T16:17:38.814 INFO:tasks.rados.rados.0.smithi165.stdout:1585: done (1 left) 2024-09-10T16:17:38.814 INFO:tasks.rados.rados.0.smithi165.stdout:1586: done (0 left) 2024-09-10T16:17:38.814 INFO:tasks.rados.rados.0.smithi165.stdout:1587: snap_create 2024-09-10T16:17:40.160 INFO:tasks.rados.rados.0.smithi165.stdout:1587: done (0 left) 2024-09-10T16:17:40.160 INFO:tasks.rados.rados.0.smithi165.stdout:1588: write (excl) oid 18 current snap is 183 2024-09-10T16:17:40.160 INFO:tasks.rados.rados.0.smithi165.stdout:1588: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-10T16:17:40.166 INFO:tasks.rados.rados.0.smithi165.stdout:1588: writing smithi16534841-18 from 729569 to 1205874 tid 1 2024-09-10T16:17:40.174 INFO:tasks.rados.rados.0.smithi165.stdout:1588: writing smithi16534841-18 from 1996349 to 2502215 tid 2 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:1588: writing smithi16534841-18 from 3004131 to 3529569 tid 3 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:1589: rollback oid 36 current snap is 183 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 169 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:1588: finishing write tid 1 to smithi16534841-18 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:1588: finishing write tid 2 to smithi16534841-18 2024-09-10T16:17:40.180 INFO:tasks.rados.rados.0.smithi165.stdout:1590: delete oid 48 current snap is 183 2024-09-10T16:17:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:1590: done (2 left) 2024-09-10T16:17:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:1591: rollback oid 10 current snap is 183 2024-09-10T16:17:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 181 2024-09-10T16:17:40.182 INFO:tasks.rados.rados.0.smithi165.stdout:1592: write oid 39 current snap is 183 2024-09-10T16:17:40.183 INFO:tasks.rados.rados.0.smithi165.stdout:1592: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-09-10T16:17:40.190 INFO:tasks.rados.rados.0.smithi165.stdout:1592: writing smithi16534841-39 from 763183 to 1389986 tid 1 2024-09-10T16:17:40.198 INFO:tasks.rados.rados.0.smithi165.stdout:1592: writing smithi16534841-39 from 1996960 to 2790142 tid 2 2024-09-10T16:17:40.202 INFO:tasks.rados.rados.0.smithi165.stdout:1592: writing smithi16534841-39 from 3226271 to 3563183 tid 3 2024-09-10T16:17:40.202 INFO:tasks.rados.rados.0.smithi165.stdout:1593: delete oid 26 current snap is 183 2024-09-10T16:17:40.202 INFO:tasks.rados.rados.0.smithi165.stdout:1589: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:17:40.202 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 0 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: finishing write tid 3 to smithi16534841-18 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1591: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1015 (ObjNum 257 snap 0 seq_num 1162717392) dirty dne 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: finishing write tid 4 to smithi16534841-18 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: finishing write tid 5 to smithi16534841-18 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: oid 18 updating version 0 to 1084 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: oid 18 updating version 1084 to 1085 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: oid 18 version 1085 is already newer than 1082 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: oid 18 version 1085 is already newer than 1083 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1085 (ObjNum 567 snap 183 seq_num 567) dirty exists 2024-09-10T16:17:40.203 INFO:tasks.rados.rados.0.smithi165.stdout:1588: left oid 18 (ObjNum 567 snap 183 seq_num 567) 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1592: finishing write tid 1 to smithi16534841-39 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1588: done (4 left) 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1589: done (3 left) 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1591: done (2 left) 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1593: done (1 left) 2024-09-10T16:17:40.204 INFO:tasks.rados.rados.0.smithi165.stdout:1594: snap_remove snap 182 2024-09-10T16:17:40.830 INFO:tasks.rados.rados.0.smithi165.stdout:1594: done (1 left) 2024-09-10T16:17:40.830 INFO:tasks.rados.rados.0.smithi165.stdout:1595: delete oid 43 current snap is 183 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: finishing write tid 2 to smithi16534841-39 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: finishing write tid 3 to smithi16534841-39 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: finishing write tid 4 to smithi16534841-39 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: finishing write tid 5 to smithi16534841-39 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: oid 39 updating version 0 to 1087 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: oid 39 updating version 1087 to 1089 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: oid 39 version 1089 is already newer than 1088 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:1592: oid 39 version 1089 is already newer than 1086 2024-09-10T16:17:40.831 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1089 (ObjNum 569 snap 183 seq_num 569) dirty exists 2024-09-10T16:17:40.832 INFO:tasks.rados.rados.0.smithi165.stdout:1592: left oid 39 (ObjNum 569 snap 183 seq_num 569) 2024-09-10T16:17:40.841 INFO:tasks.rados.rados.0.smithi165.stdout:1592: done (1 left) 2024-09-10T16:17:40.841 INFO:tasks.rados.rados.0.smithi165.stdout:1595: done (0 left) 2024-09-10T16:17:40.842 INFO:tasks.rados.rados.0.smithi165.stdout:1596: snap_remove snap 178 2024-09-10T16:17:41.844 INFO:tasks.rados.rados.0.smithi165.stdout:1596: done (0 left) 2024-09-10T16:17:41.844 INFO:tasks.rados.rados.0.smithi165.stdout:1597: copy_from oid 48 from oid 22 current snap is 183 2024-09-10T16:17:41.844 INFO:tasks.rados.rados.0.smithi165.stdout:1598: snap_remove snap 167 2024-09-10T16:17:42.842 INFO:tasks.rados.rados.0.smithi165.stdout:1598: done (1 left) 2024-09-10T16:17:42.842 INFO:tasks.rados.rados.0.smithi165.stdout:1599: rollback oid 17 current snap is 183 2024-09-10T16:17:42.842 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 170 2024-09-10T16:17:42.843 INFO:tasks.rados.rados.0.smithi165.stdout:1600: snap_create 2024-09-10T16:17:42.843 INFO:tasks.rados.rados.0.smithi165.stdout:1597: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:17:42.843 INFO:tasks.rados.rados.0.smithi165.stdout:1597: finishing copy_from to smithi16534841-48 2024-09-10T16:17:42.843 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 755 (ObjNum 554 snap 174 seq_num 554) dirty exists 2024-09-10T16:17:42.852 INFO:tasks.rados.rados.0.smithi165.stdout:1599: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:17:42.853 INFO:tasks.rados.rados.0.smithi165.stdout:1599: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:17:42.853 INFO:tasks.rados.rados.0.smithi165.stdout:1599: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:17:42.853 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1018 (ObjNum 369 snap 108 seq_num 369) dirty exists 2024-09-10T16:17:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:1597: done (2 left) 2024-09-10T16:17:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:1599: done (1 left) 2024-09-10T16:17:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:1600: done (0 left) 2024-09-10T16:17:43.818 INFO:tasks.rados.rados.0.smithi165.stdout:1601: write (excl) oid 49 current snap is 184 2024-09-10T16:17:43.819 INFO:tasks.rados.rados.0.smithi165.stdout:1601: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2024-09-10T16:17:43.823 INFO:tasks.rados.rados.0.smithi165.stdout:1601: writing smithi16534841-49 from 413604 to 982507 tid 1 2024-09-10T16:17:43.829 INFO:tasks.rados.rados.0.smithi165.stdout:1601: writing smithi16534841-49 from 1488759 to 2145621 tid 2 2024-09-10T16:17:43.835 INFO:tasks.rados.rados.0.smithi165.stdout:1601: writing smithi16534841-49 from 2624893 to 3216602 tid 3 2024-09-10T16:17:43.835 INFO:tasks.rados.rados.0.smithi165.stdout:1602: snap_remove snap 181 2024-09-10T16:17:44.818 INFO:tasks.rados.rados.0.smithi165.stdout:1602: done (1 left) 2024-09-10T16:17:44.818 INFO:tasks.rados.rados.0.smithi165.stdout:1603: snap_remove snap 179 2024-09-10T16:17:45.821 INFO:tasks.rados.rados.0.smithi165.stdout:1603: done (1 left) 2024-09-10T16:17:45.822 INFO:tasks.rados.rados.0.smithi165.stdout:1604: snap_remove snap 183 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1604: done (1 left) 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1605: rollback oid 39 current snap is 184 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 180 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: finishing write tid 1 to smithi16534841-49 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: finishing write tid 2 to smithi16534841-49 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: finishing write tid 3 to smithi16534841-49 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: finishing write tid 4 to smithi16534841-49 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: finishing write tid 5 to smithi16534841-49 2024-09-10T16:17:46.825 INFO:tasks.rados.rados.0.smithi165.stdout:1601: oid 49 updating version 0 to 1022 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1601: oid 49 version 1022 is already newer than 1021 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1601: oid 49 version 1022 is already newer than 1019 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1601: oid 49 version 1022 is already newer than 1020 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1022 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1601: left oid 49 (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1601: done (1 left) 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1606: copy_from oid 8 from oid 7 current snap is 184 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1607: rollback oid 25 current snap is 184 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 180 2024-09-10T16:17:46.826 INFO:tasks.rados.rados.0.smithi165.stdout:1608: delete oid 30 current snap is 184 2024-09-10T16:17:46.828 INFO:tasks.rados.rados.0.smithi165.stdout:1605: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:17:46.829 INFO:tasks.rados.rados.0.smithi165.stdout:1605: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:17:46.833 INFO:tasks.rados.rados.0.smithi165.stdout:1605: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:17:46.833 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1092 (ObjNum 518 snap 159 seq_num 518) dirty exists 2024-09-10T16:17:46.833 INFO:tasks.rados.rados.0.smithi165.stdout:1605: done (3 left) 2024-09-10T16:17:46.834 INFO:tasks.rados.rados.0.smithi165.stdout:1608: done (2 left) 2024-09-10T16:17:46.834 INFO:tasks.rados.rados.0.smithi165.stdout:1609: write (excl) oid 31 current snap is 184 2024-09-10T16:17:46.834 INFO:tasks.rados.rados.0.smithi165.stdout:1609: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-10T16:17:46.840 INFO:tasks.rados.rados.0.smithi165.stdout:1609: writing smithi16534841-31 from 447218 to 1166619 tid 1 2024-09-10T16:17:46.849 INFO:tasks.rados.rados.0.smithi165.stdout:1609: writing smithi16534841-31 from 1889370 to 2549901 tid 2 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1609: writing smithi16534841-31 from 2963386 to 3577639 tid 3 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1610: rollback oid 45 current snap is 184 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 175 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1606: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1609: finishing write tid 1 to smithi16534841-31 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1607: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1607: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:1607: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:17:46.856 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1138 (ObjNum 552 snap 173 seq_num 552) dirty exists 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1606: finishing copy_from to smithi16534841-8 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1127 (ObjNum 538 snap 169 seq_num 538) dirty exists 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1609: finishing write tid 2 to smithi16534841-31 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1606: done (3 left) 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1607: done (2 left) 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1611: write oid 19 current snap is 184 2024-09-10T16:17:46.857 INFO:tasks.rados.rados.0.smithi165.stdout:1611: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-10T16:17:46.861 INFO:tasks.rados.rados.0.smithi165.stdout:1611: writing smithi16534841-19 from 464025 to 1258675 tid 1 2024-09-10T16:17:46.867 INFO:tasks.rados.rados.0.smithi165.stdout:1611: writing smithi16534841-19 from 1947852 to 2352041 tid 2 2024-09-10T16:17:46.874 INFO:tasks.rados.rados.0.smithi165.stdout:1611: writing smithi16534841-19 from 2990809 to 3416334 tid 3 2024-09-10T16:17:46.874 INFO:tasks.rados.rados.0.smithi165.stdout:1612: read oid 21 snap 169 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1612: expect (ObjNum 539 snap 169 seq_num 539) 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1613: copy_from oid 5 from oid 16 current snap is 184 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1610: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 0 (ObjNum 33 snap 0 seq_num 1162619440) dirty dne 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: finishing write tid 3 to smithi16534841-31 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: finishing write tid 4 to smithi16534841-31 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: finishing write tid 5 to smithi16534841-31 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: oid 31 updating version 0 to 1189 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: oid 31 version 1189 is already newer than 1187 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: oid 31 version 1189 is already newer than 1188 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:1609: oid 31 updating version 1189 to 1190 2024-09-10T16:17:46.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1190 (ObjNum 574 snap 184 seq_num 574) dirty exists 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1609: left oid 31 (ObjNum 574 snap 184 seq_num 574) 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1611: finishing write tid 1 to smithi16534841-19 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1611: finishing write tid 2 to smithi16534841-19 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1609: done (4 left) 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1610: done (3 left) 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1614: read oid 17 snap -1 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1614: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1615: write oid 27 current snap is 184 2024-09-10T16:17:46.876 INFO:tasks.rados.rados.0.smithi165.stdout:1615: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-10T16:17:46.879 INFO:tasks.rados.rados.0.smithi165.stdout:1615: writing smithi16534841-27 from 480832 to 950731 tid 1 2024-09-10T16:17:46.886 INFO:tasks.rados.rados.0.smithi165.stdout:1615: writing smithi16534841-27 from 1606334 to 2270534 tid 2 2024-09-10T16:17:46.894 INFO:tasks.rados.rados.0.smithi165.stdout:1615: writing smithi16534841-27 from 3018232 to 3680832 tid 3 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1616: copy_from oid 25 from oid 2 current snap is 184 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1613: finishing copy_from to smithi16534841-5 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1613: got expected ENOENT (src dne) 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1613: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1611: finishing write tid 3 to smithi16534841-19 2024-09-10T16:17:46.895 INFO:tasks.rados.rados.0.smithi165.stdout:1614: expect (ObjNum 369 snap 108 seq_num 369) 2024-09-10T16:17:47.028 INFO:tasks.rados.rados.0.smithi165.stdout:1611: finishing write tid 4 to smithi16534841-19 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: finishing write tid 5 to smithi16534841-19 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: oid 19 updating version 0 to 1193 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: oid 19 version 1193 is already newer than 1191 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: oid 19 version 1193 is already newer than 1192 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: oid 19 updating version 1193 to 1194 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1194 (ObjNum 575 snap 184 seq_num 575) dirty exists 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1611: left oid 19 (ObjNum 575 snap 184 seq_num 575) 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:1612: expect (ObjNum 539 snap 169 seq_num 539) 2024-09-10T16:17:47.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:47.074 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 290828 2024-09-10T16:17:47.125 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1550299 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: finishing write tid 1 to smithi16534841-27 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: finishing write tid 2 to smithi16534841-27 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: finishing write tid 3 to smithi16534841-27 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: finishing write tid 4 to smithi16534841-27 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: finishing write tid 5 to smithi16534841-27 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: oid 27 updating version 0 to 1130 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: oid 27 version 1130 is already newer than 1129 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: oid 27 updating version 1130 to 1131 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: oid 27 version 1131 is already newer than 1128 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1131 (ObjNum 576 snap 184 seq_num 576) dirty exists 2024-09-10T16:17:47.156 INFO:tasks.rados.rados.0.smithi165.stdout:1615: left oid 27 (ObjNum 576 snap 184 seq_num 576) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1616: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1616: finishing copy_from to smithi16534841-25 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1080 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1611: done (5 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1612: done (4 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1613: done (3 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1614: done (2 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1615: done (1 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1616: done (0 left) 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1617: copy_from oid 17 from oid 25 current snap is 184 2024-09-10T16:17:47.157 INFO:tasks.rados.rados.0.smithi165.stdout:1618: snap_create 2024-09-10T16:17:47.166 INFO:tasks.rados.rados.0.smithi165.stdout:1617: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:17:47.169 INFO:tasks.rados.rados.0.smithi165.stdout:1617: finishing copy_from to smithi16534841-17 2024-09-10T16:17:47.170 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1080 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:17:47.831 INFO:tasks.rados.rados.0.smithi165.stdout:1617: done (1 left) 2024-09-10T16:17:47.831 INFO:tasks.rados.rados.0.smithi165.stdout:1618: done (0 left) 2024-09-10T16:17:47.831 INFO:tasks.rados.rados.0.smithi165.stdout:1619: snap_remove snap 169 2024-09-10T16:17:48.817 INFO:tasks.rados.rados.0.smithi165.stdout:1619: done (0 left) 2024-09-10T16:17:48.817 INFO:tasks.rados.rados.0.smithi165.stdout:1620: write oid 40 current snap is 185 2024-09-10T16:17:48.817 INFO:tasks.rados.rados.0.smithi165.stdout:1620: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-10T16:17:48.822 INFO:tasks.rados.rados.0.smithi165.stdout:1620: writing smithi16534841-40 from 497639 to 1042787 tid 1 2024-09-10T16:17:48.828 INFO:tasks.rados.rados.0.smithi165.stdout:1620: writing smithi16534841-40 from 1548463 to 1956321 tid 2 2024-09-10T16:17:48.835 INFO:tasks.rados.rados.0.smithi165.stdout:1620: writing smithi16534841-40 from 2529302 to 3093724 tid 3 2024-09-10T16:17:48.839 INFO:tasks.rados.rados.0.smithi165.stdout:1620: writing smithi16534841-40 from 3528143 to 3697639 tid 4 2024-09-10T16:17:48.839 INFO:tasks.rados.rados.0.smithi165.stdout:1621: write oid 32 current snap is 185 2024-09-10T16:17:48.839 INFO:tasks.rados.rados.0.smithi165.stdout:1621: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-10T16:17:48.845 INFO:tasks.rados.rados.0.smithi165.stdout:1621: writing smithi16534841-32 from 514446 to 1251196 tid 1 2024-09-10T16:17:48.852 INFO:tasks.rados.rados.0.smithi165.stdout:1621: writing smithi16534841-32 from 1723298 to 2391167 tid 2 2024-09-10T16:17:48.861 INFO:tasks.rados.rados.0.smithi165.stdout:1621: writing smithi16534841-32 from 3073078 to 3714446 tid 3 2024-09-10T16:17:48.861 INFO:tasks.rados.rados.0.smithi165.stdout:1622: read oid 38 snap 175 2024-09-10T16:17:48.861 INFO:tasks.rados.rados.0.smithi165.stdout:1622: expect (ObjNum 557 snap 174 seq_num 557) 2024-09-10T16:17:48.861 INFO:tasks.rados.rados.0.smithi165.stdout:1623: copy_from oid 50 from oid 6 current snap is 185 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 1 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 2 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 3 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 4 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 5 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1621: finishing write tid 1 to smithi16534841-32 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: finishing write tid 6 to smithi16534841-40 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: oid 40 updating version 0 to 578 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: oid 40 version 578 is already newer than 574 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: oid 40 version 578 is already newer than 576 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: oid 40 version 578 is already newer than 575 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:1620: oid 40 version 578 is already newer than 577 2024-09-10T16:17:48.862 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 578 (ObjNum 577 snap 185 seq_num 577) dirty exists 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1620: left oid 40 (ObjNum 577 snap 185 seq_num 577) 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1620: done (3 left) 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1624: read oid 34 snap 180 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1624: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1625: write (excl) oid 21 current snap is 185 2024-09-10T16:17:48.863 INFO:tasks.rados.rados.0.smithi165.stdout:1625: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-10T16:17:48.864 INFO:tasks.rados.rados.0.smithi165.stdout:1625: writing smithi16534841-21 from 531253 to 943252 tid 1 2024-09-10T16:17:48.870 INFO:tasks.rados.rados.0.smithi165.stdout:1625: writing smithi16534841-21 from 1381780 to 1793307 tid 2 2024-09-10T16:17:48.877 INFO:tasks.rados.rados.0.smithi165.stdout:1625: writing smithi16534841-21 from 2300501 to 2887467 tid 3 2024-09-10T16:17:48.882 INFO:tasks.rados.rados.0.smithi165.stdout:1625: writing smithi16534841-21 from 3376974 to 3731253 tid 4 2024-09-10T16:17:48.882 INFO:tasks.rados.rados.0.smithi165.stdout:1626: copy_from oid 37 from oid 46 current snap is 185 2024-09-10T16:17:48.882 INFO:tasks.rados.rados.0.smithi165.stdout:1621: finishing write tid 2 to smithi16534841-32 2024-09-10T16:17:48.882 INFO:tasks.rados.rados.0.smithi165.stdout:1624: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:17:48.889 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 747651 2024-09-10T16:17:49.009 INFO:tasks.rados.rados.0.smithi165.stdout:1621: finishing write tid 3 to smithi16534841-32 2024-09-10T16:17:49.009 INFO:tasks.rados.rados.0.smithi165.stdout:1621: finishing write tid 4 to smithi16534841-32 2024-09-10T16:17:49.009 INFO:tasks.rados.rados.0.smithi165.stdout:1621: finishing write tid 5 to smithi16534841-32 2024-09-10T16:17:49.009 INFO:tasks.rados.rados.0.smithi165.stdout:1621: oid 32 updating version 0 to 631 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1621: oid 32 updating version 631 to 632 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1621: oid 32 updating version 632 to 633 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1621: oid 32 updating version 633 to 634 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 634 (ObjNum 578 snap 185 seq_num 578) dirty exists 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1621: left oid 32 (ObjNum 578 snap 185 seq_num 578) 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1621: done (5 left) 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1624: done (4 left) 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1627: write oid 43 current snap is 185 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 1 to smithi16534841-21 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 2 to smithi16534841-21 2024-09-10T16:17:49.010 INFO:tasks.rados.rados.0.smithi165.stdout:1627: seq_num 580 ranges {548060=487248,1440262=671538,2844277=798238} 2024-09-10T16:17:49.014 INFO:tasks.rados.rados.0.smithi165.stdout:1627: writing smithi16534841-43 from 548060 to 1035308 tid 1 2024-09-10T16:17:49.020 INFO:tasks.rados.rados.0.smithi165.stdout:1627: writing smithi16534841-43 from 1440262 to 2111800 tid 2 2024-09-10T16:17:49.029 INFO:tasks.rados.rados.0.smithi165.stdout:1627: writing smithi16534841-43 from 2844277 to 3642515 tid 3 2024-09-10T16:17:49.029 INFO:tasks.rados.rados.0.smithi165.stdout:1628: snap_remove snap 184 2024-09-10T16:17:49.825 INFO:tasks.rados.rados.0.smithi165.stdout:1628: done (5 left) 2024-09-10T16:17:49.825 INFO:tasks.rados.rados.0.smithi165.stdout:1629: write oid 10 current snap is 185 2024-09-10T16:17:49.825 INFO:tasks.rados.rados.0.smithi165.stdout:1629: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-10T16:17:49.831 INFO:tasks.rados.rados.0.smithi165.stdout:1629: writing smithi16534841-10 from 564867 to 1127364 tid 1 2024-09-10T16:17:49.838 INFO:tasks.rados.rados.0.smithi165.stdout:1629: writing smithi16534841-10 from 1898744 to 2313940 tid 2 2024-09-10T16:17:49.844 INFO:tasks.rados.rados.0.smithi165.stdout:1629: writing smithi16534841-10 from 2755347 to 3481210 tid 3 2024-09-10T16:17:49.844 INFO:tasks.rados.rados.0.smithi165.stdout:1630: snap_remove snap 180 2024-09-10T16:17:50.829 INFO:tasks.rados.rados.0.smithi165.stdout:1630: done (6 left) 2024-09-10T16:17:50.829 INFO:tasks.rados.rados.0.smithi165.stdout:1631: rollback oid 32 current snap is 185 2024-09-10T16:17:50.829 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 32 to 170 2024-09-10T16:17:50.830 INFO:tasks.rados.rados.0.smithi165.stdout:1622: expect (ObjNum 557 snap 174 seq_num 557) 2024-09-10T16:17:50.830 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:50.869 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:50.945 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 3 to smithi16534841-21 2024-09-10T16:17:50.945 INFO:tasks.rados.rados.0.smithi165.stdout:1623: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:17:50.945 INFO:tasks.rados.rados.0.smithi165.stdout:1623: finishing copy_from to smithi16534841-50 2024-09-10T16:17:50.945 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 630 (ObjNum 558 snap 175 seq_num 558) dirty exists 2024-09-10T16:17:50.945 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 4 to smithi16534841-21 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1626: finishing copy_from to smithi16534841-37 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1626: got expected ENOENT (src dne) 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1626: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 5 to smithi16534841-21 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: finishing write tid 6 to smithi16534841-21 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: oid 21 updating version 0 to 1139 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: oid 21 updating version 1139 to 1140 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: oid 21 updating version 1140 to 1141 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: oid 21 updating version 1141 to 1143 2024-09-10T16:17:50.946 INFO:tasks.rados.rados.0.smithi165.stdout:1625: oid 21 version 1143 is already newer than 1142 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1143 (ObjNum 579 snap 185 seq_num 579) dirty exists 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1625: left oid 21 (ObjNum 579 snap 185 seq_num 579) 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1627: finishing write tid 1 to smithi16534841-43 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1622: done (6 left) 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1623: done (5 left) 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1625: done (4 left) 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1626: done (3 left) 2024-09-10T16:17:50.947 INFO:tasks.rados.rados.0.smithi165.stdout:1632: read oid 22 snap -1 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1632: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1633: write (excl) oid 41 current snap is 185 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: finishing write tid 2 to smithi16534841-43 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: finishing write tid 3 to smithi16534841-43 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: finishing write tid 4 to smithi16534841-43 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: finishing write tid 5 to smithi16534841-43 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: oid 43 updating version 0 to 884 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: oid 43 updating version 884 to 886 2024-09-10T16:17:50.948 INFO:tasks.rados.rados.0.smithi165.stdout:1627: oid 43 version 886 is already newer than 883 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1627: oid 43 version 886 is already newer than 885 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 886 (ObjNum 580 snap 185 seq_num 580) dirty exists 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1627: left oid 43 (ObjNum 580 snap 185 seq_num 580) 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1629: finishing write tid 1 to smithi16534841-10 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1629: finishing write tid 2 to smithi16534841-10 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1629: finishing write tid 3 to smithi16534841-10 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1629: finishing write tid 4 to smithi16534841-10 2024-09-10T16:17:50.949 INFO:tasks.rados.rados.0.smithi165.stdout:1633: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-09-10T16:17:50.950 INFO:tasks.rados.rados.0.smithi165.stdout:1633: writing smithi16534841-41 from 581674 to 1219420 tid 1 2024-09-10T16:17:50.957 INFO:tasks.rados.rados.0.smithi165.stdout:1633: writing smithi16534841-41 from 1840873 to 2399727 tid 2 2024-09-10T16:17:50.966 INFO:tasks.rados.rados.0.smithi165.stdout:1633: writing smithi16534841-41 from 3066417 to 3603552 tid 3 2024-09-10T16:17:50.967 INFO:tasks.rados.rados.0.smithi165.stdout:1627: done (4 left) 2024-09-10T16:17:50.967 INFO:tasks.rados.rados.0.smithi165.stdout:1634: copy_from oid 19 from oid 23 current snap is 185 2024-09-10T16:17:50.967 INFO:tasks.rados.rados.0.smithi165.stdout:1629: finishing write tid 5 to smithi16534841-10 2024-09-10T16:17:50.967 INFO:tasks.rados.rados.0.smithi165.stdout:1629: oid 10 updating version 0 to 1083 2024-09-10T16:17:50.967 INFO:tasks.rados.rados.0.smithi165.stdout:1629: oid 10 updating version 1083 to 1084 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:1629: oid 10 version 1084 is already newer than 1082 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:1629: oid 10 version 1084 is already newer than 1081 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1084 (ObjNum 581 snap 185 seq_num 581) dirty exists 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:1629: left oid 10 (ObjNum 581 snap 185 seq_num 581) 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:1629: done (4 left) 2024-09-10T16:17:50.968 INFO:tasks.rados.rados.0.smithi165.stdout:1635: read oid 50 snap 170 2024-09-10T16:17:50.969 INFO:tasks.rados.rados.0.smithi165.stdout:1635: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:17:50.969 INFO:tasks.rados.rados.0.smithi165.stdout:1636: delete oid 10 current snap is 185 2024-09-10T16:17:50.969 INFO:tasks.rados.rados.0.smithi165.stdout:1631: finishing rollback tid 0 to smithi16534841-32 2024-09-10T16:17:50.969 INFO:tasks.rados.rados.0.smithi165.stdout:1631: finishing rollback tid 1 to smithi16534841-32 2024-09-10T16:17:50.969 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 636 (ObjNum 129 snap 0 seq_num 1162756304) dirty dne 2024-09-10T16:17:50.970 INFO:tasks.rados.rados.0.smithi165.stdout:1632: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:17:50.970 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1704282 2024-09-10T16:17:50.995 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1235544 2024-09-10T16:17:51.050 INFO:tasks.rados.rados.0.smithi165.stdout:1633: finishing write tid 1 to smithi16534841-41 2024-09-10T16:17:51.050 INFO:tasks.rados.rados.0.smithi165.stdout:1633: finishing write tid 2 to smithi16534841-41 2024-09-10T16:17:51.050 INFO:tasks.rados.rados.0.smithi165.stdout:1635: expect (ObjNum 417 snap 137 seq_num 417) 2024-09-10T16:17:51.050 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:51.136 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: finishing write tid 3 to smithi16534841-41 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: finishing write tid 4 to smithi16534841-41 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: finishing write tid 5 to smithi16534841-41 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: oid 41 updating version 0 to 1096 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: oid 41 version 1096 is already newer than 1094 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: oid 41 version 1096 is already newer than 1095 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: oid 41 version 1096 is already newer than 1093 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1096 (ObjNum 582 snap 185 seq_num 582) dirty exists 2024-09-10T16:17:51.180 INFO:tasks.rados.rados.0.smithi165.stdout:1633: left oid 41 (ObjNum 582 snap 185 seq_num 582) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1634: finishing copy_from racing read to smithi16534841-19 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1634: finishing copy_from to smithi16534841-19 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1080 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1631: done (5 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1632: done (4 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1633: done (3 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1634: done (2 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1635: done (1 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1636: done (0 left) 2024-09-10T16:17:51.181 INFO:tasks.rados.rados.0.smithi165.stdout:1637: delete oid 14 current snap is 185 2024-09-10T16:17:51.182 INFO:tasks.rados.rados.0.smithi165.stdout:1637: done (0 left) 2024-09-10T16:17:51.182 INFO:tasks.rados.rados.0.smithi165.stdout:1638: snap_remove snap 170 2024-09-10T16:17:51.842 INFO:tasks.rados.rados.0.smithi165.stdout:1638: done (0 left) 2024-09-10T16:17:51.842 INFO:tasks.rados.rados.0.smithi165.stdout:1639: read oid 1 snap 175 2024-09-10T16:17:51.842 INFO:tasks.rados.rados.0.smithi165.stdout:1639: expect (ObjNum 536 snap 169 seq_num 536) 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1640: read oid 47 snap 175 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1640: expect (ObjNum 499 snap 157 seq_num 499) 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1641: rollback oid 2 current snap is 185 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 175 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1642: read oid 22 snap -1 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1642: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:17:51.843 INFO:tasks.rados.rados.0.smithi165.stdout:1643: snap_create 2024-09-10T16:17:51.849 INFO:tasks.rados.rados.0.smithi165.stdout:1642: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:17:51.852 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1501790 2024-09-10T16:17:51.880 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1731395 2024-09-10T16:17:51.921 INFO:tasks.rados.rados.0.smithi165.stdout:1639: expect (ObjNum 536 snap 169 seq_num 536) 2024-09-10T16:17:51.930 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1102319 2024-09-10T16:17:51.992 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1828530 2024-09-10T16:17:52.068 INFO:tasks.rados.rados.0.smithi165.stdout:1640: expect (ObjNum 499 snap 157 seq_num 499) 2024-09-10T16:17:52.068 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:52.138 INFO:tasks.rados.rados.0.smithi165.stdout:1641: finishing rollback tid 0 to smithi16534841-2 2024-09-10T16:17:52.139 INFO:tasks.rados.rados.0.smithi165.stdout:1641: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:17:52.139 INFO:tasks.rados.rados.0.smithi165.stdout:1641: finishing rollback tid 2 to smithi16534841-2 2024-09-10T16:17:52.139 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1134 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1639: done (4 left) 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1640: done (3 left) 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1641: done (2 left) 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1642: done (1 left) 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1643: done (0 left) 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1644: read oid 44 snap -1 2024-09-10T16:17:52.842 INFO:tasks.rados.rados.0.smithi165.stdout:1644: expect deleted 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1645: read oid 23 snap -1 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1645: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1646: rollback oid 29 current snap is 186 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 175 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1647: read oid 4 snap -1 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1647: expect (ObjNum 553 snap 174 seq_num 553) 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1648: read oid 38 snap -1 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1648: expect (ObjNum 526 snap 162 seq_num 526) 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1649: write (excl) oid 26 current snap is 186 2024-09-10T16:17:52.843 INFO:tasks.rados.rados.0.smithi165.stdout:1649: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2024-09-10T16:17:52.848 INFO:tasks.rados.rados.0.smithi165.stdout:1649: writing smithi16534841-26 from 632095 to 1095588 tid 1 2024-09-10T16:17:52.854 INFO:tasks.rados.rados.0.smithi165.stdout:1649: writing smithi16534841-26 from 1616319 to 2038853 tid 2 2024-09-10T16:17:52.861 INFO:tasks.rados.rados.0.smithi165.stdout:1649: writing smithi16534841-26 from 2748686 to 3235990 tid 3 2024-09-10T16:17:52.861 INFO:tasks.rados.rados.0.smithi165.stdout:1650: rollback oid 35 current snap is 186 2024-09-10T16:17:52.861 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 175 2024-09-10T16:17:52.861 INFO:tasks.rados.rados.0.smithi165.stdout:1645: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:17:52.861 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:52.940 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:53.096 INFO:tasks.rados.rados.0.smithi165.stdout:1646: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:17:53.096 INFO:tasks.rados.rados.0.smithi165.stdout:1646: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:17:53.096 INFO:tasks.rados.rados.0.smithi165.stdout:1646: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:17:53.096 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1137 (ObjNum 528 snap 163 seq_num 528) dirty exists 2024-09-10T16:17:53.096 INFO:tasks.rados.rados.0.smithi165.stdout:1648: expect (ObjNum 526 snap 162 seq_num 526) 2024-09-10T16:17:53.234 INFO:tasks.rados.rados.0.smithi165.stdout:1647: expect (ObjNum 553 snap 174 seq_num 553) 2024-09-10T16:17:53.234 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:53.284 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:53.387 INFO:tasks.rados.rados.0.smithi165.stdout:1649: finishing write tid 1 to smithi16534841-26 2024-09-10T16:17:53.387 INFO:tasks.rados.rados.0.smithi165.stdout:1649: finishing write tid 2 to smithi16534841-26 2024-09-10T16:17:53.387 INFO:tasks.rados.rados.0.smithi165.stdout:1650: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:17:53.387 INFO:tasks.rados.rados.0.smithi165.stdout:1649: finishing write tid 3 to smithi16534841-26 2024-09-10T16:17:53.387 INFO:tasks.rados.rados.0.smithi165.stdout:1644: done (6 left) 2024-09-10T16:17:53.388 INFO:tasks.rados.rados.0.smithi165.stdout:1645: done (5 left) 2024-09-10T16:17:53.388 INFO:tasks.rados.rados.0.smithi165.stdout:1646: done (4 left) 2024-09-10T16:17:53.388 INFO:tasks.rados.rados.0.smithi165.stdout:1647: done (3 left) 2024-09-10T16:17:53.388 INFO:tasks.rados.rados.0.smithi165.stdout:1648: done (2 left) 2024-09-10T16:17:53.388 INFO:tasks.rados.rados.0.smithi165.stdout:1651: read oid 4 snap -1 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1651: expect (ObjNum 553 snap 174 seq_num 553) 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1652: write oid 3 current snap is 186 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1649: finishing write tid 4 to smithi16534841-26 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1650: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1649: finishing write tid 5 to smithi16534841-26 2024-09-10T16:17:53.389 INFO:tasks.rados.rados.0.smithi165.stdout:1649: oid 26 updating version 0 to 1208 2024-09-10T16:17:53.390 INFO:tasks.rados.rados.0.smithi165.stdout:1649: oid 26 updating version 1208 to 1209 2024-09-10T16:17:53.390 INFO:tasks.rados.rados.0.smithi165.stdout:1649: oid 26 version 1209 is already newer than 1207 2024-09-10T16:17:53.390 INFO:tasks.rados.rados.0.smithi165.stdout:1649: oid 26 version 1209 is already newer than 1206 2024-09-10T16:17:53.390 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1209 (ObjNum 585 snap 186 seq_num 585) dirty exists 2024-09-10T16:17:53.390 INFO:tasks.rados.rados.0.smithi165.stdout:1649: left oid 26 (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:17:53.391 INFO:tasks.rados.rados.0.smithi165.stdout:1650: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:17:53.391 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 758 (ObjNum 540 snap 170 seq_num 540) dirty exists 2024-09-10T16:17:53.391 INFO:tasks.rados.rados.0.smithi165.stdout:1652: seq_num 586 ranges {648902=655095,2074801=566192,3176109=672793} 2024-09-10T16:17:53.392 INFO:tasks.rados.rados.0.smithi165.stdout:1652: writing smithi16534841-3 from 648902 to 1303997 tid 1 2024-09-10T16:17:53.399 INFO:tasks.rados.rados.0.smithi165.stdout:1652: writing smithi16534841-3 from 2074801 to 2640993 tid 2 2024-09-10T16:17:53.406 INFO:tasks.rados.rados.0.smithi165.stdout:1652: writing smithi16534841-3 from 3176109 to 3848902 tid 3 2024-09-10T16:17:53.406 INFO:tasks.rados.rados.0.smithi165.stdout:1649: done (3 left) 2024-09-10T16:17:53.406 INFO:tasks.rados.rados.0.smithi165.stdout:1650: done (2 left) 2024-09-10T16:17:53.407 INFO:tasks.rados.rados.0.smithi165.stdout:1653: snap_remove snap 175 2024-09-10T16:17:53.819 INFO:tasks.rados.rados.0.smithi165.stdout:1653: done (2 left) 2024-09-10T16:17:53.819 INFO:tasks.rados.rados.0.smithi165.stdout:1654: copy_from oid 27 from oid 30 current snap is 186 2024-09-10T16:17:53.819 INFO:tasks.rados.rados.0.smithi165.stdout:1651: expect (ObjNum 553 snap 174 seq_num 553) 2024-09-10T16:17:53.966 INFO:tasks.rados.rados.0.smithi165.stdout:1652: finishing write tid 1 to smithi16534841-3 2024-09-10T16:17:53.966 INFO:tasks.rados.rados.0.smithi165.stdout:1652: finishing write tid 2 to smithi16534841-3 2024-09-10T16:17:53.966 INFO:tasks.rados.rados.0.smithi165.stdout:1652: finishing write tid 3 to smithi16534841-3 2024-09-10T16:17:53.966 INFO:tasks.rados.rados.0.smithi165.stdout:1652: finishing write tid 4 to smithi16534841-3 2024-09-10T16:17:53.966 INFO:tasks.rados.rados.0.smithi165.stdout:1652: finishing write tid 5 to smithi16534841-3 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: oid 3 updating version 0 to 1147 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: oid 3 version 1147 is already newer than 1144 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: oid 3 version 1147 is already newer than 1145 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: oid 3 version 1147 is already newer than 1146 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1147 (ObjNum 586 snap 186 seq_num 586) dirty exists 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: left oid 3 (ObjNum 586 snap 186 seq_num 586) 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1651: done (2 left) 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1652: done (1 left) 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1655: read oid 24 snap -1 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1655: expect (ObjNum 562 snap 178 seq_num 562) 2024-09-10T16:17:53.967 INFO:tasks.rados.rados.0.smithi165.stdout:1656: snap_remove snap 185 2024-09-10T16:17:54.850 INFO:tasks.rados.rados.0.smithi165.stdout:1656: done (2 left) 2024-09-10T16:17:54.850 INFO:tasks.rados.rados.0.smithi165.stdout:1657: delete oid 38 current snap is 186 2024-09-10T16:17:54.851 INFO:tasks.rados.rados.0.smithi165.stdout:1654: finishing copy_from to smithi16534841-27 2024-09-10T16:17:54.851 INFO:tasks.rados.rados.0.smithi165.stdout:1654: got expected ENOENT (src dne) 2024-09-10T16:17:54.851 INFO:tasks.rados.rados.0.smithi165.stdout:1654: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:17:54.851 INFO:tasks.rados.rados.0.smithi165.stdout:1655: expect (ObjNum 562 snap 178 seq_num 562) 2024-09-10T16:17:54.851 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 386726 2024-09-10T16:17:54.923 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:55.037 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2299704 2024-09-10T16:17:55.064 INFO:tasks.rados.rados.0.smithi165.stdout:1654: done (2 left) 2024-09-10T16:17:55.064 INFO:tasks.rados.rados.0.smithi165.stdout:1655: done (1 left) 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1657: done (0 left) 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1658: read oid 17 snap -1 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1658: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1659: read oid 13 snap -1 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1659: expect (ObjNum 491 snap 155 seq_num 491) 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1660: write (excl) oid 33 current snap is 186 2024-09-10T16:17:55.065 INFO:tasks.rados.rados.0.smithi165.stdout:1660: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-10T16:17:55.070 INFO:tasks.rados.rados.0.smithi165.stdout:1660: writing smithi16534841-33 from 682516 to 1088109 tid 1 2024-09-10T16:17:55.077 INFO:tasks.rados.rados.0.smithi165.stdout:1660: writing smithi16534841-33 from 1791765 to 2361626 tid 2 2024-09-10T16:17:55.084 INFO:tasks.rados.rados.0.smithi165.stdout:1660: writing smithi16534841-33 from 2830955 to 3552075 tid 3 2024-09-10T16:17:55.084 INFO:tasks.rados.rados.0.smithi165.stdout:1661: copy_from oid 9 from oid 46 current snap is 186 2024-09-10T16:17:55.084 INFO:tasks.rados.rados.0.smithi165.stdout:1659: expect (ObjNum 491 snap 155 seq_num 491) 2024-09-10T16:17:55.093 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 611008 2024-09-10T16:17:55.171 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1491165 2024-09-10T16:17:55.201 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:55.265 INFO:tasks.rados.rados.0.smithi165.stdout:1658: expect (ObjNum 533 snap 166 seq_num 533) 2024-09-10T16:17:55.266 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:55.421 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1660: finishing write tid 1 to smithi16534841-33 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1660: finishing write tid 2 to smithi16534841-33 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1658: done (3 left) 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1659: done (2 left) 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1662: write oid 41 current snap is 186 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1661: finishing copy_from to smithi16534841-9 2024-09-10T16:17:55.500 INFO:tasks.rados.rados.0.smithi165.stdout:1661: got expected ENOENT (src dne) 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1661: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: finishing write tid 3 to smithi16534841-33 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: finishing write tid 4 to smithi16534841-33 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: finishing write tid 5 to smithi16534841-33 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: oid 33 updating version 0 to 1149 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: oid 33 version 1149 is already newer than 1148 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: oid 33 updating version 1149 to 1151 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: oid 33 version 1151 is already newer than 1150 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1151 (ObjNum 588 snap 186 seq_num 588) dirty exists 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1660: left oid 33 (ObjNum 588 snap 186 seq_num 588) 2024-09-10T16:17:55.501 INFO:tasks.rados.rados.0.smithi165.stdout:1662: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2024-09-10T16:17:55.501 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:17:55.502 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:17:55.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:17:55.504 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:17:55.505 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:17:55.505 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:17:55.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:17:55.506 INFO:tasks.rados.rados.0.smithi165.stdout:1662: writing smithi16534841-41 from 699323 to 1180165 tid 1 2024-09-10T16:17:55.512 INFO:tasks.rados.rados.0.smithi165.stdout:1662: writing smithi16534841-41 from 1850247 to 2280119 tid 2 2024-09-10T16:17:55.518 INFO:tasks.rados.rados.0.smithi165.stdout:1662: writing smithi16534841-41 from 2858378 to 3390770 tid 3 2024-09-10T16:17:55.518 INFO:tasks.rados.rados.0.smithi165.stdout:1660: done (2 left) 2024-09-10T16:17:55.519 INFO:tasks.rados.rados.0.smithi165.stdout:1661: done (1 left) 2024-09-10T16:17:55.519 INFO:tasks.rados.rados.0.smithi165.stdout:1663: write (excl) oid 25 current snap is 186 2024-09-10T16:17:55.519 INFO:tasks.rados.rados.0.smithi165.stdout:1662: finishing write tid 1 to smithi16534841-41 2024-09-10T16:17:55.519 INFO:tasks.rados.rados.0.smithi165.stdout:1663: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-09-10T16:17:55.525 INFO:tasks.rados.rados.0.smithi165.stdout:1663: writing smithi16534841-25 from 716130 to 1272221 tid 1 2024-09-10T16:17:55.531 INFO:tasks.rados.rados.0.smithi165.stdout:1663: writing smithi16534841-25 from 1792376 to 2365906 tid 2 2024-09-10T16:17:55.536 INFO:tasks.rados.rados.0.smithi165.stdout:1663: writing smithi16534841-25 from 2769448 to 3229465 tid 3 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1663: writing smithi16534841-25 from 3821956 to 3916130 tid 4 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1664: copy_from oid 3 from oid 8 current snap is 186 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: finishing write tid 2 to smithi16534841-41 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: finishing write tid 3 to smithi16534841-41 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: finishing write tid 4 to smithi16534841-41 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: finishing write tid 5 to smithi16534841-41 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: oid 41 updating version 0 to 1099 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: oid 41 updating version 1099 to 1100 2024-09-10T16:17:55.541 INFO:tasks.rados.rados.0.smithi165.stdout:1662: oid 41 version 1100 is already newer than 1098 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1662: oid 41 version 1100 is already newer than 1097 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1100 (ObjNum 589 snap 186 seq_num 589) dirty exists 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1662: left oid 41 (ObjNum 589 snap 186 seq_num 589) 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 1 to smithi16534841-25 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 2 to smithi16534841-25 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 3 to smithi16534841-25 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1662: done (2 left) 2024-09-10T16:17:55.542 INFO:tasks.rados.rados.0.smithi165.stdout:1665: delete oid 32 current snap is 186 2024-09-10T16:17:55.543 INFO:tasks.rados.rados.0.smithi165.stdout:1665: done (2 left) 2024-09-10T16:17:55.543 INFO:tasks.rados.rados.0.smithi165.stdout:1666: read oid 1 snap -1 2024-09-10T16:17:55.543 INFO:tasks.rados.rados.0.smithi165.stdout:1666: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:17:55.543 INFO:tasks.rados.rados.0.smithi165.stdout:1667: delete oid 20 current snap is 186 2024-09-10T16:17:55.543 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 4 to smithi16534841-25 2024-09-10T16:17:55.545 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 5 to smithi16534841-25 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: finishing write tid 6 to smithi16534841-25 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: oid 25 updating version 0 to 1153 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: oid 25 updating version 1153 to 1156 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: oid 25 version 1156 is already newer than 1154 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: oid 25 version 1156 is already newer than 1152 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: oid 25 version 1156 is already newer than 1155 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1156 (ObjNum 590 snap 186 seq_num 590) dirty exists 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: left oid 25 (ObjNum 590 snap 186 seq_num 590) 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1663: done (3 left) 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1667: done (2 left) 2024-09-10T16:17:55.546 INFO:tasks.rados.rados.0.smithi165.stdout:1668: rollback oid 31 current snap is 186 2024-09-10T16:17:55.547 INFO:tasks.rados.rados.0.smithi165.stdout:1668: done (2 left) 2024-09-10T16:17:55.547 INFO:tasks.rados.rados.0.smithi165.stdout:1669: write oid 48 current snap is 186 2024-09-10T16:17:55.547 INFO:tasks.rados.rados.0.smithi165.stdout:1669: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-10T16:17:55.552 INFO:tasks.rados.rados.0.smithi165.stdout:1669: writing smithi16534841-48 from 766551 to 1264742 tid 1 2024-09-10T16:17:55.559 INFO:tasks.rados.rados.0.smithi165.stdout:1669: writing smithi16534841-48 from 1684175 to 2405032 tid 2 2024-09-10T16:17:55.566 INFO:tasks.rados.rados.0.smithi165.stdout:1669: writing smithi16534841-48 from 2851717 to 3545550 tid 3 2024-09-10T16:17:55.566 INFO:tasks.rados.rados.0.smithi165.stdout:1670: write oid 13 current snap is 186 2024-09-10T16:17:55.566 INFO:tasks.rados.rados.0.smithi165.stdout:1670: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-09-10T16:17:55.573 INFO:tasks.rados.rados.0.smithi165.stdout:1670: writing smithi16534841-13 from 783358 to 1356798 tid 1 2024-09-10T16:17:55.580 INFO:tasks.rados.rados.0.smithi165.stdout:1670: writing smithi16534841-13 from 2026304 to 2607172 tid 2 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1670: writing smithi16534841-13 from 3279140 to 3900598 tid 3 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1671: rollback oid 4 current snap is 186 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1671: done (4 left) 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1672: read oid 41 snap -1 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1672: expect (ObjNum 589 snap 186 seq_num 589) 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1673: write oid 10 current snap is 186 2024-09-10T16:17:55.588 INFO:tasks.rados.rados.0.smithi165.stdout:1673: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-10T16:17:55.594 INFO:tasks.rados.rados.0.smithi165.stdout:1673: writing smithi16534841-10 from 400165 to 1048854 tid 1 2024-09-10T16:17:55.599 INFO:tasks.rados.rados.0.smithi165.stdout:1673: writing smithi16534841-10 from 1684786 to 2000165 tid 2 2024-09-10T16:17:55.600 INFO:tasks.rados.rados.0.smithi165.stdout:1674: write oid 19 current snap is 186 2024-09-10T16:17:55.600 INFO:tasks.rados.rados.0.smithi165.stdout:1674: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-10T16:17:55.607 INFO:tasks.rados.rados.0.smithi165.stdout:1674: writing smithi16534841-19 from 416972 to 1140910 tid 1 2024-09-10T16:17:55.612 INFO:tasks.rados.rados.0.smithi165.stdout:1674: writing smithi16534841-19 from 1743268 to 2016972 tid 2 2024-09-10T16:17:55.612 INFO:tasks.rados.rados.0.smithi165.stdout:1675: write oid 2 current snap is 186 2024-09-10T16:17:55.612 INFO:tasks.rados.rados.0.smithi165.stdout:1666: expect (ObjNum 472 snap 152 seq_num 472) 2024-09-10T16:17:55.620 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1797157 2024-09-10T16:17:55.754 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3839910 2024-09-10T16:17:55.758 INFO:tasks.rados.rados.0.smithi165.stdout:1664: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: finishing write tid 1 to smithi16534841-48 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1664: finishing copy_from to smithi16534841-3 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1127 (ObjNum 538 snap 169 seq_num 538) dirty exists 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: finishing write tid 2 to smithi16534841-48 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: finishing write tid 3 to smithi16534841-48 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: finishing write tid 4 to smithi16534841-48 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: finishing write tid 5 to smithi16534841-48 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: oid 48 updating version 0 to 1210 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: oid 48 updating version 1210 to 1211 2024-09-10T16:17:55.759 INFO:tasks.rados.rados.0.smithi165.stdout:1669: oid 48 updating version 1211 to 1213 2024-09-10T16:17:55.760 INFO:tasks.rados.rados.0.smithi165.stdout:1669: oid 48 version 1213 is already newer than 1212 2024-09-10T16:17:55.760 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1213 (ObjNum 593 snap 186 seq_num 593) dirty exists 2024-09-10T16:17:55.760 INFO:tasks.rados.rados.0.smithi165.stdout:1669: left oid 48 (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:17:55.760 INFO:tasks.rados.rados.0.smithi165.stdout:1670: finishing write tid 1 to smithi16534841-13 2024-09-10T16:17:55.760 INFO:tasks.rados.rados.0.smithi165.stdout:1675: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-10T16:17:55.765 INFO:tasks.rados.rados.0.smithi165.stdout:1675: writing smithi16534841-2 from 433779 to 1232966 tid 1 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1675: writing smithi16534841-2 from 1801750 to 2033779 tid 2 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1664: done (7 left) 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1666: done (6 left) 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1669: done (5 left) 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1676: read oid 30 snap -1 2024-09-10T16:17:55.770 INFO:tasks.rados.rados.0.smithi165.stdout:1676: expect deleted 2024-09-10T16:17:55.771 INFO:tasks.rados.rados.0.smithi165.stdout:1677: snap_create 2024-09-10T16:17:55.771 INFO:tasks.rados.rados.0.smithi165.stdout:1670: finishing write tid 2 to smithi16534841-13 2024-09-10T16:17:55.771 INFO:tasks.rados.rados.0.smithi165.stdout:1672: expect (ObjNum 589 snap 186 seq_num 589) 2024-09-10T16:17:55.956 INFO:tasks.rados.rados.0.smithi165.stdout:1670: finishing write tid 3 to smithi16534841-13 2024-09-10T16:17:55.956 INFO:tasks.rados.rados.0.smithi165.stdout:1670: finishing write tid 4 to smithi16534841-13 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: finishing write tid 5 to smithi16534841-13 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: oid 13 updating version 0 to 759 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: oid 13 updating version 759 to 762 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: oid 13 version 762 is already newer than 760 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: oid 13 version 762 is already newer than 761 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 762 (ObjNum 594 snap 186 seq_num 594) dirty exists 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1670: left oid 13 (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1673: finishing write tid 1 to smithi16534841-10 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1673: finishing write tid 2 to smithi16534841-10 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1673: finishing write tid 3 to smithi16534841-10 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1673: finishing write tid 4 to smithi16534841-10 2024-09-10T16:17:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:1673: oid 10 updating version 0 to 1087 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1673: oid 10 updating version 1087 to 1088 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1673: oid 10 version 1088 is already newer than 1086 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1088 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1673: left oid 10 (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: finishing write tid 1 to smithi16534841-19 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: finishing write tid 2 to smithi16534841-19 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: finishing write tid 3 to smithi16534841-19 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: finishing write tid 4 to smithi16534841-19 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: oid 19 updating version 0 to 1215 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: oid 19 updating version 1215 to 1216 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: oid 19 version 1216 is already newer than 1214 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1216 (ObjNum 596 snap 186 seq_num 596) dirty exists 2024-09-10T16:17:55.958 INFO:tasks.rados.rados.0.smithi165.stdout:1674: left oid 19 (ObjNum 596 snap 186 seq_num 596) 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: finishing write tid 1 to smithi16534841-2 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: finishing write tid 2 to smithi16534841-2 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: finishing write tid 3 to smithi16534841-2 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: finishing write tid 4 to smithi16534841-2 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: oid 2 updating version 0 to 1138 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: oid 2 updating version 1138 to 1139 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: oid 2 updating version 1139 to 1140 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1140 (ObjNum 597 snap 186 seq_num 597) dirty exists 2024-09-10T16:17:55.959 INFO:tasks.rados.rados.0.smithi165.stdout:1675: left oid 2 (ObjNum 597 snap 186 seq_num 597) 2024-09-10T16:18:10.506 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:18:10.507 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:18:10.508 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:18:10.508 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.617+0000 7f494207f640 10 monclient: get_monmap_and_config 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.617+0000 7f494207f640 10 monclient: build_initial_monmap 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.617+0000 7f494207f640 10 monclient: monmap: 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:10.618730+0000 2024-09-10T16:18:10.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:10.618730+0000 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:10.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.618+0000 7f494207f640 1 Processor -- start 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.618+0000 7f494207f640 1 -- start start 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.618+0000 7f494207f640 10 monclient: init 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 0x7f493c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): picked mon.noname-c con 0x7f493c09c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f493c09d160 0x7f493c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): picked mon.noname-a con 0x7f493c09d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f493c09f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f493c09a430 con 0x7f493c09f4c0 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f493c0a24e0 con 0x7f493c09d160 2024-09-10T16:18:10.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f493c0a2bf0 con 0x7f493c09c180 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f494207f640 10 monclient(hunting): authenticate will time out at 1469.106299s 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f493c09d160 0x7f493c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f493c09d160 0x7f493c09ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36658/0 (socket says 172.21.15.164:36658) 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 -- 172.21.15.164:0/627529889 learned_addr learned my addr 172.21.15.164:0/627529889 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): get_auth_request con 0x7f493c09d160 auth_method 0 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 1 --2- 172.21.15.164:0/627529889 >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 0x7f493c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 10 monclient(hunting): get_auth_request con 0x7f493c09c180 auth_method 0 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient(hunting): handle_auth_done global_id 124167 payload 274 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 -- 172.21.15.164:0/627529889 >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 msgr2=0x7f493c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 --2- 172.21.15.164:0/627529889 >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 0x7f493c09ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 -- 172.21.15.164:0/627529889 >> v1:172.21.15.165:6789/0 conn(0x7f493c09f4c0 legacy=0x7f493c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient: _finish_hunting 0 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 monclient: found mon.noname-a 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493affd640 1 -- 172.21.15.164:0/627529889 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f493c09c560 con 0x7f493c09d160 2024-09-10T16:18:10.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.619+0000 7f493b7fe640 1 --2- 172.21.15.164:0/627529889 >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 0x7f493c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493affd640 1 -- 172.21.15.164:0/627529889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c09d160 msgr2=0x7f493c09ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493affd640 1 --2- 172.21.15.164:0/627529889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c09d160 0x7f493c09ef80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f492400bb10 tx=0x7f492402c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83d26a6a47495366 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 1 -- 172.21.15.164:0/627529889 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4924031920 con 0x7f493c09d160 2024-09-10T16:18:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:10.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: dump: 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 1 -- 172.21.15.164:0/627529889 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4924031ac0 con 0x7f493c09d160 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 1 -- 172.21.15.164:0/627529889 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4924033400 con 0x7f493c09d160 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f494207f640 5 monclient: authenticate success, global_id 124167 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f494207f640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f494207f640 4 set_mon_vals no callback set 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:10.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.622+0000 7f493a7fc640 10 monclient: dump: 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 10 monclient: shutdown 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:10.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 -- 172.21.15.164:0/627529889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c09d160 msgr2=0x7f493c09ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 --2- 172.21.15.164:0/627529889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c09d160 0x7f493c09ef80 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f492400bb10 tx=0x7f492402c7a0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 -- 172.21.15.164:0/627529889 shutdown_connections 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 --2- 172.21.15.164:0/627529889 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c09d160 0x7f493c09ef80 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f492400bb10 tx=0x7f492402c7a0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 --2- 172.21.15.164:0/627529889 >> v2:172.21.15.165:3300/0 conn(0x7f493c09c180 0x7f493c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 -- 172.21.15.164:0/627529889 >> 172.21.15.164:0/627529889 conn(0x7f493c0a3110 msgr2=0x7f493c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 -- 172.21.15.164:0/627529889 shutdown_connections 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 1 -- 172.21.15.164:0/627529889 wait complete. 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 10 monclient: build_initial_monmap 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.623+0000 7f494207f640 10 monclient: monmap: 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:10.635 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 Processor -- start 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 -- start start 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient: init 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f493c09c180 0x7f493c0994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): picked mon.noname-b con 0x7f493c09c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 0x7f493c099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): picked mon.noname-c con 0x7f493c09f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): picked mon.noname-a con 0x7f493c1c9120 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f493c074f90 con 0x7f493c1c9120 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f493c074e10 con 0x7f493c09c180 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f493c09d930 con 0x7f493c09f4c0 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.624+0000 7f494207f640 10 monclient(hunting): authenticate will time out at 1469.111972s 2024-09-10T16:18:10.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 0x7f493c099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36664/0 (socket says 172.21.15.164:36664) 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 -- 172.21.15.164:0/2548701946 learned_addr learned my addr 172.21.15.164:0/2548701946 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): get_auth_request con 0x7f493c1c9120 auth_method 0 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): get_auth_request con 0x7f493c09f4c0 auth_method 0 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f493c09c180 0x7f493c0994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): get_auth_request con 0x7f493c09c180 auth_method 0 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient(hunting): handle_auth_done global_id 124170 payload 824 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 -- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 msgr2=0x7f493c099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 0x7f493c099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 -- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f493c09c180 msgr2=0x7f493c0994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f493c09c180 0x7f493c0994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 monclient: found mon.noname-a 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493bfff640 1 -- 172.21.15.164:0/2548701946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f493c093050 con 0x7f493c1c9120 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.625+0000 7f493affd640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 0x7f493c099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f493bfff640 1 -- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 msgr2=0x7f493c08f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f493bfff640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f493400c410 tx=0x7f493400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49340119a0 con 0x7f493c1c9120 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: dump: 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.641 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: _finish_auth 0 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: _check_auth_tickets 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4934017070 con 0x7f493c1c9120 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 5 monclient: authenticate success, global_id 124170 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 10 monclient: _renew_subs 2024-09-10T16:18:10.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4934011d80 con 0x7f493c1c9120 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f493c09d620 con 0x7f493c1c9120 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4933fff640 10 monclient: dump: 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:10.643 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f4939ffb640 4 set_mon_vals no callback set 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 10 monclient: _renew_subs 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.627+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f493c0a42a0 con 0x7f493c1c9120 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.628+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f493401c460 con 0x7f493c1c9120 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.628+0000 7f494207f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.628+0000 7f494207f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.628+0000 7f494207f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.628+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f493c09dcd0 con 0x7f493c1c9120 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.629+0000 7f4933fff640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 0x7f491c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.629+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(403..403 src has 184..403) ==== 4815+0+0 (secure 0 0 0) 0x7f4934052af0 con 0x7f493c1c9120 2024-09-10T16:18:10.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.629+0000 7f493b7fe640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 0x7f491c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.629+0000 7f493b7fe640 10 monclient: get_auth_request con 0x7f491c03e240 auth_method 0 2024-09-10T16:18:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.631+0000 7f493b7fe640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 0x7f491c040700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f492c003b80 tx=0x7f492c009910 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.633+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f49340514d0 con 0x7f493c1c9120 2024-09-10T16:18:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.633+0000 7f4933fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:10.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.633+0000 7f4933fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:10.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f494207f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:10.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f494207f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:10.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f494207f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:10.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f493c090cf0 con 0x7f493c1c9120 2024-09-10T16:18:10.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f4933fff640 1 -- 172.21.15.164:0/2548701946 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f4934012420 con 0x7f493c1c9120 2024-09-10T16:18:10.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f4933fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:10.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.875+0000 7f4933fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:10.878 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:10.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.877+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 msgr2=0x7f491c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:10.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 0x7f491c040700 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f492c003b80 tx=0x7f492c009910 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 10 monclient: shutdown 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 msgr2=0x7f493c08f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f493400c410 tx=0x7f493400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 shutdown_connections 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f491c03e240 0x7f491c040700 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f493c1c9120 0x7f493c08f480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f493c09f4c0 0x7f493c099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 --2- 172.21.15.164:0/2548701946 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f493c09c180 0x7f493c0994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 >> 172.21.15.164:0/2548701946 conn(0x7f493c0a3110 msgr2=0x7f493c1bc500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:10.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 shutdown_connections 2024-09-10T16:18:10.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.878+0000 7f494207f640 1 -- 172.21.15.164:0/2548701946 wait complete. 2024-09-10T16:18:10.889 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:18:10.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:18:10.890 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:18:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.986+0000 7f0aea471640 10 monclient: get_monmap_and_config 2024-09-10T16:18:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.986+0000 7f0aea471640 10 monclient: build_initial_monmap 2024-09-10T16:18:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.986+0000 7f0aea471640 10 monclient: monmap: 2024-09-10T16:18:10.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:10.987117+0000 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:10.987117+0000 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:10.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.986+0000 7f0aea471640 1 Processor -- start 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 -- start start 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient: init 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 0x7f0ae409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae409d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ae409ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae4134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae409e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae40a0870 con 0x7f0ae409ab50 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0ae409d3e0 con 0x7f0ae409d930 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae4074c70 con 0x7f0ae409e250 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0aea471640 10 monclient(hunting): authenticate will time out at 1469.474602s 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 0x7f0ae409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0ae409ab50 0x7f0ae4134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34216/0 (socket says 172.21.15.164:34216) 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae37fe640 1 -- 172.21.15.164:0/986707809 learned_addr learned my addr 172.21.15.164:0/986707809 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae4134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): get_auth_request con 0x7f0ae409d930 auth_method 0 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): get_auth_request con 0x7f0ae409e250 auth_method 0 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_done global_id 114254 payload 274 2024-09-10T16:18:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/986707809 >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 msgr2=0x7f0ae409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 --2- 172.21.15.164:0/986707809 >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 0x7f0ae409dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/986707809 >> v1:172.21.15.164:6789/0 conn(0x7f0ae409ab50 legacy=0x7f0ae4134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient: _finish_hunting 0 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 monclient: found mon.noname-b 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/986707809 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae409ecb0 con 0x7f0ae409e250 2024-09-10T16:18:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.987+0000 7f0ae3fff640 1 --2- 172.21.15.164:0/986707809 >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 0x7f0ae409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.990+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/986707809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 msgr2=0x7f0ae4134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.990+0000 7f0ae8a0e640 1 --2- 172.21.15.164:0/986707809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae4134c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009d90 tx=0x7f0adc02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdb5475919ed42c0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.990+0000 7f0ae2ffd640 1 -- 172.21.15.164:0/986707809 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc007950 con 0x7f0ae409e250 2024-09-10T16:18:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: dump: 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 1 -- 172.21.15.164:0/986707809 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0adc007af0 con 0x7f0ae409e250 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 5 monclient: authenticate success, global_id 114254 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 1 -- 172.21.15.164:0/986707809 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0adc031400 con 0x7f0ae409e250 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 4 set_mon_vals no callback set 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0ae2ffd640 10 monclient: dump: 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 10 monclient: shutdown 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 -- 172.21.15.164:0/986707809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 msgr2=0x7f0ae4134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 --2- 172.21.15.164:0/986707809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae4134c90 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0adc009d90 tx=0x7f0adc02a490 comp rx=0 tx=0).stop 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 -- 172.21.15.164:0/986707809 shutdown_connections 2024-09-10T16:18:11.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 --2- 172.21.15.164:0/986707809 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae4134c90 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 --2- 172.21.15.164:0/986707809 >> v2:172.21.15.164:3300/0 conn(0x7f0ae409d930 0x7f0ae409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 -- 172.21.15.164:0/986707809 >> 172.21.15.164:0/986707809 conn(0x7f0ae40a3110 msgr2=0x7f0ae40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 -- 172.21.15.164:0/986707809 shutdown_connections 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.991+0000 7f0aea471640 1 -- 172.21.15.164:0/986707809 wait complete. 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.992+0000 7f0aea471640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.992+0000 7f0aea471640 10 monclient: monmap: 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.005 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.992+0000 7f0aea471640 1 Processor -- start 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.992+0000 7f0aea471640 1 -- start start 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.992+0000 7f0aea471640 10 monclient: init 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 0x7f0ae41cd5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae409ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae409d930 0x7f0ae41cdb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae409d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae41ce060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae409e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae409ef30 con 0x7f0ae409ab50 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae409e6e0 con 0x7f0ae409d930 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0ae40a2940 con 0x7f0ae409e250 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0aea471640 10 monclient(hunting): authenticate will time out at 1469.480214s 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 0x7f0ae41cd5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae41ce060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 0x7f0ae41cd5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36676/0 (socket says 172.21.15.164:36676) 2024-09-10T16:18:11.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 -- 172.21.15.164:0/2954394377 learned_addr learned my addr 172.21.15.164:0/2954394377 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae41ce060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52158/0 (socket says 172.21.15.164:52158) 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): get_auth_request con 0x7f0ae409ab50 auth_method 0 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): get_auth_request con 0x7f0ae409e250 auth_method 0 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae37fe640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae409d930 0x7f0ae41cdb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae37fe640 10 monclient(hunting): get_auth_request con 0x7f0ae409d930 auth_method 0 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient(hunting): handle_auth_done global_id 124176 payload 824 2024-09-10T16:18:11.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 msgr2=0x7f0ae41ce060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae41ce060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae409d930 msgr2=0x7f0ae41cdb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ae409d930 0x7f0ae41cdb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 monclient: found mon.noname-a 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae3fff640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae409d620 con 0x7f0ae409ab50 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.993+0000 7f0ae8a0e640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ae409e250 0x7f0ae41ce060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae3fff640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 msgr2=0x7f0ae41cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae3fff640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 0x7f0ae41cd5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400ebe0 tx=0x7f0ad400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4017070 con 0x7f0ae409ab50 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: dump: 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0aea471640 5 monclient: authenticate success, global_id 124176 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ad400cde0 con 0x7f0ae409ab50 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0aea471640 10 monclient: _renew_subs 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0aea471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae4099d80 con 0x7f0ae409ab50 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4013690 con 0x7f0ae409ab50 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae0ff9640 10 monclient: dump: 2024-09-10T16:18:11.012 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 10 monclient: _renew_subs 2024-09-10T16:18:11.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ae4070690 con 0x7f0ae409ab50 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.995+0000 7f0ae27fc640 4 set_mon_vals no callback set 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 0x7f0ae406ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae408e150 con 0x7f0ae4134600 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae37fe640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 0x7f0ae406ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae37fe640 10 monclient: get_auth_request con 0x7f0ae4134600 auth_method 0 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.996+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0ad401a070 con 0x7f0ae409ab50 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_done global_id 124179 payload 824 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae0ff9640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ac403e240 0x7f0ac4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae37fe640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ae409dd10 con 0x7f0ae4134600 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(403..403 src has 184..403) ==== 4815+0+0 (secure 0 0 0) 0x7f0ad4052c90 con 0x7f0ae409ab50 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae8a0e640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ac403e240 0x7f0ac4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.997+0000 7f0ae8a0e640 10 monclient: get_auth_request con 0x7f0ac403e240 auth_method 0 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae37fe640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 msgr2=0x7f0ae406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae37fe640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 0x7f0ae406ef70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00d4d0 tx=0x7f0acc00d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc018030 con 0x7f0ae4134600 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:11.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 10 monclient: dump: 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0acc014910 con 0x7f0ae4134600 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:11.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 msgr2=0x7f0ae406ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 0x7f0ae406ef70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0acc00d4d0 tx=0x7f0acc00d990 comp rx=0 tx=0).stop 2024-09-10T16:18:11.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 msgr2=0x7f0ae406ef70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:10.999+0000 7f0ae0ff9640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae4134600 0x7f0ae406ef70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.000+0000 7f0ae8a0e640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ac403e240 0x7f0ac4040700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ae406de20 tx=0x7f0adc02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0aea471640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:11.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0aea471640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 0x7f0ae402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0aea471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0ae412cc00 con 0x7f0ae406f4b0 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0ae37fe640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 0x7f0ae402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0ae37fe640 10 monclient: get_auth_request con 0x7f0ae406f4b0 auth_method 0 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0ae37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0ae37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.005+0000 7f0ae37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.006+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.006+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.006+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.006+0000 7f0ae37fe640 10 monclient(hunting): handle_auth_done global_id 124182 payload 824 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.006+0000 7f0ae37fe640 1 -- 172.21.15.164:0/2954394377 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ae406dac0 con 0x7f0ae406f4b0 2024-09-10T16:18:11.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae37fe640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 msgr2=0x7f0ae402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae37fe640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 0x7f0ae402d440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0acc018000 tx=0x7f0acc0058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0acc024070 con 0x7f0ae406f4b0 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 10 monclient: dump: 2024-09-10T16:18:11.020 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f0acc01de30 con 0x7f0ae406f4b0 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:11.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 msgr2=0x7f0ae402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.008+0000 7f0ae0ff9640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 0x7f0ae402d440 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0acc018000 tx=0x7f0acc0058d0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0ae0ff9640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 msgr2=0x7f0ae402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0ae0ff9640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae406f4b0 0x7f0ae402d440 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:11.022 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 67042, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:11.023 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:11.024 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.025 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:11.026 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.027 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:11.028 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.029 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.031 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.032 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:11.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.034 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:11.035 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:11.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ac403e240 msgr2=0x7f0ac4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ac403e240 0x7f0ac4040700 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0ae406de20 tx=0x7f0adc02f000 comp rx=0 tx=0).stop 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 10 monclient: shutdown 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 msgr2=0x7f0ae41cd5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 --2- 172.21.15.164:0/2954394377 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ae409ab50 0x7f0ae41cd5e0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ad400ebe0 tx=0x7f0ad400c520 comp rx=0 tx=0).stop 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/2954394377 reap_dead start 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0ae8a0e640 1 -- 172.21.15.164:0/2954394377 reap_dead start 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 shutdown_connections 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 >> 172.21.15.164:0/2954394377 conn(0x7f0ae40a3110 msgr2=0x7f0ae4073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 shutdown_connections 2024-09-10T16:18:11.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.009+0000 7f0aea471640 1 -- 172.21.15.164:0/2954394377 wait complete. 2024-09-10T16:18:11.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:11.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:18:11.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:18:11.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 10 monclient: get_monmap_and_config 2024-09-10T16:18:11.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 10 monclient: monmap: 2024-09-10T16:18:11.136 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:11.135946+0000 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:11.135946+0000 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:11.137 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 1 Processor -- start 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 1 -- start start 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.135+0000 7fd650616640 10 monclient: init 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd64806cb70 0x7fd648134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): picked mon.noname-a con 0x7fd64806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6480a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd6480a4ab0 0x7fd6481383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): picked mon.noname-c con 0x7fd6480a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd64809ce50 con 0x7fd6480a46a0 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd64806db60 con 0x7fd64806cb70 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd64806cf50 con 0x7fd6480a4ab0 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd650616640 10 monclient(hunting): authenticate will time out at 1469.623490s 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd64806cb70 0x7fd648134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd64806cb70 0x7fd648134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36706/0 (socket says 172.21.15.164:36706) 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1748336078 learned_addr learned my addr 172.21.15.164:0/1748336078 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request con 0x7fd64806cb70 auth_method 0 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64ebb3640 1 --2- 172.21.15.164:0/1748336078 >> v2:172.21.15.165:3300/0 conn(0x7fd6480a4ab0 0x7fd6481383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64ebb3640 10 monclient(hunting): get_auth_request con 0x7fd6480a4ab0 auth_method 0 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64ebb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64ebb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64ebb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_done global_id 124185 payload 274 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1748336078 >> v2:172.21.15.165:3300/0 conn(0x7fd6480a4ab0 msgr2=0x7fd6481383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1748336078 >> v2:172.21.15.165:3300/0 conn(0x7fd6480a4ab0 0x7fd6481383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1748336078 >> v1:172.21.15.165:6789/0 conn(0x7fd6480a46a0 legacy=0x7fd648137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 monclient: found mon.noname-a 2024-09-10T16:18:11.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:11.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.136+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1748336078 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64809cb70 con 0x7fd64806cb70 2024-09-10T16:18:11.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1748336078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd64806cb70 msgr2=0x7fd648134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1748336078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd64806cb70 0x7fd648134f70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6440098a0 tx=0x7fd64402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe67bcd31fa0a6bb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 1 -- 172.21.15.164:0/1748336078 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644037070 con 0x7fd64806cb70 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: dump: 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.151 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd650616640 5 monclient: authenticate success, global_id 124185 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 1 -- 172.21.15.164:0/1748336078 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd644032640 con 0x7fd64806cb70 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd650616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd650616640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:11.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd650616640 4 set_mon_vals no callback set 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 1 -- 172.21.15.164:0/1748336078 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644032a10 con 0x7fd64806cb70 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.139+0000 7fd64d3b0640 10 monclient: dump: 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.153 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 10 monclient: shutdown 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 -- 172.21.15.164:0/1748336078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd64806cb70 msgr2=0x7fd648134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 --2- 172.21.15.164:0/1748336078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd64806cb70 0x7fd648134f70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd6440098a0 tx=0x7fd64402a110 comp rx=0 tx=0).stop 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 -- 172.21.15.164:0/1748336078 shutdown_connections 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 --2- 172.21.15.164:0/1748336078 >> v2:172.21.15.165:3300/0 conn(0x7fd6480a4ab0 0x7fd6481383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 --2- 172.21.15.164:0/1748336078 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd64806cb70 0x7fd648134f70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 -- 172.21.15.164:0/1748336078 >> 172.21.15.164:0/1748336078 conn(0x7fd6480aa900 msgr2=0x7fd6480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 -- 172.21.15.164:0/1748336078 shutdown_connections 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.140+0000 7fd650616640 1 -- 172.21.15.164:0/1748336078 wait complete. 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient: monmap: 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 Processor -- start 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 -- start start 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient: init 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd64806cb70 0x7fd64812e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): picked mon.noname-b con 0x7fd64806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 0x7fd64812e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): picked mon.noname-c con 0x7fd6480a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 0x7fd6481cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): picked mon.noname-a con 0x7fd6480a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd648138a70 con 0x7fd6480a4ab0 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd6481388f0 con 0x7fd64806cb70 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd648140040 con 0x7fd6480a46a0 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd650616640 10 monclient(hunting): authenticate will time out at 1469.628889s 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd64dbb1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 0x7fd64812e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.141+0000 7fd64ebb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 0x7fd6481cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 0x7fd6481cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36718/0 (socket says 172.21.15.164:36718) 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 1 -- 172.21.15.164:0/1742295505 learned_addr learned my addr 172.21.15.164:0/1742295505 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): get_auth_request con 0x7fd6480a4ab0 auth_method 0 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 0x7fd64812e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52172/0 (socket says 172.21.15.164:52172) 2024-09-10T16:18:11.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): get_auth_request con 0x7fd6480a46a0 auth_method 0 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd64806cb70 0x7fd64812e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request con 0x7fd64806cb70 auth_method 0 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient(hunting): handle_auth_done global_id 114260 payload 824 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd64806cb70 msgr2=0x7fd64812e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd64806cb70 0x7fd64812e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 msgr2=0x7fd6481cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 0x7fd6481cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 monclient: found mon.noname-c 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64dbb1640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd64812ee60 con 0x7fd6480a46a0 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.142+0000 7fd64ebb3640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd6480a4ab0 0x7fd6481cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd64dbb1640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 msgr2=0x7fd64812e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd64dbb1640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 0x7fd64812e920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c440 tx=0x7fd63800c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6380119a0 con 0x7fd6480a46a0 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: dump: 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.160 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd638017070 con 0x7fd6480a46a0 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 5 monclient: authenticate success, global_id 114260 2024-09-10T16:18:11.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd638011d80 con 0x7fd6480a46a0 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 10 monclient: _renew_subs 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6480a2bc0 con 0x7fd6480a46a0 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd63effd640 10 monclient: dump: 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.162 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd64cbaf640 4 set_mon_vals no callback set 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 10 monclient: _renew_subs 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.144+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6480a3460 con 0x7fd6480a46a0 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd650616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd650616640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 0x7fd6481cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd6481cdf20 con 0x7fd6481cd600 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd63801c460 con 0x7fd6480a46a0 2024-09-10T16:18:11.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 0x7fd6481cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient: get_auth_request con 0x7fd6481cd600 auth_method 0 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd63effd640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd62c03e210 0x7fd62c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_done global_id 114263 payload 824 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(403..403 src has 184..403) ==== 4815+0+0 (secure 0 0 0) 0x7fd638052ad0 con 0x7fd6480a46a0 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.145+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd648134470 con 0x7fd6481cd600 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.146+0000 7fd64ebb3640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd62c03e210 0x7fd62c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.146+0000 7fd64ebb3640 10 monclient: get_auth_request con 0x7fd62c03e210 auth_method 0 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 msgr2=0x7fd6481cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 0x7fd6481cd9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd648077610 tx=0x7fd6440046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644032e60 con 0x7fd6481cd600 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 10 monclient: dump: 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.165 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd644004220 con 0x7fd6481cd600 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 msgr2=0x7fd6481cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 0x7fd6481cd9e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd648077610 tx=0x7fd6440046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 msgr2=0x7fd6481cd9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd63effd640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6481cd600 0x7fd6481cd9e0 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd648077610 tx=0x7fd6440046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.148+0000 7fd64ebb3640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd62c03e210 0x7fd62c0406d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6481325f0 tx=0x7fd6400098d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd650616640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd650616640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 0x7fd648130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd650616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd648134f70 con 0x7fd648130250 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 0x7fd648130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient: get_auth_request con 0x7fd648130250 auth_method 0 2024-09-10T16:18:11.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 10 monclient(hunting): handle_auth_done global_id 114266 payload 824 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.154+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1742295505 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd6480770f0 con 0x7fd648130250 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd64e3b2640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 msgr2=0x7fd648130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd64e3b2640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 0x7fd648130660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd644000c00 tx=0x7fd644008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd644037040 con 0x7fd648130250 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 10 monclient: dump: 2024-09-10T16:18:11.169 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fd644003610 con 0x7fd648130250 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:11.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 msgr2=0x7fd648130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 0x7fd648130660 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd644000c00 tx=0x7fd644008030 comp rx=0 tx=0).stop 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 msgr2=0x7fd648130660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd63effd640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd648130250 0x7fd648130660 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd644000c00 tx=0x7fd644008030 comp rx=0 tx=0).stop 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 111726, 2024-09-10T16:18:11.171 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.172 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:11.173 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.174 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.175 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.176 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:11.177 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.178 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.179 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.180 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.182 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.183 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.184 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.185 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.185 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.185 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:11.185 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd62c03e210 msgr2=0x7fd62c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd62c03e210 0x7fd62c0406d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fd6481325f0 tx=0x7fd6400098d0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 10 monclient: shutdown 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 msgr2=0x7fd64812e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 --2- 172.21.15.164:0/1742295505 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6480a46a0 0x7fd64812e920 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd63800c440 tx=0x7fd63800c900 comp rx=0 tx=0).stop 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd64ebb3640 1 -- 172.21.15.164:0/1742295505 reap_dead start 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd64ebb3640 1 -- 172.21.15.164:0/1742295505 reap_dead start 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 shutdown_connections 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 >> 172.21.15.164:0/1742295505 conn(0x7fd6480aa900 msgr2=0x7fd6480a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 shutdown_connections 2024-09-10T16:18:11.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.157+0000 7fd650616640 1 -- 172.21.15.164:0/1742295505 wait complete. 2024-09-10T16:18:11.187 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:11.188 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:18:11.188 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:18:11.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.285+0000 7f154015d640 10 monclient: get_monmap_and_config 2024-09-10T16:18:11.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.285+0000 7f154015d640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.285+0000 7f154015d640 10 monclient: monmap: 2024-09-10T16:18:11.287 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:11.286400+0000 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:11.286400+0000 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:11.288 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:11.289 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 Processor -- start 2024-09-10T16:18:11.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 -- start start 2024-09-10T16:18:11.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient: init 2024-09-10T16:18:11.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f153809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f153809d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 0x7f153813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): picked mon.noname-b con 0x7f153813b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f153809fbc0 con 0x7f153809d190 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15380a2a20 con 0x7f153809c310 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f153809f860 con 0x7f153813b5d0 2024-09-10T16:18:11.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f154015d640 10 monclient(hunting): authenticate will time out at 1469.773794s 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f153d6f8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f153809d190 0x7f153809f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34222/0 (socket says 172.21.15.164:34222) 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f153d6f8640 1 -- 172.21.15.164:0/2804820571 learned_addr learned my addr 172.21.15.164:0/2804820571 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f153e6fa640 1 --2- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 0x7f153813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f153e6fa640 10 monclient(hunting): get_auth_request con 0x7f153813b5d0 auth_method 0 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.286+0000 7f153e6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient(hunting): handle_auth_done global_id 114269 payload 274 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 1 -- 172.21.15.164:0/2804820571 >> v1:172.21.15.165:6789/0 conn(0x7f153809c310 legacy=0x7f153809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 1 -- 172.21.15.164:0/2804820571 >> v1:172.21.15.164:6789/0 conn(0x7f153809d190 legacy=0x7f153809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 1 monclient: found mon.noname-b 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.287+0000 7f153e6fa640 1 -- 172.21.15.164:0/2804820571 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f153809c6f0 con 0x7f153813b5d0 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.289+0000 7f153e6fa640 1 -- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 msgr2=0x7f153813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.289+0000 7f153e6fa640 1 --2- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 0x7f153813d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1530009dc0 tx=0x7f153002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16150f91290e4cdd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 1 -- 172.21.15.164:0/2804820571 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530007a30 con 0x7f153813b5d0 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: dump: 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.305 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 5 monclient: authenticate success, global_id 114269 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 1 -- 172.21.15.164:0/2804820571 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530007bd0 con 0x7f153813b5d0 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 4 set_mon_vals no callback set 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 1 -- 172.21.15.164:0/2804820571 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530031400 con 0x7f153813b5d0 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f153cef7640 10 monclient: dump: 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.307 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 10 monclient: shutdown 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 -- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 msgr2=0x7f153813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 --2- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 0x7f153813d9c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1530009dc0 tx=0x7f153002a480 comp rx=0 tx=0).stop 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 -- 172.21.15.164:0/2804820571 shutdown_connections 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 --2- 172.21.15.164:0/2804820571 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f153813b5d0 0x7f153813d9c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 -- 172.21.15.164:0/2804820571 >> 172.21.15.164:0/2804820571 conn(0x7f15380a3110 msgr2=0x7f15380a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 -- 172.21.15.164:0/2804820571 shutdown_connections 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.290+0000 7f154015d640 1 -- 172.21.15.164:0/2804820571 wait complete. 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.291+0000 7f154015d640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.291+0000 7f154015d640 10 monclient: monmap: 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.309 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.291+0000 7f154015d640 1 Processor -- start 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.291+0000 7f154015d640 1 -- start start 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.291+0000 7f154015d640 10 monclient: init 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809c310 0x7f1538073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): picked mon.noname-b con 0x7f153809c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f153813b5d0 0x7f15381ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): picked mon.noname-a con 0x7f153813b5d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): picked mon.noname-c con 0x7f1538099770 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15380a20b0 con 0x7f153813b5d0 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f153809a550 con 0x7f153809c310 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15380a2230 con 0x7f1538099770 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f154015d640 10 monclient(hunting): authenticate will time out at 1469.779184s 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f153813b5d0 0x7f15381ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52200/0 (socket says 172.21.15.164:52200) 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): get_auth_request con 0x7f153813b5d0 auth_method 0 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 learned_addr learned my addr 172.21.15.164:0/2587437533 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): get_auth_request con 0x7f1538099770 auth_method 0 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153def9640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809c310 0x7f1538073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153def9640 10 monclient(hunting): get_auth_request con 0x7f153809c310 auth_method 0 2024-09-10T16:18:11.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153def9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153def9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153def9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient(hunting): handle_auth_done global_id 114272 payload 824 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809c310 msgr2=0x7f1538073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809c310 0x7f1538073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f153813b5d0 msgr2=0x7f15381ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f153813b5d0 0x7f15381ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 monclient: found mon.noname-c 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15380943a0 con 0x7f1538099770 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.292+0000 7f153d6f8640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f153813b5d0 0x7f15381ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 msgr2=0x7f1538099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f153e6fa640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1530009ee0 tx=0x7f1530004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530040070 con 0x7f1538099770 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 10 monclient: dump: 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1530045460 con 0x7f1538099770 2024-09-10T16:18:11.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f154015d640 5 monclient: authenticate success, global_id 114272 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f154015d640 10 monclient: _renew_subs 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f154015d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.294+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15381cd2b0 con 0x7f1538099770 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530007a30 con 0x7f1538099770 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 10 monclient: dump: 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152ffff640 4 set_mon_vals no callback set 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 10 monclient: _renew_subs 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1538094d50 con 0x7f1538099770 2024-09-10T16:18:11.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 0x7f1538092b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f153809d930 con 0x7f153809d190 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.295+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1530039040 con 0x7f1538099770 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 0x7f1538092b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient: get_auth_request con 0x7f153809d190 auth_method 0 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f152e7fc640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f151c03e240 0x7f151c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(403..403 src has 184..403) ==== 4815+0+0 (secure 0 0 0) 0x7f15300747f0 con 0x7f1538099770 2024-09-10T16:18:11.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153d6f8640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f151c03e240 0x7f151c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153d6f8640 10 monclient: get_auth_request con 0x7f151c03e240 auth_method 0 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 10 monclient(hunting): handle_auth_done global_id 134116 payload 824 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.296+0000 7f153def9640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f15380008d0 con 0x7f153809d190 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.298+0000 7f153def9640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 msgr2=0x7f1538092b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.298+0000 7f153def9640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 0x7f1538092b50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f153400e990 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.298+0000 7f153d6f8640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f151c03e240 0x7f151c040700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1528007c90 tx=0x7f1528007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1534007450 con 0x7f153809d190 2024-09-10T16:18:11.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 10 monclient: dump: 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.321 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1534007450 con 0x7f153809d190 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 msgr2=0x7f1538092b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 0x7f1538092b50 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f153400e990 comp rx=0 tx=0).stop 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 msgr2=0x7f1538092b50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.299+0000 7f152e7fc640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153809d190 0x7f1538092b50 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f153400e990 comp rx=0 tx=0).stop 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f154015d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f154015d640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 0x7f153802d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f154015d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f15380931e0 con 0x7f153802d440 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 0x7f153802d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient: get_auth_request con 0x7f153802d440 auth_method 0 2024-09-10T16:18:11.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.305+0000 7f153def9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.306+0000 7f153def9640 10 monclient(hunting): handle_auth_done global_id 134119 payload 824 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.306+0000 7f153def9640 1 -- 172.21.15.164:0/2587437533 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1538070ad0 con 0x7f153802d440 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f153def9640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 msgr2=0x7f153802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f153def9640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 0x7f153802d850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f1534003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153401eb70 con 0x7f153802d440 2024-09-10T16:18:11.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 10 monclient: dump: 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.326 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f1534028430 con 0x7f153802d440 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 msgr2=0x7f153802d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 0x7f153802d850 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f1534003760 comp rx=0 tx=0).stop 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 msgr2=0x7f153802d850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.308+0000 7f152e7fc640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f153802d440 0x7f153802d850 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f153400e4d0 tx=0x7f1534003760 comp rx=0 tx=0).stop 2024-09-10T16:18:11.327 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 15712, 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.328 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.329 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.330 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:11.331 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:11.332 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:11.333 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.334 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.335 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.336 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.338 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.339 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.340 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:11.341 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:11.342 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:11.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f151c03e240 msgr2=0x7f151c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f151c03e240 0x7f151c040700 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1528007c90 tx=0x7f1528007be0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 10 monclient: shutdown 2024-09-10T16:18:11.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 msgr2=0x7f1538099c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f153e6fa640 1 -- 172.21.15.164:0/2587437533 reap_dead start 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1530009ee0 tx=0x7f1530004a40 comp rx=0 tx=0).stop 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 shutdown_connections 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 --2- 172.21.15.164:0/2587437533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1538099770 0x7f1538099c00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 >> 172.21.15.164:0/2587437533 conn(0x7f15380a3110 msgr2=0x7f153813a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 shutdown_connections 2024-09-10T16:18:11.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.309+0000 7f154015d640 1 -- 172.21.15.164:0/2587437533 wait complete. 2024-09-10T16:18:11.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:11.345 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:18:11.345 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:18:11.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.441+0000 7f3f7bc23640 10 monclient: get_monmap_and_config 2024-09-10T16:18:11.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.441+0000 7f3f7bc23640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.441+0000 7f3f7bc23640 10 monclient: monmap: 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:11.442150+0000 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:11.442150+0000 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:11.443 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:11.444 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:11.444 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.441+0000 7f3f7bc23640 1 Processor -- start 2024-09-10T16:18:11.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 -- start start 2024-09-10T16:18:11.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient: init 2024-09-10T16:18:11.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f7406cb70 0x7f3f74134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-a con 0x7f3f7406cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-b con 0x7f3f740a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3f74137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f7409ce50 con 0x7f3f74137ca0 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3f7406db60 con 0x7f3f7406cb70 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f7406cf50 con 0x7f3f740a46a0 2024-09-10T16:18:11.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f7bc23640 10 monclient(hunting): authenticate will time out at 1469.929586s 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f7406cb70 0x7f3f74134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f7406cb70 0x7f3f74134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36736/0 (socket says 172.21.15.164:36736) 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52214/0 (socket says 172.21.15.164:52214) 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 -- 172.21.15.164:0/1086432380 learned_addr learned my addr 172.21.15.164:0/1086432380 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): get_auth_request con 0x7f3f7406cb70 auth_method 0 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): get_auth_request con 0x7f3f740a46a0 auth_method 0 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f791be640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient(hunting): handle_auth_done global_id 124194 payload 274 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 -- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 msgr2=0x7f3f740a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 -- 172.21.15.164:0/1086432380 >> v1:172.21.15.165:6789/0 conn(0x7f3f74137ca0 legacy=0x7f3f74138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.442+0000 7f3f799bf640 1 monclient: found mon.noname-a 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.443+0000 7f3f799bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.443+0000 7f3f799bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.443+0000 7f3f799bf640 1 -- 172.21.15.164:0/1086432380 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f7409cb70 con 0x7f3f7406cb70 2024-09-10T16:18:11.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.443+0000 7f3f791be640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f799bf640 1 -- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f7406cb70 msgr2=0x7f3f74134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f799bf640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f7406cb70 0x7f3f74134f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c43643a87c743d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 1 -- 172.21.15.164:0/1086432380 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c037070 con 0x7f3f7406cb70 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: dump: 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 1 -- 172.21.15.164:0/1086432380 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f6c032440 con 0x7f3f7406cb70 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f7bc23640 5 monclient: authenticate success, global_id 124194 2024-09-10T16:18:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f7bc23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f7bc23640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 1 -- 172.21.15.164:0/1086432380 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c032810 con 0x7f3f7406cb70 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f7bc23640 4 set_mon_vals no callback set 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.445+0000 7f3f789bd640 10 monclient: dump: 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.459 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 10 monclient: shutdown 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 -- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f7406cb70 msgr2=0x7f3f74134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f7406cb70 0x7f3f74134f70 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 -- 172.21.15.164:0/1086432380 shutdown_connections 2024-09-10T16:18:11.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f740a46a0 0x7f3f740a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/1086432380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f7406cb70 0x7f3f74134f70 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 -- 172.21.15.164:0/1086432380 >> 172.21.15.164:0/1086432380 conn(0x7f3f740aa900 msgr2=0x7f3f740aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 -- 172.21.15.164:0/1086432380 shutdown_connections 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.446+0000 7f3f7bc23640 1 -- 172.21.15.164:0/1086432380 wait complete. 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient: build_initial_monmap 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient: monmap: 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:11.461 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 Processor -- start 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 -- start start 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient: init 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-c con 0x7f3f7406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 0x7f3f7412e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-b con 0x7f3f740a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 0x7f3f741cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): picked mon.noname-a con 0x7f3f74137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741389c0 con 0x7f3f74137ca0 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74138840 con 0x7f3f740a46a0 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f74140040 con 0x7f3f7406cb70 2024-09-10T16:18:11.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.447+0000 7f3f7bc23640 10 monclient(hunting): authenticate will time out at 1469.934949s 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 0x7f3f741cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52218/0 (socket says 172.21.15.164:52218) 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 -- 172.21.15.164:0/2574697483 learned_addr learned my addr 172.21.15.164:0/2574697483 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): get_auth_request con 0x7f3f7406cb70 auth_method 0 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): get_auth_request con 0x7f3f74137ca0 auth_method 0 2024-09-10T16:18:11.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 0x7f3f7412e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 10 monclient(hunting): get_auth_request con 0x7f3f740a46a0 auth_method 0 2024-09-10T16:18:11.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient(hunting): handle_auth_done global_id 114278 payload 824 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 -- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 msgr2=0x7f3f7412e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 0x7f3f7412e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 -- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 msgr2=0x7f3f741cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 0x7f3f741cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 monclient: found mon.noname-c 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f799bf640 1 -- 172.21.15.164:0/2574697483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f740a3270 con 0x7f3f7406cb70 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f7a1c0640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 0x7f3f741cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.448+0000 7f3f791be640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 0x7f3f7412e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f799bf640 1 -- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 msgr2=0x7f3f7412e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f799bf640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c041020 con 0x7f3f7406cb70 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: dump: 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f6c03d630 con 0x7f3f7406cb70 2024-09-10T16:18:11.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 5 monclient: authenticate success, global_id 114278 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 10 monclient: _renew_subs 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f740a3550 con 0x7f3f7406cb70 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6c037040 con 0x7f3f7406cb70 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f727fc640 10 monclient: dump: 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 10 monclient: _renew_subs 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f73fff640 4 set_mon_vals no callback set 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.450+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f7406e720 con 0x7f3f7406cb70 2024-09-10T16:18:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3f6c044020 con 0x7f3f7406cb70 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f7bc23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f7bc23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f7bc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f74134f70 con 0x7f3f7406cb70 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f727fc640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 0x7f3f60040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(403..403 src has 184..403) ==== 4815+0+0 (secure 0 0 0) 0x7f3f6c072da0 con 0x7f3f7406cb70 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.451+0000 7f3f791be640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 0x7f3f60040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.452+0000 7f3f791be640 10 monclient: get_auth_request con 0x7f3f6003e240 auth_method 0 2024-09-10T16:18:11.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.454+0000 7f3f791be640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 0x7f3f60040700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3f740a23e0 tx=0x7f3f68008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.456+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3f6c04b070 con 0x7f3f7406cb70 2024-09-10T16:18:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.456+0000 7f3f727fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.456+0000 7f3f727fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:11.539 INFO:tasks.rados.rados.0.smithi165.stdout:1670: done (6 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1672: done (5 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1673: done (4 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1674: done (3 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1675: done (2 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1676: done (1 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1677: done (0 left) 2024-09-10T16:18:11.540 INFO:tasks.rados.rados.0.smithi165.stdout:1678: snap_remove snap 186 2024-09-10T16:18:11.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.701+0000 7f3f7bc23640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:18:11.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.701+0000 7f3f7bc23640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:18:11.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.701+0000 7f3f7bc23640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:11.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.701+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f3f7412edf0 con 0x7f3f7406cb70 2024-09-10T16:18:11.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:11.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=1411212798,config=2586702516,health=2974672015,kv=805391194,logm=2282884461}) 2024-09-10T16:18:11.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=3924261409,config=2586702516,health=2974672015,kv=1115812742,logm=3719703497}) 2024-09-10T16:18:11.705 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:11.705 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=1411212798,config=2586702516,health=2974672015,kv=805391194,logm=2282884461}) 2024-09-10T16:18:11.705 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=11,kv=12,logm=53} crc {auth=824271267,config=2586702517,health=1866747259,kv=2034054781,logm=626406677}) 2024-09-10T16:18:11.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f3f727fc640 1 -- 172.21.15.164:0/2574697483 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f3f7409ce50 con 0x7f3f7406cb70 2024-09-10T16:18:11.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f3f727fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:18:11.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.703+0000 7f3f727fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 msgr2=0x7f3f60040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 0x7f3f60040700 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f3f740a23e0 tx=0x7f3f68008040 comp rx=0 tx=0).stop 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 10 monclient: shutdown 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 msgr2=0x7f3f7412e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:11.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3f6c009870 tx=0x7f3f6c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 shutdown_connections 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f6003e240 0x7f3f60040700 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f74137ca0 0x7f3f741cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f740a46a0 0x7f3f7412e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 --2- 172.21.15.164:0/2574697483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f7406cb70 0x7f3f7412e370 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 >> 172.21.15.164:0/2574697483 conn(0x7f3f740aa900 msgr2=0x7f3f740a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 shutdown_connections 2024-09-10T16:18:11.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:11.705+0000 7f3f7bc23640 1 -- 172.21.15.164:0/2574697483 wait complete. 2024-09-10T16:18:11.717 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634057204}) 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3542179084}) 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1634057204}) 2024-09-10T16:18:12.512 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.510+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880235920}) 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378815810}) 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=975350958}) 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1378815810}) 2024-09-10T16:18:12.537 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.534+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=283254281}) 2024-09-10T16:18:12.540 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.540 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193748445}) 2024-09-10T16:18:12.540 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2820814113}) 2024-09-10T16:18:12.540 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.541 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2193748445}) 2024-09-10T16:18:12.541 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.538+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268176506}) 2024-09-10T16:18:12.542 INFO:tasks.rados.rados.0.smithi165.stdout:1678: done (0 left) 2024-09-10T16:18:12.543 INFO:tasks.rados.rados.0.smithi165.stdout:1679: snap_create 2024-09-10T16:18:12.549 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.549 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3842017223}) 2024-09-10T16:18:12.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571476332}) 2024-09-10T16:18:12.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:12.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3842017223}) 2024-09-10T16:18:12.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:12.548+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3878559842}) 2024-09-10T16:18:12.718 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:18:12.718 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:12.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.825+0000 7f64d7233640 10 monclient: get_monmap_and_config 2024-09-10T16:18:12.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.825+0000 7f64d7233640 10 monclient: build_initial_monmap 2024-09-10T16:18:12.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.825+0000 7f64d7233640 10 monclient: monmap: 2024-09-10T16:18:12.826 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:12.826113+0000 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:12.826113+0000 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:12.827 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.825+0000 7f64d7233640 1 Processor -- start 2024-09-10T16:18:12.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 1 -- start start 2024-09-10T16:18:12.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient: init 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-b con 0x7f64d006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64d00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64d00a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64d009ce50 con 0x7f64d00a46a0 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64d0138ac0 con 0x7f64d00a4a80 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f64d006db60 con 0x7f64d006cb70 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d7233640 10 monclient(hunting): authenticate will time out at 1471.313562s 2024-09-10T16:18:12.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64cffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f64d00a46a0 0x7f64d0137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34234/0 (socket says 172.21.15.164:34234) 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64cffff640 1 -- 172.21.15.164:0/1049319990 learned_addr learned my addr 172.21.15.164:0/1049319990 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): get_auth_request con 0x7f64d006cb70 auth_method 0 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_done global_id 114281 payload 274 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1049319990 >> v1:172.21.15.165:6789/0 conn(0x7f64d00a4a80 legacy=0x7f64d01383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:12.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1049319990 >> v1:172.21.15.164:6789/0 conn(0x7f64d00a46a0 legacy=0x7f64d0137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 1 monclient: found mon.noname-b 2024-09-10T16:18:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:12.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.826+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1049319990 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d009cb70 con 0x7f64d006cb70 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1049319990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 msgr2=0x7f64d0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d4fcf640 1 --2- 172.21.15.164:0/1049319990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d0134f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c4029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3364d8e676255e25 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 1 -- 172.21.15.164:0/1049319990 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4037070 con 0x7f64d006cb70 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:12.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: dump: 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:18:12.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d7233640 5 monclient: authenticate success, global_id 114281 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d7233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 1 -- 172.21.15.164:0/1049319990 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c4032440 con 0x7f64d006cb70 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d7233640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 1 -- 172.21.15.164:0/1049319990 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4032810 con 0x7f64d006cb70 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64d7233640 4 set_mon_vals no callback set 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.829+0000 7f64cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64cf7fe640 10 monclient: dump: 2024-09-10T16:18:12.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 10 monclient: shutdown 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:12.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 -- 172.21.15.164:0/1049319990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 msgr2=0x7f64d0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 --2- 172.21.15.164:0/1049319990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d0134f70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c4029fb0 comp rx=0 tx=0).stop 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 -- 172.21.15.164:0/1049319990 shutdown_connections 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 --2- 172.21.15.164:0/1049319990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d0134f70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 -- 172.21.15.164:0/1049319990 >> 172.21.15.164:0/1049319990 conn(0x7f64d00aa900 msgr2=0x7f64d00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 -- 172.21.15.164:0/1049319990 shutdown_connections 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.830+0000 7f64d7233640 1 -- 172.21.15.164:0/1049319990 wait complete. 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.831+0000 7f64d7233640 10 monclient: build_initial_monmap 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.831+0000 7f64d7233640 10 monclient: monmap: 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:12.842 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.831+0000 7f64d7233640 1 Processor -- start 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.831+0000 7f64d7233640 1 -- start start 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.831+0000 7f64d7233640 10 monclient: init 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:18:12.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-c con 0x7f64d006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64d00a46a0 0x7f64d013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-a con 0x7f64d00a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 0x7f64d0140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): picked mon.noname-b con 0x7f64d00a4a80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f64d006cf50 con 0x7f64d00a46a0 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f64d0141050 con 0x7f64d00a4a80 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f64d01411d0 con 0x7f64d006cb70 2024-09-10T16:18:12.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d7233640 10 monclient(hunting): authenticate will time out at 1471.319225s 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64d00a46a0 0x7f64d013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52230/0 (socket says 172.21.15.164:52230) 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1279915530 learned_addr learned my addr 172.21.15.164:0/1279915530 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): get_auth_request con 0x7f64d006cb70 auth_method 0 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): get_auth_request con 0x7f64d00a46a0 auth_method 0 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:12.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 0x7f64d0140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 10 monclient(hunting): get_auth_request con 0x7f64d00a4a80 auth_method 0 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:12.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient(hunting): handle_auth_done global_id 114284 payload 824 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 msgr2=0x7f64d0140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 0x7f64d0140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64d00a46a0 msgr2=0x7f64d013ffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64d00a46a0 0x7f64d013ffe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 monclient: found mon.noname-c 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1279915530 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64d00a4000 con 0x7f64d006cb70 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.832+0000 7f64d57d0640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 0x7f64d0140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.834+0000 7f64d4fcf640 1 -- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 msgr2=0x7f64d01cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.834+0000 7f64d4fcf640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:12.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4041070 con 0x7f64d006cb70 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: dump: 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:12.848 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 5 monclient: authenticate success, global_id 114284 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64c403c660 con 0x7f64d006cb70 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64c4037040 con 0x7f64d006cb70 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 10 monclient: _renew_subs 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64d006e720 con 0x7f64d006cb70 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:12.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64ceffd640 4 set_mon_vals no callback set 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64cd7fa640 10 monclient: dump: 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:12.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 10 monclient: _renew_subs 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.835+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f64d013aa70 con 0x7f64d006cb70 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f64c4044070 con 0x7f64d006cb70 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64d7233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64d7233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64d7233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64d0134f70 con 0x7f64d006cb70 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64cd7fa640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 0x7f64b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f64c40733c0 con 0x7f64d006cb70 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64cffff640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 0x7f64b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:12.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.836+0000 7f64cffff640 10 monclient: get_auth_request con 0x7f64b403e210 auth_method 0 2024-09-10T16:18:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.839+0000 7f64cffff640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 0x7f64b40406d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f64bc007c90 tx=0x7f64bc007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.840+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f64d009ce50 con 0x7f64d006cb70 2024-09-10T16:18:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.841+0000 7f64cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:12.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:12.841+0000 7f64cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340513959}) 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1851663701}) 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2340513959}) 2024-09-10T16:18:13.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.027+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2416244311}) 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021440276}) 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3480685130}) 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021440276}) 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.028+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3442732301}) 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788555501}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=429865364}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788555501}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=318441615}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088545988}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3253676141}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2088545988}) 2024-09-10T16:18:13.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.029+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2784241778}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941140496}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456331105}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1941140496}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.030+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4059248166}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552670033}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=809502732}) 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552670033}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.031+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2488571043}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695364354}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1320148643}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695364354}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.032+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719028951}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640529441}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291490016}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3640529441}) 2024-09-10T16:18:13.036 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556865639}) 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490872852}) 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2032184194}) 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3490872852}) 2024-09-10T16:18:13.037 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.033+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984561709}) 2024-09-10T16:18:13.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64d7233640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64d7233640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64d7233640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64d00a89a0 con 0x7f64d006cb70 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64cd7fa640 1 -- 172.21.15.164:0/1279915530 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f64c4071da0 con 0x7f64d006cb70 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.082+0000 7f64cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:13.084 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 msgr2=0x7f64b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 0x7f64b40406d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f64bc007c90 tx=0x7f64bc007be0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 10 monclient: shutdown 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 msgr2=0x7f64d01cd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f64c40097d0 tx=0x7f64c40046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 shutdown_connections 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f64b403e210 0x7f64b40406d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f64d00a4a80 0x7f64d0140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f64d00a46a0 0x7f64d013ffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 --2- 172.21.15.164:0/1279915530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f64d006cb70 0x7f64d01cd2b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 >> 172.21.15.164:0/1279915530 conn(0x7f64d00aa900 msgr2=0x7f64d0137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 shutdown_connections 2024-09-10T16:18:13.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.084+0000 7f64d7233640 1 -- 172.21.15.164:0/1279915530 wait complete. 2024-09-10T16:18:13.096 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:18:13.096 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:18:13.097 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:18:13.097 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.193+0000 7f9c2ba23640 10 monclient: get_monmap_and_config 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.193+0000 7f9c2ba23640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.193+0000 7f9c2ba23640 10 monclient: monmap: 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:13.194643+0000 2024-09-10T16:18:13.195 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:13.194643+0000 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:13.196 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.194+0000 7f9c2ba23640 1 Processor -- start 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.194+0000 7f9c2ba23640 1 -- start start 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.194+0000 7f9c2ba23640 10 monclient: init 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9c241348c0 0x7f9c24073780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-a con 0x7f9c241348c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9c24134ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9c24135130 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c241389c0 con 0x7f9c24135130 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2409ccd0 con 0x7f9c24134ca0 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9c2409cfb0 con 0x7f9c241348c0 2024-09-10T16:18:13.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c2ba23640 10 monclient(hunting): authenticate will time out at 1471.682173s 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9c241348c0 0x7f9c24073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c29fc0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9c24135130 0x7f9c241382b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34238/0 (socket says 172.21.15.164:34238) 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c29fc0640 1 -- 172.21.15.164:0/3264332324 learned_addr learned my addr 172.21.15.164:0/3264332324 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): get_auth_request con 0x7f9c241348c0 auth_method 0 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient(hunting): handle_auth_done global_id 124203 payload 274 2024-09-10T16:18:13.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 1 -- 172.21.15.164:0/3264332324 >> v1:172.21.15.165:6789/0 conn(0x7f9c24134ca0 legacy=0x7f9c24137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 1 -- 172.21.15.164:0/3264332324 >> v1:172.21.15.164:6789/0 conn(0x7f9c24135130 legacy=0x7f9c241382b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 1 monclient: found mon.noname-a 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.195+0000 7f9c297bf640 1 -- 172.21.15.164:0/3264332324 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c24134620 con 0x7f9c241348c0 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c297bf640 1 -- 172.21.15.164:0/3264332324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 msgr2=0x7f9c24073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c297bf640 1 --2- 172.21.15.164:0/3264332324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c24073780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097a0 tx=0x7f9c20029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39734d9a9c0b7e09 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 1 -- 172.21.15.164:0/3264332324 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20037070 con 0x7f9c241348c0 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: dump: 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.207 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 1 -- 172.21.15.164:0/3264332324 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c20032640 con 0x7f9c241348c0 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c2ba23640 5 monclient: authenticate success, global_id 124203 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c2ba23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:13.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c2ba23640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 1 -- 172.21.15.164:0/3264332324 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20032a10 con 0x7f9c241348c0 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c2ba23640 4 set_mon_vals no callback set 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.198+0000 7f9c1bfff640 10 monclient: dump: 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.209 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 10 monclient: shutdown 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 -- 172.21.15.164:0/3264332324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 msgr2=0x7f9c24073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/3264332324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c24073780 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097a0 tx=0x7f9c20029fd0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 -- 172.21.15.164:0/3264332324 shutdown_connections 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/3264332324 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c24073780 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 -- 172.21.15.164:0/3264332324 >> 172.21.15.164:0/3264332324 conn(0x7f9c240a6f80 msgr2=0x7f9c240a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 -- 172.21.15.164:0/3264332324 shutdown_connections 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 1 -- 172.21.15.164:0/3264332324 wait complete. 2024-09-10T16:18:13.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.199+0000 7f9c2ba23640 10 monclient: monmap: 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 Processor -- start 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 -- start start 2024-09-10T16:18:13.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient: init 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c241317a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-a con 0x7f9c241348c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c24134ca0 0x7f9c2413aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-c con 0x7f9c24134ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c24135130 0x7f9c2413b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): picked mon.noname-b con 0x7f9c24135130 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9c240731c0 con 0x7f9c241348c0 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9c24073040 con 0x7f9c24135130 2024-09-10T16:18:13.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9c24140f50 con 0x7f9c24134ca0 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c2ba23640 10 monclient(hunting): authenticate will time out at 1471.687760s 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c241317a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c241317a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36782/0 (socket says 172.21.15.164:36782) 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 1 -- 172.21.15.164:0/560960390 learned_addr learned my addr 172.21.15.164:0/560960390 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c28fbe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c24134ca0 0x7f9c2413aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): get_auth_request con 0x7f9c241348c0 auth_method 0 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c28fbe640 10 monclient(hunting): get_auth_request con 0x7f9c24134ca0 auth_method 0 2024-09-10T16:18:13.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c28fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c28fbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c28fbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c297bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.200+0000 7f9c29fc0640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c24135130 0x7f9c2413b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): get_auth_request con 0x7f9c24135130 auth_method 0 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c29fc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 10 monclient(hunting): handle_auth_done global_id 124206 payload 824 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c24134ca0 msgr2=0x7f9c2413aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9c24134ca0 0x7f9c2413aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c24135130 msgr2=0x7f9c2413b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9c24135130 0x7f9c2413b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 monclient: found mon.noname-a 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.201+0000 7f9c297bf640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c2413c660 con 0x7f9c241348c0 2024-09-10T16:18:13.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c297bf640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 msgr2=0x7f9c241317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c297bf640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c241317a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097a0 tx=0x7f9c200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20041070 con 0x7f9c241348c0 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: dump: 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 5 monclient: authenticate success, global_id 124206 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c2003c880 con 0x7f9c241348c0 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c20037040 con 0x7f9c241348c0 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 10 monclient: _renew_subs 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c2413e120 con 0x7f9c241348c0 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c1b7fe640 4 set_mon_vals no callback set 2024-09-10T16:18:13.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c19ffb640 10 monclient: dump: 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 10 monclient: _renew_subs 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.203+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c2413d1c0 con 0x7f9c241348c0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c2ba23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 0x7f9c2402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2413d420 con 0x7f9c24131ce0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9c20044070 con 0x7f9c241348c0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 0x7f9c2402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient: get_auth_request con 0x7f9c24131ce0 auth_method 0 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.204+0000 7f9c19ffb640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9c0803e260 0x7f9c08040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.205+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f9c20073700 con 0x7f9c241348c0 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.205+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_done global_id 124209 payload 824 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.205+0000 7f9c28fbe640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c24073800 con 0x7f9c24131ce0 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.205+0000 7f9c29fc0640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9c0803e260 0x7f9c08040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.205+0000 7f9c29fc0640 10 monclient: get_auth_request con 0x7f9c0803e260 auth_method 0 2024-09-10T16:18:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c28fbe640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 msgr2=0x7f9c2402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c28fbe640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 0x7f9c2402d440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c1400f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c140035f0 con 0x7f9c24131ce0 2024-09-10T16:18:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 10 monclient: dump: 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c29fc0640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9c0803e260 0x7f9c08040720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0055a0 tx=0x7f9c1c005530 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9c14018070 con 0x7f9c24131ce0 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 msgr2=0x7f9c2402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 0x7f9c2402d440 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c1400f9e0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 msgr2=0x7f9c2402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.207+0000 7f9c19ffb640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c24131ce0 0x7f9c2402d440 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c1400f9e0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c2ba23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 0x7f9c2402dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c2ba23640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9c2413e640 con 0x7f9c2402d980 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 0x7f9c2402dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient: get_auth_request con 0x7f9c2402d980 auth_method 0 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.213+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.214+0000 7f9c28fbe640 10 monclient(hunting): handle_auth_done global_id 124212 payload 824 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.214+0000 7f9c28fbe640 1 -- 172.21.15.164:0/560960390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c24136e80 con 0x7f9c2402d980 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c28fbe640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 msgr2=0x7f9c2402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c28fbe640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 0x7f9c2402dd90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c14001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14003c90 con 0x7f9c2402d980 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 10 monclient: dump: 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f9c14003c90 con 0x7f9c2402d980 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:13.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 msgr2=0x7f9c2402dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 0x7f9c2402dd90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c14001020 comp rx=0 tx=0).stop 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 msgr2=0x7f9c2402dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.216+0000 7f9c19ffb640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c2402d980 0x7f9c2402dd90 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9c1400f520 tx=0x7f9c14001020 comp rx=0 tx=0).stop 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:13.227 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 69250, 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.228 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.229 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.230 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:13.231 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.232 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.233 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.234 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.235 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.236 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.237 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.238 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.239 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.240 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:13.241 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9c0803e260 msgr2=0x7f9c08040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9c0803e260 0x7f9c08040720 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f9c1c0055a0 tx=0x7f9c1c005530 comp rx=0 tx=0).stop 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 10 monclient: shutdown 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 msgr2=0x7f9c241317a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 --2- 172.21.15.164:0/560960390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9c241348c0 0x7f9c241317a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9c200097a0 tx=0x7f9c200046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c29fc0640 1 -- 172.21.15.164:0/560960390 reap_dead start 2024-09-10T16:18:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c29fc0640 1 -- 172.21.15.164:0/560960390 reap_dead start 2024-09-10T16:18:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 shutdown_connections 2024-09-10T16:18:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 >> 172.21.15.164:0/560960390 conn(0x7f9c240a6f80 msgr2=0x7f9c2409fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 shutdown_connections 2024-09-10T16:18:13.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.217+0000 7f9c2ba23640 1 -- 172.21.15.164:0/560960390 wait complete. 2024-09-10T16:18:13.243 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:13.243 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:18:13.243 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:18:13.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.341+0000 7f3371db4640 10 monclient: get_monmap_and_config 2024-09-10T16:18:13.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.341+0000 7f3371db4640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.341+0000 7f3371db4640 10 monclient: monmap: 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:13.342087+0000 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:13.342087+0000 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:13.343 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.341+0000 7f3371db4640 1 Processor -- start 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 -- start start 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient: init 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f336c06cb70 0x7f336c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-a con 0x7f336c06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 0x7f336c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-c con 0x7f336c0a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 0x7f336c138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-b con 0x7f336c137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f336c06db60 con 0x7f336c06cb70 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f336c06cf50 con 0x7f336c0a46a0 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f336c09ce50 con 0x7f336c137ca0 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f3371db4640 10 monclient(hunting): authenticate will time out at 1471.829542s 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f336c06cb70 0x7f336c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 0x7f336c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f336c06cb70 0x7f336c134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36810/0 (socket says 172.21.15.164:36810) 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 -- 172.21.15.164:0/2855064684 learned_addr learned my addr 172.21.15.164:0/2855064684 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): get_auth_request con 0x7f336c06cb70 auth_method 0 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c137ca0 auth_method 0 2024-09-10T16:18:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336affd640 1 --2- 172.21.15.164:0/2855064684 >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 0x7f336c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient(hunting): handle_auth_done global_id 124215 payload 274 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 -- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 msgr2=0x7f336c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 0x7f336c138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 -- 172.21.15.164:0/2855064684 >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 msgr2=0x7f336c0a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2855064684 >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 0x7f336c0a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336b7fe640 1 -- 172.21.15.164:0/2855064684 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c09cb70 con 0x7f336c06cb70 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336affd640 10 monclient: get_auth_request con 0x7f336c0a46a0 auth_method 0 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336bfff640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 0x7f336c138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.342+0000 7f336affd640 1 --2- 172.21.15.164:0/2855064684 >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 0x7f336c0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336b7fe640 1 -- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 msgr2=0x7f336c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c134f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f335c009870 tx=0x7f335c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40927c7d200d3bf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 1 -- 172.21.15.164:0/2855064684 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c037070 con 0x7f336c06cb70 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: dump: 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.353 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 1 -- 172.21.15.164:0/2855064684 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c032440 con 0x7f336c06cb70 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f3371db4640 5 monclient: authenticate success, global_id 124215 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f3371db4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f3371db4640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 1 -- 172.21.15.164:0/2855064684 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c032810 con 0x7f336c06cb70 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f3371db4640 4 set_mon_vals no callback set 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.345+0000 7f336a7fc640 10 monclient: dump: 2024-09-10T16:18:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 10 monclient: shutdown 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 -- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 msgr2=0x7f336c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c134f70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f335c009870 tx=0x7f335c02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 -- 172.21.15.164:0/2855064684 shutdown_connections 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c137ca0 0x7f336c138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 --2- 172.21.15.164:0/2855064684 >> v2:172.21.15.165:3300/0 conn(0x7f336c0a46a0 0x7f336c0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 --2- 172.21.15.164:0/2855064684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c134f70 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 -- 172.21.15.164:0/2855064684 >> 172.21.15.164:0/2855064684 conn(0x7f336c0aa900 msgr2=0x7f336c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 -- 172.21.15.164:0/2855064684 shutdown_connections 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.346+0000 7f3371db4640 1 -- 172.21.15.164:0/2855064684 wait complete. 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.347+0000 7f3371db4640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.347+0000 7f3371db4640 10 monclient: monmap: 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:13.357 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.347+0000 7f3371db4640 1 Processor -- start 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.347+0000 7f3371db4640 1 -- start start 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.347+0000 7f3371db4640 10 monclient: init 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c1c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-a con 0x7f336c06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-c con 0x7f336c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336c137ca0 0x7f336c1cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): picked mon.noname-b con 0x7f336c137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f336c1387f0 con 0x7f336c06cb70 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f336c138670 con 0x7f336c137ca0 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f336c138970 con 0x7f336c0a46a0 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f3371db4640 10 monclient(hunting): authenticate will time out at 1471.835159s 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c1c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c1c9d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36818/0 (socket says 172.21.15.164:36818) 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 1 -- 172.21.15.164:0/2378125325 learned_addr learned my addr 172.21.15.164:0/2378125325 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): get_auth_request con 0x7f336c06cb70 auth_method 0 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): get_auth_request con 0x7f336c0a46a0 auth_method 0 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336c137ca0 0x7f336c1cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 10 monclient(hunting): get_auth_request con 0x7f336c137ca0 auth_method 0 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient(hunting): handle_auth_done global_id 114293 payload 824 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336c137ca0 msgr2=0x7f336c1cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336c137ca0 0x7f336c1cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 msgr2=0x7f336c1c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c1c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 monclient: found mon.noname-c 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336affd640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336c06e720 con 0x7f336c0a46a0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336bfff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f336c137ca0 0x7f336c1cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.348+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f336c06cb70 0x7f336c1c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f336affd640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 msgr2=0x7f336c12f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f336affd640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f335400c610 tx=0x7f335400cad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354015070 con 0x7f336c0a46a0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: dump: 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3354011a10 con 0x7f336c0a46a0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354011de0 con 0x7f336c0a46a0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 5 monclient: authenticate success, global_id 114293 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 10 monclient: _renew_subs 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336c0a34b0 con 0x7f336c0a46a0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3369ffb640 4 set_mon_vals no callback set 2024-09-10T16:18:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3363fff640 10 monclient: dump: 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 10 monclient: _renew_subs 2024-09-10T16:18:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.350+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f336c0a3070 con 0x7f336c0a46a0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f3371db4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f3371db4640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 0x7f336c1cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f336c134430 con 0x7f336c1cd7e0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336affd640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f335400c610 tx=0x7f335400cad0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f3354052cb0 osd_map(405..405 src has 184..405) 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 0x7f336c1cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3354018070 con 0x7f336c0a46a0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient: get_auth_request con 0x7f336c1cd7e0 auth_method 0 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.351+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.352+0000 7f336b7fe640 10 monclient(hunting): handle_auth_done global_id 114296 payload 824 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.352+0000 7f336b7fe640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f336c1ce100 con 0x7f336c1cd7e0 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.352+0000 7f3363fff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f334c03e210 0x7f334c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.352+0000 7f336bfff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f334c03e210 0x7f334c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.352+0000 7f336bfff640 10 monclient: get_auth_request con 0x7f334c03e210 auth_method 0 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f336b7fe640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 msgr2=0x7f336c1cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 0x7f336c1cdbc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f335c035040 tx=0x7f335c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c043070 con 0x7f336c1cd7e0 2024-09-10T16:18:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 10 monclient: dump: 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f335c03c850 con 0x7f336c1cd7e0 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 msgr2=0x7f336c1cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 0x7f336c1cdbc0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f335c035040 tx=0x7f335c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 msgr2=0x7f336c1cdbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.354+0000 7f3363fff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c1cd7e0 0x7f336c1cdbc0 secure :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f335c035040 tx=0x7f335c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.355+0000 7f336bfff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f334c03e210 0x7f334c0406d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3364007520 tx=0x7f336400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f3371db4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f3371db4640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 0x7f336c09da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f3371db4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f336c0a6200 con 0x7f336c09d630 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 0x7f336c09da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient: get_auth_request con 0x7f336c09d630 auth_method 0 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.360+0000 7f336b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.361+0000 7f336b7fe640 10 monclient(hunting): handle_auth_done global_id 114299 payload 824 2024-09-10T16:18:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.361+0000 7f336b7fe640 1 -- 172.21.15.164:0/2378125325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f336c0a4ab0 con 0x7f336c09d630 2024-09-10T16:18:13.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f336b7fe640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 msgr2=0x7f336c09da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f336b7fe640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 0x7f336c09da40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f336c1345b0 tx=0x7f335c009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c037040 con 0x7f336c09d630 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 10 monclient: dump: 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.377 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f335c003820 con 0x7f336c09d630 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 msgr2=0x7f336c09da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 0x7f336c09da40 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f336c1345b0 tx=0x7f335c009240 comp rx=0 tx=0).stop 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 msgr2=0x7f336c09da40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f3363fff640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c09d630 0x7f336c09da40 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f336c1345b0 tx=0x7f335c009240 comp rx=0 tx=0).stop 2024-09-10T16:18:13.378 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 113932, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.388 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.389 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.390 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.391 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.392 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.392 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.392 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:13.392 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:13.392 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.393 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711098648}) 2024-09-10T16:18:13.393 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4136214487}) 2024-09-10T16:18:13.393 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.393 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=711098648}) 2024-09-10T16:18:13.393 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.363+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1037563129}) 2024-09-10T16:18:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f334c03e210 msgr2=0x7f334c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f334c03e210 0x7f334c0406d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3364007520 tx=0x7f336400b040 comp rx=0 tx=0).stop 2024-09-10T16:18:13.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 10 monclient: shutdown 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 msgr2=0x7f336c12f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f335400c610 tx=0x7f335400cad0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f336bfff640 1 -- 172.21.15.164:0/2378125325 reap_dead start 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3363fff640 1 -- 172.21.15.164:0/2378125325 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (unknown 0 0 0) 0x7f3354052cb0 con 0x7f336c0a46a0 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 shutdown_connections 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 --2- 172.21.15.164:0/2378125325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f336c0a46a0 0x7f336c12f730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 >> 172.21.15.164:0/2378125325 conn(0x7f336c0aa900 msgr2=0x7f336c131580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 shutdown_connections 2024-09-10T16:18:13.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.364+0000 7f3371db4640 1 -- 172.21.15.164:0/2378125325 wait complete. 2024-09-10T16:18:13.395 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:13.396 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:18:13.396 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:18:13.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 10 monclient: get_monmap_and_config 2024-09-10T16:18:13.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 10 monclient: monmap: 2024-09-10T16:18:13.495 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:13.494964+0000 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:13.494964+0000 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:13.496 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:13.497 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 1 Processor -- start 2024-09-10T16:18:13.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 1 -- start start 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.494+0000 7fa14fcb8640 10 monclient: init 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa14806cb70 0x7fa148134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-c con 0x7fa14806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1480a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1480a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14809ce50 con 0x7fa1480a4ab0 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa148138ac0 con 0x7fa1480a46a0 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa14806db60 con 0x7fa14806cb70 2024-09-10T16:18:13.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14fcb8640 10 monclient(hunting): authenticate will time out at 1471.982455s 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14e255640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa1480a4ab0 0x7fa1481383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:34242/0 (socket says 172.21.15.164:34242) 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14e255640 1 -- 172.21.15.164:0/2023494021 learned_addr learned my addr 172.21.15.164:0/2023494021 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 1 --2- 172.21.15.164:0/2023494021 >> v2:172.21.15.165:3300/0 conn(0x7fa14806cb70 0x7fa148134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): get_auth_request con 0x7fa14806cb70 auth_method 0 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 10 monclient(hunting): handle_auth_done global_id 134134 payload 274 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.495+0000 7fa14da54640 1 -- 172.21.15.164:0/2023494021 >> v1:172.21.15.165:6789/0 conn(0x7fa1480a46a0 legacy=0x7fa148137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 1 -- 172.21.15.164:0/2023494021 >> v1:172.21.15.164:6789/0 conn(0x7fa1480a4ab0 legacy=0x7fa1481383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 1 monclient: found mon.noname-c 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:18:13.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.496+0000 7fa14da54640 1 -- 172.21.15.164:0/2023494021 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa14809cb70 con 0x7fa14806cb70 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.498+0000 7fa14da54640 1 -- 172.21.15.164:0/2023494021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14806cb70 msgr2=0x7fa148134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.498+0000 7fa14da54640 1 --2- 172.21.15.164:0/2023494021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14806cb70 0x7fa148134f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa144009dc0 tx=0x7fa14402a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f185e6fe55eccaf2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 1 -- 172.21.15.164:0/2023494021 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144007a30 con 0x7fa14806cb70 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: dump: 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.511 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 1 -- 172.21.15.164:0/2023494021 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa144007bd0 con 0x7fa14806cb70 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 1 -- 172.21.15.164:0/2023494021 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144031400 con 0x7fa14806cb70 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14ca52640 10 monclient: dump: 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 5 monclient: authenticate success, global_id 134134 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 4 set_mon_vals no callback set 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 10 monclient: shutdown 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 1 -- 172.21.15.164:0/2023494021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14806cb70 msgr2=0x7fa148134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/2023494021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14806cb70 0x7fa148134f70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa144009dc0 tx=0x7fa14402a4d0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 1 -- 172.21.15.164:0/2023494021 shutdown_connections 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/2023494021 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14806cb70 0x7fa148134f70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.499+0000 7fa14fcb8640 1 -- 172.21.15.164:0/2023494021 >> 172.21.15.164:0/2023494021 conn(0x7fa1480aa900 msgr2=0x7fa1480aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 1 -- 172.21.15.164:0/2023494021 shutdown_connections 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 1 -- 172.21.15.164:0/2023494021 wait complete. 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 10 monclient: monmap: 2024-09-10T16:18:13.515 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:13.516 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 1 Processor -- start 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 1 -- start start 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.500+0000 7fa14fcb8640 10 monclient: init 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa14806cb70 0x7fa14812d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-c con 0x7fa14806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 0x7fa14812d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-a con 0x7fa1480a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1480a4ab0 0x7fa14809e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): picked mon.noname-b con 0x7fa1480a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa148141050 con 0x7fa1480a46a0 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa14806cf50 con 0x7fa1480a4ab0 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa1481411d0 con 0x7fa14806cb70 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14fcb8640 10 monclient(hunting): authenticate will time out at 1471.988349s 2024-09-10T16:18:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa14806cb70 0x7fa14812d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 0x7fa14812d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa14806cb70 0x7fa14812d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52274/0 (socket says 172.21.15.164:52274) 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 1 -- 172.21.15.164:0/4170033173 learned_addr learned my addr 172.21.15.164:0/4170033173 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): get_auth_request con 0x7fa14806cb70 auth_method 0 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): get_auth_request con 0x7fa1480a46a0 auth_method 0 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1480a4ab0 0x7fa14809e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 10 monclient(hunting): get_auth_request con 0x7fa1480a4ab0 auth_method 0 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient(hunting): handle_auth_done global_id 124221 payload 824 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa14806cb70 msgr2=0x7fa14812d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa14806cb70 0x7fa14812d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1480a4ab0 msgr2=0x7fa14809e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1480a4ab0 0x7fa14809e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 monclient: found mon.noname-a 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14d253640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1480a1e80 con 0x7fa1480a46a0 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.501+0000 7fa14e255640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa1480a4ab0 0x7fa14809e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14d253640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 msgr2=0x7fa14812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14d253640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 0x7fa14812d7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa13800c440 tx=0x7fa13800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1380119a0 con 0x7fa1480a46a0 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: dump: 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.522 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 5 monclient: authenticate success, global_id 124221 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa138017070 con 0x7fa1480a46a0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 monclient: _renew_subs 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1480a2160 con 0x7fa1480a46a0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa138011d80 con 0x7fa1480a46a0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13e7fc640 10 monclient: dump: 2024-09-10T16:18:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa13ffff640 4 set_mon_vals no callback set 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 monclient: _renew_subs 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:18:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1480a1ba0 con 0x7fa1480a46a0 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 0x7fa14809ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.504+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa148134420 con 0x7fa14809e980 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa13801c460 con 0x7fa1480a46a0 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 0x7fa14809ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient: get_auth_request con 0x7fa14809e980 auth_method 0 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa13e7fc640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa12c03e210 0x7fa12c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7fa138052b20 con 0x7fa1480a46a0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 10 monclient(hunting): handle_auth_done global_id 134140 payload 824 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14da54640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa148134010 con 0x7fa14809e980 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14e255640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa12c03e210 0x7fa12c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.505+0000 7fa14e255640 10 monclient: get_auth_request con 0x7fa12c03e210 auth_method 0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.507+0000 7fa14da54640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 msgr2=0x7fa14809ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.507+0000 7fa14da54640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 0x7fa14809ed60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa144000f80 tx=0x7fa144004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa14e255640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa12c03e210 0x7fa12c0406d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1481352c0 tx=0x7fa140009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144031cd0 con 0x7fa14809e980 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 10 monclient: dump: 2024-09-10T16:18:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa14403c420 con 0x7fa14809e980 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 msgr2=0x7fa14809ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 0x7fa14809ed60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa144000f80 tx=0x7fa144004a40 comp rx=0 tx=0).stop 2024-09-10T16:18:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 msgr2=0x7fa14809ed60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.508+0000 7fa13e7fc640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809e980 0x7fa14809ed60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867906208}) 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=796324055}) 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3867906208}) 2024-09-10T16:18:13.529 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:18:13.509+0000 7f114c05d640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302780305}) 2024-09-10T16:18:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.514+0000 7fa14fcb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.514+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 0x7fa14812f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.514+0000 7fa14fcb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.514+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa148134f70 con 0x7fa14809f2a0 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 0x7fa14812f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient: get_auth_request con 0x7fa14809f2a0 auth_method 0 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 10 monclient(hunting): handle_auth_done global_id 134143 payload 824 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.515+0000 7fa14da54640 1 -- 172.21.15.164:0/4170033173 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa1480770f0 con 0x7fa14809f2a0 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa14da54640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 msgr2=0x7fa14812f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa14da54640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 0x7fa14812f240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa14402a510 tx=0x7fa144030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14404a040 con 0x7fa14809f2a0 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 10 monclient: dump: 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.534 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fa144004660 con 0x7fa14809f2a0 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 msgr2=0x7fa14812f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 0x7fa14812f240 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa14402a510 tx=0x7fa144030000 comp rx=0 tx=0).stop 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 msgr2=0x7fa14812f240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.518+0000 7fa13e7fc640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa14809f2a0 0x7fa14812f240 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa14402a510 tx=0x7fa144030000 comp rx=0 tx=0).stop 2024-09-10T16:18:13.535 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 110, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 17922, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.536 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.537 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.538 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:13.539 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:13.540 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.542 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.543 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:13.544 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.545 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.546 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:13.547 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:13.548 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa12c03e210 msgr2=0x7fa12c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa12c03e210 0x7fa12c0406d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa1481352c0 tx=0x7fa140009040 comp rx=0 tx=0).stop 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 10 monclient: shutdown 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 msgr2=0x7fa14812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 --2- 172.21.15.164:0/4170033173 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa1480a46a0 0x7fa14812d7c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa13800c440 tx=0x7fa13800c900 comp rx=0 tx=0).stop 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14e255640 1 -- 172.21.15.164:0/4170033173 reap_dead start 2024-09-10T16:18:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14e255640 1 -- 172.21.15.164:0/4170033173 reap_dead start 2024-09-10T16:18:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 shutdown_connections 2024-09-10T16:18:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 >> 172.21.15.164:0/4170033173 conn(0x7fa1480aa900 msgr2=0x7fa1480a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 shutdown_connections 2024-09-10T16:18:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.519+0000 7fa14fcb8640 1 -- 172.21.15.164:0/4170033173 wait complete. 2024-09-10T16:18:13.550 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:18:13.550 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:18:13.550 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:18:13.550 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:18:13.551 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:18:13.551 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:18:13.551 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:18:13.561 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:18:13.561 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:18:13.562 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.696+0000 7f271bd94640 10 monclient: get_monmap_and_config 2024-09-10T16:18:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.696+0000 7f271bd94640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.696+0000 7f271bd94640 10 monclient: monmap: 2024-09-10T16:18:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:13.697761+0000 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:13.697761+0000 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.697+0000 7f271bd94640 1 Processor -- start 2024-09-10T16:18:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.697+0000 7f271bd94640 1 -- start start 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.697+0000 7f271bd94640 10 monclient: init 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f271409d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 0x7f27141397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-a con 0x7f271409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271409e420 0x7f271413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-c con 0x7f271409e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27140a0870 con 0x7f271409d930 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f271409d3e0 con 0x7f271409ab50 2024-09-10T16:18:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2714074c70 con 0x7f271409e420 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271bd94640 10 monclient(hunting): authenticate will time out at 1472.185134s 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271932f640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 msgr2=0x7f27141397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271932f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 0x7f27141397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271409e420 0x7f271413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f2719b30640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f271409d930 0x7f271409dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:47630/0 (socket says 172.21.15.164:47630) 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f2719b30640 1 -- 172.21.15.164:0/2966377877 learned_addr learned my addr 172.21.15.164:0/2966377877 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): get_auth_request con 0x7f271409e420 auth_method 0 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient(hunting): handle_auth_done global_id 134146 payload 274 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 -- 172.21.15.164:0/2966377877 >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 msgr2=0x7f27141397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 --2- 172.21.15.164:0/2966377877 >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 0x7f27141397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 -- 172.21.15.164:0/2966377877 >> v1:172.21.15.165:6789/0 conn(0x7f271409d930 legacy=0x7f271409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 monclient: found mon.noname-c 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:18:13.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.698+0000 7f271a331640 1 -- 172.21.15.164:0/2966377877 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271409ecb0 con 0x7f271409e420 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.701+0000 7f271a331640 1 -- 172.21.15.164:0/2966377877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 msgr2=0x7f271413bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.701+0000 7f271a331640 1 --2- 172.21.15.164:0/2966377877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f271413bd10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f271409af60 tx=0x7f270c02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d279eb043a84a530 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 1 -- 172.21.15.164:0/2966377877 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c037070 con 0x7f271409e420 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: dump: 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 1 -- 172.21.15.164:0/2966377877 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f270c032640 con 0x7f271409e420 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 1 -- 172.21.15.164:0/2966377877 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f270c032a10 con 0x7f271409e420 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 5 monclient: authenticate success, global_id 134146 2024-09-10T16:18:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 4 set_mon_vals no callback set 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f2718b2e640 10 monclient: dump: 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 10 monclient: shutdown 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 -- 172.21.15.164:0/2966377877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 msgr2=0x7f271413bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 --2- 172.21.15.164:0/2966377877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f271413bd10 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f271409af60 tx=0x7f270c02a000 comp rx=0 tx=0).stop 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 -- 172.21.15.164:0/2966377877 shutdown_connections 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 --2- 172.21.15.164:0/2966377877 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f271413bd10 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 --2- 172.21.15.164:0/2966377877 >> v2:172.21.15.164:3300/0 conn(0x7f271409ab50 0x7f27141397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 -- 172.21.15.164:0/2966377877 >> 172.21.15.164:0/2966377877 conn(0x7f27140a3110 msgr2=0x7f27140a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 -- 172.21.15.164:0/2966377877 shutdown_connections 2024-09-10T16:18:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.702+0000 7f271bd94640 1 -- 172.21.15.164:0/2966377877 wait complete. 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.703+0000 7f271bd94640 10 monclient: build_initial_monmap 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.703+0000 7f271bd94640 10 monclient: monmap: 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:13.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.703+0000 7f271bd94640 1 Processor -- start 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.703+0000 7f271bd94640 1 -- start start 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.703+0000 7f271bd94640 10 monclient: init 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-c con 0x7f271409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 0x7f27141cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-a con 0x7f271409d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f27141cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): picked mon.noname-b con 0x7f271409e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f27140a2940 con 0x7f271409d930 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f271409ef30 con 0x7f271409e420 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f27140a2ac0 con 0x7f271409ab50 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271bd94640 10 monclient(hunting): authenticate will time out at 1472.191112s 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271932f640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 msgr2=0x7f27141cd5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271932f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 0x7f27141cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52282/0 (socket says 172.21.15.164:52282) 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 -- 172.21.15.164:0/3034648853 learned_addr learned my addr 172.21.15.164:0/3034648853 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): get_auth_request con 0x7f271409ab50 auth_method 0 2024-09-10T16:18:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f27141cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): get_auth_request con 0x7f271409e420 auth_method 0 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f271a331640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient(hunting): handle_auth_done global_id 114305 payload 824 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 -- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 msgr2=0x7f27141cdb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f27141cdb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 -- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 msgr2=0x7f27141cd5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 0x7f27141cd5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient: _finish_hunting 0 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 monclient: found mon.noname-c 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.704+0000 7f2719b30640 1 -- 172.21.15.164:0/3034648853 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27141331f0 con 0x7f271409ab50 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f2719b30640 1 -- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 msgr2=0x7f27141cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f2719b30640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f270400c460 tx=0x7f270400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704007070 con 0x7f271409ab50 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 10 monclient: dump: 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.719 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 10 monclient: _finish_auth 0 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 10 monclient: _check_auth_tickets 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2704007230 con 0x7f271409ab50 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f271bd94640 5 monclient: authenticate success, global_id 114305 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f271bd94640 10 monclient: _renew_subs 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f271bd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.706+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2714132290 con 0x7f271409ab50 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f27127fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2704007620 con 0x7f271409ab50 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f27127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f27127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:13.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f27127fc640 10 monclient: dump: 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 10 monclient: _renew_subs 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f2713fff640 4 set_mon_vals no callback set 2024-09-10T16:18:13.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2714134370 con 0x7f271409ab50 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.707+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f271409dd90 con 0x7f271409ab50 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.708+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f270401b400 con 0x7f271409ab50 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.708+0000 7f27127fc640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 0x7f2700040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.708+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f27040529f0 con 0x7f271409ab50 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.708+0000 7f271932f640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 0x7f2700040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:13.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.708+0000 7f271932f640 10 monclient: get_auth_request con 0x7f270003e1c0 auth_method 0 2024-09-10T16:18:13.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.711+0000 7f271932f640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 0x7f2700040680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2708003d00 tx=0x7f2708009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:13.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.712+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2704010cf0 con 0x7f271409ab50 2024-09-10T16:18:13.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.712+0000 7f27127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:13.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.712+0000 7f27127fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:13.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.953+0000 7f271bd94640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:13.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.953+0000 7f271bd94640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.953+0000 7f271bd94640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.953+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f271406daa0 con 0x7f271409ab50 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.954+0000 7f27127fc640 1 -- 172.21.15.164:0/3034648853 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f2704012400 con 0x7f271409ab50 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.954+0000 7f27127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.954+0000 7f27127fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:13.956 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:13.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 msgr2=0x7f2700040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 0x7f2700040680 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2708003d00 tx=0x7f2708009270 comp rx=0 tx=0).stop 2024-09-10T16:18:13.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 10 monclient: shutdown 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 msgr2=0x7f27141cd080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f270400c460 tx=0x7f270400c920 comp rx=0 tx=0).stop 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 shutdown_connections 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f270003e1c0 0x7f2700040680 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271409e420 0x7f27141cdb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271409d930 0x7f27141cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 --2- 172.21.15.164:0/3034648853 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f271409ab50 0x7f27141cd080 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 >> 172.21.15.164:0/3034648853 conn(0x7f27140a3110 msgr2=0x7f27141be8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 shutdown_connections 2024-09-10T16:18:13.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:13.956+0000 7f271bd94640 1 -- 172.21.15.164:0/3034648853 wait complete. 2024-09-10T16:18:16.969 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:17.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 10 monclient: get_monmap_and_config 2024-09-10T16:18:17.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 10 monclient: build_initial_monmap 2024-09-10T16:18:17.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 10 monclient: monmap: 2024-09-10T16:18:17.078 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:17.078081+0000 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:17.078081+0000 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:17.079 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 1 Processor -- start 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 1 -- start start 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.077+0000 7fcac57bb640 10 monclient: init 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient: _add_conns ranks=[4,2,0,1,3] 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcac006d160 0x7fcac0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-c con 0x7fcac006d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-b con 0x7fcac006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcac00a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcac009cb90 con 0x7fcac00a4c20 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:17.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcac009ce70 con 0x7fcac006d160 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcac009d150 con 0x7fcac006d540 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcac57bb640 10 monclient(hunting): authenticate will time out at 1475.565430s 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabf7fe640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fcac00a4c20 legacy=0x7fcac0137ce0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57974/0 (socket says 172.21.15.164:57974) 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 -- 172.21.15.164:0/1807506250 learned_addr learned my addr 172.21.15.164:0/1807506250 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): get_auth_request con 0x7fcac006d540 auth_method 0 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabeffd640 1 --2- 172.21.15.164:0/1807506250 >> v2:172.21.15.165:3300/0 conn(0x7fcac006d160 0x7fcac0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabeffd640 10 monclient(hunting): get_auth_request con 0x7fcac006d160 auth_method 0 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient(hunting): handle_auth_done global_id 114308 payload 274 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 -- 172.21.15.164:0/1807506250 >> v2:172.21.15.165:3300/0 conn(0x7fcac006d160 msgr2=0x7fcac0134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 --2- 172.21.15.164:0/1807506250 >> v2:172.21.15.165:3300/0 conn(0x7fcac006d160 0x7fcac0134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 -- 172.21.15.164:0/1807506250 >> v1:172.21.15.164:6789/0 conn(0x7fcac00a4c20 legacy=0x7fcac0137ce0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:18:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 monclient: found mon.noname-b 2024-09-10T16:18:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.078+0000 7fcabe7fc640 1 -- 172.21.15.164:0/1807506250 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac0135340 con 0x7fcac006d540 2024-09-10T16:18:17.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabe7fc640 1 -- 172.21.15.164:0/1807506250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 msgr2=0x7fcac00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:17.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabe7fc640 1 --2- 172.21.15.164:0/1807506250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009870 tx=0x7fcab402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c4f14ba8679c8e0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 1 -- 172.21.15.164:0/1807506250 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4037070 con 0x7fcac006d540 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: dump: 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.092 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: _finish_auth 0 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcac57bb640 5 monclient: authenticate success, global_id 114308 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 1 -- 172.21.15.164:0/1807506250 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab4032440 con 0x7fcac006d540 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcac57bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcac57bb640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 1 -- 172.21.15.164:0/1807506250 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab4032810 con 0x7fcac006d540 2024-09-10T16:18:17.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcac57bb640 4 set_mon_vals no callback set 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.081+0000 7fcabdffb640 10 monclient: dump: 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:17.094 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 10 monclient: shutdown 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 -- 172.21.15.164:0/1807506250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 msgr2=0x7fcac00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 --2- 172.21.15.164:0/1807506250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009870 tx=0x7fcab402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 -- 172.21.15.164:0/1807506250 shutdown_connections 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 --2- 172.21.15.164:0/1807506250 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d540 0x7fcac00a46e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 --2- 172.21.15.164:0/1807506250 >> v2:172.21.15.165:3300/0 conn(0x7fcac006d160 0x7fcac0134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 -- 172.21.15.164:0/1807506250 >> 172.21.15.164:0/1807506250 conn(0x7fcac00aacb0 msgr2=0x7fcac00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 -- 172.21.15.164:0/1807506250 shutdown_connections 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.082+0000 7fcac57bb640 1 -- 172.21.15.164:0/1807506250 wait complete. 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.083+0000 7fcac57bb640 10 monclient: build_initial_monmap 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.083+0000 7fcac57bb640 10 monclient: monmap: 2024-09-10T16:18:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.083+0000 7fcac57bb640 1 Processor -- start 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.083+0000 7fcac57bb640 1 -- start start 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.083+0000 7fcac57bb640 10 monclient: init 2024-09-10T16:18:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-c con 0x7fcac006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 0x7fcac01cd950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-a con 0x7fcac006d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 0x7fcac01cde90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): picked mon.noname-b con 0x7fcac00a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcac0138870 con 0x7fcac006d540 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcac01386f0 con 0x7fcac00a4c20 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcac01389f0 con 0x7fcac006d160 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcac57bb640 10 monclient(hunting): authenticate will time out at 1475.571246s 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabe7fc640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 msgr2=0x7fcac01cd950 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabe7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 0x7fcac01cd950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57980/0 (socket says 172.21.15.164:57980) 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 -- 172.21.15.164:0/3484849850 learned_addr learned my addr 172.21.15.164:0/3484849850 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): get_auth_request con 0x7fcac006d160 auth_method 0 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 0x7fcac01cde90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 10 monclient(hunting): get_auth_request con 0x7fcac00a4c20 auth_method 0 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient(hunting): handle_auth_done global_id 114311 payload 824 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 -- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 msgr2=0x7fcac01cde90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 0x7fcac01cde90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 -- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 msgr2=0x7fcac01cd950 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 0x7fcac01cd950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient: _finish_hunting 0 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 monclient: found mon.noname-c 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabeffd640 1 -- 172.21.15.164:0/3484849850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcac00a48e0 con 0x7fcac006d160 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.084+0000 7fcabf7fe640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 0x7fcac01cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcabeffd640 1 -- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 msgr2=0x7fcac01315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcabeffd640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcab800d830 tx=0x7fcab800dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab800b8d0 con 0x7fcac006d160 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 10 monclient: dump: 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 10 monclient: _finish_auth 0 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcab8019070 con 0x7fcac006d160 2024-09-10T16:18:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.086+0000 7fcac57bb640 5 monclient: authenticate success, global_id 114311 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 10 monclient: _renew_subs 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcac013faa0 con 0x7fcac006d160 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcaa77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcab800bc90 con 0x7fcac006d160 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcaa77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcaa77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcaa77fe640 10 monclient: dump: 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 10 monclient: _renew_subs 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcabd7fa640 4 set_mon_vals no callback set 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.087+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcac013b610 con 0x7fcac006d160 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcac57bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcac57bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcac57bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcab801e460 con 0x7fcac006d160 2024-09-10T16:18:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcac0137df0 con 0x7fcac006d160 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcaa77fe640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 0x7fcaac040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7fcab8053250 con 0x7fcac006d160 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcabe7fc640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 0x7fcaac040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.088+0000 7fcabe7fc640 10 monclient: get_auth_request con 0x7fcaac03e1c0 auth_method 0 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.091+0000 7fcabe7fc640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 0x7fcaac040680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcab4034000 tx=0x7fcab402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.092+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcac0137df0 con 0x7fcac006d160 2024-09-10T16:18:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.092+0000 7fcaa77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.092+0000 7fcaa77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:17.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcac57bb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:17.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcac57bb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcac57bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcac0134e00 con 0x7fcac006d160 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcaa77fe640 1 -- 172.21.15.164:0/3484849850 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fcab8051c30 con 0x7fcac006d160 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcaa77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.341+0000 7fcaa77fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:17.343 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":110,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 msgr2=0x7fcaac040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 0x7fcaac040680 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcab4034000 tx=0x7fcab402e000 comp rx=0 tx=0).stop 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 10 monclient: shutdown 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 msgr2=0x7fcac01315f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fcab800d830 tx=0x7fcab800dcf0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 shutdown_connections 2024-09-10T16:18:17.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcaac03e1c0 0x7fcaac040680 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcac00a4c20 0x7fcac01cde90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcac006d540 0x7fcac01cd950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 --2- 172.21.15.164:0/3484849850 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcac006d160 0x7fcac01315f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 >> 172.21.15.164:0/3484849850 conn(0x7fcac00aacb0 msgr2=0x7fcac0136cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 shutdown_connections 2024-09-10T16:18:17.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:17.343+0000 7fcac57bb640 1 -- 172.21.15.164:0/3484849850 wait complete. 2024-09-10T16:18:20.357 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.465+0000 7f88e7047640 10 monclient: get_monmap_and_config 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.465+0000 7f88e7047640 10 monclient: build_initial_monmap 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.465+0000 7f88e7047640 10 monclient: monmap: 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:20.466547+0000 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:20.466547+0000 2024-09-10T16:18:20.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:20.468 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 Processor -- start 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 -- start start 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient: init 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T16:18:20.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88e009c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f88e009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-a con 0x7f88e013b5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e009a430 con 0x7f88e009c180 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e009f860 con 0x7f88e009d160 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e4de3640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f88e009c180 legacy=0x7f88e009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f88e00a24e0 con 0x7f88e013b5d0 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.466+0000 7f88e7047640 10 monclient(hunting): authenticate will time out at 1478.953990s 2024-09-10T16:18:20.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.467+0000 7f88e55e4640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 msgr2=0x7f88e013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.467+0000 7f88e55e4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.467+0000 7f88dffff640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f88e009d160 0x7f88e009f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:41216/0 (socket says 172.21.15.164:41216) 2024-09-10T16:18:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.467+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 learned_addr learned my addr 172.21.15.164:0/1482413088 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:20.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.469+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.165:6789/0 conn(0x7f88e009d160 legacy=0x7f88e009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:20.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.668+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88e009c180 legacy=0x7f88e009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:20.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.668+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 msgr2=0x7f88e013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:20.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:20.668+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:21.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.069+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88e009c180 legacy=0x7f88e009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:21.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.069+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 msgr2=0x7f88e013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:21.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.069+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:21.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.870+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88e009c180 legacy=0x7f88e009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:21.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.870+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 msgr2=0x7f88e013d9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:21.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:21.870+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:18:23.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): tick 2024-09-10T16:18:23.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 msgr2=0x7f88e013d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.164:3300/0 conn(0x7f88e013b5d0 0x7f88e013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.165:6789/0 conn(0x7f88e009d160 legacy=0x7f88e009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88e009c180 legacy=0x7f88e009ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient: _add_conns ranks=[4,2] 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f88c4003770 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f88c4006280 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:23.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f88e00a24e0 con 0x7f88c4003770 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88c4008e20 con 0x7f88c4006280 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88de7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4003770 auth_method 0 2024-09-10T16:18:23.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:23.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:23.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4003770 auth_method 0 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:23.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:23.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:24.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4003770 auth_method 0 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:24.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.069+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:24.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88dffff640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4003770 auth_method 0 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:24.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:24.870+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:18:26.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): tick 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:26.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 msgr2=0x7f88c4005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c4003770 0x7f88c4005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 reap_dead start 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f88c4006280 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f88c4003be0 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f88c400a9a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:26.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88e00a2bf0 con 0x7f88c4003be0 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f88e00a24e0 con 0x7f88c400a9a0 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 1 -- 172.21.15.164:0/1482413088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88c4008e20 con 0x7f88c4006280 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88de7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88c4003be0 legacy=0x7f88c4009280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:26.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:26.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 10 monclient(hunting): get_auth_request con 0x7f88c400a9a0 auth_method 0 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.467+0000 7f88e55e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.468+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.468+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:26.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.468+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:26.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.468+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:26.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88c4003be0 legacy=0x7f88c4009280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:26.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.668+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 10 monclient(hunting): get_auth_request con 0x7f88c400a9a0 auth_method 0 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:26.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:26.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:26.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:26.669+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.069+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88c4003be0 legacy=0x7f88c4009280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:27.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 10 monclient(hunting): get_auth_request con 0x7f88c400a9a0 auth_method 0 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:27.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.070+0000 7f88e55e4640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:27.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.870+0000 7f88dffff640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88c4003be0 legacy=0x7f88c4009280 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:18:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient(hunting): handle_auth_done global_id 114314 payload 274 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 msgr2=0x7f88c400cd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> v1:172.21.15.164:6789/0 conn(0x7f88c4003be0 legacy=0x7f88c4009280 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient: _finish_hunting 0 2024-09-10T16:18:27.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 monclient: found mon.noname-b 2024-09-10T16:18:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.871+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e009a430 con 0x7f88c4006280 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88e4de3640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88e4de3640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f88d4003980 tx=0x7f88d402c370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5560d706bc081b33 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 1 -- 172.21.15.164:0/1482413088 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4032070 con 0x7f88c4006280 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: dump: 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: _finish_auth 0 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88e7047640 5 monclient: authenticate success, global_id 114314 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88e7047640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 1 -- 172.21.15.164:0/1482413088 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88d4006e50 con 0x7f88c4006280 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.874+0000 7f88df7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 4 set_mon_vals no callback set 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88df7fe640 1 -- 172.21.15.164:0/1482413088 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d4032660 con 0x7f88c4006280 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88df7fe640 10 monclient: dump: 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.879 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:27.880 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 10 monclient: shutdown 2024-09-10T16:18:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 1 -- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88c4008740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f88d4003980 tx=0x7f88d402c370 comp rx=0 tx=0).stop 2024-09-10T16:18:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 1 -- 172.21.15.164:0/1482413088 shutdown_connections 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.875+0000 7f88e7047640 1 --2- 172.21.15.164:0/1482413088 >> v2:172.21.15.165:3300/0 conn(0x7f88c400a9a0 0x7f88c400cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 1 --2- 172.21.15.164:0/1482413088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88c4008740 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 1 -- 172.21.15.164:0/1482413088 >> 172.21.15.164:0/1482413088 conn(0x7f88e00a3110 msgr2=0x7f88e00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 1 -- 172.21.15.164:0/1482413088 shutdown_connections 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 1 -- 172.21.15.164:0/1482413088 wait complete. 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 10 monclient: build_initial_monmap 2024-09-10T16:18:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.876+0000 7f88e7047640 10 monclient: monmap: 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.877+0000 7f88e7047640 1 Processor -- start 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.877+0000 7f88e7047640 1 -- start start 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.877+0000 7f88e7047640 10 monclient: init 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88c4003be0 0x7f88e0093c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-b con 0x7f88c4003be0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-c con 0x7f88c4006280 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 0x7f88e0091ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): picked mon.noname-a con 0x7f88c400a9a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f88e00a2230 con 0x7f88c400a9a0 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f88e00a20b0 con 0x7f88c4003be0 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f88e0074e10 con 0x7f88c4006280 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e7047640 10 monclient(hunting): authenticate will time out at 1486.365389s 2024-09-10T16:18:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e55e4640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 msgr2=0x7f88e0091ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e55e4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 0x7f88e0091ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60576/0 (socket says 172.21.15.164:60576) 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 -- 172.21.15.164:0/3115445190 learned_addr learned my addr 172.21.15.164:0/3115445190 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): get_auth_request con 0x7f88c4006280 auth_method 0 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88c4003be0 0x7f88e0093c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): get_auth_request con 0x7f88c4003be0 auth_method 0 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88e4de3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient(hunting): handle_auth_done global_id 114317 payload 824 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 -- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88c4003be0 msgr2=0x7f88e0093c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88c4003be0 0x7f88e0093c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 -- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 msgr2=0x7f88e0091ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 0x7f88e0091ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient: _finish_hunting 0 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 monclient: found mon.noname-c 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.878+0000 7f88dffff640 1 -- 172.21.15.164:0/3115445190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88e0094e40 con 0x7f88c4006280 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dffff640 1 -- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88e0091660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dffff640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f88cc007bd0 tx=0x7f88cc0078d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc01a020 con 0x7f88c4006280 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: dump: 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88cc001d30 con 0x7f88c4006280 2024-09-10T16:18:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 5 monclient: authenticate success, global_id 114317 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 10 monclient: _renew_subs 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88e0095120 con 0x7f88c4006280 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc004e40 con 0x7f88c4006280 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88dd7fa640 10 monclient: dump: 2024-09-10T16:18:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 10 monclient: _renew_subs 2024-09-10T16:18:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88deffd640 4 set_mon_vals no callback set 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.881+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88e00945e0 con 0x7f88c4006280 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f88cc01d020 con 0x7f88c4006280 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88e7047640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88e7047640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88e7047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88e009d930 con 0x7f88c4006280 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88dd7fa640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 0x7f88c4041e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.882+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f88cc053240 con 0x7f88c4006280 2024-09-10T16:18:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.883+0000 7f88e4de3640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 0x7f88c4041e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.883+0000 7f88e4de3640 10 monclient: get_auth_request con 0x7f88c403f970 auth_method 0 2024-09-10T16:18:27.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.885+0000 7f88e4de3640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 0x7f88c4041e30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f88c4008dc0 tx=0x7f88d4032000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:27.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.887+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f88cc024030 con 0x7f88c4006280 2024-09-10T16:18:27.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.887+0000 7f88dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:27.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:27.887+0000 7f88dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.133+0000 7f88e7047640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.133+0000 7f88e7047640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.133+0000 7f88e7047640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.133+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f88e0099c90 con 0x7f88c4006280 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.134+0000 7f88dd7fa640 1 -- 172.21.15.164:0/3115445190 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f88cc01ecd0 con 0x7f88c4006280 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.134+0000 7f88dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:28.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.134+0000 7f88dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:28.136 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":114,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:28.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 msgr2=0x7f88c4041e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 0x7f88c4041e30 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f88c4008dc0 tx=0x7f88d4032000 comp rx=0 tx=0).stop 2024-09-10T16:18:28.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 10 monclient: shutdown 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 msgr2=0x7f88e0091660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f88cc007bd0 tx=0x7f88cc0078d0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 shutdown_connections 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f88c403f970 0x7f88c4041e30 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f88c400a9a0 0x7f88e0091ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f88c4006280 0x7f88e0091660 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 --2- 172.21.15.164:0/3115445190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f88c4003be0 0x7f88e0093c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 >> 172.21.15.164:0/3115445190 conn(0x7f88e00a3110 msgr2=0x7f88e0139c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 shutdown_connections 2024-09-10T16:18:28.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.136+0000 7f88e7047640 1 -- 172.21.15.164:0/3115445190 wait complete. 2024-09-10T16:18:28.148 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:18:28.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:18:28.149 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.245+0000 7f1eb09ec640 10 monclient: get_monmap_and_config 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.245+0000 7f1eb09ec640 10 monclient: build_initial_monmap 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.245+0000 7f1eb09ec640 10 monclient: monmap: 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:28.246654+0000 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:28.246654+0000 2024-09-10T16:18:28.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:28.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.246+0000 7f1eb09ec640 1 Processor -- start 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.246+0000 7f1eb09ec640 1 -- start start 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.246+0000 7f1eb09ec640 10 monclient: init 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.246+0000 7f1eb09ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 0x7f1eac09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac09d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1eac134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac0a0870 con 0x7f1eac134580 2024-09-10T16:18:28.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1eac09d3e0 con 0x7f1eac09d930 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1eac09e790 con 0x7f1eac09ab50 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eb09ec640 10 monclient(hunting): authenticate will time out at 1486.734131s 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60590/0 (socket says 172.21.15.164:60590) 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 -- 172.21.15.164:0/660534265 learned_addr learned my addr 172.21.15.164:0/660534265 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): get_auth_request con 0x7f1eac09ab50 auth_method 0 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/660534265 >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 0x7f1eac09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient(hunting): handle_auth_done global_id 114320 payload 274 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 -- 172.21.15.164:0/660534265 >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 msgr2=0x7f1eac09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 --2- 172.21.15.164:0/660534265 >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 0x7f1eac09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 -- 172.21.15.164:0/660534265 >> v1:172.21.15.165:6789/0 conn(0x7f1eac134580 legacy=0x7f1eac134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 monclient: found mon.noname-b 2024-09-10T16:18:28.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1ea9d9b640 1 -- 172.21.15.164:0/660534265 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac074c70 con 0x7f1eac09ab50 2024-09-10T16:18:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eaa59c640 10 monclient: get_auth_request con 0x7f1eac09d930 auth_method 0 2024-09-10T16:18:28.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.247+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/660534265 >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 0x7f1eac09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea9d9b640 1 -- 172.21.15.164:0/660534265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 msgr2=0x7f1eac09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea9d9b640 1 --2- 172.21.15.164:0/660534265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400bb00 tx=0x7f1e9402c5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=834fb38fbdf138c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 1 -- 172.21.15.164:0/660534265 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9403a070 con 0x7f1eac09ab50 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: dump: 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: _finish_auth 0 2024-09-10T16:18:28.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: _check_auth_tickets 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 5 monclient: authenticate success, global_id 114320 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 1 -- 172.21.15.164:0/660534265 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e94032770 con 0x7f1eac09ab50 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 4 set_mon_vals no callback set 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 1 -- 172.21.15.164:0/660534265 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e94032b40 con 0x7f1eac09ab50 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1ea959a640 10 monclient: dump: 2024-09-10T16:18:28.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 10 monclient: shutdown 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:28.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 1 -- 172.21.15.164:0/660534265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 msgr2=0x7f1eac09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.250+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/660534265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400bb00 tx=0x7f1e9402c5b0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 -- 172.21.15.164:0/660534265 shutdown_connections 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/660534265 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac09ab50 0x7f1eac09e250 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/660534265 >> v2:172.21.15.165:3300/0 conn(0x7f1eac09d930 0x7f1eac09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 -- 172.21.15.164:0/660534265 >> 172.21.15.164:0/660534265 conn(0x7f1eac0a3110 msgr2=0x7f1eac0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 -- 172.21.15.164:0/660534265 shutdown_connections 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 1 -- 172.21.15.164:0/660534265 wait complete. 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 10 monclient: build_initial_monmap 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.251+0000 7f1eb09ec640 10 monclient: monmap: 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:28.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 Processor -- start 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 -- start start 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient: init 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:18:28.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1eac09ab50 0x7f1eac072780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac09ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eac09d930 0x7f1eac072cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 0x7f1eac073200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1eac09ef30 con 0x7f1eac09d930 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1eac0a02b0 con 0x7f1eac09ab50 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1eac0a2940 con 0x7f1eac134580 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:28.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eb09ec640 10 monclient(hunting): authenticate will time out at 1486.739600s 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1ea9d9b640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eac09d930 msgr2=0x7f1eac072cc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1ea9d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eac09d930 0x7f1eac072cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 0x7f1eac073200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 0x7f1eac073200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60598/0 (socket says 172.21.15.164:60598) 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 learned_addr learned my addr 172.21.15.164:0/3323681811 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): get_auth_request con 0x7f1eac134580 auth_method 0 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1eac09ab50 0x7f1eac072780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaa59c640 10 monclient(hunting): get_auth_request con 0x7f1eac09ab50 auth_method 0 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient(hunting): handle_auth_done global_id 114323 payload 824 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1eac09ab50 msgr2=0x7f1eac072780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1eac09ab50 0x7f1eac072780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eac09d930 msgr2=0x7f1eac072cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1eac09d930 0x7f1eac072cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:18:28.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 monclient: found mon.noname-c 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.252+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac09c2a0 con 0x7f1eac134580 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 msgr2=0x7f1eac073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eaad9d640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 0x7f1eac073200 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c410 tx=0x7f1e9c00c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c0119a0 con 0x7f1eac134580 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.270 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e9c017070 con 0x7f1eac134580 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 5 monclient: authenticate success, global_id 114323 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 10 monclient: _renew_subs 2024-09-10T16:18:28.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eac1352c0 con 0x7f1eac134580 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c011d80 con 0x7f1eac134580 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1ea8d99640 4 set_mon_vals no callback set 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 10 monclient: _renew_subs 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.255+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eac12cdc0 con 0x7f1eac134580 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eb09ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 0x7f1eac1c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1eac0a1e40 con 0x7f1eac1c8c00 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1e9c01c460 con 0x7f1eac134580 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 0x7f1eac1c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient: get_auth_request con 0x7f1eac1c8c00 auth_method 0 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1ea2ffd640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1e8c03e1c0 0x7f1e8c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f1e9c052930 con 0x7f1eac134580 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_done global_id 114326 payload 824 2024-09-10T16:18:28.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1eaa59c640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1eac06dac0 con 0x7f1eac1c8c00 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1ea9d9b640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1e8c03e1c0 0x7f1e8c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.256+0000 7f1ea9d9b640 10 monclient: get_auth_request con 0x7f1e8c03e1c0 auth_method 0 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1eaa59c640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 msgr2=0x7f1eac1c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 0x7f1eac1c8fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea400c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4003b90 con 0x7f1eac1c8c00 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.258+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1ea4003b90 con 0x7f1eac1c8c00 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 msgr2=0x7f1eac1c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 0x7f1eac1c8fe0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea400c850 comp rx=0 tx=0).stop 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 msgr2=0x7f1eac1c8fe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:28.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea2ffd640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c8c00 0x7f1eac1c8fe0 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea400c850 comp rx=0 tx=0).stop 2024-09-10T16:18:28.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.259+0000 7f1ea9d9b640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1e8c03e1c0 0x7f1e8c040680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1eac09a180 tx=0x7f1e94030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eb09ec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 0x7f1eac06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eb09ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1eac074210 con 0x7f1eac1c9520 2024-09-10T16:18:28.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 0x7f1eac06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient: get_auth_request con 0x7f1eac1c9520 auth_method 0 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 10 monclient(hunting): handle_auth_done global_id 114329 payload 824 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.265+0000 7f1eaa59c640 1 -- 172.21.15.164:0/3323681811 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1eac077260 con 0x7f1eac1c9520 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1eaa59c640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 msgr2=0x7f1eac06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1eaa59c640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 0x7f1eac06e7d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4025070 con 0x7f1eac1c9520 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1ea2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1ea2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.267+0000 7f1ea2ffd640 10 monclient: dump: 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.282 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f1ea401f870 con 0x7f1eac1c9520 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 msgr2=0x7f1eac06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 0x7f1eac06e7d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea4004a40 comp rx=0 tx=0).stop 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 msgr2=0x7f1eac06e7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:28.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1ea2ffd640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac1c9520 0x7f1eac06e7d0 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ea4007fa0 tx=0x7f1ea4004a40 comp rx=0 tx=0).stop 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 114, 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 128836, 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:28.284 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.285 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.286 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:28.287 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.288 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.289 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.290 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:28.291 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.292 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:28.293 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.295 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.296 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.298 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:28.298 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1e8c03e1c0 msgr2=0x7f1e8c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1e8c03e1c0 0x7f1e8c040680 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f1eac09a180 tx=0x7f1e94030000 comp rx=0 tx=0).stop 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 10 monclient: shutdown 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 msgr2=0x7f1eac073200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 --2- 172.21.15.164:0/3323681811 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1eac134580 0x7f1eac073200 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00c410 tx=0x7f1e9c00c8d0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 reap_dead start 2024-09-10T16:18:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eaad9d640 1 -- 172.21.15.164:0/3323681811 reap_dead start 2024-09-10T16:18:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 shutdown_connections 2024-09-10T16:18:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 >> 172.21.15.164:0/3323681811 conn(0x7f1eac0a3110 msgr2=0x7f1eac0a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 shutdown_connections 2024-09-10T16:18:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.268+0000 7f1eb09ec640 1 -- 172.21.15.164:0/3323681811 wait complete. 2024-09-10T16:18:28.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:18:28.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:18:28.300 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:18:28.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.398+0000 7f23e292b640 10 monclient: get_monmap_and_config 2024-09-10T16:18:28.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.398+0000 7f23e292b640 10 monclient: build_initial_monmap 2024-09-10T16:18:28.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.398+0000 7f23e292b640 10 monclient: monmap: 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:28.399546+0000 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:28.399546+0000 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:28.401 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:28.402 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:28.403 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 Processor -- start 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 -- start start 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient: init 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-b con 0x7f23dc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 0x7f23dc09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-a con 0x7f23dc09d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23dc09f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:28.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23dc09a430 con 0x7f23dc09f4c0 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f23dc0a24e0 con 0x7f23dc09d160 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc0a2bf0 con 0x7f23dc09c180 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23e292b640 10 monclient(hunting): authenticate will time out at 1486.886948s 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.399+0000 7f23db7fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 msgr2=0x7f23dc09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23db7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 0x7f23dc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60630/0 (socket says 172.21.15.164:60630) 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 -- 172.21.15.164:0/799469662 learned_addr learned my addr 172.21.15.164:0/799469662 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:28.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): get_auth_request con 0x7f23dc09c180 auth_method 0 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient(hunting): handle_auth_done global_id 114332 payload 274 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 -- 172.21.15.164:0/799469662 >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 msgr2=0x7f23dc09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 --2- 172.21.15.164:0/799469662 >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 0x7f23dc09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 -- 172.21.15.164:0/799469662 >> v1:172.21.15.165:6789/0 conn(0x7f23dc09f4c0 legacy=0x7f23dc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 monclient: found mon.noname-b 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:28.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.400+0000 7f23dbfff640 1 -- 172.21.15.164:0/799469662 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc09c560 con 0x7f23dc09c180 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23dbfff640 1 -- 172.21.15.164:0/799469662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 msgr2=0x7f23dc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23dbfff640 1 --2- 172.21.15.164:0/799469662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009870 tx=0x7f23cc02a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb880566d01ba97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 1 -- 172.21.15.164:0/799469662 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc037070 con 0x7f23dc09c180 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: dump: 2024-09-10T16:18:28.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: _finish_auth 0 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:28.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:28.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 5 monclient: authenticate success, global_id 114332 2024-09-10T16:18:28.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 1 -- 172.21.15.164:0/799469662 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23cc032640 con 0x7f23dc09c180 2024-09-10T16:18:28.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:28.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:28.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 1 -- 172.21.15.164:0/799469662 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc032a10 con 0x7f23dc09c180 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 4 set_mon_vals no callback set 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23daffd640 10 monclient: dump: 2024-09-10T16:18:28.423 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 10 monclient: shutdown 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 -- 172.21.15.164:0/799469662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 msgr2=0x7f23dc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 --2- 172.21.15.164:0/799469662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009870 tx=0x7f23cc02a150 comp rx=0 tx=0).stop 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 -- 172.21.15.164:0/799469662 shutdown_connections 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 --2- 172.21.15.164:0/799469662 >> v2:172.21.15.164:3300/0 conn(0x7f23dc09d160 0x7f23dc09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 --2- 172.21.15.164:0/799469662 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc09ea40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 -- 172.21.15.164:0/799469662 >> 172.21.15.164:0/799469662 conn(0x7f23dc0a3110 msgr2=0x7f23dc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 -- 172.21.15.164:0/799469662 shutdown_connections 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.403+0000 7f23e292b640 1 -- 172.21.15.164:0/799469662 wait complete. 2024-09-10T16:18:28.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.404+0000 7f23e292b640 10 monclient: build_initial_monmap 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.404+0000 7f23e292b640 10 monclient: monmap: 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.404+0000 7f23e292b640 1 Processor -- start 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.404+0000 7f23e292b640 1 -- start start 2024-09-10T16:18:28.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.404+0000 7f23e292b640 10 monclient: init 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-c con 0x7f23dc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23dc09d160 0x7f23dc06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-a con 0x7f23dc09d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc09f4c0 0x7f23dc06f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): picked mon.noname-b con 0x7f23dc09f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc074f90 con 0x7f23dc09d160 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc074e10 con 0x7f23dc09f4c0 2024-09-10T16:18:28.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23dc13cce0 con 0x7f23dc09c180 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e292b640 10 monclient(hunting): authenticate will time out at 1486.892460s 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23db7fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23dc09d160 msgr2=0x7f23dc06ed10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23db7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23dc09d160 0x7f23dc06ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60644/0 (socket says 172.21.15.164:60644) 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 -- 172.21.15.164:0/2066286871 learned_addr learned my addr 172.21.15.164:0/2066286871 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): get_auth_request con 0x7f23dc09c180 auth_method 0 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e0ec8640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc09f4c0 0x7f23dc06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient(hunting): handle_auth_done global_id 114335 payload 824 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc09f4c0 msgr2=0x7f23dc06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc09f4c0 0x7f23dc06f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23dc09d160 msgr2=0x7f23dc06ed10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23dc09d160 0x7f23dc06ed10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 monclient: found mon.noname-c 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:28.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23dbfff640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23dc099060 con 0x7f23dc09c180 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e0ec8640 10 monclient: get_auth_request con 0x7f23dc09f4c0 auth_method 0 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.405+0000 7f23e0ec8640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc09f4c0 0x7f23dc06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.407+0000 7f23dbfff640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 msgr2=0x7f23dc06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23dbfff640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009870 tx=0x7f23cc004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc041070 con 0x7f23dc09c180 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: dump: 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.430 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 5 monclient: authenticate success, global_id 114335 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23cc03ca80 con 0x7f23dc09c180 2024-09-10T16:18:28.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23cc037040 con 0x7f23dc09c180 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 10 monclient: _renew_subs 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23dc09d620 con 0x7f23dc09c180 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23da7fc640 4 set_mon_vals no callback set 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23d8ff9640 10 monclient: dump: 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.432 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 10 monclient: _renew_subs 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.408+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23dc070230 con 0x7f23dc09c180 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f23cc044070 con 0x7f23dc09c180 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e292b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:28.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e292b640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 0x7f23dc13b240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc0a6100 con 0x7f23dc13ae60 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23d8ff9640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23c803e1c0 0x7f23c8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(405..405 src has 184..405) ==== 4815+0+0 (secure 0 0 0) 0x7f23cc073230 con 0x7f23dc09c180 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 0x7f23dc13b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient: get_auth_request con 0x7f23dc13ae60 auth_method 0 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e0ec8640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23c803e1c0 0x7f23c8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23e0ec8640 10 monclient: get_auth_request con 0x7f23c803e1c0 auth_method 0 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 10 monclient(hunting): handle_auth_done global_id 134164 payload 824 2024-09-10T16:18:28.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.409+0000 7f23db7fe640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23dc092fb0 con 0x7f23dc13ae60 2024-09-10T16:18:28.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.411+0000 7f23db7fe640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 msgr2=0x7f23dc13b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.411+0000 7f23db7fe640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 0x7f23dc13b240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0a62e0 tx=0x7f23d000b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.411+0000 7f23e0ec8640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23c803e1c0 0x7f23c8040680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f23d40044a0 tx=0x7f23d4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0016070 con 0x7f23dc13ae60 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 10 monclient: dump: 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.437 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23d0005270 con 0x7f23dc13ae60 2024-09-10T16:18:28.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 msgr2=0x7f23dc13b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 0x7f23dc13b240 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0a62e0 tx=0x7f23d000b8e0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 msgr2=0x7f23dc13b240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:28.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.412+0000 7f23d8ff9640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13ae60 0x7f23dc13b240 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f23dc0a62e0 tx=0x7f23d000b8e0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.418+0000 7f23e292b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:18:28.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.418+0000 7f23e292b640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 0x7f23dc1d7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:28.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.418+0000 7f23e292b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:28.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.418+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23dc06ccc0 con 0x7f23dc13b780 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.418+0000 7f23db7fe640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 0x7f23dc1d7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient: get_auth_request con 0x7f23dc13b780 auth_method 0 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:28.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 10 monclient(hunting): handle_auth_done global_id 134167 payload 824 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.419+0000 7f23db7fe640 1 -- 172.21.15.164:0/2066286871 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23dc09a0b0 con 0x7f23dc13b780 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23db7fe640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 msgr2=0x7f23dc1d7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23db7fe640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 0x7f23dc1d7c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f23d0013000 tx=0x7f23d00059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d0016040 con 0x7f23dc13b780 2024-09-10T16:18:28.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23d8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23d8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.421+0000 7f23d8ff9640 10 monclient: dump: 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.446 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:28.447 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:28.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4544 (secure 0 0 0) 0x7f23d0018ab0 con 0x7f23dc13b780 2024-09-10T16:18:28.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:18:28.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:28.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 msgr2=0x7f23dc1d7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 0x7f23dc1d7c80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f23d0013000 tx=0x7f23d00059d0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 msgr2=0x7f23dc1d7c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:18:28.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23d8ff9640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23dc13b780 0x7f23dc1d7c80 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f23d0013000 tx=0x7f23d00059d0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.449 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 114, 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 32825, 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:18:28.450 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.451 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.452 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.453 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.453 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.453 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.453 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:18:28.453 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.454 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.455 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.455 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.455 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.455 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.455 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:18:28.456 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:18:28.457 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:18:28.458 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:18:28.459 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:18:28.460 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.461 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:18:28.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:18:28.463 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:18:28.464 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:18:28.465 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23c803e1c0 msgr2=0x7f23c8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23c803e1c0 0x7f23c8040680 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f23d40044a0 tx=0x7f23d4009040 comp rx=0 tx=0).stop 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 10 monclient: shutdown 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 msgr2=0x7f23dc06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f23cc009870 tx=0x7f23cc004730 comp rx=0 tx=0).stop 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e0ec8640 1 -- 172.21.15.164:0/2066286871 reap_dead start 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 shutdown_connections 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 --2- 172.21.15.164:0/2066286871 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23dc09c180 0x7f23dc06e7d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 >> 172.21.15.164:0/2066286871 conn(0x7f23dc0a3110 msgr2=0x7f23dc139b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:28.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 shutdown_connections 2024-09-10T16:18:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:28.422+0000 7f23e292b640 1 -- 172.21.15.164:0/2066286871 wait complete. 2024-09-10T16:18:28.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:18:28.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:18:28.561 INFO:tasks.rados.rados.0.smithi165.stdout:1679: done (0 left) 2024-09-10T16:18:28.561 INFO:tasks.rados.rados.0.smithi165.stdout:1680: copy_from oid 9 from oid 18 current snap is 188 2024-09-10T16:18:28.561 INFO:tasks.rados.rados.0.smithi165.stdout:1681: write (excl) oid 42 current snap is 188 2024-09-10T16:18:28.561 INFO:tasks.rados.rados.0.smithi165.stdout:1681: seq_num 598 ranges {450586=474436,1343879=588206} 2024-09-10T16:18:28.566 INFO:tasks.rados.rados.0.smithi165.stdout:1681: writing smithi16534841-42 from 450586 to 925022 tid 1 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1681: writing smithi16534841-42 from 1343879 to 1932085 tid 2 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1682: copy_from oid 43 from oid 30 current snap is 188 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1681: finishing write tid 1 to smithi16534841-42 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1683: read oid 11 snap -1 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1683: expect deleted 2024-09-10T16:18:28.574 INFO:tasks.rados.rados.0.smithi165.stdout:1684: snap_remove snap 187 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: finishing write tid 2 to smithi16534841-42 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1682: finishing copy_from to smithi16534841-43 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1682: got expected ENOENT (src dne) 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1680: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1682: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1680: finishing copy_from to smithi16534841-9 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1085 (ObjNum 567 snap 183 seq_num 567) dirty exists 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: finishing write tid 3 to smithi16534841-42 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: finishing write tid 4 to smithi16534841-42 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: oid 42 updating version 0 to 633 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: oid 42 version 633 is already newer than 631 2024-09-10T16:18:29.575 INFO:tasks.rados.rados.0.smithi165.stdout:1681: oid 42 version 633 is already newer than 632 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 633 (ObjNum 598 snap 188 seq_num 598) dirty exists 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1681: left oid 42 (ObjNum 598 snap 188 seq_num 598) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1680: done (4 left) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1681: done (3 left) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1682: done (2 left) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1683: done (1 left) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1684: done (0 left) 2024-09-10T16:18:29.576 INFO:tasks.rados.rados.0.smithi165.stdout:1685: delete oid 39 current snap is 188 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1685: done (0 left) 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1686: read oid 22 snap -1 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1686: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1687: read oid 21 snap -1 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1687: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-10T16:18:29.580 INFO:tasks.rados.rados.0.smithi165.stdout:1688: delete oid 28 current snap is 188 2024-09-10T16:18:29.588 INFO:tasks.rados.rados.0.smithi165.stdout:1688: done (2 left) 2024-09-10T16:18:29.588 INFO:tasks.rados.rados.0.smithi165.stdout:1689: rollback oid 40 current snap is 188 2024-09-10T16:18:29.588 INFO:tasks.rados.rados.0.smithi165.stdout:1689: done (2 left) 2024-09-10T16:18:29.588 INFO:tasks.rados.rados.0.smithi165.stdout:1690: write (excl) oid 25 current snap is 188 2024-09-10T16:18:29.588 INFO:tasks.rados.rados.0.smithi165.stdout:1690: seq_num 601 ranges {501007=416536,1635678=465329} 2024-09-10T16:18:29.593 INFO:tasks.rados.rados.0.smithi165.stdout:1690: writing smithi16534841-25 from 501007 to 917543 tid 1 2024-09-10T16:18:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:1690: writing smithi16534841-25 from 1635678 to 2101007 tid 2 2024-09-10T16:18:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:1691: rollback oid 7 current snap is 188 2024-09-10T16:18:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:1691: done (3 left) 2024-09-10T16:18:29.600 INFO:tasks.rados.rados.0.smithi165.stdout:1692: delete oid 43 current snap is 188 2024-09-10T16:18:29.601 INFO:tasks.rados.rados.0.smithi165.stdout:1686: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:18:29.604 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 760804 2024-09-10T16:18:29.638 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 876702 2024-09-10T16:18:29.702 INFO:tasks.rados.rados.0.smithi165.stdout:1687: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-10T16:18:29.752 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1426714 2024-09-10T16:18:29.787 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:29.842 INFO:tasks.rados.rados.0.smithi165.stdout:1690: finishing write tid 1 to smithi16534841-25 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: finishing write tid 2 to smithi16534841-25 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: finishing write tid 3 to smithi16534841-25 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: finishing write tid 4 to smithi16534841-25 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: oid 25 updating version 0 to 1158 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: oid 25 updating version 1158 to 1160 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: oid 25 version 1160 is already newer than 1159 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1160 (ObjNum 601 snap 188 seq_num 601) dirty exists 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: left oid 25 (ObjNum 601 snap 188 seq_num 601) 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1686: done (3 left) 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1687: done (2 left) 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1690: done (1 left) 2024-09-10T16:18:29.843 INFO:tasks.rados.rados.0.smithi165.stdout:1692: done (0 left) 2024-09-10T16:18:29.844 INFO:tasks.rados.rados.0.smithi165.stdout:1693: write oid 23 current snap is 188 2024-09-10T16:18:29.844 INFO:tasks.rados.rados.0.smithi165.stdout:1693: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-10T16:18:29.847 INFO:tasks.rados.rados.0.smithi165.stdout:1693: writing smithi16534841-23 from 534621 to 1101655 tid 1 2024-09-10T16:18:29.853 INFO:tasks.rados.rados.0.smithi165.stdout:1693: writing smithi16534841-23 from 1636289 to 2134621 tid 2 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1694: rollback oid 33 current snap is 188 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1694: done (1 left) 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1695: read oid 36 snap -1 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1695: expect deleted 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1696: write (excl) oid 4 current snap is 188 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1693: finishing write tid 1 to smithi16534841-23 2024-09-10T16:18:29.854 INFO:tasks.rados.rados.0.smithi165.stdout:1696: seq_num 604 ranges {551428=642283,1694771=456657} 2024-09-10T16:18:29.860 INFO:tasks.rados.rados.0.smithi165.stdout:1696: writing smithi16534841-4 from 551428 to 1193711 tid 1 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1696: writing smithi16534841-4 from 1694771 to 2151428 tid 2 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1697: delete oid 46 current snap is 188 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: finishing write tid 2 to smithi16534841-23 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: finishing write tid 3 to smithi16534841-23 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: finishing write tid 4 to smithi16534841-23 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: oid 23 updating version 0 to 1103 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: oid 23 updating version 1103 to 1104 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:1693: oid 23 version 1104 is already newer than 1102 2024-09-10T16:18:29.866 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1104 (ObjNum 603 snap 188 seq_num 603) dirty exists 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1693: left oid 23 (ObjNum 603 snap 188 seq_num 603) 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1696: finishing write tid 1 to smithi16534841-4 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1693: done (3 left) 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1695: done (2 left) 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1697: done (1 left) 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1698: read oid 27 snap -1 2024-09-10T16:18:29.867 INFO:tasks.rados.rados.0.smithi165.stdout:1698: expect (ObjNum 576 snap 184 seq_num 576) 2024-09-10T16:18:29.868 INFO:tasks.rados.rados.0.smithi165.stdout:1699: delete oid 32 current snap is 188 2024-09-10T16:18:29.869 INFO:tasks.rados.rados.0.smithi165.stdout:1696: finishing write tid 2 to smithi16534841-4 2024-09-10T16:18:29.869 INFO:tasks.rados.rados.0.smithi165.stdout:1699: done (2 left) 2024-09-10T16:18:29.869 INFO:tasks.rados.rados.0.smithi165.stdout:1700: delete oid 39 current snap is 188 2024-09-10T16:18:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:1696: finishing write tid 3 to smithi16534841-4 2024-09-10T16:18:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:1696: finishing write tid 4 to smithi16534841-4 2024-09-10T16:18:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:1696: oid 4 updating version 0 to 711 2024-09-10T16:18:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:1696: oid 4 version 711 is already newer than 709 2024-09-10T16:18:29.872 INFO:tasks.rados.rados.0.smithi165.stdout:1696: oid 4 version 711 is already newer than 710 2024-09-10T16:18:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 711 (ObjNum 604 snap 188 seq_num 604) dirty exists 2024-09-10T16:18:29.873 INFO:tasks.rados.rados.0.smithi165.stdout:1696: left oid 4 (ObjNum 604 snap 188 seq_num 604) 2024-09-10T16:18:29.874 INFO:tasks.rados.rados.0.smithi165.stdout:1698: expect (ObjNum 576 snap 184 seq_num 576) 2024-09-10T16:18:29.874 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:29.916 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1696: done (2 left) 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1698: done (1 left) 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1700: done (0 left) 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1701: copy_from oid 31 from oid 46 current snap is 188 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1702: read oid 3 snap -1 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1702: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1703: copy_from oid 11 from oid 19 current snap is 188 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1704: write (excl) oid 40 current snap is 188 2024-09-10T16:18:29.977 INFO:tasks.rados.rados.0.smithi165.stdout:1704: seq_num 608 ranges {618656=543279,1812346=406310} 2024-09-10T16:18:29.984 INFO:tasks.rados.rados.0.smithi165.stdout:1704: writing smithi16534841-40 from 618656 to 1161935 tid 1 2024-09-10T16:18:29.989 INFO:tasks.rados.rados.0.smithi165.stdout:1704: writing smithi16534841-40 from 1812346 to 2218656 tid 2 2024-09-10T16:18:29.989 INFO:tasks.rados.rados.0.smithi165.stdout:1705: write oid 18 current snap is 188 2024-09-10T16:18:29.989 INFO:tasks.rados.rados.0.smithi165.stdout:1705: seq_num 609 ranges {635463=734881,1987181=248282} 2024-09-10T16:18:29.996 INFO:tasks.rados.rados.0.smithi165.stdout:1705: writing smithi16534841-18 from 635463 to 1370344 tid 1 2024-09-10T16:18:30.002 INFO:tasks.rados.rados.0.smithi165.stdout:1705: writing smithi16534841-18 from 1987181 to 2235463 tid 2 2024-09-10T16:18:30.002 INFO:tasks.rados.rados.0.smithi165.stdout:1706: read oid 34 snap -1 2024-09-10T16:18:30.002 INFO:tasks.rados.rados.0.smithi165.stdout:1706: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1707: read oid 48 snap -1 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1707: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1708: rollback oid 8 current snap is 188 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1708: done (7 left) 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1709: read oid 8 snap -1 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1709: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1710: snap_create 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1701: finishing copy_from to smithi16534841-31 2024-09-10T16:18:30.003 INFO:tasks.rados.rados.0.smithi165.stdout:1701: got expected ENOENT (src dne) 2024-09-10T16:18:30.004 INFO:tasks.rados.rados.0.smithi165.stdout:1701: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:18:30.004 INFO:tasks.rados.rados.0.smithi165.stdout:1702: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:30.008 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1046277 2024-09-10T16:18:30.101 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1210810 2024-09-10T16:18:30.136 INFO:tasks.rados.rados.0.smithi165.stdout:1703: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:18:30.136 INFO:tasks.rados.rados.0.smithi165.stdout:1704: finishing write tid 1 to smithi16534841-40 2024-09-10T16:18:30.136 INFO:tasks.rados.rados.0.smithi165.stdout:1703: finishing copy_from to smithi16534841-11 2024-09-10T16:18:30.136 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1216 (ObjNum 596 snap 186 seq_num 596) dirty exists 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: finishing write tid 2 to smithi16534841-40 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: finishing write tid 3 to smithi16534841-40 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: finishing write tid 4 to smithi16534841-40 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: oid 40 updating version 0 to 1086 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: oid 40 updating version 1086 to 1088 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: oid 40 version 1088 is already newer than 1087 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1088 (ObjNum 608 snap 188 seq_num 608) dirty exists 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1704: left oid 40 (ObjNum 608 snap 188 seq_num 608) 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1705: finishing write tid 1 to smithi16534841-18 2024-09-10T16:18:30.137 INFO:tasks.rados.rados.0.smithi165.stdout:1709: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:30.146 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2060509 2024-09-10T16:18:30.171 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.225 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1043944 2024-09-10T16:18:30.261 INFO:tasks.rados.rados.0.smithi165.stdout:1705: finishing write tid 2 to smithi16534841-18 2024-09-10T16:18:30.261 INFO:tasks.rados.rados.0.smithi165.stdout:1705: finishing write tid 3 to smithi16534841-18 2024-09-10T16:18:30.261 INFO:tasks.rados.rados.0.smithi165.stdout:1705: finishing write tid 4 to smithi16534841-18 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:1705: oid 18 updating version 0 to 1107 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:1705: oid 18 version 1107 is already newer than 1106 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:1705: oid 18 version 1107 is already newer than 1105 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1107 (ObjNum 609 snap 188 seq_num 609) dirty exists 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:1705: left oid 18 (ObjNum 609 snap 188 seq_num 609) 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:1706: expect (ObjNum 497 snap 156 seq_num 497) 2024-09-10T16:18:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.305 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.394 INFO:tasks.rados.rados.0.smithi165.stdout:1707: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:18:30.394 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.454 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.515 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:30.577 INFO:tasks.rados.rados.0.smithi165.stdout:1701: done (8 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1702: done (7 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1703: done (6 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1704: done (5 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1705: done (4 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1706: done (3 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1707: done (2 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1709: done (1 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1710: done (0 left) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1711: read oid 22 snap 188 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1711: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1712: read oid 42 snap -1 2024-09-10T16:18:30.578 INFO:tasks.rados.rados.0.smithi165.stdout:1712: expect (ObjNum 598 snap 188 seq_num 598) 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1713: read oid 13 snap -1 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1713: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1714: read oid 2 snap -1 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1714: expect (ObjNum 597 snap 186 seq_num 597) 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1715: read oid 15 snap -1 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1715: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:18:30.579 INFO:tasks.rados.rados.0.smithi165.stdout:1716: delete oid 6 current snap is 189 2024-09-10T16:18:30.585 INFO:tasks.rados.rados.0.smithi165.stdout:1712: expect (ObjNum 598 snap 188 seq_num 598) 2024-09-10T16:18:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:1714: expect (ObjNum 597 snap 186 seq_num 597) 2024-09-10T16:18:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 284791 2024-09-10T16:18:30.802 INFO:tasks.rados.rados.0.smithi165.stdout:1715: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:18:30.862 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3375312 2024-09-10T16:18:30.901 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1961812 2024-09-10T16:18:30.936 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:31.019 INFO:tasks.rados.rados.0.smithi165.stdout:1711: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:18:31.019 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:31.055 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:31.092 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:31.129 INFO:tasks.rados.rados.0.smithi165.stdout:1713: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:31.129 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:31.434 INFO:tasks.rados.rados.0.smithi165.stdout:1711: done (5 left) 2024-09-10T16:18:31.434 INFO:tasks.rados.rados.0.smithi165.stdout:1712: done (4 left) 2024-09-10T16:18:31.434 INFO:tasks.rados.rados.0.smithi165.stdout:1713: done (3 left) 2024-09-10T16:18:31.434 INFO:tasks.rados.rados.0.smithi165.stdout:1714: done (2 left) 2024-09-10T16:18:31.435 INFO:tasks.rados.rados.0.smithi165.stdout:1715: done (1 left) 2024-09-10T16:18:31.435 INFO:tasks.rados.rados.0.smithi165.stdout:1716: done (0 left) 2024-09-10T16:18:31.435 INFO:tasks.rados.rados.0.smithi165.stdout:1717: write (excl) oid 28 current snap is 189 2024-09-10T16:18:31.435 INFO:tasks.rados.rados.0.smithi165.stdout:1717: seq_num 611 ranges {669077=485379,1587792=681285} 2024-09-10T16:18:31.440 INFO:tasks.rados.rados.0.smithi165.stdout:1717: writing smithi16534841-28 from 669077 to 1154456 tid 1 2024-09-10T16:18:31.446 INFO:tasks.rados.rados.0.smithi165.stdout:1717: writing smithi16534841-28 from 1587792 to 2269077 tid 2 2024-09-10T16:18:31.446 INFO:tasks.rados.rados.0.smithi165.stdout:1718: read oid 9 snap -1 2024-09-10T16:18:31.447 INFO:tasks.rados.rados.0.smithi165.stdout:1718: expect (ObjNum 567 snap 183 seq_num 567) 2024-09-10T16:18:31.447 INFO:tasks.rados.rados.0.smithi165.stdout:1719: read oid 24 snap -1 2024-09-10T16:18:31.447 INFO:tasks.rados.rados.0.smithi165.stdout:1719: expect (ObjNum 562 snap 178 seq_num 562) 2024-09-10T16:18:31.447 INFO:tasks.rados.rados.0.smithi165.stdout:1720: snap_remove snap 188 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1720: done (3 left) 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1721: delete oid 50 current snap is 189 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1717: finishing write tid 1 to smithi16534841-28 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1717: finishing write tid 2 to smithi16534841-28 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1717: finishing write tid 3 to smithi16534841-28 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1717: finishing write tid 4 to smithi16534841-28 2024-09-10T16:18:31.975 INFO:tasks.rados.rados.0.smithi165.stdout:1717: oid 28 updating version 0 to 1020 2024-09-10T16:18:31.976 INFO:tasks.rados.rados.0.smithi165.stdout:1717: oid 28 version 1020 is already newer than 1019 2024-09-10T16:18:31.976 INFO:tasks.rados.rados.0.smithi165.stdout:1717: oid 28 version 1020 is already newer than 1018 2024-09-10T16:18:31.976 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1020 (ObjNum 611 snap 189 seq_num 611) dirty exists 2024-09-10T16:18:31.976 INFO:tasks.rados.rados.0.smithi165.stdout:1717: left oid 28 (ObjNum 611 snap 189 seq_num 611) 2024-09-10T16:18:31.976 INFO:tasks.rados.rados.0.smithi165.stdout:1718: expect (ObjNum 567 snap 183 seq_num 567) 2024-09-10T16:18:31.984 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 744089 2024-09-10T16:18:32.109 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:32.182 INFO:tasks.rados.rados.0.smithi165.stdout:1719: expect (ObjNum 562 snap 178 seq_num 562) 2024-09-10T16:18:32.182 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:32.257 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:32.332 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:32.408 INFO:tasks.rados.rados.0.smithi165.stdout:1717: done (3 left) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1718: done (2 left) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1719: done (1 left) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1721: done (0 left) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1722: copy_from oid 38 from oid 29 current snap is 189 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1723: read oid 31 snap -1 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1723: expect (ObjNum 574 snap 184 seq_num 574) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1724: read oid 11 snap -1 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1724: expect (ObjNum 596 snap 186 seq_num 596) 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1725: copy_from oid 12 from oid 10 current snap is 189 2024-09-10T16:18:32.409 INFO:tasks.rados.rados.0.smithi165.stdout:1726: copy_from oid 16 from oid 1 current snap is 189 2024-09-10T16:18:32.410 INFO:tasks.rados.rados.0.smithi165.stdout:1727: snap_create 2024-09-10T16:18:32.414 INFO:tasks.rados.rados.0.smithi165.stdout:1724: expect (ObjNum 596 snap 186 seq_num 596) 2024-09-10T16:18:32.456 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:32.533 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1974987 2024-09-10T16:18:32.533 INFO:tasks.rados.rados.0.smithi165.stdout:1723: expect (ObjNum 574 snap 184 seq_num 574) 2024-09-10T16:18:32.535 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1252495 2024-09-10T16:18:32.575 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1944884 2024-09-10T16:18:32.607 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3604860 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:1725: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:1725: finishing copy_from to smithi16534841-12 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1088 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:1722: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:1722: finishing copy_from to smithi16534841-38 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1137 (ObjNum 528 snap 163 seq_num 528) dirty exists 2024-09-10T16:18:32.611 INFO:tasks.rados.rados.0.smithi165.stdout:1726: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:18:32.612 INFO:tasks.rados.rados.0.smithi165.stdout:1726: finishing copy_from to smithi16534841-16 2024-09-10T16:18:32.612 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1002 (ObjNum 472 snap 152 seq_num 472) dirty exists 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1722: done (5 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1723: done (4 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1724: done (3 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1725: done (2 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1726: done (1 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1727: done (0 left) 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1728: read oid 3 snap -1 2024-09-10T16:18:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:1728: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:32.979 INFO:tasks.rados.rados.0.smithi165.stdout:1729: delete oid 45 current snap is 190 2024-09-10T16:18:32.980 INFO:tasks.rados.rados.0.smithi165.stdout:1728: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:18:33.006 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2984275 2024-09-10T16:18:33.095 INFO:tasks.rados.rados.0.smithi165.stdout:1728: done (1 left) 2024-09-10T16:18:33.096 INFO:tasks.rados.rados.0.smithi165.stdout:1729: done (0 left) 2024-09-10T16:18:33.096 INFO:tasks.rados.rados.0.smithi165.stdout:1730: write oid 46 current snap is 190 2024-09-10T16:18:33.096 INFO:tasks.rados.rados.0.smithi165.stdout:1730: seq_num 614 ranges {719498=711126,2046885=272613} 2024-09-10T16:18:33.102 INFO:tasks.rados.rados.0.smithi165.stdout:1730: writing smithi16534841-46 from 719498 to 1430624 tid 1 2024-09-10T16:18:33.107 INFO:tasks.rados.rados.0.smithi165.stdout:1730: writing smithi16534841-46 from 2046885 to 2319498 tid 2 2024-09-10T16:18:33.108 INFO:tasks.rados.rados.0.smithi165.stdout:1731: snap_remove snap 189 2024-09-10T16:18:33.989 INFO:tasks.rados.rados.0.smithi165.stdout:1731: done (1 left) 2024-09-10T16:18:33.990 INFO:tasks.rados.rados.0.smithi165.stdout:1732: write oid 37 current snap is 190 2024-09-10T16:18:33.990 INFO:tasks.rados.rados.0.smithi165.stdout:1730: finishing write tid 1 to smithi16534841-46 2024-09-10T16:18:33.990 INFO:tasks.rados.rados.0.smithi165.stdout:1732: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-10T16:18:33.997 INFO:tasks.rados.rados.0.smithi165.stdout:1732: writing smithi16534841-37 from 736305 to 1522680 tid 1 2024-09-10T16:18:34.001 INFO:tasks.rados.rados.0.smithi165.stdout:1732: writing smithi16534841-37 from 2105367 to 2336305 tid 2 2024-09-10T16:18:34.001 INFO:tasks.rados.rados.0.smithi165.stdout:1733: write oid 19 current snap is 190 2024-09-10T16:18:34.002 INFO:tasks.rados.rados.0.smithi165.stdout:1733: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-10T16:18:34.008 INFO:tasks.rados.rados.0.smithi165.stdout:1733: writing smithi16534841-19 from 753112 to 1331089 tid 1 2024-09-10T16:18:34.015 INFO:tasks.rados.rados.0.smithi165.stdout:1733: writing smithi16534841-19 from 1880202 to 2353112 tid 2 2024-09-10T16:18:34.015 INFO:tasks.rados.rados.0.smithi165.stdout:1730: finishing write tid 2 to smithi16534841-46 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: finishing write tid 3 to smithi16534841-46 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: finishing write tid 4 to smithi16534841-46 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: oid 46 updating version 0 to 640 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: oid 46 updating version 640 to 642 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: oid 46 version 642 is already newer than 641 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 642 (ObjNum 614 snap 190 seq_num 614) dirty exists 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: left oid 46 (ObjNum 614 snap 190 seq_num 614) 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1734: read oid 13 snap -1 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1734: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1730: done (3 left) 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1735: delete oid 24 current snap is 190 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1732: finishing write tid 1 to smithi16534841-37 2024-09-10T16:18:34.016 INFO:tasks.rados.rados.0.smithi165.stdout:1732: finishing write tid 2 to smithi16534841-37 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: finishing write tid 3 to smithi16534841-37 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: finishing write tid 4 to smithi16534841-37 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: oid 37 updating version 0 to 585 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: oid 37 updating version 585 to 586 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: oid 37 updating version 586 to 587 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 587 (ObjNum 615 snap 190 seq_num 615) dirty exists 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1732: left oid 37 (ObjNum 615 snap 190 seq_num 615) 2024-09-10T16:18:34.017 INFO:tasks.rados.rados.0.smithi165.stdout:1733: finishing write tid 1 to smithi16534841-19 2024-09-10T16:18:34.018 INFO:tasks.rados.rados.0.smithi165.stdout:1733: finishing write tid 2 to smithi16534841-19 2024-09-10T16:18:34.018 INFO:tasks.rados.rados.0.smithi165.stdout:1733: finishing write tid 3 to smithi16534841-19 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:1733: finishing write tid 4 to smithi16534841-19 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:1733: oid 19 updating version 0 to 1220 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:1733: oid 19 version 1220 is already newer than 1219 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:1733: oid 19 version 1220 is already newer than 1218 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1220 (ObjNum 616 snap 190 seq_num 616) dirty exists 2024-09-10T16:18:34.019 INFO:tasks.rados.rados.0.smithi165.stdout:1733: left oid 19 (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:18:34.020 INFO:tasks.rados.rados.0.smithi165.stdout:1732: done (3 left) 2024-09-10T16:18:34.020 INFO:tasks.rados.rados.0.smithi165.stdout:1733: done (2 left) 2024-09-10T16:18:34.020 INFO:tasks.rados.rados.0.smithi165.stdout:1735: done (1 left) 2024-09-10T16:18:34.020 INFO:tasks.rados.rados.0.smithi165.stdout:1736: read oid 15 snap -1 2024-09-10T16:18:34.020 INFO:tasks.rados.rados.0.smithi165.stdout:1736: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1737: read oid 49 snap -1 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1737: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1738: read oid 33 snap -1 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1738: expect (ObjNum 588 snap 186 seq_num 588) 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1739: rollback oid 35 current snap is 190 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1739: done (4 left) 2024-09-10T16:18:34.021 INFO:tasks.rados.rados.0.smithi165.stdout:1740: delete oid 2 current snap is 190 2024-09-10T16:18:34.023 INFO:tasks.rados.rados.0.smithi165.stdout:1740: done (4 left) 2024-09-10T16:18:34.023 INFO:tasks.rados.rados.0.smithi165.stdout:1741: rollback oid 45 current snap is 190 2024-09-10T16:18:34.023 INFO:tasks.rados.rados.0.smithi165.stdout:1741: done (4 left) 2024-09-10T16:18:34.023 INFO:tasks.rados.rados.0.smithi165.stdout:1742: delete oid 50 current snap is 190 2024-09-10T16:18:34.025 INFO:tasks.rados.rados.0.smithi165.stdout:1742: done (4 left) 2024-09-10T16:18:34.025 INFO:tasks.rados.rados.0.smithi165.stdout:1743: snap_create 2024-09-10T16:18:34.028 INFO:tasks.rados.rados.0.smithi165.stdout:1738: expect (ObjNum 588 snap 186 seq_num 588) 2024-09-10T16:18:34.272 INFO:tasks.rados.rados.0.smithi165.stdout:1736: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:18:34.272 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.492 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3205394 2024-09-10T16:18:34.497 INFO:tasks.rados.rados.0.smithi165.stdout:1734: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:34.498 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.599 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.702 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.804 INFO:tasks.rados.rados.0.smithi165.stdout:1737: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:34.804 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.873 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:34.942 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:35.013 INFO:tasks.rados.rados.0.smithi165.stdout:1734: done (4 left) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1736: done (3 left) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1737: done (2 left) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1738: done (1 left) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1743: done (0 left) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1744: read oid 50 snap 190 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1744: expect deleted 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1745: read oid 13 snap -1 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1745: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:35.014 INFO:tasks.rados.rados.0.smithi165.stdout:1746: snap_create 2024-09-10T16:18:35.017 INFO:tasks.rados.rados.0.smithi165.stdout:1745: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:35.614 INFO:tasks.rados.rados.0.smithi165.stdout:1744: done (2 left) 2024-09-10T16:18:35.614 INFO:tasks.rados.rados.0.smithi165.stdout:1745: done (1 left) 2024-09-10T16:18:35.614 INFO:tasks.rados.rados.0.smithi165.stdout:1746: done (0 left) 2024-09-10T16:18:35.614 INFO:tasks.rados.rados.0.smithi165.stdout:1747: rollback oid 39 current snap is 192 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 190 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:1748: rollback oid 17 current snap is 192 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 191 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:1749: rollback oid 18 current snap is 192 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 191 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:1750: copy_from oid 40 from oid 10 current snap is 192 2024-09-10T16:18:35.615 INFO:tasks.rados.rados.0.smithi165.stdout:1751: delete oid 24 current snap is 192 2024-09-10T16:18:35.620 INFO:tasks.rados.rados.0.smithi165.stdout:1747: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:18:35.620 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 0 (ObjNum 305 snap 0 seq_num 4062158336) dirty dne 2024-09-10T16:18:35.621 INFO:tasks.rados.rados.0.smithi165.stdout:1749: finishing rollback tid 0 to smithi16534841-18 2024-09-10T16:18:35.621 INFO:tasks.rados.rados.0.smithi165.stdout:1748: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:18:35.621 INFO:tasks.rados.rados.0.smithi165.stdout:1748: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:18:35.622 INFO:tasks.rados.rados.0.smithi165.stdout:1748: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:18:35.622 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1091 (ObjNum 533 snap 166 seq_num 533) dirty exists 2024-09-10T16:18:35.622 INFO:tasks.rados.rados.0.smithi165.stdout:1749: finishing rollback tid 1 to smithi16534841-18 2024-09-10T16:18:35.623 INFO:tasks.rados.rados.0.smithi165.stdout:1749: finishing rollback tid 2 to smithi16534841-18 2024-09-10T16:18:35.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1110 (ObjNum 609 snap 188 seq_num 609) dirty exists 2024-09-10T16:18:35.623 INFO:tasks.rados.rados.0.smithi165.stdout:1747: done (4 left) 2024-09-10T16:18:35.624 INFO:tasks.rados.rados.0.smithi165.stdout:1748: done (3 left) 2024-09-10T16:18:35.624 INFO:tasks.rados.rados.0.smithi165.stdout:1749: done (2 left) 2024-09-10T16:18:35.624 INFO:tasks.rados.rados.0.smithi165.stdout:1751: done (1 left) 2024-09-10T16:18:35.624 INFO:tasks.rados.rados.0.smithi165.stdout:1752: snap_create 2024-09-10T16:18:35.627 INFO:tasks.rados.rados.0.smithi165.stdout:1750: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:18:35.629 INFO:tasks.rados.rados.0.smithi165.stdout:1750: finishing copy_from to smithi16534841-40 2024-09-10T16:18:35.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1088 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:18:36.614 INFO:tasks.rados.rados.0.smithi165.stdout:1750: done (1 left) 2024-09-10T16:18:36.615 INFO:tasks.rados.rados.0.smithi165.stdout:1752: done (0 left) 2024-09-10T16:18:36.615 INFO:tasks.rados.rados.0.smithi165.stdout:1753: write oid 13 current snap is 193 2024-09-10T16:18:36.615 INFO:tasks.rados.rados.0.smithi165.stdout:1753: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-10T16:18:36.619 INFO:tasks.rados.rados.0.smithi165.stdout:1753: writing smithi16534841-13 from 437147 to 991369 tid 1 2024-09-10T16:18:36.627 INFO:tasks.rados.rados.0.smithi165.stdout:1753: writing smithi16534841-13 from 1656259 to 2312129 tid 2 2024-09-10T16:18:36.627 INFO:tasks.rados.rados.0.smithi165.stdout:1754: snap_create 2024-09-10T16:18:36.628 INFO:tasks.rados.rados.0.smithi165.stdout:1753: finishing write tid 1 to smithi16534841-13 2024-09-10T16:18:36.631 INFO:tasks.rados.rados.0.smithi165.stdout:1753: finishing write tid 2 to smithi16534841-13 2024-09-10T16:18:36.631 INFO:tasks.rados.rados.0.smithi165.stdout:1753: finishing write tid 3 to smithi16534841-13 2024-09-10T16:18:36.631 INFO:tasks.rados.rados.0.smithi165.stdout:1753: finishing write tid 4 to smithi16534841-13 2024-09-10T16:18:36.632 INFO:tasks.rados.rados.0.smithi165.stdout:1753: oid 13 updating version 0 to 763 2024-09-10T16:18:36.632 INFO:tasks.rados.rados.0.smithi165.stdout:1753: oid 13 updating version 763 to 765 2024-09-10T16:18:36.632 INFO:tasks.rados.rados.0.smithi165.stdout:1753: oid 13 version 765 is already newer than 764 2024-09-10T16:18:36.632 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 765 (ObjNum 621 snap 193 seq_num 621) dirty exists 2024-09-10T16:18:36.632 INFO:tasks.rados.rados.0.smithi165.stdout:1753: left oid 13 (ObjNum 621 snap 193 seq_num 621) 2024-09-10T16:18:37.619 INFO:tasks.rados.rados.0.smithi165.stdout:1753: done (1 left) 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:1754: done (0 left) 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:1755: rollback oid 40 current snap is 194 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 192 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:1756: rollback oid 49 current snap is 194 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 192 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:1757: write (excl) oid 45 current snap is 194 2024-09-10T16:18:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:1757: seq_num 622 ranges {453954=629471,1598388=515881} 2024-09-10T16:18:37.625 INFO:tasks.rados.rados.0.smithi165.stdout:1757: writing smithi16534841-45 from 453954 to 1083425 tid 1 2024-09-10T16:18:37.631 INFO:tasks.rados.rados.0.smithi165.stdout:1757: writing smithi16534841-45 from 1598388 to 2114269 tid 2 2024-09-10T16:18:37.631 INFO:tasks.rados.rados.0.smithi165.stdout:1758: write (excl) oid 42 current snap is 194 2024-09-10T16:18:37.631 INFO:tasks.rados.rados.0.smithi165.stdout:1756: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:18:37.632 INFO:tasks.rados.rados.0.smithi165.stdout:1756: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:18:37.632 INFO:tasks.rados.rados.0.smithi165.stdout:1755: finishing rollback tid 0 to smithi16534841-40 2024-09-10T16:18:37.632 INFO:tasks.rados.rados.0.smithi165.stdout:1755: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:18:37.632 INFO:tasks.rados.rados.0.smithi165.stdout:1758: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-10T16:18:37.637 INFO:tasks.rados.rados.0.smithi165.stdout:1758: writing smithi16534841-42 from 470761 to 1175481 tid 1 2024-09-10T16:18:37.644 INFO:tasks.rados.rados.0.smithi165.stdout:1758: writing smithi16534841-42 from 1656870 to 2316409 tid 2 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:1759: write oid 46 current snap is 194 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:1756: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1094 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:1755: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1091 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:1757: finishing write tid 1 to smithi16534841-45 2024-09-10T16:18:37.645 INFO:tasks.rados.rados.0.smithi165.stdout:1759: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-10T16:18:37.650 INFO:tasks.rados.rados.0.smithi165.stdout:1759: writing smithi16534841-46 from 487568 to 983890 tid 1 2024-09-10T16:18:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:1759: writing smithi16534841-46 from 1431705 to 1951255 tid 2 2024-09-10T16:18:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:1755: done (4 left) 2024-09-10T16:18:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:1756: done (3 left) 2024-09-10T16:18:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:1760: snap_create 2024-09-10T16:18:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:1757: finishing write tid 2 to smithi16534841-45 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:1757: finishing write tid 3 to smithi16534841-45 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:1757: finishing write tid 4 to smithi16534841-45 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:1757: oid 45 updating version 0 to 888 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:1757: oid 45 updating version 888 to 890 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:1757: oid 45 version 890 is already newer than 889 2024-09-10T16:18:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 890 (ObjNum 622 snap 194 seq_num 622) dirty exists 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1757: left oid 45 (ObjNum 622 snap 194 seq_num 622) 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: finishing write tid 1 to smithi16534841-42 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: finishing write tid 2 to smithi16534841-42 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: finishing write tid 3 to smithi16534841-42 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: finishing write tid 4 to smithi16534841-42 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: oid 42 updating version 0 to 1220 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: oid 42 version 1220 is already newer than 1218 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: oid 42 version 1220 is already newer than 1219 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1220 (ObjNum 623 snap 194 seq_num 623) dirty exists 2024-09-10T16:18:37.658 INFO:tasks.rados.rados.0.smithi165.stdout:1758: left oid 42 (ObjNum 623 snap 194 seq_num 623) 2024-09-10T16:18:37.659 INFO:tasks.rados.rados.0.smithi165.stdout:1759: finishing write tid 1 to smithi16534841-46 2024-09-10T16:18:37.659 INFO:tasks.rados.rados.0.smithi165.stdout:1759: finishing write tid 2 to smithi16534841-46 2024-09-10T16:18:37.660 INFO:tasks.rados.rados.0.smithi165.stdout:1759: finishing write tid 3 to smithi16534841-46 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1759: finishing write tid 4 to smithi16534841-46 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1759: oid 46 updating version 0 to 644 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1759: oid 46 updating version 644 to 645 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1759: oid 46 version 645 is already newer than 643 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 645 (ObjNum 624 snap 194 seq_num 624) dirty exists 2024-09-10T16:18:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1759: left oid 46 (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1757: done (3 left) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1758: done (2 left) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1759: done (1 left) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1760: done (0 left) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1761: read oid 12 snap -1 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1761: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1762: rollback oid 49 current snap is 195 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 194 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:1763: rollback oid 34 current snap is 195 2024-09-10T16:18:38.622 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 192 2024-09-10T16:18:38.623 INFO:tasks.rados.rados.0.smithi165.stdout:1764: write oid 43 current snap is 195 2024-09-10T16:18:38.623 INFO:tasks.rados.rados.0.smithi165.stdout:1764: seq_num 625 ranges {504375=571571,1490187=663208} 2024-09-10T16:18:38.627 INFO:tasks.rados.rados.0.smithi165.stdout:1764: writing smithi16534841-43 from 504375 to 1075946 tid 1 2024-09-10T16:18:38.634 INFO:tasks.rados.rados.0.smithi165.stdout:1764: writing smithi16534841-43 from 1490187 to 2153395 tid 2 2024-09-10T16:18:38.634 INFO:tasks.rados.rados.0.smithi165.stdout:1765: write oid 25 current snap is 195 2024-09-10T16:18:38.634 INFO:tasks.rados.rados.0.smithi165.stdout:1765: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-10T16:18:38.640 INFO:tasks.rados.rados.0.smithi165.stdout:1765: writing smithi16534841-25 from 521182 to 1168002 tid 1 2024-09-10T16:18:38.648 INFO:tasks.rados.rados.0.smithi165.stdout:1765: writing smithi16534841-25 from 1948669 to 2471888 tid 2 2024-09-10T16:18:38.648 INFO:tasks.rados.rados.0.smithi165.stdout:1766: read oid 13 snap 190 2024-09-10T16:18:38.648 INFO:tasks.rados.rados.0.smithi165.stdout:1766: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:38.648 INFO:tasks.rados.rados.0.smithi165.stdout:1767: copy_from oid 48 from oid 32 current snap is 195 2024-09-10T16:18:38.648 INFO:tasks.rados.rados.0.smithi165.stdout:1761: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:18:38.658 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1714719 2024-09-10T16:18:38.692 INFO:tasks.rados.rados.0.smithi165.stdout:1762: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:18:38.692 INFO:tasks.rados.rados.0.smithi165.stdout:1762: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1763: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1762: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1097 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1763: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1763: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 590 (ObjNum 497 snap 156 seq_num 497) dirty exists 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1761: done (6 left) 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1762: done (5 left) 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1763: done (4 left) 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1768: copy_from oid 31 from oid 46 current snap is 195 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1764: finishing write tid 1 to smithi16534841-43 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1764: finishing write tid 2 to smithi16534841-43 2024-09-10T16:18:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1764: finishing write tid 3 to smithi16534841-43 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1764: finishing write tid 4 to smithi16534841-43 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1764: oid 43 updating version 0 to 892 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1764: oid 43 version 892 is already newer than 891 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1764: oid 43 updating version 892 to 893 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 893 (ObjNum 625 snap 195 seq_num 625) dirty exists 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1764: left oid 43 (ObjNum 625 snap 195 seq_num 625) 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1765: finishing write tid 1 to smithi16534841-25 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1767: finishing copy_from to smithi16534841-48 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1767: got expected ENOENT (src dne) 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1767: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1765: finishing write tid 2 to smithi16534841-25 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1765: finishing write tid 3 to smithi16534841-25 2024-09-10T16:18:38.694 INFO:tasks.rados.rados.0.smithi165.stdout:1765: finishing write tid 4 to smithi16534841-25 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:1765: oid 25 updating version 0 to 1162 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:1765: oid 25 version 1162 is already newer than 1161 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:1765: oid 25 updating version 1162 to 1163 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1163 (ObjNum 626 snap 195 seq_num 626) dirty exists 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:1765: left oid 25 (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:18:38.695 INFO:tasks.rados.rados.0.smithi165.stdout:1766: expect (ObjNum 594 snap 186 seq_num 594) 2024-09-10T16:18:38.902 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1170589 2024-09-10T16:18:38.979 INFO:tasks.rados.rados.0.smithi165.stdout:1768: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1768: finishing copy_from to smithi16534841-31 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 645 (ObjNum 624 snap 194 seq_num 624) dirty exists 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1764: done (4 left) 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1765: done (3 left) 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1766: done (2 left) 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1767: done (1 left) 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1768: done (0 left) 2024-09-10T16:18:38.980 INFO:tasks.rados.rados.0.smithi165.stdout:1769: snap_create 2024-09-10T16:18:39.631 INFO:tasks.rados.rados.0.smithi165.stdout:1769: done (0 left) 2024-09-10T16:18:39.631 INFO:tasks.rados.rados.0.smithi165.stdout:1770: write (excl) oid 34 current snap is 196 2024-09-10T16:18:39.631 INFO:tasks.rados.rados.0.smithi165.stdout:1770: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-10T16:18:39.637 INFO:tasks.rados.rados.0.smithi165.stdout:1770: writing smithi16534841-34 from 537989 to 1260058 tid 1 2024-09-10T16:18:39.644 INFO:tasks.rados.rados.0.smithi165.stdout:1770: writing smithi16534841-34 from 1890798 to 2537989 tid 2 2024-09-10T16:18:39.644 INFO:tasks.rados.rados.0.smithi165.stdout:1771: read oid 45 snap 191 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1771: expect deleted 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1772: read oid 23 snap -1 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1772: expect (ObjNum 603 snap 188 seq_num 603) 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1773: read oid 33 snap 193 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1773: expect (ObjNum 588 snap 186 seq_num 588) 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1774: write oid 28 current snap is 196 2024-09-10T16:18:39.645 INFO:tasks.rados.rados.0.smithi165.stdout:1774: seq_num 628 ranges {554796=797318,1949280=526888} 2024-09-10T16:18:39.652 INFO:tasks.rados.rados.0.smithi165.stdout:1774: writing smithi16534841-28 from 554796 to 1352114 tid 1 2024-09-10T16:18:39.659 INFO:tasks.rados.rados.0.smithi165.stdout:1774: writing smithi16534841-28 from 1949280 to 2476168 tid 2 2024-09-10T16:18:39.659 INFO:tasks.rados.rados.0.smithi165.stdout:1775: write oid 30 current snap is 196 2024-09-10T16:18:39.659 INFO:tasks.rados.rados.0.smithi165.stdout:1775: seq_num 629 ranges {571603=472567,1607762=670546} 2024-09-10T16:18:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:1775: writing smithi16534841-30 from 571603 to 1044170 tid 1 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1775: writing smithi16534841-30 from 1607762 to 2278308 tid 2 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1776: read oid 49 snap -1 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1776: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1777: delete oid 38 current snap is 196 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1770: finishing write tid 1 to smithi16534841-34 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1770: finishing write tid 2 to smithi16534841-34 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1770: finishing write tid 3 to smithi16534841-34 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1770: finishing write tid 4 to smithi16534841-34 2024-09-10T16:18:39.672 INFO:tasks.rados.rados.0.smithi165.stdout:1770: oid 34 updating version 0 to 593 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:1770: oid 34 version 593 is already newer than 592 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:1770: oid 34 version 593 is already newer than 591 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 593 (ObjNum 627 snap 196 seq_num 627) dirty exists 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:1770: left oid 34 (ObjNum 627 snap 196 seq_num 627) 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:1772: expect (ObjNum 603 snap 188 seq_num 603) 2024-09-10T16:18:39.673 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:39.720 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 324893 2024-09-10T16:18:39.772 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 769043 2024-09-10T16:18:39.805 INFO:tasks.rados.rados.0.smithi165.stdout:1773: expect (ObjNum 588 snap 186 seq_num 588) 2024-09-10T16:18:39.805 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:39.897 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:40.075 INFO:tasks.rados.rados.0.smithi165.stdout:1774: finishing write tid 1 to smithi16534841-28 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1770: done (7 left) 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1771: done (6 left) 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1772: done (5 left) 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1773: done (4 left) 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1777: done (3 left) 2024-09-10T16:18:40.076 INFO:tasks.rados.rados.0.smithi165.stdout:1778: snap_remove snap 193 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1778: done (3 left) 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1779: read oid 46 snap -1 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1779: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1780: read oid 36 snap -1 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1780: expect deleted 2024-09-10T16:18:40.618 INFO:tasks.rados.rados.0.smithi165.stdout:1781: snap_remove snap 194 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1781: done (5 left) 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1782: read oid 48 snap -1 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1782: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1783: snap_create 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1774: finishing write tid 2 to smithi16534841-28 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1774: finishing write tid 3 to smithi16534841-28 2024-09-10T16:18:41.627 INFO:tasks.rados.rados.0.smithi165.stdout:1774: finishing write tid 4 to smithi16534841-28 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1774: oid 28 updating version 0 to 1023 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1774: oid 28 version 1023 is already newer than 1022 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1774: oid 28 version 1023 is already newer than 1021 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1023 (ObjNum 628 snap 196 seq_num 628) dirty exists 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1774: left oid 28 (ObjNum 628 snap 196 seq_num 628) 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: finishing write tid 1 to smithi16534841-30 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: finishing write tid 2 to smithi16534841-30 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: finishing write tid 3 to smithi16534841-30 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: finishing write tid 4 to smithi16534841-30 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: oid 30 updating version 0 to 1223 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: oid 30 updating version 1223 to 1224 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:1775: oid 30 updating version 1224 to 1225 2024-09-10T16:18:41.628 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1225 (ObjNum 629 snap 196 seq_num 629) dirty exists 2024-09-10T16:18:41.629 INFO:tasks.rados.rados.0.smithi165.stdout:1775: left oid 30 (ObjNum 629 snap 196 seq_num 629) 2024-09-10T16:18:41.629 INFO:tasks.rados.rados.0.smithi165.stdout:1776: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:41.693 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:41.783 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2530555 2024-09-10T16:18:41.821 INFO:tasks.rados.rados.0.smithi165.stdout:1779: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:18:41.848 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:41.889 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 880957 2024-09-10T16:18:41.916 INFO:tasks.rados.rados.0.smithi165.stdout:1782: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:18:41.944 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2899185 2024-09-10T16:18:41.967 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1605738 2024-09-10T16:18:42.012 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:42.628 INFO:tasks.rados.rados.0.smithi165.stdout:1774: done (6 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1775: done (5 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1776: done (4 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1779: done (3 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1780: done (2 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1782: done (1 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1783: done (0 left) 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1784: write oid 39 current snap is 197 2024-09-10T16:18:42.629 INFO:tasks.rados.rados.0.smithi165.stdout:1784: seq_num 631 ranges {605217=739418,2124726=480491} 2024-09-10T16:18:42.635 INFO:tasks.rados.rados.0.smithi165.stdout:1784: writing smithi16534841-39 from 605217 to 1344635 tid 1 2024-09-10T16:18:42.643 INFO:tasks.rados.rados.0.smithi165.stdout:1784: writing smithi16534841-39 from 2124726 to 2605217 tid 2 2024-09-10T16:18:42.643 INFO:tasks.rados.rados.0.smithi165.stdout:1785: snap_remove snap 196 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1785: done (1 left) 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1786: rollback oid 34 current snap is 197 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 192 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1784: finishing write tid 1 to smithi16534841-39 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1784: finishing write tid 2 to smithi16534841-39 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1784: finishing write tid 3 to smithi16534841-39 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1784: finishing write tid 4 to smithi16534841-39 2024-09-10T16:18:43.637 INFO:tasks.rados.rados.0.smithi165.stdout:1784: oid 39 updating version 0 to 1113 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:1784: oid 39 version 1113 is already newer than 1111 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:1784: oid 39 version 1113 is already newer than 1112 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1113 (ObjNum 631 snap 197 seq_num 631) dirty exists 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:1784: left oid 39 (ObjNum 631 snap 197 seq_num 631) 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:1784: done (1 left) 2024-09-10T16:18:43.638 INFO:tasks.rados.rados.0.smithi165.stdout:1787: snap_remove snap 191 2024-09-10T16:18:44.638 INFO:tasks.rados.rados.0.smithi165.stdout:1787: done (1 left) 2024-09-10T16:18:44.638 INFO:tasks.rados.rados.0.smithi165.stdout:1788: write (excl) oid 1 current snap is 197 2024-09-10T16:18:44.638 INFO:tasks.rados.rados.0.smithi165.stdout:1786: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:18:44.638 INFO:tasks.rados.rados.0.smithi165.stdout:1786: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:18:44.638 INFO:tasks.rados.rados.0.smithi165.stdout:1786: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:18:44.639 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 596 (ObjNum 497 snap 156 seq_num 497) dirty exists 2024-09-10T16:18:44.639 INFO:tasks.rados.rados.0.smithi165.stdout:1788: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-10T16:18:44.644 INFO:tasks.rados.rados.0.smithi165.stdout:1788: writing smithi16534841-1 from 622024 to 1036691 tid 1 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1788: writing smithi16534841-1 from 1783208 to 2201081 tid 2 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1786: done (1 left) 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1789: read oid 42 snap -1 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1789: expect (ObjNum 623 snap 194 seq_num 623) 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1790: read oid 39 snap -1 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1790: expect (ObjNum 631 snap 197 seq_num 631) 2024-09-10T16:18:44.651 INFO:tasks.rados.rados.0.smithi165.stdout:1791: rollback oid 35 current snap is 197 2024-09-10T16:18:44.652 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 190 2024-09-10T16:18:44.652 INFO:tasks.rados.rados.0.smithi165.stdout:1792: snap_remove snap 192 2024-09-10T16:18:45.619 INFO:tasks.rados.rados.0.smithi165.stdout:1792: done (4 left) 2024-09-10T16:18:45.619 INFO:tasks.rados.rados.0.smithi165.stdout:1793: snap_create 2024-09-10T16:18:45.619 INFO:tasks.rados.rados.0.smithi165.stdout:1788: finishing write tid 1 to smithi16534841-1 2024-09-10T16:18:45.619 INFO:tasks.rados.rados.0.smithi165.stdout:1791: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:18:45.619 INFO:tasks.rados.rados.0.smithi165.stdout:1791: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:18:45.620 INFO:tasks.rados.rados.0.smithi165.stdout:1791: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:18:45.620 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 768 (ObjNum 540 snap 170 seq_num 540) dirty exists 2024-09-10T16:18:45.620 INFO:tasks.rados.rados.0.smithi165.stdout:1789: expect (ObjNum 623 snap 194 seq_num 623) 2024-09-10T16:18:45.620 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:45.687 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1790212 2024-09-10T16:18:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:1788: finishing write tid 2 to smithi16534841-1 2024-09-10T16:18:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:1788: finishing write tid 3 to smithi16534841-1 2024-09-10T16:18:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:1788: finishing write tid 4 to smithi16534841-1 2024-09-10T16:18:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:1788: oid 1 updating version 0 to 1116 2024-09-10T16:18:45.742 INFO:tasks.rados.rados.0.smithi165.stdout:1788: oid 1 version 1116 is already newer than 1114 2024-09-10T16:18:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:1788: oid 1 version 1116 is already newer than 1115 2024-09-10T16:18:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1116 (ObjNum 632 snap 197 seq_num 632) dirty exists 2024-09-10T16:18:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:1788: left oid 1 (ObjNum 632 snap 197 seq_num 632) 2024-09-10T16:18:45.743 INFO:tasks.rados.rados.0.smithi165.stdout:1790: expect (ObjNum 631 snap 197 seq_num 631) 2024-09-10T16:18:45.788 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2305783 2024-09-10T16:18:46.620 INFO:tasks.rados.rados.0.smithi165.stdout:1788: done (4 left) 2024-09-10T16:18:46.620 INFO:tasks.rados.rados.0.smithi165.stdout:1789: done (3 left) 2024-09-10T16:18:46.620 INFO:tasks.rados.rados.0.smithi165.stdout:1790: done (2 left) 2024-09-10T16:18:46.620 INFO:tasks.rados.rados.0.smithi165.stdout:1791: done (1 left) 2024-09-10T16:18:46.620 INFO:tasks.rados.rados.0.smithi165.stdout:1793: done (0 left) 2024-09-10T16:18:46.621 INFO:tasks.rados.rados.0.smithi165.stdout:1794: copy_from oid 34 from oid 3 current snap is 198 2024-09-10T16:18:46.621 INFO:tasks.rados.rados.0.smithi165.stdout:1795: copy_from oid 23 from oid 46 current snap is 198 2024-09-10T16:18:46.621 INFO:tasks.rados.rados.0.smithi165.stdout:1796: read oid 49 snap -1 2024-09-10T16:18:46.621 INFO:tasks.rados.rados.0.smithi165.stdout:1796: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:46.621 INFO:tasks.rados.rados.0.smithi165.stdout:1797: snap_remove snap 197 2024-09-10T16:18:47.619 INFO:tasks.rados.rados.0.smithi165.stdout:1797: done (3 left) 2024-09-10T16:18:47.619 INFO:tasks.rados.rados.0.smithi165.stdout:1798: write oid 47 current snap is 198 2024-09-10T16:18:47.619 INFO:tasks.rados.rados.0.smithi165.stdout:1798: seq_num 633 ranges {638831=489916,1841690=677884} 2024-09-10T16:18:47.625 INFO:tasks.rados.rados.0.smithi165.stdout:1798: writing smithi16534841-47 from 638831 to 1128747 tid 1 2024-09-10T16:18:47.633 INFO:tasks.rados.rados.0.smithi165.stdout:1798: writing smithi16534841-47 from 1841690 to 2519574 tid 2 2024-09-10T16:18:47.634 INFO:tasks.rados.rados.0.smithi165.stdout:1799: read oid 25 snap 190 2024-09-10T16:18:47.634 INFO:tasks.rados.rados.0.smithi165.stdout:1799: expect (ObjNum 601 snap 188 seq_num 601) 2024-09-10T16:18:47.634 INFO:tasks.rados.rados.0.smithi165.stdout:1800: snap_remove snap 195 2024-09-10T16:18:48.468 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:18:48.469 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:18:48.469 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:18:48.472 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:18:48.472 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:18:48.473 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.575+0000 7faf60f89640 10 monclient: get_monmap_and_config 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.575+0000 7faf60f89640 10 monclient: build_initial_monmap 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.575+0000 7faf60f89640 10 monclient: monmap: 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:48.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:48.576821+0000 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:48.576821+0000 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:48.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.576+0000 7faf60f89640 1 Processor -- start 2024-09-10T16:18:48.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.576+0000 7faf60f89640 1 -- start start 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.576+0000 7faf60f89640 10 monclient: init 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf5c06d160 0x7faf5c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-c con 0x7faf5c06d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf5c06d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf5c13d060 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c09ce70 con 0x7faf5c13d060 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf5c09d150 con 0x7faf5c06d540 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faf5c09cc20 con 0x7faf5c06d160 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:48.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf60f89640 10 monclient(hunting): authenticate will time out at 1507.064399s 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7faf5c13d060 0x7faf5c13f450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45862/0 (socket says 172.21.15.164:45862) 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5ad9d640 1 -- 172.21.15.164:0/928311559 learned_addr learned my addr 172.21.15.164:0/928311559 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 1 --2- 172.21.15.164:0/928311559 >> v2:172.21.15.165:3300/0 conn(0x7faf5c06d160 0x7faf5c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): get_auth_request con 0x7faf5c06d160 auth_method 0 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.577+0000 7faf5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 10 monclient(hunting): handle_auth_done global_id 134170 payload 274 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 1 -- 172.21.15.164:0/928311559 >> v1:172.21.15.165:6789/0 conn(0x7faf5c06d540 legacy=0x7faf5c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:48.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 1 -- 172.21.15.164:0/928311559 >> v1:172.21.15.164:6789/0 conn(0x7faf5c13d060 legacy=0x7faf5c13f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:48.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:18:48.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 1 monclient: found mon.noname-c 2024-09-10T16:18:48.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:48.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:18:48.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.578+0000 7faf5a59c640 1 -- 172.21.15.164:0/928311559 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0a4df0 con 0x7faf5c06d160 2024-09-10T16:18:48.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5a59c640 1 -- 172.21.15.164:0/928311559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 msgr2=0x7faf5c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:48.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5a59c640 1 --2- 172.21.15.164:0/928311559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c134e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85bb797397598a9b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 1 -- 172.21.15.164:0/928311559 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c037070 con 0x7faf5c06d160 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: dump: 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:48.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: _finish_auth 0 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: _check_auth_tickets 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 5 monclient: authenticate success, global_id 134170 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 1 -- 172.21.15.164:0/928311559 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf4c032640 con 0x7faf5c06d160 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 1 -- 172.21.15.164:0/928311559 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf4c032a10 con 0x7faf5c06d160 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 4 set_mon_vals no callback set 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:48.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf5959a640 10 monclient: dump: 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 10 monclient: shutdown 2024-09-10T16:18:48.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 1 -- 172.21.15.164:0/928311559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 msgr2=0x7faf5c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.581+0000 7faf60f89640 1 --2- 172.21.15.164:0/928311559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c134e00 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf4c0097a0 tx=0x7faf4c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 1 -- 172.21.15.164:0/928311559 shutdown_connections 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 1 --2- 172.21.15.164:0/928311559 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c134e00 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 1 -- 172.21.15.164:0/928311559 >> 172.21.15.164:0/928311559 conn(0x7faf5c0aacb0 msgr2=0x7faf5c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 1 -- 172.21.15.164:0/928311559 shutdown_connections 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 1 -- 172.21.15.164:0/928311559 wait complete. 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 10 monclient: build_initial_monmap 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.582+0000 7faf60f89640 10 monclient: monmap: 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:48.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 Processor -- start 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 -- start start 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient: init 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-b con 0x7faf5c06d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:48.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-a con 0x7faf5c06d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 0x7faf5c13f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): picked mon.noname-c con 0x7faf5c13d060 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c140cf0 con 0x7faf5c06d540 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faf5c140b70 con 0x7faf5c06d160 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:48.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf5c140e70 con 0x7faf5c13d060 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf60f89640 10 monclient(hunting): authenticate will time out at 1507.070671s 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 0x7faf5c13f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46060/0 (socket says 172.21.15.164:46060) 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 -- 172.21.15.164:0/1793408043 learned_addr learned my addr 172.21.15.164:0/1793408043 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 10 monclient(hunting): get_auth_request con 0x7faf5c06d540 auth_method 0 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): get_auth_request con 0x7faf5c13d060 auth_method 0 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:48.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 10 monclient(hunting): get_auth_request con 0x7faf5c06d160 auth_method 0 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 msgr2=0x7faf5c13ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 msgr2=0x7faf5c13ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:48.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 msgr2=0x7faf5c12c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 msgr2=0x7faf5c12c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf5a59c640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.583+0000 7faf59d9b640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 10 monclient(hunting): handle_auth_done global_id 114338 payload 824 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 msgr2=0x7faf5c12c8c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 msgr2=0x7faf5c13ed50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 monclient: found mon.noname-c 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:48.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.584+0000 7faf5ad9d640 1 -- 172.21.15.164:0/1793408043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf5c0a4aa0 con 0x7faf5c13d060 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf5ad9d640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 msgr2=0x7faf5c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf5ad9d640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 0x7faf5c13f290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faf5400cac0 tx=0x7faf5400cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf5400ad80 con 0x7faf5c13d060 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: dump: 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: _finish_auth 0 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: _check_auth_tickets 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf54012400 con 0x7faf5c13d060 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 5 monclient: authenticate success, global_id 114338 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 10 monclient: _renew_subs 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:48.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf5c06e6b0 con 0x7faf5c13d060 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf540127f0 con 0x7faf5c13d060 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf4affd640 10 monclient: dump: 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:48.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf58d99640 4 set_mon_vals no callback set 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 10 monclient: _renew_subs 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.586+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf5c12da70 con 0x7faf5c13d060 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faf540129d0 con 0x7faf5c13d060 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf60f89640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf60f89640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf60f89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf4affd640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 0x7faf3c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf5c131940 con 0x7faf5c13d060 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.587+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(426..426 src has 184..426) ==== 4783+0+0 (secure 0 0 0) 0x7faf540533a0 con 0x7faf5c13d060 2024-09-10T16:18:48.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.588+0000 7faf5a59c640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 0x7faf3c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:48.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.588+0000 7faf5a59c640 10 monclient: get_auth_request con 0x7faf3c03e210 auth_method 0 2024-09-10T16:18:48.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.590+0000 7faf5a59c640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 0x7faf3c0406d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf4c031000 tx=0x7faf4c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:48.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.592+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faf5c131940 con 0x7faf5c13d060 2024-09-10T16:18:48.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.592+0000 7faf4affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:48.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.592+0000 7faf4affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:48.619 INFO:tasks.rados.rados.0.smithi165.stdout:1800: done (5 left) 2024-09-10T16:18:48.620 INFO:tasks.rados.rados.0.smithi165.stdout:1801: write (excl) oid 2 current snap is 198 2024-09-10T16:18:48.620 INFO:tasks.rados.rados.0.smithi165.stdout:1801: seq_num 634 ranges {655638=565165,1900172=421542} 2024-09-10T16:18:48.626 INFO:tasks.rados.rados.0.smithi165.stdout:1801: writing smithi16534841-2 from 655638 to 1220803 tid 1 2024-09-10T16:18:48.633 INFO:tasks.rados.rados.0.smithi165.stdout:1801: writing smithi16534841-2 from 1900172 to 2321714 tid 2 2024-09-10T16:18:48.633 INFO:tasks.rados.rados.0.smithi165.stdout:1802: copy_from oid 14 from oid 29 current snap is 198 2024-09-10T16:18:48.633 INFO:tasks.rados.rados.0.smithi165.stdout:1796: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:18:48.633 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:48.712 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1482553 2024-09-10T16:18:48.822 INFO:tasks.rados.rados.0.smithi165.stdout:1795: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:18:48.822 INFO:tasks.rados.rados.0.smithi165.stdout:1794: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:18:48.822 INFO:tasks.rados.rados.0.smithi165.stdout:1794: finishing copy_from to smithi16534841-34 2024-09-10T16:18:48.822 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1127 (ObjNum 538 snap 169 seq_num 538) dirty exists 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1795: finishing copy_from to smithi16534841-23 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 645 (ObjNum 624 snap 194 seq_num 624) dirty exists 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: finishing write tid 1 to smithi16534841-47 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1794: done (6 left) 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1795: done (5 left) 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1796: done (4 left) 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1803: snap_create 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: finishing write tid 2 to smithi16534841-47 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: finishing write tid 3 to smithi16534841-47 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: finishing write tid 4 to smithi16534841-47 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: oid 47 updating version 0 to 1145 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: oid 47 version 1145 is already newer than 1143 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:1798: oid 47 version 1145 is already newer than 1144 2024-09-10T16:18:48.823 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1145 (ObjNum 633 snap 198 seq_num 633) dirty exists 2024-09-10T16:18:48.824 INFO:tasks.rados.rados.0.smithi165.stdout:1798: left oid 47 (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:18:48.824 INFO:tasks.rados.rados.0.smithi165.stdout:1799: expect (ObjNum 601 snap 188 seq_num 601) 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf60f89640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf60f89640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf60f89640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faf5c134e00 con 0x7faf5c13d060 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf4affd640 1 -- 172.21.15.164:0/1793408043 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7faf5c06e6b0 con 0x7faf5c13d060 2024-09-10T16:18:48.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf4affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:48.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.848+0000 7faf4affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:48.851 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":114,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 msgr2=0x7faf3c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 0x7faf3c0406d0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7faf4c031000 tx=0x7faf4c02e000 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 10 monclient: shutdown 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 msgr2=0x7faf5c13f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 0x7faf5c13f290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7faf5400cac0 tx=0x7faf5400cec0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 shutdown_connections 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf3c03e210 0x7faf3c0406d0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf5c13d060 0x7faf5c13f290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf5c06d540 0x7faf5c13ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 --2- 172.21.15.164:0/1793408043 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf5c06d160 0x7faf5c12c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:48.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 >> 172.21.15.164:0/1793408043 conn(0x7faf5c0aacb0 msgr2=0x7faf5c1327d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:48.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 shutdown_connections 2024-09-10T16:18:48.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:48.851+0000 7faf60f89640 1 -- 172.21.15.164:0/1793408043 wait complete. 2024-09-10T16:18:48.864 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:48.918 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 738477 2024-09-10T16:18:48.950 INFO:tasks.rados.rados.0.smithi165.stdout:1802: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1802: finishing copy_from to smithi16534841-14 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1137 (ObjNum 528 snap 163 seq_num 528) dirty exists 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: finishing write tid 1 to smithi16534841-2 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: finishing write tid 2 to smithi16534841-2 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: finishing write tid 3 to smithi16534841-2 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: finishing write tid 4 to smithi16534841-2 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: oid 2 updating version 0 to 1148 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: oid 2 version 1148 is already newer than 1146 2024-09-10T16:18:48.951 INFO:tasks.rados.rados.0.smithi165.stdout:1801: oid 2 version 1148 is already newer than 1147 2024-09-10T16:18:48.952 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1148 (ObjNum 634 snap 198 seq_num 634) dirty exists 2024-09-10T16:18:48.952 INFO:tasks.rados.rados.0.smithi165.stdout:1801: left oid 2 (ObjNum 634 snap 198 seq_num 634) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1798: done (4 left) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1799: done (3 left) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1801: done (2 left) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1802: done (1 left) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1803: done (0 left) 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1804: rollback oid 44 current snap is 199 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 190 2024-09-10T16:18:49.628 INFO:tasks.rados.rados.0.smithi165.stdout:1805: write oid 46 current snap is 199 2024-09-10T16:18:49.629 INFO:tasks.rados.rados.0.smithi165.stdout:1805: seq_num 635 ranges {672445=640414,1842301=681553} 2024-09-10T16:18:49.634 INFO:tasks.rados.rados.0.smithi165.stdout:1805: writing smithi16534841-46 from 672445 to 1312859 tid 1 2024-09-10T16:18:49.641 INFO:tasks.rados.rados.0.smithi165.stdout:1805: writing smithi16534841-46 from 1842301 to 2523854 tid 2 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:1806: read oid 21 snap -1 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:1806: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:1807: snap_create 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:1804: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 0 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:18:49.642 INFO:tasks.rados.rados.0.smithi165.stdout:1805: finishing write tid 1 to smithi16534841-46 2024-09-10T16:18:49.644 INFO:tasks.rados.rados.0.smithi165.stdout:1805: finishing write tid 2 to smithi16534841-46 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: finishing write tid 3 to smithi16534841-46 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: finishing write tid 4 to smithi16534841-46 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: oid 46 updating version 0 to 646 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: oid 46 updating version 646 to 648 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: oid 46 version 648 is already newer than 647 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 648 (ObjNum 635 snap 199 seq_num 635) dirty exists 2024-09-10T16:18:49.647 INFO:tasks.rados.rados.0.smithi165.stdout:1805: left oid 46 (ObjNum 635 snap 199 seq_num 635) 2024-09-10T16:18:49.652 INFO:tasks.rados.rados.0.smithi165.stdout:1806: expect (ObjNum 579 snap 185 seq_num 579) 2024-09-10T16:18:49.652 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:18:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:1804: done (3 left) 2024-09-10T16:18:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:1805: done (2 left) 2024-09-10T16:18:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:1806: done (1 left) 2024-09-10T16:18:50.620 INFO:tasks.rados.rados.0.smithi165.stdout:1807: done (0 left) 2024-09-10T16:18:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:1808: copy_from oid 41 from oid 5 current snap is 200 2024-09-10T16:18:50.621 INFO:tasks.rados.rados.0.smithi165.stdout:1809: snap_remove snap 190 2024-09-10T16:18:51.629 INFO:tasks.rados.rados.0.smithi165.stdout:1809: done (1 left) 2024-09-10T16:18:51.629 INFO:tasks.rados.rados.0.smithi165.stdout:1810: snap_remove snap 198 2024-09-10T16:18:51.865 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:51.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.971+0000 7f2d051c2640 10 monclient: get_monmap_and_config 2024-09-10T16:18:51.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.971+0000 7f2d051c2640 10 monclient: build_initial_monmap 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.971+0000 7f2d051c2640 10 monclient: monmap: 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:51.973377+0000 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:51.973377+0000 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:51.974 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:51.975 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 Processor -- start 2024-09-10T16:18:51.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 -- start start 2024-09-10T16:18:51.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient: init 2024-09-10T16:18:51.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-a con 0x7f2d0006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-b con 0x7f2d000a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2d00137ca0 0x7f2d00138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-c con 0x7f2d00137ca0 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0006db60 con 0x7f2d0006cb70 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0006cf50 con 0x7f2d00137ca0 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0009ce50 con 0x7f2d000a46a0 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2d051c2640 10 monclient(hunting): authenticate will time out at 1510.460645s 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46082/0 (socket says 172.21.15.164:46082) 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59914/0 (socket says 172.21.15.164:59914) 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 -- 172.21.15.164:0/3538143552 learned_addr learned my addr 172.21.15.164:0/3538143552 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 10 monclient(hunting): get_auth_request con 0x7f2d0006cb70 auth_method 0 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): get_auth_request con 0x7f2d000a46a0 auth_method 0 2024-09-10T16:18:51.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 -- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 msgr2=0x7f2d00134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 -- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 msgr2=0x7f2d00134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cfed9d640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.972+0000 7f2cff59e640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.165:3300/0 conn(0x7f2d00137ca0 0x7f2d00138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cff59e640 10 monclient(hunting): get_auth_request con 0x7f2d00137ca0 auth_method 0 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cff59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_done global_id 114341 payload 274 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 -- 172.21.15.164:0/3538143552 >> v2:172.21.15.165:3300/0 conn(0x7f2d00137ca0 msgr2=0x7f2d00138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.165:3300/0 conn(0x7f2d00137ca0 0x7f2d00138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 -- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 msgr2=0x7f2d00134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 10 monclient: _finish_hunting 0 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 monclient: found mon.noname-b 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:18:51.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.973+0000 7f2cfe59c640 1 -- 172.21.15.164:0/3538143552 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0009cb70 con 0x7f2d000a46a0 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.975+0000 7f2cfe59c640 1 -- 172.21.15.164:0/3538143552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 msgr2=0x7f2d000a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.975+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/3538143552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d00074d10 tx=0x7f2cf402a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c595aa14b35ea73 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 1 -- 172.21.15.164:0/3538143552 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037070 con 0x7f2d000a46a0 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: dump: 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:51.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: _finish_auth 0 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 1 -- 172.21.15.164:0/3538143552 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf4032440 con 0x7f2d000a46a0 2024-09-10T16:18:51.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 5 monclient: authenticate success, global_id 114341 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 10 monclient: get_monmap_and_config success 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 1 -- 172.21.15.164:0/3538143552 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4032810 con 0x7f2d000a46a0 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 4 set_mon_vals no callback set 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2cfdd9b640 10 monclient: dump: 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:51.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 10 monclient: shutdown 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 -- 172.21.15.164:0/3538143552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 msgr2=0x7f2d000a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 --2- 172.21.15.164:0/3538143552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f2d00074d10 tx=0x7f2cf402a000 comp rx=0 tx=0).stop 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 -- 172.21.15.164:0/3538143552 shutdown_connections 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.165:3300/0 conn(0x7f2d00137ca0 0x7f2d00138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 --2- 172.21.15.164:0/3538143552 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d000a4ab0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 --2- 172.21.15.164:0/3538143552 >> v2:172.21.15.164:3300/0 conn(0x7f2d0006cb70 0x7f2d00134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 -- 172.21.15.164:0/3538143552 >> 172.21.15.164:0/3538143552 conn(0x7f2d000aa900 msgr2=0x7f2d000aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 -- 172.21.15.164:0/3538143552 shutdown_connections 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.976+0000 7f2d051c2640 1 -- 172.21.15.164:0/3538143552 wait complete. 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.977+0000 7f2d051c2640 10 monclient: build_initial_monmap 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.977+0000 7f2d051c2640 10 monclient: monmap: 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:18:51.991 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.977+0000 7f2d051c2640 1 Processor -- start 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 -- start start 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient: init 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d0006cb70 0x7f2d00131350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-b con 0x7f2d0006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-c con 0x7f2d000a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): picked mon.noname-a con 0x7f2d00137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:18:51.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2d001387f0 con 0x7f2d00137ca0 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2d00138670 con 0x7f2d0006cb70 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2d00138970 con 0x7f2d000a46a0 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2d051c2640 10 monclient(hunting): authenticate will time out at 1510.466576s 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59926/0 (socket says 172.21.15.164:59926) 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 10 monclient(hunting): get_auth_request con 0x7f2d00137ca0 auth_method 0 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 1 -- 172.21.15.164:0/2297869052 learned_addr learned my addr 172.21.15.164:0/2297869052 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:18:51.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): get_auth_request con 0x7f2d000a46a0 auth_method 0 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 msgr2=0x7f2d0009e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:18:51.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 msgr2=0x7f2d0009e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d0006cb70 0x7f2d00131350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cff59e640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): get_auth_request con 0x7f2d0006cb70 auth_method 0 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.978+0000 7f2cfe59c640 10 monclient(hunting): handle_auth_done global_id 114344 payload 824 2024-09-10T16:18:51.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d0006cb70 msgr2=0x7f2d00131350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d0006cb70 0x7f2d00131350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 msgr2=0x7f2d0009e680 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 10 monclient: _finish_hunting 0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 monclient: found mon.noname-c 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.979+0000 7f2cfe59c640 1 -- 172.21.15.164:0/2297869052 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d000a1830 con 0x7f2d000a46a0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2cfe59c640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 msgr2=0x7f2d00131890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2cfe59c640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402a5f0 tx=0x7f2cf40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4042030 con 0x7f2d000a46a0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:51.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: dump: 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: _finish_auth 0 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:18:51.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cf403d680 con 0x7f2d000a46a0 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 5 monclient: authenticate success, global_id 114344 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 10 monclient: _renew_subs 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d000a1b10 con 0x7f2d000a46a0 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4037040 con 0x7f2d000a46a0 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2ce77fe640 10 monclient: dump: 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 10 monclient: _renew_subs 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2cfd59a640 4 set_mon_vals no callback set 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.981+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d00131dd0 con 0x7f2d000a46a0 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2cf4045070 con 0x7f2d000a46a0 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2ce77fe640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 0x7f2cec040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2d051c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:51.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(430..430 src has 184..430) ==== 4815+0+0 (secure 0 0 0) 0x7f2cf4073420 con 0x7f2d000a46a0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2d051c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2d051c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d000a4ab0 con 0x7f2d000a46a0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2cfed9d640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 0x7f2cec040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.982+0000 7f2cfed9d640 10 monclient: get_auth_request con 0x7f2cec03e240 auth_method 0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.985+0000 7f2cfed9d640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 0x7f2cec040700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009100 tx=0x7f2cf0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:18:52.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.987+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2cf40463b0 con 0x7f2d000a46a0 2024-09-10T16:18:52.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.987+0000 7f2ce77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:18:52.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:51.987+0000 7f2ce77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.228+0000 7f2d051c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.228+0000 7f2d051c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.228+0000 7f2d051c2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.228+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d000a1ce0 con 0x7f2d000a46a0 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.229+0000 7f2ce77fe640 1 -- 172.21.15.164:0/2297869052 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f2cf403e420 con 0x7f2d000a46a0 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.229+0000 7f2ce77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:18:52.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.229+0000 7f2ce77fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:18:52.232 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":114,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:18:52.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 msgr2=0x7f2cec040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:52.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 0x7f2cec040700 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2cf0009100 tx=0x7f2cf0008040 comp rx=0 tx=0).stop 2024-09-10T16:18:52.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 10 monclient: shutdown 2024-09-10T16:18:52.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 msgr2=0x7f2d00131890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402a5f0 tx=0x7f2cf40046c0 comp rx=0 tx=0).stop 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 shutdown_connections 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2cec03e240 0x7f2cec040700 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2d00137ca0 0x7f2d0009e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2d000a46a0 0x7f2d00131890 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 --2- 172.21.15.164:0/2297869052 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2d0006cb70 0x7f2d00131350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 >> 172.21.15.164:0/2297869052 conn(0x7f2d000aa900 msgr2=0x7f2d00133930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 shutdown_connections 2024-09-10T16:18:52.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:52.231+0000 7f2d051c2640 1 -- 172.21.15.164:0/2297869052 wait complete. 2024-09-10T16:18:52.636 INFO:tasks.rados.rados.0.smithi165.stdout:1810: done (1 left) 2024-09-10T16:18:52.636 INFO:tasks.rados.rados.0.smithi165.stdout:1811: delete oid 45 current snap is 200 2024-09-10T16:18:52.636 INFO:tasks.rados.rados.0.smithi165.stdout:1808: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:18:52.637 INFO:tasks.rados.rados.0.smithi165.stdout:1808: finishing copy_from to smithi16534841-41 2024-09-10T16:18:52.637 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 573 (ObjNum 397 snap 130 seq_num 397) dirty exists 2024-09-10T16:18:52.640 INFO:tasks.rados.rados.0.smithi165.stdout:1808: done (1 left) 2024-09-10T16:18:52.640 INFO:tasks.rados.rados.0.smithi165.stdout:1811: done (0 left) 2024-09-10T16:18:52.640 INFO:tasks.rados.rados.0.smithi165.stdout:1812: snap_remove snap 199 2024-09-10T16:18:53.646 INFO:tasks.rados.rados.0.smithi165.stdout:1812: done (0 left) 2024-09-10T16:18:53.646 INFO:tasks.rados.rados.0.smithi165.stdout:1813: copy_from oid 50 from oid 27 current snap is 200 2024-09-10T16:18:53.646 INFO:tasks.rados.rados.0.smithi165.stdout:1814: rollback oid 25 current snap is 200 2024-09-10T16:18:53.646 INFO:tasks.rados.rados.0.smithi165.stdout:1814: done (1 left) 2024-09-10T16:18:53.646 INFO:tasks.rados.rados.0.smithi165.stdout:1815: snap_create 2024-09-10T16:18:53.661 INFO:tasks.rados.rados.0.smithi165.stdout:1813: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:18:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:1813: finishing copy_from to smithi16534841-50 2024-09-10T16:18:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1131 (ObjNum 576 snap 184 seq_num 576) dirty exists 2024-09-10T16:18:54.649 INFO:tasks.rados.rados.0.smithi165.stdout:1813: done (1 left) 2024-09-10T16:18:54.650 INFO:tasks.rados.rados.0.smithi165.stdout:1815: done (0 left) 2024-09-10T16:18:54.650 INFO:tasks.rados.rados.0.smithi165.stdout:1816: snap_create 2024-09-10T16:18:55.244 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:18:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.352+0000 7f40851cd640 10 monclient: get_monmap_and_config 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.352+0000 7f40851cd640 10 monclient: build_initial_monmap 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.352+0000 7f40851cd640 10 monclient: monmap: 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:18:55.354383+0000 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:18:55.354383+0000 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:18:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:18:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:18:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:18:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:18:55.356 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 Processor -- start 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 -- start start 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient: init 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:18:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f408009c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f408009d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-b con 0x7f408013b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408009a430 con 0x7f408009d160 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408009f860 con 0x7f408009c180 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f40800a24e0 con 0x7f408013b5d0 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f40851cd640 10 monclient(hunting): authenticate will time out at 1513.841931s 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.353+0000 7f4077fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f408009d160 0x7f408009f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38212/0 (socket says 172.21.15.164:38212) 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 learned_addr learned my addr 172.21.15.164:0/3181125872 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:55.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f408013b5d0 auth_method 0 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.354+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.356+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v1:172.21.15.164:6789/0 conn(0x7f408009d160 legacy=0x7f408009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.356+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v1:172.21.15.165:6789/0 conn(0x7f408009c180 legacy=0x7f408009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:18:55.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:55.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f408013b5d0 auth_method 0 2024-09-10T16:18:55.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:55.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:55.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:55.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:18:55.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:55.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:55.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.555+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f408013b5d0 auth_method 0 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:55.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:55.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:55.956+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f408013b5d0 auth_method 0 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:56.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:18:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:56.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:56.757+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:18:58.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): tick 2024-09-10T16:18:58.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:18:58.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 monclient(hunting): continuing hunt 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 msgr2=0x7f408013d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f408013b5d0 0x7f408013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 >> v1:172.21.15.165:6789/0 conn(0x7f408009c180 legacy=0x7f408009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 >> v1:172.21.15.164:6789/0 conn(0x7f408009d160 legacy=0x7f408009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): picked mon.noname-a con 0x7f4060003820 addr v2:172.21.15.164:3300/0 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): picked mon.noname-c con 0x7f4060006390 addr v2:172.21.15.165:3300/0 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f40800a24e0 con 0x7f4060003820 2024-09-10T16:18:58.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4060008ed0 con 0x7f4060006390 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407d59a640 10 monclient(hunting): _renew_subs 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060003820 auth_method 0 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.354+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060003820 auth_method 0 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.555+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:18:58.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060003820 auth_method 0 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:58.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:58.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:58.956+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:18:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060003820 auth_method 0 2024-09-10T16:18:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:18:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:18:59.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:18:59.758+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): tick 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 monclient(hunting): continuing hunt 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 msgr2=0x7f4060005ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060003820 0x7f4060005ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T16:19:01.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 reap_dead start 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): picked mon.noname-c con 0x7f4060006390 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 0x7f40600092e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): picked mon.noname-b con 0x7f4060003c30 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060009820 0x7f406000bc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): picked mon.noname-a con 0x7f4060009820 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f40800a24e0 con 0x7f4060009820 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4060008ed0 con 0x7f4060006390 2024-09-10T16:19:01.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 1 -- 172.21.15.164:0/3181125872 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f406000c300 con 0x7f4060003c30 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407d59a640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.164:3300/0 conn(0x7f4060009820 0x7f406000bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 0x7f40600092e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f4060009820 auth_method 0 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060003c30 auth_method 0 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f4077fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 10 monclient(hunting): handle_auth_done global_id 144099 payload 274 2024-09-10T16:19:01.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 msgr2=0x7f40600092e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 0x7f40600092e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.355+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 msgr2=0x7f4060008810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 1 monclient: found mon.noname-a 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40800a2bf0 con 0x7f4060009820 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f4077fff640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 0x7f40600092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:01.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.356+0000 7f407ed9d640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407f59e640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060009820 msgr2=0x7f406000bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407f59e640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060009820 0x7f406000bc10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f408009a7a0 tx=0x7f407802afb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a79dc88032e1df server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 1 -- 172.21.15.164:0/3181125872 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078031070 con 0x7f4060009820 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:01.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: dump: 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 1 -- 172.21.15.164:0/3181125872 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f407802bcd0 con 0x7f4060009820 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 1 -- 172.21.15.164:0/3181125872 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078035620 con 0x7f4060009820 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f40851cd640 5 monclient: authenticate success, global_id 144099 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f40851cd640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f40851cd640 4 set_mon_vals no callback set 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.359+0000 7f407e59c640 10 monclient: dump: 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.372 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 10 monclient: shutdown 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 -- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060009820 msgr2=0x7f406000bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060009820 0x7f406000bc10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f408009a7a0 tx=0x7f407802afb0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 -- 172.21.15.164:0/3181125872 shutdown_connections 2024-09-10T16:19:01.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060009820 0x7f406000bc10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 --2- 172.21.15.164:0/3181125872 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060003c30 0x7f40600092e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 --2- 172.21.15.164:0/3181125872 >> v2:172.21.15.165:3300/0 conn(0x7f4060006390 0x7f4060008810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 -- 172.21.15.164:0/3181125872 >> 172.21.15.164:0/3181125872 conn(0x7f40800a3110 msgr2=0x7f40800a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 -- 172.21.15.164:0/3181125872 shutdown_connections 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.360+0000 7f40851cd640 1 -- 172.21.15.164:0/3181125872 wait complete. 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.361+0000 7f40851cd640 10 monclient: build_initial_monmap 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.361+0000 7f40851cd640 10 monclient: monmap: 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:01.374 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.361+0000 7f40851cd640 1 Processor -- start 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.361+0000 7f40851cd640 1 -- start start 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.361+0000 7f40851cd640 10 monclient: init 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4060003c30 0x7f408008df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-b con 0x7f4060003c30 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:01.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-a con 0x7f4060006390 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): picked mon.noname-c con 0x7f4060009820 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40800a2230 con 0x7f4060006390 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40800a20b0 con 0x7f4060003c30 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4080074e10 con 0x7f4060009820 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f40851cd640 10 monclient(hunting): authenticate will time out at 1519.849376s 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34882/0 (socket says 172.21.15.164:34882) 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 -- 172.21.15.164:0/2408265904 learned_addr learned my addr 172.21.15.164:0/2408265904 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 10 monclient(hunting): get_auth_request con 0x7f4060006390 auth_method 0 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): get_auth_request con 0x7f4060009820 auth_method 0 2024-09-10T16:19:01.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4060003c30 0x7f408008df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): get_auth_request con 0x7f4060003c30 auth_method 0 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 msgr2=0x7f408008e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 msgr2=0x7f408008e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f4077fff640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient(hunting): handle_auth_done global_id 114350 payload 824 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4060003c30 msgr2=0x7f408008df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4060003c30 0x7f408008df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 msgr2=0x7f408008e4c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 monclient: found mon.noname-c 2024-09-10T16:19:01.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.362+0000 7f407f59e640 1 -- 172.21.15.164:0/2408265904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4080090940 con 0x7f4060009820 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f407f59e640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 msgr2=0x7f408008ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f407f59e640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f407802aff0 tx=0x7f4078004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078044020 con 0x7f4060009820 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: dump: 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4078040670 con 0x7f4060009820 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 5 monclient: authenticate success, global_id 114350 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 10 monclient: _renew_subs 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4080090c20 con 0x7f4060009820 2024-09-10T16:19:01.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4078040a60 con 0x7f4060009820 2024-09-10T16:19:01.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:01.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40777fe640 10 monclient: dump: 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.384 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f407dd9b640 4 set_mon_vals no callback set 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 10 monclient: _renew_subs 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.365+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f408009f2c0 con 0x7f4060009820 2024-09-10T16:19:01.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.366+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4078047020 con 0x7f4060009820 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.366+0000 7f40851cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.366+0000 7f40851cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.366+0000 7f40851cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.366+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f408009d930 con 0x7f4060009820 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.367+0000 7f40777fe640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 0x7f4060042020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.367+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(433..433 src has 184..433) ==== 4787+0+0 (secure 0 0 0) 0x7f40780745b0 con 0x7f4060009820 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.367+0000 7f407ed9d640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 0x7f4060042020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.367+0000 7f407ed9d640 10 monclient: get_auth_request con 0x7f406003fb60 auth_method 0 2024-09-10T16:19:01.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.370+0000 7f407ed9d640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 0x7f4060042020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4070007af0 tx=0x7f4070009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.372+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f407804e030 con 0x7f4060009820 2024-09-10T16:19:01.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.372+0000 7f40777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.372+0000 7f40777fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.614+0000 7f40851cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.614+0000 7f40851cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.614+0000 7f40851cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.614+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4080073f70 con 0x7f4060009820 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.615+0000 7f40777fe640 1 -- 172.21.15.164:0/2408265904 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f40780483b0 con 0x7f4060009820 2024-09-10T16:19:01.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.615+0000 7f40777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:01.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.615+0000 7f40777fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:01.617 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 msgr2=0x7f4060042020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 0x7f4060042020 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4070007af0 tx=0x7f4070009040 comp rx=0 tx=0).stop 2024-09-10T16:19:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 10 monclient: shutdown 2024-09-10T16:19:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 msgr2=0x7f408008ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f407802aff0 tx=0x7f4078004a40 comp rx=0 tx=0).stop 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 shutdown_connections 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f406003fb60 0x7f4060042020 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4060009820 0x7f408008ea00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4060006390 0x7f408008e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 --2- 172.21.15.164:0/2408265904 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4060003c30 0x7f408008df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 >> 172.21.15.164:0/2408265904 conn(0x7f40800a3110 msgr2=0x7f408013a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 shutdown_connections 2024-09-10T16:19:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.617+0000 7f40851cd640 1 -- 172.21.15.164:0/2408265904 wait complete. 2024-09-10T16:19:01.629 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:19:01.629 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:19:01.630 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:19:01.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.729+0000 7fe9ba3c6640 10 monclient: get_monmap_and_config 2024-09-10T16:19:01.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.729+0000 7fe9ba3c6640 10 monclient: build_initial_monmap 2024-09-10T16:19:01.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.729+0000 7fe9ba3c6640 10 monclient: monmap: 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:01.730166+0000 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:01.730166+0000 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:01.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:01.732 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 Processor -- start 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 -- start start 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient: init 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe9b406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 0x7fe9b413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b4134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 0x7fe9b413f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b413d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b409ce50 con 0x7fe9b406cb70 2024-09-10T16:19:01.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe9b406db60 con 0x7fe9b4134f70 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b406cf50 con 0x7fe9b413d560 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9ba3c6640 10 monclient(hunting): authenticate will time out at 1520.217620s 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 0x7fe9b413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b3fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe9b406cb70 0x7fe9b40a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38216/0 (socket says 172.21.15.164:38216) 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b3fff640 1 -- 172.21.15.164:0/3276182855 learned_addr learned my addr 172.21.15.164:0/3276182855 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): get_auth_request con 0x7fe9b413d560 auth_method 0 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/3276182855 >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 0x7fe9b413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request con 0x7fe9b4134f70 auth_method 0 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.730+0000 7fe9b8963640 10 monclient(hunting): handle_auth_done global_id 114353 payload 274 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 1 -- 172.21.15.164:0/3276182855 >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 msgr2=0x7fe9b413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 1 --2- 172.21.15.164:0/3276182855 >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 0x7fe9b413d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 1 -- 172.21.15.164:0/3276182855 >> v1:172.21.15.164:6789/0 conn(0x7fe9b406cb70 legacy=0x7fe9b40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 1 monclient: found mon.noname-b 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b8963640 1 -- 172.21.15.164:0/3276182855 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b4135380 con 0x7fe9b413d560 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.731+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/3276182855 >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 0x7fe9b413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.733+0000 7fe9b8963640 1 -- 172.21.15.164:0/3276182855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 msgr2=0x7fe9b413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.733+0000 7fe9b8963640 1 --2- 172.21.15.164:0/3276182855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 0x7fe9b413f950 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009dc0 tx=0x7fe9ac02a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=393991ba00370bc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 1 -- 172.21.15.164:0/3276182855 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac007950 con 0x7fe9b413d560 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: dump: 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.743 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 1 -- 172.21.15.164:0/3276182855 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9ac007af0 con 0x7fe9b413d560 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 5 monclient: authenticate success, global_id 114353 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 1 -- 172.21.15.164:0/3276182855 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9ac031400 con 0x7fe9b413d560 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 4 set_mon_vals no callback set 2024-09-10T16:19:01.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9b2ffd640 10 monclient: dump: 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 10 monclient: shutdown 2024-09-10T16:19:01.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/3276182855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 msgr2=0x7fe9b413f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/3276182855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 0x7fe9b413f950 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac009dc0 tx=0x7fe9ac02a4d0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/3276182855 shutdown_connections 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/3276182855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b413d560 0x7fe9b413f950 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/3276182855 >> v2:172.21.15.165:3300/0 conn(0x7fe9b4134f70 0x7fe9b413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/3276182855 >> 172.21.15.164:0/3276182855 conn(0x7fe9b40aa900 msgr2=0x7fe9b40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/3276182855 shutdown_connections 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.736+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/3276182855 wait complete. 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.737+0000 7fe9ba3c6640 10 monclient: build_initial_monmap 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.737+0000 7fe9ba3c6640 10 monclient: monmap: 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:01.746 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.737+0000 7fe9ba3c6640 1 Processor -- start 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 -- start start 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient: init 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:01.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 0x7fe9b413c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9b406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9b4134f70 0x7fe9b412e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9b4134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b413d560 0x7fe9b412e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9b413d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b40a4db0 con 0x7fe9b413d560 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b409d130 con 0x7fe9b4134f70 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b413fe90 con 0x7fe9b406cb70 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9ba3c6640 10 monclient(hunting): authenticate will time out at 1520.225521s 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 0x7fe9b413c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b413d560 0x7fe9b412e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 0x7fe9b413c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34900/0 (socket says 172.21.15.164:34900) 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): get_auth_request con 0x7fe9b413d560 auth_method 0 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 -- 172.21.15.164:0/977755456 learned_addr learned my addr 172.21.15.164:0/977755456 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): get_auth_request con 0x7fe9b406cb70 auth_method 0 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9b4134f70 0x7fe9b412e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request con 0x7fe9b4134f70 auth_method 0 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient(hunting): handle_auth_done global_id 114356 payload 824 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9b4134f70 msgr2=0x7fe9b412e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9b4134f70 0x7fe9b412e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b413d560 msgr2=0x7fe9b412e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b413d560 0x7fe9b412e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 monclient: found mon.noname-c 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b3fff640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b413a4a0 con 0x7fe9b406cb70 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe9b4134f70 0x7fe9b412e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.738+0000 7fe9b8963640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b413d560 0x7fe9b412e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b3fff640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 msgr2=0x7fe9b413c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b3fff640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 0x7fe9b413c6a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400b460 tx=0x7fe9a400b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4015020 con 0x7fe9b406cb70 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: dump: 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 5 monclient: authenticate success, global_id 114356 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a4011ac0 con 0x7fe9b406cb70 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a40103e0 con 0x7fe9b406cb70 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 10 monclient: _renew_subs 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b406e720 con 0x7fe9b406cb70 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b0ff9640 10 monclient: dump: 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.753 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9b1ffb640 4 set_mon_vals no callback set 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 10 monclient: _renew_subs 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:01.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.741+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b412ee60 con 0x7fe9b406cb70 2024-09-10T16:19:01.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9ba3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 0x7fe9b4139190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b40a46a0 con 0x7fe9b4138db0 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe9a4018020 con 0x7fe9b406cb70 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 0x7fe9b4139190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient: get_auth_request con 0x7fe9b4138db0 auth_method 0 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b0ff9640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe99403e210 0x7fe9940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_done global_id 144108 payload 824 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b37fe640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9b413bac0 con 0x7fe9b4138db0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(433..433 src has 184..433) ==== 4787+0+0 (secure 0 0 0) 0x7fe9a4052e80 con 0x7fe9b406cb70 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b8963640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe99403e210 0x7fe9940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.742+0000 7fe9b8963640 10 monclient: get_auth_request con 0x7fe99403e210 auth_method 0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b37fe640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 msgr2=0x7fe9b4139190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 0x7fe9b4139190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe99c006fd0 tx=0x7fe99c010b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c01a030 con 0x7fe9b4138db0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:01.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.744+0000 7fe9b0ff9640 10 monclient: dump: 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe99c00ebe0 con 0x7fe9b4138db0 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:01.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 msgr2=0x7fe9b4139190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 0x7fe9b4139190 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe99c006fd0 tx=0x7fe99c010b70 comp rx=0 tx=0).stop 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 msgr2=0x7fe9b4139190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b0ff9640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b4138db0 0x7fe9b4139190 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fe99c006fd0 tx=0x7fe99c010b70 comp rx=0 tx=0).stop 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.745+0000 7fe9b8963640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe99403e210 0x7fe9940406d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac031000 tx=0x7fe9ac02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9ba3c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 0x7fe9b41302e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9ba3c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b40770f0 con 0x7fe9b41396d0 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 0x7fe9b41302e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient: get_auth_request con 0x7fe9b41396d0 auth_method 0 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 10 monclient(hunting): handle_auth_done global_id 144111 payload 824 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.751+0000 7fe9b37fe640 1 -- 172.21.15.164:0/977755456 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9b4139ae0 con 0x7fe9b41396d0 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b37fe640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 msgr2=0x7fe9b41302e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b37fe640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 0x7fe9b41302e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe99c008010 tx=0x7fe99c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe99c00f040 con 0x7fe9b41396d0 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.753+0000 7fe9b0ff9640 10 monclient: dump: 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.763 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fe99c020c20 con 0x7fe9b41396d0 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 msgr2=0x7fe9b41302e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 0x7fe9b41302e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fe99c008010 tx=0x7fe99c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 msgr2=0x7fe9b41302e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b0ff9640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe9b41396d0 0x7fe9b41302e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 13196, 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:01.765 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.766 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.767 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:01.768 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.769 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:01.770 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.771 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:01.772 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.773 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.774 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:01.775 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.776 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.778 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:01.778 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:01.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe99403e210 msgr2=0x7fe9940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe99403e210 0x7fe9940406d0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe9ac031000 tx=0x7fe9ac02f000 comp rx=0 tx=0).stop 2024-09-10T16:19:01.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 10 monclient: shutdown 2024-09-10T16:19:01.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 msgr2=0x7fe9b413c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 --2- 172.21.15.164:0/977755456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe9b406cb70 0x7fe9b413c6a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fe9a400b460 tx=0x7fe9a400b920 comp rx=0 tx=0).stop 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b8963640 1 -- 172.21.15.164:0/977755456 reap_dead start 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9b8963640 1 -- 172.21.15.164:0/977755456 reap_dead start 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 shutdown_connections 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 >> 172.21.15.164:0/977755456 conn(0x7fe9b40aa900 msgr2=0x7fe9b4133550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 shutdown_connections 2024-09-10T16:19:01.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.754+0000 7fe9ba3c6640 1 -- 172.21.15.164:0/977755456 wait complete. 2024-09-10T16:19:01.780 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:01.780 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:19:01.781 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:19:01.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.877+0000 7fa613f65640 10 monclient: get_monmap_and_config 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.877+0000 7fa613f65640 10 monclient: build_initial_monmap 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.877+0000 7fa613f65640 10 monclient: monmap: 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:01.878391+0000 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:01.878391+0000 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:01.879 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:01.880 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:01.880 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:01.880 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 Processor -- start 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 -- start start 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient: init 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa60c09c180 0x7fa60c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-c con 0x7fa60c09c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa60c09d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:01.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa60c13a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60c09a430 con 0x7fa60c09d160 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa60c0a24e0 con 0x7fa60c13a5c0 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa60c0a2bf0 con 0x7fa60c09c180 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa613f65640 10 monclient(hunting): authenticate will time out at 1520.365749s 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa611500640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa60c09d160 0x7fa60c09ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38232/0 (socket says 172.21.15.164:38232) 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.878+0000 7fa611500640 1 -- 172.21.15.164:0/1092874005 learned_addr learned my addr 172.21.15.164:0/1092874005 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 1 --2- 172.21.15.164:0/1092874005 >> v2:172.21.15.165:3300/0 conn(0x7fa60c09c180 0x7fa60c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): get_auth_request con 0x7fa60c09c180 auth_method 0 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient(hunting): handle_auth_done global_id 134194 payload 274 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 1 -- 172.21.15.164:0/1092874005 >> v1:172.21.15.165:6789/0 conn(0x7fa60c13a5c0 legacy=0x7fa60c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 1 -- 172.21.15.164:0/1092874005 >> v1:172.21.15.164:6789/0 conn(0x7fa60c09d160 legacy=0x7fa60c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 1 monclient: found mon.noname-c 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:19:01.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.879+0000 7fa611d01640 1 -- 172.21.15.164:0/1092874005 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60c09c560 con 0x7fa60c09c180 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.882+0000 7fa611d01640 1 -- 172.21.15.164:0/1092874005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c09c180 msgr2=0x7fa60c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.882+0000 7fa611d01640 1 --2- 172.21.15.164:0/1092874005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c09c180 0x7fa60c09ea40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097d0 tx=0x7fa60802a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ad4a58b9230a72f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 1 -- 172.21.15.164:0/1092874005 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608037070 con 0x7fa60c09c180 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: dump: 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.891 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 1 -- 172.21.15.164:0/1092874005 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa608032440 con 0x7fa60c09c180 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 1 -- 172.21.15.164:0/1092874005 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608032810 con 0x7fa60c09c180 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 5 monclient: authenticate success, global_id 134194 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:01.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 4 set_mon_vals no callback set 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa610cff640 10 monclient: dump: 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 10 monclient: shutdown 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 -- 172.21.15.164:0/1092874005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c09c180 msgr2=0x7fa60c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 --2- 172.21.15.164:0/1092874005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c09c180 0x7fa60c09ea40 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa6080097d0 tx=0x7fa60802a000 comp rx=0 tx=0).stop 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 -- 172.21.15.164:0/1092874005 shutdown_connections 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 --2- 172.21.15.164:0/1092874005 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c09c180 0x7fa60c09ea40 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 -- 172.21.15.164:0/1092874005 >> 172.21.15.164:0/1092874005 conn(0x7fa60c0a3110 msgr2=0x7fa60c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 -- 172.21.15.164:0/1092874005 shutdown_connections 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.883+0000 7fa613f65640 1 -- 172.21.15.164:0/1092874005 wait complete. 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.884+0000 7fa613f65640 10 monclient: build_initial_monmap 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.884+0000 7fa613f65640 10 monclient: monmap: 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:01.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.884+0000 7fa613f65640 1 Processor -- start 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.884+0000 7fa613f65640 1 -- start start 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.884+0000 7fa613f65640 10 monclient: init 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:19:01.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-a con 0x7fa60c09c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c09d160 0x7fa60c08f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-c con 0x7fa60c09d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c13a5c0 0x7fa60c08fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): picked mon.noname-b con 0x7fa60c13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c074f90 con 0x7fa60c09c180 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa60c074e10 con 0x7fa60c13a5c0 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa60c09f690 con 0x7fa60c09d160 2024-09-10T16:19:01.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa613f65640 10 monclient(hunting): authenticate will time out at 1520.372159s 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42712/0 (socket says 172.21.15.164:42712) 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 -- 172.21.15.164:0/2481956026 learned_addr learned my addr 172.21.15.164:0/2481956026 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): get_auth_request con 0x7fa60c09c180 auth_method 0 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c09d160 0x7fa60c08f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): get_auth_request con 0x7fa60c09d160 auth_method 0 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c13a5c0 0x7fa60c08fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): get_auth_request con 0x7fa60c13a5c0 auth_method 0 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa612502640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient(hunting): handle_auth_done global_id 144114 payload 824 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c09d160 msgr2=0x7fa60c08f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c09d160 0x7fa60c08f8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c13a5c0 msgr2=0x7fa60c08fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa60c13a5c0 0x7fa60c08fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient: _finish_hunting 0 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 monclient: found mon.noname-a 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611d01640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60c0a5160 con 0x7fa60c09c180 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.885+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c09d160 0x7fa60c08f8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:01.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa611d01640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 msgr2=0x7fa60c099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa611d01640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6080098f0 tx=0x7fa6080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608042020 con 0x7fa60c09c180 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.900 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa613f65640 5 monclient: authenticate success, global_id 144114 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa613f65640 10 monclient: _renew_subs 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa60803d680 con 0x7fa60c09c180 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa613f65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.887+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa60c092f70 con 0x7fa60c09c180 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa608037040 con 0x7fa60c09c180 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 10 monclient: _renew_subs 2024-09-10T16:19:01.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa60c0911d0 con 0x7fa60c09c180 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fffff640 4 set_mon_vals no callback set 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 0x7fa60c090350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa60c092c90 con 0x7fa60c02e5c0 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa608045020 con 0x7fa60c09c180 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 0x7fa60c090350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient: get_auth_request con 0x7fa60c02e5c0 auth_method 0 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.888+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa611500640 10 monclient(hunting): handle_auth_done global_id 114362 payload 824 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa611500640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa60c02e9a0 con 0x7fa60c02e5c0 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa5fe7fc640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa5f003e210 0x7fa5f00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(433..433 src has 184..433) ==== 4787+0+0 (secure 0 0 0) 0x7fa6080734a0 con 0x7fa60c09c180 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa612502640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa5f003e210 0x7fa5f00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.889+0000 7fa612502640 10 monclient: get_auth_request con 0x7fa5f003e210 auth_method 0 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa611500640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 msgr2=0x7fa60c090350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 0x7fa60c090350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600007700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa600017020 con 0x7fa60c02e5c0 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.907 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa60000fc40 con 0x7fa60c02e5c0 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 msgr2=0x7fa60c090350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 0x7fa60c090350 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600007700 comp rx=0 tx=0).stop 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 msgr2=0x7fa60c090350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa5fe7fc640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c02e5c0 0x7fa60c090350 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600007700 comp rx=0 tx=0).stop 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.891+0000 7fa612502640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa5f003e210 0x7fa5f00406d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0914b0 tx=0x7fa604003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.897+0000 7fa613f65640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.897+0000 7fa613f65640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 0x7fa60c064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.897+0000 7fa613f65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.897+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa60c09ea40 con 0x7fa60c063d50 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 0x7fa60c064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient: get_auth_request con 0x7fa60c063d50 auth_method 0 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:01.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 10 monclient(hunting): handle_auth_done global_id 114365 payload 824 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.898+0000 7fa611500640 1 -- 172.21.15.164:0/2481956026 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa60c0935f0 con 0x7fa60c063d50 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa611500640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 msgr2=0x7fa60c064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa611500640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 0x7fa60c064160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60001e2e0 con 0x7fa60c063d50 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 10 monclient: dump: 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.910 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fa600013880 con 0x7fa60c063d50 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 msgr2=0x7fa60c064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 0x7fa60c064160 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600004940 comp rx=0 tx=0).stop 2024-09-10T16:19:01.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 msgr2=0x7fa60c064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.900+0000 7fa5fe7fc640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa60c063d50 0x7fa60c064160 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fa60c09d5b0 tx=0x7fa600004940 comp rx=0 tx=0).stop 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 162469, 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:01.912 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.913 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.914 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:01.915 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:01.916 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.917 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:01.920 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:01.921 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:01.922 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.923 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:01.924 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:01.925 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:01.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa5f003e210 msgr2=0x7fa5f00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa5f003e210 0x7fa5f00406d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0914b0 tx=0x7fa604003d10 comp rx=0 tx=0).stop 2024-09-10T16:19:01.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 10 monclient: shutdown 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 msgr2=0x7fa60c099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6080098f0 tx=0x7fa6080046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa612502640 1 -- 172.21.15.164:0/2481956026 reap_dead start 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 shutdown_connections 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 --2- 172.21.15.164:0/2481956026 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa60c09c180 0x7fa60c099ae0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 >> 172.21.15.164:0/2481956026 conn(0x7fa60c0a3110 msgr2=0x7fa60c139bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 shutdown_connections 2024-09-10T16:19:01.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:01.901+0000 7fa613f65640 1 -- 172.21.15.164:0/2481956026 wait complete. 2024-09-10T16:19:01.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:01.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:19:01.928 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:19:02.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.026+0000 7f36ef105640 10 monclient: get_monmap_and_config 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.026+0000 7f36ef105640 10 monclient: build_initial_monmap 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.026+0000 7f36ef105640 10 monclient: monmap: 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:02.027393+0000 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:02.027393+0000 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:02.028 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:02.029 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:02.029 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 Processor -- start 2024-09-10T16:19:02.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 -- start start 2024-09-10T16:19:02.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient: init 2024-09-10T16:19:02.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:02.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f36e806d160 0x7f36e8134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-a con 0x7f36e806d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 0x7f36e80a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-c con 0x7f36e806d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f36e80a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36e809cb90 con 0x7f36e80a4c20 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f36e809ce70 con 0x7f36e806d160 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f36e809d150 con 0x7f36e806d540 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ef105640 10 monclient(hunting): authenticate will time out at 1520.514873s 2024-09-10T16:19:02.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f36e806d160 0x7f36e8134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f36e806d160 0x7f36e8134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42720/0 (socket says 172.21.15.164:42720) 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 1 -- 172.21.15.164:0/1154197795 learned_addr learned my addr 172.21.15.164:0/1154197795 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 10 monclient(hunting): get_auth_request con 0x7f36e806d160 auth_method 0 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.027+0000 7f36ecea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 1 --2- 172.21.15.164:0/1154197795 >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 0x7f36e80a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 10 monclient(hunting): get_auth_request con 0x7f36e806d540 auth_method 0 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient(hunting): handle_auth_done global_id 144117 payload 274 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 1 -- 172.21.15.164:0/1154197795 >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 msgr2=0x7f36e80a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1154197795 >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 0x7f36e80a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 1 -- 172.21.15.164:0/1154197795 >> v1:172.21.15.165:6789/0 conn(0x7f36e80a4c20 legacy=0x7f36e8137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient: _finish_hunting 0 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 1 monclient: found mon.noname-a 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:02.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36ecea1640 1 -- 172.21.15.164:0/1154197795 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e8135340 con 0x7f36e806d160 2024-09-10T16:19:02.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.028+0000 7f36e7fff640 1 --2- 172.21.15.164:0/1154197795 >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 0x7f36e80a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.030+0000 7f36ecea1640 1 -- 172.21.15.164:0/1154197795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d160 msgr2=0x7f36e8134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1154197795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d160 0x7f36e8134e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f36e80a9d40 tx=0x7f36dc02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ebc5e19e6d0086 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 1 -- 172.21.15.164:0/1154197795 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc037070 con 0x7f36e806d160 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:02.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: dump: 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:02.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 1 -- 172.21.15.164:0/1154197795 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36dc032640 con 0x7f36e806d160 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 5 monclient: authenticate success, global_id 144117 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 4 set_mon_vals no callback set 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 1 -- 172.21.15.164:0/1154197795 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc032a10 con 0x7f36e806d160 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36e77fe640 10 monclient: dump: 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.045 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 10 monclient: shutdown 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 -- 172.21.15.164:0/1154197795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d160 msgr2=0x7f36e8134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 --2- 172.21.15.164:0/1154197795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d160 0x7f36e8134e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f36e80a9d40 tx=0x7f36dc02a0b0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 -- 172.21.15.164:0/1154197795 shutdown_connections 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 --2- 172.21.15.164:0/1154197795 >> v2:172.21.15.165:3300/0 conn(0x7f36e806d540 0x7f36e80a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 --2- 172.21.15.164:0/1154197795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d160 0x7f36e8134e00 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 -- 172.21.15.164:0/1154197795 >> 172.21.15.164:0/1154197795 conn(0x7f36e80aacb0 msgr2=0x7f36e80ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 -- 172.21.15.164:0/1154197795 shutdown_connections 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.031+0000 7f36ef105640 1 -- 172.21.15.164:0/1154197795 wait complete. 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.032+0000 7f36ef105640 10 monclient: build_initial_monmap 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.032+0000 7f36ef105640 10 monclient: monmap: 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.047 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.032+0000 7f36ef105640 1 Processor -- start 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.032+0000 7f36ef105640 1 -- start start 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.032+0000 7f36ef105640 10 monclient: init 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e806d160 0x7f36e81cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-b con 0x7f36e806d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:02.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 0x7f36e81cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-a con 0x7f36e806d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 0x7f36e81cdb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): picked mon.noname-c con 0x7f36e80a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8138780 con 0x7f36e806d540 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f36e8138600 con 0x7f36e806d160 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f36e8138900 con 0x7f36e80a4c20 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ef105640 10 monclient(hunting): authenticate will time out at 1520.520272s 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 0x7f36e81cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 0x7f36e81cd5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42734/0 (socket says 172.21.15.164:42734) 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 0x7f36e81cdb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 1 -- 172.21.15.164:0/1380309088 learned_addr learned my addr 172.21.15.164:0/1380309088 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): get_auth_request con 0x7f36e806d540 auth_method 0 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): get_auth_request con 0x7f36e80a4c20 auth_method 0 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e806d160 0x7f36e81cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): get_auth_request con 0x7f36e806d160 auth_method 0 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient(hunting): handle_auth_done global_id 114368 payload 824 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e806d160 msgr2=0x7f36e81cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e806d160 0x7f36e81cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 msgr2=0x7f36e81cd5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 0x7f36e81cd5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient: _finish_hunting 0 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 monclient: found mon.noname-c 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36e80a1de0 con 0x7f36e80a4c20 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.033+0000 7f36e7fff640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f36e806d540 0x7f36e81cd5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.035+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 msgr2=0x7f36e81cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.035+0000 7f36ed6a2640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 0x7f36e81cdb00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c7b0 tx=0x7f36e000cc70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.035+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0016020 con 0x7f36e80a4c20 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:02.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: dump: 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: _finish_auth 0 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:02.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 5 monclient: authenticate success, global_id 114368 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36e000ac60 con 0x7f36e80a4c20 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 monclient: _renew_subs 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36e806e6b0 con 0x7f36e80a4c20 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e000ce60 con 0x7f36e80a4c20 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 10 monclient: dump: 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.054 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e6ffd640 4 set_mon_vals no callback set 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 monclient: _renew_subs 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f36e80a1340 con 0x7f36e80a4c20 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 0x7f36e812dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f36e80a2e50 con 0x7f36e80a3d70 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.036+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f36e0019020 con 0x7f36e80a4c20 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 0x7f36e812dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient: get_auth_request con 0x7f36e80a3d70 auth_method 0 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36e57fa640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36cc03e210 0x7f36cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(433..433 src has 184..433) ==== 4787+0+0 (secure 0 0 0) 0x7f36e0052bf0 con 0x7f36e80a4c20 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 10 monclient(hunting): handle_auth_done global_id 134206 payload 824 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36ecea1640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f36e8134e00 con 0x7f36e80a3d70 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36e7fff640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36cc03e210 0x7f36cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.037+0000 7f36e7fff640 10 monclient: get_auth_request con 0x7f36cc03e210 auth_method 0 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36ecea1640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 msgr2=0x7f36e812dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 0x7f36e812dfb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36dc034000 tx=0x7f36dc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc037040 con 0x7f36e80a3d70 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36e57fa640 10 monclient: dump: 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.057 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.039+0000 7f36e7fff640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36cc03e210 0x7f36cc0406d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f36e8136620 tx=0x7f36d4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f36dc03c850 con 0x7f36e80a3d70 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:02.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 msgr2=0x7f36e812dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 0x7f36e812dfb0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36dc034000 tx=0x7f36dc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 msgr2=0x7f36e812dfb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:02.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.040+0000 7f36e57fa640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e80a3d70 0x7f36e812dfb0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f36dc034000 tx=0x7f36dc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ef105640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:02.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ef105640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 0x7f36e812e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ef105640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f36e812ee40 con 0x7f36e812e4f0 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 0x7f36e812e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient: get_auth_request con 0x7f36e812e4f0 auth_method 0 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 10 monclient(hunting): handle_auth_done global_id 134209 payload 824 2024-09-10T16:19:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.046+0000 7f36ecea1640 1 -- 172.21.15.164:0/1380309088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f36e80aa1e0 con 0x7f36e812e4f0 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ecea1640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 msgr2=0x7f36e812e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ecea1640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 0x7f36e812e900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36e80a4ba0 tx=0x7f36dc008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36dc037030 con 0x7f36e812e4f0 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 10 monclient: dump: 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f36dc003c30 con 0x7f36e812e4f0 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 msgr2=0x7f36e812e900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 0x7f36e812e900 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36e80a4ba0 tx=0x7f36dc008030 comp rx=0 tx=0).stop 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 msgr2=0x7f36e812e900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36e57fa640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f36e812e4f0 0x7f36e812e900 secure :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f36e80a4ba0 tx=0x7f36dc008030 comp rx=0 tx=0).stop 2024-09-10T16:19:02.063 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 66453, 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:02.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:02.065 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:02.066 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:02.067 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:02.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:02.069 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:02.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:02.072 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:02.073 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:02.074 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:02.075 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:02.076 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:02.077 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36cc03e210 msgr2=0x7f36cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ef105640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36cc03e210 0x7f36cc0406d0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f36e8136620 tx=0x7f36d4008040 comp rx=0 tx=0).stop 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ef105640 10 monclient: shutdown 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ef105640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.049+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 msgr2=0x7f36e81cdb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ef105640 1 --2- 172.21.15.164:0/1380309088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f36e80a4c20 0x7f36e81cdb00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f36e000c7b0 tx=0x7f36e000cc70 comp rx=0 tx=0).stop 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 reap_dead start 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ed6a2640 1 -- 172.21.15.164:0/1380309088 reap_dead start 2024-09-10T16:19:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 shutdown_connections 2024-09-10T16:19:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 >> 172.21.15.164:0/1380309088 conn(0x7f36e80aacb0 msgr2=0x7f36e8132730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 shutdown_connections 2024-09-10T16:19:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.050+0000 7f36ef105640 1 -- 172.21.15.164:0/1380309088 wait complete. 2024-09-10T16:19:02.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:02.079 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:19:02.079 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:19:02.122 INFO:tasks.rados.rados.0.smithi165.stdout:1816: done (0 left) 2024-09-10T16:19:02.122 INFO:tasks.rados.rados.0.smithi165.stdout:1817: rollback oid 14 current snap is 202 2024-09-10T16:19:02.123 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 201 2024-09-10T16:19:02.123 INFO:tasks.rados.rados.0.smithi165.stdout:1818: read oid 22 snap -1 2024-09-10T16:19:02.123 INFO:tasks.rados.rados.0.smithi165.stdout:1818: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:19:02.123 INFO:tasks.rados.rados.0.smithi165.stdout:1819: write (excl) oid 30 current snap is 202 2024-09-10T16:19:02.123 INFO:tasks.rados.rados.0.smithi165.stdout:1819: seq_num 637 ranges {706059=790912,1959265=685222} 2024-09-10T16:19:02.131 INFO:tasks.rados.rados.0.smithi165.stdout:1819: writing smithi16534841-30 from 706059 to 1496971 tid 1 2024-09-10T16:19:02.138 INFO:tasks.rados.rados.0.smithi165.stdout:1819: writing smithi16534841-30 from 1959265 to 2644487 tid 2 2024-09-10T16:19:02.138 INFO:tasks.rados.rados.0.smithi165.stdout:1820: write (excl) oid 16 current snap is 202 2024-09-10T16:19:02.138 INFO:tasks.rados.rados.0.smithi165.stdout:1817: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:19:02.138 INFO:tasks.rados.rados.0.smithi165.stdout:1817: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:19:02.138 INFO:tasks.rados.rados.0.smithi165.stdout:1820: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-09-10T16:19:02.144 INFO:tasks.rados.rados.0.smithi165.stdout:1820: writing smithi16534841-16 from 722866 to 1189027 tid 1 2024-09-10T16:19:02.149 INFO:tasks.rados.rados.0.smithi165.stdout:1820: writing smithi16534841-16 from 1617747 to 2046627 tid 2 2024-09-10T16:19:02.153 INFO:tasks.rados.rados.0.smithi165.stdout:1820: writing smithi16534841-16 from 2587634 to 2722866 tid 3 2024-09-10T16:19:02.153 INFO:tasks.rados.rados.0.smithi165.stdout:1821: read oid 42 snap -1 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1821: expect (ObjNum 623 snap 194 seq_num 623) 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1822: delete oid 3 current snap is 202 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1817: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1140 (ObjNum 528 snap 163 seq_num 528) dirty exists 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:1818: expect (ObjNum 554 snap 174 seq_num 554) 2024-09-10T16:19:02.154 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:02.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.182+0000 7f177627f640 10 monclient: get_monmap_and_config 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.182+0000 7f177627f640 10 monclient: build_initial_monmap 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.182+0000 7f177627f640 10 monclient: monmap: 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:02.183430+0000 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:02.183430+0000 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:02.184 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:02.185 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 Processor -- start 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 -- start start 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient: init 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f177009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): picked mon.noname-b con 0x7f177009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 0x7f177013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): picked mon.noname-c con 0x7f177009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17700a0870 con 0x7f177009d930 2024-09-10T16:19:02.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f177009d3e0 con 0x7f177009e420 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1770074c70 con 0x7f177009ab50 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f177627f640 10 monclient(hunting): authenticate will time out at 1520.670779s 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34922/0 (socket says 172.21.15.164:34922) 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f177009d930 0x7f177009dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38248/0 (socket says 172.21.15.164:38248) 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 1 -- 172.21.15.164:0/4032873547 learned_addr learned my addr 172.21.15.164:0/4032873547 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 10 monclient(hunting): get_auth_request con 0x7f177009ab50 auth_method 0 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.183+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 1 --2- 172.21.15.164:0/4032873547 >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 0x7f177013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 10 monclient(hunting): get_auth_request con 0x7f177009e420 auth_method 0 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 10 monclient(hunting): handle_auth_done global_id 114371 payload 274 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 1 -- 172.21.15.164:0/4032873547 >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 msgr2=0x7f177013bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 1 --2- 172.21.15.164:0/4032873547 >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 0x7f177013bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 1 -- 172.21.15.164:0/4032873547 >> v1:172.21.15.164:6789/0 conn(0x7f177009d930 legacy=0x7f177009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:02.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:02.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 1 monclient: found mon.noname-b 2024-09-10T16:19:02.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:02.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:02.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f176f7fe640 1 -- 172.21.15.164:0/4032873547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177009ecb0 con 0x7f177009ab50 2024-09-10T16:19:02.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.184+0000 7f177481c640 1 --2- 172.21.15.164:0/4032873547 >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 0x7f177013bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.186+0000 7f176f7fe640 1 -- 172.21.15.164:0/4032873547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 msgr2=0x7f17701397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.186+0000 7f176f7fe640 1 --2- 172.21.15.164:0/4032873547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f176400bb10 tx=0x7f176402c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22572902072e871a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 1 -- 172.21.15.164:0/4032873547 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764031920 con 0x7f177009ab50 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: dump: 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.195 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: _finish_auth 0 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 1 -- 172.21.15.164:0/4032873547 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764031ac0 con 0x7f177009ab50 2024-09-10T16:19:02.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 5 monclient: authenticate success, global_id 114371 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 1 -- 172.21.15.164:0/4032873547 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764033400 con 0x7f177009ab50 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 4 set_mon_vals no callback set 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f176effd640 10 monclient: dump: 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.197 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 10 monclient: shutdown 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 -- 172.21.15.164:0/4032873547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 msgr2=0x7f17701397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 --2- 172.21.15.164:0/4032873547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f176400bb10 tx=0x7f176402c7e0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 -- 172.21.15.164:0/4032873547 shutdown_connections 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 --2- 172.21.15.164:0/4032873547 >> v2:172.21.15.165:3300/0 conn(0x7f177009e420 0x7f177013bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 --2- 172.21.15.164:0/4032873547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f17701397c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 -- 172.21.15.164:0/4032873547 >> 172.21.15.164:0/4032873547 conn(0x7f17700a3110 msgr2=0x7f17700a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:02.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 -- 172.21.15.164:0/4032873547 shutdown_connections 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.187+0000 7f177627f640 1 -- 172.21.15.164:0/4032873547 wait complete. 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.188+0000 7f177627f640 10 monclient: build_initial_monmap 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.188+0000 7f177627f640 10 monclient: monmap: 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.188+0000 7f177627f640 1 Processor -- start 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.188+0000 7f177627f640 1 -- start start 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.188+0000 7f177627f640 10 monclient: init 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f177012f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): picked mon.noname-c con 0x7f177009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): picked mon.noname-a con 0x7f177009d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177009e420 0x7f1770130380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): picked mon.noname-b con 0x7f177009e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17700a2940 con 0x7f177009d930 2024-09-10T16:19:02.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f177009ef30 con 0x7f177009e420 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17700a2ac0 con 0x7f177009ab50 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177627f640 10 monclient(hunting): authenticate will time out at 1520.676179s 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42740/0 (socket says 172.21.15.164:42740) 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 -- 172.21.15.164:0/637176134 learned_addr learned my addr 172.21.15.164:0/637176134 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): get_auth_request con 0x7f177009d930 auth_method 0 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f177012f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177009e420 0x7f1770130380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): get_auth_request con 0x7f177009ab50 auth_method 0 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): get_auth_request con 0x7f177009e420 auth_method 0 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f177481c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient(hunting): handle_auth_done global_id 144123 payload 824 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 -- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 msgr2=0x7f177012f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f177012f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 -- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177009e420 msgr2=0x7f1770130380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177009e420 0x7f1770130380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176f7fe640 1 -- 172.21.15.164:0/637176134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1770132730 con 0x7f177009d930 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.189+0000 7f176ffff640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f177012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176f7fe640 1 -- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 msgr2=0x7f177012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176f7fe640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1764000c00 tx=0x7f1764004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764040020 con 0x7f177009d930 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: dump: 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: _finish_auth 0 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f176403c640 con 0x7f177009d930 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f177627f640 5 monclient: authenticate success, global_id 144123 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f176403c9f0 con 0x7f177009d930 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f177627f640 10 monclient: _renew_subs 2024-09-10T16:19:02.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f177627f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1770132a10 con 0x7f177009d930 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f176cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.191+0000 7f176e7fc640 4 set_mon_vals no callback set 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f176cff9640 10 monclient: dump: 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:02.206 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 10 monclient: _renew_subs 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f177009d620 con 0x7f177009d930 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1764043020 con 0x7f177009d930 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.192+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17700a45a0 con 0x7f177009d930 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.193+0000 7f176cff9640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 0x7f175c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.193+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(434..434 src has 184..434) ==== 4787+0+0 (secure 0 0 0) 0x7f17640737f0 con 0x7f177009d930 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.193+0000 7f176ffff640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 0x7f175c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:02.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.193+0000 7f176ffff640 10 monclient: get_auth_request con 0x7f175c03e240 auth_method 0 2024-09-10T16:19:02.208 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.195+0000 7f176ffff640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 0x7f175c040700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1760007fd0 tx=0x7f1760007e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:02.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.198+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f176404a030 con 0x7f177009d930 2024-09-10T16:19:02.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.198+0000 7f176cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:02.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.198+0000 7f176cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:02.240 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3040409 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: finishing write tid 1 to smithi16534841-30 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: finishing write tid 2 to smithi16534841-30 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: finishing write tid 3 to smithi16534841-30 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: finishing write tid 4 to smithi16534841-30 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: oid 30 updating version 0 to 1239 2024-09-10T16:19:02.249 INFO:tasks.rados.rados.0.smithi165.stdout:1819: oid 30 version 1239 is already newer than 1237 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1819: oid 30 version 1239 is already newer than 1238 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1239 (ObjNum 637 snap 202 seq_num 637) dirty exists 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1819: left oid 30 (ObjNum 637 snap 202 seq_num 637) 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1820: finishing write tid 1 to smithi16534841-16 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1820: finishing write tid 2 to smithi16534841-16 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1817: done (5 left) 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1818: done (4 left) 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1819: done (3 left) 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1822: done (2 left) 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1823: write oid 20 current snap is 202 2024-09-10T16:19:02.250 INFO:tasks.rados.rados.0.smithi165.stdout:1823: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-10T16:19:02.257 INFO:tasks.rados.rados.0.smithi165.stdout:1823: writing smithi16534841-20 from 756480 to 1489492 tid 1 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1823: writing smithi16534841-20 from 2134711 to 2567260 tid 2 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1824: read oid 43 snap -1 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1824: expect (ObjNum 625 snap 195 seq_num 625) 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1825: write (excl) oid 27 current snap is 202 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1820: finishing write tid 3 to smithi16534841-16 2024-09-10T16:19:02.263 INFO:tasks.rados.rados.0.smithi165.stdout:1820: finishing write tid 4 to smithi16534841-16 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: finishing write tid 5 to smithi16534841-16 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: oid 16 updating version 0 to 1241 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: oid 16 updating version 1241 to 1242 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: oid 16 updating version 1242 to 1243 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: oid 16 updating version 1243 to 1244 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1244 (ObjNum 638 snap 202 seq_num 638) dirty exists 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1820: left oid 16 (ObjNum 638 snap 202 seq_num 638) 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:1821: expect (ObjNum 623 snap 194 seq_num 623) 2024-09-10T16:19:02.264 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 148321 2024-09-10T16:19:02.305 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 31756 2024-09-10T16:19:02.348 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 281075 2024-09-10T16:19:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:1823: finishing write tid 1 to smithi16534841-20 2024-09-10T16:19:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:1823: finishing write tid 2 to smithi16534841-20 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: finishing write tid 3 to smithi16534841-20 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: finishing write tid 4 to smithi16534841-20 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: oid 20 updating version 0 to 1166 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: oid 20 updating version 1166 to 1167 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: oid 20 version 1167 is already newer than 1165 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1167 (ObjNum 640 snap 202 seq_num 640) dirty exists 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1823: left oid 20 (ObjNum 640 snap 202 seq_num 640) 2024-09-10T16:19:02.390 INFO:tasks.rados.rados.0.smithi165.stdout:1824: expect (ObjNum 625 snap 195 seq_num 625) 2024-09-10T16:19:02.408 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:02.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.441+0000 7f177627f640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:19:02.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.441+0000 7f177627f640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:19:02.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.441+0000 7f177627f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:02.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.441+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f177009dd10 con 0x7f177009d930 2024-09-10T16:19:02.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.442+0000 7f176cff9640 1 -- 172.21.15.164:0/637176134 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f176403a320 con 0x7f177009d930 2024-09-10T16:19:02.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.442+0000 7f176cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:19:02.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.442+0000 7f176cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:02.445 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.445 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=1316105904,config=2586702516,health=2974672015,kv=805391194,logm=2561469176}) 2024-09-10T16:19:02.445 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=22,config=2,health=11,kv=11,logm=54} crc {auth=2590976483,config=2586702516,health=271994670,kv=1222545535,logm=3856047359}) 2024-09-10T16:19:02.445 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.445 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=1316105904,config=2586702516,health=2974672015,kv=805391194,logm=2561469176}) 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.442+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=1316105904,config=2586702516,health=2974672015,kv=3747982264,logm=2455472451}) 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265569561}) 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1291521429}) 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265569561}) 2024-09-10T16:19:02.446 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.444+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1436794184}) 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220382592}) 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068495742}) 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2220382592}) 2024-09-10T16:19:02.447 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.445+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2892213058}) 2024-09-10T16:19:02.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 msgr2=0x7f175c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 0x7f175c040700 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f1760007fd0 tx=0x7f1760007e60 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 10 monclient: shutdown 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 msgr2=0x7f177012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1764000c00 tx=0x7f1764004730 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 shutdown_connections 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f175c03e240 0x7f175c040700 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177009e420 0x7f1770130380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177009d930 0x7f177012fe40 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 --2- 172.21.15.164:0/637176134 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177009ab50 0x7f177012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:02.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 >> 172.21.15.164:0/637176134 conn(0x7f17700a3110 msgr2=0x7f17701bc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:02.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 shutdown_connections 2024-09-10T16:19:02.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:02.445+0000 7f177627f640 1 -- 172.21.15.164:0/637176134 wait complete. 2024-09-10T16:19:02.451 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.452 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241412282}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2029127409}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241412282}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.446+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2546351893}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554268123}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2913043163}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554268123}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=51231827}) 2024-09-10T16:19:02.453 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142531035}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=832031108}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142531035}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.447+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169049890}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214937026}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1447491338}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214937026}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.448+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733901233}) 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.454 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374416517}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1736450592}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374416517}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.449+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3026303834}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162862720}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1804345682}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2162862720}) 2024-09-10T16:19:02.455 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.450+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179741842}) 2024-09-10T16:19:02.457 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426552379}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1086434456}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426552379}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.451+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1015907099}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436673829}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532610178}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1436673829}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.452+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3421955911}) 2024-09-10T16:19:02.458 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736808051}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3256873987}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=736808051}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.453+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411658190}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157239131}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3208906682}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157239131}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.454+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2656389691}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457793082}) 2024-09-10T16:19:02.459 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2909953156}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1457793082}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4099397032}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126994682}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3028694969}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.455+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3126994682}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=293627810}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738206047}) 2024-09-10T16:19:02.460 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2106164573}) 2024-09-10T16:19:02.461 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.461 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1738206047}) 2024-09-10T16:19:02.461 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.456+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3097283928}) 2024-09-10T16:19:02.461 INFO:tasks.rados.rados.0.smithi165.stdout:1825: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671559772}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3928233243}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2671559772}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.457+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4094744737}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077317335}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=98494602}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2077317335}) 2024-09-10T16:19:02.463 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.458+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2681818459}) 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133439751}) 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258613907}) 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133439751}) 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.459+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=793473681}) 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.464 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944411638}) 2024-09-10T16:19:02.465 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:19:02.466 INFO:tasks.rados.rados.0.smithi165.stdout:1825: writing smithi16534841-27 from 773287 to 1181548 tid 1 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893047658}) 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1944411638}) 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1174715993}) 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906696085}) 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376668694}) 2024-09-10T16:19:02.470 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906696085}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.460+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923810175}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031697847}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=157714809}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031697847}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.461+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=330517853}) 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.471 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38058489}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=989402447}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=38058489}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.462+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2204378704}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501418631}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2880857665}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1501418631}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.463+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720378435}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415851858}) 2024-09-10T16:19:02.472 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3760465140}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415851858}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.464+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3229926019}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256300716}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=350139891}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=256300716}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.465+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3136038036}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359554345}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2559982671}) 2024-09-10T16:19:02.473 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2359554345}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.466+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=45743007}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314762092}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1923495040}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3314762092}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841006955}) 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.474 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142861812}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4294894392}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1142861812}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.467+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440761132}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880202337}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2333266650}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880202337}) 2024-09-10T16:19:02.475 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.468+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=878288152}) 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1825: writing smithi16534841-27 from 1793193 to 2369400 tid 2 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1820: done (4 left) 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1821: done (3 left) 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1823: done (2 left) 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1824: done (1 left) 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1826: read oid 48 snap -1 2024-09-10T16:19:02.477 INFO:tasks.rados.rados.0.smithi165.stdout:1826: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:19:02.478 INFO:tasks.rados.rados.0.smithi165.stdout:1827: snap_create 2024-09-10T16:19:02.478 INFO:tasks.rados.rados.0.smithi165.stdout:1825: finishing write tid 1 to smithi16534841-27 2024-09-10T16:19:02.481 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.481 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110075156}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861157333}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110075156}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.469+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3874924021}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168431094}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257982693}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4168431094}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.470+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773859651}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365087351}) 2024-09-10T16:19:02.482 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1663907467}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=365087351}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.471+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4217734876}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670980291}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3331149997}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2670980291}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3970696056}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177745381}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1408363972}) 2024-09-10T16:19:02.483 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1177745381}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.472+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405095063}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.473+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.473+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=15,monmap=2} crc {logm=3209979420,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgrstat=2921534902,mon_config_key=1117509908,monmap=765578399}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.474+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15} crc {logm=2078892635,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2097262121,mon_config_key=647691077}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.474+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.474+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=15,monmap=2} crc {logm=3209979420,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgrstat=2921534902,mon_config_key=1117509908,monmap=765578399}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.474+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgrstat=10,mon_config_key=17,monmap=2} crc {logm=2636693291,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgrstat=2610716910,mon_config_key=665552247,monmap=765578399}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=82,osdmap=8} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=2196505739,osd_snap=1517946612,osdmap=740938123}) 2024-09-10T16:19:02.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=78,osdmap=11} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2196505739,osd_snap=3207043233,osdmap=703465567}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=1,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=82,osdmap=8} crc {monmap=1228700967,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=2196505739,osd_snap=1517946612,osdmap=740938123}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.475+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=84,osdmap=8} crc {nvmeofgw=598337686,osd_metadata=2836976987,osd_pg_creating=2196505739,osd_snap=2334331651,osdmap=2309188667}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2093315783}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3356212077}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2093315783}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.476+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1565817274}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4030767020}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3705257298}) 2024-09-10T16:19:02.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4030767020}) 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.477+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2649376576}) 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1471508284}) 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4281865248}) 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1471508284}) 2024-09-10T16:19:02.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.478+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=842432046}) 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: finishing write tid 2 to smithi16534841-27 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: finishing write tid 3 to smithi16534841-27 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: finishing write tid 4 to smithi16534841-27 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: oid 27 updating version 0 to 1151 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: oid 27 version 1151 is already newer than 1150 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: oid 27 version 1151 is already newer than 1149 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1151 (ObjNum 641 snap 202 seq_num 641) dirty exists 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1825: left oid 27 (ObjNum 641 snap 202 seq_num 641) 2024-09-10T16:19:02.487 INFO:tasks.rados.rados.0.smithi165.stdout:1826: expect (ObjNum 593 snap 186 seq_num 593) 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=58} crc {osdmap=4139823010}) 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=1164818732}) 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=58} crc {osdmap=4139823010}) 2024-09-10T16:19:02.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:02.482+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=61} crc {osdmap=1924397906}) 2024-09-10T16:19:02.633 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3780934 2024-09-10T16:19:03.123 INFO:tasks.rados.rados.0.smithi165.stdout:1825: done (2 left) 2024-09-10T16:19:03.123 INFO:tasks.rados.rados.0.smithi165.stdout:1826: done (1 left) 2024-09-10T16:19:03.123 INFO:tasks.rados.rados.0.smithi165.stdout:1827: done (0 left) 2024-09-10T16:19:03.123 INFO:tasks.rados.rados.0.smithi165.stdout:1828: copy_from oid 50 from oid 28 current snap is 203 2024-09-10T16:19:03.123 INFO:tasks.rados.rados.0.smithi165.stdout:1829: write (excl) oid 17 current snap is 203 2024-09-10T16:19:03.124 INFO:tasks.rados.rados.0.smithi165.stdout:1829: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-10T16:19:03.130 INFO:tasks.rados.rados.0.smithi165.stdout:1829: writing smithi16534841-17 from 790094 to 1273604 tid 1 2024-09-10T16:19:03.136 INFO:tasks.rados.rados.0.smithi165.stdout:1829: writing smithi16534841-17 from 1851675 to 2287893 tid 2 2024-09-10T16:19:03.137 INFO:tasks.rados.rados.0.smithi165.stdout:1830: write oid 15 current snap is 203 2024-09-10T16:19:03.137 INFO:tasks.rados.rados.0.smithi165.stdout:1829: finishing write tid 1 to smithi16534841-17 2024-09-10T16:19:03.137 INFO:tasks.rados.rados.0.smithi165.stdout:1830: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-10T16:19:03.141 INFO:tasks.rados.rados.0.smithi165.stdout:1830: writing smithi16534841-15 from 406901 to 965660 tid 1 2024-09-10T16:19:03.147 INFO:tasks.rados.rados.0.smithi165.stdout:1830: writing smithi16534841-15 from 1393804 to 1973680 tid 2 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1830: writing smithi16534841-15 from 2608396 to 2806901 tid 3 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1831: read oid 7 snap -1 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1831: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1832: write (excl) oid 34 current snap is 203 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1828: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:19:03.153 INFO:tasks.rados.rados.0.smithi165.stdout:1832: seq_num 644 ranges {423708=634008,1852286=439887,2752172=71536} 2024-09-10T16:19:03.158 INFO:tasks.rados.rados.0.smithi165.stdout:1832: writing smithi16534841-34 from 423708 to 1057716 tid 1 2024-09-10T16:19:03.165 INFO:tasks.rados.rados.0.smithi165.stdout:1832: writing smithi16534841-34 from 1852286 to 2292173 tid 2 2024-09-10T16:19:03.168 INFO:tasks.rados.rados.0.smithi165.stdout:1832: writing smithi16534841-34 from 2752172 to 2823708 tid 3 2024-09-10T16:19:03.168 INFO:tasks.rados.rados.0.smithi165.stdout:1833: snap_create 2024-09-10T16:19:03.168 INFO:tasks.rados.rados.0.smithi165.stdout:1829: finishing write tid 2 to smithi16534841-17 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: finishing write tid 3 to smithi16534841-17 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: finishing write tid 4 to smithi16534841-17 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: oid 17 updating version 0 to 1099 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: oid 17 version 1099 is already newer than 1098 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: oid 17 updating version 1099 to 1100 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1100 (ObjNum 642 snap 203 seq_num 642) dirty exists 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1829: left oid 17 (ObjNum 642 snap 203 seq_num 642) 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1828: finishing copy_from to smithi16534841-50 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1023 (ObjNum 628 snap 196 seq_num 628) dirty exists 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1830: finishing write tid 1 to smithi16534841-15 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1830: finishing write tid 2 to smithi16534841-15 2024-09-10T16:19:03.169 INFO:tasks.rados.rados.0.smithi165.stdout:1831: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:03.312 INFO:tasks.rados.rados.0.smithi165.stdout:1830: finishing write tid 3 to smithi16534841-15 2024-09-10T16:19:03.312 INFO:tasks.rados.rados.0.smithi165.stdout:1830: finishing write tid 4 to smithi16534841-15 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: finishing write tid 5 to smithi16534841-15 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: oid 15 updating version 0 to 714 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: oid 15 updating version 714 to 715 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: oid 15 version 715 is already newer than 713 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: oid 15 version 715 is already newer than 712 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 715 (ObjNum 643 snap 203 seq_num 643) dirty exists 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1830: left oid 15 (ObjNum 643 snap 203 seq_num 643) 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: finishing write tid 1 to smithi16534841-34 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: finishing write tid 2 to smithi16534841-34 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: finishing write tid 3 to smithi16534841-34 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: finishing write tid 4 to smithi16534841-34 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: finishing write tid 5 to smithi16534841-34 2024-09-10T16:19:03.313 INFO:tasks.rados.rados.0.smithi165.stdout:1832: oid 34 updating version 0 to 1128 2024-09-10T16:19:03.314 INFO:tasks.rados.rados.0.smithi165.stdout:1832: oid 34 updating version 1128 to 1130 2024-09-10T16:19:03.314 INFO:tasks.rados.rados.0.smithi165.stdout:1832: oid 34 version 1130 is already newer than 1129 2024-09-10T16:19:03.314 INFO:tasks.rados.rados.0.smithi165.stdout:1832: oid 34 updating version 1130 to 1131 2024-09-10T16:19:03.314 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1131 (ObjNum 644 snap 203 seq_num 644) dirty exists 2024-09-10T16:19:03.314 INFO:tasks.rados.rados.0.smithi165.stdout:1832: left oid 34 (ObjNum 644 snap 203 seq_num 644) 2024-09-10T16:19:03.466 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:19:03.466 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:03.568 INFO:tasks.rados.rados.0.smithi165.stdout:1828: done (5 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1829: done (4 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1830: done (3 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1831: done (2 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1832: done (1 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1833: done (0 left) 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1834: write (excl) oid 1 current snap is 204 2024-09-10T16:19:03.569 INFO:tasks.rados.rados.0.smithi165.stdout:1834: seq_num 645 ranges {440515=709257,1910768=583545} 2024-09-10T16:19:03.574 INFO:tasks.rados.rados.0.smithi165.stdout:1834: writing smithi16534841-1 from 440515 to 1149772 tid 1 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.575+0000 7fd81eb6b640 10 monclient: get_monmap_and_config 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.575+0000 7fd81eb6b640 10 monclient: build_initial_monmap 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.575+0000 7fd81eb6b640 10 monclient: monmap: 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:03.576837+0000 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:03.576837+0000 2024-09-10T16:19:03.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:03.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.576+0000 7fd81eb6b640 1 Processor -- start 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.576+0000 7fd81eb6b640 1 -- start start 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.576+0000 7fd81eb6b640 10 monclient: init 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-10T16:19:03.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd81809c180 0x7fd81809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-c con 0x7fd81809c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd81809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81813b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-b con 0x7fd81809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd81809a430 con 0x7fd81809d160 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd8180a24e0 con 0x7fd81809c180 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd8180a2bf0 con 0x7fd81809f4c0 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:03.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81eb6b640 10 monclient(hunting): authenticate will time out at 1522.064436s 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42756/0 (socket says 172.21.15.164:42756) 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 1 -- 172.21.15.164:0/1610253685 learned_addr learned my addr 172.21.15.164:0/1610253685 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 --2- 172.21.15.164:0/1610253685 >> v2:172.21.15.165:3300/0 conn(0x7fd81809c180 0x7fd81809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): get_auth_request con 0x7fd81809d160 auth_method 0 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): get_auth_request con 0x7fd81809c180 auth_method 0 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81813b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): get_auth_request con 0x7fd81809f4c0 auth_method 0 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient(hunting): handle_auth_done global_id 134218 payload 274 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 -- 172.21.15.164:0/1610253685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 msgr2=0x7fd81813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81813b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 -- 172.21.15.164:0/1610253685 >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 msgr2=0x7fd81809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 --2- 172.21.15.164:0/1610253685 >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient: _finish_hunting 0 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 monclient: found mon.noname-c 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.577+0000 7fd81c907640 1 -- 172.21.15.164:0/1610253685 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd81809c560 con 0x7fd81809c180 2024-09-10T16:19:03.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.578+0000 7fd817fff640 1 --2- 172.21.15.164:0/1610253685 >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:03.586 INFO:tasks.rados.rados.0.smithi165.stdout:1834: writing smithi16534841-1 from 1910768 to 2494313 tid 2 2024-09-10T16:19:03.586 INFO:tasks.rados.rados.0.smithi165.stdout:1835: snap_remove snap 203 2024-09-10T16:19:03.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd81c907640 1 -- 172.21.15.164:0/1610253685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 msgr2=0x7fd81809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd81c907640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81809ea40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0097a0 tx=0x7fd80c02a020 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b9756642af62d92 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 1 -- 172.21.15.164:0/1610253685 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c037070 con 0x7fd81809c180 2024-09-10T16:19:03.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: dump: 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:03.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 1 -- 172.21.15.164:0/1610253685 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c032440 con 0x7fd81809c180 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 1 -- 172.21.15.164:0/1610253685 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c032810 con 0x7fd81809c180 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd81eb6b640 5 monclient: authenticate success, global_id 134218 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd81eb6b640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd81eb6b640 4 set_mon_vals no callback set 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.581+0000 7fd8177fe640 10 monclient: dump: 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 10 monclient: shutdown 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1610253685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 msgr2=0x7fd81809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81809ea40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd80c0097a0 tx=0x7fd80c02a020 comp rx=0 tx=0).stop 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1610253685 shutdown_connections 2024-09-10T16:19:03.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81813b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1610253685 >> v2:172.21.15.164:3300/0 conn(0x7fd81809d160 0x7fd81809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1610253685 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81809ea40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1610253685 >> 172.21.15.164:0/1610253685 conn(0x7fd8180a3110 msgr2=0x7fd8180a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1610253685 shutdown_connections 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.582+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1610253685 wait complete. 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.583+0000 7fd81eb6b640 10 monclient: build_initial_monmap 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.583+0000 7fd81eb6b640 10 monclient: monmap: 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.583+0000 7fd81eb6b640 1 Processor -- start 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.583+0000 7fd81eb6b640 1 -- start start 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.583+0000 7fd81eb6b640 10 monclient: init 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:03.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81808e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-b con 0x7fd81809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-a con 0x7fd81809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81808f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): picked mon.noname-c con 0x7fd81809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd818074f90 con 0x7fd81809d160 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd818074e10 con 0x7fd81809c180 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd81809d930 con 0x7fd81809f4c0 2024-09-10T16:19:03.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81eb6b640 10 monclient(hunting): authenticate will time out at 1522.071273s 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81808f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81808e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42760/0 (socket says 172.21.15.164:42760) 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): get_auth_request con 0x7fd81809f4c0 auth_method 0 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 1 -- 172.21.15.164:0/1914889772 learned_addr learned my addr 172.21.15.164:0/1914889772 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): get_auth_request con 0x7fd81809d160 auth_method 0 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): get_auth_request con 0x7fd81809c180 auth_method 0 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient(hunting): handle_auth_done global_id 134221 payload 824 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 -- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 msgr2=0x7fd81808f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81808f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 -- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 msgr2=0x7fd81808ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient: _finish_hunting 0 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 monclient: found mon.noname-b 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81c907640 1 -- 172.21.15.164:0/1914889772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd818092390 con 0x7fd81809c180 2024-09-10T16:19:03.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd81d108640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81808f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.584+0000 7fd817fff640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81c907640 1 -- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 msgr2=0x7fd81808e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81c907640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81808e700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd80c031000 tx=0x7fd80c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c037040 con 0x7fd81809c180 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: dump: 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: _finish_auth 0 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: _check_auth_tickets 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd80c03b8e0 con 0x7fd81809c180 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 5 monclient: authenticate success, global_id 134221 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 10 monclient: _renew_subs 2024-09-10T16:19:03.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8180a4230 con 0x7fd81809c180 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd80c03bcd0 con 0x7fd81809c180 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd8157fa640 10 monclient: dump: 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.603 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd816ffd640 4 set_mon_vals no callback set 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 10 monclient: _renew_subs 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.587+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd818091520 con 0x7fd81809c180 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd81eb6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:03.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd81eb6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd81eb6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd81809dd60 con 0x7fd81809c180 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd80c044020 con 0x7fd81809c180 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd8157fa640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 0x7fd7fc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.588+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7fd80c0728b0 con 0x7fd81809c180 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.589+0000 7fd817fff640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 0x7fd7fc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.589+0000 7fd817fff640 10 monclient: get_auth_request con 0x7fd7fc03e260 auth_method 0 2024-09-10T16:19:03.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.591+0000 7fd817fff640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 0x7fd7fc040720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8040095e0 tx=0x7fd804012040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.594+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd80c0712a0 con 0x7fd81809c180 2024-09-10T16:19:03.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.594+0000 7fd8157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:03.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.594+0000 7fd8157fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd81eb6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd81eb6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd81eb6b640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd818071ef0 con 0x7fd81809c180 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd8157fa640 1 -- 172.21.15.164:0/1914889772 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fd80c03cb50 con 0x7fd81809c180 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd8157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:03.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.838+0000 7fd8157fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:03.841 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 msgr2=0x7fd7fc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 0x7fd7fc040720 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fd8040095e0 tx=0x7fd804012040 comp rx=0 tx=0).stop 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 10 monclient: shutdown 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 msgr2=0x7fd81808e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.840+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81808e700 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd80c031000 tx=0x7fd80c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 shutdown_connections 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd7fc03e260 0x7fd7fc040720 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd81809f4c0 0x7fd81808f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd81809d160 0x7fd81808ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 --2- 172.21.15.164:0/1914889772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd81809c180 0x7fd81808e700 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 >> 172.21.15.164:0/1914889772 conn(0x7fd8180a3110 msgr2=0x7fd8180997b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 shutdown_connections 2024-09-10T16:19:03.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.841+0000 7fd81eb6b640 1 -- 172.21.15.164:0/1914889772 wait complete. 2024-09-10T16:19:03.853 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:19:03.853 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:19:03.853 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:19:03.853 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:19:03.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.951+0000 7fead58a3640 10 monclient: get_monmap_and_config 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.951+0000 7fead58a3640 10 monclient: build_initial_monmap 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.951+0000 7fead58a3640 10 monclient: monmap: 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:03.952314+0000 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:03.952314+0000 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:03.953 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:03.954 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:03.954 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:03.954 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.951+0000 7fead58a3640 1 Processor -- start 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 -- start start 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient: init 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fead006cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:03.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fead0134f70 0x7fead013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-c con 0x7fead0134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fead013d560 0x7fead013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-a con 0x7fead013d560 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fead009ce50 con 0x7fead006cb70 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fead006db60 con 0x7fead013d560 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fead006cf50 con 0x7fead0134f70 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7fead58a3640 10 monclient(hunting): authenticate will time out at 1522.439603s 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feaceffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fead006cb70 0x7fead00a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38260/0 (socket says 172.21.15.164:38260) 2024-09-10T16:19:03.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fead013d560 0x7fead013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feaceffd640 1 -- 172.21.15.164:0/1939944132 learned_addr learned my addr 172.21.15.164:0/1939944132 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): get_auth_request con 0x7fead013d560 auth_method 0 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 1 --2- 172.21.15.164:0/1939944132 >> v2:172.21.15.165:3300/0 conn(0x7fead0134f70 0x7fead013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): get_auth_request con 0x7fead0134f70 auth_method 0 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.952+0000 7feacf7fe640 10 monclient(hunting): handle_auth_done global_id 144132 payload 274 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 1 -- 172.21.15.164:0/1939944132 >> v2:172.21.15.165:3300/0 conn(0x7fead0134f70 msgr2=0x7fead013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 1 --2- 172.21.15.164:0/1939944132 >> v2:172.21.15.165:3300/0 conn(0x7fead0134f70 0x7fead013d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 1 -- 172.21.15.164:0/1939944132 >> v1:172.21.15.164:6789/0 conn(0x7fead006cb70 legacy=0x7fead00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 1 monclient: found mon.noname-a 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:03.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.953+0000 7feacf7fe640 1 -- 172.21.15.164:0/1939944132 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead0135380 con 0x7fead013d560 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.955+0000 7feacf7fe640 1 -- 172.21.15.164:0/1939944132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead013d560 msgr2=0x7fead013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.955+0000 7feacf7fe640 1 --2- 172.21.15.164:0/1939944132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead013d560 0x7fead013f950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feac8009d90 tx=0x7feac802a4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f63da2f0bb94b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.955+0000 7feacdffb640 1 -- 172.21.15.164:0/1939944132 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8007950 con 0x7fead013d560 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: dump: 2024-09-10T16:19:03.967 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.968 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: _finish_auth 0 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 1 -- 172.21.15.164:0/1939944132 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac8007af0 con 0x7fead013d560 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 5 monclient: authenticate success, global_id 144132 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 1 -- 172.21.15.164:0/1939944132 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac8031400 con 0x7fead013d560 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 4 set_mon_vals no callback set 2024-09-10T16:19:03.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7feacdffb640 10 monclient: dump: 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 10 monclient: shutdown 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 -- 172.21.15.164:0/1939944132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead013d560 msgr2=0x7fead013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 --2- 172.21.15.164:0/1939944132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead013d560 0x7fead013f950 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7feac8009d90 tx=0x7feac802a4e0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 -- 172.21.15.164:0/1939944132 shutdown_connections 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 --2- 172.21.15.164:0/1939944132 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead013d560 0x7fead013f950 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 --2- 172.21.15.164:0/1939944132 >> v2:172.21.15.165:3300/0 conn(0x7fead0134f70 0x7fead013d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 -- 172.21.15.164:0/1939944132 >> 172.21.15.164:0/1939944132 conn(0x7fead00aa900 msgr2=0x7fead00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 -- 172.21.15.164:0/1939944132 shutdown_connections 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.956+0000 7fead58a3640 1 -- 172.21.15.164:0/1939944132 wait complete. 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.957+0000 7fead58a3640 10 monclient: build_initial_monmap 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.957+0000 7fead58a3640 10 monclient: monmap: 2024-09-10T16:19:03.971 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.957+0000 7fead58a3640 1 Processor -- start 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.957+0000 7fead58a3640 1 -- start start 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.957+0000 7fead58a3640 10 monclient: init 2024-09-10T16:19:03.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fead006cb70 0x7fead01ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-b con 0x7fead006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-a con 0x7fead0134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead013d560 0x7fead01cda20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): picked mon.noname-c con 0x7fead013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fead00a4db0 con 0x7fead0134f70 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fead009d130 con 0x7fead006cb70 2024-09-10T16:19:03.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fead013fe90 con 0x7fead013d560 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7fead58a3640 10 monclient(hunting): authenticate will time out at 1522.445139s 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42782/0 (socket says 172.21.15.164:42782) 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 -- 172.21.15.164:0/2317454701 learned_addr learned my addr 172.21.15.164:0/2317454701 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): get_auth_request con 0x7fead0134f70 auth_method 0 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead013d560 0x7fead01cda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fead006cb70 0x7fead01ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): get_auth_request con 0x7fead013d560 auth_method 0 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): get_auth_request con 0x7fead006cb70 auth_method 0 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feacf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient(hunting): handle_auth_done global_id 144135 payload 824 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead013d560 msgr2=0x7fead01cda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fead013d560 0x7fead01cda20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fead006cb70 msgr2=0x7fead01ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fead006cb70 0x7fead01ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 monclient: found mon.noname-a 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:03.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.958+0000 7feace7fc640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fead006e720 con 0x7fead0134f70 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feace7fc640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 msgr2=0x7fead01cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feace7fc640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feac400e9b0 tx=0x7feac400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4017070 con 0x7fead0134f70 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: dump: 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.977 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: _finish_auth 0 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feac400cbb0 con 0x7fead0134f70 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7fead58a3640 5 monclient: authenticate success, global_id 144135 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac4013470 con 0x7fead0134f70 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7fead58a3640 10 monclient: _renew_subs 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7fead58a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fead013b1a0 con 0x7fead0134f70 2024-09-10T16:19:03.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feab7fff640 10 monclient: dump: 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.960+0000 7feacd7fa640 4 set_mon_vals no callback set 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 10 monclient: _renew_subs 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fead0137ce0 con 0x7fead0134f70 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 0x7fead012e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fead00a46a0 con 0x7fead012e020 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 0x7fead012e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient: get_auth_request con 0x7fead012e020 auth_method 0 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feac401a070 con 0x7fead0134f70 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.961+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feaceffd640 10 monclient(hunting): handle_auth_done global_id 144138 payload 824 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feab7fff640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feabc03e260 0x7feabc040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feaceffd640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fead00008d0 con 0x7fead012e020 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7feac4053120 con 0x7fead0134f70 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feacf7fe640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feabc03e260 0x7feabc040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.962+0000 7feacf7fe640 10 monclient: get_auth_request con 0x7feabc03e260 auth_method 0 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feaceffd640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 msgr2=0x7fead012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feaceffd640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 0x7fead012e400 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac000ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0017070 con 0x7fead012e020 2024-09-10T16:19:03.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 10 monclient: dump: 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feac000ee70 con 0x7fead012e020 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 msgr2=0x7fead012e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 0x7fead012e400 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac000ec60 comp rx=0 tx=0).stop 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 msgr2=0x7fead012e400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feab7fff640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e020 0x7fead012e400 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac000ec60 comp rx=0 tx=0).stop 2024-09-10T16:19:03.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.964+0000 7feacf7fe640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feabc03e260 0x7feabc040720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fead006e110 tx=0x7feac802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.970+0000 7fead58a3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.970+0000 7fead58a3640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 0x7fead0130250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.970+0000 7fead58a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.970+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fead012ed50 con 0x7fead012e940 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.970+0000 7feaceffd640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 0x7fead0130250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient: get_auth_request con 0x7fead012e940 auth_method 0 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:03.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 10 monclient(hunting): handle_auth_done global_id 144141 payload 824 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.971+0000 7feaceffd640 1 -- 172.21.15.164:0/2317454701 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fead0138420 con 0x7fead012e940 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feaceffd640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 msgr2=0x7fead0130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feaceffd640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 0x7fead0130250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac00058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feac0017050 con 0x7fead012e940 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 10 monclient: dump: 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:03.986 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7feac0006460 con 0x7fead012e940 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 msgr2=0x7fead0130250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:03.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 0x7fead0130250 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac00058d0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 msgr2=0x7fead0130250 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.973+0000 7feab7fff640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead012e940 0x7fead0130250 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feac000e7a0 tx=0x7feac00058d0 comp rx=0 tx=0).stop 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 15416, 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:03.988 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:03.989 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:03.990 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:03.991 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:03.992 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.993 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:03.994 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:03.995 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:03.996 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:03.997 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:03.998 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:03.999 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:19:04.000 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.001 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.001 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.001 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:04.001 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:04.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feabc03e260 msgr2=0x7feabc040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feabc03e260 0x7feabc040720 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fead006e110 tx=0x7feac802f000 comp rx=0 tx=0).stop 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 10 monclient: shutdown 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 msgr2=0x7fead01cd4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7feac400e9b0 tx=0x7feac400ee70 comp rx=0 tx=0).stop 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7feacf7fe640 1 -- 172.21.15.164:0/2317454701 reap_dead start 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 shutdown_connections 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 --2- 172.21.15.164:0/2317454701 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fead0134f70 0x7fead01cd4e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 >> 172.21.15.164:0/2317454701 conn(0x7fead00aa900 msgr2=0x7fead00a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 shutdown_connections 2024-09-10T16:19:04.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:03.974+0000 7fead58a3640 1 -- 172.21.15.164:0/2317454701 wait complete. 2024-09-10T16:19:04.004 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:04.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:19:04.004 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:19:04.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.101+0000 7f306b679640 10 monclient: get_monmap_and_config 2024-09-10T16:19:04.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.101+0000 7f306b679640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.101+0000 7f306b679640 10 monclient: monmap: 2024-09-10T16:19:04.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:04.102614+0000 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:04.102614+0000 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 Processor -- start 2024-09-10T16:19:04.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 -- start start 2024-09-10T16:19:04.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient: init 2024-09-10T16:19:04.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 0x7f306409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): picked mon.noname-b con 0x7f306409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f306409d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f306413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f306409a430 con 0x7f306409d160 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30640a24e0 con 0x7f306413a5c0 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f30640a2bf0 con 0x7f306409c180 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.102+0000 7f306b679640 10 monclient(hunting): authenticate will time out at 1522.589970s 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 0x7f306409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3068c14640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f306409d160 0x7f306409ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38268/0 (socket says 172.21.15.164:38268) 2024-09-10T16:19:04.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3068c14640 1 -- 172.21.15.164:0/850238039 learned_addr learned my addr 172.21.15.164:0/850238039 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): get_auth_request con 0x7f306409c180 auth_method 0 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient(hunting): handle_auth_done global_id 114386 payload 274 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 1 -- 172.21.15.164:0/850238039 >> v1:172.21.15.165:6789/0 conn(0x7f306413a5c0 legacy=0x7f306413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 1 -- 172.21.15.164:0/850238039 >> v1:172.21.15.164:6789/0 conn(0x7f306409d160 legacy=0x7f306409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 1 monclient: found mon.noname-b 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.103+0000 7f3069415640 1 -- 172.21.15.164:0/850238039 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f306409c560 con 0x7f306409c180 2024-09-10T16:19:04.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f3069415640 1 -- 172.21.15.164:0/850238039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 msgr2=0x7f306409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f3069415640 1 --2- 172.21.15.164:0/850238039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 0x7f306409ea40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f30600097a0 tx=0x7f3060029f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f9d9caccabc3317 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 1 -- 172.21.15.164:0/850238039 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060037070 con 0x7f306409c180 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: dump: 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.116 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 1 -- 172.21.15.164:0/850238039 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3060032440 con 0x7f306409c180 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 5 monclient: authenticate success, global_id 114386 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:04.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 1 -- 172.21.15.164:0/850238039 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060032810 con 0x7f306409c180 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 4 set_mon_vals no callback set 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f305bfff640 10 monclient: dump: 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.118 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 10 monclient: shutdown 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 -- 172.21.15.164:0/850238039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 msgr2=0x7f306409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 --2- 172.21.15.164:0/850238039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 0x7f306409ea40 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f30600097a0 tx=0x7f3060029f80 comp rx=0 tx=0).stop 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 -- 172.21.15.164:0/850238039 shutdown_connections 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 --2- 172.21.15.164:0/850238039 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306409c180 0x7f306409ea40 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 -- 172.21.15.164:0/850238039 >> 172.21.15.164:0/850238039 conn(0x7f30640a3110 msgr2=0x7f30640a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 -- 172.21.15.164:0/850238039 shutdown_connections 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.106+0000 7f306b679640 1 -- 172.21.15.164:0/850238039 wait complete. 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.107+0000 7f306b679640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.107+0000 7f306b679640 10 monclient: monmap: 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.107+0000 7f306b679640 1 Processor -- start 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.107+0000 7f306b679640 1 -- start start 2024-09-10T16:19:04.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.107+0000 7f306b679640 10 monclient: init 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306409c180 0x7f3064092a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): picked mon.noname-b con 0x7f306409c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 0x7f3064092f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): picked mon.noname-a con 0x7f306409d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 0x7f30640934a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): picked mon.noname-c con 0x7f306413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3064074f90 con 0x7f306409d160 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3064074e10 con 0x7f306409c180 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f306409f690 con 0x7f306413a5c0 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f306b679640 10 monclient(hunting): authenticate will time out at 1522.595283s 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 0x7f3064092f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 0x7f30640934a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 0x7f3064092f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42806/0 (socket says 172.21.15.164:42806) 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 1 -- 172.21.15.164:0/3698028851 learned_addr learned my addr 172.21.15.164:0/3698028851 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): get_auth_request con 0x7f306409d160 auth_method 0 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): get_auth_request con 0x7f306413a5c0 auth_method 0 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306409c180 0x7f3064092a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): get_auth_request con 0x7f306409c180 auth_method 0 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient(hunting): handle_auth_done global_id 114389 payload 824 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306409c180 msgr2=0x7f3064092a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f306409c180 0x7f3064092a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 msgr2=0x7f3064092f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 0x7f3064092f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 monclient: found mon.noname-c 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3069c16640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30640992f0 con 0x7f306413a5c0 2024-09-10T16:19:04.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.108+0000 7f3068c14640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f306409d160 0x7f3064092f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.110+0000 7f3069c16640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 msgr2=0x7f30640934a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.110+0000 7f3069c16640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 0x7f30640934a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c780 tx=0x7f305c00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c016020 con 0x7f306413a5c0 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: dump: 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.125 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f305c00ac60 con 0x7f306413a5c0 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 5 monclient: authenticate success, global_id 114389 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f305c012660 con 0x7f306413a5c0 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 10 monclient: _renew_subs 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f306409e1f0 con 0x7f306413a5c0 2024-09-10T16:19:04.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f3059ffb640 10 monclient: dump: 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.127 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f305b7fe640 4 set_mon_vals no callback set 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 10 monclient: _renew_subs 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.111+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3064095590 con 0x7f306413a5c0 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f305c019020 con 0x7f306413a5c0 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f306b679640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f306b679640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 0x7f3064063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f306402e9a0 con 0x7f306402e5c0 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 0x7f3064063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient: get_auth_request con 0x7f306402e5c0 auth_method 0 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3059ffb640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f304803e260 0x7f3048040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7f305c052be0 con 0x7f306413a5c0 2024-09-10T16:19:04.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 10 monclient(hunting): handle_auth_done global_id 114392 payload 824 2024-09-10T16:19:04.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3069415640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f306409ea40 con 0x7f306402e5c0 2024-09-10T16:19:04.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3068c14640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f304803e260 0x7f3048040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.112+0000 7f3068c14640 10 monclient: get_auth_request con 0x7f304803e260 auth_method 0 2024-09-10T16:19:04.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3069415640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 msgr2=0x7f3064063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3069415640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 0x7f3064063d50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3060034000 tx=0x7f30600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060037040 con 0x7f306402e5c0 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3059ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.114+0000 7f3059ffb640 10 monclient: dump: 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3068c14640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f304803e260 0x7f3048040720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3054007d50 tx=0x7f3054007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f306003c820 con 0x7f306402e5c0 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 msgr2=0x7f3064063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 0x7f3064063d50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3060034000 tx=0x7f30600046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 msgr2=0x7f3064063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.115+0000 7f3059ffb640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306402e5c0 0x7f3064063d50 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3060034000 tx=0x7f30600046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f306b679640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f306b679640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 0x7f30641d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f306b679640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3064073770 con 0x7f3064064290 2024-09-10T16:19:04.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 0x7f30641d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient: get_auth_request con 0x7f3064064290 auth_method 0 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 10 monclient(hunting): handle_auth_done global_id 114395 payload 824 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.121+0000 7f3069415640 1 -- 172.21.15.164:0/3698028851 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f30641396d0 con 0x7f3064064290 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3069415640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 msgr2=0x7f30641d57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3069415640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 0x7f30641d57c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3060029fc0 tx=0x7f3060008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3060050070 con 0x7f3064064290 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3059ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3059ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.123+0000 7f3059ffb640 10 monclient: dump: 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f306004c810 con 0x7f3064064290 2024-09-10T16:19:04.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 msgr2=0x7f30641d57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 0x7f30641d57c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3060029fc0 tx=0x7f3060008030 comp rx=0 tx=0).stop 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 msgr2=0x7f30641d57c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3059ffb640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3064064290 0x7f30641d57c0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3060029fc0 tx=0x7f3060008030 comp rx=0 tx=0).stop 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:04.137 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 164692, 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.138 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.139 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.140 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:04.141 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.142 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.143 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.144 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:04.145 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.146 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:04.147 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.148 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.149 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:04.150 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f304803e260 msgr2=0x7f3048040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f304803e260 0x7f3048040720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f3054007d50 tx=0x7f3054007ca0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 10 monclient: shutdown 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 msgr2=0x7f30640934a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f3069c16640 1 -- 172.21.15.164:0/3698028851 reap_dead start 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 0x7f30640934a0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f305c00c780 tx=0x7f305c00cc40 comp rx=0 tx=0).stop 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 shutdown_connections 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 --2- 172.21.15.164:0/3698028851 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f306413a5c0 0x7f30640934a0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 >> 172.21.15.164:0/3698028851 conn(0x7f30640a3110 msgr2=0x7f3064139c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 shutdown_connections 2024-09-10T16:19:04.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.124+0000 7f306b679640 1 -- 172.21.15.164:0/3698028851 wait complete. 2024-09-10T16:19:04.153 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:04.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:19:04.153 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:19:04.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.250+0000 7f2c11b64640 10 monclient: get_monmap_and_config 2024-09-10T16:19:04.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.250+0000 7f2c11b64640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.250+0000 7f2c11b64640 10 monclient: monmap: 2024-09-10T16:19:04.252 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:04.251450+0000 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:04.251450+0000 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.253 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:04.254 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:04.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 Processor -- start 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 -- start start 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient: init 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-10T16:19:04.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2c0c09d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-b con 0x7f2c0c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2c0c09e420 0x7f2c0c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-a con 0x7f2c0c09e420 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c0c0a0870 con 0x7f2c0c09d930 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2c0c09d3e0 con 0x7f2c0c09e420 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c074c70 con 0x7f2c0c09ab50 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.251+0000 7f2c11b64640 10 monclient(hunting): authenticate will time out at 1522.738941s 2024-09-10T16:19:04.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2c0c09e420 0x7f2c0c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:35000/0 (socket says 172.21.15.164:35000) 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2c0c09e420 0x7f2c0c13bd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42820/0 (socket says 172.21.15.164:42820) 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 1 -- 172.21.15.164:0/2917772533 learned_addr learned my addr 172.21.15.164:0/2917772533 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): get_auth_request con 0x7f2c0c09ab50 auth_method 0 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): get_auth_request con 0x7f2c0c09e420 auth_method 0 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_done global_id 144147 payload 274 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 -- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 msgr2=0x7f2c0c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 -- 172.21.15.164:0/2917772533 >> v1:172.21.15.165:6789/0 conn(0x7f2c0c09d930 legacy=0x7f2c0c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 monclient: found mon.noname-a 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c0bfff640 1 -- 172.21.15.164:0/2917772533 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c09ecb0 con 0x7f2c0c09e420 2024-09-10T16:19:04.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.252+0000 7f2c03fff640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0bfff640 1 -- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 msgr2=0x7f2c0c13bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0bfff640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c13bd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400aed0 tx=0x7f2c0402b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3efdcf38166893bf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 1 -- 172.21.15.164:0/2917772533 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04037070 con 0x7f2c0c09e420 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: dump: 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 5 monclient: authenticate success, global_id 144147 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 1 -- 172.21.15.164:0/2917772533 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c04008850 con 0x7f2c0c09e420 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:04.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 4 set_mon_vals no callback set 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 1 -- 172.21.15.164:0/2917772533 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c04008c20 con 0x7f2c0c09e420 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c0affd640 10 monclient: dump: 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.274 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 10 monclient: shutdown 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 -- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 msgr2=0x7f2c0c13bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c13bd10 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2c0400aed0 tx=0x7f2c0402b6b0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 -- 172.21.15.164:0/2917772533 shutdown_connections 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09ab50 0x7f2c0c1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 --2- 172.21.15.164:0/2917772533 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c13bd10 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 -- 172.21.15.164:0/2917772533 >> 172.21.15.164:0/2917772533 conn(0x7f2c0c0a3110 msgr2=0x7f2c0c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 -- 172.21.15.164:0/2917772533 shutdown_connections 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.255+0000 7f2c11b64640 1 -- 172.21.15.164:0/2917772533 wait complete. 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.256+0000 7f2c11b64640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.256+0000 7f2c11b64640 10 monclient: monmap: 2024-09-10T16:19:04.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:04.276 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.256+0000 7f2c11b64640 1 Processor -- start 2024-09-10T16:19:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.256+0000 7f2c11b64640 1 -- start start 2024-09-10T16:19:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.256+0000 7f2c11b64640 10 monclient: init 2024-09-10T16:19:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c09ab50 0x7f2c0c1c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-b con 0x7f2c0c09ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-c con 0x7f2c0c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c1c5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): picked mon.noname-a con 0x7f2c0c09e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c0a2940 con 0x7f2c0c09e420 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c09ef30 con 0x7f2c0c09ab50 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2c0c0a2ac0 con 0x7f2c0c09d930 2024-09-10T16:19:04.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c11b64640 10 monclient(hunting): authenticate will time out at 1522.744322s 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c1c5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c1c5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42834/0 (socket says 172.21.15.164:42834) 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 1 -- 172.21.15.164:0/3912756772 learned_addr learned my addr 172.21.15.164:0/3912756772 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): get_auth_request con 0x7f2c0c09e420 auth_method 0 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): get_auth_request con 0x7f2c0c09d930 auth_method 0 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c09ab50 0x7f2c0c1c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): get_auth_request con 0x7f2c0c09ab50 auth_method 0 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient(hunting): handle_auth_done global_id 114401 payload 824 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c09ab50 msgr2=0x7f2c0c1c8c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c09ab50 0x7f2c0c1c8c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 msgr2=0x7f2c0c1c5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c1c5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 monclient: found mon.noname-c 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c03fff640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c0c1302c0 con 0x7f2c0c09d930 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.257+0000 7f2c0bfff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2c0c09e420 0x7f2c0c1c5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.259+0000 7f2c03fff640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 msgr2=0x7f2c0c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.259+0000 7f2c03fff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400e890 tx=0x7f2bf400ed50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf400cb50 con 0x7f2c0c09d930 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: dump: 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.283 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bf400ccf0 con 0x7f2c0c09d930 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 5 monclient: authenticate success, global_id 114401 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 monclient: _renew_subs 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c0c131760 con 0x7f2c0c09d930 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bf400cea0 con 0x7f2c0c09d930 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 10 monclient: dump: 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 monclient: _renew_subs 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c0a7fc640 4 set_mon_vals no callback set 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c0c131480 con 0x7f2c0c09d930 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 0x7f2c0c02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c09dd10 con 0x7f2c0c02d440 2024-09-10T16:19:04.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c03fff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400e890 tx=0x7f2bf400ed50 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f2bf4052c50 osd_map(436..436 src has 184..436) 2024-09-10T16:19:04.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.260+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2bf40042d0 con 0x7f2c0c09d930 2024-09-10T16:19:04.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 0x7f2c0c02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient: get_auth_request con 0x7f2c0c02d440 auth_method 0 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c08ff9640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bec03e270 0x7f2bec040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_done global_id 134236 payload 824 2024-09-10T16:19:04.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0b7fe640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2c0c09c1d0 con 0x7f2c0c02d440 2024-09-10T16:19:04.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0bfff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bec03e270 0x7f2bec040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.261+0000 7f2c0bfff640 10 monclient: get_auth_request con 0x7f2bec03e270 auth_method 0 2024-09-10T16:19:04.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c0b7fe640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 msgr2=0x7f2c0c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c0b7fe640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 0x7f2c0c02d820 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00f7a0 tx=0x7f2bfc00fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc018030 con 0x7f2c0c02d440 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c08ff9640 10 monclient: dump: 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.292 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.293 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.263+0000 7f2c0bfff640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bec03e270 0x7f2bec040730 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c1310d0 tx=0x7f2c0400a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2bfc00d9b0 con 0x7f2c0c02d440 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 msgr2=0x7f2c0c02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 0x7f2c0c02d820 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00f7a0 tx=0x7f2bfc00fc60 comp rx=0 tx=0).stop 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 msgr2=0x7f2c0c02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:04.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.264+0000 7f2c08ff9640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02d440 0x7f2c0c02d820 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c11b64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c11b64640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 0x7f2c0c02e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c11b64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2c0c06dac0 con 0x7f2c0c02dd60 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 0x7f2c0c02e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient: get_auth_request con 0x7f2c0c02dd60 auth_method 0 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 10 monclient(hunting): handle_auth_done global_id 134239 payload 824 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.270+0000 7f2c0b7fe640 1 -- 172.21.15.164:0/3912756772 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2c0c073610 con 0x7f2c0c02dd60 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.272+0000 7f2c0b7fe640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 msgr2=0x7f2c0c02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.272+0000 7f2c0b7fe640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 0x7f2c0c02e170 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00f7a0 tx=0x7f2bfc003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bfc01dd70 con 0x7f2c0c02dd60 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 10 monclient: dump: 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f2bfc01dd70 con 0x7f2c0c02dd60 2024-09-10T16:19:04.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 msgr2=0x7f2c0c02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 0x7f2c0c02e170 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f2bfc00f7a0 tx=0x7f2bfc003760 comp rx=0 tx=0).stop 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 msgr2=0x7f2c0c02e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2c0c02dd60 0x7f2c0c02e170 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:04.300 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 122, 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 68677, 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.301 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.302 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.303 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:04.304 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:04.305 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.306 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.307 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.308 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.309 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.310 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:04.311 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:04.312 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:04.313 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:04.314 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bec03e270 msgr2=0x7f2bec040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2bec03e270 0x7f2bec040730 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f2c0c1310d0 tx=0x7f2c0400a000 comp rx=0 tx=0).stop 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 10 monclient: shutdown 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 msgr2=0x7f2c0c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2bf400e890 tx=0x7f2bf400ed50 comp rx=0 tx=0).stop 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c0bfff640 1 -- 172.21.15.164:0/3912756772 reap_dead start 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c08ff9640 1 -- 172.21.15.164:0/3912756772 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (unknown 0 0 0) 0x7f2bf4052c50 con 0x7f2c0c09d930 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 shutdown_connections 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 --2- 172.21.15.164:0/3912756772 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2c0c09d930 0x7f2c0c134c90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 >> 172.21.15.164:0/3912756772 conn(0x7f2c0c0a3110 msgr2=0x7f2c0c0707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 shutdown_connections 2024-09-10T16:19:04.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.273+0000 7f2c11b64640 1 -- 172.21.15.164:0/3912756772 wait complete. 2024-09-10T16:19:04.318 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:19:04.319 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:19:04.319 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:19:04.320 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:19:04.332 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:19:04.332 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:19:04.332 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.428+0000 7f6fafb24640 10 monclient: get_monmap_and_config 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.428+0000 7f6fafb24640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.428+0000 7f6fafb24640 10 monclient: monmap: 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:04.429518+0000 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:04.429518+0000 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:04.431 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:04.431 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:04.431 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:04.431 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:04.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 Processor -- start 2024-09-10T16:19:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 -- start start 2024-09-10T16:19:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient: init 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fa809d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fa809ab50 0x7f6fa81397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa809ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fa809e420 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa80a0870 con 0x7f6fa809d930 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fa8074c70 con 0x7f6fa809e420 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6fa809d3e0 con 0x7f6fa809ab50 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fafb24640 10 monclient(hunting): authenticate will time out at 1522.916896s 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6fa809ab50 0x7f6fa81397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.429+0000 7f6fae0c1640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f6fa809e420 legacy=0x7f6fa813bd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad8c0640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6fa809d930 0x7f6fa809dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35780/0 (socket says 172.21.15.164:35780) 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad8c0640 1 -- 172.21.15.164:0/2074508168 learned_addr learned my addr 172.21.15.164:0/2074508168 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): get_auth_request con 0x7f6fa809ab50 auth_method 0 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient(hunting): handle_auth_done global_id 144153 payload 274 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 1 -- 172.21.15.164:0/2074508168 >> v1:172.21.15.165:6789/0 conn(0x7f6fa809e420 legacy=0x7f6fa813bd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 1 -- 172.21.15.164:0/2074508168 >> v1:172.21.15.164:6789/0 conn(0x7f6fa809d930 legacy=0x7f6fa809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 1 monclient: found mon.noname-a 2024-09-10T16:19:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.430+0000 7f6fad0bf640 1 -- 172.21.15.164:0/2074508168 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa809ecb0 con 0x7f6fa809ab50 2024-09-10T16:19:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fad0bf640 1 -- 172.21.15.164:0/2074508168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809ab50 msgr2=0x7f6fa81397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fad0bf640 1 --2- 172.21.15.164:0/2074508168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809ab50 0x7f6fa81397c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f980097a0 tx=0x7f6f98029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9391cb02c3099ac8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 1 -- 172.21.15.164:0/2074508168 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98037070 con 0x7f6fa809ab50 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: dump: 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 1 -- 172.21.15.164:0/2074508168 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f98032640 con 0x7f6fa809ab50 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 5 monclient: authenticate success, global_id 144153 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 1 -- 172.21.15.164:0/2074508168 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f98032a10 con 0x7f6fa809ab50 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fac8be640 10 monclient: dump: 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 4 set_mon_vals no callback set 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 10 monclient: shutdown 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 -- 172.21.15.164:0/2074508168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809ab50 msgr2=0x7f6fa81397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 --2- 172.21.15.164:0/2074508168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809ab50 0x7f6fa81397c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f980097a0 tx=0x7f6f98029fd0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 -- 172.21.15.164:0/2074508168 shutdown_connections 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 --2- 172.21.15.164:0/2074508168 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809ab50 0x7f6fa81397c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 -- 172.21.15.164:0/2074508168 >> 172.21.15.164:0/2074508168 conn(0x7f6fa80a3110 msgr2=0x7f6fa80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 -- 172.21.15.164:0/2074508168 shutdown_connections 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.433+0000 7f6fafb24640 1 -- 172.21.15.164:0/2074508168 wait complete. 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.434+0000 7f6fafb24640 10 monclient: build_initial_monmap 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.434+0000 7f6fafb24640 10 monclient: monmap: 2024-09-10T16:19:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.434+0000 7f6fafb24640 1 Processor -- start 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.434+0000 7f6fafb24640 1 -- start start 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.434+0000 7f6fafb24640 10 monclient: init 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fa809ab50 0x7f6fa81cd8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-c con 0x7f6fa809ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 0x7f6fa81cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-b con 0x7f6fa809d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): picked mon.noname-a con 0x7f6fa809e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa80a2940 con 0x7f6fa809e420 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6fa809ef30 con 0x7f6fa809d930 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6fa80a2ac0 con 0x7f6fa809ab50 2024-09-10T16:19:04.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fafb24640 10 monclient(hunting): authenticate will time out at 1522.922201s 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fa809ab50 0x7f6fa81cd8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54026/0 (socket says 172.21.15.164:54026) 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 -- 172.21.15.164:0/882727777 learned_addr learned my addr 172.21.15.164:0/882727777 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): get_auth_request con 0x7f6fa809ab50 auth_method 0 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): get_auth_request con 0x7f6fa809e420 auth_method 0 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad0bf640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 msgr2=0x7f6fa81cde20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:04.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad0bf640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 0x7f6fa81cde20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fad8c0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient(hunting): handle_auth_done global_id 144156 payload 824 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fa809ab50 msgr2=0x7f6fa81cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fa809ab50 0x7f6fa81cd8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 msgr2=0x7f6fa81cde20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:04.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 0x7f6fa81cde20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient: _finish_hunting 0 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 monclient: found mon.noname-a 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.435+0000 7f6fae0c1640 1 -- 172.21.15.164:0/882727777 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fa8073a30 con 0x7f6fa809e420 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fae0c1640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 msgr2=0x7f6fa8071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fae0c1640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000b9c0 tx=0x7f6fa000be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa000cd30 con 0x7f6fa809e420 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: dump: 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.450 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fa0010070 con 0x7f6fa809e420 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fafb24640 5 monclient: authenticate success, global_id 144156 2024-09-10T16:19:04.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fa0015690 con 0x7f6fa809e420 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fafb24640 10 monclient: _renew_subs 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fafb24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6fa80a45c0 con 0x7f6fa809e420 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9e7fc640 10 monclient: dump: 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:04.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.437+0000 7f6f9ffff640 4 set_mon_vals no callback set 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 10 monclient: _renew_subs 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6fa8072e00 con 0x7f6fa809e420 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6fa001d390 con 0x7f6fa809e420 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.438+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6fa809dd90 con 0x7f6fa809e420 2024-09-10T16:19:04.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.439+0000 7f6f9e7fc640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 0x7f6f8c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.439+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7f6fa0052fc0 con 0x7f6fa809e420 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.439+0000 7f6fad8c0640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 0x7f6f8c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.439+0000 7f6fad8c0640 10 monclient: get_auth_request con 0x7f6f8c03e1f0 auth_method 0 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.441+0000 7f6fad8c0640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 0x7f6f8c0406b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4007c90 tx=0x7f6fa4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.443+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6fa001a060 con 0x7f6fa809e420 2024-09-10T16:19:04.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.443+0000 7f6f9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:04.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.443+0000 7f6f9e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6fafb24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:04.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6fafb24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6fafb24640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6fa806daa0 con 0x7f6fa809e420 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6f9e7fc640 1 -- 172.21.15.164:0/882727777 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6fa00519b0 con 0x7f6fa809e420 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6f9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.689+0000 7f6f9e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:04.691 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 msgr2=0x7f6f8c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 0x7f6f8c0406b0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6fa4007c90 tx=0x7f6fa4007be0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 10 monclient: shutdown 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 msgr2=0x7f6fa8071620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6fa000b9c0 tx=0x7f6fa000be80 comp rx=0 tx=0).stop 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 shutdown_connections 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f8c03e1f0 0x7f6f8c0406b0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6fa809e420 0x7f6fa8071620 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6fa809d930 0x7f6fa81cde20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 --2- 172.21.15.164:0/882727777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6fa809ab50 0x7f6fa81cd8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 >> 172.21.15.164:0/882727777 conn(0x7f6fa80a3110 msgr2=0x7f6fa8131910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.691+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 shutdown_connections 2024-09-10T16:19:04.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:04.692+0000 7f6fafb24640 1 -- 172.21.15.164:0/882727777 wait complete. 2024-09-10T16:19:07.704 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 10 monclient: get_monmap_and_config 2024-09-10T16:19:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 10 monclient: build_initial_monmap 2024-09-10T16:19:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 10 monclient: monmap: 2024-09-10T16:19:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:07.813078+0000 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:07.813078+0000 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:07.814 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 1 Processor -- start 2024-09-10T16:19:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 1 -- start start 2024-09-10T16:19:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.812+0000 7f3d2062b640 10 monclient: init 2024-09-10T16:19:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d1806cb70 0x7f3d18134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d1806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d180a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d181383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d180a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d1809ce50 con 0x7f3d180a46a0 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1806db60 con 0x7f3d1806cb70 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1806cf50 con 0x7f3d180a4ab0 2024-09-10T16:19:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d2062b640 10 monclient(hunting): authenticate will time out at 1526.300669s 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3d1806cb70 0x7f3d18134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d181383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1dbc6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3d180a46a0 0x7f3d18137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35788/0 (socket says 172.21.15.164:35788) 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1dbc6640 1 -- 172.21.15.164:0/2991455739 learned_addr learned my addr 172.21.15.164:0/2991455739 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): get_auth_request con 0x7f3d1806cb70 auth_method 0 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): get_auth_request con 0x7f3d180a4ab0 auth_method 0 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1e3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.813+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 10 monclient(hunting): handle_auth_done global_id 144159 payload 274 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 1 -- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 msgr2=0x7f3d181383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d181383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 1 -- 172.21.15.164:0/2991455739 >> v1:172.21.15.164:6789/0 conn(0x7f3d180a46a0 legacy=0x7f3d18137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 10 monclient: _finish_hunting 0 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 1 monclient: found mon.noname-a 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1e3c7640 1 -- 172.21.15.164:0/2991455739 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1809cb70 con 0x7f3d1806cb70 2024-09-10T16:19:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.814+0000 7f3d1ebc8640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d181383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.816+0000 7f3d1e3c7640 1 -- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d1806cb70 msgr2=0x7f3d18134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.816+0000 7f3d1e3c7640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d1806cb70 0x7f3d18134f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d140097d0 tx=0x7f3d1402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac7cf957cfbded6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 1 -- 172.21.15.164:0/2991455739 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14037070 con 0x7f3d1806cb70 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: dump: 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: _finish_auth 0 2024-09-10T16:19:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: _check_auth_tickets 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 5 monclient: authenticate success, global_id 144159 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 1 -- 172.21.15.164:0/2991455739 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d14032440 con 0x7f3d1806cb70 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 1 -- 172.21.15.164:0/2991455739 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14032810 con 0x7f3d1806cb70 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 4 set_mon_vals no callback set 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d1d3c5640 10 monclient: dump: 2024-09-10T16:19:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 10 monclient: shutdown 2024-09-10T16:19:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 -- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d1806cb70 msgr2=0x7f3d18134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d1806cb70 0x7f3d18134f70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3d140097d0 tx=0x7f3d1402a000 comp rx=0 tx=0).stop 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 -- 172.21.15.164:0/2991455739 shutdown_connections 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d181383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 --2- 172.21.15.164:0/2991455739 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d1806cb70 0x7f3d18134f70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 -- 172.21.15.164:0/2991455739 >> 172.21.15.164:0/2991455739 conn(0x7f3d180aa900 msgr2=0x7f3d180aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 -- 172.21.15.164:0/2991455739 shutdown_connections 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.817+0000 7f3d2062b640 1 -- 172.21.15.164:0/2991455739 wait complete. 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.818+0000 7f3d2062b640 10 monclient: build_initial_monmap 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.818+0000 7f3d2062b640 10 monclient: monmap: 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:07.833 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.818+0000 7f3d2062b640 1 Processor -- start 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.818+0000 7f3d2062b640 1 -- start start 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.818+0000 7f3d2062b640 10 monclient: init 2024-09-10T16:19:07.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 0x7f3d180a40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-b con 0x7f3d1806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 0x7f3d1809d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-a con 0x7f3d180a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): picked mon.noname-c con 0x7f3d180a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d18138a70 con 0x7f3d180a46a0 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d181388f0 con 0x7f3d1806cb70 2024-09-10T16:19:07.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d18140040 con 0x7f3d180a4ab0 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d2062b640 10 monclient(hunting): authenticate will time out at 1526.306324s 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 0x7f3d1809d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49998/0 (socket says 172.21.15.164:49998) 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 -- 172.21.15.164:0/402905914 learned_addr learned my addr 172.21.15.164:0/402905914 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): get_auth_request con 0x7f3d180a4ab0 auth_method 0 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:07.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): get_auth_request con 0x7f3d180a46a0 auth_method 0 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1e3c7640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 msgr2=0x7f3d180a40b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1e3c7640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 0x7f3d180a40b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient(hunting): handle_auth_done global_id 114410 payload 824 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 msgr2=0x7f3d180a40b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 0x7f3d180a40b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 msgr2=0x7f3d1809d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 0x7f3d1809d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient: _finish_hunting 0 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 monclient: found mon.noname-c 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1ebc8640 1 -- 172.21.15.164:0/402905914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d180a06a0 con 0x7f3d180a4ab0 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.819+0000 7f3d1dbc6640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 0x7f3d1809d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d1ebc8640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 msgr2=0x7f3d1809db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d1ebc8640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009f00 tx=0x7f3d1000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10018030 con 0x7f3d180a4ab0 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: dump: 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:07.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: _finish_auth 0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1000ce50 con 0x7f3d180a4ab0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 5 monclient: authenticate success, global_id 114410 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10003bb0 con 0x7f3d180a4ab0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 10 monclient: _renew_subs 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d180a0980 con 0x7f3d180a4ab0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d1cbc4640 4 set_mon_vals no callback set 2024-09-10T16:19:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d0effd640 10 monclient: dump: 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 10 monclient: _renew_subs 2024-09-10T16:19:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.822+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d18134030 con 0x7f3d180a4ab0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3d10005d10 con 0x7f3d180a4ab0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d2062b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d2062b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d2062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d18134f70 con 0x7f3d180a4ab0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d0effd640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 0x7f3cfc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7f3d10053780 con 0x7f3d180a4ab0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.823+0000 7f3d1e3c7640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 0x7f3cfc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.824+0000 7f3d1e3c7640 10 monclient: get_auth_request con 0x7f3cfc03e210 auth_method 0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.826+0000 7f3d1e3c7640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 0x7f3cfc0406d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d140097a0 tx=0x7f3d1402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.828+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3d10022070 con 0x7f3d180a4ab0 2024-09-10T16:19:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.828+0000 7f3d0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:07.828+0000 7f3d0effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d2062b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d2062b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d2062b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3d1809ed00 con 0x7f3d180a4ab0 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d0effd640 1 -- 172.21.15.164:0/402905914 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f3d10010040 con 0x7f3d180a4ab0 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.069+0000 7f3d0effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:08.071 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 msgr2=0x7f3cfc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 0x7f3cfc0406d0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d140097a0 tx=0x7f3d1402e000 comp rx=0 tx=0).stop 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 10 monclient: shutdown 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 msgr2=0x7f3d1809db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3d10009f00 tx=0x7f3d1000c490 comp rx=0 tx=0).stop 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 shutdown_connections 2024-09-10T16:19:08.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3cfc03e210 0x7f3cfc0406d0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d180a4ab0 0x7f3d1809db70 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d180a46a0 0x7f3d1809d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 --2- 172.21.15.164:0/402905914 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d1806cb70 0x7f3d180a40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 >> 172.21.15.164:0/402905914 conn(0x7f3d180aa900 msgr2=0x7f3d1812e200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 shutdown_connections 2024-09-10T16:19:08.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:08.071+0000 7f3d2062b640 1 -- 172.21.15.164:0/402905914 wait complete. 2024-09-10T16:19:11.084 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.192+0000 7f75c8260640 10 monclient: get_monmap_and_config 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.192+0000 7f75c8260640 10 monclient: build_initial_monmap 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.192+0000 7f75c8260640 10 monclient: monmap: 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:11.193388+0000 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:11.193388+0000 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:11.194 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 Processor -- start 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 -- start start 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient: init 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75c009a430 0x7f75c009a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-a con 0x7f75c009a430 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:11.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-b con 0x7f75c0074cb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 0x7f75c013c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-c con 0x7f75c00a2820 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f75c009b290 con 0x7f75c009a430 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f75c00a2320 con 0x7f75c00a2820 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f75c009cbd0 con 0x7f75c0074cb0 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.193+0000 7f75c8260640 10 monclient(hunting): authenticate will time out at 1529.680926s 2024-09-10T16:19:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75c009a430 0x7f75c009a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c67fd640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 msgr2=0x7f75c013c600 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50000/0 (socket says 172.21.15.164:50000) 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 1 -- 172.21.15.164:0/125951167 learned_addr learned my addr 172.21.15.164:0/125951167 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f75c009a430 0x7f75c009a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54052/0 (socket says 172.21.15.164:54052) 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): get_auth_request con 0x7f75c0074cb0 auth_method 0 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): get_auth_request con 0x7f75c009a430 auth_method 0 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c67fd640 1 --2- 172.21.15.164:0/125951167 >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 0x7f75c013c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient(hunting): handle_auth_done global_id 144165 payload 274 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 -- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 msgr2=0x7f75c009ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 -- 172.21.15.164:0/125951167 >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 msgr2=0x7f75c013c600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/125951167 >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 0x7f75c013c600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient: _finish_hunting 0 2024-09-10T16:19:11.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 monclient: found mon.noname-a 2024-09-10T16:19:11.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:11.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c5ffc640 1 -- 172.21.15.164:0/125951167 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c009c2c0 con 0x7f75c009a430 2024-09-10T16:19:11.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.194+0000 7f75c57fb640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c5ffc640 1 -- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c009a430 msgr2=0x7f75c009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c009a430 0x7f75c009a810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0097d0 tx=0x7f75bc02a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce84b057189ae279 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 1 -- 172.21.15.164:0/125951167 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc037070 con 0x7f75c009a430 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: dump: 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: _finish_auth 0 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: _check_auth_tickets 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 1 -- 172.21.15.164:0/125951167 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75bc032440 con 0x7f75c009a430 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 5 monclient: authenticate success, global_id 144165 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 1 -- 172.21.15.164:0/125951167 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75bc032810 con 0x7f75c009a430 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 4 set_mon_vals no callback set 2024-09-10T16:19:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c4ffa640 10 monclient: dump: 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 10 monclient: shutdown 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 -- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c009a430 msgr2=0x7f75c009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c009a430 0x7f75c009a810 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0097d0 tx=0x7f75bc02a060 comp rx=0 tx=0).stop 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 -- 172.21.15.164:0/125951167 shutdown_connections 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 --2- 172.21.15.164:0/125951167 >> v2:172.21.15.165:3300/0 conn(0x7f75c00a2820 0x7f75c013c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c0074cb0 0x7f75c009ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 --2- 172.21.15.164:0/125951167 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c009a430 0x7f75c009a810 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.197+0000 7f75c8260640 1 -- 172.21.15.164:0/125951167 >> 172.21.15.164:0/125951167 conn(0x7f75c00a3110 msgr2=0x7f75c00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.198+0000 7f75c8260640 1 -- 172.21.15.164:0/125951167 shutdown_connections 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.198+0000 7f75c8260640 1 -- 172.21.15.164:0/125951167 wait complete. 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.198+0000 7f75c8260640 10 monclient: build_initial_monmap 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.198+0000 7f75c8260640 10 monclient: monmap: 2024-09-10T16:19:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.198+0000 7f75c8260640 1 Processor -- start 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 -- start start 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient: init 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:19:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 0x7f75c0072760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-b con 0x7f75c0074cb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-c con 0x7f75c009a430 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): picked mon.noname-a con 0x7f75c00a2820 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f75c009d450 con 0x7f75c00a2820 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f75c009d2d0 con 0x7f75c0074cb0 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f75c013cb40 con 0x7f75c009a430 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c8260640 10 monclient(hunting): authenticate will time out at 1529.686589s 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c5ffc640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 msgr2=0x7f75c0072760 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50002/0 (socket says 172.21.15.164:50002) 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54066/0 (socket says 172.21.15.164:54066) 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 1 -- 172.21.15.164:0/3410726962 learned_addr learned my addr 172.21.15.164:0/3410726962 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): get_auth_request con 0x7f75c009a430 auth_method 0 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 0x7f75c0072760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): get_auth_request con 0x7f75c00a2820 auth_method 0 2024-09-10T16:19:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient(hunting): handle_auth_done global_id 144168 payload 824 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 -- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 msgr2=0x7f75c0072ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 -- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 msgr2=0x7f75c0072760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 0x7f75c0072760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient: _finish_hunting 0 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 monclient: found mon.noname-a 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c67fd640 1 -- 172.21.15.164:0/3410726962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c012cb20 con 0x7f75c00a2820 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.199+0000 7f75c57fb640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c67fd640 1 -- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 msgr2=0x7f75c00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c67fd640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75c006e780 tx=0x7f75b8009eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b80066f0 con 0x7f75c00a2820 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: dump: 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:11.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75b8006890 con 0x7f75c00a2820 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 5 monclient: authenticate success, global_id 144168 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 10 monclient: _renew_subs 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75b8015440 con 0x7f75c00a2820 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c00a5fd0 con 0x7f75c00a2820 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b67fc640 10 monclient: dump: 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75b7fff640 4 set_mon_vals no callback set 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 10 monclient: _renew_subs 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.202+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c0073d50 con 0x7f75c00a2820 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f75b801e3f0 con 0x7f75c00a2820 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c8260640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75b67fc640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 0x7f75a4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c8260640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c8260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c013c5e0 con 0x7f75c00a2820 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(436..436 src has 184..436) ==== 4787+0+0 (secure 0 0 0) 0x7f75b8053d10 con 0x7f75c00a2820 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 0x7f75a4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.203+0000 7f75c5ffc640 10 monclient: get_auth_request con 0x7f75a403e1c0 auth_method 0 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.206+0000 7f75c5ffc640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 0x7f75a4040680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f75c006d500 tx=0x7f75bc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.208+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f75b8052700 con 0x7f75c00a2820 2024-09-10T16:19:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.208+0000 7f75b67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.208+0000 7f75b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.452+0000 7f75c8260640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.452+0000 7f75c8260640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.452+0000 7f75c8260640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.452+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f75c0077260 con 0x7f75c00a2820 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.453+0000 7f75b67fc640 1 -- 172.21.15.164:0/3410726962 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f75b8011cf0 con 0x7f75c00a2820 2024-09-10T16:19:11.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.453+0000 7f75b67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:11.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.453+0000 7f75b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:11.455 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":122,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:11.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 msgr2=0x7f75a4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 0x7f75a4040680 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f75c006d500 tx=0x7f75bc02e000 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 10 monclient: shutdown 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 msgr2=0x7f75c00731e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f75c006e780 tx=0x7f75b8009eb0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 shutdown_connections 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f75a403e1c0 0x7f75a4040680 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f75c00a2820 0x7f75c00731e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f75c009a430 0x7f75c0072ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 --2- 172.21.15.164:0/3410726962 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f75c0074cb0 0x7f75c0072760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 >> 172.21.15.164:0/3410726962 conn(0x7f75c00a3110 msgr2=0x7f75c0098510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:11.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 shutdown_connections 2024-09-10T16:19:11.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:11.455+0000 7f75c8260640 1 -- 172.21.15.164:0/3410726962 wait complete. 2024-09-10T16:19:14.468 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:14.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.575+0000 7f95fbe62640 10 monclient: get_monmap_and_config 2024-09-10T16:19:14.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.575+0000 7f95fbe62640 10 monclient: build_initial_monmap 2024-09-10T16:19:14.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.575+0000 7f95fbe62640 10 monclient: monmap: 2024-09-10T16:19:14.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:14.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:14.576818+0000 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:14.576818+0000 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:14.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.576+0000 7f95fbe62640 1 Processor -- start 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 1 -- start start 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient: init 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95f40a3a50 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-c con 0x7f95f40a2320 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f95f4134cb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:14.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f40748f0 con 0x7f95f4134cb0 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f4074bd0 con 0x7f95f40a3a50 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f95f40a2b20 con 0x7f95f40a2320 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fbe62640 10 monclient(hunting): authenticate will time out at 1533.064560s 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fa3ff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f95f4134cb0 0x7f95f413f570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:60018/0 (socket says 172.21.15.164:60018) 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 learned_addr learned my addr 172.21.15.164:0/1639618582 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 msgr2=0x7f95f413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95f40a3a50 legacy=0x7f95f409c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:14.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.577+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:14.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.579+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.164:6789/0 conn(0x7f95f4134cb0 legacy=0x7f95f413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:14.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.778+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95f40a3a50 legacy=0x7f95f409c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:14.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.778+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 msgr2=0x7f95f413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:14.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:14.778+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:15.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.179+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95f40a3a50 legacy=0x7f95f409c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:15.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.179+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 msgr2=0x7f95f413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:15.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.180+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:15.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.981+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 msgr2=0x7f95f413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:15.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.981+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95f40a3a50 legacy=0x7f95f409c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:15.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:15.981+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): tick 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 msgr2=0x7f95f413d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95f40a2320 0x7f95f413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95f40a3a50 legacy=0x7f95f409c970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.164:6789/0 conn(0x7f95f4134cb0 legacy=0x7f95f413f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient: _add_conns ranks=[2,3] 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f95d8003740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f95d80061d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f95d8008d00 con 0x7f95d80061d0 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95d8008eb0 con 0x7f95d8003740 2024-09-10T16:19:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95eb7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f93fd640 10 monclient(hunting): get_auth_request con 0x7f95d80061d0 auth_method 0 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f93fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f93fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f93fd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request con 0x7f95d8003740 auth_method 0 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f9bfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.577+0000 7f95f9bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.578+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:17.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:17.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request con 0x7f95d8003740 auth_method 0 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 10 monclient(hunting): get_auth_request con 0x7f95d80061d0 auth_method 0 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:17.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:17.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:17.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:17.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:17.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:17.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:17.779+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:18.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:18.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 10 monclient(hunting): get_auth_request con 0x7f95d80061d0 auth_method 0 2024-09-10T16:19:18.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:18.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:18.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request con 0x7f95d8003740 auth_method 0 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:18.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.180+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:18.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 10 monclient(hunting): get_auth_request con 0x7f95d80061d0 auth_method 0 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request con 0x7f95d8003740 auth_method 0 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:18.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f93fd640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.981+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:18.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:18.982+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1835: done (1 left) 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1836: read oid 16 snap -1 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1836: expect (ObjNum 638 snap 202 seq_num 638) 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1837: snap_create 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1834: finishing write tid 1 to smithi16534841-1 2024-09-10T16:19:19.142 INFO:tasks.rados.rados.0.smithi165.stdout:1834: finishing write tid 2 to smithi16534841-1 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: finishing write tid 3 to smithi16534841-1 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: finishing write tid 4 to smithi16534841-1 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: oid 1 updating version 0 to 1119 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: oid 1 version 1119 is already newer than 1117 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: oid 1 version 1119 is already newer than 1118 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1119 (ObjNum 645 snap 204 seq_num 645) dirty exists 2024-09-10T16:19:19.143 INFO:tasks.rados.rados.0.smithi165.stdout:1834: left oid 1 (ObjNum 645 snap 204 seq_num 645) 2024-09-10T16:19:19.156 INFO:tasks.rados.rados.0.smithi165.stdout:1836: expect (ObjNum 638 snap 202 seq_num 638) 2024-09-10T16:19:19.157 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:19.223 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1407852 2024-09-10T16:19:19.252 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 135200 2024-09-10T16:19:20.144 INFO:tasks.rados.rados.0.smithi165.stdout:1834: done (2 left) 2024-09-10T16:19:20.145 INFO:tasks.rados.rados.0.smithi165.stdout:1836: done (1 left) 2024-09-10T16:19:20.145 INFO:tasks.rados.rados.0.smithi165.stdout:1837: done (0 left) 2024-09-10T16:19:20.145 INFO:tasks.rados.rados.0.smithi165.stdout:1838: write oid 29 current snap is 205 2024-09-10T16:19:20.145 INFO:tasks.rados.rados.0.smithi165.stdout:1838: seq_num 646 ranges {457322=784506,1969250=443556} 2024-09-10T16:19:20.150 INFO:tasks.rados.rados.0.smithi165.stdout:1838: writing smithi16534841-29 from 457322 to 1241828 tid 1 2024-09-10T16:19:20.158 INFO:tasks.rados.rados.0.smithi165.stdout:1838: writing smithi16534841-29 from 1969250 to 2412806 tid 2 2024-09-10T16:19:20.158 INFO:tasks.rados.rados.0.smithi165.stdout:1839: read oid 5 snap 201 2024-09-10T16:19:20.158 INFO:tasks.rados.rados.0.smithi165.stdout:1839: expect (ObjNum 397 snap 130 seq_num 397) 2024-09-10T16:19:20.158 INFO:tasks.rados.rados.0.smithi165.stdout:1840: write oid 12 current snap is 205 2024-09-10T16:19:20.159 INFO:tasks.rados.rados.0.smithi165.stdout:1840: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2024-09-10T16:19:20.164 INFO:tasks.rados.rados.0.smithi165.stdout:1840: writing smithi16534841-12 from 474129 to 1050237 tid 1 2024-09-10T16:19:20.171 INFO:tasks.rados.rados.0.smithi165.stdout:1840: writing smithi16534841-12 from 1627732 to 2214946 tid 2 2024-09-10T16:19:20.175 INFO:tasks.rados.rados.0.smithi165.stdout:1840: writing smithi16534841-12 from 2718088 to 2874129 tid 3 2024-09-10T16:19:20.175 INFO:tasks.rados.rados.0.smithi165.stdout:1841: write oid 1 current snap is 205 2024-09-10T16:19:20.175 INFO:tasks.rados.rados.0.smithi165.stdout:1838: finishing write tid 1 to smithi16534841-29 2024-09-10T16:19:20.175 INFO:tasks.rados.rados.0.smithi165.stdout:1841: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-10T16:19:20.181 INFO:tasks.rados.rados.0.smithi165.stdout:1841: writing smithi16534841-1 from 490936 to 1142293 tid 1 2024-09-10T16:19:20.186 INFO:tasks.rados.rados.0.smithi165.stdout:1841: writing smithi16534841-1 from 1686214 to 2133439 tid 2 2024-09-10T16:19:20.191 INFO:tasks.rados.rados.0.smithi165.stdout:1841: writing smithi16534841-1 from 2861864 to 2890936 tid 3 2024-09-10T16:19:20.191 INFO:tasks.rados.rados.0.smithi165.stdout:1842: write oid 9 current snap is 205 2024-09-10T16:19:20.191 INFO:tasks.rados.rados.0.smithi165.stdout:1838: finishing write tid 2 to smithi16534841-29 2024-09-10T16:19:20.191 INFO:tasks.rados.rados.0.smithi165.stdout:1838: finishing write tid 3 to smithi16534841-29 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1838: finishing write tid 4 to smithi16534841-29 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1838: oid 29 updating version 0 to 1154 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1838: oid 29 version 1154 is already newer than 1153 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1838: oid 29 version 1154 is already newer than 1152 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1154 (ObjNum 646 snap 205 seq_num 646) dirty exists 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1838: left oid 29 (ObjNum 646 snap 205 seq_num 646) 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:1839: expect (ObjNum 397 snap 130 seq_num 397) 2024-09-10T16:19:20.192 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:20.241 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:20.321 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2608911 2024-09-10T16:19:20.326 INFO:tasks.rados.rados.0.smithi165.stdout:1840: finishing write tid 1 to smithi16534841-12 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: finishing write tid 2 to smithi16534841-12 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: finishing write tid 3 to smithi16534841-12 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: finishing write tid 4 to smithi16534841-12 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: finishing write tid 5 to smithi16534841-12 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: oid 12 updating version 0 to 1091 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: oid 12 version 1091 is already newer than 1090 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: oid 12 version 1091 is already newer than 1089 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: oid 12 updating version 1091 to 1092 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1092 (ObjNum 647 snap 205 seq_num 647) dirty exists 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1840: left oid 12 (ObjNum 647 snap 205 seq_num 647) 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1841: finishing write tid 1 to smithi16534841-1 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1841: finishing write tid 2 to smithi16534841-1 2024-09-10T16:19:20.327 INFO:tasks.rados.rados.0.smithi165.stdout:1842: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-09-10T16:19:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:1842: writing smithi16534841-9 from 507743 to 1234349 tid 1 2024-09-10T16:19:20.338 INFO:tasks.rados.rados.0.smithi165.stdout:1842: writing smithi16534841-9 from 1744696 to 2335579 tid 2 2024-09-10T16:19:20.342 INFO:tasks.rados.rados.0.smithi165.stdout:1842: writing smithi16534841-9 from 2772934 to 2907743 tid 3 2024-09-10T16:19:20.342 INFO:tasks.rados.rados.0.smithi165.stdout:1838: done (4 left) 2024-09-10T16:19:20.342 INFO:tasks.rados.rados.0.smithi165.stdout:1839: done (3 left) 2024-09-10T16:19:20.342 INFO:tasks.rados.rados.0.smithi165.stdout:1840: done (2 left) 2024-09-10T16:19:20.342 INFO:tasks.rados.rados.0.smithi165.stdout:1843: snap_create 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: finishing write tid 3 to smithi16534841-1 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: finishing write tid 4 to smithi16534841-1 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: finishing write tid 5 to smithi16534841-1 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: oid 1 updating version 0 to 1123 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: oid 1 version 1123 is already newer than 1120 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: oid 1 version 1123 is already newer than 1121 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: oid 1 version 1123 is already newer than 1122 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1123 (ObjNum 648 snap 205 seq_num 648) dirty exists 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1841: left oid 1 (ObjNum 648 snap 205 seq_num 648) 2024-09-10T16:19:20.343 INFO:tasks.rados.rados.0.smithi165.stdout:1842: finishing write tid 1 to smithi16534841-9 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: finishing write tid 2 to smithi16534841-9 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: finishing write tid 3 to smithi16534841-9 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: finishing write tid 4 to smithi16534841-9 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: finishing write tid 5 to smithi16534841-9 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: oid 9 updating version 0 to 1141 2024-09-10T16:19:20.345 INFO:tasks.rados.rados.0.smithi165.stdout:1842: oid 9 updating version 1141 to 1142 2024-09-10T16:19:20.346 INFO:tasks.rados.rados.0.smithi165.stdout:1842: oid 9 updating version 1142 to 1144 2024-09-10T16:19:20.346 INFO:tasks.rados.rados.0.smithi165.stdout:1842: oid 9 version 1144 is already newer than 1143 2024-09-10T16:19:20.346 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1144 (ObjNum 649 snap 205 seq_num 649) dirty exists 2024-09-10T16:19:20.346 INFO:tasks.rados.rados.0.smithi165.stdout:1842: left oid 9 (ObjNum 649 snap 205 seq_num 649) 2024-09-10T16:19:20.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 10 monclient(hunting): tick 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 msgr2=0x7f95d8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8003740 0x7f95d8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.577+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 msgr2=0x7f95d8008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.164:3300/0 conn(0x7f95d80061d0 0x7f95d8008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 reap_dead start 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 0x7f95d8006680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): picked mon.noname-c con 0x7f95d8003b80 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95d8006bc0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:20.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95d800d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): picked mon.noname-b con 0x7f95d8008710 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95f409d080 con 0x7f95d8006bc0 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f95d8008d00 con 0x7f95d8003b80 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 1 -- 172.21.15.164:0/1639618582 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95d8008eb0 con 0x7f95d8008710 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95eb7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95f9bfe640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 msgr2=0x7f95d8006680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95f93fd640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95d8006bc0 legacy=0x7f95d8006ff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 0x7f95d8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95d800d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): get_auth_request con 0x7f95d8008710 auth_method 0 2024-09-10T16:19:20.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_done global_id 114419 payload 274 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 msgr2=0x7f95d8006680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 0x7f95d8006680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 >> v1:172.21.15.165:6789/0 conn(0x7f95d8006bc0 legacy=0x7f95d8006ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 monclient: found mon.noname-b 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:20.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.578+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f4074bd0 con 0x7f95d8008710 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.581+0000 7f95fa3ff640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 msgr2=0x7f95d800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.581+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95d800d310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95f41350b0 tx=0x7f95ec02a700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9a32b6da867aa00 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 1 -- 172.21.15.164:0/1639618582 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec007350 con 0x7f95d8008710 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: dump: 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.588 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: _finish_auth 0 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 1 -- 172.21.15.164:0/1639618582 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ec0074f0 con 0x7f95d8008710 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 5 monclient: authenticate success, global_id 114419 2024-09-10T16:19:20.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 1 -- 172.21.15.164:0/1639618582 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0078a0 con 0x7f95d8008710 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 4 set_mon_vals no callback set 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95f8bfc640 10 monclient: dump: 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 10 monclient: shutdown 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 1 -- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 msgr2=0x7f95d800d310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.582+0000 7f95fbe62640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95d800d310 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f95f41350b0 tx=0x7f95ec02a700 comp rx=0 tx=0).stop 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 -- 172.21.15.164:0/1639618582 shutdown_connections 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 --2- 172.21.15.164:0/1639618582 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95d800d310 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 --2- 172.21.15.164:0/1639618582 >> v2:172.21.15.165:3300/0 conn(0x7f95d8003b80 0x7f95d8006680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 -- 172.21.15.164:0/1639618582 >> 172.21.15.164:0/1639618582 conn(0x7f95f40a2f30 msgr2=0x7f95f40a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 -- 172.21.15.164:0/1639618582 shutdown_connections 2024-09-10T16:19:20.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.583+0000 7f95fbe62640 1 -- 172.21.15.164:0/1639618582 wait complete. 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient: build_initial_monmap 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient: monmap: 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 1 Processor -- start 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 1 -- start start 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient: init 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 0x7f95f41bdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-a con 0x7f95d8003b80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 0x7f95f41be480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-b con 0x7f95d8006bc0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.584+0000 7f95fbe62640 10 monclient(hunting): picked mon.noname-c con 0x7f95d8008710 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f95f409cd00 con 0x7f95d8003b80 2024-09-10T16:19:20.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f95f409cb80 con 0x7f95d8006bc0 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95f409ce80 con 0x7f95d8008710 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fbe62640 10 monclient(hunting): authenticate will time out at 1539.072031s 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f93fd640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 msgr2=0x7f95f41be480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f93fd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 0x7f95f41be480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 0x7f95f41bdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39448/0 (socket says 172.21.15.164:39448) 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 -- 172.21.15.164:0/2900389271 learned_addr learned my addr 172.21.15.164:0/2900389271 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): get_auth_request con 0x7f95d8008710 auth_method 0 2024-09-10T16:19:20.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request con 0x7f95d8003b80 auth_method 0 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient(hunting): handle_auth_done global_id 114422 payload 824 2024-09-10T16:19:20.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 -- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 msgr2=0x7f95f41be480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 0x7f95f41be480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 -- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 msgr2=0x7f95f41bdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 0x7f95f41bdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 monclient: found mon.noname-c 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95fa3ff640 1 -- 172.21.15.164:0/2900389271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95f41c18d0 con 0x7f95d8008710 2024-09-10T16:19:20.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.585+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 0x7f95f41bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:20.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.587+0000 7f95fa3ff640 1 -- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 msgr2=0x7f95f41be9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.587+0000 7f95fa3ff640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f95f41350e0 tx=0x7f95ec0017e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec032630 con 0x7f95d8008710 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: dump: 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ec043070 con 0x7f95d8008710 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 5 monclient: authenticate success, global_id 114422 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 10 monclient: _renew_subs 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95f409d620 con 0x7f95d8008710 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:20.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ec0329d0 con 0x7f95d8008710 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ea7fc640 10 monclient: dump: 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 10 monclient: _renew_subs 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95ebfff640 4 set_mon_vals no callback set 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.588+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95f41c00d0 con 0x7f95d8008710 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f95ec032b70 con 0x7f95d8008710 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95fbe62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95fbe62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95fbe62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95f4137d00 con 0x7f95d8008710 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95ea7fc640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 0x7f95d8042390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.589+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(438..438 src has 184..438) ==== 4739+0+0 (secure 0 0 0) 0x7f95ec073930 con 0x7f95d8008710 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.590+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 0x7f95d8042390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.590+0000 7f95f9bfe640 10 monclient: get_auth_request con 0x7f95d803fed0 auth_method 0 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.592+0000 7f95f9bfe640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 0x7f95d8042390 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005020 tx=0x7f95f000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.594+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f95ec072350 con 0x7f95d8008710 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.594+0000 7f95ea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.594+0000 7f95ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:20.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95fbe62640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95fbe62640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95fbe62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95f41c14a0 con 0x7f95d8008710 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95ea7fc640 1 -- 172.21.15.164:0/2900389271 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f95ec048dc0 con 0x7f95d8008710 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95ea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.842+0000 7f95ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:20.844 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":126,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 msgr2=0x7f95d8042390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 0x7f95d8042390 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f95f0005020 tx=0x7f95f000d040 comp rx=0 tx=0).stop 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 10 monclient: shutdown 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 msgr2=0x7f95f41be9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f95f41350e0 tx=0x7f95ec0017e0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 shutdown_connections 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95d803fed0 0x7f95d8042390 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95d8008710 0x7f95f41be9c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95d8006bc0 0x7f95f41be480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 --2- 172.21.15.164:0/2900389271 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95d8003b80 0x7f95f41bdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.844+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 >> 172.21.15.164:0/2900389271 conn(0x7f95f40a2f30 msgr2=0x7f95f4133070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.845+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 shutdown_connections 2024-09-10T16:19:20.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.845+0000 7f95fbe62640 1 -- 172.21.15.164:0/2900389271 wait complete. 2024-09-10T16:19:20.861 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:19:20.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:19:20.861 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 10 monclient: get_monmap_and_config 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 10 monclient: build_initial_monmap 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 10 monclient: monmap: 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:20.958933+0000 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:20.958933+0000 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.960 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 1 Processor -- start 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 1 -- start start 2024-09-10T16:19:20.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.958+0000 7fa85da57640 10 monclient: init 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa858134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-b con 0x7fa85806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa8580a46a0 0x7fa8580a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-a con 0x7fa8580a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa858137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa85809ce50 con 0x7fa858137ca0 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa85806db60 con 0x7fa8580a46a0 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa85806cf50 con 0x7fa85806cb70 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa85da57640 10 monclient(hunting): authenticate will time out at 1539.446263s 2024-09-10T16:19:20.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8577fe640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fa858137ca0 legacy=0x7fa858138130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa8580a46a0 0x7fa8580a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa858134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa8580a46a0 0x7fa8580a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60488/0 (socket says 172.21.15.164:60488) 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 -- 172.21.15.164:0/3680435910 learned_addr learned my addr 172.21.15.164:0/3680435910 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): get_auth_request con 0x7fa8580a46a0 auth_method 0 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): get_auth_request con 0x7fa85806cb70 auth_method 0 2024-09-10T16:19:20.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient(hunting): handle_auth_done global_id 144174 payload 274 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 -- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 msgr2=0x7fa858134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:20.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 --2- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa858134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 -- 172.21.15.164:0/3680435910 >> v1:172.21.15.165:6789/0 conn(0x7fa858137ca0 legacy=0x7fa858138130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient: _finish_hunting 0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 monclient: found mon.noname-a 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.959+0000 7fa8567fc640 1 -- 172.21.15.164:0/3680435910 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa85809cb70 con 0x7fa8580a46a0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa8567fc640 1 -- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 msgr2=0x7fa8580a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa8567fc640 1 --2- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa8580a4a80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa840009870 tx=0x7fa84002a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3ffc2c6c89c964a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 1 -- 172.21.15.164:0/3680435910 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840037070 con 0x7fa8580a46a0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: dump: 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: _finish_auth 0 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa85da57640 5 monclient: authenticate success, global_id 144174 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 1 -- 172.21.15.164:0/3680435910 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa840032440 con 0x7fa8580a46a0 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa85da57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa85da57640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 1 -- 172.21.15.164:0/3680435910 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840032810 con 0x7fa8580a46a0 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa85da57640 4 set_mon_vals no callback set 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.962+0000 7fa855ffb640 10 monclient: dump: 2024-09-10T16:19:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 10 monclient: shutdown 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 -- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 msgr2=0x7fa8580a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 --2- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa8580a4a80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fa840009870 tx=0x7fa84002a0a0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 -- 172.21.15.164:0/3680435910 shutdown_connections 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 --2- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa8580a4a80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 --2- 172.21.15.164:0/3680435910 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa858134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 -- 172.21.15.164:0/3680435910 >> 172.21.15.164:0/3680435910 conn(0x7fa8580aa900 msgr2=0x7fa8580aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:20.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 -- 172.21.15.164:0/3680435910 shutdown_connections 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 1 -- 172.21.15.164:0/3680435910 wait complete. 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 10 monclient: build_initial_monmap 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.963+0000 7fa85da57640 10 monclient: monmap: 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:20.982 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 Processor -- start 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 -- start start 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient: init 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa8580a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-c con 0x7fa85806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa85809eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-a con 0x7fa8580a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa858137ca0 0x7fa85809f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): picked mon.noname-b con 0x7fa858137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8581389c0 con 0x7fa8580a46a0 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa858138840 con 0x7fa858137ca0 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa858140040 con 0x7fa85806cb70 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa85da57640 10 monclient(hunting): authenticate will time out at 1539.451692s 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa8580a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa85809eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa8580a40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39454/0 (socket says 172.21.15.164:39454) 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa85809eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60496/0 (socket says 172.21.15.164:60496) 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 1 -- 172.21.15.164:0/2969547819 learned_addr learned my addr 172.21.15.164:0/2969547819 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): get_auth_request con 0x7fa85806cb70 auth_method 0 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): get_auth_request con 0x7fa8580a46a0 auth_method 0 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8577fe640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa858137ca0 msgr2=0x7fa85809f090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8577fe640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa858137ca0 0x7fa85809f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:20.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa856ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.964+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 10 monclient(hunting): handle_auth_done global_id 114428 payload 824 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa858137ca0 msgr2=0x7fa85809f090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa858137ca0 0x7fa85809f090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 msgr2=0x7fa85809eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa85809eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 monclient: found mon.noname-c 2024-09-10T16:19:20.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa856ffd640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8580a0550 con 0x7fa85806cb70 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.965+0000 7fa8567fc640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8580a46a0 0x7fa85809eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa856ffd640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 msgr2=0x7fa8580a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa856ffd640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa8580a40c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa85000efc0 tx=0x7fa85000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850018020 con 0x7fa85806cb70 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: dump: 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.987 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: _finish_auth 0 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa850014680 con 0x7fa85806cb70 2024-09-10T16:19:20.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 5 monclient: authenticate success, global_id 114428 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa850014a50 con 0x7fa85806cb70 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa84ffff640 10 monclient: dump: 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.989 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 10 monclient: _renew_subs 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa8557fa640 4 set_mon_vals no callback set 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8580ab180 con 0x7fa85806cb70 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 10 monclient: _renew_subs 2024-09-10T16:19:20.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.967+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8580a2010 con 0x7fa85806cb70 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa85da57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa85da57640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 0x7fa85809f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8580a2480 con 0x7fa85809f5d0 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 0x7fa85809f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient: get_auth_request con 0x7fa85809f5d0 auth_method 0 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa850005d10 con 0x7fa85806cb70 2024-09-10T16:19:20.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 10 monclient(hunting): handle_auth_done global_id 144180 payload 824 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa8567fc640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa8580a5b30 con 0x7fa85809f5d0 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa84ffff640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83803e1c0 0x7fa838040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.968+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(438..438 src has 184..438) ==== 4739+0+0 (secure 0 0 0) 0x7fa8500538a0 con 0x7fa85806cb70 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.969+0000 7fa8577fe640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83803e1c0 0x7fa838040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.969+0000 7fa8577fe640 10 monclient: get_auth_request con 0x7fa83803e1c0 auth_method 0 2024-09-10T16:19:20.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.970+0000 7fa8567fc640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 msgr2=0x7fa85809f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.970+0000 7fa8567fc640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 0x7fa85809f9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa840034000 tx=0x7fa8400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840037040 con 0x7fa85809f5d0 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 10 monclient: dump: 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.993 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa84003c860 con 0x7fa85809f5d0 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 msgr2=0x7fa85809f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 0x7fa85809f9b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa840034000 tx=0x7fa8400046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 msgr2=0x7fa85809f9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa84ffff640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85809f5d0 0x7fa85809f9b0 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa840034000 tx=0x7fa8400046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:20.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.971+0000 7fa8577fe640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83803e1c0 0x7fa838040680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa8580a0eb0 tx=0x7fa848006c60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa85da57640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:20.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa85da57640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 0x7fa85812f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa85da57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa85809fef0 con 0x7fa85812f460 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 0x7fa85812f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient: get_auth_request con 0x7fa85812f460 auth_method 0 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:20.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 10 monclient(hunting): handle_auth_done global_id 144183 payload 824 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.977+0000 7fa8567fc640 1 -- 172.21.15.164:0/2969547819 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa858134f70 con 0x7fa85812f460 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa8567fc640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 msgr2=0x7fa85812f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa8567fc640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 0x7fa85812f870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa840000c00 tx=0x7fa840009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa840050020 con 0x7fa85812f460 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 10 monclient: dump: 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fa840004450 con 0x7fa85812f460 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 msgr2=0x7fa85812f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:20.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 0x7fa85812f870 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa840000c00 tx=0x7fa840009240 comp rx=0 tx=0).stop 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 msgr2=0x7fa85812f870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa84ffff640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa85812f460 0x7fa85812f870 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa840000c00 tx=0x7fa840009240 comp rx=0 tx=0).stop 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:21.000 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 126, 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 32423, 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.001 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.002 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:21.003 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.004 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:21.005 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:21.006 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.007 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.008 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.009 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.010 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.011 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.012 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:21.014 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.015 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:21.016 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.017 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.017 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.017 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:21.017 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:21.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83803e1c0 msgr2=0x7fa838040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa83803e1c0 0x7fa838040680 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa8580a0eb0 tx=0x7fa848006c60 comp rx=0 tx=0).stop 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 10 monclient: shutdown 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 msgr2=0x7fa8580a40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 --2- 172.21.15.164:0/2969547819 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa85806cb70 0x7fa8580a40c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa85000efc0 tx=0x7fa85000c490 comp rx=0 tx=0).stop 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa8577fe640 1 -- 172.21.15.164:0/2969547819 reap_dead start 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa8577fe640 1 -- 172.21.15.164:0/2969547819 reap_dead start 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 shutdown_connections 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 >> 172.21.15.164:0/2969547819 conn(0x7fa8580aa900 msgr2=0x7fa858132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 shutdown_connections 2024-09-10T16:19:21.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:20.980+0000 7fa85da57640 1 -- 172.21.15.164:0/2969547819 wait complete. 2024-09-10T16:19:21.020 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:19:21.020 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:19:21.020 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.117+0000 7f42e9c7f640 10 monclient: get_monmap_and_config 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.117+0000 7f42e9c7f640 10 monclient: build_initial_monmap 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.117+0000 7f42e9c7f640 10 monclient: monmap: 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.119 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:21.118426+0000 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:21.118426+0000 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:21.120 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:21.121 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:21.121 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 Processor -- start 2024-09-10T16:19:21.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 -- start start 2024-09-10T16:19:21.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient: init 2024-09-10T16:19:21.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:21.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f42e409c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 0x7f42e409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-b con 0x7f42e409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f42e413a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e409a430 con 0x7f42e413a5c0 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e40a24e0 con 0x7f42e409c180 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42e40a2bf0 con 0x7f42e409d160 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e9c7f640 10 monclient(hunting): authenticate will time out at 1539.605805s 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e37fe640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f42e409c180 legacy=0x7f42e409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 0x7f42e409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e3fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f42e413a5c0 0x7f42e413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:60024/0 (socket says 172.21.15.164:60024) 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e3fff640 1 -- 172.21.15.164:0/2380055922 learned_addr learned my addr 172.21.15.164:0/2380055922 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e2ffd640 10 monclient(hunting): get_auth_request con 0x7f42e409d160 auth_method 0 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.118+0000 7f42e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_done global_id 114431 payload 274 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 1 -- 172.21.15.164:0/2380055922 >> v1:172.21.15.165:6789/0 conn(0x7f42e409c180 legacy=0x7f42e409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 1 -- 172.21.15.164:0/2380055922 >> v1:172.21.15.164:6789/0 conn(0x7f42e413a5c0 legacy=0x7f42e413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 1 monclient: found mon.noname-b 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.119+0000 7f42e2ffd640 1 -- 172.21.15.164:0/2380055922 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e409c560 con 0x7f42e409d160 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.121+0000 7f42e2ffd640 1 -- 172.21.15.164:0/2380055922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 msgr2=0x7f42e409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/2380055922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 0x7f42e409f150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f42d80097a0 tx=0x7f42d8029f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86006fea7567ae48 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 1 -- 172.21.15.164:0/2380055922 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8037070 con 0x7f42e409d160 2024-09-10T16:19:21.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: dump: 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:21.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 1 -- 172.21.15.164:0/2380055922 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d8032440 con 0x7f42e409d160 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 1 -- 172.21.15.164:0/2380055922 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d8032810 con 0x7f42e409d160 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 5 monclient: authenticate success, global_id 114431 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 4 set_mon_vals no callback set 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e27fc640 10 monclient: dump: 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.139 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 10 monclient: shutdown 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 -- 172.21.15.164:0/2380055922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 msgr2=0x7f42e409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/2380055922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 0x7f42e409f150 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f42d80097a0 tx=0x7f42d8029f90 comp rx=0 tx=0).stop 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 -- 172.21.15.164:0/2380055922 shutdown_connections 2024-09-10T16:19:21.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/2380055922 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409d160 0x7f42e409f150 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 -- 172.21.15.164:0/2380055922 >> 172.21.15.164:0/2380055922 conn(0x7f42e40a3110 msgr2=0x7f42e40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 -- 172.21.15.164:0/2380055922 shutdown_connections 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.122+0000 7f42e9c7f640 1 -- 172.21.15.164:0/2380055922 wait complete. 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.123+0000 7f42e9c7f640 10 monclient: build_initial_monmap 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.123+0000 7f42e9c7f640 10 monclient: monmap: 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:21.141 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.123+0000 7f42e9c7f640 1 Processor -- start 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.123+0000 7f42e9c7f640 1 -- start start 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.123+0000 7f42e9c7f640 10 monclient: init 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 0x7f42e40939f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-c con 0x7f42e409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e409d160 0x7f42e4091d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-b con 0x7f42e409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42e413a5c0 0x7f42e40922d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): picked mon.noname-a con 0x7f42e413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f42e4074f90 con 0x7f42e413a5c0 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f42e4074e10 con 0x7f42e409d160 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42e409f690 con 0x7f42e409c180 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e9c7f640 10 monclient(hunting): authenticate will time out at 1539.611406s 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 0x7f42e40939f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 0x7f42e40939f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39468/0 (socket says 172.21.15.164:39468) 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 -- 172.21.15.164:0/3260290149 learned_addr learned my addr 172.21.15.164:0/3260290149 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): get_auth_request con 0x7f42e409c180 auth_method 0 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42e413a5c0 0x7f42e40922d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): get_auth_request con 0x7f42e413a5c0 auth_method 0 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:21.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e409d160 msgr2=0x7f42e4091d90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e409d160 0x7f42e4091d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient(hunting): handle_auth_done global_id 114434 payload 824 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e409d160 msgr2=0x7f42e4091d90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:21.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f42e409d160 0x7f42e4091d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42e413a5c0 msgr2=0x7f42e40922d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f42e413a5c0 0x7f42e40922d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 monclient: found mon.noname-c 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.124+0000 7f42e37fe640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e4094900 con 0x7f42e409c180 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e37fe640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 msgr2=0x7f42e40939f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e37fe640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 0x7f42e40939f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f42d400b580 tx=0x7f42d400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4013020 con 0x7f42e409c180 2024-09-10T16:19:21.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: dump: 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.147 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: _finish_auth 0 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d4005070 con 0x7f42e409c180 2024-09-10T16:19:21.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 5 monclient: authenticate success, global_id 114434 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 monclient: _renew_subs 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e40a4cc0 con 0x7f42e409c180 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4005480 con 0x7f42e409c180 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42cbfff640 10 monclient: dump: 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.149 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 monclient: _renew_subs 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:21.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e4093100 con 0x7f42e409c180 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e1ffb640 4 set_mon_vals no callback set 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.127+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42e402e9a0 con 0x7f42e402e5c0 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient: get_auth_request con 0x7f42e402e5c0 auth_method 0 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:21.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f42d4016020 con 0x7f42e409c180 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_done global_id 114437 payload 824 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42e409ea40 con 0x7f42e402e5c0 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 msgr2=0x7f42e4092810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 msgr2=0x7f42e4092810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:21.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f42d8034000 tx=0x7f42d80046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42cbfff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42d003e210 0x7f42d00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(438..438 src has 184..438) ==== 4739+0+0 (secure 0 0 0) 0x7f42d4052710 con 0x7f42e409c180 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e3fff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42d003e210 0x7f42d00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.128+0000 7f42e3fff640 10 monclient: get_auth_request con 0x7f42d003e210 auth_method 0 2024-09-10T16:19:21.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.131+0000 7f42e3fff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42d003e210 0x7f42d00406d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0055a0 tx=0x7f42dc005530 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1841: done (2 left) 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1842: done (1 left) 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1843: done (0 left) 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1844: rollback oid 50 current snap is 206 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 202 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1845: read oid 15 snap -1 2024-09-10T16:19:21.155 INFO:tasks.rados.rados.0.smithi165.stdout:1845: expect (ObjNum 643 snap 203 seq_num 643) 2024-09-10T16:19:21.156 INFO:tasks.rados.rados.0.smithi165.stdout:1846: copy_from oid 37 from oid 32 current snap is 206 2024-09-10T16:19:21.156 INFO:tasks.rados.rados.0.smithi165.stdout:1847: snap_create 2024-09-10T16:19:21.156 INFO:tasks.rados.rados.0.smithi165.stdout:1846: finishing copy_from to smithi16534841-37 2024-09-10T16:19:21.156 INFO:tasks.rados.rados.0.smithi165.stdout:1846: got expected ENOENT (src dne) 2024-09-10T16:19:21.156 INFO:tasks.rados.rados.0.smithi165.stdout:1846: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:19:21.157 INFO:tasks.rados.rados.0.smithi165.stdout:1844: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:19:21.157 INFO:tasks.rados.rados.0.smithi165.stdout:1844: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:19:21.157 INFO:tasks.rados.rados.0.smithi165.stdout:1844: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:19:21.157 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1223 (ObjNum 576 snap 184 seq_num 576) dirty exists 2024-09-10T16:19:21.164 INFO:tasks.rados.rados.0.smithi165.stdout:1845: expect (ObjNum 643 snap 203 seq_num 643) 2024-09-10T16:19:21.212 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:21.265 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: get_auth_request con 0x7f42e402e5c0 auth_method 0 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: _init_auth method 2 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: _init_auth already have auth, reseting 2024-09-10T16:19:21.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T16:19:21.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T16:19:21.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.329+0000 7f42e2ffd640 10 monclient: handle_auth_done global_id 114437 payload 293 2024-09-10T16:19:21.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 msgr2=0x7f42e4092810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:21.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f42d8034000 tx=0x7f42d802e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:21.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d804c070 con 0x7f42e402e5c0 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.331+0000 7f42cbfff640 10 monclient: dump: 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.334 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f42d803d650 con 0x7f42e402e5c0 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 msgr2=0x7f42e4092810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f42d8034000 tx=0x7f42d802e000 comp rx=0 tx=0).stop 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 msgr2=0x7f42e4092810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:21.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.332+0000 7f42cbfff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e402e5c0 0x7f42e4092810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e9c7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 0x7f42e4064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e9c7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f42e40726d0 con 0x7f42e4063d50 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 0x7f42e4064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient: get_auth_request con 0x7f42e4063d50 auth_method 0 2024-09-10T16:19:21.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 10 monclient(hunting): handle_auth_done global_id 114440 payload 824 2024-09-10T16:19:21.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.338+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f42e4077260 con 0x7f42e4063d50 2024-09-10T16:19:21.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e2ffd640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 msgr2=0x7f42e4064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:21.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e2ffd640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 0x7f42e4064160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f42d8000c00 tx=0x7f42d8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:21.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d804c040 con 0x7f42e4063d50 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 10 monclient: dump: 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:21.343 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f42d804e820 con 0x7f42e4063d50 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 msgr2=0x7f42e4064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 0x7f42e4064160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f42d8000c00 tx=0x7f42d8008030 comp rx=0 tx=0).stop 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 msgr2=0x7f42e4064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42cbfff640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e4063d50 0x7f42e4064160 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f42d8000c00 tx=0x7f42d8008030 comp rx=0 tx=0).stop 2024-09-10T16:19:21.344 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 126, 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 181909, 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:21.345 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:21.346 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.347 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:21.348 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:21.349 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.350 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.351 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.352 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.353 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:21.354 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:21.355 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.356 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.357 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 7 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:21.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:21.359 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:21.359 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42d003e210 msgr2=0x7f42d00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f42d003e210 0x7f42d00406d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f42dc0055a0 tx=0x7f42dc005530 comp rx=0 tx=0).stop 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 10 monclient: shutdown 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 msgr2=0x7f42e40939f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 --2- 172.21.15.164:0/3260290149 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f42e409c180 0x7f42e40939f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f42d400b580 tx=0x7f42d400ba40 comp rx=0 tx=0).stop 2024-09-10T16:19:21.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e3fff640 1 -- 172.21.15.164:0/3260290149 reap_dead start 2024-09-10T16:19:21.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e3fff640 1 -- 172.21.15.164:0/3260290149 reap_dead start 2024-09-10T16:19:21.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 shutdown_connections 2024-09-10T16:19:21.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 >> 172.21.15.164:0/3260290149 conn(0x7f42e40a3110 msgr2=0x7f42e4139940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:21.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 shutdown_connections 2024-09-10T16:19:21.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:21.341+0000 7f42e9c7f640 1 -- 172.21.15.164:0/3260290149 wait complete. 2024-09-10T16:19:21.361 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:19:21.361 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:19:22.155 INFO:tasks.rados.rados.0.smithi165.stdout:1844: done (3 left) 2024-09-10T16:19:22.155 INFO:tasks.rados.rados.0.smithi165.stdout:1845: done (2 left) 2024-09-10T16:19:22.155 INFO:tasks.rados.rados.0.smithi165.stdout:1846: done (1 left) 2024-09-10T16:19:22.156 INFO:tasks.rados.rados.0.smithi165.stdout:1847: done (0 left) 2024-09-10T16:19:22.156 INFO:tasks.rados.rados.0.smithi165.stdout:1848: copy_from oid 20 from oid 34 current snap is 207 2024-09-10T16:19:22.156 INFO:tasks.rados.rados.0.smithi165.stdout:1849: read oid 23 snap 200 2024-09-10T16:19:22.156 INFO:tasks.rados.rados.0.smithi165.stdout:1849: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:19:22.156 INFO:tasks.rados.rados.0.smithi165.stdout:1850: delete oid 16 current snap is 207 2024-09-10T16:19:22.161 INFO:tasks.rados.rados.0.smithi165.stdout:1850: done (2 left) 2024-09-10T16:19:22.161 INFO:tasks.rados.rados.0.smithi165.stdout:1851: read oid 38 snap -1 2024-09-10T16:19:22.161 INFO:tasks.rados.rados.0.smithi165.stdout:1851: expect deleted 2024-09-10T16:19:22.161 INFO:tasks.rados.rados.0.smithi165.stdout:1852: write (excl) oid 14 current snap is 207 2024-09-10T16:19:22.162 INFO:tasks.rados.rados.0.smithi165.stdout:1852: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-10T16:19:22.167 INFO:tasks.rados.rados.0.smithi165.stdout:1852: writing smithi16534841-14 from 541357 to 1018461 tid 1 2024-09-10T16:19:22.175 INFO:tasks.rados.rados.0.smithi165.stdout:1852: writing smithi16534841-14 from 1745307 to 2339859 tid 2 2024-09-10T16:19:22.175 INFO:tasks.rados.rados.0.smithi165.stdout:1853: delete oid 12 current snap is 207 2024-09-10T16:19:22.175 INFO:tasks.rados.rados.0.smithi165.stdout:1849: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:19:22.183 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1005981 2024-09-10T16:19:22.207 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:22.262 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2274372 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1852: finishing write tid 1 to smithi16534841-14 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1848: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1848: finishing copy_from to smithi16534841-20 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1131 (ObjNum 644 snap 203 seq_num 644) dirty exists 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1852: finishing write tid 2 to smithi16534841-14 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1852: finishing write tid 3 to smithi16534841-14 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1852: finishing write tid 4 to smithi16534841-14 2024-09-10T16:19:22.270 INFO:tasks.rados.rados.0.smithi165.stdout:1852: oid 14 updating version 0 to 1146 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1852: oid 14 version 1146 is already newer than 1145 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1852: oid 14 updating version 1146 to 1147 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1147 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1852: left oid 14 (ObjNum 651 snap 207 seq_num 651) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1848: done (4 left) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1849: done (3 left) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1851: done (2 left) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1852: done (1 left) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1853: done (0 left) 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1854: write (excl) oid 21 current snap is 207 2024-09-10T16:19:22.271 INFO:tasks.rados.rados.0.smithi165.stdout:1854: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-10T16:19:22.276 INFO:tasks.rados.rados.0.smithi165.stdout:1854: writing smithi16534841-21 from 574971 to 1202573 tid 1 2024-09-10T16:19:22.283 INFO:tasks.rados.rados.0.smithi165.stdout:1854: writing smithi16534841-21 from 1862271 to 2460492 tid 2 2024-09-10T16:19:22.283 INFO:tasks.rados.rados.0.smithi165.stdout:1855: read oid 15 snap 200 2024-09-10T16:19:22.283 INFO:tasks.rados.rados.0.smithi165.stdout:1855: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:19:22.283 INFO:tasks.rados.rados.0.smithi165.stdout:1856: delete oid 42 current snap is 207 2024-09-10T16:19:22.284 INFO:tasks.rados.rados.0.smithi165.stdout:1854: finishing write tid 1 to smithi16534841-21 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1856: done (2 left) 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1857: delete oid 48 current snap is 207 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: finishing write tid 2 to smithi16534841-21 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: finishing write tid 3 to smithi16534841-21 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: finishing write tid 4 to smithi16534841-21 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: oid 21 updating version 0 to 1170 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: oid 21 version 1170 is already newer than 1168 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: oid 21 version 1170 is already newer than 1169 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1170 (ObjNum 653 snap 207 seq_num 653) dirty exists 2024-09-10T16:19:22.290 INFO:tasks.rados.rados.0.smithi165.stdout:1854: left oid 21 (ObjNum 653 snap 207 seq_num 653) 2024-09-10T16:19:22.293 INFO:tasks.rados.rados.0.smithi165.stdout:1854: done (2 left) 2024-09-10T16:19:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:1857: done (1 left) 2024-09-10T16:19:22.294 INFO:tasks.rados.rados.0.smithi165.stdout:1858: delete oid 33 current snap is 207 2024-09-10T16:19:22.297 INFO:tasks.rados.rados.0.smithi165.stdout:1855: expect (ObjNum 465 snap 146 seq_num 465) 2024-09-10T16:19:22.333 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2019596 2024-09-10T16:19:22.368 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:22.450 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:22.532 INFO:tasks.rados.rados.0.smithi165.stdout:1855: done (1 left) 2024-09-10T16:19:22.532 INFO:tasks.rados.rados.0.smithi165.stdout:1858: done (0 left) 2024-09-10T16:19:22.533 INFO:tasks.rados.rados.0.smithi165.stdout:1859: write (excl) oid 42 current snap is 207 2024-09-10T16:19:22.533 INFO:tasks.rados.rados.0.smithi165.stdout:1859: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-10T16:19:22.539 INFO:tasks.rados.rados.0.smithi165.stdout:1859: writing smithi16534841-42 from 642199 to 1287150 tid 1 2024-09-10T16:19:22.544 INFO:tasks.rados.rados.0.smithi165.stdout:1859: writing smithi16534841-42 from 1696199 to 2301758 tid 2 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1859: writing smithi16534841-42 from 2992318 to 3042199 tid 3 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1860: read oid 9 snap -1 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1860: expect (ObjNum 649 snap 205 seq_num 649) 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1861: delete oid 20 current snap is 207 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1859: finishing write tid 1 to smithi16534841-42 2024-09-10T16:19:22.549 INFO:tasks.rados.rados.0.smithi165.stdout:1859: finishing write tid 2 to smithi16534841-42 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1861: done (2 left) 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1862: delete oid 40 current snap is 207 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: finishing write tid 3 to smithi16534841-42 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: finishing write tid 4 to smithi16534841-42 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: finishing write tid 5 to smithi16534841-42 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: oid 42 updating version 0 to 1227 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: oid 42 version 1227 is already newer than 1225 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: oid 42 updating version 1227 to 1228 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:1859: oid 42 version 1228 is already newer than 1226 2024-09-10T16:19:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1228 (ObjNum 657 snap 207 seq_num 657) dirty exists 2024-09-10T16:19:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:1859: left oid 42 (ObjNum 657 snap 207 seq_num 657) 2024-09-10T16:19:22.553 INFO:tasks.rados.rados.0.smithi165.stdout:1860: expect (ObjNum 649 snap 205 seq_num 649) 2024-09-10T16:19:22.553 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:22.667 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2712391 2024-09-10T16:19:22.688 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1008404 2024-09-10T16:19:22.745 INFO:tasks.rados.rados.0.smithi165.stdout:1859: done (2 left) 2024-09-10T16:19:22.745 INFO:tasks.rados.rados.0.smithi165.stdout:1860: done (1 left) 2024-09-10T16:19:22.745 INFO:tasks.rados.rados.0.smithi165.stdout:1862: done (0 left) 2024-09-10T16:19:22.745 INFO:tasks.rados.rados.0.smithi165.stdout:1863: snap_remove snap 200 2024-09-10T16:19:23.165 INFO:tasks.rados.rados.0.smithi165.stdout:1863: done (0 left) 2024-09-10T16:19:23.165 INFO:tasks.rados.rados.0.smithi165.stdout:1864: write (excl) oid 6 current snap is 207 2024-09-10T16:19:23.165 INFO:tasks.rados.rados.0.smithi165.stdout:1864: seq_num 660 ranges {692620=470698,1755292=752886} 2024-09-10T16:19:23.172 INFO:tasks.rados.rados.0.smithi165.stdout:1864: writing smithi16534841-6 from 692620 to 1163318 tid 1 2024-09-10T16:19:23.179 INFO:tasks.rados.rados.0.smithi165.stdout:1864: writing smithi16534841-6 from 1755292 to 2508178 tid 2 2024-09-10T16:19:23.180 INFO:tasks.rados.rados.0.smithi165.stdout:1865: write oid 22 current snap is 207 2024-09-10T16:19:23.180 INFO:tasks.rados.rados.0.smithi165.stdout:1864: finishing write tid 1 to smithi16534841-6 2024-09-10T16:19:23.180 INFO:tasks.rados.rados.0.smithi165.stdout:1865: seq_num 661 ranges {709427=545947,1813774=612897,2985657=123770} 2024-09-10T16:19:23.186 INFO:tasks.rados.rados.0.smithi165.stdout:1865: writing smithi16534841-22 from 709427 to 1255374 tid 1 2024-09-10T16:19:23.193 INFO:tasks.rados.rados.0.smithi165.stdout:1865: writing smithi16534841-22 from 1813774 to 2426671 tid 2 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1865: writing smithi16534841-22 from 2985657 to 3109427 tid 3 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1866: delete oid 44 current snap is 207 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: finishing write tid 2 to smithi16534841-6 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: finishing write tid 3 to smithi16534841-6 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: finishing write tid 4 to smithi16534841-6 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: oid 6 updating version 0 to 640 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: oid 6 version 640 is already newer than 638 2024-09-10T16:19:23.197 INFO:tasks.rados.rados.0.smithi165.stdout:1864: oid 6 version 640 is already newer than 639 2024-09-10T16:19:23.198 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 640 (ObjNum 660 snap 207 seq_num 660) dirty exists 2024-09-10T16:19:23.198 INFO:tasks.rados.rados.0.smithi165.stdout:1864: left oid 6 (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:19:23.198 INFO:tasks.rados.rados.0.smithi165.stdout:1865: finishing write tid 1 to smithi16534841-22 2024-09-10T16:19:23.199 INFO:tasks.rados.rados.0.smithi165.stdout:1865: finishing write tid 2 to smithi16534841-22 2024-09-10T16:19:23.199 INFO:tasks.rados.rados.0.smithi165.stdout:1865: finishing write tid 3 to smithi16534841-22 2024-09-10T16:19:23.200 INFO:tasks.rados.rados.0.smithi165.stdout:1864: done (2 left) 2024-09-10T16:19:23.200 INFO:tasks.rados.rados.0.smithi165.stdout:1866: done (1 left) 2024-09-10T16:19:23.200 INFO:tasks.rados.rados.0.smithi165.stdout:1867: write oid 28 current snap is 207 2024-09-10T16:19:23.200 INFO:tasks.rados.rados.0.smithi165.stdout:1867: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2024-09-10T16:19:23.207 INFO:tasks.rados.rados.0.smithi165.stdout:1867: writing smithi16534841-28 from 743041 to 1155839 tid 1 2024-09-10T16:19:23.214 INFO:tasks.rados.rados.0.smithi165.stdout:1867: writing smithi16534841-28 from 1930738 to 2547304 tid 2 2024-09-10T16:19:23.218 INFO:tasks.rados.rados.0.smithi165.stdout:1867: writing smithi16534841-28 from 3040503 to 3143041 tid 3 2024-09-10T16:19:23.218 INFO:tasks.rados.rados.0.smithi165.stdout:1868: delete oid 36 current snap is 207 2024-09-10T16:19:23.218 INFO:tasks.rados.rados.0.smithi165.stdout:1865: finishing write tid 4 to smithi16534841-22 2024-09-10T16:19:23.218 INFO:tasks.rados.rados.0.smithi165.stdout:1865: finishing write tid 5 to smithi16534841-22 2024-09-10T16:19:23.218 INFO:tasks.rados.rados.0.smithi165.stdout:1865: oid 22 updating version 0 to 772 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:1865: oid 22 version 772 is already newer than 771 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:1865: oid 22 version 772 is already newer than 770 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:1865: oid 22 version 772 is already newer than 769 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 772 (ObjNum 661 snap 207 seq_num 661) dirty exists 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:1865: left oid 22 (ObjNum 661 snap 207 seq_num 661) 2024-09-10T16:19:23.219 INFO:tasks.rados.rados.0.smithi165.stdout:1867: finishing write tid 1 to smithi16534841-28 2024-09-10T16:19:23.221 INFO:tasks.rados.rados.0.smithi165.stdout:1865: done (2 left) 2024-09-10T16:19:23.221 INFO:tasks.rados.rados.0.smithi165.stdout:1868: done (1 left) 2024-09-10T16:19:23.221 INFO:tasks.rados.rados.0.smithi165.stdout:1869: snap_remove snap 206 2024-09-10T16:19:23.570 INFO:tasks.rados.rados.0.smithi165.stdout:1869: done (1 left) 2024-09-10T16:19:23.570 INFO:tasks.rados.rados.0.smithi165.stdout:1870: snap_remove snap 201 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1870: done (1 left) 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1871: delete oid 37 current snap is 207 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1867: finishing write tid 2 to smithi16534841-28 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1867: finishing write tid 3 to smithi16534841-28 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1867: finishing write tid 4 to smithi16534841-28 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1867: finishing write tid 5 to smithi16534841-28 2024-09-10T16:19:24.580 INFO:tasks.rados.rados.0.smithi165.stdout:1867: oid 28 updating version 0 to 1024 2024-09-10T16:19:24.581 INFO:tasks.rados.rados.0.smithi165.stdout:1867: oid 28 updating version 1024 to 1026 2024-09-10T16:19:24.581 INFO:tasks.rados.rados.0.smithi165.stdout:1867: oid 28 version 1026 is already newer than 1025 2024-09-10T16:19:24.581 INFO:tasks.rados.rados.0.smithi165.stdout:1867: oid 28 updating version 1026 to 1027 2024-09-10T16:19:24.581 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1027 (ObjNum 663 snap 207 seq_num 663) dirty exists 2024-09-10T16:19:24.581 INFO:tasks.rados.rados.0.smithi165.stdout:1867: left oid 28 (ObjNum 663 snap 207 seq_num 663) 2024-09-10T16:19:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:1867: done (1 left) 2024-09-10T16:19:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:1871: done (0 left) 2024-09-10T16:19:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:1872: write (excl) oid 16 current snap is 207 2024-09-10T16:19:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:1872: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-10T16:19:24.592 INFO:tasks.rados.rados.0.smithi165.stdout:1872: writing smithi16534841-16 from 793462 to 1432007 tid 1 2024-09-10T16:19:24.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: writing smithi16534841-16 from 2106184 to 2870077 tid 2 2024-09-10T16:19:24.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: write oid 35 current snap is 207 2024-09-10T16:19:24.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-10T16:19:24.607 INFO:tasks.rados.rados.0.smithi165.stdout:1873: writing smithi16534841-35 from 410269 to 1124063 tid 1 2024-09-10T16:19:24.612 INFO:tasks.rados.rados.0.smithi165.stdout:1873: writing smithi16534841-35 from 1648313 to 2155864 tid 2 2024-09-10T16:19:24.618 INFO:tasks.rados.rados.0.smithi165.stdout:1873: writing smithi16534841-35 from 2917489 to 3210269 tid 3 2024-09-10T16:19:24.618 INFO:tasks.rados.rados.0.smithi165.stdout:1874: read oid 50 snap -1 2024-09-10T16:19:24.618 INFO:tasks.rados.rados.0.smithi165.stdout:1874: expect (ObjNum 576 snap 184 seq_num 576) 2024-09-10T16:19:24.619 INFO:tasks.rados.rados.0.smithi165.stdout:1875: snap_remove snap 205 2024-09-10T16:19:25.581 INFO:tasks.rados.rados.0.smithi165.stdout:1875: done (3 left) 2024-09-10T16:19:25.581 INFO:tasks.rados.rados.0.smithi165.stdout:1876: write (excl) oid 23 current snap is 207 2024-09-10T16:19:25.581 INFO:tasks.rados.rados.0.smithi165.stdout:1876: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-10T16:19:25.587 INFO:tasks.rados.rados.0.smithi165.stdout:1876: writing smithi16534841-23 from 427076 to 1216119 tid 1 2024-09-10T16:19:25.594 INFO:tasks.rados.rados.0.smithi165.stdout:1876: writing smithi16534841-23 from 1706795 to 2474357 tid 2 2024-09-10T16:19:25.599 INFO:tasks.rados.rados.0.smithi165.stdout:1876: writing smithi16534841-23 from 2944912 to 3227076 tid 3 2024-09-10T16:19:25.599 INFO:tasks.rados.rados.0.smithi165.stdout:1877: delete oid 8 current snap is 207 2024-09-10T16:19:25.599 INFO:tasks.rados.rados.0.smithi165.stdout:1872: finishing write tid 1 to smithi16534841-16 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: finishing write tid 2 to smithi16534841-16 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: finishing write tid 3 to smithi16534841-16 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: finishing write tid 4 to smithi16534841-16 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: oid 16 updating version 0 to 1265 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: oid 16 updating version 1265 to 1266 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: oid 16 updating version 1266 to 1267 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1267 (ObjNum 666 snap 207 seq_num 666) dirty exists 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1872: left oid 16 (ObjNum 666 snap 207 seq_num 666) 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1873: finishing write tid 1 to smithi16534841-35 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1873: finishing write tid 2 to smithi16534841-35 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1873: finishing write tid 3 to smithi16534841-35 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1873: finishing write tid 4 to smithi16534841-35 2024-09-10T16:19:25.600 INFO:tasks.rados.rados.0.smithi165.stdout:1873: finishing write tid 5 to smithi16534841-35 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: oid 35 updating version 0 to 773 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: oid 35 updating version 773 to 774 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: oid 35 updating version 774 to 775 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: oid 35 updating version 775 to 776 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 776 (ObjNum 667 snap 207 seq_num 667) dirty exists 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1873: left oid 35 (ObjNum 667 snap 207 seq_num 667) 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:1874: expect (ObjNum 576 snap 184 seq_num 576) 2024-09-10T16:19:25.601 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:25.680 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 262612 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: finishing write tid 1 to smithi16534841-23 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: finishing write tid 2 to smithi16534841-23 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: finishing write tid 3 to smithi16534841-23 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: finishing write tid 4 to smithi16534841-23 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: finishing write tid 5 to smithi16534841-23 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: oid 23 updating version 0 to 1124 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: oid 23 updating version 1124 to 1126 2024-09-10T16:19:25.719 INFO:tasks.rados.rados.0.smithi165.stdout:1876: oid 23 version 1126 is already newer than 1125 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1876: oid 23 updating version 1126 to 1127 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1127 (ObjNum 668 snap 207 seq_num 668) dirty exists 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1876: left oid 23 (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1872: done (4 left) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1873: done (3 left) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1874: done (2 left) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1876: done (1 left) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1877: done (0 left) 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:1878: rollback oid 20 current snap is 207 2024-09-10T16:19:25.720 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 202 2024-09-10T16:19:25.721 INFO:tasks.rados.rados.0.smithi165.stdout:1879: write oid 41 current snap is 207 2024-09-10T16:19:25.721 INFO:tasks.rados.rados.0.smithi165.stdout:1879: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-10T16:19:25.725 INFO:tasks.rados.rados.0.smithi165.stdout:1879: writing smithi16534841-41 from 460690 to 1116584 tid 1 2024-09-10T16:19:25.732 INFO:tasks.rados.rados.0.smithi165.stdout:1879: writing smithi16534841-41 from 1540112 to 2311343 tid 2 2024-09-10T16:19:25.738 INFO:tasks.rados.rados.0.smithi165.stdout:1879: writing smithi16534841-41 from 2716111 to 3255889 tid 3 2024-09-10T16:19:25.738 INFO:tasks.rados.rados.0.smithi165.stdout:1880: read oid 30 snap -1 2024-09-10T16:19:25.738 INFO:tasks.rados.rados.0.smithi165.stdout:1880: expect (ObjNum 637 snap 202 seq_num 637) 2024-09-10T16:19:25.738 INFO:tasks.rados.rados.0.smithi165.stdout:1881: snap_remove snap 204 2024-09-10T16:19:26.583 INFO:tasks.rados.rados.0.smithi165.stdout:1881: done (3 left) 2024-09-10T16:19:26.583 INFO:tasks.rados.rados.0.smithi165.stdout:1882: copy_from oid 24 from oid 8 current snap is 207 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1878: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1878: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1174 (ObjNum 640 snap 202 seq_num 640) dirty exists 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: finishing write tid 1 to smithi16534841-41 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: finishing write tid 2 to smithi16534841-41 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: finishing write tid 3 to smithi16534841-41 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: finishing write tid 4 to smithi16534841-41 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: finishing write tid 5 to smithi16534841-41 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: oid 41 updating version 0 to 1129 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: oid 41 updating version 1129 to 1131 2024-09-10T16:19:26.584 INFO:tasks.rados.rados.0.smithi165.stdout:1879: oid 41 version 1131 is already newer than 1128 2024-09-10T16:19:26.585 INFO:tasks.rados.rados.0.smithi165.stdout:1879: oid 41 version 1131 is already newer than 1130 2024-09-10T16:19:26.585 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1131 (ObjNum 670 snap 207 seq_num 670) dirty exists 2024-09-10T16:19:26.585 INFO:tasks.rados.rados.0.smithi165.stdout:1879: left oid 41 (ObjNum 670 snap 207 seq_num 670) 2024-09-10T16:19:26.585 INFO:tasks.rados.rados.0.smithi165.stdout:1880: expect (ObjNum 637 snap 202 seq_num 637) 2024-09-10T16:19:26.621 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:26.689 INFO:tasks.rados.rados.0.smithi165.stdout:1882: finishing copy_from to smithi16534841-24 2024-09-10T16:19:26.689 INFO:tasks.rados.rados.0.smithi165.stdout:1882: got expected ENOENT (src dne) 2024-09-10T16:19:26.689 INFO:tasks.rados.rados.0.smithi165.stdout:1882: finishing copy_from racing read to smithi16534841-24 2024-09-10T16:19:26.689 INFO:tasks.rados.rados.0.smithi165.stdout:1878: done (3 left) 2024-09-10T16:19:26.690 INFO:tasks.rados.rados.0.smithi165.stdout:1879: done (2 left) 2024-09-10T16:19:26.690 INFO:tasks.rados.rados.0.smithi165.stdout:1880: done (1 left) 2024-09-10T16:19:26.690 INFO:tasks.rados.rados.0.smithi165.stdout:1882: done (0 left) 2024-09-10T16:19:26.690 INFO:tasks.rados.rados.0.smithi165.stdout:1883: snap_remove snap 202 2024-09-10T16:19:27.593 INFO:tasks.rados.rados.0.smithi165.stdout:1883: done (0 left) 2024-09-10T16:19:27.593 INFO:tasks.rados.rados.0.smithi165.stdout:1884: write oid 20 current snap is 207 2024-09-10T16:19:27.593 INFO:tasks.rados.rados.0.smithi165.stdout:1884: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-10T16:19:27.600 INFO:tasks.rados.rados.0.smithi165.stdout:1884: writing smithi16534841-20 from 477497 to 1208640 tid 1 2024-09-10T16:19:27.607 INFO:tasks.rados.rados.0.smithi165.stdout:1884: writing smithi16534841-20 from 1998594 to 2513483 tid 2 2024-09-10T16:19:27.613 INFO:tasks.rados.rados.0.smithi165.stdout:1884: writing smithi16534841-20 from 3143534 to 3277497 tid 3 2024-09-10T16:19:27.613 INFO:tasks.rados.rados.0.smithi165.stdout:1885: write oid 33 current snap is 207 2024-09-10T16:19:27.613 INFO:tasks.rados.rados.0.smithi165.stdout:1885: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-10T16:19:27.617 INFO:tasks.rados.rados.0.smithi165.stdout:1885: writing smithi16534841-33 from 494304 to 900696 tid 1 2024-09-10T16:19:27.625 INFO:tasks.rados.rados.0.smithi165.stdout:1885: writing smithi16534841-33 from 1540723 to 2315623 tid 2 2024-09-10T16:19:27.631 INFO:tasks.rados.rados.0.smithi165.stdout:1885: writing smithi16534841-33 from 2770957 to 3294304 tid 3 2024-09-10T16:19:27.631 INFO:tasks.rados.rados.0.smithi165.stdout:1886: write oid 17 current snap is 207 2024-09-10T16:19:27.631 INFO:tasks.rados.rados.0.smithi165.stdout:1884: finishing write tid 1 to smithi16534841-20 2024-09-10T16:19:27.631 INFO:tasks.rados.rados.0.smithi165.stdout:1884: finishing write tid 2 to smithi16534841-20 2024-09-10T16:19:27.632 INFO:tasks.rados.rados.0.smithi165.stdout:1884: finishing write tid 3 to smithi16534841-20 2024-09-10T16:19:27.632 INFO:tasks.rados.rados.0.smithi165.stdout:1886: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-10T16:19:27.636 INFO:tasks.rados.rados.0.smithi165.stdout:1886: writing smithi16534841-17 from 511111 to 992752 tid 1 2024-09-10T16:19:27.642 INFO:tasks.rados.rados.0.smithi165.stdout:1886: writing smithi16534841-17 from 1599205 to 2117763 tid 2 2024-09-10T16:19:27.648 INFO:tasks.rados.rados.0.smithi165.stdout:1886: writing smithi16534841-17 from 2682027 to 3171974 tid 3 2024-09-10T16:19:27.648 INFO:tasks.rados.rados.0.smithi165.stdout:1887: delete oid 11 current snap is 207 2024-09-10T16:19:27.648 INFO:tasks.rados.rados.0.smithi165.stdout:1884: finishing write tid 4 to smithi16534841-20 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: finishing write tid 5 to smithi16534841-20 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: oid 20 updating version 0 to 1175 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: oid 20 updating version 1175 to 1178 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: oid 20 version 1178 is already newer than 1177 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: oid 20 version 1178 is already newer than 1176 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1178 (ObjNum 671 snap 207 seq_num 671) dirty exists 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1884: left oid 20 (ObjNum 671 snap 207 seq_num 671) 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: finishing write tid 1 to smithi16534841-33 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: finishing write tid 2 to smithi16534841-33 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: finishing write tid 3 to smithi16534841-33 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: finishing write tid 4 to smithi16534841-33 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: finishing write tid 5 to smithi16534841-33 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: oid 33 updating version 0 to 1180 2024-09-10T16:19:27.649 INFO:tasks.rados.rados.0.smithi165.stdout:1885: oid 33 updating version 1180 to 1182 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1885: oid 33 version 1182 is already newer than 1179 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1885: oid 33 version 1182 is already newer than 1181 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1182 (ObjNum 672 snap 207 seq_num 672) dirty exists 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1885: left oid 33 (ObjNum 672 snap 207 seq_num 672) 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1886: finishing write tid 1 to smithi16534841-17 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1886: finishing write tid 2 to smithi16534841-17 2024-09-10T16:19:27.650 INFO:tasks.rados.rados.0.smithi165.stdout:1884: done (3 left) 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1885: done (2 left) 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1887: done (1 left) 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1888: read oid 26 snap -1 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1888: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1889: write oid 11 current snap is 207 2024-09-10T16:19:27.651 INFO:tasks.rados.rados.0.smithi165.stdout:1889: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2024-09-10T16:19:27.656 INFO:tasks.rados.rados.0.smithi165.stdout:1889: writing smithi16534841-11 from 544725 to 1176864 tid 1 2024-09-10T16:19:27.662 INFO:tasks.rados.rados.0.smithi165.stdout:1889: writing smithi16534841-11 from 1716169 to 2238396 tid 2 2024-09-10T16:19:27.668 INFO:tasks.rados.rados.0.smithi165.stdout:1889: writing smithi16534841-11 from 2736873 to 3249364 tid 3 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1890: copy_from oid 27 from oid 8 current snap is 207 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: finishing write tid 3 to smithi16534841-17 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: finishing write tid 4 to smithi16534841-17 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: finishing write tid 5 to smithi16534841-17 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: oid 17 updating version 0 to 1104 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: oid 17 version 1104 is already newer than 1103 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: oid 17 version 1104 is already newer than 1102 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: oid 17 version 1104 is already newer than 1101 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1104 (ObjNum 673 snap 207 seq_num 673) dirty exists 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1886: left oid 17 (ObjNum 673 snap 207 seq_num 673) 2024-09-10T16:19:27.669 INFO:tasks.rados.rados.0.smithi165.stdout:1888: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:27.671 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1201710 2024-09-10T16:19:27.714 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1619422 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1889: finishing write tid 1 to smithi16534841-11 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1889: finishing write tid 2 to smithi16534841-11 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1890: finishing copy_from to smithi16534841-27 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1890: got expected ENOENT (src dne) 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1890: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:19:27.766 INFO:tasks.rados.rados.0.smithi165.stdout:1889: finishing write tid 3 to smithi16534841-11 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: finishing write tid 4 to smithi16534841-11 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: finishing write tid 5 to smithi16534841-11 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: oid 11 updating version 0 to 1230 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: oid 11 updating version 1230 to 1232 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: oid 11 version 1232 is already newer than 1231 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: oid 11 updating version 1232 to 1233 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1233 (ObjNum 675 snap 207 seq_num 675) dirty exists 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1889: left oid 11 (ObjNum 675 snap 207 seq_num 675) 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1886: done (3 left) 2024-09-10T16:19:27.767 INFO:tasks.rados.rados.0.smithi165.stdout:1888: done (2 left) 2024-09-10T16:19:27.768 INFO:tasks.rados.rados.0.smithi165.stdout:1889: done (1 left) 2024-09-10T16:19:27.768 INFO:tasks.rados.rados.0.smithi165.stdout:1890: done (0 left) 2024-09-10T16:19:27.768 INFO:tasks.rados.rados.0.smithi165.stdout:1891: delete oid 20 current snap is 207 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1891: done (0 left) 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1892: rollback oid 38 current snap is 207 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1892: done (0 left) 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1893: read oid 6 snap -1 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1893: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:19:27.769 INFO:tasks.rados.rados.0.smithi165.stdout:1894: read oid 13 snap -1 2024-09-10T16:19:27.770 INFO:tasks.rados.rados.0.smithi165.stdout:1894: expect (ObjNum 621 snap 193 seq_num 621) 2024-09-10T16:19:27.770 INFO:tasks.rados.rados.0.smithi165.stdout:1895: write (excl) oid 9 current snap is 207 2024-09-10T16:19:27.770 INFO:tasks.rados.rados.0.smithi165.stdout:1895: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-10T16:19:27.774 INFO:tasks.rados.rados.0.smithi165.stdout:1895: writing smithi16534841-9 from 578339 to 1077329 tid 1 2024-09-10T16:19:27.781 INFO:tasks.rados.rados.0.smithi165.stdout:1895: writing smithi16534841-9 from 1833133 to 2359029 tid 2 2024-09-10T16:19:27.787 INFO:tasks.rados.rados.0.smithi165.stdout:1895: writing smithi16534841-9 from 2791719 to 3378339 tid 3 2024-09-10T16:19:27.787 INFO:tasks.rados.rados.0.smithi165.stdout:1896: copy_from oid 45 from oid 37 current snap is 207 2024-09-10T16:19:27.787 INFO:tasks.rados.rados.0.smithi165.stdout:1893: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:19:27.877 INFO:tasks.rados.rados.0.smithi165.stdout:1894: expect (ObjNum 621 snap 193 seq_num 621) 2024-09-10T16:19:27.923 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1893692 2024-09-10T16:19:27.950 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1392126 2024-09-10T16:19:27.996 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:28.064 INFO:tasks.rados.rados.0.smithi165.stdout:1895: finishing write tid 1 to smithi16534841-9 2024-09-10T16:19:28.064 INFO:tasks.rados.rados.0.smithi165.stdout:1895: finishing write tid 2 to smithi16534841-9 2024-09-10T16:19:28.064 INFO:tasks.rados.rados.0.smithi165.stdout:1896: finishing copy_from to smithi16534841-45 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1896: got expected ENOENT (src dne) 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1896: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: finishing write tid 3 to smithi16534841-9 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: finishing write tid 4 to smithi16534841-9 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: finishing write tid 5 to smithi16534841-9 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: oid 9 updating version 0 to 1150 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: oid 9 version 1150 is already newer than 1149 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: oid 9 updating version 1150 to 1152 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: oid 9 version 1152 is already newer than 1151 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1152 (ObjNum 677 snap 207 seq_num 677) dirty exists 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1895: left oid 9 (ObjNum 677 snap 207 seq_num 677) 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1893: done (3 left) 2024-09-10T16:19:28.065 INFO:tasks.rados.rados.0.smithi165.stdout:1894: done (2 left) 2024-09-10T16:19:28.066 INFO:tasks.rados.rados.0.smithi165.stdout:1895: done (1 left) 2024-09-10T16:19:28.066 INFO:tasks.rados.rados.0.smithi165.stdout:1896: done (0 left) 2024-09-10T16:19:28.066 INFO:tasks.rados.rados.0.smithi165.stdout:1897: write oid 36 current snap is 207 2024-09-10T16:19:28.066 INFO:tasks.rados.rados.0.smithi165.stdout:1897: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-10T16:19:28.070 INFO:tasks.rados.rados.0.smithi165.stdout:1897: writing smithi16534841-36 from 595146 to 1169385 tid 1 2024-09-10T16:19:28.077 INFO:tasks.rados.rados.0.smithi165.stdout:1897: writing smithi16534841-36 from 1891615 to 2561169 tid 2 2024-09-10T16:19:28.082 INFO:tasks.rados.rados.0.smithi165.stdout:1897: writing smithi16534841-36 from 3219142 to 3395146 tid 3 2024-09-10T16:19:28.082 INFO:tasks.rados.rados.0.smithi165.stdout:1898: write oid 40 current snap is 207 2024-09-10T16:19:28.082 INFO:tasks.rados.rados.0.smithi165.stdout:1898: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2024-09-10T16:19:28.088 INFO:tasks.rados.rados.0.smithi165.stdout:1898: writing smithi16534841-40 from 611953 to 1261441 tid 1 2024-09-10T16:19:28.095 INFO:tasks.rados.rados.0.smithi165.stdout:1898: writing smithi16534841-40 from 1950097 to 2479662 tid 2 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1898: writing smithi16534841-40 from 3246565 to 3411953 tid 3 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1899: write (excl) oid 1 current snap is 207 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: finishing write tid 1 to smithi16534841-36 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: finishing write tid 2 to smithi16534841-36 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: finishing write tid 3 to smithi16534841-36 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: finishing write tid 4 to smithi16534841-36 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: finishing write tid 5 to smithi16534841-36 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: oid 36 updating version 0 to 1141 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: oid 36 updating version 1141 to 1142 2024-09-10T16:19:28.101 INFO:tasks.rados.rados.0.smithi165.stdout:1897: oid 36 version 1142 is already newer than 1140 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:1897: oid 36 version 1142 is already newer than 1139 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1142 (ObjNum 678 snap 207 seq_num 678) dirty exists 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:1897: left oid 36 (ObjNum 678 snap 207 seq_num 678) 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:1898: finishing write tid 1 to smithi16534841-40 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:1898: finishing write tid 2 to smithi16534841-40 2024-09-10T16:19:28.102 INFO:tasks.rados.rados.0.smithi165.stdout:1899: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2024-09-10T16:19:28.107 INFO:tasks.rados.rados.0.smithi165.stdout:1899: writing smithi16534841-1 from 628760 to 1353497 tid 1 2024-09-10T16:19:28.114 INFO:tasks.rados.rados.0.smithi165.stdout:1899: writing smithi16534841-1 from 1892226 to 2565449 tid 2 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1899: writing smithi16534841-1 from 3157635 to 3428760 tid 3 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1897: done (2 left) 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1900: read oid 26 snap -1 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1900: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1901: snap_create 2024-09-10T16:19:28.119 INFO:tasks.rados.rados.0.smithi165.stdout:1898: finishing write tid 3 to smithi16534841-40 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: finishing write tid 4 to smithi16534841-40 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: finishing write tid 5 to smithi16534841-40 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: oid 40 updating version 0 to 1155 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: oid 40 updating version 1155 to 1156 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: oid 40 version 1156 is already newer than 1154 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: oid 40 version 1156 is already newer than 1153 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1156 (ObjNum 679 snap 207 seq_num 679) dirty exists 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1898: left oid 40 (ObjNum 679 snap 207 seq_num 679) 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1899: finishing write tid 1 to smithi16534841-1 2024-09-10T16:19:28.120 INFO:tasks.rados.rados.0.smithi165.stdout:1899: finishing write tid 2 to smithi16534841-1 2024-09-10T16:19:28.121 INFO:tasks.rados.rados.0.smithi165.stdout:1899: finishing write tid 3 to smithi16534841-1 2024-09-10T16:19:28.121 INFO:tasks.rados.rados.0.smithi165.stdout:1899: finishing write tid 4 to smithi16534841-1 2024-09-10T16:19:28.121 INFO:tasks.rados.rados.0.smithi165.stdout:1899: finishing write tid 5 to smithi16534841-1 2024-09-10T16:19:28.121 INFO:tasks.rados.rados.0.smithi165.stdout:1899: oid 1 updating version 0 to 1133 2024-09-10T16:19:28.122 INFO:tasks.rados.rados.0.smithi165.stdout:1899: oid 1 updating version 1133 to 1135 2024-09-10T16:19:28.122 INFO:tasks.rados.rados.0.smithi165.stdout:1899: oid 1 version 1135 is already newer than 1134 2024-09-10T16:19:28.122 INFO:tasks.rados.rados.0.smithi165.stdout:1899: oid 1 version 1135 is already newer than 1132 2024-09-10T16:19:28.122 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1135 (ObjNum 680 snap 207 seq_num 680) dirty exists 2024-09-10T16:19:28.122 INFO:tasks.rados.rados.0.smithi165.stdout:1899: left oid 1 (ObjNum 680 snap 207 seq_num 680) 2024-09-10T16:19:28.126 INFO:tasks.rados.rados.0.smithi165.stdout:1900: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:28.213 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1898: done (3 left) 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1899: done (2 left) 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1900: done (1 left) 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1901: done (0 left) 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1902: rollback oid 33 current snap is 208 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 207 2024-09-10T16:19:28.572 INFO:tasks.rados.rados.0.smithi165.stdout:1903: delete oid 30 current snap is 208 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:1903: done (1 left) 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:1904: rollback oid 15 current snap is 208 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 207 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:1905: rollback oid 9 current snap is 208 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 207 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:1906: write oid 22 current snap is 208 2024-09-10T16:19:28.577 INFO:tasks.rados.rados.0.smithi165.stdout:1906: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-10T16:19:28.583 INFO:tasks.rados.rados.0.smithi165.stdout:1906: writing smithi16534841-22 from 662374 to 1137609 tid 1 2024-09-10T16:19:28.589 INFO:tasks.rados.rados.0.smithi165.stdout:1906: writing smithi16534841-22 from 1609190 to 2286082 tid 2 2024-09-10T16:19:28.596 INFO:tasks.rados.rados.0.smithi165.stdout:1906: writing smithi16534841-22 from 2812481 to 3436582 tid 3 2024-09-10T16:19:28.596 INFO:tasks.rados.rados.0.smithi165.stdout:1907: write oid 2 current snap is 208 2024-09-10T16:19:28.597 INFO:tasks.rados.rados.0.smithi165.stdout:1902: finishing rollback tid 0 to smithi16534841-33 2024-09-10T16:19:28.597 INFO:tasks.rados.rados.0.smithi165.stdout:1907: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-10T16:19:28.603 INFO:tasks.rados.rados.0.smithi165.stdout:1907: writing smithi16534841-2 from 679181 to 1229665 tid 1 2024-09-10T16:19:28.608 INFO:tasks.rados.rados.0.smithi165.stdout:1907: writing smithi16534841-2 from 1667672 to 2204575 tid 2 2024-09-10T16:19:28.615 INFO:tasks.rados.rados.0.smithi165.stdout:1907: writing smithi16534841-2 from 2839904 to 3275277 tid 3 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1908: snap_create 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1902: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1902: finishing rollback tid 2 to smithi16534841-33 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1186 (ObjNum 672 snap 207 seq_num 672) dirty exists 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1904: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1905: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1904: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1905: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1905: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1159 (ObjNum 677 snap 207 seq_num 677) dirty exists 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:1904: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:19:28.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 718 (ObjNum 643 snap 203 seq_num 643) dirty exists 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: finishing write tid 1 to smithi16534841-22 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: finishing write tid 2 to smithi16534841-22 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: finishing write tid 3 to smithi16534841-22 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: finishing write tid 4 to smithi16534841-22 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: finishing write tid 5 to smithi16534841-22 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: oid 22 updating version 0 to 777 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: oid 22 updating version 777 to 779 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: oid 22 version 779 is already newer than 778 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: oid 22 updating version 779 to 780 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 780 (ObjNum 682 snap 208 seq_num 682) dirty exists 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1906: left oid 22 (ObjNum 682 snap 208 seq_num 682) 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1907: finishing write tid 1 to smithi16534841-2 2024-09-10T16:19:28.617 INFO:tasks.rados.rados.0.smithi165.stdout:1907: finishing write tid 2 to smithi16534841-2 2024-09-10T16:19:28.618 INFO:tasks.rados.rados.0.smithi165.stdout:1907: finishing write tid 3 to smithi16534841-2 2024-09-10T16:19:28.618 INFO:tasks.rados.rados.0.smithi165.stdout:1907: finishing write tid 4 to smithi16534841-2 2024-09-10T16:19:28.618 INFO:tasks.rados.rados.0.smithi165.stdout:1907: finishing write tid 5 to smithi16534841-2 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:1907: oid 2 updating version 0 to 1158 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:1907: oid 2 updating version 1158 to 1159 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:1907: oid 2 version 1159 is already newer than 1157 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:1907: oid 2 version 1159 is already newer than 1156 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1159 (ObjNum 683 snap 208 seq_num 683) dirty exists 2024-09-10T16:19:28.619 INFO:tasks.rados.rados.0.smithi165.stdout:1907: left oid 2 (ObjNum 683 snap 208 seq_num 683) 2024-09-10T16:19:29.605 INFO:tasks.rados.rados.0.smithi165.stdout:1902: done (5 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1904: done (4 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1905: done (3 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1906: done (2 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1907: done (1 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1908: done (0 left) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1909: read oid 7 snap -1 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1909: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1910: rollback oid 41 current snap is 209 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 208 2024-09-10T16:19:29.606 INFO:tasks.rados.rados.0.smithi165.stdout:1911: read oid 29 snap -1 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1911: expect (ObjNum 646 snap 205 seq_num 646) 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1912: read oid 3 snap -1 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1912: expect deleted 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1913: read oid 30 snap 208 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1913: expect deleted 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1914: rollback oid 44 current snap is 209 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 207 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1915: read oid 25 snap -1 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1915: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1916: rollback oid 50 current snap is 209 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 207 2024-09-10T16:19:29.607 INFO:tasks.rados.rados.0.smithi165.stdout:1917: read oid 49 snap -1 2024-09-10T16:19:29.608 INFO:tasks.rados.rados.0.smithi165.stdout:1917: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:19:29.608 INFO:tasks.rados.rados.0.smithi165.stdout:1918: snap_create 2024-09-10T16:19:29.610 INFO:tasks.rados.rados.0.smithi165.stdout:1916: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:19:29.611 INFO:tasks.rados.rados.0.smithi165.stdout:1914: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:19:29.611 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 0 (ObjNum 49 snap 0 seq_num 1446270224) dirty dne 2024-09-10T16:19:29.612 INFO:tasks.rados.rados.0.smithi165.stdout:1916: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:19:29.613 INFO:tasks.rados.rados.0.smithi165.stdout:1916: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:19:29.613 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1236 (ObjNum 576 snap 184 seq_num 576) dirty exists 2024-09-10T16:19:29.614 INFO:tasks.rados.rados.0.smithi165.stdout:1911: expect (ObjNum 646 snap 205 seq_num 646) 2024-09-10T16:19:29.775 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:29.855 INFO:tasks.rados.rados.0.smithi165.stdout:1910: finishing rollback tid 0 to smithi16534841-41 2024-09-10T16:19:29.855 INFO:tasks.rados.rados.0.smithi165.stdout:1910: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:19:29.855 INFO:tasks.rados.rados.0.smithi165.stdout:1910: finishing rollback tid 2 to smithi16534841-41 2024-09-10T16:19:29.855 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1138 (ObjNum 670 snap 207 seq_num 670) dirty exists 2024-09-10T16:19:29.855 INFO:tasks.rados.rados.0.smithi165.stdout:1915: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:29.856 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:29.927 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 647656 2024-09-10T16:19:29.974 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:30.036 INFO:tasks.rados.rados.0.smithi165.stdout:1917: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:19:30.036 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:30.178 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 957512 2024-09-10T16:19:30.237 INFO:tasks.rados.rados.0.smithi165.stdout:1909: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:30.262 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2975180 2024-09-10T16:19:30.269 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:30.317 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 429969 2024-09-10T16:19:30.606 INFO:tasks.rados.rados.0.smithi165.stdout:1909: done (9 left) 2024-09-10T16:19:30.606 INFO:tasks.rados.rados.0.smithi165.stdout:1910: done (8 left) 2024-09-10T16:19:30.606 INFO:tasks.rados.rados.0.smithi165.stdout:1911: done (7 left) 2024-09-10T16:19:30.606 INFO:tasks.rados.rados.0.smithi165.stdout:1912: done (6 left) 2024-09-10T16:19:30.606 INFO:tasks.rados.rados.0.smithi165.stdout:1913: done (5 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1914: done (4 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1915: done (3 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1916: done (2 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1917: done (1 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1918: done (0 left) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1919: copy_from oid 34 from oid 24 current snap is 210 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1920: read oid 23 snap -1 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1920: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:19:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:1921: delete oid 11 current snap is 210 2024-09-10T16:19:30.611 INFO:tasks.rados.rados.0.smithi165.stdout:1921: done (2 left) 2024-09-10T16:19:30.611 INFO:tasks.rados.rados.0.smithi165.stdout:1922: snap_create 2024-09-10T16:19:30.612 INFO:tasks.rados.rados.0.smithi165.stdout:1920: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:19:30.653 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2956832 2024-09-10T16:19:30.680 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2138019 2024-09-10T16:19:30.751 INFO:tasks.rados.rados.0.smithi165.stdout:1919: finishing copy_from to smithi16534841-34 2024-09-10T16:19:30.751 INFO:tasks.rados.rados.0.smithi165.stdout:1919: got expected ENOENT (src dne) 2024-09-10T16:19:30.751 INFO:tasks.rados.rados.0.smithi165.stdout:1919: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:19:31.622 INFO:tasks.rados.rados.0.smithi165.stdout:1919: done (2 left) 2024-09-10T16:19:31.622 INFO:tasks.rados.rados.0.smithi165.stdout:1920: done (1 left) 2024-09-10T16:19:31.622 INFO:tasks.rados.rados.0.smithi165.stdout:1922: done (0 left) 2024-09-10T16:19:31.622 INFO:tasks.rados.rados.0.smithi165.stdout:1923: write oid 41 current snap is 211 2024-09-10T16:19:31.622 INFO:tasks.rados.rados.0.smithi165.stdout:1923: seq_num 685 ranges {712795=417335,1784636=540572,2894750=457917} 2024-09-10T16:19:31.627 INFO:tasks.rados.rados.0.smithi165.stdout:1923: writing smithi16534841-41 from 712795 to 1130130 tid 1 2024-09-10T16:19:31.634 INFO:tasks.rados.rados.0.smithi165.stdout:1923: writing smithi16534841-41 from 1784636 to 2325208 tid 2 2024-09-10T16:19:31.640 INFO:tasks.rados.rados.0.smithi165.stdout:1923: writing smithi16534841-41 from 2894750 to 3352667 tid 3 2024-09-10T16:19:31.640 INFO:tasks.rados.rados.0.smithi165.stdout:1924: snap_create 2024-09-10T16:19:31.645 INFO:tasks.rados.rados.0.smithi165.stdout:1923: finishing write tid 1 to smithi16534841-41 2024-09-10T16:19:31.645 INFO:tasks.rados.rados.0.smithi165.stdout:1923: finishing write tid 2 to smithi16534841-41 2024-09-10T16:19:31.646 INFO:tasks.rados.rados.0.smithi165.stdout:1923: finishing write tid 3 to smithi16534841-41 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: finishing write tid 4 to smithi16534841-41 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: finishing write tid 5 to smithi16534841-41 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: oid 41 updating version 0 to 1139 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: oid 41 updating version 1139 to 1142 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: oid 41 version 1142 is already newer than 1141 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: oid 41 version 1142 is already newer than 1140 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1142 (ObjNum 685 snap 211 seq_num 685) dirty exists 2024-09-10T16:19:31.647 INFO:tasks.rados.rados.0.smithi165.stdout:1923: left oid 41 (ObjNum 685 snap 211 seq_num 685) 2024-09-10T16:19:32.622 INFO:tasks.rados.rados.0.smithi165.stdout:1923: done (1 left) 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:1924: done (0 left) 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:1925: read oid 6 snap 211 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:1925: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:1926: rollback oid 48 current snap is 212 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 210 2024-09-10T16:19:32.623 INFO:tasks.rados.rados.0.smithi165.stdout:1927: delete oid 27 current snap is 212 2024-09-10T16:19:32.624 INFO:tasks.rados.rados.0.smithi165.stdout:1926: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:19:32.624 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 0 (ObjNum 1201 snap 0 seq_num 4088869880) dirty dne 2024-09-10T16:19:32.628 INFO:tasks.rados.rados.0.smithi165.stdout:1926: done (2 left) 2024-09-10T16:19:32.628 INFO:tasks.rados.rados.0.smithi165.stdout:1927: done (1 left) 2024-09-10T16:19:32.628 INFO:tasks.rados.rados.0.smithi165.stdout:1928: read oid 4 snap 211 2024-09-10T16:19:32.628 INFO:tasks.rados.rados.0.smithi165.stdout:1928: expect (ObjNum 604 snap 188 seq_num 604) 2024-09-10T16:19:32.628 INFO:tasks.rados.rados.0.smithi165.stdout:1929: snap_remove snap 207 2024-09-10T16:19:33.572 INFO:tasks.rados.rados.0.smithi165.stdout:1929: done (2 left) 2024-09-10T16:19:33.572 INFO:tasks.rados.rados.0.smithi165.stdout:1930: delete oid 35 current snap is 212 2024-09-10T16:19:33.572 INFO:tasks.rados.rados.0.smithi165.stdout:1925: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:19:33.573 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 636038 2024-09-10T16:19:33.644 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1148115 2024-09-10T16:19:33.666 INFO:tasks.rados.rados.0.smithi165.stdout:1928: expect (ObjNum 604 snap 188 seq_num 604) 2024-09-10T16:19:33.666 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:33.738 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 521546 2024-09-10T16:19:33.763 INFO:tasks.rados.rados.0.smithi165.stdout:1925: done (2 left) 2024-09-10T16:19:33.763 INFO:tasks.rados.rados.0.smithi165.stdout:1928: done (1 left) 2024-09-10T16:19:33.763 INFO:tasks.rados.rados.0.smithi165.stdout:1930: done (0 left) 2024-09-10T16:19:33.763 INFO:tasks.rados.rados.0.smithi165.stdout:1931: write oid 45 current snap is 212 2024-09-10T16:19:33.764 INFO:tasks.rados.rados.0.smithi165.stdout:1931: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-10T16:19:33.770 INFO:tasks.rados.rados.0.smithi165.stdout:1931: writing smithi16534841-45 from 763216 to 1406298 tid 1 2024-09-10T16:19:33.777 INFO:tasks.rados.rados.0.smithi165.stdout:1931: writing smithi16534841-45 from 1843729 to 2531628 tid 2 2024-09-10T16:19:33.783 INFO:tasks.rados.rados.0.smithi165.stdout:1931: writing smithi16534841-45 from 3260666 to 3563216 tid 3 2024-09-10T16:19:33.783 INFO:tasks.rados.rados.0.smithi165.stdout:1932: rollback oid 26 current snap is 212 2024-09-10T16:19:33.783 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 211 2024-09-10T16:19:33.783 INFO:tasks.rados.rados.0.smithi165.stdout:1931: finishing write tid 1 to smithi16534841-45 2024-09-10T16:19:33.783 INFO:tasks.rados.rados.0.smithi165.stdout:1933: snap_remove snap 209 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1933: done (2 left) 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1934: snap_create 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: finishing write tid 2 to smithi16534841-45 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: finishing write tid 3 to smithi16534841-45 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1932: finishing rollback tid 0 to smithi16534841-26 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1932: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1932: finishing rollback tid 2 to smithi16534841-26 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1287 (ObjNum 585 snap 186 seq_num 585) dirty exists 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: finishing write tid 4 to smithi16534841-45 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: finishing write tid 5 to smithi16534841-45 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: oid 45 updating version 0 to 897 2024-09-10T16:19:34.637 INFO:tasks.rados.rados.0.smithi165.stdout:1931: oid 45 version 897 is already newer than 896 2024-09-10T16:19:34.638 INFO:tasks.rados.rados.0.smithi165.stdout:1931: oid 45 version 897 is already newer than 895 2024-09-10T16:19:34.638 INFO:tasks.rados.rados.0.smithi165.stdout:1931: oid 45 updating version 897 to 898 2024-09-10T16:19:34.638 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 898 (ObjNum 688 snap 212 seq_num 688) dirty exists 2024-09-10T16:19:34.638 INFO:tasks.rados.rados.0.smithi165.stdout:1931: left oid 45 (ObjNum 688 snap 212 seq_num 688) 2024-09-10T16:19:35.926 INFO:tasks.rados.rados.0.smithi165.stdout:1931: done (2 left) 2024-09-10T16:19:35.926 INFO:tasks.rados.rados.0.smithi165.stdout:1932: done (1 left) 2024-09-10T16:19:35.926 INFO:tasks.rados.rados.0.smithi165.stdout:1934: done (0 left) 2024-09-10T16:19:35.927 INFO:tasks.rados.rados.0.smithi165.stdout:1935: snap_remove snap 208 2024-09-10T16:19:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:1935: done (0 left) 2024-09-10T16:19:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:1936: snap_remove snap 211 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1936: done (0 left) 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1937: read oid 26 snap -1 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1937: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1938: read oid 12 snap 212 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1938: expect deleted 2024-09-10T16:19:37.661 INFO:tasks.rados.rados.0.smithi165.stdout:1939: delete oid 15 current snap is 213 2024-09-10T16:19:37.664 INFO:tasks.rados.rados.0.smithi165.stdout:1939: done (2 left) 2024-09-10T16:19:37.664 INFO:tasks.rados.rados.0.smithi165.stdout:1940: write oid 21 current snap is 213 2024-09-10T16:19:37.664 INFO:tasks.rados.rados.0.smithi165.stdout:1940: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-10T16:19:37.672 INFO:tasks.rados.rados.0.smithi165.stdout:1940: writing smithi16534841-21 from 796830 to 1590410 tid 1 2024-09-10T16:19:37.681 INFO:tasks.rados.rados.0.smithi165.stdout:1940: writing smithi16534841-21 from 2360693 to 3052261 tid 2 2024-09-10T16:19:37.681 INFO:tasks.rados.rados.0.smithi165.stdout:1941: write oid 4 current snap is 213 2024-09-10T16:19:37.681 INFO:tasks.rados.rados.0.smithi165.stdout:1941: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-10T16:19:37.686 INFO:tasks.rados.rados.0.smithi165.stdout:1941: writing smithi16534841-4 from 413637 to 882466 tid 1 2024-09-10T16:19:37.692 INFO:tasks.rados.rados.0.smithi165.stdout:1941: writing smithi16534841-4 from 1619175 to 2054401 tid 2 2024-09-10T16:19:37.699 INFO:tasks.rados.rados.0.smithi165.stdout:1941: writing smithi16534841-4 from 2542935 to 3301190 tid 3 2024-09-10T16:19:37.699 INFO:tasks.rados.rados.0.smithi165.stdout:1942: snap_remove snap 210 2024-09-10T16:19:38.579 INFO:tasks.rados.rados.0.smithi165.stdout:1942: done (4 left) 2024-09-10T16:19:38.579 INFO:tasks.rados.rados.0.smithi165.stdout:1943: write oid 20 current snap is 213 2024-09-10T16:19:38.579 INFO:tasks.rados.rados.0.smithi165.stdout:1937: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:19:38.579 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:38.692 INFO:tasks.rados.rados.0.smithi165.stdout:1940: finishing write tid 1 to smithi16534841-21 2024-09-10T16:19:38.692 INFO:tasks.rados.rados.0.smithi165.stdout:1940: finishing write tid 2 to smithi16534841-21 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: finishing write tid 3 to smithi16534841-21 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: finishing write tid 4 to smithi16534841-21 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: oid 21 updating version 0 to 1188 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: oid 21 version 1188 is already newer than 1187 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: oid 21 updating version 1188 to 1189 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1189 (ObjNum 690 snap 213 seq_num 690) dirty exists 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1940: left oid 21 (ObjNum 690 snap 213 seq_num 690) 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1941: finishing write tid 1 to smithi16534841-4 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1941: finishing write tid 2 to smithi16534841-4 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1941: finishing write tid 3 to smithi16534841-4 2024-09-10T16:19:38.693 INFO:tasks.rados.rados.0.smithi165.stdout:1943: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-10T16:19:38.698 INFO:tasks.rados.rados.0.smithi165.stdout:1943: writing smithi16534841-20 from 430444 to 1090875 tid 1 2024-09-10T16:19:38.705 INFO:tasks.rados.rados.0.smithi165.stdout:1943: writing smithi16534841-20 from 1677657 to 2372894 tid 2 2024-09-10T16:19:38.712 INFO:tasks.rados.rados.0.smithi165.stdout:1943: writing smithi16534841-20 from 2970358 to 3539885 tid 3 2024-09-10T16:19:38.712 INFO:tasks.rados.rados.0.smithi165.stdout:1937: done (4 left) 2024-09-10T16:19:38.712 INFO:tasks.rados.rados.0.smithi165.stdout:1938: done (3 left) 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1940: done (2 left) 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1944: read oid 3 snap 212 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1944: expect deleted 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1945: snap_create 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1941: finishing write tid 4 to smithi16534841-4 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1941: finishing write tid 5 to smithi16534841-4 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1941: oid 4 updating version 0 to 721 2024-09-10T16:19:38.713 INFO:tasks.rados.rados.0.smithi165.stdout:1941: oid 4 updating version 721 to 722 2024-09-10T16:19:38.714 INFO:tasks.rados.rados.0.smithi165.stdout:1941: oid 4 version 722 is already newer than 720 2024-09-10T16:19:38.714 INFO:tasks.rados.rados.0.smithi165.stdout:1941: oid 4 updating version 722 to 723 2024-09-10T16:19:38.714 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 723 (ObjNum 691 snap 213 seq_num 691) dirty exists 2024-09-10T16:19:38.714 INFO:tasks.rados.rados.0.smithi165.stdout:1941: left oid 4 (ObjNum 691 snap 213 seq_num 691) 2024-09-10T16:19:38.715 INFO:tasks.rados.rados.0.smithi165.stdout:1943: finishing write tid 1 to smithi16534841-20 2024-09-10T16:19:38.715 INFO:tasks.rados.rados.0.smithi165.stdout:1943: finishing write tid 2 to smithi16534841-20 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: finishing write tid 3 to smithi16534841-20 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: finishing write tid 4 to smithi16534841-20 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: finishing write tid 5 to smithi16534841-20 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: oid 20 updating version 0 to 1193 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: oid 20 version 1193 is already newer than 1192 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: oid 20 version 1193 is already newer than 1191 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: oid 20 version 1193 is already newer than 1190 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1193 (ObjNum 692 snap 213 seq_num 692) dirty exists 2024-09-10T16:19:38.716 INFO:tasks.rados.rados.0.smithi165.stdout:1943: left oid 20 (ObjNum 692 snap 213 seq_num 692) 2024-09-10T16:19:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:1941: done (3 left) 2024-09-10T16:19:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:1943: done (2 left) 2024-09-10T16:19:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:1944: done (1 left) 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1945: done (0 left) 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1946: read oid 7 snap -1 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1946: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1947: read oid 25 snap -1 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1947: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1948: rollback oid 6 current snap is 214 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 212 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1949: write (excl) oid 37 current snap is 214 2024-09-10T16:19:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:1949: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-10T16:19:39.670 INFO:tasks.rados.rados.0.smithi165.stdout:1949: writing smithi16534841-37 from 447251 to 1182931 tid 1 2024-09-10T16:19:39.676 INFO:tasks.rados.rados.0.smithi165.stdout:1949: writing smithi16534841-37 from 1736139 to 2175034 tid 2 2024-09-10T16:19:39.682 INFO:tasks.rados.rados.0.smithi165.stdout:1949: writing smithi16534841-37 from 2597781 to 3378580 tid 3 2024-09-10T16:19:39.682 INFO:tasks.rados.rados.0.smithi165.stdout:1950: write (excl) oid 1 current snap is 214 2024-09-10T16:19:39.682 INFO:tasks.rados.rados.0.smithi165.stdout:1950: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-10T16:19:39.687 INFO:tasks.rados.rados.0.smithi165.stdout:1950: writing smithi16534841-1 from 464058 to 874987 tid 1 2024-09-10T16:19:39.694 INFO:tasks.rados.rados.0.smithi165.stdout:1950: writing smithi16534841-1 from 1394621 to 2093527 tid 2 2024-09-10T16:19:39.700 INFO:tasks.rados.rados.0.smithi165.stdout:1950: writing smithi16534841-1 from 2625204 to 3217275 tid 3 2024-09-10T16:19:39.703 INFO:tasks.rados.rados.0.smithi165.stdout:1950: writing smithi16534841-1 from 3656107 to 3664058 tid 4 2024-09-10T16:19:39.703 INFO:tasks.rados.rados.0.smithi165.stdout:1951: delete oid 38 current snap is 214 2024-09-10T16:19:39.703 INFO:tasks.rados.rados.0.smithi165.stdout:1947: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:39.703 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 233290 2024-09-10T16:19:39.886 INFO:tasks.rados.rados.0.smithi165.stdout:1946: expect (ObjNum 538 snap 169 seq_num 538) 2024-09-10T16:19:39.890 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1435973 2024-09-10T16:19:40.013 INFO:tasks.rados.rados.0.smithi165.stdout:1948: finishing rollback tid 0 to smithi16534841-6 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1948: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1946: done (5 left) 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1947: done (4 left) 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1951: done (3 left) 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1952: delete oid 42 current snap is 214 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1948: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 643 (ObjNum 660 snap 207 seq_num 660) dirty exists 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: finishing write tid 1 to smithi16534841-37 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: finishing write tid 2 to smithi16534841-37 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: finishing write tid 3 to smithi16534841-37 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: finishing write tid 4 to smithi16534841-37 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: finishing write tid 5 to smithi16534841-37 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: oid 37 updating version 0 to 1135 2024-09-10T16:19:40.014 INFO:tasks.rados.rados.0.smithi165.stdout:1949: oid 37 version 1135 is already newer than 1134 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1949: oid 37 version 1135 is already newer than 1133 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1949: oid 37 updating version 1135 to 1136 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1136 (ObjNum 693 snap 214 seq_num 693) dirty exists 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1949: left oid 37 (ObjNum 693 snap 214 seq_num 693) 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 1 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 2 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 3 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 4 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 5 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: finishing write tid 6 to smithi16534841-1 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: oid 1 updating version 0 to 1147 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: oid 1 version 1147 is already newer than 1146 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: oid 1 version 1147 is already newer than 1145 2024-09-10T16:19:40.015 INFO:tasks.rados.rados.0.smithi165.stdout:1950: oid 1 version 1147 is already newer than 1143 2024-09-10T16:19:40.016 INFO:tasks.rados.rados.0.smithi165.stdout:1950: oid 1 version 1147 is already newer than 1144 2024-09-10T16:19:40.016 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1147 (ObjNum 694 snap 214 seq_num 694) dirty exists 2024-09-10T16:19:40.016 INFO:tasks.rados.rados.0.smithi165.stdout:1950: left oid 1 (ObjNum 694 snap 214 seq_num 694) 2024-09-10T16:19:40.016 INFO:tasks.rados.rados.0.smithi165.stdout:1948: done (3 left) 2024-09-10T16:19:40.016 INFO:tasks.rados.rados.0.smithi165.stdout:1949: done (2 left) 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1950: done (1 left) 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1952: done (0 left) 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1953: read oid 3 snap -1 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1953: expect deleted 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1954: write oid 32 current snap is 214 2024-09-10T16:19:40.017 INFO:tasks.rados.rados.0.smithi165.stdout:1954: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2024-09-10T16:19:40.022 INFO:tasks.rados.rados.0.smithi165.stdout:1954: writing smithi16534841-32 from 514479 to 1151155 tid 1 2024-09-10T16:19:40.028 INFO:tasks.rados.rados.0.smithi165.stdout:1954: writing smithi16534841-32 from 1853714 to 2299947 tid 2 2024-09-10T16:19:40.035 INFO:tasks.rados.rados.0.smithi165.stdout:1954: writing smithi16534841-32 from 2991120 to 3533360 tid 3 2024-09-10T16:19:40.036 INFO:tasks.rados.rados.0.smithi165.stdout:1955: snap_create 2024-09-10T16:19:40.036 INFO:tasks.rados.rados.0.smithi165.stdout:1954: finishing write tid 1 to smithi16534841-32 2024-09-10T16:19:40.036 INFO:tasks.rados.rados.0.smithi165.stdout:1954: finishing write tid 2 to smithi16534841-32 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: finishing write tid 3 to smithi16534841-32 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: finishing write tid 4 to smithi16534841-32 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: finishing write tid 5 to smithi16534841-32 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: oid 32 updating version 0 to 647 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: oid 32 version 647 is already newer than 646 2024-09-10T16:19:40.039 INFO:tasks.rados.rados.0.smithi165.stdout:1954: oid 32 version 647 is already newer than 645 2024-09-10T16:19:40.040 INFO:tasks.rados.rados.0.smithi165.stdout:1954: oid 32 version 647 is already newer than 644 2024-09-10T16:19:40.040 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 647 (ObjNum 697 snap 214 seq_num 697) dirty exists 2024-09-10T16:19:40.040 INFO:tasks.rados.rados.0.smithi165.stdout:1954: left oid 32 (ObjNum 697 snap 214 seq_num 697) 2024-09-10T16:19:40.668 INFO:tasks.rados.rados.0.smithi165.stdout:1953: done (2 left) 2024-09-10T16:19:40.669 INFO:tasks.rados.rados.0.smithi165.stdout:1954: done (1 left) 2024-09-10T16:19:40.669 INFO:tasks.rados.rados.0.smithi165.stdout:1955: done (0 left) 2024-09-10T16:19:40.669 INFO:tasks.rados.rados.0.smithi165.stdout:1956: write oid 5 current snap is 215 2024-09-10T16:19:40.669 INFO:tasks.rados.rados.0.smithi165.stdout:1956: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-10T16:19:40.675 INFO:tasks.rados.rados.0.smithi165.stdout:1956: writing smithi16534841-5 from 531286 to 1243211 tid 1 2024-09-10T16:19:40.682 INFO:tasks.rados.rados.0.smithi165.stdout:1956: writing smithi16534841-5 from 1912196 to 2618440 tid 2 2024-09-10T16:19:40.688 INFO:tasks.rados.rados.0.smithi165.stdout:1956: writing smithi16534841-5 from 3018543 to 3731286 tid 3 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1957: read oid 34 snap -1 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1957: expect (ObjNum 644 snap 203 seq_num 644) 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1958: write oid 27 current snap is 215 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1956: finishing write tid 1 to smithi16534841-5 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1956: finishing write tid 2 to smithi16534841-5 2024-09-10T16:19:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:1958: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-10T16:19:40.695 INFO:tasks.rados.rados.0.smithi165.stdout:1958: writing smithi16534841-27 from 548093 to 1335267 tid 1 2024-09-10T16:19:40.701 INFO:tasks.rados.rados.0.smithi165.stdout:1958: writing smithi16534841-27 from 1970678 to 2420580 tid 2 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1958: writing smithi16534841-27 from 3045966 to 3727103 tid 3 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1959: snap_create 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1956: finishing write tid 3 to smithi16534841-5 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1956: finishing write tid 4 to smithi16534841-5 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1956: finishing write tid 5 to smithi16534841-5 2024-09-10T16:19:40.709 INFO:tasks.rados.rados.0.smithi165.stdout:1956: oid 5 updating version 0 to 651 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:1956: oid 5 version 651 is already newer than 649 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:1956: oid 5 version 651 is already newer than 648 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:1956: oid 5 version 651 is already newer than 650 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 651 (ObjNum 698 snap 215 seq_num 698) dirty exists 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:1956: left oid 5 (ObjNum 698 snap 215 seq_num 698) 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:1957: expect (ObjNum 644 snap 203 seq_num 644) 2024-09-10T16:19:40.710 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:40.823 INFO:tasks.rados.rados.0.smithi165.stdout:1958: finishing write tid 1 to smithi16534841-27 2024-09-10T16:19:40.823 INFO:tasks.rados.rados.0.smithi165.stdout:1958: finishing write tid 2 to smithi16534841-27 2024-09-10T16:19:40.823 INFO:tasks.rados.rados.0.smithi165.stdout:1958: finishing write tid 3 to smithi16534841-27 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: finishing write tid 4 to smithi16534841-27 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: finishing write tid 5 to smithi16534841-27 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: oid 27 updating version 0 to 1163 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: oid 27 updating version 1163 to 1164 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: oid 27 version 1164 is already newer than 1162 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: oid 27 version 1164 is already newer than 1161 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1164 (ObjNum 699 snap 215 seq_num 699) dirty exists 2024-09-10T16:19:40.824 INFO:tasks.rados.rados.0.smithi165.stdout:1958: left oid 27 (ObjNum 699 snap 215 seq_num 699) 2024-09-10T16:19:41.362 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:19:41.362 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:19:41.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:19:41.366 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:19:41.367 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:19:41.367 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.474+0000 7fbed4fc2640 10 monclient: get_monmap_and_config 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.474+0000 7fbed4fc2640 10 monclient: build_initial_monmap 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.474+0000 7fbed4fc2640 10 monclient: monmap: 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:41.475558+0000 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:41.475558+0000 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:41.476 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:41.477 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:41.477 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:41.477 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:41.477 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 Processor -- start 2024-09-10T16:19:41.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 -- start start 2024-09-10T16:19:41.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient: init 2024-09-10T16:19:41.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-c con 0x7fbed009f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbed00a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbed00a27b0 0x7fbed013dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-a con 0x7fbed00a27b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbed009fd10 con 0x7fbed00a2380 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbed009cd40 con 0x7fbed00a27b0 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbed00a2c40 con 0x7fbed009f930 2024-09-10T16:19:41.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.475+0000 7fbed4fc2640 10 monclient(hunting): authenticate will time out at 1559.962989s 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbed00a27b0 0x7fbed013dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbecdd9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbed00a2380 0x7fbed013a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47044/0 (socket says 172.21.15.164:47044) 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbecdd9b640 1 -- 172.21.15.164:0/140471921 learned_addr learned my addr 172.21.15.164:0/140471921 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): get_auth_request con 0x7fbed00a27b0 auth_method 0 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 1 --2- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 10 monclient(hunting): get_auth_request con 0x7fbed009f930 auth_method 0 2024-09-10T16:19:41.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 1 -- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 msgr2=0x7fbed00a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 1 -- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 msgr2=0x7fbed00a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 1 --2- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:41.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbece59c640 1 --2- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient(hunting): handle_auth_done global_id 144189 payload 274 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 -- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 msgr2=0x7fbed00a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 --2- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 -- 172.21.15.164:0/140471921 >> v1:172.21.15.164:6789/0 conn(0x7fbed00a2380 legacy=0x7fbed013a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient: _finish_hunting 0 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 monclient: found mon.noname-a 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:41.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.476+0000 7fbeced9d640 1 -- 172.21.15.164:0/140471921 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed009a0d0 con 0x7fbed00a27b0 2024-09-10T16:19:41.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbeced9d640 1 -- 172.21.15.164:0/140471921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 msgr2=0x7fbed013dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbeced9d640 1 --2- 172.21.15.164:0/140471921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed013dcf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbed009f620 tx=0x7fbec802a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88cf7941b709734 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 1 -- 172.21.15.164:0/140471921 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8007950 con 0x7fbed00a27b0 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: dump: 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.490 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: _finish_auth 0 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: _check_auth_tickets 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 1 -- 172.21.15.164:0/140471921 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec8007af0 con 0x7fbed00a27b0 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 5 monclient: authenticate success, global_id 144189 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:41.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 1 -- 172.21.15.164:0/140471921 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec8031400 con 0x7fbed00a27b0 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 4 set_mon_vals no callback set 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbecd59a640 10 monclient: dump: 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.492 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 10 monclient: shutdown 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 1 -- 172.21.15.164:0/140471921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 msgr2=0x7fbed013dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/140471921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed013dcf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbed009f620 tx=0x7fbec802a480 comp rx=0 tx=0).stop 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 1 -- 172.21.15.164:0/140471921 shutdown_connections 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.479+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/140471921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed013dcf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/140471921 >> v2:172.21.15.165:3300/0 conn(0x7fbed009f930 0x7fbed00a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 1 -- 172.21.15.164:0/140471921 >> 172.21.15.164:0/140471921 conn(0x7fbed00a3110 msgr2=0x7fbed00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 1 -- 172.21.15.164:0/140471921 shutdown_connections 2024-09-10T16:19:41.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 1 -- 172.21.15.164:0/140471921 wait complete. 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 10 monclient: build_initial_monmap 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.480+0000 7fbed4fc2640 10 monclient: monmap: 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 Processor -- start 2024-09-10T16:19:41.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 -- start start 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient: init 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-c con 0x7fbed009f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-b con 0x7fbed00a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed0099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): picked mon.noname-a con 0x7fbed00a27b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbed009ee00 con 0x7fbed00a27b0 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:41.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbed009e8d0 con 0x7fbed00a2380 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbed009d930 con 0x7fbed009f930 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbed4fc2640 10 monclient(hunting): authenticate will time out at 1559.968682s 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed0099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57502/0 (socket says 172.21.15.164:57502) 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 1 -- 172.21.15.164:0/2799655256 learned_addr learned my addr 172.21.15.164:0/2799655256 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): get_auth_request con 0x7fbed009f930 auth_method 0 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:41.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): get_auth_request con 0x7fbed00a27b0 auth_method 0 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbece59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbeced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbecdd9b640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbecdd9b640 10 monclient(hunting): get_auth_request con 0x7fbed00a2380 auth_method 0 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbecdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbecdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:41.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.481+0000 7fbecdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbecdd9b640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 msgr2=0x7fbed00994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbecdd9b640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 msgr2=0x7fbed00994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbecdd9b640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbecdd9b640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 10 monclient(hunting): handle_auth_done global_id 114443 payload 824 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 msgr2=0x7fbed00994d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 msgr2=0x7fbed0099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed0099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 10 monclient: _finish_hunting 0 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 monclient: found mon.noname-c 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:41.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbece59c640 1 -- 172.21.15.164:0/2799655256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbed0095ca0 con 0x7fbed009f930 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.482+0000 7fbeced9d640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed0099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbece59c640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 msgr2=0x7fbed01cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbece59c640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbec000d7d0 tx=0x7fbec000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000b870 con 0x7fbed009f930 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: dump: 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:41.499 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: _finish_auth 0 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbec0019070 con 0x7fbed009f930 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 5 monclient: authenticate success, global_id 114443 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbec000bc70 con 0x7fbed009f930 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:41.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbebeffd640 10 monclient: dump: 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 10 monclient: _renew_subs 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbeccd99640 4 set_mon_vals no callback set 2024-09-10T16:19:41.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbed01cd3f0 con 0x7fbed009f930 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 10 monclient: _renew_subs 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.484+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbed006e7d0 con 0x7fbed009f930 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbec001e460 con 0x7fbed009f930 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbed4fc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbed4fc2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbed4fc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbed009c990 con 0x7fbed009f930 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbebeffd640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 0x7fbeb00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(459..459 src has 184..459) ==== 4739+0+0 (secure 0 0 0) 0x7fbec00531c0 con 0x7fbed009f930 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbecdd9b640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 0x7fbeb00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:41.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.485+0000 7fbecdd9b640 10 monclient: get_auth_request con 0x7fbeb003e210 auth_method 0 2024-09-10T16:19:41.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.488+0000 7fbecdd9b640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 0x7fbeb00406d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbec4007fd0 tx=0x7fbec4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:41.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.490+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbec000be10 con 0x7fbed009f930 2024-09-10T16:19:41.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.490+0000 7fbebeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:41.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.490+0000 7fbebeffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:41.671 INFO:tasks.rados.rados.0.smithi165.stdout:1956: done (3 left) 2024-09-10T16:19:41.671 INFO:tasks.rados.rados.0.smithi165.stdout:1957: done (2 left) 2024-09-10T16:19:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:1958: done (1 left) 2024-09-10T16:19:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:1959: done (0 left) 2024-09-10T16:19:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:1960: snap_remove snap 212 2024-09-10T16:19:41.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.731+0000 7fbed4fc2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:41.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.731+0000 7fbed4fc2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:41.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.731+0000 7fbed4fc2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:41.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.731+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbed006ec30 con 0x7fbed009f930 2024-09-10T16:19:41.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.732+0000 7fbebeffd640 1 -- 172.21.15.164:0/2799655256 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fbec0051be0 con 0x7fbed009f930 2024-09-10T16:19:41.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.732+0000 7fbebeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:41.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.732+0000 7fbebeffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:41.734 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":126,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:41.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 msgr2=0x7fbeb00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 0x7fbeb00406d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fbec4007fd0 tx=0x7fbec4009040 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 10 monclient: shutdown 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 msgr2=0x7fbed01cceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbec000d7d0 tx=0x7fbec000dc90 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 shutdown_connections 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbeb003e210 0x7fbeb00406d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbed00a27b0 0x7fbed0099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbed00a2380 0x7fbed00994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 --2- 172.21.15.164:0/2799655256 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbed009f930 0x7fbed01cceb0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:41.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 >> 172.21.15.164:0/2799655256 conn(0x7fbed00a3110 msgr2=0x7fbed00a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:41.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 shutdown_connections 2024-09-10T16:19:41.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:41.734+0000 7fbed4fc2640 1 -- 172.21.15.164:0/2799655256 wait complete. 2024-09-10T16:19:42.674 INFO:tasks.rados.rados.0.smithi165.stdout:1960: done (0 left) 2024-09-10T16:19:42.675 INFO:tasks.rados.rados.0.smithi165.stdout:1961: read oid 25 snap -1 2024-09-10T16:19:42.675 INFO:tasks.rados.rados.0.smithi165.stdout:1961: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:42.675 INFO:tasks.rados.rados.0.smithi165.stdout:1962: delete oid 44 current snap is 216 2024-09-10T16:19:42.683 INFO:tasks.rados.rados.0.smithi165.stdout:1961: expect (ObjNum 626 snap 195 seq_num 626) 2024-09-10T16:19:42.684 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:19:42.789 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1982031 2024-09-10T16:19:42.860 INFO:tasks.rados.rados.0.smithi165.stdout:1961: done (1 left) 2024-09-10T16:19:42.860 INFO:tasks.rados.rados.0.smithi165.stdout:1962: done (0 left) 2024-09-10T16:19:42.860 INFO:tasks.rados.rados.0.smithi165.stdout:1963: copy_from oid 5 from oid 3 current snap is 216 2024-09-10T16:19:42.861 INFO:tasks.rados.rados.0.smithi165.stdout:1964: read oid 10 snap -1 2024-09-10T16:19:42.861 INFO:tasks.rados.rados.0.smithi165.stdout:1964: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:19:42.861 INFO:tasks.rados.rados.0.smithi165.stdout:1965: delete oid 25 current snap is 216 2024-09-10T16:19:42.862 INFO:tasks.rados.rados.0.smithi165.stdout:1964: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:19:42.909 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1750137 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1963: finishing copy_from to smithi16534841-5 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1963: got expected ENOENT (src dne) 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1963: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1963: done (2 left) 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1964: done (1 left) 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1965: done (0 left) 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1966: write (excl) oid 4 current snap is 216 2024-09-10T16:19:42.912 INFO:tasks.rados.rados.0.smithi165.stdout:1966: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2024-09-10T16:19:42.918 INFO:tasks.rados.rados.0.smithi165.stdout:1966: writing smithi16534841-4 from 598514 to 1327788 tid 1 2024-09-10T16:19:42.924 INFO:tasks.rados.rados.0.smithi165.stdout:1966: writing smithi16534841-4 from 1746124 to 2343353 tid 2 2024-09-10T16:19:42.931 INFO:tasks.rados.rados.0.smithi165.stdout:1966: writing smithi16534841-4 from 3128235 to 3643188 tid 3 2024-09-10T16:19:42.931 INFO:tasks.rados.rados.0.smithi165.stdout:1967: write (excl) oid 23 current snap is 216 2024-09-10T16:19:42.931 INFO:tasks.rados.rados.0.smithi165.stdout:1967: seq_num 703 ranges {615321=404523,1804606=457240,2755658=726225} 2024-09-10T16:19:42.937 INFO:tasks.rados.rados.0.smithi165.stdout:1967: writing smithi16534841-23 from 615321 to 1019844 tid 1 2024-09-10T16:19:42.943 INFO:tasks.rados.rados.0.smithi165.stdout:1967: writing smithi16534841-23 from 1804606 to 2261846 tid 2 2024-09-10T16:19:42.950 INFO:tasks.rados.rados.0.smithi165.stdout:1967: writing smithi16534841-23 from 2755658 to 3481883 tid 3 2024-09-10T16:19:42.950 INFO:tasks.rados.rados.0.smithi165.stdout:1968: write (excl) oid 43 current snap is 216 2024-09-10T16:19:42.950 INFO:tasks.rados.rados.0.smithi165.stdout:1968: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-10T16:19:42.956 INFO:tasks.rados.rados.0.smithi165.stdout:1968: writing smithi16534841-43 from 632128 to 1111900 tid 1 2024-09-10T16:19:42.963 INFO:tasks.rados.rados.0.smithi165.stdout:1968: writing smithi16534841-43 from 1746735 to 2347633 tid 2 2024-09-10T16:19:42.971 INFO:tasks.rados.rados.0.smithi165.stdout:1968: writing smithi16534841-43 from 3066728 to 3720578 tid 3 2024-09-10T16:19:42.971 INFO:tasks.rados.rados.0.smithi165.stdout:1969: write oid 36 current snap is 216 2024-09-10T16:19:42.971 INFO:tasks.rados.rados.0.smithi165.stdout:1969: seq_num 705 ranges {648935=555021,1805217=460909,2694151=465122,3817442=31493} 2024-09-10T16:19:42.977 INFO:tasks.rados.rados.0.smithi165.stdout:1969: writing smithi16534841-36 from 648935 to 1203956 tid 1 2024-09-10T16:19:42.983 INFO:tasks.rados.rados.0.smithi165.stdout:1969: writing smithi16534841-36 from 1805217 to 2266126 tid 2 2024-09-10T16:19:42.988 INFO:tasks.rados.rados.0.smithi165.stdout:1969: writing smithi16534841-36 from 2694151 to 3159273 tid 3 2024-09-10T16:19:42.991 INFO:tasks.rados.rados.0.smithi165.stdout:1969: writing smithi16534841-36 from 3817442 to 3848935 tid 4 2024-09-10T16:19:42.991 INFO:tasks.rados.rados.0.smithi165.stdout:1970: snap_remove snap 213 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1970: done (4 left) 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1971: delete oid 38 current snap is 216 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1966: finishing write tid 1 to smithi16534841-4 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1966: finishing write tid 2 to smithi16534841-4 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1966: finishing write tid 3 to smithi16534841-4 2024-09-10T16:19:43.576 INFO:tasks.rados.rados.0.smithi165.stdout:1966: finishing write tid 4 to smithi16534841-4 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: finishing write tid 5 to smithi16534841-4 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: oid 4 updating version 0 to 724 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: oid 4 updating version 724 to 727 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: oid 4 version 727 is already newer than 726 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: oid 4 version 727 is already newer than 725 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 727 (ObjNum 702 snap 216 seq_num 702) dirty exists 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1966: left oid 4 (ObjNum 702 snap 216 seq_num 702) 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: finishing write tid 1 to smithi16534841-23 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: finishing write tid 2 to smithi16534841-23 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: finishing write tid 3 to smithi16534841-23 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: finishing write tid 4 to smithi16534841-23 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: finishing write tid 5 to smithi16534841-23 2024-09-10T16:19:43.577 INFO:tasks.rados.rados.0.smithi165.stdout:1967: oid 23 updating version 0 to 1148 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1967: oid 23 updating version 1148 to 1150 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1967: oid 23 version 1150 is already newer than 1149 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1967: oid 23 updating version 1150 to 1151 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1151 (ObjNum 703 snap 216 seq_num 703) dirty exists 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1967: left oid 23 (ObjNum 703 snap 216 seq_num 703) 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: finishing write tid 1 to smithi16534841-43 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: finishing write tid 2 to smithi16534841-43 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: finishing write tid 3 to smithi16534841-43 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: finishing write tid 4 to smithi16534841-43 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: finishing write tid 5 to smithi16534841-43 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: oid 43 updating version 0 to 900 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: oid 43 version 900 is already newer than 899 2024-09-10T16:19:43.578 INFO:tasks.rados.rados.0.smithi165.stdout:1968: oid 43 updating version 900 to 902 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1968: oid 43 version 902 is already newer than 901 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 902 (ObjNum 704 snap 216 seq_num 704) dirty exists 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1968: left oid 43 (ObjNum 704 snap 216 seq_num 704) 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 1 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 2 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 3 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 4 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 5 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: finishing write tid 6 to smithi16534841-36 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: oid 36 updating version 0 to 1146 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: oid 36 version 1146 is already newer than 1143 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: oid 36 version 1146 is already newer than 1144 2024-09-10T16:19:43.579 INFO:tasks.rados.rados.0.smithi165.stdout:1969: oid 36 version 1146 is already newer than 1145 2024-09-10T16:19:43.580 INFO:tasks.rados.rados.0.smithi165.stdout:1969: oid 36 updating version 1146 to 1147 2024-09-10T16:19:43.580 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1147 (ObjNum 705 snap 216 seq_num 705) dirty exists 2024-09-10T16:19:43.580 INFO:tasks.rados.rados.0.smithi165.stdout:1969: left oid 36 (ObjNum 705 snap 216 seq_num 705) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1966: done (4 left) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1967: done (3 left) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1968: done (2 left) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1969: done (1 left) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1971: done (0 left) 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1972: write (excl) oid 16 current snap is 216 2024-09-10T16:19:43.581 INFO:tasks.rados.rados.0.smithi165.stdout:1972: seq_num 707 ranges {682549=705519,1922181=464578,3148997=487666} 2024-09-10T16:19:43.589 INFO:tasks.rados.rados.0.smithi165.stdout:1972: writing smithi16534841-16 from 682549 to 1388068 tid 1 2024-09-10T16:19:43.595 INFO:tasks.rados.rados.0.smithi165.stdout:1972: writing smithi16534841-16 from 1922181 to 2386759 tid 2 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1972: writing smithi16534841-16 from 3148997 to 3636663 tid 3 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1973: copy_from oid 40 from oid 49 current snap is 216 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1972: finishing write tid 1 to smithi16534841-16 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1974: copy_from oid 29 from oid 19 current snap is 216 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1975: copy_from oid 8 from oid 36 current snap is 216 2024-09-10T16:19:43.602 INFO:tasks.rados.rados.0.smithi165.stdout:1976: copy_from oid 25 from oid 12 current snap is 216 2024-09-10T16:19:43.603 INFO:tasks.rados.rados.0.smithi165.stdout:1977: snap_remove snap 215 2024-09-10T16:19:44.747 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:44.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.856+0000 7fec3c265640 10 monclient: get_monmap_and_config 2024-09-10T16:19:44.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.856+0000 7fec3c265640 10 monclient: build_initial_monmap 2024-09-10T16:19:44.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.856+0000 7fec3c265640 10 monclient: monmap: 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:44.857065+0000 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:44.857065+0000 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:44.858 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:44.859 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:44.859 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.856+0000 7fec3c265640 1 Processor -- start 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 -- start start 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient: init 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:19:44.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-a con 0x7fec3406cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-b con 0x7fec340a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fec34137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3409ce50 con 0x7fec34137ca0 2024-09-10T16:19:44.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fec3406db60 con 0x7fec3406cb70 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec3406cf50 con 0x7fec340a46a0 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.857+0000 7fec3c265640 10 monclient(hunting): authenticate will time out at 1563.344945s 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56094/0 (socket says 172.21.15.164:56094) 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 learned_addr learned my addr 172.21.15.164:0/3671719443 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec340a46a0 auth_method 0 2024-09-10T16:19:44.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec3406cb70 auth_method 0 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:44.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:44.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:44.860+0000 7fec3a802640 1 -- 172.21.15.164:0/3671719443 >> v1:172.21.15.165:6789/0 conn(0x7fec34137ca0 legacy=0x7fec34138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:45.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec340a46a0 auth_method 0 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec3406cb70 auth_method 0 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:45.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.059+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:45.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.060+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:45.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:45.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:45.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec340a46a0 auth_method 0 2024-09-10T16:19:45.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec3406cb70 auth_method 0 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:45.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:45.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:45.461+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec3406cb70 auth_method 0 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec340a46a0 auth_method 0 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:46.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:46.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:46.262+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): tick 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 msgr2=0x7fec340a4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec340a46a0 0x7fec340a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:47.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 msgr2=0x7fec34134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.164:3300/0 conn(0x7fec3406cb70 0x7fec34134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 -- 172.21.15.164:0/3671719443 >> v1:172.21.15.165:6789/0 conn(0x7fec34137ca0 legacy=0x7fec34138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient: _add_conns ranks=[4,0] 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): picked mon.noname-c con 0x7fec20003720 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec20006290 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 -- 172.21.15.164:0/3671719443 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec3409cb70 con 0x7fec20006290 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 1 -- 172.21.15.164:0/3671719443 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fec3406cf50 con 0x7fec20003720 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec337fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec20003720 auth_method 0 2024-09-10T16:19:47.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.858+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:47.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:47.860+0000 7fec39800640 1 -- 172.21.15.164:0/3671719443 >> v1:172.21.15.164:6789/0 conn(0x7fec20006290 legacy=0x7fec20008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec20003720 auth_method 0 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:48.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.059+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec20003720 auth_method 0 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:48.460+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:49.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec20003720 auth_method 0 2024-09-10T16:19:49.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.261+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.262+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3154921756 0 0) 0x7fec3409cb70 con 0x7fec20006290 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 10 monclient(hunting): _negotiate 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 10 monclient(hunting): my global_id is 144195 2024-09-10T16:19:49.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.499+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec1c003520 con 0x7fec20006290 2024-09-10T16:19:49.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3091014439 0 0) 0x7fec1c003520 con 0x7fec20006290 2024-09-10T16:19:49.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 msgr2=0x7fec20005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:49.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 --2- 172.21.15.164:0/3671719443 >> v2:172.21.15.165:3300/0 conn(0x7fec20003720 0x7fec20005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:49.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec2000b030 con 0x7fec20006290 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _finish_auth 0 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fec28009a80 con 0x7fec20006290 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3c265640 5 monclient: authenticate success, global_id 144195 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3c265640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: dump: 2024-09-10T16:19:49.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 >> v1:172.21.15.164:6789/0 conn(0x7fec20006290 legacy=0x7fec20008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a802640 1 -- 172.21.15.164:0/3671719443 reap_dead start 2024-09-10T16:19:49.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 0x7fec1c003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): picked mon.a con 0x7fec1c003b10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): picked mon.b con 0x7fec1c004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c0099b0 0x7fec1c00bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): picked mon.c con 0x7fec1c0099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fec3406cf50 con 0x7fec1c003b10 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec1c00c390 con 0x7fec1c004800 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec1c00c590 con 0x7fec1c0099b0 2024-09-10T16:19:49.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec38fff640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3c265640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 0x7fec1c003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a802640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c0099b0 0x7fec1c00bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec1c003b10 auth_method 0 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.513+0000 7fec3a001640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): get_auth_request con 0x7fec1c0099b0 auth_method 0 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec1c004800 auth_method 0 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient(hunting): handle_auth_done global_id 144195 payload 293 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c0099b0 msgr2=0x7fec1c00bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c0099b0 0x7fec1c00bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 msgr2=0x7fec1c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient: _finish_hunting 0 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 monclient: found mon.a 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1c003520 con 0x7fec1c003b10 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient: _finish_auth 0 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 10 monclient: _check_auth_tickets 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec3a001640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:49.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.514+0000 7fec39800640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:49.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec3a001640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 msgr2=0x7fec1c003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec3a001640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 0x7fec1c003f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec24019cd0 tx=0x7fec1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e74956d562a8aa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2403a070 con 0x7fec1c003b10 2024-09-10T16:19:49.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: dump: 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec3c265640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:49.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec24035410 con 0x7fec1c003b10 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 1 -- 172.21.15.164:0/3671719443 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec24019950 con 0x7fec1c003b10 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec3c265640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec3c265640 4 set_mon_vals no callback set 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.517+0000 7fec38fff640 10 monclient: dump: 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 10 monclient: shutdown 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 -- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 msgr2=0x7fec1c003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 0x7fec1c003f10 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec24019cd0 tx=0x7fec1c005830 comp rx=0 tx=0).stop 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 -- 172.21.15.164:0/3671719443 shutdown_connections 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c0099b0 0x7fec1c00bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c004800 0x7fec1c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 --2- 172.21.15.164:0/3671719443 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c003b10 0x7fec1c003f10 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 -- 172.21.15.164:0/3671719443 >> 172.21.15.164:0/3671719443 conn(0x7fec340aa900 msgr2=0x7fec340aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 -- 172.21.15.164:0/3671719443 shutdown_connections 2024-09-10T16:19:49.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.518+0000 7fec3c265640 1 -- 172.21.15.164:0/3671719443 wait complete. 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.519+0000 7fec3c265640 10 monclient: build_initial_monmap 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.519+0000 7fec3c265640 10 monclient: monmap: 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 Processor -- start 2024-09-10T16:19:49.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 -- start start 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient: init 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-b con 0x7fec1c003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-c con 0x7fec1c004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): picked mon.noname-a con 0x7fec1c0099b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fec340a4c30 con 0x7fec1c0099b0 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec340a4ab0 con 0x7fec1c003b10 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec340a4db0 con 0x7fec1c004800 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3c265640 10 monclient(hunting): authenticate will time out at 1568.007745s 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56128/0 (socket says 172.21.15.164:56128) 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:45338/0 (socket says 172.21.15.164:45338) 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58008/0 (socket says 172.21.15.164:58008) 2024-09-10T16:19:49.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 1 -- 172.21.15.164:0/3635561755 learned_addr learned my addr 172.21.15.164:0/3635561755 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 10 monclient(hunting): get_auth_request con 0x7fec1c004800 auth_method 0 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec39800640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 10 monclient(hunting): get_auth_request con 0x7fec1c003b10 auth_method 0 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a001640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 10 monclient(hunting): get_auth_request con 0x7fec1c0099b0 auth_method 0 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.520+0000 7fec3a802640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec39800640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec39800640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec39800640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a001640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 msgr2=0x7fec3412e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a001640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 msgr2=0x7fec3412e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a001640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a001640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient(hunting): handle_auth_done global_id 144198 payload 824 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 msgr2=0x7fec3412e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 msgr2=0x7fec3412e400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:49.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient: _finish_hunting 0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 monclient: found mon.noname-a 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec3a802640 1 -- 172.21.15.164:0/3635561755 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec340a3440 con 0x7fec1c0099b0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.521+0000 7fec39800640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3a802640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 msgr2=0x7fec341ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3a802640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec34077410 tx=0x7fec2c00f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c018070 con 0x7fec1c0099b0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: dump: 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.538 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 5 monclient: authenticate success, global_id 144198 2024-09-10T16:19:49.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec2c0074f0 con 0x7fec1c0099b0 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec2c0059e0 con 0x7fec1c0099b0 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 10 monclient: _renew_subs 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec341340e0 con 0x7fec1c0099b0 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec327fc640 10 monclient: dump: 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.540 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec33fff640 4 set_mon_vals no callback set 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 10 monclient: _renew_subs 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.523+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec340a10e0 con 0x7fec1c0099b0 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.524+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fec2c01b070 con 0x7fec1c0099b0 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3c265640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3c265640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec327fc640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 0x7fec200410a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec34134f70 con 0x7fec1c0099b0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(462..462 src has 184..462) ==== 4795+0+0 (secure 0 0 0) 0x7fec2c051ca0 con 0x7fec1c0099b0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3a001640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 0x7fec200410a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.525+0000 7fec3a001640 10 monclient: get_auth_request con 0x7fec2003ebe0 auth_method 0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.527+0000 7fec3a001640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 0x7fec200410a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec24019cd0 tx=0x7fec24030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.529+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fec3409ce50 con 0x7fec1c0099b0 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.530+0000 7fec327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:49.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.530+0000 7fec327fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:49.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.778+0000 7fec3c265640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:49.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.778+0000 7fec3c265640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:49.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.778+0000 7fec3c265640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:49.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.778+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec340a1c40 con 0x7fec1c0099b0 2024-09-10T16:19:49.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.779+0000 7fec327fc640 1 -- 172.21.15.164:0/3635561755 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fec2c01a2c0 con 0x7fec1c0099b0 2024-09-10T16:19:49.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.779+0000 7fec327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:49.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.779+0000 7fec327fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:49.782 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":134,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 msgr2=0x7fec200410a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 0x7fec200410a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fec24019cd0 tx=0x7fec24030000 comp rx=0 tx=0).stop 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 10 monclient: shutdown 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 msgr2=0x7fec341ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec34077410 tx=0x7fec2c00f890 comp rx=0 tx=0).stop 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 shutdown_connections 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec2003ebe0 0x7fec200410a0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec1c0099b0 0x7fec341ccea0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec1c004800 0x7fec3412e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 --2- 172.21.15.164:0/3635561755 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec1c003b10 0x7fec3412e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 >> 172.21.15.164:0/3635561755 conn(0x7fec340aa900 msgr2=0x7fec340a5380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.781+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 shutdown_connections 2024-09-10T16:19:49.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.782+0000 7fec3c265640 1 -- 172.21.15.164:0/3635561755 wait complete. 2024-09-10T16:19:49.795 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:19:49.795 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:19:49.795 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.892+0000 7fb82bf3d640 10 monclient: get_monmap_and_config 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.892+0000 7fb82bf3d640 10 monclient: build_initial_monmap 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.892+0000 7fb82bf3d640 10 monclient: monmap: 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:49.894524+0000 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:49.894524+0000 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:49.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 Processor -- start 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 -- start start 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient: init 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-b con 0x7fb82409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb82409d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 0x7fb82413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-c con 0x7fb82413a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb82409a430 con 0x7fb82409d160 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb8240a24e0 con 0x7fb82413a5c0 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb8240a2bf0 con 0x7fb82409c180 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.893+0000 7fb82bf3d640 10 monclient(hunting): authenticate will time out at 1568.381905s 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56138/0 (socket says 172.21.15.164:56138) 2024-09-10T16:19:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 -- 172.21.15.164:0/4159371193 learned_addr learned my addr 172.21.15.164:0/4159371193 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): get_auth_request con 0x7fb82409c180 auth_method 0 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb82a4da640 1 --2- 172.21.15.164:0/4159371193 >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 0x7fb82413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient(hunting): handle_auth_done global_id 114449 payload 274 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 -- 172.21.15.164:0/4159371193 >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 msgr2=0x7fb82413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 --2- 172.21.15.164:0/4159371193 >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 0x7fb82413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 -- 172.21.15.164:0/4159371193 >> v1:172.21.15.165:6789/0 conn(0x7fb82409d160 legacy=0x7fb82409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient: _finish_hunting 0 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 monclient: found mon.noname-b 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb829cd9640 1 -- 172.21.15.164:0/4159371193 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb82409c560 con 0x7fb82409c180 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb82a4da640 10 monclient: get_auth_request con 0x7fb82413a5c0 auth_method 0 2024-09-10T16:19:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.894+0000 7fb82a4da640 1 --2- 172.21.15.164:0/4159371193 >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 0x7fb82413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb829cd9640 1 -- 172.21.15.164:0/4159371193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 msgr2=0x7fb82409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb829cd9640 1 --2- 172.21.15.164:0/4159371193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb82409abb0 tx=0x7fb82002a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a638fea6b6cb8752 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 1 -- 172.21.15.164:0/4159371193 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820037070 con 0x7fb82409c180 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: dump: 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: _finish_auth 0 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: _check_auth_tickets 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 5 monclient: authenticate success, global_id 114449 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 1 -- 172.21.15.164:0/4159371193 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb820032640 con 0x7fb82409c180 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 4 set_mon_vals no callback set 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 1 -- 172.21.15.164:0/4159371193 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820032a10 con 0x7fb82409c180 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb828cd7640 10 monclient: dump: 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 10 monclient: shutdown 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 -- 172.21.15.164:0/4159371193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 msgr2=0x7fb82409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/4159371193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb82409abb0 tx=0x7fb82002a080 comp rx=0 tx=0).stop 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 -- 172.21.15.164:0/4159371193 shutdown_connections 2024-09-10T16:19:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/4159371193 >> v2:172.21.15.165:3300/0 conn(0x7fb82413a5c0 0x7fb82413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/4159371193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb82409ea40 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 -- 172.21.15.164:0/4159371193 >> 172.21.15.164:0/4159371193 conn(0x7fb8240a3110 msgr2=0x7fb8240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 -- 172.21.15.164:0/4159371193 shutdown_connections 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.897+0000 7fb82bf3d640 1 -- 172.21.15.164:0/4159371193 wait complete. 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.898+0000 7fb82bf3d640 10 monclient: build_initial_monmap 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.898+0000 7fb82bf3d640 10 monclient: monmap: 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:49.912 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.898+0000 7fb82bf3d640 1 Processor -- start 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.898+0000 7fb82bf3d640 1 -- start start 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.898+0000 7fb82bf3d640 10 monclient: init 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb824093870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-c con 0x7fb82409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb82409d160 0x7fb824093db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-a con 0x7fb82409d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 0x7fb8240942f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): picked mon.noname-b con 0x7fb82413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb824074f90 con 0x7fb82409d160 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb824074e10 con 0x7fb82413a5c0 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb82409f690 con 0x7fb82409c180 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82bf3d640 10 monclient(hunting): authenticate will time out at 1568.387257s 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb824093870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb824093870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56154/0 (socket says 172.21.15.164:56154) 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 -- 172.21.15.164:0/271157356 learned_addr learned my addr 172.21.15.164:0/271157356 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): get_auth_request con 0x7fb82409c180 auth_method 0 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb82409d160 0x7fb824093db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): get_auth_request con 0x7fb82409d160 auth_method 0 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 0x7fb8240942f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 10 monclient(hunting): get_auth_request con 0x7fb82413a5c0 auth_method 0 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 msgr2=0x7fb8240942f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 msgr2=0x7fb8240942f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 0x7fb8240942f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient(hunting): handle_auth_done global_id 114452 payload 824 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb82a4da640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 0x7fb8240942f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 msgr2=0x7fb8240942f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb82413a5c0 0x7fb8240942f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb82409d160 msgr2=0x7fb824093db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb82409d160 0x7fb824093db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient: _finish_hunting 0 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 monclient: found mon.noname-c 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.899+0000 7fb829cd9640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb824097540 con 0x7fb82409c180 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.901+0000 7fb829cd9640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 msgr2=0x7fb824093870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.901+0000 7fb829cd9640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb824093870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8200090b0 tx=0x7fb8200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.901+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820041020 con 0x7fb82409c180 2024-09-10T16:19:49.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:49.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: dump: 2024-09-10T16:19:49.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 5 monclient: authenticate success, global_id 114452 2024-09-10T16:19:49.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb82003d910 con 0x7fb82409c180 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb820037040 con 0x7fb82409c180 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 monclient: _renew_subs 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb82409f070 con 0x7fb82409c180 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81b7fe640 4 set_mon_vals no callback set 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 10 monclient: dump: 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.920 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 monclient: _renew_subs 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb824097260 con 0x7fb82409c180 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 0x7fb82402e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb82409d670 con 0x7fb824099c40 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.902+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb820044020 con 0x7fb82409c180 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 0x7fb82402e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient: get_auth_request con 0x7fb824099c40 auth_method 0 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 10 monclient(hunting): handle_auth_done global_id 144204 payload 824 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb81a7fc640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb80803e260 0x7fb808040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb8294d8640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb82409d930 con 0x7fb824099c40 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(462..462 src has 184..462) ==== 4795+0+0 (secure 0 0 0) 0x7fb820073330 con 0x7fb82409c180 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb82a4da640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb80803e260 0x7fb808040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.903+0000 7fb82a4da640 10 monclient: get_auth_request con 0x7fb80803e260 auth_method 0 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb8294d8640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 msgr2=0x7fb82402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb8294d8640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 0x7fb82402e5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb82409ecc0 tx=0x7fb81400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb814018030 con 0x7fb824099c40 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb81a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb81a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb81a7fc640 10 monclient: dump: 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.923 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.905+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb81400d9b0 con 0x7fb824099c40 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb82a4da640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb80803e260 0x7fb808040720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb8240961c0 tx=0x7fb81c0079a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 msgr2=0x7fb82402e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 0x7fb82402e5c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb82409ecc0 tx=0x7fb81400fc00 comp rx=0 tx=0).stop 2024-09-10T16:19:49.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 msgr2=0x7fb82402e5c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:49.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.906+0000 7fb81a7fc640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824099c40 0x7fb82402e5c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb82bf3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 0x7fb824063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb82bf3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb824094c40 con 0x7fb824094830 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 0x7fb824063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient: get_auth_request con 0x7fb824094830 auth_method 0 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:49.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 10 monclient(hunting): handle_auth_done global_id 144207 payload 824 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.912+0000 7fb8294d8640 1 -- 172.21.15.164:0/271157356 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb824072f00 con 0x7fb824094830 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb8294d8640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 msgr2=0x7fb824063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb8294d8640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 0x7fb824063d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb814018000 tx=0x7fb814003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb81401dd20 con 0x7fb824094830 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 10 monclient: dump: 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:49.928 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fb814007020 con 0x7fb824094830 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 msgr2=0x7fb824063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 0x7fb824063d50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb814018000 tx=0x7fb814003760 comp rx=0 tx=0).stop 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 msgr2=0x7fb824063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:49.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.914+0000 7fb81a7fc640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb824094830 0x7fb824063d50 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb814018000 tx=0x7fb814003760 comp rx=0 tx=0).stop 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 61357, 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:49.930 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:49.931 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:49.932 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:49.933 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:49.934 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:49.935 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:49.936 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.937 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.938 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:49.939 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:49.940 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:49.941 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:49.942 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb80803e260 msgr2=0x7fb808040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb80803e260 0x7fb808040720 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb8240961c0 tx=0x7fb81c0079a0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 10 monclient: shutdown 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 msgr2=0x7fb824093870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 --2- 172.21.15.164:0/271157356 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb82409c180 0x7fb824093870 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb8200090b0 tx=0x7fb8200046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82a4da640 1 -- 172.21.15.164:0/271157356 reap_dead start 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82a4da640 1 -- 172.21.15.164:0/271157356 reap_dead start 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 shutdown_connections 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 >> 172.21.15.164:0/271157356 conn(0x7fb8240a3110 msgr2=0x7fb8241399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:49.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 shutdown_connections 2024-09-10T16:19:49.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:49.915+0000 7fb82bf3d640 1 -- 172.21.15.164:0/271157356 wait complete. 2024-09-10T16:19:49.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:49.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:19:49.945 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.041+0000 7fdec3b3d640 10 monclient: get_monmap_and_config 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.041+0000 7fdec3b3d640 10 monclient: build_initial_monmap 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.041+0000 7fdec3b3d640 10 monclient: monmap: 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:50.043627+0000 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:50.043627+0000 2024-09-10T16:19:50.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:50.045 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 Processor -- start 2024-09-10T16:19:50.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 -- start start 2024-09-10T16:19:50.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient: init 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdebc09ab50 0x7fdebc0a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdebc09d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:50.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdebc09e420 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:50.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc0a0870 con 0x7fdebc09d930 2024-09-10T16:19:50.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdebc074c70 con 0x7fdebc09e420 2024-09-10T16:19:50.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdebc09d3e0 con 0x7fdebc09ab50 2024-09-10T16:19:50.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.042+0000 7fdec3b3d640 10 monclient(hunting): authenticate will time out at 1568.530980s 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fdebc09ab50 0x7fdebc0a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec10d8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fdebc09d930 0x7fdebc09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35554/0 (socket says 172.21.15.164:35554) 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec10d8640 1 -- 172.21.15.164:0/1117137430 learned_addr learned my addr 172.21.15.164:0/1117137430 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): get_auth_request con 0x7fdebc09ab50 auth_method 0 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient(hunting): handle_auth_done global_id 144210 payload 274 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 1 -- 172.21.15.164:0/1117137430 >> v1:172.21.15.165:6789/0 conn(0x7fdebc09e420 legacy=0x7fdebc13c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 1 -- 172.21.15.164:0/1117137430 >> v1:172.21.15.164:6789/0 conn(0x7fdebc09d930 legacy=0x7fdebc09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient: _finish_hunting 0 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 1 monclient: found mon.noname-a 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.043+0000 7fdec18d9640 1 -- 172.21.15.164:0/1117137430 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc09ecb0 con 0x7fdebc09ab50 2024-09-10T16:19:50.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec18d9640 1 -- 172.21.15.164:0/1117137430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09ab50 msgr2=0x7fdebc0a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec18d9640 1 --2- 172.21.15.164:0/1117137430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09ab50 0x7fdebc0a1e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80097a0 tx=0x7fdeb8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fe642450c7b2827 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 1 -- 172.21.15.164:0/1117137430 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8037070 con 0x7fdebc09ab50 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: dump: 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.066 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: _finish_auth 0 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: _check_auth_tickets 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 1 -- 172.21.15.164:0/1117137430 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb8032640 con 0x7fdebc09ab50 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 5 monclient: authenticate success, global_id 144210 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 1 -- 172.21.15.164:0/1117137430 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8032a10 con 0x7fdebc09ab50 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 4 set_mon_vals no callback set 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:50.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec08d7640 10 monclient: dump: 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 10 monclient: shutdown 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 1 -- 172.21.15.164:0/1117137430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09ab50 msgr2=0x7fdebc0a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/1117137430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09ab50 0x7fdebc0a1e40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdeb80097a0 tx=0x7fdeb8029fd0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 1 -- 172.21.15.164:0/1117137430 shutdown_connections 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/1117137430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09ab50 0x7fdebc0a1e40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.046+0000 7fdec3b3d640 1 -- 172.21.15.164:0/1117137430 >> 172.21.15.164:0/1117137430 conn(0x7fdebc0a3110 msgr2=0x7fdebc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 1 -- 172.21.15.164:0/1117137430 shutdown_connections 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 1 -- 172.21.15.164:0/1117137430 wait complete. 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 10 monclient: build_initial_monmap 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 10 monclient: monmap: 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:50.069 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 1 Processor -- start 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 1 -- start start 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.047+0000 7fdec3b3d640 10 monclient: init 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdebc09ab50 0x7fdebc1357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fdebc09ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 0x7fdebc135ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fdebc09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 0x7fdebc136220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fdebc09e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc0a2940 con 0x7fdebc09e420 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdebc09ef30 con 0x7fdebc09ab50 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc0a2ac0 con 0x7fdebc09d930 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec3b3d640 10 monclient(hunting): authenticate will time out at 1568.536370s 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 0x7fdebc135ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 0x7fdebc136220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 0x7fdebc135ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56156/0 (socket says 172.21.15.164:56156) 2024-09-10T16:19:50.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 1 -- 172.21.15.164:0/424448961 learned_addr learned my addr 172.21.15.164:0/424448961 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 0x7fdebc136220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58054/0 (socket says 172.21.15.164:58054) 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): get_auth_request con 0x7fdebc09d930 auth_method 0 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): get_auth_request con 0x7fdebc09e420 auth_method 0 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec18d9640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdebc09ab50 0x7fdebc1357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec18d9640 10 monclient(hunting): get_auth_request con 0x7fdebc09ab50 auth_method 0 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec18d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec18d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec18d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient(hunting): handle_auth_done global_id 144213 payload 824 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 msgr2=0x7fdebc135ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 0x7fdebc135ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdebc09ab50 msgr2=0x7fdebc1357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdebc09ab50 0x7fdebc1357a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient: _finish_hunting 0 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 monclient: found mon.noname-a 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdebc12cf80 con 0x7fdebc09e420 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.048+0000 7fdec10d8640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc09d930 0x7fdebc135ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 msgr2=0x7fdebc136220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec20da640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 0x7fdebc136220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400c780 tx=0x7fdeb400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4016020 con 0x7fdebc09e420 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:50.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: dump: 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:50.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdeb400ac60 con 0x7fdebc09e420 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 5 monclient: authenticate success, global_id 144213 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 monclient: _renew_subs 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdebc12d260 con 0x7fdebc09e420 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb4012660 con 0x7fdebc09e420 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb27fc640 10 monclient: dump: 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.076 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 monclient: _renew_subs 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdeb3fff640 4 set_mon_vals no callback set 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdebc12b630 con 0x7fdebc09e420 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 0x7fdebc1bc820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.051+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc12bd40 con 0x7fdebc1bc440 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 0x7fdebc1bc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient: get_auth_request con 0x7fdebc1bc440 auth_method 0 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdeb4019020 con 0x7fdebc09e420 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 10 monclient(hunting): handle_auth_done global_id 114458 payload 824 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec18d9640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdebc0a45c0 con 0x7fdebc1bc440 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdeb27fc640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdea003e260 0x7fdea0040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(462..462 src has 184..462) ==== 4795+0+0 (secure 0 0 0) 0x7fdeb4052bc0 con 0x7fdebc09e420 2024-09-10T16:19:50.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec10d8640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdea003e260 0x7fdea0040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.052+0000 7fdec10d8640 10 monclient: get_auth_request con 0x7fdea003e260 auth_method 0 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdec18d9640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 msgr2=0x7fdebc1bc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdec18d9640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 0x7fdebc1bc820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8034000 tx=0x7fdeb80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8037040 con 0x7fdebc1bc440 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 10 monclient: dump: 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.079 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdeb803c850 con 0x7fdebc1bc440 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 msgr2=0x7fdebc1bc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 0x7fdebc1bc820 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8034000 tx=0x7fdeb80046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 msgr2=0x7fdebc1bc820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.054+0000 7fdeb27fc640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bc440 0x7fdebc1bc820 secure :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8034000 tx=0x7fdeb80046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.055+0000 7fdec10d8640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdea003e260 0x7fdea0040720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdeac007d50 tx=0x7fdeac007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.060+0000 7fdec3b3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.060+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 0x7fdebc1bd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.060+0000 7fdec3b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.060+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdebc12c0a0 con 0x7fdebc1bcd60 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 0x7fdebc1bd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient: get_auth_request con 0x7fdebc1bcd60 auth_method 0 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 10 monclient(hunting): handle_auth_done global_id 114461 payload 824 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.061+0000 7fdec18d9640 1 -- 172.21.15.164:0/424448961 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdebc0a3940 con 0x7fdebc1bcd60 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdec18d9640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 msgr2=0x7fdebc1bd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdec18d9640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 0x7fdebc1bd170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c00 tx=0x7fdeb8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdeb8037040 con 0x7fdebc1bcd60 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 10 monclient: dump: 2024-09-10T16:19:50.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fdeb8049c60 con 0x7fdebc1bcd60 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 msgr2=0x7fdebc1bd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 0x7fdebc1bd170 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c00 tx=0x7fdeb8008030 comp rx=0 tx=0).stop 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 msgr2=0x7fdebc1bd170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.063+0000 7fdeb27fc640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdebc1bcd60 0x7fdebc1bd170 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdeb8000c00 tx=0x7fdeb8008030 comp rx=0 tx=0).stop 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 210632, 2024-09-10T16:19:50.085 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:50.086 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:50.087 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:50.088 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:50.089 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:50.090 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.091 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.092 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:50.093 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:19:50.096 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:50.097 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdea003e260 msgr2=0x7fdea0040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdea003e260 0x7fdea0040720 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdeac007d50 tx=0x7fdeac007ca0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 10 monclient: shutdown 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 msgr2=0x7fdebc136220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 --2- 172.21.15.164:0/424448961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdebc09e420 0x7fdebc136220 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdeb400c780 tx=0x7fdeb400cc40 comp rx=0 tx=0).stop 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 reap_dead start 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec20da640 1 -- 172.21.15.164:0/424448961 reap_dead start 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 shutdown_connections 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 >> 172.21.15.164:0/424448961 conn(0x7fdebc0a3110 msgr2=0x7fdebc09be90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 shutdown_connections 2024-09-10T16:19:50.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.064+0000 7fdec3b3d640 1 -- 172.21.15.164:0/424448961 wait complete. 2024-09-10T16:19:50.100 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:50.100 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:19:50.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:19:50.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.197+0000 7f9a8a08c640 10 monclient: get_monmap_and_config 2024-09-10T16:19:50.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.197+0000 7f9a8a08c640 10 monclient: build_initial_monmap 2024-09-10T16:19:50.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.197+0000 7f9a8a08c640 10 monclient: monmap: 2024-09-10T16:19:50.200 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:50.200 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:50.199444+0000 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:50.199444+0000 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:50.201 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:50.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 Processor -- start 2024-09-10T16:19:50.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 -- start start 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient: init 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 0x7f9a8409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8409d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9a84134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a840a0870 con 0x7f9a84134580 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9a8409d3e0 con 0x7f9a8409ab50 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9a8409e790 con 0x7f9a8409d930 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a8a08c640 10 monclient(hunting): authenticate will time out at 1568.686880s 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a82ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.198+0000 7f9a837fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 0x7f9a8409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58066/0 (socket says 172.21.15.164:58066) 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 1 -- 172.21.15.164:0/828375836 learned_addr learned my addr 172.21.15.164:0/828375836 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request con 0x7f9a8409ab50 auth_method 0 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): get_auth_request con 0x7f9a8409d930 auth_method 0 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient(hunting): handle_auth_done global_id 114464 payload 274 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 1 -- 172.21.15.164:0/828375836 >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 msgr2=0x7f9a8409e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 1 --2- 172.21.15.164:0/828375836 >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 1 -- 172.21.15.164:0/828375836 >> v1:172.21.15.165:6789/0 conn(0x7f9a84134580 legacy=0x7f9a84134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 1 monclient: found mon.noname-b 2024-09-10T16:19:50.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a837fe640 1 -- 172.21.15.164:0/828375836 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a84074c70 con 0x7f9a8409d930 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.199+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/828375836 >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a837fe640 1 -- 172.21.15.164:0/828375836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 msgr2=0x7f9a8409dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a837fe640 1 --2- 172.21.15.164:0/828375836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 0x7f9a8409dd10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69ef44a40f4c3d5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 1 -- 172.21.15.164:0/828375836 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74037070 con 0x7f9a8409d930 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: dump: 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 1 -- 172.21.15.164:0/828375836 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a74032440 con 0x7f9a8409d930 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 5 monclient: authenticate success, global_id 114464 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:50.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 1 -- 172.21.15.164:0/828375836 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74032810 con 0x7f9a8409d930 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 4 set_mon_vals no callback set 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a827fc640 10 monclient: dump: 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.219 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.219 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.219 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.219 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 10 monclient: shutdown 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 -- 172.21.15.164:0/828375836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 msgr2=0x7f9a8409dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/828375836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 0x7f9a8409dd10 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f9a740097a0 tx=0x7f9a7402a050 comp rx=0 tx=0).stop 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 -- 172.21.15.164:0/828375836 shutdown_connections 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/828375836 >> v2:172.21.15.164:3300/0 conn(0x7f9a8409ab50 0x7f9a8409e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/828375836 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a8409d930 0x7f9a8409dd10 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 -- 172.21.15.164:0/828375836 >> 172.21.15.164:0/828375836 conn(0x7f9a840a3110 msgr2=0x7f9a840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 -- 172.21.15.164:0/828375836 shutdown_connections 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.202+0000 7f9a8a08c640 1 -- 172.21.15.164:0/828375836 wait complete. 2024-09-10T16:19:50.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.203+0000 7f9a8a08c640 10 monclient: build_initial_monmap 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.203+0000 7f9a8a08c640 10 monclient: monmap: 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.203+0000 7f9a8a08c640 1 Processor -- start 2024-09-10T16:19:50.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.203+0000 7f9a8a08c640 1 -- start start 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.203+0000 7f9a8a08c640 10 monclient: init 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 0x7f9a84096910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-a con 0x7f9a8409ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 0x7f9a84096e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-b con 0x7f9a8409d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a84134580 0x7f9a84097390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): picked mon.noname-c con 0x7f9a84134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9a8409ef30 con 0x7f9a8409ab50 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9a840a02b0 con 0x7f9a8409d930 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9a840a2940 con 0x7f9a84134580 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a8a08c640 10 monclient(hunting): authenticate will time out at 1568.692451s 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 0x7f9a84096910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 0x7f9a84096910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58082/0 (socket says 172.21.15.164:58082) 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 -- 172.21.15.164:0/2036135253 learned_addr learned my addr 172.21.15.164:0/2036135253 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): get_auth_request con 0x7f9a8409ab50 auth_method 0 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a84134580 0x7f9a84097390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): get_auth_request con 0x7f9a84134580 auth_method 0 2024-09-10T16:19:50.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 0x7f9a84096e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 msgr2=0x7f9a84096e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 msgr2=0x7f9a84096e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 0x7f9a84096e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient(hunting): handle_auth_done global_id 144219 payload 824 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 0x7f9a84096e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a84134580 msgr2=0x7f9a84097390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a84134580 0x7f9a84097390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 msgr2=0x7f9a84096e50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409d930 0x7f9a84096e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 monclient: found mon.noname-a 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:50.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a837fe640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a84099210 con 0x7f9a8409ab50 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.204+0000 7f9a83fff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9a84134580 0x7f9a84097390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a837fe640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 msgr2=0x7f9a84096910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a837fe640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 0x7f9a84096910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a740098c0 tx=0x7f9a740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74040020 con 0x7f9a8409ab50 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: dump: 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: _finish_auth 0 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a74038040 con 0x7f9a8409ab50 2024-09-10T16:19:50.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 5 monclient: authenticate success, global_id 144219 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 10 monclient: _renew_subs 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a840994f0 con 0x7f9a8409ab50 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a74037070 con 0x7f9a8409ab50 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a6bfff640 10 monclient: dump: 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:50.228 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a81ffb640 4 set_mon_vals no callback set 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 10 monclient: _renew_subs 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:50.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.207+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a8409d620 con 0x7f9a8409ab50 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9a74043020 con 0x7f9a8409ab50 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a8a08c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84098080 con 0x7f9a8409b000 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a6bfff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a7003e290 0x7f9a70040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(462..462 src has 184..462) ==== 4795+0+0 (secure 0 0 0) 0x7f9a74072cb0 con 0x7f9a8409ab50 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 10 monclient: get_auth_request con 0x7f9a8409b000 auth_method 0 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:50.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a83fff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a7003e290 0x7f9a70040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a83fff640 10 monclient: get_auth_request con 0x7f9a7003e290 auth_method 0 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.208+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:50.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.211+0000 7f9a83fff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a7003e290 0x7f9a70040750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9a84097a80 tx=0x7f9a7c003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 10 monclient: get_auth_request con 0x7f9a8409b000 auth_method 0 2024-09-10T16:19:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.410+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 10 monclient: get_auth_request con 0x7f9a8409b000 auth_method 0 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:50.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:50.811+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:19:51.537 INFO:tasks.rados.rados.0.smithi165.stdout:1977: done (5 left) 2024-09-10T16:19:51.537 INFO:tasks.rados.rados.0.smithi165.stdout:1978: rollback oid 35 current snap is 216 2024-09-10T16:19:51.537 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 214 2024-09-10T16:19:51.537 INFO:tasks.rados.rados.0.smithi165.stdout:1976: finishing copy_from to smithi16534841-25 2024-09-10T16:19:51.537 INFO:tasks.rados.rados.0.smithi165.stdout:1976: got expected ENOENT (src dne) 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1976: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: finishing write tid 2 to smithi16534841-16 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: finishing write tid 3 to smithi16534841-16 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: finishing write tid 4 to smithi16534841-16 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: finishing write tid 5 to smithi16534841-16 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: oid 16 updating version 0 to 1299 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: oid 16 version 1299 is already newer than 1298 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: oid 16 version 1299 is already newer than 1297 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: oid 16 updating version 1299 to 1300 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1300 (ObjNum 707 snap 216 seq_num 707) dirty exists 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1972: left oid 16 (ObjNum 707 snap 216 seq_num 707) 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1973: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1975: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:19:51.538 INFO:tasks.rados.rados.0.smithi165.stdout:1974: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1973: finishing copy_from to smithi16534841-40 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1097 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1974: finishing copy_from to smithi16534841-29 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1220 (ObjNum 616 snap 190 seq_num 616) dirty exists 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1975: finishing copy_from to smithi16534841-8 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1147 (ObjNum 705 snap 216 seq_num 705) dirty exists 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1972: done (5 left) 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1973: done (4 left) 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1974: done (3 left) 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1975: done (2 left) 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1976: done (1 left) 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1979: read oid 48 snap -1 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1979: expect deleted 2024-09-10T16:19:51.539 INFO:tasks.rados.rados.0.smithi165.stdout:1980: snap_create 2024-09-10T16:19:51.540 INFO:tasks.rados.rados.0.smithi165.stdout:1978: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:19:51.540 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 0 (ObjNum 145 snap 0 seq_num 1160734912) dirty dne 2024-09-10T16:19:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient: get_auth_request con 0x7f9a8409b000 auth_method 0 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.612+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.613+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_done global_id 154099 payload 824 2024-09-10T16:19:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.613+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9a84134a60 con 0x7f9a8409b000 2024-09-10T16:19:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78016ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a7800f040 con 0x7f9a8409b000 2024-09-10T16:19:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.615+0000 7f9a6bfff640 10 monclient: dump: 2024-09-10T16:19:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a78014d60 con 0x7f9a8409b000 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78016ac0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 msgr2=0x7f9a8409b3e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:51.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.616+0000 7f9a6bfff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b000 0x7f9a8409b3e0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78016ac0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.623+0000 7f9a8a08c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:51.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.623+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 0x7f9a841d56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.623+0000 7f9a8a08c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.623+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9a84073460 con 0x7f9a8409b920 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 0x7f9a841d56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient: get_auth_request con 0x7f9a8409b920 auth_method 0 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:51.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:51.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:51.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 10 monclient(hunting): handle_auth_done global_id 154102 payload 824 2024-09-10T16:19:51.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.624+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9a8406dac0 con 0x7f9a8409b920 2024-09-10T16:19:51.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a82ffd640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 msgr2=0x7f9a841d56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:51.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a82ffd640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 0x7f9a841d56e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a78004020 con 0x7f9a8409b920 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 10 monclient: dump: 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f9a78004020 con 0x7f9a8409b920 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 msgr2=0x7f9a841d56e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 0x7f9a841d56e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78004730 comp rx=0 tx=0).stop 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 msgr2=0x7f9a841d56e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:51.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.627+0000 7f9a6bfff640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9a8409b920 0x7f9a841d56e0 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9a84099420 tx=0x7f9a78004730 comp rx=0 tx=0).stop 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10178, 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:51.632 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:51.633 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:51.634 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:51.635 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:51.636 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:51.637 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:51.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:51.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:51.643 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:51.644 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:51.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:51.647 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:51.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:51.647 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:51.648 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a7003e290 msgr2=0x7f9a70040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9a7003e290 0x7f9a70040750 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9a84097a80 tx=0x7f9a7c003d10 comp rx=0 tx=0).stop 2024-09-10T16:19:51.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 10 monclient: shutdown 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 msgr2=0x7f9a84096910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 --2- 172.21.15.164:0/2036135253 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9a8409ab50 0x7f9a84096910 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9a740098c0 tx=0x7f9a740046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a83fff640 1 -- 172.21.15.164:0/2036135253 reap_dead start 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a83fff640 1 -- 172.21.15.164:0/2036135253 reap_dead start 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 shutdown_connections 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 >> 172.21.15.164:0/2036135253 conn(0x7f9a840a3110 msgr2=0x7f9a840955d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 shutdown_connections 2024-09-10T16:19:51.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.628+0000 7f9a8a08c640 1 -- 172.21.15.164:0/2036135253 wait complete. 2024-09-10T16:19:51.651 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:51.651 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:19:51.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:19:51.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.754+0000 7fae637bb640 10 monclient: get_monmap_and_config 2024-09-10T16:19:51.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.754+0000 7fae637bb640 10 monclient: build_initial_monmap 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.754+0000 7fae637bb640 10 monclient: monmap: 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:51.756176+0000 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:51.756176+0000 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:51.757 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.754+0000 7fae637bb640 1 Processor -- start 2024-09-10T16:19:51.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 -- start start 2024-09-10T16:19:51.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient: init 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-b con 0x7fae5c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fae5c09ab50 0x7fae5c09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-c con 0x7fae5c09ab50 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fae5c134580 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae5c0a0870 con 0x7fae5c134580 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fae5c09d3e0 con 0x7fae5c09ab50 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c09e790 con 0x7fae5c09d930 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae637bb640 10 monclient(hunting): authenticate will time out at 1570.243618s 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61d58640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fae5c134580 0x7fae5c134a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35556/0 (socket says 172.21.15.164:35556) 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61d58640 1 -- 172.21.15.164:0/148275519 learned_addr learned my addr 172.21.15.164:0/148275519 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 10 monclient(hunting): get_auth_request con 0x7fae5c09d930 auth_method 0 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 1 -- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 msgr2=0x7fae5c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 1 -- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 msgr2=0x7fae5c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae61557640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:51.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.755+0000 7fae60d56640 1 --2- 172.21.15.164:0/148275519 >> v2:172.21.15.165:3300/0 conn(0x7fae5c09ab50 0x7fae5c09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): get_auth_request con 0x7fae5c09ab50 auth_method 0 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient(hunting): handle_auth_done global_id 154105 payload 274 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 1 -- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 msgr2=0x7fae5c09dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 1 -- 172.21.15.164:0/148275519 >> v1:172.21.15.164:6789/0 conn(0x7fae5c134580 legacy=0x7fae5c134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient: _finish_hunting 0 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 1 monclient: found mon.noname-c 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:51.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:19:51.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.756+0000 7fae60d56640 1 -- 172.21.15.164:0/148275519 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c074c70 con 0x7fae5c09ab50 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae60d56640 1 -- 172.21.15.164:0/148275519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09ab50 msgr2=0x7fae5c09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae60d56640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09ab50 0x7fae5c09e250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae5000a940 tx=0x7fae5002b400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd33557475952708 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 1 -- 172.21.15.164:0/148275519 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5000b040 con 0x7fae5c09ab50 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:51.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: dump: 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: _finish_auth 0 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:19:51.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 1 -- 172.21.15.164:0/148275519 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae50030920 con 0x7fae5c09ab50 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 1 -- 172.21.15.164:0/148275519 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae50030cf0 con 0x7fae5c09ab50 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae637bb640 5 monclient: authenticate success, global_id 154105 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae637bb640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae637bb640 4 set_mon_vals no callback set 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.759+0000 7fae5bfff640 10 monclient: dump: 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 10 monclient: shutdown 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 -- 172.21.15.164:0/148275519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09ab50 msgr2=0x7fae5c09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09ab50 0x7fae5c09e250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fae5000a940 tx=0x7fae5002b400 comp rx=0 tx=0).stop 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 -- 172.21.15.164:0/148275519 shutdown_connections 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09ab50 0x7fae5c09e250 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 --2- 172.21.15.164:0/148275519 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c09d930 0x7fae5c09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 -- 172.21.15.164:0/148275519 >> 172.21.15.164:0/148275519 conn(0x7fae5c0a3110 msgr2=0x7fae5c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 -- 172.21.15.164:0/148275519 shutdown_connections 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 1 -- 172.21.15.164:0/148275519 wait complete. 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 10 monclient: build_initial_monmap 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.760+0000 7fae637bb640 10 monclient: monmap: 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.773 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 Processor -- start 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 -- start start 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient: init 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-a con 0x7fae5c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 0x7fae5c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-b con 0x7fae5c09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 0x7fae5c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): picked mon.noname-c con 0x7fae5c134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c09ef30 con 0x7fae5c09ab50 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fae5c0a02b0 con 0x7fae5c09d930 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fae5c0a2940 con 0x7fae5c134580 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae637bb640 10 monclient(hunting): authenticate will time out at 1570.249810s 2024-09-10T16:19:51.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61d58640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 0x7fae5c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58088/0 (socket says 172.21.15.164:58088) 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 1 -- 172.21.15.164:0/3809633502 learned_addr learned my addr 172.21.15.164:0/3809633502 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 10 monclient(hunting): get_auth_request con 0x7fae5c09ab50 auth_method 0 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61557640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61d58640 10 monclient(hunting): get_auth_request con 0x7fae5c134580 auth_method 0 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61d58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61d58640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.761+0000 7fae61d58640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61d58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61d58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61d58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 0x7fae5c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 10 monclient(hunting): get_auth_request con 0x7fae5c09d930 auth_method 0 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient(hunting): handle_auth_done global_id 144222 payload 824 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 -- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 msgr2=0x7fae5c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 0x7fae5c06f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 -- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 msgr2=0x7fae5c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 0x7fae5c06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient: _finish_hunting 0 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 monclient: found mon.noname-a 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61557640 1 -- 172.21.15.164:0/3809633502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae5c1351c0 con 0x7fae5c09ab50 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae61d58640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 0x7fae5c06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.762+0000 7fae60d56640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 0x7fae5c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae61557640 1 -- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 msgr2=0x7fae5c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae61557640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae5400d7d0 tx=0x7fae5400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5400b8d0 con 0x7fae5c09ab50 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: dump: 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: _finish_auth 0 2024-09-10T16:19:51.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 5 monclient: authenticate success, global_id 144222 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae54019070 con 0x7fae5c09ab50 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 10 monclient: _renew_subs 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae5c08e350 con 0x7fae5c09ab50 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae5400bcd0 con 0x7fae5c09ab50 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae59ffb640 10 monclient: dump: 2024-09-10T16:19:51.780 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae5b7fe640 4 set_mon_vals no callback set 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 10 monclient: _renew_subs 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:51.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.764+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae5c099e80 con 0x7fae5c09ab50 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fae5401e460 con 0x7fae5c09ab50 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae637bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae637bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae637bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae5c0a1e40 con 0x7fae5c09ab50 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.765+0000 7fae59ffb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 0x7fae480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.766+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(463..463 src has 184..463) ==== 4783+0+0 (secure 0 0 0) 0x7fae54053220 con 0x7fae5c09ab50 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.766+0000 7fae60d56640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 0x7fae480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:51.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.766+0000 7fae60d56640 10 monclient: get_auth_request con 0x7fae4803e210 auth_method 0 2024-09-10T16:19:51.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.768+0000 7fae60d56640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 0x7fae480406d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fae5c09af30 tx=0x7fae5000c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:51.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.770+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fae5c1351c0 con 0x7fae5c09ab50 2024-09-10T16:19:51.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.770+0000 7fae59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:51.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:51.770+0000 7fae59ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:52.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.013+0000 7fae637bb640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:19:52.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae637bb640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:19:52.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae637bb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:52.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fae5c06dac0 con 0x7fae5c09ab50 2024-09-10T16:19:52.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae59ffb640 1 -- 172.21.15.164:0/3809633502 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fae54051c20 con 0x7fae5c09ab50 2024-09-10T16:19:52.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:19:52.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.014+0000 7fae59ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=2012049149,config=2586702516,health=2278779853,kv=4029844337,logm=1229665602}) 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=1,health=12,kv=11,logm=52} crc {auth=1203410575,config=1228700967,health=3152313405,kv=80041199,logm=1255597065}) 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=11,kv=12,logm=54} crc {auth=2012049149,config=2586702516,health=2278779853,kv=4029844337,logm=1229665602}) 2024-09-10T16:19:52.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.015+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=24,config=2,health=12,kv=11,logm=51} crc {auth=1203410575,config=2586702516,health=3152313405,kv=4161620397,logm=170317298}) 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489954128}) 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509598738}) 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3489954128}) 2024-09-10T16:19:52.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.016+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3674730798}) 2024-09-10T16:19:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 msgr2=0x7fae480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 0x7fae480406d0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fae5c09af30 tx=0x7fae5000c040 comp rx=0 tx=0).stop 2024-09-10T16:19:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 10 monclient: shutdown 2024-09-10T16:19:52.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 msgr2=0x7fae5c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fae5400d7d0 tx=0x7fae5400dc90 comp rx=0 tx=0).stop 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 shutdown_connections 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fae4803e210 0x7fae480406d0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fae5c134580 0x7fae5c06f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fae5c09d930 0x7fae5c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 --2- 172.21.15.164:0/3809633502 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fae5c09ab50 0x7fae5c06e7d0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 >> 172.21.15.164:0/3809633502 conn(0x7fae5c0a3110 msgr2=0x7fae5c0a57d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 shutdown_connections 2024-09-10T16:19:52.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:52.017+0000 7fae637bb640 1 -- 172.21.15.164:0/3809633502 wait complete. 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725317563}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3734493746}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725317563}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.017+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2621195153}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041024002}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950158007}) 2024-09-10T16:19:52.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4041024002}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.018+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871488117}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831517747}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2489963655}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=831517747}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.019+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3478826113}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631885476}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1525777791}) 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631885476}) 2024-09-10T16:19:52.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.020+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3024355263}) 2024-09-10T16:19:52.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394687490}) 2024-09-10T16:19:52.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1068229422}) 2024-09-10T16:19:52.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2394687490}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.021+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=290752824}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977791267}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705359528}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977791267}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1085869794}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236037341}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2146003428}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.022+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4236037341}) 2024-09-10T16:19:52.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3538681952}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886049238}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637506063}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=886049238}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.023+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332666997}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150196727}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2826126678}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=150196727}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2422374843}) 2024-09-10T16:19:52.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422678602}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.024+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615008618}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422678602}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4138625678}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2457424434}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2334074306}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2457424434}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.025+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2032150240}) 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754156559}) 2024-09-10T16:19:52.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3747092536}) 2024-09-10T16:19:52.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3754156559}) 2024-09-10T16:19:52.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296217529}) 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.026+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979554614}) 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2759103085}) 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979554614}) 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2190272855}) 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301390937}) 2024-09-10T16:19:52.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3323066157}) 2024-09-10T16:19:52.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.027+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301390937}) 2024-09-10T16:19:52.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2131385855}) 2024-09-10T16:19:52.035 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600902123}) 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988401145}) 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1600902123}) 2024-09-10T16:19:52.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.028+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1614237041}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4078454733}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1781899988}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4078454733}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353139397}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.029+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4009843785}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2791538579}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4009843785}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=160095343}) 2024-09-10T16:19:52.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522090625}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2202560046}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=522090625}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.030+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282820186}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247644763}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4229295554}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247644763}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=802195121}) 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353358864}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1607030205}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353358864}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.031+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2413431653}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830621273}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4066970687}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2830621273}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.032+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2553158004}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545280278}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4057326076}) 2024-09-10T16:19:52.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=545280278}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3404309213}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202276113}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4028029697}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202276113}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.033+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1834184270}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563156093}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1207399481}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1563156093}) 2024-09-10T16:19:52.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.034+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2061744397}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255101111}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4269899395}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4255101111}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=222567168}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661340767}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758739740}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661340767}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.035+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273861532}) 2024-09-10T16:19:52.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178733056}) 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3524974225}) 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2178733056}) 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3923018062}) 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841122476}) 2024-09-10T16:19:52.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2259431241}) 2024-09-10T16:19:52.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.036+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=841122476}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3807649719}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278410038}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4200366727}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=278410038}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.037+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133329431}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015844464}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=838006265}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2015844464}) 2024-09-10T16:19:52.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2677851488}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709088688}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2542870059}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3709088688}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.038+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=306679982}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619206862}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2578930848}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=619206862}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.039+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4117210893}) 2024-09-10T16:19:52.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788525739}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=459204911}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3788525739}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052705211}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319726817}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4036903052}) 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=319726817}) 2024-09-10T16:19:52.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.040+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=759635304}) 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671690823}) 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1379619521}) 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3671690823}) 2024-09-10T16:19:52.164 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770975877}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968538044}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1047633091}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2968538044}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.162+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3892976421}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1938836741,mdsmap=931935893,mgr=2975079260}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=2,mgr=1} crc {logm=1589853389,mdsmap=204415920,mgr=3189685207}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1938836741,mdsmap=931935893,mgr=2975079260}) 2024-09-10T16:19:52.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.163+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2454791302,mdsmap=931935893}) 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=56} crc {mgr=2410950321,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3160369107,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=4005186995,osd_pg_creating=3329820679,osd_snap=3107868049}) 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=55} crc {mgr=2410950321,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=327616633,mon_config_key=472225872,monmap=1433040829,nvmeofgw=598337686,osd_metadata=1740073500,osd_pg_creating=3329820679,osd_snap=1067066718}) 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=56} crc {mgr=2410950321,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3160369107,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=4005186995,osd_pg_creating=3329820679,osd_snap=3107868049}) 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.164+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {mgr=2410950321,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2018402745,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3329820679,osd_snap=4137034286}) 2024-09-10T16:19:52.169 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=2020197830,osdmap=3805648295}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=29,osdmap=71} crc {osd_snap=3509460971,osdmap=3856541819}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=2020197830,osdmap=3805648295}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=28,osdmap=72} crc {osd_snap=854213322,osdmap=1606829493}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2646726843}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3151670194}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2646726843}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.165+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3567888565}) 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.170 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=416728751}) 2024-09-10T16:19:52.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3089535901}) 2024-09-10T16:19:52.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=416728751}) 2024-09-10T16:19:52.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.166+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2298324849}) 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3384258117}) 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2398106280}) 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3384258117}) 2024-09-10T16:19:52.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.168+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1441556150}) 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=4044465690}) 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=46} crc {osdmap=3292592180}) 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=40} crc {osdmap=4044465690}) 2024-09-10T16:19:52.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:19:52.172+0000 7ff656366640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=416193000}) 2024-09-10T16:19:52.601 INFO:tasks.rados.rados.0.smithi165.stdout:1978: done (2 left) 2024-09-10T16:19:52.601 INFO:tasks.rados.rados.0.smithi165.stdout:1979: done (1 left) 2024-09-10T16:19:52.601 INFO:tasks.rados.rados.0.smithi165.stdout:1980: done (0 left) 2024-09-10T16:19:52.601 INFO:tasks.rados.rados.0.smithi165.stdout:1981: delete oid 33 current snap is 217 2024-09-10T16:19:52.606 INFO:tasks.rados.rados.0.smithi165.stdout:1981: done (0 left) 2024-09-10T16:19:52.607 INFO:tasks.rados.rados.0.smithi165.stdout:1982: read oid 42 snap -1 2024-09-10T16:19:52.607 INFO:tasks.rados.rados.0.smithi165.stdout:1982: expect deleted 2024-09-10T16:19:52.607 INFO:tasks.rados.rados.0.smithi165.stdout:1983: snap_create 2024-09-10T16:19:53.036 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:19:53.036 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:53.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.141+0000 7ffb421f3640 10 monclient: get_monmap_and_config 2024-09-10T16:19:53.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.141+0000 7ffb421f3640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.141+0000 7ffb421f3640 10 monclient: monmap: 2024-09-10T16:19:53.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:53.143051+0000 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:53.143051+0000 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:53.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.141+0000 7ffb421f3640 1 Processor -- start 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 -- start start 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient: init 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffb3c06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb3c134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb3c13d560 0x7ffb3c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3c13d560 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb3c09ce50 con 0x7ffb3c06cb70 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ffb3c06db60 con 0x7ffb3c13d560 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffb3c06cf50 con 0x7ffb3c134f70 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb421f3640 10 monclient(hunting): authenticate will time out at 1571.630588s 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb3c13d560 0x7ffb3c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56244/0 (socket says 172.21.15.164:56244) 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ffb3c13d560 0x7ffb3c13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58090/0 (socket says 172.21.15.164:58090) 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 1 -- 172.21.15.164:0/3684800568 learned_addr learned my addr 172.21.15.164:0/3684800568 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): get_auth_request con 0x7ffb3c134f70 auth_method 0 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): get_auth_request con 0x7ffb3c13d560 auth_method 0 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.142+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_done global_id 144225 payload 274 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 1 -- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 msgr2=0x7ffb3c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 1 -- 172.21.15.164:0/3684800568 >> v1:172.21.15.165:6789/0 conn(0x7ffb3c06cb70 legacy=0x7ffb3c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 1 monclient: found mon.noname-a 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb3bfff640 1 -- 172.21.15.164:0/3684800568 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3c135380 con 0x7ffb3c13d560 2024-09-10T16:19:53.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.143+0000 7ffb32ffd640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.145+0000 7ffb3bfff640 1 -- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c13d560 msgr2=0x7ffb3c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.145+0000 7ffb3bfff640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c13d560 0x7ffb3c13f950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009dc0 tx=0x7ffb3402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e53ac11b53e6af8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.145+0000 7ffb3affd640 1 -- 172.21.15.164:0/3684800568 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34007950 con 0x7ffb3c13d560 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.145+0000 7ffb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: dump: 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.156 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 1 -- 172.21.15.164:0/3684800568 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb34007af0 con 0x7ffb3c13d560 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 5 monclient: authenticate success, global_id 144225 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 1 -- 172.21.15.164:0/3684800568 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34031400 con 0x7ffb3c13d560 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 4 set_mon_vals no callback set 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb3affd640 10 monclient: dump: 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.158 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 10 monclient: shutdown 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 -- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c13d560 msgr2=0x7ffb3c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c13d560 0x7ffb3c13f950 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009dc0 tx=0x7ffb3402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 -- 172.21.15.164:0/3684800568 shutdown_connections 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c134f70 0x7ffb3c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 --2- 172.21.15.164:0/3684800568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c13d560 0x7ffb3c13f950 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 -- 172.21.15.164:0/3684800568 >> 172.21.15.164:0/3684800568 conn(0x7ffb3c0aa900 msgr2=0x7ffb3c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 -- 172.21.15.164:0/3684800568 shutdown_connections 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.146+0000 7ffb421f3640 1 -- 172.21.15.164:0/3684800568 wait complete. 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.147+0000 7ffb421f3640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.147+0000 7ffb421f3640 10 monclient: monmap: 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.147+0000 7ffb421f3640 1 Processor -- start 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.147+0000 7ffb421f3640 1 -- start start 2024-09-10T16:19:53.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.147+0000 7ffb421f3640 10 monclient: init 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb3c06cb70 0x7ffb3c13afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-b con 0x7ffb3c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c134f70 0x7ffb3c13b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3c134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): picked mon.noname-c con 0x7ffb3c13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3c0a4db0 con 0x7ffb3c134f70 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffb3c09d130 con 0x7ffb3c06cb70 2024-09-10T16:19:53.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffb3c13fe90 con 0x7ffb3c13d560 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb421f3640 10 monclient(hunting): authenticate will time out at 1571.636354s 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c134f70 0x7ffb3c13b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56246/0 (socket says 172.21.15.164:56246) 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 -- 172.21.15.164:0/1363534330 learned_addr learned my addr 172.21.15.164:0/1363534330 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): get_auth_request con 0x7ffb3c13d560 auth_method 0 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): get_auth_request con 0x7ffb3c134f70 auth_method 0 2024-09-10T16:19:53.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb3c06cb70 0x7ffb3c13afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): get_auth_request con 0x7ffb3c06cb70 auth_method 0 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient(hunting): handle_auth_done global_id 114476 payload 824 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 -- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb3c06cb70 msgr2=0x7ffb3c13afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb3c06cb70 0x7ffb3c13afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 -- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c134f70 msgr2=0x7ffb3c13b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c134f70 0x7ffb3c13b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 monclient: found mon.noname-c 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.148+0000 7ffb3bfff640 1 -- 172.21.15.164:0/1363534330 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb3c09ede0 con 0x7ffb3c13d560 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb3bfff640 1 -- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 msgr2=0x7ffb3c13ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb3bfff640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009ee0 tx=0x7ffb34004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34040070 con 0x7ffb3c13d560 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: dump: 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.165 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb34045460 con 0x7ffb3c13d560 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 5 monclient: authenticate success, global_id 114476 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 10 monclient: _renew_subs 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb3c09f3a0 con 0x7ffb3c13d560 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb34007950 con 0x7ffb3c13d560 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb38ff9640 10 monclient: dump: 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.167 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb3a7fc640 4 set_mon_vals no callback set 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 10 monclient: _renew_subs 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.151+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb3c09da70 con 0x7ffb3c13d560 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ffb34039040 con 0x7ffb3c13d560 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb38ff9640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 0x7ffb1c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb421f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb421f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb421f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(464..464 src has 184..464) ==== 4755+0+0 (secure 0 0 0) 0x7ffb34074530 con 0x7ffb3c13d560 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb3c0a46a0 con 0x7ffb3c13d560 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb3b7fe640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 0x7ffb1c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.152+0000 7ffb3b7fe640 10 monclient: get_auth_request con 0x7ffb1c03e210 auth_method 0 2024-09-10T16:19:53.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.155+0000 7ffb3b7fe640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 0x7ffb1c0406d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c006fd0 tx=0x7ffb2c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.157+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ffb3c09ce50 con 0x7ffb3c13d560 2024-09-10T16:19:53.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.157+0000 7ffb38ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.157+0000 7ffb38ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:53.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.400+0000 7ffb421f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:53.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.400+0000 7ffb421f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:53.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.400+0000 7ffb421f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.401+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffb3c0770f0 con 0x7ffb3c13d560 2024-09-10T16:19:53.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.401+0000 7ffb38ff9640 1 -- 172.21.15.164:0/1363534330 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7ffb3403c420 con 0x7ffb3c13d560 2024-09-10T16:19:53.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.401+0000 7ffb38ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:53.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.401+0000 7ffb38ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:53.404 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":134,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:53.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 msgr2=0x7ffb1c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 0x7ffb1c0406d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c006fd0 tx=0x7ffb2c009060 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 10 monclient: shutdown 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 msgr2=0x7ffb3c13ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ffb34009ee0 tx=0x7ffb34004a40 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 shutdown_connections 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffb1c03e210 0x7ffb1c0406d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffb3c13d560 0x7ffb3c13ba50 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffb3c134f70 0x7ffb3c13b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 --2- 172.21.15.164:0/1363534330 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffb3c06cb70 0x7ffb3c13afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 >> 172.21.15.164:0/1363534330 conn(0x7ffb3c0aa900 msgr2=0x7ffb3c1335e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 shutdown_connections 2024-09-10T16:19:53.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.404+0000 7ffb421f3640 1 -- 172.21.15.164:0/1363534330 wait complete. 2024-09-10T16:19:53.417 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:19:53.417 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:19:53.418 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:19:53.418 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:19:53.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.514+0000 7f3a4e8dd640 10 monclient: get_monmap_and_config 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.514+0000 7f3a4e8dd640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.514+0000 7f3a4e8dd640 10 monclient: monmap: 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:53.516326+0000 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:53.516326+0000 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:53.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:53.518 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:53.518 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:53.518 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:53.518 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 Processor -- start 2024-09-10T16:19:53.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 -- start start 2024-09-10T16:19:53.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient: init 2024-09-10T16:19:53.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a4809d930 0x7f3a4809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4809d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3a4809ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a48134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4809e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a480a0870 con 0x7f3a4809ab50 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3a4809d3e0 con 0x7f3a4809d930 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3a48074c70 con 0x7f3a4809e250 2024-09-10T16:19:53.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4e8dd640 10 monclient(hunting): authenticate will time out at 1572.003743s 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3a4809d930 0x7f3a4809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a48134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a477fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3a4809ab50 0x7f3a48134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35564/0 (socket says 172.21.15.164:35564) 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a477fe640 1 -- 172.21.15.164:0/2009177551 learned_addr learned my addr 172.21.15.164:0/2009177551 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): get_auth_request con 0x7f3a4809e250 auth_method 0 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): get_auth_request con 0x7f3a4809d930 auth_method 0 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.515+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 10 monclient(hunting): handle_auth_done global_id 144231 payload 274 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 1 -- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 msgr2=0x7f3a48134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a48134c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 1 -- 172.21.15.164:0/2009177551 >> v1:172.21.15.164:6789/0 conn(0x7f3a4809ab50 legacy=0x7f3a48134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 1 monclient: found mon.noname-a 2024-09-10T16:19:53.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a47fff640 1 -- 172.21.15.164:0/2009177551 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a4809ecb0 con 0x7f3a4809d930 2024-09-10T16:19:53.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.516+0000 7f3a4ce7a640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a48134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.518+0000 7f3a47fff640 1 -- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 msgr2=0x7f3a4809dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.518+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4809dd10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a380097a0 tx=0x7f3a38029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e0ae729a3736bbd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 1 -- 172.21.15.164:0/2009177551 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38037070 con 0x7f3a4809d930 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: dump: 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.532 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 5 monclient: authenticate success, global_id 144231 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 1 -- 172.21.15.164:0/2009177551 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a38032440 con 0x7f3a4809d930 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 4 set_mon_vals no callback set 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 1 -- 172.21.15.164:0/2009177551 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38032810 con 0x7f3a4809d930 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a46ffd640 10 monclient: dump: 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.535 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.536 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.536 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.536 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 10 monclient: shutdown 2024-09-10T16:19:53.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 msgr2=0x7f3a4809dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4809dd10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3a380097a0 tx=0x7f3a38029fd0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2009177551 shutdown_connections 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a48134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2009177551 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4809dd10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2009177551 >> 172.21.15.164:0/2009177551 conn(0x7f3a480a3110 msgr2=0x7f3a480a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2009177551 shutdown_connections 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.519+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2009177551 wait complete. 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.520+0000 7f3a4e8dd640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.520+0000 7f3a4e8dd640 10 monclient: monmap: 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.538 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:53.539 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:53.540 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:53.540 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.520+0000 7f3a4e8dd640 1 Processor -- start 2024-09-10T16:19:53.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.520+0000 7f3a4e8dd640 1 -- start start 2024-09-10T16:19:53.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.520+0000 7f3a4e8dd640 10 monclient: init 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a4809ab50 0x7f3a4813e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-b con 0x7f3a4809ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-a con 0x7f3a4809d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a4813f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): picked mon.noname-c con 0x7f3a4809e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4809e660 con 0x7f3a4809d930 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4809ef30 con 0x7f3a4809ab50 2024-09-10T16:19:53.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3a480a2940 con 0x7f3a4809e250 2024-09-10T16:19:53.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4e8dd640 10 monclient(hunting): authenticate will time out at 1572.009149s 2024-09-10T16:19:53.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a4813f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a4813f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56262/0 (socket says 172.21.15.164:56262) 2024-09-10T16:19:53.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58116/0 (socket says 172.21.15.164:58116) 2024-09-10T16:19:53.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 1 -- 172.21.15.164:0/2704051627 learned_addr learned my addr 172.21.15.164:0/2704051627 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): get_auth_request con 0x7f3a4809e250 auth_method 0 2024-09-10T16:19:53.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): get_auth_request con 0x7f3a4809d930 auth_method 0 2024-09-10T16:19:53.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a4809ab50 0x7f3a4813e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a47fff640 10 monclient(hunting): get_auth_request con 0x7f3a4809ab50 auth_method 0 2024-09-10T16:19:53.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient(hunting): handle_auth_done global_id 144234 payload 824 2024-09-10T16:19:53.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 msgr2=0x7f3a4813f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a4813f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a4809ab50 msgr2=0x7f3a4813e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3a4809ab50 0x7f3a4813e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 monclient: found mon.noname-a 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a477fe640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a48074670 con 0x7f3a4809d930 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.521+0000 7f3a4ce7a640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3a4809e250 0x7f3a4813f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a477fe640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 msgr2=0x7f3a4813ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a477fe640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00b410 tx=0x7f3a3c00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c015020 con 0x7f3a4809d930 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: dump: 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3c011a60 con 0x7f3a4809d930 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3c011e50 con 0x7f3a4809d930 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a4e8dd640 5 monclient: authenticate success, global_id 144234 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a4e8dd640 10 monclient: _renew_subs 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a4e8dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a48140110 con 0x7f3a4809d930 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a44ff9640 10 monclient: dump: 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.523+0000 7f3a467fc640 4 set_mon_vals no callback set 2024-09-10T16:19:53.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 10 monclient: _renew_subs 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a48073f30 con 0x7f3a4809d930 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 0x7f3a4806eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4813fd60 con 0x7f3a4806e7b0 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 0x7f3a4806eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3a3c018020 con 0x7f3a4809d930 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient: get_auth_request con 0x7f3a4806e7b0 auth_method 0 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.524+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a47fff640 10 monclient(hunting): handle_auth_done global_id 144237 payload 824 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a47fff640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3a4809dd10 con 0x7f3a4806e7b0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a44ff9640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3a3403e3b0 0x7f3a34040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(464..464 src has 184..464) ==== 4755+0+0 (secure 0 0 0) 0x7f3a3c052b80 con 0x7f3a4809d930 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a4ce7a640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3a3403e3b0 0x7f3a34040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.525+0000 7f3a4ce7a640 10 monclient: get_auth_request con 0x7f3a3403e3b0 auth_method 0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a47fff640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 msgr2=0x7f3a4806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 0x7f3a4806eb90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3a38034000 tx=0x7f3a380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38037040 con 0x7f3a4806e7b0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 10 monclient: dump: 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3a3803c640 con 0x7f3a4806e7b0 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:53.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 msgr2=0x7f3a4806eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 0x7f3a4806eb90 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3a38034000 tx=0x7f3a380046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 msgr2=0x7f3a4806eb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:53.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a44ff9640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806e7b0 0x7f3a4806eb90 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f3a38034000 tx=0x7f3a380046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.527+0000 7f3a4ce7a640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3a3403e3b0 0x7f3a34040870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009bd0 tx=0x7f3a4000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a4e8dd640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:53.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 0x7f3a481d78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a4e8dd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3a4809d620 con 0x7f3a4806f0d0 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 0x7f3a481d78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 10 monclient: get_auth_request con 0x7f3a4806f0d0 auth_method 0 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.533+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.534+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.534+0000 7f3a47fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.534+0000 7f3a47fff640 10 monclient(hunting): handle_auth_done global_id 144240 payload 824 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.534+0000 7f3a47fff640 1 -- 172.21.15.164:0/2704051627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3a4809cd20 con 0x7f3a4806f0d0 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a47fff640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 msgr2=0x7f3a481d78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a47fff640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 0x7f3a481d78f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a38008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38037040 con 0x7f3a4806f0d0 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 10 monclient: dump: 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f3a3803de50 con 0x7f3a4806f0d0 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 msgr2=0x7f3a481d78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 0x7f3a481d78f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a38008030 comp rx=0 tx=0).stop 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 msgr2=0x7f3a481d78f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:53.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.536+0000 7f3a44ff9640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4806f0d0 0x7f3a481d78f0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f3a38000c00 tx=0x7f3a38008030 comp rx=0 tx=0).stop 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 64979, 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:53.563 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.564 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.565 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:53.566 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:53.567 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.568 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:53.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.571 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:53.572 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:53.575 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.576 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.577 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:19:53.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.579 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:53.579 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:53.581 INFO:tasks.rados.rados.0.smithi165.stdout:1982: done (1 left) 2024-09-10T16:19:53.581 INFO:tasks.rados.rados.0.smithi165.stdout:1983: done (0 left) 2024-09-10T16:19:53.581 INFO:tasks.rados.rados.0.smithi165.stdout:1984: write oid 3 current snap is 218 2024-09-10T16:19:53.581 INFO:tasks.rados.rados.0.smithi165.stdout:1984: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2024-09-10T16:19:53.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3a3403e3b0 msgr2=0x7f3a34040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3a3403e3b0 0x7f3a34040870 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f3a40009bd0 tx=0x7f3a4000b040 comp rx=0 tx=0).stop 2024-09-10T16:19:53.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 10 monclient: shutdown 2024-09-10T16:19:53.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 msgr2=0x7f3a4813ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3a3c00b410 tx=0x7f3a3c00b8d0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4ce7a640 1 -- 172.21.15.164:0/2704051627 reap_dead start 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 shutdown_connections 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 --2- 172.21.15.164:0/2704051627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3a4809d930 0x7f3a4813ee40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 >> 172.21.15.164:0/2704051627 conn(0x7f3a480a3110 msgr2=0x7f3a4809afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 shutdown_connections 2024-09-10T16:19:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.537+0000 7f3a4e8dd640 1 -- 172.21.15.164:0/2704051627 wait complete. 2024-09-10T16:19:53.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:53.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:19:53.583 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:19:53.583 INFO:tasks.rados.rados.0.smithi165.stdout:1984: writing smithi16534841-3 from 716163 to 1288533 tid 1 2024-09-10T16:19:53.589 INFO:tasks.rados.rados.0.smithi165.stdout:1984: writing smithi16534841-3 from 2039145 to 2507392 tid 2 2024-09-10T16:19:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:1984: writing smithi16534841-3 from 3203843 to 3830406 tid 3 2024-09-10T16:19:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:1985: write (excl) oid 36 current snap is 218 2024-09-10T16:19:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:1984: finishing write tid 1 to smithi16534841-3 2024-09-10T16:19:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:1984: finishing write tid 2 to smithi16534841-3 2024-09-10T16:19:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:1985: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-10T16:19:53.605 INFO:tasks.rados.rados.0.smithi165.stdout:1985: writing smithi16534841-36 from 732970 to 1380589 tid 1 2024-09-10T16:19:53.612 INFO:tasks.rados.rados.0.smithi165.stdout:1985: writing smithi16534841-36 from 2097627 to 2709532 tid 2 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1985: writing smithi16534841-36 from 3231266 to 3669101 tid 3 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1986: rollback oid 43 current snap is 218 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 214 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1984: finishing write tid 3 to smithi16534841-3 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1984: finishing write tid 4 to smithi16534841-3 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1984: finishing write tid 5 to smithi16534841-3 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1984: oid 3 updating version 0 to 1197 2024-09-10T16:19:53.618 INFO:tasks.rados.rados.0.smithi165.stdout:1984: oid 3 updating version 1197 to 1198 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1984: oid 3 updating version 1198 to 1199 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1984: oid 3 version 1199 is already newer than 1196 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1199 (ObjNum 709 snap 218 seq_num 709) dirty exists 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1984: left oid 3 (ObjNum 709 snap 218 seq_num 709) 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1985: finishing write tid 1 to smithi16534841-36 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1984: done (2 left) 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1987: copy_from oid 42 from oid 14 current snap is 218 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1988: copy_from oid 26 from oid 35 current snap is 218 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1985: finishing write tid 2 to smithi16534841-36 2024-09-10T16:19:53.619 INFO:tasks.rados.rados.0.smithi165.stdout:1989: snap_create 2024-09-10T16:19:53.620 INFO:tasks.rados.rados.0.smithi165.stdout:1988: finishing copy_from to smithi16534841-26 2024-09-10T16:19:53.620 INFO:tasks.rados.rados.0.smithi165.stdout:1988: got expected ENOENT (src dne) 2024-09-10T16:19:53.620 INFO:tasks.rados.rados.0.smithi165.stdout:1988: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: finishing write tid 3 to smithi16534841-36 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: finishing write tid 4 to smithi16534841-36 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: finishing write tid 5 to smithi16534841-36 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: oid 36 updating version 0 to 1150 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: oid 36 version 1150 is already newer than 1149 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: oid 36 version 1150 is already newer than 1148 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: oid 36 updating version 1150 to 1151 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1151 (ObjNum 710 snap 218 seq_num 710) dirty exists 2024-09-10T16:19:53.621 INFO:tasks.rados.rados.0.smithi165.stdout:1985: left oid 36 (ObjNum 710 snap 218 seq_num 710) 2024-09-10T16:19:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:1986: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:19:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:1986: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:19:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:1986: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:19:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 905 (ObjNum 625 snap 195 seq_num 625) dirty exists 2024-09-10T16:19:53.627 INFO:tasks.rados.rados.0.smithi165.stdout:1987: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:19:53.629 INFO:tasks.rados.rados.0.smithi165.stdout:1987: finishing copy_from to smithi16534841-42 2024-09-10T16:19:53.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1147 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:19:53.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.683+0000 7f5a5cb64640 10 monclient: get_monmap_and_config 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.683+0000 7f5a5cb64640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.683+0000 7f5a5cb64640 10 monclient: monmap: 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:53.685435+0000 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:53.685435+0000 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:53.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:53.687 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:53.687 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:53.687 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:53.687 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 Processor -- start 2024-09-10T16:19:53.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 -- start start 2024-09-10T16:19:53.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient: init 2024-09-10T16:19:53.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5a5809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5a5809d160 0x7f5a5809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-a con 0x7f5a5809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a5813a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a5809a430 con 0x7f5a5809c180 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a580a24e0 con 0x7f5a5813a5c0 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5a580a2bf0 con 0x7f5a5809d160 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5cb64640 10 monclient(hunting): authenticate will time out at 1572.172931s 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a4ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5a5809d160 0x7f5a5809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5a5809c180 0x7f5a5809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35576/0 (socket says 172.21.15.164:35576) 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.684+0000 7f5a5659c640 1 -- 172.21.15.164:0/837392144 learned_addr learned my addr 172.21.15.164:0/837392144 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): get_auth_request con 0x7f5a5809d160 auth_method 0 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient(hunting): handle_auth_done global_id 144243 payload 274 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 1 -- 172.21.15.164:0/837392144 >> v1:172.21.15.165:6789/0 conn(0x7f5a5813a5c0 legacy=0x7f5a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 1 -- 172.21.15.164:0/837392144 >> v1:172.21.15.164:6789/0 conn(0x7f5a5809c180 legacy=0x7f5a5809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 1 monclient: found mon.noname-a 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.685+0000 7f5a4ffff640 1 -- 172.21.15.164:0/837392144 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5809c560 con 0x7f5a5809d160 2024-09-10T16:19:53.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a4ffff640 1 -- 172.21.15.164:0/837392144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5809d160 msgr2=0x7f5a5809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a4ffff640 1 --2- 172.21.15.164:0/837392144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5809d160 0x7f5a5809f150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a400097d0 tx=0x7f5a4002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f66b3c341e6e49 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 1 -- 172.21.15.164:0/837392144 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40037070 con 0x7f5a5809d160 2024-09-10T16:19:53.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: dump: 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 1 -- 172.21.15.164:0/837392144 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a40032440 con 0x7f5a5809d160 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 5 monclient: authenticate success, global_id 144243 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 1 -- 172.21.15.164:0/837392144 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a40032810 con 0x7f5a5809d160 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 4 set_mon_vals no callback set 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a55d9b640 10 monclient: dump: 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 10 monclient: shutdown 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 -- 172.21.15.164:0/837392144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5809d160 msgr2=0x7f5a5809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/837392144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5809d160 0x7f5a5809f150 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5a400097d0 tx=0x7f5a4002a000 comp rx=0 tx=0).stop 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 -- 172.21.15.164:0/837392144 shutdown_connections 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/837392144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5809d160 0x7f5a5809f150 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 -- 172.21.15.164:0/837392144 >> 172.21.15.164:0/837392144 conn(0x7f5a580a3110 msgr2=0x7f5a580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 -- 172.21.15.164:0/837392144 shutdown_connections 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.688+0000 7f5a5cb64640 1 -- 172.21.15.164:0/837392144 wait complete. 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.689+0000 7f5a5cb64640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.689+0000 7f5a5cb64640 10 monclient: monmap: 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:53.702 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.689+0000 7f5a5cb64640 1 Processor -- start 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 -- start start 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient: init 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:53.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5809c180 0x7f5a58091de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-c con 0x7f5a5809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a5809d160 0x7f5a58092320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-b con 0x7f5a5809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 0x7f5a58092860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): picked mon.noname-a con 0x7f5a5813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5a58074f90 con 0x7f5a5813a5c0 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a58074e10 con 0x7f5a5809d160 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5809f690 con 0x7f5a5809c180 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5cb64640 10 monclient(hunting): authenticate will time out at 1572.178520s 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5809c180 0x7f5a58091de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 0x7f5a58092860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5809c180 0x7f5a58091de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56270/0 (socket says 172.21.15.164:56270) 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 0x7f5a58092860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58142/0 (socket says 172.21.15.164:58142) 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 1 -- 172.21.15.164:0/1410394595 learned_addr learned my addr 172.21.15.164:0/1410394595 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): get_auth_request con 0x7f5a5809c180 auth_method 0 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): get_auth_request con 0x7f5a5813a5c0 auth_method 0 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient(hunting): handle_auth_done global_id 144246 payload 824 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5809c180 msgr2=0x7f5a58091de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a4ffff640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a5809d160 0x7f5a58092320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5809c180 0x7f5a58091de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a5809d160 msgr2=0x7f5a58092320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a5809d160 0x7f5a58092320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 monclient: found mon.noname-a 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.690+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a5809e430 con 0x7f5a5813a5c0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 msgr2=0x7f5a58092860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a56d9d640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 0x7f5a58092860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000c750 tx=0x7f5a5000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a50016020 con 0x7f5a5813a5c0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: dump: 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.707 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a5000aa60 con 0x7f5a5813a5c0 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 5 monclient: authenticate success, global_id 144246 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 10 monclient: _renew_subs 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a5809dfa0 con 0x7f5a5813a5c0 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a5000ae50 con 0x7f5a5813a5c0 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a4effd640 10 monclient: dump: 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.709 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5559a640 4 set_mon_vals no callback set 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 10 monclient: _renew_subs 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.693+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a580946a0 con 0x7f5a5813a5c0 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5cb64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 0x7f5a58063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a5802e9a0 con 0x7f5a5802e5c0 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 0x7f5a58063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient: get_auth_request con 0x7f5a5802e5c0 auth_method 0 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5a50019020 con 0x7f5a5813a5c0 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 10 monclient(hunting): handle_auth_done global_id 114488 payload 824 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a5659c640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a580944e0 con 0x7f5a5802e5c0 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a4effd640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a3803e3b0 0x7f5a38040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7f5a50052c10 con 0x7f5a5813a5c0 2024-09-10T16:19:53.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a4ffff640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a3803e3b0 0x7f5a38040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.694+0000 7f5a4ffff640 10 monclient: get_auth_request con 0x7f5a3803e3b0 auth_method 0 2024-09-10T16:19:53.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a5659c640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 msgr2=0x7f5a58063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a5659c640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 0x7f5a58063d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800eaa0 tx=0x7f5a4800ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a4800cc80 con 0x7f5a5802e5c0 2024-09-10T16:19:53.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.696+0000 7f5a4effd640 10 monclient: dump: 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5a48019070 con 0x7f5a5802e5c0 2024-09-10T16:19:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 msgr2=0x7f5a58063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 0x7f5a58063d50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800eaa0 tx=0x7f5a4800ef60 comp rx=0 tx=0).stop 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 msgr2=0x7f5a58063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4effd640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a5802e5c0 0x7f5a58063d50 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5a4800eaa0 tx=0x7f5a4800ef60 comp rx=0 tx=0).stop 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.697+0000 7f5a4ffff640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a3803e3b0 0x7f5a38040870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5a5806d4a0 tx=0x7f5a4002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5cb64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 0x7f5a581d57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5cb64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a58072760 con 0x7f5a58064290 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 0x7f5a581d57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient: get_auth_request con 0x7f5a58064290 auth_method 0 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 10 monclient(hunting): handle_auth_done global_id 114491 payload 824 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.703+0000 7f5a5659c640 1 -- 172.21.15.164:0/1410394595 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a580a4c70 con 0x7f5a58064290 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a5659c640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 msgr2=0x7f5a581d57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a5659c640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 0x7f5a581d57c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a480070c0 tx=0x7f5a48004940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a48024070 con 0x7f5a58064290 2024-09-10T16:19:53.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a4effd640 10 monclient: dump: 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.705+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f5a48014480 con 0x7f5a58064290 2024-09-10T16:19:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 msgr2=0x7f5a581d57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 0x7f5a581d57c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5a480070c0 tx=0x7f5a48004940 comp rx=0 tx=0).stop 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 msgr2=0x7f5a581d57c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a4effd640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a58064290 0x7f5a581d57c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:53.719 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 214274, 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.720 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.721 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:53.722 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:53.723 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:53.724 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:53.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:53.727 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:53.728 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.729 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:53.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:53.731 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:53.732 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a3803e3b0 msgr2=0x7f5a38040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a3803e3b0 0x7f5a38040870 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5a5806d4a0 tx=0x7f5a4002e000 comp rx=0 tx=0).stop 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 10 monclient: shutdown 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 msgr2=0x7f5a58092860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 --2- 172.21.15.164:0/1410394595 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a5813a5c0 0x7f5a58092860 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5a5000c750 tx=0x7f5a5000cc10 comp rx=0 tx=0).stop 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 reap_dead start 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a56d9d640 1 -- 172.21.15.164:0/1410394595 reap_dead start 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 shutdown_connections 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 >> 172.21.15.164:0/1410394595 conn(0x7f5a580a3110 msgr2=0x7f5a58139dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 shutdown_connections 2024-09-10T16:19:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.706+0000 7f5a5cb64640 1 -- 172.21.15.164:0/1410394595 wait complete. 2024-09-10T16:19:53.735 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:53.735 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:19:53.735 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.832+0000 7fd0ed35d640 10 monclient: get_monmap_and_config 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.832+0000 7fd0ed35d640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.833+0000 7fd0ed35d640 10 monclient: monmap: 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:53.834879+0000 2024-09-10T16:19:53.835 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:53.834879+0000 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:53.836 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.833+0000 7fd0ed35d640 1 Processor -- start 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.833+0000 7fd0ed35d640 1 -- start start 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.833+0000 7fd0ed35d640 10 monclient: init 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e8135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e80a40d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0e8136600 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:53.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e80a4ac0 con 0x7fd0e8136600 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd0e80a1400 con 0x7fd0e80a40d0 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e80a1120 con 0x7fd0e8135ce0 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0ed35d640 10 monclient(hunting): authenticate will time out at 1572.322209s 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56298/0 (socket says 172.21.15.164:56298) 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58150/0 (socket says 172.21.15.164:58150) 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/878888840 learned_addr learned my addr 172.21.15.164:0/878888840 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request con 0x7fd0e8135ce0 auth_method 0 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): get_auth_request con 0x7fd0e80a40d0 auth_method 0 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_done global_id 114494 payload 274 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/878888840 >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 msgr2=0x7fd0e81360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/878888840 >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/878888840 >> v1:172.21.15.165:6789/0 conn(0x7fd0e8136600 legacy=0x7fd0e809b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 monclient: found mon.noname-b 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/878888840 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e80a28e0 con 0x7fd0e8135ce0 2024-09-10T16:19:53.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.834+0000 7fd0dffff640 1 --2- 172.21.15.164:0/878888840 >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/878888840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 msgr2=0x7fd0e80a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/878888840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009870 tx=0x7fd0d802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9524c12eeb7d491d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 1 -- 172.21.15.164:0/878888840 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037070 con 0x7fd0e8135ce0 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: dump: 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 1 -- 172.21.15.164:0/878888840 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d8032440 con 0x7fd0e8135ce0 2024-09-10T16:19:53.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0ed35d640 5 monclient: authenticate success, global_id 114494 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0ed35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0ed35d640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 1 -- 172.21.15.164:0/878888840 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8032810 con 0x7fd0e8135ce0 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0ed35d640 4 set_mon_vals no callback set 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.837+0000 7fd0e67fc640 10 monclient: dump: 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.850 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 10 monclient: shutdown 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 -- 172.21.15.164:0/878888840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 msgr2=0x7fd0e80a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/878888840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8009870 tx=0x7fd0d802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 -- 172.21.15.164:0/878888840 shutdown_connections 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/878888840 >> v2:172.21.15.164:3300/0 conn(0x7fd0e80a40d0 0x7fd0e81360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/878888840 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e80a23a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 -- 172.21.15.164:0/878888840 >> 172.21.15.164:0/878888840 conn(0x7fd0e80a4f50 msgr2=0x7fd0e80a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 -- 172.21.15.164:0/878888840 shutdown_connections 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 1 -- 172.21.15.164:0/878888840 wait complete. 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 10 monclient: build_initial_monmap 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.838+0000 7fd0ed35d640 10 monclient: monmap: 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:53.852 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 Processor -- start 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 -- start start 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient: init 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e80a40d0 0x7fd0e8074120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e80a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e8131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e8135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8136600 0x7fd0e8131c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e8136600 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:53.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e809b970 con 0x7fd0e8136600 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e809b7f0 con 0x7fd0e80a40d0 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e809baf0 con 0x7fd0e8135ce0 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0ed35d640 10 monclient(hunting): authenticate will time out at 1572.327896s 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e8131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0dffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e8131710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56302/0 (socket says 172.21.15.164:56302) 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0dffff640 1 -- 172.21.15.164:0/3362456432 learned_addr learned my addr 172.21.15.164:0/3362456432 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0dffff640 10 monclient(hunting): get_auth_request con 0x7fd0e8135ce0 auth_method 0 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.839+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8136600 0x7fd0e8131c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request con 0x7fd0e8136600 auth_method 0 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:53.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e80a40d0 0x7fd0e8074120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request con 0x7fd0e80a40d0 auth_method 0 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient(hunting): handle_auth_done global_id 114497 payload 824 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e80a40d0 msgr2=0x7fd0e8074120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e80a40d0 0x7fd0e8074120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8136600 msgr2=0x7fd0e8131c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8136600 0x7fd0e8131c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 monclient: found mon.noname-c 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:53.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.840+0000 7fd0dffff640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e81301e0 con 0x7fd0e8135ce0 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0dffff640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 msgr2=0x7fd0e8131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0dffff640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e8131710 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000ebe0 tx=0x7fd0d000c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0017070 con 0x7fd0e8135ce0 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: dump: 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.857 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 5 monclient: authenticate success, global_id 114497 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d000ce40 con 0x7fd0e8135ce0 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 10 monclient: _renew_subs 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d0013690 con 0x7fd0e8135ce0 2024-09-10T16:19:53.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e812e400 con 0x7fd0e8135ce0 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0e5ffb640 4 set_mon_vals no callback set 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0df7fe640 10 monclient: dump: 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:53.859 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 10 monclient: _renew_subs 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.842+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e812d9f0 con 0x7fd0e8135ce0 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0ed35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e809bf00 con 0x7fd0e812e9e0 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd0d001a070 con 0x7fd0e8135ce0 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0df7fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e240 0x7fd0c8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:53.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 10 monclient: get_auth_request con 0x7fd0e812e9e0 auth_method 0 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7fd0d0052d00 con 0x7fd0e8135ce0 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 msgr2=0x7fd0e81cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 msgr2=0x7fd0e81cb0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e240 0x7fd0c8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:53.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.843+0000 7fd0e77fe640 10 monclient: get_auth_request con 0x7fd0c803e240 auth_method 0 2024-09-10T16:19:53.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:53.846+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e240 0x7fd0c8040700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd0e81306d0 tx=0x7fd0e0003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient: get_auth_request con 0x7fd0e812e9e0 auth_method 0 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:54.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:54.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:54.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_done global_id 154123 payload 824 2024-09-10T16:19:54.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.044+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0e80008d0 con 0x7fd0e812e9e0 2024-09-10T16:19:54.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 msgr2=0x7fd0e81cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037040 con 0x7fd0e812e9e0 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 10 monclient: dump: 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.050 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd0d803c410 con 0x7fd0e812e9e0 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 msgr2=0x7fd0e81cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d80046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 msgr2=0x7fd0e81cb0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:54.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.047+0000 7fd0df7fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e812e9e0 0x7fd0e81cb0d0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8034000 tx=0x7fd0d80046c0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.053+0000 7fd0ed35d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:19:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.053+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 0x7fd0e81cba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.053+0000 7fd0ed35d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.053+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e81cbf60 con 0x7fd0e81cb610 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.053+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 0x7fd0e81cba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient: get_auth_request con 0x7fd0e81cb610 auth_method 0 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 10 monclient(hunting): handle_auth_done global_id 154126 payload 824 2024-09-10T16:19:54.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.054+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0e81d6420 con 0x7fd0e81cb610 2024-09-10T16:19:54.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.056+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 msgr2=0x7fd0e81cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.056+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 0x7fd0e81cba20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80090b0 tx=0x7fd0d8009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8037040 con 0x7fd0e81cb610 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 10 monclient: dump: 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.060 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fd0d8003d30 con 0x7fd0e81cb610 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 msgr2=0x7fd0e81cba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 0x7fd0e81cba20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd0d80090b0 tx=0x7fd0d8009240 comp rx=0 tx=0).stop 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 msgr2=0x7fd0e81cba20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0df7fe640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e81cb610 0x7fd0e81cba20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.061 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 134, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12608, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:19:54.062 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:54.063 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:54.064 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:54.065 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:19:54.066 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.067 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.068 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:19:54.069 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.070 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:19:54.071 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.072 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.073 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:19:54.074 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:19:54.075 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:19:54.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e240 msgr2=0x7fd0c8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e240 0x7fd0c8040700 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fd0e81306d0 tx=0x7fd0e0003d10 comp rx=0 tx=0).stop 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 10 monclient: shutdown 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 msgr2=0x7fd0e8131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 --2- 172.21.15.164:0/3362456432 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8135ce0 0x7fd0e8131710 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd0d000ebe0 tx=0x7fd0d000c520 comp rx=0 tx=0).stop 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0e77fe640 1 -- 172.21.15.164:0/3362456432 reap_dead start 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0e77fe640 1 -- 172.21.15.164:0/3362456432 reap_dead start 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 shutdown_connections 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 >> 172.21.15.164:0/3362456432 conn(0x7fd0e80a4f50 msgr2=0x7fd0e80a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 shutdown_connections 2024-09-10T16:19:54.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.057+0000 7fd0ed35d640 1 -- 172.21.15.164:0/3362456432 wait complete. 2024-09-10T16:19:54.078 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:19:54.078 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:19:54.078 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:19:54.078 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:19:54.078 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:19:54.078 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:19:54.079 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:19:54.090 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:19:54.090 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:19:54.090 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:54.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.224+0000 7f954af90640 10 monclient: get_monmap_and_config 2024-09-10T16:19:54.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.224+0000 7f954af90640 10 monclient: build_initial_monmap 2024-09-10T16:19:54.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.224+0000 7f954af90640 10 monclient: monmap: 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:54.226399+0000 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:54.226399+0000 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 Processor -- start 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 -- start start 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient: init 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9544074af0 addr v1:172.21.15.164:6789/0 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 0x7f95440a1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): picked mon.noname-a con 0x7f95440a1090 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): picked mon.noname-b con 0x7f954413d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954409ebc0 con 0x7f9544074af0 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f95440698a0 con 0x7f95440a1090 2024-09-10T16:19:54.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f954406bfb0 con 0x7f954413d190 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954af90640 10 monclient(hunting): authenticate will time out at 1572.713750s 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f9548d2c640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f9544074af0 legacy=0x7f9544137e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f9543fff640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 msgr2=0x7f95440a1500 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f9543fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 0x7f95440a1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56308/0 (socket says 172.21.15.164:56308) 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 1 -- 172.21.15.164:0/2761108547 learned_addr learned my addr 172.21.15.164:0/2761108547 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): get_auth_request con 0x7f954413d190 auth_method 0 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:54.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.225+0000 7f954952d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 10 monclient(hunting): handle_auth_done global_id 114500 payload 274 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 1 -- 172.21.15.164:0/2761108547 >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 msgr2=0x7f95440a1500 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 1 --2- 172.21.15.164:0/2761108547 >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 0x7f95440a1500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 1 -- 172.21.15.164:0/2761108547 >> v1:172.21.15.164:6789/0 conn(0x7f9544074af0 legacy=0x7f9544137e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 10 monclient: _finish_hunting 0 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 1 monclient: found mon.noname-b 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:54.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:19:54.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.226+0000 7f954952d640 1 -- 172.21.15.164:0/2761108547 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9544138580 con 0x7f954413d190 2024-09-10T16:19:54.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.228+0000 7f954952d640 1 -- 172.21.15.164:0/2761108547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 msgr2=0x7f954413f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.228+0000 7f954952d640 1 --2- 172.21.15.164:0/2761108547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f953c009e50 tx=0x7f953c02a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d65e671a46152c3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 1 -- 172.21.15.164:0/2761108547 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c036070 con 0x7f954413d190 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: dump: 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.241 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:54.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 1 -- 172.21.15.164:0/2761108547 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f953c007800 con 0x7f954413d190 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 5 monclient: authenticate success, global_id 114500 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 1 -- 172.21.15.164:0/2761108547 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f953c007bd0 con 0x7f954413d190 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 4 set_mon_vals no callback set 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f95437fe640 10 monclient: dump: 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 10 monclient: shutdown 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 -- 172.21.15.164:0/2761108547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 msgr2=0x7f954413f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 --2- 172.21.15.164:0/2761108547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f953c009e50 tx=0x7f953c02a570 comp rx=0 tx=0).stop 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 -- 172.21.15.164:0/2761108547 shutdown_connections 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 --2- 172.21.15.164:0/2761108547 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f954413d190 0x7f954413f580 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 --2- 172.21.15.164:0/2761108547 >> v2:172.21.15.164:3300/0 conn(0x7f95440a1090 0x7f95440a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 -- 172.21.15.164:0/2761108547 >> 172.21.15.164:0/2761108547 conn(0x7f95440a3fb0 msgr2=0x7f95440a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 -- 172.21.15.164:0/2761108547 shutdown_connections 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.229+0000 7f954af90640 1 -- 172.21.15.164:0/2761108547 wait complete. 2024-09-10T16:19:54.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.230+0000 7f954af90640 10 monclient: build_initial_monmap 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.230+0000 7f954af90640 10 monclient: monmap: 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.246 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.230+0000 7f954af90640 1 Processor -- start 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.230+0000 7f954af90640 1 -- start start 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.230+0000 7f954af90640 10 monclient: init 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:54.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 0x7f954409b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): picked mon.noname-a con 0x7f9544074af0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): picked mon.noname-c con 0x7f95440a1090 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 0x7f95441ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): picked mon.noname-b con 0x7f954413d190 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:54.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9544074ed0 con 0x7f9544074af0 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9544069370 con 0x7f954413d190 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f954413fcd0 con 0x7f95440a1090 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954af90640 10 monclient(hunting): authenticate will time out at 1572.719218s 2024-09-10T16:19:54.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9548d2c640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 msgr2=0x7f954409b8d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9548d2c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 0x7f954409b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56310/0 (socket says 172.21.15.164:56310) 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 -- 172.21.15.164:0/2109167793 learned_addr learned my addr 172.21.15.164:0/2109167793 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): get_auth_request con 0x7f95440a1090 auth_method 0 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:54.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 0x7f95441ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 10 monclient(hunting): get_auth_request con 0x7f954413d190 auth_method 0 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:54.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient(hunting): handle_auth_done global_id 114503 payload 824 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 -- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 msgr2=0x7f95441ccea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 0x7f95441ccea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 -- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 msgr2=0x7f954409b8d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 0x7f954409b8d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:54.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient: _finish_hunting 0 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 monclient: found mon.noname-c 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f9543fff640 1 -- 172.21.15.164:0/2109167793 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95441399a0 con 0x7f95440a1090 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.231+0000 7f954952d640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 0x7f95441ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f9543fff640 1 -- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 msgr2=0x7f954409be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:54.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f9543fff640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f953000b550 tx=0x7f953000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9530005070 con 0x7f95440a1090 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: dump: 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: _finish_auth 0 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: _check_auth_tickets 2024-09-10T16:19:54.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9530015070 con 0x7f95440a1090 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f954af90640 5 monclient: authenticate success, global_id 114503 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9530005450 con 0x7f95440a1090 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.233+0000 7f954af90640 10 monclient: _renew_subs 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f954af90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9544139c80 con 0x7f95440a1090 2024-09-10T16:19:54.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f95417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f95417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f9542ffd640 4 set_mon_vals no callback set 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f95417fa640 10 monclient: dump: 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.257 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f954af90640 10 monclient: _renew_subs 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f954af90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:54.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f954409cf30 con 0x7f95440a1090 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.234+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9530005630 con 0x7f95440a1090 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f954af90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f954af90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f954af90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9544138f60 con 0x7f95440a1090 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f95417fa640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 0x7f95280406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:54.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7f9530052ae0 con 0x7f95440a1090 2024-09-10T16:19:54.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f9548d2c640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 0x7f95280406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:54.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.235+0000 7f9548d2c640 10 monclient: get_auth_request con 0x7f952803e1f0 auth_method 0 2024-09-10T16:19:54.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.238+0000 7f9548d2c640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 0x7f95280406b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9538006fd0 tx=0x7f9538006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:54.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:54.239+0000 7f9543fff640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f953000b550 tx=0x7f953000ba10 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f95300514f0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:19:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.068+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f95300514f0 con 0x7f95440a1090 2024-09-10T16:19:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.068+0000 7f95417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:55.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.068+0000 7f95417fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.312+0000 7f954af90640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.312+0000 7f954af90640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.312+0000 7f954af90640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.312+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95440a14e0 con 0x7f95440a1090 2024-09-10T16:19:55.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.313+0000 7f95417fa640 1 -- 172.21.15.164:0/2109167793 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f953001a460 con 0x7f95440a1090 2024-09-10T16:19:55.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.313+0000 7f95417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:55.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.313+0000 7f95417fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:55.316 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":134,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:55.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 msgr2=0x7f95280406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:55.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 0x7f95280406b0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f9538006fd0 tx=0x7f9538006d80 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 10 monclient: shutdown 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 msgr2=0x7f954409be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f953000b550 tx=0x7f953000ba10 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 shutdown_connections 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f952803e1f0 0x7f95280406b0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f954413d190 0x7f95441ccea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95440a1090 0x7f954409be10 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 --2- 172.21.15.164:0/2109167793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9544074af0 0x7f954409b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 >> 172.21.15.164:0/2109167793 conn(0x7f95440a3fb0 msgr2=0x7f9544137a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 shutdown_connections 2024-09-10T16:19:55.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:55.315+0000 7f954af90640 1 -- 172.21.15.164:0/2109167793 wait complete. 2024-09-10T16:19:58.329 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:19:58.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.436+0000 7fa632fda640 10 monclient: get_monmap_and_config 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.436+0000 7fa632fda640 10 monclient: build_initial_monmap 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.436+0000 7fa632fda640 10 monclient: monmap: 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:19:58.437474+0000 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:19:58.437474+0000 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:19:58.438 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:19:58.439 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:19:58.439 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:19:58.439 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 Processor -- start 2024-09-10T16:19:58.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 -- start start 2024-09-10T16:19:58.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient: init 2024-09-10T16:19:58.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa62c09c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 0x7fa62c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-a con 0x7fa62c09d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa62c13a5c0 0x7fa62c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-c con 0x7fa62c13a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62c09fbc0 con 0x7fa62c09c310 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa62c0a2a20 con 0x7fa62c09d190 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:58.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa62c09c6f0 con 0x7fa62c13a5c0 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa632fda640 10 monclient(hunting): authenticate will time out at 1576.924948s 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.437+0000 7fa623fff640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 msgr2=0x7fa62c09f150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa623fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 0x7fa62c09f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa62c13a5c0 0x7fa62c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa62c13a5c0 0x7fa62c13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:55502/0 (socket says 172.21.15.164:55502) 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 1 -- 172.21.15.164:0/3701427721 learned_addr learned my addr 172.21.15.164:0/3701427721 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 10 monclient(hunting): get_auth_request con 0x7fa62c13a5c0 auth_method 0 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:58.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.438+0000 7fa631577640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:58.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa630d76640 1 -- 172.21.15.164:0/3701427721 >> v1:172.21.15.165:6789/0 conn(0x7fa62c09c310 legacy=0x7fa62c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient(hunting): handle_auth_done global_id 154132 payload 274 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 1 -- 172.21.15.164:0/3701427721 >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 msgr2=0x7fa62c09f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 1 --2- 172.21.15.164:0/3701427721 >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 0x7fa62c09f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 1 -- 172.21.15.164:0/3701427721 >> v1:172.21.15.165:6789/0 conn(0x7fa62c09c310 legacy=0x7fa62c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient: _finish_hunting 0 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 1 monclient: found mon.noname-c 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:58.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:19:58.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.440+0000 7fa631577640 1 -- 172.21.15.164:0/3701427721 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c09a4d0 con 0x7fa62c13a5c0 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.443+0000 7fa631577640 1 -- 172.21.15.164:0/3701427721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 msgr2=0x7fa62c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.443+0000 7fa631577640 1 --2- 172.21.15.164:0/3701427721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c13c9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa62c06dcd0 tx=0x7fa62402a600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae328a0cb7430cfc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 1 -- 172.21.15.164:0/3701427721 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624036070 con 0x7fa62c13a5c0 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: dump: 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.450 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: _finish_auth 0 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: _check_auth_tickets 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 1 -- 172.21.15.164:0/3701427721 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa624007800 con 0x7fa62c13a5c0 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 5 monclient: authenticate success, global_id 154132 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:19:58.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 1 -- 172.21.15.164:0/3701427721 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa624007bd0 con 0x7fa62c13a5c0 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 10 monclient: get_monmap_and_config success 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 4 set_mon_vals no callback set 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa6237fe640 10 monclient: dump: 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 10 monclient: shutdown 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 -- 172.21.15.164:0/3701427721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 msgr2=0x7fa62c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 --2- 172.21.15.164:0/3701427721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c13c9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa62c06dcd0 tx=0x7fa62402a600 comp rx=0 tx=0).stop 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 -- 172.21.15.164:0/3701427721 shutdown_connections 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 --2- 172.21.15.164:0/3701427721 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c13c9b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 --2- 172.21.15.164:0/3701427721 >> v2:172.21.15.164:3300/0 conn(0x7fa62c09d190 0x7fa62c09f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 -- 172.21.15.164:0/3701427721 >> 172.21.15.164:0/3701427721 conn(0x7fa62c0a3110 msgr2=0x7fa62c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 -- 172.21.15.164:0/3701427721 shutdown_connections 2024-09-10T16:19:58.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.444+0000 7fa632fda640 1 -- 172.21.15.164:0/3701427721 wait complete. 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.445+0000 7fa632fda640 10 monclient: build_initial_monmap 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.445+0000 7fa632fda640 10 monclient: monmap: 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.445+0000 7fa632fda640 1 Processor -- start 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.445+0000 7fa632fda640 1 -- start start 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.445+0000 7fa632fda640 10 monclient: init 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient: _reopen_session rank -1 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-c con 0x7fa62c09c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c090ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-b con 0x7fa62c13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 0x7fa62c1d37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): picked mon.noname-a con 0x7fa62c0910e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0a2230 con 0x7fa62c0910e0 2024-09-10T16:19:58.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa62c0a20b0 con 0x7fa62c13a5c0 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): start opening mon connection 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa62c074e00 con 0x7fa62c09c310 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): _renew_subs 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa632fda640 10 monclient(hunting): authenticate will time out at 1576.933372s 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa631577640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 msgr2=0x7fa62c1d37c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa631577640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 0x7fa62c1d37c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58402/0 (socket says 172.21.15.164:58402) 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 -- 172.21.15.164:0/850777217 learned_addr learned my addr 172.21.15.164:0/850777217 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): get_auth_request con 0x7fa62c09c310 auth_method 0 2024-09-10T16:19:58.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c090ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): get_auth_request con 0x7fa62c13a5c0 auth_method 0 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:19:58.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa623fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient(hunting): handle_auth_done global_id 114506 payload 824 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 -- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 msgr2=0x7fa62c090ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c090ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 -- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 msgr2=0x7fa62c1d37c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 0x7fa62c1d37c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient: _finish_hunting 0 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 monclient: found mon.noname-c 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:19:58.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.446+0000 7fa630d76640 1 -- 172.21.15.164:0/850777217 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa62c09e580 con 0x7fa62c09c310 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa630d76640 1 -- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 msgr2=0x7fa62c090660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa630d76640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c7f0 tx=0x7fa62800ccb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628017070 con 0x7fa62c09c310 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: dump: 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:58.461 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: _finish_auth 0 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: _check_auth_tickets 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6280057c0 con 0x7fa62c09c310 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 5 monclient: authenticate success, global_id 114506 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628005b70 con 0x7fa62c09c310 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 10 monclient: _renew_subs 2024-09-10T16:19:58.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa62c09d620 con 0x7fa62c09c310 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa6217fa640 10 monclient: dump: 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:19:58.463 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa622ffd640 4 set_mon_vals no callback set 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 10 monclient: _renew_subs 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.449+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa62c091930 con 0x7fa62c09c310 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa632fda640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa632fda640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa632fda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa62c09f720 con 0x7fa62c09c310 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa62801a070 con 0x7fa62c09c310 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa6217fa640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 0x7fa618040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7fa628052320 con 0x7fa62c09c310 2024-09-10T16:19:58.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa623fff640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 0x7fa618040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:19:58.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.450+0000 7fa623fff640 10 monclient: get_auth_request con 0x7fa61803e1c0 auth_method 0 2024-09-10T16:19:58.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.453+0000 7fa623fff640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 0x7fa618040680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009390 tx=0x7fa61c009320 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:19:58.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.455+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa62c09fbc0 con 0x7fa62c09c310 2024-09-10T16:19:58.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.455+0000 7fa6217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:19:58.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.455+0000 7fa6217fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:19:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.699+0000 7fa632fda640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:19:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.699+0000 7fa632fda640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.699+0000 7fa632fda640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:19:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.699+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa62c0735c0 con 0x7fa62c09c310 2024-09-10T16:19:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.700+0000 7fa6217fa640 1 -- 172.21.15.164:0/850777217 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fa6280520d0 con 0x7fa62c09c310 2024-09-10T16:19:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.700+0000 7fa6217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:19:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.700+0000 7fa6217fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:19:58.702 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":134,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 msgr2=0x7fa618040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 0x7fa618040680 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa61c009390 tx=0x7fa61c009320 comp rx=0 tx=0).stop 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 10 monclient: shutdown 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 msgr2=0x7fa62c090660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fa62800c7f0 tx=0x7fa62800ccb0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 shutdown_connections 2024-09-10T16:19:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa61803e1c0 0x7fa618040680 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa62c0910e0 0x7fa62c1d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa62c13a5c0 0x7fa62c090ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 --2- 172.21.15.164:0/850777217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa62c09c310 0x7fa62c090660 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 >> 172.21.15.164:0/850777217 conn(0x7fa62c0a3110 msgr2=0x7fa62c1397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 shutdown_connections 2024-09-10T16:19:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:19:58.702+0000 7fa632fda640 1 -- 172.21.15.164:0/850777217 wait complete. 2024-09-10T16:20:01.716 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.826+0000 7facd59c6640 10 monclient: get_monmap_and_config 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.826+0000 7facd59c6640 10 monclient: build_initial_monmap 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.826+0000 7facd59c6640 10 monclient: monmap: 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:01.828 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:01.827635+0000 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:01.827635+0000 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:01.829 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 Processor -- start 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 -- start start 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient: init 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 0x7facd0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-a con 0x7facd006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7facd00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-b con 0x7facd00a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facd009ce50 con 0x7facd00a46a0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7facd006db60 con 0x7facd006cb70 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7facd006cf50 con 0x7facd00a4ab0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.827+0000 7facd59c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7facd59c6640 10 monclient(hunting): authenticate will time out at 1580.315010s 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7facc67fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7facd00a46a0 legacy=0x7facd0137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7facceffd640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 msgr2=0x7facd0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:01.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7facceffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 0x7facd0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58406/0 (socket says 172.21.15.164:58406) 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 -- 172.21.15.164:0/1976177494 learned_addr learned my addr 172.21.15.164:0/1976177494 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): get_auth_request con 0x7facd00a4ab0 auth_method 0 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient(hunting): handle_auth_done global_id 114509 payload 274 2024-09-10T16:20:01.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 -- 172.21.15.164:0/1976177494 >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 msgr2=0x7facd0134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 --2- 172.21.15.164:0/1976177494 >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 0x7facd0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 -- 172.21.15.164:0/1976177494 >> v1:172.21.15.164:6789/0 conn(0x7facd00a46a0 legacy=0x7facd0137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 monclient: found mon.noname-b 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.828+0000 7faccf7fe640 1 -- 172.21.15.164:0/1976177494 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd009cb70 con 0x7facd00a4ab0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7faccf7fe640 1 -- 172.21.15.164:0/1976177494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 msgr2=0x7facd01383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7faccf7fe640 1 --2- 172.21.15.164:0/1976177494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e80 tx=0x7facc802a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5721bdda73e910a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 1 -- 172.21.15.164:0/1976177494 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8036070 con 0x7facd00a4ab0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:01.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: dump: 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:01.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 1 -- 172.21.15.164:0/1976177494 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facc8007800 con 0x7facd00a4ab0 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facd59c6640 5 monclient: authenticate success, global_id 114509 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facd59c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facd59c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 1 -- 172.21.15.164:0/1976177494 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facc8007bd0 con 0x7facd00a4ab0 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:01.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facd59c6640 4 set_mon_vals no callback set 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.831+0000 7facce7fc640 10 monclient: dump: 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:01.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 10 monclient: shutdown 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 -- 172.21.15.164:0/1976177494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 msgr2=0x7facd01383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 --2- 172.21.15.164:0/1976177494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7facc8009e80 tx=0x7facc802a5b0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 -- 172.21.15.164:0/1976177494 shutdown_connections 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 --2- 172.21.15.164:0/1976177494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a4ab0 0x7facd01383b0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 --2- 172.21.15.164:0/1976177494 >> v2:172.21.15.164:3300/0 conn(0x7facd006cb70 0x7facd0134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 -- 172.21.15.164:0/1976177494 >> 172.21.15.164:0/1976177494 conn(0x7facd00aa900 msgr2=0x7facd00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:01.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 -- 172.21.15.164:0/1976177494 shutdown_connections 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 1 -- 172.21.15.164:0/1976177494 wait complete. 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 10 monclient: build_initial_monmap 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.832+0000 7facd59c6640 10 monclient: monmap: 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:01.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 Processor -- start 2024-09-10T16:20:01.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 -- start start 2024-09-10T16:20:01.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient: init 2024-09-10T16:20:01.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:01.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:20:01.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd006cb70 0x7facd01312c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-b con 0x7facd006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:01.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-c con 0x7facd00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 0x7facd00a2790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): picked mon.noname-a con 0x7facd00a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7facd0138a70 con 0x7facd00a4ab0 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7facd01388f0 con 0x7facd006cb70 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7facd0140040 con 0x7facd00a46a0 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facd59c6640 10 monclient(hunting): authenticate will time out at 1580.320893s 2024-09-10T16:20:01.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7faccf7fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 msgr2=0x7facd00a2790 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7faccf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 0x7facd00a2790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.833+0000 7facc67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58418/0 (socket says 172.21.15.164:58418) 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 -- 172.21.15.164:0/1196648112 learned_addr learned my addr 172.21.15.164:0/1196648112 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): get_auth_request con 0x7facd00a46a0 auth_method 0 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:01.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facceffd640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd006cb70 0x7facd01312c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facceffd640 10 monclient(hunting): get_auth_request con 0x7facd006cb70 auth_method 0 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient(hunting): handle_auth_done global_id 114512 payload 824 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 -- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd006cb70 msgr2=0x7facd01312c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd006cb70 0x7facd01312c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 -- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 msgr2=0x7facd00a2790 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:01.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 0x7facd00a2790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 monclient: found mon.noname-c 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.834+0000 7facc67fc640 1 -- 172.21.15.164:0/1196648112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facd013aae0 con 0x7facd00a46a0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facc67fc640 1 -- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 msgr2=0x7facd0131800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facc67fc640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7facb800b550 tx=0x7facb800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8005070 con 0x7facd00a46a0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: dump: 2024-09-10T16:20:01.852 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: _finish_auth 0 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: _check_auth_tickets 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 5 monclient: authenticate success, global_id 114512 2024-09-10T16:20:01.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7facb8015070 con 0x7facd00a46a0 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facb8005450 con 0x7facd00a46a0 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facccff9640 10 monclient: dump: 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:01.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 10 monclient: _renew_subs 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7faccdffb640 4 set_mon_vals no callback set 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facd006e720 con 0x7facd00a46a0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 10 monclient: _renew_subs 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.836+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7facd01392e0 con 0x7facd00a46a0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7facb8005630 con 0x7facd00a46a0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facd59c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facd59c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facd59c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:01.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facd0134f70 con 0x7facd00a46a0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facccff9640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 0x7facb00406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.837+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7facb8052b60 con 0x7facd00a46a0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.838+0000 7facceffd640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 0x7facb00406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.838+0000 7facceffd640 10 monclient: get_auth_request con 0x7facb003e1f0 auth_method 0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.840+0000 7facceffd640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 0x7facb00406b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7facc00038b0 tx=0x7facc0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.842+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7facb8051570 con 0x7facd00a46a0 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.842+0000 7facccff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:01.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:01.842+0000 7facccff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facd59c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facd59c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facd59c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7facd00770f0 con 0x7facd00a46a0 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facccff9640 1 -- 172.21.15.164:0/1196648112 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7facb801a460 con 0x7facd00a46a0 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facccff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.084+0000 7facccff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:02.086 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":134,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 msgr2=0x7facb00406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 0x7facb00406b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7facc00038b0 tx=0x7facc0008040 comp rx=0 tx=0).stop 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 10 monclient: shutdown 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 msgr2=0x7facd0131800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7facb800b550 tx=0x7facb800ba10 comp rx=0 tx=0).stop 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 shutdown_connections 2024-09-10T16:20:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7facb003e1f0 0x7facb00406b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7facd00a4ab0 0x7facd00a2790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7facd00a46a0 0x7facd0131800 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 --2- 172.21.15.164:0/1196648112 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7facd006cb70 0x7facd01312c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.086+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 >> 172.21.15.164:0/1196648112 conn(0x7facd00aa900 msgr2=0x7facd00a5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.087+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 shutdown_connections 2024-09-10T16:20:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:02.087+0000 7facd59c6640 1 -- 172.21.15.164:0/1196648112 wait complete. 2024-09-10T16:20:05.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:05.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 10 monclient: get_monmap_and_config 2024-09-10T16:20:05.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 10 monclient: build_initial_monmap 2024-09-10T16:20:05.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 10 monclient: monmap: 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:05.208938+0000 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:05.208938+0000 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:05.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 1 Processor -- start 2024-09-10T16:20:05.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 1 -- start start 2024-09-10T16:20:05.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.208+0000 7ff1e0d73640 10 monclient: init 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-c con 0x7ff1dc06cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff1dc0a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-b con 0x7ff1dc0a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1dc09ce50 con 0x7ff1dc0a46a0 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff1dc06db60 con 0x7ff1dc06cb70 2024-09-10T16:20:05.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff1dc06cf50 con 0x7ff1dc0a4ab0 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1e0d73640 10 monclient(hunting): authenticate will time out at 1583.696506s 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1d9d9b640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7ff1dc0a46a0 legacy=0x7ff1dc137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54988/0 (socket says 172.21.15.164:54988) 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 learned_addr learned my addr 172.21.15.164:0/2197281563 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request con 0x7ff1dc0a4ab0 auth_method 0 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1dc06cb70 auth_method 0 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.209+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:05.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request con 0x7ff1dc0a4ab0 auth_method 0 2024-09-10T16:20:05.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1dc06cb70 auth_method 0 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:05.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.410+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.164:6789/0 conn(0x7ff1dc0a46a0 legacy=0x7ff1dc137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:05.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request con 0x7ff1dc0a4ab0 auth_method 0 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.164:6789/0 conn(0x7ff1dc0a46a0 legacy=0x7ff1dc137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1dc06cb70 auth_method 0 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:05.811+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:06.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:06.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.164:6789/0 conn(0x7ff1dc0a46a0 legacy=0x7ff1dc137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:06.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request con 0x7ff1dc0a4ab0 auth_method 0 2024-09-10T16:20:06.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:06.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.612+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1dc06cb70 auth_method 0 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:06.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:06.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:06.613+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:20:08.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): tick 2024-09-10T16:20:08.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 msgr2=0x7ff1dc1383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1dc0a4ab0 0x7ff1dc1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 msgr2=0x7ff1dc134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.165:3300/0 conn(0x7ff1dc06cb70 0x7ff1dc134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.164:6789/0 conn(0x7ff1dc0a46a0 legacy=0x7ff1dc137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 0x7ff1bc005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): picked mon.noname-a con 0x7ff1bc0036d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff1bc006220 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:08.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 -- 172.21.15.164:0/2197281563 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1dc09cb70 con 0x7ff1bc006220 2024-09-10T16:20:08.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 1 -- 172.21.15.164:0/2197281563 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff1dc06cf50 con 0x7ff1bc0036d0 2024-09-10T16:20:08.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1cbfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:08.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 msgr2=0x7ff1bc005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:08.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.209+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 0x7ff1bc005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:08.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.212+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.165:6789/0 conn(0x7ff1bc006220 legacy=0x7ff1bc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:08.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.410+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 msgr2=0x7ff1bc005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:08.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.410+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 0x7ff1bc005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:08.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655324389 0 0) 0x7ff1dc09cb70 con 0x7ff1bc006220 2024-09-10T16:20:08.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 10 monclient(hunting): _negotiate 2024-09-10T16:20:08.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:08.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:08.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 10 monclient(hunting): my global_id is 154144 2024-09-10T16:20:08.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.620+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1c4003520 con 0x7ff1bc006220 2024-09-10T16:20:08.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2184216326 0 0) 0x7ff1c4003520 con 0x7ff1bc006220 2024-09-10T16:20:08.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 msgr2=0x7ff1bc005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:08.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 --2- 172.21.15.164:0/2197281563 >> v2:172.21.15.164:3300/0 conn(0x7ff1bc0036d0 0x7ff1bc005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _finish_hunting 0 2024-09-10T16:20:08.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1bc00aff0 con 0x7ff1bc006220 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _finish_auth 0 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _check_auth_tickets 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7ff1d0004500 con 0x7ff1bc006220 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: dump: 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:08.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 >> v1:172.21.15.165:6789/0 conn(0x7ff1bc006220 legacy=0x7ff1bc0086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 reap_dead start 2024-09-10T16:20:08.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 0x7ff1c4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): picked mon.c con 0x7ff1c4003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): picked mon.b con 0x7ff1c4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1c400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): picked mon.a con 0x7ff1c4005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc06cf50 con 0x7ff1c4005410 2024-09-10T16:20:08.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff1c400c320 con 0x7ff1c4004800 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff1c400c520 con 0x7ff1c4003a20 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d959a640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1dad9d640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 msgr2=0x7ff1c400ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1dad9d640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1c400ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 0x7ff1c4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:08.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1c4003a20 auth_method 0 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): get_auth_request con 0x7ff1c4004800 auth_method 0 2024-09-10T16:20:08.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:08.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient(hunting): handle_auth_done global_id 154144 payload 293 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4004800 msgr2=0x7ff1c4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 msgr2=0x7ff1c400ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1c400ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient: _finish_hunting 0 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 monclient: found mon.c 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1c4003520 con 0x7ff1c4003a20 2024-09-10T16:20:08.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient: _finish_auth 0 2024-09-10T16:20:08.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 10 monclient: _check_auth_tickets 2024-09-10T16:20:08.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1da59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:08.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1e0d73640 5 monclient: authenticate success, global_id 154144 2024-09-10T16:20:08.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.622+0000 7ff1e0d73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:08.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1da59c640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 msgr2=0x7ff1c4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:08.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1da59c640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 0x7ff1c4003e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc003a10 tx=0x7ff1c4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bda6a8ef9aa2751a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:08.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc031070 con 0x7ff1c4003a20 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: dump: 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:08.642 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1e0d73640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1cc02b850 con 0x7ff1c4003a20 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:08.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 1 -- 172.21.15.164:0/2197281563 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc003610 con 0x7ff1c4003a20 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1e0d73640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1e0d73640 4 set_mon_vals no callback set 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.625+0000 7ff1d959a640 10 monclient: dump: 2024-09-10T16:20:08.644 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.645 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 10 monclient: shutdown 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:08.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 -- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 msgr2=0x7ff1c4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 0x7ff1c4003e20 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc003a10 tx=0x7ff1c4005830 comp rx=0 tx=0).stop 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 -- 172.21.15.164:0/2197281563 shutdown_connections 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1c400ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4004800 0x7ff1c4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/2197281563 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4003a20 0x7ff1c4003e20 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 -- 172.21.15.164:0/2197281563 >> 172.21.15.164:0/2197281563 conn(0x7ff1dc0aa900 msgr2=0x7ff1dc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:08.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.626+0000 7ff1e0d73640 1 -- 172.21.15.164:0/2197281563 shutdown_connections 2024-09-10T16:20:08.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.627+0000 7ff1e0d73640 1 -- 172.21.15.164:0/2197281563 wait complete. 2024-09-10T16:20:08.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.628+0000 7ff1e0d73640 10 monclient: build_initial_monmap 2024-09-10T16:20:08.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.628+0000 7ff1e0d73640 10 monclient: monmap: 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:08.649 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.628+0000 7ff1e0d73640 1 Processor -- start 2024-09-10T16:20:08.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.628+0000 7ff1e0d73640 1 -- start start 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.628+0000 7ff1e0d73640 10 monclient: init 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 0x7ff1dc1cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-b con 0x7ff1c4003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:08.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-c con 0x7ff1c4004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1dc0a1e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): picked mon.noname-a con 0x7ff1c4005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc1350f0 con 0x7ff1c4005410 2024-09-10T16:20:08.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc134f70 con 0x7ff1c4003a20 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff1dc135270 con 0x7ff1c4004800 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1e0d73640 10 monclient(hunting): authenticate will time out at 1587.116217s 2024-09-10T16:20:08.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1dad9d640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 msgr2=0x7ff1dc0a1e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1dad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1dc0a1e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55032/0 (socket says 172.21.15.164:55032) 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 0x7ff1dc1cce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/3834002835 learned_addr learned my addr 172.21.15.164:0/3834002835 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:08.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): get_auth_request con 0x7ff1c4004800 auth_method 0 2024-09-10T16:20:08.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:08.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:08.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:08.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): get_auth_request con 0x7ff1c4003a20 auth_method 0 2024-09-10T16:20:08.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:08.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:08.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:08.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:08.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:08.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:08.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:08.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:08.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient(hunting): handle_auth_done global_id 114515 payload 824 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 msgr2=0x7ff1dc1cce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 0x7ff1dc1cce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 msgr2=0x7ff1dc0a1e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1dc0a1e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:20:08.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 monclient: found mon.noname-c 2024-09-10T16:20:08.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:08.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/3834002835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc121f50 con 0x7ff1c4004800 2024-09-10T16:20:08.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.629+0000 7ff1da59c640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 0x7ff1dc1cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:08.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1d9d9b640 1 -- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 msgr2=0x7ff1dc1cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:08.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1d9d9b640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000f8b0 tx=0x7ff1d000fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:08.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00064f0 con 0x7ff1c4004800 2024-09-10T16:20:08.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:08.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:08.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: dump: 2024-09-10T16:20:08.661 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:08.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:08.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.662 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.662 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:08.662 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.663 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:08.663 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:08.663 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:08.663 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: _finish_auth 0 2024-09-10T16:20:08.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:08.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:08.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 5 monclient: authenticate success, global_id 114515 2024-09-10T16:20:08.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d0018070 con 0x7ff1c4004800 2024-09-10T16:20:08.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 10 monclient: _renew_subs 2024-09-10T16:20:08.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc122230 con 0x7ff1c4004800 2024-09-10T16:20:08.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:08.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d00049e0 con 0x7ff1c4004800 2024-09-10T16:20:08.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:08.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:08.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1caffd640 10 monclient: dump: 2024-09-10T16:20:08.666 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:08.666 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:08.666 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:08.667 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:08.668 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:08.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 10 monclient: _renew_subs 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1d8d99640 4 set_mon_vals no callback set 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.632+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc133eb0 con 0x7ff1c4004800 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.633+0000 7ff1e0d73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.633+0000 7ff1e0d73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:08.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.633+0000 7ff1e0d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.633+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1dc137ca0 con 0x7ff1c4004800 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.633+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff1d0004b80 con 0x7ff1c4004800 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.634+0000 7ff1caffd640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 0x7ff1bc040f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.634+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(465..465 src has 184..465) ==== 4767+0+0 (secure 0 0 0) 0x7ff1d0052d30 con 0x7ff1c4004800 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.634+0000 7ff1da59c640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 0x7ff1bc040f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:08.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.634+0000 7ff1da59c640 10 monclient: get_auth_request con 0x7ff1bc03ea60 auth_method 0 2024-09-10T16:20:08.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.637+0000 7ff1da59c640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 0x7ff1bc040f20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc031000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:08.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.638+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff1d001ccc0 con 0x7ff1c4004800 2024-09-10T16:20:08.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.638+0000 7ff1caffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:08.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.638+0000 7ff1caffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:08.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.884+0000 7ff1e0d73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:08.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1e0d73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1e0d73640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff1dc1216a0 con 0x7ff1c4004800 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1caffd640 1 -- 172.21.15.164:0/3834002835 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7ff1d0012350 con 0x7ff1c4004800 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1caffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.885+0000 7ff1caffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:08.887 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":136,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 msgr2=0x7ff1bc040f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 0x7ff1bc040f20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff1cc006fd0 tx=0x7ff1cc031000 comp rx=0 tx=0).stop 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 10 monclient: shutdown 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 msgr2=0x7ff1dc1cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff1d000f8b0 tx=0x7ff1d000fd70 comp rx=0 tx=0).stop 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 shutdown_connections 2024-09-10T16:20:08.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff1bc03ea60 0x7ff1bc040f20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff1c4005410 0x7ff1dc0a1e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff1c4004800 0x7ff1dc1cd360 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 --2- 172.21.15.164:0/3834002835 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff1c4003a20 0x7ff1dc1cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 >> 172.21.15.164:0/3834002835 conn(0x7ff1dc0aa900 msgr2=0x7ff1dc132f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 shutdown_connections 2024-09-10T16:20:08.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.887+0000 7ff1e0d73640 1 -- 172.21.15.164:0/3834002835 wait complete. 2024-09-10T16:20:08.900 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:20:08.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:20:08.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:20:08.963 INFO:tasks.rados.rados.0.smithi165.stdout:1985: done (4 left) 2024-09-10T16:20:08.963 INFO:tasks.rados.rados.0.smithi165.stdout:1986: done (3 left) 2024-09-10T16:20:08.963 INFO:tasks.rados.rados.0.smithi165.stdout:1987: done (2 left) 2024-09-10T16:20:08.963 INFO:tasks.rados.rados.0.smithi165.stdout:1988: done (1 left) 2024-09-10T16:20:08.963 INFO:tasks.rados.rados.0.smithi165.stdout:1989: done (0 left) 2024-09-10T16:20:08.964 INFO:tasks.rados.rados.0.smithi165.stdout:1990: snap_create 2024-09-10T16:20:09.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.998+0000 7f4505f91640 10 monclient: get_monmap_and_config 2024-09-10T16:20:09.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.998+0000 7f4505f91640 10 monclient: build_initial_monmap 2024-09-10T16:20:09.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.998+0000 7f4505f91640 10 monclient: monmap: 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:08.999596+0000 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:08.999596+0000 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:09.001 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:09.002 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:09.003 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 1 Processor -- start 2024-09-10T16:20:09.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 1 -- start start 2024-09-10T16:20:09.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 10 monclient: init 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4500137bb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f45001386a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:08.999+0000 7f4505f91640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-b con 0x7f45000a2e80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45000a1c90 con 0x7f4500137bb0 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45000a1f70 con 0x7f45001386a0 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45000a2250 con 0x7f45000a2e80 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f4505f91640 10 monclient(hunting): authenticate will time out at 1587.487044s 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44ff7fe640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f4500137bb0 legacy=0x7f4500137f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55036/0 (socket says 172.21.15.164:55036) 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 -- 172.21.15.164:0/1041496045 learned_addr learned my addr 172.21.15.164:0/1041496045 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): get_auth_request con 0x7f45000a2e80 auth_method 0 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient(hunting): handle_auth_done global_id 114518 payload 274 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 -- 172.21.15.164:0/1041496045 >> v1:172.21.15.165:6789/0 conn(0x7f45001386a0 legacy=0x7f450013d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 -- 172.21.15.164:0/1041496045 >> v1:172.21.15.164:6789/0 conn(0x7f4500137bb0 legacy=0x7f4500137f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:09.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 monclient: found mon.noname-b 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.000+0000 7f44fffff640 1 -- 172.21.15.164:0/1041496045 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45000a3330 con 0x7f45000a2e80 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fffff640 1 -- 172.21.15.164:0/1041496045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 msgr2=0x7f4500140870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fffff640 1 --2- 172.21.15.164:0/1041496045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f44f80098d0 tx=0x7f44f802a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba8262f722580e8c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 1 -- 172.21.15.164:0/1041496045 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8037070 con 0x7f45000a2e80 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: dump: 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.018 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.019 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.019 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.019 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.020 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.020 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:09.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 1 -- 172.21.15.164:0/1041496045 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f8032640 con 0x7f45000a2e80 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 5 monclient: authenticate success, global_id 114518 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 1 -- 172.21.15.164:0/1041496045 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f8032a10 con 0x7f45000a2e80 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 4 set_mon_vals no callback set 2024-09-10T16:20:09.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f44fe7fc640 10 monclient: dump: 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 10 monclient: shutdown 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 -- 172.21.15.164:0/1041496045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 msgr2=0x7f4500140870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 --2- 172.21.15.164:0/1041496045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f44f80098d0 tx=0x7f44f802a0a0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 -- 172.21.15.164:0/1041496045 shutdown_connections 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 --2- 172.21.15.164:0/1041496045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f45000a2e80 0x7f4500140870 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 -- 172.21.15.164:0/1041496045 >> 172.21.15.164:0/1041496045 conn(0x7f45000a7320 msgr2=0x7f45000a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 -- 172.21.15.164:0/1041496045 shutdown_connections 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.003+0000 7f4505f91640 1 -- 172.21.15.164:0/1041496045 wait complete. 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.004+0000 7f4505f91640 10 monclient: build_initial_monmap 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.004+0000 7f4505f91640 10 monclient: monmap: 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:09.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.004+0000 7f4505f91640 1 Processor -- start 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.004+0000 7f4505f91640 1 -- start start 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.004+0000 7f4505f91640 10 monclient: init 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:20:09.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45000a2e80 0x7f450013f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-a con 0x7f45000a2e80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-c con 0x7f4500137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45001386a0 0x7f450013cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): picked mon.noname-b con 0x7f45001386a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4500077060 con 0x7f45000a2e80 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4500076ee0 con 0x7f45001386a0 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4500140db0 con 0x7f4500137bb0 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:09.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44ff7fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45000a2e80 msgr2=0x7f450013f3a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f4505f91640 10 monclient(hunting): authenticate will time out at 1587.492479s 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44ff7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45000a2e80 0x7f450013f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55046/0 (socket says 172.21.15.164:55046) 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 -- 172.21.15.164:0/2342957499 learned_addr learned my addr 172.21.15.164:0/2342957499 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): get_auth_request con 0x7f4500137bb0 auth_method 0 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient(hunting): handle_auth_done global_id 114521 payload 824 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44fffff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45001386a0 0x7f450013cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45001386a0 msgr2=0x7f450013cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f45001386a0 0x7f450013cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45000a2e80 msgr2=0x7f450013f3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f45000a2e80 0x7f450013f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient: _finish_hunting 0 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 monclient: found mon.noname-c 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.005+0000 7f44feffd640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f450011d9a0 con 0x7f4500137bb0 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44feffd640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 msgr2=0x7f45001345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44feffd640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c460 tx=0x7f44f400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4016020 con 0x7f4500137bb0 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: dump: 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.028 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: _finish_auth 0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44f4012a80 con 0x7f4500137bb0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 5 monclient: authenticate success, global_id 114521 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 monclient: _renew_subs 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f450013fd30 con 0x7f4500137bb0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f4011380 con 0x7f4500137bb0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44e7fff640 10 monclient: dump: 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 monclient: _renew_subs 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f450013e890 con 0x7f4500137bb0 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f44fdffb640 4 set_mon_vals no callback set 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 0x7f450013d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.008+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f450013dbe0 con 0x7f450013d2c0 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 0x7f450013d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient: get_auth_request con 0x7f450013d2c0 auth_method 0 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f44f4019020 con 0x7f4500137bb0 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_done global_id 114524 payload 824 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44ff7fe640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f45001401c0 con 0x7f450013d2c0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44e7fff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f44ec03e1c0 0x7f44ec040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(466..466 src has 184..466) ==== 4739+0+0 (secure 0 0 0) 0x7f44f40528c0 con 0x7f4500137bb0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44fffff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f44ec03e1c0 0x7f44ec040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.009+0000 7f44fffff640 10 monclient: get_auth_request con 0x7f44ec03e1c0 auth_method 0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44ff7fe640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 msgr2=0x7f450013d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44ff7fe640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 0x7f450013d6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44f000bcc0 tx=0x7f44f0009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0005560 con 0x7f450013d2c0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 10 monclient: dump: 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f44f0005560 con 0x7f450013d2c0 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:09.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 msgr2=0x7f450013d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 0x7f450013d6a0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44f000bcc0 tx=0x7f44f0009f30 comp rx=0 tx=0).stop 2024-09-10T16:20:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 msgr2=0x7f450013d6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.011+0000 7f44e7fff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f450013d2c0 0x7f450013d6a0 secure :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f44f000bcc0 tx=0x7f44f0009f30 comp rx=0 tx=0).stop 2024-09-10T16:20:09.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.012+0000 7f44fffff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f44ec03e1c0 0x7f44ec040680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f45000a7ed0 tx=0x7f44f802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.017+0000 7f4505f91640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f4505f91640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 0x7f4500139530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f4505f91640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f45001cdda0 con 0x7f4500139120 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 0x7f4500139530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient: get_auth_request con 0x7f4500139120 auth_method 0 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 10 monclient(hunting): handle_auth_done global_id 114527 payload 824 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.018+0000 7f44ff7fe640 1 -- 172.21.15.164:0/2342957499 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f45001c5f10 con 0x7f4500139120 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44ff7fe640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 msgr2=0x7f4500139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44ff7fe640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 0x7f4500139530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44f0014000 tx=0x7f44f0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44f0019040 con 0x7f4500139120 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 10 monclient: dump: 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.037 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f44f001aa20 con 0x7f4500139120 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 msgr2=0x7f4500139530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 0x7f4500139530 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44f0014000 tx=0x7f44f0005dc0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 msgr2=0x7f4500139530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:09.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.020+0000 7f44e7fff640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500139120 0x7f4500139530 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f44f0014000 tx=0x7f44f0005dc0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 136, 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 229589, 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:09.039 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.040 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.041 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:09.042 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.043 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:09.044 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.046 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.047 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.048 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.049 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:20:09.050 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 7 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:09.051 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f44ec03e1c0 msgr2=0x7f44ec040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f44ec03e1c0 0x7f44ec040680 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f45000a7ed0 tx=0x7f44f802e000 comp rx=0 tx=0).stop 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 10 monclient: shutdown 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 msgr2=0x7f45001345e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f44fffff640 1 -- 172.21.15.164:0/2342957499 reap_dead start 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f44f400c460 tx=0x7f44f400c920 comp rx=0 tx=0).stop 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 shutdown_connections 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 --2- 172.21.15.164:0/2342957499 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4500137bb0 0x7f45001345e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 >> 172.21.15.164:0/2342957499 conn(0x7f45000a7320 msgr2=0x7f45001367c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:09.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 shutdown_connections 2024-09-10T16:20:09.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.021+0000 7f4505f91640 1 -- 172.21.15.164:0/2342957499 wait complete. 2024-09-10T16:20:09.054 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:20:09.054 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:20:09.054 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.150+0000 7feb81c39640 10 monclient: get_monmap_and_config 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.150+0000 7feb81c39640 10 monclient: build_initial_monmap 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.150+0000 7feb81c39640 10 monclient: monmap: 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:09.151754+0000 2024-09-10T16:20:09.152 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:09.151754+0000 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:09.153 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.151+0000 7feb81c39640 1 Processor -- start 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.151+0000 7feb81c39640 1 -- start start 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.151+0000 7feb81c39640 10 monclient: init 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb7c09c180 0x7feb7c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-a con 0x7feb7c09c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7feb7c09d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7feb7c13a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb7c09a430 con 0x7feb7c09d160 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb7b7fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7feb7c09c180 msgr2=0x7feb7c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb7c0a24e0 con 0x7feb7c13a5c0 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7feb7c0a2bf0 con 0x7feb7c09c180 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb7affd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7feb7c09d160 legacy=0x7feb7c09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb7b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7feb7c09c180 0x7feb7c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb81c39640 10 monclient(hunting): authenticate will time out at 1587.639293s 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb7bfff640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7feb7c13a5c0 0x7feb7c13c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:50622/0 (socket says 172.21.15.164:50622) 2024-09-10T16:20:09.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.152+0000 7feb7bfff640 1 -- 172.21.15.164:0/1294781633 learned_addr learned my addr 172.21.15.164:0/1294781633 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.154+0000 7feb7bfff640 1 -- 172.21.15.164:0/1294781633 >> v1:172.21.15.165:6789/0 conn(0x7feb7c13a5c0 legacy=0x7feb7c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2673052423 0 0) 0x7feb7c0a24e0 con 0x7feb7c13a5c0 2024-09-10T16:20:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:20:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): my global_id is 154150 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb64003520 con 0x7feb7c13a5c0 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3735551415 0 0) 0x7feb64003520 con 0x7feb7c13a5c0 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 >> v2:172.21.15.164:3300/0 conn(0x7feb7c09c180 msgr2=0x7feb7c09ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 --2- 172.21.15.164:0/1294781633 >> v2:172.21.15.164:3300/0 conn(0x7feb7c09c180 0x7feb7c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 >> v1:172.21.15.164:6789/0 conn(0x7feb7c09d160 legacy=0x7feb7c09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:20:09.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c09c560 con 0x7feb7c13a5c0 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7feb74001a60 con 0x7feb7c13a5c0 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb81c39640 5 monclient: authenticate success, global_id 154150 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb81c39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: dump: 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 >> v1:172.21.15.165:6789/0 conn(0x7feb7c13a5c0 legacy=0x7feb7c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64003a20 0x7feb64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): picked mon.a con 0x7feb64003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:09.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 0x7feb64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): picked mon.c con 0x7feb64004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb64005410 0x7feb6400ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): picked mon.b con 0x7feb64005410 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0a2bf0 con 0x7feb64003a20 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb6400c320 con 0x7feb64005410 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7b7fe640 1 -- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64003a20 msgr2=0x7feb64003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feb6400c520 con 0x7feb64004800 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7b7fe640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64003a20 0x7feb64003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:20:09.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb81c39640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 0x7feb64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): get_auth_request con 0x7feb64004800 auth_method 0 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb64005410 0x7feb6400ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): get_auth_request con 0x7feb64005410 auth_method 0 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient(hunting): handle_auth_done global_id 154150 payload 293 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 -- 172.21.15.164:0/1294781633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb64005410 msgr2=0x7feb6400ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb64005410 0x7feb6400ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 -- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64003a20 msgr2=0x7feb64003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64003a20 0x7feb64003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient: _finish_hunting 0 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 monclient: found mon.c 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7bfff640 1 -- 172.21.15.164:0/1294781633 reap_dead start 2024-09-10T16:20:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 1 -- 172.21.15.164:0/1294781633 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb64003520 con 0x7feb64004800 2024-09-10T16:20:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient: _finish_auth 0 2024-09-10T16:20:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:09.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.155+0000 7feb7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:09.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7affd640 1 -- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 msgr2=0x7feb64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7affd640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 0x7feb64004c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb700098f0 tx=0x7feb64005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f82f747d8657b57d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb700094f0 con 0x7feb64004800 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: dump: 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.172 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb81c39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7002e070 con 0x7feb64004800 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb81c39640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb81c39640 4 set_mon_vals no callback set 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 1 -- 172.21.15.164:0/1294781633 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70003be0 con 0x7feb64004800 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.158+0000 7feb7a7fc640 10 monclient: dump: 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.173 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 10 monclient: shutdown 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 -- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 msgr2=0x7feb64004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 0x7feb64004c80 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7feb700098f0 tx=0x7feb64005830 comp rx=0 tx=0).stop 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 -- 172.21.15.164:0/1294781633 shutdown_connections 2024-09-10T16:20:09.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 --2- 172.21.15.164:0/1294781633 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb64004800 0x7feb64004c80 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 -- 172.21.15.164:0/1294781633 >> 172.21.15.164:0/1294781633 conn(0x7feb7c0a3110 msgr2=0x7feb7c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 -- 172.21.15.164:0/1294781633 shutdown_connections 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.159+0000 7feb81c39640 1 -- 172.21.15.164:0/1294781633 wait complete. 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.160+0000 7feb81c39640 10 monclient: build_initial_monmap 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.160+0000 7feb81c39640 10 monclient: monmap: 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:09.175 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.160+0000 7feb81c39640 1 Processor -- start 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.160+0000 7feb81c39640 1 -- start start 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.160+0000 7feb81c39640 10 monclient: init 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64004800 0x7feb7c08e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-a con 0x7feb64004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 0x7feb7c08f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-c con 0x7feb7c08ed30 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7b7fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64004800 msgr2=0x7feb7c08e7f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c1cd770 0x7feb7c1cfb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): picked mon.noname-b con 0x7feb7c1cd770 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c074f90 con 0x7feb64004800 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c074e10 con 0x7feb7c1cd770 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64004800 0x7feb7c08e7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c09d930 con 0x7feb7c08ed30 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb63fff640 1 -- DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb81c39640 10 monclient(hunting): authenticate will time out at 1587.648194s 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 0x7feb7c08f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 0x7feb7c08f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55082/0 (socket says 172.21.15.164:55082) 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 -- 172.21.15.164:0/746091654 learned_addr learned my addr 172.21.15.164:0/746091654 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:09.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): get_auth_request con 0x7feb7c08ed30 auth_method 0 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient(hunting): handle_auth_done global_id 114530 payload 824 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7bfff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c1cd770 0x7feb7c1cfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c1cd770 msgr2=0x7feb7c1cfb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c1cd770 0x7feb7c1cfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64004800 msgr2=0x7feb7c08e7f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7feb64004800 0x7feb7c08e7f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient: _finish_hunting 0 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 monclient: found mon.noname-c 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.161+0000 7feb7affd640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c091f50 con 0x7feb7c08ed30 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb7affd640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 msgr2=0x7feb7c08f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb7affd640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 0x7feb7c08f190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb7002a7b0 tx=0x7feb70005a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70043070 con 0x7feb7c08ed30 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: dump: 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.179 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: _finish_auth 0 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb70005690 con 0x7feb7c08ed30 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb81c39640 5 monclient: authenticate success, global_id 114530 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb81c39640 10 monclient: _renew_subs 2024-09-10T16:20:09.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb81c39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c092230 con 0x7feb7c08ed30 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7003e6b0 con 0x7feb7c08ed30 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb63fff640 10 monclient: dump: 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.181 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 10 monclient: _renew_subs 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.163+0000 7feb79ffb640 4 set_mon_vals no callback set 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c0901b0 con 0x7feb7c08ed30 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 0x7feb7c063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c02e9a0 con 0x7feb7c02e5c0 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.164+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feb70033470 con 0x7feb7c08ed30 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb63fff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feb6803e1c0 0x7feb68040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 0x7feb7c063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient: get_auth_request con 0x7feb7c02e5c0 auth_method 0 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(466..466 src has 184..466) ==== 4739+0+0 (secure 0 0 0) 0x7feb700728f0 con 0x7feb7c08ed30 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7bfff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feb6803e1c0 0x7feb68040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7bfff640 10 monclient: get_auth_request con 0x7feb6803e1c0 auth_method 0 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_done global_id 154153 payload 824 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.165+0000 7feb7b7fe640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb7c0a48a0 con 0x7feb7c02e5c0 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.167+0000 7feb7bfff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feb6803e1c0 0x7feb68040680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feb7c09ab20 tx=0x7feb74009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.167+0000 7feb7b7fe640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 msgr2=0x7feb7c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.167+0000 7feb7b7fe640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 0x7feb7c063d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00bd00 tx=0x7feb6c009f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c015670 con 0x7feb7c02e5c0 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 10 monclient: dump: 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.184 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb6c010070 con 0x7feb7c02e5c0 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 msgr2=0x7feb7c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 0x7feb7c063d50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00bd00 tx=0x7feb6c009f70 comp rx=0 tx=0).stop 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 msgr2=0x7feb7c063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:09.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.168+0000 7feb63fff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c02e5c0 0x7feb7c063d50 secure :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00bd00 tx=0x7feb6c009f70 comp rx=0 tx=0).stop 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb81c39640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb81c39640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 0x7feb7c1d40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb81c39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0a5590 con 0x7feb7c064290 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 0x7feb7c1d40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient: get_auth_request con 0x7feb7c064290 auth_method 0 2024-09-10T16:20:09.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 10 monclient(hunting): handle_auth_done global_id 154156 payload 824 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.174+0000 7feb7b7fe640 1 -- 172.21.15.164:0/746091654 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb7c09dcc0 con 0x7feb7c064290 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb7b7fe640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 msgr2=0x7feb7c1d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb7b7fe640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 0x7feb7c1d40c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00bd00 tx=0x7feb6c005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb6c01a980 con 0x7feb7c064290 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:09.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.177+0000 7feb63fff640 10 monclient: dump: 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7feb6c01a980 con 0x7feb7c064290 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:09.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 msgr2=0x7feb7c1d40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 0x7feb7c1d40c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00bd00 tx=0x7feb6c005de0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 msgr2=0x7feb7c1d40c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb63fff640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7feb7c064290 0x7feb7c1d40c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:20:09.190 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 136, 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27728, 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.191 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.192 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:09.193 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:09.194 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:09.195 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.196 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:09.197 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.198 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:09.199 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:09.200 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.201 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:09.202 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feb6803e1c0 msgr2=0x7feb68040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7feb6803e1c0 0x7feb68040680 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feb7c09ab20 tx=0x7feb74009040 comp rx=0 tx=0).stop 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 10 monclient: shutdown 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 msgr2=0x7feb7c08f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 --2- 172.21.15.164:0/746091654 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7feb7c08ed30 0x7feb7c08f190 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7feb7002a7b0 tx=0x7feb70005a00 comp rx=0 tx=0).stop 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb7bfff640 1 -- 172.21.15.164:0/746091654 reap_dead start 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb7bfff640 1 -- 172.21.15.164:0/746091654 reap_dead start 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 shutdown_connections 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.178+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 >> 172.21.15.164:0/746091654 conn(0x7feb7c0a3110 msgr2=0x7feb7c1387d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.179+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 shutdown_connections 2024-09-10T16:20:09.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:09.179+0000 7feb81c39640 1 -- 172.21.15.164:0/746091654 wait complete. 2024-09-10T16:20:09.205 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:20:09.205 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:20:09.963 INFO:tasks.rados.rados.0.smithi165.stdout:1990: done (0 left) 2024-09-10T16:20:09.963 INFO:tasks.rados.rados.0.smithi165.stdout:1991: copy_from oid 28 from oid 49 current snap is 220 2024-09-10T16:20:09.963 INFO:tasks.rados.rados.0.smithi165.stdout:1992: read oid 22 snap -1 2024-09-10T16:20:09.963 INFO:tasks.rados.rados.0.smithi165.stdout:1992: expect (ObjNum 682 snap 208 seq_num 682) 2024-09-10T16:20:09.964 INFO:tasks.rados.rados.0.smithi165.stdout:1993: copy_from oid 17 from oid 15 current snap is 220 2024-09-10T16:20:09.964 INFO:tasks.rados.rados.0.smithi165.stdout:1994: rollback oid 25 current snap is 220 2024-09-10T16:20:09.964 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 219 2024-09-10T16:20:09.964 INFO:tasks.rados.rados.0.smithi165.stdout:1995: delete oid 41 current snap is 220 2024-09-10T16:20:09.966 INFO:tasks.rados.rados.0.smithi165.stdout:1993: finishing copy_from to smithi16534841-17 2024-09-10T16:20:09.966 INFO:tasks.rados.rados.0.smithi165.stdout:1993: got expected ENOENT (src dne) 2024-09-10T16:20:09.966 INFO:tasks.rados.rados.0.smithi165.stdout:1993: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:20:09.971 INFO:tasks.rados.rados.0.smithi165.stdout:1994: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:20:09.971 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1194 (ObjNum 1025 snap 0 seq_num 4609) dirty dne 2024-09-10T16:20:09.972 INFO:tasks.rados.rados.0.smithi165.stdout:1993: done (4 left) 2024-09-10T16:20:09.972 INFO:tasks.rados.rados.0.smithi165.stdout:1994: done (3 left) 2024-09-10T16:20:09.972 INFO:tasks.rados.rados.0.smithi165.stdout:1995: done (2 left) 2024-09-10T16:20:09.972 INFO:tasks.rados.rados.0.smithi165.stdout:1996: write oid 3 current snap is 220 2024-09-10T16:20:09.973 INFO:tasks.rados.rados.0.smithi165.stdout:1996: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2024-09-10T16:20:09.980 INFO:tasks.rados.rados.0.smithi165.stdout:1996: writing smithi16534841-3 from 766584 to 1564701 tid 1 2024-09-10T16:20:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:1996: writing smithi16534841-3 from 2098238 to 2713812 tid 2 2024-09-10T16:20:09.992 INFO:tasks.rados.rados.0.smithi165.stdout:1996: writing smithi16534841-3 from 3169759 to 3630138 tid 3 2024-09-10T16:20:09.993 INFO:tasks.rados.rados.0.smithi165.stdout:1997: delete oid 21 current snap is 220 2024-09-10T16:20:09.993 INFO:tasks.rados.rados.0.smithi165.stdout:1991: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:20:09.993 INFO:tasks.rados.rados.0.smithi165.stdout:1992: expect (ObjNum 682 snap 208 seq_num 682) 2024-09-10T16:20:09.993 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:10.192 INFO:tasks.rados.rados.0.smithi165.stdout:1991: finishing copy_from to smithi16534841-28 2024-09-10T16:20:10.192 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1097 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:20:10.192 INFO:tasks.rados.rados.0.smithi165.stdout:1996: finishing write tid 1 to smithi16534841-3 2024-09-10T16:20:10.192 INFO:tasks.rados.rados.0.smithi165.stdout:1996: finishing write tid 2 to smithi16534841-3 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: finishing write tid 3 to smithi16534841-3 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: finishing write tid 4 to smithi16534841-3 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: finishing write tid 5 to smithi16534841-3 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: oid 3 updating version 0 to 1200 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: oid 3 updating version 1200 to 1202 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: oid 3 version 1202 is already newer than 1201 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: oid 3 updating version 1202 to 1203 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1203 (ObjNum 712 snap 220 seq_num 712) dirty exists 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: left oid 3 (ObjNum 712 snap 220 seq_num 712) 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1991: done (3 left) 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1992: done (2 left) 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1996: done (1 left) 2024-09-10T16:20:10.193 INFO:tasks.rados.rados.0.smithi165.stdout:1997: done (0 left) 2024-09-10T16:20:10.194 INFO:tasks.rados.rados.0.smithi165.stdout:1998: rollback oid 21 current snap is 220 2024-09-10T16:20:10.194 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 216 2024-09-10T16:20:10.194 INFO:tasks.rados.rados.0.smithi165.stdout:1999: write (excl) oid 34 current snap is 220 2024-09-10T16:20:10.194 INFO:tasks.rados.rados.0.smithi165.stdout:1999: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-10T16:20:10.197 INFO:tasks.rados.rados.0.smithi165.stdout:1999: writing smithi16534841-34 from 400198 to 948813 tid 1 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:1999: writing smithi16534841-34 from 1415202 to 2000198 tid 2 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:2000: snap_create 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:1998: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:1998: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1206 (ObjNum 690 snap 213 seq_num 690) dirty exists 2024-09-10T16:20:10.203 INFO:tasks.rados.rados.0.smithi165.stdout:1999: finishing write tid 1 to smithi16534841-34 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: finishing write tid 2 to smithi16534841-34 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: finishing write tid 3 to smithi16534841-34 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: finishing write tid 4 to smithi16534841-34 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: oid 34 updating version 0 to 1137 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: oid 34 updating version 1137 to 1138 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:1999: oid 34 updating version 1138 to 1139 2024-09-10T16:20:10.206 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1139 (ObjNum 714 snap 220 seq_num 714) dirty exists 2024-09-10T16:20:10.207 INFO:tasks.rados.rados.0.smithi165.stdout:1999: left oid 34 (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:20:10.971 INFO:tasks.rados.rados.0.smithi165.stdout:1998: done (2 left) 2024-09-10T16:20:10.971 INFO:tasks.rados.rados.0.smithi165.stdout:1999: done (1 left) 2024-09-10T16:20:10.971 INFO:tasks.rados.rados.0.smithi165.stdout:2000: done (0 left) 2024-09-10T16:20:10.971 INFO:tasks.rados.rados.0.smithi165.stdout:2001: snap_remove snap 216 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:2001: done (0 left) 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:2002: rollback oid 23 current snap is 221 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 214 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:2003: read oid 2 snap -1 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:2003: expect (ObjNum 683 snap 208 seq_num 683) 2024-09-10T16:20:11.471 INFO:tasks.rados.rados.0.smithi165.stdout:2004: delete oid 36 current snap is 221 2024-09-10T16:20:11.476 INFO:tasks.rados.rados.0.smithi165.stdout:2004: done (2 left) 2024-09-10T16:20:11.476 INFO:tasks.rados.rados.0.smithi165.stdout:2005: read oid 26 snap -1 2024-09-10T16:20:11.477 INFO:tasks.rados.rados.0.smithi165.stdout:2005: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:20:11.477 INFO:tasks.rados.rados.0.smithi165.stdout:2006: rollback oid 20 current snap is 221 2024-09-10T16:20:11.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 214 2024-09-10T16:20:11.477 INFO:tasks.rados.rados.0.smithi165.stdout:2007: write oid 46 current snap is 221 2024-09-10T16:20:11.477 INFO:tasks.rados.rados.0.smithi165.stdout:2007: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-10T16:20:11.484 INFO:tasks.rados.rados.0.smithi165.stdout:2007: writing smithi16534841-46 from 433812 to 849278 tid 1 2024-09-10T16:20:11.492 INFO:tasks.rados.rados.0.smithi165.stdout:2007: writing smithi16534841-46 from 1648519 to 2033812 tid 2 2024-09-10T16:20:11.492 INFO:tasks.rados.rados.0.smithi165.stdout:2008: rollback oid 21 current snap is 221 2024-09-10T16:20:11.492 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 217 2024-09-10T16:20:11.492 INFO:tasks.rados.rados.0.smithi165.stdout:2003: expect (ObjNum 683 snap 208 seq_num 683) 2024-09-10T16:20:11.573 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2834702 2024-09-10T16:20:11.643 INFO:tasks.rados.rados.0.smithi165.stdout:2002: finishing rollback tid 0 to smithi16534841-23 2024-09-10T16:20:11.643 INFO:tasks.rados.rados.0.smithi165.stdout:2002: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:20:11.643 INFO:tasks.rados.rados.0.smithi165.stdout:2005: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:20:11.686 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:11.752 INFO:tasks.rados.rados.0.smithi165.stdout:2002: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1155 (ObjNum 668 snap 207 seq_num 668) dirty exists 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2006: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: finishing write tid 1 to smithi16534841-46 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: finishing write tid 2 to smithi16534841-46 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: finishing write tid 3 to smithi16534841-46 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: finishing write tid 4 to smithi16534841-46 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: oid 46 updating version 0 to 651 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: oid 46 version 651 is already newer than 649 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: oid 46 version 651 is already newer than 650 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 651 (ObjNum 716 snap 221 seq_num 716) dirty exists 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2007: left oid 46 (ObjNum 716 snap 221 seq_num 716) 2024-09-10T16:20:11.753 INFO:tasks.rados.rados.0.smithi165.stdout:2006: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2002: done (5 left) 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2003: done (4 left) 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2005: done (3 left) 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2007: done (2 left) 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2009: delete oid 45 current snap is 221 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2006: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1209 (ObjNum 692 snap 213 seq_num 692) dirty exists 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2008: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2008: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:2008: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:20:11.754 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1212 (ObjNum 690 snap 213 seq_num 690) dirty exists 2024-09-10T16:20:11.755 INFO:tasks.rados.rados.0.smithi165.stdout:2006: done (2 left) 2024-09-10T16:20:11.755 INFO:tasks.rados.rados.0.smithi165.stdout:2008: done (1 left) 2024-09-10T16:20:11.755 INFO:tasks.rados.rados.0.smithi165.stdout:2009: done (0 left) 2024-09-10T16:20:11.755 INFO:tasks.rados.rados.0.smithi165.stdout:2010: delete oid 20 current snap is 221 2024-09-10T16:20:11.757 INFO:tasks.rados.rados.0.smithi165.stdout:2010: done (0 left) 2024-09-10T16:20:11.757 INFO:tasks.rados.rados.0.smithi165.stdout:2011: read oid 16 snap -1 2024-09-10T16:20:11.757 INFO:tasks.rados.rados.0.smithi165.stdout:2011: expect (ObjNum 707 snap 216 seq_num 707) 2024-09-10T16:20:11.757 INFO:tasks.rados.rados.0.smithi165.stdout:2012: copy_from oid 21 from oid 5 current snap is 221 2024-09-10T16:20:11.758 INFO:tasks.rados.rados.0.smithi165.stdout:2013: read oid 35 snap 217 2024-09-10T16:20:11.758 INFO:tasks.rados.rados.0.smithi165.stdout:2013: expect deleted 2024-09-10T16:20:11.758 INFO:tasks.rados.rados.0.smithi165.stdout:2014: write oid 18 current snap is 221 2024-09-10T16:20:11.758 INFO:tasks.rados.rados.0.smithi165.stdout:2014: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-10T16:20:11.763 INFO:tasks.rados.rados.0.smithi165.stdout:2014: writing smithi16534841-18 from 484233 to 1125446 tid 1 2024-09-10T16:20:11.773 INFO:tasks.rados.rados.0.smithi165.stdout:2014: writing smithi16534841-18 from 1707612 to 2084233 tid 2 2024-09-10T16:20:11.773 INFO:tasks.rados.rados.0.smithi165.stdout:2015: read oid 3 snap -1 2024-09-10T16:20:11.773 INFO:tasks.rados.rados.0.smithi165.stdout:2015: expect (ObjNum 712 snap 220 seq_num 712) 2024-09-10T16:20:11.773 INFO:tasks.rados.rados.0.smithi165.stdout:2016: delete oid 1 current snap is 221 2024-09-10T16:20:11.773 INFO:tasks.rados.rados.0.smithi165.stdout:2011: expect (ObjNum 707 snap 216 seq_num 707) 2024-09-10T16:20:11.922 INFO:tasks.rados.rados.0.smithi165.stdout:2014: finishing write tid 1 to smithi16534841-18 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: finishing write tid 2 to smithi16534841-18 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: finishing write tid 3 to smithi16534841-18 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2012: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: finishing write tid 4 to smithi16534841-18 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: oid 18 updating version 0 to 1156 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: oid 18 updating version 1156 to 1158 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: oid 18 version 1158 is already newer than 1157 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1158 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: left oid 18 (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2011: done (5 left) 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2013: done (4 left) 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2014: done (3 left) 2024-09-10T16:20:11.923 INFO:tasks.rados.rados.0.smithi165.stdout:2016: done (2 left) 2024-09-10T16:20:11.924 INFO:tasks.rados.rados.0.smithi165.stdout:2017: write (excl) oid 14 current snap is 221 2024-09-10T16:20:11.924 INFO:tasks.rados.rados.0.smithi165.stdout:2012: finishing copy_from to smithi16534841-21 2024-09-10T16:20:11.924 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 651 (ObjNum 698 snap 215 seq_num 698) dirty exists 2024-09-10T16:20:11.924 INFO:tasks.rados.rados.0.smithi165.stdout:2015: expect (ObjNum 712 snap 220 seq_num 712) 2024-09-10T16:20:11.946 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2490145 2024-09-10T16:20:12.096 INFO:tasks.rados.rados.0.smithi165.stdout:2017: seq_num 721 ranges {517847=791711,2108223=9624} 2024-09-10T16:20:12.103 INFO:tasks.rados.rados.0.smithi165.stdout:2017: writing smithi16534841-14 from 517847 to 1309558 tid 1 2024-09-10T16:20:12.108 INFO:tasks.rados.rados.0.smithi165.stdout:2017: writing smithi16534841-14 from 2108223 to 2117847 tid 2 2024-09-10T16:20:12.108 INFO:tasks.rados.rados.0.smithi165.stdout:2012: done (2 left) 2024-09-10T16:20:12.108 INFO:tasks.rados.rados.0.smithi165.stdout:2015: done (1 left) 2024-09-10T16:20:12.108 INFO:tasks.rados.rados.0.smithi165.stdout:2018: delete oid 41 current snap is 221 2024-09-10T16:20:12.109 INFO:tasks.rados.rados.0.smithi165.stdout:2017: finishing write tid 1 to smithi16534841-14 2024-09-10T16:20:12.109 INFO:tasks.rados.rados.0.smithi165.stdout:2018: done (1 left) 2024-09-10T16:20:12.110 INFO:tasks.rados.rados.0.smithi165.stdout:2019: delete oid 5 current snap is 221 2024-09-10T16:20:12.110 INFO:tasks.rados.rados.0.smithi165.stdout:2017: finishing write tid 2 to smithi16534841-14 2024-09-10T16:20:12.111 INFO:tasks.rados.rados.0.smithi165.stdout:2017: finishing write tid 3 to smithi16534841-14 2024-09-10T16:20:12.111 INFO:tasks.rados.rados.0.smithi165.stdout:2017: finishing write tid 4 to smithi16534841-14 2024-09-10T16:20:12.111 INFO:tasks.rados.rados.0.smithi165.stdout:2017: oid 14 updating version 0 to 1162 2024-09-10T16:20:12.112 INFO:tasks.rados.rados.0.smithi165.stdout:2017: oid 14 version 1162 is already newer than 1160 2024-09-10T16:20:12.112 INFO:tasks.rados.rados.0.smithi165.stdout:2017: oid 14 version 1162 is already newer than 1161 2024-09-10T16:20:12.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1162 (ObjNum 721 snap 221 seq_num 721) dirty exists 2024-09-10T16:20:12.112 INFO:tasks.rados.rados.0.smithi165.stdout:2017: left oid 14 (ObjNum 721 snap 221 seq_num 721) 2024-09-10T16:20:12.114 INFO:tasks.rados.rados.0.smithi165.stdout:2017: done (1 left) 2024-09-10T16:20:12.114 INFO:tasks.rados.rados.0.smithi165.stdout:2019: done (0 left) 2024-09-10T16:20:12.114 INFO:tasks.rados.rados.0.smithi165.stdout:2020: write oid 12 current snap is 221 2024-09-10T16:20:12.114 INFO:tasks.rados.rados.0.smithi165.stdout:2020: seq_num 724 ranges {568268=733811,2000022=168246} 2024-09-10T16:20:12.121 INFO:tasks.rados.rados.0.smithi165.stdout:2020: writing smithi16534841-12 from 568268 to 1302079 tid 1 2024-09-10T16:20:12.126 INFO:tasks.rados.rados.0.smithi165.stdout:2020: writing smithi16534841-12 from 2000022 to 2168268 tid 2 2024-09-10T16:20:12.126 INFO:tasks.rados.rados.0.smithi165.stdout:2021: copy_from oid 36 from oid 18 current snap is 221 2024-09-10T16:20:12.126 INFO:tasks.rados.rados.0.smithi165.stdout:2020: finishing write tid 1 to smithi16534841-12 2024-09-10T16:20:12.126 INFO:tasks.rados.rados.0.smithi165.stdout:2022: copy_from oid 20 from oid 9 current snap is 221 2024-09-10T16:20:12.126 INFO:tasks.rados.rados.0.smithi165.stdout:2023: write (excl) oid 48 current snap is 221 2024-09-10T16:20:12.127 INFO:tasks.rados.rados.0.smithi165.stdout:2023: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-10T16:20:12.131 INFO:tasks.rados.rados.0.smithi165.stdout:2023: writing smithi16534841-48 from 585075 to 994135 tid 1 2024-09-10T16:20:12.139 INFO:tasks.rados.rados.0.smithi165.stdout:2023: writing smithi16534841-48 from 1542151 to 2185075 tid 2 2024-09-10T16:20:12.139 INFO:tasks.rados.rados.0.smithi165.stdout:2024: delete oid 43 current snap is 221 2024-09-10T16:20:12.139 INFO:tasks.rados.rados.0.smithi165.stdout:2020: finishing write tid 2 to smithi16534841-12 2024-09-10T16:20:12.139 INFO:tasks.rados.rados.0.smithi165.stdout:2020: finishing write tid 3 to smithi16534841-12 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2020: finishing write tid 4 to smithi16534841-12 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2020: oid 12 updating version 0 to 1095 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2020: oid 12 updating version 1095 to 1096 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2020: oid 12 version 1096 is already newer than 1094 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1096 (ObjNum 724 snap 221 seq_num 724) dirty exists 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2020: left oid 12 (ObjNum 724 snap 221 seq_num 724) 2024-09-10T16:20:12.140 INFO:tasks.rados.rados.0.smithi165.stdout:2023: finishing write tid 1 to smithi16534841-48 2024-09-10T16:20:12.141 INFO:tasks.rados.rados.0.smithi165.stdout:2021: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:20:12.141 INFO:tasks.rados.rados.0.smithi165.stdout:2022: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:20:12.144 INFO:tasks.rados.rados.0.smithi165.stdout:2021: finishing copy_from to smithi16534841-36 2024-09-10T16:20:12.144 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1158 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:20:12.146 INFO:tasks.rados.rados.0.smithi165.stdout:2023: finishing write tid 2 to smithi16534841-48 2024-09-10T16:20:12.146 INFO:tasks.rados.rados.0.smithi165.stdout:2023: finishing write tid 3 to smithi16534841-48 2024-09-10T16:20:12.146 INFO:tasks.rados.rados.0.smithi165.stdout:2023: finishing write tid 4 to smithi16534841-48 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2023: oid 48 updating version 0 to 1303 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2023: oid 48 updating version 1303 to 1304 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2023: oid 48 updating version 1304 to 1305 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1305 (ObjNum 725 snap 221 seq_num 725) dirty exists 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2023: left oid 48 (ObjNum 725 snap 221 seq_num 725) 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2020: done (4 left) 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2021: done (3 left) 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2023: done (2 left) 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2024: done (1 left) 2024-09-10T16:20:12.147 INFO:tasks.rados.rados.0.smithi165.stdout:2025: copy_from oid 32 from oid 16 current snap is 221 2024-09-10T16:20:12.148 INFO:tasks.rados.rados.0.smithi165.stdout:2026: rollback oid 24 current snap is 221 2024-09-10T16:20:12.148 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 220 2024-09-10T16:20:12.148 INFO:tasks.rados.rados.0.smithi165.stdout:2027: copy_from oid 35 from oid 41 current snap is 221 2024-09-10T16:20:12.148 INFO:tasks.rados.rados.0.smithi165.stdout:2028: copy_from oid 45 from oid 26 current snap is 221 2024-09-10T16:20:12.148 INFO:tasks.rados.rados.0.smithi165.stdout:2029: snap_remove snap 214 2024-09-10T16:20:12.979 INFO:tasks.rados.rados.0.smithi165.stdout:2029: done (5 left) 2024-09-10T16:20:12.979 INFO:tasks.rados.rados.0.smithi165.stdout:2030: read oid 27 snap -1 2024-09-10T16:20:12.979 INFO:tasks.rados.rados.0.smithi165.stdout:2030: expect (ObjNum 699 snap 215 seq_num 699) 2024-09-10T16:20:12.979 INFO:tasks.rados.rados.0.smithi165.stdout:2031: write oid 25 current snap is 221 2024-09-10T16:20:12.980 INFO:tasks.rados.rados.0.smithi165.stdout:2031: seq_num 727 ranges {618689=559558,1659115=559574} 2024-09-10T16:20:12.986 INFO:tasks.rados.rados.0.smithi165.stdout:2031: writing smithi16534841-25 from 618689 to 1178247 tid 1 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2031: writing smithi16534841-25 from 1659115 to 2218689 tid 2 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2032: snap_create 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2022: finishing copy_from to smithi16534841-20 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1159 (ObjNum 677 snap 207 seq_num 677) dirty exists 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2027: finishing copy_from to smithi16534841-35 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2027: got expected ENOENT (src dne) 2024-09-10T16:20:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2027: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2026: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 0 (ObjNum 129 snap 0 seq_num 1160024320) dirty dne 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2028: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2025: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2025: finishing copy_from to smithi16534841-32 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1300 (ObjNum 707 snap 216 seq_num 707) dirty exists 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2028: finishing copy_from to smithi16534841-45 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1287 (ObjNum 585 snap 186 seq_num 585) dirty exists 2024-09-10T16:20:12.994 INFO:tasks.rados.rados.0.smithi165.stdout:2030: expect (ObjNum 699 snap 215 seq_num 699) 2024-09-10T16:20:13.092 INFO:tasks.rados.rados.0.smithi165.stdout:2031: finishing write tid 1 to smithi16534841-25 2024-09-10T16:20:13.092 INFO:tasks.rados.rados.0.smithi165.stdout:2031: finishing write tid 2 to smithi16534841-25 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: finishing write tid 3 to smithi16534841-25 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: finishing write tid 4 to smithi16534841-25 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: oid 25 updating version 0 to 1214 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: oid 25 updating version 1214 to 1216 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: oid 25 version 1216 is already newer than 1215 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1216 (ObjNum 727 snap 221 seq_num 727) dirty exists 2024-09-10T16:20:13.093 INFO:tasks.rados.rados.0.smithi165.stdout:2031: left oid 25 (ObjNum 727 snap 221 seq_num 727) 2024-09-10T16:20:14.201 INFO:tasks.rados.rados.0.smithi165.stdout:2022: done (7 left) 2024-09-10T16:20:14.201 INFO:tasks.rados.rados.0.smithi165.stdout:2025: done (6 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2026: done (5 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2027: done (4 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2028: done (3 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2030: done (2 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2031: done (1 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2032: done (0 left) 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2033: write oid 27 current snap is 222 2024-09-10T16:20:14.202 INFO:tasks.rados.rados.0.smithi165.stdout:2033: seq_num 728 ranges {635496=634807,1717597=517899} 2024-09-10T16:20:14.208 INFO:tasks.rados.rados.0.smithi165.stdout:2033: writing smithi16534841-27 from 635496 to 1270303 tid 1 2024-09-10T16:20:14.213 INFO:tasks.rados.rados.0.smithi165.stdout:2033: writing smithi16534841-27 from 1717597 to 2235496 tid 2 2024-09-10T16:20:14.214 INFO:tasks.rados.rados.0.smithi165.stdout:2034: snap_create 2024-09-10T16:20:14.217 INFO:tasks.rados.rados.0.smithi165.stdout:2033: finishing write tid 1 to smithi16534841-27 2024-09-10T16:20:14.217 INFO:tasks.rados.rados.0.smithi165.stdout:2033: finishing write tid 2 to smithi16534841-27 2024-09-10T16:20:14.218 INFO:tasks.rados.rados.0.smithi165.stdout:2033: finishing write tid 3 to smithi16534841-27 2024-09-10T16:20:14.218 INFO:tasks.rados.rados.0.smithi165.stdout:2033: finishing write tid 4 to smithi16534841-27 2024-09-10T16:20:14.218 INFO:tasks.rados.rados.0.smithi165.stdout:2033: oid 27 updating version 0 to 1221 2024-09-10T16:20:14.218 INFO:tasks.rados.rados.0.smithi165.stdout:2033: oid 27 updating version 1221 to 1222 2024-09-10T16:20:14.218 INFO:tasks.rados.rados.0.smithi165.stdout:2033: oid 27 updating version 1222 to 1223 2024-09-10T16:20:14.219 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1223 (ObjNum 728 snap 222 seq_num 728) dirty exists 2024-09-10T16:20:14.219 INFO:tasks.rados.rados.0.smithi165.stdout:2033: left oid 27 (ObjNum 728 snap 222 seq_num 728) 2024-09-10T16:20:15.205 INFO:tasks.rados.rados.0.smithi165.stdout:2033: done (1 left) 2024-09-10T16:20:15.206 INFO:tasks.rados.rados.0.smithi165.stdout:2034: done (0 left) 2024-09-10T16:20:15.206 INFO:tasks.rados.rados.0.smithi165.stdout:2035: copy_from oid 47 from oid 48 current snap is 223 2024-09-10T16:20:15.206 INFO:tasks.rados.rados.0.smithi165.stdout:2036: write oid 5 current snap is 223 2024-09-10T16:20:15.206 INFO:tasks.rados.rados.0.smithi165.stdout:2036: seq_num 729 ranges {652303=710056,2059726=192577} 2024-09-10T16:20:15.213 INFO:tasks.rados.rados.0.smithi165.stdout:2036: writing smithi16534841-5 from 652303 to 1362359 tid 1 2024-09-10T16:20:15.218 INFO:tasks.rados.rados.0.smithi165.stdout:2036: writing smithi16534841-5 from 2059726 to 2252303 tid 2 2024-09-10T16:20:15.219 INFO:tasks.rados.rados.0.smithi165.stdout:2037: delete oid 4 current snap is 223 2024-09-10T16:20:15.219 INFO:tasks.rados.rados.0.smithi165.stdout:2036: finishing write tid 1 to smithi16534841-5 2024-09-10T16:20:15.220 INFO:tasks.rados.rados.0.smithi165.stdout:2035: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:20:15.221 INFO:tasks.rados.rados.0.smithi165.stdout:2036: finishing write tid 2 to smithi16534841-5 2024-09-10T16:20:15.221 INFO:tasks.rados.rados.0.smithi165.stdout:2036: finishing write tid 3 to smithi16534841-5 2024-09-10T16:20:15.221 INFO:tasks.rados.rados.0.smithi165.stdout:2036: finishing write tid 4 to smithi16534841-5 2024-09-10T16:20:15.221 INFO:tasks.rados.rados.0.smithi165.stdout:2036: oid 5 updating version 0 to 1301 2024-09-10T16:20:15.222 INFO:tasks.rados.rados.0.smithi165.stdout:2036: oid 5 updating version 1301 to 1303 2024-09-10T16:20:15.222 INFO:tasks.rados.rados.0.smithi165.stdout:2036: oid 5 version 1303 is already newer than 1302 2024-09-10T16:20:15.222 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1303 (ObjNum 729 snap 223 seq_num 729) dirty exists 2024-09-10T16:20:15.222 INFO:tasks.rados.rados.0.smithi165.stdout:2036: left oid 5 (ObjNum 729 snap 223 seq_num 729) 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:2036: done (2 left) 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:2037: done (1 left) 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:2038: rollback oid 41 current snap is 223 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 222 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:2039: write oid 45 current snap is 223 2024-09-10T16:20:15.223 INFO:tasks.rados.rados.0.smithi165.stdout:2039: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-10T16:20:15.229 INFO:tasks.rados.rados.0.smithi165.stdout:2039: writing smithi16534841-45 from 685917 to 1262824 tid 1 2024-09-10T16:20:15.235 INFO:tasks.rados.rados.0.smithi165.stdout:2039: writing smithi16534841-45 from 1893043 to 2285917 tid 2 2024-09-10T16:20:15.235 INFO:tasks.rados.rados.0.smithi165.stdout:2040: write oid 33 current snap is 223 2024-09-10T16:20:15.235 INFO:tasks.rados.rados.0.smithi165.stdout:2035: finishing copy_from to smithi16534841-47 2024-09-10T16:20:15.236 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1305 (ObjNum 725 snap 221 seq_num 725) dirty exists 2024-09-10T16:20:15.236 INFO:tasks.rados.rados.0.smithi165.stdout:2040: seq_num 732 ranges {702724=652156,1951525=351199} 2024-09-10T16:20:15.243 INFO:tasks.rados.rados.0.smithi165.stdout:2040: writing smithi16534841-33 from 702724 to 1354880 tid 1 2024-09-10T16:20:15.248 INFO:tasks.rados.rados.0.smithi165.stdout:2040: writing smithi16534841-33 from 1951525 to 2302724 tid 2 2024-09-10T16:20:15.248 INFO:tasks.rados.rados.0.smithi165.stdout:2035: done (3 left) 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2041: read oid 28 snap -1 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2041: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2042: write oid 26 current snap is 223 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2038: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1152 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2039: finishing write tid 1 to smithi16534841-45 2024-09-10T16:20:15.249 INFO:tasks.rados.rados.0.smithi165.stdout:2042: seq_num 733 ranges {719531=727405,1893654=425877} 2024-09-10T16:20:15.256 INFO:tasks.rados.rados.0.smithi165.stdout:2042: writing smithi16534841-26 from 719531 to 1446936 tid 1 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2042: writing smithi16534841-26 from 1893654 to 2319531 tid 2 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2038: done (4 left) 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2043: read oid 17 snap 218 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2043: expect (ObjNum 673 snap 207 seq_num 673) 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2044: delete oid 15 current snap is 223 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2039: finishing write tid 2 to smithi16534841-45 2024-09-10T16:20:15.262 INFO:tasks.rados.rados.0.smithi165.stdout:2039: finishing write tid 3 to smithi16534841-45 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2039: finishing write tid 4 to smithi16534841-45 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2039: oid 45 updating version 0 to 1289 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2039: oid 45 updating version 1289 to 1290 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2039: oid 45 version 1290 is already newer than 1288 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1290 (ObjNum 731 snap 223 seq_num 731) dirty exists 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2039: left oid 45 (ObjNum 731 snap 223 seq_num 731) 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: finishing write tid 1 to smithi16534841-33 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: finishing write tid 2 to smithi16534841-33 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: finishing write tid 3 to smithi16534841-33 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: finishing write tid 4 to smithi16534841-33 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: oid 33 updating version 0 to 1217 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: oid 33 updating version 1217 to 1219 2024-09-10T16:20:15.263 INFO:tasks.rados.rados.0.smithi165.stdout:2040: oid 33 version 1219 is already newer than 1218 2024-09-10T16:20:15.264 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1219 (ObjNum 732 snap 223 seq_num 732) dirty exists 2024-09-10T16:20:15.264 INFO:tasks.rados.rados.0.smithi165.stdout:2040: left oid 33 (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:15.264 INFO:tasks.rados.rados.0.smithi165.stdout:2041: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:15.289 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3066171 2024-09-10T16:20:15.319 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: finishing write tid 1 to smithi16534841-26 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: finishing write tid 2 to smithi16534841-26 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: finishing write tid 3 to smithi16534841-26 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: finishing write tid 4 to smithi16534841-26 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: oid 26 updating version 0 to 1309 2024-09-10T16:20:15.452 INFO:tasks.rados.rados.0.smithi165.stdout:2042: oid 26 updating version 1309 to 1311 2024-09-10T16:20:15.453 INFO:tasks.rados.rados.0.smithi165.stdout:2042: oid 26 version 1311 is already newer than 1310 2024-09-10T16:20:15.453 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1311 (ObjNum 733 snap 223 seq_num 733) dirty exists 2024-09-10T16:20:15.453 INFO:tasks.rados.rados.0.smithi165.stdout:2042: left oid 26 (ObjNum 733 snap 223 seq_num 733) 2024-09-10T16:20:15.453 INFO:tasks.rados.rados.0.smithi165.stdout:2043: expect (ObjNum 673 snap 207 seq_num 673) 2024-09-10T16:20:15.548 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:15.645 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:15.741 INFO:tasks.rados.rados.0.smithi165.stdout:2039: done (5 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2040: done (4 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2041: done (3 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2042: done (2 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2043: done (1 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2044: done (0 left) 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2045: write (excl) oid 12 current snap is 223 2024-09-10T16:20:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:2045: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-10T16:20:15.748 INFO:tasks.rados.rados.0.smithi165.stdout:2045: writing smithi16534841-12 from 753145 to 1231048 tid 1 2024-09-10T16:20:15.754 INFO:tasks.rados.rados.0.smithi165.stdout:2045: writing smithi16534841-12 from 2010618 to 2353145 tid 2 2024-09-10T16:20:15.754 INFO:tasks.rados.rados.0.smithi165.stdout:2046: snap_remove snap 220 2024-09-10T16:20:16.215 INFO:tasks.rados.rados.0.smithi165.stdout:2046: done (1 left) 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2047: read oid 14 snap 221 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2047: expect (ObjNum 721 snap 221 seq_num 721) 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2048: rollback oid 35 current snap is 223 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 222 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: finishing write tid 1 to smithi16534841-12 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: finishing write tid 2 to smithi16534841-12 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: finishing write tid 3 to smithi16534841-12 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: finishing write tid 4 to smithi16534841-12 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: oid 12 updating version 0 to 1098 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: oid 12 version 1098 is already newer than 1097 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: oid 12 updating version 1098 to 1099 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1099 (ObjNum 735 snap 223 seq_num 735) dirty exists 2024-09-10T16:20:16.216 INFO:tasks.rados.rados.0.smithi165.stdout:2045: left oid 12 (ObjNum 735 snap 223 seq_num 735) 2024-09-10T16:20:16.217 INFO:tasks.rados.rados.0.smithi165.stdout:2045: done (2 left) 2024-09-10T16:20:16.217 INFO:tasks.rados.rados.0.smithi165.stdout:2049: write oid 3 current snap is 223 2024-09-10T16:20:16.217 INFO:tasks.rados.rados.0.smithi165.stdout:2049: seq_num 736 ranges {769952=553152,2069100=300852} 2024-09-10T16:20:16.223 INFO:tasks.rados.rados.0.smithi165.stdout:2049: writing smithi16534841-3 from 769952 to 1323104 tid 1 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2049: writing smithi16534841-3 from 2069100 to 2369952 tid 2 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2050: read oid 6 snap 221 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2050: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2051: read oid 42 snap -1 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2051: expect (ObjNum 651 snap 207 seq_num 651) 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2052: copy_from oid 8 from oid 30 current snap is 223 2024-09-10T16:20:16.230 INFO:tasks.rados.rados.0.smithi165.stdout:2048: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:20:16.231 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 0 (ObjNum 129 snap 0 seq_num 1162666544) dirty dne 2024-09-10T16:20:16.231 INFO:tasks.rados.rados.0.smithi165.stdout:2047: expect (ObjNum 721 snap 221 seq_num 721) 2024-09-10T16:20:16.352 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:16.413 INFO:tasks.rados.rados.0.smithi165.stdout:2049: finishing write tid 1 to smithi16534841-3 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2052: finishing copy_from to smithi16534841-8 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2052: got expected ENOENT (src dne) 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2052: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: finishing write tid 2 to smithi16534841-3 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: finishing write tid 3 to smithi16534841-3 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: finishing write tid 4 to smithi16534841-3 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: oid 3 updating version 0 to 1221 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: oid 3 updating version 1221 to 1222 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: oid 3 version 1222 is already newer than 1220 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1222 (ObjNum 736 snap 223 seq_num 736) dirty exists 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2049: left oid 3 (ObjNum 736 snap 223 seq_num 736) 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2047: done (5 left) 2024-09-10T16:20:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:2048: done (4 left) 2024-09-10T16:20:16.415 INFO:tasks.rados.rados.0.smithi165.stdout:2049: done (3 left) 2024-09-10T16:20:16.415 INFO:tasks.rados.rados.0.smithi165.stdout:2052: done (2 left) 2024-09-10T16:20:16.415 INFO:tasks.rados.rados.0.smithi165.stdout:2053: write oid 17 current snap is 223 2024-09-10T16:20:16.415 INFO:tasks.rados.rados.0.smithi165.stdout:2053: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-10T16:20:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:2053: writing smithi16534841-17 from 786759 to 1415160 tid 1 2024-09-10T16:20:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:2053: writing smithi16534841-17 from 2011229 to 2386759 tid 2 2024-09-10T16:20:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:2054: read oid 21 snap -1 2024-09-10T16:20:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:2054: expect (ObjNum 698 snap 215 seq_num 698) 2024-09-10T16:20:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:2055: write (excl) oid 20 current snap is 223 2024-09-10T16:20:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:2051: expect (ObjNum 651 snap 207 seq_num 651) 2024-09-10T16:20:16.510 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:16.675 INFO:tasks.rados.rados.0.smithi165.stdout:2050: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:16.675 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:16.718 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2642488 2024-09-10T16:20:16.763 INFO:tasks.rados.rados.0.smithi165.stdout:2053: finishing write tid 1 to smithi16534841-17 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: finishing write tid 2 to smithi16534841-17 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: finishing write tid 3 to smithi16534841-17 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: finishing write tid 4 to smithi16534841-17 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: oid 17 updating version 0 to 1106 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: oid 17 updating version 1106 to 1107 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: oid 17 version 1107 is already newer than 1105 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1107 (ObjNum 737 snap 223 seq_num 737) dirty exists 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2053: left oid 17 (ObjNum 737 snap 223 seq_num 737) 2024-09-10T16:20:16.764 INFO:tasks.rados.rados.0.smithi165.stdout:2054: expect (ObjNum 698 snap 215 seq_num 698) 2024-09-10T16:20:16.885 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:16.953 INFO:tasks.rados.rados.0.smithi165.stdout:2055: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-10T16:20:16.957 INFO:tasks.rados.rados.0.smithi165.stdout:2055: writing smithi16534841-20 from 403566 to 823569 tid 1 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2055: writing smithi16534841-20 from 1386064 to 1932982 tid 2 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2050: done (4 left) 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2051: done (3 left) 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2053: done (2 left) 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2054: done (1 left) 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2056: copy_from oid 15 from oid 45 current snap is 223 2024-09-10T16:20:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2057: snap_create 2024-09-10T16:20:16.971 INFO:tasks.rados.rados.0.smithi165.stdout:2055: finishing write tid 1 to smithi16534841-20 2024-09-10T16:20:16.971 INFO:tasks.rados.rados.0.smithi165.stdout:2055: finishing write tid 2 to smithi16534841-20 2024-09-10T16:20:16.973 INFO:tasks.rados.rados.0.smithi165.stdout:2055: finishing write tid 3 to smithi16534841-20 2024-09-10T16:20:16.973 INFO:tasks.rados.rados.0.smithi165.stdout:2055: finishing write tid 4 to smithi16534841-20 2024-09-10T16:20:16.973 INFO:tasks.rados.rados.0.smithi165.stdout:2055: oid 20 updating version 0 to 1225 2024-09-10T16:20:16.973 INFO:tasks.rados.rados.0.smithi165.stdout:2055: oid 20 version 1225 is already newer than 1224 2024-09-10T16:20:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:2055: oid 20 version 1225 is already newer than 1223 2024-09-10T16:20:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1225 (ObjNum 738 snap 223 seq_num 738) dirty exists 2024-09-10T16:20:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:2055: left oid 20 (ObjNum 738 snap 223 seq_num 738) 2024-09-10T16:20:16.977 INFO:tasks.rados.rados.0.smithi165.stdout:2056: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:20:16.979 INFO:tasks.rados.rados.0.smithi165.stdout:2056: finishing copy_from to smithi16534841-15 2024-09-10T16:20:16.979 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1290 (ObjNum 731 snap 223 seq_num 731) dirty exists 2024-09-10T16:20:17.469 INFO:tasks.rados.rados.0.smithi165.stdout:2055: done (2 left) 2024-09-10T16:20:17.469 INFO:tasks.rados.rados.0.smithi165.stdout:2056: done (1 left) 2024-09-10T16:20:17.470 INFO:tasks.rados.rados.0.smithi165.stdout:2057: done (0 left) 2024-09-10T16:20:17.470 INFO:tasks.rados.rados.0.smithi165.stdout:2058: copy_from oid 12 from oid 32 current snap is 224 2024-09-10T16:20:17.470 INFO:tasks.rados.rados.0.smithi165.stdout:2059: delete oid 42 current snap is 224 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:2059: done (1 left) 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:2060: read oid 33 snap -1 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:2060: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:2061: rollback oid 17 current snap is 224 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 222 2024-09-10T16:20:17.473 INFO:tasks.rados.rados.0.smithi165.stdout:2062: copy_from oid 46 from oid 10 current snap is 224 2024-09-10T16:20:17.474 INFO:tasks.rados.rados.0.smithi165.stdout:2063: snap_create 2024-09-10T16:20:17.476 INFO:tasks.rados.rados.0.smithi165.stdout:2060: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:17.476 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:17.534 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1389505 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:2061: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:2061: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:2061: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1110 (ObjNum 673 snap 207 seq_num 673) dirty exists 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:2058: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:2058: finishing copy_from to smithi16534841-12 2024-09-10T16:20:17.548 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1300 (ObjNum 707 snap 216 seq_num 707) dirty exists 2024-09-10T16:20:17.549 INFO:tasks.rados.rados.0.smithi165.stdout:2062: finishing copy_from racing read to smithi16534841-46 2024-09-10T16:20:17.549 INFO:tasks.rados.rados.0.smithi165.stdout:2062: finishing copy_from to smithi16534841-46 2024-09-10T16:20:17.549 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1088 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2058: done (4 left) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2060: done (3 left) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2061: done (2 left) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2062: done (1 left) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2063: done (0 left) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2064: read oid 6 snap 223 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2064: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:18.477 INFO:tasks.rados.rados.0.smithi165.stdout:2065: read oid 49 snap -1 2024-09-10T16:20:18.478 INFO:tasks.rados.rados.0.smithi165.stdout:2065: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:18.478 INFO:tasks.rados.rados.0.smithi165.stdout:2066: write (excl) oid 32 current snap is 225 2024-09-10T16:20:18.478 INFO:tasks.rados.rados.0.smithi165.stdout:2066: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-10T16:20:18.482 INFO:tasks.rados.rados.0.smithi165.stdout:2066: writing smithi16534841-32 from 437180 to 1007681 tid 1 2024-09-10T16:20:18.488 INFO:tasks.rados.rados.0.smithi165.stdout:2066: writing smithi16534841-32 from 1503028 to 2053615 tid 2 2024-09-10T16:20:18.488 INFO:tasks.rados.rados.0.smithi165.stdout:2067: write (excl) oid 12 current snap is 225 2024-09-10T16:20:18.488 INFO:tasks.rados.rados.0.smithi165.stdout:2067: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-10T16:20:18.494 INFO:tasks.rados.rados.0.smithi165.stdout:2067: writing smithi16534841-12 from 453987 to 1099737 tid 1 2024-09-10T16:20:18.502 INFO:tasks.rados.rados.0.smithi165.stdout:2067: writing smithi16534841-12 from 1845157 to 2453987 tid 2 2024-09-10T16:20:18.502 INFO:tasks.rados.rados.0.smithi165.stdout:2068: write (excl) oid 43 current snap is 225 2024-09-10T16:20:18.502 INFO:tasks.rados.rados.0.smithi165.stdout:2068: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-10T16:20:18.508 INFO:tasks.rados.rados.0.smithi165.stdout:2068: writing smithi16534841-43 from 470794 to 1191793 tid 1 2024-09-10T16:20:18.515 INFO:tasks.rados.rados.0.smithi165.stdout:2068: writing smithi16534841-43 from 1903639 to 2457895 tid 2 2024-09-10T16:20:18.516 INFO:tasks.rados.rados.0.smithi165.stdout:2069: write oid 7 current snap is 225 2024-09-10T16:20:18.516 INFO:tasks.rados.rados.0.smithi165.stdout:2069: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-10T16:20:18.522 INFO:tasks.rados.rados.0.smithi165.stdout:2069: writing smithi16534841-7 from 487601 to 1283849 tid 1 2024-09-10T16:20:18.532 INFO:tasks.rados.rados.0.smithi165.stdout:2069: writing smithi16534841-7 from 1962121 to 2487601 tid 2 2024-09-10T16:20:18.532 INFO:tasks.rados.rados.0.smithi165.stdout:2070: read oid 1 snap -1 2024-09-10T16:20:18.532 INFO:tasks.rados.rados.0.smithi165.stdout:2070: expect deleted 2024-09-10T16:20:18.532 INFO:tasks.rados.rados.0.smithi165.stdout:2071: snap_create 2024-09-10T16:20:18.532 INFO:tasks.rados.rados.0.smithi165.stdout:2065: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:18.556 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2764379 2024-09-10T16:20:18.716 INFO:tasks.rados.rados.0.smithi165.stdout:2064: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:18.750 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3021779 2024-09-10T16:20:18.762 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 896104 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: finishing write tid 1 to smithi16534841-32 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: finishing write tid 2 to smithi16534841-32 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: finishing write tid 3 to smithi16534841-32 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: finishing write tid 4 to smithi16534841-32 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: oid 32 updating version 0 to 1305 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: oid 32 version 1305 is already newer than 1304 2024-09-10T16:20:18.786 INFO:tasks.rados.rados.0.smithi165.stdout:2066: oid 32 updating version 1305 to 1306 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1306 (ObjNum 740 snap 225 seq_num 740) dirty exists 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2066: left oid 32 (ObjNum 740 snap 225 seq_num 740) 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: finishing write tid 1 to smithi16534841-12 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: finishing write tid 2 to smithi16534841-12 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: finishing write tid 3 to smithi16534841-12 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: finishing write tid 4 to smithi16534841-12 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: oid 12 updating version 0 to 1301 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: oid 12 updating version 1301 to 1303 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: oid 12 version 1303 is already newer than 1302 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1303 (ObjNum 741 snap 225 seq_num 741) dirty exists 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2067: left oid 12 (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2068: finishing write tid 1 to smithi16534841-43 2024-09-10T16:20:18.787 INFO:tasks.rados.rados.0.smithi165.stdout:2068: finishing write tid 2 to smithi16534841-43 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: finishing write tid 3 to smithi16534841-43 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: finishing write tid 4 to smithi16534841-43 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: oid 43 updating version 0 to 1293 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: oid 43 version 1293 is already newer than 1291 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: oid 43 version 1293 is already newer than 1292 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1293 (ObjNum 742 snap 225 seq_num 742) dirty exists 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2068: left oid 43 (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: finishing write tid 1 to smithi16534841-7 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: finishing write tid 2 to smithi16534841-7 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: finishing write tid 3 to smithi16534841-7 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: finishing write tid 4 to smithi16534841-7 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: oid 7 updating version 0 to 1316 2024-09-10T16:20:18.788 INFO:tasks.rados.rados.0.smithi165.stdout:2069: oid 7 updating version 1316 to 1317 2024-09-10T16:20:18.789 INFO:tasks.rados.rados.0.smithi165.stdout:2069: oid 7 version 1317 is already newer than 1315 2024-09-10T16:20:18.789 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1317 (ObjNum 743 snap 225 seq_num 743) dirty exists 2024-09-10T16:20:18.789 INFO:tasks.rados.rados.0.smithi165.stdout:2069: left oid 7 (ObjNum 743 snap 225 seq_num 743) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2064: done (7 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2065: done (6 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2066: done (5 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2067: done (4 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2068: done (3 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2069: done (2 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2070: done (1 left) 2024-09-10T16:20:19.476 INFO:tasks.rados.rados.0.smithi165.stdout:2071: done (0 left) 2024-09-10T16:20:19.477 INFO:tasks.rados.rados.0.smithi165.stdout:2072: copy_from oid 4 from oid 12 current snap is 226 2024-09-10T16:20:19.477 INFO:tasks.rados.rados.0.smithi165.stdout:2073: snap_remove snap 225 2024-09-10T16:20:20.488 INFO:tasks.rados.rados.0.smithi165.stdout:2073: done (1 left) 2024-09-10T16:20:20.488 INFO:tasks.rados.rados.0.smithi165.stdout:2074: read oid 7 snap -1 2024-09-10T16:20:20.488 INFO:tasks.rados.rados.0.smithi165.stdout:2074: expect (ObjNum 743 snap 225 seq_num 743) 2024-09-10T16:20:20.489 INFO:tasks.rados.rados.0.smithi165.stdout:2075: snap_create 2024-09-10T16:20:20.489 INFO:tasks.rados.rados.0.smithi165.stdout:2072: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:20:20.489 INFO:tasks.rados.rados.0.smithi165.stdout:2072: finishing copy_from to smithi16534841-4 2024-09-10T16:20:20.489 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1303 (ObjNum 741 snap 225 seq_num 741) dirty exists 2024-09-10T16:20:20.497 INFO:tasks.rados.rados.0.smithi165.stdout:2074: expect (ObjNum 743 snap 225 seq_num 743) 2024-09-10T16:20:20.539 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 775668 2024-09-10T16:20:21.468 INFO:tasks.rados.rados.0.smithi165.stdout:2072: done (2 left) 2024-09-10T16:20:21.468 INFO:tasks.rados.rados.0.smithi165.stdout:2074: done (1 left) 2024-09-10T16:20:21.468 INFO:tasks.rados.rados.0.smithi165.stdout:2075: done (0 left) 2024-09-10T16:20:21.468 INFO:tasks.rados.rados.0.smithi165.stdout:2076: read oid 47 snap 222 2024-09-10T16:20:21.468 INFO:tasks.rados.rados.0.smithi165.stdout:2076: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:20:21.469 INFO:tasks.rados.rados.0.smithi165.stdout:2077: read oid 33 snap -1 2024-09-10T16:20:21.469 INFO:tasks.rados.rados.0.smithi165.stdout:2077: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:21.469 INFO:tasks.rados.rados.0.smithi165.stdout:2078: delete oid 36 current snap is 227 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2078: done (2 left) 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2079: read oid 4 snap -1 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2079: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2080: read oid 49 snap -1 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2080: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:21.478 INFO:tasks.rados.rados.0.smithi165.stdout:2081: rollback oid 22 current snap is 227 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 223 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2082: read oid 46 snap -1 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2082: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2083: read oid 44 snap -1 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2083: expect deleted 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2084: write oid 13 current snap is 227 2024-09-10T16:20:21.479 INFO:tasks.rados.rados.0.smithi165.stdout:2084: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-10T16:20:21.485 INFO:tasks.rados.rados.0.smithi165.stdout:2084: writing smithi16534841-13 from 521215 to 1067961 tid 1 2024-09-10T16:20:21.493 INFO:tasks.rados.rados.0.smithi165.stdout:2084: writing smithi16534841-13 from 1562732 to 2264315 tid 2 2024-09-10T16:20:21.493 INFO:tasks.rados.rados.0.smithi165.stdout:2085: delete oid 2 current snap is 227 2024-09-10T16:20:21.493 INFO:tasks.rados.rados.0.smithi165.stdout:2077: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:21.493 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 16450 2024-09-10T16:20:21.571 INFO:tasks.rados.rados.0.smithi165.stdout:2082: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:20:21.607 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 158716 2024-09-10T16:20:21.627 INFO:tasks.rados.rados.0.smithi165.stdout:2076: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:20:21.666 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:21.707 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:21.747 INFO:tasks.rados.rados.0.smithi165.stdout:2081: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:20:21.748 INFO:tasks.rados.rados.0.smithi165.stdout:2081: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:20:21.748 INFO:tasks.rados.rados.0.smithi165.stdout:2081: finishing rollback tid 2 to smithi16534841-22 2024-09-10T16:20:21.748 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 784 (ObjNum 682 snap 208 seq_num 682) dirty exists 2024-09-10T16:20:21.748 INFO:tasks.rados.rados.0.smithi165.stdout:2080: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:20:21.751 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 680180 2024-09-10T16:20:21.813 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:21.950 INFO:tasks.rados.rados.0.smithi165.stdout:2084: finishing write tid 1 to smithi16534841-13 2024-09-10T16:20:21.951 INFO:tasks.rados.rados.0.smithi165.stdout:2079: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:20:21.989 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:22.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 114341 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: finishing write tid 2 to smithi16534841-13 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: finishing write tid 3 to smithi16534841-13 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: finishing write tid 4 to smithi16534841-13 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: oid 13 updating version 0 to 786 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: oid 13 version 786 is already newer than 785 2024-09-10T16:20:22.069 INFO:tasks.rados.rados.0.smithi165.stdout:2084: oid 13 updating version 786 to 787 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 787 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2084: left oid 13 (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2076: done (8 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2077: done (7 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2079: done (6 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2080: done (5 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2081: done (4 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2082: done (3 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2083: done (2 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2084: done (1 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2085: done (0 left) 2024-09-10T16:20:22.070 INFO:tasks.rados.rados.0.smithi165.stdout:2086: snap_remove snap 223 2024-09-10T16:20:22.501 INFO:tasks.rados.rados.0.smithi165.stdout:2086: done (0 left) 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2087: copy_from oid 27 from oid 12 current snap is 227 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2088: read oid 47 snap -1 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2088: expect (ObjNum 725 snap 221 seq_num 725) 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2089: copy_from oid 4 from oid 13 current snap is 227 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2090: rollback oid 39 current snap is 227 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 226 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2091: write (excl) oid 22 current snap is 227 2024-09-10T16:20:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2091: seq_num 747 ranges {554829=413597,1396049=705252} 2024-09-10T16:20:22.507 INFO:tasks.rados.rados.0.smithi165.stdout:2091: writing smithi16534841-22 from 554829 to 968426 tid 1 2024-09-10T16:20:22.514 INFO:tasks.rados.rados.0.smithi165.stdout:2091: writing smithi16534841-22 from 1396049 to 2101301 tid 2 2024-09-10T16:20:22.514 INFO:tasks.rados.rados.0.smithi165.stdout:2092: snap_remove snap 222 2024-09-10T16:20:24.134 INFO:tasks.rados.rados.0.smithi165.stdout:2092: done (5 left) 2024-09-10T16:20:24.135 INFO:tasks.rados.rados.0.smithi165.stdout:2093: copy_from oid 17 from oid 2 current snap is 227 2024-09-10T16:20:24.135 INFO:tasks.rados.rados.0.smithi165.stdout:2088: expect (ObjNum 725 snap 221 seq_num 725) 2024-09-10T16:20:24.158 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2130114 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2090: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2090: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2090: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1162 (ObjNum 631 snap 197 seq_num 631) dirty exists 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2091: finishing write tid 1 to smithi16534841-22 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2087: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2091: finishing write tid 2 to smithi16534841-22 2024-09-10T16:20:24.182 INFO:tasks.rados.rados.0.smithi165.stdout:2089: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: finishing write tid 3 to smithi16534841-22 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: finishing write tid 4 to smithi16534841-22 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: oid 22 updating version 0 to 790 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: oid 22 version 790 is already newer than 788 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: oid 22 version 790 is already newer than 789 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 790 (ObjNum 747 snap 227 seq_num 747) dirty exists 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2091: left oid 22 (ObjNum 747 snap 227 seq_num 747) 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2089: finishing copy_from to smithi16534841-4 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 787 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2087: finishing copy_from to smithi16534841-27 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1303 (ObjNum 741 snap 225 seq_num 741) dirty exists 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2093: finishing copy_from to smithi16534841-17 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2093: got expected ENOENT (src dne) 2024-09-10T16:20:24.183 INFO:tasks.rados.rados.0.smithi165.stdout:2093: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2087: done (5 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2088: done (4 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2089: done (3 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2090: done (2 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2091: done (1 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2093: done (0 left) 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2094: rollback oid 10 current snap is 227 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 221 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2095: write oid 40 current snap is 227 2024-09-10T16:20:24.184 INFO:tasks.rados.rados.0.smithi165.stdout:2095: seq_num 748 ranges {571636=488846,1854531=565263} 2024-09-10T16:20:24.187 INFO:tasks.rados.rados.0.smithi165.stdout:2095: writing smithi16534841-40 from 571636 to 1060482 tid 1 2024-09-10T16:20:24.195 INFO:tasks.rados.rados.0.smithi165.stdout:2095: writing smithi16534841-40 from 1854531 to 2419794 tid 2 2024-09-10T16:20:24.195 INFO:tasks.rados.rados.0.smithi165.stdout:2096: read oid 6 snap -1 2024-09-10T16:20:24.195 INFO:tasks.rados.rados.0.smithi165.stdout:2096: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:24.196 INFO:tasks.rados.rados.0.smithi165.stdout:2097: snap_remove snap 224 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2097: done (3 left) 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2098: read oid 43 snap -1 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2098: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2094: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2094: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2094: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1113 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2095: finishing write tid 1 to smithi16534841-40 2024-09-10T16:20:24.624 INFO:tasks.rados.rados.0.smithi165.stdout:2095: finishing write tid 2 to smithi16534841-40 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: finishing write tid 3 to smithi16534841-40 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: finishing write tid 4 to smithi16534841-40 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: oid 40 updating version 0 to 1163 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: oid 40 updating version 1163 to 1164 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: oid 40 updating version 1164 to 1165 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1165 (ObjNum 748 snap 227 seq_num 748) dirty exists 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2095: left oid 40 (ObjNum 748 snap 227 seq_num 748) 2024-09-10T16:20:24.625 INFO:tasks.rados.rados.0.smithi165.stdout:2096: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:20:24.661 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:24.729 INFO:tasks.rados.rados.0.smithi165.stdout:2094: done (3 left) 2024-09-10T16:20:24.729 INFO:tasks.rados.rados.0.smithi165.stdout:2095: done (2 left) 2024-09-10T16:20:24.729 INFO:tasks.rados.rados.0.smithi165.stdout:2096: done (1 left) 2024-09-10T16:20:24.729 INFO:tasks.rados.rados.0.smithi165.stdout:2099: rollback oid 47 current snap is 227 2024-09-10T16:20:24.729 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 219 2024-09-10T16:20:24.730 INFO:tasks.rados.rados.0.smithi165.stdout:2100: write (excl) oid 9 current snap is 227 2024-09-10T16:20:24.730 INFO:tasks.rados.rados.0.smithi165.stdout:2100: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-10T16:20:24.735 INFO:tasks.rados.rados.0.smithi165.stdout:2100: writing smithi16534841-9 from 588443 to 1152538 tid 1 2024-09-10T16:20:24.743 INFO:tasks.rados.rados.0.smithi165.stdout:2100: writing smithi16534841-9 from 1796660 to 2505581 tid 2 2024-09-10T16:20:24.743 INFO:tasks.rados.rados.0.smithi165.stdout:2101: delete oid 11 current snap is 227 2024-09-10T16:20:24.743 INFO:tasks.rados.rados.0.smithi165.stdout:2098: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:24.781 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1950494 2024-09-10T16:20:24.788 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1579390 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2099: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2099: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2099: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1309 (ObjNum 633 snap 198 seq_num 633) dirty exists 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2100: finishing write tid 1 to smithi16534841-9 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2100: finishing write tid 2 to smithi16534841-9 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2100: finishing write tid 3 to smithi16534841-9 2024-09-10T16:20:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:2100: finishing write tid 4 to smithi16534841-9 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2100: oid 9 updating version 0 to 1167 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2100: oid 9 version 1167 is already newer than 1166 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2100: oid 9 updating version 1167 to 1168 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1168 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2100: left oid 9 (ObjNum 749 snap 227 seq_num 749) 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2098: done (3 left) 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2099: done (2 left) 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2100: done (1 left) 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2101: done (0 left) 2024-09-10T16:20:24.805 INFO:tasks.rados.rados.0.smithi165.stdout:2102: write (excl) oid 50 current snap is 227 2024-09-10T16:20:24.806 INFO:tasks.rados.rados.0.smithi165.stdout:2102: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-10T16:20:24.810 INFO:tasks.rados.rados.0.smithi165.stdout:2102: writing smithi16534841-50 from 622057 to 1336650 tid 1 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:2102: writing smithi16534841-50 from 1913624 to 2622057 tid 2 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:2103: rollback oid 14 current snap is 227 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 219 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:2102: finishing write tid 1 to smithi16534841-50 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:2104: write oid 24 current snap is 227 2024-09-10T16:20:24.818 INFO:tasks.rados.rados.0.smithi165.stdout:2104: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-10T16:20:24.825 INFO:tasks.rados.rados.0.smithi165.stdout:2104: writing smithi16534841-24 from 638864 to 1428706 tid 1 2024-09-10T16:20:24.831 INFO:tasks.rados.rados.0.smithi165.stdout:2104: writing smithi16534841-24 from 1972106 to 2428354 tid 2 2024-09-10T16:20:24.832 INFO:tasks.rados.rados.0.smithi165.stdout:2105: snap_remove snap 226 2024-09-10T16:20:25.633 INFO:tasks.rados.rados.0.smithi165.stdout:2105: done (3 left) 2024-09-10T16:20:25.634 INFO:tasks.rados.rados.0.smithi165.stdout:2106: snap_remove snap 221 2024-09-10T16:20:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:2106: done (3 left) 2024-09-10T16:20:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:2107: write oid 48 current snap is 227 2024-09-10T16:20:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:2103: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:20:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:2107: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-10T16:20:26.474 INFO:tasks.rados.rados.0.smithi165.stdout:2107: writing smithi16534841-48 from 655671 to 1237115 tid 1 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2107: writing smithi16534841-48 from 2030588 to 2655671 tid 2 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2108: snap_create 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2103: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2102: finishing write tid 2 to smithi16534841-50 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2103: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1171 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:20:26.483 INFO:tasks.rados.rados.0.smithi165.stdout:2102: finishing write tid 3 to smithi16534841-50 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2102: finishing write tid 4 to smithi16534841-50 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2102: oid 50 updating version 0 to 1241 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2102: oid 50 version 1241 is already newer than 1240 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2102: oid 50 updating version 1241 to 1242 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1242 (ObjNum 751 snap 227 seq_num 751) dirty exists 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2102: left oid 50 (ObjNum 751 snap 227 seq_num 751) 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: finishing write tid 1 to smithi16534841-24 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: finishing write tid 2 to smithi16534841-24 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: finishing write tid 3 to smithi16534841-24 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: finishing write tid 4 to smithi16534841-24 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: oid 24 updating version 0 to 1310 2024-09-10T16:20:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2104: oid 24 updating version 1310 to 1311 2024-09-10T16:20:26.485 INFO:tasks.rados.rados.0.smithi165.stdout:2104: oid 24 updating version 1311 to 1312 2024-09-10T16:20:26.485 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1312 (ObjNum 752 snap 227 seq_num 752) dirty exists 2024-09-10T16:20:26.485 INFO:tasks.rados.rados.0.smithi165.stdout:2104: left oid 24 (ObjNum 752 snap 227 seq_num 752) 2024-09-10T16:20:26.485 INFO:tasks.rados.rados.0.smithi165.stdout:2107: finishing write tid 1 to smithi16534841-48 2024-09-10T16:20:26.489 INFO:tasks.rados.rados.0.smithi165.stdout:2107: finishing write tid 2 to smithi16534841-48 2024-09-10T16:20:26.490 INFO:tasks.rados.rados.0.smithi165.stdout:2107: finishing write tid 3 to smithi16534841-48 2024-09-10T16:20:26.490 INFO:tasks.rados.rados.0.smithi165.stdout:2107: finishing write tid 4 to smithi16534841-48 2024-09-10T16:20:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:2107: oid 48 updating version 0 to 1330 2024-09-10T16:20:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:2107: oid 48 updating version 1330 to 1331 2024-09-10T16:20:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:2107: oid 48 updating version 1331 to 1332 2024-09-10T16:20:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1332 (ObjNum 753 snap 227 seq_num 753) dirty exists 2024-09-10T16:20:26.491 INFO:tasks.rados.rados.0.smithi165.stdout:2107: left oid 48 (ObjNum 753 snap 227 seq_num 753) 2024-09-10T16:20:27.470 INFO:tasks.rados.rados.0.smithi165.stdout:2102: done (4 left) 2024-09-10T16:20:27.470 INFO:tasks.rados.rados.0.smithi165.stdout:2103: done (3 left) 2024-09-10T16:20:27.470 INFO:tasks.rados.rados.0.smithi165.stdout:2104: done (2 left) 2024-09-10T16:20:27.471 INFO:tasks.rados.rados.0.smithi165.stdout:2107: done (1 left) 2024-09-10T16:20:27.471 INFO:tasks.rados.rados.0.smithi165.stdout:2108: done (0 left) 2024-09-10T16:20:27.471 INFO:tasks.rados.rados.0.smithi165.stdout:2109: write oid 31 current snap is 228 2024-09-10T16:20:27.471 INFO:tasks.rados.rados.0.smithi165.stdout:2109: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-10T16:20:27.477 INFO:tasks.rados.rados.0.smithi165.stdout:2109: writing smithi16534841-31 from 672478 to 1329171 tid 1 2024-09-10T16:20:27.484 INFO:tasks.rados.rados.0.smithi165.stdout:2109: writing smithi16534841-31 from 2089070 to 2548987 tid 2 2024-09-10T16:20:27.484 INFO:tasks.rados.rados.0.smithi165.stdout:2110: read oid 37 snap -1 2024-09-10T16:20:27.484 INFO:tasks.rados.rados.0.smithi165.stdout:2110: expect (ObjNum 693 snap 214 seq_num 693) 2024-09-10T16:20:27.484 INFO:tasks.rados.rados.0.smithi165.stdout:2111: read oid 27 snap 219 2024-09-10T16:20:27.485 INFO:tasks.rados.rados.0.smithi165.stdout:2111: expect (ObjNum 699 snap 215 seq_num 699) 2024-09-10T16:20:27.485 INFO:tasks.rados.rados.0.smithi165.stdout:2112: write (excl) oid 12 current snap is 228 2024-09-10T16:20:27.485 INFO:tasks.rados.rados.0.smithi165.stdout:2112: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-10T16:20:27.492 INFO:tasks.rados.rados.0.smithi165.stdout:2112: writing smithi16534841-12 from 689285 to 1421227 tid 1 2024-09-10T16:20:27.500 INFO:tasks.rados.rados.0.smithi165.stdout:2112: writing smithi16534841-12 from 2147552 to 2689285 tid 2 2024-09-10T16:20:27.500 INFO:tasks.rados.rados.0.smithi165.stdout:2113: copy_from oid 47 from oid 7 current snap is 228 2024-09-10T16:20:27.500 INFO:tasks.rados.rados.0.smithi165.stdout:2109: finishing write tid 1 to smithi16534841-31 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: finishing write tid 2 to smithi16534841-31 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: finishing write tid 3 to smithi16534841-31 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: finishing write tid 4 to smithi16534841-31 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: oid 31 updating version 0 to 1335 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: oid 31 updating version 1335 to 1336 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: oid 31 version 1336 is already newer than 1334 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1336 (ObjNum 754 snap 228 seq_num 754) dirty exists 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2109: left oid 31 (ObjNum 754 snap 228 seq_num 754) 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:2111: expect (ObjNum 699 snap 215 seq_num 699) 2024-09-10T16:20:27.501 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:27.543 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:27.606 INFO:tasks.rados.rados.0.smithi165.stdout:2110: expect (ObjNum 693 snap 214 seq_num 693) 2024-09-10T16:20:27.629 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 472538 2024-09-10T16:20:27.662 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 281156 2024-09-10T16:20:27.699 INFO:tasks.rados.rados.0.smithi165.stdout:2112: finishing write tid 1 to smithi16534841-12 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: finishing write tid 2 to smithi16534841-12 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: finishing write tid 3 to smithi16534841-12 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: finishing write tid 4 to smithi16534841-12 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: oid 12 updating version 0 to 1305 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: oid 12 version 1305 is already newer than 1304 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: oid 12 updating version 1305 to 1306 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1306 (ObjNum 755 snap 228 seq_num 755) dirty exists 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2112: left oid 12 (ObjNum 755 snap 228 seq_num 755) 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2113: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2113: finishing copy_from to smithi16534841-47 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1317 (ObjNum 743 snap 225 seq_num 743) dirty exists 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2109: done (4 left) 2024-09-10T16:20:27.700 INFO:tasks.rados.rados.0.smithi165.stdout:2110: done (3 left) 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2111: done (2 left) 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2112: done (1 left) 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2113: done (0 left) 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2114: copy_from oid 12 from oid 46 current snap is 228 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2115: copy_from oid 15 from oid 9 current snap is 228 2024-09-10T16:20:27.701 INFO:tasks.rados.rados.0.smithi165.stdout:2116: snap_create 2024-09-10T16:20:27.704 INFO:tasks.rados.rados.0.smithi165.stdout:2114: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:20:27.704 INFO:tasks.rados.rados.0.smithi165.stdout:2114: finishing copy_from to smithi16534841-12 2024-09-10T16:20:27.704 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1088 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:20:27.711 INFO:tasks.rados.rados.0.smithi165.stdout:2115: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:20:27.712 INFO:tasks.rados.rados.0.smithi165.stdout:2115: finishing copy_from to smithi16534841-15 2024-09-10T16:20:27.712 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1168 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:20:28.477 INFO:tasks.rados.rados.0.smithi165.stdout:2114: done (2 left) 2024-09-10T16:20:28.477 INFO:tasks.rados.rados.0.smithi165.stdout:2115: done (1 left) 2024-09-10T16:20:28.477 INFO:tasks.rados.rados.0.smithi165.stdout:2116: done (0 left) 2024-09-10T16:20:28.477 INFO:tasks.rados.rados.0.smithi165.stdout:2117: snap_create 2024-09-10T16:20:29.205 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:20:29.206 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:20:29.206 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:20:29.209 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:20:29.209 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:20:29.209 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:29.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.317+0000 7fd59635d640 10 monclient: get_monmap_and_config 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.317+0000 7fd59635d640 10 monclient: build_initial_monmap 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.317+0000 7fd59635d640 10 monclient: monmap: 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:29.318188+0000 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:29.318188+0000 2024-09-10T16:20:29.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:29.320 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 Processor -- start 2024-09-10T16:20:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 -- start start 2024-09-10T16:20:29.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient: init 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd59006cb70 0x7fd590134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-c con 0x7fd59006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-a con 0x7fd5900a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd590137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:29.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59009ce50 con 0x7fd590137ca0 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd59006db60 con 0x7fd5900a46a0 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd59006cf50 con 0x7fd59006cb70 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd59635d640 10 monclient(hunting): authenticate will time out at 1607.805796s 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd5948fa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd590137ca0 0x7fd590138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35924/0 (socket says 172.21.15.164:35924) 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36376/0 (socket says 172.21.15.164:36376) 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd5948fa640 1 -- 172.21.15.164:0/2501990343 learned_addr learned my addr 172.21.15.164:0/2501990343 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:29.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 10 monclient(hunting): get_auth_request con 0x7fd5900a46a0 auth_method 0 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.318+0000 7fd58f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58f7fe640 1 -- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 msgr2=0x7fd5900a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58f7fe640 1 -- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 msgr2=0x7fd5900a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 --2- 172.21.15.164:0/2501990343 >> v2:172.21.15.165:3300/0 conn(0x7fd59006cb70 0x7fd590134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): get_auth_request con 0x7fd59006cb70 auth_method 0 2024-09-10T16:20:29.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient(hunting): handle_auth_done global_id 154159 payload 274 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 -- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 msgr2=0x7fd5900a4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 --2- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 -- 172.21.15.164:0/2501990343 >> v1:172.21.15.164:6789/0 conn(0x7fd590137ca0 legacy=0x7fd590138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 monclient: found mon.noname-c 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.319+0000 7fd58ffff640 1 -- 172.21.15.164:0/2501990343 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59009cb70 con 0x7fd59006cb70 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58ffff640 1 -- 172.21.15.164:0/2501990343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd59006cb70 msgr2=0x7fd590134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58ffff640 1 --2- 172.21.15.164:0/2501990343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd59006cb70 0x7fd590134f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5800097a0 tx=0x7fd58002a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bdddad94d1ea3c04 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 1 -- 172.21.15.164:0/2501990343 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580037070 con 0x7fd59006cb70 2024-09-10T16:20:29.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: dump: 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.334 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: _finish_auth 0 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 1 -- 172.21.15.164:0/2501990343 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd580032440 con 0x7fd59006cb70 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:29.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 1 -- 172.21.15.164:0/2501990343 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580032810 con 0x7fd59006cb70 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd59635d640 5 monclient: authenticate success, global_id 154159 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd59635d640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd59635d640 4 set_mon_vals no callback set 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.322+0000 7fd58effd640 10 monclient: dump: 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:29.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:29.337 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 10 monclient: shutdown 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 -- 172.21.15.164:0/2501990343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd59006cb70 msgr2=0x7fd590134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 --2- 172.21.15.164:0/2501990343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd59006cb70 0x7fd590134f70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd5800097a0 tx=0x7fd58002a010 comp rx=0 tx=0).stop 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 -- 172.21.15.164:0/2501990343 shutdown_connections 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 --2- 172.21.15.164:0/2501990343 >> v2:172.21.15.164:3300/0 conn(0x7fd5900a46a0 0x7fd5900a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 --2- 172.21.15.164:0/2501990343 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd59006cb70 0x7fd590134f70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 -- 172.21.15.164:0/2501990343 >> 172.21.15.164:0/2501990343 conn(0x7fd5900aa900 msgr2=0x7fd5900aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 -- 172.21.15.164:0/2501990343 shutdown_connections 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.323+0000 7fd59635d640 1 -- 172.21.15.164:0/2501990343 wait complete. 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.324+0000 7fd59635d640 10 monclient: build_initial_monmap 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.324+0000 7fd59635d640 10 monclient: monmap: 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:29.339 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:29.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.324+0000 7fd59635d640 1 Processor -- start 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.324+0000 7fd59635d640 1 -- start start 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.324+0000 7fd59635d640 10 monclient: init 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-c con 0x7fd59006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 0x7fd590131b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-b con 0x7fd5900a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): picked mon.noname-a con 0x7fd590137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd5901389c0 con 0x7fd590137ca0 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:29.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd590138840 con 0x7fd5900a46a0 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd590140040 con 0x7fd59006cb70 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd59635d640 10 monclient(hunting): authenticate will time out at 1607.812136s 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39260/0 (socket says 172.21.15.164:39260) 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 -- 172.21.15.164:0/1493195802 learned_addr learned my addr 172.21.15.164:0/1493195802 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:29.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): get_auth_request con 0x7fd59006cb70 auth_method 0 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 10 monclient(hunting): get_auth_request con 0x7fd590137ca0 auth_method 0 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:29.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 msgr2=0x7fd59013dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 msgr2=0x7fd59013dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd5948fa640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 0x7fd590131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 10 monclient(hunting): get_auth_request con 0x7fd5900a46a0 auth_method 0 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient(hunting): handle_auth_done global_id 114533 payload 824 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 msgr2=0x7fd590131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 0x7fd590131b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 msgr2=0x7fd59013dc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:29.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 monclient: found mon.noname-c 2024-09-10T16:20:29.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:29.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:29.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58ffff640 1 -- 172.21.15.164:0/1493195802 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59009f200 con 0x7fd59006cb70 2024-09-10T16:20:29.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.325+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 0x7fd590131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.327+0000 7fd58ffff640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 msgr2=0x7fd590131610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.327+0000 7fd58ffff640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd580000c00 tx=0x7fd5800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.327+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580041020 con 0x7fd59006cb70 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: dump: 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:29.350 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: _finish_auth 0 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:20:29.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd58003d910 con 0x7fd59006cb70 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 5 monclient: authenticate success, global_id 114533 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 10 monclient: _renew_subs 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59009e960 con 0x7fd59006cb70 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd580037040 con 0x7fd59006cb70 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:29.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58cff9640 10 monclient: dump: 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:29.353 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 10 monclient: _renew_subs 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd58e7fc640 4 set_mon_vals no callback set 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.328+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59009fbe0 con 0x7fd59006cb70 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd59635d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd59635d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd59635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd580044020 con 0x7fd59006cb70 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd590134f70 con 0x7fd59006cb70 2024-09-10T16:20:29.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd58cff9640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 0x7fd57c040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(486..486 src has 184..486) ==== 4755+0+0 (secure 0 0 0) 0x7fd5800733e0 con 0x7fd59006cb70 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 0x7fd57c040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.329+0000 7fd58f7fe640 10 monclient: get_auth_request con 0x7fd57c03e270 auth_method 0 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.332+0000 7fd58f7fe640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 0x7fd57c040730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd59009e890 tx=0x7fd584008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.333+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd58004b030 con 0x7fd59006cb70 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.333+0000 7fd58cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:29.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.333+0000 7fd58cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2117: done (0 left) 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2118: rollback oid 11 current snap is 230 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 228 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2119: copy_from oid 32 from oid 6 current snap is 230 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2120: read oid 36 snap -1 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2120: expect deleted 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2121: rollback oid 15 current snap is 230 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 228 2024-09-10T16:20:29.478 INFO:tasks.rados.rados.0.smithi165.stdout:2122: rollback oid 24 current snap is 230 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 229 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2123: read oid 42 snap -1 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2123: expect deleted 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2124: read oid 31 snap -1 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2124: expect (ObjNum 754 snap 228 seq_num 754) 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2125: rollback oid 48 current snap is 230 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 229 2024-09-10T16:20:29.479 INFO:tasks.rados.rados.0.smithi165.stdout:2126: snap_remove snap 217 2024-09-10T16:20:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.583+0000 7fd59635d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd59635d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd59635d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd5900770f0 con 0x7fd59006cb70 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd58cff9640 1 -- 172.21.15.164:0/1493195802 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fd59009ce50 con 0x7fd59006cb70 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd58cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.584+0000 7fd58cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:29.586 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":136,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 msgr2=0x7fd57c040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 0x7fd57c040730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fd59009e890 tx=0x7fd584008040 comp rx=0 tx=0).stop 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 10 monclient: shutdown 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 msgr2=0x7fd590131610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:29.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd580000c00 tx=0x7fd5800046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 shutdown_connections 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd57c03e270 0x7fd57c040730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd590137ca0 0x7fd59013dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd5900a46a0 0x7fd590131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 --2- 172.21.15.164:0/1493195802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd59006cb70 0x7fd590131610 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 >> 172.21.15.164:0/1493195802 conn(0x7fd5900aa900 msgr2=0x7fd590133660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 shutdown_connections 2024-09-10T16:20:29.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:29.586+0000 7fd59635d640 1 -- 172.21.15.164:0/1493195802 wait complete. 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2126: done (8 left) 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2127: copy_from oid 37 from oid 33 current snap is 230 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2118: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 0 (ObjNum 129 snap 0 seq_num 1162936456) dirty dne 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2122: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2122: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:20:30.481 INFO:tasks.rados.rados.0.smithi165.stdout:2122: finishing rollback tid 2 to smithi16534841-24 2024-09-10T16:20:30.482 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1320 (ObjNum 752 snap 227 seq_num 752) dirty exists 2024-09-10T16:20:30.482 INFO:tasks.rados.rados.0.smithi165.stdout:2124: expect (ObjNum 754 snap 228 seq_num 754) 2024-09-10T16:20:30.532 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 812764 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2121: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2121: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2121: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1306 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2119: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2125: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2125: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:2125: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:20:30.615 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1340 (ObjNum 753 snap 227 seq_num 753) dirty exists 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2119: finishing copy_from to smithi16534841-32 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 643 (ObjNum 660 snap 207 seq_num 660) dirty exists 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2118: done (8 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2119: done (7 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2120: done (6 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2121: done (5 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2122: done (4 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2123: done (3 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2124: done (2 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2125: done (1 left) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2128: read oid 43 snap 227 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2128: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:30.616 INFO:tasks.rados.rados.0.smithi165.stdout:2129: rollback oid 14 current snap is 230 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 219 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2127: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2127: finishing copy_from to smithi16534841-37 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1219 (ObjNum 732 snap 223 seq_num 732) dirty exists 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2127: done (2 left) 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2130: rollback oid 1 current snap is 230 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 219 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2131: write (excl) oid 11 current snap is 230 2024-09-10T16:20:30.617 INFO:tasks.rados.rados.0.smithi165.stdout:2131: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-10T16:20:30.620 INFO:tasks.rados.rados.0.smithi165.stdout:2131: writing smithi16534841-11 from 706092 to 1113283 tid 1 2024-09-10T16:20:30.627 INFO:tasks.rados.rados.0.smithi165.stdout:2131: writing smithi16534841-11 from 1806034 to 2269620 tid 2 2024-09-10T16:20:30.627 INFO:tasks.rados.rados.0.smithi165.stdout:2132: read oid 29 snap -1 2024-09-10T16:20:30.627 INFO:tasks.rados.rados.0.smithi165.stdout:2132: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2133: read oid 33 snap -1 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2133: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2134: delete oid 2 current snap is 230 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2130: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2130: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1164 (ObjNum 694 snap 214 seq_num 694) dirty exists 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:2128: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:30.628 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:30.655 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2129: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2129: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2129: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1174 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2131: finishing write tid 1 to smithi16534841-11 2024-09-10T16:20:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2133: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:20:30.712 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1451459 2024-09-10T16:20:30.726 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 47683 2024-09-10T16:20:30.754 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1527883 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: finishing write tid 2 to smithi16534841-11 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: finishing write tid 3 to smithi16534841-11 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: finishing write tid 4 to smithi16534841-11 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: oid 11 updating version 0 to 1243 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: oid 11 updating version 1243 to 1245 2024-09-10T16:20:30.767 INFO:tasks.rados.rados.0.smithi165.stdout:2131: oid 11 version 1245 is already newer than 1244 2024-09-10T16:20:30.768 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1245 (ObjNum 756 snap 230 seq_num 756) dirty exists 2024-09-10T16:20:30.768 INFO:tasks.rados.rados.0.smithi165.stdout:2131: left oid 11 (ObjNum 756 snap 230 seq_num 756) 2024-09-10T16:20:30.768 INFO:tasks.rados.rados.0.smithi165.stdout:2132: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:20:30.817 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:30.903 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2017210 2024-09-10T16:20:30.906 INFO:tasks.rados.rados.0.smithi165.stdout:2128: done (6 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2129: done (5 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2130: done (4 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2131: done (3 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2132: done (2 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2133: done (1 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2134: done (0 left) 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2135: rollback oid 30 current snap is 230 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 219 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2136: rollback oid 17 current snap is 230 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 228 2024-09-10T16:20:30.907 INFO:tasks.rados.rados.0.smithi165.stdout:2137: rollback oid 26 current snap is 230 2024-09-10T16:20:30.908 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 219 2024-09-10T16:20:30.908 INFO:tasks.rados.rados.0.smithi165.stdout:2138: write (excl) oid 42 current snap is 230 2024-09-10T16:20:30.908 INFO:tasks.rados.rados.0.smithi165.stdout:2138: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-10T16:20:30.913 INFO:tasks.rados.rados.0.smithi165.stdout:2138: writing smithi16534841-42 from 739706 to 1297395 tid 1 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2138: writing smithi16534841-42 from 1806645 to 2273900 tid 2 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2139: read oid 8 snap -1 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2139: expect (ObjNum 705 snap 216 seq_num 705) 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2140: snap_create 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2135: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 0 (ObjNum 177 snap 0 seq_num 1162457120) dirty dne 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2137: finishing rollback tid 0 to smithi16534841-26 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2136: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2136: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:2136: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:20:30.919 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1116 (ObjNum 673 snap 207 seq_num 673) dirty exists 2024-09-10T16:20:30.920 INFO:tasks.rados.rados.0.smithi165.stdout:2137: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:20:30.920 INFO:tasks.rados.rados.0.smithi165.stdout:2137: finishing rollback tid 2 to smithi16534841-26 2024-09-10T16:20:30.920 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1350 (ObjNum 585 snap 186 seq_num 585) dirty exists 2024-09-10T16:20:30.920 INFO:tasks.rados.rados.0.smithi165.stdout:2138: finishing write tid 1 to smithi16534841-42 2024-09-10T16:20:30.921 INFO:tasks.rados.rados.0.smithi165.stdout:2138: finishing write tid 2 to smithi16534841-42 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: finishing write tid 3 to smithi16534841-42 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: finishing write tid 4 to smithi16534841-42 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: oid 42 updating version 0 to 1248 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: oid 42 version 1248 is already newer than 1246 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: oid 42 version 1248 is already newer than 1247 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1248 (ObjNum 758 snap 230 seq_num 758) dirty exists 2024-09-10T16:20:30.922 INFO:tasks.rados.rados.0.smithi165.stdout:2138: left oid 42 (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:20:30.924 INFO:tasks.rados.rados.0.smithi165.stdout:2139: expect (ObjNum 705 snap 216 seq_num 705) 2024-09-10T16:20:31.018 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3550012 2024-09-10T16:20:31.026 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:31.466 INFO:tasks.rados.rados.0.smithi165.stdout:2135: done (5 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2136: done (4 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2137: done (3 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2138: done (2 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2139: done (1 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2140: done (0 left) 2024-09-10T16:20:31.467 INFO:tasks.rados.rados.0.smithi165.stdout:2141: snap_create 2024-09-10T16:20:32.471 INFO:tasks.rados.rados.0.smithi165.stdout:2141: done (0 left) 2024-09-10T16:20:32.471 INFO:tasks.rados.rados.0.smithi165.stdout:2142: snap_create 2024-09-10T16:20:32.599 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:32.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 10 monclient: get_monmap_and_config 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 10 monclient: build_initial_monmap 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 10 monclient: monmap: 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:32.707912+0000 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:32.707912+0000 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:32.709 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:32.710 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:32.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 1 Processor -- start 2024-09-10T16:20:32.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 1 -- start start 2024-09-10T16:20:32.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.707+0000 7f658d072640 10 monclient: init 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): picked mon.noname-a con 0x7f6588074af0 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f65880a1090 0x7f65880a1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): picked mon.noname-c con 0x7f65880a1090 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f65880a3310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): picked mon.noname-b con 0x7f65880a2ec0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f65880698a0 con 0x7f6588074af0 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f658806bfb0 con 0x7f65880a1090 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6588068d70 con 0x7f65880a2ec0 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658d072640 10 monclient(hunting): authenticate will time out at 1611.195168s 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f65880a3310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36382/0 (socket says 172.21.15.164:36382) 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 -- 172.21.15.164:0/1184757570 learned_addr learned my addr 172.21.15.164:0/1184757570 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:32.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 10 monclient(hunting): get_auth_request con 0x7f6588074af0 auth_method 0 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): get_auth_request con 0x7f65880a2ec0 auth_method 0 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 -- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 msgr2=0x7f6588137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 -- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 msgr2=0x7f6588137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:32.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f6586d9d640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658659c640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.165:3300/0 conn(0x7f65880a1090 0x7f65880a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658659c640 10 monclient(hunting): get_auth_request con 0x7f65880a1090 auth_method 0 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient(hunting): handle_auth_done global_id 114536 payload 274 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 -- 172.21.15.164:0/1184757570 >> v2:172.21.15.165:3300/0 conn(0x7f65880a1090 msgr2=0x7f65880a1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.165:3300/0 conn(0x7f65880a1090 0x7f65880a1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 -- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 msgr2=0x7f6588137e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient: _finish_hunting 0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 monclient: found mon.noname-b 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.708+0000 7f658759e640 1 -- 172.21.15.164:0/1184757570 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f658809ebc0 con 0x7f65880a2ec0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f658759e640 1 -- 172.21.15.164:0/1184757570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 msgr2=0x7f65880a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f658759e640 1 --2- 172.21.15.164:0/1184757570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f65880a3310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6588135a30 tx=0x7f658002a4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cf9a8cc81a66c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 1 -- 172.21.15.164:0/1184757570 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580007950 con 0x7f65880a2ec0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: dump: 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:32.721 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: _finish_auth 0 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f658d072640 5 monclient: authenticate success, global_id 114536 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 1 -- 172.21.15.164:0/1184757570 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6580007af0 con 0x7f65880a2ec0 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:32.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f658d072640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f658d072640 4 set_mon_vals no callback set 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 1 -- 172.21.15.164:0/1184757570 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580031400 con 0x7f65880a2ec0 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.711+0000 7f6585d9b640 10 monclient: dump: 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.723 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:32.724 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.724 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:32.724 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:32.724 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:32.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 10 monclient: shutdown 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 -- 172.21.15.164:0/1184757570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 msgr2=0x7f65880a3310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 --2- 172.21.15.164:0/1184757570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f65880a3310 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f6588135a30 tx=0x7f658002a4e0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 -- 172.21.15.164:0/1184757570 shutdown_connections 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 --2- 172.21.15.164:0/1184757570 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f65880a3310 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.165:3300/0 conn(0x7f65880a1090 0x7f65880a1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 --2- 172.21.15.164:0/1184757570 >> v2:172.21.15.164:3300/0 conn(0x7f6588074af0 0x7f6588137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 -- 172.21.15.164:0/1184757570 >> 172.21.15.164:0/1184757570 conn(0x7f65880a3fb0 msgr2=0x7f65880a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 -- 172.21.15.164:0/1184757570 shutdown_connections 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.712+0000 7f658d072640 1 -- 172.21.15.164:0/1184757570 wait complete. 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 10 monclient: build_initial_monmap 2024-09-10T16:20:32.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 10 monclient: monmap: 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 1 Processor -- start 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 1 -- start start 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 10 monclient: init 2024-09-10T16:20:32.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.713+0000 7f658d072640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): picked mon.noname-a con 0x7f6588074af0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65880a1090 0x7f65881cd4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): picked mon.noname-b con 0x7f65880a1090 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f658812c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): picked mon.noname-c con 0x7f65880a2ec0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6588074ed0 con 0x7f6588074af0 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6588069370 con 0x7f65880a1090 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f65880a3850 con 0x7f65880a2ec0 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658d072640 10 monclient(hunting): authenticate will time out at 1611.201080s 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f658812c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36392/0 (socket says 172.21.15.164:36392) 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 -- 172.21.15.164:0/3441142269 learned_addr learned my addr 172.21.15.164:0/3441142269 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): get_auth_request con 0x7f65880a2ec0 auth_method 0 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 10 monclient(hunting): get_auth_request con 0x7f6588074af0 auth_method 0 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 msgr2=0x7f65881ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 msgr2=0x7f65881ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:20:32.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f6586d9d640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658659c640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65880a1090 0x7f65881cd4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658659c640 10 monclient(hunting): get_auth_request con 0x7f65880a1090 auth_method 0 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient(hunting): handle_auth_done global_id 114539 payload 824 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65880a1090 msgr2=0x7f65881cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:32.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65880a1090 0x7f65881cd4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 msgr2=0x7f65881ccfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient: _finish_hunting 0 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 monclient: found mon.noname-c 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.714+0000 7f658759e640 1 -- 172.21.15.164:0/3441142269 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6588130ab0 con 0x7f65880a2ec0 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658759e640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 msgr2=0x7f658812c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658759e640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f658812c570 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6580031000 tx=0x7f6580004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580040070 con 0x7f65880a2ec0 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: dump: 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:32.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:20:32.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65800456a0 con 0x7f65880a2ec0 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658d072640 5 monclient: authenticate success, global_id 114539 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6580045a90 con 0x7f65880a2ec0 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658d072640 10 monclient: _renew_subs 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658d072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f658812fae0 con 0x7f65880a2ec0 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f656f7fe640 10 monclient: dump: 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:32.737 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.716+0000 7f658559a640 4 set_mon_vals no callback set 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 10 monclient: _renew_subs 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f658812df00 con 0x7f65880a2ec0 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6580039040 con 0x7f65880a2ec0 2024-09-10T16:20:32.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.717+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6588130180 con 0x7f65880a2ec0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.718+0000 7f656f7fe640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 0x7f65740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.718+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(490..490 src has 184..490) ==== 4771+0+0 (secure 0 0 0) 0x7f65800746b0 con 0x7f65880a2ec0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.718+0000 7f6586d9d640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 0x7f65740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.718+0000 7f6586d9d640 10 monclient: get_auth_request con 0x7f657403e210 auth_method 0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.721+0000 7f6586d9d640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 0x7f65740406d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f657c0091d0 tx=0x7f657c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.722+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f65800730b0 con 0x7f65880a2ec0 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.722+0000 7f656f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:32.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.722+0000 7f656f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f658d072640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f658d072640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f658d072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f65880a3c30 con 0x7f65880a2ec0 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f656f7fe640 1 -- 172.21.15.164:0/3441142269 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f65800743a0 con 0x7f65880a2ec0 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f656f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:32.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.969+0000 7f656f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:32.972 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":136,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:32.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 msgr2=0x7f65740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:32.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 0x7f65740406d0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f657c0091d0 tx=0x7f657c008040 comp rx=0 tx=0).stop 2024-09-10T16:20:32.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 10 monclient: shutdown 2024-09-10T16:20:32.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:32.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 msgr2=0x7f658812c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f658812c570 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f6580031000 tx=0x7f6580004a40 comp rx=0 tx=0).stop 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 shutdown_connections 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f657403e210 0x7f65740406d0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f65880a2ec0 0x7f658812c570 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f65880a1090 0x7f65881cd4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 --2- 172.21.15.164:0/3441142269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6588074af0 0x7f65881ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.971+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 >> 172.21.15.164:0/3441142269 conn(0x7f65880a3fb0 msgr2=0x7f658809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.972+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 shutdown_connections 2024-09-10T16:20:32.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:32.972+0000 7f658d072640 1 -- 172.21.15.164:0/3441142269 wait complete. 2024-09-10T16:20:33.470 INFO:tasks.rados.rados.0.smithi165.stdout:2142: done (0 left) 2024-09-10T16:20:33.470 INFO:tasks.rados.rados.0.smithi165.stdout:2143: read oid 13 snap -1 2024-09-10T16:20:33.470 INFO:tasks.rados.rados.0.smithi165.stdout:2143: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:33.470 INFO:tasks.rados.rados.0.smithi165.stdout:2144: snap_create 2024-09-10T16:20:33.478 INFO:tasks.rados.rados.0.smithi165.stdout:2143: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:33.478 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:33.570 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1413715 2024-09-10T16:20:33.658 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2036920 2024-09-10T16:20:34.475 INFO:tasks.rados.rados.0.smithi165.stdout:2143: done (1 left) 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2144: done (0 left) 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2145: copy_from oid 37 from oid 33 current snap is 234 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2146: read oid 31 snap 219 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2146: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2147: rollback oid 15 current snap is 234 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 230 2024-09-10T16:20:34.476 INFO:tasks.rados.rados.0.smithi165.stdout:2148: delete oid 16 current snap is 234 2024-09-10T16:20:34.485 INFO:tasks.rados.rados.0.smithi165.stdout:2147: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:20:34.486 INFO:tasks.rados.rados.0.smithi165.stdout:2147: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:20:34.486 INFO:tasks.rados.rados.0.smithi165.stdout:2147: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:20:34.486 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1309 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:20:34.487 INFO:tasks.rados.rados.0.smithi165.stdout:2146: expect (ObjNum 624 snap 194 seq_num 624) 2024-09-10T16:20:34.525 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:34.563 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:34.600 INFO:tasks.rados.rados.0.smithi165.stdout:2145: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2145: finishing copy_from to smithi16534841-37 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1219 (ObjNum 732 snap 223 seq_num 732) dirty exists 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2145: done (3 left) 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2146: done (2 left) 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2147: done (1 left) 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2148: done (0 left) 2024-09-10T16:20:34.601 INFO:tasks.rados.rados.0.smithi165.stdout:2149: delete oid 21 current snap is 234 2024-09-10T16:20:34.604 INFO:tasks.rados.rados.0.smithi165.stdout:2149: done (0 left) 2024-09-10T16:20:34.605 INFO:tasks.rados.rados.0.smithi165.stdout:2150: rollback oid 29 current snap is 234 2024-09-10T16:20:34.605 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 231 2024-09-10T16:20:34.605 INFO:tasks.rados.rados.0.smithi165.stdout:2151: write (excl) oid 47 current snap is 234 2024-09-10T16:20:34.605 INFO:tasks.rados.rados.0.smithi165.stdout:2151: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-10T16:20:34.611 INFO:tasks.rados.rados.0.smithi165.stdout:2151: writing smithi16534841-47 from 790127 to 1289916 tid 1 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:2151: writing smithi16534841-47 from 1698444 to 2429379 tid 2 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:2150: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:2150: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:2150: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1323 (ObjNum 616 snap 190 seq_num 616) dirty exists 2024-09-10T16:20:34.618 INFO:tasks.rados.rados.0.smithi165.stdout:2152: read oid 13 snap -1 2024-09-10T16:20:34.619 INFO:tasks.rados.rados.0.smithi165.stdout:2152: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:34.619 INFO:tasks.rados.rados.0.smithi165.stdout:2150: done (2 left) 2024-09-10T16:20:34.619 INFO:tasks.rados.rados.0.smithi165.stdout:2153: delete oid 3 current snap is 234 2024-09-10T16:20:34.619 INFO:tasks.rados.rados.0.smithi165.stdout:2151: finishing write tid 1 to smithi16534841-47 2024-09-10T16:20:34.621 INFO:tasks.rados.rados.0.smithi165.stdout:2152: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:34.628 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 885806 2024-09-10T16:20:34.796 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2267652 2024-09-10T16:20:34.810 INFO:tasks.rados.rados.0.smithi165.stdout:2151: finishing write tid 2 to smithi16534841-47 2024-09-10T16:20:34.810 INFO:tasks.rados.rados.0.smithi165.stdout:2151: finishing write tid 3 to smithi16534841-47 2024-09-10T16:20:34.810 INFO:tasks.rados.rados.0.smithi165.stdout:2151: finishing write tid 4 to smithi16534841-47 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2151: oid 47 updating version 0 to 1324 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2151: oid 47 updating version 1324 to 1325 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2151: oid 47 updating version 1325 to 1326 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1326 (ObjNum 761 snap 234 seq_num 761) dirty exists 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2151: left oid 47 (ObjNum 761 snap 234 seq_num 761) 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2151: done (2 left) 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2152: done (1 left) 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2153: done (0 left) 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2154: read oid 43 snap -1 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2154: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2155: write oid 8 current snap is 234 2024-09-10T16:20:34.811 INFO:tasks.rados.rados.0.smithi165.stdout:2155: seq_num 763 ranges {423741=650287,1699055=618251} 2024-09-10T16:20:34.816 INFO:tasks.rados.rados.0.smithi165.stdout:2155: writing smithi16534841-8 from 423741 to 1074028 tid 1 2024-09-10T16:20:34.823 INFO:tasks.rados.rados.0.smithi165.stdout:2155: writing smithi16534841-8 from 1699055 to 2317306 tid 2 2024-09-10T16:20:34.823 INFO:tasks.rados.rados.0.smithi165.stdout:2156: write oid 21 current snap is 234 2024-09-10T16:20:34.823 INFO:tasks.rados.rados.0.smithi165.stdout:2156: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-10T16:20:34.829 INFO:tasks.rados.rados.0.smithi165.stdout:2156: writing smithi16534841-21 from 440548 to 1166084 tid 1 2024-09-10T16:20:34.835 INFO:tasks.rados.rados.0.smithi165.stdout:2156: writing smithi16534841-21 from 1757537 to 2235799 tid 2 2024-09-10T16:20:34.838 INFO:tasks.rados.rados.0.smithi165.stdout:2156: writing smithi16534841-21 from 2813990 to 2840548 tid 3 2024-09-10T16:20:34.839 INFO:tasks.rados.rados.0.smithi165.stdout:2157: write (excl) oid 9 current snap is 234 2024-09-10T16:20:34.839 INFO:tasks.rados.rados.0.smithi165.stdout:2157: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-10T16:20:34.843 INFO:tasks.rados.rados.0.smithi165.stdout:2157: writing smithi16534841-9 from 457355 to 858140 tid 1 2024-09-10T16:20:34.850 INFO:tasks.rados.rados.0.smithi165.stdout:2157: writing smithi16534841-9 from 1416019 to 2037939 tid 2 2024-09-10T16:20:34.855 INFO:tasks.rados.rados.0.smithi165.stdout:2157: writing smithi16534841-9 from 2725060 to 2857355 tid 3 2024-09-10T16:20:34.855 INFO:tasks.rados.rados.0.smithi165.stdout:2158: rollback oid 3 current snap is 234 2024-09-10T16:20:34.855 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 231 2024-09-10T16:20:34.855 INFO:tasks.rados.rados.0.smithi165.stdout:2154: expect (ObjNum 742 snap 225 seq_num 742) 2024-09-10T16:20:34.855 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:34.934 INFO:tasks.rados.rados.0.smithi165.stdout:2155: finishing write tid 1 to smithi16534841-8 2024-09-10T16:20:34.935 INFO:tasks.rados.rados.0.smithi165.stdout:2154: done (4 left) 2024-09-10T16:20:34.935 INFO:tasks.rados.rados.0.smithi165.stdout:2159: snap_remove snap 232 2024-09-10T16:20:35.514 INFO:tasks.rados.rados.0.smithi165.stdout:2159: done (4 left) 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2160: rollback oid 1 current snap is 234 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 218 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: finishing write tid 2 to smithi16534841-8 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: finishing write tid 3 to smithi16534841-8 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: finishing write tid 4 to smithi16534841-8 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: oid 8 updating version 0 to 1327 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: oid 8 updating version 1327 to 1329 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: oid 8 version 1329 is already newer than 1328 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1329 (ObjNum 763 snap 234 seq_num 763) dirty exists 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2155: left oid 8 (ObjNum 763 snap 234 seq_num 763) 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2156: finishing write tid 1 to smithi16534841-21 2024-09-10T16:20:35.515 INFO:tasks.rados.rados.0.smithi165.stdout:2156: finishing write tid 2 to smithi16534841-21 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: finishing write tid 3 to smithi16534841-21 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: finishing write tid 4 to smithi16534841-21 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: finishing write tid 5 to smithi16534841-21 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: oid 21 updating version 0 to 1231 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: oid 21 version 1231 is already newer than 1229 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: oid 21 version 1231 is already newer than 1230 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: oid 21 version 1231 is already newer than 1228 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1231 (ObjNum 764 snap 234 seq_num 764) dirty exists 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2156: left oid 21 (ObjNum 764 snap 234 seq_num 764) 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2157: finishing write tid 1 to smithi16534841-9 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2157: finishing write tid 2 to smithi16534841-9 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2157: finishing write tid 3 to smithi16534841-9 2024-09-10T16:20:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2157: finishing write tid 4 to smithi16534841-9 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: finishing write tid 5 to smithi16534841-9 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: oid 9 updating version 0 to 1178 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: oid 9 version 1178 is already newer than 1175 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: oid 9 version 1178 is already newer than 1177 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: oid 9 version 1178 is already newer than 1176 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1178 (ObjNum 765 snap 234 seq_num 765) dirty exists 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: left oid 9 (ObjNum 765 snap 234 seq_num 765) 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2158: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2158: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1233 (ObjNum 736 snap 223 seq_num 736) dirty exists 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2155: done (4 left) 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2156: done (3 left) 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2157: done (2 left) 2024-09-10T16:20:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2158: done (1 left) 2024-09-10T16:20:35.518 INFO:tasks.rados.rados.0.smithi165.stdout:2161: snap_remove snap 219 2024-09-10T16:20:35.984 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:36.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.092+0000 7f7a5f73d640 10 monclient: get_monmap_and_config 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.092+0000 7f7a5f73d640 10 monclient: build_initial_monmap 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.092+0000 7f7a5f73d640 10 monclient: monmap: 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:36.093430+0000 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:36.093430+0000 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:36.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 Processor -- start 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 -- start start 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient: init 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T16:20:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7a5809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5809d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a5813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a5809a430 con 0x7f7a5809c180 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7a580a24e0 con 0x7f7a5809d160 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7a580a2bf0 con 0x7f7a5813a5c0 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5f73d640 10 monclient(hunting): authenticate will time out at 1614.580851s 2024-09-10T16:20:36.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5dcda640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5d4d9640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7a5809c180 0x7f7a5809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:43148/0 (socket says 172.21.15.164:43148) 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.093+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 learned_addr learned my addr 172.21.15.164:0/1044915479 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a5809d160 auth_method 0 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.094+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:36.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.096+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v1:172.21.15.164:6789/0 conn(0x7f7a5809c180 legacy=0x7f7a5809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:36.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.295+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a5809d160 auth_method 0 2024-09-10T16:20:36.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.296+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.696+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a5809d160 auth_method 0 2024-09-10T16:20:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:36.698+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:37.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:37.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.498+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a5809d160 auth_method 0 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:37.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:37.499+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 10 monclient(hunting): tick 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 monclient(hunting): continuing hunt 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a5813c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:39.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a5813c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 msgr2=0x7f7a5809f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.165:3300/0 conn(0x7f7a5809d160 0x7f7a5809f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.093+0000 7f7a4effd640 1 -- 172.21.15.164:0/1044915479 >> v1:172.21.15.164:6789/0 conn(0x7f7a5809c180 legacy=0x7f7a5809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 10 monclient(hunting): picked mon.noname-a con 0x7f7a3c003770 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7a3c0062c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 1 -- 172.21.15.164:0/1044915479 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a5809c560 con 0x7f7a3c0062c0 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 1 -- 172.21.15.164:0/1044915479 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7a580a24e0 con 0x7f7a3c003770 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a4effd640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:39.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a3c003770 auth_method 0 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.094+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:39.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.096+0000 7f7a5ccd8640 1 -- 172.21.15.164:0/1044915479 >> v1:172.21.15.165:6789/0 conn(0x7f7a3c0062c0 legacy=0x7f7a3c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:39.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:39.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a3c003770 auth_method 0 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:39.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.296+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:39.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:39.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a3c003770 auth_method 0 2024-09-10T16:20:39.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:39.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:39.697+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:40.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a3c003770 auth_method 0 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:40.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.498+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:20:40.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2573510697 0 0) 0x7f7a5809c560 con 0x7f7a3c0062c0 2024-09-10T16:20:40.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): _negotiate 2024-09-10T16:20:40.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:40.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): my global_id is 154171 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a44003520 con 0x7f7a3c0062c0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1223705945 0 0) 0x7f7a44003520 con 0x7f7a3c0062c0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 msgr2=0x7f7a3c005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 --2- 172.21.15.164:0/1044915479 >> v2:172.21.15.164:3300/0 conn(0x7f7a3c003770 0x7f7a3c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c00b090 con 0x7f7a3c0062c0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _finish_auth 0 2024-09-10T16:20:40.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f7a4800d680 con 0x7f7a3c0062c0 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5f73d640 5 monclient: authenticate success, global_id 154171 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5f73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: dump: 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 >> v1:172.21.15.165:6789/0 conn(0x7f7a3c0062c0 legacy=0x7f7a3c008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5dcda640 1 -- 172.21.15.164:0/1044915479 reap_dead start 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a44003b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): picked mon.c con 0x7f7a5813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 0x7f7a44004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): picked mon.b con 0x7f7a44004170 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:40.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004f40 0x7f7a4400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): picked mon.a con 0x7f7a44004f40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7a580a24e0 con 0x7f7a44004f40 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a4400bf00 con 0x7f7a44004170 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7a4400c080 con 0x7f7a5813a5c0 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:40.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a4ffff640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5f73d640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a44003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 0x7f7a44004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.555+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a44004170 auth_method 0 2024-09-10T16:20:40.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_done global_id 154171 payload 293 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 msgr2=0x7f7a44004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 0x7f7a44004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004f40 msgr2=0x7f7a4400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:40.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004f40 0x7f7a4400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient: _finish_hunting 0 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 monclient: found mon.c 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a44003520 con 0x7f7a5813a5c0 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient: _finish_auth 0 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 10 monclient: _check_auth_tickets 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5d4d9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:40.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.556+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 0x7f7a44004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:40.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a44003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:40.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a44003b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005e60 tx=0x7f7a44005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a81cbc554e30cf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:40.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54031070 con 0x7f7a5813a5c0 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: dump: 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:40.573 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a5402b9d0 con 0x7f7a5813a5c0 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a5f73d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 1 -- 172.21.15.164:0/1044915479 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54005ab0 con 0x7f7a5813a5c0 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a5f73d640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:40.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a5f73d640 4 set_mon_vals no callback set 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.559+0000 7f7a4ffff640 10 monclient: dump: 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 10 monclient: shutdown 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 -- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a44003b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a44003b10 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005e60 tx=0x7f7a44005830 comp rx=0 tx=0).stop 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 -- 172.21.15.164:0/1044915479 shutdown_connections 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004f40 0x7f7a4400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004170 0x7f7a44004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/1044915479 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a44003b10 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 -- 172.21.15.164:0/1044915479 >> 172.21.15.164:0/1044915479 conn(0x7f7a580a3110 msgr2=0x7f7a580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 -- 172.21.15.164:0/1044915479 shutdown_connections 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.560+0000 7f7a5f73d640 1 -- 172.21.15.164:0/1044915479 wait complete. 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.561+0000 7f7a5f73d640 10 monclient: build_initial_monmap 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.561+0000 7f7a5f73d640 10 monclient: monmap: 2024-09-10T16:20:40.576 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 Processor -- start 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 -- start start 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient: init 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:40.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-a con 0x7f7a44004170 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004f40 0x7f7a58092280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-b con 0x7f7a44004f40 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): picked mon.noname-c con 0x7f7a5813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58074f90 con 0x7f7a44004170 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7a58074e10 con 0x7f7a44004f40 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7a5809ea40 con 0x7f7a5813a5c0 2024-09-10T16:20:40.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5f73d640 10 monclient(hunting): authenticate will time out at 1619.049885s 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.562+0000 7f7a5d4d9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:52586/0 (socket says 172.21.15.164:52586) 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/4100741385 learned_addr learned my addr 172.21.15.164:0/4100741385 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request con 0x7f7a44004170 auth_method 0 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request con 0x7f7a5813a5c0 auth_method 0 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004f40 0x7f7a58092280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request con 0x7f7a44004f40 auth_method 0 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a580927c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a580927c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5dcda640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:40.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5ccd8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient(hunting): handle_auth_done global_id 164097 payload 824 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 msgr2=0x7f7a580927c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004f40 msgr2=0x7f7a58092280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004f40 0x7f7a58092280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient: _finish_hunting 0 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 monclient: found mon.noname-a 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.563+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/4100741385 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a58095df0 con 0x7f7a44004170 2024-09-10T16:20:40.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5d4d9640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 msgr2=0x7f7a58099670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5d4d9640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005f80 tx=0x7f7a54001460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54044020 con 0x7f7a44004170 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: dump: 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.582 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: _finish_auth 0 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a54004bf0 con 0x7f7a44004170 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5f73d640 5 monclient: authenticate success, global_id 164097 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5f73d640 10 monclient: _renew_subs 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5f73d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a58094170 con 0x7f7a44004170 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a54005040 con 0x7f7a44004170 2024-09-10T16:20:40.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.566+0000 7f7a4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.566+0000 7f7a4dffb640 10 monclient: dump: 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:40.584 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.566+0000 7f7a5f73d640 10 monclient: _renew_subs 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.565+0000 7f7a4f7fe640 4 set_mon_vals no callback set 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.566+0000 7f7a5f73d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.566+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a58095b10 con 0x7f7a44004170 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7a54047020 con 0x7f7a44004170 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a5f73d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:40.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a5f73d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a5f73d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a5809ee20 con 0x7f7a44004170 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a4dffb640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 0x7f7a3c0410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.567+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(493..493 src has 184..493) ==== 4815+0+0 (secure 0 0 0) 0x7f7a54073cd0 con 0x7f7a44004170 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.568+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 0x7f7a3c0410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.568+0000 7f7a5ccd8640 10 monclient: get_auth_request con 0x7f7a3c03ec00 auth_method 0 2024-09-10T16:20:40.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.570+0000 7f7a5ccd8640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 0x7f7a3c0410c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800d310 tx=0x7f7a48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:40.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.573+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7a5404e030 con 0x7f7a44004170 2024-09-10T16:20:40.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.573+0000 7f7a4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:40.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.573+0000 7f7a4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a5f73d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a5f73d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a5f73d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a580a5720 con 0x7f7a44004170 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a4dffb640 1 -- 172.21.15.164:0/4100741385 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f7a540726b0 con 0x7f7a44004170 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:40.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.819+0000 7f7a4dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:40.822 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":140,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:40.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 msgr2=0x7f7a3c0410c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 0x7f7a3c0410c0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7a4800d310 tx=0x7f7a48008040 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 10 monclient: shutdown 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 msgr2=0x7f7a58099670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f7a54005f80 tx=0x7f7a54001460 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 shutdown_connections 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7a3c03ec00 0x7f7a3c0410c0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7a5813a5c0 0x7f7a580927c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7a44004f40 0x7f7a58092280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 --2- 172.21.15.164:0/4100741385 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7a44004170 0x7f7a58099670 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:40.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 >> 172.21.15.164:0/4100741385 conn(0x7f7a580a3110 msgr2=0x7f7a580729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:40.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 shutdown_connections 2024-09-10T16:20:40.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:40.822+0000 7f7a5f73d640 1 -- 172.21.15.164:0/4100741385 wait complete. 2024-09-10T16:20:42.008 INFO:tasks.rados.rados.0.smithi165.stdout:2161: done (1 left) 2024-09-10T16:20:42.008 INFO:tasks.rados.rados.0.smithi165.stdout:2162: read oid 13 snap -1 2024-09-10T16:20:42.008 INFO:tasks.rados.rados.0.smithi165.stdout:2162: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:42.008 INFO:tasks.rados.rados.0.smithi165.stdout:2163: copy_from oid 32 from oid 36 current snap is 234 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:2160: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:2160: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:2160: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1167 (ObjNum 694 snap 214 seq_num 694) dirty exists 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:2160: done (2 left) 2024-09-10T16:20:42.009 INFO:tasks.rados.rados.0.smithi165.stdout:2164: delete oid 45 current snap is 234 2024-09-10T16:20:42.014 INFO:tasks.rados.rados.0.smithi165.stdout:2163: finishing copy_from to smithi16534841-32 2024-09-10T16:20:42.014 INFO:tasks.rados.rados.0.smithi165.stdout:2163: got expected ENOENT (src dne) 2024-09-10T16:20:42.016 INFO:tasks.rados.rados.0.smithi165.stdout:2163: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:20:42.017 INFO:tasks.rados.rados.0.smithi165.stdout:2163: done (2 left) 2024-09-10T16:20:42.017 INFO:tasks.rados.rados.0.smithi165.stdout:2164: done (1 left) 2024-09-10T16:20:42.017 INFO:tasks.rados.rados.0.smithi165.stdout:2165: read oid 32 snap 229 2024-09-10T16:20:42.017 INFO:tasks.rados.rados.0.smithi165.stdout:2165: expect (ObjNum 740 snap 225 seq_num 740) 2024-09-10T16:20:42.017 INFO:tasks.rados.rados.0.smithi165.stdout:2166: snap_create 2024-09-10T16:20:42.020 INFO:tasks.rados.rados.0.smithi165.stdout:2162: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:20:42.020 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:42.096 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:42.168 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:42.241 INFO:tasks.rados.rados.0.smithi165.stdout:2165: expect (ObjNum 740 snap 225 seq_num 740) 2024-09-10T16:20:43.015 INFO:tasks.rados.rados.0.smithi165.stdout:2162: done (2 left) 2024-09-10T16:20:43.015 INFO:tasks.rados.rados.0.smithi165.stdout:2165: done (1 left) 2024-09-10T16:20:43.016 INFO:tasks.rados.rados.0.smithi165.stdout:2166: done (0 left) 2024-09-10T16:20:43.016 INFO:tasks.rados.rados.0.smithi165.stdout:2167: read oid 42 snap 234 2024-09-10T16:20:43.016 INFO:tasks.rados.rados.0.smithi165.stdout:2167: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:20:43.016 INFO:tasks.rados.rados.0.smithi165.stdout:2168: write oid 5 current snap is 235 2024-09-10T16:20:43.016 INFO:tasks.rados.rados.0.smithi165.stdout:2168: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-10T16:20:43.020 INFO:tasks.rados.rados.0.smithi165.stdout:2168: writing smithi16534841-5 from 490969 to 1042252 tid 1 2024-09-10T16:20:43.028 INFO:tasks.rados.rados.0.smithi165.stdout:2168: writing smithi16534841-5 from 1816630 to 2442219 tid 2 2024-09-10T16:20:43.028 INFO:tasks.rados.rados.0.smithi165.stdout:2169: write oid 26 current snap is 235 2024-09-10T16:20:43.028 INFO:tasks.rados.rados.0.smithi165.stdout:2169: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-10T16:20:43.034 INFO:tasks.rados.rados.0.smithi165.stdout:2169: writing smithi16534841-26 from 507776 to 1250661 tid 1 2024-09-10T16:20:43.042 INFO:tasks.rados.rados.0.smithi165.stdout:2169: writing smithi16534841-26 from 1991465 to 2477065 tid 2 2024-09-10T16:20:43.042 INFO:tasks.rados.rados.0.smithi165.stdout:2170: delete oid 25 current snap is 235 2024-09-10T16:20:43.042 INFO:tasks.rados.rados.0.smithi165.stdout:2167: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:20:43.053 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2017906 2024-09-10T16:20:43.094 INFO:tasks.rados.rados.0.smithi165.stdout:2168: finishing write tid 1 to smithi16534841-5 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: finishing write tid 2 to smithi16534841-5 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: finishing write tid 3 to smithi16534841-5 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: finishing write tid 4 to smithi16534841-5 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: oid 5 updating version 0 to 1308 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: oid 5 updating version 1308 to 1309 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: oid 5 version 1309 is already newer than 1307 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1309 (ObjNum 767 snap 235 seq_num 767) dirty exists 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2168: left oid 5 (ObjNum 767 snap 235 seq_num 767) 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2169: finishing write tid 1 to smithi16534841-26 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2169: finishing write tid 2 to smithi16534841-26 2024-09-10T16:20:43.095 INFO:tasks.rados.rados.0.smithi165.stdout:2167: done (3 left) 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2168: done (2 left) 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2170: done (1 left) 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2171: copy_from oid 22 from oid 48 current snap is 235 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: finishing write tid 3 to smithi16534841-26 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: finishing write tid 4 to smithi16534841-26 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: oid 26 updating version 0 to 1364 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: oid 26 updating version 1364 to 1366 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: oid 26 version 1366 is already newer than 1365 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1366 (ObjNum 768 snap 235 seq_num 768) dirty exists 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: left oid 26 (ObjNum 768 snap 235 seq_num 768) 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2169: done (1 left) 2024-09-10T16:20:43.096 INFO:tasks.rados.rados.0.smithi165.stdout:2172: snap_create 2024-09-10T16:20:43.104 INFO:tasks.rados.rados.0.smithi165.stdout:2171: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:20:43.106 INFO:tasks.rados.rados.0.smithi165.stdout:2171: finishing copy_from to smithi16534841-22 2024-09-10T16:20:43.106 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1340 (ObjNum 753 snap 227 seq_num 753) dirty exists 2024-09-10T16:20:43.835 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:43.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.942+0000 7f3056ba2640 10 monclient: get_monmap_and_config 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.942+0000 7f3056ba2640 10 monclient: build_initial_monmap 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.942+0000 7f3056ba2640 10 monclient: monmap: 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:43.943403+0000 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:43.943403+0000 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:43.944 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:43.945 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:43.945 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:43.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 Processor -- start 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 -- start start 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient: init 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-10T16:20:43.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f305009c180 0x7f305009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f305009c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f305009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f305013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f305009a430 con 0x7f305009d160 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f30500a24e0 con 0x7f305009c180 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f30500a2bf0 con 0x7f305013a5c0 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f3056ba2640 10 monclient(hunting): authenticate will time out at 1622.430834s 2024-09-10T16:20:43.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43072/0 (socket says 172.21.15.164:43072) 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 1 -- 172.21.15.164:0/4038121777 learned_addr learned my addr 172.21.15.164:0/4038121777 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 10 monclient(hunting): get_auth_request con 0x7f305013a5c0 auth_method 0 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.943+0000 7f305513f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:43.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient(hunting): handle_auth_done global_id 114542 payload 274 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 1 -- 172.21.15.164:0/4038121777 >> v2:172.21.15.165:3300/0 conn(0x7f305009c180 msgr2=0x7f305009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 1 --2- 172.21.15.164:0/4038121777 >> v2:172.21.15.165:3300/0 conn(0x7f305009c180 0x7f305009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 1 -- 172.21.15.164:0/4038121777 >> v1:172.21.15.165:6789/0 conn(0x7f305009d160 legacy=0x7f305009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient: _finish_hunting 0 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 1 monclient: found mon.noname-b 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:43.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.944+0000 7f305513f640 1 -- 172.21.15.164:0/4038121777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305009c560 con 0x7f305013a5c0 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f305513f640 1 -- 172.21.15.164:0/4038121777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 msgr2=0x7f305013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f305513f640 1 --2- 172.21.15.164:0/4038121777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3048009e80 tx=0x7f304802a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3aa24dd380be7914 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 1 -- 172.21.15.164:0/4038121777 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048036070 con 0x7f305013a5c0 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:43.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: dump: 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:43.957 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 1 -- 172.21.15.164:0/4038121777 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3048007800 con 0x7f305013a5c0 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 5 monclient: authenticate success, global_id 114542 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 4 set_mon_vals no callback set 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 1 -- 172.21.15.164:0/4038121777 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048007bd0 con 0x7f305013a5c0 2024-09-10T16:20:43.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f304f7fe640 10 monclient: dump: 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.959 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 10 monclient: shutdown 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 -- 172.21.15.164:0/4038121777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 msgr2=0x7f305013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 --2- 172.21.15.164:0/4038121777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3048009e80 tx=0x7f304802a580 comp rx=0 tx=0).stop 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 -- 172.21.15.164:0/4038121777 shutdown_connections 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 --2- 172.21.15.164:0/4038121777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305013a5c0 0x7f305013c9b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 --2- 172.21.15.164:0/4038121777 >> v2:172.21.15.165:3300/0 conn(0x7f305009c180 0x7f305009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 -- 172.21.15.164:0/4038121777 >> 172.21.15.164:0/4038121777 conn(0x7f30500a3110 msgr2=0x7f30500a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:43.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 -- 172.21.15.164:0/4038121777 shutdown_connections 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.947+0000 7f3056ba2640 1 -- 172.21.15.164:0/4038121777 wait complete. 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.948+0000 7f3056ba2640 10 monclient: build_initial_monmap 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.948+0000 7f3056ba2640 10 monclient: monmap: 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.948+0000 7f3056ba2640 1 Processor -- start 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.948+0000 7f3056ba2640 1 -- start start 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.948+0000 7f3056ba2640 10 monclient: init 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-c con 0x7f305009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 0x7f30500925a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-b con 0x7f305009d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): picked mon.noname-a con 0x7f305013a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3050074f90 con 0x7f305013a5c0 2024-09-10T16:20:43.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3050074e10 con 0x7f305009d160 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f305009f690 con 0x7f305009c180 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f3056ba2640 10 monclient(hunting): authenticate will time out at 1622.436609s 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43074/0 (socket says 172.21.15.164:43074) 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:52594/0 (socket says 172.21.15.164:52594) 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 1 -- 172.21.15.164:0/131440126 learned_addr learned my addr 172.21.15.164:0/131440126 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): get_auth_request con 0x7f305009c180 auth_method 0 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:43.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): get_auth_request con 0x7f305013a5c0 auth_method 0 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305493e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f305513f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f304ffff640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 0x7f30500925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:43.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f304ffff640 10 monclient(hunting): get_auth_request con 0x7f305009d160 auth_method 0 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f304ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f304ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.949+0000 7f304ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 10 monclient(hunting): handle_auth_done global_id 164100 payload 824 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 -- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 msgr2=0x7f3050139f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 -- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 msgr2=0x7f30500925a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 0x7f30500925a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 10 monclient: _finish_hunting 0 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 monclient: found mon.noname-a 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:43.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305513f640 1 -- 172.21.15.164:0/131440126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f305009d620 con 0x7f305013a5c0 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f305493e640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.950+0000 7f304ffff640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 0x7f30500925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f305513f640 1 -- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 msgr2=0x7f3050092ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f305513f640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3048033000 tx=0x7f3048004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048041020 con 0x7f305013a5c0 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: dump: 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.966 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 5 monclient: authenticate success, global_id 164100 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3048037040 con 0x7f305013a5c0 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 10 monclient: _renew_subs 2024-09-10T16:20:43.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3048036070 con 0x7f305013a5c0 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3050095c10 con 0x7f305013a5c0 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304d7fa640 10 monclient: dump: 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:43.968 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f304effd640 4 set_mon_vals no callback set 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 10 monclient: _renew_subs 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.952+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30500961d0 con 0x7f305013a5c0 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.953+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3048044020 con 0x7f305013a5c0 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.953+0000 7f3056ba2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.953+0000 7f3056ba2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.953+0000 7f3056ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.953+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f305009ea40 con 0x7f305013a5c0 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.954+0000 7f304d7fa640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 0x7f3034040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:43.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.954+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(495..495 src has 184..495) ==== 4803+0+0 (secure 0 0 0) 0x7f3048073aa0 con 0x7f305013a5c0 2024-09-10T16:20:43.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.954+0000 7f305493e640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 0x7f3034040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:43.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.954+0000 7f305493e640 10 monclient: get_auth_request con 0x7f303403e260 auth_method 0 2024-09-10T16:20:43.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.956+0000 7f305493e640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 0x7f3034040720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3044009e10 tx=0x7f30440091c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:43.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.958+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f304804b030 con 0x7f305013a5c0 2024-09-10T16:20:43.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.958+0000 7f304d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:43.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:43.958+0000 7f304d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:44.019 INFO:tasks.rados.rados.0.smithi165.stdout:2171: done (1 left) 2024-09-10T16:20:44.019 INFO:tasks.rados.rados.0.smithi165.stdout:2172: done (0 left) 2024-09-10T16:20:44.019 INFO:tasks.rados.rados.0.smithi165.stdout:2173: read oid 5 snap -1 2024-09-10T16:20:44.019 INFO:tasks.rados.rados.0.smithi165.stdout:2173: expect (ObjNum 767 snap 235 seq_num 767) 2024-09-10T16:20:44.020 INFO:tasks.rados.rados.0.smithi165.stdout:2174: delete oid 13 current snap is 236 2024-09-10T16:20:44.027 INFO:tasks.rados.rados.0.smithi165.stdout:2174: done (1 left) 2024-09-10T16:20:44.027 INFO:tasks.rados.rados.0.smithi165.stdout:2175: read oid 34 snap 235 2024-09-10T16:20:44.027 INFO:tasks.rados.rados.0.smithi165.stdout:2175: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:20:44.027 INFO:tasks.rados.rados.0.smithi165.stdout:2176: snap_create 2024-09-10T16:20:44.037 INFO:tasks.rados.rados.0.smithi165.stdout:2175: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:20:44.041 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 692174 2024-09-10T16:20:44.069 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 529932 2024-09-10T16:20:44.105 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 149795 2024-09-10T16:20:44.124 INFO:tasks.rados.rados.0.smithi165.stdout:2173: expect (ObjNum 767 snap 235 seq_num 767) 2024-09-10T16:20:44.169 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:44.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f3056ba2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:44.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f3056ba2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:44.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f3056ba2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:44.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f305008f690 con 0x7f305013a5c0 2024-09-10T16:20:44.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f304d7fa640 1 -- 172.21.15.164:0/131440126 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f305009d620 con 0x7f305013a5c0 2024-09-10T16:20:44.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f304d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:44.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.206+0000 7f304d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:44.208 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 msgr2=0x7f3034040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 0x7f3034040720 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f3044009e10 tx=0x7f30440091c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 10 monclient: shutdown 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 msgr2=0x7f3050092ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3048033000 tx=0x7f3048004a40 comp rx=0 tx=0).stop 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 shutdown_connections 2024-09-10T16:20:44.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f303403e260 0x7f3034040720 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f305013a5c0 0x7f3050092ae0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f305009d160 0x7f30500925a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 --2- 172.21.15.164:0/131440126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f305009c180 0x7f3050139f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 >> 172.21.15.164:0/131440126 conn(0x7f30500a3110 msgr2=0x7f3050073c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 shutdown_connections 2024-09-10T16:20:44.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.208+0000 7f3056ba2640 1 -- 172.21.15.164:0/131440126 wait complete. 2024-09-10T16:20:44.221 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:20:44.221 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:20:44.221 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:20:44.316 INFO:tasks.rados.rados.0.smithi165.stdout:2173: done (2 left) 2024-09-10T16:20:44.316 INFO:tasks.rados.rados.0.smithi165.stdout:2175: done (1 left) 2024-09-10T16:20:44.316 INFO:tasks.rados.rados.0.smithi165.stdout:2176: done (0 left) 2024-09-10T16:20:44.317 INFO:tasks.rados.rados.0.smithi165.stdout:2177: write (excl) oid 30 current snap is 237 2024-09-10T16:20:44.317 INFO:tasks.rados.rados.0.smithi165.stdout:2177: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.320+0000 7fed77c72640 10 monclient: get_monmap_and_config 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.320+0000 7fed77c72640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.320+0000 7fed77c72640 10 monclient: monmap: 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:44.321541+0000 2024-09-10T16:20:44.322 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:44.321541+0000 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:44.323 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.324 INFO:tasks.rados.rados.0.smithi165.stdout:2177: writing smithi16534841-30 from 558197 to 1126829 tid 1 2024-09-10T16:20:44.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.321+0000 7fed77c72640 1 Processor -- start 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.321+0000 7fed77c72640 1 -- start start 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.321+0000 7fed77c72640 10 monclient: init 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fed700a0410 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fed700a2550 0x7fed7013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-c con 0x7fed700a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed700a29e0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed700a0210 con 0x7fed700a0410 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed7009d930 con 0x7fed700a29e0 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fed7009d3f0 con 0x7fed700a2550 2024-09-10T16:20:44.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed77c72640 10 monclient(hunting): authenticate will time out at 1622.809158s 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed75a0e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fed700a0410 0x7fed700a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44252/0 (socket says 172.21.15.164:44252) 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed75a0e640 1 -- 172.21.15.164:0/2956952791 learned_addr learned my addr 172.21.15.164:0/2956952791 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 1 --2- 172.21.15.164:0/2956952791 >> v2:172.21.15.165:3300/0 conn(0x7fed700a2550 0x7fed7013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): get_auth_request con 0x7fed700a2550 auth_method 0 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient(hunting): handle_auth_done global_id 154180 payload 274 2024-09-10T16:20:44.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 1 -- 172.21.15.164:0/2956952791 >> v1:172.21.15.165:6789/0 conn(0x7fed700a29e0 legacy=0x7fed7013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 1 -- 172.21.15.164:0/2956952791 >> v1:172.21.15.164:6789/0 conn(0x7fed700a0410 legacy=0x7fed700a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.322+0000 7fed7520d640 1 -- 172.21.15.164:0/2956952791 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed7009e7d0 con 0x7fed700a2550 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed7520d640 1 -- 172.21.15.164:0/2956952791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a2550 msgr2=0x7fed7013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed7520d640 1 --2- 172.21.15.164:0/2956952791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a2550 0x7fed7013b5d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed7009f490 tx=0x7fed60029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c808f900ea06daf7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.330 INFO:tasks.rados.rados.0.smithi165.stdout:2177: writing smithi16534841-30 from 1650558 to 2283485 tid 2 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 1 -- 172.21.15.164:0/2956952791 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60037070 con 0x7fed700a2550 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: dump: 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 1 -- 172.21.15.164:0/2956952791 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed60032440 con 0x7fed700a2550 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed77c72640 5 monclient: authenticate success, global_id 154180 2024-09-10T16:20:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed77c72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed77c72640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 1 -- 172.21.15.164:0/2956952791 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60032810 con 0x7fed700a2550 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed77c72640 4 set_mon_vals no callback set 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.325+0000 7fed74a0c640 10 monclient: dump: 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.339 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 10 monclient: shutdown 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 -- 172.21.15.164:0/2956952791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a2550 msgr2=0x7fed7013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 --2- 172.21.15.164:0/2956952791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a2550 0x7fed7013b5d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fed7009f490 tx=0x7fed60029fb0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 -- 172.21.15.164:0/2956952791 shutdown_connections 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 --2- 172.21.15.164:0/2956952791 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a2550 0x7fed7013b5d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 -- 172.21.15.164:0/2956952791 >> 172.21.15.164:0/2956952791 conn(0x7fed700a3110 msgr2=0x7fed700a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 -- 172.21.15.164:0/2956952791 shutdown_connections 2024-09-10T16:20:44.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.326+0000 7fed77c72640 1 -- 172.21.15.164:0/2956952791 wait complete. 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.327+0000 7fed77c72640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.327+0000 7fed77c72640 10 monclient: monmap: 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.327+0000 7fed77c72640 1 Processor -- start 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.327+0000 7fed77c72640 1 -- start start 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.327+0000 7fed77c72640 10 monclient: init 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a0410 0x7fed701c8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-b con 0x7fed700a0410 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-c con 0x7fed700a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed700a29e0 0x7fed701b0a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): picked mon.noname-a con 0x7fed700a29e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed700a07f0 con 0x7fed700a29e0 2024-09-10T16:20:44.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed7009c910 con 0x7fed700a0410 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed700a0970 con 0x7fed700a2550 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed77c72640 10 monclient(hunting): authenticate will time out at 1622.815372s 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed700a29e0 0x7fed701b0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38296/0 (socket says 172.21.15.164:38296) 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 -- 172.21.15.164:0/3823773390 learned_addr learned my addr 172.21.15.164:0/3823773390 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): get_auth_request con 0x7fed700a2550 auth_method 0 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): get_auth_request con 0x7fed700a29e0 auth_method 0 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed75a0e640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a0410 0x7fed701c8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed75a0e640 10 monclient(hunting): get_auth_request con 0x7fed700a0410 auth_method 0 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed75a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed75a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed75a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient(hunting): handle_auth_done global_id 114548 payload 824 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a0410 msgr2=0x7fed701c8fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed700a0410 0x7fed701c8fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed700a29e0 msgr2=0x7fed701b0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed700a29e0 0x7fed701b0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7520d640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed701b1c40 con 0x7fed700a2550 2024-09-10T16:20:44.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.328+0000 7fed7620f640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed700a29e0 0x7fed701b0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed7520d640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 msgr2=0x7fed701b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed7520d640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed60000c00 tx=0x7fed600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60041070 con 0x7fed700a2550 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: dump: 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.346 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed6003c890 con 0x7fed700a2550 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 5 monclient: authenticate success, global_id 114548 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 10 monclient: _renew_subs 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed701b3cc0 con 0x7fed700a2550 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed60037040 con 0x7fed700a2550 2024-09-10T16:20:44.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed667fc640 10 monclient: dump: 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.349 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed67fff640 4 set_mon_vals no callback set 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 10 monclient: _renew_subs 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.331+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fed701b27a0 con 0x7fed700a2550 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed77c72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed77c72640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 0x7fed701b0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed7009a4e0 con 0x7fed701c9510 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 0x7fed701b0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient: get_auth_request con 0x7fed701c9510 auth_method 0 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed7520d640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed60000c00 tx=0x7fed600046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7fed60073490 osd_map(497..497 src has 184..497) 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fed60044070 con 0x7fed700a2550 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 10 monclient(hunting): handle_auth_done global_id 164106 payload 824 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed75a0e640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed7009d670 con 0x7fed701c9510 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.332+0000 7fed667fc640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed5403e210 0x7fed540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.333+0000 7fed7620f640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed5403e210 0x7fed540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.333+0000 7fed7620f640 10 monclient: get_auth_request con 0x7fed5403e210 auth_method 0 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.334+0000 7fed75a0e640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 msgr2=0x7fed701b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.334+0000 7fed75a0e640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 0x7fed701b0f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00e500 tx=0x7fed6c00e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c004e30 con 0x7fed701c9510 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 10 monclient: dump: 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.352 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed6c018070 con 0x7fed701c9510 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:44.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 msgr2=0x7fed701b0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 0x7fed701b0f40 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00e500 tx=0x7fed6c00e9c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 msgr2=0x7fed701b0f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed667fc640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701c9510 0x7fed701b0f40 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.335+0000 7fed7620f640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed5403e210 0x7fed540406d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fed701b2530 tx=0x7fed68003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2177: writing smithi16534841-30 from 2773245 to 2958197 tid 3 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2178: write (excl) oid 12 current snap is 237 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2177: finishing write tid 1 to smithi16534841-30 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2178: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2178: writing smithi16534841-12 from 575004 to 1218885 tid 1 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2178: writing smithi16534841-12 from 1709040 to 2201978 tid 2 2024-09-10T16:20:44.359 INFO:tasks.rados.rados.0.smithi165.stdout:2178: writing smithi16534841-12 from 2917021 to 2975004 tid 3 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2179: rollback oid 9 current snap is 237 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 231 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2177: finishing write tid 2 to smithi16534841-30 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2177: finishing write tid 3 to smithi16534841-30 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2177: finishing write tid 4 to smithi16534841-30 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2177: finishing write tid 5 to smithi16534841-30 2024-09-10T16:20:44.360 INFO:tasks.rados.rados.0.smithi165.stdout:2177: oid 30 updating version 0 to 1369 2024-09-10T16:20:44.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed77c72640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed77c72640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 0x7fed700707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed77c72640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed7009fbc0 con 0x7fed701b1480 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 0x7fed700707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient: get_auth_request con 0x7fed701b1480 auth_method 0 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.341+0000 7fed75a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.342+0000 7fed75a0e640 10 monclient(hunting): handle_auth_done global_id 164109 payload 824 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.342+0000 7fed75a0e640 1 -- 172.21.15.164:0/3823773390 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed701b1890 con 0x7fed701b1480 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed75a0e640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 msgr2=0x7fed700707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed75a0e640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 0x7fed700707e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed6c017000 tx=0x7fed6c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed6c0066f0 con 0x7fed701b1480 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 10 monclient: dump: 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7fed6c0066f0 con 0x7fed701b1480 2024-09-10T16:20:44.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 msgr2=0x7fed700707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 0x7fed700707e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fed6c017000 tx=0x7fed6c004940 comp rx=0 tx=0).stop 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 msgr2=0x7fed700707e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed667fc640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed701b1480 0x7fed700707e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:44.368 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 15046, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:44.369 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.370 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.371 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:44.372 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.373 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.374 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.375 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.376 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.377 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.378 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.379 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:44.380 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:2177: oid 30 version 1369 is already newer than 1367 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:2177: oid 30 updating version 1369 to 1370 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:2177: oid 30 version 1370 is already newer than 1368 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1370 (ObjNum 771 snap 237 seq_num 771) dirty exists 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:2177: left oid 30 (ObjNum 771 snap 237 seq_num 771) 2024-09-10T16:20:44.381 INFO:tasks.rados.rados.0.smithi165.stdout:2178: finishing write tid 1 to smithi16534841-12 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: finishing write tid 2 to smithi16534841-12 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2177: done (2 left) 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2180: read oid 27 snap -1 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2180: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2181: rollback oid 7 current snap is 237 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 7 to 227 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2182: delete oid 47 current snap is 237 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: finishing write tid 3 to smithi16534841-12 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: finishing write tid 4 to smithi16534841-12 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: finishing write tid 5 to smithi16534841-12 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: oid 12 updating version 0 to 1309 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: oid 12 updating version 1309 to 1310 2024-09-10T16:20:44.382 INFO:tasks.rados.rados.0.smithi165.stdout:2178: oid 12 version 1310 is already newer than 1308 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2178: oid 12 version 1310 is already newer than 1307 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1310 (ObjNum 772 snap 237 seq_num 772) dirty exists 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2178: left oid 12 (ObjNum 772 snap 237 seq_num 772) 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2181: finishing rollback tid 0 to smithi16534841-7 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2181: finishing rollback tid 1 to smithi16534841-7 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2181: finishing rollback tid 2 to smithi16534841-7 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1374 (ObjNum 743 snap 225 seq_num 743) dirty exists 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2179: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2179: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2179: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1181 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:20:44.383 INFO:tasks.rados.rados.0.smithi165.stdout:2180: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.344+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed5403e210 msgr2=0x7fed540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fed5403e210 0x7fed540406d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fed701b2530 tx=0x7fed68003d10 comp rx=0 tx=0).stop 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 10 monclient: shutdown 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 msgr2=0x7fed701b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fed60000c00 tx=0x7fed600046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed7620f640 1 -- 172.21.15.164:0/3823773390 reap_dead start 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed667fc640 1 -- 172.21.15.164:0/3823773390 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(497..497 src has 184..497) ==== 4815+0+0 (unknown 0 0 0) 0x7fed60073490 con 0x7fed700a2550 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 shutdown_connections 2024-09-10T16:20:44.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 --2- 172.21.15.164:0/3823773390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed700a2550 0x7fed701b04c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 >> 172.21.15.164:0/3823773390 conn(0x7fed700a3110 msgr2=0x7fed700a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 shutdown_connections 2024-09-10T16:20:44.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.345+0000 7fed77c72640 1 -- 172.21.15.164:0/3823773390 wait complete. 2024-09-10T16:20:44.386 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:44.386 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:20:44.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:20:44.434 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:20:44.474 INFO:tasks.rados.rados.0.smithi165.stdout:2178: done (4 left) 2024-09-10T16:20:44.474 INFO:tasks.rados.rados.0.smithi165.stdout:2179: done (3 left) 2024-09-10T16:20:44.474 INFO:tasks.rados.rados.0.smithi165.stdout:2180: done (2 left) 2024-09-10T16:20:44.474 INFO:tasks.rados.rados.0.smithi165.stdout:2181: done (1 left) 2024-09-10T16:20:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:2182: done (0 left) 2024-09-10T16:20:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:2183: rollback oid 24 current snap is 237 2024-09-10T16:20:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 234 2024-09-10T16:20:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:2184: copy_from oid 35 from oid 34 current snap is 237 2024-09-10T16:20:44.475 INFO:tasks.rados.rados.0.smithi165.stdout:2185: delete oid 46 current snap is 237 2024-09-10T16:20:44.476 INFO:tasks.rados.rados.0.smithi165.stdout:2185: done (2 left) 2024-09-10T16:20:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:2186: copy_from oid 12 from oid 29 current snap is 237 2024-09-10T16:20:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:2187: delete oid 9 current snap is 237 2024-09-10T16:20:44.481 INFO:tasks.rados.rados.0.smithi165.stdout:2183: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:20:44.481 INFO:tasks.rados.rados.0.smithi165.stdout:2183: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:20:44.482 INFO:tasks.rados.rados.0.smithi165.stdout:2183: finishing rollback tid 2 to smithi16534841-24 2024-09-10T16:20:44.482 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1333 (ObjNum 752 snap 227 seq_num 752) dirty exists 2024-09-10T16:20:44.482 INFO:tasks.rados.rados.0.smithi165.stdout:2183: done (3 left) 2024-09-10T16:20:44.482 INFO:tasks.rados.rados.0.smithi165.stdout:2187: done (2 left) 2024-09-10T16:20:44.482 INFO:tasks.rados.rados.0.smithi165.stdout:2188: snap_remove snap 234 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.527+0000 7fa88bdad640 10 monclient: get_monmap_and_config 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.527+0000 7fa88bdad640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.527+0000 7fa88bdad640 10 monclient: monmap: 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:44.528511+0000 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:44.528511+0000 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:44.530 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 Processor -- start 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 -- start start 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient: init 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8840a0410 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:44.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8840a2550 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa8840a29e0 0x7fa88413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-c con 0x7fa8840a29e0 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8840a0210 con 0x7fa8840a2550 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa88409d930 con 0x7fa8840a0410 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa88409d3f0 con 0x7fa8840a29e0 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.528+0000 7fa88bdad640 10 monclient(hunting): authenticate will time out at 1623.015907s 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa889348640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa8840a2550 0x7fa88413b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44256/0 (socket says 172.21.15.164:44256) 2024-09-10T16:20:44.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa889348640 1 -- 172.21.15.164:0/742231271 learned_addr learned my addr 172.21.15.164:0/742231271 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 1 --2- 172.21.15.164:0/742231271 >> v2:172.21.15.165:3300/0 conn(0x7fa8840a29e0 0x7fa88413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): get_auth_request con 0x7fa8840a29e0 auth_method 0 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient(hunting): handle_auth_done global_id 154186 payload 274 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 1 -- 172.21.15.164:0/742231271 >> v1:172.21.15.165:6789/0 conn(0x7fa8840a0410 legacy=0x7fa8840a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 1 -- 172.21.15.164:0/742231271 >> v1:172.21.15.164:6789/0 conn(0x7fa8840a2550 legacy=0x7fa88413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:20:44.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.529+0000 7fa88a34a640 1 -- 172.21.15.164:0/742231271 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88409e7d0 con 0x7fa8840a29e0 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88a34a640 1 -- 172.21.15.164:0/742231271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a29e0 msgr2=0x7fa88413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88a34a640 1 --2- 172.21.15.164:0/742231271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a29e0 0x7fa88413ed00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa884075190 tx=0x7fa87c02a460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ffc290e27e86748 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 1 -- 172.21.15.164:0/742231271 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c007a30 con 0x7fa8840a29e0 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: dump: 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 1 -- 172.21.15.164:0/742231271 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c007bd0 con 0x7fa8840a29e0 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88bdad640 5 monclient: authenticate success, global_id 154186 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88bdad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:44.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88bdad640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa88bdad640 4 set_mon_vals no callback set 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 1 -- 172.21.15.164:0/742231271 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c031400 con 0x7fa8840a29e0 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.532+0000 7fa888b47640 10 monclient: dump: 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.546 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 10 monclient: shutdown 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 -- 172.21.15.164:0/742231271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a29e0 msgr2=0x7fa88413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 --2- 172.21.15.164:0/742231271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a29e0 0x7fa88413ed00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa884075190 tx=0x7fa87c02a460 comp rx=0 tx=0).stop 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 -- 172.21.15.164:0/742231271 shutdown_connections 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 --2- 172.21.15.164:0/742231271 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a29e0 0x7fa88413ed00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 -- 172.21.15.164:0/742231271 >> 172.21.15.164:0/742231271 conn(0x7fa8840a3110 msgr2=0x7fa8840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 -- 172.21.15.164:0/742231271 shutdown_connections 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.533+0000 7fa88bdad640 1 -- 172.21.15.164:0/742231271 wait complete. 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient: monmap: 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 Processor -- start 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 -- start start 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient: init 2024-09-10T16:20:44.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 0x7fa8841cd100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-c con 0x7fa8840a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a2550 0x7fa8841ba910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-b con 0x7fa8840a2550 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 0x7fa8841c9b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): picked mon.noname-a con 0x7fa8840a29e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840a07f0 con 0x7fa8840a29e0 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa88409c910 con 0x7fa8840a2550 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8840a0970 con 0x7fa8840a0410 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88bdad640 10 monclient(hunting): authenticate will time out at 1623.021876s 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88a34a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 0x7fa8841c9b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa889b49640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 0x7fa8841cd100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa88a34a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 0x7fa8841c9b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60390/0 (socket says 172.21.15.164:60390) 2024-09-10T16:20:44.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 learned_addr learned my addr 172.21.15.164:0/811634506 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.534+0000 7fa889b49640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 0x7fa8841cd100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38306/0 (socket says 172.21.15.164:38306) 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): get_auth_request con 0x7fa8840a29e0 auth_method 0 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): get_auth_request con 0x7fa8840a0410 auth_method 0 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a2550 0x7fa8841ba910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): get_auth_request con 0x7fa8840a2550 auth_method 0 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889348640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient(hunting): handle_auth_done global_id 164112 payload 824 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 msgr2=0x7fa8841cd100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 0x7fa8841cd100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a2550 msgr2=0x7fa8841ba910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa8840a2550 0x7fa8841ba910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 monclient: found mon.noname-a 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8840a57e0 con 0x7fa8840a29e0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.535+0000 7fa889b49640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8840a0410 0x7fa8841cd100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 msgr2=0x7fa8841c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88a34a640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 0x7fa8841c9b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa87c02aa50 tx=0x7fa87c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c040070 con 0x7fa8840a29e0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: dump: 2024-09-10T16:20:44.553 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 5 monclient: authenticate success, global_id 164112 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa87c045660 con 0x7fa8840a29e0 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 10 monclient: _renew_subs 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88409d620 con 0x7fa8840a29e0 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87c0314c0 con 0x7fa8840a29e0 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87a7fc640 10 monclient: dump: 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.555 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 10 monclient: _renew_subs 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa87bfff640 4 set_mon_vals no callback set 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.537+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8841b63b0 con 0x7fa8840a29e0 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa88bdad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa88bdad640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 0x7fa8841ca0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa884074750 con 0x7fa8841bae50 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 0x7fa8841ca0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient: get_auth_request con 0x7fa8841bae50 auth_method 0 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa87c039040 con 0x7fa8840a29e0 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 10 monclient(hunting): handle_auth_done global_id 114554 payload 824 2024-09-10T16:20:44.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.538+0000 7fa889b49640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa88409fbc0 con 0x7fa8841bae50 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.539+0000 7fa87a7fc640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa86803e210 0x7fa8680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.539+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(497..497 src has 184..497) ==== 4815+0+0 (secure 0 0 0) 0x7fa87c074880 con 0x7fa8840a29e0 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.539+0000 7fa889348640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa86803e210 0x7fa8680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.539+0000 7fa889348640 10 monclient: get_auth_request con 0x7fa86803e210 auth_method 0 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa889b49640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 msgr2=0x7fa8841ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa889b49640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 0x7fa8841ca0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa88000fb10 tx=0x7fa8800044c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880018020 con 0x7fa8841bae50 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 10 monclient: dump: 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.558 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa88000dd90 con 0x7fa8841bae50 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 msgr2=0x7fa8841ca0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 0x7fa8841ca0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa88000fb10 tx=0x7fa8800044c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 msgr2=0x7fa8841ca0c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa87a7fc640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841bae50 0x7fa8841ca0c0 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fa88000fb10 tx=0x7fa8800044c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.541+0000 7fa889348640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa86803e210 0x7fa8680406d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa874007c90 tx=0x7fa874007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa88bdad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa88bdad640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 0x7fa8841ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa88bdad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa8841d84b0 con 0x7fa8841b4bb0 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 0x7fa8841ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient: get_auth_request con 0x7fa8841b4bb0 auth_method 0 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 10 monclient(hunting): handle_auth_done global_id 114557 payload 824 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.547+0000 7fa889b49640 1 -- 172.21.15.164:0/811634506 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa8841d8660 con 0x7fa8841b4bb0 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa889b49640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 msgr2=0x7fa8841ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa889b49640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 0x7fa8841ca600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa880019000 tx=0x7fa8800036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880018030 con 0x7fa8841b4bb0 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 10 monclient: dump: 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.563 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fa880014b80 con 0x7fa8841b4bb0 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 msgr2=0x7fa8841ca600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 0x7fa8841ca600 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa880019000 tx=0x7fa8800036f0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 msgr2=0x7fa8841ca600 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa87a7fc640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa8841b4bb0 0x7fa8841ca600 secure :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa880019000 tx=0x7fa8800036f0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:20:44.565 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 265118, 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.566 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:44.567 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.568 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.569 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.570 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.571 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.572 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.573 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.575 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.577 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.578 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:44.579 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa86803e210 msgr2=0x7fa8680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa86803e210 0x7fa8680406d0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa874007c90 tx=0x7fa874007be0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 10 monclient: shutdown 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 msgr2=0x7fa8841c9b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 --2- 172.21.15.164:0/811634506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa8840a29e0 0x7fa8841c9b80 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa87c02aa50 tx=0x7fa87c004a40 comp rx=0 tx=0).stop 2024-09-10T16:20:44.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 reap_dead start 2024-09-10T16:20:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88a34a640 1 -- 172.21.15.164:0/811634506 reap_dead start 2024-09-10T16:20:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 shutdown_connections 2024-09-10T16:20:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.550+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 >> 172.21.15.164:0/811634506 conn(0x7fa8840a3110 msgr2=0x7fa884072350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.551+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 shutdown_connections 2024-09-10T16:20:44.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.551+0000 7fa88bdad640 1 -- 172.21.15.164:0/811634506 wait complete. 2024-09-10T16:20:44.582 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:44.582 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:20:44.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.679+0000 7faf2b4a9640 10 monclient: get_monmap_and_config 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.679+0000 7faf2b4a9640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.679+0000 7faf2b4a9640 10 monclient: monmap: 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:44.680444+0000 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:44.680444+0000 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:44.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:44.682 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:44.682 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 Processor -- start 2024-09-10T16:20:44.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 -- start start 2024-09-10T16:20:44.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient: init 2024-09-10T16:20:44.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faf240a0bc0 0x7faf2409d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-c con 0x7faf240a0bc0 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faf24136260 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faf2413b070 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf240a1740 con 0x7faf2413b070 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf240690a0 con 0x7faf24136260 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faf24069660 con 0x7faf240a0bc0 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.680+0000 7faf2b4a9640 10 monclient(hunting): authenticate will time out at 1623.167911s 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29a46640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7faf2413b070 0x7faf2413d460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44258/0 (socket says 172.21.15.164:44258) 2024-09-10T16:20:44.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29a46640 1 -- 172.21.15.164:0/2085702976 learned_addr learned my addr 172.21.15.164:0/2085702976 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 1 --2- 172.21.15.164:0/2085702976 >> v2:172.21.15.165:3300/0 conn(0x7faf240a0bc0 0x7faf2409d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): get_auth_request con 0x7faf240a0bc0 auth_method 0 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient(hunting): handle_auth_done global_id 154192 payload 274 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 1 -- 172.21.15.164:0/2085702976 >> v1:172.21.15.165:6789/0 conn(0x7faf24136260 legacy=0x7faf2409dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 1 -- 172.21.15.164:0/2085702976 >> v1:172.21.15.164:6789/0 conn(0x7faf2413b070 legacy=0x7faf2413d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:20:44.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.681+0000 7faf29245640 1 -- 172.21.15.164:0/2085702976 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf24069380 con 0x7faf240a0bc0 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.684+0000 7faf29245640 1 -- 172.21.15.164:0/2085702976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf240a0bc0 msgr2=0x7faf2409d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.684+0000 7faf29245640 1 --2- 172.21.15.164:0/2085702976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf240a0bc0 0x7faf2409d960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf200097a0 tx=0x7faf20029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4503055b7e9eac56 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 1 -- 172.21.15.164:0/2085702976 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20037070 con 0x7faf240a0bc0 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: dump: 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.695 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 1 -- 172.21.15.164:0/2085702976 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf20032640 con 0x7faf240a0bc0 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 1 -- 172.21.15.164:0/2085702976 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20032a10 con 0x7faf240a0bc0 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf2b4a9640 5 monclient: authenticate success, global_id 154192 2024-09-10T16:20:44.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf2b4a9640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf2b4a9640 4 set_mon_vals no callback set 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.686+0000 7faf1bfff640 10 monclient: dump: 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 10 monclient: shutdown 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 -- 172.21.15.164:0/2085702976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf240a0bc0 msgr2=0x7faf2409d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/2085702976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf240a0bc0 0x7faf2409d960 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7faf200097a0 tx=0x7faf20029fd0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 -- 172.21.15.164:0/2085702976 shutdown_connections 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/2085702976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf240a0bc0 0x7faf2409d960 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 -- 172.21.15.164:0/2085702976 >> 172.21.15.164:0/2085702976 conn(0x7faf240a3fb0 msgr2=0x7faf240a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 -- 172.21.15.164:0/2085702976 shutdown_connections 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 1 -- 172.21.15.164:0/2085702976 wait complete. 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.687+0000 7faf2b4a9640 10 monclient: monmap: 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.698 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 Processor -- start 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 -- start start 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient: init 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:44.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 0x7faf24138f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-c con 0x7faf240a0bc0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf24136260 0x7faf24139480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-b con 0x7faf24136260 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf2413b070 0x7faf241399c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): picked mon.noname-a con 0x7faf2413b070 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faf2413eb80 con 0x7faf2413b070 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faf240a0fa0 con 0x7faf24136260 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faf2413ed00 con 0x7faf240a0bc0 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf2b4a9640 10 monclient(hunting): authenticate will time out at 1623.175868s 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 0x7faf24138f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 0x7faf24138f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38328/0 (socket says 172.21.15.164:38328) 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 1 -- 172.21.15.164:0/554453456 learned_addr learned my addr 172.21.15.164:0/554453456 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 10 monclient(hunting): get_auth_request con 0x7faf240a0bc0 auth_method 0 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29245640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29a46640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf2413b070 0x7faf241399c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29a46640 10 monclient(hunting): get_auth_request con 0x7faf2413b070 auth_method 0 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29a46640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29a46640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.688+0000 7faf29a46640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29a46640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29a46640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29a46640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf24136260 0x7faf24139480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): get_auth_request con 0x7faf24136260 auth_method 0 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient(hunting): handle_auth_done global_id 114560 payload 824 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf24136260 msgr2=0x7faf24139480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf24136260 0x7faf24139480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf2413b070 msgr2=0x7faf241399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf2413b070 0x7faf241399c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29245640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf24071c60 con 0x7faf240a0bc0 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.689+0000 7faf29a46640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faf2413b070 0x7faf241399c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf29245640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 msgr2=0x7faf24138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf29245640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 0x7faf24138f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf200098c0 tx=0x7faf200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20042020 con 0x7faf240a0bc0 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: dump: 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 5 monclient: authenticate success, global_id 114560 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf20030880 con 0x7faf240a0bc0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 10 monclient: _renew_subs 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf2409d280 con 0x7faf240a0bc0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf20037040 con 0x7faf240a0bc0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf19ffb640 10 monclient: dump: 2024-09-10T16:20:44.705 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf1b7fe640 4 set_mon_vals no callback set 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 10 monclient: _renew_subs 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.691+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7faf24071980 con 0x7faf240a0bc0 2024-09-10T16:20:44.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.692+0000 7faf2b4a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.692+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 0x7faf241cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.692+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.692+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faf2409d960 con 0x7faf241cae50 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.692+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faf20045020 con 0x7faf240a0bc0 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf19ffb640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf0803e210 0x7faf080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 0x7faf241cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(497..497 src has 184..497) ==== 4815+0+0 (secure 0 0 0) 0x7faf20073400 con 0x7faf240a0bc0 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient: get_auth_request con 0x7faf241cae50 auth_method 0 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf29a46640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf0803e210 0x7faf080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf29a46640 10 monclient: get_auth_request con 0x7faf0803e210 auth_method 0 2024-09-10T16:20:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 10 monclient(hunting): handle_auth_done global_id 154198 payload 824 2024-09-10T16:20:44.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.693+0000 7faf28a44640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faf24074b80 con 0x7faf241cae50 2024-09-10T16:20:44.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf28a44640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 msgr2=0x7faf241cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf28a44640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 0x7faf241cb230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f7a0 tx=0x7faf1400fc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf29a46640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf0803e210 0x7faf080406d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005c00 tx=0x7faf1c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf14018030 con 0x7faf241cae50 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.695+0000 7faf19ffb640 10 monclient: dump: 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.711 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faf1400d9b0 con 0x7faf241cae50 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 msgr2=0x7faf241cb230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 0x7faf241cb230 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f7a0 tx=0x7faf1400fc60 comp rx=0 tx=0).stop 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 msgr2=0x7faf241cb230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.696+0000 7faf19ffb640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cae50 0x7faf241cb230 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f7a0 tx=0x7faf1400fc60 comp rx=0 tx=0).stop 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf2b4a9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 0x7faf24133d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf2b4a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faf240714a0 con 0x7faf241cb770 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 0x7faf24133d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient: get_auth_request con 0x7faf241cb770 auth_method 0 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 10 monclient(hunting): handle_auth_done global_id 154201 payload 824 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.702+0000 7faf28a44640 1 -- 172.21.15.164:0/554453456 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faf240772c0 con 0x7faf241cb770 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.704+0000 7faf28a44640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 msgr2=0x7faf24133d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.704+0000 7faf28a44640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 0x7faf24133d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f7a0 tx=0x7faf14003760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf1401dd70 con 0x7faf241cb770 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 10 monclient: dump: 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7faf1401dd70 con 0x7faf241cb770 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 msgr2=0x7faf24133d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 0x7faf24133d90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7faf1400f7a0 tx=0x7faf14003760 comp rx=0 tx=0).stop 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 msgr2=0x7faf24133d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf19ffb640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faf241cb770 0x7faf24133d90 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:44.715 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 63256, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:44.716 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:44.717 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.718 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:44.719 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:44.720 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:44.723 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:44.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.726 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:44.727 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:44.728 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:44.728 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:44.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf0803e210 msgr2=0x7faf080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faf0803e210 0x7faf080406d0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7faf1c005c00 tx=0x7faf1c009040 comp rx=0 tx=0).stop 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 10 monclient: shutdown 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 msgr2=0x7faf24138f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 --2- 172.21.15.164:0/554453456 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faf240a0bc0 0x7faf24138f40 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7faf200098c0 tx=0x7faf200046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf29a46640 1 -- 172.21.15.164:0/554453456 reap_dead start 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf29a46640 1 -- 172.21.15.164:0/554453456 reap_dead start 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 shutdown_connections 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 >> 172.21.15.164:0/554453456 conn(0x7faf240a3fb0 msgr2=0x7faf2413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 shutdown_connections 2024-09-10T16:20:44.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.705+0000 7faf2b4a9640 1 -- 172.21.15.164:0/554453456 wait complete. 2024-09-10T16:20:44.731 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:44.731 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:20:44.731 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.826+0000 7f10159ad640 10 monclient: get_monmap_and_config 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.826+0000 7f10159ad640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.826+0000 7f10159ad640 10 monclient: monmap: 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:44.827663+0000 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:44.827663+0000 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:44.829 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 Processor -- start 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 -- start start 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient: init 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f101006cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 0x7f101013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-b con 0x7f1010134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f101013d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f101009ce50 con 0x7f101006cb70 2024-09-10T16:20:44.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1010135380 con 0x7f101013d560 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f101006db60 con 0x7f1010134f70 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.827+0000 7f10159ad640 10 monclient(hunting): authenticate will time out at 1623.314952s 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 0x7f101013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f100effd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f101006cb70 0x7f10100a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44260/0 (socket says 172.21.15.164:44260) 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f100effd640 1 -- 172.21.15.164:0/3797370750 learned_addr learned my addr 172.21.15.164:0/3797370750 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): get_auth_request con 0x7f1010134f70 auth_method 0 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient(hunting): handle_auth_done global_id 114563 payload 274 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 1 -- 172.21.15.164:0/3797370750 >> v1:172.21.15.165:6789/0 conn(0x7f101013d560 legacy=0x7f101013f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 1 -- 172.21.15.164:0/3797370750 >> v1:172.21.15.164:6789/0 conn(0x7f101006cb70 legacy=0x7f10100a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 1 monclient: found mon.noname-b 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.828+0000 7f10067fc640 1 -- 172.21.15.164:0/3797370750 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101009cb70 con 0x7f1010134f70 2024-09-10T16:20:44.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10067fc640 1 -- 172.21.15.164:0/3797370750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 msgr2=0x7f101013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10067fc640 1 --2- 172.21.15.164:0/3797370750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 0x7f101013d020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80097d0 tx=0x7f0ff8029fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ac073e9b6f3a354 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 1 -- 172.21.15.164:0/3797370750 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff8037070 con 0x7f1010134f70 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: dump: 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 1 -- 172.21.15.164:0/3797370750 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ff8032440 con 0x7f1010134f70 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 5 monclient: authenticate success, global_id 114563 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 1 -- 172.21.15.164:0/3797370750 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ff80327f0 con 0x7f1010134f70 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f100e7fc640 10 monclient: dump: 2024-09-10T16:20:44.839 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 4 set_mon_vals no callback set 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 10 monclient: shutdown 2024-09-10T16:20:44.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 -- 172.21.15.164:0/3797370750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 msgr2=0x7f101013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 --2- 172.21.15.164:0/3797370750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 0x7f101013d020 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f0ff80097d0 tx=0x7f0ff8029fa0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 -- 172.21.15.164:0/3797370750 shutdown_connections 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 --2- 172.21.15.164:0/3797370750 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1010134f70 0x7f101013d020 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 -- 172.21.15.164:0/3797370750 >> 172.21.15.164:0/3797370750 conn(0x7f10100aa900 msgr2=0x7f10100aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 -- 172.21.15.164:0/3797370750 shutdown_connections 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.831+0000 7f10159ad640 1 -- 172.21.15.164:0/3797370750 wait complete. 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.832+0000 7f10159ad640 10 monclient: build_initial_monmap 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.832+0000 7f10159ad640 10 monclient: monmap: 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:44.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.832+0000 7f10159ad640 1 Processor -- start 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.832+0000 7f10159ad640 1 -- start start 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.832+0000 7f10159ad640 10 monclient: init 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-c con 0x7f101006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:44.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-a con 0x7f1010134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101013d560 0x7f1010131ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): picked mon.noname-b con 0x7f101013d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f10100a4db0 con 0x7f1010134f70 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f101006cf50 con 0x7f101013d560 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1010141070 con 0x7f101006cb70 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10159ad640 10 monclient(hunting): authenticate will time out at 1623.320497s 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38336/0 (socket says 172.21.15.164:38336) 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60416/0 (socket says 172.21.15.164:60416) 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 -- 172.21.15.164:0/2201865857 learned_addr learned my addr 172.21.15.164:0/2201865857 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): get_auth_request con 0x7f101006cb70 auth_method 0 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): get_auth_request con 0x7f1010134f70 auth_method 0 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101013d560 0x7f1010131ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): get_auth_request con 0x7f101013d560 auth_method 0 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:44.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient(hunting): handle_auth_done global_id 114566 payload 824 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 -- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101013d560 msgr2=0x7f1010131ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101013d560 0x7f1010131ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 -- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 msgr2=0x7f1010131580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient: _finish_hunting 0 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 monclient: found mon.noname-c 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f100effd640 1 -- 172.21.15.164:0/2201865857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f101012f8b0 con 0x7f101006cb70 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.833+0000 7f10067fc640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100effd640 1 -- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 msgr2=0x7f10101cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100effd640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f100000ba50 tx=0x7f100000bf10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000019070 con 0x7f101006cb70 2024-09-10T16:20:44.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: dump: 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: _finish_auth 0 2024-09-10T16:20:44.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f100000ce10 con 0x7f101006cb70 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 5 monclient: authenticate success, global_id 114566 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 10 monclient: _renew_subs 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f101012fe70 con 0x7f101006cb70 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1000014680 con 0x7f101006cb70 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100cff9640 10 monclient: dump: 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:44.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 10 monclient: _renew_subs 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10101cd690 con 0x7f101006cb70 2024-09-10T16:20:44.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.836+0000 7f100d7fa640 4 set_mon_vals no callback set 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10159ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10159ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10159ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10100a46a0 con 0x7f101006cb70 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f100001c070 con 0x7f101006cb70 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f100cff9640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 0x7f0ff0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(497..497 src has 184..497) ==== 4815+0+0 (secure 0 0 0) 0x7f1000052b00 con 0x7f101006cb70 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10067fc640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 0x7f0ff0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:44.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.837+0000 7f10067fc640 10 monclient: get_auth_request con 0x7f0ff003e290 auth_method 0 2024-09-10T16:20:44.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.840+0000 7f10067fc640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 0x7f0ff0040750 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f101006e110 tx=0x7f0ff802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:44.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.841+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f100001dd70 con 0x7f101006cb70 2024-09-10T16:20:44.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.841+0000 7f100cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:44.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:44.841+0000 7f100cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.080+0000 7f10159ad640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:20:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.080+0000 7f10159ad640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:20:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.080+0000 7f10159ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:45.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.081+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f10100770f0 con 0x7f101006cb70 2024-09-10T16:20:45.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.084+0000 7f100cff9640 1 -- 172.21.15.164:0/2201865857 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f100001bcd0 con 0x7f101006cb70 2024-09-10T16:20:45.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.084+0000 7f100cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:20:45.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.084+0000 7f100cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.086+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 msgr2=0x7f0ff0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.086+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 0x7f0ff0040750 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f101006e110 tx=0x7f0ff802e000 comp rx=0 tx=0).stop 2024-09-10T16:20:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.086+0000 7f10159ad640 10 monclient: shutdown 2024-09-10T16:20:45.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.086+0000 7f10159ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.086+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 msgr2=0x7f10101cd150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f100000ba50 tx=0x7f100000bf10 comp rx=0 tx=0).stop 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 shutdown_connections 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ff003e290 0x7f0ff0040750 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f101013d560 0x7f1010131ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1010134f70 0x7f1010131580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 --2- 172.21.15.164:0/2201865857 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f101006cb70 0x7f10101cd150 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 >> 172.21.15.164:0/2201865857 conn(0x7f10100aa900 msgr2=0x7f1010133650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 shutdown_connections 2024-09-10T16:20:45.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:45.087+0000 7f10159ad640 1 -- 172.21.15.164:0/2201865857 wait complete. 2024-09-10T16:20:45.098 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=10,kv=12,logm=52} crc {auth=2710032825,config=2586702516,health=4141920480,kv=805391194,logm=698395123}) 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=1414617414,config=2586702516,health=343610311,kv=805391194,logm=2891872346}) 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=10,kv=12,logm=52} crc {auth=2710032825,config=2586702516,health=4141920480,kv=805391194,logm=698395123}) 2024-09-10T16:20:45.309 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.307+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=3799294535,config=2586702516,health=3152313405,kv=3293307581,logm=1513249977}) 2024-09-10T16:20:45.312 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.312 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949049526}) 2024-09-10T16:20:45.312 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3620098861}) 2024-09-10T16:20:45.312 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.312 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3949049526}) 2024-09-10T16:20:45.313 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.311+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=325559241}) 2024-09-10T16:20:45.313 INFO:tasks.rados.rados.0.smithi165.stdout:2188: done (2 left) 2024-09-10T16:20:45.313 INFO:tasks.rados.rados.0.smithi165.stdout:2189: snap_remove snap 233 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659561674}) 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1760440086}) 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1659561674}) 2024-09-10T16:20:45.783 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.782+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907996703}) 2024-09-10T16:20:45.784 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.784 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910255728}) 2024-09-10T16:20:45.784 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146143528}) 2024-09-10T16:20:45.784 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.784 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910255728}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4088153940}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953714597}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2928959302}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3953714597}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.783+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=933126604}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962939241}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3298111708}) 2024-09-10T16:20:45.786 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.787 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962939241}) 2024-09-10T16:20:45.787 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.784+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=488656873}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3483668093}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3159661963}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3483668093}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.785+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2321514406}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480944709}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=964340855}) 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.788 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480944709}) 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227980169}) 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503409751}) 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176745504}) 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503409751}) 2024-09-10T16:20:45.789 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.786+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4180247342}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936777914}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409555369}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=936777914}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.787+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396625126}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980789408}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132060355}) 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.791 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=980789408}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=445080042}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132511795}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160516719}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=132511795}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.788+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=188979961}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3089872276}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15140004}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3089872276}) 2024-09-10T16:20:45.792 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.789+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=29377274}) 2024-09-10T16:20:45.794 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.794 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425139617}) 2024-09-10T16:20:45.794 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1315567636}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=425139617}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.790+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237158903}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742247221}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1561247754}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=742247221}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4220524113}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492538226}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2938437258}) 2024-09-10T16:20:45.795 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492538226}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.791+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4230597301}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567289598}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779366572}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3567289598}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.792+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3152368580}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051422248}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1054581801}) 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.796 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051422248}) 2024-09-10T16:20:45.797 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3666542680}) 2024-09-10T16:20:45.800 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.800 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094192905}) 2024-09-10T16:20:45.800 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2525666212}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4094192905}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.793+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3628850019}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558633005}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1404644338}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558633005}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.794+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3812945759}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530590210}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089106595}) 2024-09-10T16:20:45.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530590210}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3245270187}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136055059}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1372705389}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.795+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3136055059}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1218425849}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157967490}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=96858448}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3157967490}) 2024-09-10T16:20:45.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.796+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3427603060}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116582061}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094276743}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2116582061}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1500294507}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906462983}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.797+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136599181}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1906462983}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1210808315}) 2024-09-10T16:20:45.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685671724}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=371268745}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2685671724}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.798+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3544062819}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393978594}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4084507803}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393978594}) 2024-09-10T16:20:45.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.799+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4027136555}) 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431195922}) 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3098733233}) 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431195922}) 2024-09-10T16:20:45.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:45.800+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4253163954}) 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180802881}) 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2065438654}) 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=180802881}) 2024-09-10T16:20:46.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.083+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2793910407}) 2024-09-10T16:20:46.090 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920421516}) 2024-09-10T16:20:46.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112304513}) 2024-09-10T16:20:46.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1920421516}) 2024-09-10T16:20:46.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.089+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=999575704}) 2024-09-10T16:20:46.099 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:20:46.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:46.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879382691}) 2024-09-10T16:20:46.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2323356276}) 2024-09-10T16:20:46.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=879382691}) 2024-09-10T16:20:46.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2877103251}) 2024-09-10T16:20:46.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1715524561}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=76515014}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1715524561}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.140+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1698638018}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974057105}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374877560}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1974057105}) 2024-09-10T16:20:46.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.141+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3196541552}) 2024-09-10T16:20:46.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707456470}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2387386404}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707456470}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.142+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3414381511}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760122732}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1002716520}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760122732}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3188317958}) 2024-09-10T16:20:46.145 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070567786}) 2024-09-10T16:20:46.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009179434}) 2024-09-10T16:20:46.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2070567786}) 2024-09-10T16:20:46.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.143+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012657545}) 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667126618}) 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=230026367}) 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=667126618}) 2024-09-10T16:20:46.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.144+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=24980742}) 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.199+0000 7ff4c19a0640 10 monclient: get_monmap_and_config 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.199+0000 7ff4c19a0640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.199+0000 7ff4c19a0640 10 monclient: monmap: 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:46.200677+0000 2024-09-10T16:20:46.201 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:46.200677+0000 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:46.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 1 Processor -- start 2024-09-10T16:20:46.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 1 -- start start 2024-09-10T16:20:46.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 10 monclient: init 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff4bc1329f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-a con 0x7ff4bc0a2f10 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.200+0000 7ff4c19a0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff4bc13b2b0 0x7ff4bc13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-c con 0x7ff4bc13b2b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4bc1334e0 con 0x7ff4bc1329f0 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc0a0730 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff4bc0a21a0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4c19a0640 10 monclient(hunting): authenticate will time out at 1624.688055s 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60424/0 (socket says 172.21.15.164:60424) 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 -- 172.21.15.164:0/3765180600 learned_addr learned my addr 172.21.15.164:0/3765180600 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): get_auth_request con 0x7ff4bc0a2f10 auth_method 0 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4bb7fe640 1 --2- 172.21.15.164:0/3765180600 >> v2:172.21.15.165:3300/0 conn(0x7ff4bc13b2b0 0x7ff4bc13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff4bc13b2b0 auth_method 0 2024-09-10T16:20:46.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient(hunting): handle_auth_done global_id 164121 payload 274 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 -- 172.21.15.164:0/3765180600 >> v2:172.21.15.165:3300/0 conn(0x7ff4bc13b2b0 msgr2=0x7ff4bc13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 --2- 172.21.15.164:0/3765180600 >> v2:172.21.15.165:3300/0 conn(0x7ff4bc13b2b0 0x7ff4bc13d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 -- 172.21.15.164:0/3765180600 >> v1:172.21.15.165:6789/0 conn(0x7ff4bc1329f0 legacy=0x7ff4bc132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 monclient: found mon.noname-a 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.201+0000 7ff4ba7fc640 1 -- 172.21.15.164:0/3765180600 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc0a2480 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4ba7fc640 1 -- 172.21.15.164:0/3765180600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc0a2f10 msgr2=0x7ff4bc13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4ba7fc640 1 --2- 172.21.15.164:0/3765180600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0a1470 tx=0x7ff4b002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fda946d2f2362e4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 1 -- 172.21.15.164:0/3765180600 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0037070 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 1 -- 172.21.15.164:0/3765180600 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b0032640 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 5 monclient: authenticate success, global_id 164121 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 4 set_mon_vals no callback set 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 1 -- 172.21.15.164:0/3765180600 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b0032a10 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4b9ffb640 10 monclient: dump: 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 10 monclient: shutdown 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 -- 172.21.15.164:0/3765180600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc0a2f10 msgr2=0x7ff4bc13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/3765180600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0a1470 tx=0x7ff4b002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 -- 172.21.15.164:0/3765180600 shutdown_connections 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/3765180600 >> v2:172.21.15.165:3300/0 conn(0x7ff4bc13b2b0 0x7ff4bc13d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/3765180600 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc0a2f10 0x7ff4bc13ad70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 -- 172.21.15.164:0/3765180600 >> 172.21.15.164:0/3765180600 conn(0x7ff4bc0a7320 msgr2=0x7ff4bc0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 -- 172.21.15.164:0/3765180600 shutdown_connections 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.204+0000 7ff4c19a0640 1 -- 172.21.15.164:0/3765180600 wait complete. 2024-09-10T16:20:46.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.205+0000 7ff4c19a0640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.205+0000 7ff4c19a0640 10 monclient: monmap: 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.205+0000 7ff4c19a0640 1 Processor -- start 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.205+0000 7ff4c19a0640 1 -- start start 2024-09-10T16:20:46.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.205+0000 7ff4c19a0640 10 monclient: init 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 0x7ff4bc131b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-c con 0x7ff4bc0a2f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 0x7ff4bc132090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-b con 0x7ff4bc1329f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): picked mon.noname-a con 0x7ff4bc13b2b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc0a1a80 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff4bc0a1900 con 0x7ff4bc1329f0 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff4bc13dbe0 con 0x7ff4bc0a2f10 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4c19a0640 10 monclient(hunting): authenticate will time out at 1624.693381s 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 0x7ff4bc131b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60434/0 (socket says 172.21.15.164:60434) 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 -- 172.21.15.164:0/2782716558 learned_addr learned my addr 172.21.15.164:0/2782716558 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): get_auth_request con 0x7ff4bc13b2b0 auth_method 0 2024-09-10T16:20:46.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): get_auth_request con 0x7ff4bc0a2f10 auth_method 0 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 0x7ff4bc132090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 10 monclient(hunting): get_auth_request con 0x7ff4bc1329f0 auth_method 0 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient(hunting): handle_auth_done global_id 164124 payload 824 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 -- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 msgr2=0x7ff4bc131b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 0x7ff4bc131b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 -- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 msgr2=0x7ff4bc132090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 0x7ff4bc132090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 monclient: found mon.noname-a 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4bb7fe640 1 -- 172.21.15.164:0/2782716558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4bc073230 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4baffd640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 0x7ff4bc131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.206+0000 7ff4ba7fc640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 0x7ff4bc132090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.208+0000 7ff4bb7fe640 1 -- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 msgr2=0x7ff4bc1cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4bb7fe640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007e40 tx=0x7ff4b400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4013790 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: dump: 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.222 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4b4019070 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 5 monclient: authenticate success, global_id 164124 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 10 monclient: _renew_subs 2024-09-10T16:20:46.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4b4013b90 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4bc13afa0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4a3fff640 10 monclient: dump: 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4b97fa640 4 set_mon_vals no callback set 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 10 monclient: _renew_subs 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.209+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4bc070ed0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff4b401e460 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4a3fff640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 0x7ff4a8040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4c19a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4c19a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4c19a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(498..498 src has 184..498) ==== 4783+0+0 (secure 0 0 0) 0x7ff4b4053620 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4bc0745f0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4baffd640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 0x7ff4a8040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.210+0000 7ff4baffd640 10 monclient: get_auth_request con 0x7ff4a803e260 auth_method 0 2024-09-10T16:20:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.213+0000 7ff4baffd640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 0x7ff4a8040720 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc136a70 tx=0x7ff4ac007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.215+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff4bc0745f0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.215+0000 7ff4a3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.215+0000 7ff4a3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293005218}) 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503542512}) 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4293005218}) 2024-09-10T16:20:46.311 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.309+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=15540412}) 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464446210}) 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1395777433}) 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464446210}) 2024-09-10T16:20:46.314 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.313+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060199771}) 2024-09-10T16:20:46.320 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.320 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841150668}) 2024-09-10T16:20:46.320 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1292300676}) 2024-09-10T16:20:46.320 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.321 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3841150668}) 2024-09-10T16:20:46.321 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.317+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3782219088}) 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:2189: done (2 left) 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:2190: snap_create 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:2184: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:2186: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:2186: finishing copy_from to smithi16534841-12 2024-09-10T16:20:46.321 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1323 (ObjNum 616 snap 190 seq_num 616) dirty exists 2024-09-10T16:20:46.322 INFO:tasks.rados.rados.0.smithi165.stdout:2184: finishing copy_from to smithi16534841-35 2024-09-10T16:20:46.322 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1139 (ObjNum 714 snap 220 seq_num 714) dirty exists 2024-09-10T16:20:46.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582973230}) 2024-09-10T16:20:46.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3432142560}) 2024-09-10T16:20:46.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582973230}) 2024-09-10T16:20:46.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.408+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1086686489}) 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253662179}) 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1304229447}) 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253662179}) 2024-09-10T16:20:46.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.409+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1723021925}) 2024-09-10T16:20:46.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674949044}) 2024-09-10T16:20:46.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054666188}) 2024-09-10T16:20:46.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2674949044}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.410+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4236991734}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81825571}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4207226443}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=81825571}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2396185593}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=1337202857,mdsmap=204415920}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=277450254,mdsmap=931935893}) 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=1337202857,mdsmap=204415920}) 2024-09-10T16:20:46.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.411+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=2068485171,mdsmap=931935893,mgr=2679724451}) 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.412+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.412+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=49} crc {mdsmap=1228700967,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2942752402,mon_config_key=801590129,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1804203552,osd_snap=2963306210}) 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.412+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=52} crc {mdsmap=1228700967,mgr=2390680851,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3012012654,mon_config_key=2394523661,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1804203552,osd_snap=230881409}) 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mdsmap=1,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=49} crc {mdsmap=1228700967,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2942752402,mon_config_key=801590129,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1804203552,osd_snap=2963306210}) 2024-09-10T16:20:46.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mdsmap=1,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=51} crc {mdsmap=1228700967,mgr=3878148497,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=240366202,mon_config_key=1923930453,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1804203552,osd_snap=1632440777}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=46,osdmap=54} crc {osd_snap=2179575387,osdmap=1702948035}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=46,osdmap=54} crc {osd_snap=3933713617,osdmap=2360890776}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=46,osdmap=54} crc {osd_snap=2179575387,osdmap=1702948035}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.413+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=47,osdmap=53} crc {osd_snap=1456352327,osdmap=2478878396}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1163557694}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2301238590}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1163557694}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.414+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=926690006}) 2024-09-10T16:20:46.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2123864729}) 2024-09-10T16:20:46.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=483108506}) 2024-09-10T16:20:46.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2123864729}) 2024-09-10T16:20:46.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.415+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3950713146}) 2024-09-10T16:20:46.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4052558694}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1329902268}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4052558694}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.417+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3361593904}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1394186958}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1503228834}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1394186958}) 2024-09-10T16:20:46.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.419+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3901573455}) 2024-09-10T16:20:46.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=141498335}) 2024-09-10T16:20:46.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=15} crc {osdmap=597101139}) 2024-09-10T16:20:46.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:20:46.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=141498335}) 2024-09-10T16:20:46.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:20:46.424+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=1358328719}) 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4c19a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4c19a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4c19a0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff4bc132dd0 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4a3fff640 1 -- 172.21.15.164:0/2782716558 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7ff4b401cc60 con 0x7ff4bc13b2b0 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4a3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:46.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.475+0000 7ff4a3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:46.478 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:46.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 msgr2=0x7ff4a8040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 0x7ff4a8040720 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc136a70 tx=0x7ff4ac007e30 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 10 monclient: shutdown 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 msgr2=0x7ff4bc1cad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.477+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff4b4007e40 tx=0x7ff4b400e4b0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 shutdown_connections 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4a803e260 0x7ff4a8040720 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff4bc13b2b0 0x7ff4bc1cad70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff4bc1329f0 0x7ff4bc132090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 --2- 172.21.15.164:0/2782716558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff4bc0a2f10 0x7ff4bc131b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 >> 172.21.15.164:0/2782716558 conn(0x7ff4bc0a7320 msgr2=0x7ff4bc0a5600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 shutdown_connections 2024-09-10T16:20:46.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.478+0000 7ff4c19a0640 1 -- 172.21.15.164:0/2782716558 wait complete. 2024-09-10T16:20:46.489 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:20:46.490 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:20:46.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:20:46.490 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:20:46.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 10 monclient: get_monmap_and_config 2024-09-10T16:20:46.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 10 monclient: monmap: 2024-09-10T16:20:46.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:46.629152+0000 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:46.629152+0000 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:46.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 1 Processor -- start 2024-09-10T16:20:46.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 1 -- start start 2024-09-10T16:20:46.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.628+0000 7f96f319f640 10 monclient: init 2024-09-10T16:20:46.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec09d680 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f96ec06d1f0 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f96ec06d5d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ec1389a0 con 0x7f96ec06d1f0 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96ec06e110 con 0x7f96ec06d5d0 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96ec0ab520 con 0x7f96ec09d680 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f319f640 10 monclient(hunting): authenticate will time out at 1625.116532s 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38352/0 (socket says 172.21.15.164:38352) 2024-09-10T16:20:46.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 -- 172.21.15.164:0/313312125 learned_addr learned my addr 172.21.15.164:0/313312125 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): get_auth_request con 0x7f96ec09d680 auth_method 0 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient(hunting): handle_auth_done global_id 114572 payload 274 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 -- 172.21.15.164:0/313312125 >> v1:172.21.15.165:6789/0 conn(0x7f96ec06d5d0 legacy=0x7f96ec138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 -- 172.21.15.164:0/313312125 >> v1:172.21.15.164:6789/0 conn(0x7f96ec06d1f0 legacy=0x7f96ec137b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 monclient: found mon.noname-b 2024-09-10T16:20:46.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.629+0000 7f96f219d640 1 -- 172.21.15.164:0/313312125 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec06e3f0 con 0x7f96ec09d680 2024-09-10T16:20:46.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f219d640 1 -- 172.21.15.164:0/313312125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 msgr2=0x7f96ec09da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f219d640 1 --2- 172.21.15.164:0/313312125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e802a070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=265077bac087b602 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 1 -- 172.21.15.164:0/313312125 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8037070 con 0x7f96ec09d680 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: dump: 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.642 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 1 -- 172.21.15.164:0/313312125 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e8032640 con 0x7f96ec09d680 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f319f640 5 monclient: authenticate success, global_id 114572 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f319f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f319f640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 1 -- 172.21.15.164:0/313312125 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8032a10 con 0x7f96ec09d680 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f319f640 4 set_mon_vals no callback set 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.632+0000 7f96f119b640 10 monclient: dump: 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 10 monclient: shutdown 2024-09-10T16:20:46.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 -- 172.21.15.164:0/313312125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 msgr2=0x7f96ec09da60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 --2- 172.21.15.164:0/313312125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e802a070 comp rx=0 tx=0).stop 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 -- 172.21.15.164:0/313312125 shutdown_connections 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 --2- 172.21.15.164:0/313312125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec09d680 0x7f96ec09da60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 -- 172.21.15.164:0/313312125 >> 172.21.15.164:0/313312125 conn(0x7f96ec0a70e0 msgr2=0x7f96ec0a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 -- 172.21.15.164:0/313312125 shutdown_connections 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.633+0000 7f96f319f640 1 -- 172.21.15.164:0/313312125 wait complete. 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient: monmap: 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:46.645 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 Processor -- start 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 -- start start 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient: init 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-c con 0x7f96ec06d1f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ec06d5d0 0x7f96ec09a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-b con 0x7f96ec06d5d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec09d680 0x7f96ec12f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): picked mon.noname-a con 0x7f96ec09d680 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec0a63e0 con 0x7f96ec09d680 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec0a6260 con 0x7f96ec06d5d0 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f96ec140ed0 con 0x7f96ec06d1f0 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f319f640 10 monclient(hunting): authenticate will time out at 1625.121893s 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f219d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f299e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec09d680 0x7f96ec12f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.634+0000 7f96f219d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38368/0 (socket says 172.21.15.164:38368) 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 -- 172.21.15.164:0/720193729 learned_addr learned my addr 172.21.15.164:0/720193729 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): get_auth_request con 0x7f96ec06d1f0 auth_method 0 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): get_auth_request con 0x7f96ec09d680 auth_method 0 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f299e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f199c640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ec06d5d0 0x7f96ec09a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f199c640 10 monclient(hunting): get_auth_request con 0x7f96ec06d5d0 auth_method 0 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f199c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f199c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient(hunting): handle_auth_done global_id 114575 payload 824 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ec06d5d0 msgr2=0x7f96ec09a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f96ec06d5d0 0x7f96ec09a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec09d680 msgr2=0x7f96ec12f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec09d680 0x7f96ec12f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 monclient: found mon.noname-c 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.635+0000 7f96f219d640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96ec133280 con 0x7f96ec06d1f0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f219d640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 msgr2=0x7f96ec09a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f219d640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8041070 con 0x7f96ec06d1f0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: dump: 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.650 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e803c880 con 0x7f96ec06d1f0 2024-09-10T16:20:46.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 5 monclient: authenticate success, global_id 114575 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 10 monclient: _renew_subs 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96ec0a86c0 con 0x7f96ec06d1f0 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e8037040 con 0x7f96ec06d1f0 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96e2ffd640 10 monclient: dump: 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.652 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f099a640 4 set_mon_vals no callback set 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 10 monclient: _renew_subs 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.637+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96ec135220 con 0x7f96ec06d1f0 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f319f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f319f640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 0x7f96ec12fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec09da60 con 0x7f96ec1329e0 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 0x7f96ec12fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient: get_auth_request con 0x7f96ec1329e0 auth_method 0 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f96e8044070 con 0x7f96ec06d1f0 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 10 monclient(hunting): handle_auth_done global_id 164130 payload 824 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.638+0000 7f96f199c640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f96ec0008d0 con 0x7f96ec1329e0 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.639+0000 7f96e2ffd640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96d003e240 0x7f96d0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.639+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7f96e8073560 con 0x7f96ec06d1f0 2024-09-10T16:20:46.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.639+0000 7f96f299e640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96d003e240 0x7f96d0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.639+0000 7f96f299e640 10 monclient: get_auth_request con 0x7f96d003e240 auth_method 0 2024-09-10T16:20:46.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96f199c640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 msgr2=0x7f96ec12fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96f199c640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 0x7f96ec12fbe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0a15a0 tx=0x7f96dc00e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc007450 con 0x7f96ec1329e0 2024-09-10T16:20:46.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 10 monclient: dump: 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f96dc007450 con 0x7f96ec1329e0 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 msgr2=0x7f96ec12fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 0x7f96ec12fbe0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0a15a0 tx=0x7f96dc00e960 comp rx=0 tx=0).stop 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 msgr2=0x7f96ec12fbe0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96e2ffd640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec1329e0 0x7f96ec12fbe0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0a15a0 tx=0x7f96dc00e960 comp rx=0 tx=0).stop 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.641+0000 7f96f299e640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96d003e240 0x7f96d0040700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f96ec09b120 tx=0x7f96e4003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f319f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:46.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f319f640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 0x7f96ec130530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f319f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f96ec130a70 con 0x7f96ec130120 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 0x7f96ec130530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient: get_auth_request con 0x7f96ec130120 auth_method 0 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 10 monclient(hunting): handle_auth_done global_id 164133 payload 824 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.647+0000 7f96f199c640 1 -- 172.21.15.164:0/720193729 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96ec09ad30 con 0x7f96ec130120 2024-09-10T16:20:46.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f199c640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 msgr2=0x7f96ec130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f199c640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 0x7f96ec130530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009010 tx=0x7f96dc003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96dc01eb70 con 0x7f96ec130120 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 10 monclient: dump: 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.662 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7f96dc01eb70 con 0x7f96ec130120 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 msgr2=0x7f96ec130530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 0x7f96ec130530 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f96dc009010 tx=0x7f96dc003760 comp rx=0 tx=0).stop 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 msgr2=0x7f96ec130530 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96e2ffd640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f96ec130120 0x7f96ec130530 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:46.664 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 17352, 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:46.665 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.666 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.667 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:46.668 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.669 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.670 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.671 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.672 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.673 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.674 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.675 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.676 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:46.677 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96d003e240 msgr2=0x7f96d0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f96d003e240 0x7f96d0040700 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f96ec09b120 tx=0x7f96e4003d10 comp rx=0 tx=0).stop 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 10 monclient: shutdown 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 msgr2=0x7f96ec09a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f96e80098a0 tx=0x7f96e80046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f299e640 1 -- 172.21.15.164:0/720193729 reap_dead start 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 shutdown_connections 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 --2- 172.21.15.164:0/720193729 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f96ec06d1f0 0x7f96ec09a2b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 >> 172.21.15.164:0/720193729 conn(0x7f96ec0a70e0 msgr2=0x7f96ec0a92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 shutdown_connections 2024-09-10T16:20:46.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.650+0000 7f96f319f640 1 -- 172.21.15.164:0/720193729 wait complete. 2024-09-10T16:20:46.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:46.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:20:46.680 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.776+0000 7fd698a9b640 10 monclient: get_monmap_and_config 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.776+0000 7fd698a9b640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.776+0000 7fd698a9b640 10 monclient: monmap: 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:46.777459+0000 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:46.777459+0000 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:46.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 Processor -- start 2024-09-10T16:20:46.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 -- start start 2024-09-10T16:20:46.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient: init 2024-09-10T16:20:46.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd69409f930 0x7fd6940a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-a con 0x7fd69409f930 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd6940a2380 0x7fd6940a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-c con 0x7fd6940a2380 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd69413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd69409fd10 con 0x7fd69413a5c0 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd69409cd20 con 0x7fd69409f930 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd69409a0d0 con 0x7fd6940a2380 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd698a9b640 10 monclient(hunting): authenticate will time out at 1625.264797s 2024-09-10T16:20:46.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd69409f930 0x7fd6940a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd69409f930 0x7fd6940a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60466/0 (socket says 172.21.15.164:60466) 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 1 -- 172.21.15.164:0/3141005828 learned_addr learned my addr 172.21.15.164:0/3141005828 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 10 monclient(hunting): get_auth_request con 0x7fd69409f930 auth_method 0 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.777+0000 7fd69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd68bfff640 1 --2- 172.21.15.164:0/3141005828 >> v2:172.21.15.165:3300/0 conn(0x7fd6940a2380 0x7fd6940a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd68bfff640 10 monclient(hunting): get_auth_request con 0x7fd6940a2380 auth_method 0 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd68bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient(hunting): handle_auth_done global_id 164136 payload 274 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 1 -- 172.21.15.164:0/3141005828 >> v2:172.21.15.165:3300/0 conn(0x7fd6940a2380 msgr2=0x7fd6940a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 1 --2- 172.21.15.164:0/3141005828 >> v2:172.21.15.165:3300/0 conn(0x7fd6940a2380 0x7fd6940a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 1 -- 172.21.15.164:0/3141005828 >> v1:172.21.15.165:6789/0 conn(0x7fd69413a5c0 legacy=0x7fd69413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 1 monclient: found mon.noname-a 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.778+0000 7fd69259c640 1 -- 172.21.15.164:0/3141005828 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69409e840 con 0x7fd69409f930 2024-09-10T16:20:46.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.780+0000 7fd69259c640 1 -- 172.21.15.164:0/3141005828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69409f930 msgr2=0x7fd6940a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.780+0000 7fd69259c640 1 --2- 172.21.15.164:0/3141005828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69409f930 0x7fd6940a1e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6940a2cf0 tx=0x7fd68402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=335d5d7c1fd431d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 1 -- 172.21.15.164:0/3141005828 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd684037070 con 0x7fd69409f930 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: dump: 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.792 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 5 monclient: authenticate success, global_id 164136 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 1 -- 172.21.15.164:0/3141005828 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd684032640 con 0x7fd69409f930 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 4 set_mon_vals no callback set 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 1 -- 172.21.15.164:0/3141005828 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd684032a10 con 0x7fd69409f930 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd691d9b640 10 monclient: dump: 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 10 monclient: shutdown 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 -- 172.21.15.164:0/3141005828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69409f930 msgr2=0x7fd6940a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 --2- 172.21.15.164:0/3141005828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69409f930 0x7fd6940a1e40 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd6940a2cf0 tx=0x7fd68402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 -- 172.21.15.164:0/3141005828 shutdown_connections 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 --2- 172.21.15.164:0/3141005828 >> v2:172.21.15.165:3300/0 conn(0x7fd6940a2380 0x7fd6940a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 --2- 172.21.15.164:0/3141005828 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69409f930 0x7fd6940a1e40 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 -- 172.21.15.164:0/3141005828 >> 172.21.15.164:0/3141005828 conn(0x7fd6940a3110 msgr2=0x7fd6940a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 -- 172.21.15.164:0/3141005828 shutdown_connections 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.781+0000 7fd698a9b640 1 -- 172.21.15.164:0/3141005828 wait complete. 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.782+0000 7fd698a9b640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.782+0000 7fd698a9b640 10 monclient: monmap: 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:46.795 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.782+0000 7fd698a9b640 1 Processor -- start 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.782+0000 7fd698a9b640 1 -- start start 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.782+0000 7fd698a9b640 10 monclient: init 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd69409f930 0x7fd6941c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-b con 0x7fd69409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:46.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-c con 0x7fd6940a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 0x7fd6941cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): picked mon.noname-a con 0x7fd69413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd69409d930 con 0x7fd69413a5c0 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd69409ebe0 con 0x7fd69409f930 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd69409c8b0 con 0x7fd6940a2380 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd698a9b640 10 monclient(hunting): authenticate will time out at 1625.270302s 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 0x7fd6941cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38384/0 (socket says 172.21.15.164:38384) 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 0x7fd6941cace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60474/0 (socket says 172.21.15.164:60474) 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 -- 172.21.15.164:0/2991830978 learned_addr learned my addr 172.21.15.164:0/2991830978 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): get_auth_request con 0x7fd6940a2380 auth_method 0 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): get_auth_request con 0x7fd69413a5c0 auth_method 0 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd69409f930 0x7fd6941c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): get_auth_request con 0x7fd69409f930 auth_method 0 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient(hunting): handle_auth_done global_id 114578 payload 824 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd69409f930 msgr2=0x7fd6941c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd69409f930 0x7fd6941c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 msgr2=0x7fd6941cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 0x7fd6941cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 monclient: found mon.noname-c 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd68bfff640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd69412edf0 con 0x7fd6940a2380 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.783+0000 7fd692d9d640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd69413a5c0 0x7fd6941cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.785+0000 7fd68bfff640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 msgr2=0x7fd694135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.785+0000 7fd68bfff640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d800 tx=0x7fd67c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00b8d0 con 0x7fd6940a2380 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: dump: 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 5 monclient: authenticate success, global_id 114578 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd67c019070 con 0x7fd6940a2380 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd67c00bc90 con 0x7fd6940a2380 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 monclient: _renew_subs 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd69412f0d0 con 0x7fd6940a2380 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd69159a640 4 set_mon_vals no callback set 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd68affd640 10 monclient: dump: 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.802 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 monclient: _renew_subs 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd69412d3f0 con 0x7fd6940a2380 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 0x7fd69402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.786+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd6940a0760 con 0x7fd69412ce00 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 0x7fd69402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient: get_auth_request con 0x7fd69412ce00 auth_method 0 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd67c01e460 con 0x7fd6940a2380 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 10 monclient(hunting): handle_auth_done global_id 114581 payload 824 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd69259c640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd6940717d0 con 0x7fd69412ce00 2024-09-10T16:20:46.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd68affd640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd67403e210 0x7fd6740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7fd67c053220 con 0x7fd6940a2380 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd692d9d640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd67403e210 0x7fd6740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.787+0000 7fd692d9d640 10 monclient: get_auth_request con 0x7fd67403e210 auth_method 0 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd69259c640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 msgr2=0x7fd69402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd69259c640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 0x7fd69402d440 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd684034000 tx=0x7fd6840046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd684037040 con 0x7fd69412ce00 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 10 monclient: dump: 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.805 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd68403c850 con 0x7fd69412ce00 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 msgr2=0x7fd69402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 0x7fd69402d440 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd684034000 tx=0x7fd6840046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 msgr2=0x7fd69402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.789+0000 7fd68affd640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd69412ce00 0x7fd69402d440 secure :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fd684034000 tx=0x7fd6840046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.790+0000 7fd692d9d640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd67403e210 0x7fd6740406d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00f1d0 tx=0x7fd68c003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.795+0000 7fd698a9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:46.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.795+0000 7fd698a9b640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 0x7fd69402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd698a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd694077260 con 0x7fd694136150 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 0x7fd69402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient: get_auth_request con 0x7fd694136150 auth_method 0 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 10 monclient(hunting): handle_auth_done global_id 114584 payload 824 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.796+0000 7fd69259c640 1 -- 172.21.15.164:0/2991830978 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd69412f460 con 0x7fd694136150 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd69259c640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 msgr2=0x7fd69402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd69259c640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 0x7fd69402d980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6940a08e0 tx=0x7fd684008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd684037030 con 0x7fd694136150 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 10 monclient: dump: 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.810 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7fd684003c30 con 0x7fd694136150 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 msgr2=0x7fd69402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 0x7fd69402d980 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6940a08e0 tx=0x7fd684008030 comp rx=0 tx=0).stop 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 msgr2=0x7fd69402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.798+0000 7fd68affd640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd694136150 0x7fd69402d980 secure :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fd6940a08e0 tx=0x7fd684008030 comp rx=0 tx=0).stop 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:46.811 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 267366, 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.812 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.813 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.814 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:46.815 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.816 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.817 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:46.818 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.819 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.820 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.821 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:46.822 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.823 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:20:46.824 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.826 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.826 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:46.826 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:46.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd67403e210 msgr2=0x7fd6740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd67403e210 0x7fd6740406d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fd68c00f1d0 tx=0x7fd68c003d10 comp rx=0 tx=0).stop 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 10 monclient: shutdown 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 msgr2=0x7fd694135c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd692d9d640 1 -- 172.21.15.164:0/2991830978 reap_dead start 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fd67c00d800 tx=0x7fd67c00dcc0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 shutdown_connections 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 --2- 172.21.15.164:0/2991830978 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd6940a2380 0x7fd694135c10 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 >> 172.21.15.164:0/2991830978 conn(0x7fd6940a3110 msgr2=0x7fd694138b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 shutdown_connections 2024-09-10T16:20:46.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.799+0000 7fd698a9b640 1 -- 172.21.15.164:0/2991830978 wait complete. 2024-09-10T16:20:46.829 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:46.829 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:20:46.829 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.926+0000 7f96923a1640 10 monclient: get_monmap_and_config 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.926+0000 7f96923a1640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.926+0000 7f96923a1640 10 monclient: monmap: 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:46.927783+0000 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:46.927783+0000 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:46.929 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.927+0000 7f96923a1640 1 Processor -- start 2024-09-10T16:20:46.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.927+0000 7f96923a1640 1 -- start start 2024-09-10T16:20:46.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.927+0000 7f96923a1640 10 monclient: init 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f968c0a3320 0x7f968c1346d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-a con 0x7f968c0a3320 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 0x7f968c135010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-b con 0x7f968c134c10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 0x7f968c131560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-c con 0x7f968c131150 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f968c135550 con 0x7f968c0a3320 2024-09-10T16:20:46.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f968c0a3700 con 0x7f968c131150 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f968c09fb60 con 0x7f968c134c10 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f96923a1640 10 monclient(hunting): authenticate will time out at 1625.415357s 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f968c0a3320 0x7f968c1346d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f968c0a3320 0x7f968c1346d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60490/0 (socket says 172.21.15.164:60490) 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 -- 172.21.15.164:0/2524167004 learned_addr learned my addr 172.21.15.164:0/2524167004 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): get_auth_request con 0x7f968c0a3320 auth_method 0 2024-09-10T16:20:46.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 0x7f968c135010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): get_auth_request con 0x7f968c134c10 auth_method 0 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient(hunting): handle_auth_done global_id 164142 payload 274 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9691ba0640 1 --2- 172.21.15.164:0/2524167004 >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 0x7f968c131560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 -- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 msgr2=0x7f968c135010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 0x7f968c135010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 -- 172.21.15.164:0/2524167004 >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 msgr2=0x7f968c131560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 --2- 172.21.15.164:0/2524167004 >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 0x7f968c131560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 monclient: found mon.noname-a 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f969139f640 1 -- 172.21.15.164:0/2524167004 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c09fe40 con 0x7f968c0a3320 2024-09-10T16:20:46.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9691ba0640 10 monclient: get_auth_request con 0x7f968c131150 auth_method 0 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9690b9e640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 0x7f968c135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.928+0000 7f9691ba0640 1 --2- 172.21.15.164:0/2524167004 >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 0x7f968c131560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f969139f640 1 -- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c0a3320 msgr2=0x7f968c1346d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f969139f640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c0a3320 0x7f968c1346d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f968c0a08b0 tx=0x7f968802a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fc93b9d29ae3ce1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 1 -- 172.21.15.164:0/2524167004 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688037070 con 0x7f968c0a3320 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: dump: 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.941 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 1 -- 172.21.15.164:0/2524167004 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9688032640 con 0x7f968c0a3320 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f96923a1640 5 monclient: authenticate success, global_id 164142 2024-09-10T16:20:46.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f96923a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 1 -- 172.21.15.164:0/2524167004 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688032a10 con 0x7f968c0a3320 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f96923a1640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f96923a1640 4 set_mon_vals no callback set 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.931+0000 7f9683fff640 10 monclient: dump: 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.943 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 10 monclient: shutdown 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 -- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c0a3320 msgr2=0x7f968c1346d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c0a3320 0x7f968c1346d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f968c0a08b0 tx=0x7f968802a100 comp rx=0 tx=0).stop 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 -- 172.21.15.164:0/2524167004 shutdown_connections 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 --2- 172.21.15.164:0/2524167004 >> v2:172.21.15.165:3300/0 conn(0x7f968c131150 0x7f968c131560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c134c10 0x7f968c135010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 --2- 172.21.15.164:0/2524167004 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c0a3320 0x7f968c1346d0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 -- 172.21.15.164:0/2524167004 >> 172.21.15.164:0/2524167004 conn(0x7f968c0a4570 msgr2=0x7f968c0a4980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 -- 172.21.15.164:0/2524167004 shutdown_connections 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 1 -- 172.21.15.164:0/2524167004 wait complete. 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 10 monclient: build_initial_monmap 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.932+0000 7f96923a1640 10 monclient: monmap: 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:46.945 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 Processor -- start 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 -- start start 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient: init 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-c con 0x7f968c0a3320 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c131150 0x7f968c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-a con 0x7f968c131150 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c134c10 0x7f968c06fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): picked mon.noname-b con 0x7f968c134c10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f968c131aa0 con 0x7f968c131150 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f968c09f390 con 0x7f968c134c10 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f968c131c20 con 0x7f968c0a3320 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f96923a1640 10 monclient(hunting): authenticate will time out at 1625.420734s 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:38418/0 (socket says 172.21.15.164:38418) 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c131150 0x7f968c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 1 -- 172.21.15.164:0/4289409390 learned_addr learned my addr 172.21.15.164:0/4289409390 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:46.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): get_auth_request con 0x7f968c0a3320 auth_method 0 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): get_auth_request con 0x7f968c131150 auth_method 0 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f969139f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.933+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 10 monclient(hunting): handle_auth_done global_id 114590 payload 824 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c134c10 msgr2=0x7f968c06fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f9691ba0640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c134c10 0x7f968c06fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c134c10 0x7f968c06fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c131150 msgr2=0x7f968c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c131150 0x7f968c06f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 10 monclient: _finish_hunting 0 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 monclient: found mon.noname-c 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f969139f640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f968c072220 con 0x7f968c0a3320 2024-09-10T16:20:46.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f9691ba0640 10 monclient: get_auth_request con 0x7f968c134c10 auth_method 0 2024-09-10T16:20:46.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f9690b9e640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f968c131150 0x7f968c06f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:20:46.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.934+0000 7f9691ba0640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c134c10 0x7f968c06fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f969139f640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 msgr2=0x7f968c073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f969139f640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9688009990 tx=0x7f96880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688041020 con 0x7f968c0a3320 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: dump: 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.955 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: _finish_auth 0 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f968803d850 con 0x7f968c0a3320 2024-09-10T16:20:46.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 5 monclient: authenticate success, global_id 114590 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9688037040 con 0x7f968c0a3320 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 10 monclient: _renew_subs 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f968c130060 con 0x7f968c0a3320 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96837fe640 4 set_mon_vals no callback set 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f9681ffb640 10 monclient: dump: 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.957 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 10 monclient: _renew_subs 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.936+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f968c071f40 con 0x7f968c0a3320 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f96923a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f96923a1640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 0x7f968c02e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f968c0741b0 con 0x7f968c12e590 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9688044020 con 0x7f968c0a3320 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 0x7f968c02e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient: get_auth_request con 0x7f968c12e590 auth_method 0 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9681ffb640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f967003e260 0x7f9670040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.937+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7f96880731f0 con 0x7f968c0a3320 2024-09-10T16:20:46.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.938+0000 7f9690b9e640 10 monclient(hunting): handle_auth_done global_id 154222 payload 824 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.938+0000 7f9690b9e640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f968c131560 con 0x7f968c12e590 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.938+0000 7f9691ba0640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f967003e260 0x7f9670040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.938+0000 7f9691ba0640 10 monclient: get_auth_request con 0x7f967003e260 auth_method 0 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9690b9e640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 msgr2=0x7f968c02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9690b9e640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 0x7f968c02e5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f967c011d20 tx=0x7f967c00f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c019070 con 0x7f968c12e590 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 10 monclient: dump: 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.960 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9691ba0640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f967003e260 0x7f9670040720 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f968c0697b0 tx=0x7f9684007490 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f967c004240 con 0x7f968c12e590 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 msgr2=0x7f968c02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 0x7f968c02e5c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f967c011d20 tx=0x7f967c00f610 comp rx=0 tx=0).stop 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 msgr2=0x7f968c02e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.940+0000 7f9681ffb640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c12e590 0x7f968c02e5c0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f967c011d20 tx=0x7f967c00f610 comp rx=0 tx=0).stop 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.946+0000 7f96923a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.946+0000 7f96923a1640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 0x7f968c063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.946+0000 7f96923a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.946+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f968c0703e0 con 0x7f968c06ffd0 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 0x7f968c063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient: get_auth_request con 0x7f968c06ffd0 auth_method 0 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:46.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 10 monclient(hunting): handle_auth_done global_id 154225 payload 824 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.947+0000 7f9690b9e640 1 -- 172.21.15.164:0/4289409390 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f968c1bbb00 con 0x7f968c06ffd0 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.949+0000 7f9690b9e640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 msgr2=0x7f968c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.949+0000 7f9690b9e640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 0x7f968c063d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f967c01b000 tx=0x7f967c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f967c019040 con 0x7f968c06ffd0 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 10 monclient: dump: 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:46.963 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f967c01e870 con 0x7f968c06ffd0 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 msgr2=0x7f968c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 0x7f968c063d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f967c01b000 tx=0x7f967c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 msgr2=0x7f968c063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:20:46.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f9681ffb640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f968c06ffd0 0x7f968c063d50 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f967c01b000 tx=0x7f967c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 144, 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 65500, 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:20:46.965 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.966 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.967 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:20:46.968 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:20:46.969 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:20:46.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:20:46.974 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.976 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:20:46.977 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:20:46.978 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:20:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f967003e260 msgr2=0x7f9670040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.950+0000 7f96923a1640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f967003e260 0x7f9670040720 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f968c0697b0 tx=0x7f9684007490 comp rx=0 tx=0).stop 2024-09-10T16:20:46.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 10 monclient: shutdown 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 msgr2=0x7f968c073aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f9688009990 tx=0x7f96880046c0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f9691ba0640 1 -- 172.21.15.164:0/4289409390 reap_dead start 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 shutdown_connections 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 --2- 172.21.15.164:0/4289409390 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f968c0a3320 0x7f968c073aa0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 >> 172.21.15.164:0/4289409390 conn(0x7f968c0a4570 msgr2=0x7f968c0a2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 shutdown_connections 2024-09-10T16:20:46.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:46.951+0000 7f96923a1640 1 -- 172.21.15.164:0/4289409390 wait complete. 2024-09-10T16:20:46.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:20:46.981 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:20:46.981 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:20:46.981 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:20:46.981 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:20:46.981 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:20:46.981 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:20:46.994 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:20:46.994 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:20:46.994 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:47.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.130+0000 7fe840b2b640 10 monclient: get_monmap_and_config 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.130+0000 7fe840b2b640 10 monclient: build_initial_monmap 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.130+0000 7fe840b2b640 10 monclient: monmap: 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:47.131448+0000 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:47.131448+0000 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:47.132 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:47.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 Processor -- start 2024-09-10T16:20:47.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 -- start start 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient: init 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe83c09f930 0x7fe83c0a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe83c09f930 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe83c0a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 0x7fe83c13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe83c0a27b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe83c09fd10 con 0x7fe83c0a2380 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:47.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe83c09cd40 con 0x7fe83c09f930 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe83c0a2c40 con 0x7fe83c0a27b0 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe840b2b640 10 monclient(hunting): authenticate will time out at 1625.618761s 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe83c09f930 0x7fe83c0a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe831d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fe83c0a2380 0x7fe83c13a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44270/0 (socket says 172.21.15.164:44270) 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe831d9b640 1 -- 172.21.15.164:0/1084486065 learned_addr learned my addr 172.21.15.164:0/1084486065 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 1 --2- 172.21.15.164:0/1084486065 >> v2:172.21.15.164:3300/0 conn(0x7fe83c09f930 0x7fe83c0a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60512/0 (socket says 172.21.15.164:60512) 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): get_auth_request con 0x7fe83c09f930 auth_method 0 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:47.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.131+0000 7fe83a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83ad9d640 1 --2- 172.21.15.164:0/1084486065 >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 0x7fe83c13dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 10 monclient(hunting): handle_auth_done global_id 164148 payload 274 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 1 -- 172.21.15.164:0/1084486065 >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 msgr2=0x7fe83c13dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 1 --2- 172.21.15.164:0/1084486065 >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 0x7fe83c13dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 1 -- 172.21.15.164:0/1084486065 >> v1:172.21.15.164:6789/0 conn(0x7fe83c0a2380 legacy=0x7fe83c13a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 1 monclient: found mon.noname-a 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83a59c640 1 -- 172.21.15.164:0/1084486065 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c09a0d0 con 0x7fe83c09f930 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83ad9d640 10 monclient: get_auth_request con 0x7fe83c0a27b0 auth_method 0 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.132+0000 7fe83ad9d640 1 --2- 172.21.15.164:0/1084486065 >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 0x7fe83c13dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe83a59c640 1 -- 172.21.15.164:0/1084486065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c09f930 msgr2=0x7fe83c0a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe83a59c640 1 --2- 172.21.15.164:0/1084486065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c09f930 0x7fe83c0a1e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c06d470 tx=0x7fe82c029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4bd8515b9e2aab58 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 1 -- 172.21.15.164:0/1084486065 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c037070 con 0x7fe83c09f930 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 10 monclient: dump: 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:47.147 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 10 monclient: _finish_auth 0 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.134+0000 7fe839d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 1 -- 172.21.15.164:0/1084486065 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82c032440 con 0x7fe83c09f930 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 1 -- 172.21.15.164:0/1084486065 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82c032810 con 0x7fe83c09f930 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 5 monclient: authenticate success, global_id 164148 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:47.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe839d9b640 10 monclient: dump: 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 4 set_mon_vals no callback set 2024-09-10T16:20:47.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 10 monclient: shutdown 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 -- 172.21.15.164:0/1084486065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c09f930 msgr2=0x7fe83c0a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 --2- 172.21.15.164:0/1084486065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c09f930 0x7fe83c0a1e40 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe83c06d470 tx=0x7fe82c029fb0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 -- 172.21.15.164:0/1084486065 shutdown_connections 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 --2- 172.21.15.164:0/1084486065 >> v2:172.21.15.165:3300/0 conn(0x7fe83c0a27b0 0x7fe83c13dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 --2- 172.21.15.164:0/1084486065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c09f930 0x7fe83c0a1e40 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 -- 172.21.15.164:0/1084486065 >> 172.21.15.164:0/1084486065 conn(0x7fe83c0a3110 msgr2=0x7fe83c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 -- 172.21.15.164:0/1084486065 shutdown_connections 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.135+0000 7fe840b2b640 1 -- 172.21.15.164:0/1084486065 wait complete. 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.136+0000 7fe840b2b640 10 monclient: build_initial_monmap 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.136+0000 7fe840b2b640 10 monclient: monmap: 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:47.150 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.136+0000 7fe840b2b640 1 Processor -- start 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.136+0000 7fe840b2b640 1 -- start start 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.136+0000 7fe840b2b640 10 monclient: init 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:20:47.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 0x7fe83c1cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-b con 0x7fe83c09f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-a con 0x7fe83c0a2380 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 0x7fe83c09a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): picked mon.noname-c con 0x7fe83c0a27b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c09ee00 con 0x7fe83c0a2380 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe83c09e8d0 con 0x7fe83c09f930 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe83c09d930 con 0x7fe83c0a27b0 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:47.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe840b2b640 10 monclient(hunting): authenticate will time out at 1625.624275s 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83ad9d640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 msgr2=0x7fe83c09a460 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 0x7fe83c09a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60522/0 (socket says 172.21.15.164:60522) 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 -- 172.21.15.164:0/630356855 learned_addr learned my addr 172.21.15.164:0/630356855 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): get_auth_request con 0x7fe83c0a2380 auth_method 0 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 0x7fe83c1cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:47.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 10 monclient(hunting): get_auth_request con 0x7fe83c09f930 auth_method 0 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient(hunting): handle_auth_done global_id 164151 payload 824 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 -- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 msgr2=0x7fe83c09a460 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 0x7fe83c09a460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 -- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 msgr2=0x7fe83c1cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 0x7fe83c1cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 monclient: found mon.noname-a 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe831d9b640 1 -- 172.21.15.164:0/630356855 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe83c1b7940 con 0x7fe83c0a2380 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.137+0000 7fe83a59c640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 0x7fe83c1cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe831d9b640 1 -- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 msgr2=0x7fe83c1cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe831d9b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c460 tx=0x7fe82400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824007070 con 0x7fe83c0a2380 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 10 monclient: dump: 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:47.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 10 monclient: _finish_auth 0 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe824007230 con 0x7fe83c0a2380 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.139+0000 7fe840b2b640 5 monclient: authenticate success, global_id 164151 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 10 monclient: _renew_subs 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe83c09be00 con 0x7fe83c0a2380 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe8337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:47.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe824007620 con 0x7fe83c0a2380 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe8337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe8337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe8337fe640 10 monclient: dump: 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:47.157 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe83959a640 4 set_mon_vals no callback set 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 10 monclient: _renew_subs 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.140+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe83c1b5e20 con 0x7fe83c0a2380 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe82401b400 con 0x7fe83c0a2380 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe840b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe840b2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe840b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe83c0a0760 con 0x7fe83c0a2380 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe8337fe640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7fe824052a20 con 0x7fe83c0a2380 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe83a59c640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:47.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.141+0000 7fe83a59c640 10 monclient: get_auth_request con 0x7fe81c03e1f0 auth_method 0 2024-09-10T16:20:47.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.144+0000 7fe83a59c640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 0x7fe81c0406b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe82c034000 tx=0x7fe82c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:47.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.146+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe824051420 con 0x7fe83c0a2380 2024-09-10T16:20:47.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.146+0000 7fe8337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:47.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.146+0000 7fe8337fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:47.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe840b2b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe840b2b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe840b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe83c1b7100 con 0x7fe83c0a2380 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe8337fe640 1 -- 172.21.15.164:0/630356855 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fe824012400 con 0x7fe83c0a2380 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe8337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.387+0000 7fe8337fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:47.389 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 msgr2=0x7fe81c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 0x7fe81c0406b0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fe82c034000 tx=0x7fe82c02e000 comp rx=0 tx=0).stop 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 10 monclient: shutdown 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 msgr2=0x7fe83c1cdca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe82400c460 tx=0x7fe82400c920 comp rx=0 tx=0).stop 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 shutdown_connections 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe81c03e1f0 0x7fe81c0406b0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe83c0a27b0 0x7fe83c09a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe83c0a2380 0x7fe83c1cdca0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 --2- 172.21.15.164:0/630356855 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe83c09f930 0x7fe83c1cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:47.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 >> 172.21.15.164:0/630356855 conn(0x7fe83c0a3110 msgr2=0x7fe83c0a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:47.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 shutdown_connections 2024-09-10T16:20:47.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:47.389+0000 7fe840b2b640 1 -- 172.21.15.164:0/630356855 wait complete. 2024-09-10T16:20:50.403 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:50.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.509+0000 7f07765ec640 10 monclient: get_monmap_and_config 2024-09-10T16:20:50.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.509+0000 7f07765ec640 10 monclient: build_initial_monmap 2024-09-10T16:20:50.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.509+0000 7f07765ec640 10 monclient: monmap: 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:50.511638+0000 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:50.511638+0000 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:50.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.510+0000 7f07765ec640 1 Processor -- start 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.510+0000 7f07765ec640 1 -- start start 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.510+0000 7f07765ec640 10 monclient: init 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 0x7f077009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-b con 0x7f077009c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 0x7f077009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-c con 0x7f077009d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f077009f4c0 0x7f077013b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-a con 0x7f077009f4c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f077009fbc0 con 0x7f077009f4c0 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f07700a2a20 con 0x7f077009d190 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f077009c6f0 con 0x7f077009c310 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f07765ec640 10 monclient(hunting): authenticate will time out at 1628.999130s 2024-09-10T16:20:50.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f076ffff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 msgr2=0x7f077009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f076ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 0x7f077009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f077009f4c0 0x7f077013b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f077009f4c0 0x7f077013b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60530/0 (socket says 172.21.15.164:60530) 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 -- 172.21.15.164:0/3580653611 learned_addr learned my addr 172.21.15.164:0/3580653611 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): get_auth_request con 0x7f077009f4c0 auth_method 0 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:50.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f076f7fe640 1 --2- 172.21.15.164:0/3580653611 >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 0x7f077009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient(hunting): handle_auth_done global_id 164154 payload 274 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 -- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 msgr2=0x7f077009ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 --2- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 0x7f077009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 -- 172.21.15.164:0/3580653611 >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 msgr2=0x7f077009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 --2- 172.21.15.164:0/3580653611 >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 0x7f077009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient: _finish_hunting 0 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 monclient: found mon.noname-a 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f0774b89640 1 -- 172.21.15.164:0/3580653611 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f077009a4d0 con 0x7f077009f4c0 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f076f7fe640 10 monclient: get_auth_request con 0x7f077009d190 auth_method 0 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.511+0000 7f076f7fe640 1 --2- 172.21.15.164:0/3580653611 >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 0x7f077009ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f0774b89640 1 -- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009f4c0 msgr2=0x7f077013b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f0774b89640 1 --2- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009f4c0 0x7f077013b5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f077006d5b0 tx=0x7f076802a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=692c43048d1860ed server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 1 -- 172.21.15.164:0/3580653611 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768036070 con 0x7f077009f4c0 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: dump: 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: _finish_auth 0 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:50.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 1 -- 172.21.15.164:0/3580653611 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0768007800 con 0x7f077009f4c0 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f07765ec640 5 monclient: authenticate success, global_id 164154 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f07765ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 1 -- 172.21.15.164:0/3580653611 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768007bd0 con 0x7f077009f4c0 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f07765ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f07765ec640 4 set_mon_vals no callback set 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.514+0000 7f076effd640 10 monclient: dump: 2024-09-10T16:20:50.530 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 10 monclient: shutdown 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:50.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 -- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009f4c0 msgr2=0x7f077013b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 --2- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009f4c0 0x7f077013b5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f077006d5b0 tx=0x7f076802a590 comp rx=0 tx=0).stop 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 -- 172.21.15.164:0/3580653611 shutdown_connections 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 --2- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009f4c0 0x7f077013b5c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 --2- 172.21.15.164:0/3580653611 >> v2:172.21.15.165:3300/0 conn(0x7f077009d190 0x7f077009ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 --2- 172.21.15.164:0/3580653611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f077009c310 0x7f077009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 -- 172.21.15.164:0/3580653611 >> 172.21.15.164:0/3580653611 conn(0x7f07700a3110 msgr2=0x7f07700a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 -- 172.21.15.164:0/3580653611 shutdown_connections 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 1 -- 172.21.15.164:0/3580653611 wait complete. 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 10 monclient: build_initial_monmap 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.515+0000 7f07765ec640 10 monclient: monmap: 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:50.533 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:50.534 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 Processor -- start 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 -- start start 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient: init 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-a con 0x7f077009c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f077009f4c0 0x7f07701bbb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-b con 0x7f077009f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 0x7f07701b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): picked mon.noname-c con 0x7f07701b1540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07700a2230 con 0x7f077009c310 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07700a20b0 con 0x7f077009f4c0 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0770074e00 con 0x7f07701b1540 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f07765ec640 10 monclient(hunting): authenticate will time out at 1629.004942s 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f0774b89640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 msgr2=0x7f07701b19b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:20:50.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.516+0000 7f0774b89640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 0x7f07701b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60534/0 (socket says 172.21.15.164:60534) 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 -- 172.21.15.164:0/1782389230 learned_addr learned my addr 172.21.15.164:0/1782389230 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): get_auth_request con 0x7f077009c310 auth_method 0 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:50.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f077009f4c0 0x7f07701bbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): get_auth_request con 0x7f077009f4c0 auth_method 0 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:50.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient(hunting): handle_auth_done global_id 164157 payload 824 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 -- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 msgr2=0x7f07701b19b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 0x7f07701b19b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 -- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f077009f4c0 msgr2=0x7f07701bbb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f077009f4c0 0x7f07701bbb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:50.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 monclient: found mon.noname-a 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.517+0000 7f076ffff640 1 -- 172.21.15.164:0/1782389230 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07701b4ba0 con 0x7f077009c310 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076ffff640 1 -- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 msgr2=0x7f07701bb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076ffff640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f076000b550 tx=0x7f076000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0760005070 con 0x7f077009c310 2024-09-10T16:20:50.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: dump: 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:50.541 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: _finish_auth 0 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 5 monclient: authenticate success, global_id 164157 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0760015070 con 0x7f077009c310 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 10 monclient: _renew_subs 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07701b4e80 con 0x7f077009c310 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0760005450 con 0x7f077009c310 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076cff9640 10 monclient: dump: 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:50.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 10 monclient: _renew_subs 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077009d620 con 0x7f077009c310 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.519+0000 7f076e7fc640 4 set_mon_vals no callback set 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.520+0000 7f07765ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.520+0000 7f07765ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.520+0000 7f07765ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.520+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f077009ea40 con 0x7f077009c310 2024-09-10T16:20:50.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.521+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f07600055f0 con 0x7f077009c310 2024-09-10T16:20:50.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.521+0000 7f076cff9640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 0x7f075c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:50.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.521+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7f07600529c0 con 0x7f077009c310 2024-09-10T16:20:50.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.521+0000 7f076f7fe640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 0x7f075c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:50.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.521+0000 7f076f7fe640 10 monclient: get_auth_request con 0x7f075c03e210 auth_method 0 2024-09-10T16:20:50.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.524+0000 7f076f7fe640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 0x7f075c0406d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f07701b5090 tx=0x7f0764008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:50.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.524+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:20:50.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.526+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0760010e20 con 0x7f077009c310 2024-09-10T16:20:50.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.526+0000 7f076cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:50.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.526+0000 7f076cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:50.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f07765ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:50.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f07765ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:50.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f07765ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:50.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f07701b2e00 con 0x7f077009c310 2024-09-10T16:20:50.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f076cff9640 1 -- 172.21.15.164:0/1782389230 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f0760019ce0 con 0x7f077009c310 2024-09-10T16:20:50.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f076cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:50.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.765+0000 7f076cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:50.769 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:50.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 msgr2=0x7f075c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:50.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 0x7f075c0406d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f07701b5090 tx=0x7f0764008040 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 10 monclient: shutdown 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 msgr2=0x7f07701bb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.767+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f076000b550 tx=0x7f076000ba10 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 shutdown_connections 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f075c03e210 0x7f075c0406d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f07701b1540 0x7f07701b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f077009f4c0 0x7f07701bbb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 --2- 172.21.15.164:0/1782389230 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f077009c310 0x7f07701bb5e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 >> 172.21.15.164:0/1782389230 conn(0x7f07700a3110 msgr2=0x7f077009d930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 shutdown_connections 2024-09-10T16:20:50.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:50.768+0000 7f07765ec640 1 -- 172.21.15.164:0/1782389230 wait complete. 2024-09-10T16:20:53.782 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.889+0000 7fbc98764640 10 monclient: get_monmap_and_config 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.889+0000 7fbc98764640 10 monclient: build_initial_monmap 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.889+0000 7fbc98764640 10 monclient: monmap: 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:53.891490+0000 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:53.891490+0000 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:53.892 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:53.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 Processor -- start 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 -- start start 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient: init 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 0x7fbc9009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbc9009d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:53.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbc9013a5c0 0x7fbc9013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9013a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbc9009a430 con 0x7fbc9009d160 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbc900a24e0 con 0x7fbc9013a5c0 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbc900a2bf0 con 0x7fbc9009c180 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.890+0000 7fbc98764640 10 monclient(hunting): authenticate will time out at 1632.378993s 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96500640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 msgr2=0x7fbc9009ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:20:53.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96500640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 0x7fbc9009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbc9013a5c0 0x7fbc9013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc95cff640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fbc9009d160 0x7fbc9009ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:39126/0 (socket says 172.21.15.164:39126) 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbc9013a5c0 0x7fbc9013c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:51868/0 (socket says 172.21.15.164:51868) 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc95cff640 1 -- 172.21.15.164:0/3497414115 learned_addr learned my addr 172.21.15.164:0/3497414115 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): get_auth_request con 0x7fbc9013a5c0 auth_method 0 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:53.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient(hunting): handle_auth_done global_id 154234 payload 274 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 -- 172.21.15.164:0/3497414115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 msgr2=0x7fbc9009ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 --2- 172.21.15.164:0/3497414115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 0x7fbc9009ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 -- 172.21.15.164:0/3497414115 >> v1:172.21.15.165:6789/0 conn(0x7fbc9009d160 legacy=0x7fbc9009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient: _finish_hunting 0 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 monclient: found mon.noname-c 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:20:53.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.891+0000 7fbc96d01640 1 -- 172.21.15.164:0/3497414115 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc9009c560 con 0x7fbc9013a5c0 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc96d01640 1 -- 172.21.15.164:0/3497414115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9013a5c0 msgr2=0x7fbc9013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc96d01640 1 --2- 172.21.15.164:0/3497414115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9013a5c0 0x7fbc9013c9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc9009abb0 tx=0x7fbc8802a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0cc5aafad853e3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 1 -- 172.21.15.164:0/3497414115 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88007a30 con 0x7fbc9013a5c0 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 10 monclient: dump: 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.906 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 10 monclient: _finish_auth 0 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.894+0000 7fbc954fe640 10 monclient: _check_auth_tickets 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:53.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 1 -- 172.21.15.164:0/3497414115 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc88007bd0 con 0x7fbc9013a5c0 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 5 monclient: authenticate success, global_id 154234 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 10 monclient: get_monmap_and_config success 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 1 -- 172.21.15.164:0/3497414115 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc88031400 con 0x7fbc9013a5c0 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 4 set_mon_vals no callback set 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc954fe640 10 monclient: dump: 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:53.908 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 10 monclient: shutdown 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:53.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 -- 172.21.15.164:0/3497414115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9013a5c0 msgr2=0x7fbc9013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 --2- 172.21.15.164:0/3497414115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9013a5c0 0x7fbc9013c9b0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fbc9009abb0 tx=0x7fbc8802a4b0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 -- 172.21.15.164:0/3497414115 shutdown_connections 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 --2- 172.21.15.164:0/3497414115 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9013a5c0 0x7fbc9013c9b0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 --2- 172.21.15.164:0/3497414115 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9009c180 0x7fbc9009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 -- 172.21.15.164:0/3497414115 >> 172.21.15.164:0/3497414115 conn(0x7fbc900a3110 msgr2=0x7fbc900a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 -- 172.21.15.164:0/3497414115 shutdown_connections 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.895+0000 7fbc98764640 1 -- 172.21.15.164:0/3497414115 wait complete. 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 10 monclient: build_initial_monmap 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 10 monclient: monmap: 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:20:53.910 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 1 Processor -- start 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 1 -- start start 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 10 monclient: init 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.896+0000 7fbc98764640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:53.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9009c180 0x7fbc9008f610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-b con 0x7fbc9009c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-a con 0x7fbc9009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 0x7fbc90090090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): picked mon.noname-c con 0x7fbc9013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbc90074f90 con 0x7fbc9009d160 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbc90074e10 con 0x7fbc9009c180 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbc9009f690 con 0x7fbc9013a5c0 2024-09-10T16:20:53.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc98764640 10 monclient(hunting): authenticate will time out at 1632.385075s 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96d01640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 msgr2=0x7fbc90090090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96d01640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 0x7fbc90090090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60546/0 (socket says 172.21.15.164:60546) 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 -- 172.21.15.164:0/1937832232 learned_addr learned my addr 172.21.15.164:0/1937832232 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): get_auth_request con 0x7fbc9009d160 auth_method 0 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:53.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9009c180 0x7fbc9008f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): get_auth_request con 0x7fbc9009c180 auth_method 0 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc96500640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient(hunting): handle_auth_done global_id 164160 payload 824 2024-09-10T16:20:53.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 -- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 msgr2=0x7fbc90090090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 0x7fbc90090090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 -- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9009c180 msgr2=0x7fbc9008f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9009c180 0x7fbc9008f610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient: _finish_hunting 0 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 monclient: found mon.noname-a 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.897+0000 7fbc95cff640 1 -- 172.21.15.164:0/1937832232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc900a4c10 con 0x7fbc9009d160 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc95cff640 1 -- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 msgr2=0x7fbc9008fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc95cff640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c490 tx=0x7fbc8000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc80017070 con 0x7fbc9009d160 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:53.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: dump: 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: _finish_auth 0 2024-09-10T16:20:53.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc800070d0 con 0x7fbc9009d160 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 5 monclient: authenticate success, global_id 164160 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 10 monclient: _renew_subs 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc900960c0 con 0x7fbc9009d160 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc800074c0 con 0x7fbc9009d160 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc86ffd640 10 monclient: dump: 2024-09-10T16:20:53.917 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc94cfd640 4 set_mon_vals no callback set 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 10 monclient: _renew_subs 2024-09-10T16:20:53.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.899+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc90095de0 con 0x7fbc9009d160 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.900+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbc8001a070 con 0x7fbc9009d160 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.900+0000 7fbc98764640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.900+0000 7fbc98764640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.900+0000 7fbc98764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.900+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc9009ea40 con 0x7fbc9009d160 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.901+0000 7fbc86ffd640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 0x7fbc74040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.901+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7fbc80052160 con 0x7fbc9009d160 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.901+0000 7fbc96500640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 0x7fbc74040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.901+0000 7fbc96500640 10 monclient: get_auth_request con 0x7fbc7403e1c0 auth_method 0 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.903+0000 7fbc96500640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 0x7fbc74040680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c003980 tx=0x7fbc8c009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:20:53.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.903+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:20:53.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.905+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbc900a4c10 con 0x7fbc9009d160 2024-09-10T16:20:53.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.905+0000 7fbc86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:20:53.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:53.905+0000 7fbc86ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.144+0000 7fbc98764640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.144+0000 7fbc98764640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.144+0000 7fbc98764640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.144+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbc90077260 con 0x7fbc9009d160 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.145+0000 7fbc86ffd640 1 -- 172.21.15.164:0/1937832232 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7fbc80050b60 con 0x7fbc9009d160 2024-09-10T16:20:54.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.145+0000 7fbc86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:20:54.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.145+0000 7fbc86ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:20:54.148 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":144,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:20:54.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 msgr2=0x7fbc74040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 0x7fbc74040680 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c003980 tx=0x7fbc8c009110 comp rx=0 tx=0).stop 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 10 monclient: shutdown 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 msgr2=0x7fbc9008fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbc8000c490 tx=0x7fbc8000c890 comp rx=0 tx=0).stop 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 shutdown_connections 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbc7403e1c0 0x7fbc74040680 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbc9013a5c0 0x7fbc90090090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbc9009d160 0x7fbc9008fb50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:54.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 --2- 172.21.15.164:0/1937832232 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbc9009c180 0x7fbc9008f610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:20:54.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 >> 172.21.15.164:0/1937832232 conn(0x7fbc900a3110 msgr2=0x7fbc90139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:20:54.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 shutdown_connections 2024-09-10T16:20:54.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:54.147+0000 7fbc98764640 1 -- 172.21.15.164:0/1937832232 wait complete. 2024-09-10T16:20:57.161 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 10 monclient: get_monmap_and_config 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 10 monclient: build_initial_monmap 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 10 monclient: monmap: 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:20:57.270902+0000 2024-09-10T16:20:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:20:57.270902+0000 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:20:57.272 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 1 Processor -- start 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 1 -- start start 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.270+0000 7f360e164640 10 monclient: init 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient: _reopen_session rank -1 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f360809d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): picked mon.noname-a con 0x7f360809ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f360809e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36080a0870 con 0x7f360809e420 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3608074c70 con 0x7f360809d930 2024-09-10T16:20:57.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f360809d3e0 con 0x7f360809ab50 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): _renew_subs 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f360e164640 10 monclient(hunting): authenticate will time out at 1635.758356s 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3607fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f360809e420 0x7f360813bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:48436/0 (socket says 172.21.15.164:48436) 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3607fff640 1 -- 172.21.15.164:0/4207893096 learned_addr learned my addr 172.21.15.164:0/4207893096 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f360809ab50 auth_method 0 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:57.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.271+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:20:57.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.273+0000 7f3607fff640 1 -- 172.21.15.164:0/4207893096 >> v1:172.21.15.164:6789/0 conn(0x7f360809e420 legacy=0x7f360813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:57.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.273+0000 7f36077fe640 1 -- 172.21.15.164:0/4207893096 >> v1:172.21.15.165:6789/0 conn(0x7f360809d930 legacy=0x7f360809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:20:57.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f360809ab50 auth_method 0 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:57.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:20:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f360809ab50 auth_method 0 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:57.875+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:20:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:20:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f360809ab50 auth_method 0 2024-09-10T16:20:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:20:58.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:20:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:20:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:20:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:20:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:20:58.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:20:58.676+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:00.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): tick 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 msgr2=0x7f36081397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.164:3300/0 conn(0x7f360809ab50 0x7f36081397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 -- 172.21.15.164:0/4207893096 >> v1:172.21.15.165:6789/0 conn(0x7f360809d930 legacy=0x7f360809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 -- 172.21.15.164:0/4207893096 >> v1:172.21.15.164:6789/0 conn(0x7f360809e420 legacy=0x7f360813bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient: _add_conns ranks=[2,4] 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): picked mon.noname-b con 0x7f35f4003830 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:00.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): picked mon.noname-c con 0x7f35f4006390 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 -- 172.21.15.164:0/4207893096 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f360809d3e0 con 0x7f35f4006390 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 1 -- 172.21.15.164:0/4207893096 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f35f4008f10 con 0x7f35f4003830 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36057fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36077fe640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 msgr2=0x7f35f4005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f36077fe640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f35f4006390 auth_method 0 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:00.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.271+0000 7f3606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:00.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.272+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:00.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.272+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:00.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.272+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:00.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.272+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:00.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f36077fe640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 msgr2=0x7f35f4005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:00.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f36077fe640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f35f4006390 auth_method 0 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:00.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:00.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:00.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:00.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.473+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:00.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:00.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f35f4006390 auth_method 0 2024-09-10T16:21:00.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:00.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:00.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 msgr2=0x7f35f4008850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f36077fe640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 msgr2=0x7f35f4005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f36077fe640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:00.874+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.675+0000 7f36077fe640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 msgr2=0x7f35f4005c60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.675+0000 7f36077fe640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> v2:172.21.15.165:3300/0 conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f35f4006390 auth_method 0 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:01.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient(hunting): handle_auth_done global_id 154240 payload 274 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 msgr2=0x7f35f4005c60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f35f4003830 0x7f35f4005c60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 1 monclient: found mon.noname-c 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:21:01.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.676+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f360809ecb0 con 0x7f35f4006390 2024-09-10T16:21:01.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.679+0000 7f3606ffd640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 msgr2=0x7f35f4008850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:01.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.679+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f35f4008850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006fd0 tx=0x7f35fc02d3e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4dfeb3aee193d430 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:01.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 1 -- 172.21.15.164:0/4207893096 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc00b040 con 0x7f35f4006390 2024-09-10T16:21:01.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:01.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:01.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: dump: 2024-09-10T16:21:01.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f3606ffd640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f35f4008850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006fd0 tx=0x7f35fc02d3e0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f35fc0162d0 config(0 keys) 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:01.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f360e164640 5 monclient: authenticate success, global_id 154240 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f360e164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 1 -- 172.21.15.164:0/4207893096 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc0162d0 con 0x7f35f4006390 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f36067fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f360e164640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:01.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.680+0000 7f360e164640 4 set_mon_vals no callback set 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 10 monclient: shutdown 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 -- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 msgr2=0x7f35f4008850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f35f4008850 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f35fc006fd0 tx=0x7f35fc02d3e0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f3607fff640 1 -- 172.21.15.164:0/4207893096 reap_dead start 2024-09-10T16:21:01.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f36067fc640 1 -- 172.21.15.164:0/4207893096 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 0 0 0) 0x7f35fc004260 con 0x7f35f4006390 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f36067fc640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 -- 172.21.15.164:0/4207893096 shutdown_connections 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 --2- 172.21.15.164:0/4207893096 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f35f4008850 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 -- 172.21.15.164:0/4207893096 >> 172.21.15.164:0/4207893096 conn(0x7f36080a3110 msgr2=0x7f36080a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 -- 172.21.15.164:0/4207893096 shutdown_connections 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 1 -- 172.21.15.164:0/4207893096 wait complete. 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 10 monclient: build_initial_monmap 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.681+0000 7f360e164640 10 monclient: monmap: 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:01.689 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.682+0000 7f360e164640 1 Processor -- start 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.682+0000 7f360e164640 1 -- start start 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.682+0000 7f360e164640 10 monclient: init 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f3608073980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): picked mon.noname-b con 0x7f35f4006390 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:01.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): picked mon.noname-a con 0x7f3608134ac0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 0x7f360812e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): picked mon.noname-c con 0x7f360812dc40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f36080a2940 con 0x7f3608134ac0 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f360809ef30 con 0x7f35f4006390 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f36080a2ac0 con 0x7f360812dc40 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f360e164640 10 monclient(hunting): authenticate will time out at 1640.170257s 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3607fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 msgr2=0x7f360812e0a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3607fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 0x7f360812e0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:01.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44982/0 (socket says 172.21.15.164:44982) 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 -- 172.21.15.164:0/496618594 learned_addr learned my addr 172.21.15.164:0/496618594 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): get_auth_request con 0x7f3608134ac0 auth_method 0 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f3608073980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): get_auth_request con 0x7f35f4006390 auth_method 0 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:01.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f36077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient(hunting): handle_auth_done global_id 164163 payload 824 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 -- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 msgr2=0x7f360812e0a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 0x7f360812e0a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 -- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 msgr2=0x7f3608073980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f3608073980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 monclient: found mon.noname-a 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:01.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.683+0000 7f3606ffd640 1 -- 172.21.15.164:0/496618594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3608135220 con 0x7f3608134ac0 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f3606ffd640 1 -- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 msgr2=0x7f360812d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f3606ffd640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35fc016de0 tx=0x7f35fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc042020 con 0x7f3608134ac0 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: dump: 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:01.694 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: _finish_auth 0 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f35fc035610 con 0x7f3608134ac0 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 5 monclient: authenticate success, global_id 164163 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35fc03a040 con 0x7f3608134ac0 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:01.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f35effff640 10 monclient: dump: 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 10 monclient: _renew_subs 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:01.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f360809d620 con 0x7f3608134ac0 2024-09-10T16:21:01.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f3605ffb640 4 set_mon_vals no callback set 2024-09-10T16:21:01.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 10 monclient: _renew_subs 2024-09-10T16:21:01.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:01.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.686+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36080741f0 con 0x7f3608134ac0 2024-09-10T16:21:01.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f35fc045020 con 0x7f3608134ac0 2024-09-10T16:21:01.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f360e164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:01.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f360e164640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f360e164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f360809dd10 con 0x7f3608134ac0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f35effff640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 0x7f35f4041570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.687+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7f35fc074580 con 0x7f3608134ac0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.688+0000 7f36077fe640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 0x7f35f4041570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.688+0000 7f36077fe640 10 monclient: get_auth_request con 0x7f35f403f0b0 auth_method 0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.690+0000 7f36077fe640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 0x7f35f4041570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35f80012e0 tx=0x7f35f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.692+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f35fc04c070 con 0x7f3608134ac0 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.692+0000 7f35effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:01.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.692+0000 7f35effff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f360e164640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f360e164640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f360e164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f360806dac0 con 0x7f3608134ac0 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f35effff640 1 -- 172.21.15.164:0/496618594 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f35fc072f80 con 0x7f3608134ac0 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f35effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:01.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.938+0000 7f35effff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:01.941 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":148,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:01.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 msgr2=0x7f35f4041570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:01.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 0x7f35f4041570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f35f80012e0 tx=0x7f35f8009040 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 10 monclient: shutdown 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 msgr2=0x7f360812d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f35fc016de0 tx=0x7f35fc004730 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 shutdown_connections 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f35f403f0b0 0x7f35f4041570 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f360812dc40 0x7f360812e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3608134ac0 0x7f360812d700 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 --2- 172.21.15.164:0/496618594 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f35f4006390 0x7f3608073980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:01.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 >> 172.21.15.164:0/496618594 conn(0x7f36080a3110 msgr2=0x7f36081bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:01.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 shutdown_connections 2024-09-10T16:21:01.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:01.941+0000 7f360e164640 1 -- 172.21.15.164:0/496618594 wait complete. 2024-09-10T16:21:01.954 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:21:01.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:21:01.954 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 10 monclient: get_monmap_and_config 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 10 monclient: build_initial_monmap 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 10 monclient: monmap: 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:02.053908+0000 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:02.053908+0000 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:02.055 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 1 Processor -- start 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 1 -- start start 2024-09-10T16:21:02.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.053+0000 7f73e9d5d640 10 monclient: init 2024-09-10T16:21:02.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:02.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient: _add_conns ranks=[2,4,3,1,0] 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 0x7f73e40a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f73e4135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 0x7f73e41360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f73e40a40d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73e4136600 0x7f73e409b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f73e4136600 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f73e40a1400 con 0x7f73e4136600 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f73e40a4ac0 con 0x7f73e40a40d0 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f73e40a1120 con 0x7f73e4135ce0 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e9d5d640 10 monclient(hunting): authenticate will time out at 1640.541420s 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e37fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 msgr2=0x7f73e40a23a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 0x7f73e40a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73e4136600 0x7f73e409b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f73e4136600 0x7f73e409b0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44992/0 (socket says 172.21.15.164:44992) 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 -- 172.21.15.164:0/901581558 learned_addr learned my addr 172.21.15.164:0/901581558 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): get_auth_request con 0x7f73e4136600 auth_method 0 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/901581558 >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 0x7f73e41360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 10 monclient(hunting): get_auth_request con 0x7f73e40a40d0 auth_method 0 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient(hunting): handle_auth_done global_id 164166 payload 274 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 -- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 msgr2=0x7f73e40a23a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 --2- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 0x7f73e40a23a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 -- 172.21.15.164:0/901581558 >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 msgr2=0x7f73e41360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 --2- 172.21.15.164:0/901581558 >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 0x7f73e41360c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:02.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 monclient: found mon.noname-a 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e3fff640 1 -- 172.21.15.164:0/901581558 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e40a28e0 con 0x7f73e4136600 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.054+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/901581558 >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 0x7f73e41360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e3fff640 1 -- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 msgr2=0x7f73e409b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e3fff640 1 --2- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e409b0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f73e40a2b10 tx=0x7f73dc02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ad694ad301c303f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 1 -- 172.21.15.164:0/901581558 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc037070 con 0x7f73e4136600 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: dump: 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.064 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 1 -- 172.21.15.164:0/901581558 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73dc032640 con 0x7f73e4136600 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e9d5d640 5 monclient: authenticate success, global_id 164166 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e9d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:02.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e9d5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 1 -- 172.21.15.164:0/901581558 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc032a10 con 0x7f73e4136600 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e9d5d640 4 set_mon_vals no callback set 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.057+0000 7f73e27fc640 10 monclient: dump: 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.066 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 10 monclient: shutdown 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 -- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 msgr2=0x7f73e409b0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e409b0e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f73e40a2b10 tx=0x7f73dc02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 -- 172.21.15.164:0/901581558 shutdown_connections 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e409b0e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/901581558 >> v2:172.21.15.165:3300/0 conn(0x7f73e40a40d0 0x7f73e41360c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/901581558 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e4135ce0 0x7f73e40a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 -- 172.21.15.164:0/901581558 >> 172.21.15.164:0/901581558 conn(0x7f73e40a4f50 msgr2=0x7f73e40a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:02.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 -- 172.21.15.164:0/901581558 shutdown_connections 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.058+0000 7f73e9d5d640 1 -- 172.21.15.164:0/901581558 wait complete. 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.059+0000 7f73e9d5d640 10 monclient: build_initial_monmap 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.059+0000 7f73e9d5d640 10 monclient: monmap: 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:02.068 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.059+0000 7f73e9d5d640 1 Processor -- start 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.059+0000 7f73e9d5d640 1 -- start start 2024-09-10T16:21:02.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.059+0000 7f73e9d5d640 10 monclient: init 2024-09-10T16:21:02.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:02.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:21:02.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e40a40d0 0x7f73e412d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f73e40a40d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73e4135ce0 0x7f73e412da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f73e4135ce0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e412df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e37fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e40a40d0 msgr2=0x7f73e412d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f73e4136600 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f73e409b7a0 con 0x7f73e4136600 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f73e409b620 con 0x7f73e4135ce0 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f73e409b920 con 0x7f73e40a40d0 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e40a40d0 0x7f73e412d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e9d5d640 10 monclient(hunting): authenticate will time out at 1640.547117s 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e412df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e412df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45002/0 (socket says 172.21.15.164:45002) 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 learned_addr learned my addr 172.21.15.164:0/2804681279 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): get_auth_request con 0x7f73e4136600 auth_method 0 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73e4135ce0 0x7f73e412da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient(hunting): handle_auth_done global_id 164169 payload 824 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e40a40d0 msgr2=0x7f73e412d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f73e40a40d0 0x7f73e412d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73e4135ce0 msgr2=0x7f73e412da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73e4135ce0 0x7f73e412da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 monclient: found mon.noname-a 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e412fbb0 con 0x7f73e4136600 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e2ffd640 10 monclient: get_auth_request con 0x7f73e4135ce0 auth_method 0 2024-09-10T16:21:02.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.060+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f73e4135ce0 0x7f73e412da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 msgr2=0x7f73e412df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e3fff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e412df90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f73dc035040 tx=0x7f73dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc041020 con 0x7f73e4136600 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: dump: 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.081 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: _finish_auth 0 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f73dc03d910 con 0x7f73e4136600 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 5 monclient: authenticate success, global_id 164169 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 10 monclient: _renew_subs 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73e412fe90 con 0x7f73e4136600 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73dc037040 con 0x7f73e4136600 2024-09-10T16:21:02.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73cbfff640 10 monclient: dump: 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 10 monclient: _renew_subs 2024-09-10T16:21:02.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e1ffb640 4 set_mon_vals no callback set 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.062+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73e409c140 con 0x7f73e4136600 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e9d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 0x7f73e41cb440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f73e409bcc0 con 0x7f73e41cb060 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 0x7f73e41cb440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f73dc044020 con 0x7f73e4136600 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient: get_auth_request con 0x7f73e41cb060 auth_method 0 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.063+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73e37fe640 10 monclient(hunting): handle_auth_done global_id 164172 payload 824 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73e37fe640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f73e40008d0 con 0x7f73e41cb060 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73cbfff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f73d003e1c0 0x7f73d0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7f73dc0731e0 con 0x7f73e4136600 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f73d003e1c0 0x7f73d0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.064+0000 7f73e2ffd640 10 monclient: get_auth_request con 0x7f73d003e1c0 auth_method 0 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73e37fe640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 msgr2=0x7f73e41cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73e37fe640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 0x7f73e41cb440 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73d400bcc0 tx=0x7f73d4009f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4005560 con 0x7f73e41cb060 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 10 monclient: dump: 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f73d4005560 con 0x7f73e41cb060 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:02.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 msgr2=0x7f73e41cb440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 0x7f73e41cb440 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f73d400bcc0 tx=0x7f73d4009f30 comp rx=0 tx=0).stop 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 msgr2=0x7f73e41cb440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.066+0000 7f73cbfff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb060 0x7f73e41cb440 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.067+0000 7f73e2ffd640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f73d003e1c0 0x7f73d0040680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f73d8006fd0 tx=0x7f73d8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e9d5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 0x7f73e409cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e9d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f73e4130770 con 0x7f73e41cb980 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 0x7f73e409cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient: get_auth_request con 0x7f73e41cb980 auth_method 0 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 10 monclient(hunting): handle_auth_done global_id 164175 payload 824 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.072+0000 7f73e37fe640 1 -- 172.21.15.164:0/2804681279 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f73e41cbd90 con 0x7f73e41cb980 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e37fe640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 msgr2=0x7f73e409cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e37fe640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 0x7f73e409cb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f73d4014000 tx=0x7f73d4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f73d4019040 con 0x7f73e41cb980 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 10 monclient: dump: 2024-09-10T16:21:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f73d4014430 con 0x7f73e41cb980 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 msgr2=0x7f73e409cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 0x7f73e409cb70 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f73d4014000 tx=0x7f73d4005dc0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 msgr2=0x7f73e409cb70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73cbfff640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e41cb980 0x7f73e409cb70 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f73d4014000 tx=0x7f73d4005dc0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:02.090 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 148, 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 32778, 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.091 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.092 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:02.093 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:02.094 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.095 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.096 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:02.097 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:02.098 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.099 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.100 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.101 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:02.102 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f73d003e1c0 msgr2=0x7f73d0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f73d003e1c0 0x7f73d0040680 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f73d8006fd0 tx=0x7f73d8008040 comp rx=0 tx=0).stop 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 10 monclient: shutdown 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 msgr2=0x7f73e412df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 --2- 172.21.15.164:0/2804681279 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f73e4136600 0x7f73e412df90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f73dc035040 tx=0x7f73dc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 reap_dead start 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e3fff640 1 -- 172.21.15.164:0/2804681279 reap_dead start 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 shutdown_connections 2024-09-10T16:21:02.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.075+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 >> 172.21.15.164:0/2804681279 conn(0x7f73e40a4f50 msgr2=0x7f73e40a6710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.076+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 shutdown_connections 2024-09-10T16:21:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.076+0000 7f73e9d5d640 1 -- 172.21.15.164:0/2804681279 wait complete. 2024-09-10T16:21:02.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:21:02.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:21:02.105 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:21:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.202+0000 7fec60048640 10 monclient: get_monmap_and_config 2024-09-10T16:21:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.202+0000 7fec60048640 10 monclient: build_initial_monmap 2024-09-10T16:21:02.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.202+0000 7fec60048640 10 monclient: monmap: 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:02.203164+0000 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:02.203164+0000 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:02.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.202+0000 7fec60048640 1 Processor -- start 2024-09-10T16:21:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 -- start start 2024-09-10T16:21:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient: init 2024-09-10T16:21:02.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec5806cb70 0x7fec58134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): picked mon.noname-a con 0x7fec5806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fec580a46a0 0x7fec580a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): picked mon.noname-c con 0x7fec580a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec58137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec5809ce50 con 0x7fec58137ca0 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fec5806db60 con 0x7fec5806cb70 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fec5806cf50 con 0x7fec580a46a0 2024-09-10T16:21:02.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec60048640 10 monclient(hunting): authenticate will time out at 1640.690674s 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec5806cb70 0x7fec58134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5e5e5640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fec58137ca0 0x7fec58138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:48446/0 (socket says 172.21.15.164:48446) 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1224949376 learned_addr learned my addr 172.21.15.164:0/1224949376 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): get_auth_request con 0x7fec5806cb70 auth_method 0 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.203+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5d5e3640 1 --2- 172.21.15.164:0/1224949376 >> v2:172.21.15.165:3300/0 conn(0x7fec580a46a0 0x7fec580a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5d5e3640 10 monclient(hunting): get_auth_request con 0x7fec580a46a0 auth_method 0 2024-09-10T16:21:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5d5e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5d5e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5d5e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 10 monclient(hunting): handle_auth_done global_id 164178 payload 274 2024-09-10T16:21:02.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 1 -- 172.21.15.164:0/1224949376 >> v2:172.21.15.165:3300/0 conn(0x7fec580a46a0 msgr2=0x7fec580a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1224949376 >> v2:172.21.15.165:3300/0 conn(0x7fec580a46a0 0x7fec580a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 1 -- 172.21.15.164:0/1224949376 >> v1:172.21.15.164:6789/0 conn(0x7fec58137ca0 legacy=0x7fec58138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 10 monclient: _finish_hunting 0 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 1 monclient: found mon.noname-a 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.204+0000 7fec5dde4640 1 -- 172.21.15.164:0/1224949376 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec5809cb70 con 0x7fec5806cb70 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.206+0000 7fec5dde4640 1 -- 172.21.15.164:0/1224949376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec5806cb70 msgr2=0x7fec58134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.206+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1224949376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec5806cb70 0x7fec58134f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec540097d0 tx=0x7fec5402a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8173094d493b837 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 1 -- 172.21.15.164:0/1224949376 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54037070 con 0x7fec5806cb70 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:02.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: dump: 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: _finish_auth 0 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: _check_auth_tickets 2024-09-10T16:21:02.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 1 -- 172.21.15.164:0/1224949376 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec54032440 con 0x7fec5806cb70 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 5 monclient: authenticate success, global_id 164178 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 1 -- 172.21.15.164:0/1224949376 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54032810 con 0x7fec5806cb70 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 4 set_mon_vals no callback set 2024-09-10T16:21:02.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec5cde2640 10 monclient: dump: 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.222 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 10 monclient: shutdown 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 -- 172.21.15.164:0/1224949376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec5806cb70 msgr2=0x7fec58134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 --2- 172.21.15.164:0/1224949376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec5806cb70 0x7fec58134f70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fec540097d0 tx=0x7fec5402a040 comp rx=0 tx=0).stop 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 -- 172.21.15.164:0/1224949376 shutdown_connections 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 --2- 172.21.15.164:0/1224949376 >> v2:172.21.15.165:3300/0 conn(0x7fec580a46a0 0x7fec580a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 --2- 172.21.15.164:0/1224949376 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec5806cb70 0x7fec58134f70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 -- 172.21.15.164:0/1224949376 >> 172.21.15.164:0/1224949376 conn(0x7fec580aa900 msgr2=0x7fec580aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 -- 172.21.15.164:0/1224949376 shutdown_connections 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.207+0000 7fec60048640 1 -- 172.21.15.164:0/1224949376 wait complete. 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.208+0000 7fec60048640 10 monclient: build_initial_monmap 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.208+0000 7fec60048640 10 monclient: monmap: 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:02.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.208+0000 7fec60048640 1 Processor -- start 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.208+0000 7fec60048640 1 -- start start 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.208+0000 7fec60048640 10 monclient: init 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec5806cb70 0x7fec5812e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): picked mon.noname-b con 0x7fec5806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:02.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec580a46a0 0x7fec5812e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): picked mon.noname-c con 0x7fec580a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 0x7fec581cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): picked mon.noname-a con 0x7fec58137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fec581389c0 con 0x7fec58137ca0 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec58138840 con 0x7fec5806cb70 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec58140040 con 0x7fec580a46a0 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec60048640 10 monclient(hunting): authenticate will time out at 1640.696248s 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5d5e3640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec580a46a0 msgr2=0x7fec5812e9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5d5e3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec580a46a0 0x7fec5812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:02.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 0x7fec581cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 0x7fec581cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45050/0 (socket says 172.21.15.164:45050) 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 learned_addr learned my addr 172.21.15.164:0/1730116125 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): get_auth_request con 0x7fec58137ca0 auth_method 0 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec5806cb70 0x7fec5812e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): get_auth_request con 0x7fec5806cb70 auth_method 0 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient(hunting): handle_auth_done global_id 164181 payload 824 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec580a46a0 msgr2=0x7fec5812e9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec580a46a0 0x7fec5812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec5806cb70 msgr2=0x7fec5812e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec5806cb70 0x7fec5812e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient: _finish_hunting 0 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 monclient: found mon.noname-a 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:02.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.209+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec580a1210 con 0x7fec58137ca0 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 msgr2=0x7fec581cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec5e5e5640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 0x7fec581cd1d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec5000caf0 tx=0x7fec5000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec50012400 con 0x7fec58137ca0 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: dump: 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.229 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec60048640 5 monclient: authenticate success, global_id 164181 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec500125c0 con 0x7fec58137ca0 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec60048640 10 monclient: _renew_subs 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec500129b0 con 0x7fec58137ca0 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec60048640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec5812eef0 con 0x7fec58137ca0 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.211+0000 7fec4bfff640 4 set_mon_vals no callback set 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec4a7fc640 10 monclient: dump: 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 10 monclient: _renew_subs 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec580a3570 con 0x7fec58137ca0 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 0x7fec581cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec581ce030 con 0x7fec581cd710 2024-09-10T16:21:02.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.212+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fec50012b90 con 0x7fec58137ca0 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 0x7fec581cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient: get_auth_request con 0x7fec581cd710 auth_method 0 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec4a7fc640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec3c03e1f0 0x7fec3c0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(499..499 src has 184..499) ==== 4783+0+0 (secure 0 0 0) 0x7fec50053360 con 0x7fec58137ca0 2024-09-10T16:21:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5d5e3640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec3c03e1f0 0x7fec3c0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 10 monclient(hunting): handle_auth_done global_id 154255 payload 824 2024-09-10T16:21:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5dde4640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec58134f70 con 0x7fec581cd710 2024-09-10T16:21:02.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.213+0000 7fec5d5e3640 10 monclient: get_auth_request con 0x7fec3c03e1f0 auth_method 0 2024-09-10T16:21:02.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.215+0000 7fec5dde4640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 msgr2=0x7fec581cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.215+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 0x7fec581cdaf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec54034000 tx=0x7fec540046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec5d5e3640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec3c03e1f0 0x7fec3c0406b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fec580a0dc0 tx=0x7fec4c0092e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54037040 con 0x7fec581cd710 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 10 monclient: dump: 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.237 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec54004220 con 0x7fec581cd710 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 msgr2=0x7fec581cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 0x7fec581cdaf0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fec54034000 tx=0x7fec540046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 msgr2=0x7fec581cdaf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.216+0000 7fec4a7fc640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec581cd710 0x7fec581cdaf0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec60048640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec60048640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 0x7fec58130780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:02.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec60048640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec580770f0 con 0x7fec58130370 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 0x7fec58130780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec5dde4640 10 monclient: get_auth_request con 0x7fec58130370 auth_method 0 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec5dde4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec5dde4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.222+0000 7fec5dde4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.223+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.223+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.223+0000 7fec5dde4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.223+0000 7fec5dde4640 10 monclient(hunting): handle_auth_done global_id 154258 payload 824 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.223+0000 7fec5dde4640 1 -- 172.21.15.164:0/1730116125 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec580a3ce0 con 0x7fec58130370 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec5dde4640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 msgr2=0x7fec58130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec5dde4640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 0x7fec58130780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec54009720 tx=0x7fec54008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:02.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec54033e60 con 0x7fec58130370 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 10 monclient: dump: 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:02.240 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fec54003a20 con 0x7fec58130370 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.225+0000 7fec4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 msgr2=0x7fec58130780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec4a7fc640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 0x7fec58130780 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec54009720 tx=0x7fec54008030 comp rx=0 tx=0).stop 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec4a7fc640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 msgr2=0x7fec58130780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec4a7fc640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec58130370 0x7fec58130780 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fec54009720 tx=0x7fec54008030 comp rx=0 tx=0).stop 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:02.241 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 148, 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 80775, 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.242 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.243 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.244 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:02.245 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:02.246 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.247 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:02.248 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:02.249 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:02.251 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:02.252 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.253 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:02.254 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:02.255 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:02.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec3c03e1f0 msgr2=0x7fec3c0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec3c03e1f0 0x7fec3c0406b0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fec580a0dc0 tx=0x7fec4c0092e0 comp rx=0 tx=0).stop 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 10 monclient: shutdown 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 msgr2=0x7fec581cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 --2- 172.21.15.164:0/1730116125 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec58137ca0 0x7fec581cd1d0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fec5000caf0 tx=0x7fec5000a060 comp rx=0 tx=0).stop 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 reap_dead start 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec5e5e5640 1 -- 172.21.15.164:0/1730116125 reap_dead start 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 shutdown_connections 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 >> 172.21.15.164:0/1730116125 conn(0x7fec580aa900 msgr2=0x7fec580a7390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 shutdown_connections 2024-09-10T16:21:02.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:02.226+0000 7fec60048640 1 -- 172.21.15.164:0/1730116125 wait complete. 2024-09-10T16:21:02.258 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:21:02.258 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:21:02.338 INFO:tasks.rados.rados.0.smithi165.stdout:2184: done (2 left) 2024-09-10T16:21:02.338 INFO:tasks.rados.rados.0.smithi165.stdout:2186: done (1 left) 2024-09-10T16:21:02.338 INFO:tasks.rados.rados.0.smithi165.stdout:2190: done (0 left) 2024-09-10T16:21:02.338 INFO:tasks.rados.rados.0.smithi165.stdout:2191: write oid 47 current snap is 238 2024-09-10T16:21:02.338 INFO:tasks.rados.rados.0.smithi165.stdout:2191: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-10T16:21:02.344 INFO:tasks.rados.rados.0.smithi165.stdout:2191: writing smithi16534841-47 from 642232 to 1303462 tid 1 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2191: writing smithi16534841-47 from 1942968 to 2726891 tid 2 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2192: read oid 15 snap -1 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2192: expect (ObjNum 749 snap 227 seq_num 749) 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2193: read oid 37 snap -1 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2193: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2194: write (excl) oid 13 current snap is 238 2024-09-10T16:21:02.353 INFO:tasks.rados.rados.0.smithi165.stdout:2194: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-10T16:21:02.360 INFO:tasks.rados.rados.0.smithi165.stdout:2194: writing smithi16534841-13 from 659039 to 1395518 tid 1 2024-09-10T16:21:02.368 INFO:tasks.rados.rados.0.smithi165.stdout:2194: writing smithi16534841-13 from 2001450 to 2645384 tid 2 2024-09-10T16:21:02.368 INFO:tasks.rados.rados.0.smithi165.stdout:2195: delete oid 25 current snap is 238 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: finishing write tid 1 to smithi16534841-47 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: finishing write tid 2 to smithi16534841-47 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: finishing write tid 3 to smithi16534841-47 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: finishing write tid 4 to smithi16534841-47 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: oid 47 updating version 0 to 1335 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: oid 47 updating version 1335 to 1336 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: oid 47 version 1336 is already newer than 1334 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1336 (ObjNum 776 snap 238 seq_num 776) dirty exists 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2191: left oid 47 (ObjNum 776 snap 238 seq_num 776) 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:2193: expect (ObjNum 732 snap 223 seq_num 732) 2024-09-10T16:21:02.369 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:02.395 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:02.439 INFO:tasks.rados.rados.0.smithi165.stdout:2192: expect (ObjNum 749 snap 227 seq_num 749) 2024-09-10T16:21:02.460 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1266022 2024-09-10T16:21:02.554 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:02.614 INFO:tasks.rados.rados.0.smithi165.stdout:2194: finishing write tid 1 to smithi16534841-13 2024-09-10T16:21:02.614 INFO:tasks.rados.rados.0.smithi165.stdout:2191: done (4 left) 2024-09-10T16:21:02.614 INFO:tasks.rados.rados.0.smithi165.stdout:2192: done (3 left) 2024-09-10T16:21:02.614 INFO:tasks.rados.rados.0.smithi165.stdout:2193: done (2 left) 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2195: done (1 left) 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2196: write (excl) oid 33 current snap is 238 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: finishing write tid 2 to smithi16534841-13 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: finishing write tid 3 to smithi16534841-13 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: finishing write tid 4 to smithi16534841-13 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: oid 13 updating version 0 to 1342 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: oid 13 updating version 1342 to 1343 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: oid 13 updating version 1343 to 1344 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1344 (ObjNum 777 snap 238 seq_num 777) dirty exists 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2194: left oid 13 (ObjNum 777 snap 238 seq_num 777) 2024-09-10T16:21:02.615 INFO:tasks.rados.rados.0.smithi165.stdout:2196: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-10T16:21:02.620 INFO:tasks.rados.rados.0.smithi165.stdout:2196: writing smithi16534841-33 from 692653 to 1179630 tid 1 2024-09-10T16:21:02.626 INFO:tasks.rados.rados.0.smithi165.stdout:2196: writing smithi16534841-33 from 1602061 to 2249664 tid 2 2024-09-10T16:21:02.631 INFO:tasks.rados.rados.0.smithi165.stdout:2196: writing smithi16534841-33 from 2992629 to 3092653 tid 3 2024-09-10T16:21:02.631 INFO:tasks.rados.rados.0.smithi165.stdout:2194: done (1 left) 2024-09-10T16:21:02.631 INFO:tasks.rados.rados.0.smithi165.stdout:2197: write (excl) oid 20 current snap is 238 2024-09-10T16:21:02.631 INFO:tasks.rados.rados.0.smithi165.stdout:2196: finishing write tid 1 to smithi16534841-33 2024-09-10T16:21:02.631 INFO:tasks.rados.rados.0.smithi165.stdout:2197: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-10T16:21:02.637 INFO:tasks.rados.rados.0.smithi165.stdout:2197: writing smithi16534841-20 from 709460 to 1271686 tid 1 2024-09-10T16:21:02.646 INFO:tasks.rados.rados.0.smithi165.stdout:2197: writing smithi16534841-20 from 2060543 to 2851804 tid 2 2024-09-10T16:21:02.647 INFO:tasks.rados.rados.0.smithi165.stdout:2198: snap_remove snap 227 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2198: done (2 left) 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2199: snap_create 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: finishing write tid 2 to smithi16534841-33 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: finishing write tid 3 to smithi16534841-33 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: finishing write tid 4 to smithi16534841-33 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: finishing write tid 5 to smithi16534841-33 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: oid 33 updating version 0 to 1237 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: oid 33 version 1237 is already newer than 1235 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: oid 33 updating version 1237 to 1238 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: oid 33 version 1238 is already newer than 1236 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1238 (ObjNum 779 snap 238 seq_num 779) dirty exists 2024-09-10T16:21:03.341 INFO:tasks.rados.rados.0.smithi165.stdout:2196: left oid 33 (ObjNum 779 snap 238 seq_num 779) 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: finishing write tid 1 to smithi16534841-20 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: finishing write tid 2 to smithi16534841-20 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: finishing write tid 3 to smithi16534841-20 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: finishing write tid 4 to smithi16534841-20 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: oid 20 updating version 0 to 1239 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: oid 20 updating version 1239 to 1240 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: oid 20 updating version 1240 to 1241 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1241 (ObjNum 780 snap 238 seq_num 780) dirty exists 2024-09-10T16:21:03.342 INFO:tasks.rados.rados.0.smithi165.stdout:2197: left oid 20 (ObjNum 780 snap 238 seq_num 780) 2024-09-10T16:21:04.312 INFO:tasks.rados.rados.0.smithi165.stdout:2196: done (2 left) 2024-09-10T16:21:04.312 INFO:tasks.rados.rados.0.smithi165.stdout:2197: done (1 left) 2024-09-10T16:21:04.312 INFO:tasks.rados.rados.0.smithi165.stdout:2199: done (0 left) 2024-09-10T16:21:04.312 INFO:tasks.rados.rados.0.smithi165.stdout:2200: snap_create 2024-09-10T16:21:05.320 INFO:tasks.rados.rados.0.smithi165.stdout:2200: done (0 left) 2024-09-10T16:21:05.320 INFO:tasks.rados.rados.0.smithi165.stdout:2201: read oid 12 snap -1 2024-09-10T16:21:05.320 INFO:tasks.rados.rados.0.smithi165.stdout:2201: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:05.320 INFO:tasks.rados.rados.0.smithi165.stdout:2202: copy_from oid 6 from oid 14 current snap is 240 2024-09-10T16:21:05.321 INFO:tasks.rados.rados.0.smithi165.stdout:2203: snap_remove snap 238 2024-09-10T16:21:06.325 INFO:tasks.rados.rados.0.smithi165.stdout:2203: done (2 left) 2024-09-10T16:21:06.325 INFO:tasks.rados.rados.0.smithi165.stdout:2204: copy_from oid 2 from oid 33 current snap is 240 2024-09-10T16:21:06.325 INFO:tasks.rados.rados.0.smithi165.stdout:2201: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:06.456 INFO:tasks.rados.rados.0.smithi165.stdout:2202: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2202: finishing copy_from to smithi16534841-6 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1174 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2201: done (2 left) 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2202: done (1 left) 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2205: read oid 45 snap -1 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2205: expect deleted 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2206: snap_create 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2204: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:2204: finishing copy_from to smithi16534841-2 2024-09-10T16:21:06.457 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1238 (ObjNum 779 snap 238 seq_num 779) dirty exists 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2204: done (2 left) 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2205: done (1 left) 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2206: done (0 left) 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2207: read oid 34 snap 240 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2207: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2208: copy_from oid 33 from oid 36 current snap is 241 2024-09-10T16:21:07.328 INFO:tasks.rados.rados.0.smithi165.stdout:2209: snap_remove snap 235 2024-09-10T16:21:08.338 INFO:tasks.rados.rados.0.smithi165.stdout:2209: done (2 left) 2024-09-10T16:21:08.338 INFO:tasks.rados.rados.0.smithi165.stdout:2210: write (excl) oid 20 current snap is 241 2024-09-10T16:21:08.339 INFO:tasks.rados.rados.0.smithi165.stdout:2210: seq_num 781 ranges {726267=637475,2119025=651272} 2024-09-10T16:21:08.345 INFO:tasks.rados.rados.0.smithi165.stdout:2210: writing smithi16534841-20 from 726267 to 1363742 tid 1 2024-09-10T16:21:08.354 INFO:tasks.rados.rados.0.smithi165.stdout:2210: writing smithi16534841-20 from 2119025 to 2770297 tid 2 2024-09-10T16:21:08.355 INFO:tasks.rados.rados.0.smithi165.stdout:2211: snap_remove snap 228 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2211: done (3 left) 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2212: read oid 19 snap -1 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2212: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2213: read oid 25 snap -1 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2213: expect deleted 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2214: delete oid 29 current snap is 241 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2208: finishing copy_from to smithi16534841-33 2024-09-10T16:21:09.338 INFO:tasks.rados.rados.0.smithi165.stdout:2208: got expected ENOENT (src dne) 2024-09-10T16:21:09.339 INFO:tasks.rados.rados.0.smithi165.stdout:2208: finishing copy_from racing read to smithi16534841-33 2024-09-10T16:21:09.339 INFO:tasks.rados.rados.0.smithi165.stdout:2207: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:21:09.339 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:09.408 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:09.438 INFO:tasks.rados.rados.0.smithi165.stdout:2210: finishing write tid 1 to smithi16534841-20 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: finishing write tid 2 to smithi16534841-20 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: finishing write tid 3 to smithi16534841-20 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: finishing write tid 4 to smithi16534841-20 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: oid 20 updating version 0 to 1256 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: oid 20 updating version 1256 to 1257 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: oid 20 version 1257 is already newer than 1251 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1257 (ObjNum 781 snap 241 seq_num 781) dirty exists 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: left oid 20 (ObjNum 781 snap 241 seq_num 781) 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2207: done (5 left) 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2208: done (4 left) 2024-09-10T16:21:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2210: done (3 left) 2024-09-10T16:21:09.440 INFO:tasks.rados.rados.0.smithi165.stdout:2214: done (2 left) 2024-09-10T16:21:09.440 INFO:tasks.rados.rados.0.smithi165.stdout:2215: read oid 18 snap -1 2024-09-10T16:21:09.440 INFO:tasks.rados.rados.0.smithi165.stdout:2215: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:21:09.440 INFO:tasks.rados.rados.0.smithi165.stdout:2216: delete oid 43 current snap is 241 2024-09-10T16:21:09.440 INFO:tasks.rados.rados.0.smithi165.stdout:2212: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:09.561 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1877292 2024-09-10T16:21:09.571 INFO:tasks.rados.rados.0.smithi165.stdout:2215: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:21:09.607 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1472625 2024-09-10T16:21:09.635 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1386158 2024-09-10T16:21:09.653 INFO:tasks.rados.rados.0.smithi165.stdout:2212: done (3 left) 2024-09-10T16:21:09.653 INFO:tasks.rados.rados.0.smithi165.stdout:2213: done (2 left) 2024-09-10T16:21:09.653 INFO:tasks.rados.rados.0.smithi165.stdout:2215: done (1 left) 2024-09-10T16:21:09.654 INFO:tasks.rados.rados.0.smithi165.stdout:2216: done (0 left) 2024-09-10T16:21:09.654 INFO:tasks.rados.rados.0.smithi165.stdout:2217: snap_remove snap 239 2024-09-10T16:21:10.345 INFO:tasks.rados.rados.0.smithi165.stdout:2217: done (0 left) 2024-09-10T16:21:10.345 INFO:tasks.rados.rados.0.smithi165.stdout:2218: delete oid 50 current snap is 241 2024-09-10T16:21:10.348 INFO:tasks.rados.rados.0.smithi165.stdout:2218: done (0 left) 2024-09-10T16:21:10.348 INFO:tasks.rados.rados.0.smithi165.stdout:2219: write (excl) oid 20 current snap is 241 2024-09-10T16:21:10.348 INFO:tasks.rados.rados.0.smithi165.stdout:2219: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2024-09-10T16:21:10.356 INFO:tasks.rados.rados.0.smithi165.stdout:2219: writing smithi16534841-20 from 793495 to 1448319 tid 1 2024-09-10T16:21:10.363 INFO:tasks.rados.rados.0.smithi165.stdout:2219: writing smithi16534841-20 from 1952953 to 2611563 tid 2 2024-09-10T16:21:10.366 INFO:tasks.rados.rados.0.smithi165.stdout:2219: writing smithi16534841-20 from 3157167 to 3193495 tid 3 2024-09-10T16:21:10.367 INFO:tasks.rados.rados.0.smithi165.stdout:2220: snap_create 2024-09-10T16:21:10.367 INFO:tasks.rados.rados.0.smithi165.stdout:2219: finishing write tid 1 to smithi16534841-20 2024-09-10T16:21:10.371 INFO:tasks.rados.rados.0.smithi165.stdout:2219: finishing write tid 2 to smithi16534841-20 2024-09-10T16:21:10.372 INFO:tasks.rados.rados.0.smithi165.stdout:2219: finishing write tid 3 to smithi16534841-20 2024-09-10T16:21:10.373 INFO:tasks.rados.rados.0.smithi165.stdout:2219: finishing write tid 4 to smithi16534841-20 2024-09-10T16:21:10.373 INFO:tasks.rados.rados.0.smithi165.stdout:2219: finishing write tid 5 to smithi16534841-20 2024-09-10T16:21:10.373 INFO:tasks.rados.rados.0.smithi165.stdout:2219: oid 20 updating version 0 to 1271 2024-09-10T16:21:10.373 INFO:tasks.rados.rados.0.smithi165.stdout:2219: oid 20 updating version 1271 to 1272 2024-09-10T16:21:10.373 INFO:tasks.rados.rados.0.smithi165.stdout:2219: oid 20 updating version 1272 to 1273 2024-09-10T16:21:10.374 INFO:tasks.rados.rados.0.smithi165.stdout:2219: oid 20 version 1273 is already newer than 1268 2024-09-10T16:21:10.374 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1273 (ObjNum 785 snap 241 seq_num 785) dirty exists 2024-09-10T16:21:10.374 INFO:tasks.rados.rados.0.smithi165.stdout:2219: left oid 20 (ObjNum 785 snap 241 seq_num 785) 2024-09-10T16:21:11.350 INFO:tasks.rados.rados.0.smithi165.stdout:2219: done (1 left) 2024-09-10T16:21:11.350 INFO:tasks.rados.rados.0.smithi165.stdout:2220: done (0 left) 2024-09-10T16:21:11.350 INFO:tasks.rados.rados.0.smithi165.stdout:2221: write (excl) oid 39 current snap is 242 2024-09-10T16:21:11.351 INFO:tasks.rados.rados.0.smithi165.stdout:2221: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2024-09-10T16:21:11.356 INFO:tasks.rados.rados.0.smithi165.stdout:2221: writing smithi16534841-39 from 410302 to 1140375 tid 1 2024-09-10T16:21:11.363 INFO:tasks.rados.rados.0.smithi165.stdout:2221: writing smithi16534841-39 from 1895082 to 2297350 tid 2 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2221: writing smithi16534841-39 from 2951884 to 3210302 tid 3 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2222: read oid 41 snap -1 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2222: expect deleted 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2223: write oid 37 current snap is 242 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2221: finishing write tid 1 to smithi16534841-39 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2221: finishing write tid 2 to smithi16534841-39 2024-09-10T16:21:11.369 INFO:tasks.rados.rados.0.smithi165.stdout:2223: seq_num 787 ranges {427109=405322,1553564=545926,2579307=647802} 2024-09-10T16:21:11.372 INFO:tasks.rados.rados.0.smithi165.stdout:2223: writing smithi16534841-37 from 427109 to 832431 tid 1 2024-09-10T16:21:11.380 INFO:tasks.rados.rados.0.smithi165.stdout:2223: writing smithi16534841-37 from 1553564 to 2099490 tid 2 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2223: writing smithi16534841-37 from 2579307 to 3227109 tid 3 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2224: rollback oid 7 current snap is 242 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 7 to 241 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: finishing write tid 3 to smithi16534841-39 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: finishing write tid 4 to smithi16534841-39 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: finishing write tid 5 to smithi16534841-39 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: oid 39 updating version 0 to 1170 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: oid 39 updating version 1170 to 1171 2024-09-10T16:21:11.387 INFO:tasks.rados.rados.0.smithi165.stdout:2221: oid 39 version 1171 is already newer than 1169 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2221: oid 39 version 1171 is already newer than 1168 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1171 (ObjNum 786 snap 242 seq_num 786) dirty exists 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2221: left oid 39 (ObjNum 786 snap 242 seq_num 786) 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2223: finishing write tid 1 to smithi16534841-37 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2223: finishing write tid 2 to smithi16534841-37 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2221: done (3 left) 2024-09-10T16:21:11.388 INFO:tasks.rados.rados.0.smithi165.stdout:2222: done (2 left) 2024-09-10T16:21:11.389 INFO:tasks.rados.rados.0.smithi165.stdout:2225: read oid 26 snap -1 2024-09-10T16:21:11.389 INFO:tasks.rados.rados.0.smithi165.stdout:2225: expect (ObjNum 768 snap 235 seq_num 768) 2024-09-10T16:21:11.389 INFO:tasks.rados.rados.0.smithi165.stdout:2226: delete oid 8 current snap is 242 2024-09-10T16:21:11.389 INFO:tasks.rados.rados.0.smithi165.stdout:2223: finishing write tid 3 to smithi16534841-37 2024-09-10T16:21:11.391 INFO:tasks.rados.rados.0.smithi165.stdout:2223: finishing write tid 4 to smithi16534841-37 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: finishing write tid 5 to smithi16534841-37 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: oid 37 updating version 0 to 1220 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: oid 37 updating version 1220 to 1222 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: oid 37 version 1222 is already newer than 1221 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: oid 37 updating version 1222 to 1223 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1223 (ObjNum 787 snap 242 seq_num 787) dirty exists 2024-09-10T16:21:11.392 INFO:tasks.rados.rados.0.smithi165.stdout:2223: left oid 37 (ObjNum 787 snap 242 seq_num 787) 2024-09-10T16:21:11.393 INFO:tasks.rados.rados.0.smithi165.stdout:2223: done (3 left) 2024-09-10T16:21:11.393 INFO:tasks.rados.rados.0.smithi165.stdout:2226: done (2 left) 2024-09-10T16:21:11.393 INFO:tasks.rados.rados.0.smithi165.stdout:2227: snap_create 2024-09-10T16:21:11.399 INFO:tasks.rados.rados.0.smithi165.stdout:2224: finishing rollback tid 0 to smithi16534841-7 2024-09-10T16:21:11.399 INFO:tasks.rados.rados.0.smithi165.stdout:2224: finishing rollback tid 1 to smithi16534841-7 2024-09-10T16:21:11.400 INFO:tasks.rados.rados.0.smithi165.stdout:2224: finishing rollback tid 2 to smithi16534841-7 2024-09-10T16:21:11.400 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1408 (ObjNum 743 snap 225 seq_num 743) dirty exists 2024-09-10T16:21:11.402 INFO:tasks.rados.rados.0.smithi165.stdout:2225: expect (ObjNum 768 snap 235 seq_num 768) 2024-09-10T16:21:11.402 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:11.472 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2906342 2024-09-10T16:21:12.360 INFO:tasks.rados.rados.0.smithi165.stdout:2224: done (2 left) 2024-09-10T16:21:12.360 INFO:tasks.rados.rados.0.smithi165.stdout:2225: done (1 left) 2024-09-10T16:21:12.360 INFO:tasks.rados.rados.0.smithi165.stdout:2227: done (0 left) 2024-09-10T16:21:12.360 INFO:tasks.rados.rados.0.smithi165.stdout:2228: read oid 18 snap -1 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2228: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2229: read oid 21 snap -1 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2229: expect (ObjNum 764 snap 234 seq_num 764) 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2230: read oid 20 snap -1 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2230: expect (ObjNum 785 snap 241 seq_num 785) 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2231: rollback oid 11 current snap is 243 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 237 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2232: write (excl) oid 34 current snap is 243 2024-09-10T16:21:12.361 INFO:tasks.rados.rados.0.smithi165.stdout:2232: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2024-09-10T16:21:12.365 INFO:tasks.rados.rados.0.smithi165.stdout:2232: writing smithi16534841-34 from 460723 to 1016543 tid 1 2024-09-10T16:21:12.373 INFO:tasks.rados.rados.0.smithi165.stdout:2232: writing smithi16534841-34 from 1670528 to 2220123 tid 2 2024-09-10T16:21:12.379 INFO:tasks.rados.rados.0.smithi165.stdout:2232: writing smithi16534841-34 from 2634153 to 3260723 tid 3 2024-09-10T16:21:12.379 INFO:tasks.rados.rados.0.smithi165.stdout:2233: read oid 12 snap -1 2024-09-10T16:21:12.379 INFO:tasks.rados.rados.0.smithi165.stdout:2233: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:12.379 INFO:tasks.rados.rados.0.smithi165.stdout:2234: copy_from oid 36 from oid 45 current snap is 243 2024-09-10T16:21:12.380 INFO:tasks.rados.rados.0.smithi165.stdout:2228: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:21:12.380 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:12.416 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:12.459 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 704980 2024-09-10T16:21:12.485 INFO:tasks.rados.rados.0.smithi165.stdout:2231: finishing rollback tid 0 to smithi16534841-11 2024-09-10T16:21:12.486 INFO:tasks.rados.rados.0.smithi165.stdout:2231: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:21:12.486 INFO:tasks.rados.rados.0.smithi165.stdout:2231: finishing rollback tid 2 to smithi16534841-11 2024-09-10T16:21:12.486 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1252 (ObjNum 756 snap 230 seq_num 756) dirty exists 2024-09-10T16:21:12.486 INFO:tasks.rados.rados.0.smithi165.stdout:2229: expect (ObjNum 764 snap 234 seq_num 764) 2024-09-10T16:21:12.487 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:12.580 INFO:tasks.rados.rados.0.smithi165.stdout:2232: finishing write tid 1 to smithi16534841-34 2024-09-10T16:21:12.580 INFO:tasks.rados.rados.0.smithi165.stdout:2230: expect (ObjNum 785 snap 241 seq_num 785) 2024-09-10T16:21:12.668 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:12.843 INFO:tasks.rados.rados.0.smithi165.stdout:2232: finishing write tid 2 to smithi16534841-34 2024-09-10T16:21:12.844 INFO:tasks.rados.rados.0.smithi165.stdout:2234: finishing copy_from to smithi16534841-36 2024-09-10T16:21:12.844 INFO:tasks.rados.rados.0.smithi165.stdout:2234: got expected ENOENT (src dne) 2024-09-10T16:21:12.844 INFO:tasks.rados.rados.0.smithi165.stdout:2234: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:21:12.844 INFO:tasks.rados.rados.0.smithi165.stdout:2233: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: finishing write tid 3 to smithi16534841-34 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: finishing write tid 4 to smithi16534841-34 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: finishing write tid 5 to smithi16534841-34 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: oid 34 updating version 0 to 1224 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: oid 34 updating version 1224 to 1227 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: oid 34 version 1227 is already newer than 1226 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: oid 34 version 1227 is already newer than 1225 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1227 (ObjNum 789 snap 243 seq_num 789) dirty exists 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2232: left oid 34 (ObjNum 789 snap 243 seq_num 789) 2024-09-10T16:21:12.988 INFO:tasks.rados.rados.0.smithi165.stdout:2228: done (6 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2229: done (5 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2230: done (4 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2231: done (3 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2232: done (2 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2233: done (1 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2234: done (0 left) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2235: read oid 34 snap -1 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2235: expect (ObjNum 789 snap 243 seq_num 789) 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2236: write (excl) oid 35 current snap is 243 2024-09-10T16:21:12.989 INFO:tasks.rados.rados.0.smithi165.stdout:2236: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2024-09-10T16:21:12.993 INFO:tasks.rados.rados.0.smithi165.stdout:2236: writing smithi16534841-35 from 477530 to 1108599 tid 1 2024-09-10T16:21:12.998 INFO:tasks.rados.rados.0.smithi165.stdout:2236: writing smithi16534841-35 from 1612657 to 2022263 tid 2 2024-09-10T16:21:13.004 INFO:tasks.rados.rados.0.smithi165.stdout:2236: writing smithi16534841-35 from 2545223 to 3062819 tid 3 2024-09-10T16:21:13.004 INFO:tasks.rados.rados.0.smithi165.stdout:2237: write oid 13 current snap is 243 2024-09-10T16:21:13.005 INFO:tasks.rados.rados.0.smithi165.stdout:2237: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2024-09-10T16:21:13.009 INFO:tasks.rados.rados.0.smithi165.stdout:2237: writing smithi16534841-13 from 494337 to 917008 tid 1 2024-09-10T16:21:13.014 INFO:tasks.rados.rados.0.smithi165.stdout:2237: writing smithi16534841-13 from 1387492 to 1940756 tid 2 2024-09-10T16:21:13.021 INFO:tasks.rados.rados.0.smithi165.stdout:2237: writing smithi16534841-13 from 2688999 to 3134220 tid 3 2024-09-10T16:21:13.021 INFO:tasks.rados.rados.0.smithi165.stdout:2238: read oid 33 snap -1 2024-09-10T16:21:13.022 INFO:tasks.rados.rados.0.smithi165.stdout:2238: expect (ObjNum 779 snap 238 seq_num 779) 2024-09-10T16:21:13.022 INFO:tasks.rados.rados.0.smithi165.stdout:2239: read oid 38 snap -1 2024-09-10T16:21:13.022 INFO:tasks.rados.rados.0.smithi165.stdout:2239: expect deleted 2024-09-10T16:21:13.022 INFO:tasks.rados.rados.0.smithi165.stdout:2240: snap_create 2024-09-10T16:21:13.022 INFO:tasks.rados.rados.0.smithi165.stdout:2235: expect (ObjNum 789 snap 243 seq_num 789) 2024-09-10T16:21:13.078 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:13.185 INFO:tasks.rados.rados.0.smithi165.stdout:2236: finishing write tid 1 to smithi16534841-35 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: finishing write tid 2 to smithi16534841-35 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: finishing write tid 3 to smithi16534841-35 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: finishing write tid 4 to smithi16534841-35 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: finishing write tid 5 to smithi16534841-35 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: oid 35 updating version 0 to 1346 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: oid 35 updating version 1346 to 1347 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: oid 35 version 1347 is already newer than 1345 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: oid 35 updating version 1347 to 1348 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1348 (ObjNum 790 snap 243 seq_num 790) dirty exists 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2236: left oid 35 (ObjNum 790 snap 243 seq_num 790) 2024-09-10T16:21:13.186 INFO:tasks.rados.rados.0.smithi165.stdout:2237: finishing write tid 1 to smithi16534841-13 2024-09-10T16:21:13.187 INFO:tasks.rados.rados.0.smithi165.stdout:2237: finishing write tid 2 to smithi16534841-13 2024-09-10T16:21:13.187 INFO:tasks.rados.rados.0.smithi165.stdout:2237: finishing write tid 3 to smithi16534841-13 2024-09-10T16:21:13.187 INFO:tasks.rados.rados.0.smithi165.stdout:2238: expect (ObjNum 779 snap 238 seq_num 779) 2024-09-10T16:21:13.248 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2084431 2024-09-10T16:21:13.265 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: finishing write tid 4 to smithi16534841-13 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: finishing write tid 5 to smithi16534841-13 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: oid 13 updating version 0 to 1352 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: oid 13 version 1352 is already newer than 1350 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: oid 13 version 1352 is already newer than 1351 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: oid 13 version 1352 is already newer than 1349 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1352 (ObjNum 791 snap 243 seq_num 791) dirty exists 2024-09-10T16:21:13.311 INFO:tasks.rados.rados.0.smithi165.stdout:2237: left oid 13 (ObjNum 791 snap 243 seq_num 791) 2024-09-10T16:21:13.366 INFO:tasks.rados.rados.0.smithi165.stdout:2235: done (5 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2236: done (4 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2237: done (3 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2238: done (2 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2239: done (1 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2240: done (0 left) 2024-09-10T16:21:13.367 INFO:tasks.rados.rados.0.smithi165.stdout:2241: delete oid 3 current snap is 244 2024-09-10T16:21:13.371 INFO:tasks.rados.rados.0.smithi165.stdout:2241: done (0 left) 2024-09-10T16:21:13.372 INFO:tasks.rados.rados.0.smithi165.stdout:2242: read oid 32 snap 230 2024-09-10T16:21:13.372 INFO:tasks.rados.rados.0.smithi165.stdout:2242: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:21:13.372 INFO:tasks.rados.rados.0.smithi165.stdout:2243: delete oid 20 current snap is 244 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2243: done (1 left) 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2244: read oid 10 snap -1 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2244: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2245: copy_from oid 29 from oid 23 current snap is 244 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2246: rollback oid 16 current snap is 244 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 242 2024-09-10T16:21:13.375 INFO:tasks.rados.rados.0.smithi165.stdout:2247: snap_create 2024-09-10T16:21:13.383 INFO:tasks.rados.rados.0.smithi165.stdout:2246: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:21:13.383 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1352 (ObjNum 33 snap 0 seq_num 1127806872) dirty dne 2024-09-10T16:21:13.384 INFO:tasks.rados.rados.0.smithi165.stdout:2244: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:21:13.448 INFO:tasks.rados.rados.0.smithi165.stdout:2245: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:21:13.448 INFO:tasks.rados.rados.0.smithi165.stdout:2242: expect (ObjNum 660 snap 207 seq_num 660) 2024-09-10T16:21:13.448 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 606326 2024-09-10T16:21:13.481 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1278377 2024-09-10T16:21:13.505 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:13.540 INFO:tasks.rados.rados.0.smithi165.stdout:2245: finishing copy_from to smithi16534841-29 2024-09-10T16:21:13.540 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1155 (ObjNum 668 snap 207 seq_num 668) dirty exists 2024-09-10T16:21:14.318 INFO:tasks.rados.rados.0.smithi165.stdout:2242: done (4 left) 2024-09-10T16:21:14.318 INFO:tasks.rados.rados.0.smithi165.stdout:2244: done (3 left) 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:2245: done (2 left) 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:2246: done (1 left) 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:2247: done (0 left) 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:2248: rollback oid 41 current snap is 245 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 242 2024-09-10T16:21:14.319 INFO:tasks.rados.rados.0.smithi165.stdout:2249: delete oid 2 current snap is 245 2024-09-10T16:21:14.325 INFO:tasks.rados.rados.0.smithi165.stdout:2248: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:21:14.325 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1152 (ObjNum 0 snap 0 seq_num 304) dirty dne 2024-09-10T16:21:14.325 INFO:tasks.rados.rados.0.smithi165.stdout:2248: done (1 left) 2024-09-10T16:21:14.325 INFO:tasks.rados.rados.0.smithi165.stdout:2249: done (0 left) 2024-09-10T16:21:14.325 INFO:tasks.rados.rados.0.smithi165.stdout:2250: copy_from oid 25 from oid 35 current snap is 245 2024-09-10T16:21:14.326 INFO:tasks.rados.rados.0.smithi165.stdout:2251: write oid 3 current snap is 245 2024-09-10T16:21:14.326 INFO:tasks.rados.rados.0.smithi165.stdout:2251: seq_num 795 ranges {561565=723667,1905067=560602,3082338=279227} 2024-09-10T16:21:14.331 INFO:tasks.rados.rados.0.smithi165.stdout:2251: writing smithi16534841-3 from 561565 to 1285232 tid 1 2024-09-10T16:21:14.338 INFO:tasks.rados.rados.0.smithi165.stdout:2251: writing smithi16534841-3 from 1905067 to 2465669 tid 2 2024-09-10T16:21:14.344 INFO:tasks.rados.rados.0.smithi165.stdout:2251: writing smithi16534841-3 from 3082338 to 3361565 tid 3 2024-09-10T16:21:14.344 INFO:tasks.rados.rados.0.smithi165.stdout:2252: rollback oid 15 current snap is 245 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 240 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2250: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2253: copy_from oid 14 from oid 2 current snap is 245 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2254: read oid 31 snap 236 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2254: expect (ObjNum 754 snap 228 seq_num 754) 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2255: rollback oid 47 current snap is 245 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 218 2024-09-10T16:21:14.345 INFO:tasks.rados.rados.0.smithi165.stdout:2256: snap_create 2024-09-10T16:21:14.348 INFO:tasks.rados.rados.0.smithi165.stdout:2251: finishing write tid 1 to smithi16534841-3 2024-09-10T16:21:14.349 INFO:tasks.rados.rados.0.smithi165.stdout:2253: finishing copy_from to smithi16534841-14 2024-09-10T16:21:14.349 INFO:tasks.rados.rados.0.smithi165.stdout:2253: got expected ENOENT (src dne) 2024-09-10T16:21:14.349 INFO:tasks.rados.rados.0.smithi165.stdout:2253: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:21:14.351 INFO:tasks.rados.rados.0.smithi165.stdout:2255: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:21:14.351 INFO:tasks.rados.rados.0.smithi165.stdout:2250: finishing copy_from to smithi16534841-25 2024-09-10T16:21:14.351 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1348 (ObjNum 790 snap 243 seq_num 790) dirty exists 2024-09-10T16:21:14.352 INFO:tasks.rados.rados.0.smithi165.stdout:2254: expect (ObjNum 754 snap 228 seq_num 754) 2024-09-10T16:21:14.398 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 598187 2024-09-10T16:21:14.491 INFO:tasks.rados.rados.0.smithi165.stdout:2255: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:21:14.491 INFO:tasks.rados.rados.0.smithi165.stdout:2255: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:21:14.491 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1342 (ObjNum 633 snap 198 seq_num 633) dirty exists 2024-09-10T16:21:14.491 INFO:tasks.rados.rados.0.smithi165.stdout:2251: finishing write tid 2 to smithi16534841-3 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:2252: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:2252: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:2251: finishing write tid 3 to smithi16534841-3 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:2252: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1312 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:21:14.492 INFO:tasks.rados.rados.0.smithi165.stdout:2251: finishing write tid 4 to smithi16534841-3 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: finishing write tid 5 to smithi16534841-3 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: oid 3 updating version 0 to 1278 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: oid 3 updating version 1278 to 1351 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: oid 3 version 1351 is already newer than 1349 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: oid 3 version 1351 is already newer than 1350 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1351 (ObjNum 795 snap 245 seq_num 795) dirty exists 2024-09-10T16:21:14.493 INFO:tasks.rados.rados.0.smithi165.stdout:2251: left oid 3 (ObjNum 795 snap 245 seq_num 795) 2024-09-10T16:21:15.336 INFO:tasks.rados.rados.0.smithi165.stdout:2250: done (6 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2251: done (5 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2252: done (4 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2253: done (3 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2254: done (2 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2255: done (1 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2256: done (0 left) 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2257: read oid 20 snap -1 2024-09-10T16:21:15.337 INFO:tasks.rados.rados.0.smithi165.stdout:2257: expect deleted 2024-09-10T16:21:15.338 INFO:tasks.rados.rados.0.smithi165.stdout:2258: copy_from oid 34 from oid 24 current snap is 246 2024-09-10T16:21:15.338 INFO:tasks.rados.rados.0.smithi165.stdout:2259: write (excl) oid 25 current snap is 246 2024-09-10T16:21:15.338 INFO:tasks.rados.rados.0.smithi165.stdout:2259: seq_num 796 ranges {578372=798916,1963549=420613,2826114=417934} 2024-09-10T16:21:15.340 INFO:tasks.rados.rados.0.smithi165.stdout:2259: writing smithi16534841-25 from 578372 to 1377288 tid 1 2024-09-10T16:21:15.341 INFO:tasks.rados.rados.0.smithi165.stdout:2259: writing smithi16534841-25 from 1963549 to 2384162 tid 2 2024-09-10T16:21:15.346 INFO:tasks.rados.rados.0.smithi165.stdout:2259: writing smithi16534841-25 from 2826114 to 3244048 tid 3 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2260: copy_from oid 26 from oid 41 current snap is 246 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2259: finishing write tid 1 to smithi16534841-25 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2259: finishing write tid 2 to smithi16534841-25 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2257: done (3 left) 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2261: rollback oid 12 current snap is 246 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 230 2024-09-10T16:21:15.347 INFO:tasks.rados.rados.0.smithi165.stdout:2262: snap_remove snap 243 2024-09-10T16:21:16.329 INFO:tasks.rados.rados.0.smithi165.stdout:2262: done (4 left) 2024-09-10T16:21:16.329 INFO:tasks.rados.rados.0.smithi165.stdout:2263: read oid 42 snap 241 2024-09-10T16:21:16.329 INFO:tasks.rados.rados.0.smithi165.stdout:2263: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:21:16.329 INFO:tasks.rados.rados.0.smithi165.stdout:2264: write (excl) oid 9 current snap is 246 2024-09-10T16:21:16.329 INFO:tasks.rados.rados.0.smithi165.stdout:2264: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2024-09-10T16:21:16.334 INFO:tasks.rados.rados.0.smithi165.stdout:2264: writing smithi16534841-9 from 595179 to 1069344 tid 1 2024-09-10T16:21:16.341 INFO:tasks.rados.rados.0.smithi165.stdout:2264: writing smithi16534841-9 from 1622031 to 2186302 tid 2 2024-09-10T16:21:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:2264: writing smithi16534841-9 from 2737184 to 3366390 tid 3 2024-09-10T16:21:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:2265: write (excl) oid 45 current snap is 246 2024-09-10T16:21:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:2260: finishing copy_from to smithi16534841-26 2024-09-10T16:21:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:2260: got expected ENOENT (src dne) 2024-09-10T16:21:16.348 INFO:tasks.rados.rados.0.smithi165.stdout:2265: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2024-09-10T16:21:16.354 INFO:tasks.rados.rados.0.smithi165.stdout:2265: writing smithi16534841-45 from 611986 to 1161400 tid 1 2024-09-10T16:21:16.358 INFO:tasks.rados.rados.0.smithi165.stdout:2265: writing smithi16534841-45 from 1564160 to 1988442 tid 2 2024-09-10T16:21:16.366 INFO:tasks.rados.rados.0.smithi165.stdout:2265: writing smithi16534841-45 from 2764607 to 3205085 tid 3 2024-09-10T16:21:16.366 INFO:tasks.rados.rados.0.smithi165.stdout:2266: write (excl) oid 8 current snap is 246 2024-09-10T16:21:16.366 INFO:tasks.rados.rados.0.smithi165.stdout:2258: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:21:16.366 INFO:tasks.rados.rados.0.smithi165.stdout:2260: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:21:16.366 INFO:tasks.rados.rados.0.smithi165.stdout:2259: finishing write tid 3 to smithi16534841-25 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: finishing write tid 4 to smithi16534841-25 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: finishing write tid 5 to smithi16534841-25 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: oid 25 updating version 0 to 1353 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: oid 25 updating version 1353 to 1355 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: oid 25 version 1355 is already newer than 1352 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: oid 25 version 1355 is already newer than 1354 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1355 (ObjNum 796 snap 246 seq_num 796) dirty exists 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2259: left oid 25 (ObjNum 796 snap 246 seq_num 796) 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2258: finishing copy_from to smithi16534841-34 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1333 (ObjNum 752 snap 227 seq_num 752) dirty exists 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2261: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2261: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:21:16.367 INFO:tasks.rados.rados.0.smithi165.stdout:2261: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:21:16.368 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1326 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:21:16.368 INFO:tasks.rados.rados.0.smithi165.stdout:2263: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:21:16.397 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1016034 2024-09-10T16:21:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1555468 2024-09-10T16:21:16.438 INFO:tasks.rados.rados.0.smithi165.stdout:2264: finishing write tid 1 to smithi16534841-9 2024-09-10T16:21:16.438 INFO:tasks.rados.rados.0.smithi165.stdout:2264: finishing write tid 2 to smithi16534841-9 2024-09-10T16:21:16.438 INFO:tasks.rados.rados.0.smithi165.stdout:2266: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2024-09-10T16:21:16.445 INFO:tasks.rados.rados.0.smithi165.stdout:2266: writing smithi16534841-8 from 628793 to 1369809 tid 1 2024-09-10T16:21:16.452 INFO:tasks.rados.rados.0.smithi165.stdout:2266: writing smithi16534841-8 from 2138995 to 2706935 tid 2 2024-09-10T16:21:16.456 INFO:tasks.rados.rados.0.smithi165.stdout:2266: writing smithi16534841-8 from 3192030 to 3428793 tid 3 2024-09-10T16:21:16.456 INFO:tasks.rados.rados.0.smithi165.stdout:2258: done (7 left) 2024-09-10T16:21:16.456 INFO:tasks.rados.rados.0.smithi165.stdout:2259: done (6 left) 2024-09-10T16:21:16.456 INFO:tasks.rados.rados.0.smithi165.stdout:2260: done (5 left) 2024-09-10T16:21:16.457 INFO:tasks.rados.rados.0.smithi165.stdout:2261: done (4 left) 2024-09-10T16:21:16.457 INFO:tasks.rados.rados.0.smithi165.stdout:2263: done (3 left) 2024-09-10T16:21:16.457 INFO:tasks.rados.rados.0.smithi165.stdout:2267: snap_remove snap 231 2024-09-10T16:21:18.037 INFO:tasks.rados.rados.0.smithi165.stdout:2267: done (3 left) 2024-09-10T16:21:18.037 INFO:tasks.rados.rados.0.smithi165.stdout:2268: snap_create 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: finishing write tid 3 to smithi16534841-9 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: finishing write tid 4 to smithi16534841-9 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: finishing write tid 5 to smithi16534841-9 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: oid 9 updating version 0 to 1186 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: oid 9 version 1186 is already newer than 1185 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: oid 9 version 1186 is already newer than 1183 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: oid 9 version 1186 is already newer than 1184 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1186 (ObjNum 797 snap 246 seq_num 797) dirty exists 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2264: left oid 9 (ObjNum 797 snap 246 seq_num 797) 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2265: finishing write tid 1 to smithi16534841-45 2024-09-10T16:21:18.038 INFO:tasks.rados.rados.0.smithi165.stdout:2265: finishing write tid 2 to smithi16534841-45 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: finishing write tid 3 to smithi16534841-45 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: finishing write tid 4 to smithi16534841-45 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: finishing write tid 5 to smithi16534841-45 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: oid 45 updating version 0 to 1297 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: oid 45 updating version 1297 to 1298 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: oid 45 version 1298 is already newer than 1296 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: oid 45 updating version 1298 to 1299 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1299 (ObjNum 798 snap 246 seq_num 798) dirty exists 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2265: left oid 45 (ObjNum 798 snap 246 seq_num 798) 2024-09-10T16:21:18.039 INFO:tasks.rados.rados.0.smithi165.stdout:2266: finishing write tid 1 to smithi16534841-8 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: finishing write tid 2 to smithi16534841-8 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: finishing write tid 3 to smithi16534841-8 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: finishing write tid 4 to smithi16534841-8 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: finishing write tid 5 to smithi16534841-8 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: oid 8 updating version 0 to 1343 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: oid 8 updating version 1343 to 1345 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: oid 8 updating version 1345 to 1346 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: oid 8 version 1346 is already newer than 1344 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1346 (ObjNum 799 snap 246 seq_num 799) dirty exists 2024-09-10T16:21:18.040 INFO:tasks.rados.rados.0.smithi165.stdout:2266: left oid 8 (ObjNum 799 snap 246 seq_num 799) 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:2264: done (3 left) 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:2265: done (2 left) 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:2266: done (1 left) 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:2268: done (0 left) 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:2269: rollback oid 35 current snap is 247 2024-09-10T16:21:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 241 2024-09-10T16:21:19.042 INFO:tasks.rados.rados.0.smithi165.stdout:2270: snap_create 2024-09-10T16:21:19.047 INFO:tasks.rados.rados.0.smithi165.stdout:2269: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:21:19.048 INFO:tasks.rados.rados.0.smithi165.stdout:2269: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:21:19.051 INFO:tasks.rados.rados.0.smithi165.stdout:2269: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:21:19.051 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1355 (ObjNum 714 snap 220 seq_num 714) dirty exists 2024-09-10T16:21:19.319 INFO:tasks.rados.rados.0.smithi165.stdout:2269: done (1 left) 2024-09-10T16:21:19.319 INFO:tasks.rados.rados.0.smithi165.stdout:2270: done (0 left) 2024-09-10T16:21:19.319 INFO:tasks.rados.rados.0.smithi165.stdout:2271: copy_from oid 1 from oid 28 current snap is 248 2024-09-10T16:21:19.319 INFO:tasks.rados.rados.0.smithi165.stdout:2272: copy_from oid 10 from oid 38 current snap is 248 2024-09-10T16:21:19.319 INFO:tasks.rados.rados.0.smithi165.stdout:2273: snap_create 2024-09-10T16:21:19.326 INFO:tasks.rados.rados.0.smithi165.stdout:2272: finishing copy_from to smithi16534841-10 2024-09-10T16:21:19.326 INFO:tasks.rados.rados.0.smithi165.stdout:2272: got expected ENOENT (src dne) 2024-09-10T16:21:19.326 INFO:tasks.rados.rados.0.smithi165.stdout:2272: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:21:19.337 INFO:tasks.rados.rados.0.smithi165.stdout:2271: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:21:19.339 INFO:tasks.rados.rados.0.smithi165.stdout:2271: finishing copy_from to smithi16534841-1 2024-09-10T16:21:19.339 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1097 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:21:20.329 INFO:tasks.rados.rados.0.smithi165.stdout:2271: done (2 left) 2024-09-10T16:21:20.329 INFO:tasks.rados.rados.0.smithi165.stdout:2272: done (1 left) 2024-09-10T16:21:20.329 INFO:tasks.rados.rados.0.smithi165.stdout:2273: done (0 left) 2024-09-10T16:21:20.329 INFO:tasks.rados.rados.0.smithi165.stdout:2274: delete oid 50 current snap is 249 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:2274: done (0 left) 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:2275: read oid 20 snap 242 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:2275: expect (ObjNum 785 snap 241 seq_num 785) 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:2276: rollback oid 3 current snap is 249 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 230 2024-09-10T16:21:20.333 INFO:tasks.rados.rados.0.smithi165.stdout:2277: snap_remove snap 246 2024-09-10T16:21:21.330 INFO:tasks.rados.rados.0.smithi165.stdout:2277: done (2 left) 2024-09-10T16:21:21.330 INFO:tasks.rados.rados.0.smithi165.stdout:2278: read oid 24 snap -1 2024-09-10T16:21:21.331 INFO:tasks.rados.rados.0.smithi165.stdout:2278: expect (ObjNum 752 snap 227 seq_num 752) 2024-09-10T16:21:21.331 INFO:tasks.rados.rados.0.smithi165.stdout:2279: snap_remove snap 244 2024-09-10T16:21:22.259 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:21:22.259 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:21:22.259 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:21:22.262 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:21:22.262 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:21:22.263 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:21:22.339 INFO:tasks.rados.rados.0.smithi165.stdout:2279: done (3 left) 2024-09-10T16:21:22.340 INFO:tasks.rados.rados.0.smithi165.stdout:2280: snap_create 2024-09-10T16:21:22.340 INFO:tasks.rados.rados.0.smithi165.stdout:2275: expect (ObjNum 785 snap 241 seq_num 785) 2024-09-10T16:21:22.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.373+0000 7f0ee88ef640 10 monclient: get_monmap_and_config 2024-09-10T16:21:22.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.373+0000 7f0ee88ef640 10 monclient: build_initial_monmap 2024-09-10T16:21:22.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.373+0000 7f0ee88ef640 10 monclient: monmap: 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:22.374201+0000 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:22.374201+0000 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:22.375 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.373+0000 7f0ee88ef640 1 Processor -- start 2024-09-10T16:21:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 -- start start 2024-09-10T16:21:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient: init 2024-09-10T16:21:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:22.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ee40a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ee40a4a80 0x7f0ee41383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee40a4a80 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee409ce50 con 0x7f0ee40a46a0 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee406db60 con 0x7f0ee40a4a80 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee40a4f10 con 0x7f0ee406cb70 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:22.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee88ef640 10 monclient(hunting): authenticate will time out at 1660.861782s 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee1d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0ee40a46a0 0x7f0ee4137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57662/0 (socket says 172.21.15.164:57662) 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee1d9b640 1 -- 172.21.15.164:0/4089351624 learned_addr learned my addr 172.21.15.164:0/4089351624 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 10 monclient(hunting): get_auth_request con 0x7f0ee406cb70 auth_method 0 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee2d9d640 1 --2- 172.21.15.164:0/4089351624 >> v2:172.21.15.165:3300/0 conn(0x7f0ee40a4a80 0x7f0ee41383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 1 -- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 msgr2=0x7f0ee4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee2d9d640 10 monclient(hunting): get_auth_request con 0x7f0ee40a4a80 auth_method 0 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.374+0000 7f0ee259c640 1 -- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 msgr2=0x7f0ee4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:22.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee259c640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee259c640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient(hunting): handle_auth_done global_id 154261 payload 274 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 msgr2=0x7f0ee4134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/4089351624 >> v1:172.21.15.164:6789/0 conn(0x7f0ee40a46a0 legacy=0x7f0ee4137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:22.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:21:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 1 monclient: found mon.noname-c 2024-09-10T16:21:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:21:22.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.375+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/4089351624 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee409cb70 con 0x7f0ee40a4a80 2024-09-10T16:21:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/4089351624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a4a80 msgr2=0x7f0ee41383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee2d9d640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a4a80 0x7f0ee41383b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009d90 tx=0x7f0edc02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35754b2af95f7590 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 1 -- 172.21.15.164:0/4089351624 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc007950 con 0x7f0ee40a4a80 2024-09-10T16:21:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:22.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: dump: 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: _finish_auth 0 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: _check_auth_tickets 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 1 -- 172.21.15.164:0/4089351624 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc007af0 con 0x7f0ee40a4a80 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 5 monclient: authenticate success, global_id 154261 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 1 -- 172.21.15.164:0/4089351624 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc031400 con 0x7f0ee40a4a80 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 4 set_mon_vals no callback set 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee159a640 10 monclient: dump: 2024-09-10T16:21:22.390 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 10 monclient: shutdown 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:22.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 -- 172.21.15.164:0/4089351624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a4a80 msgr2=0x7f0ee41383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a4a80 0x7f0ee41383b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009d90 tx=0x7f0edc02a490 comp rx=0 tx=0).stop 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 -- 172.21.15.164:0/4089351624 shutdown_connections 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a4a80 0x7f0ee41383b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/4089351624 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee406cb70 0x7f0ee4134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 -- 172.21.15.164:0/4089351624 >> 172.21.15.164:0/4089351624 conn(0x7f0ee40aa900 msgr2=0x7f0ee40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 -- 172.21.15.164:0/4089351624 shutdown_connections 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.378+0000 7f0ee88ef640 1 -- 172.21.15.164:0/4089351624 wait complete. 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.379+0000 7f0ee88ef640 10 monclient: build_initial_monmap 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.379+0000 7f0ee88ef640 10 monclient: monmap: 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:22.392 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.379+0000 7f0ee88ef640 1 Processor -- start 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 -- start start 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient: init 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-a con 0x7f0ee406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a46a0 0x7f0ee413db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-b con 0x7f0ee40a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): picked mon.noname-c con 0x7f0ee40a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee41388f0 con 0x7f0ee406cb70 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee406cf50 con 0x7f0ee40a46a0 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee4138a70 con 0x7f0ee40a4a80 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:22.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee88ef640 10 monclient(hunting): authenticate will time out at 1660.867491s 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58716/0 (socket says 172.21.15.164:58716) 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 -- 172.21.15.164:0/235458329 learned_addr learned my addr 172.21.15.164:0/235458329 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): get_auth_request con 0x7f0ee406cb70 auth_method 0 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 10 monclient(hunting): get_auth_request con 0x7f0ee40a4a80 auth_method 0 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:22.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a46a0 0x7f0ee413db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): get_auth_request con 0x7f0ee40a46a0 auth_method 0 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 msgr2=0x7f0ee413e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 msgr2=0x7f0ee413e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee2d9d640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:22.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient(hunting): handle_auth_done global_id 164184 payload 824 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 msgr2=0x7f0ee413e0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a46a0 msgr2=0x7f0ee413db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a46a0 0x7f0ee413db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient: _finish_hunting 0 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 monclient: found mon.noname-a 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.380+0000 7f0ee259c640 1 -- 172.21.15.164:0/235458329 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee409e960 con 0x7f0ee406cb70 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee259c640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 msgr2=0x7f0ee4131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:22.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee259c640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00d7d0 tx=0x7f0ecc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00b8d0 con 0x7f0ee406cb70 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: dump: 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:22.398 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 5 monclient: authenticate success, global_id 164184 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ecc019070 con 0x7f0ee406cb70 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ecc00bcd0 con 0x7f0ee406cb70 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0edaffd640 10 monclient: dump: 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:22.399 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 10 monclient: _renew_subs 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee0d99640 4 set_mon_vals no callback set 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:22.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee406e720 con 0x7f0ee406cb70 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 10 monclient: _renew_subs 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.383+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee409fbe0 con 0x7f0ee406cb70 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0ecc01e460 con 0x7f0ee406cb70 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee88ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee88ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee88ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee4134f70 con 0x7f0ee406cb70 2024-09-10T16:21:22.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0edaffd640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 0x7f0ec40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:22.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(520..520 src has 481..520) ==== 4799+0+0 (secure 0 0 0) 0x7f0ecc053200 con 0x7f0ee406cb70 2024-09-10T16:21:22.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee1d9b640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 0x7f0ec40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:22.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.384+0000 7f0ee1d9b640 10 monclient: get_auth_request con 0x7f0ec403e210 auth_method 0 2024-09-10T16:21:22.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.387+0000 7f0ee1d9b640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 0x7f0ec40406d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4007c90 tx=0x7f0ed4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:22.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.389+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0ee409e960 con 0x7f0ee406cb70 2024-09-10T16:21:22.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.389+0000 7f0edaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:22.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.389+0000 7f0edaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:22.504 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:22.593 INFO:tasks.rados.rados.0.smithi165.stdout:2276: finishing rollback tid 0 to smithi16534841-3 2024-09-10T16:21:22.593 INFO:tasks.rados.rados.0.smithi165.stdout:2276: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:21:22.593 INFO:tasks.rados.rados.0.smithi165.stdout:2276: finishing rollback tid 2 to smithi16534841-3 2024-09-10T16:21:22.593 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1358 (ObjNum 736 snap 223 seq_num 736) dirty exists 2024-09-10T16:21:22.594 INFO:tasks.rados.rados.0.smithi165.stdout:2278: expect (ObjNum 752 snap 227 seq_num 752) 2024-09-10T16:21:22.615 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.634+0000 7f0ee88ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.634+0000 7f0ee88ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.634+0000 7f0ee88ef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.634+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0ee413f630 con 0x7f0ee406cb70 2024-09-10T16:21:22.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.635+0000 7f0edaffd640 1 -- 172.21.15.164:0/235458329 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f0ecc051bf0 con 0x7f0ee406cb70 2024-09-10T16:21:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.635+0000 7f0edaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:22.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.635+0000 7f0edaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:22.637 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":148,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 msgr2=0x7f0ec40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 0x7f0ec40406d0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4007c90 tx=0x7f0ed4007be0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 10 monclient: shutdown 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 msgr2=0x7f0ee4131810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ecc00d7d0 tx=0x7f0ecc00dc90 comp rx=0 tx=0).stop 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 shutdown_connections 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0ec403e210 0x7f0ec40406d0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ee40a4a80 0x7f0ee413e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ee40a46a0 0x7f0ee413db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 --2- 172.21.15.164:0/235458329 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ee406cb70 0x7f0ee4131810 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 >> 172.21.15.164:0/235458329 conn(0x7f0ee40aa900 msgr2=0x7f0ee4137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 shutdown_connections 2024-09-10T16:21:22.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:22.637+0000 7f0ee88ef640 1 -- 172.21.15.164:0/235458329 wait complete. 2024-09-10T16:21:22.645 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:23.345 INFO:tasks.rados.rados.0.smithi165.stdout:2275: done (3 left) 2024-09-10T16:21:23.345 INFO:tasks.rados.rados.0.smithi165.stdout:2276: done (2 left) 2024-09-10T16:21:23.345 INFO:tasks.rados.rados.0.smithi165.stdout:2278: done (1 left) 2024-09-10T16:21:23.345 INFO:tasks.rados.rados.0.smithi165.stdout:2280: done (0 left) 2024-09-10T16:21:23.346 INFO:tasks.rados.rados.0.smithi165.stdout:2281: rollback oid 40 current snap is 250 2024-09-10T16:21:23.346 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 248 2024-09-10T16:21:23.346 INFO:tasks.rados.rados.0.smithi165.stdout:2282: snap_create 2024-09-10T16:21:23.353 INFO:tasks.rados.rados.0.smithi165.stdout:2281: finishing rollback tid 0 to smithi16534841-40 2024-09-10T16:21:23.356 INFO:tasks.rados.rados.0.smithi165.stdout:2281: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:21:23.356 INFO:tasks.rados.rados.0.smithi165.stdout:2281: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:21:23.356 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1189 (ObjNum 748 snap 227 seq_num 748) dirty exists 2024-09-10T16:21:24.321 INFO:tasks.rados.rados.0.smithi165.stdout:2281: done (1 left) 2024-09-10T16:21:24.322 INFO:tasks.rados.rados.0.smithi165.stdout:2282: done (0 left) 2024-09-10T16:21:24.322 INFO:tasks.rados.rados.0.smithi165.stdout:2283: delete oid 50 current snap is 251 2024-09-10T16:21:24.325 INFO:tasks.rados.rados.0.smithi165.stdout:2283: done (0 left) 2024-09-10T16:21:24.325 INFO:tasks.rados.rados.0.smithi165.stdout:2284: read oid 29 snap -1 2024-09-10T16:21:24.325 INFO:tasks.rados.rados.0.smithi165.stdout:2284: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:21:24.325 INFO:tasks.rados.rados.0.smithi165.stdout:2285: read oid 17 snap -1 2024-09-10T16:21:24.326 INFO:tasks.rados.rados.0.smithi165.stdout:2285: expect (ObjNum 673 snap 207 seq_num 673) 2024-09-10T16:21:24.326 INFO:tasks.rados.rados.0.smithi165.stdout:2286: write (excl) oid 45 current snap is 251 2024-09-10T16:21:24.326 INFO:tasks.rados.rados.0.smithi165.stdout:2286: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2024-09-10T16:21:24.332 INFO:tasks.rados.rados.0.smithi165.stdout:2286: writing smithi16534841-45 from 679214 to 1245977 tid 1 2024-09-10T16:21:24.339 INFO:tasks.rados.rados.0.smithi165.stdout:2286: writing smithi16534841-45 from 1798088 to 2513355 tid 2 2024-09-10T16:21:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:2286: writing smithi16534841-45 from 3157946 to 3479214 tid 3 2024-09-10T16:21:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:2287: delete oid 27 current snap is 251 2024-09-10T16:21:24.345 INFO:tasks.rados.rados.0.smithi165.stdout:2284: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:21:24.402 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 946571 2024-09-10T16:21:24.476 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2375983 2024-09-10T16:21:24.497 INFO:tasks.rados.rados.0.smithi165.stdout:2285: expect (ObjNum 673 snap 207 seq_num 673) 2024-09-10T16:21:24.595 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 407523 2024-09-10T16:21:24.783 INFO:tasks.rados.rados.0.smithi165.stdout:2286: finishing write tid 1 to smithi16534841-45 2024-09-10T16:21:24.783 INFO:tasks.rados.rados.0.smithi165.stdout:2286: finishing write tid 2 to smithi16534841-45 2024-09-10T16:21:24.783 INFO:tasks.rados.rados.0.smithi165.stdout:2286: finishing write tid 3 to smithi16534841-45 2024-09-10T16:21:24.783 INFO:tasks.rados.rados.0.smithi165.stdout:2286: finishing write tid 4 to smithi16534841-45 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: finishing write tid 5 to smithi16534841-45 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: oid 45 updating version 0 to 1301 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: oid 45 updating version 1301 to 1303 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: oid 45 version 1303 is already newer than 1302 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: oid 45 version 1303 is already newer than 1300 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1303 (ObjNum 802 snap 251 seq_num 802) dirty exists 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: left oid 45 (ObjNum 802 snap 251 seq_num 802) 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2284: done (3 left) 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2285: done (2 left) 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2286: done (1 left) 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2287: done (0 left) 2024-09-10T16:21:24.784 INFO:tasks.rados.rados.0.smithi165.stdout:2288: delete oid 16 current snap is 251 2024-09-10T16:21:24.786 INFO:tasks.rados.rados.0.smithi165.stdout:2288: done (0 left) 2024-09-10T16:21:24.786 INFO:tasks.rados.rados.0.smithi165.stdout:2289: delete oid 29 current snap is 251 2024-09-10T16:21:24.789 INFO:tasks.rados.rados.0.smithi165.stdout:2289: done (0 left) 2024-09-10T16:21:24.789 INFO:tasks.rados.rados.0.smithi165.stdout:2290: snap_create 2024-09-10T16:21:25.356 INFO:tasks.rados.rados.0.smithi165.stdout:2290: done (0 left) 2024-09-10T16:21:25.356 INFO:tasks.rados.rados.0.smithi165.stdout:2291: rollback oid 20 current snap is 252 2024-09-10T16:21:25.356 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 218 2024-09-10T16:21:25.356 INFO:tasks.rados.rados.0.smithi165.stdout:2292: snap_remove snap 247 2024-09-10T16:21:25.651 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.758+0000 7f746c6a3640 10 monclient: get_monmap_and_config 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.758+0000 7f746c6a3640 10 monclient: build_initial_monmap 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.758+0000 7f746c6a3640 10 monclient: monmap: 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:25.759645+0000 2024-09-10T16:21:25.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:25.759645+0000 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:25.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 1 Processor -- start 2024-09-10T16:21:25.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 1 -- start start 2024-09-10T16:21:25.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 10 monclient: init 2024-09-10T16:21:25.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-b con 0x7f746409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f746409d160 0x7f746409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.759+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-c con 0x7f746409d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f746409f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f746409a430 con 0x7f746409f4c0 2024-09-10T16:21:25.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f74640a24e0 con 0x7f746409d160 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f74640a2bf0 con 0x7f746409c180 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746c6a3640 10 monclient(hunting): authenticate will time out at 1664.247043s 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746ac40640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f746409f4c0 0x7f746413b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35788/0 (socket says 172.21.15.164:35788) 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746ac40640 1 -- 172.21.15.164:0/169132773 learned_addr learned my addr 172.21.15.164:0/169132773 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 10 monclient(hunting): get_auth_request con 0x7f746409c180 auth_method 0 2024-09-10T16:21:25.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 1 -- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f746409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 1 -- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f746409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f746a43f640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 --2- 172.21.15.164:0/169132773 >> v2:172.21.15.165:3300/0 conn(0x7f746409d160 0x7f746409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): get_auth_request con 0x7f746409d160 auth_method 0 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:25.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient(hunting): handle_auth_done global_id 154267 payload 274 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 -- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f746409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 -- 172.21.15.164:0/169132773 >> v1:172.21.15.164:6789/0 conn(0x7f746409f4c0 legacy=0x7f746413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient: _finish_hunting 0 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 monclient: found mon.noname-c 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:21:25.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.760+0000 7f7469c3e640 1 -- 172.21.15.164:0/169132773 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f746409c560 con 0x7f746409d160 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.763+0000 7f7469c3e640 1 -- 172.21.15.164:0/169132773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409d160 msgr2=0x7f746409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.763+0000 7f7469c3e640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409d160 0x7f746409ef80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74580097a0 tx=0x7f745802a050 comp rx=0 tx=0).ready entity=mon.1 client_cookie=39e115ed69499dd2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 1 -- 172.21.15.164:0/169132773 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458037070 con 0x7f746409d160 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: dump: 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:25.775 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: _finish_auth 0 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: _check_auth_tickets 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 1 -- 172.21.15.164:0/169132773 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7458032440 con 0x7f746409d160 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:25.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 1 -- 172.21.15.164:0/169132773 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458032810 con 0x7f746409d160 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 5 monclient: authenticate success, global_id 154267 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 4 set_mon_vals no callback set 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746943d640 10 monclient: dump: 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 10 monclient: shutdown 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 -- 172.21.15.164:0/169132773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409d160 msgr2=0x7f746409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409d160 0x7f746409ef80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f74580097a0 tx=0x7f745802a050 comp rx=0 tx=0).stop 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 -- 172.21.15.164:0/169132773 shutdown_connections 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409d160 0x7f746409ef80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 --2- 172.21.15.164:0/169132773 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f746409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 -- 172.21.15.164:0/169132773 >> 172.21.15.164:0/169132773 conn(0x7f74640a3110 msgr2=0x7f74640a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 -- 172.21.15.164:0/169132773 shutdown_connections 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.764+0000 7f746c6a3640 1 -- 172.21.15.164:0/169132773 wait complete. 2024-09-10T16:21:25.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.765+0000 7f746c6a3640 10 monclient: build_initial_monmap 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.765+0000 7f746c6a3640 10 monclient: monmap: 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.765+0000 7f746c6a3640 1 Processor -- start 2024-09-10T16:21:25.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.765+0000 7f746c6a3640 1 -- start start 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.765+0000 7f746c6a3640 10 monclient: init 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-c con 0x7f746409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-a con 0x7f746409d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409f4c0 0x7f74641ca7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): picked mon.noname-b con 0x7f746409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7464074f90 con 0x7f746409d160 2024-09-10T16:21:25.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7464074e10 con 0x7f746409f4c0 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f746413cce0 con 0x7f746409c180 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746c6a3640 10 monclient(hunting): authenticate will time out at 1664.253290s 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42830/0 (socket says 172.21.15.164:42830) 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 -- 172.21.15.164:0/1842390445 learned_addr learned my addr 172.21.15.164:0/1842390445 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): get_auth_request con 0x7f746409d160 auth_method 0 2024-09-10T16:21:25.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 10 monclient(hunting): get_auth_request con 0x7f746409c180 auth_method 0 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f74641c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f74641c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:25.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746a43f640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746ac40640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409f4c0 0x7f74641ca7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746ac40640 10 monclient(hunting): get_auth_request con 0x7f746409f4c0 auth_method 0 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746ac40640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746ac40640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f746ac40640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient(hunting): handle_auth_done global_id 164187 payload 824 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 msgr2=0x7f74641c9d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409f4c0 msgr2=0x7f74641ca7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409f4c0 0x7f74641ca7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient: _finish_hunting 0 2024-09-10T16:21:25.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 monclient: found mon.noname-a 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.766+0000 7f7469c3e640 1 -- 172.21.15.164:0/1842390445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7464093ba0 con 0x7f746409d160 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.768+0000 7f7469c3e640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 msgr2=0x7f74641ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.768+0000 7f7469c3e640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7458000c00 tx=0x7f74580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458041020 con 0x7f746409d160 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: dump: 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:25.784 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745803d8f0 con 0x7f746409d160 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 5 monclient: authenticate success, global_id 164187 2024-09-10T16:21:25.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 10 monclient: _renew_subs 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7464092960 con 0x7f746409d160 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7458037040 con 0x7f746409d160 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7456ffd640 10 monclient: dump: 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:25.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f7468c3c640 4 set_mon_vals no callback set 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 10 monclient: _renew_subs 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.769+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74640a55e0 con 0x7f746409d160 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7458044020 con 0x7f746409d160 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746c6a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746c6a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746c6a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f746409ea40 con 0x7f746409d160 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f7456ffd640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 0x7f74480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:25.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(523..523 src has 481..523) ==== 4771+0+0 (secure 0 0 0) 0x7f74580733f0 con 0x7f746409d160 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746a43f640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 0x7f74480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.770+0000 7f746a43f640 10 monclient: get_auth_request con 0x7f744803e210 auth_method 0 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.773+0000 7f746a43f640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 0x7f74480406d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7460006fd0 tx=0x7f7460008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.774+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f745804b030 con 0x7f746409d160 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.775+0000 7f7456ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:25.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:25.775+0000 7f7456ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:26.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.017+0000 7f746c6a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:26.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.017+0000 7f746c6a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:26.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.017+0000 7f746c6a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:26.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.017+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f74640924e0 con 0x7f746409d160 2024-09-10T16:21:26.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.018+0000 7f7456ffd640 1 -- 172.21.15.164:0/1842390445 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f745803a350 con 0x7f746409d160 2024-09-10T16:21:26.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.018+0000 7f7456ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:26.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.018+0000 7f7456ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:26.020 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":148,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:26.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 msgr2=0x7f74480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:26.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 0x7f74480406d0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f7460006fd0 tx=0x7f7460008040 comp rx=0 tx=0).stop 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 10 monclient: shutdown 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 msgr2=0x7f74641ca2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7458000c00 tx=0x7f74580046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 shutdown_connections 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f744803e210 0x7f74480406d0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:26.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f746409f4c0 0x7f74641ca7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:26.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f746409d160 0x7f74641ca2a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:26.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 --2- 172.21.15.164:0/1842390445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f746409c180 0x7f74641c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:26.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 >> 172.21.15.164:0/1842390445 conn(0x7f74640a3110 msgr2=0x7f7464073ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:26.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 shutdown_connections 2024-09-10T16:21:26.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:26.020+0000 7f746c6a3640 1 -- 172.21.15.164:0/1842390445 wait complete. 2024-09-10T16:21:26.360 INFO:tasks.rados.rados.0.smithi165.stdout:2292: done (1 left) 2024-09-10T16:21:26.360 INFO:tasks.rados.rados.0.smithi165.stdout:2293: copy_from oid 22 from oid 26 current snap is 252 2024-09-10T16:21:26.360 INFO:tasks.rados.rados.0.smithi165.stdout:2291: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:2291: finishing rollback tid 2 to smithi16534841-20 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1360 (ObjNum 692 snap 213 seq_num 692) dirty exists 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:2291: done (1 left) 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:2294: copy_from oid 12 from oid 13 current snap is 252 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:2295: write (excl) oid 20 current snap is 252 2024-09-10T16:21:26.361 INFO:tasks.rados.rados.0.smithi165.stdout:2295: seq_num 806 ranges {746442=584112,1748369=722605,2983991=562451} 2024-09-10T16:21:26.368 INFO:tasks.rados.rados.0.smithi165.stdout:2295: writing smithi16534841-20 from 746442 to 1330554 tid 1 2024-09-10T16:21:26.375 INFO:tasks.rados.rados.0.smithi165.stdout:2295: writing smithi16534841-20 from 1748369 to 2470974 tid 2 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:2295: writing smithi16534841-20 from 2983991 to 3546442 tid 3 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:2296: delete oid 2 current snap is 252 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:2293: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:2295: finishing write tid 1 to smithi16534841-20 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:2293: finishing copy_from to smithi16534841-22 2024-09-10T16:21:26.382 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1366 (ObjNum 768 snap 235 seq_num 768) dirty exists 2024-09-10T16:21:26.383 INFO:tasks.rados.rados.0.smithi165.stdout:2294: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:21:26.387 INFO:tasks.rados.rados.0.smithi165.stdout:2295: finishing write tid 2 to smithi16534841-20 2024-09-10T16:21:26.390 INFO:tasks.rados.rados.0.smithi165.stdout:2294: finishing copy_from to smithi16534841-12 2024-09-10T16:21:26.390 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1352 (ObjNum 791 snap 243 seq_num 791) dirty exists 2024-09-10T16:21:26.390 INFO:tasks.rados.rados.0.smithi165.stdout:2295: finishing write tid 3 to smithi16534841-20 2024-09-10T16:21:26.390 INFO:tasks.rados.rados.0.smithi165.stdout:2295: finishing write tid 4 to smithi16534841-20 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: finishing write tid 5 to smithi16534841-20 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: oid 20 updating version 0 to 1362 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: oid 20 updating version 1362 to 1364 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: oid 20 version 1364 is already newer than 1363 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: oid 20 version 1364 is already newer than 1361 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1364 (ObjNum 806 snap 252 seq_num 806) dirty exists 2024-09-10T16:21:26.391 INFO:tasks.rados.rados.0.smithi165.stdout:2295: left oid 20 (ObjNum 806 snap 252 seq_num 806) 2024-09-10T16:21:26.393 INFO:tasks.rados.rados.0.smithi165.stdout:2293: done (3 left) 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2294: done (2 left) 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2295: done (1 left) 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2296: done (0 left) 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2297: copy_from oid 23 from oid 22 current snap is 252 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2298: read oid 16 snap -1 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2298: expect deleted 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2299: copy_from oid 45 from oid 13 current snap is 252 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2300: read oid 46 snap -1 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2300: expect deleted 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2301: write oid 2 current snap is 252 2024-09-10T16:21:26.394 INFO:tasks.rados.rados.0.smithi165.stdout:2301: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2024-09-10T16:21:26.402 INFO:tasks.rados.rados.0.smithi165.stdout:2301: writing smithi16534841-2 from 780056 to 1514666 tid 1 2024-09-10T16:21:26.410 INFO:tasks.rados.rados.0.smithi165.stdout:2301: writing smithi16534841-2 from 2148980 to 2875254 tid 2 2024-09-10T16:21:26.414 INFO:tasks.rados.rados.0.smithi165.stdout:2301: writing smithi16534841-2 from 3322484 to 3580056 tid 3 2024-09-10T16:21:26.414 INFO:tasks.rados.rados.0.smithi165.stdout:2302: rollback oid 9 current snap is 252 2024-09-10T16:21:26.414 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 230 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2299: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2301: finishing write tid 1 to smithi16534841-2 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2297: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2298: done (5 left) 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2300: done (4 left) 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2303: delete oid 37 current snap is 252 2024-09-10T16:21:26.415 INFO:tasks.rados.rados.0.smithi165.stdout:2299: finishing copy_from to smithi16534841-45 2024-09-10T16:21:26.416 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1352 (ObjNum 791 snap 243 seq_num 791) dirty exists 2024-09-10T16:21:26.416 INFO:tasks.rados.rados.0.smithi165.stdout:2297: finishing copy_from to smithi16534841-23 2024-09-10T16:21:26.416 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1366 (ObjNum 768 snap 235 seq_num 768) dirty exists 2024-09-10T16:21:26.419 INFO:tasks.rados.rados.0.smithi165.stdout:2302: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:21:26.419 INFO:tasks.rados.rados.0.smithi165.stdout:2301: finishing write tid 2 to smithi16534841-2 2024-09-10T16:21:26.419 INFO:tasks.rados.rados.0.smithi165.stdout:2301: finishing write tid 3 to smithi16534841-2 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2302: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: finishing write tid 4 to smithi16534841-2 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2302: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1192 (ObjNum 749 snap 227 seq_num 749) dirty exists 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: finishing write tid 5 to smithi16534841-2 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: oid 2 updating version 0 to 1350 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: oid 2 version 1350 is already newer than 1349 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: oid 2 updating version 1350 to 1352 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: oid 2 version 1352 is already newer than 1351 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1352 (ObjNum 808 snap 252 seq_num 808) dirty exists 2024-09-10T16:21:26.420 INFO:tasks.rados.rados.0.smithi165.stdout:2301: left oid 2 (ObjNum 808 snap 252 seq_num 808) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2297: done (4 left) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2299: done (3 left) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2301: done (2 left) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2302: done (1 left) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2303: done (0 left) 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2304: copy_from oid 36 from oid 29 current snap is 252 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2305: write oid 24 current snap is 252 2024-09-10T16:21:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:2305: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2024-09-10T16:21:26.426 INFO:tasks.rados.rados.0.smithi165.stdout:2305: writing smithi16534841-24 from 413670 to 898778 tid 1 2024-09-10T16:21:26.434 INFO:tasks.rados.rados.0.smithi165.stdout:2305: writing smithi16534841-24 from 1465944 to 2195887 tid 2 2024-09-10T16:21:26.441 INFO:tasks.rados.rados.0.smithi165.stdout:2305: writing smithi16534841-24 from 2977330 to 3502131 tid 3 2024-09-10T16:21:26.441 INFO:tasks.rados.rados.0.smithi165.stdout:2306: copy_from oid 8 from oid 5 current snap is 252 2024-09-10T16:21:26.441 INFO:tasks.rados.rados.0.smithi165.stdout:2304: finishing copy_from to smithi16534841-36 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2304: got expected ENOENT (src dne) 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2304: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2305: finishing write tid 1 to smithi16534841-24 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2305: finishing write tid 2 to smithi16534841-24 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2304: done (2 left) 2024-09-10T16:21:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:2307: delete oid 3 current snap is 252 2024-09-10T16:21:26.444 INFO:tasks.rados.rados.0.smithi165.stdout:2307: done (2 left) 2024-09-10T16:21:26.444 INFO:tasks.rados.rados.0.smithi165.stdout:2308: copy_from oid 16 from oid 25 current snap is 252 2024-09-10T16:21:26.444 INFO:tasks.rados.rados.0.smithi165.stdout:2309: snap_create 2024-09-10T16:21:26.444 INFO:tasks.rados.rados.0.smithi165.stdout:2305: finishing write tid 3 to smithi16534841-24 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: finishing write tid 4 to smithi16534841-24 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: finishing write tid 5 to smithi16534841-24 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: oid 24 updating version 0 to 1355 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: oid 24 version 1355 is already newer than 1354 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: oid 24 updating version 1355 to 1356 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: oid 24 version 1356 is already newer than 1353 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1356 (ObjNum 810 snap 252 seq_num 810) dirty exists 2024-09-10T16:21:26.446 INFO:tasks.rados.rados.0.smithi165.stdout:2305: left oid 24 (ObjNum 810 snap 252 seq_num 810) 2024-09-10T16:21:26.454 INFO:tasks.rados.rados.0.smithi165.stdout:2306: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:21:26.457 INFO:tasks.rados.rados.0.smithi165.stdout:2306: finishing copy_from to smithi16534841-8 2024-09-10T16:21:26.457 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1309 (ObjNum 767 snap 235 seq_num 767) dirty exists 2024-09-10T16:21:26.458 INFO:tasks.rados.rados.0.smithi165.stdout:2308: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:21:26.461 INFO:tasks.rados.rados.0.smithi165.stdout:2308: finishing copy_from to smithi16534841-16 2024-09-10T16:21:26.461 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1355 (ObjNum 796 snap 246 seq_num 796) dirty exists 2024-09-10T16:21:27.364 INFO:tasks.rados.rados.0.smithi165.stdout:2305: done (3 left) 2024-09-10T16:21:27.364 INFO:tasks.rados.rados.0.smithi165.stdout:2306: done (2 left) 2024-09-10T16:21:27.364 INFO:tasks.rados.rados.0.smithi165.stdout:2308: done (1 left) 2024-09-10T16:21:27.364 INFO:tasks.rados.rados.0.smithi165.stdout:2309: done (0 left) 2024-09-10T16:21:27.364 INFO:tasks.rados.rados.0.smithi165.stdout:2310: delete oid 12 current snap is 253 2024-09-10T16:21:27.371 INFO:tasks.rados.rados.0.smithi165.stdout:2310: done (0 left) 2024-09-10T16:21:27.371 INFO:tasks.rados.rados.0.smithi165.stdout:2311: delete oid 9 current snap is 253 2024-09-10T16:21:27.374 INFO:tasks.rados.rados.0.smithi165.stdout:2311: done (0 left) 2024-09-10T16:21:27.374 INFO:tasks.rados.rados.0.smithi165.stdout:2312: snap_create 2024-09-10T16:21:28.366 INFO:tasks.rados.rados.0.smithi165.stdout:2312: done (0 left) 2024-09-10T16:21:28.366 INFO:tasks.rados.rados.0.smithi165.stdout:2313: snap_remove snap 229 2024-09-10T16:21:29.032 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:21:29.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.144+0000 7f15ae1ec640 10 monclient: get_monmap_and_config 2024-09-10T16:21:29.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.144+0000 7f15ae1ec640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.144+0000 7f15ae1ec640 10 monclient: monmap: 2024-09-10T16:21:29.145 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:29.144918+0000 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:29.144918+0000 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:29.146 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:29.147 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:29.147 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:29.147 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.144+0000 7f15ae1ec640 1 Processor -- start 2024-09-10T16:21:29.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 -- start start 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient: init 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-b con 0x7f15a806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f15a80a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15a80a4a80 0x7f15a81383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-a con 0x7f15a80a4a80 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a809ce50 con 0x7f15a80a46a0 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f15a806db60 con 0x7f15a80a4a80 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15a80a4f10 con 0x7f15a806cb70 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15ae1ec640 10 monclient(hunting): authenticate will time out at 1667.632764s 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15a80a4a80 0x7f15a81383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f15a80a4a80 0x7f15a81383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42842/0 (socket says 172.21.15.164:42842) 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55806/0 (socket says 172.21.15.164:55806) 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 1 -- 172.21.15.164:0/453037397 learned_addr learned my addr 172.21.15.164:0/453037397 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): get_auth_request con 0x7f15a80a4a80 auth_method 0 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 10 monclient(hunting): get_auth_request con 0x7f15a806cb70 auth_method 0 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 1 -- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 msgr2=0x7f15a8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 1 -- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 msgr2=0x7f15a8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.145+0000 7f15a77fe640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a77fe640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 10 monclient(hunting): handle_auth_done global_id 164190 payload 274 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 1 -- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 msgr2=0x7f15a8134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 1 -- 172.21.15.164:0/453037397 >> v1:172.21.15.165:6789/0 conn(0x7f15a80a46a0 legacy=0x7f15a8137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 1 monclient: found mon.noname-a 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.146+0000 7f15a7fff640 1 -- 172.21.15.164:0/453037397 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a809cb70 con 0x7f15a80a4a80 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.148+0000 7f15a7fff640 1 -- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 msgr2=0x7f15a81383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.148+0000 7f15a7fff640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81383b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009e50 tx=0x7f15a002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38a0c07aa0f751f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 1 -- 172.21.15.164:0/453037397 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0036070 con 0x7f15a80a4a80 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: dump: 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.160 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 1 -- 172.21.15.164:0/453037397 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a0007800 con 0x7f15a80a4a80 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 5 monclient: authenticate success, global_id 164190 2024-09-10T16:21:29.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 4 set_mon_vals no callback set 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 1 -- 172.21.15.164:0/453037397 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0007bd0 con 0x7f15a80a4a80 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15a6ffd640 10 monclient: dump: 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.162 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 10 monclient: shutdown 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 -- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 msgr2=0x7f15a81383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81383b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f15a0009e50 tx=0x7f15a002a560 comp rx=0 tx=0).stop 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 -- 172.21.15.164:0/453037397 shutdown_connections 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81383b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/453037397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a806cb70 0x7f15a8134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 -- 172.21.15.164:0/453037397 >> 172.21.15.164:0/453037397 conn(0x7f15a80aa900 msgr2=0x7f15a80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:29.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 -- 172.21.15.164:0/453037397 shutdown_connections 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.149+0000 7f15ae1ec640 1 -- 172.21.15.164:0/453037397 wait complete. 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.150+0000 7f15ae1ec640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.150+0000 7f15ae1ec640 10 monclient: monmap: 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:29.164 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.150+0000 7f15ae1ec640 1 Processor -- start 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.150+0000 7f15ae1ec640 1 -- start start 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.150+0000 7f15ae1ec640 10 monclient: init 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a806cb70 0x7f15a812e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-b con 0x7f15a806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-c con 0x7f15a80a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): picked mon.noname-a con 0x7f15a80a4a80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f15a81388f0 con 0x7f15a80a4a80 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f15a806cf50 con 0x7f15a806cb70 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f15a8138a70 con 0x7f15a80a46a0 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15ae1ec640 10 monclient(hunting): authenticate will time out at 1667.638390s 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42852/0 (socket says 172.21.15.164:42852) 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 -- 172.21.15.164:0/2678957371 learned_addr learned my addr 172.21.15.164:0/2678957371 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:29.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): get_auth_request con 0x7f15a80a4a80 auth_method 0 2024-09-10T16:21:29.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55820/0 (socket says 172.21.15.164:55820) 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 10 monclient(hunting): get_auth_request con 0x7f15a80a46a0 auth_method 0 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 msgr2=0x7f15a812e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 msgr2=0x7f15a812e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a77fe640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a806cb70 0x7f15a812e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a77fe640 10 monclient(hunting): get_auth_request con 0x7f15a806cb70 auth_method 0 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f159ffff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient(hunting): handle_auth_done global_id 164193 payload 824 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 msgr2=0x7f15a812e9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a806cb70 msgr2=0x7f15a812e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a806cb70 0x7f15a812e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 monclient: found mon.noname-a 2024-09-10T16:21:29.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.151+0000 7f15a7fff640 1 -- 172.21.15.164:0/2678957371 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a806e720 con 0x7f15a80a4a80 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a7fff640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 msgr2=0x7f15a81cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a7fff640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15a0034040 tx=0x7f15a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0041020 con 0x7f15a80a4a80 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: dump: 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.170 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.171 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.172 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f15a0031940 con 0x7f15a80a4a80 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a0036040 con 0x7f15a80a4a80 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 5 monclient: authenticate success, global_id 164193 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 10 monclient: _renew_subs 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15a80a1120 con 0x7f15a80a4a80 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a4ff9640 10 monclient: dump: 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.173 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15a67fc640 4 set_mon_vals no callback set 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 10 monclient: _renew_subs 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.154+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15a80a1c80 con 0x7f15a80a4a80 2024-09-10T16:21:29.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f15a0044020 con 0x7f15a80a4a80 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15ae1ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15ae1ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15ae1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15a8134f70 con 0x7f15a80a4a80 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15a4ff9640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 0x7f1588040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(526..526 src has 481..526) ==== 4771+0+0 (secure 0 0 0) 0x7f15a0073a10 con 0x7f15a80a4a80 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15a77fe640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 0x7f1588040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.155+0000 7f15a77fe640 10 monclient: get_auth_request con 0x7f158803e290 auth_method 0 2024-09-10T16:21:29.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.158+0000 7f15a77fe640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 0x7f1588040750 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f15a80a2200 tx=0x7f1598008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.160+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f15a004b070 con 0x7f15a80a4a80 2024-09-10T16:21:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.160+0000 7f15a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.160+0000 7f15a4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:29.321 INFO:tasks.rados.rados.0.smithi165.stdout:2313: done (0 left) 2024-09-10T16:21:29.321 INFO:tasks.rados.rados.0.smithi165.stdout:2314: snap_remove snap 250 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.411+0000 7f15ae1ec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.411+0000 7f15ae1ec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.411+0000 7f15ae1ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.411+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f15a80a4480 con 0x7f15a80a4a80 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.412+0000 7f15a4ff9640 1 -- 172.21.15.164:0/2678957371 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f15a809ce50 con 0x7f15a80a4a80 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.412+0000 7f15a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:29.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.412+0000 7f15a4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:29.414 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 msgr2=0x7f1588040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 0x7f1588040750 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f15a80a2200 tx=0x7f1598008040 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 10 monclient: shutdown 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 msgr2=0x7f15a81cd060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f15a0034040 tx=0x7f15a0004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 shutdown_connections 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158803e290 0x7f1588040750 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f15a80a4a80 0x7f15a81cd060 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f15a80a46a0 0x7f15a812e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 --2- 172.21.15.164:0/2678957371 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f15a806cb70 0x7f15a812e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 >> 172.21.15.164:0/2678957371 conn(0x7f15a80aa900 msgr2=0x7f15a8132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:29.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 shutdown_connections 2024-09-10T16:21:29.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.414+0000 7f15ae1ec640 1 -- 172.21.15.164:0/2678957371 wait complete. 2024-09-10T16:21:29.426 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:21:29.426 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:21:29.426 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:21:29.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.523+0000 7fd0ee60f640 10 monclient: get_monmap_and_config 2024-09-10T16:21:29.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.523+0000 7fd0ee60f640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.523+0000 7fd0ee60f640 10 monclient: monmap: 2024-09-10T16:21:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:29.524810+0000 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:29.524810+0000 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:29.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.524+0000 7fd0ee60f640 1 Processor -- start 2024-09-10T16:21:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.524+0000 7fd0ee60f640 1 -- start start 2024-09-10T16:21:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.524+0000 7fd0ee60f640 10 monclient: init 2024-09-10T16:21:29.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0e809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd0e809d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd0e813a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e809fbc0 con 0x7fd0e809d190 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0e80a2a20 con 0x7fd0e813a5c0 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e809c6f0 con 0x7fd0e809c310 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0ee60f640 10 monclient(hunting): authenticate will time out at 1668.012238s 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0e809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e77fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd0e809d190 0x7fd0e809ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35798/0 (socket says 172.21.15.164:35798) 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e77fe640 1 -- 172.21.15.164:0/198328617 learned_addr learned my addr 172.21.15.164:0/198328617 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request con 0x7fd0e809c310 auth_method 0 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 msgr2=0x7fd0e809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 msgr2=0x7fd0e809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0e809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.525+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0e809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e77fe640 1 -- 172.21.15.164:0/198328617 >> v1:172.21.15.164:6789/0 conn(0x7fd0e809d190 legacy=0x7fd0e809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3474390813 0 0) 0x7fd0d0001450 con 0x7fd0e809d190 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient(hunting): _negotiate 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient(hunting): my global_id is 164196 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd0d4003520 con 0x7fd0e809d190 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0ecbac640 1 -- 172.21.15.164:0/198328617 >> v1:172.21.15.165:6789/0 conn(0x7fd0e813a5c0 legacy=0x7fd0e813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1190296641 0 0) 0x7fd0d4003520 con 0x7fd0e809d190 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 msgr2=0x7fd0e809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0e809ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 >> v1:172.21.15.165:6789/0 conn(0x7fd0e813a5c0 legacy=0x7fd0e813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e809a4d0 con 0x7fd0e809d190 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0ee60f640 5 monclient: authenticate success, global_id 164196 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fd0d0001b70 con 0x7fd0e809d190 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0ee60f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: dump: 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 >> v1:172.21.15.164:6789/0 conn(0x7fd0e809d190 legacy=0x7fd0e809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:21:29.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0d4003aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.527+0000 7fd0e6ffd640 10 monclient(hunting): picked mon.c con 0x7fd0e809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d40040e0 0x7fd0d4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): picked mon.a con 0x7fd0d40040e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0d4004db0 0x7fd0d4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): picked mon.b con 0x7fd0d4004db0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e809c6f0 con 0x7fd0d40040e0 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d400bcc0 con 0x7fd0d4004db0 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0d400bec0 con 0x7fd0e809c310 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ee60f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:29.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d40040e0 0x7fd0d4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0d4003aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request con 0x7fd0e809c310 auth_method 0 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request con 0x7fd0d40040e0 auth_method 0 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ecbac640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0d4004db0 0x7fd0d4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ecbac640 10 monclient(hunting): get_auth_request con 0x7fd0d4004db0 auth_method 0 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ecbac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ecbac640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0ecbac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_done global_id 164196 payload 293 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0d4004db0 msgr2=0x7fd0d4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0d4004db0 0x7fd0d4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d40040e0 msgr2=0x7fd0d4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0d40040e0 0x7fd0d4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 monclient: found mon.c 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d4003520 con 0x7fd0e809c310 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.528+0000 7fd0e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e7fff640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 msgr2=0x7fd0d4003aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0d4003aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b9e0 tx=0x7fd0d4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3cb6fa332eed6ef7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d8030070 con 0x7fd0e809c310 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:29.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: dump: 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0d8034400 con 0x7fd0e809c310 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:29.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 4 set_mon_vals no callback set 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 1 -- 172.21.15.164:0/198328617 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d800b5e0 con 0x7fd0e809c310 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0e6ffd640 10 monclient: dump: 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.551 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 10 monclient: shutdown 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 -- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 msgr2=0x7fd0d4003aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/198328617 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e809c310 0x7fd0d4003aa0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fd0d800b9e0 tx=0x7fd0d4005830 comp rx=0 tx=0).stop 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ecbac640 1 -- 172.21.15.164:0/198328617 reap_dead start 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 -- 172.21.15.164:0/198328617 shutdown_connections 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 -- 172.21.15.164:0/198328617 >> 172.21.15.164:0/198328617 conn(0x7fd0e80a3110 msgr2=0x7fd0e80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 -- 172.21.15.164:0/198328617 shutdown_connections 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.531+0000 7fd0ee60f640 1 -- 172.21.15.164:0/198328617 wait complete. 2024-09-10T16:21:29.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.532+0000 7fd0ee60f640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.532+0000 7fd0ee60f640 10 monclient: monmap: 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.532+0000 7fd0ee60f640 1 Processor -- start 2024-09-10T16:21:29.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.532+0000 7fd0ee60f640 1 -- start start 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.532+0000 7fd0ee60f640 10 monclient: init 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e809c310 0x7fd0e80918b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-b con 0x7fd0e809c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-c con 0x7fd0e8099ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 0x7fd0e81d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): picked mon.noname-a con 0x7fd0e8092330 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e80a2230 con 0x7fd0e8092330 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e80a20b0 con 0x7fd0e809c310 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd0e8074e00 con 0x7fd0e8099ab0 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ee60f640 10 monclient(hunting): authenticate will time out at 1668.020386s 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 0x7fd0e81d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55856/0 (socket says 172.21.15.164:55856) 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 -- 172.21.15.164:0/2131865268 learned_addr learned my addr 172.21.15.164:0/2131865268 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request con 0x7fd0e8099ab0 auth_method 0 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): get_auth_request con 0x7fd0e8092330 auth_method 0 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e809c310 0x7fd0e80918b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request con 0x7fd0e809c310 auth_method 0 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 msgr2=0x7fd0e8091df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 msgr2=0x7fd0e8091df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient(hunting): handle_auth_done global_id 164199 payload 824 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 msgr2=0x7fd0e8091df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd0e8099ab0 0x7fd0e8091df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e809c310 msgr2=0x7fd0e80918b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd0e809c310 0x7fd0e80918b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 monclient: found mon.noname-a 2024-09-10T16:21:29.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.533+0000 7fd0ecbac640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0e80943b0 con 0x7fd0e8092330 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ecbac640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 msgr2=0x7fd0e81d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ecbac640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 0x7fd0e81d17a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0e809aeb0 tx=0x7fd0e000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e000ee50 con 0x7fd0e8092330 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: dump: 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.558 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0e0018070 con 0x7fd0e8092330 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 5 monclient: authenticate success, global_id 164199 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 10 monclient: _renew_subs 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0e8094a80 con 0x7fd0e8092330 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0e000c8d0 con 0x7fd0e8092330 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e4ff9640 10 monclient: dump: 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.560 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0e67fc640 4 set_mon_vals no callback set 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 10 monclient: _renew_subs 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.536+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0e8093000 con 0x7fd0e8092330 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0ee60f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 0x7fd0e808df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8093230 con 0x7fd0e8094fa0 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd0e000caf0 con 0x7fd0e8092330 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 0x7fd0e808df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient: get_auth_request con 0x7fd0e8094fa0 auth_method 0 2024-09-10T16:21:29.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_done global_id 164202 payload 824 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e4ff9640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e210 0x7fd0c80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e7fff640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd0e809f720 con 0x7fd0e8094fa0 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(527..527 src has 481..527) ==== 4843+0+0 (secure 0 0 0) 0x7fd0e0052b60 con 0x7fd0e8092330 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e210 0x7fd0c80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.537+0000 7fd0e77fe640 10 monclient: get_auth_request con 0x7fd0c803e210 auth_method 0 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e7fff640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 msgr2=0x7fd0e808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 0x7fd0e808df80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8035000 tx=0x7fd0d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d803d800 con 0x7fd0e8094fa0 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.539+0000 7fd0e4ff9640 10 monclient: dump: 2024-09-10T16:21:29.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd0d800b6a0 con 0x7fd0e8094fa0 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 msgr2=0x7fd0e808df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 0x7fd0e808df80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8035000 tx=0x7fd0d8004730 comp rx=0 tx=0).stop 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 msgr2=0x7fd0e808df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e4ff9640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8094fa0 0x7fd0e808df80 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8035000 tx=0x7fd0d8004730 comp rx=0 tx=0).stop 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.540+0000 7fd0e77fe640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e210 0x7fd0c80406d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8093ab0 tx=0x7fd0d0006040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0ee60f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 0x7fd0e808e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0ee60f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd0e8077260 con 0x7fd0e80927a0 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 0x7fd0e808e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient: get_auth_request con 0x7fd0e80927a0 auth_method 0 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 10 monclient(hunting): handle_auth_done global_id 164205 payload 824 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.546+0000 7fd0e7fff640 1 -- 172.21.15.164:0/2131865268 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd0e81d6e80 con 0x7fd0e80927a0 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e7fff640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 msgr2=0x7fd0e808e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e7fff640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 0x7fd0e808e4c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80a46c0 tx=0x7fd0d802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0d804d820 con 0x7fd0e80927a0 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 10 monclient: dump: 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fd0d804d820 con 0x7fd0e80927a0 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 msgr2=0x7fd0e808e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 0x7fd0e808e4c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd0e80a46c0 tx=0x7fd0d802e000 comp rx=0 tx=0).stop 2024-09-10T16:21:29.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 msgr2=0x7fd0e808e4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0e4ff9640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e80927a0 0x7fd0e808e4c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 60251, 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:29.570 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:29.571 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:29.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:29.573 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:29.574 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.575 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.576 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:29.577 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:29.579 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.580 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:29.581 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:29.582 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:29.583 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:29.583 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:29.583 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:29.583 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:29.583 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:29.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e210 msgr2=0x7fd0c80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd0c803e210 0x7fd0c80406d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fd0e8093ab0 tx=0x7fd0d0006040 comp rx=0 tx=0).stop 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 10 monclient: shutdown 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 msgr2=0x7fd0e81d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ecbac640 1 -- 172.21.15.164:0/2131865268 reap_dead start 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 0x7fd0e81d17a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd0e809aeb0 tx=0x7fd0e000ec00 comp rx=0 tx=0).stop 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 shutdown_connections 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 --2- 172.21.15.164:0/2131865268 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd0e8092330 0x7fd0e81d17a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 >> 172.21.15.164:0/2131865268 conn(0x7fd0e80a3110 msgr2=0x7fd0e8139900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 shutdown_connections 2024-09-10T16:21:29.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.549+0000 7fd0ee60f640 1 -- 172.21.15.164:0/2131865268 wait complete. 2024-09-10T16:21:29.586 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:29.586 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:21:29.586 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:21:29.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.683+0000 7f2980d64640 10 monclient: get_monmap_and_config 2024-09-10T16:21:29.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.683+0000 7f2980d64640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.684+0000 7f2980d64640 10 monclient: monmap: 2024-09-10T16:21:29.685 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.685 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:29.684881+0000 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:29.684881+0000 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:29.686 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.684+0000 7f2980d64640 1 Processor -- start 2024-09-10T16:21:29.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.684+0000 7f2980d64640 1 -- start start 2024-09-10T16:21:29.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.684+0000 7f2980d64640 10 monclient: init 2024-09-10T16:21:29.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f297c1329f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f297c0a2f10 0x7f297c13ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-c con 0x7f297c0a2f10 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-b con 0x7f297c13b2b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f297c1334e0 con 0x7f297c1329f0 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f297c0a0730 con 0x7f297c0a2f10 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f297c0a21a0 con 0x7f297c13b2b0 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2980d64640 10 monclient(hunting): authenticate will time out at 1668.172379s 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55864/0 (socket says 172.21.15.164:55864) 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 -- 172.21.15.164:0/126564540 learned_addr learned my addr 172.21.15.164:0/126564540 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 10 monclient(hunting): get_auth_request con 0x7f297c13b2b0 auth_method 0 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 -- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 msgr2=0x7f297c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 -- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 msgr2=0x7f297c13d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f297ad9d640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 --2- 172.21.15.164:0/126564540 >> v2:172.21.15.165:3300/0 conn(0x7f297c0a2f10 0x7f297c13ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): get_auth_request con 0x7f297c0a2f10 auth_method 0 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient(hunting): handle_auth_done global_id 154282 payload 274 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 -- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 msgr2=0x7f297c13d6a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 -- 172.21.15.164:0/126564540 >> v1:172.21.15.165:6789/0 conn(0x7f297c1329f0 legacy=0x7f297c132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 monclient: found mon.noname-c 2024-09-10T16:21:29.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:21:29.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.685+0000 7f2979d9b640 1 -- 172.21.15.164:0/126564540 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c0a2480 con 0x7f297c0a2f10 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.688+0000 7f2979d9b640 1 -- 172.21.15.164:0/126564540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c0a2f10 msgr2=0x7f297c13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.688+0000 7f2979d9b640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c0a2f10 0x7f297c13ad70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f2964029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92bc5560ca23b319 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 1 -- 172.21.15.164:0/126564540 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964037070 con 0x7f297c0a2f10 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: dump: 2024-09-10T16:21:29.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 1 -- 172.21.15.164:0/126564540 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2964032440 con 0x7f297c0a2f10 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 5 monclient: authenticate success, global_id 154282 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 1 -- 172.21.15.164:0/126564540 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964032810 con 0x7f297c0a2f10 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 4 set_mon_vals no callback set 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f297959a640 10 monclient: dump: 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 10 monclient: shutdown 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 -- 172.21.15.164:0/126564540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c0a2f10 msgr2=0x7f297c13ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c0a2f10 0x7f297c13ad70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f2964029fd0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 -- 172.21.15.164:0/126564540 shutdown_connections 2024-09-10T16:21:29.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c13b2b0 0x7f297c13d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 --2- 172.21.15.164:0/126564540 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c0a2f10 0x7f297c13ad70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 -- 172.21.15.164:0/126564540 >> 172.21.15.164:0/126564540 conn(0x7f297c0a7320 msgr2=0x7f297c0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 -- 172.21.15.164:0/126564540 shutdown_connections 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.689+0000 7f2980d64640 1 -- 172.21.15.164:0/126564540 wait complete. 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.690+0000 7f2980d64640 10 monclient: build_initial_monmap 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.690+0000 7f2980d64640 10 monclient: monmap: 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:29.703 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.690+0000 7f2980d64640 1 Processor -- start 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.690+0000 7f2980d64640 1 -- start start 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.690+0000 7f2980d64640 10 monclient: init 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 0x7f297c1cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-a con 0x7f297c0a2f10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-c con 0x7f297c1329f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c13b2b0 0x7f297c1cb770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): picked mon.noname-b con 0x7f297c13b2b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f297c0a1a80 con 0x7f297c0a2f10 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f297c0a1900 con 0x7f297c13b2b0 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f297c13dbe0 con 0x7f297c1329f0 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2980d64640 10 monclient(hunting): authenticate will time out at 1668.178293s 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 0x7f297c1cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55876/0 (socket says 172.21.15.164:55876) 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 learned_addr learned my addr 172.21.15.164:0/3106746684 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:29.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 10 monclient(hunting): get_auth_request con 0x7f297c1329f0 auth_method 0 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): get_auth_request con 0x7f297c0a2f10 auth_method 0 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 msgr2=0x7f297c1cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 msgr2=0x7f297c1cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:29.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297ad9d640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c13b2b0 0x7f297c1cb770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297ad9d640 10 monclient(hunting): get_auth_request con 0x7f297c13b2b0 auth_method 0 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient(hunting): handle_auth_done global_id 164208 payload 824 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 msgr2=0x7f297c1cb230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c1329f0 0x7f297c1cb230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c13b2b0 msgr2=0x7f297c1cb770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f297c13b2b0 0x7f297c1cb770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 monclient: found mon.noname-a 2024-09-10T16:21:29.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.691+0000 7f297a59c640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f297c070d30 con 0x7f297c0a2f10 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.693+0000 7f297a59c640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 msgr2=0x7f297c1cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.693+0000 7f297a59c640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 0x7f297c1cacf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c780 tx=0x7f296c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c016020 con 0x7f297c0a2f10 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: dump: 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f296c00aca0 con 0x7f297c0a2f10 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 5 monclient: authenticate success, global_id 164208 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 monclient: _renew_subs 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f297c073090 con 0x7f297c0a2f10 2024-09-10T16:21:29.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296c012660 con 0x7f297c0a2f10 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2972ffd640 10 monclient: dump: 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:29.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 monclient: _renew_subs 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2978d99640 4 set_mon_vals no callback set 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f297c09e0f0 con 0x7f297c0a2f10 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.694+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f297c132dd0 con 0x7f297c131690 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131690 auth_method 0 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f296c019020 con 0x7f297c0a2f10 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2972ffd640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f295c03e210 0x7f295c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(527..527 src has 481..527) ==== 4843+0+0 (secure 0 0 0) 0x7f296c052d70 con 0x7f297c0a2f10 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f297ad9d640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f295c03e210 0x7f295c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.695+0000 7f297ad9d640 10 monclient: get_auth_request con 0x7f295c03e210 auth_method 0 2024-09-10T16:21:29.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.698+0000 7f297ad9d640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f295c03e210 0x7f295c0406d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2974009100 tx=0x7f2974008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131690 auth_method 0 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:29.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:21:29.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:29.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.896+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:29.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:29.897+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131690 auth_method 0 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:21:30.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:30.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:30.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:30.298+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:30.322 INFO:tasks.rados.rados.0.smithi165.stdout:2314: done (0 left) 2024-09-10T16:21:30.322 INFO:tasks.rados.rados.0.smithi165.stdout:2315: write oid 13 current snap is 254 2024-09-10T16:21:30.323 INFO:tasks.rados.rados.0.smithi165.stdout:2315: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2024-09-10T16:21:30.327 INFO:tasks.rados.rados.0.smithi165.stdout:2315: writing smithi16534841-13 from 480898 to 983355 tid 1 2024-09-10T16:21:30.335 INFO:tasks.rados.rados.0.smithi165.stdout:2315: writing smithi16534841-13 from 1699872 to 2437153 tid 2 2024-09-10T16:21:30.343 INFO:tasks.rados.rados.0.smithi165.stdout:2315: writing smithi16534841-13 from 3087022 to 3680898 tid 3 2024-09-10T16:21:30.343 INFO:tasks.rados.rados.0.smithi165.stdout:2316: snap_create 2024-09-10T16:21:30.343 INFO:tasks.rados.rados.0.smithi165.stdout:2315: finishing write tid 1 to smithi16534841-13 2024-09-10T16:21:30.345 INFO:tasks.rados.rados.0.smithi165.stdout:2315: finishing write tid 2 to smithi16534841-13 2024-09-10T16:21:30.346 INFO:tasks.rados.rados.0.smithi165.stdout:2315: finishing write tid 3 to smithi16534841-13 2024-09-10T16:21:30.349 INFO:tasks.rados.rados.0.smithi165.stdout:2315: finishing write tid 4 to smithi16534841-13 2024-09-10T16:21:30.349 INFO:tasks.rados.rados.0.smithi165.stdout:2315: finishing write tid 5 to smithi16534841-13 2024-09-10T16:21:30.349 INFO:tasks.rados.rados.0.smithi165.stdout:2315: oid 13 updating version 0 to 1368 2024-09-10T16:21:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:2315: oid 13 updating version 1368 to 1370 2024-09-10T16:21:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:2315: oid 13 version 1370 is already newer than 1367 2024-09-10T16:21:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:2315: oid 13 version 1370 is already newer than 1369 2024-09-10T16:21:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1370 (ObjNum 814 snap 254 seq_num 814) dirty exists 2024-09-10T16:21:30.350 INFO:tasks.rados.rados.0.smithi165.stdout:2315: left oid 13 (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:21:31.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:31.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131690 auth_method 0 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:31.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:31.099+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:31.326 INFO:tasks.rados.rados.0.smithi165.stdout:2315: done (1 left) 2024-09-10T16:21:31.326 INFO:tasks.rados.rados.0.smithi165.stdout:2316: done (0 left) 2024-09-10T16:21:31.326 INFO:tasks.rados.rados.0.smithi165.stdout:2317: delete oid 2 current snap is 255 2024-09-10T16:21:31.332 INFO:tasks.rados.rados.0.smithi165.stdout:2317: done (0 left) 2024-09-10T16:21:31.332 INFO:tasks.rados.rados.0.smithi165.stdout:2318: snap_remove snap 248 2024-09-10T16:21:32.331 INFO:tasks.rados.rados.0.smithi165.stdout:2318: done (0 left) 2024-09-10T16:21:32.331 INFO:tasks.rados.rados.0.smithi165.stdout:2319: delete oid 44 current snap is 255 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2319: done (0 left) 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2320: read oid 13 snap -1 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2320: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2321: read oid 29 snap 240 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2321: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2322: write (excl) oid 17 current snap is 255 2024-09-10T16:21:32.341 INFO:tasks.rados.rados.0.smithi165.stdout:2322: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2024-09-10T16:21:32.348 INFO:tasks.rados.rados.0.smithi165.stdout:2322: writing smithi16534841-17 from 531319 to 1259523 tid 1 2024-09-10T16:21:32.353 INFO:tasks.rados.rados.0.smithi165.stdout:2322: writing smithi16534841-17 from 1758965 to 2243573 tid 2 2024-09-10T16:21:32.359 INFO:tasks.rados.rados.0.smithi165.stdout:2322: writing smithi16534841-17 from 2652938 to 3172996 tid 3 2024-09-10T16:21:32.359 INFO:tasks.rados.rados.0.smithi165.stdout:2323: write (excl) oid 10 current snap is 255 2024-09-10T16:21:32.359 INFO:tasks.rados.rados.0.smithi165.stdout:2323: seq_num 818 ranges {548126=403453,1417447=744619,2680361=447683,3680450=67676} 2024-09-10T16:21:32.364 INFO:tasks.rados.rados.0.smithi165.stdout:2323: writing smithi16534841-10 from 548126 to 951579 tid 1 2024-09-10T16:21:32.371 INFO:tasks.rados.rados.0.smithi165.stdout:2323: writing smithi16534841-10 from 1417447 to 2162066 tid 2 2024-09-10T16:21:32.376 INFO:tasks.rados.rados.0.smithi165.stdout:2323: writing smithi16534841-10 from 2680361 to 3128044 tid 3 2024-09-10T16:21:32.380 INFO:tasks.rados.rados.0.smithi165.stdout:2323: writing smithi16534841-10 from 3680450 to 3748126 tid 4 2024-09-10T16:21:32.380 INFO:tasks.rados.rados.0.smithi165.stdout:2324: snap_remove snap 241 2024-09-10T16:21:32.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.691+0000 7f2971ffb640 10 monclient: tick 2024-09-10T16:21:32.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.691+0000 7f2971ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:21:32.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.691+0000 7f2971ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:32.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.691+0000 7f2971ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131690 auth_method 0 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 10 monclient(hunting): handle_auth_done global_id 174101 payload 824 2024-09-10T16:21:32.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.701+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f297c072e70 con 0x7f297c131690 2024-09-10T16:21:32.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f29640046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964037040 con 0x7f297c131690 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 10 monclient: dump: 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.706 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2964047620 con 0x7f297c131690 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f29640046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 msgr2=0x7f297c131a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:32.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.704+0000 7f2972ffd640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131690 0x7f297c131a70 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f29640097a0 tx=0x7f29640046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2980d64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2980d64640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 0x7f297c1356c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2980d64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f297c071890 con 0x7f297c131fb0 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 0x7f297c1356c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient: get_auth_request con 0x7f297c131fb0 auth_method 0 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 10 monclient(hunting): handle_auth_done global_id 174104 payload 824 2024-09-10T16:21:32.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.711+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f297c06d1c0 con 0x7f297c131fb0 2024-09-10T16:21:32.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2979d9b640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 msgr2=0x7f297c1356c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2979d9b640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 0x7f297c1356c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2964033000 tx=0x7f2964008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2964050070 con 0x7f297c131fb0 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 10 monclient: dump: 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f296404c2c0 con 0x7f297c131fb0 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 msgr2=0x7f297c1356c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 0x7f297c1356c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2964033000 tx=0x7f2964008030 comp rx=0 tx=0).stop 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 msgr2=0x7f297c1356c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.714+0000 7f2972ffd640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f297c131fb0 0x7f297c1356c0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2964033000 tx=0x7f2964008030 comp rx=0 tx=0).stop 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:32.717 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10356, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:32.718 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.720 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:32.721 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.722 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.723 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.724 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:32.726 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.728 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:32.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:32.730 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f295c03e210 msgr2=0x7f295c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f295c03e210 0x7f295c0406d0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f2974009100 tx=0x7f2974008040 comp rx=0 tx=0).stop 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 10 monclient: shutdown 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 msgr2=0x7f297c1cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 --2- 172.21.15.164:0/3106746684 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f297c0a2f10 0x7f297c1cacf0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f296c00c780 tx=0x7f296c00cc40 comp rx=0 tx=0).stop 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f297ad9d640 1 -- 172.21.15.164:0/3106746684 reap_dead start 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f297ad9d640 1 -- 172.21.15.164:0/3106746684 reap_dead start 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 shutdown_connections 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 >> 172.21.15.164:0/3106746684 conn(0x7f297c0a7320 msgr2=0x7f297c13cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 shutdown_connections 2024-09-10T16:21:32.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.715+0000 7f2980d64640 1 -- 172.21.15.164:0/3106746684 wait complete. 2024-09-10T16:21:32.733 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:32.733 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:21:32.733 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:21:32.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.837+0000 7f8232ee2640 10 monclient: get_monmap_and_config 2024-09-10T16:21:32.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.837+0000 7f8232ee2640 10 monclient: build_initial_monmap 2024-09-10T16:21:32.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.837+0000 7f8232ee2640 10 monclient: monmap: 2024-09-10T16:21:32.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:32.838127+0000 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:32.838127+0000 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:32.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.837+0000 7f8232ee2640 1 Processor -- start 2024-09-10T16:21:32.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 -- start start 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient: init 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-a con 0x7f822c06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-b con 0x7f822c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f822c137ca0 0x7f822c138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-c con 0x7f822c137ca0 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f822c06db60 con 0x7f822c06cb70 2024-09-10T16:21:32.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f822c06cf50 con 0x7f822c137ca0 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f822c09ce50 con 0x7f822c0a46a0 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8232ee2640 10 monclient(hunting): authenticate will time out at 1671.325510s 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42898/0 (socket says 172.21.15.164:42898) 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 1 -- 172.21.15.164:0/1857196383 learned_addr learned my addr 172.21.15.164:0/1857196383 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): get_auth_request con 0x7f822c06cb70 auth_method 0 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): get_auth_request con 0x7f822c0a46a0 auth_method 0 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f823147f640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.165:3300/0 conn(0x7f822c137ca0 0x7f822c138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f823147f640 10 monclient(hunting): get_auth_request con 0x7f822c137ca0 auth_method 0 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f823147f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f823147f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f823147f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient(hunting): handle_auth_done global_id 174107 payload 274 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 -- 172.21.15.164:0/1857196383 >> v2:172.21.15.165:3300/0 conn(0x7f822c137ca0 msgr2=0x7f822c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.165:3300/0 conn(0x7f822c137ca0 0x7f822c138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 -- 172.21.15.164:0/1857196383 >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 msgr2=0x7f822c134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 monclient: found mon.noname-b 2024-09-10T16:21:32.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.838+0000 7f822bfff640 1 -- 172.21.15.164:0/1857196383 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c09cb70 con 0x7f822c0a46a0 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.839+0000 7f8230c7e640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822bfff640 1 -- 172.21.15.164:0/1857196383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 msgr2=0x7f822c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822bfff640 1 --2- 172.21.15.164:0/1857196383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a4ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f822c074d10 tx=0x7f821802a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=851a9bc41768a6bb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 1 -- 172.21.15.164:0/1857196383 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218037070 con 0x7f822c0a46a0 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: dump: 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.849 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.850 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 1 -- 172.21.15.164:0/1857196383 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8218032440 con 0x7f822c0a46a0 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f8232ee2640 5 monclient: authenticate success, global_id 174107 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f8232ee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f8232ee2640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 1 -- 172.21.15.164:0/1857196383 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218032810 con 0x7f822c0a46a0 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f8232ee2640 4 set_mon_vals no callback set 2024-09-10T16:21:32.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.841+0000 7f822b7fe640 10 monclient: dump: 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 10 monclient: shutdown 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 -- 172.21.15.164:0/1857196383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 msgr2=0x7f822c0a4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 --2- 172.21.15.164:0/1857196383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a4ab0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f822c074d10 tx=0x7f821802a000 comp rx=0 tx=0).stop 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 -- 172.21.15.164:0/1857196383 shutdown_connections 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.165:3300/0 conn(0x7f822c137ca0 0x7f822c138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 --2- 172.21.15.164:0/1857196383 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a4ab0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 --2- 172.21.15.164:0/1857196383 >> v2:172.21.15.164:3300/0 conn(0x7f822c06cb70 0x7f822c134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 -- 172.21.15.164:0/1857196383 >> 172.21.15.164:0/1857196383 conn(0x7f822c0aa900 msgr2=0x7f822c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 -- 172.21.15.164:0/1857196383 shutdown_connections 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.842+0000 7f8232ee2640 1 -- 172.21.15.164:0/1857196383 wait complete. 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.843+0000 7f8232ee2640 10 monclient: build_initial_monmap 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.843+0000 7f8232ee2640 10 monclient: monmap: 2024-09-10T16:21:32.853 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.843+0000 7f8232ee2640 1 Processor -- start 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.843+0000 7f8232ee2640 1 -- start start 2024-09-10T16:21:32.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.843+0000 7f8232ee2640 10 monclient: init 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c06cb70 0x7f822c1cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-b con 0x7f822c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-c con 0x7f822c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f822c137ca0 0x7f822c0a3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): picked mon.noname-a con 0x7f822c137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f822c1387f0 con 0x7f822c137ca0 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f822c138670 con 0x7f822c06cb70 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f822c138970 con 0x7f822c0a46a0 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8232ee2640 10 monclient(hunting): authenticate will time out at 1671.331121s 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f822c137ca0 0x7f822c0a3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55950/0 (socket says 172.21.15.164:55950) 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 -- 172.21.15.164:0/2291210603 learned_addr learned my addr 172.21.15.164:0/2291210603 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:32.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): get_auth_request con 0x7f822c0a46a0 auth_method 0 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): get_auth_request con 0x7f822c137ca0 auth_method 0 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c06cb70 0x7f822c1cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): get_auth_request con 0x7f822c06cb70 auth_method 0 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient(hunting): handle_auth_done global_id 174110 payload 824 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c06cb70 msgr2=0x7f822c1cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c06cb70 0x7f822c1cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f822c137ca0 msgr2=0x7f822c0a3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f822c137ca0 0x7f822c0a3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 monclient: found mon.noname-c 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f822bfff640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f822c0a5b10 con 0x7f822c0a46a0 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.844+0000 7f823147f640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f822c137ca0 0x7f822c0a3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f822bfff640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 msgr2=0x7f822c0a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f822bfff640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a3830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f821802a5f0 tx=0x7f82180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218041030 con 0x7f822c0a46a0 2024-09-10T16:21:32.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: dump: 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.866 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: _finish_auth 0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: _check_auth_tickets 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f821803d680 con 0x7f822c0a46a0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f8232ee2640 5 monclient: authenticate success, global_id 174110 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f8232ee2640 10 monclient: _renew_subs 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f8232ee2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f822c06e720 con 0x7f822c0a46a0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8218037040 con 0x7f822c0a46a0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:32.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f82297fa640 10 monclient: dump: 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.846+0000 7f822affd640 4 set_mon_vals no callback set 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 10 monclient: _renew_subs 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f822c09ffa0 con 0x7f822c0a46a0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 0x7f822c09e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f822c0a4ab0 con 0x7f822c09e5d0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.847+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8218044070 con 0x7f822c0a46a0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 0x7f822c09e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f82297fa640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f821003e240 0x7f8210040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient: get_auth_request con 0x7f822c09e5d0 auth_method 0 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(530..530 src has 481..530) ==== 4799+0+0 (secure 0 0 0) 0x7f8218073020 con 0x7f822c0a46a0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f823147f640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f821003e240 0x7f8210040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f823147f640 10 monclient: get_auth_request con 0x7f821003e240 auth_method 0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 10 monclient(hunting): handle_auth_done global_id 154294 payload 824 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.848+0000 7f8230c7e640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f822c0a0b00 con 0x7f822c09e5d0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.850+0000 7f8230c7e640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 msgr2=0x7f822c09e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.850+0000 7f8230c7e640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 0x7f822c09e9b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f8220011cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.850+0000 7f823147f640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f821003e240 0x7f8210040700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8224007740 tx=0x7f8224009820 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822000fcb0 con 0x7f822c09e5d0 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 10 monclient: dump: 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f822000fcb0 con 0x7f822c09e5d0 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 msgr2=0x7f822c09e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 0x7f822c09e9b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f8220011cd0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 msgr2=0x7f822c09e9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.851+0000 7f82297fa640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09e5d0 0x7f822c09e9b0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f8220011cd0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.857+0000 7f8232ee2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.857+0000 7f8232ee2640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 0x7f822c135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.857+0000 7f8232ee2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.857+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f822c0770f0 con 0x7f822c09eef0 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 0x7f822c135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient: get_auth_request con 0x7f822c09eef0 auth_method 0 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 10 monclient(hunting): handle_auth_done global_id 154297 payload 824 2024-09-10T16:21:32.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.858+0000 7f8230c7e640 1 -- 172.21.15.164:0/2291210603 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f822c09f300 con 0x7f822c09eef0 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f8230c7e640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 msgr2=0x7f822c135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f8230c7e640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 0x7f822c135870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f82200046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220014850 con 0x7f822c09eef0 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f82297fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f82297fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:32.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.860+0000 7f82297fa640 10 monclient: dump: 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f822002b470 con 0x7f822c09eef0 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:32.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 msgr2=0x7f822c135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 0x7f822c135870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f82200046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 msgr2=0x7f822c135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f82297fa640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f822c09eef0 0x7f822c135870 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8220006ca0 tx=0x7f82200046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:32.876 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 111412, 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.877 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.878 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:32.879 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:32.880 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:32.881 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.882 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:32.883 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:32.884 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.885 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.886 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.887 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:32.888 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:32.889 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:32.890 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:32.890 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:32.890 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:32.890 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:32.890 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f821003e240 msgr2=0x7f8210040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f821003e240 0x7f8210040700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f8224007740 tx=0x7f8224009820 comp rx=0 tx=0).stop 2024-09-10T16:21:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 10 monclient: shutdown 2024-09-10T16:21:32.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 msgr2=0x7f822c0a3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 --2- 172.21.15.164:0/2291210603 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f822c0a46a0 0x7f822c0a3830 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f821802a5f0 tx=0x7f82180046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f823147f640 1 -- 172.21.15.164:0/2291210603 reap_dead start 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f823147f640 1 -- 172.21.15.164:0/2291210603 reap_dead start 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 shutdown_connections 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 >> 172.21.15.164:0/2291210603 conn(0x7f822c0aa900 msgr2=0x7f822c132e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 shutdown_connections 2024-09-10T16:21:32.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.861+0000 7f8232ee2640 1 -- 172.21.15.164:0/2291210603 wait complete. 2024-09-10T16:21:32.892 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:32.892 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:21:32.893 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:21:32.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 10 monclient: get_monmap_and_config 2024-09-10T16:21:32.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 10 monclient: build_initial_monmap 2024-09-10T16:21:32.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 10 monclient: monmap: 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:32.990064+0000 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:32.990064+0000 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:32.991 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:32.992 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 1 Processor -- start 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 1 -- start start 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.989+0000 7f6116d5d640 10 monclient: init 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f611009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:32.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f611009ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f611009e250 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61100a0870 con 0x7f611009e250 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61101353a0 con 0x7f611009ab50 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f611009d3e0 con 0x7f611009d930 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6116d5d640 10 monclient(hunting): authenticate will time out at 1671.477438s 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f61152fa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f611009e250 0x7f6110134c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35812/0 (socket says 172.21.15.164:35812) 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f61152fa640 1 -- 172.21.15.164:0/3301869054 learned_addr learned my addr 172.21.15.164:0/3301869054 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): get_auth_request con 0x7f611009d930 auth_method 0 2024-09-10T16:21:32.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient(hunting): handle_auth_done global_id 174113 payload 274 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 1 -- 172.21.15.164:0/3301869054 >> v1:172.21.15.165:6789/0 conn(0x7f611009ab50 legacy=0x7f6110134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 1 -- 172.21.15.164:0/3301869054 >> v1:172.21.15.164:6789/0 conn(0x7f611009e250 legacy=0x7f6110134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient: _finish_hunting 0 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 1 monclient: found mon.noname-b 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:32.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:21:32.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.990+0000 7f6114af9640 1 -- 172.21.15.164:0/3301869054 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6110074cc0 con 0x7f611009d930 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f6114af9640 1 -- 172.21.15.164:0/3301869054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 msgr2=0x7f611009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f6114af9640 1 --2- 172.21.15.164:0/3301869054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611009dd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f610800aed0 tx=0x7f610802b650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bbe8ad0c61eda97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 1 -- 172.21.15.164:0/3301869054 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108037070 con 0x7f611009d930 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: dump: 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.003 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f6116d5d640 5 monclient: authenticate success, global_id 174113 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 1 -- 172.21.15.164:0/3301869054 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6108008850 con 0x7f611009d930 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 1 -- 172.21.15.164:0/3301869054 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108008c20 con 0x7f611009d930 2024-09-10T16:21:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f6116d5d640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f6116d5d640 4 set_mon_vals no callback set 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.993+0000 7f610f7fe640 10 monclient: dump: 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 10 monclient: shutdown 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 -- 172.21.15.164:0/3301869054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 msgr2=0x7f611009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3301869054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611009dd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f610800aed0 tx=0x7f610802b650 comp rx=0 tx=0).stop 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 -- 172.21.15.164:0/3301869054 shutdown_connections 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3301869054 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611009dd10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 -- 172.21.15.164:0/3301869054 >> 172.21.15.164:0/3301869054 conn(0x7f61100a3110 msgr2=0x7f61100a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 -- 172.21.15.164:0/3301869054 shutdown_connections 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.994+0000 7f6116d5d640 1 -- 172.21.15.164:0/3301869054 wait complete. 2024-09-10T16:21:33.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 10 monclient: build_initial_monmap 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 10 monclient: monmap: 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 1 Processor -- start 2024-09-10T16:21:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 1 -- start start 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 10 monclient: init 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.995+0000 7f6116d5d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f611009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611012f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f611009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f611009e250 0x7f611012f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f611009e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f611009e6e0 con 0x7f611009ab50 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f61100a02b0 con 0x7f611009e250 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f611009ef30 con 0x7f611009d930 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6116d5d640 10 monclient(hunting): authenticate will time out at 1671.483066s 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42918/0 (socket says 172.21.15.164:42918) 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 -- 172.21.15.164:0/3897693689 learned_addr learned my addr 172.21.15.164:0/3897693689 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:33.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): get_auth_request con 0x7f611009ab50 auth_method 0 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611012f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): get_auth_request con 0x7f611009d930 auth_method 0 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:33.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f61152fa640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f611009e250 0x7f611012f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f61152fa640 10 monclient(hunting): get_auth_request con 0x7f611009e250 auth_method 0 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f61152fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f61152fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f61152fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient(hunting): handle_auth_done global_id 164217 payload 824 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 -- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 msgr2=0x7f611012f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611012f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 -- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f611009e250 msgr2=0x7f611012f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f611009e250 0x7f611012f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient: _finish_hunting 0 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 monclient: found mon.noname-a 2024-09-10T16:21:33.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f6114af9640 1 -- 172.21.15.164:0/3897693689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6110132eb0 con 0x7f611009ab50 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.996+0000 7f610ffff640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611012f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6114af9640 1 -- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 msgr2=0x7f61101335d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6114af9640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f610802b230 tx=0x7f6108004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108037050 con 0x7f611009ab50 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: dump: 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.012 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6108043070 con 0x7f611009ab50 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6116d5d640 5 monclient: authenticate success, global_id 164217 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:33.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6108032df0 con 0x7f611009ab50 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6116d5d640 10 monclient: _renew_subs 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6116d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6110132610 con 0x7f611009ab50 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610effd640 4 set_mon_vals no callback set 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.998+0000 7f610d7fa640 10 monclient: dump: 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:33.014 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.999+0000 7f6116d5d640 10 monclient: _renew_subs 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.999+0000 7f6116d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.999+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6110131890 con 0x7f611009ab50 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.999+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6108048460 con 0x7f611009ab50 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:32.999+0000 7f610d7fa640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 0x7f60f00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f6116d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f6116d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:33.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f6116d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f611009d620 con 0x7f611009ab50 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(530..530 src has 481..530) ==== 4799+0+0 (secure 0 0 0) 0x7f6108073fa0 con 0x7f611009ab50 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f610ffff640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 0x7f60f00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.000+0000 7f610ffff640 10 monclient: get_auth_request con 0x7f60f003e210 auth_method 0 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.002+0000 7f610ffff640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 0x7f60f00406d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6110131610 tx=0x7f6100007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.004+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6108047ce0 con 0x7f611009ab50 2024-09-10T16:21:33.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.004+0000 7f610d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:33.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.004+0000 7f610d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.246+0000 7f6116d5d640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:21:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.246+0000 7f6116d5d640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:21:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.246+0000 7f6116d5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.246+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f611009d620 con 0x7f611009ab50 2024-09-10T16:21:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.247+0000 7f610d7fa640 1 -- 172.21.15.164:0/3897693689 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f610803e450 con 0x7f611009ab50 2024-09-10T16:21:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.247+0000 7f610d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:21:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.247+0000 7f610d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:33.249 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=3766392112,config=2586702516,health=3152313405,kv=3185102672,logm=2250318012}) 2024-09-10T16:21:33.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=1756151356,config=2586702516,health=3152313406,kv=2249883377,logm=755301561}) 2024-09-10T16:21:33.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=12,kv=11,logm=52} crc {auth=3766392112,config=2586702516,health=3152313405,kv=3185102672,logm=2250318012}) 2024-09-10T16:21:33.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.248+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=12,kv=12,logm=48} crc {auth=696830358,config=2586702516,health=3152313405,kv=805391194,logm=1907287113}) 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510055137}) 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2960846323}) 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3510055137}) 2024-09-10T16:21:33.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084180484}) 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 msgr2=0x7f60f00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 0x7f60f00406d0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f6110131610 tx=0x7f6100007e30 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 10 monclient: shutdown 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 msgr2=0x7f61101335d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f610802b230 tx=0x7f6108004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 shutdown_connections 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60f003e210 0x7f60f00406d0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.249+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f611009e250 0x7f611012f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f611009d930 0x7f611012f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f6116d5d640 1 --2- 172.21.15.164:0/3897693689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f611009ab50 0x7f61101335d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 >> 172.21.15.164:0/3897693689 conn(0x7f61100a3110 msgr2=0x7f61100729f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:33.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 shutdown_connections 2024-09-10T16:21:33.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f6116d5d640 1 -- 172.21.15.164:0/3897693689 wait complete. 2024-09-10T16:21:33.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968239522}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425593069}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3968239522}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.250+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2704841480}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278785}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4250298528}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563278785}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.251+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401788911}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079004458}) 2024-09-10T16:21:33.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=259422325}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4079004458}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.252+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2644613232}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712030559}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1065793526}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1712030559}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.253+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2495300728}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290481768}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3454732161}) 2024-09-10T16:21:33.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2290481768}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.254+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408056186}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776583124}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2414787775}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1776583124}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.255+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433313073}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035576022}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1762617310}) 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035576022}) 2024-09-10T16:21:33.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785587154}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087452364}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81345302}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.256+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3087452364}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3862675720}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138703729}) 2024-09-10T16:21:33.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019543055}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=138703729}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.257+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1835662205}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450754981}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2350316133}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2450754981}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.258+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=292597375}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713754936}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=597452727}) 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2713754936}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.259+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=349429820}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941791818}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2765327824}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2941791818}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4131923737}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333843789}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073010818}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333843789}) 2024-09-10T16:21:33.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.260+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864675329}) 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096992643}) 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=858001183}) 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4096992643}) 2024-09-10T16:21:33.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.261+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=729247108}) 2024-09-10T16:21:33.268 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:21:33.270 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.270 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541422105}) 2024-09-10T16:21:33.270 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1717333797}) 2024-09-10T16:21:33.270 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1541422105}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.262+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904911669}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839852730}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4138409096}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839852730}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.263+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2305957136}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922300825}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168280291}) 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.271 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922300825}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.264+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=704761623}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814391408}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281640538}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3814391408}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.265+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=105071415}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232568060}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351319278}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2232568060}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.266+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178164923}) 2024-09-10T16:21:33.272 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.273 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745530180}) 2024-09-10T16:21:33.273 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1554357526}) 2024-09-10T16:21:33.273 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.273 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2745530180}) 2024-09-10T16:21:33.273 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.267+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=191671475}) 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119177579}) 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=68959898}) 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119177579}) 2024-09-10T16:21:33.329 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.327+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4244423067}) 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058143650}) 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752157542}) 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1058143650}) 2024-09-10T16:21:33.336 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.334+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=375836411}) 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298437950}) 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172346727}) 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298437950}) 2024-09-10T16:21:33.368 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.366+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1966811161}) 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.373+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.374+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589211158}) 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.374+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3395066099}) 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.374+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.374+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=589211158}) 2024-09-10T16:21:33.375 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.374+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2445180547}) 2024-09-10T16:21:33.376 INFO:tasks.rados.rados.0.smithi165.stdout:2324: done (4 left) 2024-09-10T16:21:33.376 INFO:tasks.rados.rados.0.smithi165.stdout:2325: write (excl) oid 14 current snap is 255 2024-09-10T16:21:33.376 INFO:tasks.rados.rados.0.smithi165.stdout:2325: seq_num 819 ranges {564933=478702,1759576=488277,2991431=658955} 2024-09-10T16:21:33.382 INFO:tasks.rados.rados.0.smithi165.stdout:2325: writing smithi16534841-14 from 564933 to 1043635 tid 1 2024-09-10T16:21:33.389 INFO:tasks.rados.rados.0.smithi165.stdout:2325: writing smithi16534841-14 from 1759576 to 2247853 tid 2 2024-09-10T16:21:33.397 INFO:tasks.rados.rados.0.smithi165.stdout:2325: writing smithi16534841-14 from 2991431 to 3650386 tid 3 2024-09-10T16:21:33.397 INFO:tasks.rados.rados.0.smithi165.stdout:2326: snap_remove snap 245 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386689229}) 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290137938}) 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1386689229}) 2024-09-10T16:21:33.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.396+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1978590949}) 2024-09-10T16:21:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517011084}) 2024-09-10T16:21:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247236879}) 2024-09-10T16:21:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3517011084}) 2024-09-10T16:21:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.546+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1408892987}) 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849373881}) 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266440511}) 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849373881}) 2024-09-10T16:21:33.550 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.548+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060167968}) 2024-09-10T16:21:33.563 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.563 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106656929}) 2024-09-10T16:21:33.564 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2957564598}) 2024-09-10T16:21:33.564 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.564 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3106656929}) 2024-09-10T16:21:33.564 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.562+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3754169903}) 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788449059}) 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190315661}) 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=788449059}) 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.563+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1970251183}) 2024-09-10T16:21:33.565 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.566 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996285450}) 2024-09-10T16:21:33.566 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2807570238}) 2024-09-10T16:21:33.566 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.566 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3996285450}) 2024-09-10T16:21:33.566 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.564+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1754250550}) 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90202369}) 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3438293396}) 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=90202369}) 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3929711328}) 2024-09-10T16:21:33.567 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.565+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.568 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453275085}) 2024-09-10T16:21:33.568 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2064571858}) 2024-09-10T16:21:33.568 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.568 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1453275085}) 2024-09-10T16:21:33.568 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=607227778}) 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024799676}) 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190112836}) 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.566+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4024799676}) 2024-09-10T16:21:33.569 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.567+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666445371}) 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842771686}) 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2593840717}) 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2842771686}) 2024-09-10T16:21:33.690 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.688+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1798109591}) 2024-09-10T16:21:33.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189277211}) 2024-09-10T16:21:33.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=997059451}) 2024-09-10T16:21:33.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3189277211}) 2024-09-10T16:21:33.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.689+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3162923557}) 2024-09-10T16:21:33.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266332639}) 2024-09-10T16:21:33.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=442334542}) 2024-09-10T16:21:33.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.693 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2266332639}) 2024-09-10T16:21:33.693 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.690+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2087393584}) 2024-09-10T16:21:33.693 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.693 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124455454}) 2024-09-10T16:21:33.693 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2549576942}) 2024-09-10T16:21:33.694 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.694 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4124455454}) 2024-09-10T16:21:33.694 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.691+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4206380331}) 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843057093}) 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=263013836}) 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=843057093}) 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.692+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691171870}) 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.695 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206311086}) 2024-09-10T16:21:33.696 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4021204239}) 2024-09-10T16:21:33.696 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.696 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2206311086}) 2024-09-10T16:21:33.696 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.693+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132125835}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077581009}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1935406207}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1077581009}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.694+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903795621}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622795896}) 2024-09-10T16:21:33.697 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2370824238}) 2024-09-10T16:21:33.698 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.698 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622795896}) 2024-09-10T16:21:33.698 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.695+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2752375706}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787684512}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=748909526}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2787684512}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.696+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=146684807}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1248713251}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570987679}) 2024-09-10T16:21:33.699 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.700 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1248713251}) 2024-09-10T16:21:33.700 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.697+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2203746262}) 2024-09-10T16:21:33.701 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.701 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901686724}) 2024-09-10T16:21:33.701 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110585548}) 2024-09-10T16:21:33.701 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=901686724}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.698+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2549203701}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177738873}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2519634516}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4177738873}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.699+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269990116}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975650330}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2992109952}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=975650330}) 2024-09-10T16:21:33.702 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.700+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=775719778}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=5,mgr_command_descs=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=1} crc {logm=849556731,mdsmap=931935893,mgr=762987792,mgr_command_descs=3192186153,mgrstat=1851809270,mon_config_key=1698199361,monmap=1685985870,nvmeofgw=773687688}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=62,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13} crc {logm=1640784321,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186154,mgr_metadata=2169216343,mgrstat=491079286,mon_config_key=2432472233}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=60,mdsmap=3,mgr=5,mgr_command_descs=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=1} crc {logm=849556731,mdsmap=931935893,mgr=762987792,mgr_command_descs=3192186153,mgrstat=1851809270,mon_config_key=1698199361,monmap=1685985870,nvmeofgw=773687688}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.701+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=1} crc {logm=3032206915,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1561490661,mon_config_key=2406530038,monmap=1433040829,nvmeofgw=773687688}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {nvmeofgw=3140514009,osd_metadata=4005186995,osd_pg_creating=1471129809,osd_snap=2247410651}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {nvmeofgw=3140514009,osd_metadata=4005186995,osd_pg_creating=1471129809,osd_snap=1124569676}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {nvmeofgw=3140514009,osd_metadata=4005186995,osd_pg_creating=1471129809,osd_snap=2247410651}) 2024-09-10T16:21:33.704 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.702+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=91} crc {nvmeofgw=3140514009,osd_metadata=4005186995,osd_pg_creating=1471129809,osd_snap=1166689523}) 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4097941461,osdmap=3070885220}) 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=11,osdmap=86} crc {osd_snap=4097941461,osdmap=1812188335}) 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4097941461,osdmap=3070885220}) 2024-09-10T16:21:33.709 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:21:33.708+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=4097941461,osdmap=2080336271}) 2024-09-10T16:21:34.269 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:21:34.270 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:21:34.322 INFO:tasks.rados.rados.0.smithi165.stdout:2326: done (5 left) 2024-09-10T16:21:34.322 INFO:tasks.rados.rados.0.smithi165.stdout:2327: snap_remove snap 253 2024-09-10T16:21:34.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.378+0000 7f39f5389640 10 monclient: get_monmap_and_config 2024-09-10T16:21:34.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.378+0000 7f39f5389640 10 monclient: build_initial_monmap 2024-09-10T16:21:34.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.378+0000 7f39f5389640 10 monclient: monmap: 2024-09-10T16:21:34.380 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:34.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:34.379647+0000 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:34.379647+0000 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:34.381 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 Processor -- start 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 -- start start 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient: init 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f39f006cd00 0x7f39f0135000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-c con 0x7f39f006cd00 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-b con 0x7f39f00a46e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 0x7f39f0138040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-a con 0x7f39f0137bf0 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:34.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f39f006d0e0 con 0x7f39f0137bf0 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f39f006df10 con 0x7f39f006cd00 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f39f009cb90 con 0x7f39f00a46e0 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.379+0000 7f39f5389640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39f5389640 10 monclient(hunting): authenticate will time out at 1672.867008s 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 0x7f39f0138040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34212/0 (socket says 172.21.15.164:34212) 2024-09-10T16:21:34.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 -- 172.21.15.164:0/2356116473 learned_addr learned my addr 172.21.15.164:0/2356116473 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): get_auth_request con 0x7f39f0137bf0 auth_method 0 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): get_auth_request con 0x7f39f00a46e0 auth_method 0 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.165:3300/0 conn(0x7f39f006cd00 0x7f39f0135000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): get_auth_request con 0x7f39f006cd00 auth_method 0 2024-09-10T16:21:34.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:34.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient(hunting): handle_auth_done global_id 174119 payload 274 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 -- 172.21.15.164:0/2356116473 >> v2:172.21.15.165:3300/0 conn(0x7f39f006cd00 msgr2=0x7f39f0135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.165:3300/0 conn(0x7f39f006cd00 0x7f39f0135000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 -- 172.21.15.164:0/2356116473 >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 msgr2=0x7f39f0138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 0x7f39f0138040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 monclient: found mon.noname-b 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39e67fc640 1 -- 172.21.15.164:0/2356116473 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f009ce70 con 0x7f39f00a46e0 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.380+0000 7f39ef7fe640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 0x7f39f0138040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:34.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39e67fc640 1 -- 172.21.15.164:0/2356116473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 msgr2=0x7f39f00a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39e67fc640 1 --2- 172.21.15.164:0/2356116473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39f006d420 tx=0x7f39d802a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8aa2f3e4f5eb08f2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 1 -- 172.21.15.164:0/2356116473 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8037070 con 0x7f39f00a46e0 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: dump: 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:34.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 1 -- 172.21.15.164:0/2356116473 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d8032640 con 0x7f39f00a46e0 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 1 -- 172.21.15.164:0/2356116473 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8032a10 con 0x7f39f00a46e0 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:34.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39ee7fc640 10 monclient: dump: 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:34.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 5 monclient: authenticate success, global_id 174119 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 4 set_mon_vals no callback set 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 10 monclient: shutdown 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 -- 172.21.15.164:0/2356116473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 msgr2=0x7f39f00a4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 --2- 172.21.15.164:0/2356116473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f39f006d420 tx=0x7f39d802a0b0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 -- 172.21.15.164:0/2356116473 shutdown_connections 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.164:3300/0 conn(0x7f39f0137bf0 0x7f39f0138040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 --2- 172.21.15.164:0/2356116473 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f00a46e0 0x7f39f00a4b00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 --2- 172.21.15.164:0/2356116473 >> v2:172.21.15.165:3300/0 conn(0x7f39f006cd00 0x7f39f0135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 -- 172.21.15.164:0/2356116473 >> 172.21.15.164:0/2356116473 conn(0x7f39f00aaa70 msgr2=0x7f39f00aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 -- 172.21.15.164:0/2356116473 shutdown_connections 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.383+0000 7f39f5389640 1 -- 172.21.15.164:0/2356116473 wait complete. 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.384+0000 7f39f5389640 10 monclient: build_initial_monmap 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.384+0000 7f39f5389640 10 monclient: monmap: 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:34.401 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.384+0000 7f39f5389640 1 Processor -- start 2024-09-10T16:21:34.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 -- start start 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient: init 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-c con 0x7f39f006cd00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-a con 0x7f39f00a46e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39f0137bf0 0x7f39f00a3de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): picked mon.noname-b con 0x7f39f0137bf0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f39f0138700 con 0x7f39f00a46e0 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f39f0138580 con 0x7f39f0137bf0 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f39f0138880 con 0x7f39f006cd00 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39f5389640 10 monclient(hunting): authenticate will time out at 1672.872486s 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34220/0 (socket says 172.21.15.164:34220) 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 1 -- 172.21.15.164:0/807691102 learned_addr learned my addr 172.21.15.164:0/807691102 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:34.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): get_auth_request con 0x7f39f006cd00 auth_method 0 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): get_auth_request con 0x7f39f00a46e0 auth_method 0 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:34.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient(hunting): handle_auth_done global_id 164223 payload 824 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39ef7fe640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39f0137bf0 0x7f39f00a3de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 -- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 msgr2=0x7f39f01cd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 -- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39f0137bf0 msgr2=0x7f39f00a3de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39f0137bf0 0x7f39f00a3de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 monclient: found mon.noname-a 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:34.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39e67fc640 1 -- 172.21.15.164:0/807691102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f006e670 con 0x7f39f00a46e0 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.385+0000 7f39eeffd640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e67fc640 1 -- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 msgr2=0x7f39f00a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e67fc640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39d8031000 tx=0x7f39d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d8037040 con 0x7f39f00a46e0 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: dump: 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.407 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: _finish_auth 0 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39d803b640 con 0x7f39f00a46e0 2024-09-10T16:21:34.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 5 monclient: authenticate success, global_id 164223 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 10 monclient: _renew_subs 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39d803ba30 con 0x7f39f00a46e0 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f00a2090 con 0x7f39f00a46e0 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39e7fff640 10 monclient: dump: 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:34.409 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39edffb640 4 set_mon_vals no callback set 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 10 monclient: _renew_subs 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.388+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f01cd580 con 0x7f39f00a46e0 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39e67fc640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39d8031000 tx=0x7f39d80046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f39d8072c90 osd_map(532..532 src has 481..532) 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f39d8044020 con 0x7f39f00a46e0 2024-09-10T16:21:34.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39e7fff640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 0x7f39d00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39f5389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39f5389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39f5389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39f00a1c20 con 0x7f39f00a46e0 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39eeffd640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 0x7f39d00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:34.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.389+0000 7f39eeffd640 10 monclient: get_auth_request con 0x7f39d003e210 auth_method 0 2024-09-10T16:21:34.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.391+0000 7f39eeffd640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 0x7f39d00406d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009f50 tx=0x7f39e0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:34.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:34.394+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(532..532 src has 481..532) ==== 4859+0+0 (secure 0 0 0) 0x7f39d8072c90 con 0x7f39f00a46e0 2024-09-10T16:21:35.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.217+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f39f006e670 con 0x7f39f00a46e0 2024-09-10T16:21:35.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.217+0000 7f39e7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.217+0000 7f39e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:35.331 INFO:tasks.rados.rados.0.smithi165.stdout:2327: done (5 left) 2024-09-10T16:21:35.331 INFO:tasks.rados.rados.0.smithi165.stdout:2328: write oid 46 current snap is 255 2024-09-10T16:21:35.332 INFO:tasks.rados.rados.0.smithi165.stdout:2328: seq_num 820 ranges {581740=553951,1818058=748288,3018854=470227} 2024-09-10T16:21:35.337 INFO:tasks.rados.rados.0.smithi165.stdout:2328: writing smithi16534841-46 from 581740 to 1135691 tid 1 2024-09-10T16:21:35.346 INFO:tasks.rados.rados.0.smithi165.stdout:2328: writing smithi16534841-46 from 1818058 to 2566346 tid 2 2024-09-10T16:21:35.351 INFO:tasks.rados.rados.0.smithi165.stdout:2328: writing smithi16534841-46 from 3018854 to 3489081 tid 3 2024-09-10T16:21:35.351 INFO:tasks.rados.rados.0.smithi165.stdout:2329: copy_from oid 16 from oid 6 current snap is 255 2024-09-10T16:21:35.351 INFO:tasks.rados.rados.0.smithi165.stdout:2320: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:21:35.397 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3381924 2024-09-10T16:21:35.399 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.469+0000 7f39f5389640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.469+0000 7f39f5389640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.469+0000 7f39f5389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.469+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f39f0137740 con 0x7f39f00a46e0 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.470+0000 7f39e7fff640 1 -- 172.21.15.164:0/807691102 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f39d803c380 con 0x7f39f00a46e0 2024-09-10T16:21:35.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.470+0000 7f39e7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:35.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.470+0000 7f39e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:35.472 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":150,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 msgr2=0x7f39d00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 0x7f39d00406d0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f39e0009f50 tx=0x7f39e0009190 comp rx=0 tx=0).stop 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 10 monclient: shutdown 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 msgr2=0x7f39f00a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f39d8031000 tx=0x7f39d80046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 shutdown_connections 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f39d003e210 0x7f39d00406d0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f39f0137bf0 0x7f39f00a3de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f39f00a46e0 0x7f39f00a38a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 --2- 172.21.15.164:0/807691102 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f39f006cd00 0x7f39f01cd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.472+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 >> 172.21.15.164:0/807691102 conn(0x7f39f00aaa70 msgr2=0x7f39f01bfc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.473+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 shutdown_connections 2024-09-10T16:21:35.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.473+0000 7f39f5389640 1 -- 172.21.15.164:0/807691102 wait complete. 2024-09-10T16:21:35.485 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:21:35.485 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:21:35.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:21:35.485 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:21:35.529 INFO:tasks.rados.rados.0.smithi165.stdout:2321: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.583+0000 7f2085d1f640 10 monclient: get_monmap_and_config 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.583+0000 7f2085d1f640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.583+0000 7f2085d1f640 10 monclient: monmap: 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:35.584646+0000 2024-09-10T16:21:35.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:35.584646+0000 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:35.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.584+0000 7f2085d1f640 1 Processor -- start 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.584+0000 7f2085d1f640 1 -- start start 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.584+0000 7f2085d1f640 10 monclient: init 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2080074af0 0x7f2080137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-a con 0x7f2080074af0 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20800a1090 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 0x7f20800a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-c con 0x7f20800a1500 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f208009ebc0 con 0x7f20800a1090 2024-09-10T16:21:35.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f20800698a0 con 0x7f2080074af0 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f208006bfb0 con 0x7f20800a1500 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f2085d1f640 10 monclient(hunting): authenticate will time out at 1674.072110s 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2080074af0 0x7f2080137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2080074af0 0x7f2080137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48270/0 (socket says 172.21.15.164:48270) 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 -- 172.21.15.164:0/1525303171 learned_addr learned my addr 172.21.15.164:0/1525303171 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): get_auth_request con 0x7f2080074af0 auth_method 0 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 1 --2- 172.21.15.164:0/1525303171 >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 0x7f20800a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 10 monclient(hunting): get_auth_request con 0x7f20800a1500 auth_method 0 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient(hunting): handle_auth_done global_id 164226 payload 274 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 -- 172.21.15.164:0/1525303171 >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 msgr2=0x7f20800a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 --2- 172.21.15.164:0/1525303171 >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 0x7f20800a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 -- 172.21.15.164:0/1525303171 >> v1:172.21.15.165:6789/0 conn(0x7f20800a1090 legacy=0x7f20800a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207f7fe640 1 -- 172.21.15.164:0/1525303171 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800a1950 con 0x7f2080074af0 2024-09-10T16:21:35.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.585+0000 7f207ffff640 1 --2- 172.21.15.164:0/1525303171 >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 0x7f20800a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:21:35.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207f7fe640 1 -- 172.21.15.164:0/1525303171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2080074af0 msgr2=0x7f2080137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207f7fe640 1 --2- 172.21.15.164:0/1525303171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2080074af0 0x7f2080137e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f208006fc40 tx=0x7f207002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea9500780e6605d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 1 -- 172.21.15.164:0/1525303171 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070036070 con 0x7f2080074af0 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: dump: 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 5 monclient: authenticate success, global_id 164226 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 1 -- 172.21.15.164:0/1525303171 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2070007800 con 0x7f2080074af0 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:35.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 4 set_mon_vals no callback set 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 1 -- 172.21.15.164:0/1525303171 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070007bd0 con 0x7f2080074af0 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f207e7fc640 10 monclient: dump: 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.602 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 10 monclient: shutdown 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 -- 172.21.15.164:0/1525303171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2080074af0 msgr2=0x7f2080137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 --2- 172.21.15.164:0/1525303171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2080074af0 0x7f2080137e70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f208006fc40 tx=0x7f207002a570 comp rx=0 tx=0).stop 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 -- 172.21.15.164:0/1525303171 shutdown_connections 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 --2- 172.21.15.164:0/1525303171 >> v2:172.21.15.165:3300/0 conn(0x7f20800a1500 0x7f20800a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 --2- 172.21.15.164:0/1525303171 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2080074af0 0x7f2080137e70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 -- 172.21.15.164:0/1525303171 >> 172.21.15.164:0/1525303171 conn(0x7f20800a3fb0 msgr2=0x7f20800a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 -- 172.21.15.164:0/1525303171 shutdown_connections 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.588+0000 7f2085d1f640 1 -- 172.21.15.164:0/1525303171 wait complete. 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.589+0000 7f2085d1f640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.589+0000 7f2085d1f640 10 monclient: monmap: 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.589+0000 7f2085d1f640 1 Processor -- start 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 -- start start 2024-09-10T16:21:35.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient: init 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 0x7f20801be080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-c con 0x7f2080074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20800a1090 0x7f20801be5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-b con 0x7f20800a1090 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20800a1500 0x7f20801beb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): picked mon.noname-a con 0x7f20800a1500 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2080069370 con 0x7f20800a1500 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2080074ed0 con 0x7f20800a1090 2024-09-10T16:21:35.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f20800a3b10 con 0x7f2080074af0 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f2085d1f640 10 monclient(hunting): authenticate will time out at 1674.077494s 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 0x7f20801be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20800a1500 0x7f20801beb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 0x7f20801be080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34232/0 (socket says 172.21.15.164:34232) 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 -- 172.21.15.164:0/885853418 learned_addr learned my addr 172.21.15.164:0/885853418 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): get_auth_request con 0x7f2080074af0 auth_method 0 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): get_auth_request con 0x7f20800a1500 auth_method 0 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20800a1090 0x7f20801be5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): get_auth_request con 0x7f20800a1090 auth_method 0 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient(hunting): handle_auth_done global_id 174125 payload 824 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20800a1090 msgr2=0x7f20801be5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f20800a1090 0x7f20801be5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20800a1500 msgr2=0x7f20801beb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20800a1500 0x7f20801beb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207f7fe640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20801323a0 con 0x7f2080074af0 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.590+0000 7f207ffff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20800a1500 0x7f20801beb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f207f7fe640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 msgr2=0x7f20801be080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f207f7fe640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 0x7f20801be080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2070000f80 tx=0x7f2070004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070036050 con 0x7f2080074af0 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: dump: 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2070042070 con 0x7f2080074af0 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 5 monclient: authenticate success, global_id 174125 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 10 monclient: _renew_subs 2024-09-10T16:21:35.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f208009e6a0 con 0x7f2080074af0 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2070031e10 con 0x7f2080074af0 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2067fff640 10 monclient: dump: 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 10 monclient: _renew_subs 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20801bf970 con 0x7f2080074af0 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.593+0000 7f207dffb640 4 set_mon_vals no callback set 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2085d1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2085d1f640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 0x7f20801cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2070047460 con 0x7f2080074af0 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20801c1360 con 0x7f20801cd310 2024-09-10T16:21:35.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 0x7f20801cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient: get_auth_request con 0x7f20801cd310 auth_method 0 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2067fff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f206c03e240 0x7f206c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(533..533 src has 481..533) ==== 4847+0+0 (secure 0 0 0) 0x7f2070073e00 con 0x7f2080074af0 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 10 monclient(hunting): handle_auth_done global_id 164232 payload 824 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207effd640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2080137e70 con 0x7f20801cd310 2024-09-10T16:21:35.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207ffff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f206c03e240 0x7f206c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.594+0000 7f207ffff640 10 monclient: get_auth_request con 0x7f206c03e240 auth_method 0 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f207effd640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 msgr2=0x7f20801cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f207effd640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 0x7f20801cd6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f207400fc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074018030 con 0x7f20801cd310 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f2067fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.596+0000 7f2067fff640 10 monclient: dump: 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f207ffff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f206c03e240 0x7f206c040700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20801bf8a0 tx=0x7f2078004730 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f207400d9b0 con 0x7f20801cd310 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 msgr2=0x7f20801cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 0x7f20801cd6f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f207400fc00 comp rx=0 tx=0).stop 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 msgr2=0x7f20801cd6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.597+0000 7f2067fff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cd310 0x7f20801cd6f0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f207400fc00 comp rx=0 tx=0).stop 2024-09-10T16:21:35.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f2085d1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f2085d1f640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 0x7f208009b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f2085d1f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f20801385c0 con 0x7f20801cdc30 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 0x7f208009b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient: get_auth_request con 0x7f20801cdc30 auth_method 0 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 10 monclient(hunting): handle_auth_done global_id 164235 payload 824 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.603+0000 7f207effd640 1 -- 172.21.15.164:0/885853418 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20801c0380 con 0x7f20801cdc30 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.605+0000 7f207effd640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 msgr2=0x7f208009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.605+0000 7f207effd640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 0x7f208009b3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f2074003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.605+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207401dbe0 con 0x7f20801cdc30 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.605+0000 7f2067fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 10 monclient: dump: 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f207401dbe0 con 0x7f20801cdc30 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 msgr2=0x7f208009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 0x7f208009b3b0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f2074003760 comp rx=0 tx=0).stop 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 msgr2=0x7f208009b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2067fff640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f20801cdc30 0x7f208009b3b0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f20801bfe60 tx=0x7f2074003760 comp rx=0 tx=0).stop 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 66308, 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:35.621 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 8, 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.622 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.623 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:35.624 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.625 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.626 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.627 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:35.628 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.629 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.630 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.631 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.632 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:35.633 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:35.634 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:35.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f206c03e240 msgr2=0x7f206c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f206c03e240 0x7f206c040700 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f20801bf8a0 tx=0x7f2078004730 comp rx=0 tx=0).stop 2024-09-10T16:21:35.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 10 monclient: shutdown 2024-09-10T16:21:35.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 msgr2=0x7f20801be080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 --2- 172.21.15.164:0/885853418 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2080074af0 0x7f20801be080 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2070000f80 tx=0x7f2070004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f207ffff640 1 -- 172.21.15.164:0/885853418 reap_dead start 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f207ffff640 1 -- 172.21.15.164:0/885853418 reap_dead start 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 shutdown_connections 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 >> 172.21.15.164:0/885853418 conn(0x7f20800a3fb0 msgr2=0x7f20801328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 shutdown_connections 2024-09-10T16:21:35.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.606+0000 7f2085d1f640 1 -- 172.21.15.164:0/885853418 wait complete. 2024-09-10T16:21:35.636 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:35.636 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:21:35.637 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:21:35.678 INFO:tasks.rados.rados.0.smithi165.stdout:2322: finishing write tid 1 to smithi16534841-17 2024-09-10T16:21:35.678 INFO:tasks.rados.rados.0.smithi165.stdout:2322: finishing write tid 2 to smithi16534841-17 2024-09-10T16:21:35.678 INFO:tasks.rados.rados.0.smithi165.stdout:2322: finishing write tid 3 to smithi16534841-17 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: finishing write tid 4 to smithi16534841-17 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2320: done (6 left) 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2321: done (5 left) 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2330: copy_from oid 33 from oid 47 current snap is 255 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: finishing write tid 5 to smithi16534841-17 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: oid 17 updating version 0 to 1118 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: oid 17 updating version 1118 to 1120 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: oid 17 version 1120 is already newer than 1117 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: oid 17 version 1120 is already newer than 1119 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1120 (ObjNum 817 snap 255 seq_num 817) dirty exists 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2322: left oid 17 (ObjNum 817 snap 255 seq_num 817) 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 1 to smithi16534841-10 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 2 to smithi16534841-10 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 3 to smithi16534841-10 2024-09-10T16:21:35.679 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 4 to smithi16534841-10 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 5 to smithi16534841-10 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: finishing write tid 6 to smithi16534841-10 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: oid 10 updating version 0 to 1123 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: oid 10 updating version 1123 to 1124 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: oid 10 version 1124 is already newer than 1122 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: oid 10 updating version 1124 to 1125 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: oid 10 version 1125 is already newer than 1121 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1125 (ObjNum 818 snap 255 seq_num 818) dirty exists 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2323: left oid 10 (ObjNum 818 snap 255 seq_num 818) 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2325: finishing write tid 1 to smithi16534841-14 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2325: finishing write tid 2 to smithi16534841-14 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2325: finishing write tid 3 to smithi16534841-14 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2325: finishing write tid 4 to smithi16534841-14 2024-09-10T16:21:35.680 INFO:tasks.rados.rados.0.smithi165.stdout:2325: finishing write tid 5 to smithi16534841-14 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: oid 14 updating version 0 to 1196 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: oid 14 version 1196 is already newer than 1195 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: oid 14 updating version 1196 to 1197 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: oid 14 version 1197 is already newer than 1194 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1197 (ObjNum 819 snap 255 seq_num 819) dirty exists 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: left oid 14 (ObjNum 819 snap 255 seq_num 819) 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2328: finishing write tid 1 to smithi16534841-46 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2328: finishing write tid 2 to smithi16534841-46 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2328: finishing write tid 3 to smithi16534841-46 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2328: finishing write tid 4 to smithi16534841-46 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2322: done (5 left) 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2323: done (4 left) 2024-09-10T16:21:35.681 INFO:tasks.rados.rados.0.smithi165.stdout:2325: done (3 left) 2024-09-10T16:21:35.682 INFO:tasks.rados.rados.0.smithi165.stdout:2331: snap_remove snap 237 2024-09-10T16:21:35.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.734+0000 7fc163989640 10 monclient: get_monmap_and_config 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.734+0000 7fc163989640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.734+0000 7fc163989640 10 monclient: monmap: 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:35.735118+0000 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:35.735118+0000 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:35.736 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:35.737 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:35.737 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.734+0000 7fc163989640 1 Processor -- start 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 -- start start 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient: init 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 0x7fc15c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): picked mon.noname-c con 0x7fc15c06d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): picked mon.noname-a con 0x7fc15c06d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): picked mon.noname-b con 0x7fc15c0a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c09cb90 con 0x7fc15c06d540 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c09ce70 con 0x7fc15c06d160 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c09d150 con 0x7fc15c0a4c20 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc163989640 10 monclient(hunting): authenticate will time out at 1674.222564s 2024-09-10T16:21:35.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34244/0 (socket says 172.21.15.164:34244) 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 -- 172.21.15.164:0/515641116 learned_addr learned my addr 172.21.15.164:0/515641116 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48294/0 (socket says 172.21.15.164:48294) 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): get_auth_request con 0x7fc15c0a4c20 auth_method 0 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): get_auth_request con 0x7fc15c06d540 auth_method 0 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161725640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 0x7fc15c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc160f24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161725640 10 monclient(hunting): get_auth_request con 0x7fc15c06d160 auth_method 0 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161725640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161725640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient(hunting): handle_auth_done global_id 174128 payload 274 2024-09-10T16:21:35.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 -- 172.21.15.164:0/515641116 >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 msgr2=0x7fc15c134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 0x7fc15c134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 -- 172.21.15.164:0/515641116 >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 msgr2=0x7fc15c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.735+0000 7fc161f26640 1 monclient: found mon.noname-b 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.736+0000 7fc161f26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.736+0000 7fc161f26640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.736+0000 7fc161f26640 1 -- 172.21.15.164:0/515641116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c135340 con 0x7fc15c0a4c20 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.736+0000 7fc160f24640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.736+0000 7fc161725640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 0x7fc15c134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc161f26640 1 -- 172.21.15.164:0/515641116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 msgr2=0x7fc15c137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc161f26640 1 --2- 172.21.15.164:0/515641116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc15c06d130 tx=0x7fc15402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80959ea13b6e8cfa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 1 -- 172.21.15.164:0/515641116 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154037070 con 0x7fc15c0a4c20 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: dump: 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.746 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 1 -- 172.21.15.164:0/515641116 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc154032640 con 0x7fc15c0a4c20 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc163989640 5 monclient: authenticate success, global_id 174128 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc163989640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 1 -- 172.21.15.164:0/515641116 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154032a10 con 0x7fc15c0a4c20 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc15bfff640 10 monclient: dump: 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.747 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc163989640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.738+0000 7fc163989640 4 set_mon_vals no callback set 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 10 monclient: shutdown 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 -- 172.21.15.164:0/515641116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 msgr2=0x7fc15c137ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 --2- 172.21.15.164:0/515641116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc15c06d130 tx=0x7fc15402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 -- 172.21.15.164:0/515641116 shutdown_connections 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 --2- 172.21.15.164:0/515641116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c137ce0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.164:3300/0 conn(0x7fc15c06d540 0x7fc15c0a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 --2- 172.21.15.164:0/515641116 >> v2:172.21.15.165:3300/0 conn(0x7fc15c06d160 0x7fc15c134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 -- 172.21.15.164:0/515641116 >> 172.21.15.164:0/515641116 conn(0x7fc15c0aacb0 msgr2=0x7fc15c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 -- 172.21.15.164:0/515641116 shutdown_connections 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.739+0000 7fc163989640 1 -- 172.21.15.164:0/515641116 wait complete. 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient: monmap: 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:35.749 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 Processor -- start 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 -- start start 2024-09-10T16:21:35.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient: init 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 0x7fc15c130080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): picked mon.noname-a con 0x7fc15c06d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc15c06d540 0x7fc15c1305c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): picked mon.noname-b con 0x7fc15c06d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c130b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): picked mon.noname-c con 0x7fc15c0a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c1385b0 con 0x7fc15c06d160 2024-09-10T16:21:35.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c138430 con 0x7fc15c06d540 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c138730 con 0x7fc15c0a4c20 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc163989640 10 monclient(hunting): authenticate will time out at 1674.227860s 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 0x7fc15c130080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161f26640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c130b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 0x7fc15c130080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48298/0 (socket says 172.21.15.164:48298) 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 1 -- 172.21.15.164:0/2176929928 learned_addr learned my addr 172.21.15.164:0/2176929928 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 10 monclient(hunting): get_auth_request con 0x7fc15c06d160 auth_method 0 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161725640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161f26640 10 monclient(hunting): get_auth_request con 0x7fc15c0a4c20 auth_method 0 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161f26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161f26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.740+0000 7fc161f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc160f24640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc15c06d540 0x7fc15c1305c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient(hunting): handle_auth_done global_id 174131 payload 824 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc15c06d540 msgr2=0x7fc15c1305c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc15c06d540 0x7fc15c1305c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 msgr2=0x7fc15c130080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 0x7fc15c130080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 monclient: found mon.noname-c 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161f26640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c06e6b0 con 0x7fc15c0a4c20 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.741+0000 7fc161725640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc15c06d160 0x7fc15c130080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc161f26640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 msgr2=0x7fc15c130b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc161f26640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c130b00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc154031000 tx=0x7fc1540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154041020 con 0x7fc15c0a4c20 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: dump: 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc15403d850 con 0x7fc15c0a4c20 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 5 monclient: authenticate success, global_id 174131 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 10 monclient: _renew_subs 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c1361c0 con 0x7fc15c0a4c20 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc154037040 con 0x7fc15c0a4c20 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc159ffb640 10 monclient: dump: 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 10 monclient: _renew_subs 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc15b7fe640 4 set_mon_vals no callback set 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.743+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c09efb0 con 0x7fc15c0a4c20 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc163989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc163989640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 0x7fc15c1cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c0a4210 con 0x7fc15c1cce90 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc154044020 con 0x7fc15c0a4c20 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 0x7fc15c1cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient: get_auth_request con 0x7fc15c1cce90 auth_method 0 2024-09-10T16:21:35.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.744+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc161725640 10 monclient(hunting): handle_auth_done global_id 174134 payload 824 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc159ffb640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc14803e260 0x7fc148040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc161725640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc15c0a46e0 con 0x7fc15c1cce90 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(533..533 src has 481..533) ==== 4847+0+0 (secure 0 0 0) 0x7fc154073120 con 0x7fc15c0a4c20 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc160f24640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc14803e260 0x7fc148040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.745+0000 7fc160f24640 10 monclient: get_auth_request con 0x7fc14803e260 auth_method 0 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc161725640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 msgr2=0x7fc15c1cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc161725640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 0x7fc15c1cd270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc14c003f20 tx=0x7fc14c004410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c019020 con 0x7fc15c1cce90 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 10 monclient: dump: 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.765 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc14c015b40 con 0x7fc15c1cce90 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 msgr2=0x7fc15c1cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 0x7fc15c1cd270 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc14c003f20 tx=0x7fc14c004410 comp rx=0 tx=0).stop 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 msgr2=0x7fc15c1cd270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc159ffb640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cce90 0x7fc15c1cd270 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.747+0000 7fc160f24640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc14803e260 0x7fc148040720 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc150009560 tx=0x7fc150008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc163989640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc163989640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 0x7fc15c0a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc163989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c0aa1e0 con 0x7fc15c1cd7b0 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 0x7fc15c0a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient: get_auth_request con 0x7fc15c1cd7b0 auth_method 0 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.753+0000 7fc161725640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.754+0000 7fc161725640 10 monclient(hunting): handle_auth_done global_id 174137 payload 824 2024-09-10T16:21:35.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.754+0000 7fc161725640 1 -- 172.21.15.164:0/2176929928 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc15c09ed80 con 0x7fc15c1cd7b0 2024-09-10T16:21:35.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc161725640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 msgr2=0x7fc15c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc161725640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 0x7fc15c0a5300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc14c003f20 tx=0x7fc14c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c019030 con 0x7fc15c1cd7b0 2024-09-10T16:21:35.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 10 monclient: dump: 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fc14c020220 con 0x7fc15c1cd7b0 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 msgr2=0x7fc15c0a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 0x7fc15c0a5300 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc14c003f20 tx=0x7fc14c004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 msgr2=0x7fc15c0a5300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.756+0000 7fc159ffb640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c1cd7b0 0x7fc15c0a5300 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fc14c003f20 tx=0x7fc14c004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:35.770 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 13398, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 8, 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:35.771 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.772 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.773 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.774 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.775 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.776 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.777 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.779 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:35.780 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.781 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.782 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:35.783 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:35.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc14803e260 msgr2=0x7fc148040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc14803e260 0x7fc148040720 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc150009560 tx=0x7fc150008040 comp rx=0 tx=0).stop 2024-09-10T16:21:35.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 10 monclient: shutdown 2024-09-10T16:21:35.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 msgr2=0x7fc15c130b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c130b00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc154031000 tx=0x7fc1540046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc161f26640 1 -- 172.21.15.164:0/2176929928 reap_dead start 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 shutdown_connections 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 --2- 172.21.15.164:0/2176929928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc15c0a4c20 0x7fc15c130b00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 >> 172.21.15.164:0/2176929928 conn(0x7fc15c0aacb0 msgr2=0x7fc15c132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 shutdown_connections 2024-09-10T16:21:35.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.757+0000 7fc163989640 1 -- 172.21.15.164:0/2176929928 wait complete. 2024-09-10T16:21:35.785 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:35.785 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:21:35.786 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:21:35.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.882+0000 7f90f72e2640 10 monclient: get_monmap_and_config 2024-09-10T16:21:35.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.882+0000 7f90f72e2640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.882+0000 7f90f72e2640 10 monclient: monmap: 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:35.883822+0000 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:35.883822+0000 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:35.885 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:35.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.883+0000 7f90f72e2640 1 Processor -- start 2024-09-10T16:21:35.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.883+0000 7f90f72e2640 1 -- start start 2024-09-10T16:21:35.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.883+0000 7f90f72e2640 10 monclient: init 2024-09-10T16:21:35.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:21:35.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90f006d160 0x7f90f0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-a con 0x7f90f006d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:35.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-b con 0x7f90f006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f90f00a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f90f009cb90 con 0x7f90f00a4c20 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f90f009ce70 con 0x7f90f006d160 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f90f009d150 con 0x7f90f006d540 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f72e2640 10 monclient(hunting): authenticate will time out at 1674.371243s 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90f006d160 0x7f90f0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34274/0 (socket says 172.21.15.164:34274) 2024-09-10T16:21:35.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 1 -- 172.21.15.164:0/449638506 learned_addr learned my addr 172.21.15.164:0/449638506 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f90f006d160 0x7f90f0134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48302/0 (socket says 172.21.15.164:48302) 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): get_auth_request con 0x7f90f006d540 auth_method 0 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f587f640 1 --1- 172.21.15.164:0/449638506 >> v1:172.21.15.164:6789/0 conn(0x7f90f00a4c20 0x7f90f0137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42054/0 (socket says 172.21.15.164:42054) 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): get_auth_request con 0x7f90f006d160 auth_method 0 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient(hunting): handle_auth_done global_id 164244 payload 274 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 -- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 msgr2=0x7f90f00a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 -- 172.21.15.164:0/449638506 >> v1:172.21.15.164:6789/0 conn(0x7f90f00a4c20 legacy=0x7f90f0137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 monclient: found mon.noname-a 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f507e640 1 -- 172.21.15.164:0/449638506 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f0135340 con 0x7f90f006d160 2024-09-10T16:21:35.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.884+0000 7f90f487d640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:35.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f507e640 1 -- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 msgr2=0x7f90f0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f507e640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f0134e00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097a0 tx=0x7f90e8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaae0e1ce78ba7a6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 1 -- 172.21.15.164:0/449638506 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8037070 con 0x7f90f006d160 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: dump: 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.899 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 1 -- 172.21.15.164:0/449638506 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e8032440 con 0x7f90f006d160 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f72e2640 5 monclient: authenticate success, global_id 164244 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f72e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 1 -- 172.21.15.164:0/449638506 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8032810 con 0x7f90f006d160 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f72e2640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90f72e2640 4 set_mon_vals no callback set 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.887+0000 7f90effff640 10 monclient: dump: 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.901 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 10 monclient: shutdown 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 -- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 msgr2=0x7f90f0134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f0134e00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f90e80097a0 tx=0x7f90e8029fd0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 -- 172.21.15.164:0/449638506 shutdown_connections 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f00a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 --2- 172.21.15.164:0/449638506 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f0134e00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 -- 172.21.15.164:0/449638506 >> 172.21.15.164:0/449638506 conn(0x7f90f00aacb0 msgr2=0x7f90f00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 -- 172.21.15.164:0/449638506 shutdown_connections 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 1 -- 172.21.15.164:0/449638506 wait complete. 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 10 monclient: build_initial_monmap 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.888+0000 7f90f72e2640 10 monclient: monmap: 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:35.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 Processor -- start 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 -- start start 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient: init 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f01cd250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-a con 0x7f90f006d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f01cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-c con 0x7f90f006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f00a4c20 0x7f90f01cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): picked mon.noname-b con 0x7f90f00a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f90f0138870 con 0x7f90f006d160 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90f01386f0 con 0x7f90f00a4c20 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f90f01389f0 con 0x7f90f006d540 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f72e2640 10 monclient(hunting): authenticate will time out at 1674.376770s 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f01cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f01cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f01cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48308/0 (socket says 172.21.15.164:48308) 2024-09-10T16:21:35.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 1 -- 172.21.15.164:0/3524815581 learned_addr learned my addr 172.21.15.164:0/3524815581 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): get_auth_request con 0x7f90f006d160 auth_method 0 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f01cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34290/0 (socket says 172.21.15.164:34290) 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): get_auth_request con 0x7f90f006d540 auth_method 0 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.889+0000 7f90f487d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f00a4c20 0x7f90f01cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 10 monclient(hunting): get_auth_request con 0x7f90f00a4c20 auth_method 0 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 10 monclient(hunting): handle_auth_done global_id 174143 payload 824 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f00a4c20 msgr2=0x7f90f01cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f00a4c20 0x7f90f01cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 msgr2=0x7f90f01cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f01cd250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 10 monclient: _finish_hunting 0 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 monclient: found mon.noname-c 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f487d640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90f013e870 con 0x7f90f006d540 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f507e640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f90f006d160 0x7f90f01cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.890+0000 7f90f587f640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f00a4c20 0x7f90f01cdcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f487d640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 msgr2=0x7f90f01cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f487d640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f01cd790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f90e400e930 tx=0x7f90e400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e400cc10 con 0x7f90f006d540 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: dump: 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: _finish_auth 0 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: _check_auth_tickets 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90e400cdb0 con 0x7f90f006d540 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 5 monclient: authenticate success, global_id 174143 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e4004270 con 0x7f90f006d540 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 10 monclient: _renew_subs 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90f013eb50 con 0x7f90f006d540 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90edffb640 10 monclient: dump: 2024-09-10T16:21:35.910 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90ef7fe640 4 set_mon_vals no callback set 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 10 monclient: _renew_subs 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:21:35.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.892+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f90f013e590 con 0x7f90f006d540 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f72e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f90e4003610 con 0x7f90f006d540 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f72e2640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 0x7f90f012e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90f0134e00 con 0x7f90f012dfb0 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90edffb640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90dc03e240 0x7f90dc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 0x7f90f012e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(533..533 src has 481..533) ==== 4847+0+0 (secure 0 0 0) 0x7f90e4052d70 con 0x7f90f006d540 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient: get_auth_request con 0x7f90f012dfb0 auth_method 0 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f587f640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90dc03e240 0x7f90dc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.893+0000 7f90f587f640 10 monclient: get_auth_request con 0x7f90dc03e240 auth_method 0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.894+0000 7f90f507e640 10 monclient(hunting): handle_auth_done global_id 154318 payload 824 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.894+0000 7f90f507e640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f90f00aa1e0 con 0x7f90f012dfb0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90f507e640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 msgr2=0x7f90f012e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90f507e640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 0x7f90f012e390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f90f013ce60 tx=0x7f90e80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90f587f640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90dc03e240 0x7f90dc040700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f90f006d9a0 tx=0x7f90e0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8043070 con 0x7f90f012dfb0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 10 monclient: dump: 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f90e803ca60 con 0x7f90f012dfb0 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:35.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:35.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 msgr2=0x7f90f012e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 0x7f90f012e390 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f90f013ce60 tx=0x7f90e80046c0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 msgr2=0x7f90f012e390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.896+0000 7f90edffb640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012dfb0 0x7f90f012e390 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.902+0000 7f90f72e2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f72e2640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 0x7f90f01301e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f72e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f90f013f970 con 0x7f90f012e8d0 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 0x7f90f01301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient: get_auth_request con 0x7f90f012e8d0 auth_method 0 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:35.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 10 monclient(hunting): handle_auth_done global_id 154321 payload 824 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.903+0000 7f90f507e640 1 -- 172.21.15.164:0/3524815581 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f90f00770b0 con 0x7f90f012e8d0 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f507e640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 msgr2=0x7f90f01301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f507e640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 0x7f90f01301e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f90f0134f80 tx=0x7f90e8008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90e8037040 con 0x7f90f012e8d0 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 10 monclient: dump: 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:35.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f90e8003c40 con 0x7f90f012e8d0 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 msgr2=0x7f90f01301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 0x7f90f01301e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f90f0134f80 tx=0x7f90e8008030 comp rx=0 tx=0).stop 2024-09-10T16:21:35.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 msgr2=0x7f90f01301e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90edffb640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f90f012e8d0 0x7f90f01301e0 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f90f0134f80 tx=0x7f90e8008030 comp rx=0 tx=0).stop 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 150, 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 114457, 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:35.920 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 8, 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.921 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.922 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:35.923 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:35.924 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.925 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.926 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.927 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.928 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:35.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.930 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.931 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.932 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:35.933 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90dc03e240 msgr2=0x7f90dc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f90dc03e240 0x7f90dc040700 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f90f006d9a0 tx=0x7f90e0009040 comp rx=0 tx=0).stop 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 10 monclient: shutdown 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 msgr2=0x7f90f01cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 --2- 172.21.15.164:0/3524815581 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f90f006d540 0x7f90f01cd790 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f90e400e930 tx=0x7f90e400edf0 comp rx=0 tx=0).stop 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f587f640 1 -- 172.21.15.164:0/3524815581 reap_dead start 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f587f640 1 -- 172.21.15.164:0/3524815581 reap_dead start 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 shutdown_connections 2024-09-10T16:21:35.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 >> 172.21.15.164:0/3524815581 conn(0x7f90f00aacb0 msgr2=0x7f90f0132660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:35.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 shutdown_connections 2024-09-10T16:21:35.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:35.906+0000 7f90f72e2640 1 -- 172.21.15.164:0/3524815581 wait complete. 2024-09-10T16:21:35.936 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:21:35.936 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:21:35.936 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-10T16:21:35.936 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:21:35.936 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:21:35.937 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:21:35.937 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:21:35.947 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:21:35.947 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:21:35.947 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:21:35.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:21:50.948 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:21:50.949 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:21:50.950 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:21:50.950 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:21:51.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 10 monclient: get_monmap_and_config 2024-09-10T16:21:51.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 10 monclient: build_initial_monmap 2024-09-10T16:21:51.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 10 monclient: monmap: 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:51.058072+0000 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:51.058072+0000 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:51.059 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:51.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 1 Processor -- start 2024-09-10T16:21:51.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 1 -- start start 2024-09-10T16:21:51.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.056+0000 7fde98b6b640 10 monclient: init 2024-09-10T16:21:51.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient: _add_conns ranks=[0,2,1,4,3] 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde9406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-b con 0x7fde94134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde9413d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9409ce50 con 0x7fde9406cb70 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde94135380 con 0x7fde9413d560 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fde9406db60 con 0x7fde94134f70 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde98b6b640 10 monclient(hunting): authenticate will time out at 1689.545600s 2024-09-10T16:21:51.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde91d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 msgr2=0x7fde9413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:51.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde91d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:51.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde9259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fde9406cb70 0x7fde940a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45856/0 (socket says 172.21.15.164:45856) 2024-09-10T16:21:51.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.057+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 learned_addr learned my addr 172.21.15.164:0/3059335311 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:51.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.059+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.164:6789/0 conn(0x7fde9406cb70 legacy=0x7fde940a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:51.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.060+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.165:6789/0 conn(0x7fde9413d560 legacy=0x7fde9413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:51.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.259+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 msgr2=0x7fde9413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:51.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.259+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.660+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 msgr2=0x7fde9413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:51.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:51.660+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:52.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:52.462+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 msgr2=0x7fde9413d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:52.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:52.462+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): tick 2024-09-10T16:21:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:21:54.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 msgr2=0x7fde9413d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde94134f70 0x7fde9413d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.165:6789/0 conn(0x7fde9413d560 legacy=0x7fde9413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.164:6789/0 conn(0x7fde9406cb70 legacy=0x7fde940a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fde74003810 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): picked mon.noname-c con 0x7fde74006380 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fde74008e90 con 0x7fde74003810 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:54.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fde74009040 con 0x7fde74006380 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde8bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 10 monclient(hunting): get_auth_request con 0x7fde74003810 auth_method 0 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 10 monclient(hunting): get_auth_request con 0x7fde74006380 auth_method 0 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.058+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:54.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 10 monclient(hunting): get_auth_request con 0x7fde74003810 auth_method 0 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 10 monclient(hunting): get_auth_request con 0x7fde74006380 auth_method 0 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.259+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:21:54.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 10 monclient(hunting): get_auth_request con 0x7fde74003810 auth_method 0 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:54.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 10 monclient(hunting): get_auth_request con 0x7fde74006380 auth_method 0 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:54.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:54.660+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 10 monclient(hunting): get_auth_request con 0x7fde74003810 auth_method 0 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.461+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde9259c640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:55.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 10 monclient(hunting): get_auth_request con 0x7fde74006380 auth_method 0 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:21:55.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:55.462+0000 7fde91d9b640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:21:56.975 INFO:tasks.rados.rados.0.smithi165.stdout:2331: done (3 left) 2024-09-10T16:21:56.975 INFO:tasks.rados.rados.0.smithi165.stdout:2332: write oid 44 current snap is 255 2024-09-10T16:21:56.975 INFO:tasks.rados.rados.0.smithi165.stdout:2332: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2024-09-10T16:21:56.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: writing smithi16534841-44 from 598547 to 1227747 tid 1 2024-09-10T16:21:56.988 INFO:tasks.rados.rados.0.smithi165.stdout:2332: writing smithi16534841-44 from 1876540 to 2368486 tid 2 2024-09-10T16:21:56.997 INFO:tasks.rados.rados.0.smithi165.stdout:2332: writing smithi16534841-44 from 3046277 to 3727776 tid 3 2024-09-10T16:21:56.997 INFO:tasks.rados.rados.0.smithi165.stdout:2333: read oid 23 snap 230 2024-09-10T16:21:56.997 INFO:tasks.rados.rados.0.smithi165.stdout:2333: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:21:56.997 INFO:tasks.rados.rados.0.smithi165.stdout:2334: snap_remove snap 242 2024-09-10T16:21:57.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): tick 2024-09-10T16:21:57.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:21:57.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 monclient(hunting): continuing hunt 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 msgr2=0x7fde74008800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.165:3300/0 conn(0x7fde74006380 0x7fde74008800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 msgr2=0x7fde74005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde74003810 0x7fde74005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 reap_dead start 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient: _add_conns ranks=[0,1,3,2,4] 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde74003c80 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde74007f10 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde7400c4b0 0x7fde7400e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): picked mon.noname-a con 0x7fde7400c4b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:57.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde9409cb70 con 0x7fde74003c80 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde7400fdf0 con 0x7fde74007f10 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 1 -- 172.21.15.164:0/3059335311 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fde74008e90 con 0x7fde7400c4b0 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde8bfff640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 --2- 172.21.15.164:0/3059335311 >> v2:172.21.15.164:3300/0 conn(0x7fde7400c4b0 0x7fde7400e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): get_auth_request con 0x7fde7400c4b0 auth_method 0 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient(hunting): handle_auth_done global_id 164250 payload 274 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.165:6789/0 conn(0x7fde74007f10 legacy=0x7fde740082f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 >> v1:172.21.15.164:6789/0 conn(0x7fde74003c80 legacy=0x7fde740067f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 monclient: found mon.noname-a 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.059+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde74011020 con 0x7fde7400c4b0 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde92d9d640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde7400c4b0 msgr2=0x7fde7400e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde92d9d640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde7400c4b0 0x7fde7400e8a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde940a6cc0 tx=0x7fde8c029f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f968dbcac0cc3ba6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 1 -- 172.21.15.164:0/3059335311 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c007560 con 0x7fde7400c4b0 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: dump: 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 1 -- 172.21.15.164:0/3059335311 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde8c02e070 con 0x7fde7400c4b0 2024-09-10T16:21:57.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 1 -- 172.21.15.164:0/3059335311 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8c02aa50 con 0x7fde7400c4b0 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde98b6b640 5 monclient: authenticate success, global_id 164250 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde98b6b640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde98b6b640 4 set_mon_vals no callback set 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.062+0000 7fde9159a640 10 monclient: dump: 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.068 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 10 monclient: shutdown 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 -- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde7400c4b0 msgr2=0x7fde7400e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde7400c4b0 0x7fde7400e8a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fde940a6cc0 tx=0x7fde8c029f50 comp rx=0 tx=0).stop 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 -- 172.21.15.164:0/3059335311 shutdown_connections 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 --2- 172.21.15.164:0/3059335311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde7400c4b0 0x7fde7400e8a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 -- 172.21.15.164:0/3059335311 >> 172.21.15.164:0/3059335311 conn(0x7fde940aa900 msgr2=0x7fde940aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 -- 172.21.15.164:0/3059335311 shutdown_connections 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.063+0000 7fde98b6b640 1 -- 172.21.15.164:0/3059335311 wait complete. 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.064+0000 7fde98b6b640 10 monclient: build_initial_monmap 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.064+0000 7fde98b6b640 10 monclient: monmap: 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.064+0000 7fde98b6b640 1 Processor -- start 2024-09-10T16:21:57.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 -- start start 2024-09-10T16:21:57.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient: init 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-a con 0x7fde74003c80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 0x7fde9409e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-c con 0x7fde74007f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 0x7fde9409e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): picked mon.noname-b con 0x7fde7400c4b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fde940a46a0 con 0x7fde74003c80 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fde9406cf50 con 0x7fde7400c4b0 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fde940a4820 con 0x7fde74007f10 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde98b6b640 10 monclient(hunting): authenticate will time out at 1695.552621s 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde91d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 msgr2=0x7fde9409e180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde91d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 0x7fde9409e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39364/0 (socket says 172.21.15.164:39364) 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 -- 172.21.15.164:0/2274482917 learned_addr learned my addr 172.21.15.164:0/2274482917 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): get_auth_request con 0x7fde74003c80 auth_method 0 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde92d9d640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 0x7fde9409e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde92d9d640 10 monclient(hunting): get_auth_request con 0x7fde7400c4b0 auth_method 0 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde92d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde92d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde92d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient(hunting): handle_auth_done global_id 164253 payload 824 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 -- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 msgr2=0x7fde9409e180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 0x7fde9409e180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 -- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 msgr2=0x7fde9409e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 0x7fde9409e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 1 monclient: found mon.noname-a 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.065+0000 7fde9259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.066+0000 7fde9259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.066+0000 7fde9259c640 1 -- 172.21.15.164:0/2274482917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde940a1a60 con 0x7fde74003c80 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.066+0000 7fde92d9d640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 0x7fde9409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde9259c640 1 -- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 msgr2=0x7fde9409dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde9259c640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde84007b50 tx=0x7fde84001ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde8401b020 con 0x7fde74003c80 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: dump: 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.079 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde84014890 con 0x7fde74003c80 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 5 monclient: authenticate success, global_id 164253 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 10 monclient: _renew_subs 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde940a9fe0 con 0x7fde74003c80 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde84014c60 con 0x7fde74003c80 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8affd640 10 monclient: dump: 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde8bfff640 4 set_mon_vals no callback set 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 10 monclient: _renew_subs 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.068+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde9409f700 con 0x7fde74003c80 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fde8401e020 con 0x7fde74003c80 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde98b6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde98b6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde98b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fde940a4c50 con 0x7fde74003c80 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde8affd640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 0x7fde74042740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.069+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(534..534 src has 481..534) ==== 4863+0+0 (secure 0 0 0) 0x7fde84053a70 con 0x7fde74003c80 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.070+0000 7fde91d9b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 0x7fde74042740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.070+0000 7fde91d9b640 10 monclient: get_auth_request con 0x7fde74040280 auth_method 0 2024-09-10T16:21:57.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.072+0000 7fde91d9b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 0x7fde74042740 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fde940a1040 tx=0x7fde7c005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.074+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fde84025020 con 0x7fde74003c80 2024-09-10T16:21:57.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.074+0000 7fde8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.074+0000 7fde8affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:57.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.317+0000 7fde98b6b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:21:57.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.317+0000 7fde98b6b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:57.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.317+0000 7fde98b6b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.317+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fde940770f0 con 0x7fde74003c80 2024-09-10T16:21:57.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.318+0000 7fde8affd640 1 -- 172.21.15.164:0/2274482917 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fde9409ce50 con 0x7fde74003c80 2024-09-10T16:21:57.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.318+0000 7fde8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:21:57.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.318+0000 7fde8affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:57.320 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":154,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:21:57.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 msgr2=0x7fde74042740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 0x7fde74042740 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fde940a1040 tx=0x7fde7c005040 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 10 monclient: shutdown 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 msgr2=0x7fde9409dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fde84007b50 tx=0x7fde84001ee0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 shutdown_connections 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fde74040280 0x7fde74042740 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fde7400c4b0 0x7fde9409e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fde74007f10 0x7fde9409e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 --2- 172.21.15.164:0/2274482917 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fde74003c80 0x7fde9409dc40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 >> 172.21.15.164:0/2274482917 conn(0x7fde940aa900 msgr2=0x7fde940a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 shutdown_connections 2024-09-10T16:21:57.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.320+0000 7fde98b6b640 1 -- 172.21.15.164:0/2274482917 wait complete. 2024-09-10T16:21:57.332 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:21:57.333 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:21:57.333 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.429+0000 7f7eccfa0640 10 monclient: get_monmap_and_config 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.429+0000 7f7eccfa0640 10 monclient: build_initial_monmap 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.429+0000 7f7eccfa0640 10 monclient: monmap: 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:57.430808+0000 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:57.430808+0000 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:57.432 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.430+0000 7f7eccfa0640 1 Processor -- start 2024-09-10T16:21:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.430+0000 7f7eccfa0640 1 -- start start 2024-09-10T16:21:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.430+0000 7f7eccfa0640 10 monclient: init 2024-09-10T16:21:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7ec809f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7ec80a2550 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ec80a2980 0x7f7ec813ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec80a2980 addr v2:172.21.15.164:3300/0 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec809fd10 con 0x7f7ec80a2550 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7ec809cd20 con 0x7f7ec809f930 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7ec809a0d0 con 0x7f7ec80a2980 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7eccfa0640 10 monclient(hunting): authenticate will time out at 1695.918245s 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7ec80a2980 0x7f7ec813ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec5d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7ec80a2550 0x7f7ec813b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:43924/0 (socket says 172.21.15.164:43924) 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec5d9b640 1 -- 172.21.15.164:0/4286114857 learned_addr learned my addr 172.21.15.164:0/4286114857 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): get_auth_request con 0x7f7ec80a2980 auth_method 0 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient(hunting): handle_auth_done global_id 164256 payload 274 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/4286114857 >> v1:172.21.15.165:6789/0 conn(0x7f7ec809f930 legacy=0x7f7ec80a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/4286114857 >> v1:172.21.15.164:6789/0 conn(0x7f7ec80a2550 legacy=0x7f7ec813b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 1 monclient: found mon.noname-a 2024-09-10T16:21:57.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.431+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/4286114857 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec809e840 con 0x7f7ec80a2980 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/4286114857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec80a2980 msgr2=0x7f7ec813ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec6d9d640 1 --2- 172.21.15.164:0/4286114857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec80a2980 0x7f7ec813ed00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7ec809cef0 tx=0x7f7ec002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c283becda4d50b2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 1 -- 172.21.15.164:0/4286114857 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0007a30 con 0x7f7ec80a2980 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: dump: 2024-09-10T16:21:57.443 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 1 -- 172.21.15.164:0/4286114857 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ec0007bd0 con 0x7f7ec80a2980 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7eccfa0640 5 monclient: authenticate success, global_id 164256 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7eccfa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7eccfa0640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7eccfa0640 4 set_mon_vals no callback set 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 1 -- 172.21.15.164:0/4286114857 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec0031400 con 0x7f7ec80a2980 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.434+0000 7f7ec559a640 10 monclient: dump: 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.445 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 10 monclient: shutdown 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 -- 172.21.15.164:0/4286114857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec80a2980 msgr2=0x7f7ec813ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/4286114857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec80a2980 0x7f7ec813ed00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f7ec809cef0 tx=0x7f7ec002a480 comp rx=0 tx=0).stop 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 -- 172.21.15.164:0/4286114857 shutdown_connections 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/4286114857 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec80a2980 0x7f7ec813ed00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 -- 172.21.15.164:0/4286114857 >> 172.21.15.164:0/4286114857 conn(0x7f7ec80a3110 msgr2=0x7f7ec80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 -- 172.21.15.164:0/4286114857 shutdown_connections 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 1 -- 172.21.15.164:0/4286114857 wait complete. 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 10 monclient: build_initial_monmap 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.435+0000 7f7eccfa0640 10 monclient: monmap: 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:57.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 Processor -- start 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 -- start start 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient: init 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 0x7f7ec81cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-a con 0x7f7ec809f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ec80a2550 0x7f7ec81cdca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-c con 0x7f7ec80a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ec80a2980 0x7f7ec809a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): picked mon.noname-b con 0x7f7ec80a2980 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec809d930 con 0x7f7ec809f930 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec809ebe0 con 0x7f7ec80a2980 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7ec809c8b0 con 0x7f7ec80a2550 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7eccfa0640 10 monclient(hunting): authenticate will time out at 1695.923800s 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec5d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ec80a2550 msgr2=0x7f7ec81cdca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec5d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ec80a2550 0x7f7ec81cdca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:57.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 0x7f7ec81cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 0x7f7ec81cd760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39376/0 (socket says 172.21.15.164:39376) 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 1 -- 172.21.15.164:0/2527041126 learned_addr learned my addr 172.21.15.164:0/2527041126 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): get_auth_request con 0x7f7ec809f930 auth_method 0 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.436+0000 7f7ec659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 10 monclient(hunting): handle_auth_done global_id 164259 payload 824 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec6d9d640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ec80a2980 0x7f7ec809a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ec80a2550 msgr2=0x7f7ec81cdca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:57.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7ec80a2550 0x7f7ec81cdca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ec80a2980 msgr2=0x7f7ec809a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ec80a2980 0x7f7ec809a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 monclient: found mon.noname-a 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec659c640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ec81b7920 con 0x7f7ec809f930 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec6d9d640 10 monclient: get_auth_request con 0x7f7ec80a2980 auth_method 0 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.437+0000 7f7ec6d9d640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7ec80a2980 0x7f7ec809a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7ec659c640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 msgr2=0x7f7ec81cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7ec659c640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 0x7f7ec81cd760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800c430 tx=0x7f7eb800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb80119a0 con 0x7f7ec809f930 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: dump: 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7eb8017070 con 0x7f7ec809f930 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 5 monclient: authenticate success, global_id 164259 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 10 monclient: _renew_subs 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ec809c100 con 0x7f7ec809f930 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7eb8011d80 con 0x7f7ec809f930 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eb6ffd640 10 monclient: dump: 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.453 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 10 monclient: _renew_subs 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ec81b7640 con 0x7f7ec809f930 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.439+0000 7f7ec4d99640 4 set_mon_vals no callback set 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7eb801c460 con 0x7f7ec809f930 2024-09-10T16:21:57.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eccfa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 0x7f7ec809ad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec80a0760 con 0x7f7ec809a9a0 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 0x7f7ec809ad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient: get_auth_request con 0x7f7ec809a9a0 auth_method 0 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eb6ffd640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ea803e1f0 0x7f7ea80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(534..534 src has 481..534) ==== 4863+0+0 (secure 0 0 0) 0x7f7eb8052d40 con 0x7f7ec809f930 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec6d9d640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ea803e1f0 0x7f7ea80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec6d9d640 10 monclient: get_auth_request con 0x7f7ea803e1f0 auth_method 0 2024-09-10T16:21:57.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_done global_id 164262 payload 824 2024-09-10T16:21:57.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.440+0000 7f7ec5d9b640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ec81bb1f0 con 0x7f7ec809a9a0 2024-09-10T16:21:57.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7ec5d9b640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 msgr2=0x7f7ec809ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7ec5d9b640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 0x7f7ec809ad80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ec81b6000 tx=0x7f7ebc00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc017070 con 0x7f7ec809a9a0 2024-09-10T16:21:57.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 10 monclient: dump: 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7ec6d9d640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ea803e1f0 0x7f7ea80406b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0034000 tx=0x7f7ec002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ebc007070 con 0x7f7ec809a9a0 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 msgr2=0x7f7ec809ad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 0x7f7ec809ad80 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ec81b6000 tx=0x7f7ebc00c8e0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 msgr2=0x7f7ec809ad80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.443+0000 7f7eb6ffd640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809a9a0 0x7f7ec809ad80 secure :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f7ec81b6000 tx=0x7f7ebc00c8e0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7eccfa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 0x7f7ec8071480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7eccfa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7ec8077260 con 0x7f7ec809b2c0 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 0x7f7ec8071480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient: get_auth_request con 0x7f7ec809b2c0 auth_method 0 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 10 monclient(hunting): handle_auth_done global_id 164265 payload 824 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.449+0000 7f7ec5d9b640 1 -- 172.21.15.164:0/2527041126 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7ec81ce1e0 con 0x7f7ec809b2c0 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7ec5d9b640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 msgr2=0x7f7ec8071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7ec5d9b640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 0x7f7ec8071480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc012000 tx=0x7f7ebc004a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ebc017030 con 0x7f7ec809b2c0 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 10 monclient: dump: 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f7ebc01ad50 con 0x7f7ec809b2c0 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 msgr2=0x7f7ec8071480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 0x7f7ec8071480 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc012000 tx=0x7f7ebc004a10 comp rx=0 tx=0).stop 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 msgr2=0x7f7ec8071480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eb6ffd640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809b2c0 0x7f7ec8071480 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7ebc012000 tx=0x7f7ebc004a10 comp rx=0 tx=0).stop 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 154, 2024-09-10T16:21:57.463 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 88155, 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.464 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:57.465 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:57.467 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.468 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.469 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.470 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.471 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:57.472 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.473 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.474 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.475 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:57.476 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ea803e1f0 msgr2=0x7f7ea80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7ea803e1f0 0x7f7ea80406b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7ec0034000 tx=0x7f7ec002f000 comp rx=0 tx=0).stop 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 10 monclient: shutdown 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 msgr2=0x7f7ec81cd760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 --2- 172.21.15.164:0/2527041126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7ec809f930 0x7f7ec81cd760 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f7eb800c430 tx=0x7f7eb800c8f0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/2527041126 reap_dead start 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7ec6d9d640 1 -- 172.21.15.164:0/2527041126 reap_dead start 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 shutdown_connections 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 >> 172.21.15.164:0/2527041126 conn(0x7f7ec80a3110 msgr2=0x7f7ec80a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 shutdown_connections 2024-09-10T16:21:57.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.452+0000 7f7eccfa0640 1 -- 172.21.15.164:0/2527041126 wait complete. 2024-09-10T16:21:57.482 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:21:57.482 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:21:57.482 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.579+0000 7f3f361b4640 10 monclient: get_monmap_and_config 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.579+0000 7f3f361b4640 10 monclient: build_initial_monmap 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.579+0000 7f3f361b4640 10 monclient: monmap: 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:21:57.580424+0000 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:21:57.580424+0000 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:21:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:21:57.582 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:21:57.582 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:21:57.582 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:21:57.582 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 Processor -- start 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 -- start start 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient: init 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-10T16:21:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f3009d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f301397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f3009e420 0x7f3f3013bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3009e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f300a0870 con 0x7f3f3009d930 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3f3009d3e0 con 0x7f3f3009e420 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f30074c70 con 0x7f3f3009ab50 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f361b4640 10 monclient(hunting): authenticate will time out at 1696.067796s 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f2effd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 msgr2=0x7f3f301397c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:57.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f2effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f2f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3f3009d930 0x7f3f3009dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:43934/0 (socket says 172.21.15.164:43934) 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.580+0000 7f3f2f7fe640 1 -- 172.21.15.164:0/2184987270 learned_addr learned my addr 172.21.15.164:0/2184987270 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/2184987270 >> v2:172.21.15.165:3300/0 conn(0x7f3f3009e420 0x7f3f3013bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request con 0x7f3f3009e420 auth_method 0 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_done global_id 154327 payload 274 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 -- 172.21.15.164:0/2184987270 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 msgr2=0x7f3f301397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:21:57.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/2184987270 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f301397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 -- 172.21.15.164:0/2184987270 >> v1:172.21.15.164:6789/0 conn(0x7f3f3009d930 legacy=0x7f3f3009dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 monclient: found mon.noname-c 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:21:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.581+0000 7f3f2ffff640 1 -- 172.21.15.164:0/2184987270 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3009ecb0 con 0x7f3f3009e420 2024-09-10T16:21:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2ffff640 1 -- 172.21.15.164:0/2184987270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009e420 msgr2=0x7f3f3013bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/2184987270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009e420 0x7f3f3013bd10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009dc0 tx=0x7f3f2802a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=313aa937bf8d47fc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 1 -- 172.21.15.164:0/2184987270 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28007a30 con 0x7f3f3009e420 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: dump: 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 1 -- 172.21.15.164:0/2184987270 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f28007bd0 con 0x7f3f3009e420 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 1 -- 172.21.15.164:0/2184987270 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28031400 con 0x7f3f3009e420 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f2e7fc640 10 monclient: dump: 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 5 monclient: authenticate success, global_id 154327 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 10 monclient: get_monmap_and_config success 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 4 set_mon_vals no callback set 2024-09-10T16:21:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 10 monclient: shutdown 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 1 -- 172.21.15.164:0/2184987270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009e420 msgr2=0x7f3f3013bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.584+0000 7f3f361b4640 1 --2- 172.21.15.164:0/2184987270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009e420 0x7f3f3013bd10 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009dc0 tx=0x7f3f2802a580 comp rx=0 tx=0).stop 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 -- 172.21.15.164:0/2184987270 shutdown_connections 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 --2- 172.21.15.164:0/2184987270 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009e420 0x7f3f3013bd10 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 --2- 172.21.15.164:0/2184987270 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f301397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 -- 172.21.15.164:0/2184987270 >> 172.21.15.164:0/2184987270 conn(0x7f3f300a3110 msgr2=0x7f3f300a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 -- 172.21.15.164:0/2184987270 shutdown_connections 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 -- 172.21.15.164:0/2184987270 wait complete. 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 10 monclient: build_initial_monmap 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 10 monclient: monmap: 2024-09-10T16:21:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.585+0000 7f3f361b4640 1 Processor -- start 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 -- start start 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient: init 2024-09-10T16:21:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient: _reopen_session rank -1 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f30134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009d930 0x7f3f3006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3009d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 0x7f3f3006ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3009e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300a2940 con 0x7f3f3009e420 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3009ef30 con 0x7f3f3009d930 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f300a2ac0 con 0x7f3f3009ab50 2024-09-10T16:21:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): _renew_subs 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2f7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 msgr2=0x7f3f30134e60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f361b4640 10 monclient(hunting): authenticate will time out at 1696.073585s 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f30134e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 0x7f3f3006ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 0x7f3f3006ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39402/0 (socket says 172.21.15.164:39402) 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 learned_addr learned my addr 172.21.15.164:0/3879034562 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request con 0x7f3f3009e420 auth_method 0 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2effd640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009d930 0x7f3f3006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2effd640 10 monclient(hunting): get_auth_request con 0x7f3f3009d930 auth_method 0 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.586+0000 7f3f2ffff640 10 monclient(hunting): handle_auth_done global_id 164268 payload 824 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 msgr2=0x7f3f30134e60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f3009ab50 0x7f3f30134e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009d930 msgr2=0x7f3f3006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3009d930 0x7f3f3006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:21:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 monclient: found mon.noname-a 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.587+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3009c910 con 0x7f3f3009e420 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 msgr2=0x7f3f3006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f2ffff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 0x7f3f3006ed10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009ee0 tx=0x7f3f28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28040070 con 0x7f3f3009e420 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: dump: 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: _finish_auth 0 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: _check_auth_tickets 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 5 monclient: authenticate success, global_id 164268 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f28045460 con 0x7f3f3009e420 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 10 monclient: _renew_subs 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f3013b070 con 0x7f3f3009e420 2024-09-10T16:21:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f28007a30 con 0x7f3f3009e420 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f17fff640 10 monclient: dump: 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 10 monclient: _renew_subs 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f3013b350 con 0x7f3f3009e420 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.589+0000 7f3f2dffb640 4 set_mon_vals no callback set 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f361b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f361b4640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 0x7f3f3006f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f300a4de0 con 0x7f3f3006f250 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3f28039040 con 0x7f3f3009e420 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f17fff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f1c03e1c0 0x7f3f1c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(534..534 src has 481..534) ==== 4863+0+0 (secure 0 0 0) 0x7f3f280747d0 con 0x7f3f3009e420 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 0x7f3f3006f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient: get_auth_request con 0x7f3f3006f250 auth_method 0 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2effd640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f1c03e1c0 0x7f3f1c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.590+0000 7f3f2effd640 10 monclient: get_auth_request con 0x7f3f1c03e1c0 auth_method 0 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.591+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_done global_id 154333 payload 824 2024-09-10T16:21:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.591+0000 7f3f2f7fe640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f3009dd10 con 0x7f3f3006f250 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f2f7fe640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 msgr2=0x7f3f3006f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f2f7fe640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 0x7f3f3006f630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f20004810 tx=0x7f3f2000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f2effd640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f1c03e1c0 0x7f3f1c040680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3f24003980 tx=0x7f3f24008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20017070 con 0x7f3f3006f250 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 10 monclient: dump: 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f20007070 con 0x7f3f3006f250 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:21:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 msgr2=0x7f3f3006f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 0x7f3f3006f630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f20004810 tx=0x7f3f2000c8e0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 msgr2=0x7f3f3006f630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.593+0000 7f3f17fff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f3006f250 0x7f3f3006f630 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f3f20004810 tx=0x7f3f2000c8e0 comp rx=0 tx=0).stop 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.599+0000 7f3f361b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.599+0000 7f3f361b4640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 0x7f3f30135bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.599+0000 7f3f361b4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.599+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f3006fb70 con 0x7f3f301357a0 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 0x7f3f30135bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient: get_auth_request con 0x7f3f301357a0 auth_method 0 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:21:57.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 10 monclient(hunting): handle_auth_done global_id 154336 payload 824 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.600+0000 7f3f2f7fe640 1 -- 172.21.15.164:0/3879034562 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f3013ae50 con 0x7f3f301357a0 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.602+0000 7f3f2f7fe640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 msgr2=0x7f3f30135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.602+0000 7f3f2f7fe640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 0x7f3f30135bb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f20007fd0 tx=0x7f3f20004a10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f20017040 con 0x7f3f301357a0 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 10 monclient: dump: 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:21:57.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f3f20012400 con 0x7f3f301357a0 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 msgr2=0x7f3f30135bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 0x7f3f30135bb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f20007fd0 tx=0x7f3f20004a10 comp rx=0 tx=0).stop 2024-09-10T16:21:57.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 msgr2=0x7f3f30135bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f17fff640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f301357a0 0x7f3f30135bb0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3f20007fd0 tx=0x7f3f20004a10 comp rx=0 tx=0).stop 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 154, 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 136153, 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:21:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:21:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:21:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:21:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:21:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:21:57.623 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:21:57.624 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:21:57.625 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:21:57.626 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:21:57.627 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:21:57.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f1c03e1c0 msgr2=0x7f3f1c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f1c03e1c0 0x7f3f1c040680 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f3f24003980 tx=0x7f3f24008040 comp rx=0 tx=0).stop 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 10 monclient: shutdown 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 msgr2=0x7f3f3006ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 --2- 172.21.15.164:0/3879034562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f3009e420 0x7f3f3006ed10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f3f28009ee0 tx=0x7f3f28004a40 comp rx=0 tx=0).stop 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 reap_dead start 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f2ffff640 1 -- 172.21.15.164:0/3879034562 reap_dead start 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 shutdown_connections 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 >> 172.21.15.164:0/3879034562 conn(0x7f3f300a3110 msgr2=0x7f3f3009b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 shutdown_connections 2024-09-10T16:21:57.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:21:57.603+0000 7f3f361b4640 1 -- 172.21.15.164:0/3879034562 wait complete. 2024-09-10T16:21:57.630 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:21:57.630 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:21:57.978 INFO:tasks.rados.rados.0.smithi165.stdout:2334: done (5 left) 2024-09-10T16:21:57.978 INFO:tasks.rados.rados.0.smithi165.stdout:2335: rollback oid 27 current snap is 255 2024-09-10T16:21:57.978 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 249 2024-09-10T16:21:57.978 INFO:tasks.rados.rados.0.smithi165.stdout:2328: finishing write tid 5 to smithi16534841-46 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2328: oid 46 updating version 0 to 1093 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2328: oid 46 version 1093 is already newer than 1091 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2328: oid 46 version 1093 is already newer than 1090 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2328: oid 46 version 1093 is already newer than 1092 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1093 (ObjNum 820 snap 255 seq_num 820) dirty exists 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2328: left oid 46 (ObjNum 820 snap 255 seq_num 820) 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2329: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2329: finishing copy_from to smithi16534841-16 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1174 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2330: finishing copy_from racing read to smithi16534841-33 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2330: finishing copy_from to smithi16534841-33 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1342 (ObjNum 633 snap 198 seq_num 633) dirty exists 2024-09-10T16:21:57.979 INFO:tasks.rados.rados.0.smithi165.stdout:2332: finishing write tid 1 to smithi16534841-44 2024-09-10T16:21:57.980 INFO:tasks.rados.rados.0.smithi165.stdout:2332: finishing write tid 2 to smithi16534841-44 2024-09-10T16:21:57.980 INFO:tasks.rados.rados.0.smithi165.stdout:2332: finishing write tid 3 to smithi16534841-44 2024-09-10T16:21:57.980 INFO:tasks.rados.rados.0.smithi165.stdout:2332: finishing write tid 4 to smithi16534841-44 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: finishing write tid 5 to smithi16534841-44 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: oid 44 updating version 0 to 1096 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: oid 44 version 1096 is already newer than 1095 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: oid 44 version 1096 is already newer than 1094 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: oid 44 updating version 1096 to 1097 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1097 (ObjNum 821 snap 255 seq_num 821) dirty exists 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2332: left oid 44 (ObjNum 821 snap 255 seq_num 821) 2024-09-10T16:21:57.981 INFO:tasks.rados.rados.0.smithi165.stdout:2333: expect (ObjNum 668 snap 207 seq_num 668) 2024-09-10T16:21:58.027 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:58.088 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:21:58.146 INFO:tasks.rados.rados.0.smithi165.stdout:2335: finishing rollback tid 1 to smithi16534841-27 2024-09-10T16:21:58.146 INFO:tasks.rados.rados.0.smithi165.stdout:2335: finishing rollback tid 2 to smithi16534841-27 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1359 (ObjNum 741 snap 225 seq_num 741) dirty exists 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2328: done (5 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2329: done (4 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2330: done (3 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2332: done (2 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2333: done (1 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2335: done (0 left) 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2336: write (excl) oid 31 current snap is 255 2024-09-10T16:21:58.147 INFO:tasks.rados.rados.0.smithi165.stdout:2336: seq_num 822 ranges {615354=420802,1651375=751957,3190053=492771} 2024-09-10T16:21:58.151 INFO:tasks.rados.rados.0.smithi165.stdout:2336: writing smithi16534841-31 from 615354 to 1036156 tid 1 2024-09-10T16:21:58.159 INFO:tasks.rados.rados.0.smithi165.stdout:2336: writing smithi16534841-31 from 1651375 to 2403332 tid 2 2024-09-10T16:21:58.166 INFO:tasks.rados.rados.0.smithi165.stdout:2336: writing smithi16534841-31 from 3190053 to 3682824 tid 3 2024-09-10T16:21:58.166 INFO:tasks.rados.rados.0.smithi165.stdout:2337: rollback oid 39 current snap is 255 2024-09-10T16:21:58.166 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 240 2024-09-10T16:21:58.166 INFO:tasks.rados.rados.0.smithi165.stdout:2336: finishing write tid 1 to smithi16534841-31 2024-09-10T16:21:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:2336: finishing write tid 2 to smithi16534841-31 2024-09-10T16:21:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:2338: copy_from oid 41 from oid 6 current snap is 255 2024-09-10T16:21:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:2339: delete oid 37 current snap is 255 2024-09-10T16:21:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:2339: done (3 left) 2024-09-10T16:21:58.167 INFO:tasks.rados.rados.0.smithi165.stdout:2340: snap_remove snap 252 2024-09-10T16:21:58.986 INFO:tasks.rados.rados.0.smithi165.stdout:2340: done (3 left) 2024-09-10T16:21:58.986 INFO:tasks.rados.rados.0.smithi165.stdout:2341: delete oid 12 current snap is 255 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: finishing write tid 3 to smithi16534841-31 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: finishing write tid 4 to smithi16534841-31 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: finishing write tid 5 to smithi16534841-31 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: oid 31 updating version 0 to 1433 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: oid 31 updating version 1433 to 1436 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: oid 31 version 1436 is already newer than 1434 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: oid 31 version 1436 is already newer than 1435 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1436 (ObjNum 822 snap 255 seq_num 822) dirty exists 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2336: left oid 31 (ObjNum 822 snap 255 seq_num 822) 2024-09-10T16:21:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2337: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2337: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2337: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1369 (ObjNum 631 snap 197 seq_num 631) dirty exists 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2338: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2338: finishing copy_from to smithi16534841-41 2024-09-10T16:21:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1174 (ObjNum 651 snap 207 seq_num 651) dirty exists 2024-09-10T16:21:58.991 INFO:tasks.rados.rados.0.smithi165.stdout:2336: done (3 left) 2024-09-10T16:21:58.991 INFO:tasks.rados.rados.0.smithi165.stdout:2337: done (2 left) 2024-09-10T16:21:58.991 INFO:tasks.rados.rados.0.smithi165.stdout:2338: done (1 left) 2024-09-10T16:21:58.991 INFO:tasks.rados.rados.0.smithi165.stdout:2341: done (0 left) 2024-09-10T16:21:58.991 INFO:tasks.rados.rados.0.smithi165.stdout:2342: snap_remove snap 249 2024-09-10T16:21:59.988 INFO:tasks.rados.rados.0.smithi165.stdout:2342: done (0 left) 2024-09-10T16:21:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2343: delete oid 5 current snap is 255 2024-09-10T16:21:59.995 INFO:tasks.rados.rados.0.smithi165.stdout:2343: done (0 left) 2024-09-10T16:21:59.995 INFO:tasks.rados.rados.0.smithi165.stdout:2344: copy_from oid 6 from oid 12 current snap is 255 2024-09-10T16:21:59.995 INFO:tasks.rados.rados.0.smithi165.stdout:2345: read oid 3 snap -1 2024-09-10T16:21:59.995 INFO:tasks.rados.rados.0.smithi165.stdout:2345: expect deleted 2024-09-10T16:21:59.995 INFO:tasks.rados.rados.0.smithi165.stdout:2346: snap_remove snap 251 2024-09-10T16:22:00.959 INFO:tasks.rados.rados.0.smithi165.stdout:2346: done (2 left) 2024-09-10T16:22:00.959 INFO:tasks.rados.rados.0.smithi165.stdout:2347: write oid 34 current snap is 255 2024-09-10T16:22:00.959 INFO:tasks.rados.rados.0.smithi165.stdout:2347: seq_num 826 ranges {682582=721798,2168950=642942,3583392=299190} 2024-09-10T16:22:00.967 INFO:tasks.rados.rados.0.smithi165.stdout:2347: writing smithi16534841-34 from 682582 to 1404380 tid 1 2024-09-10T16:22:00.975 INFO:tasks.rados.rados.0.smithi165.stdout:2347: writing smithi16534841-34 from 2168950 to 2811892 tid 2 2024-09-10T16:22:00.982 INFO:tasks.rados.rados.0.smithi165.stdout:2347: writing smithi16534841-34 from 3583392 to 3882582 tid 3 2024-09-10T16:22:00.982 INFO:tasks.rados.rados.0.smithi165.stdout:2348: read oid 37 snap 254 2024-09-10T16:22:00.982 INFO:tasks.rados.rados.0.smithi165.stdout:2348: expect deleted 2024-09-10T16:22:00.982 INFO:tasks.rados.rados.0.smithi165.stdout:2349: rollback oid 26 current snap is 255 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 230 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2344: finishing copy_from to smithi16534841-6 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2344: got expected ENOENT (src dne) 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2344: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2347: finishing write tid 1 to smithi16534841-34 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2344: done (4 left) 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2345: done (3 left) 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2350: rollback oid 32 current snap is 255 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 32 to 240 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2351: read oid 30 snap -1 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2351: expect (ObjNum 771 snap 237 seq_num 771) 2024-09-10T16:22:00.983 INFO:tasks.rados.rados.0.smithi165.stdout:2352: write oid 12 current snap is 255 2024-09-10T16:22:00.984 INFO:tasks.rados.rados.0.smithi165.stdout:2352: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2024-09-10T16:22:00.990 INFO:tasks.rados.rados.0.smithi165.stdout:2352: writing smithi16534841-12 from 699389 to 1496436 tid 1 2024-09-10T16:22:00.996 INFO:tasks.rados.rados.0.smithi165.stdout:2352: writing smithi16534841-12 from 2111079 to 2614032 tid 2 2024-09-10T16:22:01.002 INFO:tasks.rados.rados.0.smithi165.stdout:2352: writing smithi16534841-12 from 3094462 to 3676299 tid 3 2024-09-10T16:22:01.002 INFO:tasks.rados.rados.0.smithi165.stdout:2353: write oid 49 current snap is 255 2024-09-10T16:22:01.002 INFO:tasks.rados.rados.0.smithi165.stdout:2353: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2024-09-10T16:22:01.008 INFO:tasks.rados.rados.0.smithi165.stdout:2353: writing smithi16534841-49 from 716196 to 1188492 tid 1 2024-09-10T16:22:01.015 INFO:tasks.rados.rados.0.smithi165.stdout:2353: writing smithi16534841-49 from 1769561 to 2416172 tid 2 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2353: writing smithi16534841-49 from 3121885 to 3914994 tid 3 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2354: delete oid 11 current snap is 255 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: finishing write tid 2 to smithi16534841-34 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2350: finishing rollback tid 0 to smithi16534841-32 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: finishing write tid 3 to smithi16534841-34 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: finishing write tid 4 to smithi16534841-34 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2349: finishing rollback tid 0 to smithi16534841-26 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: finishing write tid 5 to smithi16534841-34 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: oid 34 updating version 0 to 1338 2024-09-10T16:22:01.024 INFO:tasks.rados.rados.0.smithi165.stdout:2347: oid 34 version 1338 is already newer than 1336 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2347: oid 34 version 1338 is already newer than 1337 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2347: oid 34 version 1338 is already newer than 1335 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1338 (ObjNum 826 snap 255 seq_num 826) dirty exists 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2347: left oid 34 (ObjNum 826 snap 255 seq_num 826) 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2350: finishing rollback tid 1 to smithi16534841-32 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2350: finishing rollback tid 2 to smithi16534841-32 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1313 (ObjNum 660 snap 207 seq_num 660) dirty exists 2024-09-10T16:22:01.025 INFO:tasks.rados.rados.0.smithi165.stdout:2351: expect (ObjNum 771 snap 237 seq_num 771) 2024-09-10T16:22:01.026 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1477669 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2349: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2349: finishing rollback tid 2 to smithi16534841-26 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1448 (ObjNum 585 snap 186 seq_num 585) dirty exists 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2352: finishing write tid 1 to smithi16534841-12 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2352: finishing write tid 2 to smithi16534841-12 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2352: finishing write tid 3 to smithi16534841-12 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2352: finishing write tid 4 to smithi16534841-12 2024-09-10T16:22:01.112 INFO:tasks.rados.rados.0.smithi165.stdout:2352: finishing write tid 5 to smithi16534841-12 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2352: oid 12 updating version 0 to 1355 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2352: oid 12 version 1355 is already newer than 1354 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2352: oid 12 updating version 1355 to 1357 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2352: oid 12 version 1357 is already newer than 1356 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1357 (ObjNum 827 snap 255 seq_num 827) dirty exists 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2352: left oid 12 (ObjNum 827 snap 255 seq_num 827) 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2353: finishing write tid 1 to smithi16534841-49 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2353: finishing write tid 2 to smithi16534841-49 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2353: finishing write tid 3 to smithi16534841-49 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2347: done (7 left) 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2348: done (6 left) 2024-09-10T16:22:01.113 INFO:tasks.rados.rados.0.smithi165.stdout:2349: done (5 left) 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2350: done (4 left) 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2351: done (3 left) 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2352: done (2 left) 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2354: done (1 left) 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2355: snap_create 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: finishing write tid 4 to smithi16534841-49 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: finishing write tid 5 to smithi16534841-49 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: oid 49 updating version 0 to 1126 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: oid 49 updating version 1126 to 1129 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: oid 49 version 1129 is already newer than 1127 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: oid 49 version 1129 is already newer than 1128 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1129 (ObjNum 828 snap 255 seq_num 828) dirty exists 2024-09-10T16:22:01.114 INFO:tasks.rados.rados.0.smithi165.stdout:2353: left oid 49 (ObjNum 828 snap 255 seq_num 828) 2024-09-10T16:22:02.493 INFO:tasks.rados.rados.0.smithi165.stdout:2353: done (1 left) 2024-09-10T16:22:02.493 INFO:tasks.rados.rados.0.smithi165.stdout:2355: done (0 left) 2024-09-10T16:22:02.493 INFO:tasks.rados.rados.0.smithi165.stdout:2356: write oid 6 current snap is 256 2024-09-10T16:22:02.494 INFO:tasks.rados.rados.0.smithi165.stdout:2356: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2024-09-10T16:22:02.501 INFO:tasks.rados.rados.0.smithi165.stdout:2356: writing smithi16534841-6 from 749810 to 1488957 tid 1 2024-09-10T16:22:02.507 INFO:tasks.rados.rados.0.smithi165.stdout:2356: writing smithi16534841-6 from 2002878 to 2653158 tid 2 2024-09-10T16:22:02.513 INFO:tasks.rados.rados.0.smithi165.stdout:2356: writing smithi16534841-6 from 3293084 to 3708737 tid 3 2024-09-10T16:22:02.513 INFO:tasks.rados.rados.0.smithi165.stdout:2357: snap_create 2024-09-10T16:22:02.513 INFO:tasks.rados.rados.0.smithi165.stdout:2356: finishing write tid 1 to smithi16534841-6 2024-09-10T16:22:02.513 INFO:tasks.rados.rados.0.smithi165.stdout:2356: finishing write tid 2 to smithi16534841-6 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: finishing write tid 3 to smithi16534841-6 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: finishing write tid 4 to smithi16534841-6 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: finishing write tid 5 to smithi16534841-6 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: oid 6 updating version 0 to 1315 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: oid 6 version 1315 is already newer than 1314 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: oid 6 updating version 1315 to 1317 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: oid 6 version 1317 is already newer than 1316 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1317 (ObjNum 830 snap 256 seq_num 830) dirty exists 2024-09-10T16:22:02.517 INFO:tasks.rados.rados.0.smithi165.stdout:2356: left oid 6 (ObjNum 830 snap 256 seq_num 830) 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:2356: done (1 left) 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:2357: done (0 left) 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:2358: rollback oid 41 current snap is 257 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 254 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:2359: write (excl) oid 50 current snap is 257 2024-09-10T16:22:03.496 INFO:tasks.rados.rados.0.smithi165.stdout:2359: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2024-09-10T16:22:03.502 INFO:tasks.rados.rados.0.smithi165.stdout:2359: writing smithi16534841-50 from 766617 to 1181013 tid 1 2024-09-10T16:22:03.509 INFO:tasks.rados.rados.0.smithi165.stdout:2359: writing smithi16534841-50 from 1945007 to 2455298 tid 2 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2359: writing smithi16534841-50 from 3204154 to 3831079 tid 3 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2360: read oid 26 snap -1 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2360: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2361: snap_create 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2358: finishing rollback tid 0 to smithi16534841-41 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2358: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1371 (ObjNum 513 snap 0 seq_num 1162045496) dirty dne 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2359: finishing write tid 1 to smithi16534841-50 2024-09-10T16:22:03.518 INFO:tasks.rados.rados.0.smithi165.stdout:2359: finishing write tid 2 to smithi16534841-50 2024-09-10T16:22:03.520 INFO:tasks.rados.rados.0.smithi165.stdout:2359: finishing write tid 3 to smithi16534841-50 2024-09-10T16:22:03.523 INFO:tasks.rados.rados.0.smithi165.stdout:2359: finishing write tid 4 to smithi16534841-50 2024-09-10T16:22:03.523 INFO:tasks.rados.rados.0.smithi165.stdout:2359: finishing write tid 5 to smithi16534841-50 2024-09-10T16:22:03.523 INFO:tasks.rados.rados.0.smithi165.stdout:2359: oid 50 updating version 0 to 1255 2024-09-10T16:22:03.524 INFO:tasks.rados.rados.0.smithi165.stdout:2359: oid 50 updating version 1255 to 1256 2024-09-10T16:22:03.524 INFO:tasks.rados.rados.0.smithi165.stdout:2359: oid 50 version 1256 is already newer than 1254 2024-09-10T16:22:03.524 INFO:tasks.rados.rados.0.smithi165.stdout:2359: oid 50 updating version 1256 to 1257 2024-09-10T16:22:03.524 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1257 (ObjNum 831 snap 257 seq_num 831) dirty exists 2024-09-10T16:22:03.524 INFO:tasks.rados.rados.0.smithi165.stdout:2359: left oid 50 (ObjNum 831 snap 257 seq_num 831) 2024-09-10T16:22:03.525 INFO:tasks.rados.rados.0.smithi165.stdout:2360: expect (ObjNum 585 snap 186 seq_num 585) 2024-09-10T16:22:03.616 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2561961 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2358: done (3 left) 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2359: done (2 left) 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2360: done (1 left) 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2361: done (0 left) 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2362: write (excl) oid 10 current snap is 258 2024-09-10T16:22:04.499 INFO:tasks.rados.rados.0.smithi165.stdout:2362: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2024-09-10T16:22:04.505 INFO:tasks.rados.rados.0.smithi165.stdout:2362: writing smithi16534841-10 from 783424 to 1273069 tid 1 2024-09-10T16:22:04.513 INFO:tasks.rados.rados.0.smithi165.stdout:2362: writing smithi16534841-10 from 2003489 to 2657438 tid 2 2024-09-10T16:22:04.519 INFO:tasks.rados.rados.0.smithi165.stdout:2362: writing smithi16534841-10 from 3231577 to 3786127 tid 3 2024-09-10T16:22:04.519 INFO:tasks.rados.rados.0.smithi165.stdout:2363: snap_remove snap 240 2024-09-10T16:22:05.501 INFO:tasks.rados.rados.0.smithi165.stdout:2363: done (1 left) 2024-09-10T16:22:05.501 INFO:tasks.rados.rados.0.smithi165.stdout:2364: read oid 1 snap 230 2024-09-10T16:22:05.501 INFO:tasks.rados.rados.0.smithi165.stdout:2364: expect (ObjNum 694 snap 214 seq_num 694) 2024-09-10T16:22:05.501 INFO:tasks.rados.rados.0.smithi165.stdout:2365: read oid 19 snap -1 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2365: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2366: copy_from oid 47 from oid 31 current snap is 258 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: finishing write tid 1 to smithi16534841-10 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: finishing write tid 2 to smithi16534841-10 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: finishing write tid 3 to smithi16534841-10 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: finishing write tid 4 to smithi16534841-10 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: finishing write tid 5 to smithi16534841-10 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: oid 10 updating version 0 to 1132 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: oid 10 version 1132 is already newer than 1131 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: oid 10 updating version 1132 to 1133 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: oid 10 version 1133 is already newer than 1130 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1133 (ObjNum 832 snap 258 seq_num 832) dirty exists 2024-09-10T16:22:05.502 INFO:tasks.rados.rados.0.smithi165.stdout:2362: left oid 10 (ObjNum 832 snap 258 seq_num 832) 2024-09-10T16:22:05.503 INFO:tasks.rados.rados.0.smithi165.stdout:2362: done (3 left) 2024-09-10T16:22:05.503 INFO:tasks.rados.rados.0.smithi165.stdout:2367: delete oid 16 current snap is 258 2024-09-10T16:22:05.514 INFO:tasks.rados.rados.0.smithi165.stdout:2364: expect (ObjNum 694 snap 214 seq_num 694) 2024-09-10T16:22:05.672 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:05.754 INFO:tasks.rados.rados.0.smithi165.stdout:2365: expect (ObjNum 616 snap 190 seq_num 616) 2024-09-10T16:22:05.851 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 408830 2024-09-10T16:22:05.899 INFO:tasks.rados.rados.0.smithi165.stdout:2366: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:22:05.899 INFO:tasks.rados.rados.0.smithi165.stdout:2366: finishing copy_from to smithi16534841-47 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1436 (ObjNum 822 snap 255 seq_num 822) dirty exists 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2364: done (3 left) 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2365: done (2 left) 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2366: done (1 left) 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2367: done (0 left) 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2368: write (excl) oid 10 current snap is 258 2024-09-10T16:22:05.900 INFO:tasks.rados.rados.0.smithi165.stdout:2368: seq_num 834 ranges {417038=640143,1720453=296585} 2024-09-10T16:22:05.904 INFO:tasks.rados.rados.0.smithi165.stdout:2368: writing smithi16534841-10 from 417038 to 1057181 tid 1 2024-09-10T16:22:05.910 INFO:tasks.rados.rados.0.smithi165.stdout:2368: writing smithi16534841-10 from 1720453 to 2017038 tid 2 2024-09-10T16:22:05.910 INFO:tasks.rados.rados.0.smithi165.stdout:2369: rollback oid 17 current snap is 258 2024-09-10T16:22:05.910 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 230 2024-09-10T16:22:05.910 INFO:tasks.rados.rados.0.smithi165.stdout:2370: snap_create 2024-09-10T16:22:05.913 INFO:tasks.rados.rados.0.smithi165.stdout:2368: finishing write tid 1 to smithi16534841-10 2024-09-10T16:22:05.918 INFO:tasks.rados.rados.0.smithi165.stdout:2368: finishing write tid 2 to smithi16534841-10 2024-09-10T16:22:05.918 INFO:tasks.rados.rados.0.smithi165.stdout:2368: finishing write tid 3 to smithi16534841-10 2024-09-10T16:22:05.918 INFO:tasks.rados.rados.0.smithi165.stdout:2368: finishing write tid 4 to smithi16534841-10 2024-09-10T16:22:05.918 INFO:tasks.rados.rados.0.smithi165.stdout:2368: oid 10 updating version 0 to 1136 2024-09-10T16:22:05.918 INFO:tasks.rados.rados.0.smithi165.stdout:2368: oid 10 version 1136 is already newer than 1134 2024-09-10T16:22:05.919 INFO:tasks.rados.rados.0.smithi165.stdout:2368: oid 10 version 1136 is already newer than 1135 2024-09-10T16:22:05.919 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1136 (ObjNum 834 snap 258 seq_num 834) dirty exists 2024-09-10T16:22:05.919 INFO:tasks.rados.rados.0.smithi165.stdout:2368: left oid 10 (ObjNum 834 snap 258 seq_num 834) 2024-09-10T16:22:05.919 INFO:tasks.rados.rados.0.smithi165.stdout:2369: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:22:05.919 INFO:tasks.rados.rados.0.smithi165.stdout:2369: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:22:05.920 INFO:tasks.rados.rados.0.smithi165.stdout:2369: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:22:05.920 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1139 (ObjNum 673 snap 207 seq_num 673) dirty exists 2024-09-10T16:22:05.960 INFO:tasks.rados.rados.0.smithi165.stdout:2368: done (2 left) 2024-09-10T16:22:05.960 INFO:tasks.rados.rados.0.smithi165.stdout:2369: done (1 left) 2024-09-10T16:22:05.960 INFO:tasks.rados.rados.0.smithi165.stdout:2370: done (0 left) 2024-09-10T16:22:05.960 INFO:tasks.rados.rados.0.smithi165.stdout:2371: delete oid 20 current snap is 259 2024-09-10T16:22:05.968 INFO:tasks.rados.rados.0.smithi165.stdout:2371: done (0 left) 2024-09-10T16:22:05.968 INFO:tasks.rados.rados.0.smithi165.stdout:2372: read oid 33 snap 257 2024-09-10T16:22:05.968 INFO:tasks.rados.rados.0.smithi165.stdout:2372: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:22:05.969 INFO:tasks.rados.rados.0.smithi165.stdout:2373: snap_create 2024-09-10T16:22:05.970 INFO:tasks.rados.rados.0.smithi165.stdout:2372: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:22:06.030 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2352249 2024-09-10T16:22:06.959 INFO:tasks.rados.rados.0.smithi165.stdout:2372: done (1 left) 2024-09-10T16:22:06.960 INFO:tasks.rados.rados.0.smithi165.stdout:2373: done (0 left) 2024-09-10T16:22:06.960 INFO:tasks.rados.rados.0.smithi165.stdout:2374: rollback oid 21 current snap is 260 2024-09-10T16:22:06.960 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 230 2024-09-10T16:22:06.960 INFO:tasks.rados.rados.0.smithi165.stdout:2375: write (excl) oid 5 current snap is 260 2024-09-10T16:22:06.960 INFO:tasks.rados.rados.0.smithi165.stdout:2375: seq_num 836 ranges {450652=790641,1721064=329588} 2024-09-10T16:22:06.966 INFO:tasks.rados.rados.0.smithi165.stdout:2375: writing smithi16534841-5 from 450652 to 1241293 tid 1 2024-09-10T16:22:06.971 INFO:tasks.rados.rados.0.smithi165.stdout:2375: writing smithi16534841-5 from 1721064 to 2050652 tid 2 2024-09-10T16:22:06.971 INFO:tasks.rados.rados.0.smithi165.stdout:2376: snap_remove snap 218 2024-09-10T16:22:07.972 INFO:tasks.rados.rados.0.smithi165.stdout:2376: done (2 left) 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2377: delete oid 46 current snap is 260 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2374: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2374: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2374: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1375 (ObjNum 698 snap 215 seq_num 698) dirty exists 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2375: finishing write tid 1 to smithi16534841-5 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2375: finishing write tid 2 to smithi16534841-5 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2375: finishing write tid 3 to smithi16534841-5 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2375: finishing write tid 4 to smithi16534841-5 2024-09-10T16:22:07.973 INFO:tasks.rados.rados.0.smithi165.stdout:2375: oid 5 updating version 0 to 1320 2024-09-10T16:22:07.974 INFO:tasks.rados.rados.0.smithi165.stdout:2375: oid 5 version 1320 is already newer than 1318 2024-09-10T16:22:07.974 INFO:tasks.rados.rados.0.smithi165.stdout:2375: oid 5 version 1320 is already newer than 1319 2024-09-10T16:22:07.974 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1320 (ObjNum 836 snap 260 seq_num 836) dirty exists 2024-09-10T16:22:07.974 INFO:tasks.rados.rados.0.smithi165.stdout:2375: left oid 5 (ObjNum 836 snap 260 seq_num 836) 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2374: done (2 left) 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2375: done (1 left) 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2377: done (0 left) 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2378: read oid 42 snap 236 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2378: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2379: copy_from oid 30 from oid 45 current snap is 260 2024-09-10T16:22:07.980 INFO:tasks.rados.rados.0.smithi165.stdout:2380: write oid 31 current snap is 260 2024-09-10T16:22:07.981 INFO:tasks.rados.rados.0.smithi165.stdout:2380: seq_num 838 ranges {484266=657492,1554381=529885} 2024-09-10T16:22:07.988 INFO:tasks.rados.rados.0.smithi165.stdout:2380: writing smithi16534841-31 from 484266 to 1141758 tid 1 2024-09-10T16:22:07.994 INFO:tasks.rados.rados.0.smithi165.stdout:2380: writing smithi16534841-31 from 1554381 to 2084266 tid 2 2024-09-10T16:22:07.994 INFO:tasks.rados.rados.0.smithi165.stdout:2381: write (excl) oid 40 current snap is 260 2024-09-10T16:22:07.994 INFO:tasks.rados.rados.0.smithi165.stdout:2381: seq_num 839 ranges {501073=732741,1896510=204563} 2024-09-10T16:22:08.001 INFO:tasks.rados.rados.0.smithi165.stdout:2381: writing smithi16534841-40 from 501073 to 1233814 tid 1 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:2381: writing smithi16534841-40 from 1896510 to 2101073 tid 2 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:2382: read oid 3 snap 259 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:2382: expect deleted 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:2383: rollback oid 10 current snap is 260 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 254 2024-09-10T16:22:08.007 INFO:tasks.rados.rados.0.smithi165.stdout:2378: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:22:08.040 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:08.092 INFO:tasks.rados.rados.0.smithi165.stdout:2379: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: finishing write tid 1 to smithi16534841-40 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: finishing write tid 2 to smithi16534841-40 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: finishing write tid 3 to smithi16534841-40 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: finishing write tid 4 to smithi16534841-40 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: oid 40 updating version 0 to 1200 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: oid 40 version 1200 is already newer than 1199 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: oid 40 version 1200 is already newer than 1198 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1200 (ObjNum 839 snap 260 seq_num 839) dirty exists 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: left oid 40 (ObjNum 839 snap 260 seq_num 839) 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2383: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2383: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2378: done (5 left) 2024-09-10T16:22:08.093 INFO:tasks.rados.rados.0.smithi165.stdout:2381: done (4 left) 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2382: done (3 left) 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2384: copy_from oid 38 from oid 42 current snap is 260 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2383: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1142 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2379: finishing copy_from to smithi16534841-30 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1352 (ObjNum 791 snap 243 seq_num 791) dirty exists 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: finishing write tid 1 to smithi16534841-31 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: finishing write tid 2 to smithi16534841-31 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: finishing write tid 3 to smithi16534841-31 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: finishing write tid 4 to smithi16534841-31 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: oid 31 updating version 0 to 1466 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: oid 31 version 1466 is already newer than 1464 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:2380: oid 31 version 1466 is already newer than 1465 2024-09-10T16:22:08.094 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1466 (ObjNum 838 snap 260 seq_num 838) dirty exists 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2380: left oid 31 (ObjNum 838 snap 260 seq_num 838) 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2379: done (3 left) 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2380: done (2 left) 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2383: done (1 left) 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2385: delete oid 31 current snap is 260 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2385: done (1 left) 2024-09-10T16:22:08.095 INFO:tasks.rados.rados.0.smithi165.stdout:2386: read oid 21 snap -1 2024-09-10T16:22:08.096 INFO:tasks.rados.rados.0.smithi165.stdout:2386: expect (ObjNum 698 snap 215 seq_num 698) 2024-09-10T16:22:08.096 INFO:tasks.rados.rados.0.smithi165.stdout:2387: write oid 14 current snap is 260 2024-09-10T16:22:08.096 INFO:tasks.rados.rados.0.smithi165.stdout:2387: seq_num 841 ranges {534687=483239,1613474=412283} 2024-09-10T16:22:08.100 INFO:tasks.rados.rados.0.smithi165.stdout:2387: writing smithi16534841-14 from 534687 to 1017926 tid 1 2024-09-10T16:22:08.106 INFO:tasks.rados.rados.0.smithi165.stdout:2387: writing smithi16534841-14 from 1613474 to 2025757 tid 2 2024-09-10T16:22:08.106 INFO:tasks.rados.rados.0.smithi165.stdout:2388: read oid 27 snap 230 2024-09-10T16:22:08.106 INFO:tasks.rados.rados.0.smithi165.stdout:2388: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:22:08.106 INFO:tasks.rados.rados.0.smithi165.stdout:2389: write (excl) oid 6 current snap is 260 2024-09-10T16:22:08.106 INFO:tasks.rados.rados.0.smithi165.stdout:2389: seq_num 842 ranges {551494=558488,1671956=479538} 2024-09-10T16:22:08.112 INFO:tasks.rados.rados.0.smithi165.stdout:2389: writing smithi16534841-6 from 551494 to 1109982 tid 1 2024-09-10T16:22:08.118 INFO:tasks.rados.rados.0.smithi165.stdout:2389: writing smithi16534841-6 from 1671956 to 2151494 tid 2 2024-09-10T16:22:08.118 INFO:tasks.rados.rados.0.smithi165.stdout:2390: snap_remove snap 257 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2390: done (5 left) 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2391: read oid 33 snap 258 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2391: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2392: read oid 20 snap -1 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2392: expect deleted 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2393: write (excl) oid 44 current snap is 260 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:2386: expect (ObjNum 698 snap 215 seq_num 698) 2024-09-10T16:22:08.977 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:2384: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:2384: finishing copy_from to smithi16534841-38 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1248 (ObjNum 758 snap 230 seq_num 758) dirty exists 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:2387: finishing write tid 1 to smithi16534841-14 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:2387: finishing write tid 2 to smithi16534841-14 2024-09-10T16:22:09.168 INFO:tasks.rados.rados.0.smithi165.stdout:2393: seq_num 843 ranges {568301=633737,1614085=415952} 2024-09-10T16:22:09.174 INFO:tasks.rados.rados.0.smithi165.stdout:2393: writing smithi16534841-44 from 568301 to 1202038 tid 1 2024-09-10T16:22:09.179 INFO:tasks.rados.rados.0.smithi165.stdout:2393: writing smithi16534841-44 from 1614085 to 2030037 tid 2 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:2384: done (7 left) 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:2386: done (6 left) 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:2394: rollback oid 3 current snap is 260 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 254 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:2388: expect (ObjNum 741 snap 225 seq_num 741) 2024-09-10T16:22:09.180 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:09.220 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: finishing write tid 3 to smithi16534841-14 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: finishing write tid 4 to smithi16534841-14 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: oid 14 updating version 0 to 1202 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: oid 14 updating version 1202 to 1203 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: oid 14 version 1203 is already newer than 1201 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1203 (ObjNum 841 snap 260 seq_num 841) dirty exists 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2387: left oid 14 (ObjNum 841 snap 260 seq_num 841) 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2389: finishing write tid 1 to smithi16534841-6 2024-09-10T16:22:09.289 INFO:tasks.rados.rados.0.smithi165.stdout:2389: finishing write tid 2 to smithi16534841-6 2024-09-10T16:22:09.290 INFO:tasks.rados.rados.0.smithi165.stdout:2389: finishing write tid 3 to smithi16534841-6 2024-09-10T16:22:09.290 INFO:tasks.rados.rados.0.smithi165.stdout:2387: done (6 left) 2024-09-10T16:22:09.290 INFO:tasks.rados.rados.0.smithi165.stdout:2388: done (5 left) 2024-09-10T16:22:09.290 INFO:tasks.rados.rados.0.smithi165.stdout:2395: write (excl) oid 22 current snap is 260 2024-09-10T16:22:09.290 INFO:tasks.rados.rados.0.smithi165.stdout:2395: seq_num 844 ranges {585108=425339,1788920=396188} 2024-09-10T16:22:09.293 INFO:tasks.rados.rados.0.smithi165.stdout:2395: writing smithi16534841-22 from 585108 to 1010447 tid 1 2024-09-10T16:22:09.300 INFO:tasks.rados.rados.0.smithi165.stdout:2395: writing smithi16534841-22 from 1788920 to 2185108 tid 2 2024-09-10T16:22:09.301 INFO:tasks.rados.rados.0.smithi165.stdout:2396: write (excl) oid 9 current snap is 260 2024-09-10T16:22:09.301 INFO:tasks.rados.rados.0.smithi165.stdout:2396: seq_num 845 ranges {601915=500588,1847402=354513} 2024-09-10T16:22:09.306 INFO:tasks.rados.rados.0.smithi165.stdout:2396: writing smithi16534841-9 from 601915 to 1102503 tid 1 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2396: writing smithi16534841-9 from 1847402 to 2201915 tid 2 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2397: write oid 23 current snap is 260 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2389: finishing write tid 4 to smithi16534841-6 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2389: oid 6 updating version 0 to 1323 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2389: oid 6 version 1323 is already newer than 1321 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2389: oid 6 version 1323 is already newer than 1322 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1323 (ObjNum 842 snap 260 seq_num 842) dirty exists 2024-09-10T16:22:09.313 INFO:tasks.rados.rados.0.smithi165.stdout:2389: left oid 6 (ObjNum 842 snap 260 seq_num 842) 2024-09-10T16:22:09.314 INFO:tasks.rados.rados.0.smithi165.stdout:2391: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:22:09.330 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2272582 2024-09-10T16:22:09.382 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:2393: finishing write tid 1 to smithi16534841-44 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:2394: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1365 (ObjNum 129 snap 0 seq_num 1162126080) dirty dne 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:2393: finishing write tid 2 to smithi16534841-44 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:2393: finishing write tid 3 to smithi16534841-44 2024-09-10T16:22:09.424 INFO:tasks.rados.rados.0.smithi165.stdout:2393: finishing write tid 4 to smithi16534841-44 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2393: oid 44 updating version 0 to 1101 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2393: oid 44 version 1101 is already newer than 1100 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2393: oid 44 version 1101 is already newer than 1099 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1101 (ObjNum 843 snap 260 seq_num 843) dirty exists 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2393: left oid 44 (ObjNum 843 snap 260 seq_num 843) 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2395: finishing write tid 1 to smithi16534841-22 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2395: finishing write tid 2 to smithi16534841-22 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2395: finishing write tid 3 to smithi16534841-22 2024-09-10T16:22:09.425 INFO:tasks.rados.rados.0.smithi165.stdout:2397: seq_num 846 ranges {618722=575837,1905884=312838} 2024-09-10T16:22:09.429 INFO:tasks.rados.rados.0.smithi165.stdout:2397: writing smithi16534841-23 from 618722 to 1194559 tid 1 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2397: writing smithi16534841-23 from 1905884 to 2218722 tid 2 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2389: done (7 left) 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2391: done (6 left) 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2392: done (5 left) 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2393: done (4 left) 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2394: done (3 left) 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2398: snap_create 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2395: finishing write tid 4 to smithi16534841-22 2024-09-10T16:22:09.436 INFO:tasks.rados.rados.0.smithi165.stdout:2395: oid 22 updating version 0 to 1372 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2395: oid 22 updating version 1372 to 1373 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2395: oid 22 version 1373 is already newer than 1371 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1373 (ObjNum 844 snap 260 seq_num 844) dirty exists 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2395: left oid 22 (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: finishing write tid 1 to smithi16534841-9 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: finishing write tid 2 to smithi16534841-9 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: finishing write tid 3 to smithi16534841-9 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: finishing write tid 4 to smithi16534841-9 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: oid 9 updating version 0 to 1204 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: oid 9 updating version 1204 to 1205 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: oid 9 updating version 1205 to 1206 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1206 (ObjNum 845 snap 260 seq_num 845) dirty exists 2024-09-10T16:22:09.437 INFO:tasks.rados.rados.0.smithi165.stdout:2396: left oid 9 (ObjNum 845 snap 260 seq_num 845) 2024-09-10T16:22:09.438 INFO:tasks.rados.rados.0.smithi165.stdout:2397: finishing write tid 1 to smithi16534841-23 2024-09-10T16:22:09.438 INFO:tasks.rados.rados.0.smithi165.stdout:2397: finishing write tid 2 to smithi16534841-23 2024-09-10T16:22:09.438 INFO:tasks.rados.rados.0.smithi165.stdout:2397: finishing write tid 3 to smithi16534841-23 2024-09-10T16:22:09.438 INFO:tasks.rados.rados.0.smithi165.stdout:2397: finishing write tid 4 to smithi16534841-23 2024-09-10T16:22:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2397: oid 23 updating version 0 to 1373 2024-09-10T16:22:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2397: oid 23 version 1373 is already newer than 1372 2024-09-10T16:22:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2397: oid 23 updating version 1373 to 1374 2024-09-10T16:22:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1374 (ObjNum 846 snap 260 seq_num 846) dirty exists 2024-09-10T16:22:09.439 INFO:tasks.rados.rados.0.smithi165.stdout:2397: left oid 23 (ObjNum 846 snap 260 seq_num 846) 2024-09-10T16:22:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:2395: done (3 left) 2024-09-10T16:22:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:2396: done (2 left) 2024-09-10T16:22:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:2397: done (1 left) 2024-09-10T16:22:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:2398: done (0 left) 2024-09-10T16:22:09.987 INFO:tasks.rados.rados.0.smithi165.stdout:2399: snap_create 2024-09-10T16:22:10.961 INFO:tasks.rados.rados.0.smithi165.stdout:2399: done (0 left) 2024-09-10T16:22:10.961 INFO:tasks.rados.rados.0.smithi165.stdout:2400: delete oid 16 current snap is 262 2024-09-10T16:22:10.965 INFO:tasks.rados.rados.0.smithi165.stdout:2400: done (0 left) 2024-09-10T16:22:10.965 INFO:tasks.rados.rados.0.smithi165.stdout:2401: copy_from oid 38 from oid 34 current snap is 262 2024-09-10T16:22:10.965 INFO:tasks.rados.rados.0.smithi165.stdout:2402: read oid 35 snap -1 2024-09-10T16:22:10.965 INFO:tasks.rados.rados.0.smithi165.stdout:2402: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:22:10.965 INFO:tasks.rados.rados.0.smithi165.stdout:2403: read oid 13 snap -1 2024-09-10T16:22:10.966 INFO:tasks.rados.rados.0.smithi165.stdout:2403: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:22:10.966 INFO:tasks.rados.rados.0.smithi165.stdout:2404: read oid 46 snap -1 2024-09-10T16:22:10.966 INFO:tasks.rados.rados.0.smithi165.stdout:2404: expect deleted 2024-09-10T16:22:10.966 INFO:tasks.rados.rados.0.smithi165.stdout:2405: write (excl) oid 21 current snap is 262 2024-09-10T16:22:10.966 INFO:tasks.rados.rados.0.smithi165.stdout:2405: seq_num 848 ranges {652336=726335,1906495=345841} 2024-09-10T16:22:10.972 INFO:tasks.rados.rados.0.smithi165.stdout:2405: writing smithi16534841-21 from 652336 to 1378671 tid 1 2024-09-10T16:22:10.978 INFO:tasks.rados.rados.0.smithi165.stdout:2405: writing smithi16534841-21 from 1906495 to 2252336 tid 2 2024-09-10T16:22:10.978 INFO:tasks.rados.rados.0.smithi165.stdout:2406: copy_from oid 26 from oid 50 current snap is 262 2024-09-10T16:22:10.978 INFO:tasks.rados.rados.0.smithi165.stdout:2402: expect (ObjNum 714 snap 220 seq_num 714) 2024-09-10T16:22:11.065 INFO:tasks.rados.rados.0.smithi165.stdout:2403: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:22:11.129 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:11.193 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:11.260 INFO:tasks.rados.rados.0.smithi165.stdout:2401: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2401: finishing copy_from to smithi16534841-38 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1338 (ObjNum 826 snap 255 seq_num 826) dirty exists 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2405: finishing write tid 1 to smithi16534841-21 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2401: done (5 left) 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2402: done (4 left) 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2403: done (3 left) 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2404: done (2 left) 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2407: rollback oid 3 current snap is 262 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 258 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2405: finishing write tid 2 to smithi16534841-21 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2405: finishing write tid 3 to smithi16534841-21 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2405: finishing write tid 4 to smithi16534841-21 2024-09-10T16:22:11.261 INFO:tasks.rados.rados.0.smithi165.stdout:2405: oid 21 updating version 0 to 1388 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2405: oid 21 updating version 1388 to 1390 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2405: oid 21 version 1390 is already newer than 1389 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1390 (ObjNum 848 snap 262 seq_num 848) dirty exists 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2405: left oid 21 (ObjNum 848 snap 262 seq_num 848) 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2406: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2406: finishing copy_from to smithi16534841-26 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1257 (ObjNum 831 snap 257 seq_num 831) dirty exists 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2405: done (2 left) 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2406: done (1 left) 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2408: read oid 7 snap -1 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2408: expect (ObjNum 743 snap 225 seq_num 743) 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2409: read oid 4 snap 255 2024-09-10T16:22:11.262 INFO:tasks.rados.rados.0.smithi165.stdout:2409: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2410: read oid 12 snap 261 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2410: expect (ObjNum 827 snap 255 seq_num 827) 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2411: rollback oid 36 current snap is 262 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 256 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2412: read oid 37 snap -1 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2412: expect deleted 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2413: delete oid 28 current snap is 262 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:2411: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:22:11.263 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 0 (ObjNum 65 snap 0 seq_num 0) dirty dne 2024-09-10T16:22:11.264 INFO:tasks.rados.rados.0.smithi165.stdout:2407: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:22:11.264 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1365 (ObjNum 65 snap 0 seq_num 326610248) dirty dne 2024-09-10T16:22:11.264 INFO:tasks.rados.rados.0.smithi165.stdout:2410: expect (ObjNum 827 snap 255 seq_num 827) 2024-09-10T16:22:11.342 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1816124 2024-09-10T16:22:11.370 INFO:tasks.rados.rados.0.smithi165.stdout:2409: expect (ObjNum 745 snap 227 seq_num 745) 2024-09-10T16:22:11.565 INFO:tasks.rados.rados.0.smithi165.stdout:2408: expect (ObjNum 743 snap 225 seq_num 743) 2024-09-10T16:22:11.565 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:11.641 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1299457 2024-09-10T16:22:11.668 INFO:tasks.rados.rados.0.smithi165.stdout:2407: done (6 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2408: done (5 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2409: done (4 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2410: done (3 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2411: done (2 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2412: done (1 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2413: done (0 left) 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2414: write (excl) oid 17 current snap is 262 2024-09-10T16:22:11.669 INFO:tasks.rados.rados.0.smithi165.stdout:2414: seq_num 850 ranges {685950=476833,1623459=570617} 2024-09-10T16:22:11.674 INFO:tasks.rados.rados.0.smithi165.stdout:2414: writing smithi16534841-17 from 685950 to 1162783 tid 1 2024-09-10T16:22:11.680 INFO:tasks.rados.rados.0.smithi165.stdout:2414: writing smithi16534841-17 from 1623459 to 2194076 tid 2 2024-09-10T16:22:11.680 INFO:tasks.rados.rados.0.smithi165.stdout:2415: copy_from oid 21 from oid 34 current snap is 262 2024-09-10T16:22:11.680 INFO:tasks.rados.rados.0.smithi165.stdout:2414: finishing write tid 1 to smithi16534841-17 2024-09-10T16:22:11.680 INFO:tasks.rados.rados.0.smithi165.stdout:2416: delete oid 19 current snap is 262 2024-09-10T16:22:11.684 INFO:tasks.rados.rados.0.smithi165.stdout:2416: done (2 left) 2024-09-10T16:22:11.684 INFO:tasks.rados.rados.0.smithi165.stdout:2417: delete oid 28 current snap is 262 2024-09-10T16:22:11.685 INFO:tasks.rados.rados.0.smithi165.stdout:2414: finishing write tid 2 to smithi16534841-17 2024-09-10T16:22:11.686 INFO:tasks.rados.rados.0.smithi165.stdout:2414: finishing write tid 3 to smithi16534841-17 2024-09-10T16:22:11.686 INFO:tasks.rados.rados.0.smithi165.stdout:2417: done (2 left) 2024-09-10T16:22:11.686 INFO:tasks.rados.rados.0.smithi165.stdout:2418: snap_remove snap 254 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2418: done (2 left) 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2419: delete oid 38 current snap is 262 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2414: finishing write tid 4 to smithi16534841-17 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2414: oid 17 updating version 0 to 1143 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2414: oid 17 updating version 1143 to 1145 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2414: oid 17 version 1145 is already newer than 1144 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1145 (ObjNum 850 snap 262 seq_num 850) dirty exists 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2414: left oid 17 (ObjNum 850 snap 262 seq_num 850) 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2415: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:2415: finishing copy_from to smithi16534841-21 2024-09-10T16:22:12.225 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1338 (ObjNum 826 snap 255 seq_num 826) dirty exists 2024-09-10T16:22:12.227 INFO:tasks.rados.rados.0.smithi165.stdout:2414: done (2 left) 2024-09-10T16:22:12.227 INFO:tasks.rados.rados.0.smithi165.stdout:2415: done (1 left) 2024-09-10T16:22:12.227 INFO:tasks.rados.rados.0.smithi165.stdout:2419: done (0 left) 2024-09-10T16:22:12.227 INFO:tasks.rados.rados.0.smithi165.stdout:2420: delete oid 49 current snap is 262 2024-09-10T16:22:12.233 INFO:tasks.rados.rados.0.smithi165.stdout:2420: done (0 left) 2024-09-10T16:22:12.234 INFO:tasks.rados.rados.0.smithi165.stdout:2421: snap_remove snap 259 2024-09-10T16:22:13.226 INFO:tasks.rados.rados.0.smithi165.stdout:2421: done (0 left) 2024-09-10T16:22:13.227 INFO:tasks.rados.rados.0.smithi165.stdout:2422: read oid 40 snap -1 2024-09-10T16:22:13.227 INFO:tasks.rados.rados.0.smithi165.stdout:2422: expect (ObjNum 839 snap 260 seq_num 839) 2024-09-10T16:22:13.227 INFO:tasks.rados.rados.0.smithi165.stdout:2423: write (excl) oid 15 current snap is 262 2024-09-10T16:22:13.227 INFO:tasks.rados.rados.0.smithi165.stdout:2423: seq_num 855 ranges {769985=569431,1915869=437966} 2024-09-10T16:22:13.234 INFO:tasks.rados.rados.0.smithi165.stdout:2423: writing smithi16534841-15 from 769985 to 1339416 tid 1 2024-09-10T16:22:13.240 INFO:tasks.rados.rados.0.smithi165.stdout:2423: writing smithi16534841-15 from 1915869 to 2353835 tid 2 2024-09-10T16:22:13.240 INFO:tasks.rados.rados.0.smithi165.stdout:2424: snap_remove snap 260 2024-09-10T16:22:14.233 INFO:tasks.rados.rados.0.smithi165.stdout:2424: done (2 left) 2024-09-10T16:22:14.233 INFO:tasks.rados.rados.0.smithi165.stdout:2425: snap_create 2024-09-10T16:22:14.233 INFO:tasks.rados.rados.0.smithi165.stdout:2422: expect (ObjNum 839 snap 260 seq_num 839) 2024-09-10T16:22:14.329 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: finishing write tid 1 to smithi16534841-15 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: finishing write tid 2 to smithi16534841-15 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: finishing write tid 3 to smithi16534841-15 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: finishing write tid 4 to smithi16534841-15 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: oid 15 updating version 0 to 1314 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: oid 15 version 1314 is already newer than 1313 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: oid 15 updating version 1314 to 1315 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1315 (ObjNum 855 snap 262 seq_num 855) dirty exists 2024-09-10T16:22:14.377 INFO:tasks.rados.rados.0.smithi165.stdout:2423: left oid 15 (ObjNum 855 snap 262 seq_num 855) 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2422: done (2 left) 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2423: done (1 left) 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2425: done (0 left) 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2426: read oid 13 snap -1 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2426: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2427: write (excl) oid 17 current snap is 263 2024-09-10T16:22:15.233 INFO:tasks.rados.rados.0.smithi165.stdout:2427: seq_num 856 ranges {786792=644680,1857998=528794} 2024-09-10T16:22:15.240 INFO:tasks.rados.rados.0.smithi165.stdout:2427: writing smithi16534841-17 from 786792 to 1431472 tid 1 2024-09-10T16:22:15.246 INFO:tasks.rados.rados.0.smithi165.stdout:2427: writing smithi16534841-17 from 1857998 to 2386792 tid 2 2024-09-10T16:22:15.246 INFO:tasks.rados.rados.0.smithi165.stdout:2428: read oid 22 snap -1 2024-09-10T16:22:15.246 INFO:tasks.rados.rados.0.smithi165.stdout:2428: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:22:15.246 INFO:tasks.rados.rados.0.smithi165.stdout:2429: write oid 9 current snap is 263 2024-09-10T16:22:15.246 INFO:tasks.rados.rados.0.smithi165.stdout:2429: seq_num 857 ranges {403599=719929,1916480=441635} 2024-09-10T16:22:15.251 INFO:tasks.rados.rados.0.smithi165.stdout:2429: writing smithi16534841-9 from 403599 to 1123528 tid 1 2024-09-10T16:22:15.259 INFO:tasks.rados.rados.0.smithi165.stdout:2429: writing smithi16534841-9 from 1916480 to 2358115 tid 2 2024-09-10T16:22:15.259 INFO:tasks.rados.rados.0.smithi165.stdout:2430: copy_from oid 36 from oid 39 current snap is 263 2024-09-10T16:22:15.259 INFO:tasks.rados.rados.0.smithi165.stdout:2426: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:22:15.374 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 611483 2024-09-10T16:22:15.431 INFO:tasks.rados.rados.0.smithi165.stdout:2427: finishing write tid 1 to smithi16534841-17 2024-09-10T16:22:15.432 INFO:tasks.rados.rados.0.smithi165.stdout:2427: finishing write tid 2 to smithi16534841-17 2024-09-10T16:22:15.432 INFO:tasks.rados.rados.0.smithi165.stdout:2427: finishing write tid 3 to smithi16534841-17 2024-09-10T16:22:15.432 INFO:tasks.rados.rados.0.smithi165.stdout:2427: finishing write tid 4 to smithi16534841-17 2024-09-10T16:22:15.432 INFO:tasks.rados.rados.0.smithi165.stdout:2427: oid 17 updating version 0 to 1148 2024-09-10T16:22:15.432 INFO:tasks.rados.rados.0.smithi165.stdout:2427: oid 17 updating version 1148 to 1149 2024-09-10T16:22:15.433 INFO:tasks.rados.rados.0.smithi165.stdout:2427: oid 17 version 1149 is already newer than 1147 2024-09-10T16:22:15.433 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1149 (ObjNum 856 snap 263 seq_num 856) dirty exists 2024-09-10T16:22:15.433 INFO:tasks.rados.rados.0.smithi165.stdout:2427: left oid 17 (ObjNum 856 snap 263 seq_num 856) 2024-09-10T16:22:15.433 INFO:tasks.rados.rados.0.smithi165.stdout:2428: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:22:15.473 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:15.544 INFO:tasks.rados.rados.0.smithi165.stdout:2429: finishing write tid 1 to smithi16534841-9 2024-09-10T16:22:15.544 INFO:tasks.rados.rados.0.smithi165.stdout:2429: finishing write tid 2 to smithi16534841-9 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: finishing write tid 3 to smithi16534841-9 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: finishing write tid 4 to smithi16534841-9 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: oid 9 updating version 0 to 1209 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: oid 9 version 1209 is already newer than 1208 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: oid 9 version 1209 is already newer than 1207 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1209 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2429: left oid 9 (ObjNum 857 snap 263 seq_num 857) 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2430: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2430: finishing copy_from to smithi16534841-36 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1369 (ObjNum 631 snap 197 seq_num 631) dirty exists 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2426: done (4 left) 2024-09-10T16:22:15.545 INFO:tasks.rados.rados.0.smithi165.stdout:2427: done (3 left) 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2428: done (2 left) 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2429: done (1 left) 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2430: done (0 left) 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2431: rollback oid 38 current snap is 263 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 262 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2432: read oid 20 snap -1 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2432: expect deleted 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2433: delete oid 5 current snap is 263 2024-09-10T16:22:15.546 INFO:tasks.rados.rados.0.smithi165.stdout:2431: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:22:15.547 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1339 (ObjNum 513 snap 0 seq_num 1446270006) dirty dne 2024-09-10T16:22:15.547 INFO:tasks.rados.rados.0.smithi165.stdout:2431: done (2 left) 2024-09-10T16:22:15.547 INFO:tasks.rados.rados.0.smithi165.stdout:2432: done (1 left) 2024-09-10T16:22:15.547 INFO:tasks.rados.rados.0.smithi165.stdout:2433: done (0 left) 2024-09-10T16:22:15.547 INFO:tasks.rados.rados.0.smithi165.stdout:2434: delete oid 45 current snap is 263 2024-09-10T16:22:15.550 INFO:tasks.rados.rados.0.smithi165.stdout:2434: done (0 left) 2024-09-10T16:22:15.550 INFO:tasks.rados.rados.0.smithi165.stdout:2435: write (excl) oid 39 current snap is 263 2024-09-10T16:22:15.550 INFO:tasks.rados.rados.0.smithi165.stdout:2435: seq_num 860 ranges {454020=662029,1691926=588962} 2024-09-10T16:22:15.556 INFO:tasks.rados.rados.0.smithi165.stdout:2435: writing smithi16534841-39 from 454020 to 1116049 tid 1 2024-09-10T16:22:15.562 INFO:tasks.rados.rados.0.smithi165.stdout:2435: writing smithi16534841-39 from 1691926 to 2280888 tid 2 2024-09-10T16:22:15.562 INFO:tasks.rados.rados.0.smithi165.stdout:2436: delete oid 16 current snap is 263 2024-09-10T16:22:15.562 INFO:tasks.rados.rados.0.smithi165.stdout:2435: finishing write tid 1 to smithi16534841-39 2024-09-10T16:22:15.565 INFO:tasks.rados.rados.0.smithi165.stdout:2436: done (1 left) 2024-09-10T16:22:15.565 INFO:tasks.rados.rados.0.smithi165.stdout:2437: delete oid 43 current snap is 263 2024-09-10T16:22:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:2435: finishing write tid 2 to smithi16534841-39 2024-09-10T16:22:15.566 INFO:tasks.rados.rados.0.smithi165.stdout:2435: finishing write tid 3 to smithi16534841-39 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: finishing write tid 4 to smithi16534841-39 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: oid 39 updating version 0 to 1377 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: oid 39 version 1377 is already newer than 1376 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: oid 39 version 1377 is already newer than 1375 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1377 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: left oid 39 (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2435: done (1 left) 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2437: done (0 left) 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2438: read oid 8 snap -1 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2438: expect (ObjNum 767 snap 235 seq_num 767) 2024-09-10T16:22:15.567 INFO:tasks.rados.rados.0.smithi165.stdout:2439: snap_create 2024-09-10T16:22:15.570 INFO:tasks.rados.rados.0.smithi165.stdout:2438: expect (ObjNum 767 snap 235 seq_num 767) 2024-09-10T16:22:15.579 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1310201 2024-09-10T16:22:15.642 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 839149 2024-09-10T16:22:15.963 INFO:tasks.rados.rados.0.smithi165.stdout:2438: done (1 left) 2024-09-10T16:22:15.964 INFO:tasks.rados.rados.0.smithi165.stdout:2439: done (0 left) 2024-09-10T16:22:15.964 INFO:tasks.rados.rados.0.smithi165.stdout:2440: snap_create 2024-09-10T16:22:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2440: done (0 left) 2024-09-10T16:22:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2441: write oid 6 current snap is 265 2024-09-10T16:22:16.964 INFO:tasks.rados.rados.0.smithi165.stdout:2441: seq_num 863 ranges {504441=487776,1467372=736289} 2024-09-10T16:22:16.969 INFO:tasks.rados.rados.0.smithi165.stdout:2441: writing smithi16534841-6 from 504441 to 992217 tid 1 2024-09-10T16:22:16.975 INFO:tasks.rados.rados.0.smithi165.stdout:2441: writing smithi16534841-6 from 1467372 to 2203661 tid 2 2024-09-10T16:22:16.975 INFO:tasks.rados.rados.0.smithi165.stdout:2442: delete oid 24 current snap is 265 2024-09-10T16:22:16.976 INFO:tasks.rados.rados.0.smithi165.stdout:2441: finishing write tid 1 to smithi16534841-6 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2441: finishing write tid 2 to smithi16534841-6 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2442: done (1 left) 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2443: delete oid 28 current snap is 265 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2441: finishing write tid 3 to smithi16534841-6 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2441: finishing write tid 4 to smithi16534841-6 2024-09-10T16:22:16.983 INFO:tasks.rados.rados.0.smithi165.stdout:2441: oid 6 updating version 0 to 1325 2024-09-10T16:22:16.984 INFO:tasks.rados.rados.0.smithi165.stdout:2441: oid 6 updating version 1325 to 1326 2024-09-10T16:22:16.984 INFO:tasks.rados.rados.0.smithi165.stdout:2441: oid 6 updating version 1326 to 1327 2024-09-10T16:22:16.984 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1327 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-10T16:22:16.984 INFO:tasks.rados.rados.0.smithi165.stdout:2441: left oid 6 (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2441: done (1 left) 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2443: done (0 left) 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2444: rollback oid 40 current snap is 265 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 255 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2445: copy_from oid 25 from oid 31 current snap is 265 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2446: write oid 28 current snap is 265 2024-09-10T16:22:16.985 INFO:tasks.rados.rados.0.smithi165.stdout:2446: seq_num 866 ranges {554862=713523,1926465=599969} 2024-09-10T16:22:16.991 INFO:tasks.rados.rados.0.smithi165.stdout:2446: writing smithi16534841-28 from 554862 to 1268385 tid 1 2024-09-10T16:22:16.998 INFO:tasks.rados.rados.0.smithi165.stdout:2446: writing smithi16534841-28 from 1926465 to 2526434 tid 2 2024-09-10T16:22:16.998 INFO:tasks.rados.rados.0.smithi165.stdout:2447: snap_remove snap 256 2024-09-10T16:22:17.631 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:22:17.631 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:22:17.631 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:22:17.635 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:22:17.635 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:22:17.635 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:22:17.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:22:32.636 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:22:32.637 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:22:32.637 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:22:32.637 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.744+0000 7f59d86fe640 10 monclient: get_monmap_and_config 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.744+0000 7f59d86fe640 10 monclient: build_initial_monmap 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.744+0000 7f59d86fe640 10 monclient: monmap: 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:32.745753+0000 2024-09-10T16:22:32.746 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:32.745753+0000 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:32.747 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.745+0000 7f59d86fe640 1 Processor -- start 2024-09-10T16:22:32.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.745+0000 7f59d86fe640 1 -- start start 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.745+0000 7f59d86fe640 10 monclient: init 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f59d009d930 0x7f59d009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-c con 0x7f59d009d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f59d009ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d009e250 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d00a0870 con 0x7f59d009e250 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59d01353a0 con 0x7f59d009ab50 2024-09-10T16:22:32.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f59d009d3e0 con 0x7f59d009d930 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d86fe640 10 monclient(hunting): authenticate will time out at 1731.233207s 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d6c9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f59d009e250 0x7f59d0134c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35122/0 (socket says 172.21.15.164:35122) 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d6c9b640 1 -- 172.21.15.164:0/2657248631 learned_addr learned my addr 172.21.15.164:0/2657248631 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 1 --2- 172.21.15.164:0/2657248631 >> v2:172.21.15.165:3300/0 conn(0x7f59d009d930 0x7f59d009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): get_auth_request con 0x7f59d009d930 auth_method 0 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:32.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient(hunting): handle_auth_done global_id 154339 payload 274 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 1 -- 172.21.15.164:0/2657248631 >> v1:172.21.15.165:6789/0 conn(0x7f59d009ab50 legacy=0x7f59d0134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 1 -- 172.21.15.164:0/2657248631 >> v1:172.21.15.164:6789/0 conn(0x7f59d009e250 legacy=0x7f59d0134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient: _finish_hunting 0 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 1 monclient: found mon.noname-c 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:22:32.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.746+0000 7f59d649a640 1 -- 172.21.15.164:0/2657248631 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d0074cc0 con 0x7f59d009d930 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.749+0000 7f59d649a640 1 -- 172.21.15.164:0/2657248631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009d930 msgr2=0x7f59d009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d649a640 1 --2- 172.21.15.164:0/2657248631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009d930 0x7f59d009dd10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c0029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9ca00541292e6a3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 1 -- 172.21.15.164:0/2657248631 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0037070 con 0x7f59d009d930 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: dump: 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.759 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: _finish_auth 0 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: _check_auth_tickets 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 1 -- 172.21.15.164:0/2657248631 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59c0032640 con 0x7f59d009d930 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 1 -- 172.21.15.164:0/2657248631 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0032a10 con 0x7f59d009d930 2024-09-10T16:22:32.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d5498640 10 monclient: dump: 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 5 monclient: authenticate success, global_id 154339 2024-09-10T16:22:32.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 4 set_mon_vals no callback set 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 10 monclient: shutdown 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 1 -- 172.21.15.164:0/2657248631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009d930 msgr2=0x7f59d009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.750+0000 7f59d86fe640 1 --2- 172.21.15.164:0/2657248631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009d930 0x7f59d009dd10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 -- 172.21.15.164:0/2657248631 shutdown_connections 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 --2- 172.21.15.164:0/2657248631 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009d930 0x7f59d009dd10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 -- 172.21.15.164:0/2657248631 >> 172.21.15.164:0/2657248631 conn(0x7f59d00a3110 msgr2=0x7f59d00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 -- 172.21.15.164:0/2657248631 shutdown_connections 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 -- 172.21.15.164:0/2657248631 wait complete. 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 10 monclient: build_initial_monmap 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 10 monclient: monmap: 2024-09-10T16:22:32.762 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.751+0000 7f59d86fe640 1 Processor -- start 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 -- start start 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient: init 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:32.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-a con 0x7f59d009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-c con 0x7f59d009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 0x7f59d012fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): picked mon.noname-b con 0x7f59d009e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f59d009e6e0 con 0x7f59d009ab50 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f59d00a02b0 con 0x7f59d009e250 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f59d009ef30 con 0x7f59d009d930 2024-09-10T16:22:32.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d86fe640 10 monclient(hunting): authenticate will time out at 1731.239441s 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35936/0 (socket says 172.21.15.164:35936) 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 -- 172.21.15.164:0/1909930745 learned_addr learned my addr 172.21.15.164:0/1909930745 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): get_auth_request con 0x7f59d009ab50 auth_method 0 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 10 monclient(hunting): get_auth_request con 0x7f59d009d930 auth_method 0 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:32.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 msgr2=0x7f59d012f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 msgr2=0x7f59d012f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d5c99640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 0x7f59d012fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 10 monclient(hunting): get_auth_request con 0x7f59d009e250 auth_method 0 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:32.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient(hunting): handle_auth_done global_id 164271 payload 824 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 msgr2=0x7f59d012f6c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 msgr2=0x7f59d012fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 0x7f59d012fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient: _finish_hunting 0 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 monclient: found mon.noname-a 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d649a640 1 -- 172.21.15.164:0/1909930745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59d01328b0 con 0x7f59d009ab50 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.752+0000 7f59d6c9b640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 0x7f59d012fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d649a640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 msgr2=0x7f59d012f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d649a640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:32.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0041070 con 0x7f59d009ab50 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: dump: 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:32.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59c003c880 con 0x7f59d009ab50 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 5 monclient: authenticate success, global_id 164271 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 10 monclient: _renew_subs 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59d0132b90 con 0x7f59d009ab50 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59c0037040 con 0x7f59d009ab50 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:32.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59c6ffd640 10 monclient: dump: 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d4c97640 4 set_mon_vals no callback set 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 10 monclient: _renew_subs 2024-09-10T16:22:32.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.755+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59d00a5b80 con 0x7f59d009ab50 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f59c0044070 con 0x7f59d009ab50 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d86fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d86fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d86fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59d009d620 con 0x7f59d009ab50 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59c6ffd640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 0x7f59b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(554..554 src has 481..554) ==== 4755+0+0 (secure 0 0 0) 0x7f59c0073810 con 0x7f59d009ab50 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d5c99640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 0x7f59b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.756+0000 7f59d5c99640 10 monclient: get_auth_request con 0x7f59b403e210 auth_method 0 2024-09-10T16:22:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.759+0000 7f59d5c99640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 0x7f59b40406d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f59d0131960 tx=0x7f59cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.761+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f59c0044c60 con 0x7f59d009ab50 2024-09-10T16:22:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.761+0000 7f59c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:32.761+0000 7f59c6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.002+0000 7f59d86fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:22:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59d86fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59d86fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f59d0131670 con 0x7f59d009ab50 2024-09-10T16:22:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59c6ffd640 1 -- 172.21.15.164:0/1909930745 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f59c0072220 con 0x7f59d009ab50 2024-09-10T16:22:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:33.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.003+0000 7f59c6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:33.005 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":158,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 msgr2=0x7f59b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 0x7f59b40406d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f59d0131960 tx=0x7f59cc009040 comp rx=0 tx=0).stop 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 10 monclient: shutdown 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 msgr2=0x7f59d012f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f59c00097a0 tx=0x7f59c00046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 shutdown_connections 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59b403e210 0x7f59b40406d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f59d009e250 0x7f59d012fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f59d009d930 0x7f59d012f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 --2- 172.21.15.164:0/1909930745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f59d009ab50 0x7f59d012f180 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 >> 172.21.15.164:0/1909930745 conn(0x7f59d00a3110 msgr2=0x7f59d0074140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 shutdown_connections 2024-09-10T16:22:33.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.005+0000 7f59d86fe640 1 -- 172.21.15.164:0/1909930745 wait complete. 2024-09-10T16:22:33.017 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:22:33.017 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:22:33.017 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.114+0000 7ff132257640 10 monclient: get_monmap_and_config 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.114+0000 7ff132257640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.114+0000 7ff132257640 10 monclient: monmap: 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.116 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:33.115678+0000 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:33.115678+0000 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:33.117 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.115+0000 7ff132257640 1 Processor -- start 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.115+0000 7ff132257640 1 -- start start 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.115+0000 7ff132257640 10 monclient: init 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient: _add_conns ranks=[0,2,3,1,4] 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff12c09d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 0x7ff12c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): picked mon.noname-b con 0x7ff12c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff12c09e420 0x7ff12c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c09e420 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12c0a0870 con 0x7ff12c09d930 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff12c09d3e0 con 0x7ff12c09e420 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c074c70 con 0x7ff12c09ab50 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff132257640 10 monclient(hunting): authenticate will time out at 1731.603128s 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 0x7ff12c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff12c09d930 0x7ff12c09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35138/0 (socket says 172.21.15.164:35138) 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12b7fe640 1 -- 172.21.15.164:0/3369173974 learned_addr learned my addr 172.21.15.164:0/3369173974 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): get_auth_request con 0x7ff12c09ab50 auth_method 0 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 1 --2- 172.21.15.164:0/3369173974 >> v2:172.21.15.164:3300/0 conn(0x7ff12c09e420 0x7ff12c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): get_auth_request con 0x7ff12c09e420 auth_method 0 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient(hunting): handle_auth_done global_id 184100 payload 274 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 -- 172.21.15.164:0/3369173974 >> v2:172.21.15.164:3300/0 conn(0x7ff12c09e420 msgr2=0x7ff12c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 --2- 172.21.15.164:0/3369173974 >> v2:172.21.15.164:3300/0 conn(0x7ff12c09e420 0x7ff12c13bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 -- 172.21.15.164:0/3369173974 >> v1:172.21.15.164:6789/0 conn(0x7ff12c09d930 legacy=0x7ff12c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 monclient: found mon.noname-b 2024-09-10T16:22:33.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.116+0000 7ff12affd640 1 -- 172.21.15.164:0/3369173974 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c09ecb0 con 0x7ff12c09ab50 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12affd640 1 -- 172.21.15.164:0/3369173974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 msgr2=0x7ff12c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12affd640 1 --2- 172.21.15.164:0/3369173974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 0x7ff12c1397c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff1200097a0 tx=0x7ff12002a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5641db11a224a6a2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 1 -- 172.21.15.164:0/3369173974 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120037070 con 0x7ff12c09ab50 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: dump: 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.130 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 1 -- 172.21.15.164:0/3369173974 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff120032440 con 0x7ff12c09ab50 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff132257640 5 monclient: authenticate success, global_id 184100 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff132257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:33.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 1 -- 172.21.15.164:0/3369173974 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120032810 con 0x7ff12c09ab50 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff132257640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff132257640 4 set_mon_vals no callback set 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.119+0000 7ff12a7fc640 10 monclient: dump: 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.132 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 10 monclient: shutdown 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 -- 172.21.15.164:0/3369173974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 msgr2=0x7ff12c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 --2- 172.21.15.164:0/3369173974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 0x7ff12c1397c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ff1200097a0 tx=0x7ff12002a050 comp rx=0 tx=0).stop 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 -- 172.21.15.164:0/3369173974 shutdown_connections 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 --2- 172.21.15.164:0/3369173974 >> v2:172.21.15.164:3300/0 conn(0x7ff12c09e420 0x7ff12c13bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 --2- 172.21.15.164:0/3369173974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09ab50 0x7ff12c1397c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 -- 172.21.15.164:0/3369173974 >> 172.21.15.164:0/3369173974 conn(0x7ff12c0a3110 msgr2=0x7ff12c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 -- 172.21.15.164:0/3369173974 shutdown_connections 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.120+0000 7ff132257640 1 -- 172.21.15.164:0/3369173974 wait complete. 2024-09-10T16:22:33.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 10 monclient: monmap: 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 1 Processor -- start 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 1 -- start start 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 10 monclient: init 2024-09-10T16:22:33.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.121+0000 7ff132257640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): picked mon.noname-a con 0x7ff12c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09d930 0x7ff12c099b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): picked mon.noname-c con 0x7ff12c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff12c09e420 0x7ff12c137790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): picked mon.noname-b con 0x7ff12c09e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0a2940 con 0x7ff12c09ab50 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c09ef30 con 0x7ff12c09e420 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff12c0a2ac0 con 0x7ff12c09d930 2024-09-10T16:22:33.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff132257640 10 monclient(hunting): authenticate will time out at 1731.609061s 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09d930 0x7ff12c099b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35946/0 (socket says 172.21.15.164:35946) 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 -- 172.21.15.164:0/2433032858 learned_addr learned my addr 172.21.15.164:0/2433032858 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): get_auth_request con 0x7ff12c09ab50 auth_method 0 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): get_auth_request con 0x7ff12c09d930 auth_method 0 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff12c09e420 0x7ff12c137790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 10 monclient(hunting): get_auth_request con 0x7ff12c09e420 auth_method 0 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient(hunting): handle_auth_done global_id 164277 payload 824 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09d930 msgr2=0x7ff12c099b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09d930 0x7ff12c099b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff12c09e420 msgr2=0x7ff12c137790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff12c09e420 0x7ff12c137790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12b7fe640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12c09d620 con 0x7ff12c09ab50 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12affd640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff12c09d930 0x7ff12c099b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.122+0000 7ff12bfff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff12c09e420 0x7ff12c137790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:22:33.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff12b7fe640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 msgr2=0x7ff12c0995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff12b7fe640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00cc10 tx=0x7ff11c00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c012400 con 0x7ff12c09ab50 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: dump: 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.139 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 5 monclient: authenticate success, global_id 164277 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff11c0125e0 con 0x7ff12c09ab50 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 10 monclient: _renew_subs 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff12c13aa20 con 0x7ff12c09ab50 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff11c0129d0 con 0x7ff12c09ab50 2024-09-10T16:22:33.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff113fff640 10 monclient: dump: 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 10 monclient: _renew_subs 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff129ffb640 4 set_mon_vals no callback set 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.124+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff12c139160 con 0x7ff12c09ab50 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff132257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff132257640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 0x7ff12c137cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c13b0c0 con 0x7ff12c1395d0 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 0x7ff12c137cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient: get_auth_request con 0x7ff12c1395d0 auth_method 0 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff11c012bf0 con 0x7ff12c09ab50 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.125+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff12affd640 10 monclient(hunting): handle_auth_done global_id 164280 payload 824 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff12affd640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff12c09dd10 con 0x7ff12c1395d0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff113fff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff11803e210 0x7ff1180406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(554..554 src has 481..554) ==== 4755+0+0 (secure 0 0 0) 0x7ff11c053480 con 0x7ff12c09ab50 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff12bfff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff11803e210 0x7ff1180406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.126+0000 7ff12bfff640 10 monclient: get_auth_request con 0x7ff11803e210 auth_method 0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff12affd640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 msgr2=0x7ff12c137cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff12affd640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 0x7ff12c137cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff12c06d500 tx=0x7ff1200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120043070 con 0x7ff12c1395d0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 10 monclient: dump: 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.143 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff12003ca60 con 0x7ff12c1395d0 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 msgr2=0x7ff12c137cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 0x7ff12c137cd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff12c06d500 tx=0x7ff1200046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 msgr2=0x7ff12c137cd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.128+0000 7ff113fff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c1395d0 0x7ff12c137cd0 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff12c06d500 tx=0x7ff1200046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.129+0000 7ff12bfff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff11803e210 0x7ff1180406d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff124009b10 tx=0x7ff124008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff132257640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:33.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff132257640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 0x7ff12c1385f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff132257640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff12c0a6170 con 0x7ff12c138210 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 0x7ff12c1385f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient: get_auth_request con 0x7ff12c138210 auth_method 0 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.134+0000 7ff12affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.135+0000 7ff12affd640 10 monclient(hunting): handle_auth_done global_id 164283 payload 824 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.135+0000 7ff12affd640 1 -- 172.21.15.164:0/2433032858 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff12c06daa0 con 0x7ff12c138210 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff12affd640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 msgr2=0x7ff12c1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff12affd640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 0x7ff12c1385f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff120000c00 tx=0x7ff120008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff120037040 con 0x7ff12c138210 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 10 monclient: dump: 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7ff1200039c0 con 0x7ff12c138210 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 msgr2=0x7ff12c1385f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 0x7ff12c1385f0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff120000c00 tx=0x7ff120008030 comp rx=0 tx=0).stop 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 msgr2=0x7ff12c1385f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff113fff640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c138210 0x7ff12c1385f0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff120000c00 tx=0x7ff120008030 comp rx=0 tx=0).stop 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:33.149 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 123839, 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.150 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.151 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:33.152 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.153 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:33.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.155 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:33.156 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.157 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:33.158 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.159 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.160 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.162 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:33.162 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff11803e210 msgr2=0x7ff1180406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff11803e210 0x7ff1180406d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff124009b10 tx=0x7ff124008040 comp rx=0 tx=0).stop 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 10 monclient: shutdown 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 msgr2=0x7ff12c0995f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff11c00cc10 tx=0x7ff11c00a430 comp rx=0 tx=0).stop 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff12bfff640 1 -- 172.21.15.164:0/2433032858 reap_dead start 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.137+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 shutdown_connections 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.138+0000 7ff132257640 1 --2- 172.21.15.164:0/2433032858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff12c09ab50 0x7ff12c0995f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.138+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 >> 172.21.15.164:0/2433032858 conn(0x7ff12c0a3110 msgr2=0x7ff12c09af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.138+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 shutdown_connections 2024-09-10T16:22:33.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.138+0000 7ff132257640 1 -- 172.21.15.164:0/2433032858 wait complete. 2024-09-10T16:22:33.164 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:33.164 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:22:33.164 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.261+0000 7f60d9e60640 10 monclient: get_monmap_and_config 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.261+0000 7f60d9e60640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.261+0000 7f60d9e60640 10 monclient: monmap: 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:33.262413+0000 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:33.262413+0000 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:33.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:33.264 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:33.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 Processor -- start 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 -- start start 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient: init 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f60d409c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f60d409d190 0x7f60d409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-a con 0x7f60d409d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f60d413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d409fbc0 con 0x7f60d409c310 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f60d40a2a20 con 0x7f60d413a5c0 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f60d409c6f0 con 0x7f60d409d190 2024-09-10T16:22:33.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d9e60640 10 monclient(hunting): authenticate will time out at 1731.749831s 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f60d409d190 0x7f60d409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f60d409c310 0x7f60d409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35154/0 (socket says 172.21.15.164:35154) 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d37fe640 1 -- 172.21.15.164:0/3083932326 learned_addr learned my addr 172.21.15.164:0/3083932326 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request con 0x7f60d409d190 auth_method 0 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.262+0000 7f60d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_done global_id 164286 payload 274 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 1 -- 172.21.15.164:0/3083932326 >> v1:172.21.15.165:6789/0 conn(0x7f60d413a5c0 legacy=0x7f60d413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 1 -- 172.21.15.164:0/3083932326 >> v1:172.21.15.164:6789/0 conn(0x7f60d409c310 legacy=0x7f60d409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 1 monclient: found mon.noname-a 2024-09-10T16:22:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.263+0000 7f60d2ffd640 1 -- 172.21.15.164:0/3083932326 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d409a4d0 con 0x7f60d409d190 2024-09-10T16:22:33.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d2ffd640 1 -- 172.21.15.164:0/3083932326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409d190 msgr2=0x7f60d409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/3083932326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409d190 0x7f60d409f150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097a0 tx=0x7f60c8029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be4ff95cefbb7c49 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 1 -- 172.21.15.164:0/3083932326 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8037070 con 0x7f60d409d190 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: dump: 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 1 -- 172.21.15.164:0/3083932326 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c8032640 con 0x7f60d409d190 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 5 monclient: authenticate success, global_id 164286 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 1 -- 172.21.15.164:0/3083932326 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8032a10 con 0x7f60d409d190 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 4 set_mon_vals no callback set 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d27fc640 10 monclient: dump: 2024-09-10T16:22:33.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 10 monclient: shutdown 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 -- 172.21.15.164:0/3083932326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409d190 msgr2=0x7f60d409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 --2- 172.21.15.164:0/3083932326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409d190 0x7f60d409f150 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f60c80097a0 tx=0x7f60c8029fd0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 -- 172.21.15.164:0/3083932326 shutdown_connections 2024-09-10T16:22:33.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 --2- 172.21.15.164:0/3083932326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d409d190 0x7f60d409f150 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 -- 172.21.15.164:0/3083932326 >> 172.21.15.164:0/3083932326 conn(0x7f60d40a3110 msgr2=0x7f60d40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 -- 172.21.15.164:0/3083932326 shutdown_connections 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.266+0000 7f60d9e60640 1 -- 172.21.15.164:0/3083932326 wait complete. 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.267+0000 7f60d9e60640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.267+0000 7f60d9e60640 10 monclient: monmap: 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:33.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.267+0000 7f60d9e60640 1 Processor -- start 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.267+0000 7f60d9e60640 1 -- start start 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.267+0000 7f60d9e60640 10 monclient: init 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 0x7f60d41cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-c con 0x7f60d409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 0x7f60d41cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-a con 0x7f60d413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d4092390 0x7f60d4092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): picked mon.noname-b con 0x7f60d4092390 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f60d40a2230 con 0x7f60d413a5c0 2024-09-10T16:22:33.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f60d40a20b0 con 0x7f60d4092390 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d4074e00 con 0x7f60d409c310 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d9e60640 10 monclient(hunting): authenticate will time out at 1731.755366s 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 0x7f60d41cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 0x7f60d41cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 0x7f60d41cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39668/0 (socket says 172.21.15.164:39668) 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 0x7f60d41cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35990/0 (socket says 172.21.15.164:35990) 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 -- 172.21.15.164:0/1605554089 learned_addr learned my addr 172.21.15.164:0/1605554089 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): get_auth_request con 0x7f60d409c310 auth_method 0 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request con 0x7f60d413a5c0 auth_method 0 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d3fff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d4092390 0x7f60d4092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d3fff640 10 monclient(hunting): get_auth_request con 0x7f60d4092390 auth_method 0 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient(hunting): handle_auth_done global_id 184106 payload 824 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d4092390 msgr2=0x7f60d4092800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f60d4092390 0x7f60d4092800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 msgr2=0x7f60d41cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 0x7f60d41cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 monclient: found mon.noname-c 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d37fe640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60d4095440 con 0x7f60d409c310 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.268+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f60d413a5c0 0x7f60d41cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d37fe640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 msgr2=0x7f60d41cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d37fe640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 0x7f60d41cda00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c460 tx=0x7f60c400c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4007070 con 0x7f60d409c310 2024-09-10T16:22:33.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: dump: 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.281 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f60c4007230 con 0x7f60d409c310 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 5 monclient: authenticate success, global_id 184106 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 10 monclient: _renew_subs 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60d4095720 con 0x7f60d409c310 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c4007620 con 0x7f60d409c310 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60bbfff640 10 monclient: dump: 2024-09-10T16:22:33.282 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 10 monclient: _renew_subs 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f60d409d620 con 0x7f60d409c310 2024-09-10T16:22:33.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.271+0000 7f60d1ffb640 4 set_mon_vals no callback set 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d9e60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d9e60640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 0x7f60d4092d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d406ec90 con 0x7f60d409d190 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f60c401b400 con 0x7f60d409c310 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 0x7f60d4092d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient: get_auth_request con 0x7f60d409d190 auth_method 0 2024-09-10T16:22:33.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60bbfff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60c003e290 0x7f60c0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_done global_id 184109 payload 824 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(554..554 src has 481..554) ==== 4755+0+0 (secure 0 0 0) 0x7f60c4052920 con 0x7f60d409c310 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d2ffd640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f60d409f7b0 con 0x7f60d409d190 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d3fff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60c003e290 0x7f60c0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.272+0000 7f60d3fff640 10 monclient: get_auth_request con 0x7f60c003e290 auth_method 0 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.274+0000 7f60d2ffd640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 msgr2=0x7f60d4092d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.274+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 0x7f60d4092d40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60c8034000 tx=0x7f60c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8037040 con 0x7f60d409d190 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 10 monclient: dump: 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f60c803cb10 con 0x7f60d409d190 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 msgr2=0x7f60d4092d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 0x7f60d4092d40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60c8034000 tx=0x7f60c80046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 msgr2=0x7f60d4092d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60bbfff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409d190 0x7f60d4092d40 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f60c8034000 tx=0x7f60c80046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.275+0000 7f60d3fff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60c003e290 0x7f60c0040750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f60cc005520 tx=0x7f60cc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d9e60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d9e60640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 0x7f60d41389f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d9e60640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f60d409ed00 con 0x7f60d4093280 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 0x7f60d41389f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient: get_auth_request con 0x7f60d4093280 auth_method 0 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 10 monclient(hunting): handle_auth_done global_id 184112 payload 824 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.281+0000 7f60d2ffd640 1 -- 172.21.15.164:0/1605554089 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f60d4093f80 con 0x7f60d4093280 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d2ffd640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 msgr2=0x7f60d41389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d2ffd640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 0x7f60d41389f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f60c8000c00 tx=0x7f60c8008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60c8050070 con 0x7f60d4093280 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 10 monclient: dump: 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f60c8003ac0 con 0x7f60d4093280 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:33.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 msgr2=0x7f60d41389f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 0x7f60d41389f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f60c8000c00 tx=0x7f60c8008030 comp rx=0 tx=0).stop 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 msgr2=0x7f60d41389f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60bbfff640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d4093280 0x7f60d41389f0 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f60c8000c00 tx=0x7f60c8008030 comp rx=0 tx=0).stop 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 15563, 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:33.293 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.294 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.295 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:33.296 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.297 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.298 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.299 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.300 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:33.301 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:33.303 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:33.304 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60c003e290 msgr2=0x7f60c0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f60c003e290 0x7f60c0040750 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f60cc005520 tx=0x7f60cc009040 comp rx=0 tx=0).stop 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 10 monclient: shutdown 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 msgr2=0x7f60d41cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 --2- 172.21.15.164:0/1605554089 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f60d409c310 0x7f60d41cda00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f60c400c460 tx=0x7f60c400c920 comp rx=0 tx=0).stop 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d3fff640 1 -- 172.21.15.164:0/1605554089 reap_dead start 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d3fff640 1 -- 172.21.15.164:0/1605554089 reap_dead start 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 shutdown_connections 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.284+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 >> 172.21.15.164:0/1605554089 conn(0x7f60d40a3110 msgr2=0x7f60d41376b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.285+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 shutdown_connections 2024-09-10T16:22:33.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.285+0000 7f60d9e60640 1 -- 172.21.15.164:0/1605554089 wait complete. 2024-09-10T16:22:33.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:33.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:22:33.307 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.403+0000 7fa768685640 10 monclient: get_monmap_and_config 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.403+0000 7fa768685640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.403+0000 7fa768685640 10 monclient: monmap: 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:33.404737+0000 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:33.404737+0000 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.405 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:33.406 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.404+0000 7fa768685640 1 Processor -- start 2024-09-10T16:22:33.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.404+0000 7fa768685640 1 -- start start 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.404+0000 7fa768685640 10 monclient: init 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): picked mon.noname-b con 0x7fa760134a50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): picked mon.noname-a con 0x7fa76009e9b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7600a1270 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7600a05a0 con 0x7fa7600a1270 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa7601358b0 con 0x7fa76009e9b0 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa76009ffe0 con 0x7fa760134a50 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa768685640 10 monclient(hunting): authenticate will time out at 1731.892156s 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39690/0 (socket says 172.21.15.164:39690) 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35992/0 (socket says 172.21.15.164:35992) 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 -- 172.21.15.164:0/806359095 learned_addr learned my addr 172.21.15.164:0/806359095 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): get_auth_request con 0x7fa760134a50 auth_method 0 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): get_auth_request con 0x7fa76009e9b0 auth_method 0 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient(hunting): handle_auth_done global_id 184115 payload 274 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 -- 172.21.15.164:0/806359095 >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 msgr2=0x7fa760135370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 --2- 172.21.15.164:0/806359095 >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 -- 172.21.15.164:0/806359095 >> v1:172.21.15.164:6789/0 conn(0x7fa7600a1270 legacy=0x7fa76009a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 monclient: found mon.noname-b 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa766421640 1 -- 172.21.15.164:0/806359095 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7600a02c0 con 0x7fa760134a50 2024-09-10T16:22:33.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.405+0000 7fa765c20640 1 --2- 172.21.15.164:0/806359095 >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa766421640 1 -- 172.21.15.164:0/806359095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 msgr2=0x7fa760134e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa766421640 1 --2- 172.21.15.164:0/806359095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009870 tx=0x7fa75c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e0ba2ada39e7df5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 1 -- 172.21.15.164:0/806359095 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c037070 con 0x7fa760134a50 2024-09-10T16:22:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: dump: 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa768685640 5 monclient: authenticate success, global_id 184115 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 1 -- 172.21.15.164:0/806359095 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa75c032440 con 0x7fa760134a50 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa768685640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa768685640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa768685640 4 set_mon_vals no callback set 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 1 -- 172.21.15.164:0/806359095 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c032810 con 0x7fa760134a50 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.408+0000 7fa76541f640 10 monclient: dump: 2024-09-10T16:22:33.418 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 10 monclient: shutdown 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 -- 172.21.15.164:0/806359095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 msgr2=0x7fa760134e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 --2- 172.21.15.164:0/806359095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa75c009870 tx=0x7fa75c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 -- 172.21.15.164:0/806359095 shutdown_connections 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 --2- 172.21.15.164:0/806359095 >> v2:172.21.15.164:3300/0 conn(0x7fa76009e9b0 0x7fa760135370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 --2- 172.21.15.164:0/806359095 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa760134e30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 -- 172.21.15.164:0/806359095 >> 172.21.15.164:0/806359095 conn(0x7fa7600a3fb0 msgr2=0x7fa7600a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 -- 172.21.15.164:0/806359095 shutdown_connections 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 1 -- 172.21.15.164:0/806359095 wait complete. 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.409+0000 7fa768685640 10 monclient: monmap: 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:33.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 Processor -- start 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 -- start start 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient: init 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7601c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): picked mon.noname-a con 0x7fa76009e9b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:33.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa7601c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): picked mon.noname-b con 0x7fa7600a1270 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): picked mon.noname-c con 0x7fa760134a50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa76009a960 con 0x7fa76009e9b0 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa76009a7e0 con 0x7fa7600a1270 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa76009aae0 con 0x7fa760134a50 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa768685640 10 monclient(hunting): authenticate will time out at 1731.897545s 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7601c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7601c8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:35994/0 (socket says 172.21.15.164:35994) 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 1 -- 172.21.15.164:0/2032777088 learned_addr learned my addr 172.21.15.164:0/2032777088 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): get_auth_request con 0x7fa760134a50 auth_method 0 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): get_auth_request con 0x7fa76009e9b0 auth_method 0 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient(hunting): handle_auth_done global_id 184118 payload 824 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa7601c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 msgr2=0x7fa7601c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa7601c9500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 msgr2=0x7fa7601c8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7601c8fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 monclient: found mon.noname-c 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa76012fe90 con 0x7fa760134a50 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa765c20640 10 monclient: get_auth_request con 0x7fa7600a1270 auth_method 0 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7601c8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.410+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa7601c9500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.411+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 msgr2=0x7fa76012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.411+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 msgr2=0x7fa76012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.411+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7fa75800c780 tx=0x7fa75800cc40 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:33.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.411+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:33.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766c22640 10 monclient: get_auth_request con 0x7fa760134a50 auth_method 0 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766c22640 0 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 msgr2=0x7fa76012c570 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa760134a50 0x7fa76012c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7440054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fa76009e9b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:33.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 0x7fa744008380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fa744005ef0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa74400af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fa7600a1270 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa74400b880 con 0x7fa76009e9b0 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa74400ba00 con 0x7fa7600a1270 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa74400bc00 con 0x7fa744005ef0 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa756ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 0x7fa744008380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): get_auth_request con 0x7fa744005ef0 auth_method 0 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7440054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766421640 10 monclient(hunting): get_auth_request con 0x7fa76009e9b0 auth_method 0 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766421640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa766421640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.612+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:22:33.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 10 monclient(hunting): handle_auth_done global_id 184118 payload 293 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 msgr2=0x7fa74400af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7600a1270 0x7fa74400af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 msgr2=0x7fa7440054b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa76009e9b0 0x7fa7440054b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 monclient: found mon.noname-c 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.613+0000 7fa765c20640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa74400bec0 con 0x7fa744005ef0 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa765c20640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 msgr2=0x7fa744008380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa765c20640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 0x7fa744008380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa75000bf60 tx=0x7fa758003d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75000bb60 con 0x7fa744005ef0 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: dump: 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa750010670 con 0x7fa744005ef0 2024-09-10T16:22:33.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 5 monclient: authenticate success, global_id 184118 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 10 monclient: _renew_subs 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7600a6be0 con 0x7fa744005ef0 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa750010a40 con 0x7fa744005ef0 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa756ffd640 10 monclient: dump: 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 10 monclient: _renew_subs 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa764c1e640 4 set_mon_vals no callback set 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.615+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa76012fbb0 con 0x7fa744005ef0 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa750010be0 con 0x7fa744005ef0 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa768685640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa768685640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 0x7fa7601c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa76009a0d0 con 0x7fa7601c8fc0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa760134a50 0x7fa744048a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(554..554 src has 481..554) ==== 4755+0+0 (secure 0 0 0) 0x7fa750054bd0 con 0x7fa744005ef0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 0x7fa7601c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa766421640 10 monclient: get_auth_request con 0x7fa7601c8fc0 auth_method 0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa766421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa766421640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.616+0000 7fa766421640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa760134a50 0x7fa744048a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766c22640 10 monclient: get_auth_request con 0x7fa760134a50 auth_method 0 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766421640 10 monclient(hunting): handle_auth_done global_id 154351 payload 824 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.617+0000 7fa766421640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7600008d0 con 0x7fa7601c8fc0 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa766421640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 msgr2=0x7fa7601c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 0x7fa7601c93a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa75c035040 tx=0x7fa75c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa766c22640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa760134a50 0x7fa744048a90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa758004970 tx=0x7fa758011000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c042070 con 0x7fa7601c8fc0 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.619+0000 7fa756ffd640 10 monclient: dump: 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.628 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa75c03c830 con 0x7fa7601c8fc0 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 msgr2=0x7fa7601c93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 0x7fa7601c93a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa75c035040 tx=0x7fa75c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 msgr2=0x7fa7601c93a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.620+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c8fc0 0x7fa7601c93a0 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fa75c035040 tx=0x7fa75c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa768685640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:33.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa768685640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 0x7fa760071030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa768685640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa76012dd00 con 0x7fa7601c98e0 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 0x7fa760071030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient: get_auth_request con 0x7fa7601c98e0 auth_method 0 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 10 monclient(hunting): handle_auth_done global_id 154354 payload 824 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.626+0000 7fa766421640 1 -- 172.21.15.164:0/2032777088 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa76012d6f0 con 0x7fa7601c98e0 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa766421640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 msgr2=0x7fa760071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa766421640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 0x7fa760071030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa75c02a0e0 tx=0x7fa75c009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa75c051070 con 0x7fa7601c98e0 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 10 monclient: dump: 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fa75c04da90 con 0x7fa7601c98e0 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 msgr2=0x7fa760071030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 0x7fa760071030 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa75c02a0e0 tx=0x7fa75c009240 comp rx=0 tx=0).stop 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 msgr2=0x7fa760071030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.629+0000 7fa756ffd640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7601c98e0 0x7fa760071030 secure :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa75c02a0e0 tx=0x7fa75c009240 comp rx=0 tx=0).stop 2024-09-10T16:22:33.634 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 172180, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.635 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.636 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.637 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:33.638 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:33.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.641 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:33.643 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.644 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:33.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:22:33.646 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:33.647 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:33.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa760134a50 msgr2=0x7fa744048a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa760134a50 0x7fa744048a90 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fa758004970 tx=0x7fa758011000 comp rx=0 tx=0).stop 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 10 monclient: shutdown 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 msgr2=0x7fa744008380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 --2- 172.21.15.164:0/2032777088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa744005ef0 0x7fa744008380 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa75000bf60 tx=0x7fa758003d10 comp rx=0 tx=0).stop 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 reap_dead start 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa766c22640 1 -- 172.21.15.164:0/2032777088 reap_dead start 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 shutdown_connections 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 >> 172.21.15.164:0/2032777088 conn(0x7fa7600a3fb0 msgr2=0x7fa7600a1db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 shutdown_connections 2024-09-10T16:22:33.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.630+0000 7fa768685640 1 -- 172.21.15.164:0/2032777088 wait complete. 2024-09-10T16:22:33.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:33.650 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:22:33.650 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2447: done (3 left) 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2448: delete oid 2 current snap is 265 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2445: finishing copy_from to smithi16534841-25 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2445: got expected ENOENT (src dne) 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2445: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2444: finishing rollback tid 0 to smithi16534841-40 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2446: finishing write tid 1 to smithi16534841-28 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2444: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2444: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1212 (ObjNum 748 snap 227 seq_num 748) dirty exists 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2446: finishing write tid 2 to smithi16534841-28 2024-09-10T16:22:33.674 INFO:tasks.rados.rados.0.smithi165.stdout:2446: finishing write tid 3 to smithi16534841-28 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:2446: finishing write tid 4 to smithi16534841-28 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:2446: oid 28 updating version 0 to 1100 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:2446: oid 28 updating version 1100 to 1101 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:2446: oid 28 version 1101 is already newer than 1099 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1101 (ObjNum 866 snap 265 seq_num 866) dirty exists 2024-09-10T16:22:33.675 INFO:tasks.rados.rados.0.smithi165.stdout:2446: left oid 28 (ObjNum 866 snap 265 seq_num 866) 2024-09-10T16:22:33.679 INFO:tasks.rados.rados.0.smithi165.stdout:2444: done (3 left) 2024-09-10T16:22:33.679 INFO:tasks.rados.rados.0.smithi165.stdout:2445: done (2 left) 2024-09-10T16:22:33.679 INFO:tasks.rados.rados.0.smithi165.stdout:2446: done (1 left) 2024-09-10T16:22:33.680 INFO:tasks.rados.rados.0.smithi165.stdout:2448: done (0 left) 2024-09-10T16:22:33.680 INFO:tasks.rados.rados.0.smithi165.stdout:2449: read oid 1 snap -1 2024-09-10T16:22:33.680 INFO:tasks.rados.rados.0.smithi165.stdout:2449: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:22:33.680 INFO:tasks.rados.rados.0.smithi165.stdout:2450: delete oid 37 current snap is 265 2024-09-10T16:22:33.681 INFO:tasks.rados.rados.0.smithi165.stdout:2450: done (1 left) 2024-09-10T16:22:33.681 INFO:tasks.rados.rados.0.smithi165.stdout:2451: rollback oid 43 current snap is 265 2024-09-10T16:22:33.681 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 230 2024-09-10T16:22:33.681 INFO:tasks.rados.rados.0.smithi165.stdout:2452: snap_create 2024-09-10T16:22:33.689 INFO:tasks.rados.rados.0.smithi165.stdout:2451: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:22:33.690 INFO:tasks.rados.rados.0.smithi165.stdout:2451: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:22:33.690 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1355 (ObjNum 742 snap 225 seq_num 742) dirty exists 2024-09-10T16:22:33.693 INFO:tasks.rados.rados.0.smithi165.stdout:2449: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.751+0000 7f8850c8c640 10 monclient: get_monmap_and_config 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.751+0000 7f8850c8c640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.751+0000 7f8850c8c640 10 monclient: monmap: 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:33.752759+0000 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:33.752759+0000 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.753 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:33.754 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.752+0000 7f8850c8c640 1 Processor -- start 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.752+0000 7f8850c8c640 1 -- start start 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.752+0000 7f8850c8c640 10 monclient: init 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.752+0000 7f8850c8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.752+0000 7f8850c8c640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f884c09d930 0x7f884c09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-c con 0x7f884c09d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-b con 0x7f884c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f884c134580 0x7f884c134a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f884c134580 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:33.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f884c09d3e0 con 0x7f884c134580 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f884c09e790 con 0x7f884c09d930 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f884c0a0870 con 0x7f884c09ab50 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8850c8c640 10 monclient(hunting): authenticate will time out at 1732.240065s 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f884c134580 0x7f884c134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f884c134580 0x7f884c134a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36014/0 (socket says 172.21.15.164:36014) 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 -- 172.21.15.164:0/4293044760 learned_addr learned my addr 172.21.15.164:0/4293044760 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): get_auth_request con 0x7f884c134580 auth_method 0 2024-09-10T16:22:33.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): get_auth_request con 0x7f884c09ab50 auth_method 0 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 1 --2- 172.21.15.164:0/4293044760 >> v2:172.21.15.165:3300/0 conn(0x7f884c09d930 0x7f884c09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): get_auth_request con 0x7f884c09d930 auth_method 0 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 1 -- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 msgr2=0x7f884c09e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 1 -- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 msgr2=0x7f884c09e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f8843fff640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient(hunting): handle_auth_done global_id 164298 payload 274 2024-09-10T16:22:33.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 -- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 msgr2=0x7f884c09e250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 -- 172.21.15.164:0/4293044760 >> v2:172.21.15.165:3300/0 conn(0x7f884c09d930 msgr2=0x7f884c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 --2- 172.21.15.164:0/4293044760 >> v2:172.21.15.165:3300/0 conn(0x7f884c09d930 0x7f884c09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 monclient: found mon.noname-a 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.753+0000 7f884ad9d640 1 -- 172.21.15.164:0/4293044760 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c074c70 con 0x7f884c134580 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f884ad9d640 1 -- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c134580 msgr2=0x7f884c134a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f884ad9d640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c134580 0x7f884c134a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f884c06d500 tx=0x7f884402c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c9460357026ddf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 1 -- 172.21.15.164:0/4293044760 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884403b070 con 0x7f884c134580 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: dump: 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 5 monclient: authenticate success, global_id 164298 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 1 -- 172.21.15.164:0/4293044760 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8844033880 con 0x7f884c134580 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 1 -- 172.21.15.164:0/4293044760 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8844033c50 con 0x7f884c134580 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 4 set_mon_vals no callback set 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8849d9b640 10 monclient: dump: 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.769 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 10 monclient: shutdown 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 -- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c134580 msgr2=0x7f884c134a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c134580 0x7f884c134a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f884c06d500 tx=0x7f884402c840 comp rx=0 tx=0).stop 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 -- 172.21.15.164:0/4293044760 shutdown_connections 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c134580 0x7f884c134a30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 --2- 172.21.15.164:0/4293044760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09ab50 0x7f884c09e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 --2- 172.21.15.164:0/4293044760 >> v2:172.21.15.165:3300/0 conn(0x7f884c09d930 0x7f884c09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.756+0000 7f8850c8c640 1 -- 172.21.15.164:0/4293044760 >> 172.21.15.164:0/4293044760 conn(0x7f884c0a3110 msgr2=0x7f884c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.757+0000 7f8850c8c640 1 -- 172.21.15.164:0/4293044760 shutdown_connections 2024-09-10T16:22:33.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.757+0000 7f8850c8c640 1 -- 172.21.15.164:0/4293044760 wait complete. 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.757+0000 7f8850c8c640 10 monclient: build_initial_monmap 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.757+0000 7f8850c8c640 10 monclient: monmap: 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 Processor -- start 2024-09-10T16:22:33.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 -- start start 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient: init 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f884c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-c con 0x7f884c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f884c134580 0x7f884c12fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): picked mon.noname-b con 0x7f884c134580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f884c09ef30 con 0x7f884c09ab50 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f884c0a02b0 con 0x7f884c134580 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f884c0a2940 con 0x7f884c09d930 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8850c8c640 10 monclient(hunting): authenticate will time out at 1732.245520s 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:36018/0 (socket says 172.21.15.164:36018) 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:39730/0 (socket says 172.21.15.164:39730) 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 -- 172.21.15.164:0/1890817425 learned_addr learned my addr 172.21.15.164:0/1890817425 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): get_auth_request con 0x7f884c09ab50 auth_method 0 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): get_auth_request con 0x7f884c09d930 auth_method 0 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f884c134580 0x7f884c12fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): get_auth_request con 0x7f884c134580 auth_method 0 2024-09-10T16:22:33.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient(hunting): handle_auth_done global_id 164301 payload 824 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 -- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 msgr2=0x7f884c12f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 -- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f884c134580 msgr2=0x7f884c12fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f884c134580 0x7f884c12fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 monclient: found mon.noname-a 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:33.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f884a59c640 1 -- 172.21.15.164:0/1890817425 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c131580 con 0x7f884c09ab50 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.758+0000 7f8843fff640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f884a59c640 1 -- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 msgr2=0x7f884c073e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f884a59c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f883c00e9b0 tx=0x7f883c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c017070 con 0x7f884c09ab50 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: dump: 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.776 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: _finish_auth 0 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 5 monclient: authenticate success, global_id 164301 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f883c00cbb0 con 0x7f884c09ab50 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 10 monclient: _renew_subs 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f884c1312e0 con 0x7f884c09ab50 2024-09-10T16:22:33.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f883c013470 con 0x7f884c09ab50 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8842ffd640 10 monclient: dump: 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:33.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f884959a640 4 set_mon_vals no callback set 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 10 monclient: _renew_subs 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.761+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f884c1329d0 con 0x7f884c09ab50 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8850c8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8850c8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8850c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f884c0a1e40 con 0x7f884c09ab50 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f883c01a070 con 0x7f884c09ab50 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8842ffd640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 0x7f882c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(555..555 src has 481..555) ==== 4799+0+0 (secure 0 0 0) 0x7f883c052d40 con 0x7f884c09ab50 2024-09-10T16:22:33.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8843fff640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 0x7f882c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:33.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.762+0000 7f8843fff640 10 monclient: get_auth_request con 0x7f882c03e210 auth_method 0 2024-09-10T16:22:33.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.765+0000 7f8843fff640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 0x7f882c0406d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f884c130c30 tx=0x7f8834009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:33.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.767+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f883c023d00 con 0x7f884c09ab50 2024-09-10T16:22:33.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.767+0000 7f8842ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:33.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:33.767+0000 7f8842ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:33.821 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.009+0000 7f8850c8c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:22:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.009+0000 7f8850c8c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:22:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.009+0000 7f8850c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:34.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.009+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f884c06dac0 con 0x7f884c09ab50 2024-09-10T16:22:34.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f8842ffd640 1 -- 172.21.15.164:0/1890817425 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f883c01b3b0 con 0x7f884c09ab50 2024-09-10T16:22:34.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f8842ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:22:34.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f8842ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:34.011 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1331992362,config=2586702516,health=2510573225,kv=842815299,logm=2535757661}) 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=1,health=11,kv=11,logm=49} crc {auth=3471566198,config=1228700967,health=3686469583,kv=805391194,logm=1306703594}) 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=28,config=2,health=12,kv=12,logm=46} crc {auth=1331992362,config=2586702516,health=2510573225,kv=842815299,logm=2535757661}) 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.010+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=28,config=2,health=11,kv=10,logm=49} crc {auth=2765079930,config=2586702516,health=1382759333,kv=4208923166,logm=2136711891}) 2024-09-10T16:22:34.012 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.013 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733568532}) 2024-09-10T16:22:34.013 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503162712}) 2024-09-10T16:22:34.013 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.013 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3733568532}) 2024-09-10T16:22:34.013 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.011+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492364420}) 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156192519}) 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2869721651}) 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156192519}) 2024-09-10T16:22:34.014 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.012+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134438595}) 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 msgr2=0x7f882c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 0x7f882c0406d0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f884c130c30 tx=0x7f8834009190 comp rx=0 tx=0).stop 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 10 monclient: shutdown 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 msgr2=0x7f884c073e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f883c00e9b0 tx=0x7f883c00ee70 comp rx=0 tx=0).stop 2024-09-10T16:22:34.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 shutdown_connections 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f882c03e210 0x7f882c0406d0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f884c134580 0x7f884c12fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f884c09d930 0x7f884c12f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 --2- 172.21.15.164:0/1890817425 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f884c09ab50 0x7f884c073e10 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 >> 172.21.15.164:0/1890817425 conn(0x7f884c0a3110 msgr2=0x7f884c099560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 shutdown_connections 2024-09-10T16:22:34.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:34.013+0000 7f8850c8c640 1 -- 172.21.15.164:0/1890817425 wait complete. 2024-09-10T16:22:34.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780593900}) 2024-09-10T16:22:34.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4029583388}) 2024-09-10T16:22:34.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780593900}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.014+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3492590178}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849358708}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182171486}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849358708}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3336161811}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995878055}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3721147366}) 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2995878055}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.015+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=326185984}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240343722}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258221671}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240343722}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.016+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=513416333}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291400237}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1552521925}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=291400237}) 2024-09-10T16:22:34.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2162449558}) 2024-09-10T16:22:34.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629747093}) 2024-09-10T16:22:34.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=406407704}) 2024-09-10T16:22:34.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2629747093}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.017+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=564256660}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192100020}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=958063130}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1192100020}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.018+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=713806053}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185242063}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4006495588}) 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3185242063}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1648766289}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747539767}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2735025338}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747539767}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.019+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2329299471}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080906147}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1079761136}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4080906147}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.020+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=691319229}) 2024-09-10T16:22:34.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204567290}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=762580801}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=204567290}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900651271}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534798311}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=52852004}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=534798311}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.021+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636928749}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267050052}) 2024-09-10T16:22:34.027 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1909683828}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267050052}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.022+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253010243}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090630952}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=947021043}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1090630952}) 2024-09-10T16:22:34.028 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104139898}) 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349654928}) 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355011679}) 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349654928}) 2024-09-10T16:22:34.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.023+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4171178179}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091547409}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=232207057}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091547409}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.024+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1771594560}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208323889}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=354565886}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2208323889}) 2024-09-10T16:22:34.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.025+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=616027416}) 2024-09-10T16:22:34.033 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675168690}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1783379588}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1675168690}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.026+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4012012523}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10825072}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4162092507}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=10825072}) 2024-09-10T16:22:34.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2127562719}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741527219}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=621814070}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=741527219}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.027+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1611492904}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189609505}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517493633}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=189609505}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.028+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3019344535}) 2024-09-10T16:22:34.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107092874}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4271389460}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107092874}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1020700923}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961553876}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3794087209}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=961553876}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.029+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1562524023}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342700941}) 2024-09-10T16:22:34.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1891942198}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342700941}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.030+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551492267}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705733488}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988836812}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2705733488}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.031+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442932841}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025359557}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4184178740}) 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025359557}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2000912533}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143839680}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2781162516}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143839680}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.032+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=457518233}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650173716}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582600075}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650173716}) 2024-09-10T16:22:34.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3708400296}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.033+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969314814}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1245335889}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2969314814}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3558212557}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075693273}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2607008894}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3075693273}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.034+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3957636874}) 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.045 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956347261}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3963461316}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2956347261}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=988702133}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914735823}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3203568164}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=914735823}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.035+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1865960204}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838696732}) 2024-09-10T16:22:34.046 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128748383}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838696732}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.036+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3011785965}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751208448}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565696228}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1751208448}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1309106621}) 2024-09-10T16:22:34.047 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840611781}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4023498373}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1840611781}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.037+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3405956361}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781215946}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3141573928}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781215946}) 2024-09-10T16:22:34.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.038+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1861484182}) 2024-09-10T16:22:34.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223005220}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=503583672}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223005220}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.039+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3562170281}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821263259}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1493416653}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821263259}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3525186227}) 2024-09-10T16:22:34.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868747416}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1172549062}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=868747416}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.040+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806829340}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1103674438}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085255185}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1103674438}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.041+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=973795228}) 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990941267}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1916687891}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2990941267}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2267178848}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958683520}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3954382519}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958683520}) 2024-09-10T16:22:34.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.042+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=935138360}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820619525}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884850981}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2820619525}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.043+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146374605}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294272619}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4128812653}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294272619}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.044+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=422619523}) 2024-09-10T16:22:34.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564093378}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1775583351}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2564093378}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1388757221}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747483983}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2625236326}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=747483983}) 2024-09-10T16:22:34.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.045+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1662456488}) 2024-09-10T16:22:34.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135931642}) 2024-09-10T16:22:34.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570096132}) 2024-09-10T16:22:34.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135931642}) 2024-09-10T16:22:34.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.058+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021465476}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270457277}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3975581317}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1270457277}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.059+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280516324}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2748380724,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=240724071}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9} crc {logm=3036574068,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=957204152}) 2024-09-10T16:22:34.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2748380724,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=240724071}) 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=1} crc {logm=1413530426,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=542852026,mon_config_key=1867747704}) 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=15,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mon_config_key=754067386,monmap=765578399,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2204866889,osd_snap=411287996}) 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=72} crc {mon_config_key=1004069617,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2204866889,osd_snap=2634911310}) 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=15,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=73} crc {mon_config_key=754067386,monmap=765578399,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2204866889,osd_snap=411287996}) 2024-09-10T16:22:34.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.060+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=72} crc {mon_config_key=1963201189,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=2204866889,osd_snap=2814490976}) 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=4095400094,osdmap=3970924982}) 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=3877438466,osdmap=3797149480}) 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=4095400094,osdmap=3970924982}) 2024-09-10T16:22:34.064 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.061+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=2596344019,osdmap=4020096966}) 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=4248596058}) 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=3270536945}) 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=60} crc {osdmap=4248596058}) 2024-09-10T16:22:34.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:22:34.065+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=60} crc {osdmap=3009642325}) 2024-09-10T16:22:34.681 INFO:tasks.rados.rados.0.smithi165.stdout:2449: done (2 left) 2024-09-10T16:22:34.681 INFO:tasks.rados.rados.0.smithi165.stdout:2451: done (1 left) 2024-09-10T16:22:34.681 INFO:tasks.rados.rados.0.smithi165.stdout:2452: done (0 left) 2024-09-10T16:22:34.681 INFO:tasks.rados.rados.0.smithi165.stdout:2453: write (excl) oid 31 current snap is 266 2024-09-10T16:22:34.681 INFO:tasks.rados.rados.0.smithi165.stdout:2453: seq_num 869 ranges {605283=655623,1701911=747296} 2024-09-10T16:22:34.687 INFO:tasks.rados.rados.0.smithi165.stdout:2453: writing smithi16534841-31 from 605283 to 1260906 tid 1 2024-09-10T16:22:34.696 INFO:tasks.rados.rados.0.smithi165.stdout:2453: writing smithi16534841-31 from 1701911 to 2449207 tid 2 2024-09-10T16:22:34.696 INFO:tasks.rados.rados.0.smithi165.stdout:2454: read oid 22 snap -1 2024-09-10T16:22:34.696 INFO:tasks.rados.rados.0.smithi165.stdout:2454: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:22:34.696 INFO:tasks.rados.rados.0.smithi165.stdout:2455: write (excl) oid 30 current snap is 266 2024-09-10T16:22:34.696 INFO:tasks.rados.rados.0.smithi165.stdout:2453: finishing write tid 1 to smithi16534841-31 2024-09-10T16:22:34.697 INFO:tasks.rados.rados.0.smithi165.stdout:2455: seq_num 870 ranges {622090=730872,1760393=607307} 2024-09-10T16:22:34.702 INFO:tasks.rados.rados.0.smithi165.stdout:2455: writing smithi16534841-30 from 622090 to 1352962 tid 1 2024-09-10T16:22:34.708 INFO:tasks.rados.rados.0.smithi165.stdout:2455: writing smithi16534841-30 from 1760393 to 2367700 tid 2 2024-09-10T16:22:34.708 INFO:tasks.rados.rados.0.smithi165.stdout:2456: write oid 50 current snap is 266 2024-09-10T16:22:34.708 INFO:tasks.rados.rados.0.smithi165.stdout:2456: seq_num 871 ranges {638897=406121,1818875=750965} 2024-09-10T16:22:34.713 INFO:tasks.rados.rados.0.smithi165.stdout:2456: writing smithi16534841-50 from 638897 to 1045018 tid 1 2024-09-10T16:22:34.722 INFO:tasks.rados.rados.0.smithi165.stdout:2456: writing smithi16534841-50 from 1818875 to 2569840 tid 2 2024-09-10T16:22:34.722 INFO:tasks.rados.rados.0.smithi165.stdout:2457: write oid 34 current snap is 266 2024-09-10T16:22:34.722 INFO:tasks.rados.rados.0.smithi165.stdout:2457: seq_num 872 ranges {655704=481370,1761004=494623} 2024-09-10T16:22:34.728 INFO:tasks.rados.rados.0.smithi165.stdout:2457: writing smithi16534841-34 from 655704 to 1137074 tid 1 2024-09-10T16:22:34.734 INFO:tasks.rados.rados.0.smithi165.stdout:2457: writing smithi16534841-34 from 1761004 to 2255627 tid 2 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2458: rollback oid 3 current snap is 266 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 264 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: finishing write tid 2 to smithi16534841-31 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: finishing write tid 3 to smithi16534841-31 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: finishing write tid 4 to smithi16534841-31 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: oid 31 updating version 0 to 1516 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: oid 31 updating version 1516 to 1517 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: oid 31 updating version 1517 to 1518 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1518 (ObjNum 869 snap 266 seq_num 869) dirty exists 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2453: left oid 31 (ObjNum 869 snap 266 seq_num 869) 2024-09-10T16:22:34.735 INFO:tasks.rados.rados.0.smithi165.stdout:2454: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:22:34.773 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:34.851 INFO:tasks.rados.rados.0.smithi165.stdout:2455: finishing write tid 1 to smithi16534841-30 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: finishing write tid 2 to smithi16534841-30 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: finishing write tid 3 to smithi16534841-30 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: finishing write tid 4 to smithi16534841-30 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: oid 30 updating version 0 to 1520 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: oid 30 updating version 1520 to 1522 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: oid 30 version 1522 is already newer than 1521 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1522 (ObjNum 870 snap 266 seq_num 870) dirty exists 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2455: left oid 30 (ObjNum 870 snap 266 seq_num 870) 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2456: finishing write tid 1 to smithi16534841-50 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2456: finishing write tid 2 to smithi16534841-50 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2456: finishing write tid 3 to smithi16534841-50 2024-09-10T16:22:34.852 INFO:tasks.rados.rados.0.smithi165.stdout:2456: finishing write tid 4 to smithi16534841-50 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2456: oid 50 updating version 0 to 1260 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2456: oid 50 version 1260 is already newer than 1259 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2456: oid 50 version 1260 is already newer than 1258 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1260 (ObjNum 871 snap 266 seq_num 871) dirty exists 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2456: left oid 50 (ObjNum 871 snap 266 seq_num 871) 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2457: finishing write tid 1 to smithi16534841-34 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2453: done (5 left) 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2454: done (4 left) 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2455: done (3 left) 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2456: done (2 left) 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2459: copy_from oid 43 from oid 49 current snap is 266 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2457: finishing write tid 2 to smithi16534841-34 2024-09-10T16:22:34.853 INFO:tasks.rados.rados.0.smithi165.stdout:2457: finishing write tid 3 to smithi16534841-34 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: finishing write tid 4 to smithi16534841-34 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: oid 34 updating version 0 to 1339 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: oid 34 updating version 1339 to 1340 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: oid 34 updating version 1340 to 1341 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1341 (ObjNum 872 snap 266 seq_num 872) dirty exists 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: left oid 34 (ObjNum 872 snap 266 seq_num 872) 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2458: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1365 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2457: done (2 left) 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2458: done (1 left) 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2460: write oid 11 current snap is 266 2024-09-10T16:22:34.854 INFO:tasks.rados.rados.0.smithi165.stdout:2460: seq_num 873 ranges {672511=556619,1819486=754634} 2024-09-10T16:22:34.857 INFO:tasks.rados.rados.0.smithi165.stdout:2460: writing smithi16534841-11 from 672511 to 1229130 tid 1 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2460: writing smithi16534841-11 from 1819486 to 2574120 tid 2 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2461: read oid 26 snap 265 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2461: expect (ObjNum 831 snap 257 seq_num 831) 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2462: snap_create 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2459: finishing copy_from to smithi16534841-43 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2459: got expected ENOENT (src dne) 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2459: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:22:34.865 INFO:tasks.rados.rados.0.smithi165.stdout:2460: finishing write tid 1 to smithi16534841-11 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: finishing write tid 2 to smithi16534841-11 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: finishing write tid 3 to smithi16534841-11 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: finishing write tid 4 to smithi16534841-11 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: oid 11 updating version 0 to 1262 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: oid 11 version 1262 is already newer than 1261 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:2460: oid 11 updating version 1262 to 1263 2024-09-10T16:22:34.868 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1263 (ObjNum 873 snap 266 seq_num 873) dirty exists 2024-09-10T16:22:34.869 INFO:tasks.rados.rados.0.smithi165.stdout:2460: left oid 11 (ObjNum 873 snap 266 seq_num 873) 2024-09-10T16:22:34.876 INFO:tasks.rados.rados.0.smithi165.stdout:2461: expect (ObjNum 831 snap 257 seq_num 831) 2024-09-10T16:22:34.876 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 749589 2024-09-10T16:22:34.915 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:34.979 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3662783 2024-09-10T16:22:35.033 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:22:35.034 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.128+0000 7efc2fb82640 10 monclient: get_monmap_and_config 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.128+0000 7efc2fb82640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.128+0000 7efc2fb82640 10 monclient: monmap: 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:35.129673+0000 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:35.129673+0000 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.130 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:35.131 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 1 Processor -- start 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 1 -- start start 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 10 monclient: init 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efc2809d930 0x7efc2809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.129+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-c con 0x7efc2809d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-b con 0x7efc2809ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-a con 0x7efc28134580 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7efc2809d3e0 con 0x7efc28134580 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efc2809e790 con 0x7efc2809d930 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efc280a0870 con 0x7efc2809ab50 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2fb82640 10 monclient(hunting): authenticate will time out at 1733.617043s 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45628/0 (socket says 172.21.15.164:45628) 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54922/0 (socket says 172.21.15.164:54922) 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 1 -- 172.21.15.164:0/1510214890 learned_addr learned my addr 172.21.15.164:0/1510214890 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): get_auth_request con 0x7efc2809ab50 auth_method 0 2024-09-10T16:22:35.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): get_auth_request con 0x7efc28134580 auth_method 0 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.165:3300/0 conn(0x7efc2809d930 0x7efc2809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): get_auth_request con 0x7efc2809d930 auth_method 0 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient(hunting): handle_auth_done global_id 184127 payload 274 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 -- 172.21.15.164:0/1510214890 >> v2:172.21.15.165:3300/0 conn(0x7efc2809d930 msgr2=0x7efc2809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.165:3300/0 conn(0x7efc2809d930 0x7efc2809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 -- 172.21.15.164:0/1510214890 >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 msgr2=0x7efc28134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 monclient: found mon.noname-b 2024-09-10T16:22:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:22:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2d11d640 1 -- 172.21.15.164:0/1510214890 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc28074c70 con 0x7efc2809ab50 2024-09-10T16:22:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.130+0000 7efc2e11f640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2d11d640 1 -- 172.21.15.164:0/1510214890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 msgr2=0x7efc2809e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2d11d640 1 --2- 172.21.15.164:0/1510214890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc2806d500 tx=0x7efc1000a7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=481f05bc38e82ceb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 1 -- 172.21.15.164:0/1510214890 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10036070 con 0x7efc2809ab50 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: dump: 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 5 monclient: authenticate success, global_id 184127 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 1 -- 172.21.15.164:0/1510214890 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc10008850 con 0x7efc2809ab50 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 4 set_mon_vals no callback set 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 1 -- 172.21.15.164:0/1510214890 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10008c00 con 0x7efc2809ab50 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2c91c640 10 monclient: dump: 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 10 monclient: shutdown 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 -- 172.21.15.164:0/1510214890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 msgr2=0x7efc2809e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 --2- 172.21.15.164:0/1510214890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7efc2806d500 tx=0x7efc1000a7b0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 -- 172.21.15.164:0/1510214890 shutdown_connections 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.164:3300/0 conn(0x7efc28134580 0x7efc28134a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 --2- 172.21.15.164:0/1510214890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc2809e250 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 --2- 172.21.15.164:0/1510214890 >> v2:172.21.15.165:3300/0 conn(0x7efc2809d930 0x7efc2809dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.133+0000 7efc2fb82640 1 -- 172.21.15.164:0/1510214890 >> 172.21.15.164:0/1510214890 conn(0x7efc280a3110 msgr2=0x7efc280a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 1 -- 172.21.15.164:0/1510214890 shutdown_connections 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 1 -- 172.21.15.164:0/1510214890 wait complete. 2024-09-10T16:22:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 10 monclient: monmap: 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 1 Processor -- start 2024-09-10T16:22:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 1 -- start start 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.134+0000 7efc2fb82640 10 monclient: init 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc281c9570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-c con 0x7efc2809ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-a con 0x7efc2809d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc28134580 0x7efc28073100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): picked mon.noname-b con 0x7efc28134580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efc2809ef30 con 0x7efc2809d930 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efc280a02b0 con 0x7efc28134580 2024-09-10T16:22:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efc280a2940 con 0x7efc2809ab50 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2fb82640 10 monclient(hunting): authenticate will time out at 1733.622462s 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc281c9570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45634/0 (socket says 172.21.15.164:45634) 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 -- 172.21.15.164:0/2165854801 learned_addr learned my addr 172.21.15.164:0/2165854801 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): get_auth_request con 0x7efc2809d930 auth_method 0 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): get_auth_request con 0x7efc2809ab50 auth_method 0 2024-09-10T16:22:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc28134580 0x7efc28073100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): get_auth_request con 0x7efc28134580 auth_method 0 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2e11f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient(hunting): handle_auth_done global_id 164307 payload 824 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 -- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 msgr2=0x7efc281c9570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc281c9570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 -- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc28134580 msgr2=0x7efc28073100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc28134580 0x7efc28073100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 monclient: found mon.noname-a 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d11d640 1 -- 172.21.15.164:0/2165854801 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc28135340 con 0x7efc2809d930 2024-09-10T16:22:35.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.135+0000 7efc2d91e640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc281c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2d11d640 1 -- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 msgr2=0x7efc28072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2d11d640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc10034040 tx=0x7efc10004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc10036050 con 0x7efc2809d930 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: dump: 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.153 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 5 monclient: authenticate success, global_id 164307 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc10042070 con 0x7efc2809d930 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc1000a2e0 con 0x7efc2809d930 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 10 monclient: _renew_subs 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc2809d620 con 0x7efc2809d930 2024-09-10T16:22:35.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc27fff640 4 set_mon_vals no callback set 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc267fc640 10 monclient: dump: 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 10 monclient: _renew_subs 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.138+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc28134310 con 0x7efc2809d930 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efc10004490 con 0x7efc2809d930 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2fb82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2fb82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2fb82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc267fc640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 0x7efc08040780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc28134110 con 0x7efc2809d930 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(556..556 src has 481..556) ==== 4771+0+0 (secure 0 0 0) 0x7efc10073bd0 con 0x7efc2809d930 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2d91e640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 0x7efc08040780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.139+0000 7efc2d91e640 10 monclient: get_auth_request con 0x7efc0803e2c0 auth_method 0 2024-09-10T16:22:35.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.142+0000 7efc2d91e640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 0x7efc08040780 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efc18009f30 tx=0x7efc18009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.144+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efc28134110 con 0x7efc2809d930 2024-09-10T16:22:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.144+0000 7efc267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.144+0000 7efc267fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:35.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc2fb82640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:22:35.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc2fb82640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc2fb82640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efc280a1e40 con 0x7efc2809d930 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc267fc640 1 -- 172.21.15.164:0/2165854801 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7efc10042ce0 con 0x7efc2809d930 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.383+0000 7efc267fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:35.385 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":158,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 msgr2=0x7efc08040780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 0x7efc08040780 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7efc18009f30 tx=0x7efc18009190 comp rx=0 tx=0).stop 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 10 monclient: shutdown 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 msgr2=0x7efc28072bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7efc10034040 tx=0x7efc10004a40 comp rx=0 tx=0).stop 2024-09-10T16:22:35.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 shutdown_connections 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efc0803e2c0 0x7efc08040780 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efc28134580 0x7efc28073100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efc2809d930 0x7efc28072bc0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 --2- 172.21.15.164:0/2165854801 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efc2809ab50 0x7efc281c9570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 >> 172.21.15.164:0/2165854801 conn(0x7efc280a3110 msgr2=0x7efc281bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 shutdown_connections 2024-09-10T16:22:35.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.385+0000 7efc2fb82640 1 -- 172.21.15.164:0/2165854801 wait complete. 2024-09-10T16:22:35.397 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:22:35.397 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:22:35.397 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:22:35.398 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:22:35.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.495+0000 7f6df1517640 10 monclient: get_monmap_and_config 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.495+0000 7f6df1517640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.495+0000 7f6df1517640 10 monclient: monmap: 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:35.496421+0000 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:35.496421+0000 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:35.497 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:35.498 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:35.498 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:35.498 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 Processor -- start 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 -- start start 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient: init 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6dec06cb70 0x7f6dec134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-a con 0x7f6dec06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 0x7f6dec0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-c con 0x7f6dec0a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 0x7f6dec138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-b con 0x7f6dec137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6dec06db60 con 0x7f6dec06cb70 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6dec06cf50 con 0x7f6dec0a46a0 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec09ce50 con 0x7f6dec137ca0 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6df1517640 10 monclient(hunting): authenticate will time out at 1733.983782s 2024-09-10T16:22:35.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6dec06cb70 0x7f6dec134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 0x7f6dec138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6dec06cb70 0x7f6dec134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45642/0 (socket says 172.21.15.164:45642) 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 1 -- 172.21.15.164:0/4261209247 learned_addr learned my addr 172.21.15.164:0/4261209247 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 10 monclient(hunting): get_auth_request con 0x7f6dec06cb70 auth_method 0 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deb7fe640 10 monclient(hunting): get_auth_request con 0x7f6dec137ca0 auth_method 0 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.496+0000 7f6deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6de3fff640 1 --2- 172.21.15.164:0/4261209247 >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 0x7f6dec0a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient(hunting): handle_auth_done global_id 164310 payload 274 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 -- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 msgr2=0x7f6dec138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 0x7f6dec138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 -- 172.21.15.164:0/4261209247 >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 msgr2=0x7f6dec0a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 --2- 172.21.15.164:0/4261209247 >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 0x7f6dec0a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 monclient: found mon.noname-a 2024-09-10T16:22:35.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deaffd640 1 -- 172.21.15.164:0/4261209247 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec09cb70 con 0x7f6dec06cb70 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6de3fff640 10 monclient: get_auth_request con 0x7f6dec0a46a0 auth_method 0 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6deb7fe640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 0x7f6dec138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.497+0000 7f6de3fff640 1 --2- 172.21.15.164:0/4261209247 >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 0x7f6dec0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.499+0000 7f6deaffd640 1 -- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 msgr2=0x7f6dec134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.499+0000 7f6deaffd640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec134f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009870 tx=0x7f6ddc02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f375d7c9ad80489c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 1 -- 172.21.15.164:0/4261209247 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc037070 con 0x7f6dec06cb70 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: dump: 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.506 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 1 -- 172.21.15.164:0/4261209247 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ddc032440 con 0x7f6dec06cb70 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 5 monclient: authenticate success, global_id 164310 2024-09-10T16:22:35.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 1 -- 172.21.15.164:0/4261209247 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc032810 con 0x7f6dec06cb70 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 4 set_mon_vals no callback set 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6dea7fc640 10 monclient: dump: 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 10 monclient: shutdown 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 -- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 msgr2=0x7f6dec134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec134f70 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc009870 tx=0x7f6ddc02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 -- 172.21.15.164:0/4261209247 shutdown_connections 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec137ca0 0x7f6dec138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 --2- 172.21.15.164:0/4261209247 >> v2:172.21.15.165:3300/0 conn(0x7f6dec0a46a0 0x7f6dec0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 --2- 172.21.15.164:0/4261209247 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec134f70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 -- 172.21.15.164:0/4261209247 >> 172.21.15.164:0/4261209247 conn(0x7f6dec0aa900 msgr2=0x7f6dec0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 -- 172.21.15.164:0/4261209247 shutdown_connections 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.500+0000 7f6df1517640 1 -- 172.21.15.164:0/4261209247 wait complete. 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.501+0000 7f6df1517640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.501+0000 7f6df1517640 10 monclient: monmap: 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:35.510 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.501+0000 7f6df1517640 1 Processor -- start 2024-09-10T16:22:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.501+0000 7f6df1517640 1 -- start start 2024-09-10T16:22:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.501+0000 7f6df1517640 10 monclient: init 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec1c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-a con 0x7f6dec06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 0x7f6dec12f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-c con 0x7f6dec0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dec137ca0 0x7f6dec1cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): picked mon.noname-b con 0x7f6dec137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1387f0 con 0x7f6dec06cb70 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec138670 con 0x7f6dec137ca0 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec138970 con 0x7f6dec0a46a0 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6df1517640 10 monclient(hunting): authenticate will time out at 1733.989112s 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 0x7f6dec12f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec1c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 0x7f6dec12f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54946/0 (socket says 172.21.15.164:54946) 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 -- 172.21.15.164:0/655888979 learned_addr learned my addr 172.21.15.164:0/655888979 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): get_auth_request con 0x7f6dec06cb70 auth_method 0 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): get_auth_request con 0x7f6dec0a46a0 auth_method 0 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dec137ca0 0x7f6dec1cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): get_auth_request con 0x7f6dec137ca0 auth_method 0 2024-09-10T16:22:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient(hunting): handle_auth_done global_id 184136 payload 824 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dec137ca0 msgr2=0x7f6dec1cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6dec137ca0 0x7f6dec1cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 msgr2=0x7f6dec1c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec1c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 monclient: found mon.noname-c 2024-09-10T16:22:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6de3fff640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec06e720 con 0x7f6dec0a46a0 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.502+0000 7f6deaffd640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec06cb70 0x7f6dec1c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de3fff640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 msgr2=0x7f6dec12f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de3fff640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 0x7f6dec12f730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c820 tx=0x7f6dd400cc20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4015070 con 0x7f6dec0a46a0 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: dump: 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 5 monclient: authenticate success, global_id 184136 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dd4007a30 con 0x7f6dec0a46a0 2024-09-10T16:22:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 10 monclient: _renew_subs 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec0a34b0 con 0x7f6dec0a46a0 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dd4007de0 con 0x7f6dec0a46a0 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de37fe640 10 monclient: dump: 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6de9ffb640 4 set_mon_vals no callback set 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 10 monclient: _renew_subs 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.504+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec0a3070 con 0x7f6dec0a46a0 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6dd4018070 con 0x7f6dec0a46a0 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6df1517640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6df1517640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 0x7f6dec1cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec134430 con 0x7f6dec1cd7e0 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6de37fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6dcc03e210 0x7f6dcc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6deaffd640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 0x7f6dec1cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6deaffd640 10 monclient: get_auth_request con 0x7f6dec1cd7e0 auth_method 0 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.505+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(556..556 src has 481..556) ==== 4771+0+0 (secure 0 0 0) 0x7f6dd4052c80 con 0x7f6dec0a46a0 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deaffd640 10 monclient(hunting): handle_auth_done global_id 164316 payload 824 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deaffd640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6dec1ce100 con 0x7f6dec1cd7e0 2024-09-10T16:22:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deb7fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6dcc03e210 0x7f6dcc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.506+0000 7f6deb7fe640 10 monclient: get_auth_request con 0x7f6dcc03e210 auth_method 0 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6deaffd640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 msgr2=0x7f6dec1cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6deaffd640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 0x7f6dec1cdbc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc034000 tx=0x7f6ddc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc043070 con 0x7f6dec1cd7e0 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 10 monclient: dump: 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ddc03c850 con 0x7f6dec1cd7e0 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 msgr2=0x7f6dec1cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 0x7f6dec1cdbc0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc034000 tx=0x7f6ddc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 msgr2=0x7f6dec1cdbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6deb7fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6dcc03e210 0x7f6dcc0406d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6de4011420 tx=0x7f6de400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.508+0000 7f6de37fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec1cd7e0 0x7f6dec1cdbc0 secure :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc034000 tx=0x7f6ddc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6df1517640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6df1517640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 0x7f6dec09da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6df1517640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec0a6200 con 0x7f6dec09d630 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 0x7f6dec09da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient: get_auth_request con 0x7f6dec09d630 auth_method 0 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.514+0000 7f6deaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.515+0000 7f6deaffd640 10 monclient(hunting): handle_auth_done global_id 164319 payload 824 2024-09-10T16:22:35.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.515+0000 7f6deaffd640 1 -- 172.21.15.164:0/655888979 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6dec0a4ab0 con 0x7f6dec09d630 2024-09-10T16:22:35.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6deaffd640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 msgr2=0x7f6dec09da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6deaffd640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 0x7f6dec09da40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc02a110 tx=0x7f6ddc03c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc037040 con 0x7f6dec09d630 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 10 monclient: dump: 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.531 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f6ddc048c80 con 0x7f6dec09d630 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 msgr2=0x7f6dec09da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 0x7f6dec09da40 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc02a110 tx=0x7f6ddc03c000 comp rx=0 tx=0).stop 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 msgr2=0x7f6dec09da40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.517+0000 7f6de37fe640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6dec09d630 0x7f6dec09da40 secure :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc02a110 tx=0x7f6ddc03c000 comp rx=0 tx=0).stop 2024-09-10T16:22:35.532 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 126219, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:22:35.533 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.534 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.535 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:35.536 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.537 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.538 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.539 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.540 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.541 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:35.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.543 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:35.544 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:35.545 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6dcc03e210 msgr2=0x7f6dcc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6dcc03e210 0x7f6dcc0406d0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6de4011420 tx=0x7f6de400b040 comp rx=0 tx=0).stop 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 10 monclient: shutdown 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 msgr2=0x7f6dec12f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 --2- 172.21.15.164:0/655888979 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6dec0a46a0 0x7f6dec12f730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6dd400c820 tx=0x7f6dd400cc20 comp rx=0 tx=0).stop 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6deb7fe640 1 -- 172.21.15.164:0/655888979 reap_dead start 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6deb7fe640 1 -- 172.21.15.164:0/655888979 reap_dead start 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 shutdown_connections 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 >> 172.21.15.164:0/655888979 conn(0x7f6dec0aa900 msgr2=0x7f6dec131580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 shutdown_connections 2024-09-10T16:22:35.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.518+0000 7f6df1517640 1 -- 172.21.15.164:0/655888979 wait complete. 2024-09-10T16:22:35.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:35.548 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:22:35.548 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:22:35.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 10 monclient: get_monmap_and_config 2024-09-10T16:22:35.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 10 monclient: monmap: 2024-09-10T16:22:35.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.647 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:35.646955+0000 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:35.646955+0000 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:35.648 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 1 Processor -- start 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 1 -- start start 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.646+0000 7ffa45de8640 10 monclient: init 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ffa4006d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-b con 0x7ffa4006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffa400a4c20 0x7ffa4013f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-c con 0x7ffa400a4c20 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:35.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa40135510 con 0x7ffa4006d160 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ffa4009cb90 con 0x7ffa400a4c20 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4009ce70 con 0x7ffa4006d540 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa45de8640 10 monclient(hunting): authenticate will time out at 1734.134520s 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ffa4006d160 0x7ffa40134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38900/0 (socket says 172.21.15.164:38900) 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54954/0 (socket says 172.21.15.164:54954) 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/630435431 learned_addr learned my addr 172.21.15.164:0/630435431 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:35.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): get_auth_request con 0x7ffa4006d540 auth_method 0 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3ffff640 1 --2- 172.21.15.164:0/630435431 >> v2:172.21.15.165:3300/0 conn(0x7ffa400a4c20 0x7ffa4013f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request con 0x7ffa400a4c20 auth_method 0 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient(hunting): handle_auth_done global_id 184139 payload 274 2024-09-10T16:22:35.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 -- 172.21.15.164:0/630435431 >> v2:172.21.15.165:3300/0 conn(0x7ffa400a4c20 msgr2=0x7ffa4013f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 --2- 172.21.15.164:0/630435431 >> v2:172.21.15.165:3300/0 conn(0x7ffa400a4c20 0x7ffa4013f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 -- 172.21.15.164:0/630435431 >> v1:172.21.15.164:6789/0 conn(0x7ffa4006d160 legacy=0x7ffa40134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 monclient: found mon.noname-b 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:22:35.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.647+0000 7ffa3effd640 1 -- 172.21.15.164:0/630435431 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa4009d150 con 0x7ffa4006d540 2024-09-10T16:22:35.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3effd640 1 -- 172.21.15.164:0/630435431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 msgr2=0x7ffa400a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3effd640 1 --2- 172.21.15.164:0/630435431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa280097a0 tx=0x7ffa28029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dce583374c80c79e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 1 -- 172.21.15.164:0/630435431 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28037070 con 0x7ffa4006d540 2024-09-10T16:22:35.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: dump: 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.662 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa45de8640 5 monclient: authenticate success, global_id 184139 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 1 -- 172.21.15.164:0/630435431 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa28032440 con 0x7ffa4006d540 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa45de8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa45de8640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa45de8640 4 set_mon_vals no callback set 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 1 -- 172.21.15.164:0/630435431 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28032810 con 0x7ffa4006d540 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:35.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.650+0000 7ffa3e7fc640 10 monclient: dump: 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 10 monclient: shutdown 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 -- 172.21.15.164:0/630435431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 msgr2=0x7ffa400a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 --2- 172.21.15.164:0/630435431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ffa280097a0 tx=0x7ffa28029fd0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 -- 172.21.15.164:0/630435431 shutdown_connections 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 --2- 172.21.15.164:0/630435431 >> v2:172.21.15.165:3300/0 conn(0x7ffa400a4c20 0x7ffa4013f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 --2- 172.21.15.164:0/630435431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d540 0x7ffa400a46e0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 -- 172.21.15.164:0/630435431 >> 172.21.15.164:0/630435431 conn(0x7ffa400aacb0 msgr2=0x7ffa400ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 -- 172.21.15.164:0/630435431 shutdown_connections 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 1 -- 172.21.15.164:0/630435431 wait complete. 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.651+0000 7ffa45de8640 10 monclient: monmap: 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:35.665 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 Processor -- start 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 -- start start 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient: init 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 0x7ffa401cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-c con 0x7ffa4006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 0x7ffa400a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-a con 0x7ffa4006d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa400a4c20 0x7ffa400a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): picked mon.noname-b con 0x7ffa400a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4013fb20 con 0x7ffa4006d540 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffa4013f9a0 con 0x7ffa400a4c20 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.652+0000 7ffa45de8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa4013fca0 con 0x7ffa4006d160 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa45de8640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa45de8640 10 monclient(hunting): authenticate will time out at 1734.140012s 2024-09-10T16:22:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 0x7ffa401cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 0x7ffa401cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54962/0 (socket says 172.21.15.164:54962) 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 0x7ffa400a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/827737123 learned_addr learned my addr 172.21.15.164:0/827737123 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request con 0x7ffa4006d160 auth_method 0 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): get_auth_request con 0x7ffa4006d540 auth_method 0 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3ffff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa400a4c20 0x7ffa400a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request con 0x7ffa400a4c20 auth_method 0 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient(hunting): handle_auth_done global_id 164322 payload 824 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 msgr2=0x7ffa401cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 0x7ffa401cd280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa400a4c20 msgr2=0x7ffa400a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffa400a4c20 0x7ffa400a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 monclient: found mon.noname-a 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3effd640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa400a4510 con 0x7ffa4006d540 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.653+0000 7ffa3f7fe640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4006d160 0x7ffa401cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa3effd640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 msgr2=0x7ffa400a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa3effd640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 0x7ffa400a3a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a83d0 tx=0x7ffa280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28041020 con 0x7ffa4006d540 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: dump: 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2803d8f0 con 0x7ffa4006d540 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 5 monclient: authenticate success, global_id 164322 2024-09-10T16:22:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 10 monclient: _renew_subs 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa28037040 con 0x7ffa4006d540 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa400a2560 con 0x7ffa4006d540 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa37fff640 10 monclient: dump: 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa3dffb640 4 set_mon_vals no callback set 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 10 monclient: _renew_subs 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.655+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa400a0d60 con 0x7ffa4006d540 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa45de8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa45de8640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 0x7ffa4009ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa400a46e0 con 0x7ffa4009e950 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 0x7ffa4009ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient: get_auth_request con 0x7ffa4009e950 auth_method 0 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3effd640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 0x7ffa400a3a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a83d0 tx=0x7ffa280046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7ffa28073340 osd_map(556..556 src has 481..556) 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ffa28044020 con 0x7ffa4006d540 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.656+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.657+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_done global_id 184145 payload 824 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.657+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa400a13d0 con 0x7ffa4009e950 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.657+0000 7ffa37fff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffa2003e210 0x7ffa200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.657+0000 7ffa3ffff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffa2003e210 0x7ffa200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.657+0000 7ffa3ffff640 10 monclient: get_auth_request con 0x7ffa2003e210 auth_method 0 2024-09-10T16:22:35.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.659+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 msgr2=0x7ffa4009ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.659+0000 7ffa3f7fe640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 0x7ffa4009ed30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000d3d0 tx=0x7ffa3000d890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.659+0000 7ffa3ffff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffa2003e210 0x7ffa200406d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa300075b0 con 0x7ffa4009e950 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 10 monclient: dump: 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa30018070 con 0x7ffa4009e950 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 msgr2=0x7ffa4009ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 0x7ffa4009ed30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000d3d0 tx=0x7ffa3000d890 comp rx=0 tx=0).stop 2024-09-10T16:22:35.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 msgr2=0x7ffa4009ed30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.661+0000 7ffa37fff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009e950 0x7ffa4009ed30 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7ffa3000d3d0 tx=0x7ffa3000d890 comp rx=0 tx=0).stop 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.667+0000 7ffa45de8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.667+0000 7ffa45de8640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 0x7ffa400a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.667+0000 7ffa45de8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.667+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffa400a0200 con 0x7ffa4009f270 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 0x7ffa400a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient: get_auth_request con 0x7ffa4009f270 auth_method 0 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 10 monclient(hunting): handle_auth_done global_id 184148 payload 824 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.668+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/827737123 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa400aa1e0 con 0x7ffa4009f270 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa3f7fe640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 msgr2=0x7ffa400a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa3f7fe640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 0x7ffa400a5300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa30015040 tx=0x7ffa30004cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3001dcc0 con 0x7ffa4009f270 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 10 monclient: dump: 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.680 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7ffa3001dcc0 con 0x7ffa4009f270 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 msgr2=0x7ffa400a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 0x7ffa400a5300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa30015040 tx=0x7ffa30004cc0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 msgr2=0x7ffa400a5300 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.670+0000 7ffa37fff640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffa4009f270 0x7ffa400a5300 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa30015040 tx=0x7ffa30004cc0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 17949, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:22:35.682 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.683 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.684 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:35.685 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.686 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.687 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.688 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.689 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.690 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.691 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:35.692 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.693 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:35.694 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:35.694 INFO:tasks.rados.rados.0.smithi165.stdout:2459: done (3 left) 2024-09-10T16:22:35.695 INFO:tasks.rados.rados.0.smithi165.stdout:2460: done (2 left) 2024-09-10T16:22:35.695 INFO:tasks.rados.rados.0.smithi165.stdout:2461: done (1 left) 2024-09-10T16:22:35.695 INFO:tasks.rados.rados.0.smithi165.stdout:2462: done (0 left) 2024-09-10T16:22:35.695 INFO:tasks.rados.rados.0.smithi165.stdout:2463: snap_create 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffa2003e210 msgr2=0x7ffa200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffa2003e210 0x7ffa200406d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ffa38006fd0 tx=0x7ffa38008040 comp rx=0 tx=0).stop 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 10 monclient: shutdown 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 msgr2=0x7ffa400a3a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 --2- 172.21.15.164:0/827737123 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffa4006d540 0x7ffa400a3a90 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffa400a83d0 tx=0x7ffa280046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa3ffff640 1 -- 172.21.15.164:0/827737123 reap_dead start 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa37fff640 1 -- 172.21.15.164:0/827737123 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(556..556 src has 481..556) ==== 4771+0+0 (unknown 0 0 0) 0x7ffa28073340 con 0x7ffa4006d540 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa3ffff640 1 -- 172.21.15.164:0/827737123 reap_dead start 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 shutdown_connections 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 >> 172.21.15.164:0/827737123 conn(0x7ffa400aacb0 msgr2=0x7ffa4012e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 shutdown_connections 2024-09-10T16:22:35.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.671+0000 7ffa45de8640 1 -- 172.21.15.164:0/827737123 wait complete. 2024-09-10T16:22:35.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:35.697 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:22:35.697 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:22:35.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.794+0000 7fbae3ab8640 10 monclient: get_monmap_and_config 2024-09-10T16:22:35.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.794+0000 7fbae3ab8640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.794+0000 7fbae3ab8640 10 monclient: monmap: 2024-09-10T16:22:35.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:35.795789+0000 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:35.795789+0000 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:35.797 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.795+0000 7fbae3ab8640 1 Processor -- start 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.795+0000 7fbae3ab8640 1 -- start start 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.795+0000 7fbae3ab8640 10 monclient: init 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbadc09c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbadc09d190 0x7fbadc09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc09d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbadc13a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc09fbc0 con 0x7fbadc09c310 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbadc0a2a20 con 0x7fbadc13a5c0 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbadc09c6f0 con 0x7fbadc09d190 2024-09-10T16:22:35.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae3ab8640 10 monclient(hunting): authenticate will time out at 1734.283284s 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1854640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbadc09c310 0x7fbadc09ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38916/0 (socket says 172.21.15.164:38916) 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1854640 1 -- 172.21.15.164:0/687986339 learned_addr learned my addr 172.21.15.164:0/687986339 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 1 --2- 172.21.15.164:0/687986339 >> v2:172.21.15.165:3300/0 conn(0x7fbadc09d190 0x7fbadc09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): get_auth_request con 0x7fbadc09d190 auth_method 0 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient(hunting): handle_auth_done global_id 154378 payload 274 2024-09-10T16:22:35.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 1 -- 172.21.15.164:0/687986339 >> v1:172.21.15.165:6789/0 conn(0x7fbadc13a5c0 legacy=0x7fbadc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 1 -- 172.21.15.164:0/687986339 >> v1:172.21.15.164:6789/0 conn(0x7fbadc09c310 legacy=0x7fbadc09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 1 monclient: found mon.noname-c 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:22:35.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.796+0000 7fbae1053640 1 -- 172.21.15.164:0/687986339 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc09a4d0 con 0x7fbadc09d190 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae1053640 1 -- 172.21.15.164:0/687986339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 msgr2=0x7fbadc09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae1053640 1 --2- 172.21.15.164:0/687986339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc09f150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0097a0 tx=0x7fbacc029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44c33192c2cb98fa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae0852640 1 -- 172.21.15.164:0/687986339 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc037070 con 0x7fbadc09d190 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae0852640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae0852640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.799+0000 7fbae0852640 10 monclient: dump: 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.809 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 1 -- 172.21.15.164:0/687986339 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbacc032640 con 0x7fbadc09d190 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 1 -- 172.21.15.164:0/687986339 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc032a10 con 0x7fbadc09d190 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae0852640 10 monclient: dump: 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 5 monclient: authenticate success, global_id 154378 2024-09-10T16:22:35.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 4 set_mon_vals no callback set 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 10 monclient: shutdown 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 -- 172.21.15.164:0/687986339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 msgr2=0x7fbadc09f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/687986339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc09f150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0097a0 tx=0x7fbacc029fd0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 -- 172.21.15.164:0/687986339 shutdown_connections 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/687986339 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc09f150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 -- 172.21.15.164:0/687986339 >> 172.21.15.164:0/687986339 conn(0x7fbadc0a3110 msgr2=0x7fbadc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 -- 172.21.15.164:0/687986339 shutdown_connections 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.800+0000 7fbae3ab8640 1 -- 172.21.15.164:0/687986339 wait complete. 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient: build_initial_monmap 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient: monmap: 2024-09-10T16:22:35.812 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 Processor -- start 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 -- start start 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient: init 2024-09-10T16:22:35.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09c310 0x7fbadc092870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-b con 0x7fbadc09c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 0x7fbadc092db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-a con 0x7fbadc13a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbadc0932f0 0x7fbadc1d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient(hunting): picked mon.noname-c con 0x7fbadc0932f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.801+0000 7fbae3ab8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0a2230 con 0x7fbadc13a5c0 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc0a20b0 con 0x7fbadc09c310 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbadc074e00 con 0x7fbadc0932f0 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae3ab8640 10 monclient(hunting): authenticate will time out at 1734.289015s 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 0x7fbadc092db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 0x7fbadc092db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:45694/0 (socket says 172.21.15.164:45694) 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 -- 172.21.15.164:0/3891765261 learned_addr learned my addr 172.21.15.164:0/3891765261 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): get_auth_request con 0x7fbadc13a5c0 auth_method 0 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbadc0932f0 0x7fbadc1d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): get_auth_request con 0x7fbadc0932f0 auth_method 0 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09c310 0x7fbadc092870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): get_auth_request con 0x7fbadc09c310 auth_method 0 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient(hunting): handle_auth_done global_id 164325 payload 824 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbadc0932f0 msgr2=0x7fbadc1d17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbadc0932f0 0x7fbadc1d17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09c310 msgr2=0x7fbadc092870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09c310 0x7fbadc092870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient: _finish_hunting 0 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 monclient: found mon.noname-a 2024-09-10T16:22:35.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae1053640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbadc0990b0 con 0x7fbadc13a5c0 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.802+0000 7fbae2055640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbadc0932f0 0x7fbadc1d17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae1053640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 msgr2=0x7fbadc092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae1053640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 0x7fbadc092db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0098c0 tx=0x7fbacc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc042020 con 0x7fbadc13a5c0 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: dump: 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.818 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbacc030880 con 0x7fbadc13a5c0 2024-09-10T16:22:35.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 5 monclient: authenticate success, global_id 164325 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 10 monclient: _renew_subs 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbadc0a4c90 con 0x7fbadc13a5c0 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbacc037040 con 0x7fbadc13a5c0 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad27fc640 10 monclient: dump: 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.820 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbad3fff640 4 set_mon_vals no callback set 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 10 monclient: _renew_subs 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.804+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbadc095170 con 0x7fbadc13a5c0 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae3ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc02e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc093700 con 0x7fbadc09d190 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbacc045020 con 0x7fbadc13a5c0 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae1854640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc02e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae1854640 10 monclient: get_auth_request con 0x7fbadc09d190 auth_method 0 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae1854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae1854640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.805+0000 7fbae1854640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbad27fc640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbac003e240 0x7fbac0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(557..557 src has 481..557) ==== 4771+0+0 (secure 0 0 0) 0x7fbacc073700 con 0x7fbadc13a5c0 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae1854640 10 monclient(hunting): handle_auth_done global_id 154384 payload 824 2024-09-10T16:22:35.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae2055640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbac003e240 0x7fbac0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae1854640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbadc09e2b0 con 0x7fbadc09d190 2024-09-10T16:22:35.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.806+0000 7fbae2055640 10 monclient: get_auth_request con 0x7fbac003e240 auth_method 0 2024-09-10T16:22:35.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbae1854640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 msgr2=0x7fbadc02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbae1854640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc02e5c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbad800ea70 tx=0x7fbad800ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad801a070 con 0x7fbadc09d190 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbad27fc640 10 monclient: dump: 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.808+0000 7fbae2055640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbac003e240 0x7fbac0040700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbad400f1d0 tx=0x7fbad4003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbad800cc50 con 0x7fbadc09d190 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 msgr2=0x7fbadc02e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc02e5c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbad800ea70 tx=0x7fbad800ef30 comp rx=0 tx=0).stop 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 msgr2=0x7fbadc02e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.809+0000 7fbad27fc640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc09d190 0x7fbadc02e5c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae3ab8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 0x7fbadc063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae3ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:35.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbadc09f7b0 con 0x7fbadc093c50 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 0x7fbadc063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient: get_auth_request con 0x7fbadc093c50 auth_method 0 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 10 monclient(hunting): handle_auth_done global_id 154387 payload 824 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.815+0000 7fbae1854640 1 -- 172.21.15.164:0/3891765261 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbadc073770 con 0x7fbadc093c50 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.817+0000 7fbae1854640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 msgr2=0x7fbadc063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.817+0000 7fbae1854640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 0x7fbadc063d50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad8017000 tx=0x7fbad8004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:35.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbad801a050 con 0x7fbadc093c50 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 10 monclient: dump: 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:35.829 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fbad801b040 con 0x7fbadc093c50 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 msgr2=0x7fbadc063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 0x7fbadc063d50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad8017000 tx=0x7fbad8004940 comp rx=0 tx=0).stop 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 msgr2=0x7fbadc063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.818+0000 7fbad27fc640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbadc093c50 0x7fbadc063d50 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fbad8017000 tx=0x7fbad8004940 comp rx=0 tx=0).stop 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:35.830 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 158, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 174369, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:35.831 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.832 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.833 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.834 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:35.835 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:35.836 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.837 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.838 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.839 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:35.840 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:35.841 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.842 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:22:35.843 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:35.844 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:35.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbac003e240 msgr2=0x7fbac0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbac003e240 0x7fbac0040700 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fbad400f1d0 tx=0x7fbad4003d10 comp rx=0 tx=0).stop 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 10 monclient: shutdown 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 msgr2=0x7fbadc092db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 --2- 172.21.15.164:0/3891765261 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbadc13a5c0 0x7fbadc092db0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fbacc0098c0 tx=0x7fbacc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae2055640 1 -- 172.21.15.164:0/3891765261 reap_dead start 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae2055640 1 -- 172.21.15.164:0/3891765261 reap_dead start 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 shutdown_connections 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 >> 172.21.15.164:0/3891765261 conn(0x7fbadc0a3110 msgr2=0x7fbadc1399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 shutdown_connections 2024-09-10T16:22:35.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:35.819+0000 7fbae3ab8640 1 -- 172.21.15.164:0/3891765261 wait complete. 2024-09-10T16:22:35.847 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:22:35.847 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:22:35.847 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-10T16:22:35.847 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:22:35.847 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:22:35.847 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:22:35.848 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:22:35.857 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:22:35.857 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:22:35.857 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:22:35.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:22:50.858 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:22:50.858 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:22:50.859 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:22:50.859 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:22:50.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.965+0000 7ff0398fe640 10 monclient: get_monmap_and_config 2024-09-10T16:22:50.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.965+0000 7ff0398fe640 10 monclient: build_initial_monmap 2024-09-10T16:22:50.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.965+0000 7ff0398fe640 10 monclient: monmap: 2024-09-10T16:22:50.968 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:50.968 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:50.968 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:50.967292+0000 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:50.967292+0000 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:50.969 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:50.970 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:50.970 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:50.970 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:50.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 Processor -- start 2024-09-10T16:22:50.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 -- start start 2024-09-10T16:22:50.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient: init 2024-09-10T16:22:50.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff0340a4540 addr v2:172.21.15.165:3300/0 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0341385c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff0340a60b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0340a71c0 con 0x7ff0341385c0 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff03409c9f0 con 0x7ff0340a60b0 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:50.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff03409ccd0 con 0x7ff0340a4540 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0398fe640 10 monclient(hunting): authenticate will time out at 1749.454829s 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0337fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff02bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff0341385c0 0x7ff0341345e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38310/0 (socket says 172.21.15.164:38310) 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 learned_addr learned my addr 172.21.15.164:0/408393947 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff0340a60b0 auth_method 0 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.966+0000 7ff0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:22:50.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff0340a4540 auth_method 0 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:22:50.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.967+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:50.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:50.969+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> v1:172.21.15.164:6789/0 conn(0x7ff0341385c0 legacy=0x7ff0341345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:51.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff0340a60b0 auth_method 0 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:22:51.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff0340a4540 auth_method 0 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:51.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.168+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff0340a4540 auth_method 0 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:22:51.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff0340a60b0 auth_method 0 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:51.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:51.569+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:22:52.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:52.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff0340a4540 auth_method 0 2024-09-10T16:22:52.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff0340a60b0 auth_method 0 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:52.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:22:52.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:22:52.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:22:52.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:52.370+0000 7ff0337fe640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 10 monclient(hunting): tick 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 msgr2=0x7ff0340a4920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.165:3300/0 conn(0x7ff0340a4540 0x7ff0340a4920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 -- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 msgr2=0x7ff034134cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:53.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 --2- 172.21.15.164:0/408393947 >> v2:172.21.15.164:3300/0 conn(0x7ff0340a60b0 0x7ff034134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.966+0000 7ff0317fa640 1 -- 172.21.15.164:0/408393947 >> v1:172.21.15.164:6789/0 conn(0x7ff0341385c0 legacy=0x7ff0341345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 10 monclient: _add_conns ranks=[1,2] 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff014003720 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 10 monclient(hunting): picked mon.noname-b con 0x7ff014007410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 1 -- 172.21.15.164:0/408393947 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff03409cfb0 con 0x7ff014003720 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 1 -- 172.21.15.164:0/408393947 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff03409c9f0 con 0x7ff014007410 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff0317fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 msgr2=0x7ff0140098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:53.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.968+0000 7ff02bfff640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:53.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:53.970+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> v1:172.21.15.165:6789/0 conn(0x7ff014003720 legacy=0x7ff014005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:54.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:54.169+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 msgr2=0x7ff0140098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:54.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:54.169+0000 7ff02bfff640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:22:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:54.570+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 msgr2=0x7ff0140098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:54.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:54.570+0000 7ff02bfff640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:22:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.371+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 msgr2=0x7ff0140098b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.371+0000 7ff02bfff640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:22:55.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314960686 0 0) 0x7ff03409cfb0 con 0x7ff014003720 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 10 monclient(hunting): _negotiate 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 10 monclient(hunting): my global_id is 154390 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff020003520 con 0x7ff014003720 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1528726146 0 0) 0x7ff020003520 con 0x7ff014003720 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 msgr2=0x7ff0140098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.876+0000 7ff0327fc640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff014007410 0x7ff0140098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _finish_hunting 0 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:22:55.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff01400b0f0 con 0x7ff014003720 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _finish_auth 0 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _check_auth_tickets 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7ff02400d680 con 0x7ff014003720 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0398fe640 5 monclient: authenticate success, global_id 154390 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0398fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: dump: 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:55.882 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:55.883 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 >> v1:172.21.15.165:6789/0 conn(0x7ff014003720 legacy=0x7ff014005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0337fe640 1 -- 172.21.15.164:0/408393947 reap_dead start 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 0x7ff020003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): picked mon.b con 0x7ff020003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff020004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): picked mon.c con 0x7ff020004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0200053c0 0x7ff02000ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): picked mon.a con 0x7ff0200053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:55.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff03409c9f0 con 0x7ff0200053c0 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff02000c2e0 con 0x7ff020003a20 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff02000c4e0 con 0x7ff020004800 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0327fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff0398fe640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff02bfff640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 msgr2=0x7ff020004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff02bfff640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff020004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 0x7ff020003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff020003a20 auth_method 0 2024-09-10T16:22:55.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient(hunting): handle_auth_done global_id 154390 payload 293 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 msgr2=0x7ff020004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff020004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0200053c0 msgr2=0x7ff02000ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0200053c0 0x7ff02000ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 monclient: found mon.b 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:55.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:22:55.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff020003520 con 0x7ff020003a20 2024-09-10T16:22:55.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient: _finish_auth 0 2024-09-10T16:22:55.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:22:55.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.877+0000 7ff032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:55.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff032ffd640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 msgr2=0x7ff020003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:55.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff032ffd640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 0x7ff020003e20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff020005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=872e6dedd5aea55b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:55.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024031070 con 0x7ff020003a20 2024-09-10T16:22:55.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: dump: 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:55.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff02402bc40 con 0x7ff020003a20 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0398fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 1 -- 172.21.15.164:0/408393947 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff02400de40 con 0x7ff020003a20 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0398fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0398fe640 4 set_mon_vals no callback set 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.881+0000 7ff0327fc640 10 monclient: dump: 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:55.896 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 10 monclient: shutdown 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 -- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 msgr2=0x7ff020003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 0x7ff020003e20 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff020005830 comp rx=0 tx=0).stop 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 -- 172.21.15.164:0/408393947 shutdown_connections 2024-09-10T16:22:55.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0200053c0 0x7ff02000ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff020004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 --2- 172.21.15.164:0/408393947 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff020003a20 0x7ff020003e20 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 -- 172.21.15.164:0/408393947 >> 172.21.15.164:0/408393947 conn(0x7ff0340aaa00 msgr2=0x7ff0340aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 -- 172.21.15.164:0/408393947 shutdown_connections 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.882+0000 7ff0398fe640 1 -- 172.21.15.164:0/408393947 wait complete. 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.883+0000 7ff0398fe640 10 monclient: build_initial_monmap 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.883+0000 7ff0398fe640 10 monclient: monmap: 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:55.898 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.883+0000 7ff0398fe640 1 Processor -- start 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.883+0000 7ff0398fe640 1 -- start start 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.883+0000 7ff0398fe640 10 monclient: init 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-a con 0x7ff020003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:55.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff0341cd220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-c con 0x7ff020004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0200053c0 0x7ff0340a1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): picked mon.noname-b con 0x7ff0200053c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0340a4aa0 con 0x7ff020003a20 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff0340a4920 con 0x7ff0200053c0 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff0340a4c20 con 0x7ff020004800 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0398fe640 10 monclient(hunting): authenticate will time out at 1754.371408s 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff02bfff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 msgr2=0x7ff0341cd220 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:55.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff02bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff0341cd220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43618/0 (socket says 172.21.15.164:43618) 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 -- 172.21.15.164:0/2994076856 learned_addr learned my addr 172.21.15.164:0/2994076856 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): get_auth_request con 0x7ff020003a20 auth_method 0 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0200053c0 0x7ff0340a1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:55.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): get_auth_request con 0x7ff0200053c0 auth_method 0 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff0337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient(hunting): handle_auth_done global_id 164328 payload 824 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 -- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 msgr2=0x7ff0341cd220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff0341cd220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 -- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0200053c0 msgr2=0x7ff0340a1fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0200053c0 0x7ff0340a1fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:22:55.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 monclient: found mon.noname-a 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.884+0000 7ff032ffd640 1 -- 172.21.15.164:0/2994076856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0340a9de0 con 0x7ff020003a20 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff032ffd640 1 -- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 msgr2=0x7ff0341ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff032ffd640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff0240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024006030 con 0x7ff020003a20 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: dump: 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:55.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: _finish_auth 0 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff024044070 con 0x7ff020003a20 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 5 monclient: authenticate success, global_id 164328 2024-09-10T16:22:55.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 10 monclient: _renew_subs 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff034122550 con 0x7ff020003a20 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff024006450 con 0x7ff020003a20 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff030ff9640 10 monclient: dump: 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:55.905 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff031ffb640 4 set_mon_vals no callback set 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 10 monclient: _renew_subs 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.887+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff03406e7d0 con 0x7ff020003a20 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff0240065f0 con 0x7ff020003a20 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff0398fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff0398fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:55.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff0398fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0341345e0 con 0x7ff020003a20 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.888+0000 7ff030ff9640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 0x7ff014041000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.889+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(558..558 src has 481..558) ==== 4771+0+0 (secure 0 0 0) 0x7ff024073af0 con 0x7ff020003a20 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.889+0000 7ff02bfff640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 0x7ff014041000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.889+0000 7ff02bfff640 10 monclient: get_auth_request con 0x7ff01403eb40 auth_method 0 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.891+0000 7ff02bfff640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 0x7ff014041000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff0341321a0 tx=0x7ff01c007a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.893+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff0340a9de0 con 0x7ff020003a20 2024-09-10T16:22:55.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.893+0000 7ff030ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:55.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:55.893+0000 7ff030ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff0398fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff0398fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff0398fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff034077260 con 0x7ff020003a20 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff030ff9640 1 -- 172.21.15.164:0/2994076856 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7ff0240724f0 con 0x7ff020003a20 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff030ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.141+0000 7ff030ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:56.143 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":162,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:22:56.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.143+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 msgr2=0x7ff014041000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.143+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 0x7ff014041000 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7ff0341321a0 tx=0x7ff01c007a80 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.143+0000 7ff0398fe640 10 monclient: shutdown 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.143+0000 7ff0398fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.143+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 msgr2=0x7ff0341ccce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff024006fd0 tx=0x7ff0240046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 shutdown_connections 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff01403eb40 0x7ff014041000 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff0200053c0 0x7ff0340a1fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff020004800 0x7ff0341cd220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 --2- 172.21.15.164:0/2994076856 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff020003a20 0x7ff0341ccce0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 >> 172.21.15.164:0/2994076856 conn(0x7ff0340aaa00 msgr2=0x7ff0341333e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 shutdown_connections 2024-09-10T16:22:56.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.144+0000 7ff0398fe640 1 -- 172.21.15.164:0/2994076856 wait complete. 2024-09-10T16:22:56.156 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:22:56.156 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:22:56.156 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:22:56.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.254+0000 7fcc608b0640 10 monclient: get_monmap_and_config 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.254+0000 7fcc608b0640 10 monclient: build_initial_monmap 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.254+0000 7fcc608b0640 10 monclient: monmap: 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:56.255400+0000 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:56.255400+0000 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:56.256 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:56.257 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:56.257 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:56.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 Processor -- start 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 -- start start 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient: init 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:22:56.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 0x7fcc5c0a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-b con 0x7fcc5c09f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcc5c0a2380 0x7fcc5c0a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5c0a2380 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcc5c13a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc5c09fd10 con 0x7fcc5c13a5c0 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fcc5c0a2cc0 con 0x7fcc5c0a2380 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5c09cd70 con 0x7fcc5c09f930 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc608b0640 10 monclient(hunting): authenticate will time out at 1754.742798s 2024-09-10T16:22:56.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc5a59c640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 msgr2=0x7fcc5c0a1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc5a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 0x7fcc5c0a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcc5c0a2380 0x7fcc5c0a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcc5c0a2380 0x7fcc5c0a2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43620/0 (socket says 172.21.15.164:43620) 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 1 -- 172.21.15.164:0/4206588438 learned_addr learned my addr 172.21.15.164:0/4206588438 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): get_auth_request con 0x7fcc5c0a2380 auth_method 0 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.255+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_done global_id 164331 payload 274 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 1 -- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 msgr2=0x7fcc5c0a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 0x7fcc5c0a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 1 -- 172.21.15.164:0/4206588438 >> v1:172.21.15.165:6789/0 conn(0x7fcc5c13a5c0 legacy=0x7fcc5c13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 1 monclient: found mon.noname-a 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.256+0000 7fcc59d9b640 1 -- 172.21.15.164:0/4206588438 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c09a120 con 0x7fcc5c0a2380 2024-09-10T16:22:56.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.258+0000 7fcc59d9b640 1 -- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c0a2380 msgr2=0x7fcc5c0a2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.258+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c0a2380 0x7fcc5c0a2780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65eca795902ba282 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 1 -- 172.21.15.164:0/4206588438 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44037070 con 0x7fcc5c0a2380 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: dump: 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.268 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: _finish_auth 0 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: _check_auth_tickets 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 5 monclient: authenticate success, global_id 164331 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 1 -- 172.21.15.164:0/4206588438 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc44032640 con 0x7fcc5c0a2380 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:56.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 4 set_mon_vals no callback set 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 1 -- 172.21.15.164:0/4206588438 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44032a10 con 0x7fcc5c0a2380 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc5959a640 10 monclient: dump: 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.270 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 10 monclient: shutdown 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 -- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c0a2380 msgr2=0x7fcc5c0a2780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 --2- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c0a2380 0x7fcc5c0a2780 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcc44009870 tx=0x7fcc4402a0d0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 -- 172.21.15.164:0/4206588438 shutdown_connections 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 --2- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c0a2380 0x7fcc5c0a2780 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 --2- 172.21.15.164:0/4206588438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c09f930 0x7fcc5c0a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 -- 172.21.15.164:0/4206588438 >> 172.21.15.164:0/4206588438 conn(0x7fcc5c0a3110 msgr2=0x7fcc5c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 -- 172.21.15.164:0/4206588438 shutdown_connections 2024-09-10T16:22:56.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.259+0000 7fcc608b0640 1 -- 172.21.15.164:0/4206588438 wait complete. 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.260+0000 7fcc608b0640 10 monclient: build_initial_monmap 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.260+0000 7fcc608b0640 10 monclient: monmap: 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.260+0000 7fcc608b0640 1 Processor -- start 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.260+0000 7fcc608b0640 1 -- start start 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.260+0000 7fcc608b0640 10 monclient: init 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 0x7fcc5c0737d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-a con 0x7fcc5c09f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c0a2380 0x7fcc5c09a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-c con 0x7fcc5c0a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5c13a5c0 0x7fcc5c09a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): picked mon.noname-b con 0x7fcc5c13a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c09d930 con 0x7fcc5c09f930 2024-09-10T16:22:56.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c09ee00 con 0x7fcc5c13a5c0 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcc5c09c8b0 con 0x7fcc5c0a2380 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc608b0640 10 monclient(hunting): authenticate will time out at 1754.748367s 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc59d9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c0a2380 msgr2=0x7fcc5c09a430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc59d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c0a2380 0x7fcc5c09a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 0x7fcc5c0737d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 0x7fcc5c0737d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43624/0 (socket says 172.21.15.164:43624) 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 -- 172.21.15.164:0/1557426045 learned_addr learned my addr 172.21.15.164:0/1557426045 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): get_auth_request con 0x7fcc5c09f930 auth_method 0 2024-09-10T16:22:56.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient(hunting): handle_auth_done global_id 164334 payload 824 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c0a2380 msgr2=0x7fcc5c09a430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcc5c0a2380 0x7fcc5c09a430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5c13a5c0 msgr2=0x7fcc5c09a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcc5c13a5c0 0x7fcc5c09a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 monclient: found mon.noname-a 2024-09-10T16:22:56.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.261+0000 7fcc5a59c640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc5c131320 con 0x7fcc5c09f930 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.263+0000 7fcc5a59c640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 msgr2=0x7fcc5c0737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.263+0000 7fcc5a59c640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 0x7fcc5c0737d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b550 tx=0x7fcc4c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c005070 con 0x7fcc5c09f930 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: dump: 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: _finish_auth 0 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc4c015070 con 0x7fcc5c09f930 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 5 monclient: authenticate success, global_id 164334 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc4c005450 con 0x7fcc5c09f930 2024-09-10T16:22:56.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc52ffd640 10 monclient: dump: 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.278 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 monclient: _renew_subs 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc5c09c130 con 0x7fcc5c09f930 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc58d99640 4 set_mon_vals no callback set 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 monclient: _renew_subs 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc5c09d620 con 0x7fcc5c09f930 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 0x7fcc5c09aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c0a1810 con 0x7fcc5c02e5c0 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 0x7fcc5c09aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.264+0000 7fcc59d9b640 10 monclient: get_auth_request con 0x7fcc5c02e5c0 auth_method 0 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcc4c0055f0 con 0x7fcc5c09f930 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc52ffd640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcc3c03e1c0 0x7fcc3c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_done global_id 164337 payload 824 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc59d9b640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcc5c02e9a0 con 0x7fcc5c02e5c0 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(558..558 src has 481..558) ==== 4771+0+0 (secure 0 0 0) 0x7fcc4c052c30 con 0x7fcc5c09f930 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc5ad9d640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcc3c03e1c0 0x7fcc3c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.265+0000 7fcc5ad9d640 10 monclient: get_auth_request con 0x7fcc3c03e1c0 auth_method 0 2024-09-10T16:22:56.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc59d9b640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 msgr2=0x7fcc5c09aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 0x7fcc5c09aeb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc44034000 tx=0x7fcc440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44042070 con 0x7fcc5c02e5c0 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc52ffd640 10 monclient: dump: 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.284 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.267+0000 7fcc5ad9d640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcc3c03e1c0 0x7fcc3c040680 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c09b4c0 tx=0x7fcc54007d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcc44032a00 con 0x7fcc5c02e5c0 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 msgr2=0x7fcc5c09aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 0x7fcc5c09aeb0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc44034000 tx=0x7fcc440046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 msgr2=0x7fcc5c09aeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:56.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.268+0000 7fcc52ffd640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c02e5c0 0x7fcc5c09aeb0 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcc44034000 tx=0x7fcc440046c0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc608b0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc608b0640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 0x7fcc5c064130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc608b0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcc5c0a0760 con 0x7fcc5c063d50 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 0x7fcc5c064130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient: get_auth_request con 0x7fcc5c063d50 auth_method 0 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 10 monclient(hunting): handle_auth_done global_id 164340 payload 824 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.274+0000 7fcc59d9b640 1 -- 172.21.15.164:0/1557426045 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcc5c0718e0 con 0x7fcc5c063d50 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.276+0000 7fcc59d9b640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 msgr2=0x7fcc5c064130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.276+0000 7fcc59d9b640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 0x7fcc5c064130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc44009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44037040 con 0x7fcc5c063d50 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 10 monclient: dump: 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7fcc44003610 con 0x7fcc5c063d50 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 msgr2=0x7fcc5c064130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 0x7fcc5c064130 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc44009240 comp rx=0 tx=0).stop 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 msgr2=0x7fcc5c064130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:56.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc52ffd640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c063d50 0x7fcc5c064130 secure :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcc44000c00 tx=0x7fcc44009240 comp rx=0 tx=0).stop 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 162, 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 146980, 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:22:56.289 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.290 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.291 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:56.292 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.293 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:56.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.295 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.296 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.297 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:56.299 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.300 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:56.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.302 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:56.302 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:56.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcc3c03e1c0 msgr2=0x7fcc3c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcc3c03e1c0 0x7fcc3c040680 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fcc5c09b4c0 tx=0x7fcc54007d30 comp rx=0 tx=0).stop 2024-09-10T16:22:56.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 10 monclient: shutdown 2024-09-10T16:22:56.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 msgr2=0x7fcc5c0737d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 --2- 172.21.15.164:0/1557426045 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcc5c09f930 0x7fcc5c0737d0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c00b550 tx=0x7fcc4c00ba10 comp rx=0 tx=0).stop 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc5ad9d640 1 -- 172.21.15.164:0/1557426045 reap_dead start 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc5ad9d640 1 -- 172.21.15.164:0/1557426045 reap_dead start 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 shutdown_connections 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 >> 172.21.15.164:0/1557426045 conn(0x7fcc5c0a3110 msgr2=0x7fcc5c138b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 shutdown_connections 2024-09-10T16:22:56.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.277+0000 7fcc608b0640 1 -- 172.21.15.164:0/1557426045 wait complete. 2024-09-10T16:22:56.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:22:56.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:22:56.305 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:22:56.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.402+0000 7f370628c640 10 monclient: get_monmap_and_config 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.402+0000 7f370628c640 10 monclient: build_initial_monmap 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.402+0000 7f370628c640 10 monclient: monmap: 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:22:56.403121+0000 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:22:56.403121+0000 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:22:56.404 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:22:56.405 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:22:56.405 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 Processor -- start 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 -- start start 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient: init 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f37000a0410 addr v1:172.21.15.165:6789/0 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f37000a2550 addr v1:172.21.15.164:6789/0 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f37000a29e0 0x7f370013ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): picked mon.noname-a con 0x7f37000a29e0 addr v2:172.21.15.164:3300/0 2024-09-10T16:22:56.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37000a0210 con 0x7f37000a2550 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f370009d930 con 0x7f37000a0410 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f370009d3f0 con 0x7f37000a29e0 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f370628c640 10 monclient(hunting): authenticate will time out at 1754.890733s 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f37000a29e0 0x7f370013ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f36f7fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f37000a2550 0x7f370013b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:35934/0 (socket says 172.21.15.164:35934) 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f36f7fff640 1 -- 172.21.15.164:0/1419581246 learned_addr learned my addr 172.21.15.164:0/1419581246 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): get_auth_request con 0x7f37000a29e0 auth_method 0 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.403+0000 7f3704829640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 10 monclient(hunting): handle_auth_done global_id 164343 payload 274 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 1 -- 172.21.15.164:0/1419581246 >> v1:172.21.15.165:6789/0 conn(0x7f37000a0410 legacy=0x7f37000a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 1 -- 172.21.15.164:0/1419581246 >> v1:172.21.15.164:6789/0 conn(0x7f37000a2550 legacy=0x7f370013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 10 monclient: _finish_hunting 0 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 1 monclient: found mon.noname-a 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.404+0000 7f3704829640 1 -- 172.21.15.164:0/1419581246 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370009e7d0 con 0x7f37000a29e0 2024-09-10T16:22:56.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.406+0000 7f3704829640 1 -- 172.21.15.164:0/1419581246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a29e0 msgr2=0x7f370013ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.406+0000 7f3704829640 1 --2- 172.21.15.164:0/1419581246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a29e0 0x7f370013ed00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3700075190 tx=0x7f36f802a460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91e275760d93b74b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 1 -- 172.21.15.164:0/1419581246 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8007a30 con 0x7f37000a29e0 2024-09-10T16:22:56.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: dump: 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.417 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: _finish_auth 0 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 5 monclient: authenticate success, global_id 164343 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 1 -- 172.21.15.164:0/1419581246 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f8007bd0 con 0x7f37000a29e0 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 10 monclient: get_monmap_and_config success 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 4 set_mon_vals no callback set 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 1 -- 172.21.15.164:0/1419581246 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8031400 con 0x7f37000a29e0 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f36ff7fe640 10 monclient: dump: 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 10 monclient: shutdown 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 -- 172.21.15.164:0/1419581246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a29e0 msgr2=0x7f370013ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 --2- 172.21.15.164:0/1419581246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a29e0 0x7f370013ed00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3700075190 tx=0x7f36f802a460 comp rx=0 tx=0).stop 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 -- 172.21.15.164:0/1419581246 shutdown_connections 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 --2- 172.21.15.164:0/1419581246 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a29e0 0x7f370013ed00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 -- 172.21.15.164:0/1419581246 >> 172.21.15.164:0/1419581246 conn(0x7f37000a3110 msgr2=0x7f37000a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 -- 172.21.15.164:0/1419581246 shutdown_connections 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.407+0000 7f370628c640 1 -- 172.21.15.164:0/1419581246 wait complete. 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 10 monclient: build_initial_monmap 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 10 monclient: monmap: 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:22:56.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 1 Processor -- start 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 1 -- start start 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 10 monclient: init 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.408+0000 7f370628c640 10 monclient: _reopen_session rank -1 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 0x7f37001cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): picked mon.noname-a con 0x7f37000a0410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000a2550 0x7f37001ba740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): picked mon.noname-b con 0x7f37000a2550 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37000a29e0 0x7f37001bac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): picked mon.noname-c con 0x7f37000a29e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f37000a07f0 con 0x7f37000a0410 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f370009c910 con 0x7f37000a2550 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f37000a0970 con 0x7f37000a29e0 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): _renew_subs 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f370628c640 10 monclient(hunting): authenticate will time out at 1754.896073s 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f3704829640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37000a29e0 msgr2=0x7f37001bac80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f3704829640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37000a29e0 0x7f37001bac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:22:56.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 0x7f37001cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 0x7f37001cd050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:43664/0 (socket says 172.21.15.164:43664) 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 -- 172.21.15.164:0/3838779312 learned_addr learned my addr 172.21.15.164:0/3838779312 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): get_auth_request con 0x7f37000a0410 auth_method 0 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000a2550 0x7f37001ba740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): get_auth_request con 0x7f37000a2550 auth_method 0 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient(hunting): handle_auth_done global_id 164346 payload 824 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37000a29e0 msgr2=0x7f37001bac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f37000a29e0 0x7f37001bac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000a2550 msgr2=0x7f37001ba740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f37000a2550 0x7f37001ba740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient: _finish_hunting 0 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 monclient: found mon.noname-a 2024-09-10T16:22:56.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.409+0000 7f36fffff640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f370009bca0 con 0x7f37000a0410 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fffff640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 msgr2=0x7f37001cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fffff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 0x7f37001cd050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c410 tx=0x7f36f000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0017070 con 0x7f37000a0410 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: dump: 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.425 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f00037f0 con 0x7f37000a0410 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 5 monclient: authenticate success, global_id 164346 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 10 monclient: _renew_subs 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37001b6200 con 0x7f37000a0410 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0003be0 con 0x7f37000a0410 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36fd7fa640 10 monclient: dump: 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.427 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 10 monclient: _renew_subs 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f36feffd640 4 set_mon_vals no callback set 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.411+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37001cd590 con 0x7f37000a0410 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f370628c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f370628c640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 0x7f370009a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f370009d620 con 0x7f370009a430 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f36f001a070 con 0x7f37000a0410 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f36f7fff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 0x7f370009a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.412+0000 7f36fd7fa640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36e003e210 0x7f36e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient: get_auth_request con 0x7f370009a430 auth_method 0 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(558..558 src has 481..558) ==== 4771+0+0 (secure 0 0 0) 0x7f36f00521d0 con 0x7f37000a0410 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f3704829640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36e003e210 0x7f36e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f3704829640 10 monclient: get_auth_request con 0x7f36e003e210 auth_method 0 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 10 monclient(hunting): handle_auth_done global_id 154399 payload 824 2024-09-10T16:22:56.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.413+0000 7f36f7fff640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f370009fb30 con 0x7f370009a430 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36f7fff640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 msgr2=0x7f370009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36f7fff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 0x7f370009a810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36e8007fd0 tx=0x7f36e800ef00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f3704829640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36e003e210 0x7f36e00406d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3700074de0 tx=0x7f36f802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e801a070 con 0x7f370009a430 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.415+0000 7f36fd7fa640 10 monclient: dump: 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f36e800cc50 con 0x7f370009a430 2024-09-10T16:22:56.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 msgr2=0x7f370009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 0x7f370009a810 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36e8007fd0 tx=0x7f36e800ef00 comp rx=0 tx=0).stop 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 msgr2=0x7f370009a810 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.416+0000 7f36fd7fa640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009a430 0x7f370009a810 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f36e8007fd0 tx=0x7f36e800ef00 comp rx=0 tx=0).stop 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f370628c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f370628c640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 0x7f37000703d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f370628c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f37001b5d40 con 0x7f370009ad50 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 0x7f37000703d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient: get_auth_request con 0x7f370009ad50 auth_method 0 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:22:56.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 10 monclient(hunting): handle_auth_done global_id 154402 payload 824 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.422+0000 7f36f7fff640 1 -- 172.21.15.164:0/3838779312 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3700077260 con 0x7f370009ad50 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36f7fff640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 msgr2=0x7f37000703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36f7fff640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 0x7f37000703d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36e8017000 tx=0x7f36e8004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e801a070 con 0x7f370009ad50 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 10 monclient: dump: 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:22:56.435 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f36e801b040 con 0x7f370009ad50 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 msgr2=0x7f37000703d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 0x7f37000703d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36e8017000 tx=0x7f36e8004940 comp rx=0 tx=0).stop 2024-09-10T16:22:56.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 msgr2=0x7f37000703d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f36fd7fa640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f370009ad50 0x7f37000703d0 secure :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f36e8017000 tx=0x7f36e8004940 comp rx=0 tx=0).stop 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 162, 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 194975, 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:22:56.437 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.438 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.439 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:22:56.440 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:22:56.441 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:22:56.442 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.444 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.445 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.447 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.448 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:22:56.449 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:22:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:22:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:22:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:22:56.450 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:22:56.450 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:22:56.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36e003e210 msgr2=0x7f36e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f36e003e210 0x7f36e00406d0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f3700074de0 tx=0x7f36f802f000 comp rx=0 tx=0).stop 2024-09-10T16:22:56.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 10 monclient: shutdown 2024-09-10T16:22:56.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:22:56.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 msgr2=0x7f37001cd050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 1 --2- 172.21.15.164:0/3838779312 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f37000a0410 0x7f37001cd050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f36f000c410 tx=0x7f36f000c8d0 comp rx=0 tx=0).stop 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f3704829640 1 -- 172.21.15.164:0/3838779312 reap_dead start 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f3704829640 1 -- 172.21.15.164:0/3838779312 reap_dead start 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.425+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 shutdown_connections 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.426+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 >> 172.21.15.164:0/3838779312 conn(0x7f37000a3110 msgr2=0x7f37000a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.426+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 shutdown_connections 2024-09-10T16:22:56.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:22:56.426+0000 7f370628c640 1 -- 172.21.15.164:0/3838779312 wait complete. 2024-09-10T16:22:56.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:22:56.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:22:56.879 INFO:tasks.rados.rados.0.smithi165.stdout:2463: done (0 left) 2024-09-10T16:22:56.880 INFO:tasks.rados.rados.0.smithi165.stdout:2464: read oid 34 snap -1 2024-09-10T16:22:56.880 INFO:tasks.rados.rados.0.smithi165.stdout:2464: expect (ObjNum 872 snap 266 seq_num 872) 2024-09-10T16:22:56.880 INFO:tasks.rados.rados.0.smithi165.stdout:2465: write oid 27 current snap is 268 2024-09-10T16:22:56.880 INFO:tasks.rados.rados.0.smithi165.stdout:2465: seq_num 874 ranges {689318=631868,1877968=498292} 2024-09-10T16:22:56.886 INFO:tasks.rados.rados.0.smithi165.stdout:2465: writing smithi16534841-27 from 689318 to 1321186 tid 1 2024-09-10T16:22:56.892 INFO:tasks.rados.rados.0.smithi165.stdout:2465: writing smithi16534841-27 from 1877968 to 2376260 tid 2 2024-09-10T16:22:56.892 INFO:tasks.rados.rados.0.smithi165.stdout:2464: expect (ObjNum 872 snap 266 seq_num 872) 2024-09-10T16:22:56.893 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:56.942 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:56.990 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:22:57.039 INFO:tasks.rados.rados.0.smithi165.stdout:2465: finishing write tid 1 to smithi16534841-27 2024-09-10T16:22:57.039 INFO:tasks.rados.rados.0.smithi165.stdout:2465: finishing write tid 2 to smithi16534841-27 2024-09-10T16:22:57.039 INFO:tasks.rados.rados.0.smithi165.stdout:2465: finishing write tid 3 to smithi16534841-27 2024-09-10T16:22:57.039 INFO:tasks.rados.rados.0.smithi165.stdout:2465: finishing write tid 4 to smithi16534841-27 2024-09-10T16:22:57.039 INFO:tasks.rados.rados.0.smithi165.stdout:2465: oid 27 updating version 0 to 1440 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2465: oid 27 version 1440 is already newer than 1439 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2465: oid 27 version 1440 is already newer than 1438 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1440 (ObjNum 874 snap 268 seq_num 874) dirty exists 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2465: left oid 27 (ObjNum 874 snap 268 seq_num 874) 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2466: read oid 2 snap -1 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2466: expect deleted 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2464: done (2 left) 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2465: done (1 left) 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2467: rollback oid 4 current snap is 268 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 230 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2468: rollback oid 45 current snap is 268 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 230 2024-09-10T16:22:57.040 INFO:tasks.rados.rados.0.smithi165.stdout:2469: rollback oid 35 current snap is 268 2024-09-10T16:22:57.041 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 265 2024-09-10T16:22:57.041 INFO:tasks.rados.rados.0.smithi165.stdout:2470: rollback oid 37 current snap is 268 2024-09-10T16:22:57.041 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 261 2024-09-10T16:22:57.041 INFO:tasks.rados.rados.0.smithi165.stdout:2471: snap_remove snap 267 2024-09-10T16:22:57.885 INFO:tasks.rados.rados.0.smithi165.stdout:2471: done (5 left) 2024-09-10T16:22:57.885 INFO:tasks.rados.rados.0.smithi165.stdout:2472: snap_create 2024-09-10T16:22:57.885 INFO:tasks.rados.rados.0.smithi165.stdout:2470: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:22:57.885 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1334 (ObjNum 33 snap 0 seq_num 1127806872) dirty dne 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2469: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2469: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2468: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2468: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1357 (ObjNum 731 snap 223 seq_num 731) dirty exists 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2469: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1376 (ObjNum 714 snap 220 seq_num 714) dirty exists 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2467: finishing rollback tid 0 to smithi16534841-4 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2467: finishing rollback tid 1 to smithi16534841-4 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:2467: finishing rollback tid 2 to smithi16534841-4 2024-09-10T16:22:57.886 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1318 (ObjNum 745 snap 227 seq_num 745) dirty exists 2024-09-10T16:22:58.896 INFO:tasks.rados.rados.0.smithi165.stdout:2466: done (5 left) 2024-09-10T16:22:58.896 INFO:tasks.rados.rados.0.smithi165.stdout:2467: done (4 left) 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2468: done (3 left) 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2469: done (2 left) 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2470: done (1 left) 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2472: done (0 left) 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2473: rollback oid 37 current snap is 269 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 264 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2474: write (excl) oid 34 current snap is 269 2024-09-10T16:22:58.897 INFO:tasks.rados.rados.0.smithi165.stdout:2474: seq_num 875 ranges {706125=423470,1652803=758303} 2024-09-10T16:22:58.902 INFO:tasks.rados.rados.0.smithi165.stdout:2474: writing smithi16534841-34 from 706125 to 1129595 tid 1 2024-09-10T16:22:58.909 INFO:tasks.rados.rados.0.smithi165.stdout:2474: writing smithi16534841-34 from 1652803 to 2411106 tid 2 2024-09-10T16:22:58.909 INFO:tasks.rados.rados.0.smithi165.stdout:2475: write (excl) oid 12 current snap is 269 2024-09-10T16:22:58.909 INFO:tasks.rados.rados.0.smithi165.stdout:2473: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:22:58.910 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1334 (ObjNum -232809232 snap 32725 seq_num 304) dirty dne 2024-09-10T16:22:58.910 INFO:tasks.rados.rados.0.smithi165.stdout:2474: finishing write tid 1 to smithi16534841-34 2024-09-10T16:22:58.910 INFO:tasks.rados.rados.0.smithi165.stdout:2475: seq_num 876 ranges {722932=498719,1994932=501961} 2024-09-10T16:22:58.915 INFO:tasks.rados.rados.0.smithi165.stdout:2475: writing smithi16534841-12 from 722932 to 1221651 tid 1 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2475: writing smithi16534841-12 from 1994932 to 2496893 tid 2 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2473: done (2 left) 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2476: delete oid 26 current snap is 269 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2474: finishing write tid 2 to smithi16534841-34 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2474: finishing write tid 3 to smithi16534841-34 2024-09-10T16:22:58.924 INFO:tasks.rados.rados.0.smithi165.stdout:2474: finishing write tid 4 to smithi16534841-34 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:2474: oid 34 updating version 0 to 1344 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:2474: oid 34 version 1344 is already newer than 1342 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:2474: oid 34 version 1344 is already newer than 1343 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1344 (ObjNum 875 snap 269 seq_num 875) dirty exists 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:2474: left oid 34 (ObjNum 875 snap 269 seq_num 875) 2024-09-10T16:22:58.925 INFO:tasks.rados.rados.0.smithi165.stdout:2475: finishing write tid 1 to smithi16534841-12 2024-09-10T16:22:58.926 INFO:tasks.rados.rados.0.smithi165.stdout:2475: finishing write tid 2 to smithi16534841-12 2024-09-10T16:22:58.927 INFO:tasks.rados.rados.0.smithi165.stdout:2474: done (2 left) 2024-09-10T16:22:58.927 INFO:tasks.rados.rados.0.smithi165.stdout:2476: done (1 left) 2024-09-10T16:22:58.927 INFO:tasks.rados.rados.0.smithi165.stdout:2477: write (excl) oid 46 current snap is 269 2024-09-10T16:22:58.927 INFO:tasks.rados.rados.0.smithi165.stdout:2477: seq_num 878 ranges {756546=649217,2111896=505630} 2024-09-10T16:22:58.934 INFO:tasks.rados.rados.0.smithi165.stdout:2477: writing smithi16534841-46 from 756546 to 1405763 tid 1 2024-09-10T16:22:58.941 INFO:tasks.rados.rados.0.smithi165.stdout:2477: writing smithi16534841-46 from 2111896 to 2617526 tid 2 2024-09-10T16:22:58.941 INFO:tasks.rados.rados.0.smithi165.stdout:2478: write oid 4 current snap is 269 2024-09-10T16:22:58.941 INFO:tasks.rados.rados.0.smithi165.stdout:2478: seq_num 879 ranges {773353=724466,2170378=602975} 2024-09-10T16:22:58.949 INFO:tasks.rados.rados.0.smithi165.stdout:2478: writing smithi16534841-4 from 773353 to 1497819 tid 1 2024-09-10T16:22:58.956 INFO:tasks.rados.rados.0.smithi165.stdout:2478: writing smithi16534841-4 from 2170378 to 2773353 tid 2 2024-09-10T16:22:58.957 INFO:tasks.rados.rados.0.smithi165.stdout:2479: write (excl) oid 6 current snap is 269 2024-09-10T16:22:58.957 INFO:tasks.rados.rados.0.smithi165.stdout:2475: finishing write tid 3 to smithi16534841-12 2024-09-10T16:22:58.957 INFO:tasks.rados.rados.0.smithi165.stdout:2479: seq_num 880 ranges {790160=799715,2112507=509299} 2024-09-10T16:22:58.965 INFO:tasks.rados.rados.0.smithi165.stdout:2479: writing smithi16534841-6 from 790160 to 1589875 tid 1 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2479: writing smithi16534841-6 from 2112507 to 2621806 tid 2 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2480: write oid 31 current snap is 269 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2475: finishing write tid 4 to smithi16534841-12 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2475: oid 12 updating version 0 to 1360 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2475: oid 12 version 1360 is already newer than 1358 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2475: oid 12 version 1360 is already newer than 1359 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1360 (ObjNum 876 snap 269 seq_num 876) dirty exists 2024-09-10T16:22:58.971 INFO:tasks.rados.rados.0.smithi165.stdout:2475: left oid 12 (ObjNum 876 snap 269 seq_num 876) 2024-09-10T16:22:58.972 INFO:tasks.rados.rados.0.smithi165.stdout:2477: finishing write tid 1 to smithi16534841-46 2024-09-10T16:22:58.972 INFO:tasks.rados.rados.0.smithi165.stdout:2480: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2024-09-10T16:22:58.975 INFO:tasks.rados.rados.0.smithi165.stdout:2480: writing smithi16534841-31 from 406967 to 881931 tid 1 2024-09-10T16:22:58.982 INFO:tasks.rados.rados.0.smithi165.stdout:2480: writing smithi16534841-31 from 1370989 to 2140299 tid 2 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2480: writing smithi16534841-31 from 2677186 to 2806967 tid 3 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2475: done (4 left) 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2481: snap_create 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: finishing write tid 2 to smithi16534841-46 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: finishing write tid 3 to smithi16534841-46 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: finishing write tid 4 to smithi16534841-46 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: oid 46 updating version 0 to 1102 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: oid 46 updating version 1102 to 1104 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:2477: oid 46 version 1104 is already newer than 1103 2024-09-10T16:22:58.987 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1104 (ObjNum 878 snap 269 seq_num 878) dirty exists 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2477: left oid 46 (ObjNum 878 snap 269 seq_num 878) 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: finishing write tid 1 to smithi16534841-4 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: finishing write tid 2 to smithi16534841-4 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: finishing write tid 3 to smithi16534841-4 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: finishing write tid 4 to smithi16534841-4 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: oid 4 updating version 0 to 1321 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: oid 4 version 1321 is already newer than 1320 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: oid 4 version 1321 is already newer than 1319 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1321 (ObjNum 879 snap 269 seq_num 879) dirty exists 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2478: left oid 4 (ObjNum 879 snap 269 seq_num 879) 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2479: finishing write tid 1 to smithi16534841-6 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2479: finishing write tid 2 to smithi16534841-6 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2479: finishing write tid 3 to smithi16534841-6 2024-09-10T16:22:58.988 INFO:tasks.rados.rados.0.smithi165.stdout:2479: finishing write tid 4 to smithi16534841-6 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:2479: oid 6 updating version 0 to 1328 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:2479: oid 6 updating version 1328 to 1329 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:2479: oid 6 updating version 1329 to 1330 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1330 (ObjNum 880 snap 269 seq_num 880) dirty exists 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:2479: left oid 6 (ObjNum 880 snap 269 seq_num 880) 2024-09-10T16:22:58.989 INFO:tasks.rados.rados.0.smithi165.stdout:2480: finishing write tid 1 to smithi16534841-31 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: finishing write tid 2 to smithi16534841-31 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: finishing write tid 3 to smithi16534841-31 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: finishing write tid 4 to smithi16534841-31 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: finishing write tid 5 to smithi16534841-31 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: oid 31 updating version 0 to 1528 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: oid 31 version 1528 is already newer than 1526 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: oid 31 updating version 1528 to 1529 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: oid 31 version 1529 is already newer than 1527 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1529 (ObjNum 881 snap 269 seq_num 881) dirty exists 2024-09-10T16:22:58.990 INFO:tasks.rados.rados.0.smithi165.stdout:2480: left oid 31 (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2477: done (4 left) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2478: done (3 left) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2479: done (2 left) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2480: done (1 left) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2481: done (0 left) 2024-09-10T16:22:59.989 INFO:tasks.rados.rados.0.smithi165.stdout:2482: delete oid 36 current snap is 270 2024-09-10T16:22:59.992 INFO:tasks.rados.rados.0.smithi165.stdout:2482: done (0 left) 2024-09-10T16:22:59.992 INFO:tasks.rados.rados.0.smithi165.stdout:2483: rollback oid 7 current snap is 270 2024-09-10T16:22:59.992 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 7 to 269 2024-09-10T16:22:59.993 INFO:tasks.rados.rados.0.smithi165.stdout:2484: read oid 6 snap -1 2024-09-10T16:22:59.993 INFO:tasks.rados.rados.0.smithi165.stdout:2484: expect (ObjNum 880 snap 269 seq_num 880) 2024-09-10T16:22:59.993 INFO:tasks.rados.rados.0.smithi165.stdout:2485: delete oid 25 current snap is 270 2024-09-10T16:22:59.999 INFO:tasks.rados.rados.0.smithi165.stdout:2485: done (2 left) 2024-09-10T16:22:59.999 INFO:tasks.rados.rados.0.smithi165.stdout:2486: rollback oid 43 current snap is 270 2024-09-10T16:22:59.999 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 265 2024-09-10T16:22:59.999 INFO:tasks.rados.rados.0.smithi165.stdout:2487: delete oid 29 current snap is 270 2024-09-10T16:22:59.999 INFO:tasks.rados.rados.0.smithi165.stdout:2483: finishing rollback tid 0 to smithi16534841-7 2024-09-10T16:23:00.000 INFO:tasks.rados.rados.0.smithi165.stdout:2483: finishing rollback tid 1 to smithi16534841-7 2024-09-10T16:23:00.001 INFO:tasks.rados.rados.0.smithi165.stdout:2483: finishing rollback tid 2 to smithi16534841-7 2024-09-10T16:23:00.002 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1533 (ObjNum 743 snap 225 seq_num 743) dirty exists 2024-09-10T16:23:00.003 INFO:tasks.rados.rados.0.smithi165.stdout:2486: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:23:00.004 INFO:tasks.rados.rados.0.smithi165.stdout:2486: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:23:00.004 INFO:tasks.rados.rados.0.smithi165.stdout:2483: done (3 left) 2024-09-10T16:23:00.004 INFO:tasks.rados.rados.0.smithi165.stdout:2487: done (2 left) 2024-09-10T16:23:00.004 INFO:tasks.rados.rados.0.smithi165.stdout:2488: snap_create 2024-09-10T16:23:00.004 INFO:tasks.rados.rados.0.smithi165.stdout:2484: expect (ObjNum 880 snap 269 seq_num 880) 2024-09-10T16:23:00.050 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:00.095 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:00.141 INFO:tasks.rados.rados.0.smithi165.stdout:2486: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:23:00.141 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1360 (ObjNum 742 snap 225 seq_num 742) dirty exists 2024-09-10T16:23:00.866 INFO:tasks.rados.rados.0.smithi165.stdout:2484: done (2 left) 2024-09-10T16:23:00.866 INFO:tasks.rados.rados.0.smithi165.stdout:2486: done (1 left) 2024-09-10T16:23:00.866 INFO:tasks.rados.rados.0.smithi165.stdout:2488: done (0 left) 2024-09-10T16:23:00.866 INFO:tasks.rados.rados.0.smithi165.stdout:2489: delete oid 19 current snap is 271 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2489: done (0 left) 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2490: rollback oid 14 current snap is 271 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 266 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2491: read oid 10 snap -1 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2491: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2492: copy_from oid 12 from oid 46 current snap is 271 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2493: read oid 24 snap -1 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2493: expect deleted 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:2494: rollback oid 13 current snap is 271 2024-09-10T16:23:00.870 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 269 2024-09-10T16:23:00.871 INFO:tasks.rados.rados.0.smithi165.stdout:2495: snap_create 2024-09-10T16:23:00.876 INFO:tasks.rados.rados.0.smithi165.stdout:2491: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2490: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2490: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2490: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1215 (ObjNum 841 snap 260 seq_num 841) dirty exists 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2494: finishing rollback tid 0 to smithi16534841-13 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2494: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2494: finishing rollback tid 2 to smithi16534841-13 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1379 (ObjNum 814 snap 254 seq_num 814) dirty exists 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2492: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:2492: finishing copy_from to smithi16534841-12 2024-09-10T16:23:00.945 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1104 (ObjNum 878 snap 269 seq_num 878) dirty exists 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2490: done (5 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2491: done (4 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2492: done (3 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2493: done (2 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2494: done (1 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2495: done (0 left) 2024-09-10T16:23:01.872 INFO:tasks.rados.rados.0.smithi165.stdout:2496: delete oid 10 current snap is 272 2024-09-10T16:23:01.874 INFO:tasks.rados.rados.0.smithi165.stdout:2496: done (0 left) 2024-09-10T16:23:01.874 INFO:tasks.rados.rados.0.smithi165.stdout:2497: snap_create 2024-09-10T16:23:02.877 INFO:tasks.rados.rados.0.smithi165.stdout:2497: done (0 left) 2024-09-10T16:23:02.877 INFO:tasks.rados.rados.0.smithi165.stdout:2498: write (excl) oid 11 current snap is 273 2024-09-10T16:23:02.877 INFO:tasks.rados.rados.0.smithi165.stdout:2498: seq_num 887 ranges {507809=642811,1721881=663964} 2024-09-10T16:23:02.883 INFO:tasks.rados.rados.0.smithi165.stdout:2498: writing smithi16534841-11 from 507809 to 1150620 tid 1 2024-09-10T16:23:02.891 INFO:tasks.rados.rados.0.smithi165.stdout:2498: writing smithi16534841-11 from 1721881 to 2385845 tid 2 2024-09-10T16:23:02.891 INFO:tasks.rados.rados.0.smithi165.stdout:2499: rollback oid 1 current snap is 273 2024-09-10T16:23:02.891 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 258 2024-09-10T16:23:02.891 INFO:tasks.rados.rados.0.smithi165.stdout:2498: finishing write tid 1 to smithi16534841-11 2024-09-10T16:23:02.892 INFO:tasks.rados.rados.0.smithi165.stdout:2500: read oid 3 snap -1 2024-09-10T16:23:02.892 INFO:tasks.rados.rados.0.smithi165.stdout:2500: expect deleted 2024-09-10T16:23:02.892 INFO:tasks.rados.rados.0.smithi165.stdout:2501: read oid 30 snap 258 2024-09-10T16:23:02.892 INFO:tasks.rados.rados.0.smithi165.stdout:2501: expect (ObjNum 771 snap 237 seq_num 771) 2024-09-10T16:23:02.892 INFO:tasks.rados.rados.0.smithi165.stdout:2502: delete oid 49 current snap is 273 2024-09-10T16:23:02.895 INFO:tasks.rados.rados.0.smithi165.stdout:2498: finishing write tid 2 to smithi16534841-11 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: finishing write tid 3 to smithi16534841-11 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: finishing write tid 4 to smithi16534841-11 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: oid 11 updating version 0 to 1265 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: oid 11 version 1265 is already newer than 1264 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: oid 11 updating version 1265 to 1266 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1266 (ObjNum 887 snap 273 seq_num 887) dirty exists 2024-09-10T16:23:02.896 INFO:tasks.rados.rados.0.smithi165.stdout:2498: left oid 11 (ObjNum 887 snap 273 seq_num 887) 2024-09-10T16:23:02.897 INFO:tasks.rados.rados.0.smithi165.stdout:2498: done (4 left) 2024-09-10T16:23:02.897 INFO:tasks.rados.rados.0.smithi165.stdout:2500: done (3 left) 2024-09-10T16:23:02.897 INFO:tasks.rados.rados.0.smithi165.stdout:2502: done (2 left) 2024-09-10T16:23:02.897 INFO:tasks.rados.rados.0.smithi165.stdout:2503: delete oid 44 current snap is 273 2024-09-10T16:23:02.897 INFO:tasks.rados.rados.0.smithi165.stdout:2499: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:23:02.898 INFO:tasks.rados.rados.0.smithi165.stdout:2499: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:23:02.898 INFO:tasks.rados.rados.0.smithi165.stdout:2499: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:23:02.898 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1380 (ObjNum 572 snap 184 seq_num 572) dirty exists 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:2499: done (2 left) 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:2503: done (1 left) 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:2504: rollback oid 10 current snap is 273 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 236 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:2505: write oid 11 current snap is 273 2024-09-10T16:23:02.902 INFO:tasks.rados.rados.0.smithi165.stdout:2505: seq_num 890 ranges {558230=584911,1897327=527644,2923993=34237} 2024-09-10T16:23:02.908 INFO:tasks.rados.rados.0.smithi165.stdout:2505: writing smithi16534841-11 from 558230 to 1143141 tid 1 2024-09-10T16:23:02.915 INFO:tasks.rados.rados.0.smithi165.stdout:2505: writing smithi16534841-11 from 1897327 to 2424971 tid 2 2024-09-10T16:23:02.918 INFO:tasks.rados.rados.0.smithi165.stdout:2505: writing smithi16534841-11 from 2923993 to 2958230 tid 3 2024-09-10T16:23:02.918 INFO:tasks.rados.rados.0.smithi165.stdout:2506: write oid 7 current snap is 273 2024-09-10T16:23:02.919 INFO:tasks.rados.rados.0.smithi165.stdout:2501: expect (ObjNum 771 snap 237 seq_num 771) 2024-09-10T16:23:02.945 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:02.977 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2504: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2504: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2505: finishing write tid 1 to smithi16534841-11 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2505: finishing write tid 2 to smithi16534841-11 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2505: finishing write tid 3 to smithi16534841-11 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2505: finishing write tid 4 to smithi16534841-11 2024-09-10T16:23:03.011 INFO:tasks.rados.rados.0.smithi165.stdout:2505: finishing write tid 5 to smithi16534841-11 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2505: oid 11 updating version 0 to 1270 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2505: oid 11 version 1270 is already newer than 1269 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2505: oid 11 version 1270 is already newer than 1268 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2505: oid 11 version 1270 is already newer than 1267 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1270 (ObjNum 890 snap 273 seq_num 890) dirty exists 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2505: left oid 11 (ObjNum 890 snap 273 seq_num 890) 2024-09-10T16:23:03.012 INFO:tasks.rados.rados.0.smithi165.stdout:2506: seq_num 891 ranges {575037=660160,1955809=671302} 2024-09-10T16:23:03.017 INFO:tasks.rados.rados.0.smithi165.stdout:2506: writing smithi16534841-7 from 575037 to 1235197 tid 1 2024-09-10T16:23:03.025 INFO:tasks.rados.rados.0.smithi165.stdout:2506: writing smithi16534841-7 from 1955809 to 2627111 tid 2 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2501: done (3 left) 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2504: done (2 left) 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2505: done (1 left) 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2507: read oid 3 snap -1 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2507: expect deleted 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2508: rollback oid 8 current snap is 273 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 264 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2506: finishing write tid 1 to smithi16534841-7 2024-09-10T16:23:03.026 INFO:tasks.rados.rados.0.smithi165.stdout:2509: snap_create 2024-09-10T16:23:03.029 INFO:tasks.rados.rados.0.smithi165.stdout:2508: finishing rollback tid 0 to smithi16534841-8 2024-09-10T16:23:03.030 INFO:tasks.rados.rados.0.smithi165.stdout:2508: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2508: finishing rollback tid 2 to smithi16534841-8 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1443 (ObjNum 767 snap 235 seq_num 767) dirty exists 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: finishing write tid 2 to smithi16534841-7 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: finishing write tid 3 to smithi16534841-7 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: finishing write tid 4 to smithi16534841-7 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: oid 7 updating version 0 to 1536 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: oid 7 updating version 1536 to 1538 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: oid 7 version 1538 is already newer than 1537 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1538 (ObjNum 891 snap 273 seq_num 891) dirty exists 2024-09-10T16:23:03.031 INFO:tasks.rados.rados.0.smithi165.stdout:2506: left oid 7 (ObjNum 891 snap 273 seq_num 891) 2024-09-10T16:23:03.999 INFO:tasks.rados.rados.0.smithi165.stdout:2506: done (3 left) 2024-09-10T16:23:04.000 INFO:tasks.rados.rados.0.smithi165.stdout:2507: done (2 left) 2024-09-10T16:23:04.000 INFO:tasks.rados.rados.0.smithi165.stdout:2508: done (1 left) 2024-09-10T16:23:04.000 INFO:tasks.rados.rados.0.smithi165.stdout:2509: done (0 left) 2024-09-10T16:23:04.000 INFO:tasks.rados.rados.0.smithi165.stdout:2510: write oid 30 current snap is 274 2024-09-10T16:23:04.000 INFO:tasks.rados.rados.0.smithi165.stdout:2510: seq_num 892 ranges {591844=735409,1897938=531313,2862486=129358} 2024-09-10T16:23:04.006 INFO:tasks.rados.rados.0.smithi165.stdout:2510: writing smithi16534841-30 from 591844 to 1327253 tid 1 2024-09-10T16:23:04.012 INFO:tasks.rados.rados.0.smithi165.stdout:2510: writing smithi16534841-30 from 1897938 to 2429251 tid 2 2024-09-10T16:23:04.016 INFO:tasks.rados.rados.0.smithi165.stdout:2510: writing smithi16534841-30 from 2862486 to 2991844 tid 3 2024-09-10T16:23:04.016 INFO:tasks.rados.rados.0.smithi165.stdout:2511: copy_from oid 5 from oid 12 current snap is 274 2024-09-10T16:23:04.016 INFO:tasks.rados.rados.0.smithi165.stdout:2512: snap_remove snap 273 2024-09-10T16:23:05.004 INFO:tasks.rados.rados.0.smithi165.stdout:2512: done (2 left) 2024-09-10T16:23:05.004 INFO:tasks.rados.rados.0.smithi165.stdout:2513: snap_remove snap 262 2024-09-10T16:23:05.867 INFO:tasks.rados.rados.0.smithi165.stdout:2513: done (2 left) 2024-09-10T16:23:05.868 INFO:tasks.rados.rados.0.smithi165.stdout:2514: read oid 13 snap -1 2024-09-10T16:23:05.868 INFO:tasks.rados.rados.0.smithi165.stdout:2514: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:23:05.868 INFO:tasks.rados.rados.0.smithi165.stdout:2515: rollback oid 29 current snap is 274 2024-09-10T16:23:05.868 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 255 2024-09-10T16:23:05.868 INFO:tasks.rados.rados.0.smithi165.stdout:2516: snap_remove snap 236 2024-09-10T16:23:06.872 INFO:tasks.rados.rados.0.smithi165.stdout:2516: done (4 left) 2024-09-10T16:23:06.872 INFO:tasks.rados.rados.0.smithi165.stdout:2517: rollback oid 46 current snap is 274 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 258 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: finishing write tid 1 to smithi16534841-30 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: finishing write tid 2 to smithi16534841-30 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: finishing write tid 3 to smithi16534841-30 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: finishing write tid 4 to smithi16534841-30 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: finishing write tid 5 to smithi16534841-30 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: oid 30 updating version 0 to 1542 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: oid 30 version 1542 is already newer than 1541 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: oid 30 version 1542 is already newer than 1540 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: oid 30 updating version 1542 to 1543 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1543 (ObjNum 892 snap 274 seq_num 892) dirty exists 2024-09-10T16:23:06.873 INFO:tasks.rados.rados.0.smithi165.stdout:2510: left oid 30 (ObjNum 892 snap 274 seq_num 892) 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:2511: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:2511: finishing copy_from to smithi16534841-5 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1104 (ObjNum 878 snap 269 seq_num 878) dirty exists 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:2515: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1348 (ObjNum 129 snap 0 seq_num 1162448112) dirty dne 2024-09-10T16:23:06.874 INFO:tasks.rados.rados.0.smithi165.stdout:2514: expect (ObjNum 814 snap 254 seq_num 814) 2024-09-10T16:23:06.986 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3346275 2024-09-10T16:23:07.054 INFO:tasks.rados.rados.0.smithi165.stdout:2517: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2517: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2517: finishing rollback tid 2 to smithi16534841-46 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1108 (ObjNum 820 snap 255 seq_num 820) dirty exists 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2510: done (4 left) 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2511: done (3 left) 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2514: done (2 left) 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2515: done (1 left) 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2517: done (0 left) 2024-09-10T16:23:07.055 INFO:tasks.rados.rados.0.smithi165.stdout:2518: delete oid 48 current snap is 274 2024-09-10T16:23:07.057 INFO:tasks.rados.rados.0.smithi165.stdout:2518: done (0 left) 2024-09-10T16:23:07.057 INFO:tasks.rados.rados.0.smithi165.stdout:2519: delete oid 21 current snap is 274 2024-09-10T16:23:07.060 INFO:tasks.rados.rados.0.smithi165.stdout:2519: done (0 left) 2024-09-10T16:23:07.061 INFO:tasks.rados.rados.0.smithi165.stdout:2520: delete oid 34 current snap is 274 2024-09-10T16:23:07.064 INFO:tasks.rados.rados.0.smithi165.stdout:2520: done (0 left) 2024-09-10T16:23:07.064 INFO:tasks.rados.rados.0.smithi165.stdout:2521: snap_create 2024-09-10T16:23:08.019 INFO:tasks.rados.rados.0.smithi165.stdout:2521: done (0 left) 2024-09-10T16:23:08.019 INFO:tasks.rados.rados.0.smithi165.stdout:2522: snap_create 2024-09-10T16:23:09.679 INFO:tasks.rados.rados.0.smithi165.stdout:2522: done (0 left) 2024-09-10T16:23:09.679 INFO:tasks.rados.rados.0.smithi165.stdout:2523: write oid 38 current snap is 276 2024-09-10T16:23:09.679 INFO:tasks.rados.rados.0.smithi165.stdout:2523: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2024-09-10T16:23:09.685 INFO:tasks.rados.rados.0.smithi165.stdout:2523: writing smithi16534841-38 from 659072 to 1295477 tid 1 2024-09-10T16:23:09.690 INFO:tasks.rados.rados.0.smithi165.stdout:2523: writing smithi16534841-38 from 1731866 to 2154164 tid 2 2024-09-10T16:23:09.695 INFO:tasks.rados.rados.0.smithi165.stdout:2523: writing smithi16534841-38 from 2855825 to 3059072 tid 3 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2524: rollback oid 44 current snap is 276 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 266 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2523: finishing write tid 1 to smithi16534841-38 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2523: finishing write tid 2 to smithi16534841-38 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2525: rollback oid 2 current snap is 276 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 261 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2526: rollback oid 49 current snap is 276 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 272 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2527: read oid 42 snap -1 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2527: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:23:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2528: write oid 16 current snap is 276 2024-09-10T16:23:09.697 INFO:tasks.rados.rados.0.smithi165.stdout:2528: seq_num 897 ranges {675879=711654,2073995=682309} 2024-09-10T16:23:09.703 INFO:tasks.rados.rados.0.smithi165.stdout:2528: writing smithi16534841-16 from 675879 to 1387533 tid 1 2024-09-10T16:23:09.711 INFO:tasks.rados.rados.0.smithi165.stdout:2528: writing smithi16534841-16 from 2073995 to 2756304 tid 2 2024-09-10T16:23:09.711 INFO:tasks.rados.rados.0.smithi165.stdout:2529: write (excl) oid 22 current snap is 276 2024-09-10T16:23:09.711 INFO:tasks.rados.rados.0.smithi165.stdout:2525: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:23:09.711 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:23:09.711 INFO:tasks.rados.rados.0.smithi165.stdout:2529: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2024-09-10T16:23:09.717 INFO:tasks.rados.rados.0.smithi165.stdout:2529: writing smithi16534841-22 from 692686 to 1195942 tid 1 2024-09-10T16:23:09.723 INFO:tasks.rados.rados.0.smithi165.stdout:2529: writing smithi16534841-22 from 1848830 to 2274797 tid 2 2024-09-10T16:23:09.729 INFO:tasks.rados.rados.0.smithi165.stdout:2529: writing smithi16534841-22 from 2910671 to 3092686 tid 3 2024-09-10T16:23:09.729 INFO:tasks.rados.rados.0.smithi165.stdout:2525: done (6 left) 2024-09-10T16:23:09.729 INFO:tasks.rados.rados.0.smithi165.stdout:2530: delete oid 15 current snap is 276 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2524: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: finishing write tid 3 to smithi16534841-38 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2526: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1146 (ObjNum 513 snap 0 seq_num 1162085032) dirty dne 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: finishing write tid 4 to smithi16534841-38 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: finishing write tid 5 to smithi16534841-38 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: oid 38 updating version 0 to 1374 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: oid 38 version 1374 is already newer than 1372 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: oid 38 version 1374 is already newer than 1373 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: oid 38 version 1374 is already newer than 1371 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1374 (ObjNum 896 snap 276 seq_num 896) dirty exists 2024-09-10T16:23:09.730 INFO:tasks.rados.rados.0.smithi165.stdout:2523: left oid 38 (ObjNum 896 snap 276 seq_num 896) 2024-09-10T16:23:09.731 INFO:tasks.rados.rados.0.smithi165.stdout:2524: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:23:09.731 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1110 (ObjNum 843 snap 260 seq_num 843) dirty exists 2024-09-10T16:23:09.731 INFO:tasks.rados.rados.0.smithi165.stdout:2527: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:23:09.731 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:09.760 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:09.791 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2528: finishing write tid 1 to smithi16534841-16 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2523: done (6 left) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2524: done (5 left) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2526: done (4 left) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2527: done (3 left) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2530: done (2 left) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2531: read oid 42 snap 269 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2531: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:23:09.823 INFO:tasks.rados.rados.0.smithi165.stdout:2532: snap_create 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: finishing write tid 2 to smithi16534841-16 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: finishing write tid 3 to smithi16534841-16 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: finishing write tid 4 to smithi16534841-16 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: oid 16 updating version 0 to 1563 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: oid 16 version 1563 is already newer than 1562 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: oid 16 updating version 1563 to 1564 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1564 (ObjNum 897 snap 276 seq_num 897) dirty exists 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2528: left oid 16 (ObjNum 897 snap 276 seq_num 897) 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2529: finishing write tid 1 to smithi16534841-22 2024-09-10T16:23:09.824 INFO:tasks.rados.rados.0.smithi165.stdout:2529: finishing write tid 2 to smithi16534841-22 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: finishing write tid 3 to smithi16534841-22 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: finishing write tid 4 to smithi16534841-22 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: finishing write tid 5 to smithi16534841-22 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: oid 22 updating version 0 to 1380 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: oid 22 updating version 1380 to 1381 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: oid 22 updating version 1381 to 1382 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: oid 22 updating version 1382 to 1383 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1383 (ObjNum 898 snap 276 seq_num 898) dirty exists 2024-09-10T16:23:09.825 INFO:tasks.rados.rados.0.smithi165.stdout:2529: left oid 22 (ObjNum 898 snap 276 seq_num 898) 2024-09-10T16:23:09.826 INFO:tasks.rados.rados.0.smithi165.stdout:2531: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:23:09.845 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:10.026 INFO:tasks.rados.rados.0.smithi165.stdout:2528: done (3 left) 2024-09-10T16:23:10.026 INFO:tasks.rados.rados.0.smithi165.stdout:2529: done (2 left) 2024-09-10T16:23:10.026 INFO:tasks.rados.rados.0.smithi165.stdout:2531: done (1 left) 2024-09-10T16:23:10.026 INFO:tasks.rados.rados.0.smithi165.stdout:2532: done (0 left) 2024-09-10T16:23:10.026 INFO:tasks.rados.rados.0.smithi165.stdout:2533: delete oid 40 current snap is 277 2024-09-10T16:23:10.030 INFO:tasks.rados.rados.0.smithi165.stdout:2533: done (0 left) 2024-09-10T16:23:10.030 INFO:tasks.rados.rados.0.smithi165.stdout:2534: rollback oid 4 current snap is 277 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 269 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2535: copy_from oid 26 from oid 17 current snap is 277 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2536: read oid 14 snap -1 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2536: expect (ObjNum 841 snap 260 seq_num 841) 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2537: read oid 45 snap -1 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2537: expect (ObjNum 731 snap 223 seq_num 731) 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2538: copy_from oid 22 from oid 36 current snap is 277 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2539: snap_create 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2538: finishing copy_from to smithi16534841-22 2024-09-10T16:23:10.031 INFO:tasks.rados.rados.0.smithi165.stdout:2538: got expected ENOENT (src dne) 2024-09-10T16:23:10.032 INFO:tasks.rados.rados.0.smithi165.stdout:2538: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:23:10.036 INFO:tasks.rados.rados.0.smithi165.stdout:2537: expect (ObjNum 731 snap 223 seq_num 731) 2024-09-10T16:23:10.123 INFO:tasks.rados.rados.0.smithi165.stdout:2536: expect (ObjNum 841 snap 260 seq_num 841) 2024-09-10T16:23:10.124 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:10.278 INFO:tasks.rados.rados.0.smithi165.stdout:2534: finishing rollback tid 0 to smithi16534841-4 2024-09-10T16:23:10.278 INFO:tasks.rados.rados.0.smithi165.stdout:2534: finishing rollback tid 1 to smithi16534841-4 2024-09-10T16:23:10.278 INFO:tasks.rados.rados.0.smithi165.stdout:2534: finishing rollback tid 2 to smithi16534841-4 2024-09-10T16:23:10.279 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1325 (ObjNum 879 snap 269 seq_num 879) dirty exists 2024-09-10T16:23:10.279 INFO:tasks.rados.rados.0.smithi165.stdout:2535: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:23:10.279 INFO:tasks.rados.rados.0.smithi165.stdout:2535: finishing copy_from to smithi16534841-26 2024-09-10T16:23:10.279 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1149 (ObjNum 856 snap 263 seq_num 856) dirty exists 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2534: done (5 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2535: done (4 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2536: done (3 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2537: done (2 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2538: done (1 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2539: done (0 left) 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2540: write (excl) oid 48 current snap is 278 2024-09-10T16:23:10.871 INFO:tasks.rados.rados.0.smithi165.stdout:2540: seq_num 901 ranges {743107=729003,1907923=689647} 2024-09-10T16:23:10.878 INFO:tasks.rados.rados.0.smithi165.stdout:2540: writing smithi16534841-48 from 743107 to 1472110 tid 1 2024-09-10T16:23:10.885 INFO:tasks.rados.rados.0.smithi165.stdout:2540: writing smithi16534841-48 from 1907923 to 2597570 tid 2 2024-09-10T16:23:10.885 INFO:tasks.rados.rados.0.smithi165.stdout:2541: rollback oid 36 current snap is 278 2024-09-10T16:23:10.885 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 36 to 265 2024-09-10T16:23:10.885 INFO:tasks.rados.rados.0.smithi165.stdout:2542: snap_remove snap 230 2024-09-10T16:23:11.878 INFO:tasks.rados.rados.0.smithi165.stdout:2542: done (2 left) 2024-09-10T16:23:11.878 INFO:tasks.rados.rados.0.smithi165.stdout:2543: snap_remove snap 255 2024-09-10T16:23:12.890 INFO:tasks.rados.rados.0.smithi165.stdout:2543: done (2 left) 2024-09-10T16:23:12.890 INFO:tasks.rados.rados.0.smithi165.stdout:2544: delete oid 22 current snap is 278 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2541: finishing rollback tid 1 to smithi16534841-36 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2541: finishing rollback tid 2 to smithi16534841-36 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1376 (ObjNum 631 snap 197 seq_num 631) dirty exists 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: finishing write tid 1 to smithi16534841-48 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: finishing write tid 2 to smithi16534841-48 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: finishing write tid 3 to smithi16534841-48 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: finishing write tid 4 to smithi16534841-48 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: oid 48 updating version 0 to 1567 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: oid 48 updating version 1567 to 1568 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: oid 48 version 1568 is already newer than 1566 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1568 (ObjNum 901 snap 278 seq_num 901) dirty exists 2024-09-10T16:23:12.891 INFO:tasks.rados.rados.0.smithi165.stdout:2540: left oid 48 (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2540: done (2 left) 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2541: done (1 left) 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2544: done (0 left) 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2545: read oid 22 snap 270 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2545: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2546: copy_from oid 28 from oid 50 current snap is 278 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2547: copy_from oid 26 from oid 19 current snap is 278 2024-09-10T16:23:12.897 INFO:tasks.rados.rados.0.smithi165.stdout:2548: delete oid 6 current snap is 278 2024-09-10T16:23:12.901 INFO:tasks.rados.rados.0.smithi165.stdout:2547: finishing copy_from to smithi16534841-26 2024-09-10T16:23:12.901 INFO:tasks.rados.rados.0.smithi165.stdout:2547: got expected ENOENT (src dne) 2024-09-10T16:23:12.901 INFO:tasks.rados.rados.0.smithi165.stdout:2547: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:23:12.902 INFO:tasks.rados.rados.0.smithi165.stdout:2547: done (3 left) 2024-09-10T16:23:12.902 INFO:tasks.rados.rados.0.smithi165.stdout:2548: done (2 left) 2024-09-10T16:23:12.902 INFO:tasks.rados.rados.0.smithi165.stdout:2549: write (excl) oid 26 current snap is 278 2024-09-10T16:23:12.902 INFO:tasks.rados.rados.0.smithi165.stdout:2549: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2024-09-10T16:23:12.909 INFO:tasks.rados.rados.0.smithi165.stdout:2549: writing smithi16534841-26 from 793528 to 1348278 tid 1 2024-09-10T16:23:12.915 INFO:tasks.rados.rados.0.smithi165.stdout:2549: writing smithi16534841-26 from 2083369 to 2520343 tid 2 2024-09-10T16:23:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:2549: writing smithi16534841-26 from 2958856 to 3193528 tid 3 2024-09-10T16:23:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:2550: snap_remove snap 277 2024-09-10T16:23:13.895 INFO:tasks.rados.rados.0.smithi165.stdout:2550: done (3 left) 2024-09-10T16:23:13.895 INFO:tasks.rados.rados.0.smithi165.stdout:2551: write oid 13 current snap is 278 2024-09-10T16:23:13.895 INFO:tasks.rados.rados.0.smithi165.stdout:2551: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2024-09-10T16:23:13.901 INFO:tasks.rados.rados.0.smithi165.stdout:2551: writing smithi16534841-13 from 410335 to 1156687 tid 1 2024-09-10T16:23:13.908 INFO:tasks.rados.rados.0.smithi165.stdout:2551: writing smithi16534841-13 from 1741851 to 2438836 tid 2 2024-09-10T16:23:13.912 INFO:tasks.rados.rados.0.smithi165.stdout:2551: writing smithi16534841-13 from 3102632 to 3210335 tid 3 2024-09-10T16:23:13.912 INFO:tasks.rados.rados.0.smithi165.stdout:2552: copy_from oid 46 from oid 10 current snap is 278 2024-09-10T16:23:13.912 INFO:tasks.rados.rados.0.smithi165.stdout:2545: expect (ObjNum 844 snap 260 seq_num 844) 2024-09-10T16:23:13.912 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:13.954 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:14.000 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1503759 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2546: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2546: finishing copy_from to smithi16534841-28 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1260 (ObjNum 871 snap 266 seq_num 871) dirty exists 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: finishing write tid 1 to smithi16534841-26 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: finishing write tid 2 to smithi16534841-26 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: finishing write tid 3 to smithi16534841-26 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: finishing write tid 4 to smithi16534841-26 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: finishing write tid 5 to smithi16534841-26 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: oid 26 updating version 0 to 1597 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: oid 26 updating version 1597 to 1599 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: oid 26 version 1599 is already newer than 1596 2024-09-10T16:23:14.022 INFO:tasks.rados.rados.0.smithi165.stdout:2549: oid 26 version 1599 is already newer than 1598 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1599 (ObjNum 904 snap 278 seq_num 904) dirty exists 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2549: left oid 26 (ObjNum 904 snap 278 seq_num 904) 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2545: done (4 left) 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2546: done (3 left) 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2549: done (2 left) 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2553: write (excl) oid 33 current snap is 278 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: finishing write tid 1 to smithi16534841-13 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: finishing write tid 2 to smithi16534841-13 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: finishing write tid 3 to smithi16534841-13 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: finishing write tid 4 to smithi16534841-13 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: finishing write tid 5 to smithi16534841-13 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: oid 13 updating version 0 to 1386 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: oid 13 updating version 1386 to 1387 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: oid 13 updating version 1387 to 1388 2024-09-10T16:23:14.023 INFO:tasks.rados.rados.0.smithi165.stdout:2551: oid 13 version 1388 is already newer than 1385 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1388 (ObjNum 905 snap 278 seq_num 905) dirty exists 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:2551: left oid 13 (ObjNum 905 snap 278 seq_num 905) 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:2552: finishing copy_from to smithi16534841-46 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:2552: finishing copy_from racing read to smithi16534841-46 2024-09-10T16:23:14.024 INFO:tasks.rados.rados.0.smithi165.stdout:2553: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2024-09-10T16:23:14.025 INFO:tasks.rados.rados.0.smithi165.stdout:2553: writing smithi16534841-33 from 427142 to 848743 tid 1 2024-09-10T16:23:14.031 INFO:tasks.rados.rados.0.smithi165.stdout:2553: writing smithi16534841-33 from 1400333 to 1840976 tid 2 2024-09-10T16:23:14.038 INFO:tasks.rados.rados.0.smithi165.stdout:2553: writing smithi16534841-33 from 2613702 to 3064028 tid 3 2024-09-10T16:23:14.038 INFO:tasks.rados.rados.0.smithi165.stdout:2551: done (2 left) 2024-09-10T16:23:14.038 INFO:tasks.rados.rados.0.smithi165.stdout:2552: done (1 left) 2024-09-10T16:23:14.038 INFO:tasks.rados.rados.0.smithi165.stdout:2554: snap_remove snap 276 2024-09-10T16:23:14.899 INFO:tasks.rados.rados.0.smithi165.stdout:2554: done (1 left) 2024-09-10T16:23:14.899 INFO:tasks.rados.rados.0.smithi165.stdout:2555: delete oid 32 current snap is 278 2024-09-10T16:23:14.899 INFO:tasks.rados.rados.0.smithi165.stdout:2553: finishing write tid 1 to smithi16534841-33 2024-09-10T16:23:14.899 INFO:tasks.rados.rados.0.smithi165.stdout:2553: finishing write tid 2 to smithi16534841-33 2024-09-10T16:23:14.899 INFO:tasks.rados.rados.0.smithi165.stdout:2553: finishing write tid 3 to smithi16534841-33 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: finishing write tid 4 to smithi16534841-33 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: finishing write tid 5 to smithi16534841-33 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: oid 33 updating version 0 to 1442 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: oid 33 version 1442 is already newer than 1440 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: oid 33 version 1442 is already newer than 1441 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: oid 33 version 1442 is already newer than 1439 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1442 (ObjNum 906 snap 278 seq_num 906) dirty exists 2024-09-10T16:23:14.900 INFO:tasks.rados.rados.0.smithi165.stdout:2553: left oid 33 (ObjNum 906 snap 278 seq_num 906) 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2553: done (1 left) 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2555: done (0 left) 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2556: rollback oid 18 current snap is 278 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 270 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2557: rollback oid 40 current snap is 278 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 274 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2558: read oid 2 snap 261 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2558: expect deleted 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2559: write oid 47 current snap is 278 2024-09-10T16:23:14.906 INFO:tasks.rados.rados.0.smithi165.stdout:2559: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2024-09-10T16:23:14.912 INFO:tasks.rados.rados.0.smithi165.stdout:2559: writing smithi16534841-47 from 460756 to 1032855 tid 1 2024-09-10T16:23:14.917 INFO:tasks.rados.rados.0.smithi165.stdout:2559: writing smithi16534841-47 from 1517297 to 1961609 tid 2 2024-09-10T16:23:14.924 INFO:tasks.rados.rados.0.smithi165.stdout:2559: writing smithi16534841-47 from 2668548 to 3141418 tid 3 2024-09-10T16:23:14.924 INFO:tasks.rados.rados.0.smithi165.stdout:2560: snap_remove snap 275 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:2560: done (4 left) 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:2561: rollback oid 50 current snap is 278 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 274 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:2556: finishing rollback tid 0 to smithi16534841-18 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:2556: finishing rollback tid 1 to smithi16534841-18 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:2556: finishing rollback tid 2 to smithi16534841-18 2024-09-10T16:23:15.874 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1383 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2557: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2557: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1218 (ObjNum 748 snap 227 seq_num 748) dirty exists 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: finishing write tid 1 to smithi16534841-47 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: finishing write tid 2 to smithi16534841-47 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: finishing write tid 3 to smithi16534841-47 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: finishing write tid 4 to smithi16534841-47 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: finishing write tid 5 to smithi16534841-47 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: oid 47 updating version 0 to 1454 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: oid 47 version 1454 is already newer than 1452 2024-09-10T16:23:15.875 INFO:tasks.rados.rados.0.smithi165.stdout:2559: oid 47 version 1454 is already newer than 1451 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2559: oid 47 version 1454 is already newer than 1453 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1454 (ObjNum 908 snap 278 seq_num 908) dirty exists 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2559: left oid 47 (ObjNum 908 snap 278 seq_num 908) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2556: done (4 left) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2557: done (3 left) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2558: done (2 left) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2559: done (1 left) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2562: read oid 1 snap 265 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2562: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2563: write (excl) oid 7 current snap is 278 2024-09-10T16:23:15.876 INFO:tasks.rados.rados.0.smithi165.stdout:2563: seq_num 909 ranges {477563=647348,1859426=587970,2979618=297945} 2024-09-10T16:23:15.880 INFO:tasks.rados.rados.0.smithi165.stdout:2563: writing smithi16534841-7 from 477563 to 1124911 tid 1 2024-09-10T16:23:15.887 INFO:tasks.rados.rados.0.smithi165.stdout:2563: writing smithi16534841-7 from 1859426 to 2447396 tid 2 2024-09-10T16:23:15.892 INFO:tasks.rados.rados.0.smithi165.stdout:2563: writing smithi16534841-7 from 2979618 to 3277563 tid 3 2024-09-10T16:23:15.892 INFO:tasks.rados.rados.0.smithi165.stdout:2564: write oid 22 current snap is 278 2024-09-10T16:23:15.892 INFO:tasks.rados.rados.0.smithi165.stdout:2564: seq_num 910 ranges {494370=722597,1917908=447981,3007041=287329} 2024-09-10T16:23:15.899 INFO:tasks.rados.rados.0.smithi165.stdout:2564: writing smithi16534841-22 from 494370 to 1216967 tid 1 2024-09-10T16:23:15.905 INFO:tasks.rados.rados.0.smithi165.stdout:2564: writing smithi16534841-22 from 1917908 to 2365889 tid 2 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2564: writing smithi16534841-22 from 3007041 to 3294370 tid 3 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2565: write oid 23 current snap is 278 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2561: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2561: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2561: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1273 (ObjNum 871 snap 266 seq_num 871) dirty exists 2024-09-10T16:23:15.911 INFO:tasks.rados.rados.0.smithi165.stdout:2562: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:15.932 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2489839 2024-09-10T16:23:15.971 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:16.098 INFO:tasks.rados.rados.0.smithi165.stdout:2563: finishing write tid 1 to smithi16534841-7 2024-09-10T16:23:16.098 INFO:tasks.rados.rados.0.smithi165.stdout:2565: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2024-09-10T16:23:16.104 INFO:tasks.rados.rados.0.smithi165.stdout:2565: writing smithi16534841-23 from 511177 to 1309023 tid 1 2024-09-10T16:23:16.111 INFO:tasks.rados.rados.0.smithi165.stdout:2565: writing smithi16534841-23 from 1976390 to 2568029 tid 2 2024-09-10T16:23:16.116 INFO:tasks.rados.rados.0.smithi165.stdout:2565: writing smithi16534841-23 from 3034464 to 3311177 tid 3 2024-09-10T16:23:16.116 INFO:tasks.rados.rados.0.smithi165.stdout:2561: done (4 left) 2024-09-10T16:23:16.116 INFO:tasks.rados.rados.0.smithi165.stdout:2562: done (3 left) 2024-09-10T16:23:16.116 INFO:tasks.rados.rados.0.smithi165.stdout:2566: read oid 27 snap -1 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2566: expect (ObjNum 874 snap 268 seq_num 874) 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2567: read oid 20 snap -1 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2567: expect deleted 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2568: read oid 25 snap -1 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2568: expect deleted 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2569: read oid 13 snap -1 2024-09-10T16:23:16.117 INFO:tasks.rados.rados.0.smithi165.stdout:2569: expect (ObjNum 905 snap 278 seq_num 905) 2024-09-10T16:23:16.118 INFO:tasks.rados.rados.0.smithi165.stdout:2570: rollback oid 9 current snap is 278 2024-09-10T16:23:16.118 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 263 2024-09-10T16:23:16.118 INFO:tasks.rados.rados.0.smithi165.stdout:2563: finishing write tid 2 to smithi16534841-7 2024-09-10T16:23:16.118 INFO:tasks.rados.rados.0.smithi165.stdout:2563: finishing write tid 3 to smithi16534841-7 2024-09-10T16:23:16.118 INFO:tasks.rados.rados.0.smithi165.stdout:2563: finishing write tid 4 to smithi16534841-7 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: finishing write tid 5 to smithi16534841-7 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: oid 7 updating version 0 to 1604 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: oid 7 updating version 1604 to 1606 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: oid 7 version 1606 is already newer than 1605 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: oid 7 version 1606 is already newer than 1603 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1606 (ObjNum 909 snap 278 seq_num 909) dirty exists 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2563: left oid 7 (ObjNum 909 snap 278 seq_num 909) 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: finishing write tid 1 to smithi16534841-22 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: finishing write tid 2 to smithi16534841-22 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: finishing write tid 3 to smithi16534841-22 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: finishing write tid 4 to smithi16534841-22 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: finishing write tid 5 to smithi16534841-22 2024-09-10T16:23:16.119 INFO:tasks.rados.rados.0.smithi165.stdout:2564: oid 22 updating version 0 to 1392 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2564: oid 22 version 1392 is already newer than 1391 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2564: oid 22 version 1392 is already newer than 1390 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2564: oid 22 version 1392 is already newer than 1389 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1392 (ObjNum 910 snap 278 seq_num 910) dirty exists 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2564: left oid 22 (ObjNum 910 snap 278 seq_num 910) 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2565: finishing write tid 1 to smithi16534841-23 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2565: finishing write tid 2 to smithi16534841-23 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2563: done (7 left) 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2564: done (6 left) 2024-09-10T16:23:16.120 INFO:tasks.rados.rados.0.smithi165.stdout:2571: snap_create 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: finishing write tid 3 to smithi16534841-23 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: finishing write tid 4 to smithi16534841-23 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: finishing write tid 5 to smithi16534841-23 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: oid 23 updating version 0 to 1386 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: oid 23 version 1386 is already newer than 1385 2024-09-10T16:23:16.121 INFO:tasks.rados.rados.0.smithi165.stdout:2565: oid 23 updating version 1386 to 1387 2024-09-10T16:23:16.122 INFO:tasks.rados.rados.0.smithi165.stdout:2565: oid 23 version 1387 is already newer than 1384 2024-09-10T16:23:16.122 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1387 (ObjNum 911 snap 278 seq_num 911) dirty exists 2024-09-10T16:23:16.122 INFO:tasks.rados.rados.0.smithi165.stdout:2565: left oid 23 (ObjNum 911 snap 278 seq_num 911) 2024-09-10T16:23:16.123 INFO:tasks.rados.rados.0.smithi165.stdout:2570: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:23:16.123 INFO:tasks.rados.rados.0.smithi165.stdout:2570: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:23:16.125 INFO:tasks.rados.rados.0.smithi165.stdout:2570: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:23:16.125 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1221 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-10T16:23:16.125 INFO:tasks.rados.rados.0.smithi165.stdout:2566: expect (ObjNum 874 snap 268 seq_num 874) 2024-09-10T16:23:16.126 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 127402 2024-09-10T16:23:16.228 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:16.283 INFO:tasks.rados.rados.0.smithi165.stdout:2569: expect (ObjNum 905 snap 278 seq_num 905) 2024-09-10T16:23:16.284 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:16.418 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3154279 2024-09-10T16:23:16.453 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:23:16.453 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:23:16.453 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:23:16.457 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:23:16.457 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:23:16.457 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:23:16.457 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:23:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:2565: done (6 left) 2024-09-10T16:23:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:2566: done (5 left) 2024-09-10T16:23:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:2567: done (4 left) 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2568: done (3 left) 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2569: done (2 left) 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2570: done (1 left) 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2571: done (0 left) 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2572: rollback oid 22 current snap is 279 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 272 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2573: copy_from oid 49 from oid 30 current snap is 279 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2574: write oid 16 current snap is 279 2024-09-10T16:23:21.841 INFO:tasks.rados.rados.0.smithi165.stdout:2574: seq_num 912 ranges {527984=473095,1634872=451650,2661887=634311} 2024-09-10T16:23:21.845 INFO:tasks.rados.rados.0.smithi165.stdout:2574: writing smithi16534841-16 from 527984 to 1001079 tid 1 2024-09-10T16:23:21.850 INFO:tasks.rados.rados.0.smithi165.stdout:2574: writing smithi16534841-16 from 1634872 to 2086522 tid 2 2024-09-10T16:23:21.857 INFO:tasks.rados.rados.0.smithi165.stdout:2574: writing smithi16534841-16 from 2661887 to 3296198 tid 3 2024-09-10T16:23:21.857 INFO:tasks.rados.rados.0.smithi165.stdout:2575: snap_remove snap 264 2024-09-10T16:23:22.839 INFO:tasks.rados.rados.0.smithi165.stdout:2575: done (3 left) 2024-09-10T16:23:22.840 INFO:tasks.rados.rados.0.smithi165.stdout:2576: read oid 25 snap 272 2024-09-10T16:23:22.840 INFO:tasks.rados.rados.0.smithi165.stdout:2576: expect deleted 2024-09-10T16:23:22.840 INFO:tasks.rados.rados.0.smithi165.stdout:2577: snap_create 2024-09-10T16:23:22.852 INFO:tasks.rados.rados.0.smithi165.stdout:2572: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:23:22.852 INFO:tasks.rados.rados.0.smithi165.stdout:2572: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:23:22.852 INFO:tasks.rados.rados.0.smithi165.stdout:2572: finishing rollback tid 2 to smithi16534841-22 2024-09-10T16:23:22.852 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1395 (ObjNum 844 snap 260 seq_num 844) dirty exists 2024-09-10T16:23:22.857 INFO:tasks.rados.rados.0.smithi165.stdout:2574: finishing write tid 1 to smithi16534841-16 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: finishing write tid 2 to smithi16534841-16 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: finishing write tid 3 to smithi16534841-16 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: finishing write tid 4 to smithi16534841-16 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: finishing write tid 5 to smithi16534841-16 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: oid 16 updating version 0 to 1609 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: oid 16 updating version 1609 to 1611 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: oid 16 version 1611 is already newer than 1610 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:2574: oid 16 version 1611 is already newer than 1608 2024-09-10T16:23:22.866 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1611 (ObjNum 912 snap 279 seq_num 912) dirty exists 2024-09-10T16:23:22.867 INFO:tasks.rados.rados.0.smithi165.stdout:2574: left oid 16 (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:22.872 INFO:tasks.rados.rados.0.smithi165.stdout:2573: finishing copy_from racing read to smithi16534841-49 2024-09-10T16:23:22.874 INFO:tasks.rados.rados.0.smithi165.stdout:2573: finishing copy_from to smithi16534841-49 2024-09-10T16:23:22.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1543 (ObjNum 892 snap 274 seq_num 892) dirty exists 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2572: done (4 left) 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2573: done (3 left) 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2574: done (2 left) 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2576: done (1 left) 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2577: done (0 left) 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2578: write oid 22 current snap is 280 2024-09-10T16:23:23.841 INFO:tasks.rados.rados.0.smithi165.stdout:2578: seq_num 913 ranges {544791=664697,1693354=595308,2689310=445583} 2024-09-10T16:23:23.847 INFO:tasks.rados.rados.0.smithi165.stdout:2578: writing smithi16534841-22 from 544791 to 1209488 tid 1 2024-09-10T16:23:23.853 INFO:tasks.rados.rados.0.smithi165.stdout:2578: writing smithi16534841-22 from 1693354 to 2288662 tid 2 2024-09-10T16:23:23.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: writing smithi16534841-22 from 2689310 to 3134893 tid 3 2024-09-10T16:23:23.858 INFO:tasks.rados.rados.0.smithi165.stdout:2579: read oid 16 snap -1 2024-09-10T16:23:23.858 INFO:tasks.rados.rados.0.smithi165.stdout:2579: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:23.858 INFO:tasks.rados.rados.0.smithi165.stdout:2580: write (excl) oid 13 current snap is 280 2024-09-10T16:23:23.858 INFO:tasks.rados.rados.0.smithi165.stdout:2580: seq_num 914 ranges {561598=739946,1751836=455319,2716733=644865} 2024-09-10T16:23:23.864 INFO:tasks.rados.rados.0.smithi165.stdout:2580: writing smithi16534841-13 from 561598 to 1301544 tid 1 2024-09-10T16:23:23.869 INFO:tasks.rados.rados.0.smithi165.stdout:2580: writing smithi16534841-13 from 1751836 to 2207155 tid 2 2024-09-10T16:23:23.876 INFO:tasks.rados.rados.0.smithi165.stdout:2580: writing smithi16534841-13 from 2716733 to 3361598 tid 3 2024-09-10T16:23:23.876 INFO:tasks.rados.rados.0.smithi165.stdout:2581: write (excl) oid 3 current snap is 280 2024-09-10T16:23:23.876 INFO:tasks.rados.rados.0.smithi165.stdout:2581: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2024-09-10T16:23:23.881 INFO:tasks.rados.rados.0.smithi165.stdout:2581: writing smithi16534841-3 from 578405 to 993600 tid 1 2024-09-10T16:23:23.887 INFO:tasks.rados.rados.0.smithi165.stdout:2581: writing smithi16534841-3 from 1410318 to 2009295 tid 2 2024-09-10T16:23:23.895 INFO:tasks.rados.rados.0.smithi165.stdout:2581: writing smithi16534841-3 from 2744156 to 3328636 tid 3 2024-09-10T16:23:23.895 INFO:tasks.rados.rados.0.smithi165.stdout:2582: snap_remove snap 270 2024-09-10T16:23:24.851 INFO:tasks.rados.rados.0.smithi165.stdout:2582: done (4 left) 2024-09-10T16:23:24.851 INFO:tasks.rados.rados.0.smithi165.stdout:2583: read oid 33 snap 274 2024-09-10T16:23:24.851 INFO:tasks.rados.rados.0.smithi165.stdout:2583: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:23:24.852 INFO:tasks.rados.rados.0.smithi165.stdout:2584: snap_remove snap 272 2024-09-10T16:23:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:2584: done (5 left) 2024-09-10T16:23:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:2585: copy_from oid 34 from oid 49 current snap is 280 2024-09-10T16:23:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:2581: finishing write tid 1 to smithi16534841-3 2024-09-10T16:23:25.856 INFO:tasks.rados.rados.0.smithi165.stdout:2581: finishing write tid 2 to smithi16534841-3 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: finishing write tid 3 to smithi16534841-3 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: finishing write tid 4 to smithi16534841-3 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: finishing write tid 5 to smithi16534841-3 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: oid 3 updating version 0 to 1456 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: oid 3 version 1456 is already newer than 1454 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: oid 3 version 1456 is already newer than 1455 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: oid 3 version 1456 is already newer than 1453 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1456 (ObjNum 915 snap 280 seq_num 915) dirty exists 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2581: left oid 3 (ObjNum 915 snap 280 seq_num 915) 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2578: finishing write tid 1 to smithi16534841-22 2024-09-10T16:23:25.857 INFO:tasks.rados.rados.0.smithi165.stdout:2578: finishing write tid 2 to smithi16534841-22 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: finishing write tid 3 to smithi16534841-22 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: finishing write tid 4 to smithi16534841-22 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: finishing write tid 5 to smithi16534841-22 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: oid 22 updating version 0 to 1397 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: oid 22 updating version 1397 to 1398 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: oid 22 updating version 1398 to 1399 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: oid 22 version 1399 is already newer than 1396 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1399 (ObjNum 913 snap 280 seq_num 913) dirty exists 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2578: left oid 22 (ObjNum 913 snap 280 seq_num 913) 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:2579: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:25.858 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:25.927 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2206828 2024-09-10T16:23:25.945 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:25.997 INFO:tasks.rados.rados.0.smithi165.stdout:2580: finishing write tid 1 to smithi16534841-13 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: finishing write tid 2 to smithi16534841-13 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: finishing write tid 3 to smithi16534841-13 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: finishing write tid 4 to smithi16534841-13 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: finishing write tid 5 to smithi16534841-13 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: oid 13 updating version 0 to 1400 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: oid 13 updating version 1400 to 1403 2024-09-10T16:23:25.998 INFO:tasks.rados.rados.0.smithi165.stdout:2580: oid 13 version 1403 is already newer than 1402 2024-09-10T16:23:25.999 INFO:tasks.rados.rados.0.smithi165.stdout:2580: oid 13 version 1403 is already newer than 1401 2024-09-10T16:23:25.999 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1403 (ObjNum 914 snap 280 seq_num 914) dirty exists 2024-09-10T16:23:25.999 INFO:tasks.rados.rados.0.smithi165.stdout:2580: left oid 13 (ObjNum 914 snap 280 seq_num 914) 2024-09-10T16:23:25.999 INFO:tasks.rados.rados.0.smithi165.stdout:2583: expect (ObjNum 633 snap 198 seq_num 633) 2024-09-10T16:23:26.089 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2296821 2024-09-10T16:23:26.100 INFO:tasks.rados.rados.0.smithi165.stdout:2585: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:23:26.100 INFO:tasks.rados.rados.0.smithi165.stdout:2585: finishing copy_from to smithi16534841-34 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1543 (ObjNum 892 snap 274 seq_num 892) dirty exists 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2578: done (5 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2579: done (4 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2580: done (3 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2581: done (2 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2583: done (1 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2585: done (0 left) 2024-09-10T16:23:26.101 INFO:tasks.rados.rados.0.smithi165.stdout:2586: write oid 12 current snap is 280 2024-09-10T16:23:26.102 INFO:tasks.rados.rados.0.smithi165.stdout:2586: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2024-09-10T16:23:26.105 INFO:tasks.rados.rados.0.smithi165.stdout:2586: writing smithi16534841-12 from 595212 to 1085656 tid 1 2024-09-10T16:23:26.112 INFO:tasks.rados.rados.0.smithi165.stdout:2586: writing smithi16534841-12 from 1868800 to 2327788 tid 2 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2586: writing smithi16534841-12 from 2887932 to 3395212 tid 3 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2587: copy_from oid 37 from oid 39 current snap is 280 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2586: finishing write tid 1 to smithi16534841-12 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2586: finishing write tid 2 to smithi16534841-12 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2588: read oid 29 snap 271 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2588: expect deleted 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2589: rollback oid 21 current snap is 280 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 278 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2590: write oid 6 current snap is 280 2024-09-10T16:23:26.119 INFO:tasks.rados.rados.0.smithi165.stdout:2590: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2024-09-10T16:23:26.125 INFO:tasks.rados.rados.0.smithi165.stdout:2590: writing smithi16534841-6 from 612019 to 1177712 tid 1 2024-09-10T16:23:26.131 INFO:tasks.rados.rados.0.smithi165.stdout:2590: writing smithi16534841-6 from 1810929 to 2413575 tid 2 2024-09-10T16:23:26.137 INFO:tasks.rados.rados.0.smithi165.stdout:2590: writing smithi16534841-6 from 3082649 to 3412019 tid 3 2024-09-10T16:23:26.137 INFO:tasks.rados.rados.0.smithi165.stdout:2591: snap_create 2024-09-10T16:23:26.137 INFO:tasks.rados.rados.0.smithi165.stdout:2586: finishing write tid 3 to smithi16534841-12 2024-09-10T16:23:26.137 INFO:tasks.rados.rados.0.smithi165.stdout:2586: finishing write tid 4 to smithi16534841-12 2024-09-10T16:23:26.137 INFO:tasks.rados.rados.0.smithi165.stdout:2586: finishing write tid 5 to smithi16534841-12 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2586: oid 12 updating version 0 to 1363 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2586: oid 12 version 1363 is already newer than 1361 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2586: oid 12 version 1363 is already newer than 1362 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2586: oid 12 updating version 1363 to 1364 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1364 (ObjNum 916 snap 280 seq_num 916) dirty exists 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2586: left oid 12 (ObjNum 916 snap 280 seq_num 916) 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2589: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1421 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2590: finishing write tid 1 to smithi16534841-6 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2587: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2587: finishing copy_from to smithi16534841-37 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1377 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-10T16:23:26.138 INFO:tasks.rados.rados.0.smithi165.stdout:2590: finishing write tid 2 to smithi16534841-6 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: finishing write tid 3 to smithi16534841-6 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: finishing write tid 4 to smithi16534841-6 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: finishing write tid 5 to smithi16534841-6 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: oid 6 updating version 0 to 1333 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: oid 6 updating version 1333 to 1336 2024-09-10T16:23:26.140 INFO:tasks.rados.rados.0.smithi165.stdout:2590: oid 6 version 1336 is already newer than 1334 2024-09-10T16:23:26.141 INFO:tasks.rados.rados.0.smithi165.stdout:2590: oid 6 version 1336 is already newer than 1335 2024-09-10T16:23:26.141 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1336 (ObjNum 917 snap 280 seq_num 917) dirty exists 2024-09-10T16:23:26.141 INFO:tasks.rados.rados.0.smithi165.stdout:2590: left oid 6 (ObjNum 917 snap 280 seq_num 917) 2024-09-10T16:23:26.479 INFO:tasks.rados.rados.0.smithi165.stdout:2586: done (5 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2587: done (4 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2588: done (3 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2589: done (2 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2590: done (1 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2591: done (0 left) 2024-09-10T16:23:26.480 INFO:tasks.rados.rados.0.smithi165.stdout:2592: delete oid 29 current snap is 281 2024-09-10T16:23:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2592: done (0 left) 2024-09-10T16:23:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2593: read oid 4 snap -1 2024-09-10T16:23:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2593: expect (ObjNum 879 snap 269 seq_num 879) 2024-09-10T16:23:26.484 INFO:tasks.rados.rados.0.smithi165.stdout:2594: snap_remove snap 278 2024-09-10T16:23:27.486 INFO:tasks.rados.rados.0.smithi165.stdout:2594: done (1 left) 2024-09-10T16:23:27.486 INFO:tasks.rados.rados.0.smithi165.stdout:2595: snap_remove snap 271 2024-09-10T16:23:28.489 INFO:tasks.rados.rados.0.smithi165.stdout:2595: done (1 left) 2024-09-10T16:23:28.489 INFO:tasks.rados.rados.0.smithi165.stdout:2596: rollback oid 21 current snap is 281 2024-09-10T16:23:28.489 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 265 2024-09-10T16:23:28.489 INFO:tasks.rados.rados.0.smithi165.stdout:2593: expect (ObjNum 879 snap 269 seq_num 879) 2024-09-10T16:23:28.490 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:28.544 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 437246 2024-09-10T16:23:28.620 INFO:tasks.rados.rados.0.smithi165.stdout:2596: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:2596: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1471 (ObjNum 826 snap 255 seq_num 826) dirty exists 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:2593: done (1 left) 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:2596: done (0 left) 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:2597: write (excl) oid 35 current snap is 281 2024-09-10T16:23:28.621 INFO:tasks.rados.rados.0.smithi165.stdout:2597: seq_num 919 ranges {645633=716191,1927893=606315,3137495=308138} 2024-09-10T16:23:28.627 INFO:tasks.rados.rados.0.smithi165.stdout:2597: writing smithi16534841-35 from 645633 to 1361824 tid 1 2024-09-10T16:23:28.634 INFO:tasks.rados.rados.0.smithi165.stdout:2597: writing smithi16534841-35 from 1927893 to 2534208 tid 2 2024-09-10T16:23:28.639 INFO:tasks.rados.rados.0.smithi165.stdout:2597: writing smithi16534841-35 from 3137495 to 3445633 tid 3 2024-09-10T16:23:28.639 INFO:tasks.rados.rados.0.smithi165.stdout:2598: snap_remove snap 261 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2598: done (1 left) 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2599: copy_from oid 36 from oid 17 current snap is 281 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: finishing write tid 1 to smithi16534841-35 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: finishing write tid 2 to smithi16534841-35 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: finishing write tid 3 to smithi16534841-35 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: finishing write tid 4 to smithi16534841-35 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: finishing write tid 5 to smithi16534841-35 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: oid 35 updating version 0 to 1405 2024-09-10T16:23:29.492 INFO:tasks.rados.rados.0.smithi165.stdout:2597: oid 35 version 1405 is already newer than 1404 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:2597: oid 35 updating version 1405 to 1407 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:2597: oid 35 version 1407 is already newer than 1406 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1407 (ObjNum 919 snap 281 seq_num 919) dirty exists 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:2597: left oid 35 (ObjNum 919 snap 281 seq_num 919) 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:2597: done (1 left) 2024-09-10T16:23:29.493 INFO:tasks.rados.rados.0.smithi165.stdout:2600: delete oid 29 current snap is 281 2024-09-10T16:23:29.498 INFO:tasks.rados.rados.0.smithi165.stdout:2600: done (1 left) 2024-09-10T16:23:29.498 INFO:tasks.rados.rados.0.smithi165.stdout:2601: write oid 8 current snap is 281 2024-09-10T16:23:29.498 INFO:tasks.rados.rados.0.smithi165.stdout:2601: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2024-09-10T16:23:29.505 INFO:tasks.rados.rados.0.smithi165.stdout:2601: writing smithi16534841-8 from 679247 to 1262289 tid 1 2024-09-10T16:23:29.513 INFO:tasks.rados.rados.0.smithi165.stdout:2601: writing smithi16534841-8 from 2044857 to 2654841 tid 2 2024-09-10T16:23:29.517 INFO:tasks.rados.rados.0.smithi165.stdout:2601: writing smithi16534841-8 from 3192341 to 3479247 tid 3 2024-09-10T16:23:29.518 INFO:tasks.rados.rados.0.smithi165.stdout:2602: write oid 50 current snap is 281 2024-09-10T16:23:29.518 INFO:tasks.rados.rados.0.smithi165.stdout:2602: seq_num 922 ranges {696054=658291,2103339=753642} 2024-09-10T16:23:29.525 INFO:tasks.rados.rados.0.smithi165.stdout:2602: writing smithi16534841-50 from 696054 to 1354345 tid 1 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2602: writing smithi16534841-50 from 2103339 to 2856981 tid 2 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2603: rollback oid 3 current snap is 281 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 263 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2599: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2599: finishing copy_from to smithi16534841-36 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1149 (ObjNum 856 snap 263 seq_num 856) dirty exists 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2601: finishing write tid 1 to smithi16534841-8 2024-09-10T16:23:29.534 INFO:tasks.rados.rados.0.smithi165.stdout:2601: finishing write tid 2 to smithi16534841-8 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: finishing write tid 3 to smithi16534841-8 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: finishing write tid 4 to smithi16534841-8 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: finishing write tid 5 to smithi16534841-8 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: oid 8 updating version 0 to 1488 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: oid 8 updating version 1488 to 1490 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: oid 8 updating version 1490 to 1491 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: oid 8 version 1491 is already newer than 1489 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1491 (ObjNum 921 snap 281 seq_num 921) dirty exists 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: left oid 8 (ObjNum 921 snap 281 seq_num 921) 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2602: finishing write tid 1 to smithi16534841-50 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2599: done (3 left) 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2601: done (2 left) 2024-09-10T16:23:29.535 INFO:tasks.rados.rados.0.smithi165.stdout:2604: write (excl) oid 4 current snap is 281 2024-09-10T16:23:29.536 INFO:tasks.rados.rados.0.smithi165.stdout:2604: seq_num 923 ranges {712861=733540,2161821=613653,3247187=265674} 2024-09-10T16:23:29.542 INFO:tasks.rados.rados.0.smithi165.stdout:2604: writing smithi16534841-4 from 712861 to 1446401 tid 1 2024-09-10T16:23:29.549 INFO:tasks.rados.rados.0.smithi165.stdout:2604: writing smithi16534841-4 from 2161821 to 2775474 tid 2 2024-09-10T16:23:29.554 INFO:tasks.rados.rados.0.smithi165.stdout:2604: writing smithi16534841-4 from 3247187 to 3512861 tid 3 2024-09-10T16:23:29.554 INFO:tasks.rados.rados.0.smithi165.stdout:2605: snap_create 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: finishing write tid 2 to smithi16534841-50 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: finishing write tid 3 to smithi16534841-50 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: finishing write tid 4 to smithi16534841-50 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: oid 50 updating version 0 to 1275 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: oid 50 version 1275 is already newer than 1274 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: oid 50 updating version 1275 to 1276 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1276 (ObjNum 922 snap 281 seq_num 922) dirty exists 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2602: left oid 50 (ObjNum 922 snap 281 seq_num 922) 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2603: finishing rollback tid 0 to smithi16534841-3 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2603: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1474 (ObjNum 129 snap 0 seq_num 1160024320) dirty dne 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2604: finishing write tid 1 to smithi16534841-4 2024-09-10T16:23:29.555 INFO:tasks.rados.rados.0.smithi165.stdout:2604: finishing write tid 2 to smithi16534841-4 2024-09-10T16:23:29.556 INFO:tasks.rados.rados.0.smithi165.stdout:2604: finishing write tid 3 to smithi16534841-4 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: finishing write tid 4 to smithi16534841-4 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: finishing write tid 5 to smithi16534841-4 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: oid 4 updating version 0 to 1329 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: oid 4 version 1329 is already newer than 1326 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: oid 4 version 1329 is already newer than 1327 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:2604: oid 4 version 1329 is already newer than 1328 2024-09-10T16:23:29.559 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1329 (ObjNum 923 snap 281 seq_num 923) dirty exists 2024-09-10T16:23:29.560 INFO:tasks.rados.rados.0.smithi165.stdout:2604: left oid 4 (ObjNum 923 snap 281 seq_num 923) 2024-09-10T16:23:30.507 INFO:tasks.rados.rados.0.smithi165.stdout:2602: done (3 left) 2024-09-10T16:23:30.507 INFO:tasks.rados.rados.0.smithi165.stdout:2603: done (2 left) 2024-09-10T16:23:30.507 INFO:tasks.rados.rados.0.smithi165.stdout:2604: done (1 left) 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2605: done (0 left) 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2606: read oid 6 snap -1 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2606: expect (ObjNum 917 snap 280 seq_num 917) 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2607: rollback oid 14 current snap is 282 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 281 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2608: read oid 46 snap -1 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2608: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:30.508 INFO:tasks.rados.rados.0.smithi165.stdout:2609: snap_create 2024-09-10T16:23:30.513 INFO:tasks.rados.rados.0.smithi165.stdout:2607: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:23:30.513 INFO:tasks.rados.rados.0.smithi165.stdout:2607: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:23:30.513 INFO:tasks.rados.rados.0.smithi165.stdout:2607: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:23:30.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1224 (ObjNum 841 snap 260 seq_num 841) dirty exists 2024-09-10T16:23:30.514 INFO:tasks.rados.rados.0.smithi165.stdout:2608: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:30.519 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 930528 2024-09-10T16:23:30.533 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1196331 2024-09-10T16:23:30.569 INFO:tasks.rados.rados.0.smithi165.stdout:2606: expect (ObjNum 917 snap 280 seq_num 917) 2024-09-10T16:23:30.569 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:30.607 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:31.458 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:23:31.459 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:23:31.459 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:23:31.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.569+0000 7f6634594640 10 monclient: get_monmap_and_config 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.569+0000 7f6634594640 10 monclient: build_initial_monmap 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.569+0000 7f6634594640 10 monclient: monmap: 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:31.570546+0000 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:31.570546+0000 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:31.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:31.572 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:31.572 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:31.572 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:31.572 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 Processor -- start 2024-09-10T16:23:31.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 -- start start 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient: init 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f662c09d930 0x7f662c09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient(hunting): picked mon.noname-c con 0x7f662c09d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f662c09ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 0x7f662c134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient(hunting): picked mon.noname-b con 0x7f662c09e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f662c0a0870 con 0x7f662c09ab50 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:31.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f662c09d3e0 con 0x7f662c09d930 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.570+0000 7f6634594640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f662c074c70 con 0x7f662c09e250 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6634594640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6634594640 10 monclient(hunting): authenticate will time out at 1790.058011s 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6631b2f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f662c09ab50 0x7f662c134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:46194/0 (socket says 172.21.15.164:46194) 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6631b2f640 1 -- 172.21.15.164:0/2195733719 learned_addr learned my addr 172.21.15.164:0/2195733719 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 --2- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 0x7f662c134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): get_auth_request con 0x7f662c09e250 auth_method 0 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:31.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 1 --2- 172.21.15.164:0/2195733719 >> v2:172.21.15.165:3300/0 conn(0x7f662c09d930 0x7f662c09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): get_auth_request con 0x7f662c09d930 auth_method 0 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient(hunting): handle_auth_done global_id 194099 payload 274 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 -- 172.21.15.164:0/2195733719 >> v2:172.21.15.165:3300/0 conn(0x7f662c09d930 msgr2=0x7f662c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 --2- 172.21.15.164:0/2195733719 >> v2:172.21.15.165:3300/0 conn(0x7f662c09d930 0x7f662c09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:31.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 -- 172.21.15.164:0/2195733719 >> v1:172.21.15.164:6789/0 conn(0x7f662c09ab50 legacy=0x7f662c134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient: _finish_hunting 0 2024-09-10T16:23:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 monclient: found mon.noname-b 2024-09-10T16:23:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:23:31.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.571+0000 7f6632b31640 1 -- 172.21.15.164:0/2195733719 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c09ecb0 con 0x7f662c09e250 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6632b31640 1 -- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 msgr2=0x7f662c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6632b31640 1 --2- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 0x7f662c134c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6624009d90 tx=0x7f662402a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e0f7af67c67056c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 1 -- 172.21.15.164:0/2195733719 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624007950 con 0x7f662c09e250 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: dump: 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:31.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: _finish_auth 0 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: _check_auth_tickets 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 5 monclient: authenticate success, global_id 194099 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 1 -- 172.21.15.164:0/2195733719 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6624007af0 con 0x7f662c09e250 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 4 set_mon_vals no callback set 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 1 -- 172.21.15.164:0/2195733719 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6624031400 con 0x7f662c09e250 2024-09-10T16:23:31.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f663132e640 10 monclient: dump: 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 10 monclient: shutdown 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 1 -- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 msgr2=0x7f662c134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.574+0000 7f6634594640 1 --2- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 0x7f662c134c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6624009d90 tx=0x7f662402a490 comp rx=0 tx=0).stop 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 -- 172.21.15.164:0/2195733719 shutdown_connections 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 --2- 172.21.15.164:0/2195733719 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09e250 0x7f662c134c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 --2- 172.21.15.164:0/2195733719 >> v2:172.21.15.165:3300/0 conn(0x7f662c09d930 0x7f662c09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 -- 172.21.15.164:0/2195733719 >> 172.21.15.164:0/2195733719 conn(0x7f662c0a3110 msgr2=0x7f662c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 -- 172.21.15.164:0/2195733719 shutdown_connections 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 -- 172.21.15.164:0/2195733719 wait complete. 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 10 monclient: build_initial_monmap 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 10 monclient: monmap: 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:31.588 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 Processor -- start 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 1 -- start start 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.575+0000 7f6634594640 10 monclient: init 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:23:31.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): picked mon.noname-c con 0x7f662c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 0x7f662c13e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): picked mon.noname-a con 0x7f662c09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c09e250 0x7f662c136dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): picked mon.noname-b con 0x7f662c09e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f662c09ef30 con 0x7f662c09d930 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f662c09e6e0 con 0x7f662c09e250 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f662c0a2940 con 0x7f662c09ab50 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6634594640 10 monclient(hunting): authenticate will time out at 1790.063410s 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 0x7f662c13e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46566/0 (socket says 172.21.15.164:46566) 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 -- 172.21.15.164:0/1478076297 learned_addr learned my addr 172.21.15.164:0/1478076297 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): get_auth_request con 0x7f662c09ab50 auth_method 0 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): get_auth_request con 0x7f662c09d930 auth_method 0 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:31.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c09e250 0x7f662c136dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): get_auth_request con 0x7f662c09e250 auth_method 0 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632b31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient(hunting): handle_auth_done global_id 194102 payload 824 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 -- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c09e250 msgr2=0x7f662c136dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:31.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c09e250 0x7f662c136dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 -- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 msgr2=0x7f662c13e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 0x7f662c13e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient: _finish_hunting 0 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 monclient: found mon.noname-c 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6632330640 1 -- 172.21.15.164:0/1478076297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c09d620 con 0x7f662c09ab50 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.576+0000 7f6631b2f640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 0x7f662c13e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6632330640 1 -- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 msgr2=0x7f662c1393a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6632330640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f662800e9e0 tx=0x7f662800eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628017070 con 0x7f662c09ab50 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:31.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: dump: 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: _finish_auth 0 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f662800cb80 con 0x7f662c09ab50 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 5 monclient: authenticate success, global_id 194102 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 10 monclient: _renew_subs 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c09ccc0 con 0x7f662c09ab50 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6628013470 con 0x7f662c09ab50 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6622ffd640 10 monclient: dump: 2024-09-10T16:23:31.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6630b2d640 4 set_mon_vals no callback set 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 10 monclient: _renew_subs 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:31.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.579+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c13a300 con 0x7f662c09ab50 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6632330640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f662800e9e0 tx=0x7f662800eea0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f6628052d90 osd_map(588..588 src has 481..588) 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f662801a070 con 0x7f662c09ab50 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6634594640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6634594640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6634594640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f662c09dd10 con 0x7f662c09ab50 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6622ffd640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 0x7f66100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6631b2f640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 0x7f66100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.580+0000 7f6631b2f640 10 monclient: get_auth_request con 0x7f661003e210 auth_method 0 2024-09-10T16:23:31.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.583+0000 7f6631b2f640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 0x7f66100406d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f661c006d50 tx=0x7f661c006ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:31.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:31.585+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(588..588 src has 481..588) ==== 4787+0+0 (secure 0 0 0) 0x7f6628052d90 con 0x7f662c09ab50 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2606: done (3 left) 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2607: done (2 left) 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2608: done (1 left) 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2609: done (0 left) 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2610: write oid 34 current snap is 283 2024-09-10T16:23:31.894 INFO:tasks.rados.rados.0.smithi165.stdout:2610: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2024-09-10T16:23:31.900 INFO:tasks.rados.rados.0.smithi165.stdout:2610: writing smithi16534841-34 from 729668 to 1138457 tid 1 2024-09-10T16:23:31.908 INFO:tasks.rados.rados.0.smithi165.stdout:2610: writing smithi16534841-34 from 1820303 to 2577614 tid 2 2024-09-10T16:23:31.913 INFO:tasks.rados.rados.0.smithi165.stdout:2610: writing smithi16534841-34 from 3274610 to 3529668 tid 3 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2611: read oid 16 snap -1 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2611: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2612: write oid 35 current snap is 283 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2610: finishing write tid 1 to smithi16534841-34 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2610: finishing write tid 2 to smithi16534841-34 2024-09-10T16:23:31.914 INFO:tasks.rados.rados.0.smithi165.stdout:2612: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2024-09-10T16:23:31.920 INFO:tasks.rados.rados.0.smithi165.stdout:2612: writing smithi16534841-35 from 746475 to 1230513 tid 1 2024-09-10T16:23:31.926 INFO:tasks.rados.rados.0.smithi165.stdout:2612: writing smithi16534841-35 from 1762432 to 2379754 tid 2 2024-09-10T16:23:31.931 INFO:tasks.rados.rados.0.smithi165.stdout:2612: writing smithi16534841-35 from 2785680 to 3315586 tid 3 2024-09-10T16:23:31.932 INFO:tasks.rados.rados.0.smithi165.stdout:2613: write oid 25 current snap is 283 2024-09-10T16:23:31.932 INFO:tasks.rados.rados.0.smithi165.stdout:2613: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2024-09-10T16:23:31.939 INFO:tasks.rados.rados.0.smithi165.stdout:2613: writing smithi16534841-25 from 763282 to 1322569 tid 1 2024-09-10T16:23:31.946 INFO:tasks.rados.rados.0.smithi165.stdout:2613: writing smithi16534841-25 from 1820914 to 2581894 tid 2 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2613: writing smithi16534841-25 from 3213103 to 3563282 tid 3 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2614: snap_create 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: finishing write tid 3 to smithi16534841-34 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: finishing write tid 4 to smithi16534841-34 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: finishing write tid 5 to smithi16534841-34 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: oid 34 updating version 0 to 1546 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: oid 34 version 1546 is already newer than 1545 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: oid 34 version 1546 is already newer than 1544 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: oid 34 updating version 1546 to 1547 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1547 (ObjNum 924 snap 283 seq_num 924) dirty exists 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2610: left oid 34 (ObjNum 924 snap 283 seq_num 924) 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:2611: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:31.952 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:32.002 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:32.102 INFO:tasks.rados.rados.0.smithi165.stdout:2612: finishing write tid 1 to smithi16534841-35 2024-09-10T16:23:32.102 INFO:tasks.rados.rados.0.smithi165.stdout:2612: finishing write tid 2 to smithi16534841-35 2024-09-10T16:23:32.102 INFO:tasks.rados.rados.0.smithi165.stdout:2612: finishing write tid 3 to smithi16534841-35 2024-09-10T16:23:32.102 INFO:tasks.rados.rados.0.smithi165.stdout:2612: finishing write tid 4 to smithi16534841-35 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: finishing write tid 5 to smithi16534841-35 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: oid 35 updating version 0 to 1411 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: oid 35 version 1411 is already newer than 1410 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: oid 35 version 1411 is already newer than 1409 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: oid 35 version 1411 is already newer than 1408 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1411 (ObjNum 925 snap 283 seq_num 925) dirty exists 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2612: left oid 35 (ObjNum 925 snap 283 seq_num 925) 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: finishing write tid 1 to smithi16534841-25 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: finishing write tid 2 to smithi16534841-25 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: finishing write tid 3 to smithi16534841-25 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: finishing write tid 4 to smithi16534841-25 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: finishing write tid 5 to smithi16534841-25 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: oid 25 updating version 0 to 1484 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: oid 25 version 1484 is already newer than 1482 2024-09-10T16:23:32.103 INFO:tasks.rados.rados.0.smithi165.stdout:2613: oid 25 version 1484 is already newer than 1481 2024-09-10T16:23:32.104 INFO:tasks.rados.rados.0.smithi165.stdout:2613: oid 25 version 1484 is already newer than 1483 2024-09-10T16:23:32.104 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1484 (ObjNum 926 snap 283 seq_num 926) dirty exists 2024-09-10T16:23:32.104 INFO:tasks.rados.rados.0.smithi165.stdout:2613: left oid 25 (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:23:32.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.408+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6628051780 con 0x7f662c09ab50 2024-09-10T16:23:32.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.408+0000 7f6622ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:32.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.408+0000 7f6622ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:32.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6634594640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6634594640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6634594640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f662c06dac0 con 0x7f662c09ab50 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6622ffd640 1 -- 172.21.15.164:0/1478076297 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6628023d00 con 0x7f662c09ab50 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6622ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.657+0000 7f6622ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:32.659 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:32.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 msgr2=0x7f66100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 0x7f66100406d0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f661c006d50 tx=0x7f661c006ca0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 10 monclient: shutdown 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 msgr2=0x7f662c1393a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f662800e9e0 tx=0x7f662800eea0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 shutdown_connections 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f661003e210 0x7f66100406d0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f662c09e250 0x7f662c136dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f662c09d930 0x7f662c13e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.659+0000 7f6634594640 1 --2- 172.21.15.164:0/1478076297 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f662c09ab50 0x7f662c1393a0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.660+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 >> 172.21.15.164:0/1478076297 conn(0x7f662c0a3110 msgr2=0x7f662c099450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:32.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.660+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 shutdown_connections 2024-09-10T16:23:32.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.660+0000 7f6634594640 1 -- 172.21.15.164:0/1478076297 wait complete. 2024-09-10T16:23:32.671 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:23:32.672 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:23:32.672 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.769+0000 7ff5984e6640 10 monclient: get_monmap_and_config 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.769+0000 7ff5984e6640 10 monclient: build_initial_monmap 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.769+0000 7ff5984e6640 10 monclient: monmap: 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:32.770712+0000 2024-09-10T16:23:32.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:32.770712+0000 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:32.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.770+0000 7ff5984e6640 1 Processor -- start 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.770+0000 7ff5984e6640 1 -- start start 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.770+0000 7ff5984e6640 10 monclient: init 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff590074af0 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:32.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff5900a1090 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5900a1500 0x7ff5901408c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff5900a1500 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff59009ebc0 con 0x7ff5900a1090 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff590138580 con 0x7ff590074af0 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900698a0 con 0x7ff5900a1500 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff5984e6640 10 monclient(hunting): authenticate will time out at 1791.258189s 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5900a1500 0x7ff5901408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff595a81640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff5900a1090 0x7ff59013d190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:46202/0 (socket says 172.21.15.164:46202) 2024-09-10T16:23:32.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff595a81640 1 -- 172.21.15.164:0/4159188472 learned_addr learned my addr 172.21.15.164:0/4159188472 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): get_auth_request con 0x7ff5900a1500 auth_method 0 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient(hunting): handle_auth_done global_id 164352 payload 274 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 1 -- 172.21.15.164:0/4159188472 >> v1:172.21.15.165:6789/0 conn(0x7ff590074af0 legacy=0x7ff590137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 1 -- 172.21.15.164:0/4159188472 >> v1:172.21.15.164:6789/0 conn(0x7ff5900a1090 legacy=0x7ff59013d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient: _finish_hunting 0 2024-09-10T16:23:32.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 1 monclient: found mon.noname-a 2024-09-10T16:23:32.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:32.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.771+0000 7ff596a83640 1 -- 172.21.15.164:0/4159188472 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900a1950 con 0x7ff5900a1500 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff596a83640 1 -- 172.21.15.164:0/4159188472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1500 msgr2=0x7ff5901408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff596a83640 1 --2- 172.21.15.164:0/4159188472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1500 0x7ff5901408c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff588009d90 tx=0x7ff58802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79d0c9470d869f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 1 -- 172.21.15.164:0/4159188472 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588007a30 con 0x7ff5900a1500 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: dump: 2024-09-10T16:23:32.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: _finish_auth 0 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: _check_auth_tickets 2024-09-10T16:23:32.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 1 -- 172.21.15.164:0/4159188472 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff588007bd0 con 0x7ff5900a1500 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff5984e6640 5 monclient: authenticate success, global_id 164352 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff5984e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 1 -- 172.21.15.164:0/4159188472 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff588031400 con 0x7ff5900a1500 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff595280640 10 monclient: dump: 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.785 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff5984e6640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.774+0000 7ff5984e6640 4 set_mon_vals no callback set 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 10 monclient: shutdown 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 -- 172.21.15.164:0/4159188472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1500 msgr2=0x7ff5901408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 --2- 172.21.15.164:0/4159188472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1500 0x7ff5901408c0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff588009d90 tx=0x7ff58802a490 comp rx=0 tx=0).stop 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 -- 172.21.15.164:0/4159188472 shutdown_connections 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 --2- 172.21.15.164:0/4159188472 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1500 0x7ff5901408c0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 -- 172.21.15.164:0/4159188472 >> 172.21.15.164:0/4159188472 conn(0x7ff5900a3fb0 msgr2=0x7ff5900a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 -- 172.21.15.164:0/4159188472 shutdown_connections 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 1 -- 172.21.15.164:0/4159188472 wait complete. 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 10 monclient: build_initial_monmap 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.775+0000 7ff5984e6640 10 monclient: monmap: 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:32.787 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 Processor -- start 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 -- start start 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient: init 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 0x7ff590131890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff590074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:32.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff5900a1090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5900a1500 0x7ff59012cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff5900a1500 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff590069370 con 0x7ff5900a1090 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff590068d70 con 0x7ff5900a1500 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff590074ed0 con 0x7ff590074af0 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff5984e6640 10 monclient(hunting): authenticate will time out at 1791.263627s 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 0x7ff590131890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 0x7ff590131890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46574/0 (socket says 172.21.15.164:46574) 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 1 -- 172.21.15.164:0/3029208996 learned_addr learned my addr 172.21.15.164:0/3029208996 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58700/0 (socket says 172.21.15.164:58700) 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): get_auth_request con 0x7ff590074af0 auth_method 0 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): get_auth_request con 0x7ff5900a1090 auth_method 0 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff595a81640 10 monclient(hunting): handle_auth_done global_id 164355 payload 824 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.776+0000 7ff596a83640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5900a1500 0x7ff59012cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 msgr2=0x7ff590131890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 0x7ff590131890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5900a1500 msgr2=0x7ff59012cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5900a1500 0x7ff59012cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 10 monclient: _finish_hunting 0 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 monclient: found mon.noname-a 2024-09-10T16:23:32.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff595a81640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff59012fae0 con 0x7ff5900a1090 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.777+0000 7ff596282640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff590074af0 0x7ff590131890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff595a81640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 msgr2=0x7ff59012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff595a81640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c3e0 tx=0x7ff58000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580017070 con 0x7ff5900a1090 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: dump: 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.792 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: _finish_auth 0 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5800070d0 con 0x7ff5900a1090 2024-09-10T16:23:32.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5800074c0 con 0x7ff5900a1090 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 5 monclient: authenticate success, global_id 164355 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 10 monclient: _renew_subs 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59012fdc0 con 0x7ff5900a1090 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff594a7f640 4 set_mon_vals no callback set 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff586ffd640 10 monclient: dump: 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.794 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 10 monclient: _renew_subs 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.779+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff59012dd80 con 0x7ff5900a1090 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff5984e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff5984e6640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 0x7ff59012cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff590137e70 con 0x7ff590131dd0 2024-09-10T16:23:32.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 0x7ff59012cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient: get_auth_request con 0x7ff590131dd0 auth_method 0 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff58001a070 con 0x7ff5900a1090 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 10 monclient(hunting): handle_auth_done global_id 164358 payload 824 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596282640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff59012ff90 con 0x7ff590131dd0 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff586ffd640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57403e210 0x7ff5740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(589..589 src has 481..589) ==== 4787+0+0 (secure 0 0 0) 0x7ff5800521c0 con 0x7ff5900a1090 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596a83640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57403e210 0x7ff5740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.780+0000 7ff596a83640 10 monclient: get_auth_request con 0x7ff57403e210 auth_method 0 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff596282640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 msgr2=0x7ff59012cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff596282640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 0x7ff59012cff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c018020 con 0x7ff590131dd0 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff586ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff586ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.782+0000 7ff586ffd640 10 monclient: dump: 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.799 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff58c00de50 con 0x7ff590131dd0 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 msgr2=0x7ff59012cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 0x7ff59012cff0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c004580 comp rx=0 tx=0).stop 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 msgr2=0x7ff59012cff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff586ffd640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff590131dd0 0x7ff59012cff0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c004580 comp rx=0 tx=0).stop 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.783+0000 7ff596a83640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57403e210 0x7ff5740406d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff590135a30 tx=0x7ff58802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff5984e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff5984e6640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 0x7ff59009b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff5984e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff59012f8b0 con 0x7ff59012e8a0 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 0x7ff59009b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient: get_auth_request con 0x7ff59012e8a0 auth_method 0 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 10 monclient(hunting): handle_auth_done global_id 164361 payload 824 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.789+0000 7ff596282640 1 -- 172.21.15.164:0/3029208996 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5900772c0 con 0x7ff59012e8a0 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff596282640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 msgr2=0x7ff59009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff596282640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 0x7ff59009b3b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c0036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c018030 con 0x7ff59012e8a0 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff586ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff586ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.791+0000 7ff586ffd640 10 monclient: dump: 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.802 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7ff58c014b80 con 0x7ff59012e8a0 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 msgr2=0x7ff59009b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 0x7ff59009b3b0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c0036f0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 msgr2=0x7ff59009b3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff586ffd640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff59012e8a0 0x7ff59009b3b0 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff58c00fb10 tx=0x7ff58c0036f0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 183493, 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:32.804 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.805 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.806 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:32.807 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.808 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:32.809 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.810 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:32.811 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.812 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.813 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:32.814 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.815 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.816 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.817 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.817 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:32.817 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57403e210 msgr2=0x7ff5740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57403e210 0x7ff5740406d0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7ff590135a30 tx=0x7ff58802f000 comp rx=0 tx=0).stop 2024-09-10T16:23:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 10 monclient: shutdown 2024-09-10T16:23:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:32.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 msgr2=0x7ff59012c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff596a83640 1 -- 172.21.15.164:0/3029208996 reap_dead start 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff58000c3e0 tx=0x7ff58000c8a0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 shutdown_connections 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 --2- 172.21.15.164:0/3029208996 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5900a1090 0x7ff59012c570 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 >> 172.21.15.164:0/3029208996 conn(0x7ff5900a3fb0 msgr2=0x7ff5900a6c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 shutdown_connections 2024-09-10T16:23:32.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.792+0000 7ff5984e6640 1 -- 172.21.15.164:0/3029208996 wait complete. 2024-09-10T16:23:32.819 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:32.819 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:23:32.820 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2610: done (4 left) 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2611: done (3 left) 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2612: done (2 left) 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2613: done (1 left) 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2614: done (0 left) 2024-09-10T16:23:32.900 INFO:tasks.rados.rados.0.smithi165.stdout:2615: delete oid 21 current snap is 284 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2615: done (0 left) 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2616: read oid 41 snap 282 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2616: expect deleted 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2617: copy_from oid 24 from oid 23 current snap is 284 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2618: read oid 10 snap 266 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2618: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2619: rollback oid 37 current snap is 284 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 258 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2620: read oid 1 snap -1 2024-09-10T16:23:32.904 INFO:tasks.rados.rados.0.smithi165.stdout:2620: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:32.905 INFO:tasks.rados.rados.0.smithi165.stdout:2621: snap_create 2024-09-10T16:23:32.908 INFO:tasks.rados.rados.0.smithi165.stdout:2619: finishing rollback tid 0 to smithi16534841-37 2024-09-10T16:23:32.908 INFO:tasks.rados.rados.0.smithi165.stdout:2618: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:32.912 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 767545 2024-09-10T16:23:32.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 10 monclient: get_monmap_and_config 2024-09-10T16:23:32.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 10 monclient: build_initial_monmap 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 10 monclient: monmap: 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:32.924167+0000 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:32.924167+0000 2024-09-10T16:23:32.925 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:32.926 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.932 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1177869 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 1 Processor -- start 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 1 -- start start 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.923+0000 7f574c7bc640 10 monclient: init 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-a con 0x7f574406cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:32.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57440a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-b con 0x7f57440a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f574409ce50 con 0x7f57440a46a0 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f574406db60 con 0x7f574406cb70 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f574406cf50 con 0x7f57440a4ab0 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574c7bc640 10 monclient(hunting): authenticate will time out at 1791.411609s 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46584/0 (socket says 172.21.15.164:46584) 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58726/0 (socket says 172.21.15.164:58726) 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 1 -- 172.21.15.164:0/3812097945 learned_addr learned my addr 172.21.15.164:0/3812097945 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f5749d57640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f57440a46a0 0x7f5744137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:46204/0 (socket says 172.21.15.164:46204) 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): get_auth_request con 0x7f57440a4ab0 auth_method 0 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): get_auth_request con 0x7f574406cb70 auth_method 0 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 10 monclient(hunting): handle_auth_done global_id 194108 payload 274 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 1 -- 172.21.15.164:0/3812097945 >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 msgr2=0x7f5744134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.924+0000 7f574ad59640 1 --2- 172.21.15.164:0/3812097945 >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 1 -- 172.21.15.164:0/3812097945 >> v1:172.21.15.164:6789/0 conn(0x7f57440a46a0 legacy=0x7f5744137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 10 monclient: _finish_hunting 0 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 1 monclient: found mon.noname-b 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:23:32.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574ad59640 1 -- 172.21.15.164:0/3812097945 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f574409cb70 con 0x7f57440a4ab0 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.925+0000 7f574a558640 1 --2- 172.21.15.164:0/3812097945 >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574ad59640 1 -- 172.21.15.164:0/3812097945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 msgr2=0x7f57441383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574ad59640 1 --2- 172.21.15.164:0/3812097945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f573c009e80 tx=0x7f573c02a580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93fa19c3c17d63b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 1 -- 172.21.15.164:0/3812097945 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c036070 con 0x7f57440a4ab0 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: dump: 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.940 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: _finish_auth 0 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: _check_auth_tickets 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 1 -- 172.21.15.164:0/3812097945 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f573c007800 con 0x7f57440a4ab0 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574c7bc640 5 monclient: authenticate success, global_id 194108 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574c7bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:32.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574c7bc640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 1 -- 172.21.15.164:0/3812097945 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f573c007bd0 con 0x7f57440a4ab0 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f574c7bc640 4 set_mon_vals no callback set 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.927+0000 7f5749556640 10 monclient: dump: 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.942 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 10 monclient: shutdown 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 -- 172.21.15.164:0/3812097945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 msgr2=0x7f57441383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 --2- 172.21.15.164:0/3812097945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f573c009e80 tx=0x7f573c02a580 comp rx=0 tx=0).stop 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 -- 172.21.15.164:0/3812097945 shutdown_connections 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 --2- 172.21.15.164:0/3812097945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441383b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 --2- 172.21.15.164:0/3812097945 >> v2:172.21.15.164:3300/0 conn(0x7f574406cb70 0x7f5744134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 -- 172.21.15.164:0/3812097945 >> 172.21.15.164:0/3812097945 conn(0x7f57440aa900 msgr2=0x7f57440aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:32.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 -- 172.21.15.164:0/3812097945 shutdown_connections 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.928+0000 7f574c7bc640 1 -- 172.21.15.164:0/3812097945 wait complete. 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient: build_initial_monmap 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient: monmap: 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 Processor -- start 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 -- start start 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient: init 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-a con 0x7f574406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57440a46a0 0x7f574412e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-b con 0x7f57440a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient(hunting): picked mon.noname-c con 0x7f57440a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.929+0000 7f574c7bc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5744138a70 con 0x7f574406cb70 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f57441388f0 con 0x7f57440a46a0 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5744140040 con 0x7f57440a4ab0 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574c7bc640 10 monclient(hunting): authenticate will time out at 1791.417018s 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58730/0 (socket says 172.21.15.164:58730) 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46592/0 (socket says 172.21.15.164:46592) 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 -- 172.21.15.164:0/2036518384 learned_addr learned my addr 172.21.15.164:0/2036518384 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): get_auth_request con 0x7f574406cb70 auth_method 0 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): get_auth_request con 0x7f57440a4ab0 auth_method 0 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57440a46a0 0x7f574412e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 10 monclient(hunting): get_auth_request con 0x7f57440a46a0 auth_method 0 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient(hunting): handle_auth_done global_id 164367 payload 824 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 msgr2=0x7f57441cd0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441cd0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57440a46a0 msgr2=0x7f574412e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57440a46a0 0x7f574412e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient: _finish_hunting 0 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 monclient: found mon.noname-a 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:32.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574a558640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57441344a0 con 0x7f574406cb70 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f57440a46a0 0x7f574412e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.930+0000 7f574ad59640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57440a4ab0 0x7f57441cd0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574a558640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 msgr2=0x7f574412e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574a558640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f574000e990 tx=0x7f574000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740017070 con 0x7f574406cb70 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: dump: 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.949 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: _finish_auth 0 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f574000cb90 con 0x7f574406cb70 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 5 monclient: authenticate success, global_id 164367 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:32.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 10 monclient: _renew_subs 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5740013470 con 0x7f574406cb70 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5744133f50 con 0x7f574406cb70 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f573affd640 10 monclient: dump: 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.951 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f5748d55640 4 set_mon_vals no callback set 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 10 monclient: _renew_subs 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.932+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57440a2b70 con 0x7f574406cb70 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f574c7bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f574c7bc640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 0x7f57441cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f57441cdf20 con 0x7f57441cd600 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f574001a070 con 0x7f574406cb70 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 0x7f57441cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient: get_auth_request con 0x7f57441cd600 auth_method 0 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f5749d57640 10 monclient(hunting): handle_auth_done global_id 194114 payload 824 2024-09-10T16:23:32.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.933+0000 7f573affd640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f572803e210 0x7f57280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.934+0000 7f5749d57640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5744134f70 con 0x7f57441cd600 2024-09-10T16:23:32.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.934+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(590..590 src has 481..590) ==== 4787+0+0 (secure 0 0 0) 0x7f5740052c00 con 0x7f574406cb70 2024-09-10T16:23:32.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.934+0000 7f574ad59640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f572803e210 0x7f57280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.934+0000 7f574ad59640 10 monclient: get_auth_request con 0x7f572803e210 auth_method 0 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f5749d57640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 msgr2=0x7f57441cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 0x7f57441cd9e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f57440a1ba0 tx=0x7f573400db90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57340098f0 con 0x7f57441cd600 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 10 monclient: dump: 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.960 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5734019070 con 0x7f57441cd600 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 msgr2=0x7f57441cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 0x7f57441cd9e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f57440a1ba0 tx=0x7f573400db90 comp rx=0 tx=0).stop 2024-09-10T16:23:32.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 msgr2=0x7f57441cd9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f573affd640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f57441cd600 0x7f57441cd9e0 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f57440a1ba0 tx=0x7f573400db90 comp rx=0 tx=0).stop 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.936+0000 7f574ad59640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f572803e210 0x7f57280406d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f573c009e50 tx=0x7f573c009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f574c7bc640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f574c7bc640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 0x7f5744130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f574c7bc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f57440a2fe0 con 0x7f5744130250 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 0x7f5744130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient: get_auth_request con 0x7f5744130250 auth_method 0 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:32.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 10 monclient(hunting): handle_auth_done global_id 194117 payload 824 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.942+0000 7f5749d57640 1 -- 172.21.15.164:0/2036518384 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f57440a3370 con 0x7f5744130250 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f5749d57640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 msgr2=0x7f5744130630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f5749d57640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 0x7f5744130630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5734018000 tx=0x7f57340058c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57340047e0 con 0x7f5744130250 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 10 monclient: dump: 2024-09-10T16:23:32.963 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f573401e7c0 con 0x7f5744130250 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:32.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 msgr2=0x7f5744130630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 0x7f5744130630 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5734018000 tx=0x7f57340058c0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 msgr2=0x7f5744130630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.945+0000 7f573affd640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5744130250 0x7f5744130630 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5734018000 tx=0x7f57340058c0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:23:32.965 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 16400, 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.966 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.967 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:32.968 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:32.969 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:32.970 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.971 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.972 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:32.973 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:32.974 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:32.975 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.976 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:32.977 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:2619: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:23:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1549 (ObjNum 129 snap 0 seq_num 1162089392) dirty dne 2024-09-10T16:23:32.978 INFO:tasks.rados.rados.0.smithi165.stdout:2620: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f572803e210 msgr2=0x7f57280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f572803e210 0x7f57280406d0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f573c009e50 tx=0x7f573c009000 comp rx=0 tx=0).stop 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 10 monclient: shutdown 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 msgr2=0x7f574412e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f574000e990 tx=0x7f574000ee50 comp rx=0 tx=0).stop 2024-09-10T16:23:32.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574ad59640 1 -- 172.21.15.164:0/2036518384 reap_dead start 2024-09-10T16:23:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 shutdown_connections 2024-09-10T16:23:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 --2- 172.21.15.164:0/2036518384 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f574406cb70 0x7f574412e400 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 >> 172.21.15.164:0/2036518384 conn(0x7f57440aa900 msgr2=0x7f57440a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 shutdown_connections 2024-09-10T16:23:32.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:32.946+0000 7f574c7bc640 1 -- 172.21.15.164:0/2036518384 wait complete. 2024-09-10T16:23:32.980 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:32.980 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:23:32.980 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:23:33.033 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 864752 2024-09-10T16:23:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.079+0000 7f078afa0640 10 monclient: get_monmap_and_config 2024-09-10T16:23:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.079+0000 7f078afa0640 10 monclient: build_initial_monmap 2024-09-10T16:23:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.079+0000 7f078afa0640 10 monclient: monmap: 2024-09-10T16:23:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:33.080 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:33.080092+0000 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:33.080092+0000 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:33.081 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.079+0000 7f078afa0640 1 Processor -- start 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 1 -- start start 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient: init 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f078406cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0784134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f078413e740 0x7f0784140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-a con 0x7f078413e740 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f078409ce50 con 0x7f0784134f70 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0784135380 con 0x7f078406cb70 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f078406db60 con 0x7f078413e740 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078afa0640 10 monclient(hunting): authenticate will time out at 1791.567711s 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f0788d3c640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f078406cb70 0x7f07840a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:53022/0 (socket says 172.21.15.164:53022) 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f078413e740 0x7f0784140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f0788d3c640 1 -- 172.21.15.164:0/3104500130 learned_addr learned my addr 172.21.15.164:0/3104500130 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f0783fff640 1 --1- 172.21.15.164:0/3104500130 >> v1:172.21.15.164:6789/0 conn(0x7f0784134f70 0x7f078413d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:46210/0 (socket says 172.21.15.164:46210) 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): get_auth_request con 0x7f078413e740 auth_method 0 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.080+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 10 monclient(hunting): handle_auth_done global_id 164370 payload 274 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 1 -- 172.21.15.164:0/3104500130 >> v1:172.21.15.165:6789/0 conn(0x7f078406cb70 legacy=0x7f07840a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 1 -- 172.21.15.164:0/3104500130 >> v1:172.21.15.164:6789/0 conn(0x7f0784134f70 legacy=0x7f078413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 10 monclient: _finish_hunting 0 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 1 monclient: found mon.noname-a 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.081+0000 7f078953d640 1 -- 172.21.15.164:0/3104500130 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078409cb70 con 0x7f078413e740 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.083+0000 7f078953d640 1 -- 172.21.15.164:0/3104500130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 msgr2=0x7f0784140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.083+0000 7f078953d640 1 --2- 172.21.15.164:0/3104500130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784140b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f077c009dc0 tx=0x7f077c02a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9f5a1f212e3059a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 1 -- 172.21.15.164:0/3104500130 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c007a30 con 0x7f078413e740 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: dump: 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.092 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: _finish_auth 0 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: _check_auth_tickets 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 1 -- 172.21.15.164:0/3104500130 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f077c007bd0 con 0x7f078413e740 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 5 monclient: authenticate success, global_id 164370 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 1 -- 172.21.15.164:0/3104500130 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f077c031400 con 0x7f078413e740 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 4 set_mon_vals no callback set 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f07837fe640 10 monclient: dump: 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 10 monclient: shutdown 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 -- 172.21.15.164:0/3104500130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 msgr2=0x7f0784140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 --2- 172.21.15.164:0/3104500130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784140b30 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f077c009dc0 tx=0x7f077c02a4d0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 -- 172.21.15.164:0/3104500130 shutdown_connections 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 --2- 172.21.15.164:0/3104500130 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784140b30 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 -- 172.21.15.164:0/3104500130 >> 172.21.15.164:0/3104500130 conn(0x7f07840aa900 msgr2=0x7f07840aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 -- 172.21.15.164:0/3104500130 shutdown_connections 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.084+0000 7f078afa0640 1 -- 172.21.15.164:0/3104500130 wait complete. 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.085+0000 7f078afa0640 10 monclient: build_initial_monmap 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.085+0000 7f078afa0640 10 monclient: monmap: 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:33.096 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.085+0000 7f078afa0640 1 Processor -- start 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.085+0000 7f078afa0640 1 -- start start 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.085+0000 7f078afa0640 10 monclient: init 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 0x7f078413cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-c con 0x7f078406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0784134f70 0x7f078413d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-b con 0x7f0784134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784130180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): picked mon.noname-a con 0x7f078413e740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f07840a4db0 con 0x7f078413e740 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f078406cf50 con 0x7f0784134f70 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0784141070 con 0x7f078406cb70 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078afa0640 10 monclient(hunting): authenticate will time out at 1791.573250s 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 0x7f078413cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784130180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784130180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58762/0 (socket says 172.21.15.164:58762) 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 0x7f078413cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46612/0 (socket says 172.21.15.164:46612) 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 1 -- 172.21.15.164:0/1908104638 learned_addr learned my addr 172.21.15.164:0/1908104638 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): get_auth_request con 0x7f078413e740 auth_method 0 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): get_auth_request con 0x7f078406cb70 auth_method 0 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0784134f70 0x7f078413d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): get_auth_request con 0x7f0784134f70 auth_method 0 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient(hunting): handle_auth_done global_id 194120 payload 824 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0784134f70 msgr2=0x7f078413d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0784134f70 0x7f078413d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 msgr2=0x7f0784130180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784130180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient: _finish_hunting 0 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 monclient: found mon.noname-c 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:33.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f0788d3c640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f078413a110 con 0x7f078406cb70 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.086+0000 7f078953d640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f078413e740 0x7f0784130180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.088+0000 7f0788d3c640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 msgr2=0x7f078413cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.088+0000 7f0788d3c640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 0x7f078413cdf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f077400b410 tx=0x7f077400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774015020 con 0x7f078406cb70 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: dump: 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: _finish_auth 0 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: _check_auth_tickets 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 5 monclient: authenticate success, global_id 194120 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 monclient: _renew_subs 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0774011a60 con 0x7f078406cb70 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f078413a450 con 0x7f078406cb70 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:33.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0774011e50 con 0x7f078406cb70 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 10 monclient: dump: 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 monclient: _renew_subs 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f0782ffd640 4 set_mon_vals no callback set 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0784138db0 con 0x7f078406cb70 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 0x7f07841306c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f078413a8c0 con 0x7f07841cd370 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.089+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0774018020 con 0x7f078406cb70 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 0x7f07841306c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient: get_auth_request con 0x7f07841cd370 auth_method 0 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f07817fa640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f076403e3e0 0x7f07640408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(590..590 src has 481..590) ==== 4787+0+0 (secure 0 0 0) 0x7f0774052bd0 con 0x7f078406cb70 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f078953d640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f076403e3e0 0x7f07640408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f078953d640 10 monclient: get_auth_request con 0x7f076403e3e0 auth_method 0 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 10 monclient(hunting): handle_auth_done global_id 154417 payload 824 2024-09-10T16:23:33.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.090+0000 7f0783fff640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07840a46a0 con 0x7f07841cd370 2024-09-10T16:23:33.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f0783fff640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 msgr2=0x7f07841306c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f0783fff640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 0x7f07841306c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f078413e020 tx=0x7f076c00eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c01a070 con 0x7f07841cd370 2024-09-10T16:23:33.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f07817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f07817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f078953d640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f076403e3e0 0x7f07640408a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0784132030 tx=0x7f077c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.092+0000 7f07817fa640 10 monclient: dump: 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f076c004b40 con 0x7f07841cd370 2024-09-10T16:23:33.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 msgr2=0x7f07841306c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 0x7f07841306c0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f078413e020 tx=0x7f076c00eb90 comp rx=0 tx=0).stop 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 msgr2=0x7f07841306c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.093+0000 7f07817fa640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841cd370 0x7f07841306c0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f078413e020 tx=0x7f076c00eb90 comp rx=0 tx=0).stop 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f078afa0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f078afa0640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 0x7f0784130c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f078afa0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f07840770f0 con 0x7f07841391b0 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 0x7f0784130c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient: get_auth_request con 0x7f07841391b0 auth_method 0 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 10 monclient(hunting): handle_auth_done global_id 154420 payload 824 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.099+0000 7f0783fff640 1 -- 172.21.15.164:0/1908104638 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f07841d84b0 con 0x7f07841391b0 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f0783fff640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 msgr2=0x7f0784130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f0783fff640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 0x7f0784130c00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f076c017000 tx=0x7f076c004930 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076c01a040 con 0x7f07841391b0 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 10 monclient: dump: 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f076c01ca30 con 0x7f07841391b0 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 msgr2=0x7f0784130c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 0x7f0784130c00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f076c017000 tx=0x7f076c004930 comp rx=0 tx=0).stop 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 msgr2=0x7f0784130c00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f07817fa640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f07841391b0 0x7f0784130c00 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f076c017000 tx=0x7f076c004930 comp rx=0 tx=0).stop 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 231653, 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:33.112 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:33.113 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:33.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:33.115 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:33.116 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:33.117 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:33.119 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:33.120 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:33.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:33.122 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:33.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:33.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:33.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:33.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:33.125 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:33.125 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f076403e3e0 msgr2=0x7f07640408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f076403e3e0 0x7f07640408a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f0784132030 tx=0x7f077c02f000 comp rx=0 tx=0).stop 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 10 monclient: shutdown 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 msgr2=0x7f078413cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 --2- 172.21.15.164:0/1908104638 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f078406cb70 0x7f078413cdf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f077400b410 tx=0x7f077400b8d0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078953d640 1 -- 172.21.15.164:0/1908104638 reap_dead start 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078953d640 1 -- 172.21.15.164:0/1908104638 reap_dead start 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 shutdown_connections 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 >> 172.21.15.164:0/1908104638 conn(0x7f07840aa900 msgr2=0x7f07840a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:33.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 shutdown_connections 2024-09-10T16:23:33.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.102+0000 7f078afa0640 1 -- 172.21.15.164:0/1908104638 wait complete. 2024-09-10T16:23:33.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:33.128 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:23:33.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:23:33.157 INFO:tasks.rados.rados.0.smithi165.stdout:2617: finishing copy_from racing read to smithi16534841-24 2024-09-10T16:23:33.157 INFO:tasks.rados.rados.0.smithi165.stdout:2617: finishing copy_from to smithi16534841-24 2024-09-10T16:23:33.157 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1387 (ObjNum 911 snap 278 seq_num 911) dirty exists 2024-09-10T16:23:33.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.224+0000 7f7d6d533640 10 monclient: get_monmap_and_config 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.224+0000 7f7d6d533640 10 monclient: build_initial_monmap 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.224+0000 7f7d6d533640 10 monclient: monmap: 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:33.225274+0000 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:33.225274+0000 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:33.227 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:33.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.224+0000 7f7d6d533640 1 Processor -- start 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 1 -- start start 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient: init 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:23:33.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7d680a4540 0x7f7d680a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-c con 0x7f7d680a4540 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7d681385c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7d680a60b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d680a71c0 con 0x7f7d680a60b0 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d68135400 con 0x7f7d681385c0 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6809c9f0 con 0x7f7d680a4540 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d6d533640 10 monclient(hunting): authenticate will time out at 1791.712556s 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d677fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7d680a60b0 0x7f7d68134cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:46220/0 (socket says 172.21.15.164:46220) 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d677fe640 1 -- 172.21.15.164:0/2011138483 learned_addr learned my addr 172.21.15.164:0/2011138483 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:33.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2011138483 >> v2:172.21.15.165:3300/0 conn(0x7f7d680a4540 0x7f7d680a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d66ffd640 10 monclient(hunting): get_auth_request con 0x7f7d680a4540 auth_method 0 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.225+0000 7f7d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient(hunting): handle_auth_done global_id 154423 payload 274 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 1 -- 172.21.15.164:0/2011138483 >> v1:172.21.15.165:6789/0 conn(0x7f7d681385c0 legacy=0x7f7d681345e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 1 -- 172.21.15.164:0/2011138483 >> v1:172.21.15.164:6789/0 conn(0x7f7d680a60b0 legacy=0x7f7d68134cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 1 monclient: found mon.noname-c 2024-09-10T16:23:33.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:33.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:23:33.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.226+0000 7f7d66ffd640 1 -- 172.21.15.164:0/2011138483 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6809ccd0 con 0x7f7d680a4540 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d66ffd640 1 -- 172.21.15.164:0/2011138483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 msgr2=0x7f7d680a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2011138483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d680a4920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7d580097d0 tx=0x7f7d5802a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e150c1ab747635ae server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 1 -- 172.21.15.164:0/2011138483 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58037070 con 0x7f7d680a4540 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:33.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: dump: 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: _finish_auth 0 2024-09-10T16:23:33.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 1 -- 172.21.15.164:0/2011138483 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d58032440 con 0x7f7d680a4540 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 5 monclient: authenticate success, global_id 154423 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 4 set_mon_vals no callback set 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 1 -- 172.21.15.164:0/2011138483 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d58032810 con 0x7f7d680a4540 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d65ffb640 10 monclient: dump: 2024-09-10T16:23:33.240 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 10 monclient: shutdown 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 -- 172.21.15.164:0/2011138483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 msgr2=0x7f7d680a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2011138483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d680a4920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7d580097d0 tx=0x7f7d5802a000 comp rx=0 tx=0).stop 2024-09-10T16:23:33.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 -- 172.21.15.164:0/2011138483 shutdown_connections 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2011138483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d680a4920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 -- 172.21.15.164:0/2011138483 >> 172.21.15.164:0/2011138483 conn(0x7f7d680aaa00 msgr2=0x7f7d680aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 -- 172.21.15.164:0/2011138483 shutdown_connections 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.229+0000 7f7d6d533640 1 -- 172.21.15.164:0/2011138483 wait complete. 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.230+0000 7f7d6d533640 10 monclient: build_initial_monmap 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.230+0000 7f7d6d533640 10 monclient: monmap: 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.230+0000 7f7d6d533640 1 Processor -- start 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.230+0000 7f7d6d533640 1 -- start start 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.230+0000 7f7d6d533640 10 monclient: init 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d681cd050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-b con 0x7f7d680a4540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-a con 0x7f7d680a60b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:33.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 0x7f7d6809e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): picked mon.noname-c con 0x7f7d680a3ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7d68137dd0 con 0x7f7d680a60b0 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7d680a4e60 con 0x7f7d680a4540 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7d68137f50 con 0x7f7d680a3ca0 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d6d533640 10 monclient(hunting): authenticate will time out at 1791.718348s 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:58772/0 (socket says 172.21.15.164:58772) 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 -- 172.21.15.164:0/2489426160 learned_addr learned my addr 172.21.15.164:0/2489426160 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:33.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): get_auth_request con 0x7f7d680a60b0 auth_method 0 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 0x7f7d6809e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): get_auth_request con 0x7f7d680a3ca0 auth_method 0 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:33.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d681cd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 10 monclient(hunting): get_auth_request con 0x7f7d680a4540 auth_method 0 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient(hunting): handle_auth_done global_id 164376 payload 824 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 -- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 msgr2=0x7f7d6809e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 0x7f7d6809e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 -- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 msgr2=0x7f7d681cd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d681cd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient: _finish_hunting 0 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 monclient: found mon.noname-a 2024-09-10T16:23:33.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d667fc640 1 -- 172.21.15.164:0/2489426160 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d680a1c90 con 0x7f7d680a60b0 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d677fe640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 0x7f7d6809e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.231+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d681cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d667fc640 1 -- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 msgr2=0x7f7d680a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d667fc640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00c420 tx=0x7f7d5c00c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c007020 con 0x7f7d680a60b0 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: dump: 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.247 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c0071c0 con 0x7f7d680a60b0 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 5 monclient: authenticate success, global_id 164376 2024-09-10T16:23:33.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 10 monclient: _renew_subs 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c007590 con 0x7f7d680a60b0 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d680a0a50 con 0x7f7d680a60b0 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d4f7fe640 10 monclient: dump: 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:33.249 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d657fa640 4 set_mon_vals no callback set 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 10 monclient: _renew_subs 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.234+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d68132f70 con 0x7f7d680a60b0 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7d5c019020 con 0x7f7d680a60b0 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d4f7fe640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 0x7f7d540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d6d533640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:33.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d6d533640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d6d533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(590..590 src has 481..590) ==== 4787+0+0 (secure 0 0 0) 0x7f7d5c0524e0 con 0x7f7d680a60b0 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d680a4920 con 0x7f7d680a60b0 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 0x7f7d540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.235+0000 7f7d66ffd640 10 monclient: get_auth_request con 0x7f7d5403e210 auth_method 0 2024-09-10T16:23:33.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.238+0000 7f7d66ffd640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 0x7f7d540406d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d6809d3d0 tx=0x7f7d5802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.240+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7d5c01a370 con 0x7f7d680a60b0 2024-09-10T16:23:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.240+0000 7f7d4f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:33.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.240+0000 7f7d4f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:33.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.481+0000 7f7d6d533640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:23:33.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.481+0000 7f7d6d533640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:23:33.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.481+0000 7f7d6d533640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:33.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.481+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7d680a05d0 con 0x7f7d680a60b0 2024-09-10T16:23:33.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f7d4f7fe640 1 -- 172.21.15.164:0/2489426160 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7d5c0117f0 con 0x7f7d680a60b0 2024-09-10T16:23:33.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f7d4f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:23:33.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f7d4f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:33.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=11,logm=51} crc {auth=3032461610,config=1228700967,health=406487438,kv=805391194,logm=1770838921}) 2024-09-10T16:23:33.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3719307618,config=2586702516,health=1273036416,kv=3029507449,logm=3870840689}) 2024-09-10T16:23:33.484 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=1,health=11,kv=11,logm=51} crc {auth=3032461610,config=1228700967,health=406487438,kv=805391194,logm=1770838921}) 2024-09-10T16:23:33.485 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.482+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=110439343,config=2586702516,health=1273036416,kv=805391194,logm=1553314972}) 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679685255}) 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=891310336}) 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1679685255}) 2024-09-10T16:23:33.486 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.483+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=451991539}) 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 msgr2=0x7f7d540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 0x7f7d540406d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f7d6809d3d0 tx=0x7f7d5802e000 comp rx=0 tx=0).stop 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 10 monclient: shutdown 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 msgr2=0x7f7d680a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c00c420 tx=0x7f7d5c00c8e0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 shutdown_connections 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7d5403e210 0x7f7d540406d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7d680a3ca0 0x7f7d6809e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7d680a60b0 0x7f7d680a3760 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.484+0000 7f7d6d533640 1 --2- 172.21.15.164:0/2489426160 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7d680a4540 0x7f7d681cd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 >> 172.21.15.164:0/2489426160 conn(0x7f7d680aaa00 msgr2=0x7f7d680a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:33.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 shutdown_connections 2024-09-10T16:23:33.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.488 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344471815}) 2024-09-10T16:23:33.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f7d6d533640 1 -- 172.21.15.164:0/2489426160 wait complete. 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176910610}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344471815}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.485+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3727164420}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905427899}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615129502}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905427899}) 2024-09-10T16:23:33.491 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.486+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3641550364}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330849254}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1539652902}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2330849254}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.487+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3235245266}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697273380}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1022169117}) 2024-09-10T16:23:33.492 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697273380}) 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.488+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4107126683}) 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283579831}) 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1369448699}) 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3283579831}) 2024-09-10T16:23:33.493 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.489+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2641375669}) 2024-09-10T16:23:33.495 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.495 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371870692}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565479178}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371870692}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.490+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3838529687}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004889759}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=63379560}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4004889759}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.491+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823194694}) 2024-09-10T16:23:33.496 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248051356}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241425934}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=248051356}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.492+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2905122769}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273919035}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3535326026}) 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.497 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4273919035}) 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.493+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3809869787}) 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934880615}) 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2790343214}) 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=934880615}) 2024-09-10T16:23:33.498 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.494+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=98051691}) 2024-09-10T16:23:33.499 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.499 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547854824}) 2024-09-10T16:23:33.499 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4152057003}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3547854824}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.495+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3888600478}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733837891}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=165757450}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733837891}) 2024-09-10T16:23:33.500 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.496+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3750105216}) 2024-09-10T16:23:33.501 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:23:33.506 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.506 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928893978}) 2024-09-10T16:23:33.506 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3296211275}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928893978}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.497+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2965989869}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727954443}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3272922279}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727954443}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.498+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4223393186}) 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.507 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696534532}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1487858819}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=696534532}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.499+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2375527517}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3238625827}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763212081}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3238625827}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.500+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4071523156}) 2024-09-10T16:23:33.508 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627426608}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=402686003}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=627426608}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.501+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125225361}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3818416937}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1093853548}) 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.509 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3818416937}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.502+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1255522977}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434021169}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2494556177}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434021169}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.503+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1981573728}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654659263}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1728308221}) 2024-09-10T16:23:33.510 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654659263}) 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.504+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=13156706}) 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=564956628}) 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=375725950}) 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=564956628}) 2024-09-10T16:23:33.511 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1690263420}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205905939}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258397331}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.505+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205905939}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1017941651}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755231710}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2848716814}) 2024-09-10T16:23:33.514 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=755231710}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.506+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2401792127}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893633709}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1474789847}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2893633709}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.507+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692646635}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977806885}) 2024-09-10T16:23:33.515 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191413027}) 2024-09-10T16:23:33.516 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.516 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2977806885}) 2024-09-10T16:23:33.516 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.508+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1817325142}) 2024-09-10T16:23:33.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.517 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259873626}) 2024-09-10T16:23:33.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217275534}) 2024-09-10T16:23:33.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=259873626}) 2024-09-10T16:23:33.518 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.516+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4269884508}) 2024-09-10T16:23:33.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280850906}) 2024-09-10T16:23:33.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013422575}) 2024-09-10T16:23:33.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.519 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2280850906}) 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.517+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3092395401}) 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061570838}) 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2172233467}) 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3061570838}) 2024-09-10T16:23:33.520 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.518+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232268680}) 2024-09-10T16:23:33.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292435971}) 2024-09-10T16:23:33.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=611564444}) 2024-09-10T16:23:33.522 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3292435971}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.519+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964530391}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426120582}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2422446847}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426120582}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.520+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2928099920}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858693969}) 2024-09-10T16:23:33.523 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425048249}) 2024-09-10T16:23:33.524 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.524 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858693969}) 2024-09-10T16:23:33.524 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3460724068}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311533713}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2750791312}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.521+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2311533713}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1702292120}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551491354}) 2024-09-10T16:23:33.530 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1548226603}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3551491354}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.522+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550921496}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739241322}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333200089}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1739241322}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.523+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2291031809}) 2024-09-10T16:23:33.531 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364165902}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594554372}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1364165902}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.524+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3732430275}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3963542711}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=14286695}) 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3963542711}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.525+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2473688474}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794922203}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3604180601}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=794922203}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.526+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3441138629}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712382472}) 2024-09-10T16:23:33.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425649669}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3712382472}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.527+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=873852812}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313798462}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235458230}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1313798462}) 2024-09-10T16:23:33.534 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.528+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3745508272}) 2024-09-10T16:23:33.541 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.541 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377373288}) 2024-09-10T16:23:33.541 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=724207998}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1377373288}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.529+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2674323163}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374472189}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2912538353}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374472189}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.530+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783371350}) 2024-09-10T16:23:33.542 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302561353}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3756553756}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1302561353}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.531+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1852173084}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758841561}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4235372656}) 2024-09-10T16:23:33.543 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3758841561}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.532+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=312890480}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027930185}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2947409933}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027930185}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.533+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1261506938}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619179333}) 2024-09-10T16:23:33.544 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3796708027}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3619179333}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.534+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168458385}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143276881}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1849571063}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4143276881}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.535+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815280765}) 2024-09-10T16:23:33.545 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404309625}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=92060531}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1404309625}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.536+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702741155}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323551501}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513511812}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=323551501}) 2024-09-10T16:23:33.546 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.537+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975273343}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388200901}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1248248468}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=388200901}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.538+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1162600065}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968551262}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1358772023}) 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.547 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1968551262}) 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010700923}) 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.539+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297101240}) 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3687112924}) 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297101240}) 2024-09-10T16:23:33.548 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3908337687}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165207281}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2793011531}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1165207281}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.540+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3400400671}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93100173}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2659715130}) 2024-09-10T16:23:33.551 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93100173}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.541+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1558909809}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399879490}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2574848516}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1399879490}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.542+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1071541986}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=117113895,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=629821609,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=2567526607,osd_pg_creating=3567796918,osd_snap=3249825209}) 2024-09-10T16:23:33.552 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=10,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=41} crc {logm=1752304944,mdsmap=931935893,mgr=1154067016,mgr_metadata=2169216343,mgrstat=1127919619,mon_config_key=884308292,monmap=1433040829,nvmeofgw=598337687,osd_metadata=4005186995,osd_pg_creating=3567796918,osd_snap=4103478389}) 2024-09-10T16:23:33.553 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.553 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=35} crc {logm=117113895,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=629821609,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=2567526607,osd_pg_creating=3567796918,osd_snap=3249825209}) 2024-09-10T16:23:33.553 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.544+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=11,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=2,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=38} crc {logm=117113895,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=629821609,mon_config_key=665552247,monmap=3140514009,nvmeofgw=1596038039,osd_metadata=3014334576,osd_pg_creating=3567796918,osd_snap=3256701758}) 2024-09-10T16:23:33.577 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.577 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=75,osdmap=25} crc {osd_snap=440035572,osdmap=3620142205}) 2024-09-10T16:23:33.578 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=75,osdmap=25} crc {osd_snap=2246348132,osdmap=1256928091}) 2024-09-10T16:23:33.578 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.578 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=75,osdmap=25} crc {osd_snap=440035572,osdmap=3620142205}) 2024-09-10T16:23:33.578 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.576+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=74,osdmap=26} crc {osd_snap=388901972,osdmap=3442720315}) 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1904926597}) 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=547104793}) 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1904926597}) 2024-09-10T16:23:33.663 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.661+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2199384550}) 2024-09-10T16:23:33.897 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.897 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=4021118449}) 2024-09-10T16:23:33.897 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=53} crc {osdmap=179142119}) 2024-09-10T16:23:33.898 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:23:33.898 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=4021118449}) 2024-09-10T16:23:33.898 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:23:33.896+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=1449906679}) 2024-09-10T16:23:34.502 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:23:34.502 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2616: done (5 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2617: done (4 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2618: done (3 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2619: done (2 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2620: done (1 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2621: done (0 left) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2622: read oid 39 snap 281 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2622: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:23:34.557 INFO:tasks.rados.rados.0.smithi165.stdout:2623: delete oid 18 current snap is 285 2024-09-10T16:23:34.561 INFO:tasks.rados.rados.0.smithi165.stdout:2622: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:23:34.577 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1916742 2024-09-10T16:23:34.589 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.606+0000 7fab13840640 10 monclient: get_monmap_and_config 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.606+0000 7fab13840640 10 monclient: build_initial_monmap 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.606+0000 7fab13840640 10 monclient: monmap: 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.608 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:34.607518+0000 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:34.607518+0000 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 Processor -- start 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 -- start start 2024-09-10T16:23:34.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient: init 2024-09-10T16:23:34.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:34.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:23:34.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab0c09d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:34.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fab0c09ab50 0x7fab0c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fab0c09e420 0x7fab0c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): picked mon.noname-c con 0x7fab0c09e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab0c0a0870 con 0x7fab0c09d930 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fab0c09d3e0 con 0x7fab0c09ab50 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fab0c074c70 con 0x7fab0c09e420 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.607+0000 7fab13840640 10 monclient(hunting): authenticate will time out at 1793.094922s 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fab0c09ab50 0x7fab0c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab115dc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fab0c09d930 0x7fab0c09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52640/0 (socket says 172.21.15.164:52640) 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab115dc640 1 -- 172.21.15.164:0/300593929 learned_addr learned my addr 172.21.15.164:0/300593929 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:34.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): get_auth_request con 0x7fab0c09ab50 auth_method 0 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 1 --2- 172.21.15.164:0/300593929 >> v2:172.21.15.165:3300/0 conn(0x7fab0c09e420 0x7fab0c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): get_auth_request con 0x7fab0c09e420 auth_method 0 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient(hunting): handle_auth_done global_id 164379 payload 274 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 -- 172.21.15.164:0/300593929 >> v2:172.21.15.165:3300/0 conn(0x7fab0c09e420 msgr2=0x7fab0c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 --2- 172.21.15.164:0/300593929 >> v2:172.21.15.165:3300/0 conn(0x7fab0c09e420 0x7fab0c13bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 -- 172.21.15.164:0/300593929 >> v1:172.21.15.164:6789/0 conn(0x7fab0c09d930 legacy=0x7fab0c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient: _finish_hunting 0 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 monclient: found mon.noname-a 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.608+0000 7fab10ddb640 1 -- 172.21.15.164:0/300593929 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c09ecb0 con 0x7fab0c09ab50 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab10ddb640 1 -- 172.21.15.164:0/300593929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 msgr2=0x7fab0c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab10ddb640 1 --2- 172.21.15.164:0/300593929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c1397c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fab0c09af60 tx=0x7faafc02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eca6c6b5bffb0fe server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 1 -- 172.21.15.164:0/300593929 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc037070 con 0x7fab0c09ab50 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:34.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: dump: 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: _finish_auth 0 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:34.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab13840640 5 monclient: authenticate success, global_id 164379 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab13840640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 1 -- 172.21.15.164:0/300593929 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faafc032440 con 0x7fab0c09ab50 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab13840640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab13840640 4 set_mon_vals no callback set 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 1 -- 172.21.15.164:0/300593929 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc032810 con 0x7fab0c09ab50 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.611+0000 7fab03fff640 10 monclient: dump: 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 10 monclient: shutdown 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 -- 172.21.15.164:0/300593929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 msgr2=0x7fab0c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 --2- 172.21.15.164:0/300593929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c1397c0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fab0c09af60 tx=0x7faafc02a000 comp rx=0 tx=0).stop 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 -- 172.21.15.164:0/300593929 shutdown_connections 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 --2- 172.21.15.164:0/300593929 >> v2:172.21.15.165:3300/0 conn(0x7fab0c09e420 0x7fab0c13bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 --2- 172.21.15.164:0/300593929 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c1397c0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 -- 172.21.15.164:0/300593929 >> 172.21.15.164:0/300593929 conn(0x7fab0c0a3110 msgr2=0x7fab0c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:34.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 -- 172.21.15.164:0/300593929 shutdown_connections 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 1 -- 172.21.15.164:0/300593929 wait complete. 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 10 monclient: build_initial_monmap 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.612+0000 7fab13840640 10 monclient: monmap: 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 Processor -- start 2024-09-10T16:23:34.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 -- start start 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient: init 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): picked mon.noname-a con 0x7fab0c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): picked mon.noname-c con 0x7fab0c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab0c09e420 0x7fab0c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): picked mon.noname-b con 0x7fab0c09e420 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c0a2940 con 0x7fab0c09ab50 2024-09-10T16:23:34.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c09ef30 con 0x7fab0c09e420 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fab0c0a2ac0 con 0x7fab0c09d930 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab13840640 10 monclient(hunting): authenticate will time out at 1793.100527s 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54024/0 (socket says 172.21.15.164:54024) 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59672/0 (socket says 172.21.15.164:59672) 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 -- 172.21.15.164:0/3804198658 learned_addr learned my addr 172.21.15.164:0/3804198658 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): get_auth_request con 0x7fab0c09d930 auth_method 0 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): get_auth_request con 0x7fab0c09ab50 auth_method 0 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab0c09e420 0x7fab0c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): get_auth_request con 0x7fab0c09e420 auth_method 0 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab11ddd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient(hunting): handle_auth_done global_id 194126 payload 824 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 -- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab0c09e420 msgr2=0x7fab0c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab0c09e420 0x7fab0c06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 -- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 msgr2=0x7fab0c134e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient: _finish_hunting 0 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 monclient: found mon.noname-c 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab10ddb640 1 -- 172.21.15.164:0/3804198658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c1353c0 con 0x7fab0c09d930 2024-09-10T16:23:34.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.613+0000 7fab115dc640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab10ddb640 1 -- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 msgr2=0x7fab0c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab10ddb640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faafc009890 tx=0x7faafc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc041020 con 0x7fab0c09d930 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:34.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: dump: 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: _finish_auth 0 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faafc03caa0 con 0x7fab0c09d930 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 5 monclient: authenticate success, global_id 194126 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 10 monclient: _renew_subs 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faafc037040 con 0x7fab0c09d930 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c13b170 con 0x7fab0c09d930 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab01ffb640 10 monclient: dump: 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab037fe640 4 set_mon_vals no callback set 2024-09-10T16:23:34.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 10 monclient: _renew_subs 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.616+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c09d620 con 0x7fab0c09d930 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faafc044020 con 0x7fab0c09d930 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab13840640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab13840640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab13840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab0c09dd10 con 0x7fab0c09d930 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab01ffb640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 0x7faaf00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7faafc073160 con 0x7fab0c09d930 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab115dc640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 0x7faaf00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.617+0000 7fab115dc640 10 monclient: get_auth_request con 0x7faaf003e210 auth_method 0 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.620+0000 7fab115dc640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 0x7faaf00406d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab08007c90 tx=0x7fab08007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:34.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.622+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7faafc04b070 con 0x7fab0c09d930 2024-09-10T16:23:34.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.622+0000 7fab01ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:34.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.622+0000 7fab01ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:34.636 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1608042 2024-09-10T16:23:34.656 INFO:tasks.rados.rados.0.smithi165.stdout:2622: done (1 left) 2024-09-10T16:23:34.656 INFO:tasks.rados.rados.0.smithi165.stdout:2623: done (0 left) 2024-09-10T16:23:34.656 INFO:tasks.rados.rados.0.smithi165.stdout:2624: delete oid 18 current snap is 285 2024-09-10T16:23:34.659 INFO:tasks.rados.rados.0.smithi165.stdout:2624: done (0 left) 2024-09-10T16:23:34.659 INFO:tasks.rados.rados.0.smithi165.stdout:2625: snap_remove snap 263 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab13840640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab13840640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab13840640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fab0c06daa0 con 0x7fab0c09d930 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab01ffb640 1 -- 172.21.15.164:0/3804198658 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7faafc071b50 con 0x7fab0c09d930 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab01ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:34.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.866+0000 7fab01ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:34.869 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 msgr2=0x7faaf00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 0x7faaf00406d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fab08007c90 tx=0x7fab08007be0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 10 monclient: shutdown 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 msgr2=0x7fab0c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7faafc009890 tx=0x7faafc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 shutdown_connections 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faaf003e210 0x7faaf00406d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fab0c09e420 0x7fab0c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fab0c09d930 0x7fab0c06e7d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 --2- 172.21.15.164:0/3804198658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fab0c09ab50 0x7fab0c134e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 >> 172.21.15.164:0/3804198658 conn(0x7fab0c0a3110 msgr2=0x7fab0c0a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:34.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 shutdown_connections 2024-09-10T16:23:34.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.869+0000 7fab13840640 1 -- 172.21.15.164:0/3804198658 wait complete. 2024-09-10T16:23:34.881 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:23:34.881 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:23:34.881 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:23:34.881 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:23:34.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.978+0000 7f23fc59b640 10 monclient: get_monmap_and_config 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.978+0000 7f23fc59b640 10 monclient: build_initial_monmap 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.978+0000 7f23fc59b640 10 monclient: monmap: 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:34.979278+0000 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:34.979278+0000 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:34.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:34.981 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:34.981 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:34.981 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:34.981 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.978+0000 7f23fc59b640 1 Processor -- start 2024-09-10T16:23:34.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 1 -- start start 2024-09-10T16:23:34.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient: init 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f23f41329f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 0x7f23f413ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-b con 0x7f23f40a2f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23f413b2b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f41334e0 con 0x7f23f413b2b0 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f40a0730 con 0x7f23f41329f0 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23f40a21a0 con 0x7f23f40a2f10 2024-09-10T16:23:34.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fc59b640 10 monclient(hunting): authenticate will time out at 1793.466811s 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23f9b36640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 0x7f23f413ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fab38640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f23f413b2b0 0x7f23f413d6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52642/0 (socket says 172.21.15.164:52642) 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23fab38640 1 -- 172.21.15.164:0/420015974 learned_addr learned my addr 172.21.15.164:0/420015974 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23f9b36640 10 monclient(hunting): get_auth_request con 0x7f23f40a2f10 auth_method 0 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23f9b36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23f9b36640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.979+0000 7f23f9b36640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:34.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient(hunting): handle_auth_done global_id 194129 payload 274 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 1 -- 172.21.15.164:0/420015974 >> v1:172.21.15.165:6789/0 conn(0x7f23f41329f0 legacy=0x7f23f4132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 1 -- 172.21.15.164:0/420015974 >> v1:172.21.15.164:6789/0 conn(0x7f23f413b2b0 legacy=0x7f23f413d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient: _finish_hunting 0 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 1 monclient: found mon.noname-b 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:23:34.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.980+0000 7f23f9b36640 1 -- 172.21.15.164:0/420015974 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f40a2480 con 0x7f23f40a2f10 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.982+0000 7f23f9b36640 1 -- 172.21.15.164:0/420015974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 msgr2=0x7f23f413ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.982+0000 7f23f9b36640 1 --2- 172.21.15.164:0/420015974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 0x7f23f413ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23e80097d0 tx=0x7f23e8029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cc59f5911ac528b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 1 -- 172.21.15.164:0/420015974 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8037070 con 0x7f23f40a2f10 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: dump: 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.995 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: _finish_auth 0 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: _check_auth_tickets 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:34.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 1 -- 172.21.15.164:0/420015974 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e8032440 con 0x7f23f40a2f10 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 5 monclient: authenticate success, global_id 194129 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 4 set_mon_vals no callback set 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 1 -- 172.21.15.164:0/420015974 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8032810 con 0x7f23f40a2f10 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23f9335640 10 monclient: dump: 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:34.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 10 monclient: shutdown 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:34.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 -- 172.21.15.164:0/420015974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 msgr2=0x7f23f413ad70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 --2- 172.21.15.164:0/420015974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 0x7f23f413ad70 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f23e80097d0 tx=0x7f23e8029fb0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 -- 172.21.15.164:0/420015974 shutdown_connections 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 --2- 172.21.15.164:0/420015974 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f40a2f10 0x7f23f413ad70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 -- 172.21.15.164:0/420015974 >> 172.21.15.164:0/420015974 conn(0x7f23f40a7320 msgr2=0x7f23f40a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 -- 172.21.15.164:0/420015974 shutdown_connections 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.983+0000 7f23fc59b640 1 -- 172.21.15.164:0/420015974 wait complete. 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.984+0000 7f23fc59b640 10 monclient: build_initial_monmap 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.984+0000 7f23fc59b640 10 monclient: monmap: 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:34.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.984+0000 7f23fc59b640 1 Processor -- start 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.984+0000 7f23fc59b640 1 -- start start 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.984+0000 7f23fc59b640 10 monclient: init 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23f40a2f10 0x7f23f4131500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-b con 0x7f23f40a2f10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-c con 0x7f23f41329f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413b2b0 0x7f23f4139770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): picked mon.noname-a con 0x7f23f413b2b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23f40a1a80 con 0x7f23f413b2b0 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23f40a1900 con 0x7f23f40a2f10 2024-09-10T16:23:35.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f23f413edc0 con 0x7f23f41329f0 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fc59b640 10 monclient(hunting): authenticate will time out at 1793.472260s 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54036/0 (socket says 172.21.15.164:54036) 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 -- 172.21.15.164:0/3828654520 learned_addr learned my addr 172.21.15.164:0/3828654520 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): get_auth_request con 0x7f23f41329f0 auth_method 0 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413b2b0 0x7f23f4139770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23f40a2f10 0x7f23f4131500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): get_auth_request con 0x7f23f413b2b0 auth_method 0 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): get_auth_request con 0x7f23f40a2f10 auth_method 0 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient(hunting): handle_auth_done global_id 194132 payload 824 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23f40a2f10 msgr2=0x7f23f4131500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23f40a2f10 0x7f23f4131500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413b2b0 msgr2=0x7f23f4139770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413b2b0 0x7f23f4139770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 monclient: found mon.noname-c 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23f9b36640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f413ab30 con 0x7f23f41329f0 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.985+0000 7f23fab38640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413b2b0 0x7f23f4139770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.987+0000 7f23f9b36640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 msgr2=0x7f23f4131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.987+0000 7f23f9b36640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009720 tx=0x7f23e80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8041070 con 0x7f23f41329f0 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: dump: 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.006 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e803c660 con 0x7f23f41329f0 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 5 monclient: authenticate success, global_id 194132 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 monclient: _renew_subs 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f40a0210 con 0x7f23f41329f0 2024-09-10T16:23:35.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e8037040 con 0x7f23f41329f0 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f2ffd640 10 monclient: dump: 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.008 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23f8b34640 4 set_mon_vals no callback set 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 monclient: _renew_subs 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f4131f80 con 0x7f23f41329f0 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 0x7f23f4139cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.009 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.988+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23f4132dd0 con 0x7f23f409fe30 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f23e8044070 con 0x7f23f41329f0 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 0x7f23f4139cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient: get_auth_request con 0x7f23f409fe30 auth_method 0 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 10 monclient(hunting): handle_auth_done global_id 164388 payload 824 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23f2ffd640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23e003e210 0x7f23e00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fa337640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23f40008d0 con 0x7f23f409fe30 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7f23e80734a0 con 0x7f23f41329f0 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fab38640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23e003e210 0x7f23e00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.989+0000 7f23fab38640 10 monclient: get_auth_request con 0x7f23e003e210 auth_method 0 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23fa337640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 msgr2=0x7f23f4139cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23fa337640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 0x7f23f4139cb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00f7a0 tx=0x7f23ec00fc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec018030 con 0x7f23f409fe30 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 10 monclient: dump: 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.012 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23ec00d9b0 con 0x7f23f409fe30 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 msgr2=0x7f23f4139cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 0x7f23f4139cb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00f7a0 tx=0x7f23ec00fc60 comp rx=0 tx=0).stop 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 msgr2=0x7f23f4139cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.991+0000 7f23f2ffd640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f409fe30 0x7f23f4139cb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.992+0000 7f23fab38640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23e003e210 0x7f23e00406d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005c00 tx=0x7f23e4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.997+0000 7f23fc59b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:35.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.997+0000 7f23fc59b640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 0x7f23f41347f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fc59b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f23f413a5d0 con 0x7f23f413a1f0 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 0x7f23f41347f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient: get_auth_request con 0x7f23f413a1f0 auth_method 0 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 10 monclient(hunting): handle_auth_done global_id 164391 payload 824 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:34.998+0000 7f23fa337640 1 -- 172.21.15.164:0/3828654520 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23f40739a0 con 0x7f23f413a1f0 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23fa337640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 msgr2=0x7f23f41347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23fa337640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 0x7f23f41347f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00f7a0 tx=0x7f23ec003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23ec01dd70 con 0x7f23f413a1f0 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 10 monclient: dump: 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.017 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f23ec01dd70 con 0x7f23f413a1f0 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:35.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 msgr2=0x7f23f41347f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 0x7f23f41347f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f23ec00f7a0 tx=0x7f23ec003760 comp rx=0 tx=0).stop 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 msgr2=0x7f23f41347f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.000+0000 7f23f2ffd640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23f413a1f0 0x7f23f41347f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:23:35.019 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 185702, 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:35.020 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.021 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.022 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.023 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.024 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:35.025 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.026 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.027 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.028 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.029 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.030 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.031 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.032 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.033 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.034 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.035 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:35.035 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:35.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23e003e210 msgr2=0x7f23e00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f23e003e210 0x7f23e00406d0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f23e4005c00 tx=0x7f23e4009040 comp rx=0 tx=0).stop 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 10 monclient: shutdown 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 msgr2=0x7f23f4131a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f23e8009720 tx=0x7f23e80046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fab38640 1 -- 172.21.15.164:0/3828654520 reap_dead start 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 shutdown_connections 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 --2- 172.21.15.164:0/3828654520 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f23f41329f0 0x7f23f4131a40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 >> 172.21.15.164:0/3828654520 conn(0x7f23f40a7320 msgr2=0x7f23f40a6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 shutdown_connections 2024-09-10T16:23:35.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.001+0000 7f23fc59b640 1 -- 172.21.15.164:0/3828654520 wait complete. 2024-09-10T16:23:35.038 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:35.038 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:23:35.038 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:23:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.134+0000 7f4663552640 10 monclient: get_monmap_and_config 2024-09-10T16:23:35.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.134+0000 7f4663552640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.134+0000 7f4663552640 10 monclient: monmap: 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:35.135832+0000 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:35.135832+0000 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:35.137 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:35.138 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.135+0000 7f4663552640 1 Processor -- start 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.135+0000 7f4663552640 1 -- start start 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.135+0000 7f4663552640 10 monclient: init 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f465c06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:35.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f465c06d660 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 0x7f465c141090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): picked mon.noname-b con 0x7f465c13eca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c0a45b0 con 0x7f465c06cb70 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465c06e3b0 con 0x7f465c06d660 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f465c06e690 con 0x7f465c13eca0 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4663552640 10 monclient(hunting): authenticate will time out at 1793.623293s 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f46612ee640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f465c06cb70 0x7f465c06cf50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52652/0 (socket says 172.21.15.164:52652) 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f46612ee640 1 -- 172.21.15.164:0/3858351035 learned_addr learned my addr 172.21.15.164:0/3858351035 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:35.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 1 --2- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 0x7f465c141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): get_auth_request con 0x7f465c13eca0 auth_method 0 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient(hunting): handle_auth_done global_id 194135 payload 274 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 1 -- 172.21.15.164:0/3858351035 >> v1:172.21.15.165:6789/0 conn(0x7f465c06d660 legacy=0x7f465c13d580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 1 -- 172.21.15.164:0/3858351035 >> v1:172.21.15.164:6789/0 conn(0x7f465c06cb70 legacy=0x7f465c06cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:35.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 1 monclient: found mon.noname-b 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.136+0000 7f4661aef640 1 -- 172.21.15.164:0/3858351035 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c0a47f0 con 0x7f465c13eca0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4661aef640 1 -- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 msgr2=0x7f465c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4661aef640 1 --2- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 0x7f465c141090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4654009dc0 tx=0x7f465402a470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bdc3bf5a05a5078 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 1 -- 172.21.15.164:0/3858351035 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654007a30 con 0x7f465c13eca0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: dump: 2024-09-10T16:23:35.145 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4663552640 5 monclient: authenticate success, global_id 194135 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4663552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 1 -- 172.21.15.164:0/3858351035 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4654007bd0 con 0x7f465c13eca0 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4663552640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f4663552640 4 set_mon_vals no callback set 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 1 -- 172.21.15.164:0/3858351035 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4654031400 con 0x7f465c13eca0 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.139+0000 7f465bfff640 10 monclient: dump: 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 10 monclient: shutdown 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 -- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 msgr2=0x7f465c141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 --2- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 0x7f465c141090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4654009dc0 tx=0x7f465402a470 comp rx=0 tx=0).stop 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 -- 172.21.15.164:0/3858351035 shutdown_connections 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 --2- 172.21.15.164:0/3858351035 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c13eca0 0x7f465c141090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 -- 172.21.15.164:0/3858351035 >> 172.21.15.164:0/3858351035 conn(0x7f465c0a4ee0 msgr2=0x7f465c0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 -- 172.21.15.164:0/3858351035 shutdown_connections 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.140+0000 7f4663552640 1 -- 172.21.15.164:0/3858351035 wait complete. 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient: monmap: 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 Processor -- start 2024-09-10T16:23:35.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 -- start start 2024-09-10T16:23:35.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient: init 2024-09-10T16:23:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 0x7f465c137b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): picked mon.noname-c con 0x7f465c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f465c06d660 0x7f465c138070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): picked mon.noname-b con 0x7f465c06d660 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f465c13eca0 0x7f465c071130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): picked mon.noname-a con 0x7f465c13eca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f465c0a41e0 con 0x7f465c13eca0 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f465c0a4060 con 0x7f465c06d660 2024-09-10T16:23:35.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f465c0a0190 con 0x7f465c06cb70 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f4663552640 10 monclient(hunting): authenticate will time out at 1793.628959s 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.141+0000 7f46612ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 0x7f465c137b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 0x7f465c137b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54052/0 (socket says 172.21.15.164:54052) 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 -- 172.21.15.164:0/4075649831 learned_addr learned my addr 172.21.15.164:0/4075649831 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): get_auth_request con 0x7f465c06cb70 auth_method 0 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f465c13eca0 0x7f465c071130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): get_auth_request con 0x7f465c13eca0 auth_method 0 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient(hunting): handle_auth_done global_id 194138 payload 824 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f465c06d660 0x7f465c138070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f465c06d660 msgr2=0x7f465c138070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f465c06d660 0x7f465c138070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f465c13eca0 msgr2=0x7f465c071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f465c13eca0 0x7f465c071130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 monclient: found mon.noname-c 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f46612ee640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f465c073d20 con 0x7f465c06cb70 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4660aed640 10 monclient: get_auth_request con 0x7f465c06d660 auth_method 0 2024-09-10T16:23:35.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f465c06d660 0x7f465c138070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.142+0000 7f4661aef640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f465c13eca0 0x7f465c071130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f46612ee640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 msgr2=0x7f465c137b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f46612ee640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 0x7f465c137b30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f464c00c460 tx=0x7f464c00c920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c007070 con 0x7f465c06cb70 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.162 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4663552640 5 monclient: authenticate success, global_id 194138 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f464c007230 con 0x7f465c06cb70 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f464c007620 con 0x7f465c06cb70 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4663552640 10 monclient: _renew_subs 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4663552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f465c09f5d0 con 0x7f465c06cb70 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.164 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.144+0000 7f465b7fe640 4 set_mon_vals no callback set 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 10 monclient: _renew_subs 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f465c073760 con 0x7f465c06cb70 2024-09-10T16:23:35.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f464c01b400 con 0x7f465c06cb70 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 0x7f465c071a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f465c071f90 con 0x7f465c071670 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 0x7f465c071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4659ffb640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f464803e210 0x7f46480406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4660aed640 10 monclient: get_auth_request con 0x7f465c071670 auth_method 0 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4660aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4660aed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.145+0000 7f4660aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7f464c0529e0 con 0x7f465c06cb70 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4661aef640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f464803e210 0x7f46480406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4661aef640 10 monclient: get_auth_request con 0x7f464803e210 auth_method 0 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4660aed640 10 monclient(hunting): handle_auth_done global_id 194141 payload 824 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.146+0000 7f4660aed640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f465c0a92f0 con 0x7f465c071670 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4660aed640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 msgr2=0x7f465c071a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 0x7f465c071a50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f465000d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650012a10 con 0x7f465c071670 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.168 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4661aef640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f464803e210 0x7f46480406d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f465c0a5f10 tx=0x7f465402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4650012a10 con 0x7f465c071670 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 msgr2=0x7f465c071a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 0x7f465c071a50 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f465000d950 comp rx=0 tx=0).stop 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 msgr2=0x7f465c071a50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.148+0000 7f4659ffb640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c071670 0x7f465c071a50 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f465000d950 comp rx=0 tx=0).stop 2024-09-10T16:23:35.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.154+0000 7f4663552640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.154+0000 7f4663552640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 0x7f465c09d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4663552640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f465c072860 con 0x7f465c09d270 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 0x7f465c09d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient: get_auth_request con 0x7f465c09d270 auth_method 0 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 10 monclient(hunting): handle_auth_done global_id 194144 payload 824 2024-09-10T16:23:35.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.155+0000 7f4660aed640 1 -- 172.21.15.164:0/4075649831 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f465c09be40 con 0x7f465c09d270 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4660aed640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 msgr2=0x7f465c09d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4660aed640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 0x7f465c09d680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f4650003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4650006bf0 con 0x7f465c09d270 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.172 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f4650006bf0 con 0x7f465c09d270 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:35.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 msgr2=0x7f465c09d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 0x7f465c09d680 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f4650003760 comp rx=0 tx=0).stop 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 msgr2=0x7f465c09d680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.157+0000 7f4659ffb640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c09d270 0x7f465c09d680 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f465c072310 tx=0x7f4650003760 comp rx=0 tx=0).stop 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:23:35.174 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 18612, 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.175 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.176 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.177 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.178 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.179 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:35.180 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.181 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.182 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.183 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.184 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:35.185 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.186 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:35.187 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:35.188 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f464803e210 msgr2=0x7f46480406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f464803e210 0x7f46480406d0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f465c0a5f10 tx=0x7f465402f000 comp rx=0 tx=0).stop 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 10 monclient: shutdown 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 msgr2=0x7f465c137b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 --2- 172.21.15.164:0/4075649831 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f465c06cb70 0x7f465c137b30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f464c00c460 tx=0x7f464c00c920 comp rx=0 tx=0).stop 2024-09-10T16:23:35.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4661aef640 1 -- 172.21.15.164:0/4075649831 reap_dead start 2024-09-10T16:23:35.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4661aef640 1 -- 172.21.15.164:0/4075649831 reap_dead start 2024-09-10T16:23:35.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 shutdown_connections 2024-09-10T16:23:35.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 >> 172.21.15.164:0/4075649831 conn(0x7f465c0a4ee0 msgr2=0x7f465c0a1390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 shutdown_connections 2024-09-10T16:23:35.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.158+0000 7f4663552640 1 -- 172.21.15.164:0/4075649831 wait complete. 2024-09-10T16:23:35.191 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:35.191 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:23:35.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:23:35.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.288+0000 7f0bf695d640 10 monclient: get_monmap_and_config 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.288+0000 7f0bf695d640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.288+0000 7f0bf695d640 10 monclient: monmap: 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:35.289418+0000 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:35.289418+0000 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.290 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:35.291 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:35.291 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 Processor -- start 2024-09-10T16:23:35.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 -- start start 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient: init 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 0x7f0bf006cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0bf009bcc0 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0bf00a40d0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf006e900 con 0x7f0bf00a40d0 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0bf006e340 con 0x7f0bf009bcc0 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf00a4840 con 0x7f0bf006cb70 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf695d640 10 monclient(hunting): authenticate will time out at 1793.776812s 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf4efa640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0bf00a40d0 0x7f0bf013f600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52668/0 (socket says 172.21.15.164:52668) 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0bf4efa640 1 -- 172.21.15.164:0/416349503 learned_addr learned my addr 172.21.15.164:0/416349503 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0beffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 0x7f0bf006cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0beffff640 10 monclient(hunting): get_auth_request con 0x7f0bf006cb70 auth_method 0 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.289+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 194147 payload 274 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 1 -- 172.21.15.164:0/416349503 >> v1:172.21.15.165:6789/0 conn(0x7f0bf009bcc0 legacy=0x7f0bf006d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 1 -- 172.21.15.164:0/416349503 >> v1:172.21.15.164:6789/0 conn(0x7f0bf00a40d0 legacy=0x7f0bf013f600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 1 monclient: found mon.noname-b 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.290+0000 7f0beffff640 1 -- 172.21.15.164:0/416349503 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf00a01e0 con 0x7f0bf006cb70 2024-09-10T16:23:35.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.292+0000 7f0beffff640 1 -- 172.21.15.164:0/416349503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 msgr2=0x7f0bf006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.292+0000 7f0beffff640 1 --2- 172.21.15.164:0/416349503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 0x7f0bf006cf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0be00097d0 tx=0x7f0be0029fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b067b75010cdbc8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 1 -- 172.21.15.164:0/416349503 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0037070 con 0x7f0bf006cb70 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: dump: 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.306 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 5 monclient: authenticate success, global_id 194147 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 1 -- 172.21.15.164:0/416349503 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be0032440 con 0x7f0bf006cb70 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:35.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 4 set_mon_vals no callback set 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 1 -- 172.21.15.164:0/416349503 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be00327f0 con 0x7f0bf006cb70 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0beeffd640 10 monclient: dump: 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.308 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 10 monclient: shutdown 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 -- 172.21.15.164:0/416349503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 msgr2=0x7f0bf006cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 --2- 172.21.15.164:0/416349503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 0x7f0bf006cf50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0be00097d0 tx=0x7f0be0029fa0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 -- 172.21.15.164:0/416349503 shutdown_connections 2024-09-10T16:23:35.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 --2- 172.21.15.164:0/416349503 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf006cb70 0x7f0bf006cf50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 -- 172.21.15.164:0/416349503 >> 172.21.15.164:0/416349503 conn(0x7f0bf00a4f50 msgr2=0x7f0bf00a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 -- 172.21.15.164:0/416349503 shutdown_connections 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.293+0000 7f0bf695d640 1 -- 172.21.15.164:0/416349503 wait complete. 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.294+0000 7f0bf695d640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.294+0000 7f0bf695d640 10 monclient: monmap: 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:35.310 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.294+0000 7f0bf695d640 1 Processor -- start 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.294+0000 7f0bf695d640 1 -- start start 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.294+0000 7f0bf695d640 10 monclient: init 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf006cb70 0x7f0bf01cd450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-b con 0x7f0bf006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:35.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 0x7f0bf009a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-a con 0x7f0bf00a40d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 0x7f0bf00933a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): picked mon.noname-c con 0x7f0bf009abe0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf01410b0 con 0x7f0bf00a40d0 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0140f30 con 0x7f0bf006cb70 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0bf0141230 con 0x7f0bf009abe0 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf695d640 10 monclient(hunting): authenticate will time out at 1793.782367s 2024-09-10T16:23:35.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 0x7f0bf009a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 0x7f0bf00933a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 0x7f0bf009a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59728/0 (socket says 172.21.15.164:59728) 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 1 -- 172.21.15.164:0/2541691396 learned_addr learned my addr 172.21.15.164:0/2541691396 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): get_auth_request con 0x7f0bf009abe0 auth_method 0 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): get_auth_request con 0x7f0bf00a40d0 auth_method 0 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf006cb70 0x7f0bf01cd450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0beffff640 10 monclient(hunting): get_auth_request con 0x7f0bf006cb70 auth_method 0 2024-09-10T16:23:35.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient(hunting): handle_auth_done global_id 194150 payload 824 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf006cb70 msgr2=0x7f0bf01cd450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf006cb70 0x7f0bf01cd450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 msgr2=0x7f0bf009a6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 0x7f0bf009a6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 monclient: found mon.noname-c 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bf4efa640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0bf009c140 con 0x7f0bf009abe0 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.295+0000 7f0bef7fe640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0bf00a40d0 0x7f0bf009a6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.297+0000 7f0bf4efa640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 msgr2=0x7f0bf00933a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.297+0000 7f0bf4efa640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 0x7f0bf00933a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0be800caf0 tx=0x7f0be800cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be800ade0 con 0x7f0bf009abe0 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: dump: 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.316 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 5 monclient: authenticate success, global_id 194150 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0be8012400 con 0x7f0bf009abe0 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 10 monclient: _renew_subs 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be80127f0 con 0x7f0bf009abe0 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0bf0097170 con 0x7f0bf009abe0 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bee7fc640 4 set_mon_vals no callback set 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0becff9640 10 monclient: dump: 2024-09-10T16:23:35.318 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 10 monclient: _renew_subs 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:23:35.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0bf00968d0 con 0x7f0bf009abe0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.298+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf006cf50 con 0x7f0bf009bcc0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0be80129d0 con 0x7f0bf009abe0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 0x7f0bd00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7f0be8053300 con 0x7f0bf009abe0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient: get_auth_request con 0x7f0bf009bcc0 auth_method 0 2024-09-10T16:23:35.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0bef7fe640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 0x7f0bd00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0bef7fe640 10 monclient: get_auth_request con 0x7f0bd003e210 auth_method 0 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 154441 payload 824 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.299+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bf00a16e0 con 0x7f0bf009bcc0 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 msgr2=0x7f0bf002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0be0034000 tx=0x7f0be00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0bef7fe640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 0x7f0bd00406d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00a1ab0 tx=0x7f0bd8007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 msgr2=0x7f0bf002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 msgr2=0x7f0bf002d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0be0034000 tx=0x7f0be00046c0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f0be0034000 tx=0x7f0be00046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 msgr2=0x7f0bf002d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 0x7f0bd0046680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 msgr2=0x7f0bf002d440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf009bcc0 0x7f0bf002d440 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0bd0046cc0 con 0x7f0bd0044290 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 0x7f0bd0046680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient: get_auth_request con 0x7f0bd0044290 auth_method 0 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.302+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 154444 payload 824 2024-09-10T16:23:35.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.303+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0be801be50 con 0x7f0bd0044290 2024-09-10T16:23:35.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 msgr2=0x7f0bd0046680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 0x7f0bd0046680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be004e070 con 0x7f0bd0044290 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 10 monclient: dump: 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0be0047d00 con 0x7f0bd0044290 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 msgr2=0x7f0bd0046680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 0x7f0bd0046680 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f0be0000c00 tx=0x7f0be0008030 comp rx=0 tx=0).stop 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 msgr2=0x7f0bd0046680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.305+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bd0044290 0x7f0bd0046680 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.311+0000 7f0bf695d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.311+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 0x7f0bf002dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.311+0000 7f0bf695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.311+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0bf0072860 con 0x7f0bf002d980 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 0x7f0bf002dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient: get_auth_request con 0x7f0bf002d980 auth_method 0 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 10 monclient(hunting): handle_auth_done global_id 154447 payload 824 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.312+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0bf0095970 con 0x7f0bf002d980 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.314+0000 7f0beffff640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 msgr2=0x7f0bf002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.314+0000 7f0beffff640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 0x7f0bf002dd90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0be0029fe0 tx=0x7f0be0044000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0be0037040 con 0x7f0bf002d980 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 10 monclient: dump: 2024-09-10T16:23:35.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f0be0059670 con 0x7f0bf002d980 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:35.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 msgr2=0x7f0bf002dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 0x7f0bf002dd90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0be0029fe0 tx=0x7f0be0044000 comp rx=0 tx=0).stop 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 msgr2=0x7f0bf002dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0becff9640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0bf002d980 0x7f0bf002dd90 secure :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0be0029fe0 tx=0x7f0be0044000 comp rx=0 tx=0).stop 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf4efa640 1 -- 172.21.15.164:0/2541691396 reap_dead start 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:35.332 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 166, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 233866, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.333 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.334 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.335 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:35.336 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:35.337 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:35.338 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:35.339 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:35.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.341 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:35.345 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:35.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:23:35.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:35.348 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 msgr2=0x7f0bd00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 0x7f0bd00406d0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f0bf00a1ab0 tx=0x7f0bd8007ef0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 10 monclient: shutdown 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 msgr2=0x7f0bf00933a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 0x7f0bf00933a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f0be800caf0 tx=0x7f0be800cef0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 shutdown_connections 2024-09-10T16:23:35.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0bd003e210 0x7f0bd00406d0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 --2- 172.21.15.164:0/2541691396 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0bf009abe0 0x7f0bf00933a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 >> 172.21.15.164:0/2541691396 conn(0x7f0bf00a4f50 msgr2=0x7f0bf00a0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 shutdown_connections 2024-09-10T16:23:35.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.315+0000 7f0bf695d640 1 -- 172.21.15.164:0/2541691396 wait complete. 2024-09-10T16:23:35.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:23:35.351 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:23:35.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:23:35.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:23:35.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:23:35.352 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:23:35.352 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:23:35.362 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:23:35.362 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:23:35.362 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.498+0000 7f122dbf4640 10 monclient: get_monmap_and_config 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.498+0000 7f122dbf4640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.498+0000 7f122dbf4640 10 monclient: monmap: 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:35.499273+0000 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:35.499273+0000 2024-09-10T16:23:35.500 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:35.501 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 Processor -- start 2024-09-10T16:23:35.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 -- start start 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient: init 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 0x7f12280a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-b con 0x7f122809f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12280a2380 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12280a2780 0x7f122813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-a con 0x7f12280a2780 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122809fd10 con 0x7f12280a2380 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f12277fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 msgr2=0x7f12280a1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:35.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f122809cd20 con 0x7f12280a2780 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f122809a0d0 con 0x7f122809f930 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f12277fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 0x7f12280a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f122dbf4640 10 monclient(hunting): authenticate will time out at 1793.986761s 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12280a2780 0x7f122813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f12280a2780 0x7f122813dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59738/0 (socket says 172.21.15.164:59738) 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 1 -- 172.21.15.164:0/2178693902 learned_addr learned my addr 172.21.15.164:0/2178693902 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:35.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): get_auth_request con 0x7f12280a2780 auth_method 0 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.499+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 10 monclient(hunting): handle_auth_done global_id 164400 payload 274 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 1 -- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 msgr2=0x7f12280a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 1 --2- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 0x7f12280a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 1 -- 172.21.15.164:0/2178693902 >> v1:172.21.15.165:6789/0 conn(0x7f12280a2380 legacy=0x7f122813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 1 monclient: found mon.noname-a 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.500+0000 7f1227fff640 1 -- 172.21.15.164:0/2178693902 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122809e840 con 0x7f12280a2780 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.502+0000 7f1227fff640 1 -- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 msgr2=0x7f122813dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.502+0000 7f1227fff640 1 --2- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f122813dcf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1220009e50 tx=0x7f122002a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b189877eb21d822 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 1 -- 172.21.15.164:0/2178693902 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220036070 con 0x7f12280a2780 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:35.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: dump: 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 5 monclient: authenticate success, global_id 164400 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 1 -- 172.21.15.164:0/2178693902 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1220007800 con 0x7f12280a2780 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 4 set_mon_vals no callback set 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 1 -- 172.21.15.164:0/2178693902 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220007bd0 con 0x7f12280a2780 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f12267fc640 10 monclient: dump: 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 10 monclient: shutdown 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 -- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 msgr2=0x7f122813dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 --2- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f122813dcf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1220009e50 tx=0x7f122002a570 comp rx=0 tx=0).stop 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 -- 172.21.15.164:0/2178693902 shutdown_connections 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 --2- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f122813dcf0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 --2- 172.21.15.164:0/2178693902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f122809f930 0x7f12280a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 -- 172.21.15.164:0/2178693902 >> 172.21.15.164:0/2178693902 conn(0x7f12280a3110 msgr2=0x7f12280a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 -- 172.21.15.164:0/2178693902 shutdown_connections 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.503+0000 7f122dbf4640 1 -- 172.21.15.164:0/2178693902 wait complete. 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.504+0000 7f122dbf4640 10 monclient: build_initial_monmap 2024-09-10T16:23:35.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.504+0000 7f122dbf4640 10 monclient: monmap: 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.504+0000 7f122dbf4640 1 Processor -- start 2024-09-10T16:23:35.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.504+0000 7f122dbf4640 1 -- start start 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.504+0000 7f122dbf4640 10 monclient: init 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f122809f930 0x7f12281b0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-b con 0x7f122809f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 0x7f12281ba670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-c con 0x7f12280a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): picked mon.noname-a con 0x7f12280a2780 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f122809d930 con 0x7f12280a2780 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f12280a2bf0 con 0x7f122809f930 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f122809c8b0 con 0x7f12280a2380 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f122dbf4640 10 monclient(hunting): authenticate will time out at 1793.992451s 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1226ffd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 msgr2=0x7f12281ba670 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1226ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 0x7f12281ba670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59742/0 (socket says 172.21.15.164:59742) 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 -- 172.21.15.164:0/3212263795 learned_addr learned my addr 172.21.15.164:0/3212263795 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:35.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): get_auth_request con 0x7f12280a2780 auth_method 0 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f12277fe640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f122809f930 0x7f12281b0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f12277fe640 10 monclient(hunting): get_auth_request con 0x7f122809f930 auth_method 0 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f12277fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f12277fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:35.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f12277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient(hunting): handle_auth_done global_id 164403 payload 824 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 -- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 msgr2=0x7f12281ba670 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 0x7f12281ba670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 -- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f122809f930 msgr2=0x7f12281b0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f122809f930 0x7f12281b0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 monclient: found mon.noname-a 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.505+0000 7f1227fff640 1 -- 172.21.15.164:0/3212263795 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12281b3850 con 0x7f12280a2780 2024-09-10T16:23:35.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f1227fff640 1 -- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 msgr2=0x7f12281babb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f1227fff640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1220034040 tx=0x7f1220004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220041020 con 0x7f12280a2780 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: dump: 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: _finish_auth 0 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:35.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1220031940 con 0x7f12280a2780 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 5 monclient: authenticate success, global_id 164403 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 10 monclient: _renew_subs 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12281b3b30 con 0x7f12280a2780 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1220036040 con 0x7f12280a2780 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f120ffff640 10 monclient: dump: 2024-09-10T16:23:35.526 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 10 monclient: _renew_subs 2024-09-10T16:23:35.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f1225ffb640 4 set_mon_vals no callback set 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.508+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12281b1040 con 0x7f12280a2780 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1220044020 con 0x7f12280a2780 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f122dbf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f122dbf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f122dbf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12280a0760 con 0x7f12280a2780 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f120ffff640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 0x7f1214040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7f1220073b80 con 0x7f12280a2780 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f12277fe640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 0x7f1214040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:35.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.509+0000 7f12277fe640 10 monclient: get_auth_request con 0x7f121403e1c0 auth_method 0 2024-09-10T16:23:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.512+0000 7f12277fe640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 0x7f1214040680 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12281b2400 tx=0x7f1218008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.514+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f122004b070 con 0x7f12280a2780 2024-09-10T16:23:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.514+0000 7f120ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:35.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.514+0000 7f120ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:35.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f122dbf4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f122dbf4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f122dbf4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1228077260 con 0x7f12280a2780 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f120ffff640 1 -- 172.21.15.164:0/3212263795 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f1220072570 con 0x7f12280a2780 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f120ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.757+0000 7f120ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:35.759 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 msgr2=0x7f1214040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 0x7f1214040680 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f12281b2400 tx=0x7f1218008040 comp rx=0 tx=0).stop 2024-09-10T16:23:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 10 monclient: shutdown 2024-09-10T16:23:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:35.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 msgr2=0x7f12281babb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f1220034040 tx=0x7f1220004a40 comp rx=0 tx=0).stop 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 shutdown_connections 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f121403e1c0 0x7f1214040680 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f12280a2780 0x7f12281babb0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f12280a2380 0x7f12281ba670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 --2- 172.21.15.164:0/3212263795 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f122809f930 0x7f12281b0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 >> 172.21.15.164:0/3212263795 conn(0x7f12280a3110 msgr2=0x7f12281bd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 shutdown_connections 2024-09-10T16:23:35.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:35.759+0000 7f122dbf4640 1 -- 172.21.15.164:0/3212263795 wait complete. 2024-09-10T16:23:38.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:38.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.881+0000 7efdd1f73640 10 monclient: get_monmap_and_config 2024-09-10T16:23:38.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.881+0000 7efdd1f73640 10 monclient: build_initial_monmap 2024-09-10T16:23:38.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.881+0000 7efdd1f73640 10 monclient: monmap: 2024-09-10T16:23:38.882 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:38.882 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:38.881907+0000 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:38.881907+0000 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:38.883 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.881+0000 7efdd1f73640 1 Processor -- start 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 -- start start 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient: init 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 0x7efdcc0a2f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-c con 0x7efdcc09ce10 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:38.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efdcc0a3450 0x7efdcc09d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-a con 0x7efdcc0a3450 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1336b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-b con 0x7efdcc09d730 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7efdcc0a22e0 con 0x7efdcc0a3450 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7efdcc073ee0 con 0x7efdcc09ce10 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc0a1c90 con 0x7efdcc09d730 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdd1f73640 10 monclient(hunting): authenticate will time out at 1797.369574s 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdcbfff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 msgr2=0x7efdcc1336b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:38.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdcbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1336b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efdcc0a3450 0x7efdcc09d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7efdcc0a3450 0x7efdcc09d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59754/0 (socket says 172.21.15.164:59754) 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 1 -- 172.21.15.164:0/4135419051 learned_addr learned my addr 172.21.15.164:0/4135419051 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): get_auth_request con 0x7efdcc0a3450 auth_method 0 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.882+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/4135419051 >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 0x7efdcc0a2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 10 monclient(hunting): handle_auth_done global_id 164406 payload 274 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 -- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 msgr2=0x7efdcc1336b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:38.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 --2- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1336b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 -- 172.21.15.164:0/4135419051 >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 msgr2=0x7efdcc0a2f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 --2- 172.21.15.164:0/4135419051 >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 0x7efdcc0a2f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 monclient: found mon.noname-a 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdc3fff640 1 -- 172.21.15.164:0/4135419051 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc0a1f70 con 0x7efdcc0a3450 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdcb7fe640 10 monclient: get_auth_request con 0x7efdcc09ce10 auth_method 0 2024-09-10T16:23:38.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.883+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/4135419051 >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 0x7efdcc0a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:23:38.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.885+0000 7efdc3fff640 1 -- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc0a3450 msgr2=0x7efdcc09d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:38.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdc3fff640 1 --2- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc0a3450 0x7efdcc09d1f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efdcc073e70 tx=0x7efdb402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abda57c4e809eb6a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 1 -- 172.21.15.164:0/4135419051 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4037070 con 0x7efdcc0a3450 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: dump: 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:38.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: _finish_auth 0 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 1 -- 172.21.15.164:0/4135419051 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdb4032640 con 0x7efdcc0a3450 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 1 -- 172.21.15.164:0/4135419051 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdb4032a10 con 0x7efdcc0a3450 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 5 monclient: authenticate success, global_id 164406 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 4 set_mon_vals no callback set 2024-09-10T16:23:38.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdcaffd640 10 monclient: dump: 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:38.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 10 monclient: shutdown 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 1 -- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc0a3450 msgr2=0x7efdcc09d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.886+0000 7efdd1f73640 1 --2- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc0a3450 0x7efdcc09d1f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7efdcc073e70 tx=0x7efdb402a0d0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 -- 172.21.15.164:0/4135419051 shutdown_connections 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 --2- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1336b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 --2- 172.21.15.164:0/4135419051 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc0a3450 0x7efdcc09d1f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 --2- 172.21.15.164:0/4135419051 >> v2:172.21.15.165:3300/0 conn(0x7efdcc09ce10 0x7efdcc0a2f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 -- 172.21.15.164:0/4135419051 >> 172.21.15.164:0/4135419051 conn(0x7efdcc0a7320 msgr2=0x7efdcc0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 -- 172.21.15.164:0/4135419051 shutdown_connections 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 1 -- 172.21.15.164:0/4135419051 wait complete. 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 10 monclient: build_initial_monmap 2024-09-10T16:23:38.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.887+0000 7efdd1f73640 10 monclient: monmap: 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 Processor -- start 2024-09-10T16:23:38.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 -- start start 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient: init 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-a con 0x7efdcc09ce10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-c con 0x7efdcc09d730 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdcc1200b0 0x7efdcc120560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): picked mon.noname-b con 0x7efdcc1200b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc133d70 con 0x7efdcc09ce10 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdc3fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 msgr2=0x7efdcc1cd7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efdcc133bf0 con 0x7efdcc1200b0 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efdcc133ef0 con 0x7efdcc09d730 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdc3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1cd7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdd1f73640 10 monclient(hunting): authenticate will time out at 1797.375628s 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59766/0 (socket says 172.21.15.164:59766) 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 -- 172.21.15.164:0/1337621722 learned_addr learned my addr 172.21.15.164:0/1337621722 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): get_auth_request con 0x7efdcc09ce10 auth_method 0 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:38.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient(hunting): handle_auth_done global_id 164409 payload 824 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 -- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 msgr2=0x7efdcc1cd7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1cd7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 -- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdcc1200b0 msgr2=0x7efdcc120560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdcc1200b0 0x7efdcc120560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 monclient: found mon.noname-a 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:38.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.888+0000 7efdcb7fe640 1 -- 172.21.15.164:0/1337621722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efdcc09e0f0 con 0x7efdcc09ce10 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdcb7fe640 1 -- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 msgr2=0x7efdcc1cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00c7f0 tx=0x7efdbc00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdcb7fe640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00c7f0 tx=0x7efdbc00cbf0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7efdbc00ce40 mon_map magic: 0 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc015070 con 0x7efdcc09ce10 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: dump: 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 5 monclient: authenticate success, global_id 164409 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdbc007c40 con 0x7efdcc09ce10 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 10 monclient: _renew_subs 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:38.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efdcc121b20 con 0x7efdcc09ce10 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdc8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 10 monclient: _renew_subs 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdca7fc640 4 set_mon_vals no callback set 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.891+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efdcc123e80 con 0x7efdcc09ce10 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdbc00ce40 con 0x7efdcc09ce10 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 10 monclient: dump: 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:38.905 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efdbc018070 con 0x7efdcc09ce10 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdd1f73640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdd1f73640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdd1f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efdcc134350 con 0x7efdcc09ce10 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc8ff9640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 0x7efdac040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc3fff640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 0x7efdac040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:38.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.892+0000 7efdc3fff640 10 monclient: get_auth_request con 0x7efdac03e1c0 auth_method 0 2024-09-10T16:23:38.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.895+0000 7efdc3fff640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 0x7efdac040680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0a7f10 tx=0x7efdb402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:38.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:38.897+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7efdbc052f10 con 0x7efdcc09ce10 2024-09-10T16:23:39.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.514+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7efdbc051900 con 0x7efdcc09ce10 2024-09-10T16:23:39.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.514+0000 7efdc8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:39.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.514+0000 7efdc8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdd1f73640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdd1f73640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdd1f73640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7efdcc1226e0 con 0x7efdcc09ce10 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdc8ff9640 1 -- 172.21.15.164:0/1337621722 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7efdbc011c90 con 0x7efdcc09ce10 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdc8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.757+0000 7efdc8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:39.759 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 msgr2=0x7efdac040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 0x7efdac040680 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7efdcc0a7f10 tx=0x7efdb402e000 comp rx=0 tx=0).stop 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 10 monclient: shutdown 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 msgr2=0x7efdcc1cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:39.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7efdbc00c7f0 tx=0x7efdbc00cbf0 comp rx=0 tx=0).stop 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 shutdown_connections 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdac03e1c0 0x7efdac040680 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efdcc1200b0 0x7efdcc120560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efdcc09d730 0x7efdcc1cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 --2- 172.21.15.164:0/1337621722 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efdcc09ce10 0x7efdcc1cd280 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 >> 172.21.15.164:0/1337621722 conn(0x7efdcc0a7320 msgr2=0x7efdcc0a5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.759+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 shutdown_connections 2024-09-10T16:23:39.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:39.760+0000 7efdd1f73640 1 -- 172.21.15.164:0/1337621722 wait complete. 2024-09-10T16:23:42.772 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 10 monclient: get_monmap_and_config 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 10 monclient: build_initial_monmap 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 10 monclient: monmap: 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:42.881998+0000 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:42.881998+0000 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:42.883 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 1 Processor -- start 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 1 -- start start 2024-09-10T16:23:42.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.881+0000 7fef9d8cd640 10 monclient: init 2024-09-10T16:23:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:23:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef9806cb70 0x7fef98134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-a con 0x7fef9806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:42.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef980a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 0x7fef981383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-c con 0x7fef980a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef9809ce50 con 0x7fef980a46a0 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fef9806db60 con 0x7fef9806cb70 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fef9806cf50 con 0x7fef980a4ab0 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef9d8cd640 10 monclient(hunting): authenticate will time out at 1801.369291s 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef9806cb70 0x7fef98134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef967fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fef980a46a0 0x7fef98137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:52672/0 (socket says 172.21.15.164:52672) 2024-09-10T16:23:42.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fef9806cb70 0x7fef98134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59782/0 (socket says 172.21.15.164:59782) 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef967fc640 1 -- 172.21.15.164:0/802471034 learned_addr learned my addr 172.21.15.164:0/802471034 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): get_auth_request con 0x7fef9806cb70 auth_method 0 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 1 --2- 172.21.15.164:0/802471034 >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 0x7fef981383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 10 monclient(hunting): get_auth_request con 0x7fef980a4ab0 auth_method 0 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:42.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient(hunting): handle_auth_done global_id 164412 payload 274 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 -- 172.21.15.164:0/802471034 >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 msgr2=0x7fef981383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 --2- 172.21.15.164:0/802471034 >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 0x7fef981383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 -- 172.21.15.164:0/802471034 >> v1:172.21.15.164:6789/0 conn(0x7fef980a46a0 legacy=0x7fef98137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 monclient: found mon.noname-a 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef96ffd640 1 -- 172.21.15.164:0/802471034 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef9809cb70 con 0x7fef9806cb70 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.882+0000 7fef977fe640 1 --2- 172.21.15.164:0/802471034 >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 0x7fef981383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef96ffd640 1 -- 172.21.15.164:0/802471034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 msgr2=0x7fef98134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef96ffd640 1 --2- 172.21.15.164:0/802471034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef98134f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fef880097d0 tx=0x7fef8802a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72d6a6d6896dbb8b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:42.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 1 -- 172.21.15.164:0/802471034 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88037070 con 0x7fef9806cb70 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: dump: 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: _finish_auth 0 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef9d8cd640 5 monclient: authenticate success, global_id 164412 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef9d8cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 1 -- 172.21.15.164:0/802471034 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88032440 con 0x7fef9806cb70 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef9d8cd640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef9d8cd640 4 set_mon_vals no callback set 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 1 -- 172.21.15.164:0/802471034 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88032810 con 0x7fef9806cb70 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.885+0000 7fef95ffb640 10 monclient: dump: 2024-09-10T16:23:42.896 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 10 monclient: shutdown 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:42.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 -- 172.21.15.164:0/802471034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 msgr2=0x7fef98134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/802471034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef98134f70 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fef880097d0 tx=0x7fef8802a040 comp rx=0 tx=0).stop 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 -- 172.21.15.164:0/802471034 shutdown_connections 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/802471034 >> v2:172.21.15.165:3300/0 conn(0x7fef980a4ab0 0x7fef981383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/802471034 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef98134f70 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 -- 172.21.15.164:0/802471034 >> 172.21.15.164:0/802471034 conn(0x7fef980aa900 msgr2=0x7fef980aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 -- 172.21.15.164:0/802471034 shutdown_connections 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.886+0000 7fef9d8cd640 1 -- 172.21.15.164:0/802471034 wait complete. 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.887+0000 7fef9d8cd640 10 monclient: build_initial_monmap 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.887+0000 7fef9d8cd640 10 monclient: monmap: 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:42.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.887+0000 7fef9d8cd640 1 Processor -- start 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.887+0000 7fef9d8cd640 1 -- start start 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.887+0000 7fef9d8cd640 10 monclient: init 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:23:42.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-a con 0x7fef9806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef980a46a0 0x7fef980a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-b con 0x7fef980a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 0x7fef980a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): picked mon.noname-c con 0x7fef980a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fef98138a70 con 0x7fef9806cb70 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fef981388f0 con 0x7fef980a46a0 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fef98140040 con 0x7fef980a4ab0 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:42.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef9d8cd640 10 monclient(hunting): authenticate will time out at 1801.375193s 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef977fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 msgr2=0x7fef980a3ea0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef977fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 0x7fef980a3ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59786/0 (socket says 172.21.15.164:59786) 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 -- 172.21.15.164:0/853202921 learned_addr learned my addr 172.21.15.164:0/853202921 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): get_auth_request con 0x7fef9806cb70 auth_method 0 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:42.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef980a46a0 0x7fef980a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): get_auth_request con 0x7fef980a46a0 auth_method 0 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient(hunting): handle_auth_done global_id 164415 payload 824 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 -- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 msgr2=0x7fef980a3ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 0x7fef980a3ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 -- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef980a46a0 msgr2=0x7fef980a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:42.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef980a46a0 0x7fef980a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:23:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 monclient: found mon.noname-a 2024-09-10T16:23:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:42.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.888+0000 7fef96ffd640 1 -- 172.21.15.164:0/853202921 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef980a2150 con 0x7fef9806cb70 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef96ffd640 1 -- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 msgr2=0x7fef981cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef96ffd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fef88035040 tx=0x7fef880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef88037040 con 0x7fef9806cb70 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:42.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 10 monclient: dump: 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 10 monclient: _finish_auth 0 2024-09-10T16:23:42.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fef88004260 con 0x7fef9806cb70 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef9d8cd640 5 monclient: authenticate success, global_id 164415 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef9d8cd640 10 monclient: _renew_subs 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.890+0000 7fef9d8cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef980a2430 con 0x7fef9806cb70 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef8ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef8803c670 con 0x7fef9806cb70 2024-09-10T16:23:42.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef8ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef8ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef8ffff640 10 monclient: dump: 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:42.908 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef957fa640 4 set_mon_vals no callback set 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef9d8cd640 10 monclient: _renew_subs 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef9d8cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.891+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef980a00d0 con 0x7fef9806cb70 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef9d8cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fef8803a070 con 0x7fef9806cb70 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef9d8cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef9d8cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef98134f70 con 0x7fef9806cb70 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef8ffff640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 0x7fef78040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(591..591 src has 481..591) ==== 4787+0+0 (secure 0 0 0) 0x7fef88073110 con 0x7fef9806cb70 2024-09-10T16:23:42.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef967fc640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 0x7fef78040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.892+0000 7fef967fc640 10 monclient: get_auth_request con 0x7fef7803e1c0 auth_method 0 2024-09-10T16:23:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.895+0000 7fef967fc640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 0x7fef78040680 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fef80007c90 tx=0x7fef80007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.896+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fef8804a030 con 0x7fef9806cb70 2024-09-10T16:23:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.896+0000 7fef8ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:42.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:42.896+0000 7fef8ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef9d8cd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef9d8cd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef9d8cd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fef981cd710 con 0x7fef9806cb70 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef8ffff640 1 -- 172.21.15.164:0/853202921 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7fef88041030 con 0x7fef9806cb70 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef8ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.138+0000 7fef8ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:43.140 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":166,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 msgr2=0x7fef78040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 0x7fef78040680 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fef80007c90 tx=0x7fef80007be0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 10 monclient: shutdown 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 msgr2=0x7fef981cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fef88035040 tx=0x7fef880046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 shutdown_connections 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fef7803e1c0 0x7fef78040680 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fef980a4ab0 0x7fef980a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fef980a46a0 0x7fef980a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 --2- 172.21.15.164:0/853202921 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fef9806cb70 0x7fef981cd1d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 >> 172.21.15.164:0/853202921 conn(0x7fef980aa900 msgr2=0x7fef980a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 shutdown_connections 2024-09-10T16:23:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:43.140+0000 7fef9d8cd640 1 -- 172.21.15.164:0/853202921 wait complete. 2024-09-10T16:23:46.153 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.259+0000 7fe661da1640 10 monclient: get_monmap_and_config 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.259+0000 7fe661da1640 10 monclient: build_initial_monmap 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.259+0000 7fe661da1640 10 monclient: monmap: 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:46.261577+0000 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:46.261577+0000 2024-09-10T16:23:46.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:46.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 1 Processor -- start 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 1 -- start start 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient: init 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-a con 0x7fe65c09ba80 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-c con 0x7fe65c06cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.260+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-b con 0x7fe65c06d4e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:46.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe65c0a4810 con 0x7fe65c09ba80 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe65c06ea40 con 0x7fe65c06cb70 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe65c09be60 con 0x7fe65c06d4e0 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe661da1640 10 monclient(hunting): authenticate will time out at 1804.749024s 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe6615a0640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 msgr2=0x7fe65c13e580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe6615a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:53900/0 (socket says 172.21.15.164:53900) 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 learned_addr learned my addr 172.21.15.164:0/3968367775 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 10 monclient(hunting): get_auth_request con 0x7fe65c09ba80 auth_method 0 2024-09-10T16:23:46.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe65c06cb70 auth_method 0 2024-09-10T16:23:46.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.261+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe6615a0640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 msgr2=0x7fe65c13e580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe6615a0640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 10 monclient(hunting): get_auth_request con 0x7fe65c09ba80 auth_method 0 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:46.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.462+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe65c06cb70 auth_method 0 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.463+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:23:46.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe6615a0640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 msgr2=0x7fe65c13e580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe6615a0640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 10 monclient(hunting): get_auth_request con 0x7fe65c09ba80 auth_method 0 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.863+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:23:46.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:46.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe65c06cb70 auth_method 0 2024-09-10T16:23:46.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:46.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:46.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:46.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:46.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:46.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:46.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:46.865+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:23:47.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:47.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 10 monclient(hunting): get_auth_request con 0x7fe65c09ba80 auth_method 0 2024-09-10T16:23:47.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:47.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:47.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.664+0000 7fe660d9f640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.665+0000 7fe6615a0640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 msgr2=0x7fe65c13e580 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:47.668 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.665+0000 7fe6615a0640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe65c06cb70 auth_method 0 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:23:47.669 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:47.666+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): tick 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 msgr2=0x7fe65c13e580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe65c06d4e0 0x7fe65c13e580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 msgr2=0x7fe65c06cfa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.165:3300/0 conn(0x7fe65c06cb70 0x7fe65c06cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 -- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 msgr2=0x7fe65c0a42d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 --2- 172.21.15.164:0/3968367775 >> v2:172.21.15.164:3300/0 conn(0x7fe65c09ba80 0x7fe65c0a42d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient: _add_conns ranks=[0,1] 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6480036c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6480073f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 -- 172.21.15.164:0/3968367775 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe65c06dbf0 con 0x7fe6480036c0 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 1 -- 172.21.15.164:0/3968367775 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64800c130 con 0x7fe6480073f0 2024-09-10T16:23:49.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.261+0000 7fe6527fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:49.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.263+0000 7fe660d9f640 1 -- 172.21.15.164:0/3968367775 >> v1:172.21.15.164:6789/0 conn(0x7fe6480036c0 legacy=0x7fe648005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.263+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> v1:172.21.15.165:6789/0 conn(0x7fe6480073f0 legacy=0x7fe6480098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 707223476 0 0) 0x7fe64800c130 con 0x7fe6480073f0 2024-09-10T16:23:49.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 10 monclient(hunting): _negotiate 2024-09-10T16:23:49.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 10 monclient(hunting): my global_id is 154459 2024-09-10T16:23:49.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.573+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe644003520 con 0x7fe6480073f0 2024-09-10T16:23:49.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 412713364 0 0) 0x7fe644003520 con 0x7fe6480073f0 2024-09-10T16:23:49.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 >> v1:172.21.15.164:6789/0 conn(0x7fe6480036c0 legacy=0x7fe648005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _finish_hunting 0 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64800d380 con 0x7fe6480073f0 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _finish_auth 0 2024-09-10T16:23:49.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _check_auth_tickets 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fe64c00d680 con 0x7fe6480073f0 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: dump: 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.579 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 >> v1:172.21.15.165:6789/0 conn(0x7fe6480073f0 legacy=0x7fe6480098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6615a0640 1 -- 172.21.15.164:0/3968367775 reap_dead start 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440039b0 0x7fe644003db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): picked mon.a con 0x7fe6440039b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:49.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe644004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): picked mon.b con 0x7fe644004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 0x7fe6440141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): picked mon.c con 0x7fe6440053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe644014a40 con 0x7fe6440039b0 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe644014bf0 con 0x7fe644004800 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe644014df0 con 0x7fe6440053c0 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6537fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:23:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.574+0000 7fe6615a0640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 msgr2=0x7fe6440141d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe6615a0640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 0x7fe6440141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe644004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe644004800 auth_method 0 2024-09-10T16:23:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.575+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 10 monclient(hunting): handle_auth_done global_id 154459 payload 293 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 msgr2=0x7fe6440141d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 0x7fe6440141d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440039b0 msgr2=0x7fe644003db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440039b0 0x7fe644003db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 monclient: found mon.b 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64800d380 con 0x7fe644004800 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 10 monclient: _finish_auth 0 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.576+0000 7fe653fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:49.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.577+0000 7fe661da1640 5 monclient: authenticate success, global_id 154459 2024-09-10T16:23:49.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.577+0000 7fe661da1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:49.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.579+0000 7fe653fff640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 msgr2=0x7fe644004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.579+0000 7fe653fff640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe644004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe64800b0c0 tx=0x7fe644005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76ada1e5280afed2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.586 INFO:tasks.rados.rados.0.smithi165.stdout:2625: done (0 left) 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2626: rollback oid 5 current snap is 285 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 269 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2627: copy_from oid 31 from oid 29 current snap is 285 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2628: read oid 6 snap 258 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2628: expect (ObjNum 830 snap 256 seq_num 830) 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2629: write (excl) oid 26 current snap is 285 2024-09-10T16:23:49.587 INFO:tasks.rados.rados.0.smithi165.stdout:2629: seq_num 930 ranges {430510=576636,1654842=768318,2922795=502619} 2024-09-10T16:23:49.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c031070 con 0x7fe644004800 2024-09-10T16:23:49.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:49.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: dump: 2024-09-10T16:23:49.588 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe661da1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c02bc70 con 0x7fe644004800 2024-09-10T16:23:49.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe661da1640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 1 -- 172.21.15.164:0/3968367775 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c0132e0 con 0x7fe644004800 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe661da1640 4 set_mon_vals no callback set 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.582+0000 7fe6537fe640 10 monclient: dump: 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 10 monclient: shutdown 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 -- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 msgr2=0x7fe644004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe644004c80 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe64800b0c0 tx=0x7fe644005830 comp rx=0 tx=0).stop 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 -- 172.21.15.164:0/3968367775 shutdown_connections 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440053c0 0x7fe6440141d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe644004c80 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 --2- 172.21.15.164:0/3968367775 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440039b0 0x7fe644003db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 -- 172.21.15.164:0/3968367775 >> 172.21.15.164:0/3968367775 conn(0x7fe65c0a4ee0 msgr2=0x7fe65c0a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 -- 172.21.15.164:0/3968367775 shutdown_connections 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.583+0000 7fe661da1640 1 -- 172.21.15.164:0/3968367775 wait complete. 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.584+0000 7fe661da1640 10 monclient: build_initial_monmap 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.584+0000 7fe661da1640 10 monclient: monmap: 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.584+0000 7fe661da1640 1 Processor -- start 2024-09-10T16:23:49.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.585+0000 7fe661da1640 1 -- start start 2024-09-10T16:23:49.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.585+0000 7fe661da1640 10 monclient: init 2024-09-10T16:23:49.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.585+0000 7fe661da1640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:49.593 INFO:tasks.rados.rados.0.smithi165.stdout:2629: writing smithi16534841-26 from 430510 to 1007146 tid 1 2024-09-10T16:23:49.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.585+0000 7fe661da1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:23:49.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 0x7fe65c09a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-c con 0x7fe6440039b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:49.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-b con 0x7fe644004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe660d9f640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 msgr2=0x7fe65c09a270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe660d9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 0x7fe65c09a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 0x7fe65c08efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): picked mon.noname-a con 0x7fe6440053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c06e130 con 0x7fe6440053c0 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe65c06e410 con 0x7fe644004800 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe65c06cfa0 con 0x7fe6440039b0 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe661da1640 10 monclient(hunting): authenticate will time out at 1808.074267s 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:38008/0 (socket says 172.21.15.164:38008) 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 -- 172.21.15.164:0/1896619529 learned_addr learned my addr 172.21.15.164:0/1896619529 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): get_auth_request con 0x7fe644004800 auth_method 0 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 0x7fe65c08efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:49.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 10 monclient(hunting): get_auth_request con 0x7fe6440053c0 auth_method 0 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient(hunting): handle_auth_done global_id 154462 payload 824 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 -- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 msgr2=0x7fe65c09a270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 0x7fe65c09a270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 -- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 msgr2=0x7fe65c08efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 0x7fe65c08efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient: _finish_hunting 0 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 monclient: found mon.noname-b 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe653fff640 1 -- 172.21.15.164:0/1896619529 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe65c1c93a0 con 0x7fe644004800 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.586+0000 7fe6615a0640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 0x7fe65c08efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe653fff640 1 -- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 msgr2=0x7fe65c09a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe653fff640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00d860 tx=0x7fe64c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c044020 con 0x7fe644004800 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 10 monclient: dump: 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 10 monclient: _finish_auth 0 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 10 monclient: _check_auth_tickets 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe64c035ba0 con 0x7fe644004800 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 5 monclient: authenticate success, global_id 154462 2024-09-10T16:23:49.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 10 monclient: _renew_subs 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe65c09f5d0 con 0x7fe644004800 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 10 monclient: _renew_subs 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.589+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe65c09acf0 con 0x7fe644004800 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6517fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe64c040470 con 0x7fe644004800 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6517fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6517fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6527fc640 4 set_mon_vals no callback set 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.590+0000 7fe6517fa640 10 monclient: dump: 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.591+0000 7fe661da1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.591+0000 7fe661da1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.591+0000 7fe661da1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.591+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe63c005080 con 0x7fe644004800 2024-09-10T16:23:49.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.592+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe64c047020 con 0x7fe644004800 2024-09-10T16:23:49.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.592+0000 7fe6517fa640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 0x7fe6480414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.592+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(592..592 src has 481..592) ==== 4815+0+0 (secure 0 0 0) 0x7fe64c074380 con 0x7fe644004800 2024-09-10T16:23:49.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.592+0000 7fe660d9f640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 0x7fe6480414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.592+0000 7fe660d9f640 10 monclient: get_auth_request con 0x7fe64803f020 auth_method 0 2024-09-10T16:23:49.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.595+0000 7fe660d9f640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 0x7fe6480414e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe658009560 tx=0x7fe6580094f0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.599+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe64c04e030 con 0x7fe644004800 2024-09-10T16:23:49.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.599+0000 7fe6517fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:49.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.599+0000 7fe6517fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2629: writing smithi16534841-26 from 1654842 to 2423160 tid 2 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2629: writing smithi16534841-26 from 2922795 to 3425414 tid 3 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2630: copy_from oid 11 from oid 10 current snap is 285 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2627: finishing copy_from to smithi16534841-31 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2627: got expected ENOENT (src dne) 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2627: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2626: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2629: finishing write tid 1 to smithi16534841-26 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:2626: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:23:49.606 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1338 (ObjNum 33 snap 0 seq_num 326127714) dirty dne 2024-09-10T16:23:49.607 INFO:tasks.rados.rados.0.smithi165.stdout:2628: expect (ObjNum 830 snap 256 seq_num 830) 2024-09-10T16:23:49.607 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.846+0000 7fe661da1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.846+0000 7fe661da1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.846+0000 7fe661da1640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.846+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe63c005370 con 0x7fe644004800 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.847+0000 7fe6517fa640 1 -- 172.21.15.164:0/1896619529 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fe65c09f5d0 con 0x7fe644004800 2024-09-10T16:23:49.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.847+0000 7fe6517fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:23:49.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.847+0000 7fe6517fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:49.850 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":170,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 msgr2=0x7fe6480414e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 0x7fe6480414e0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fe658009560 tx=0x7fe6580094f0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 10 monclient: shutdown 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 msgr2=0x7fe65c09a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fe64c00d860 tx=0x7fe64c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 shutdown_connections 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe64803f020 0x7fe6480414e0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6440053c0 0x7fe65c08efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe644004800 0x7fe65c09a7b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 --2- 172.21.15.164:0/1896619529 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6440039b0 0x7fe65c09a270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 >> 172.21.15.164:0/1896619529 conn(0x7fe65c0a4ee0 msgr2=0x7fe65c0a7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 shutdown_connections 2024-09-10T16:23:49.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.849+0000 7fe661da1640 1 -- 172.21.15.164:0/1896619529 wait complete. 2024-09-10T16:23:49.862 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:23:49.863 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:23:49.863 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:23:49.880 INFO:tasks.rados.rados.0.smithi165.stdout:2629: finishing write tid 2 to smithi16534841-26 2024-09-10T16:23:49.880 INFO:tasks.rados.rados.0.smithi165.stdout:2629: finishing write tid 3 to smithi16534841-26 2024-09-10T16:23:49.880 INFO:tasks.rados.rados.0.smithi165.stdout:2629: finishing write tid 4 to smithi16534841-26 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: finishing write tid 5 to smithi16534841-26 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: oid 26 updating version 0 to 1662 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: oid 26 version 1662 is already newer than 1661 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: oid 26 version 1662 is already newer than 1659 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: oid 26 version 1662 is already newer than 1660 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1662 (ObjNum 930 snap 285 seq_num 930) dirty exists 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: left oid 26 (ObjNum 930 snap 285 seq_num 930) 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2630: finishing copy_from racing read to smithi16534841-11 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2630: finishing copy_from to smithi16534841-11 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2626: done (4 left) 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2627: done (3 left) 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2628: done (2 left) 2024-09-10T16:23:49.881 INFO:tasks.rados.rados.0.smithi165.stdout:2629: done (1 left) 2024-09-10T16:23:49.882 INFO:tasks.rados.rados.0.smithi165.stdout:2630: done (0 left) 2024-09-10T16:23:49.882 INFO:tasks.rados.rados.0.smithi165.stdout:2631: delete oid 43 current snap is 285 2024-09-10T16:23:49.884 INFO:tasks.rados.rados.0.smithi165.stdout:2631: done (0 left) 2024-09-10T16:23:49.884 INFO:tasks.rados.rados.0.smithi165.stdout:2632: write oid 37 current snap is 285 2024-09-10T16:23:49.884 INFO:tasks.rados.rados.0.smithi165.stdout:2632: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2024-09-10T16:23:49.891 INFO:tasks.rados.rados.0.smithi165.stdout:2632: writing smithi16534841-37 from 464124 to 1191258 tid 1 2024-09-10T16:23:49.898 INFO:tasks.rados.rados.0.smithi165.stdout:2632: writing smithi16534841-37 from 1771806 to 2543793 tid 2 2024-09-10T16:23:49.904 INFO:tasks.rados.rados.0.smithi165.stdout:2632: writing smithi16534841-37 from 2977641 to 3619157 tid 3 2024-09-10T16:23:49.905 INFO:tasks.rados.rados.0.smithi165.stdout:2633: write oid 34 current snap is 285 2024-09-10T16:23:49.905 INFO:tasks.rados.rados.0.smithi165.stdout:2632: finishing write tid 1 to smithi16534841-37 2024-09-10T16:23:49.905 INFO:tasks.rados.rados.0.smithi165.stdout:2633: seq_num 933 ranges {480931=402383,1313935=515645,2488711=452788,3443230=237701} 2024-09-10T16:23:49.909 INFO:tasks.rados.rados.0.smithi165.stdout:2633: writing smithi16534841-34 from 480931 to 883314 tid 1 2024-09-10T16:23:49.916 INFO:tasks.rados.rados.0.smithi165.stdout:2633: writing smithi16534841-34 from 1313935 to 1829580 tid 2 2024-09-10T16:23:49.920 INFO:tasks.rados.rados.0.smithi165.stdout:2633: writing smithi16534841-34 from 2488711 to 2941499 tid 3 2024-09-10T16:23:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:2633: writing smithi16534841-34 from 3443230 to 3680931 tid 4 2024-09-10T16:23:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:2634: copy_from oid 6 from oid 17 current snap is 285 2024-09-10T16:23:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:2632: finishing write tid 2 to smithi16534841-37 2024-09-10T16:23:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:2632: finishing write tid 3 to smithi16534841-37 2024-09-10T16:23:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:2632: finishing write tid 4 to smithi16534841-37 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: finishing write tid 5 to smithi16534841-37 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: oid 37 updating version 0 to 1550 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: oid 37 updating version 1550 to 1553 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: oid 37 version 1553 is already newer than 1552 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: oid 37 version 1553 is already newer than 1551 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1553 (ObjNum 932 snap 285 seq_num 932) dirty exists 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: left oid 37 (ObjNum 932 snap 285 seq_num 932) 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 1 to smithi16534841-34 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 2 to smithi16534841-34 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 3 to smithi16534841-34 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2632: done (2 left) 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2635: read oid 47 snap -1 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2635: expect (ObjNum 908 snap 278 seq_num 908) 2024-09-10T16:23:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:2636: snap_remove snap 282 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 10 monclient: get_monmap_and_config 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 10 monclient: build_initial_monmap 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 10 monclient: monmap: 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:49.970971+0000 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:49.970971+0000 2024-09-10T16:23:49.971 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:49.972 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 1 Processor -- start 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 1 -- start start 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.969+0000 7f9b81b00640 10 monclient: init 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9b7c06cb70 0x7f9b7c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7c06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9b7c0a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 0x7f9b7c1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7c0a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b7c09ce50 con 0x7f9b7c0a46a0 2024-09-10T16:23:49.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7c06db60 con 0x7f9b7c06cb70 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7c06cf50 con 0x7f9b7c0a4ab0 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b81b00640 10 monclient(hunting): authenticate will time out at 1808.458567s 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7bfff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 msgr2=0x7f9b7c1383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 0x7f9b7c1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9b7c06cb70 0x7f9b7c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b73fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9b7c0a46a0 0x7f9b7c137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44800/0 (socket says 172.21.15.164:44800) 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b73fff640 1 -- 172.21.15.164:0/2294283802 learned_addr learned my addr 172.21.15.164:0/2294283802 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): get_auth_request con 0x7f9b7c06cb70 auth_method 0 2024-09-10T16:23:49.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_done global_id 164424 payload 274 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 msgr2=0x7f9b7c1383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 --2- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 0x7f9b7c1383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/2294283802 >> v1:172.21.15.164:6789/0 conn(0x7f9b7c0a46a0 legacy=0x7f9b7c137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:49.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:49.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.970+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/2294283802 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7c09cb70 con 0x7f9b7c06cb70 2024-09-10T16:23:49.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 msgr2=0x7f9b7c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7b7fe640 1 --2- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c134f70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097d0 tx=0x7f9b6c02a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be62b04988bfa3df server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 1 -- 172.21.15.164:0/2294283802 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c037070 con 0x7f9b7c06cb70 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: dump: 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.984 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: _finish_auth 0 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: _check_auth_tickets 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b81b00640 5 monclient: authenticate success, global_id 164424 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b81b00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 1 -- 172.21.15.164:0/2294283802 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6c032640 con 0x7f9b7c06cb70 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b81b00640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:49.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 1 -- 172.21.15.164:0/2294283802 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c032a10 con 0x7f9b7c06cb70 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b81b00640 4 set_mon_vals no callback set 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.973+0000 7f9b7affd640 10 monclient: dump: 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.986 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 10 monclient: shutdown 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 -- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 msgr2=0x7f9b7c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 --2- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c134f70 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097d0 tx=0x7f9b6c02a010 comp rx=0 tx=0).stop 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 -- 172.21.15.164:0/2294283802 shutdown_connections 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 --2- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a4ab0 0x7f9b7c1383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 --2- 172.21.15.164:0/2294283802 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c134f70 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 -- 172.21.15.164:0/2294283802 >> 172.21.15.164:0/2294283802 conn(0x7f9b7c0aa900 msgr2=0x7f9b7c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 -- 172.21.15.164:0/2294283802 shutdown_connections 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.974+0000 7f9b81b00640 1 -- 172.21.15.164:0/2294283802 wait complete. 2024-09-10T16:23:49.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient: build_initial_monmap 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient: monmap: 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 Processor -- start 2024-09-10T16:23:49.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 -- start start 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient: init 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c09eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-a con 0x7f9b7c06cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a46a0 0x7f9b7c09f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-c con 0x7f9b7c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b7c0a4ab0 0x7f9b7c09f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7c0a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7c138a70 con 0x7f9b7c06cb70 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7c1388f0 con 0x7f9b7c0a4ab0 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9b7c140040 con 0x7f9b7c0a46a0 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b81b00640 10 monclient(hunting): authenticate will time out at 1808.463984s 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.975+0000 7f9b73fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a46a0 msgr2=0x7f9b7c09f030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b73fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a46a0 0x7f9b7c09f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c09eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c09eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:53972/0 (socket says 172.21.15.164:53972) 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/226338928 learned_addr learned my addr 172.21.15.164:0/226338928 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): get_auth_request con 0x7f9b7c06cb70 auth_method 0 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient(hunting): handle_auth_done global_id 164427 payload 824 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a46a0 msgr2=0x7f9b7c09f030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9b7c0a46a0 0x7f9b7c09f030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b7c0a4ab0 msgr2=0x7f9b7c09f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9b7c0a4ab0 0x7f9b7c09f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:49.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.976+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7c0a2240 con 0x7f9b7c06cb70 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b7b7fe640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 msgr2=0x7f9b7c09eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b7b7fe640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c09eaf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097d0 tx=0x7f9b6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c041070 con 0x7f9b7c06cb70 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: dump: 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: _finish_auth 0 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6c03ca80 con 0x7f9b7c06cb70 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 5 monclient: authenticate success, global_id 164427 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 10 monclient: _renew_subs 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b7c0a2520 con 0x7f9b7c06cb70 2024-09-10T16:23:49.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6c037040 con 0x7f9b7c06cb70 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b78ff9640 10 monclient: dump: 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:49.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 10 monclient: _renew_subs 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7c06e720 con 0x7f9b7c06cb70 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.978+0000 7f9b7a7fc640 4 set_mon_vals no callback set 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b81b00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b81b00640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 0x7f9b7c1cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7c134f70 con 0x7f9b7c1cd3d0 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9b6c044070 con 0x7f9b7c06cb70 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 0x7f9b7c1cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient: get_auth_request con 0x7f9b7c1cd3d0 auth_method 0 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:49.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b78ff9640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b5c03e210 0x7f9b5c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(592..592 src has 481..592) ==== 4815+0+0 (secure 0 0 0) 0x7f9b6c0732a0 con 0x7f9b7c06cb70 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 10 monclient(hunting): handle_auth_done global_id 164430 payload 824 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.979+0000 7f9b73fff640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9b7c0a2020 con 0x7f9b7c1cd3d0 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.980+0000 7f9b7bfff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b5c03e210 0x7f9b5c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:49.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.980+0000 7f9b7bfff640 10 monclient: get_auth_request con 0x7f9b5c03e210 auth_method 0 2024-09-10T16:23:49.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b73fff640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 msgr2=0x7f9b7c1cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b73fff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 0x7f9b7c1cd7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0a2450 tx=0x7f9b6400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64017070 con 0x7f9b7c1cd3d0 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 10 monclient: dump: 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:49.999 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b640070d0 con 0x7f9b7c1cd3d0 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 msgr2=0x7f9b7c1cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 0x7f9b7c1cd7b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c0a2450 tx=0x7f9b6400c810 comp rx=0 tx=0).stop 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 msgr2=0x7f9b7c1cd7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b78ff9640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cd3d0 0x7f9b7c1cd7b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.982+0000 7f9b7bfff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b5c03e210 0x7f9b5c0406d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9b74003d00 tx=0x7f9b74009210 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b81b00640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:50.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b81b00640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 0x7f9b7c12f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b81b00640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9b7c0a0b80 con 0x7f9b7c1cdcf0 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 0x7f9b7c12f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient: get_auth_request con 0x7f9b7c1cdcf0 auth_method 0 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 10 monclient(hunting): handle_auth_done global_id 164433 payload 824 2024-09-10T16:23:50.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.988+0000 7f9b73fff640 1 -- 172.21.15.164:0/226338928 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9b7c0770f0 con 0x7f9b7c1cdcf0 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b73fff640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 msgr2=0x7f9b7c12f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b73fff640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 0x7f9b7c12f460 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c1350f0 tx=0x7f9b64004970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64017070 con 0x7f9b7c1cdcf0 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 10 monclient: dump: 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.002 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f9b640223f0 con 0x7f9b7c1cdcf0 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 msgr2=0x7f9b7c12f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 0x7f9b7c12f460 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c1350f0 tx=0x7f9b64004970 comp rx=0 tx=0).stop 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 msgr2=0x7f9b7c12f460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b78ff9640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c1cdcf0 0x7f9b7c12f460 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9b7c1350f0 tx=0x7f9b64004970 comp rx=0 tx=0).stop 2024-09-10T16:23:50.003 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 170, 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 200694, 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:50.004 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.005 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.006 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:50.007 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.008 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.009 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.010 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:50.011 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.012 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.013 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:50.014 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.015 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:50.016 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:50.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b5c03e210 msgr2=0x7f9b5c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b5c03e210 0x7f9b5c0406d0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f9b74003d00 tx=0x7f9b74009210 comp rx=0 tx=0).stop 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 10 monclient: shutdown 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 msgr2=0x7f9b7c09eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 --2- 172.21.15.164:0/226338928 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9b7c06cb70 0x7f9b7c09eaf0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f9b6c0097d0 tx=0x7f9b6c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b7bfff640 1 -- 172.21.15.164:0/226338928 reap_dead start 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b7bfff640 1 -- 172.21.15.164:0/226338928 reap_dead start 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 shutdown_connections 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 >> 172.21.15.164:0/226338928 conn(0x7f9b7c0aa900 msgr2=0x7f9b7c0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 shutdown_connections 2024-09-10T16:23:50.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:49.991+0000 7f9b81b00640 1 -- 172.21.15.164:0/226338928 wait complete. 2024-09-10T16:23:50.019 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:23:50.019 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:23:50.019 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:23:50.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 10 monclient: get_monmap_and_config 2024-09-10T16:23:50.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 10 monclient: build_initial_monmap 2024-09-10T16:23:50.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 10 monclient: monmap: 2024-09-10T16:23:50.118 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:23:50.118010+0000 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:23:50.118010+0000 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:23:50.119 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 1 Processor -- start 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 1 -- start start 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.116+0000 7fbfeb471640 10 monclient: init 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient: _add_conns ranks=[1,3,2,0,4] 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbfe406cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbfe4134f70 0x7fbfe413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe4134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 0x7fbfe413f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe413d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:50.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbfe409ce50 con 0x7fbfe406cb70 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbfe406db60 con 0x7fbfe4134f70 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe406cf50 con 0x7fbfe413d560 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfeb471640 10 monclient(hunting): authenticate will time out at 1808.605499s 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe9a0e640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 msgr2=0x7fbfe413f950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe9a0e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 0x7fbfe413f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbfe4134f70 0x7fbfe413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbfe4134f70 0x7fbfe413d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54000/0 (socket says 172.21.15.164:54000) 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 -- 172.21.15.164:0/870102494 learned_addr learned my addr 172.21.15.164:0/870102494 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): get_auth_request con 0x7fbfe4134f70 auth_method 0 2024-09-10T16:23:50.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_done global_id 164436 payload 274 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 -- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 msgr2=0x7fbfe413f950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 --2- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 0x7fbfe413f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 -- 172.21.15.164:0/870102494 >> v1:172.21.15.165:6789/0 conn(0x7fbfe406cb70 legacy=0x7fbfe40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient: _finish_hunting 0 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 monclient: found mon.noname-a 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:50.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:50.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.117+0000 7fbfe8a0c640 1 -- 172.21.15.164:0/870102494 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe4135380 con 0x7fbfe4134f70 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe8a0c640 1 -- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe4134f70 msgr2=0x7fbfe413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe8a0c640 1 --2- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe4134f70 0x7fbfe413d020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80098a0 tx=0x7fbfd802a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1758a4df1c2a7c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 1 -- 172.21.15.164:0/870102494 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8037070 con 0x7fbfe4134f70 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:50.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: dump: 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: _finish_auth 0 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:23:50.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 1 -- 172.21.15.164:0/870102494 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfd8032640 con 0x7fbfe4134f70 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfeb471640 5 monclient: authenticate success, global_id 164436 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfeb471640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfeb471640 10 monclient: get_monmap_and_config success 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 1 -- 172.21.15.164:0/870102494 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd8032a10 con 0x7fbfe4134f70 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfeb471640 4 set_mon_vals no callback set 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.120+0000 7fbfe3fff640 10 monclient: dump: 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.134 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 10 monclient: shutdown 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 -- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe4134f70 msgr2=0x7fbfe413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 --2- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe4134f70 0x7fbfe413d020 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fbfd80098a0 tx=0x7fbfd802a0e0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 -- 172.21.15.164:0/870102494 shutdown_connections 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 --2- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe413d560 0x7fbfe413f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 --2- 172.21.15.164:0/870102494 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe4134f70 0x7fbfe413d020 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 -- 172.21.15.164:0/870102494 >> 172.21.15.164:0/870102494 conn(0x7fbfe40aa900 msgr2=0x7fbfe40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 -- 172.21.15.164:0/870102494 shutdown_connections 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.121+0000 7fbfeb471640 1 -- 172.21.15.164:0/870102494 wait complete. 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient: build_initial_monmap 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient: monmap: 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:23:50.136 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 Processor -- start 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 -- start start 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient: init 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient: _reopen_session rank -1 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe406cb70 0x7fbfe41cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-c con 0x7fbfe406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe4134f70 0x7fbfe41cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-b con 0x7fbfe4134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): picked mon.noname-a con 0x7fbfe413d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40a4db0 con 0x7fbfe413d560 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe409d130 con 0x7fbfe4134f70 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbfe413fe90 con 0x7fbfe406cb70 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): _renew_subs 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfeb471640 10 monclient(hunting): authenticate will time out at 1808.610850s 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe920d640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe406cb70 msgr2=0x7fbfe41cd440 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:23:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe920d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe406cb70 0x7fbfe41cd440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54002/0 (socket says 172.21.15.164:54002) 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 learned_addr learned my addr 172.21.15.164:0/3218439709 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 10 monclient(hunting): get_auth_request con 0x7fbfe413d560 auth_method 0 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.122+0000 7fbfe9a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe4134f70 0x7fbfe41cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): get_auth_request con 0x7fbfe4134f70 auth_method 0 2024-09-10T16:23:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe8a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient(hunting): handle_auth_done global_id 164439 payload 824 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe406cb70 msgr2=0x7fbfe41cd440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbfe406cb70 0x7fbfe41cd440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe4134f70 msgr2=0x7fbfe41cd980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe4134f70 0x7fbfe41cd980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient: _finish_hunting 0 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 monclient: found mon.noname-a 2024-09-10T16:23:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.123+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfe409f4e0 con 0x7fbfe413d560 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 msgr2=0x7fbfe41cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe9a0e640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b410 tx=0x7fbfdc00b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe9a0e640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b410 tx=0x7fbfdc00b8d0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7fbfdc017070 mon_map magic: 0 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc017070 con 0x7fbfe413d560 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: dump: 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfdc0060d0 con 0x7fbfe413d560 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 5 monclient: authenticate success, global_id 164439 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 10 monclient: _renew_subs 2024-09-10T16:23:50.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfe409f7c0 con 0x7fbfe413d560 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 10 monclient: _renew_subs 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfe37fe640 4 set_mon_vals no callback set 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.125+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfe409e290 con 0x7fbfe413d560 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfdc0064e0 con 0x7fbfe413d560 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 10 monclient: dump: 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.143 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbfdc01a070 con 0x7fbfe413d560 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfeb471640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfeb471640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 0x7fbfe41c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe40a46a0 con 0x7fbfe409ef20 2024-09-10T16:23:50.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe1ffb640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfd003e1c0 0x7fbfd0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 0x7fbfe41c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient: get_auth_request con 0x7fbfe409ef20 auth_method 0 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe8a0c640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfd003e1c0 0x7fbfd0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.126+0000 7fbfe8a0c640 10 monclient: get_auth_request con 0x7fbfd003e1c0 auth_method 0 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.127+0000 7fbfe920d640 10 monclient(hunting): handle_auth_done global_id 154468 payload 824 2024-09-10T16:23:50.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.127+0000 7fbfe920d640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfe413ccb0 con 0x7fbfe409ef20 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe920d640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 msgr2=0x7fbfe41c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe920d640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 0x7fbfe41c9f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400c610 tx=0x7fbfd400cad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe8a0c640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfd003e1c0 0x7fbfd0040680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbfe406e110 tx=0x7fbfd802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4017070 con 0x7fbfe409ef20 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 10 monclient: dump: 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.146 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfd4005e30 con 0x7fbfe409ef20 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 msgr2=0x7fbfe41c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 0x7fbfe41c9f00 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400c610 tx=0x7fbfd400cad0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 msgr2=0x7fbfe41c9f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:50.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.129+0000 7fbfe1ffb640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe409ef20 0x7fbfe41c9f00 secure :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400c610 tx=0x7fbfd400cad0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.135+0000 7fbfeb471640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.135+0000 7fbfeb471640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 0x7fbfe41ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.135+0000 7fbfeb471640 10 monclient(hunting): start opening mon connection 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.135+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbfe41cad60 con 0x7fbfe41ca440 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 0x7fbfe41ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient: get_auth_request con 0x7fbfe41ca440 auth_method 0 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:23:50.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 10 monclient(hunting): handle_auth_done global_id 154471 payload 824 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.136+0000 7fbfe920d640 1 -- 172.21.15.164:0/3218439709 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfe41d87f0 con 0x7fbfe41ca440 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe920d640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 msgr2=0x7fbfe41ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe920d640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 0x7fbfe41ca820 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400a0d0 tx=0x7fbfd4003ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4017040 con 0x7fbfe41ca440 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:23:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.138+0000 7fbfe1ffb640 10 monclient: dump: 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7fbfd4012400 con 0x7fbfe41ca440 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:23:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 msgr2=0x7fbfe41ca820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 0x7fbfe41ca820 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fbfd400a0d0 tx=0x7fbfd4003ac0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 msgr2=0x7fbfe41ca820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbfe41ca440 0x7fbfe41ca820 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 170, 2024-09-10T16:23:50.152 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 248690, 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.153 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.154 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:23:50.155 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:23:50.156 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:23:50.157 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.158 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.159 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:23:50.160 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:23:50.161 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.162 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.163 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.164 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:23:50.165 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:23:50.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfd003e1c0 msgr2=0x7fbfd0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbfd003e1c0 0x7fbfd0040680 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7fbfe406e110 tx=0x7fbfd802e000 comp rx=0 tx=0).stop 2024-09-10T16:23:50.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 10 monclient: shutdown 2024-09-10T16:23:50.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 msgr2=0x7fbfe41cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 --2- 172.21.15.164:0/3218439709 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbfe413d560 0x7fbfe41cdec0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbfdc00b410 tx=0x7fbfdc00b8d0 comp rx=0 tx=0).stop 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 reap_dead start 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe9a0e640 1 -- 172.21.15.164:0/3218439709 reap_dead start 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfe1ffb640 1 -- 172.21.15.164:0/3218439709 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(592..592 src has 481..592) ==== 4815+0+0 (unknown 0 0 0) 0x7fbfdc0520b0 con 0x7fbfe413d560 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 shutdown_connections 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 >> 172.21.15.164:0/3218439709 conn(0x7fbfe40aa900 msgr2=0x7fbfe40a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 shutdown_connections 2024-09-10T16:23:50.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:23:50.139+0000 7fbfeb471640 1 -- 172.21.15.164:0/3218439709 wait complete. 2024-09-10T16:23:50.168 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:23:50.169 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:23:50.585 INFO:tasks.rados.rados.0.smithi165.stdout:2635: expect (ObjNum 908 snap 278 seq_num 908) 2024-09-10T16:23:50.803 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2493709 2024-09-10T16:23:50.832 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 4 to smithi16534841-34 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 5 to smithi16534841-34 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2635: done (3 left) 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2636: done (2 left) 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2637: rollback oid 37 current snap is 285 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 283 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: finishing write tid 6 to smithi16534841-34 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: oid 34 updating version 0 to 1555 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: oid 34 updating version 1555 to 1557 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: oid 34 updating version 1557 to 1558 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: oid 34 version 1558 is already newer than 1554 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:2633: oid 34 version 1558 is already newer than 1556 2024-09-10T16:23:50.833 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1558 (ObjNum 933 snap 285 seq_num 933) dirty exists 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2633: left oid 34 (ObjNum 933 snap 285 seq_num 933) 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2634: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2634: finishing copy_from to smithi16534841-6 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1149 (ObjNum 856 snap 263 seq_num 856) dirty exists 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2633: done (2 left) 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2634: done (1 left) 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2638: read oid 16 snap -1 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2638: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:50.834 INFO:tasks.rados.rados.0.smithi165.stdout:2639: snap_remove snap 258 2024-09-10T16:23:51.467 INFO:tasks.rados.rados.0.smithi165.stdout:2639: done (2 left) 2024-09-10T16:23:51.467 INFO:tasks.rados.rados.0.smithi165.stdout:2640: read oid 3 snap -1 2024-09-10T16:23:51.467 INFO:tasks.rados.rados.0.smithi165.stdout:2640: expect deleted 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2641: read oid 21 snap -1 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2641: expect deleted 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2642: read oid 13 snap -1 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2642: expect (ObjNum 914 snap 280 seq_num 914) 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2643: write oid 45 current snap is 285 2024-09-10T16:23:51.468 INFO:tasks.rados.rados.0.smithi165.stdout:2643: seq_num 934 ranges {497738=477632,1772417=775656,3316134=381604} 2024-09-10T16:23:51.473 INFO:tasks.rados.rados.0.smithi165.stdout:2643: writing smithi16534841-45 from 497738 to 975370 tid 1 2024-09-10T16:23:51.483 INFO:tasks.rados.rados.0.smithi165.stdout:2643: writing smithi16534841-45 from 1772417 to 2548073 tid 2 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:2643: writing smithi16534841-45 from 3316134 to 3697738 tid 3 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:2644: copy_from oid 30 from oid 28 current snap is 285 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:2637: finishing rollback tid 0 to smithi16534841-37 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:2637: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:2637: finishing rollback tid 2 to smithi16534841-37 2024-09-10T16:23:51.491 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1561 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-10T16:23:51.492 INFO:tasks.rados.rados.0.smithi165.stdout:2638: expect (ObjNum 912 snap 279 seq_num 912) 2024-09-10T16:23:51.543 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:51.645 INFO:tasks.rados.rados.0.smithi165.stdout:2637: done (6 left) 2024-09-10T16:23:51.645 INFO:tasks.rados.rados.0.smithi165.stdout:2638: done (5 left) 2024-09-10T16:23:51.645 INFO:tasks.rados.rados.0.smithi165.stdout:2640: done (4 left) 2024-09-10T16:23:51.645 INFO:tasks.rados.rados.0.smithi165.stdout:2641: done (3 left) 2024-09-10T16:23:51.645 INFO:tasks.rados.rados.0.smithi165.stdout:2645: snap_remove snap 280 2024-09-10T16:23:52.593 INFO:tasks.rados.rados.0.smithi165.stdout:2645: done (3 left) 2024-09-10T16:23:52.593 INFO:tasks.rados.rados.0.smithi165.stdout:2646: read oid 1 snap -1 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2646: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2647: read oid 15 snap 279 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2647: expect deleted 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2648: read oid 39 snap 268 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2648: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2649: write oid 42 current snap is 285 2024-09-10T16:23:52.594 INFO:tasks.rados.rados.0.smithi165.stdout:2649: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2024-09-10T16:23:52.599 INFO:tasks.rados.rados.0.smithi165.stdout:2649: writing smithi16534841-42 from 514545 to 1067426 tid 1 2024-09-10T16:23:52.606 INFO:tasks.rados.rados.0.smithi165.stdout:2649: writing smithi16534841-42 from 1830899 to 2350213 tid 2 2024-09-10T16:23:52.613 INFO:tasks.rados.rados.0.smithi165.stdout:2649: writing smithi16534841-42 from 2943557 to 3418889 tid 3 2024-09-10T16:23:52.613 INFO:tasks.rados.rados.0.smithi165.stdout:2650: snap_remove snap 281 2024-09-10T16:23:52.613 INFO:tasks.rados.rados.0.smithi165.stdout:2643: finishing write tid 1 to smithi16534841-45 2024-09-10T16:23:52.613 INFO:tasks.rados.rados.0.smithi165.stdout:2642: expect (ObjNum 914 snap 280 seq_num 914) 2024-09-10T16:23:52.656 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2079189 2024-09-10T16:23:52.829 INFO:tasks.rados.rados.0.smithi165.stdout:2643: finishing write tid 2 to smithi16534841-45 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: finishing write tid 3 to smithi16534841-45 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: finishing write tid 4 to smithi16534841-45 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: finishing write tid 5 to smithi16534841-45 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: oid 45 updating version 0 to 1365 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: oid 45 version 1365 is already newer than 1363 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: oid 45 version 1365 is already newer than 1364 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: oid 45 version 1365 is already newer than 1362 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1365 (ObjNum 934 snap 285 seq_num 934) dirty exists 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2643: left oid 45 (ObjNum 934 snap 285 seq_num 934) 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2644: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:2644: finishing copy_from to smithi16534841-30 2024-09-10T16:23:52.830 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1260 (ObjNum 871 snap 266 seq_num 871) dirty exists 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2642: done (7 left) 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2643: done (6 left) 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2644: done (5 left) 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2647: done (4 left) 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2650: done (3 left) 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2651: copy_from oid 29 from oid 44 current snap is 285 2024-09-10T16:23:53.602 INFO:tasks.rados.rados.0.smithi165.stdout:2646: expect (ObjNum 572 snap 184 seq_num 572) 2024-09-10T16:23:53.812 INFO:tasks.rados.rados.0.smithi165.stdout:2648: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: finishing write tid 1 to smithi16534841-42 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: finishing write tid 2 to smithi16534841-42 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: finishing write tid 3 to smithi16534841-42 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: finishing write tid 4 to smithi16534841-42 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: finishing write tid 5 to smithi16534841-42 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: oid 42 updating version 0 to 1280 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: oid 42 version 1280 is already newer than 1277 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: oid 42 version 1280 is already newer than 1278 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:2649: oid 42 version 1280 is already newer than 1279 2024-09-10T16:23:53.921 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1280 (ObjNum 935 snap 285 seq_num 935) dirty exists 2024-09-10T16:23:53.922 INFO:tasks.rados.rados.0.smithi165.stdout:2649: left oid 42 (ObjNum 935 snap 285 seq_num 935) 2024-09-10T16:23:53.922 INFO:tasks.rados.rados.0.smithi165.stdout:2646: done (3 left) 2024-09-10T16:23:53.922 INFO:tasks.rados.rados.0.smithi165.stdout:2648: done (2 left) 2024-09-10T16:23:53.922 INFO:tasks.rados.rados.0.smithi165.stdout:2649: done (1 left) 2024-09-10T16:23:53.922 INFO:tasks.rados.rados.0.smithi165.stdout:2652: snap_remove snap 279 2024-09-10T16:23:54.606 INFO:tasks.rados.rados.0.smithi165.stdout:2652: done (1 left) 2024-09-10T16:23:54.606 INFO:tasks.rados.rados.0.smithi165.stdout:2653: write oid 5 current snap is 285 2024-09-10T16:23:54.606 INFO:tasks.rados.rados.0.smithi165.stdout:2653: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2024-09-10T16:23:54.613 INFO:tasks.rados.rados.0.smithi165.stdout:2653: writing smithi16534841-5 from 531352 to 1275835 tid 1 2024-09-10T16:23:54.621 INFO:tasks.rados.rados.0.smithi165.stdout:2653: writing smithi16534841-5 from 2005734 to 2785059 tid 2 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2653: writing smithi16534841-5 from 3487333 to 3731352 tid 3 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2654: rollback oid 46 current snap is 285 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 266 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2651: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2651: finishing copy_from to smithi16534841-29 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1110 (ObjNum 843 snap 260 seq_num 843) dirty exists 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2653: finishing write tid 1 to smithi16534841-5 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2651: done (2 left) 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2655: write oid 43 current snap is 285 2024-09-10T16:23:54.627 INFO:tasks.rados.rados.0.smithi165.stdout:2655: seq_num 937 ranges {548159=419732,1664216=522983,2714756=614229} 2024-09-10T16:23:54.632 INFO:tasks.rados.rados.0.smithi165.stdout:2655: writing smithi16534841-43 from 548159 to 967891 tid 1 2024-09-10T16:23:54.639 INFO:tasks.rados.rados.0.smithi165.stdout:2655: writing smithi16534841-43 from 1664216 to 2187199 tid 2 2024-09-10T16:23:54.645 INFO:tasks.rados.rados.0.smithi165.stdout:2655: writing smithi16534841-43 from 2714756 to 3328985 tid 3 2024-09-10T16:23:54.645 INFO:tasks.rados.rados.0.smithi165.stdout:2656: snap_remove snap 265 2024-09-10T16:23:54.645 INFO:tasks.rados.rados.0.smithi165.stdout:2653: finishing write tid 2 to smithi16534841-5 2024-09-10T16:23:54.645 INFO:tasks.rados.rados.0.smithi165.stdout:2653: finishing write tid 3 to smithi16534841-5 2024-09-10T16:23:54.645 INFO:tasks.rados.rados.0.smithi165.stdout:2653: finishing write tid 4 to smithi16534841-5 2024-09-10T16:23:55.621 INFO:tasks.rados.rados.0.smithi165.stdout:2656: done (3 left) 2024-09-10T16:23:55.621 INFO:tasks.rados.rados.0.smithi165.stdout:2657: read oid 21 snap -1 2024-09-10T16:23:55.621 INFO:tasks.rados.rados.0.smithi165.stdout:2657: expect deleted 2024-09-10T16:23:55.621 INFO:tasks.rados.rados.0.smithi165.stdout:2658: copy_from oid 39 from oid 13 current snap is 285 2024-09-10T16:23:55.621 INFO:tasks.rados.rados.0.smithi165.stdout:2653: finishing write tid 5 to smithi16534841-5 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2653: oid 5 updating version 0 to 1339 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2653: oid 5 updating version 1339 to 1341 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2653: oid 5 version 1341 is already newer than 1340 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2653: oid 5 updating version 1341 to 1342 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1342 (ObjNum 936 snap 285 seq_num 936) dirty exists 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2653: left oid 5 (ObjNum 936 snap 285 seq_num 936) 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2654: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2654: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1154 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2655: finishing write tid 1 to smithi16534841-43 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2655: finishing write tid 2 to smithi16534841-43 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2655: finishing write tid 3 to smithi16534841-43 2024-09-10T16:23:55.622 INFO:tasks.rados.rados.0.smithi165.stdout:2655: finishing write tid 4 to smithi16534841-43 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: finishing write tid 5 to smithi16534841-43 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: oid 43 updating version 0 to 1367 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: oid 43 version 1367 is already newer than 1366 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: oid 43 updating version 1367 to 1369 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: oid 43 version 1369 is already newer than 1368 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1369 (ObjNum 937 snap 285 seq_num 937) dirty exists 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: left oid 43 (ObjNum 937 snap 285 seq_num 937) 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2653: done (4 left) 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2654: done (3 left) 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2655: done (2 left) 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2659: copy_from oid 20 from oid 5 current snap is 285 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:2660: rollback oid 18 current snap is 285 2024-09-10T16:23:55.623 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 268 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2661: rollback oid 34 current snap is 285 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 269 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2662: copy_from oid 17 from oid 2 current snap is 285 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2663: rollback oid 24 current snap is 285 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 274 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2664: copy_from oid 47 from oid 48 current snap is 285 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2665: rollback oid 14 current snap is 285 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 268 2024-09-10T16:23:55.624 INFO:tasks.rados.rados.0.smithi165.stdout:2666: delete oid 49 current snap is 285 2024-09-10T16:23:55.628 INFO:tasks.rados.rados.0.smithi165.stdout:2657: done (9 left) 2024-09-10T16:23:55.628 INFO:tasks.rados.rados.0.smithi165.stdout:2666: done (8 left) 2024-09-10T16:23:55.628 INFO:tasks.rados.rados.0.smithi165.stdout:2667: write oid 4 current snap is 285 2024-09-10T16:23:55.628 INFO:tasks.rados.rados.0.smithi165.stdout:2667: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2024-09-10T16:23:55.635 INFO:tasks.rados.rados.0.smithi165.stdout:2667: writing smithi16534841-4 from 581773 to 1152003 tid 1 2024-09-10T16:23:55.641 INFO:tasks.rados.rados.0.smithi165.stdout:2667: writing smithi16534841-4 from 1664827 to 2191479 tid 2 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2667: writing smithi16534841-4 from 2653249 to 3290022 tid 3 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2668: rollback oid 41 current snap is 285 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 266 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2660: finishing rollback tid 1 to smithi16534841-18 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2662: finishing copy_from to smithi16534841-17 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2662: got expected ENOENT (src dne) 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2661: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2661: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:23:55.650 INFO:tasks.rados.rados.0.smithi165.stdout:2661: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1564 (ObjNum 875 snap 269 seq_num 875) dirty exists 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2660: finishing rollback tid 2 to smithi16534841-18 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2662: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2665: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2660: done (9 left) 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2661: done (8 left) 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2662: done (7 left) 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2669: read oid 17 snap 269 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2669: expect (ObjNum 856 snap 263 seq_num 856) 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2670: delete oid 38 current snap is 285 2024-09-10T16:23:55.651 INFO:tasks.rados.rados.0.smithi165.stdout:2665: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2665: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1227 (ObjNum 841 snap 260 seq_num 841) dirty exists 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2663: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2663: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1516 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2658: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2659: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2659: finishing copy_from to smithi16534841-20 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1342 (ObjNum 936 snap 285 seq_num 936) dirty exists 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:2668: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:23:55.652 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:23:55.653 INFO:tasks.rados.rados.0.smithi165.stdout:2664: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:23:55.654 INFO:tasks.rados.rados.0.smithi165.stdout:2659: done (8 left) 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2663: done (7 left) 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2665: done (6 left) 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2668: done (5 left) 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2670: done (4 left) 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2671: rollback oid 28 current snap is 285 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 28 to 268 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2672: rollback oid 5 current snap is 285 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 284 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2673: rollback oid 6 current snap is 285 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 268 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2674: write (excl) oid 1 current snap is 285 2024-09-10T16:23:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:2674: seq_num 941 ranges {615387=720728,1781791=530321,3108095=707292} 2024-09-10T16:23:55.661 INFO:tasks.rados.rados.0.smithi165.stdout:2674: writing smithi16534841-1 from 615387 to 1336115 tid 1 2024-09-10T16:23:55.667 INFO:tasks.rados.rados.0.smithi165.stdout:2674: writing smithi16534841-1 from 1781791 to 2312112 tid 2 2024-09-10T16:23:55.676 INFO:tasks.rados.rados.0.smithi165.stdout:2674: writing smithi16534841-1 from 3108095 to 3815387 tid 3 2024-09-10T16:23:55.676 INFO:tasks.rados.rados.0.smithi165.stdout:2675: write (excl) oid 36 current snap is 285 2024-09-10T16:23:55.676 INFO:tasks.rados.rados.0.smithi165.stdout:2675: seq_num 942 ranges {632194=795977,2123920=790332,3419165=413029} 2024-09-10T16:23:55.684 INFO:tasks.rados.rados.0.smithi165.stdout:2675: writing smithi16534841-36 from 632194 to 1428171 tid 1 2024-09-10T16:23:55.691 INFO:tasks.rados.rados.0.smithi165.stdout:2675: writing smithi16534841-36 from 2123920 to 2914252 tid 2 2024-09-10T16:23:55.697 INFO:tasks.rados.rados.0.smithi165.stdout:2675: writing smithi16534841-36 from 3419165 to 3832194 tid 3 2024-09-10T16:23:55.697 INFO:tasks.rados.rados.0.smithi165.stdout:2676: delete oid 43 current snap is 285 2024-09-10T16:23:55.697 INFO:tasks.rados.rados.0.smithi165.stdout:2664: finishing copy_from to smithi16534841-47 2024-09-10T16:23:55.697 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1568 (ObjNum 901 snap 278 seq_num 901) dirty exists 2024-09-10T16:23:55.697 INFO:tasks.rados.rados.0.smithi165.stdout:2669: expect (ObjNum 856 snap 263 seq_num 856) 2024-09-10T16:23:55.775 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2166991 2024-09-10T16:23:55.829 INFO:tasks.rados.rados.0.smithi165.stdout:2667: finishing write tid 1 to smithi16534841-4 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: finishing write tid 2 to smithi16534841-4 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: finishing write tid 3 to smithi16534841-4 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: finishing write tid 4 to smithi16534841-4 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: finishing write tid 5 to smithi16534841-4 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: oid 4 updating version 0 to 1330 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: oid 4 updating version 1330 to 1333 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: oid 4 version 1333 is already newer than 1331 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: oid 4 version 1333 is already newer than 1332 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1333 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2667: left oid 4 (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:23:55.830 INFO:tasks.rados.rados.0.smithi165.stdout:2671: finishing rollback tid 0 to smithi16534841-28 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2672: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2658: finishing copy_from to smithi16534841-39 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1403 (ObjNum 914 snap 280 seq_num 914) dirty exists 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2672: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2658: done (9 left) 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2664: done (8 left) 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2667: done (7 left) 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2669: done (6 left) 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2676: done (5 left) 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2677: write oid 12 current snap is 285 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2672: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1345 (ObjNum 878 snap 269 seq_num 878) dirty exists 2024-09-10T16:23:55.831 INFO:tasks.rados.rados.0.smithi165.stdout:2673: finishing rollback tid 0 to smithi16534841-6 2024-09-10T16:23:55.832 INFO:tasks.rados.rados.0.smithi165.stdout:2677: seq_num 944 ranges {665808=662828,1957237=677648,3074011=609486} 2024-09-10T16:23:55.836 INFO:tasks.rados.rados.0.smithi165.stdout:2677: writing smithi16534841-12 from 665808 to 1328636 tid 1 2024-09-10T16:23:55.843 INFO:tasks.rados.rados.0.smithi165.stdout:2677: writing smithi16534841-12 from 1957237 to 2634885 tid 2 2024-09-10T16:23:55.851 INFO:tasks.rados.rados.0.smithi165.stdout:2677: writing smithi16534841-12 from 3074011 to 3683497 tid 3 2024-09-10T16:23:55.851 INFO:tasks.rados.rados.0.smithi165.stdout:2672: done (5 left) 2024-09-10T16:23:55.851 INFO:tasks.rados.rados.0.smithi165.stdout:2678: read oid 10 snap 283 2024-09-10T16:23:55.851 INFO:tasks.rados.rados.0.smithi165.stdout:2678: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2679: rollback oid 9 current snap is 285 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 284 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2671: finishing rollback tid 1 to smithi16534841-28 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2671: finishing rollback tid 2 to smithi16534841-28 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1263 (ObjNum 866 snap 265 seq_num 866) dirty exists 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2673: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2673: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1348 (ObjNum 863 snap 265 seq_num 863) dirty exists 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2674: finishing write tid 1 to smithi16534841-1 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2674: finishing write tid 2 to smithi16534841-1 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2674: finishing write tid 3 to smithi16534841-1 2024-09-10T16:23:55.852 INFO:tasks.rados.rados.0.smithi165.stdout:2674: finishing write tid 4 to smithi16534841-1 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: finishing write tid 5 to smithi16534841-1 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: oid 1 updating version 0 to 1404 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: oid 1 updating version 1404 to 1405 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: oid 1 updating version 1405 to 1407 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: oid 1 version 1407 is already newer than 1406 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1407 (ObjNum 941 snap 285 seq_num 941) dirty exists 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2674: left oid 1 (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: finishing write tid 1 to smithi16534841-36 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: finishing write tid 2 to smithi16534841-36 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: finishing write tid 3 to smithi16534841-36 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: finishing write tid 4 to smithi16534841-36 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: finishing write tid 5 to smithi16534841-36 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: oid 36 updating version 0 to 1379 2024-09-10T16:23:55.853 INFO:tasks.rados.rados.0.smithi165.stdout:2675: oid 36 updating version 1379 to 1380 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2675: oid 36 version 1380 is already newer than 1378 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2675: oid 36 updating version 1380 to 1381 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1381 (ObjNum 942 snap 285 seq_num 942) dirty exists 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2675: left oid 36 (ObjNum 942 snap 285 seq_num 942) 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2677: finishing write tid 1 to smithi16534841-12 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2677: finishing write tid 2 to smithi16534841-12 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2671: done (6 left) 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2673: done (5 left) 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2674: done (4 left) 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2675: done (3 left) 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2680: rollback oid 22 current snap is 285 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 284 2024-09-10T16:23:55.854 INFO:tasks.rados.rados.0.smithi165.stdout:2681: read oid 2 snap -1 2024-09-10T16:23:55.855 INFO:tasks.rados.rados.0.smithi165.stdout:2681: expect deleted 2024-09-10T16:23:55.855 INFO:tasks.rados.rados.0.smithi165.stdout:2682: snap_remove snap 266 2024-09-10T16:23:56.469 INFO:tasks.rados.rados.0.smithi165.stdout:2682: done (5 left) 2024-09-10T16:23:56.470 INFO:tasks.rados.rados.0.smithi165.stdout:2683: read oid 47 snap -1 2024-09-10T16:23:56.470 INFO:tasks.rados.rados.0.smithi165.stdout:2683: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:23:56.470 INFO:tasks.rados.rados.0.smithi165.stdout:2681: done (5 left) 2024-09-10T16:23:56.470 INFO:tasks.rados.rados.0.smithi165.stdout:2684: write (excl) oid 24 current snap is 285 2024-09-10T16:23:56.470 INFO:tasks.rados.rados.0.smithi165.stdout:2684: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2024-09-10T16:23:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:2684: writing smithi16534841-24 from 682615 to 1420692 tid 1 2024-09-10T16:23:56.485 INFO:tasks.rados.rados.0.smithi165.stdout:2684: writing smithi16534841-24 from 2015719 to 2553378 tid 2 2024-09-10T16:23:56.491 INFO:tasks.rados.rados.0.smithi165.stdout:2684: writing smithi16534841-24 from 3217787 to 3638545 tid 3 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2685: read oid 18 snap -1 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2685: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2686: read oid 49 snap -1 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2686: expect deleted 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2687: write oid 29 current snap is 285 2024-09-10T16:23:56.492 INFO:tasks.rados.rados.0.smithi165.stdout:2687: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2024-09-10T16:23:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2687: writing smithi16534841-29 from 699422 to 1112748 tid 1 2024-09-10T16:23:56.504 INFO:tasks.rados.rados.0.smithi165.stdout:2687: writing smithi16534841-29 from 1557848 to 2239165 tid 2 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2687: writing smithi16534841-29 from 3012504 to 3760887 tid 3 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2688: delete oid 35 current snap is 285 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2677: finishing write tid 3 to smithi16534841-12 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2679: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2677: finishing write tid 4 to smithi16534841-12 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2680: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2679: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:23:56.513 INFO:tasks.rados.rados.0.smithi165.stdout:2678: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:23:56.533 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:56.558 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 290338 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: finishing write tid 5 to smithi16534841-12 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: oid 12 updating version 0 to 1366 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: oid 12 version 1366 is already newer than 1365 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: oid 12 updating version 1366 to 1367 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: oid 12 updating version 1367 to 1368 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1368 (ObjNum 944 snap 285 seq_num 944) dirty exists 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: left oid 12 (ObjNum 944 snap 285 seq_num 944) 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2677: done (9 left) 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2678: done (8 left) 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2688: done (7 left) 2024-09-10T16:23:56.579 INFO:tasks.rados.rados.0.smithi165.stdout:2689: read oid 30 snap 274 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2689: expect (ObjNum 892 snap 274 seq_num 892) 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2690: copy_from oid 45 from oid 26 current snap is 285 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2679: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1230 (ObjNum 857 snap 263 seq_num 857) dirty exists 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2680: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2680: finishing rollback tid 2 to smithi16534841-22 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1414 (ObjNum 913 snap 280 seq_num 913) dirty exists 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:2683: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:23:56.580 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:56.628 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2917678 2024-09-10T16:23:56.643 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1223601 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: finishing write tid 1 to smithi16534841-24 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: finishing write tid 2 to smithi16534841-24 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: finishing write tid 3 to smithi16534841-24 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: finishing write tid 4 to smithi16534841-24 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: finishing write tid 5 to smithi16534841-24 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: oid 24 updating version 0 to 1569 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: oid 24 updating version 1569 to 1570 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: oid 24 updating version 1570 to 1571 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: oid 24 updating version 1571 to 1572 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1572 (ObjNum 945 snap 285 seq_num 945) dirty exists 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2684: left oid 24 (ObjNum 945 snap 285 seq_num 945) 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:2685: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:23:56.664 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:23:56.712 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1358671 2024-09-10T16:23:56.733 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 466633 2024-09-10T16:23:56.764 INFO:tasks.rados.rados.0.smithi165.stdout:2687: finishing write tid 1 to smithi16534841-29 2024-09-10T16:23:56.764 INFO:tasks.rados.rados.0.smithi165.stdout:2687: finishing write tid 2 to smithi16534841-29 2024-09-10T16:23:56.764 INFO:tasks.rados.rados.0.smithi165.stdout:2687: finishing write tid 3 to smithi16534841-29 2024-09-10T16:23:56.764 INFO:tasks.rados.rados.0.smithi165.stdout:2687: finishing write tid 4 to smithi16534841-29 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: finishing write tid 5 to smithi16534841-29 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: oid 29 updating version 0 to 1576 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: oid 29 version 1576 is already newer than 1573 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: oid 29 version 1576 is already newer than 1575 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: oid 29 version 1576 is already newer than 1574 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1576 (ObjNum 946 snap 285 seq_num 946) dirty exists 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2687: left oid 29 (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:23:56.765 INFO:tasks.rados.rados.0.smithi165.stdout:2689: expect (ObjNum 892 snap 274 seq_num 892) 2024-09-10T16:23:56.934 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2921967 2024-09-10T16:23:56.935 INFO:tasks.rados.rados.0.smithi165.stdout:2690: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:23:56.935 INFO:tasks.rados.rados.0.smithi165.stdout:2690: finishing copy_from to smithi16534841-45 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1662 (ObjNum 930 snap 285 seq_num 930) dirty exists 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2679: done (8 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2680: done (7 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2683: done (6 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2684: done (5 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2685: done (4 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2686: done (3 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2687: done (2 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2689: done (1 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2690: done (0 left) 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2691: rollback oid 17 current snap is 285 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 269 2024-09-10T16:23:56.936 INFO:tasks.rados.rados.0.smithi165.stdout:2692: snap_create 2024-09-10T16:23:56.941 INFO:tasks.rados.rados.0.smithi165.stdout:2691: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:23:56.941 INFO:tasks.rados.rados.0.smithi165.stdout:2691: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:23:56.941 INFO:tasks.rados.rados.0.smithi165.stdout:2691: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:23:56.941 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1547 (ObjNum 856 snap 263 seq_num 856) dirty exists 2024-09-10T16:23:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:2691: done (1 left) 2024-09-10T16:23:57.620 INFO:tasks.rados.rados.0.smithi165.stdout:2692: done (0 left) 2024-09-10T16:23:57.621 INFO:tasks.rados.rados.0.smithi165.stdout:2693: snap_remove snap 274 2024-09-10T16:23:58.623 INFO:tasks.rados.rados.0.smithi165.stdout:2693: done (0 left) 2024-09-10T16:23:58.623 INFO:tasks.rados.rados.0.smithi165.stdout:2694: snap_remove snap 269 2024-09-10T16:23:59.633 INFO:tasks.rados.rados.0.smithi165.stdout:2694: done (0 left) 2024-09-10T16:23:59.633 INFO:tasks.rados.rados.0.smithi165.stdout:2695: write oid 3 current snap is 286 2024-09-10T16:23:59.633 INFO:tasks.rados.rados.0.smithi165.stdout:2695: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2024-09-10T16:23:59.640 INFO:tasks.rados.rados.0.smithi165.stdout:2695: writing smithi16534841-3 from 733036 to 1296860 tid 1 2024-09-10T16:23:59.649 INFO:tasks.rados.rados.0.smithi165.stdout:2695: writing smithi16534841-3 from 2074812 to 2759798 tid 2 2024-09-10T16:23:59.656 INFO:tasks.rados.rados.0.smithi165.stdout:2695: writing smithi16534841-3 from 3183703 to 3933036 tid 3 2024-09-10T16:23:59.656 INFO:tasks.rados.rados.0.smithi165.stdout:2696: copy_from oid 23 from oid 4 current snap is 286 2024-09-10T16:23:59.656 INFO:tasks.rados.rados.0.smithi165.stdout:2695: finishing write tid 1 to smithi16534841-3 2024-09-10T16:23:59.656 INFO:tasks.rados.rados.0.smithi165.stdout:2697: delete oid 13 current snap is 286 2024-09-10T16:23:59.660 INFO:tasks.rados.rados.0.smithi165.stdout:2697: done (2 left) 2024-09-10T16:23:59.660 INFO:tasks.rados.rados.0.smithi165.stdout:2698: copy_from oid 27 from oid 7 current snap is 286 2024-09-10T16:23:59.661 INFO:tasks.rados.rados.0.smithi165.stdout:2699: copy_from oid 34 from oid 12 current snap is 286 2024-09-10T16:23:59.661 INFO:tasks.rados.rados.0.smithi165.stdout:2700: snap_create 2024-09-10T16:23:59.664 INFO:tasks.rados.rados.0.smithi165.stdout:2695: finishing write tid 2 to smithi16534841-3 2024-09-10T16:23:59.665 INFO:tasks.rados.rados.0.smithi165.stdout:2695: finishing write tid 3 to smithi16534841-3 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: finishing write tid 4 to smithi16534841-3 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: finishing write tid 5 to smithi16534841-3 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: oid 3 updating version 0 to 1528 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: oid 3 version 1528 is already newer than 1522 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: oid 3 updating version 1528 to 1529 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: oid 3 version 1529 is already newer than 1523 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1529 (ObjNum 948 snap 286 seq_num 948) dirty exists 2024-09-10T16:23:59.668 INFO:tasks.rados.rados.0.smithi165.stdout:2695: left oid 3 (ObjNum 948 snap 286 seq_num 948) 2024-09-10T16:23:59.676 INFO:tasks.rados.rados.0.smithi165.stdout:2696: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:23:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:2698: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:23:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:2696: finishing copy_from to smithi16534841-23 2024-09-10T16:23:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1333 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:23:59.683 INFO:tasks.rados.rados.0.smithi165.stdout:2698: finishing copy_from to smithi16534841-27 2024-09-10T16:23:59.683 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1606 (ObjNum 909 snap 278 seq_num 909) dirty exists 2024-09-10T16:23:59.684 INFO:tasks.rados.rados.0.smithi165.stdout:2699: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:23:59.690 INFO:tasks.rados.rados.0.smithi165.stdout:2699: finishing copy_from to smithi16534841-34 2024-09-10T16:23:59.690 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1368 (ObjNum 944 snap 285 seq_num 944) dirty exists 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2695: done (4 left) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2696: done (3 left) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2698: done (2 left) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2699: done (1 left) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2700: done (0 left) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2701: copy_from oid 10 from oid 35 current snap is 287 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2702: read oid 25 snap -1 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2702: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2703: read oid 11 snap -1 2024-09-10T16:24:00.636 INFO:tasks.rados.rados.0.smithi165.stdout:2703: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:00.637 INFO:tasks.rados.rados.0.smithi165.stdout:2704: delete oid 32 current snap is 287 2024-09-10T16:24:00.640 INFO:tasks.rados.rados.0.smithi165.stdout:2703: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:00.640 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 316944 2024-09-10T16:24:00.672 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:00.694 INFO:tasks.rados.rados.0.smithi165.stdout:2701: finishing copy_from to smithi16534841-10 2024-09-10T16:24:00.694 INFO:tasks.rados.rados.0.smithi165.stdout:2701: got expected ENOENT (src dne) 2024-09-10T16:24:00.694 INFO:tasks.rados.rados.0.smithi165.stdout:2701: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:24:00.694 INFO:tasks.rados.rados.0.smithi165.stdout:2702: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:00.733 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 144049 2024-09-10T16:24:00.777 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1203638 2024-09-10T16:24:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:2701: done (3 left) 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2702: done (2 left) 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2703: done (1 left) 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2704: done (0 left) 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2705: copy_from oid 7 from oid 42 current snap is 287 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2706: read oid 6 snap -1 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2706: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2707: write oid 28 current snap is 287 2024-09-10T16:24:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:2707: seq_num 951 ranges {783457=505924,1850258=548666,2865972=721096} 2024-09-10T16:24:00.809 INFO:tasks.rados.rados.0.smithi165.stdout:2707: writing smithi16534841-28 from 783457 to 1289381 tid 1 2024-09-10T16:24:00.817 INFO:tasks.rados.rados.0.smithi165.stdout:2707: writing smithi16534841-28 from 1850258 to 2398924 tid 2 2024-09-10T16:24:00.823 INFO:tasks.rados.rados.0.smithi165.stdout:2707: writing smithi16534841-28 from 2865972 to 3587068 tid 3 2024-09-10T16:24:00.823 INFO:tasks.rados.rados.0.smithi165.stdout:2708: rollback oid 41 current snap is 287 2024-09-10T16:24:00.823 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 284 2024-09-10T16:24:00.823 INFO:tasks.rados.rados.0.smithi165.stdout:2706: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:00.836 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1051112 2024-09-10T16:24:00.867 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2707: finishing write tid 1 to smithi16534841-28 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2705: finishing copy_from racing read to smithi16534841-7 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2705: finishing copy_from to smithi16534841-7 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1280 (ObjNum 935 snap 285 seq_num 935) dirty exists 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2707: finishing write tid 2 to smithi16534841-28 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2707: finishing write tid 3 to smithi16534841-28 2024-09-10T16:24:00.954 INFO:tasks.rados.rados.0.smithi165.stdout:2708: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 0 (ObjNum 129 snap 0 seq_num 1162058544) dirty dne 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: finishing write tid 4 to smithi16534841-28 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: finishing write tid 5 to smithi16534841-28 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: oid 28 updating version 0 to 1265 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: oid 28 updating version 1265 to 1266 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: oid 28 version 1266 is already newer than 1264 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: oid 28 updating version 1266 to 1267 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1267 (ObjNum 951 snap 287 seq_num 951) dirty exists 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: left oid 28 (ObjNum 951 snap 287 seq_num 951) 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2705: done (3 left) 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2706: done (2 left) 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2707: done (1 left) 2024-09-10T16:24:00.955 INFO:tasks.rados.rados.0.smithi165.stdout:2708: done (0 left) 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2709: read oid 32 snap 285 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2709: expect deleted 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2710: copy_from oid 40 from oid 10 current snap is 287 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2711: copy_from oid 3 from oid 46 current snap is 287 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2712: read oid 47 snap -1 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2712: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2713: write oid 48 current snap is 287 2024-09-10T16:24:00.956 INFO:tasks.rados.rados.0.smithi165.stdout:2713: seq_num 952 ranges {400264=581173,1508740=491524} 2024-09-10T16:24:00.959 INFO:tasks.rados.rados.0.smithi165.stdout:2713: writing smithi16534841-48 from 400264 to 981437 tid 1 2024-09-10T16:24:00.965 INFO:tasks.rados.rados.0.smithi165.stdout:2713: writing smithi16534841-48 from 1508740 to 2000264 tid 2 2024-09-10T16:24:00.965 INFO:tasks.rados.rados.0.smithi165.stdout:2714: rollback oid 39 current snap is 287 2024-09-10T16:24:00.965 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 283 2024-09-10T16:24:00.965 INFO:tasks.rados.rados.0.smithi165.stdout:2711: finishing copy_from to smithi16534841-3 2024-09-10T16:24:00.966 INFO:tasks.rados.rados.0.smithi165.stdout:2711: got expected ENOENT (src dne) 2024-09-10T16:24:00.966 INFO:tasks.rados.rados.0.smithi165.stdout:2711: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:24:00.966 INFO:tasks.rados.rados.0.smithi165.stdout:2712: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:24:01.049 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2512044 2024-09-10T16:24:01.056 INFO:tasks.rados.rados.0.smithi165.stdout:2710: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2710: finishing copy_from to smithi16534841-40 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: finishing write tid 1 to smithi16534841-48 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2714: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2714: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: finishing write tid 2 to smithi16534841-48 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2714: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1410 (ObjNum 860 snap 263 seq_num 860) dirty exists 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: finishing write tid 3 to smithi16534841-48 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: finishing write tid 4 to smithi16534841-48 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: oid 48 updating version 0 to 1722 2024-09-10T16:24:01.057 INFO:tasks.rados.rados.0.smithi165.stdout:2713: oid 48 updating version 1722 to 1723 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2713: oid 48 updating version 1723 to 1724 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1724 (ObjNum 952 snap 287 seq_num 952) dirty exists 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2713: left oid 48 (ObjNum 952 snap 287 seq_num 952) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2709: done (5 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2710: done (4 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2711: done (3 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2712: done (2 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2713: done (1 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2714: done (0 left) 2024-09-10T16:24:01.058 INFO:tasks.rados.rados.0.smithi165.stdout:2715: snap_remove snap 286 2024-09-10T16:24:01.469 INFO:tasks.rados.rados.0.smithi165.stdout:2715: done (0 left) 2024-09-10T16:24:01.469 INFO:tasks.rados.rados.0.smithi165.stdout:2716: snap_remove snap 268 2024-09-10T16:24:02.472 INFO:tasks.rados.rados.0.smithi165.stdout:2716: done (0 left) 2024-09-10T16:24:02.472 INFO:tasks.rados.rados.0.smithi165.stdout:2717: read oid 35 snap -1 2024-09-10T16:24:02.472 INFO:tasks.rados.rados.0.smithi165.stdout:2717: expect deleted 2024-09-10T16:24:02.472 INFO:tasks.rados.rados.0.smithi165.stdout:2718: write (excl) oid 28 current snap is 287 2024-09-10T16:24:02.472 INFO:tasks.rados.rados.0.smithi165.stdout:2718: seq_num 953 ranges {417071=656422,1567222=449849} 2024-09-10T16:24:02.478 INFO:tasks.rados.rados.0.smithi165.stdout:2718: writing smithi16534841-28 from 417071 to 1073493 tid 1 2024-09-10T16:24:02.484 INFO:tasks.rados.rados.0.smithi165.stdout:2718: writing smithi16534841-28 from 1567222 to 2017071 tid 2 2024-09-10T16:24:02.484 INFO:tasks.rados.rados.0.smithi165.stdout:2719: rollback oid 42 current snap is 287 2024-09-10T16:24:02.485 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 283 2024-09-10T16:24:02.485 INFO:tasks.rados.rados.0.smithi165.stdout:2717: done (2 left) 2024-09-10T16:24:02.485 INFO:tasks.rados.rados.0.smithi165.stdout:2720: read oid 29 snap -1 2024-09-10T16:24:02.485 INFO:tasks.rados.rados.0.smithi165.stdout:2720: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:24:02.485 INFO:tasks.rados.rados.0.smithi165.stdout:2721: snap_remove snap 285 2024-09-10T16:24:03.478 INFO:tasks.rados.rados.0.smithi165.stdout:2721: done (3 left) 2024-09-10T16:24:03.478 INFO:tasks.rados.rados.0.smithi165.stdout:2722: rollback oid 15 current snap is 287 2024-09-10T16:24:03.478 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 283 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:2719: finishing rollback tid 0 to smithi16534841-42 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:2718: finishing write tid 1 to smithi16534841-28 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:2719: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:2719: finishing rollback tid 2 to smithi16534841-42 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1283 (ObjNum 758 snap 230 seq_num 758) dirty exists 2024-09-10T16:24:03.479 INFO:tasks.rados.rados.0.smithi165.stdout:2720: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: finishing write tid 2 to smithi16534841-28 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: finishing write tid 3 to smithi16534841-28 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: finishing write tid 4 to smithi16534841-28 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: oid 28 updating version 0 to 1268 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: oid 28 updating version 1268 to 1269 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: oid 28 updating version 1269 to 1270 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1270 (ObjNum 953 snap 287 seq_num 953) dirty exists 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2718: left oid 28 (ObjNum 953 snap 287 seq_num 953) 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:2722: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:24:03.678 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-10T16:24:03.679 INFO:tasks.rados.rados.0.smithi165.stdout:2718: done (3 left) 2024-09-10T16:24:03.679 INFO:tasks.rados.rados.0.smithi165.stdout:2719: done (2 left) 2024-09-10T16:24:03.679 INFO:tasks.rados.rados.0.smithi165.stdout:2720: done (1 left) 2024-09-10T16:24:03.679 INFO:tasks.rados.rados.0.smithi165.stdout:2722: done (0 left) 2024-09-10T16:24:03.679 INFO:tasks.rados.rados.0.smithi165.stdout:2723: snap_remove snap 284 2024-09-10T16:24:04.486 INFO:tasks.rados.rados.0.smithi165.stdout:2723: done (0 left) 2024-09-10T16:24:04.487 INFO:tasks.rados.rados.0.smithi165.stdout:2724: write oid 36 current snap is 287 2024-09-10T16:24:04.487 INFO:tasks.rados.rados.0.smithi165.stdout:2724: seq_num 954 ranges {433878=731671,1909351=124527} 2024-09-10T16:24:04.493 INFO:tasks.rados.rados.0.smithi165.stdout:2724: writing smithi16534841-36 from 433878 to 1165549 tid 1 2024-09-10T16:24:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:2724: writing smithi16534841-36 from 1909351 to 2033878 tid 2 2024-09-10T16:24:04.498 INFO:tasks.rados.rados.0.smithi165.stdout:2725: delete oid 28 current snap is 287 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2725: done (1 left) 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2726: read oid 30 snap 283 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2726: expect (ObjNum 892 snap 274 seq_num 892) 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2727: copy_from oid 28 from oid 11 current snap is 287 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2728: copy_from oid 25 from oid 43 current snap is 287 2024-09-10T16:24:04.501 INFO:tasks.rados.rados.0.smithi165.stdout:2729: snap_create 2024-09-10T16:24:04.506 INFO:tasks.rados.rados.0.smithi165.stdout:2724: finishing write tid 1 to smithi16534841-36 2024-09-10T16:24:04.506 INFO:tasks.rados.rados.0.smithi165.stdout:2724: finishing write tid 2 to smithi16534841-36 2024-09-10T16:24:04.508 INFO:tasks.rados.rados.0.smithi165.stdout:2726: expect (ObjNum 892 snap 274 seq_num 892) 2024-09-10T16:24:04.700 INFO:tasks.rados.rados.0.smithi165.stdout:2728: finishing copy_from to smithi16534841-25 2024-09-10T16:24:04.700 INFO:tasks.rados.rados.0.smithi165.stdout:2728: got expected ENOENT (src dne) 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2728: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2727: finishing copy_from racing read to smithi16534841-28 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: finishing write tid 3 to smithi16534841-36 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: finishing write tid 4 to smithi16534841-36 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: oid 36 updating version 0 to 1382 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: oid 36 updating version 1382 to 1384 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: oid 36 version 1384 is already newer than 1383 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1384 (ObjNum 954 snap 287 seq_num 954) dirty exists 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2724: left oid 36 (ObjNum 954 snap 287 seq_num 954) 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:2727: finishing copy_from to smithi16534841-28 2024-09-10T16:24:04.701 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:24:05.485 INFO:tasks.rados.rados.0.smithi165.stdout:2724: done (4 left) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2726: done (3 left) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2727: done (2 left) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2728: done (1 left) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2729: done (0 left) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2730: read oid 6 snap -1 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2730: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:05.486 INFO:tasks.rados.rados.0.smithi165.stdout:2731: snap_remove snap 283 2024-09-10T16:24:06.474 INFO:tasks.rados.rados.0.smithi165.stdout:2731: done (1 left) 2024-09-10T16:24:06.474 INFO:tasks.rados.rados.0.smithi165.stdout:2732: copy_from oid 13 from oid 50 current snap is 288 2024-09-10T16:24:06.474 INFO:tasks.rados.rados.0.smithi165.stdout:2730: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:06.517 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:06.615 INFO:tasks.rados.rados.0.smithi165.stdout:2732: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2732: finishing copy_from to smithi16534841-13 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1276 (ObjNum 922 snap 281 seq_num 922) dirty exists 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2730: done (1 left) 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2732: done (0 left) 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2733: read oid 40 snap -1 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2733: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2734: rollback oid 46 current snap is 288 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 287 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2735: snap_create 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:2734: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:24:06.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 0 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-10T16:24:06.617 INFO:tasks.rados.rados.0.smithi165.stdout:2733: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:06.619 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1221914 2024-09-10T16:24:06.664 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1991425 2024-09-10T16:24:07.500 INFO:tasks.rados.rados.0.smithi165.stdout:2733: done (2 left) 2024-09-10T16:24:07.500 INFO:tasks.rados.rados.0.smithi165.stdout:2734: done (1 left) 2024-09-10T16:24:07.500 INFO:tasks.rados.rados.0.smithi165.stdout:2735: done (0 left) 2024-09-10T16:24:07.501 INFO:tasks.rados.rados.0.smithi165.stdout:2736: snap_remove snap 288 2024-09-10T16:24:08.507 INFO:tasks.rados.rados.0.smithi165.stdout:2736: done (0 left) 2024-09-10T16:24:08.507 INFO:tasks.rados.rados.0.smithi165.stdout:2737: read oid 40 snap -1 2024-09-10T16:24:08.507 INFO:tasks.rados.rados.0.smithi165.stdout:2737: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:08.507 INFO:tasks.rados.rados.0.smithi165.stdout:2738: snap_remove snap 287 2024-09-10T16:24:09.511 INFO:tasks.rados.rados.0.smithi165.stdout:2738: done (1 left) 2024-09-10T16:24:09.511 INFO:tasks.rados.rados.0.smithi165.stdout:2739: delete oid 20 current snap is 289 2024-09-10T16:24:09.511 INFO:tasks.rados.rados.0.smithi165.stdout:2737: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:09.514 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 625658 2024-09-10T16:24:09.552 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:09.573 INFO:tasks.rados.rados.0.smithi165.stdout:2737: done (1 left) 2024-09-10T16:24:09.573 INFO:tasks.rados.rados.0.smithi165.stdout:2739: done (0 left) 2024-09-10T16:24:09.574 INFO:tasks.rados.rados.0.smithi165.stdout:2740: write (excl) oid 2 current snap is 289 2024-09-10T16:24:09.574 INFO:tasks.rados.rados.0.smithi165.stdout:2740: seq_num 957 ranges {484299=557418,1684797=399502} 2024-09-10T16:24:09.579 INFO:tasks.rados.rados.0.smithi165.stdout:2740: writing smithi16534841-2 from 484299 to 1041717 tid 1 2024-09-10T16:24:09.585 INFO:tasks.rados.rados.0.smithi165.stdout:2740: writing smithi16534841-2 from 1684797 to 2084299 tid 2 2024-09-10T16:24:09.585 INFO:tasks.rados.rados.0.smithi165.stdout:2741: delete oid 9 current snap is 289 2024-09-10T16:24:09.587 INFO:tasks.rados.rados.0.smithi165.stdout:2741: done (1 left) 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2742: read oid 6 snap -1 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2742: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2743: copy_from oid 38 from oid 30 current snap is 289 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2744: read oid 42 snap -1 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2744: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2745: write (excl) oid 45 current snap is 289 2024-09-10T16:24:09.588 INFO:tasks.rados.rados.0.smithi165.stdout:2745: seq_num 959 ranges {517913=424269,1401761=446989} 2024-09-10T16:24:09.592 INFO:tasks.rados.rados.0.smithi165.stdout:2745: writing smithi16534841-45 from 517913 to 942182 tid 1 2024-09-10T16:24:09.599 INFO:tasks.rados.rados.0.smithi165.stdout:2745: writing smithi16534841-45 from 1401761 to 1848750 tid 2 2024-09-10T16:24:09.599 INFO:tasks.rados.rados.0.smithi165.stdout:2746: snap_create 2024-09-10T16:24:09.599 INFO:tasks.rados.rados.0.smithi165.stdout:2740: finishing write tid 1 to smithi16534841-2 2024-09-10T16:24:09.599 INFO:tasks.rados.rados.0.smithi165.stdout:2740: finishing write tid 2 to smithi16534841-2 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: finishing write tid 3 to smithi16534841-2 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: finishing write tid 4 to smithi16534841-2 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: oid 2 updating version 0 to 1608 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: oid 2 version 1608 is already newer than 1607 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: oid 2 updating version 1608 to 1609 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1609 (ObjNum 957 snap 289 seq_num 957) dirty exists 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2740: left oid 2 (ObjNum 957 snap 289 seq_num 957) 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:2744: expect (ObjNum 758 snap 230 seq_num 758) 2024-09-10T16:24:09.600 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:09.630 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:09.696 INFO:tasks.rados.rados.0.smithi165.stdout:2742: expect (ObjNum 863 snap 265 seq_num 863) 2024-09-10T16:24:09.742 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:09.811 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1397218 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: finishing write tid 1 to smithi16534841-45 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2743: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2743: finishing copy_from to smithi16534841-38 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1260 (ObjNum 871 snap 266 seq_num 871) dirty exists 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: finishing write tid 2 to smithi16534841-45 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: finishing write tid 3 to smithi16534841-45 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: finishing write tid 4 to smithi16534841-45 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: oid 45 updating version 0 to 1663 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: oid 45 updating version 1663 to 1664 2024-09-10T16:24:09.883 INFO:tasks.rados.rados.0.smithi165.stdout:2745: oid 45 updating version 1664 to 1665 2024-09-10T16:24:09.884 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1665 (ObjNum 959 snap 289 seq_num 959) dirty exists 2024-09-10T16:24:09.884 INFO:tasks.rados.rados.0.smithi165.stdout:2745: left oid 45 (ObjNum 959 snap 289 seq_num 959) 2024-09-10T16:24:10.169 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:24:10.170 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:24:10.170 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:24:10.173 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:24:10.174 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:24:10.174 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.282+0000 7f81dd423640 10 monclient: get_monmap_and_config 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.282+0000 7f81dd423640 10 monclient: build_initial_monmap 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.282+0000 7f81dd423640 10 monclient: monmap: 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:10.283734+0000 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:10.283734+0000 2024-09-10T16:24:10.284 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:10.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.283+0000 7f81dd423640 1 Processor -- start 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.283+0000 7f81dd423640 1 -- start start 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.283+0000 7f81dd423640 10 monclient: init 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f81d806cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81d813d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-b con 0x7f81d8134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f81d813d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d809ce50 con 0x7f81d813d560 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81d8135380 con 0x7f81d806cb70 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81d806db60 con 0x7f81d8134f70 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81dd423640 10 monclient(hunting): authenticate will time out at 1828.771295s 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81d77fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f81d813d560 0x7f81d813f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51460/0 (socket says 172.21.15.164:51460) 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81d813d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 learned_addr learned my addr 172.21.15.164:0/356094311 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d8134f70 auth_method 0 2024-09-10T16:24:10.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 msgr2=0x7f81d813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 msgr2=0x7f81d813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81d813d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:10.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.284+0000 7f81cffff640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81d813d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:10.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 >> v1:172.21.15.164:6789/0 conn(0x7f81d813d560 legacy=0x7f81d813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2091821273 0 0) 0x7f81d00017d0 con 0x7f81d813d560 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient(hunting): _negotiate 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient(hunting): my global_id is 164442 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81c4003520 con 0x7f81d813d560 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d6ffd640 1 -- 172.21.15.164:0/356094311 >> v1:172.21.15.165:6789/0 conn(0x7f81d806cb70 legacy=0x7f81d80a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 114567405 0 0) 0x7f81c4003520 con 0x7f81d813d560 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 msgr2=0x7f81d813d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81d813d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 >> v1:172.21.15.165:6789/0 conn(0x7f81d806cb70 legacy=0x7f81d80a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:10.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: _finish_hunting 0 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d809cb70 con 0x7f81d813d560 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: _finish_auth 0 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f81d0001ef0 con 0x7f81d813d560 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81dd423640 5 monclient: authenticate success, global_id 164442 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81dd423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:10.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: dump: 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.286+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 >> v1:172.21.15.164:6789/0 conn(0x7f81d813d560 legacy=0x7f81d813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4003af0 0x7f81c4003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): picked mon.b con 0x7f81c4003af0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81c4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): picked mon.c con 0x7f81d8134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 0x7f81c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): picked mon.a con 0x7f81c4004f40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81c400bf00 con 0x7f81c4004f40 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f81c400c080 con 0x7f81c4003af0 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81c400c200 con 0x7f81d8134f70 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d67fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81dd423640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81c4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 0x7f81c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d8134f70 auth_method 0 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:10.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4003af0 0x7f81c4003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): get_auth_request con 0x7f81c4004f40 auth_method 0 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): get_auth_request con 0x7f81c4003af0 auth_method 0 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:10.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient(hunting): handle_auth_done global_id 164442 payload 293 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 msgr2=0x7f81c4004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81c4004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4003af0 msgr2=0x7f81c4003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4003af0 0x7f81c4003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:24:10.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 monclient: found mon.a 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c4003520 con 0x7f81c4004f40 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.287+0000 7f81cffff640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d8134f70 0x7f81c4004800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 msgr2=0x7f81c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d77fe640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 0x7f81c400b9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f81d0002110 tx=0x7f81c4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e9439c31068d291 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0005440 con 0x7f81c4004f40 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:10.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: dump: 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d002f030 con 0x7f81c4004f40 2024-09-10T16:24:10.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 1 -- 172.21.15.164:0/356094311 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d00080d0 con 0x7f81c4004f40 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 4 set_mon_vals no callback set 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d67fc640 10 monclient: dump: 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.308 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 10 monclient: shutdown 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 1 -- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 msgr2=0x7f81c400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81dd423640 1 --2- 172.21.15.164:0/356094311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81c4004f40 0x7f81c400b9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f81d0002110 tx=0x7f81c4005830 comp rx=0 tx=0).stop 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.290+0000 7f81d77fe640 1 -- 172.21.15.164:0/356094311 reap_dead start 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 1 -- 172.21.15.164:0/356094311 shutdown_connections 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 1 -- 172.21.15.164:0/356094311 >> 172.21.15.164:0/356094311 conn(0x7f81d80aa900 msgr2=0x7f81d80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 1 -- 172.21.15.164:0/356094311 shutdown_connections 2024-09-10T16:24:10.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 1 -- 172.21.15.164:0/356094311 wait complete. 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 10 monclient: build_initial_monmap 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 10 monclient: monmap: 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:10.310 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.291+0000 7f81dd423640 1 Processor -- start 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 -- start start 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient: init 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4004f40 0x7f81d809dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-b con 0x7f81c4004f40 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-c con 0x7f81d809e110 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): picked mon.noname-a con 0x7f81d81d1720 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f81d80a4db0 con 0x7f81d81d1720 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f81d806cf50 con 0x7f81c4004f40 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f81d813b010 con 0x7f81d809e110 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81dd423640 10 monclient(hunting): authenticate will time out at 1828.779485s 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37904/0 (socket says 172.21.15.164:37904) 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55928/0 (socket says 172.21.15.164:55928) 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 -- 172.21.15.164:0/1658081107 learned_addr learned my addr 172.21.15.164:0/1658081107 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): get_auth_request con 0x7f81d81d1720 auth_method 0 2024-09-10T16:24:10.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 10 monclient(hunting): get_auth_request con 0x7f81d809e110 auth_method 0 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 msgr2=0x7f81d809e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 msgr2=0x7f81d809e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:10.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81cffff640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d6ffd640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4004f40 0x7f81d809dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d6ffd640 10 monclient(hunting): get_auth_request con 0x7f81c4004f40 auth_method 0 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient(hunting): handle_auth_done global_id 164445 payload 824 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 msgr2=0x7f81d809e590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4004f40 msgr2=0x7f81d809dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4004f40 0x7f81d809dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 monclient: found mon.noname-a 2024-09-10T16:24:10.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.292+0000 7f81d77fe640 1 -- 172.21.15.164:0/1658081107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81d80a19f0 con 0x7f81d81d1720 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81d77fe640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 msgr2=0x7f81d81d3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81d77fe640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81d00057d0 tx=0x7f81d809cd40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c000b5e0 con 0x7f81d81d1720 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: dump: 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:10.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81d002bc60 con 0x7f81d81d1720 2024-09-10T16:24:10.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 5 monclient: authenticate success, global_id 164445 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81d0042680 con 0x7f81d81d1720 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 10 monclient: _renew_subs 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81d8133fd0 con 0x7f81d81d1720 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81cf7fe640 10 monclient: dump: 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:10.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81d5ffb640 4 set_mon_vals no callback set 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 10 monclient: _renew_subs 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.295+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81d806e720 con 0x7f81d81d1720 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f81d00080d0 con 0x7f81d81d1720 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81dd423640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81dd423640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81dd423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81d80a46a0 con 0x7f81d81d1720 2024-09-10T16:24:10.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81cf7fe640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 0x7f81b80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:10.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(612..612 src has 481..612) ==== 4783+0+0 (secure 0 0 0) 0x7f81d00731a0 con 0x7f81d81d1720 2024-09-10T16:24:10.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81d6ffd640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 0x7f81b80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:10.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.296+0000 7f81d6ffd640 10 monclient: get_auth_request con 0x7f81b803e210 auth_method 0 2024-09-10T16:24:10.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.299+0000 7f81d6ffd640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 0x7f81b80406d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f81c8000c00 tx=0x7f81c8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:10.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.301+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f81d809ce50 con 0x7f81d81d1720 2024-09-10T16:24:10.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.301+0000 7f81cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:10.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.301+0000 7f81cf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.552+0000 7f81dd423640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:10.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.553+0000 7f81dd423640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.553+0000 7f81dd423640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:10.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:10.553+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f81d80770f0 con 0x7f81d81d1720 2024-09-10T16:24:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:13.292+0000 7f81ce7fc640 10 monclient: tick 2024-09-10T16:24:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:13.292+0000 7f81ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:13.292+0000 7f81ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:13.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:13.292+0000 7f81ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:14.292+0000 7f81ce7fc640 10 monclient: tick 2024-09-10T16:24:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:14.292+0000 7f81ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:14.292+0000 7f81ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.292+0000 7f81ce7fc640 10 monclient: tick 2024-09-10T16:24:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.292+0000 7f81ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:15.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.292+0000 7f81ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:15.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.497+0000 7f81cf7fe640 1 -- 172.21.15.164:0/1658081107 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f81d002be10 con 0x7f81d81d1720 2024-09-10T16:24:15.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.498+0000 7f81cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:15.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.498+0000 7f81cf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:15.499 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":174,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 msgr2=0x7f81b80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 0x7f81b80406d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f81c8000c00 tx=0x7f81c8009040 comp rx=0 tx=0).stop 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 10 monclient: shutdown 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 msgr2=0x7f81d81d3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f81d00057d0 tx=0x7f81d809cd40 comp rx=0 tx=0).stop 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 shutdown_connections 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f81b803e210 0x7f81b80406d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:15.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f81d81d1720 0x7f81d81d3b10 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f81d809e110 0x7f81d809e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 --2- 172.21.15.164:0/1658081107 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f81c4004f40 0x7f81d809dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 >> 172.21.15.164:0/1658081107 conn(0x7f81d80aa900 msgr2=0x7f81d80a9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 shutdown_connections 2024-09-10T16:24:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:15.502+0000 7f81dd423640 1 -- 172.21.15.164:0/1658081107 wait complete. 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2740: done (5 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2742: done (4 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2743: done (3 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2744: done (2 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2745: done (1 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2746: done (0 left) 2024-09-10T16:24:15.569 INFO:tasks.rados.rados.0.smithi165.stdout:2747: snap_create 2024-09-10T16:24:16.474 INFO:tasks.rados.rados.0.smithi165.stdout:2747: done (0 left) 2024-09-10T16:24:16.474 INFO:tasks.rados.rados.0.smithi165.stdout:2748: write oid 15 current snap is 291 2024-09-10T16:24:16.475 INFO:tasks.rados.rados.0.smithi165.stdout:2748: seq_num 960 ranges {534720=499518,1460243=674477} 2024-09-10T16:24:16.480 INFO:tasks.rados.rados.0.smithi165.stdout:2748: writing smithi16534841-15 from 534720 to 1034238 tid 1 2024-09-10T16:24:16.487 INFO:tasks.rados.rados.0.smithi165.stdout:2748: writing smithi16534841-15 from 1460243 to 2134720 tid 2 2024-09-10T16:24:16.487 INFO:tasks.rados.rados.0.smithi165.stdout:2749: snap_create 2024-09-10T16:24:16.487 INFO:tasks.rados.rados.0.smithi165.stdout:2748: finishing write tid 1 to smithi16534841-15 2024-09-10T16:24:16.490 INFO:tasks.rados.rados.0.smithi165.stdout:2748: finishing write tid 2 to smithi16534841-15 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: finishing write tid 3 to smithi16534841-15 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: finishing write tid 4 to smithi16534841-15 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: oid 15 updating version 0 to 1335 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: oid 15 updating version 1335 to 1336 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: oid 15 version 1336 is already newer than 1334 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1336 (ObjNum 960 snap 291 seq_num 960) dirty exists 2024-09-10T16:24:16.491 INFO:tasks.rados.rados.0.smithi165.stdout:2748: left oid 15 (ObjNum 960 snap 291 seq_num 960) 2024-09-10T16:24:17.475 INFO:tasks.rados.rados.0.smithi165.stdout:2748: done (1 left) 2024-09-10T16:24:17.475 INFO:tasks.rados.rados.0.smithi165.stdout:2749: done (0 left) 2024-09-10T16:24:17.475 INFO:tasks.rados.rados.0.smithi165.stdout:2750: snap_remove snap 291 2024-09-10T16:24:18.485 INFO:tasks.rados.rados.0.smithi165.stdout:2750: done (0 left) 2024-09-10T16:24:18.485 INFO:tasks.rados.rados.0.smithi165.stdout:2751: read oid 47 snap 290 2024-09-10T16:24:18.485 INFO:tasks.rados.rados.0.smithi165.stdout:2751: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:24:18.485 INFO:tasks.rados.rados.0.smithi165.stdout:2752: write (excl) oid 30 current snap is 292 2024-09-10T16:24:18.486 INFO:tasks.rados.rados.0.smithi165.stdout:2752: seq_num 961 ranges {551527=574767,1918725=232802} 2024-09-10T16:24:18.491 INFO:tasks.rados.rados.0.smithi165.stdout:2752: writing smithi16534841-30 from 551527 to 1126294 tid 1 2024-09-10T16:24:18.498 INFO:tasks.rados.rados.0.smithi165.stdout:2752: writing smithi16534841-30 from 1918725 to 2151527 tid 2 2024-09-10T16:24:18.498 INFO:tasks.rados.rados.0.smithi165.stdout:2753: write oid 44 current snap is 292 2024-09-10T16:24:18.498 INFO:tasks.rados.rados.0.smithi165.stdout:2753: seq_num 962 ranges {568334=650016,1860854=307480} 2024-09-10T16:24:18.504 INFO:tasks.rados.rados.0.smithi165.stdout:2753: writing smithi16534841-44 from 568334 to 1218350 tid 1 2024-09-10T16:24:18.510 INFO:tasks.rados.rados.0.smithi165.stdout:2753: writing smithi16534841-44 from 1860854 to 2168334 tid 2 2024-09-10T16:24:18.510 INFO:tasks.rados.rados.0.smithi165.stdout:2754: read oid 41 snap 290 2024-09-10T16:24:18.510 INFO:tasks.rados.rados.0.smithi165.stdout:2754: expect deleted 2024-09-10T16:24:18.510 INFO:tasks.rados.rados.0.smithi165.stdout:2755: write (excl) oid 40 current snap is 292 2024-09-10T16:24:18.510 INFO:tasks.rados.rados.0.smithi165.stdout:2755: seq_num 963 ranges {585141=725265,1919336=265805} 2024-09-10T16:24:18.517 INFO:tasks.rados.rados.0.smithi165.stdout:2755: writing smithi16534841-40 from 585141 to 1310406 tid 1 2024-09-10T16:24:18.519 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:18.522 INFO:tasks.rados.rados.0.smithi165.stdout:2755: writing smithi16534841-40 from 1919336 to 2185141 tid 2 2024-09-10T16:24:18.522 INFO:tasks.rados.rados.0.smithi165.stdout:2756: copy_from oid 21 from oid 49 current snap is 292 2024-09-10T16:24:18.523 INFO:tasks.rados.rados.0.smithi165.stdout:2751: expect (ObjNum 901 snap 278 seq_num 901) 2024-09-10T16:24:18.523 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:18.586 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2988822 2024-09-10T16:24:18.594 INFO:tasks.rados.rados.0.smithi165.stdout:2752: finishing write tid 1 to smithi16534841-30 2024-09-10T16:24:18.594 INFO:tasks.rados.rados.0.smithi165.stdout:2752: finishing write tid 2 to smithi16534841-30 2024-09-10T16:24:18.594 INFO:tasks.rados.rados.0.smithi165.stdout:2752: finishing write tid 3 to smithi16534841-30 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: finishing write tid 4 to smithi16534841-30 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: oid 30 updating version 0 to 1762 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: oid 30 version 1762 is already newer than 1761 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: oid 30 version 1762 is already newer than 1760 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1762 (ObjNum 961 snap 292 seq_num 961) dirty exists 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: left oid 30 (ObjNum 961 snap 292 seq_num 961) 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2753: finishing write tid 1 to smithi16534841-44 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2751: done (5 left) 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2752: done (4 left) 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2754: done (3 left) 2024-09-10T16:24:18.595 INFO:tasks.rados.rados.0.smithi165.stdout:2757: snap_remove snap 289 2024-09-10T16:24:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.623+0000 7fca148c6640 10 monclient: get_monmap_and_config 2024-09-10T16:24:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.623+0000 7fca148c6640 10 monclient: build_initial_monmap 2024-09-10T16:24:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.623+0000 7fca148c6640 10 monclient: monmap: 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:18.624345+0000 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:18.624345+0000 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:18.625 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 Processor -- start 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 -- start start 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient: init 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca1009c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca1009d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fca1013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-b con 0x7fca1013b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1009fbc0 con 0x7fca1009c310 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca100a2a20 con 0x7fca1009d190 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fca1009f860 con 0x7fca1013b5d0 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca148c6640 10 monclient(hunting): authenticate will time out at 1837.111726s 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fca1013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0e59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fca1009c310 0x7fca1009ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:58206/0 (socket says 172.21.15.164:58206) 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0e59c640 1 -- 172.21.15.164:0/3840375536 learned_addr learned my addr 172.21.15.164:0/3840375536 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request con 0x7fca1013b5d0 auth_method 0 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 msgr2=0x7fca1013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:24:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 msgr2=0x7fca1013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.624+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fca1013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.625+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fca1013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 1 -- 172.21.15.164:0/3840375536 >> v1:172.21.15.164:6789/0 conn(0x7fca1009c310 legacy=0x7fca1009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 1 -- 172.21.15.164:0/3840375536 >> v1:172.21.15.165:6789/0 conn(0x7fca1009d190 legacy=0x7fca1009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1488824319 0 0) 0x7fca00001450 con 0x7fca1009c310 2024-09-10T16:24:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _negotiate 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): my global_id is 164448 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc9fc003520 con 0x7fca1009c310 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1515028337 0 0) 0x7fca100a2a20 con 0x7fca1009d190 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _negotiate 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): my global_id is 154480 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca00001450 con 0x7fca1009d190 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 342719003 0 0) 0x7fc9fc003520 con 0x7fca1009c310 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 msgr2=0x7fca1013d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fca1013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 >> v1:172.21.15.165:6789/0 conn(0x7fca1009d190 legacy=0x7fca1009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca1009c6f0 con 0x7fca1009c310 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _finish_auth 0 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca148c6640 5 monclient: authenticate success, global_id 164448 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fca00001b70 con 0x7fca1009c310 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca148c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: dump: 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:24:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 >> v1:172.21.15.164:6789/0 conn(0x7fca1009c310 legacy=0x7fca1009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fc9fc003cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): picked mon.c con 0x7fca1013b5d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9fc004280 0x7fc9fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): picked mon.b con 0x7fc9fc004280 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fc9fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): picked mon.a con 0x7fc9fc004f20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca1009f860 con 0x7fc9fc004f20 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc9fc00bf00 con 0x7fc9fc004280 2024-09-10T16:24:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc9fc00c080 con 0x7fca1013b5d0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0dd9b640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca148c6640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fc9fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request con 0x7fc9fc004f20 auth_method 0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9fc004280 0x7fc9fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fc9fc003cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): get_auth_request con 0x7fc9fc004280 auth_method 0 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): get_auth_request con 0x7fca1013b5d0 auth_method 0 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca07fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 10 monclient(hunting): handle_auth_done global_id 164448 payload 293 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 msgr2=0x7fc9fc003cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca1013b5d0 0x7fc9fc003cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9fc004280 msgr2=0x7fc9fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.627+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9fc004280 0x7fc9fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 1 monclient: found mon.a 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9fc003520 con 0x7fc9fc004f20 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 10 monclient: _finish_auth 0 2024-09-10T16:24:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 10 monclient: _check_auth_tickets 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.628+0000 7fca0ed9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 msgr2=0x7fc9fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fc9fc00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca08009610 tx=0x7fc9fc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0ad09165dae8a8b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0803a070 con 0x7fc9fc004f20 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: dump: 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca0802d940 con 0x7fc9fc004f20 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca148c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 1 -- 172.21.15.164:0/3840375536 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca08009bd0 con 0x7fc9fc004f20 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca148c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca148c6640 4 set_mon_vals no callback set 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.630+0000 7fca0dd9b640 10 monclient: dump: 2024-09-10T16:24:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 10 monclient: shutdown 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 -- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 msgr2=0x7fc9fc00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 --2- 172.21.15.164:0/3840375536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fc9fc00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fca08009610 tx=0x7fc9fc005830 comp rx=0 tx=0).stop 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca0ed9d640 1 -- 172.21.15.164:0/3840375536 reap_dead start 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 -- 172.21.15.164:0/3840375536 shutdown_connections 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 -- 172.21.15.164:0/3840375536 >> 172.21.15.164:0/3840375536 conn(0x7fca100a3110 msgr2=0x7fca100a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 -- 172.21.15.164:0/3840375536 shutdown_connections 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.631+0000 7fca148c6640 1 -- 172.21.15.164:0/3840375536 wait complete. 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.632+0000 7fca148c6640 10 monclient: build_initial_monmap 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.632+0000 7fca148c6640 10 monclient: monmap: 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.632+0000 7fca148c6640 1 Processor -- start 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.632+0000 7fca148c6640 1 -- start start 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.632+0000 7fca148c6640 10 monclient: init 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9fc004f20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-c con 0x7fca100995c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 0x7fca10090810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): picked mon.noname-b con 0x7fca100903a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca100a20b0 con 0x7fc9fc004f20 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fca1009a7b0 con 0x7fca100903a0 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fca100a2230 con 0x7fca100995c0 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca148c6640 10 monclient(hunting): authenticate will time out at 1837.120099s 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60234/0 (socket says 172.21.15.164:60234) 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 -- 172.21.15.164:0/1922263290 learned_addr learned my addr 172.21.15.164:0/1922263290 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): get_auth_request con 0x7fc9fc004f20 auth_method 0 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 10 monclient(hunting): get_auth_request con 0x7fca100995c0 auth_method 0 2024-09-10T16:24:18.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 msgr2=0x7fca1008fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 msgr2=0x7fca1008fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca07fff640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 0x7fca10090810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request con 0x7fca100903a0 auth_method 0 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:18.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient(hunting): handle_auth_done global_id 164451 payload 824 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 msgr2=0x7fca1008fe60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 msgr2=0x7fca10090810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 0x7fca10090810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 monclient: found mon.noname-a 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0e59c640 1 -- 172.21.15.164:0/1922263290 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca10091810 con 0x7fc9fc004f20 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.633+0000 7fca0ed9d640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 0x7fca10090810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca0e59c640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 msgr2=0x7fca1013ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:18.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca0e59c640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca00007b50 tx=0x7fca00007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0000fba0 con 0x7fc9fc004f20 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: dump: 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:18.655 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca00018070 con 0x7fc9fc004f20 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 5 monclient: authenticate success, global_id 164451 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 10 monclient: _renew_subs 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca1009f2e0 con 0x7fc9fc004f20 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000133f0 con 0x7fc9fc004f20 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca06ffd640 10 monclient: dump: 2024-09-10T16:24:18.656 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 10 monclient: _renew_subs 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca0d59a640 4 set_mon_vals no callback set 2024-09-10T16:24:18.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.635+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca10093b70 con 0x7fc9fc004f20 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.636+0000 7fca148c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.636+0000 7fca148c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.636+0000 7fca148c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.636+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1009d930 con 0x7fc9fc004f20 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.636+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fca00013590 con 0x7fc9fc004f20 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.637+0000 7fca06ffd640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 0x7fc9f0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.637+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(616..616 src has 481..616) ==== 4799+0+0 (secure 0 0 0) 0x7fca00052f10 con 0x7fc9fc004f20 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.637+0000 7fca07fff640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 0x7fc9f0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:18.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.637+0000 7fca07fff640 10 monclient: get_auth_request con 0x7fc9f003e240 auth_method 0 2024-09-10T16:24:18.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.639+0000 7fca07fff640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 0x7fc9f0040700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80014b0 tx=0x7fc9f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:18.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.641+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fca1009fbc0 con 0x7fc9fc004f20 2024-09-10T16:24:18.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.641+0000 7fca06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:18.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.641+0000 7fca06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:18.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca148c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:18.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca148c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca148c6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca10092370 con 0x7fc9fc004f20 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca06ffd640 1 -- 172.21.15.164:0/1922263290 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fca0001d460 con 0x7fc9fc004f20 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.884+0000 7fca06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:18.886 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 msgr2=0x7fc9f0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 0x7fc9f0040700 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fc9f80014b0 tx=0x7fc9f8009040 comp rx=0 tx=0).stop 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 10 monclient: shutdown 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 msgr2=0x7fca1013ac80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fca00007b50 tx=0x7fca00007720 comp rx=0 tx=0).stop 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 shutdown_connections 2024-09-10T16:24:18.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc9f003e240 0x7fc9f0040700 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca100903a0 0x7fca10090810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca100995c0 0x7fca1008fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 --2- 172.21.15.164:0/1922263290 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9fc004f20 0x7fca1013ac80 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 >> 172.21.15.164:0/1922263290 conn(0x7fca100a3110 msgr2=0x7fca1009db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 shutdown_connections 2024-09-10T16:24:18.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.886+0000 7fca148c6640 1 -- 172.21.15.164:0/1922263290 wait complete. 2024-09-10T16:24:18.899 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:24:18.899 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:24:18.899 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:24:18.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 10 monclient: get_monmap_and_config 2024-09-10T16:24:18.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 10 monclient: build_initial_monmap 2024-09-10T16:24:18.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 10 monclient: monmap: 2024-09-10T16:24:18.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:18.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:18.997001+0000 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:18.997001+0000 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:18.998 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 1 Processor -- start 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 1 -- start start 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.996+0000 7f22fce52640 10 monclient: init 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:24:19.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f22f806cb70 0x7f22f8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-a con 0x7f22f806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f22f80a46a0 0x7f22f80a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-c con 0x7f22f80a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-b con 0x7f22f8137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f22f806db60 con 0x7f22f806cb70 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f22f806cf50 con 0x7f22f80a46a0 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f22f809ce50 con 0x7f22f8137ca0 2024-09-10T16:24:19.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22fce52640 10 monclient(hunting): authenticate will time out at 1837.484410s 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f22f806cb70 0x7f22f8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37370/0 (socket says 172.21.15.164:37370) 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2111344621 learned_addr learned my addr 172.21.15.164:0/2111344621 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 10 monclient(hunting): get_auth_request con 0x7f22f8137ca0 auth_method 0 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): get_auth_request con 0x7f22f806cb70 auth_method 0 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 msgr2=0x7f22f8138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 msgr2=0x7f22f8138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f6d9d640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f5d9b640 1 --2- 172.21.15.164:0/2111344621 >> v2:172.21.15.165:3300/0 conn(0x7f22f80a46a0 0x7f22f80a4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f5d9b640 10 monclient(hunting): get_auth_request con 0x7f22f80a46a0 auth_method 0 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient(hunting): handle_auth_done global_id 164454 payload 274 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 -- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 msgr2=0x7f22f8138130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 -- 172.21.15.164:0/2111344621 >> v2:172.21.15.165:3300/0 conn(0x7f22f80a46a0 msgr2=0x7f22f80a4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 --2- 172.21.15.164:0/2111344621 >> v2:172.21.15.165:3300/0 conn(0x7f22f80a46a0 0x7f22f80a4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient: _finish_hunting 0 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 monclient: found mon.noname-a 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:18.997+0000 7f22f659c640 1 -- 172.21.15.164:0/2111344621 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f809cb70 con 0x7f22f806cb70 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.000+0000 7f22f659c640 1 -- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f806cb70 msgr2=0x7f22f8134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.000+0000 7f22f659c640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f806cb70 0x7f22f8134f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009d90 tx=0x7f22f002a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4546547f03c78b0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 1 -- 172.21.15.164:0/2111344621 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0007950 con 0x7f22f806cb70 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: dump: 2024-09-10T16:24:19.012 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: _finish_auth 0 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: _check_auth_tickets 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:19.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 5 monclient: authenticate success, global_id 164454 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 1 -- 172.21.15.164:0/2111344621 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22f0007af0 con 0x7f22f806cb70 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 1 -- 172.21.15.164:0/2111344621 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0031400 con 0x7f22f806cb70 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 4 set_mon_vals no callback set 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22f559a640 10 monclient: dump: 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.014 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 10 monclient: shutdown 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 -- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f806cb70 msgr2=0x7f22f8134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f806cb70 0x7f22f8134f70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009d90 tx=0x7f22f002a4d0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 -- 172.21.15.164:0/2111344621 shutdown_connections 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f8137ca0 0x7f22f8138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 --2- 172.21.15.164:0/2111344621 >> v2:172.21.15.165:3300/0 conn(0x7f22f80a46a0 0x7f22f80a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.015 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 --2- 172.21.15.164:0/2111344621 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f806cb70 0x7f22f8134f70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 -- 172.21.15.164:0/2111344621 >> 172.21.15.164:0/2111344621 conn(0x7f22f80aa900 msgr2=0x7f22f80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 -- 172.21.15.164:0/2111344621 shutdown_connections 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.001+0000 7f22fce52640 1 -- 172.21.15.164:0/2111344621 wait complete. 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.002+0000 7f22fce52640 10 monclient: build_initial_monmap 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.002+0000 7f22fce52640 10 monclient: monmap: 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:19.016 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.002+0000 7f22fce52640 1 Processor -- start 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.002+0000 7f22fce52640 1 -- start start 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.002+0000 7f22fce52640 10 monclient: init 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22f806cb70 0x7f22f81cd420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-b con 0x7f22f806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-c con 0x7f22f80a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): picked mon.noname-a con 0x7f22f8137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:19.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f22f81387f0 con 0x7f22f8137ca0 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f22f8138670 con 0x7f22f806cb70 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f22f8138970 con 0x7f22f80a46a0 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22fce52640 10 monclient(hunting): authenticate will time out at 1837.490341s 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60260/0 (socket says 172.21.15.164:60260) 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37372/0 (socket says 172.21.15.164:37372) 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 learned_addr learned my addr 172.21.15.164:0/2896180958 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:19.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): get_auth_request con 0x7f22f8137ca0 auth_method 0 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 10 monclient(hunting): get_auth_request con 0x7f22f80a46a0 auth_method 0 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22f806cb70 0x7f22f81cd420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): get_auth_request con 0x7f22f806cb70 auth_method 0 2024-09-10T16:24:19.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 msgr2=0x7f22f80a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 msgr2=0x7f22f80a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f5d9b640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient(hunting): handle_auth_done global_id 164457 payload 824 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 msgr2=0x7f22f80a3a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f22f80a46a0 0x7f22f80a3a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22f806cb70 msgr2=0x7f22f81cd420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f22f806cb70 0x7f22f81cd420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 monclient: found mon.noname-a 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.003+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f806e720 con 0x7f22f8137ca0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.005+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 msgr2=0x7f22f809e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.005+0000 7f22f6d9d640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d620 tx=0x7f22e800dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8018030 con 0x7f22f8137ca0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: dump: 2024-09-10T16:24:19.021 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22e8004020 con 0x7f22f8137ca0 2024-09-10T16:24:19.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 5 monclient: authenticate success, global_id 164457 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e80041f0 con 0x7f22f8137ca0 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 10 monclient: _renew_subs 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f809ffc0 con 0x7f22f8137ca0 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22f4d99640 4 set_mon_vals no callback set 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22eeffd640 10 monclient: dump: 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.023 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 10 monclient: _renew_subs 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f80a0b20 con 0x7f22f8137ca0 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.006+0000 7f22fce52640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 0x7f22f809ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f22f80a5e10 con 0x7f22f80a3fc0 2024-09-10T16:24:19.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f22e801b070 con 0x7f22f8137ca0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 0x7f22f809ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient: get_auth_request con 0x7f22f80a3fc0 auth_method 0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 10 monclient(hunting): handle_auth_done global_id 164460 payload 824 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22eeffd640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22d803e210 0x7f22d80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f659c640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22f81ce0d0 con 0x7f22f80a3fc0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(616..616 src has 481..616) ==== 4799+0+0 (secure 0 0 0) 0x7f22e8052de0 con 0x7f22f8137ca0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f5d9b640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22d803e210 0x7f22d80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.007+0000 7f22f5d9b640 10 monclient: get_auth_request con 0x7f22d803e210 auth_method 0 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22f659c640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 msgr2=0x7f22f809ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22f659c640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 0x7f22f809ea90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0031cd0 con 0x7f22f80a3fc0 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 10 monclient: dump: 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.026 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22f0007e80 con 0x7f22f80a3fc0 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 msgr2=0x7f22f809ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.009+0000 7f22eeffd640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 0x7f22f809ea90 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f0004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.010+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 msgr2=0x7f22f809ea90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.010+0000 7f22eeffd640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f80a3fc0 0x7f22f809ea90 secure :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f22f0034000 tx=0x7f22f0004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:19.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.010+0000 7f22f5d9b640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22d803e210 0x7f22d80406d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22f80a1090 tx=0x7f22e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22fce52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:19.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22fce52640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 0x7f22f81bdac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22fce52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f22f809f3e0 con 0x7f22f809efd0 2024-09-10T16:24:19.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 0x7f22f81bdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient: get_auth_request con 0x7f22f809efd0 auth_method 0 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 10 monclient(hunting): handle_auth_done global_id 164463 payload 824 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.016+0000 7f22f659c640 1 -- 172.21.15.164:0/2896180958 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22f80a2120 con 0x7f22f809efd0 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22f659c640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 msgr2=0x7f22f81bdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22f659c640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 0x7f22f81bdac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f22f80a5f90 tx=0x7f22f0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22f0049040 con 0x7f22f809efd0 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 10 monclient: dump: 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.031 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f22f0004720 con 0x7f22f809efd0 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 msgr2=0x7f22f81bdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 0x7f22f81bdac0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f22f80a5f90 tx=0x7f22f0030000 comp rx=0 tx=0).stop 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 msgr2=0x7f22f81bdac0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.018+0000 7f22eeffd640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f809efd0 0x7f22f81bdac0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.032 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 229721, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:24:19.033 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:19.034 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:19.035 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:19.036 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:19.037 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:19.038 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:19.039 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.040 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:19.041 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:19.042 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:19.043 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.044 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:19.045 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22d803e210 msgr2=0x7f22d80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22d803e210 0x7f22d80406d0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f22f80a1090 tx=0x7f22e0008040 comp rx=0 tx=0).stop 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 10 monclient: shutdown 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 msgr2=0x7f22f809e550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22f6d9d640 1 -- 172.21.15.164:0/2896180958 reap_dead start 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d620 tx=0x7f22e800dae0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 shutdown_connections 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 --2- 172.21.15.164:0/2896180958 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f22f8137ca0 0x7f22f809e550 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 >> 172.21.15.164:0/2896180958 conn(0x7f22f80aa900 msgr2=0x7f22f8133cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 shutdown_connections 2024-09-10T16:24:19.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.019+0000 7f22fce52640 1 -- 172.21.15.164:0/2896180958 wait complete. 2024-09-10T16:24:19.048 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:19.048 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:24:19.048 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:24:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.145+0000 7f2b2b67f640 10 monclient: get_monmap_and_config 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.145+0000 7f2b2b67f640 10 monclient: build_initial_monmap 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.145+0000 7f2b2b67f640 10 monclient: monmap: 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:19.146407+0000 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:19.146407+0000 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:19.148 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 Processor -- start 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 -- start start 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient: init 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2b2409c180 0x7f2b2409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b2409d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b2413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b2409a430 con 0x7f2b2409d160 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2b240a24e0 con 0x7f2b2413a5c0 2024-09-10T16:24:19.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2b240a2bf0 con 0x7f2b2409c180 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2b67f640 10 monclient(hunting): authenticate will time out at 1837.633828s 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2941b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2b2409c180 0x7f2b2409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b28c1a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2b2409d160 0x7f2b2409ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:58216/0 (socket says 172.21.15.164:58216) 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b28c1a640 1 -- 172.21.15.164:0/45961777 learned_addr learned my addr 172.21.15.164:0/45961777 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2941b640 10 monclient(hunting): get_auth_request con 0x7f2b2409c180 auth_method 0 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.146+0000 7f2b2941b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient(hunting): handle_auth_done global_id 164466 payload 274 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 1 -- 172.21.15.164:0/45961777 >> v1:172.21.15.165:6789/0 conn(0x7f2b2413a5c0 legacy=0x7f2b2413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 1 -- 172.21.15.164:0/45961777 >> v1:172.21.15.164:6789/0 conn(0x7f2b2409d160 legacy=0x7f2b2409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient: _finish_hunting 0 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 1 monclient: found mon.noname-a 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.147+0000 7f2b2941b640 1 -- 172.21.15.164:0/45961777 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2409c560 con 0x7f2b2409c180 2024-09-10T16:24:19.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.149+0000 7f2b2941b640 1 -- 172.21.15.164:0/45961777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2409c180 msgr2=0x7f2b2409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.149+0000 7f2b2941b640 1 --2- 172.21.15.164:0/45961777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2409c180 0x7f2b2409ea40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2b200097d0 tx=0x7f2b2002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9923a6ba82a20a09 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 1 -- 172.21.15.164:0/45961777 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20037070 con 0x7f2b2409c180 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: dump: 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.159 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: _finish_auth 0 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 1 -- 172.21.15.164:0/45961777 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b20032440 con 0x7f2b2409c180 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 5 monclient: authenticate success, global_id 164466 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 1 -- 172.21.15.164:0/45961777 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20032810 con 0x7f2b2409c180 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 4 set_mon_vals no callback set 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b1bfff640 10 monclient: dump: 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 10 monclient: shutdown 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 -- 172.21.15.164:0/45961777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2409c180 msgr2=0x7f2b2409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/45961777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2409c180 0x7f2b2409ea40 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2b200097d0 tx=0x7f2b2002a000 comp rx=0 tx=0).stop 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 -- 172.21.15.164:0/45961777 shutdown_connections 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/45961777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2409c180 0x7f2b2409ea40 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 -- 172.21.15.164:0/45961777 >> 172.21.15.164:0/45961777 conn(0x7f2b240a3110 msgr2=0x7f2b240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 -- 172.21.15.164:0/45961777 shutdown_connections 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.150+0000 7f2b2b67f640 1 -- 172.21.15.164:0/45961777 wait complete. 2024-09-10T16:24:19.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.151+0000 7f2b2b67f640 10 monclient: build_initial_monmap 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.151+0000 7f2b2b67f640 10 monclient: monmap: 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.151+0000 7f2b2b67f640 1 Processor -- start 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.151+0000 7f2b2b67f640 1 -- start start 2024-09-10T16:24:19.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.151+0000 7f2b2b67f640 10 monclient: init 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-c con 0x7f2b2409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b2409d160 0x7f2b2408fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-b con 0x7f2b2409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 0x7f2b24090300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): picked mon.noname-a con 0x7f2b2413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2b24074f90 con 0x7f2b2413a5c0 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2b24074e10 con 0x7f2b2409d160 2024-09-10T16:24:19.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2b2409f690 con 0x7f2b2409c180 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2b67f640 10 monclient(hunting): authenticate will time out at 1837.639112s 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 0x7f2b24090300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37388/0 (socket says 172.21.15.164:37388) 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 learned_addr learned my addr 172.21.15.164:0/1686009458 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 10 monclient(hunting): get_auth_request con 0x7f2b2409c180 auth_method 0 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): get_auth_request con 0x7f2b2413a5c0 auth_method 0 2024-09-10T16:24:19.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 msgr2=0x7f2b2408f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 msgr2=0x7f2b2408f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b28c1a640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b2409d160 0x7f2b2408fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b28c1a640 10 monclient(hunting): get_auth_request con 0x7f2b2409d160 auth_method 0 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b28c1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b28c1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b28c1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient(hunting): handle_auth_done global_id 164469 payload 824 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 msgr2=0x7f2b2408f880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2409c180 0x7f2b2408f880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b2409d160 msgr2=0x7f2b2408fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2b2409d160 0x7f2b2408fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient: _finish_hunting 0 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 monclient: found mon.noname-a 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.152+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2409d620 con 0x7f2b2413a5c0 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 msgr2=0x7f2b24090300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b29c1c640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 0x7f2b24090300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00c750 tx=0x7f2b1c00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c016020 con 0x7f2b2413a5c0 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: dump: 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.168 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: _finish_auth 0 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b1c00aa60 con 0x7f2b2413a5c0 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c00ae50 con 0x7f2b2413a5c0 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b19ffb640 10 monclient: dump: 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:19.169 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b2b67f640 5 monclient: authenticate success, global_id 164469 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b2b67f640 10 monclient: _renew_subs 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b2b67f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b24090c20 con 0x7f2b2413a5c0 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.154+0000 7f2b1b7fe640 4 set_mon_vals no callback set 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 10 monclient: _renew_subs 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:19.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b24090840 con 0x7f2b2413a5c0 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2b1c019020 con 0x7f2b2413a5c0 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.155+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2b24096980 con 0x7f2b2402e5c0 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b19ffb640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2b0803e400 0x7f2b080408c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b2402e5c0 auth_method 0 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:19.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(616..616 src has 481..616) ==== 4799+0+0 (secure 0 0 0) 0x7f2b1c052c50 con 0x7f2b2413a5c0 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b28c1a640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2b0803e400 0x7f2b080408c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.156+0000 7f2b28c1a640 10 monclient: get_auth_request con 0x7f2b0803e400 auth_method 0 2024-09-10T16:24:19.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.158+0000 7f2b28c1a640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2b0803e400 0x7f2b080408c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2b24091a60 tx=0x7f2b14009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b2402e5c0 auth_method 0 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:19.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:24:19.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:19.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.357+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:24:19.494 INFO:tasks.rados.rados.0.smithi165.stdout:2757: done (3 left) 2024-09-10T16:24:19.494 INFO:tasks.rados.rados.0.smithi165.stdout:2758: write (excl) oid 24 current snap is 292 2024-09-10T16:24:19.494 INFO:tasks.rados.rados.0.smithi165.stdout:2753: finishing write tid 2 to smithi16534841-44 2024-09-10T16:24:19.494 INFO:tasks.rados.rados.0.smithi165.stdout:2758: seq_num 964 ranges {601948=400514,1577818=624130} 2024-09-10T16:24:19.499 INFO:tasks.rados.rados.0.smithi165.stdout:2758: writing smithi16534841-24 from 601948 to 1002462 tid 1 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:2758: writing smithi16534841-24 from 1577818 to 2201948 tid 2 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:2759: read oid 19 snap -1 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:2759: expect deleted 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:2760: rollback oid 15 current snap is 292 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 290 2024-09-10T16:24:19.507 INFO:tasks.rados.rados.0.smithi165.stdout:2753: finishing write tid 3 to smithi16534841-44 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2753: finishing write tid 4 to smithi16534841-44 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2753: oid 44 updating version 0 to 1157 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2753: oid 44 version 1157 is already newer than 1155 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2753: oid 44 version 1157 is already newer than 1156 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1157 (ObjNum 962 snap 292 seq_num 962) dirty exists 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2753: left oid 44 (ObjNum 962 snap 292 seq_num 962) 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2755: finishing write tid 1 to smithi16534841-40 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2756: finishing copy_from to smithi16534841-21 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2756: got expected ENOENT (src dne) 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2756: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2755: finishing write tid 2 to smithi16534841-40 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2755: finishing write tid 3 to smithi16534841-40 2024-09-10T16:24:19.508 INFO:tasks.rados.rados.0.smithi165.stdout:2755: finishing write tid 4 to smithi16534841-40 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2755: oid 40 updating version 0 to 1233 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2755: oid 40 updating version 1233 to 1234 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2755: oid 40 version 1234 is already newer than 1232 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1234 (ObjNum 963 snap 292 seq_num 963) dirty exists 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2755: left oid 40 (ObjNum 963 snap 292 seq_num 963) 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2758: finishing write tid 1 to smithi16534841-24 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2753: done (5 left) 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2755: done (4 left) 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2756: done (3 left) 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2761: read oid 9 snap -1 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2761: expect deleted 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2762: write (excl) oid 45 current snap is 292 2024-09-10T16:24:19.509 INFO:tasks.rados.rados.0.smithi165.stdout:2762: seq_num 965 ranges {618755=475763,1636300=457996} 2024-09-10T16:24:19.512 INFO:tasks.rados.rados.0.smithi165.stdout:2762: writing smithi16534841-45 from 618755 to 1094518 tid 1 2024-09-10T16:24:19.518 INFO:tasks.rados.rados.0.smithi165.stdout:2762: writing smithi16534841-45 from 1636300 to 2094296 tid 2 2024-09-10T16:24:19.519 INFO:tasks.rados.rados.0.smithi165.stdout:2763: write oid 14 current snap is 292 2024-09-10T16:24:19.519 INFO:tasks.rados.rados.0.smithi165.stdout:2763: seq_num 966 ranges {635562=667365,2094782=140780} 2024-09-10T16:24:19.525 INFO:tasks.rados.rados.0.smithi165.stdout:2763: writing smithi16534841-14 from 635562 to 1302927 tid 1 2024-09-10T16:24:19.531 INFO:tasks.rados.rados.0.smithi165.stdout:2763: writing smithi16534841-14 from 2094782 to 2235562 tid 2 2024-09-10T16:24:19.531 INFO:tasks.rados.rados.0.smithi165.stdout:2764: write oid 47 current snap is 292 2024-09-10T16:24:19.531 INFO:tasks.rados.rados.0.smithi165.stdout:2764: seq_num 967 ranges {652369=742614,2153264=99105} 2024-09-10T16:24:19.538 INFO:tasks.rados.rados.0.smithi165.stdout:2764: writing smithi16534841-47 from 652369 to 1394983 tid 1 2024-09-10T16:24:19.544 INFO:tasks.rados.rados.0.smithi165.stdout:2764: writing smithi16534841-47 from 2153264 to 2252369 tid 2 2024-09-10T16:24:19.548 INFO:tasks.rados.rados.0.smithi165.stdout:2760: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:24:19.548 INFO:tasks.rados.rados.0.smithi165.stdout:2760: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:24:19.548 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1338 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:24:19.548 INFO:tasks.rados.rados.0.smithi165.stdout:2762: finishing write tid 1 to smithi16534841-45 2024-09-10T16:24:19.548 INFO:tasks.rados.rados.0.smithi165.stdout:2758: finishing write tid 2 to smithi16534841-24 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2765: read oid 10 snap -1 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2765: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2759: done (7 left) 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2760: done (6 left) 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2761: done (5 left) 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2766: copy_from oid 43 from oid 4 current snap is 292 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2758: finishing write tid 3 to smithi16534841-24 2024-09-10T16:24:19.549 INFO:tasks.rados.rados.0.smithi165.stdout:2758: finishing write tid 4 to smithi16534841-24 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:2758: oid 24 updating version 0 to 1610 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:2758: oid 24 updating version 1610 to 1612 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:2758: oid 24 version 1612 is already newer than 1611 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1612 (ObjNum 964 snap 292 seq_num 964) dirty exists 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:2758: left oid 24 (ObjNum 964 snap 292 seq_num 964) 2024-09-10T16:24:19.550 INFO:tasks.rados.rados.0.smithi165.stdout:2762: finishing write tid 2 to smithi16534841-45 2024-09-10T16:24:19.551 INFO:tasks.rados.rados.0.smithi165.stdout:2758: done (5 left) 2024-09-10T16:24:19.551 INFO:tasks.rados.rados.0.smithi165.stdout:2767: delete oid 20 current snap is 292 2024-09-10T16:24:19.551 INFO:tasks.rados.rados.0.smithi165.stdout:2762: finishing write tid 3 to smithi16534841-45 2024-09-10T16:24:19.551 INFO:tasks.rados.rados.0.smithi165.stdout:2762: finishing write tid 4 to smithi16534841-45 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:2762: oid 45 updating version 0 to 1666 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:2762: oid 45 updating version 1666 to 1667 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:2762: oid 45 updating version 1667 to 1668 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1668 (ObjNum 965 snap 292 seq_num 965) dirty exists 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:2762: left oid 45 (ObjNum 965 snap 292 seq_num 965) 2024-09-10T16:24:19.552 INFO:tasks.rados.rados.0.smithi165.stdout:2763: finishing write tid 1 to smithi16534841-14 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: finishing write tid 2 to smithi16534841-14 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: finishing write tid 3 to smithi16534841-14 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: finishing write tid 4 to smithi16534841-14 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: oid 14 updating version 0 to 1236 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: oid 14 updating version 1236 to 1237 2024-09-10T16:24:19.553 INFO:tasks.rados.rados.0.smithi165.stdout:2763: oid 14 version 1237 is already newer than 1235 2024-09-10T16:24:19.554 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1237 (ObjNum 966 snap 292 seq_num 966) dirty exists 2024-09-10T16:24:19.554 INFO:tasks.rados.rados.0.smithi165.stdout:2763: left oid 14 (ObjNum 966 snap 292 seq_num 966) 2024-09-10T16:24:19.555 INFO:tasks.rados.rados.0.smithi165.stdout:2765: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:19.555 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1436136 2024-09-10T16:24:19.599 INFO:tasks.rados.rados.0.smithi165.stdout:2764: finishing write tid 1 to smithi16534841-47 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: finishing write tid 2 to smithi16534841-47 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: finishing write tid 3 to smithi16534841-47 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: finishing write tid 4 to smithi16534841-47 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: oid 47 updating version 0 to 1614 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: oid 47 version 1614 is already newer than 1613 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:2764: oid 47 updating version 1614 to 1615 2024-09-10T16:24:19.600 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1615 (ObjNum 967 snap 292 seq_num 967) dirty exists 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2764: left oid 47 (ObjNum 967 snap 292 seq_num 967) 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2766: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2762: done (5 left) 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2763: done (4 left) 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2764: done (3 left) 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2765: done (2 left) 2024-09-10T16:24:19.601 INFO:tasks.rados.rados.0.smithi165.stdout:2767: done (1 left) 2024-09-10T16:24:19.602 INFO:tasks.rados.rados.0.smithi165.stdout:2768: snap_create 2024-09-10T16:24:19.602 INFO:tasks.rados.rados.0.smithi165.stdout:2766: finishing copy_from to smithi16534841-43 2024-09-10T16:24:19.602 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1333 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b2402e5c0 auth_method 0 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:19.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:24:19.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:19.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:19.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:19.758+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:24:20.498 INFO:tasks.rados.rados.0.smithi165.stdout:2766: done (1 left) 2024-09-10T16:24:20.499 INFO:tasks.rados.rados.0.smithi165.stdout:2768: done (0 left) 2024-09-10T16:24:20.499 INFO:tasks.rados.rados.0.smithi165.stdout:2769: snap_create 2024-09-10T16:24:20.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b2402e5c0 auth_method 0 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 10 monclient(hunting): handle_auth_done global_id 204101 payload 824 2024-09-10T16:24:20.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2b2402e9a0 con 0x7f2b2402e5c0 2024-09-10T16:24:20.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:24:20.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:24:20.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002a040 tx=0x7f2b20004730 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:20.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:20.560+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2769: done (0 left) 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2770: read oid 26 snap 292 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2770: expect (ObjNum 930 snap 285 seq_num 930) 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2771: rollback oid 50 current snap is 294 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 292 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2772: rollback oid 23 current snap is 294 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 293 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2773: rollback oid 31 current snap is 294 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 292 2024-09-10T16:24:21.477 INFO:tasks.rados.rados.0.smithi165.stdout:2774: delete oid 3 current snap is 294 2024-09-10T16:24:21.485 INFO:tasks.rados.rados.0.smithi165.stdout:2771: finishing rollback tid 0 to smithi16534841-50 2024-09-10T16:24:21.487 INFO:tasks.rados.rados.0.smithi165.stdout:2772: finishing rollback tid 0 to smithi16534841-23 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:2772: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:2771: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:2772: finishing rollback tid 2 to smithi16534841-23 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1413 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:2771: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:24:21.489 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1286 (ObjNum 922 snap 281 seq_num 922) dirty exists 2024-09-10T16:24:21.490 INFO:tasks.rados.rados.0.smithi165.stdout:2771: done (4 left) 2024-09-10T16:24:21.490 INFO:tasks.rados.rados.0.smithi165.stdout:2772: done (3 left) 2024-09-10T16:24:21.490 INFO:tasks.rados.rados.0.smithi165.stdout:2774: done (2 left) 2024-09-10T16:24:21.490 INFO:tasks.rados.rados.0.smithi165.stdout:2775: write (excl) oid 27 current snap is 294 2024-09-10T16:24:21.490 INFO:tasks.rados.rados.0.smithi165.stdout:2775: seq_num 970 ranges {702790=568361,1812357=490433} 2024-09-10T16:24:21.497 INFO:tasks.rados.rados.0.smithi165.stdout:2775: writing smithi16534841-27 from 702790 to 1271151 tid 1 2024-09-10T16:24:21.503 INFO:tasks.rados.rados.0.smithi165.stdout:2775: writing smithi16534841-27 from 1812357 to 2302790 tid 2 2024-09-10T16:24:21.503 INFO:tasks.rados.rados.0.smithi165.stdout:2776: copy_from oid 34 from oid 18 current snap is 294 2024-09-10T16:24:21.503 INFO:tasks.rados.rados.0.smithi165.stdout:2770: expect (ObjNum 930 snap 285 seq_num 930) 2024-09-10T16:24:21.503 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 406792 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2773: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2773: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2773: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1768 (ObjNum 881 snap 269 seq_num 881) dirty exists 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2775: finishing write tid 1 to smithi16534841-27 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2775: finishing write tid 2 to smithi16534841-27 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2775: finishing write tid 3 to smithi16534841-27 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2775: finishing write tid 4 to smithi16534841-27 2024-09-10T16:24:21.774 INFO:tasks.rados.rados.0.smithi165.stdout:2775: oid 27 updating version 0 to 1618 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2775: oid 27 version 1618 is already newer than 1617 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2775: oid 27 version 1618 is already newer than 1616 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1618 (ObjNum 970 snap 294 seq_num 970) dirty exists 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2775: left oid 27 (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2776: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2776: finishing copy_from to smithi16534841-34 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2770: done (3 left) 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2773: done (2 left) 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2775: done (1 left) 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2776: done (0 left) 2024-09-10T16:24:21.775 INFO:tasks.rados.rados.0.smithi165.stdout:2777: delete oid 17 current snap is 294 2024-09-10T16:24:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:2777: done (0 left) 2024-09-10T16:24:21.776 INFO:tasks.rados.rados.0.smithi165.stdout:2778: snap_create 2024-09-10T16:24:22.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.152+0000 7f2b18ff9640 10 monclient: tick 2024-09-10T16:24:22.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.152+0000 7f2b18ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:24:22.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.152+0000 7f2b18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:22.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.152+0000 7f2b18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b2402e5c0 auth_method 0 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: _init_auth method 2 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: _init_auth already have auth, reseting 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:22.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.162+0000 7f2b2941b640 10 monclient: handle_auth_done global_id 204101 payload 293 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002a040 tx=0x7f2b2003e040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20037040 con 0x7f2b2402e5c0 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 10 monclient: dump: 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.167 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2b20047730 con 0x7f2b2402e5c0 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002a040 tx=0x7f2b2003e040 comp rx=0 tx=0).stop 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 msgr2=0x7f2b24063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:22.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.165+0000 7f2b19ffb640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b2402e5c0 0x7f2b24063d50 secure :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b2002a040 tx=0x7f2b2003e040 comp rx=0 tx=0).stop 2024-09-10T16:24:22.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2b67f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:22.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 0x7f2b241d77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2b67f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2b240a5160 con 0x7f2b24064290 2024-09-10T16:24:22.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 0x7f2b241d77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient: get_auth_request con 0x7f2b24064290 auth_method 0 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.172+0000 7f2b2941b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.173+0000 7f2b2941b640 10 monclient(hunting): handle_auth_done global_id 204104 payload 824 2024-09-10T16:24:22.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.173+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2b2409ea40 con 0x7f2b24064290 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b2941b640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 msgr2=0x7f2b241d77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b2941b640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 0x7f2b241d77e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b240a4b40 tx=0x7f2b200046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b20037040 con 0x7f2b24064290 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 10 monclient: dump: 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.177 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f2b20003810 con 0x7f2b24064290 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 msgr2=0x7f2b241d77e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 0x7f2b241d77e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b240a4b40 tx=0x7f2b200046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 msgr2=0x7f2b241d77e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:22.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.175+0000 7f2b19ffb640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2b24064290 0x7f2b241d77e0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2b240a4b40 tx=0x7f2b200046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 11912, 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:22.179 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.180 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.181 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:22.182 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.183 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.184 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:22.185 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.186 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.187 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.188 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:22.189 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.190 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:22.191 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:22.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2b0803e400 msgr2=0x7f2b080408c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2b0803e400 0x7f2b080408c0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f2b24091a60 tx=0x7f2b14009040 comp rx=0 tx=0).stop 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 10 monclient: shutdown 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 msgr2=0x7f2b24090300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 --2- 172.21.15.164:0/1686009458 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2b2413a5c0 0x7f2b24090300 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00c750 tx=0x7f2b1c00cc10 comp rx=0 tx=0).stop 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 reap_dead start 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b29c1c640 1 -- 172.21.15.164:0/1686009458 reap_dead start 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 shutdown_connections 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 >> 172.21.15.164:0/1686009458 conn(0x7f2b240a3110 msgr2=0x7f2b24139bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 shutdown_connections 2024-09-10T16:24:22.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.176+0000 7f2b2b67f640 1 -- 172.21.15.164:0/1686009458 wait complete. 2024-09-10T16:24:22.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:22.194 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:24:22.194 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:24:22.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.299+0000 7f1c85240640 10 monclient: get_monmap_and_config 2024-09-10T16:24:22.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.299+0000 7f1c85240640 10 monclient: build_initial_monmap 2024-09-10T16:24:22.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.299+0000 7f1c85240640 10 monclient: monmap: 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:22.300061+0000 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:22.300061+0000 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:22.301 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 Processor -- start 2024-09-10T16:24:22.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 -- start start 2024-09-10T16:24:22.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient: init 2024-09-10T16:24:22.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1c8009d930 0x7f1c8009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8009d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c80134580 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c800a0870 con 0x7f1c80134580 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1c8009d3e0 con 0x7f1c8009d930 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8009e790 con 0x7f1c8009ab50 2024-09-10T16:24:22.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c85240640 10 monclient(hunting): authenticate will time out at 1840.787682s 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1c8009d930 0x7f1c8009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37458/0 (socket says 172.21.15.164:37458) 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1c8009d930 0x7f1c8009dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60308/0 (socket says 172.21.15.164:60308) 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 1 -- 172.21.15.164:0/705945884 learned_addr learned my addr 172.21.15.164:0/705945884 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): get_auth_request con 0x7f1c8009ab50 auth_method 0 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request con 0x7f1c8009d930 auth_method 0 2024-09-10T16:24:22.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.300+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_done global_id 164472 payload 274 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 msgr2=0x7f1c8009e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/705945884 >> v1:172.21.15.164:6789/0 conn(0x7f1c80134580 legacy=0x7f1c80134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:22.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:24:22.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 1 monclient: found mon.noname-a 2024-09-10T16:24:22.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:22.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/705945884 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c80074c70 con 0x7f1c8009d930 2024-09-10T16:24:22.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.301+0000 7f1c77fff640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:22.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.303+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c8009d930 msgr2=0x7f1c8009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.303+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c8009d930 0x7f1c8009dd10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000ba40 tx=0x7f1c7002c4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3985aefd89310c14 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 1 -- 172.21.15.164:0/705945884 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7003a070 con 0x7f1c8009d930 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: dump: 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: _finish_auth 0 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: _check_auth_tickets 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 1 -- 172.21.15.164:0/705945884 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c70032690 con 0x7f1c8009d930 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 1 -- 172.21.15.164:0/705945884 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70032a60 con 0x7f1c8009d930 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 5 monclient: authenticate success, global_id 164472 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c7e59c640 10 monclient: dump: 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 4 set_mon_vals no callback set 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 10 monclient: shutdown 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:22.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 -- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c8009d930 msgr2=0x7f1c8009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c8009d930 0x7f1c8009dd10 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f1c7000ba40 tx=0x7f1c7002c4f0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 -- 172.21.15.164:0/705945884 shutdown_connections 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009ab50 0x7f1c8009e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 --2- 172.21.15.164:0/705945884 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c8009d930 0x7f1c8009dd10 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 -- 172.21.15.164:0/705945884 >> 172.21.15.164:0/705945884 conn(0x7f1c800a3110 msgr2=0x7f1c800a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 -- 172.21.15.164:0/705945884 shutdown_connections 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.304+0000 7f1c85240640 1 -- 172.21.15.164:0/705945884 wait complete. 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.305+0000 7f1c85240640 10 monclient: build_initial_monmap 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.305+0000 7f1c85240640 10 monclient: monmap: 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:22.319 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.305+0000 7f1c85240640 1 Processor -- start 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.305+0000 7f1c85240640 1 -- start start 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.305+0000 7f1c85240640 10 monclient: init 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c8009ab50 0x7f1c8012fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8009ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 0x7f1c801ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-c con 0x7f1c8009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 0x7f1c80133be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): picked mon.noname-a con 0x7f1c80134580 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8009ef30 con 0x7f1c80134580 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1c800a02b0 con 0x7f1c8009ab50 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1c800a2940 con 0x7f1c8009d930 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c85240640 10 monclient(hunting): authenticate will time out at 1840.793435s 2024-09-10T16:24:22.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 0x7f1c80133be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 0x7f1c801ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 0x7f1c80133be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60322/0 (socket says 172.21.15.164:60322) 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 0x7f1c801ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37460/0 (socket says 172.21.15.164:37460) 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 1 -- 172.21.15.164:0/3384486483 learned_addr learned my addr 172.21.15.164:0/3384486483 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): get_auth_request con 0x7f1c80134580 auth_method 0 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): get_auth_request con 0x7f1c8009d930 auth_method 0 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c8009ab50 0x7f1c8012fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request con 0x7f1c8009ab50 auth_method 0 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient(hunting): handle_auth_done global_id 204110 payload 824 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c8009ab50 msgr2=0x7f1c8012fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c8009ab50 0x7f1c8012fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 msgr2=0x7f1c80133be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 0x7f1c80133be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 monclient: found mon.noname-c 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c77fff640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c80074360 con 0x7f1c8009d930 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.306+0000 7f1c7f59e640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1c80134580 0x7f1c80133be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:22.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c77fff640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 msgr2=0x7f1c801ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c77fff640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 0x7f1c801ca1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e930 tx=0x7f1c6800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c6800cbb0 con 0x7f1c8009d930 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: dump: 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 5 monclient: authenticate success, global_id 204110 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c6800cd50 con 0x7f1c8009d930 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 10 monclient: _renew_subs 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8009d620 con 0x7f1c8009d930 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c68004270 con 0x7f1c8009d930 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c777fe640 10 monclient: dump: 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c7dd9b640 4 set_mon_vals no callback set 2024-09-10T16:24:22.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 10 monclient: _renew_subs 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.309+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c801ca6f0 con 0x7f1c8009d930 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c85240640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1c68003610 con 0x7f1c8009d930 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c85240640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 0x7f1c801cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1c800a1e40 con 0x7f1c80134120 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c777fe640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1c6003e210 0x7f1c600406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(619..619 src has 481..619) ==== 4799+0+0 (secure 0 0 0) 0x7f1c68052d00 con 0x7f1c8009d930 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 0x7f1c801cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient: get_auth_request con 0x7f1c80134120 auth_method 0 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7f59e640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1c6003e210 0x7f1c600406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7f59e640 10 monclient: get_auth_request con 0x7f1c6003e210 auth_method 0 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_done global_id 154498 payload 824 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.310+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c8006dac0 con 0x7f1c80134120 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 msgr2=0x7f1c801cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 0x7f1c801cd270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1c70037000 tx=0x7f1c700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c7f59e640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1c6003e210 0x7f1c600406d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1c78007fd0 tx=0x7f1c78003de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c70044070 con 0x7f1c80134120 2024-09-10T16:24:22.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 10 monclient: dump: 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1c7003f670 con 0x7f1c80134120 2024-09-10T16:24:22.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:22.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 msgr2=0x7f1c801cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 0x7f1c801cd270 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1c70037000 tx=0x7f1c700046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 msgr2=0x7f1c801cd270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:22.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.313+0000 7f1c777fe640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c80134120 0x7f1c801cd270 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c85240640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c85240640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 0x7f1c801cdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c85240640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1c801ce100 con 0x7f1c801cd7b0 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 0x7f1c801cdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c7ed9d640 10 monclient: get_auth_request con 0x7f1c801cd7b0 auth_method 0 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c7ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.319+0000 7f1c7ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.320+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.320+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.320+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.320+0000 7f1c7ed9d640 10 monclient(hunting): handle_auth_done global_id 154501 payload 824 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.320+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/3384486483 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1c80077260 con 0x7f1c801cd7b0 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c7ed9d640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 msgr2=0x7f1c801cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c7ed9d640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 0x7f1c801cdbc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000c00 tx=0x7f1c70006f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7003a040 con 0x7f1c801cd7b0 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 10 monclient: dump: 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.334 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f1c7003fe90 con 0x7f1c801cd7b0 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:22.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 msgr2=0x7f1c801cdbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 0x7f1c801cdbc0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1c70000c00 tx=0x7f1c70006f90 comp rx=0 tx=0).stop 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 msgr2=0x7f1c801cdbc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.322+0000 7f1c777fe640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1c801cd7b0 0x7f1c801cdbc0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:22.336 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 280873, 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.337 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.338 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:22.339 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:22.340 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.341 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.342 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.343 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:22.344 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:22.345 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:22.346 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.347 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.348 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:22.349 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:22.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1c6003e210 msgr2=0x7f1c600406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1c6003e210 0x7f1c600406d0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f1c78007fd0 tx=0x7f1c78003de0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 10 monclient: shutdown 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 msgr2=0x7f1c801ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 --2- 172.21.15.164:0/3384486483 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1c8009d930 0x7f1c801ca1b0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1c6800e930 tx=0x7f1c6800edf0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c7f59e640 1 -- 172.21.15.164:0/3384486483 reap_dead start 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c7f59e640 1 -- 172.21.15.164:0/3384486483 reap_dead start 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 shutdown_connections 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 >> 172.21.15.164:0/3384486483 conn(0x7f1c800a3110 msgr2=0x7f1c800a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 shutdown_connections 2024-09-10T16:24:22.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.323+0000 7f1c85240640 1 -- 172.21.15.164:0/3384486483 wait complete. 2024-09-10T16:24:22.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:22.352 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:24:22.352 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:24:22.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.448+0000 7f791e982640 10 monclient: get_monmap_and_config 2024-09-10T16:24:22.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.448+0000 7f791e982640 10 monclient: build_initial_monmap 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.448+0000 7f791e982640 10 monclient: monmap: 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:22.449236+0000 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:22.449236+0000 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:22.450 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:22.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 Processor -- start 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 -- start start 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient: init 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f791809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:22.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f791809d160 0x7f791809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): picked mon.noname-c con 0x7f791809d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f791813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): picked mon.noname-b con 0x7f791813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f791809a430 con 0x7f791809c180 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f79180a24e0 con 0x7f791809d160 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79180a2bf0 con 0x7f791813a5c0 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791e982640 10 monclient(hunting): authenticate will time out at 1840.936586s 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f791813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f7917fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f791809c180 0x7f791809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:58236/0 (socket says 172.21.15.164:58236) 2024-09-10T16:24:22.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f7917fff640 1 -- 172.21.15.164:0/3613299099 learned_addr learned my addr 172.21.15.164:0/3613299099 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): get_auth_request con 0x7f791813a5c0 auth_method 0 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 1 --2- 172.21.15.164:0/3613299099 >> v2:172.21.15.165:3300/0 conn(0x7f791809d160 0x7f791809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): get_auth_request con 0x7f791809d160 auth_method 0 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient(hunting): handle_auth_done global_id 204113 payload 274 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 -- 172.21.15.164:0/3613299099 >> v2:172.21.15.165:3300/0 conn(0x7f791809d160 msgr2=0x7f791809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 --2- 172.21.15.164:0/3613299099 >> v2:172.21.15.165:3300/0 conn(0x7f791809d160 0x7f791809f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 -- 172.21.15.164:0/3613299099 >> v1:172.21.15.164:6789/0 conn(0x7f791809c180 legacy=0x7f791809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient: _finish_hunting 0 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 monclient: found mon.noname-b 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:24:22.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.449+0000 7f791cf1f640 1 -- 172.21.15.164:0/3613299099 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f791809c560 con 0x7f791813a5c0 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791cf1f640 1 -- 172.21.15.164:0/3613299099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 msgr2=0x7f791813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791cf1f640 1 --2- 172.21.15.164:0/3613299099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f791813c9b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7910009dc0 tx=0x7f791002a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa5165f762b7dbf0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 1 -- 172.21.15.164:0/3613299099 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910007950 con 0x7f791813a5c0 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:24:22.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: dump: 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:22.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791e982640 5 monclient: authenticate success, global_id 204113 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 1 -- 172.21.15.164:0/3613299099 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7910007af0 con 0x7f791813a5c0 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791e982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791e982640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f791e982640 4 set_mon_vals no callback set 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 1 -- 172.21.15.164:0/3613299099 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7910031400 con 0x7f791813a5c0 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.452+0000 7f7916ffd640 10 monclient: dump: 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.464 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 10 monclient: shutdown 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 -- 172.21.15.164:0/3613299099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 msgr2=0x7f791813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 --2- 172.21.15.164:0/3613299099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f791813c9b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7910009dc0 tx=0x7f791002a4d0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 -- 172.21.15.164:0/3613299099 shutdown_connections 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 --2- 172.21.15.164:0/3613299099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f791813c9b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 --2- 172.21.15.164:0/3613299099 >> v2:172.21.15.165:3300/0 conn(0x7f791809d160 0x7f791809f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 -- 172.21.15.164:0/3613299099 >> 172.21.15.164:0/3613299099 conn(0x7f79180a3110 msgr2=0x7f79180a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 -- 172.21.15.164:0/3613299099 shutdown_connections 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.453+0000 7f791e982640 1 -- 172.21.15.164:0/3613299099 wait complete. 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 10 monclient: build_initial_monmap 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 10 monclient: monmap: 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:22.466 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 1 Processor -- start 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 1 -- start start 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 10 monclient: init 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.454+0000 7f791e982640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): picked mon.noname-a con 0x7f791809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f791809d160 0x7f7918099620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): picked mon.noname-b con 0x7f791809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f7918099b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): picked mon.noname-c con 0x7f791813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7918074f90 con 0x7f791809c180 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7918074e10 con 0x7f791809d160 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f791809f690 con 0x7f791813a5c0 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791e982640 10 monclient(hunting): authenticate will time out at 1840.942063s 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f7918099b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60336/0 (socket says 172.21.15.164:60336) 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 -- 172.21.15.164:0/3907678065 learned_addr learned my addr 172.21.15.164:0/3907678065 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): get_auth_request con 0x7f791809c180 auth_method 0 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): get_auth_request con 0x7f791813a5c0 auth_method 0 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f791809d160 0x7f7918099620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): get_auth_request con 0x7f791809d160 auth_method 0 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f79177fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient(hunting): handle_auth_done global_id 164478 payload 824 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 -- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 msgr2=0x7f7918099b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f7918099b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 -- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f791809d160 msgr2=0x7f7918099620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f791809d160 0x7f7918099620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 monclient: found mon.noname-a 2024-09-10T16:24:22.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f7917fff640 1 -- 172.21.15.164:0/3907678065 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79181ca680 con 0x7f791809c180 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.455+0000 7f791cf1f640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f7918099b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7917fff640 1 -- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 msgr2=0x7f791809d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7917fff640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f790800c460 tx=0x7f790800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908007070 con 0x7f791809c180 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: dump: 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.471 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: _finish_auth 0 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7908007230 con 0x7f791809c180 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f791e982640 5 monclient: authenticate success, global_id 164478 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f791e982640 10 monclient: _renew_subs 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f791e982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7918095930 con 0x7f791809c180 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7908007620 con 0x7f791809c180 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f7914ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f7914ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f7914ff9640 10 monclient: dump: 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:22.473 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 10 monclient: _renew_subs 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.457+0000 7f79167fc640 4 set_mon_vals no callback set 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f791809e170 con 0x7f791809c180 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f790801b400 con 0x7f791809c180 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.458+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f791809ea40 con 0x7f791809c180 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.459+0000 7f7914ff9640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 0x7f7904040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:22.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.459+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(619..619 src has 481..619) ==== 4799+0+0 (secure 0 0 0) 0x7f7908052990 con 0x7f791809c180 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.459+0000 7f79177fe640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 0x7f7904040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.459+0000 7f79177fe640 10 monclient: get_auth_request con 0x7f790403e240 auth_method 0 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.461+0000 7f79177fe640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 0x7f7904040700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f791809ea10 tx=0x7f790c009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.463+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7918095930 con 0x7f791809c180 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.463+0000 7f7914ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:22.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.463+0000 7f7914ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:22.502 INFO:tasks.rados.rados.0.smithi165.stdout:2778: done (0 left) 2024-09-10T16:24:22.503 INFO:tasks.rados.rados.0.smithi165.stdout:2779: copy_from oid 49 from oid 11 current snap is 295 2024-09-10T16:24:22.503 INFO:tasks.rados.rados.0.smithi165.stdout:2780: delete oid 18 current snap is 295 2024-09-10T16:24:22.505 INFO:tasks.rados.rados.0.smithi165.stdout:2780: done (1 left) 2024-09-10T16:24:22.505 INFO:tasks.rados.rados.0.smithi165.stdout:2781: copy_from oid 30 from oid 14 current snap is 295 2024-09-10T16:24:22.505 INFO:tasks.rados.rados.0.smithi165.stdout:2782: snap_create 2024-09-10T16:24:22.510 INFO:tasks.rados.rados.0.smithi165.stdout:2779: finishing copy_from racing read to smithi16534841-49 2024-09-10T16:24:22.511 INFO:tasks.rados.rados.0.smithi165.stdout:2779: finishing copy_from to smithi16534841-49 2024-09-10T16:24:22.511 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1152 (ObjNum 595 snap 186 seq_num 595) dirty exists 2024-09-10T16:24:22.515 INFO:tasks.rados.rados.0.smithi165.stdout:2781: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:24:22.518 INFO:tasks.rados.rados.0.smithi165.stdout:2781: finishing copy_from to smithi16534841-30 2024-09-10T16:24:22.518 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1237 (ObjNum 966 snap 292 seq_num 966) dirty exists 2024-09-10T16:24:22.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f791e982640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:24:22.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f791e982640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:24:22.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f791e982640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:22.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7918077260 con 0x7f791809c180 2024-09-10T16:24:22.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f7914ff9640 1 -- 172.21.15.164:0/3907678065 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f79181ca680 con 0x7f791809c180 2024-09-10T16:24:22.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f7914ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:24:22.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.709+0000 7f7914ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:22.713 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.713 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=12,logm=43} crc {auth=1495078969,config=2586702516,health=3727880303,kv=805391194,logm=1871096805}) 2024-09-10T16:24:22.713 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=3249894822,config=2586702516,health=327601140,kv=805391194,logm=2492041683}) 2024-09-10T16:24:22.713 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=12,kv=12,logm=43} crc {auth=1495078969,config=2586702516,health=3727880303,kv=805391194,logm=1871096805}) 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.710+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=31,config=2,health=11,kv=11,logm=45} crc {auth=1605719676,config=2586702516,health=67149542,kv=2631199854,logm=2219614742}) 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137828540}) 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2297923898}) 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137828540}) 2024-09-10T16:24:22.714 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.711+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972574796}) 2024-09-10T16:24:22.715 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.715 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199002041}) 2024-09-10T16:24:22.715 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4109151266}) 2024-09-10T16:24:22.716 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.716 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1199002041}) 2024-09-10T16:24:22.716 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.712+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2972778521}) 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 msgr2=0x7f7904040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 0x7f7904040700 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f791809ea10 tx=0x7f790c009190 comp rx=0 tx=0).stop 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 10 monclient: shutdown 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 msgr2=0x7f791809d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f790800c460 tx=0x7f790800c920 comp rx=0 tx=0).stop 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 shutdown_connections 2024-09-10T16:24:22.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f790403e240 0x7f7904040700 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f791813a5c0 0x7f7918099b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f791809d160 0x7f7918099620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 --2- 172.21.15.164:0/3907678065 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f791809c180 0x7f791809d930 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:22.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 >> 172.21.15.164:0/3907678065 conn(0x7f79180a3110 msgr2=0x7f7918139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:22.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 shutdown_connections 2024-09-10T16:24:22.718 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.718 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845843021}) 2024-09-10T16:24:22.718 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3165735141}) 2024-09-10T16:24:22.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f791e982640 1 -- 172.21.15.164:0/3907678065 wait complete. 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=845843021}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.713+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=795361229}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853269175}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=170333214}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3853269175}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.714+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=593391112}) 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.728 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3944903153}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1472253998}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3944903153}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.715+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386423688}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037597343}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2582609924}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2037597343}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442750307}) 2024-09-10T16:24:22.729 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720898496}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390511089}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.716+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.717+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1720898496}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.717+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582544333}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1855555525}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=825827795}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1855555525}) 2024-09-10T16:24:22.730 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.718+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3806936477}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187784447}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=757107649}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3187784447}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=425068234}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178991629}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2305440651}) 2024-09-10T16:24:22.731 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3178991629}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.719+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719672418}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249295739}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274665870}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249295739}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.720+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1891317235}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341175077}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1884697732}) 2024-09-10T16:24:22.732 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3341175077}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.721+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2986012681}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663493550}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2472788757}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=663493550}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.722+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2502948141}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028420718}) 2024-09-10T16:24:22.733 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4277730525}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2028420718}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.723+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566049359}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434542923}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2842625709}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3434542923}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.724+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4043003931}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797957414}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3356842385}) 2024-09-10T16:24:22.734 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3797957414}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1379470234}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128554438}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3615107004}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128554438}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.725+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1649559449}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366461927}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2924295968}) 2024-09-10T16:24:22.735 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2366461927}) 2024-09-10T16:24:22.736 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.726+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182894686}) 2024-09-10T16:24:22.737 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:24:22.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486615143}) 2024-09-10T16:24:22.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170686595}) 2024-09-10T16:24:22.752 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486615143}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1513918402}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941960652}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037688734}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941960652}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.727+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2750424659}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461983717}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993846001}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1461983717}) 2024-09-10T16:24:22.753 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.728+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3853503400}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140503622}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=269477413}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4140503622}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.729+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=652692136}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958024838}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=574572139}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2958024838}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.730+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3850440425}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646899517}) 2024-09-10T16:24:22.754 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2697249742}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2646899517}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187685496}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545845841}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2751950455}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545845841}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.731+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=911479781}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583455984}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=396719152}) 2024-09-10T16:24:22.755 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583455984}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=755610025}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174796962}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.732+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2795945623}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3174796962}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523361541}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916327791}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1908020479}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1916327791}) 2024-09-10T16:24:22.756 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.733+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2052799157}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864890047}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301872643}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864890047}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3352524385}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802621808}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=391061573}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802621808}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.734+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618749077}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122545830}) 2024-09-10T16:24:22.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1447271941}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122545830}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3748720403}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006749856}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=186737904}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006749856}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.735+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=179634024}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988772507}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3956096319}) 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=988772507}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.736+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618154998}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907976709}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823691093}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907976709}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4064543950}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190347276}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524878304}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4190347276}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.737+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3126356515}) 2024-09-10T16:24:22.759 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706104641}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=313090480}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706104641}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.738+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122314229}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288090347}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675964656}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288090347}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3302275987}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225706091}) 2024-09-10T16:24:22.760 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3097816782}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1225706091}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.739+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168654322}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2098359596}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3645155754}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2098359596}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3050642102}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095345222}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470697951}) 2024-09-10T16:24:22.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3095345222}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.740+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2794359633}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808476307}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4214253092}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808476307}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.741+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036099073}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225523740}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2256220602}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225523740}) 2024-09-10T16:24:22.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1990676599}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595942405}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=123611175}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1595942405}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.742+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249921594}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172477808}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136946293}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=172477808}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1860217889}) 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253473205}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059062255}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3253473205}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.743+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689296881}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584113846}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2136201564}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584113846}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.744+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=876019704}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923906238}) 2024-09-10T16:24:22.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339837571}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3923906238}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=596404827}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098280328}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=765572725}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1098280328}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.745+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=972164699}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188438461}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1890569417}) 2024-09-10T16:24:22.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=188438461}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.746+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1198642740}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383543813}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2839275897}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1383543813}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270382198}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560539115}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3718797784}) 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3560539115}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.747+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867902505}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041594154}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1385813592}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3041594154}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.748+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3869731149}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731774750}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1111669534}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731774750}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268696609}) 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154139934}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=127441315}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=154139934}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.749+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4276153299}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425863646}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501444309}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425863646}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=52001822}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068376268}) 2024-09-10T16:24:22.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=292506121}) 2024-09-10T16:24:22.769 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.769 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068376268}) 2024-09-10T16:24:22.769 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.750+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2820304343}) 2024-09-10T16:24:22.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488805126}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2352109464}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=488805126}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.751+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=747752470}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616116121}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288668422}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2616116121}) 2024-09-10T16:24:22.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.752+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=228463292}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469597332}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037049829}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=469597332}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3127005172}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.753+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=1} crc {logm=361347527,mdsmap=931935893,mgr=3221683499,mgr_command_descs=3192186153,mgrstat=1475719001}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=88,mdsmap=3,mgr=8,mgr_command_descs=1} crc {logm=820255319,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153}) 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.775 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=88,mdsmap=3,mgr=7,mgr_command_descs=1,mgrstat=1} crc {logm=361347527,mdsmap=931935893,mgr=3221683499,mgr_command_descs=3192186153,mgrstat=1475719001}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=89,mdsmap=2,mgr=8,mgr_command_descs=1} crc {logm=2456116885,mdsmap=204415920,mgr=1154067016,mgr_command_descs=3192186153}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=64} crc {mgrstat=3080576412,mon_config_key=1698199361,monmap=1433040829,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=3563206469,osd_snap=2712066355}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=63} crc {mgrstat=3080576412,mon_config_key=2124418738,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=3563206469,osd_snap=2142791427}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=64} crc {mgrstat=3080576412,mon_config_key=1698199361,monmap=1433040829,nvmeofgw=598337686,osd_metadata=141423795,osd_pg_creating=3563206469,osd_snap=2712066355}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.754+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=8,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=63} crc {mgrstat=3080576412,mon_config_key=2412854751,monmap=1433040829,nvmeofgw=598337686,osd_metadata=3297165161,osd_pg_creating=3563206469,osd_snap=4286655238}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=4182210601,osdmap=203245627}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=1830057999,osdmap=2545053510}) 2024-09-10T16:24:22.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=4182210601,osdmap=203245627}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.755+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=1326855961,osdmap=3902097905}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=315131901}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3608290951}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=315131901}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.756+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=974880611}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=3566051296}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=66} crc {osdmap=1339617883}) 2024-09-10T16:24:22.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:24:22.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=67} crc {osdmap=3566051296}) 2024-09-10T16:24:22.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:24:22.760+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=64} crc {osdmap=3318936503}) 2024-09-10T16:24:23.517 INFO:tasks.rados.rados.0.smithi165.stdout:2779: done (2 left) 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:2781: done (1 left) 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:2782: done (0 left) 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:2783: rollback oid 32 current snap is 296 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 32 to 295 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:2784: write oid 28 current snap is 296 2024-09-10T16:24:23.518 INFO:tasks.rados.rados.0.smithi165.stdout:2784: seq_num 973 ranges {753211=794108,1987803=365408} 2024-09-10T16:24:23.525 INFO:tasks.rados.rados.0.smithi165.stdout:2784: writing smithi16534841-28 from 753211 to 1547319 tid 1 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:2784: writing smithi16534841-28 from 1987803 to 2353211 tid 2 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:2785: rollback oid 20 current snap is 296 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 292 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:2783: finishing rollback tid 1 to smithi16534841-32 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 0 (ObjNum 801 snap 0 seq_num 326783444) dirty dne 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:2783: done (2 left) 2024-09-10T16:24:23.530 INFO:tasks.rados.rados.0.smithi165.stdout:2786: snap_create 2024-09-10T16:24:23.531 INFO:tasks.rados.rados.0.smithi165.stdout:2784: finishing write tid 1 to smithi16534841-28 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: finishing write tid 2 to smithi16534841-28 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: finishing write tid 3 to smithi16534841-28 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: finishing write tid 4 to smithi16534841-28 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: oid 28 updating version 0 to 1274 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: oid 28 version 1274 is already newer than 1273 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: oid 28 version 1274 is already newer than 1272 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1274 (ObjNum 973 snap 296 seq_num 973) dirty exists 2024-09-10T16:24:23.533 INFO:tasks.rados.rados.0.smithi165.stdout:2784: left oid 28 (ObjNum 973 snap 296 seq_num 973) 2024-09-10T16:24:23.534 INFO:tasks.rados.rados.0.smithi165.stdout:2785: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:24:23.534 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:24:23.738 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:24:23.738 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:24.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.846+0000 7effa2117640 10 monclient: get_monmap_and_config 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.846+0000 7effa2117640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.846+0000 7effa2117640 10 monclient: monmap: 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:23.847804+0000 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:23.847804+0000 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:24.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.847+0000 7effa2117640 1 Processor -- start 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.847+0000 7effa2117640 1 -- start start 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.847+0000 7effa2117640 10 monclient: init 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): picked mon.noname-b con 0x7eff9c0a1e40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7eff9c091700 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7eff9c092210 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9c09ee60 con 0x7eff9c092210 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff9c09f140 con 0x7eff9c091700 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c0a0640 con 0x7eff9c0a1e40 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7effa2117640 10 monclient(hunting): authenticate will time out at 1842.335212s 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:37480/0 (socket says 172.21.15.164:37480) 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 1 -- 172.21.15.164:0/243447099 learned_addr learned my addr 172.21.15.164:0/243447099 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7eff9c092210 0x7eff9c08d040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:58240/0 (socket says 172.21.15.164:58240) 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 10 monclient(hunting): get_auth_request con 0x7eff9c0a1e40 auth_method 0 2024-09-10T16:24:24.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.848+0000 7eff9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:23.850+0000 7eff9bfff640 1 -- 172.21.15.164:0/243447099 >> v1:172.21.15.164:6789/0 conn(0x7eff9c092210 legacy=0x7eff9c08d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9affd640 1 -- 172.21.15.164:0/243447099 >> v1:172.21.15.165:6789/0 conn(0x7eff9c091700 legacy=0x7eff9c091b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008859696 0 0) 0x7eff9c09ee60 con 0x7eff9c092210 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): my global_id is 164481 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff84003520 con 0x7eff9c092210 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1950330261 0 0) 0x7eff9c09f140 con 0x7eff9c091700 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 10 monclient(hunting): my global_id is 154510 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff9c09ee60 con 0x7eff9c091700 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_done global_id 204119 payload 274 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 1 -- 172.21.15.164:0/243447099 >> v1:172.21.15.165:6789/0 conn(0x7eff9c091700 legacy=0x7eff9c091b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 1 -- 172.21.15.164:0/243447099 >> v1:172.21.15.164:6789/0 conn(0x7eff9c092210 legacy=0x7eff9c08d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 1 monclient: found mon.noname-b 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.019+0000 7eff9b7fe640 1 -- 172.21.15.164:0/243447099 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c0a2220 con 0x7eff9c0a1e40 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9b7fe640 1 -- 172.21.15.164:0/243447099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 msgr2=0x7eff9c0a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9b7fe640 1 --2- 172.21.15.164:0/243447099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00c150 tx=0x7eff8c02b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a6e52875820f77f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c008700 con 0x7eff9c0a1e40 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: dump: 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.032 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7effa2117640 5 monclient: authenticate success, global_id 204119 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8c036070 con 0x7eff9c0a1e40 2024-09-10T16:24:24.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7effa2117640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7effa2117640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 1 -- 172.21.15.164:0/243447099 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8c008ac0 con 0x7eff9c0a1e40 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7effa2117640 4 set_mon_vals no callback set 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.022+0000 7eff9a7fc640 10 monclient: dump: 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.034 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 10 monclient: shutdown 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 -- 172.21.15.164:0/243447099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 msgr2=0x7eff9c0a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 --2- 172.21.15.164:0/243447099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7eff8c00c150 tx=0x7eff8c02b910 comp rx=0 tx=0).stop 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 -- 172.21.15.164:0/243447099 shutdown_connections 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 --2- 172.21.15.164:0/243447099 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c0a0100 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 -- 172.21.15.164:0/243447099 >> 172.21.15.164:0/243447099 conn(0x7eff9c0a2f30 msgr2=0x7eff9c0a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 -- 172.21.15.164:0/243447099 shutdown_connections 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.023+0000 7effa2117640 1 -- 172.21.15.164:0/243447099 wait complete. 2024-09-10T16:24:24.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.024+0000 7effa2117640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.024+0000 7effa2117640 10 monclient: monmap: 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.024+0000 7effa2117640 1 Processor -- start 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.024+0000 7effa2117640 1 -- start start 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.024+0000 7effa2117640 10 monclient: init 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff9c091700 0x7eff9c1da290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): picked mon.noname-b con 0x7eff9c091700 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): picked mon.noname-a con 0x7eff9c092210 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c1dad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): picked mon.noname-c con 0x7eff9c0a1e40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c08dae0 con 0x7eff9c092210 2024-09-10T16:24:24.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7eff9c08d960 con 0x7eff9c091700 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7eff9c08dc60 con 0x7eff9c0a1e40 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7effa2117640 10 monclient(hunting): authenticate will time out at 1842.512441s 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c1dad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:60340/0 (socket says 172.21.15.164:60340) 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 1 -- 172.21.15.164:0/3707417224 learned_addr learned my addr 172.21.15.164:0/3707417224 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): get_auth_request con 0x7eff9c0a1e40 auth_method 0 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): get_auth_request con 0x7eff9c092210 auth_method 0 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff9c091700 0x7eff9c1da290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): get_auth_request con 0x7eff9c091700 auth_method 0 2024-09-10T16:24:24.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient(hunting): handle_auth_done global_id 204122 payload 824 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 -- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff9c091700 msgr2=0x7eff9c1da290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff9c091700 0x7eff9c1da290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 -- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 msgr2=0x7eff9c1da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 monclient: found mon.noname-c 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9bfff640 1 -- 172.21.15.164:0/3707417224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff9c0741a0 con 0x7eff9c0a1e40 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.025+0000 7eff9affd640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.028+0000 7eff9bfff640 1 -- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 msgr2=0x7eff9c1dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.028+0000 7eff9bfff640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c1dad10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff94007fd0 tx=0x7eff9400c820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.028+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff940047e0 con 0x7eff9c0a1e40 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:24:24.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: dump: 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff94004980 con 0x7eff9c0a1e40 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff94004d70 con 0x7eff9c0a1e40 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 5 monclient: authenticate success, global_id 204122 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 10 monclient: _renew_subs 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff9c073a50 con 0x7eff9c0a1e40 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff83fff640 10 monclient: dump: 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.043 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7eff99ffb640 4 set_mon_vals no callback set 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 10 monclient: _renew_subs 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.030+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff9c0a1870 con 0x7eff9c0a1e40 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7eff940123f0 con 0x7eff9c0a1e40 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7eff83fff640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 0x7eff88040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(621..621 src has 481..621) ==== 4755+0+0 (secure 0 0 0) 0x7eff94052110 con 0x7eff9c0a1e40 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7effa2117640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7effa2117640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7effa2117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff9c09d620 con 0x7eff9c0a1e40 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7eff9b7fe640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 0x7eff88040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.031+0000 7eff9b7fe640 10 monclient: get_auth_request con 0x7eff8803e260 auth_method 0 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.034+0000 7eff9b7fe640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 0x7eff88040720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0926c0 tx=0x7eff8c00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.036+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7eff940113a0 con 0x7eff9c0a1e40 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.036+0000 7eff83fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.036+0000 7eff83fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:24.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.280+0000 7effa2117640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:24.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.280+0000 7effa2117640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:24.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.280+0000 7effa2117640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.280+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7eff9c09f140 con 0x7eff9c0a1e40 2024-09-10T16:24:24.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.281+0000 7eff83fff640 1 -- 172.21.15.164:0/3707417224 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7eff94010300 con 0x7eff9c0a1e40 2024-09-10T16:24:24.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.281+0000 7eff83fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:24.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.281+0000 7eff83fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:24.283 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 msgr2=0x7eff88040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 0x7eff88040720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7eff9c0926c0 tx=0x7eff8c00a000 comp rx=0 tx=0).stop 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 10 monclient: shutdown 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 msgr2=0x7eff9c1dad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c1dad10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7eff94007fd0 tx=0x7eff9400c820 comp rx=0 tx=0).stop 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 shutdown_connections 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7eff8803e260 0x7eff88040720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7eff9c0a1e40 0x7eff9c1dad10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7eff9c092210 0x7eff9c1da7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 --2- 172.21.15.164:0/3707417224 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7eff9c091700 0x7eff9c1da290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 >> 172.21.15.164:0/3707417224 conn(0x7eff9c0a2f30 msgr2=0x7eff9c0a5930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 shutdown_connections 2024-09-10T16:24:24.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.283+0000 7effa2117640 1 -- 172.21.15.164:0/3707417224 wait complete. 2024-09-10T16:24:24.295 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:24:24.295 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:24:24.295 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:24:24.295 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.392+0000 7f01604ed640 10 monclient: get_monmap_and_config 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.392+0000 7f01604ed640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.392+0000 7f01604ed640 10 monclient: monmap: 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:24.393826+0000 2024-09-10T16:24:24.394 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:24.393826+0000 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:24.395 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.393+0000 7f01604ed640 1 Processor -- start 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.393+0000 7f01604ed640 1 -- start start 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.393+0000 7f01604ed640 10 monclient: init 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f015809c180 0x7f015809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-a con 0x7f015809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f015809d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-b con 0x7f015813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f015809a430 con 0x7f015809d160 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f01580a24e0 con 0x7f015809c180 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01580a2bf0 con 0x7f015813a5c0 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f01604ed640 10 monclient(hunting): authenticate will time out at 1842.881240s 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f015809c180 0x7f015809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f015809c180 0x7f015809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42828/0 (socket says 172.21.15.164:42828) 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44868/0 (socket says 172.21.15.164:44868) 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 -- 172.21.15.164:0/733552226 learned_addr learned my addr 172.21.15.164:0/733552226 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): get_auth_request con 0x7f015813a5c0 auth_method 0 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): get_auth_request con 0x7f015809c180 auth_method 0 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient(hunting): handle_auth_done global_id 164487 payload 274 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 -- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 msgr2=0x7f015813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 -- 172.21.15.164:0/733552226 >> v1:172.21.15.165:6789/0 conn(0x7f015809d160 legacy=0x7f015809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015e289640 1 -- 172.21.15.164:0/733552226 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015809c560 con 0x7f015809c180 2024-09-10T16:24:24.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.394+0000 7f015ea8a640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:24.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015e289640 1 -- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015809c180 msgr2=0x7f015809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015e289640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015809c180 0x7f015809ea40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0148009e80 tx=0x7f014802a580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23f3ebbe3651a5f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 1 -- 172.21.15.164:0/733552226 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148036070 con 0x7f015809c180 2024-09-10T16:24:24.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: dump: 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.410 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 1 -- 172.21.15.164:0/733552226 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0148007800 con 0x7f015809c180 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f01604ed640 5 monclient: authenticate success, global_id 164487 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f01604ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f01604ed640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 1 -- 172.21.15.164:0/733552226 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148007bd0 con 0x7f015809c180 2024-09-10T16:24:24.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f01604ed640 4 set_mon_vals no callback set 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.397+0000 7f015d287640 10 monclient: dump: 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.412 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 10 monclient: shutdown 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 -- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015809c180 msgr2=0x7f015809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015809c180 0x7f015809ea40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0148009e80 tx=0x7f014802a580 comp rx=0 tx=0).stop 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 -- 172.21.15.164:0/733552226 shutdown_connections 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015813a5c0 0x7f015813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 --2- 172.21.15.164:0/733552226 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015809c180 0x7f015809ea40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 -- 172.21.15.164:0/733552226 >> 172.21.15.164:0/733552226 conn(0x7f01580a3110 msgr2=0x7f01580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 -- 172.21.15.164:0/733552226 shutdown_connections 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.398+0000 7f01604ed640 1 -- 172.21.15.164:0/733552226 wait complete. 2024-09-10T16:24:24.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient: monmap: 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 Processor -- start 2024-09-10T16:24:24.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 -- start start 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient: init 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 0x7f015806ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-c con 0x7f015809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015809d160 0x7f015806ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-b con 0x7f015809d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 0x7f015806f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): picked mon.noname-a con 0x7f015813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0158074f90 con 0x7f015813a5c0 2024-09-10T16:24:24.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0158074e10 con 0x7f015809d160 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f015809f690 con 0x7f015809c180 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f01604ed640 10 monclient(hunting): authenticate will time out at 1842.886882s 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 0x7f015806ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015ea8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 0x7f015806f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 0x7f015806ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44874/0 (socket says 172.21.15.164:44874) 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 1 -- 172.21.15.164:0/2782252382 learned_addr learned my addr 172.21.15.164:0/2782252382 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 10 monclient(hunting): get_auth_request con 0x7f015809c180 auth_method 0 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.399+0000 7f015e289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): get_auth_request con 0x7f015813a5c0 auth_method 0 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015da88640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015809d160 0x7f015806ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015da88640 10 monclient(hunting): get_auth_request con 0x7f015809d160 auth_method 0 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015da88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015da88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015da88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient(hunting): handle_auth_done global_id 164490 payload 824 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 msgr2=0x7f015806ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 0x7f015806ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015809d160 msgr2=0x7f015806ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f015809d160 0x7f015806ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0158132b40 con 0x7f015813a5c0 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.400+0000 7f015e289640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f015809c180 0x7f015806ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 msgr2=0x7f015806f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f015ea8a640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 0x7f015806f4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f015000e880 tx=0x7f015000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150017070 con 0x7f015813a5c0 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: dump: 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f015000cb50 con 0x7f015813a5c0 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 5 monclient: authenticate success, global_id 164490 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 10 monclient: _renew_subs 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0158070430 con 0x7f015813a5c0 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0150004020 con 0x7f015813a5c0 2024-09-10T16:24:24.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f0156ffd640 10 monclient: dump: 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 10 monclient: _renew_subs 2024-09-10T16:24:24.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015806ff90 con 0x7f015813a5c0 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.402+0000 7f015ca86640 4 set_mon_vals no callback set 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f01604ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f01604ed640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 0x7f015802e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f015809ea40 con 0x7f015806fa00 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 0x7f015802e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient: get_auth_request con 0x7f015806fa00 auth_method 0 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0150004240 con 0x7f015813a5c0 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 10 monclient(hunting): handle_auth_done global_id 164493 payload 824 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f015e289640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f01580720f0 con 0x7f015806fa00 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f0156ffd640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f014403e210 0x7f01440406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.403+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(621..621 src has 481..621) ==== 4755+0+0 (secure 0 0 0) 0x7f0150052fa0 con 0x7f015813a5c0 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.404+0000 7f015da88640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f014403e210 0x7f01440406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.404+0000 7f015da88640 10 monclient: get_auth_request con 0x7f014403e210 auth_method 0 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.405+0000 7f015e289640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 msgr2=0x7f015802e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.405+0000 7f015e289640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 0x7f015802e5c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f014802a5c0 tx=0x7f0148004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148042070 con 0x7f015806fa00 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 10 monclient: dump: 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.425 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0148037040 con 0x7f015806fa00 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 msgr2=0x7f015802e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 0x7f015802e5c0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f014802a5c0 tx=0x7f0148004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 msgr2=0x7f015802e5c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f0156ffd640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015806fa00 0x7f015802e5c0 secure :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f014802a5c0 tx=0x7f0148004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.406+0000 7f015da88640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f014403e210 0x7f01440406d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f01580a6390 tx=0x7f014c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f01604ed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:24.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f01604ed640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 0x7f0158064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f01604ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0158077260 con 0x7f0158063d50 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 0x7f0158064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient: get_auth_request con 0x7f0158063d50 auth_method 0 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 10 monclient(hunting): handle_auth_done global_id 164496 payload 824 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.412+0000 7f015e289640 1 -- 172.21.15.164:0/2782252382 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f01581327c0 con 0x7f0158063d50 2024-09-10T16:24:24.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.414+0000 7f015e289640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 msgr2=0x7f0158064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.414+0000 7f015e289640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 0x7f0158064160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f015809ebc0 tx=0x7f014802f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0148036040 con 0x7f0158063d50 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 10 monclient: dump: 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f0148004660 con 0x7f0158063d50 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 msgr2=0x7f0158064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 0x7f0158064160 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f015809ebc0 tx=0x7f014802f000 comp rx=0 tx=0).stop 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 msgr2=0x7f0158064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f0156ffd640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0158063d50 0x7f0158064160 secure :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f015809ebc0 tx=0x7f014802f000 comp rx=0 tx=0).stop 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 235117, 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:24.432 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.434 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:24.435 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:24.436 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.437 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.439 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:24.442 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:24.443 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.444 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:24.445 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:24.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f014403e210 msgr2=0x7f01440406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f014403e210 0x7f01440406d0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f01580a6390 tx=0x7f014c009040 comp rx=0 tx=0).stop 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 10 monclient: shutdown 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 msgr2=0x7f015806f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 --2- 172.21.15.164:0/2782252382 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f015813a5c0 0x7f015806f4c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f015000e880 tx=0x7f015000ed40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 reap_dead start 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f015ea8a640 1 -- 172.21.15.164:0/2782252382 reap_dead start 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 shutdown_connections 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 >> 172.21.15.164:0/2782252382 conn(0x7f01580a3110 msgr2=0x7f0158139ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 shutdown_connections 2024-09-10T16:24:24.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.415+0000 7f01604ed640 1 -- 172.21.15.164:0/2782252382 wait complete. 2024-09-10T16:24:24.448 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:24.448 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:24:24.448 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:24:24.530 INFO:tasks.rados.rados.0.smithi165.stdout:2784: done (2 left) 2024-09-10T16:24:24.530 INFO:tasks.rados.rados.0.smithi165.stdout:2785: done (1 left) 2024-09-10T16:24:24.530 INFO:tasks.rados.rados.0.smithi165.stdout:2786: done (0 left) 2024-09-10T16:24:24.530 INFO:tasks.rados.rados.0.smithi165.stdout:2787: delete oid 33 current snap is 297 2024-09-10T16:24:24.534 INFO:tasks.rados.rados.0.smithi165.stdout:2787: done (0 left) 2024-09-10T16:24:24.534 INFO:tasks.rados.rados.0.smithi165.stdout:2788: rollback oid 1 current snap is 297 2024-09-10T16:24:24.534 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 292 2024-09-10T16:24:24.534 INFO:tasks.rados.rados.0.smithi165.stdout:2789: write (excl) oid 50 current snap is 297 2024-09-10T16:24:24.534 INFO:tasks.rados.rados.0.smithi165.stdout:2789: seq_num 975 ranges {786825=660959,2104767=282058} 2024-09-10T16:24:24.542 INFO:tasks.rados.rados.0.smithi165.stdout:2789: writing smithi16534841-50 from 786825 to 1447784 tid 1 2024-09-10T16:24:24.547 INFO:tasks.rados.rados.0.smithi165.stdout:2789: writing smithi16534841-50 from 2104767 to 2386825 tid 2 2024-09-10T16:24:24.547 INFO:tasks.rados.rados.0.smithi165.stdout:2790: write oid 15 current snap is 297 2024-09-10T16:24:24.547 INFO:tasks.rados.rados.0.smithi165.stdout:2788: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:24:24.547 INFO:tasks.rados.rados.0.smithi165.stdout:2790: seq_num 976 ranges {403632=736208,1763249=619999} 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 10 monclient: get_monmap_and_config 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 10 monclient: monmap: 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:24.550969+0000 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:24.550969+0000 2024-09-10T16:24:24.551 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:24.552 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 1 Processor -- start 2024-09-10T16:24:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 1 -- start start 2024-09-10T16:24:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.550+0000 7f6e5edc6640 10 monclient: init 2024-09-10T16:24:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e5806cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e58134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6e5813e740 0x7f6e58140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5813e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e5809ce50 con 0x7f6e58134f70 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e58135380 con 0x7f6e5806cb70 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6e5806db60 con 0x7f6e5813e740 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5edc6640 10 monclient(hunting): authenticate will time out at 1843.038422s 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e57fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6e58134f70 0x7f6e5813d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39688/0 (socket says 172.21.15.164:39688) 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e57fff640 1 -- 172.21.15.164:0/785981822 learned_addr learned my addr 172.21.15.164:0/785981822 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:24.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 1 --2- 172.21.15.164:0/785981822 >> v2:172.21.15.165:3300/0 conn(0x7f6e5813e740 0x7f6e58140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): get_auth_request con 0x7f6e5813e740 auth_method 0 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient(hunting): handle_auth_done global_id 154519 payload 274 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 1 -- 172.21.15.164:0/785981822 >> v1:172.21.15.165:6789/0 conn(0x7f6e5806cb70 legacy=0x7f6e580a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 1 -- 172.21.15.164:0/785981822 >> v1:172.21.15.164:6789/0 conn(0x7f6e58134f70 legacy=0x7f6e5813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 1 monclient: found mon.noname-c 2024-09-10T16:24:24.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:24:24.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.551+0000 7f6e5d363640 1 -- 172.21.15.164:0/785981822 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5809cb70 con 0x7f6e5813e740 2024-09-10T16:24:24.557 INFO:tasks.rados.rados.0.smithi165.stdout:2790: writing smithi16534841-15 from 403632 to 1139840 tid 1 2024-09-10T16:24:24.563 INFO:tasks.rados.rados.0.smithi165.stdout:2790: writing smithi16534841-15 from 1763249 to 2383248 tid 2 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:2791: write (excl) oid 10 current snap is 297 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:2788: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:2788: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1417 (ObjNum 941 snap 285 seq_num 941) dirty exists 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:2789: finishing write tid 1 to smithi16534841-50 2024-09-10T16:24:24.564 INFO:tasks.rados.rados.0.smithi165.stdout:2791: seq_num 977 ranges {420439=411457,1421731=480010} 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.554+0000 7f6e5d363640 1 -- 172.21.15.164:0/785981822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5813e740 msgr2=0x7f6e58140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.554+0000 7f6e5d363640 1 --2- 172.21.15.164:0/785981822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5813e740 0x7f6e58140b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009dc0 tx=0x7f6e5002a4d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d86ef58d57bdd1e6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 1 -- 172.21.15.164:0/785981822 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50007a30 con 0x7f6e5813e740 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: dump: 2024-09-10T16:24:24.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 1 -- 172.21.15.164:0/785981822 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e50007bd0 con 0x7f6e5813e740 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 5 monclient: authenticate success, global_id 154519 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 1 -- 172.21.15.164:0/785981822 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50031400 con 0x7f6e5813e740 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 4 set_mon_vals no callback set 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e577fe640 10 monclient: dump: 2024-09-10T16:24:24.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 10 monclient: shutdown 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 -- 172.21.15.164:0/785981822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5813e740 msgr2=0x7f6e58140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/785981822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5813e740 0x7f6e58140b30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009dc0 tx=0x7f6e5002a4d0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 -- 172.21.15.164:0/785981822 shutdown_connections 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/785981822 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5813e740 0x7f6e58140b30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 -- 172.21.15.164:0/785981822 >> 172.21.15.164:0/785981822 conn(0x7f6e580aa900 msgr2=0x7f6e580aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 -- 172.21.15.164:0/785981822 shutdown_connections 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.555+0000 7f6e5edc6640 1 -- 172.21.15.164:0/785981822 wait complete. 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.556+0000 7f6e5edc6640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.556+0000 7f6e5edc6640 10 monclient: monmap: 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:24.569 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.556+0000 7f6e5edc6640 1 Processor -- start 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.556+0000 7f6e5edc6640 1 -- start start 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.556+0000 7f6e5edc6640 10 monclient: init 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5806cb70 0x7f6e5812d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-b con 0x7f6e5806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-a con 0x7f6e58134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e5813e740 0x7f6e5812fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): picked mon.noname-c con 0x7f6e5813e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6e580a4db0 con 0x7f6e58134f70 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5806cf50 con 0x7f6e5806cb70 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e58141070 con 0x7f6e5813e740 2024-09-10T16:24:24.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5edc6640 10 monclient(hunting): authenticate will time out at 1843.044604s 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e5813e740 0x7f6e5812fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5806cb70 0x7f6e5812d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42874/0 (socket says 172.21.15.164:42874) 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 1 -- 172.21.15.164:0/3947590847 learned_addr learned my addr 172.21.15.164:0/3947590847 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): get_auth_request con 0x7f6e58134f70 auth_method 0 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): get_auth_request con 0x7f6e5813e740 auth_method 0 2024-09-10T16:24:24.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): get_auth_request con 0x7f6e5806cb70 auth_method 0 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.557+0000 7f6e5cb62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 10 monclient(hunting): handle_auth_done global_id 164499 payload 824 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e5813e740 msgr2=0x7f6e5812fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e5813e740 0x7f6e5812fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5806cb70 msgr2=0x7f6e5812d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6e5806cb70 0x7f6e5812d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e57fff640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e5806e720 con 0x7f6e58134f70 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.558+0000 7f6e5d363640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e5813e740 0x7f6e5812fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e57fff640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 msgr2=0x7f6e5812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e57fff640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400c460 tx=0x7f6e4400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44007070 con 0x7f6e58134f70 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: dump: 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 5 monclient: authenticate success, global_id 164499 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 10 monclient: _renew_subs 2024-09-10T16:24:24.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e44007230 con 0x7f6e58134f70 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e58131960 con 0x7f6e58134f70 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e44007620 con 0x7f6e58134f70 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e557fa640 10 monclient: dump: 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e56ffd640 4 set_mon_vals no callback set 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 10 monclient: _renew_subs 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.560+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e5812dd00 con 0x7f6e58134f70 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6e4401b400 con 0x7f6e58134f70 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e557fa640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e3c03e270 0x7f6e3c040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(622..622 src has 481..622) ==== 4755+0+0 (secure 0 0 0) 0x7f6e44052990 con 0x7f6e58134f70 2024-09-10T16:24:24.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e5edc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 0x7f6e58130420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.561+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e580a46a0 con 0x7f6e58139110 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 0x7f6e58130420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5cb62640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e3c03e270 0x7f6e3c040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient: get_auth_request con 0x7f6e58139110 auth_method 0 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5cb62640 10 monclient: get_auth_request con 0x7f6e3c03e270 auth_method 0 2024-09-10T16:24:24.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 10 monclient(hunting): handle_auth_done global_id 204134 payload 824 2024-09-10T16:24:24.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.562+0000 7f6e5d363640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6e580008d0 con 0x7f6e58139110 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2791: writing smithi16534841-10 from 420439 to 831896 tid 1 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2791: writing smithi16534841-10 from 1421731 to 1901741 tid 2 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2788: done (3 left) 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2792: read oid 25 snap 294 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2792: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:24.583 INFO:tasks.rados.rados.0.smithi165.stdout:2793: read oid 22 snap -1 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2793: expect (ObjNum 913 snap 280 seq_num 913) 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2794: read oid 11 snap -1 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2794: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2795: delete oid 32 current snap is 297 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: finishing write tid 2 to smithi16534841-50 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: finishing write tid 3 to smithi16534841-50 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: finishing write tid 4 to smithi16534841-50 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: oid 50 updating version 0 to 1289 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: oid 50 version 1289 is already newer than 1288 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: oid 50 version 1289 is already newer than 1287 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1289 (ObjNum 975 snap 297 seq_num 975) dirty exists 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2789: left oid 50 (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:24:24.584 INFO:tasks.rados.rados.0.smithi165.stdout:2790: finishing write tid 1 to smithi16534841-15 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: finishing write tid 2 to smithi16534841-15 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: finishing write tid 3 to smithi16534841-15 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: finishing write tid 4 to smithi16534841-15 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: oid 15 updating version 0 to 1341 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: oid 15 version 1341 is already newer than 1340 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: oid 15 version 1341 is already newer than 1339 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1341 (ObjNum 976 snap 297 seq_num 976) dirty exists 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2790: left oid 15 (ObjNum 976 snap 297 seq_num 976) 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2791: finishing write tid 1 to smithi16534841-10 2024-09-10T16:24:24.585 INFO:tasks.rados.rados.0.smithi165.stdout:2794: expect (ObjNum 595 snap 186 seq_num 595) 2024-09-10T16:24:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.564+0000 7f6e5d363640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 msgr2=0x7f6e58130420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.564+0000 7f6e5d363640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 0x7f6e58130420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009d90 tx=0x7f6e50004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.564+0000 7f6e5cb62640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e3c03e270 0x7f6e3c040730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006fd0 tx=0x7f6e4c009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50046440 con 0x7f6e58139110 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 10 monclient: dump: 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6e50046440 con 0x7f6e58139110 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 msgr2=0x7f6e58130420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 0x7f6e58130420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009d90 tx=0x7f6e50004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 msgr2=0x7f6e58130420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.565+0000 7f6e557fa640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58139110 0x7f6e58130420 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009d90 tx=0x7f6e50004a40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5edc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 0x7f6e58130d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5edc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6e581312b0 con 0x7f6e58130960 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 0x7f6e58130d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient: get_auth_request con 0x7f6e58130960 auth_method 0 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 10 monclient(hunting): handle_auth_done global_id 204137 payload 824 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.573+0000 7f6e5d363640 1 -- 172.21.15.164:0/3947590847 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6e581d8610 con 0x7f6e58130960 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e5d363640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 msgr2=0x7f6e58130d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e5d363640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 0x7f6e58130d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e5002a510 tx=0x7f6e50030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e50004600 con 0x7f6e58130960 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 10 monclient: dump: 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f6e5004c040 con 0x7f6e58130960 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:24.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 msgr2=0x7f6e58130d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 0x7f6e58130d70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f6e5002a510 tx=0x7f6e50030000 comp rx=0 tx=0).stop 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 msgr2=0x7f6e58130d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.576+0000 7f6e557fa640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6e58130960 0x7f6e58130d70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:24.591 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 14313, 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 8, 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:24.592 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:24.593 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:24.595 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.596 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.599 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:24.601 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.603 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.604 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:24.604 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e3c03e270 msgr2=0x7f6e3c040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6e3c03e270 0x7f6e3c040730 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c006fd0 tx=0x7f6e4c009060 comp rx=0 tx=0).stop 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 10 monclient: shutdown 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 msgr2=0x7f6e5812d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6e4400c460 tx=0x7f6e4400c920 comp rx=0 tx=0).stop 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5d363640 1 -- 172.21.15.164:0/3947590847 reap_dead start 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 shutdown_connections 2024-09-10T16:24:24.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 --2- 172.21.15.164:0/3947590847 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6e58134f70 0x7f6e5812d7c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 >> 172.21.15.164:0/3947590847 conn(0x7f6e580aa900 msgr2=0x7f6e580a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 shutdown_connections 2024-09-10T16:24:24.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.577+0000 7f6e5edc6640 1 -- 172.21.15.164:0/3947590847 wait complete. 2024-09-10T16:24:24.607 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:24.607 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:24:24.607 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:24:24.608 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 343579 2024-09-10T16:24:24.637 INFO:tasks.rados.rados.0.smithi165.stdout:2791: finishing write tid 2 to smithi16534841-10 2024-09-10T16:24:24.637 INFO:tasks.rados.rados.0.smithi165.stdout:2791: finishing write tid 3 to smithi16534841-10 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2791: finishing write tid 4 to smithi16534841-10 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2791: oid 10 updating version 0 to 1551 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2791: oid 10 version 1551 is already newer than 1550 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2791: oid 10 version 1551 is already newer than 1549 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1551 (ObjNum 977 snap 297 seq_num 977) dirty exists 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2791: left oid 10 (ObjNum 977 snap 297 seq_num 977) 2024-09-10T16:24:24.638 INFO:tasks.rados.rados.0.smithi165.stdout:2792: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:24.647 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2863525 2024-09-10T16:24:24.664 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 855340 2024-09-10T16:24:24.696 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1435320 2024-09-10T16:24:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.705+0000 7f80b49ad640 10 monclient: get_monmap_and_config 2024-09-10T16:24:24.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.705+0000 7f80b49ad640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.705+0000 7f80b49ad640 10 monclient: monmap: 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:24.706519+0000 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:24.706519+0000 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:24.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 Processor -- start 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 -- start start 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient: init 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-b con 0x7f80b009d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-a con 0x7f80b009ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:24.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f80b0134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80b00a0870 con 0x7f80b0134580 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f80b009d3e0 con 0x7f80b009ab50 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f80b009e790 con 0x7f80b009d930 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80b49ad640 10 monclient(hunting): authenticate will time out at 1843.193869s 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80ae59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.706+0000 7f80a7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:44910/0 (socket says 172.21.15.164:44910) 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42876/0 (socket says 172.21.15.164:42876) 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 -- 172.21.15.164:0/540140642 learned_addr learned my addr 172.21.15.164:0/540140642 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): get_auth_request con 0x7f80b009d930 auth_method 0 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): get_auth_request con 0x7f80b009ab50 auth_method 0 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient(hunting): handle_auth_done global_id 204140 payload 274 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 -- 172.21.15.164:0/540140642 >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 msgr2=0x7f80b009e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 --2- 172.21.15.164:0/540140642 >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 -- 172.21.15.164:0/540140642 >> v1:172.21.15.165:6789/0 conn(0x7f80b0134580 legacy=0x7f80b0134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 monclient: found mon.noname-b 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:24:24.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80ae59c640 1 -- 172.21.15.164:0/540140642 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b0074c70 con 0x7f80b009d930 2024-09-10T16:24:24.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.707+0000 7f80a7fff640 1 --2- 172.21.15.164:0/540140642 >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:24:24.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80ae59c640 1 -- 172.21.15.164:0/540140642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 msgr2=0x7f80b009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80ae59c640 1 --2- 172.21.15.164:0/540140642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009870 tx=0x7f80a002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c657cc0442ca6118 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 1 -- 172.21.15.164:0/540140642 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0037070 con 0x7f80b009d930 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: dump: 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.721 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 1 -- 172.21.15.164:0/540140642 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a0032640 con 0x7f80b009d930 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 5 monclient: authenticate success, global_id 204140 2024-09-10T16:24:24.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 1 -- 172.21.15.164:0/540140642 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0032a10 con 0x7f80b009d930 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 4 set_mon_vals no callback set 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80add9b640 10 monclient: dump: 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.723 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 10 monclient: shutdown 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 1 -- 172.21.15.164:0/540140642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 msgr2=0x7f80b009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.710+0000 7f80b49ad640 1 --2- 172.21.15.164:0/540140642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009870 tx=0x7f80a002a130 comp rx=0 tx=0).stop 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 -- 172.21.15.164:0/540140642 shutdown_connections 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 --2- 172.21.15.164:0/540140642 >> v2:172.21.15.164:3300/0 conn(0x7f80b009ab50 0x7f80b009e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 --2- 172.21.15.164:0/540140642 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b009d930 0x7f80b009dd10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 -- 172.21.15.164:0/540140642 >> 172.21.15.164:0/540140642 conn(0x7f80b00a3110 msgr2=0x7f80b00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 -- 172.21.15.164:0/540140642 shutdown_connections 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 1 -- 172.21.15.164:0/540140642 wait complete. 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.711+0000 7f80b49ad640 10 monclient: monmap: 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 Processor -- start 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 -- start start 2024-09-10T16:24:24.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient: init 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-a con 0x7f80b009ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b009d930 0x7f80b00741e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-b con 0x7f80b009d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b0134580 0x7f80b00702b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): picked mon.noname-c con 0x7f80b0134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f80b009ef30 con 0x7f80b009ab50 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f80b00a02b0 con 0x7f80b009d930 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f80b00a2940 con 0x7f80b0134580 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80b49ad640 10 monclient(hunting): authenticate will time out at 1843.199671s 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42892/0 (socket says 172.21.15.164:42892) 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b0134580 0x7f80b00702b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 1 -- 172.21.15.164:0/2685573446 learned_addr learned my addr 172.21.15.164:0/2685573446 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): get_auth_request con 0x7f80b009ab50 auth_method 0 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): get_auth_request con 0x7f80b0134580 auth_method 0 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b009d930 0x7f80b00741e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): get_auth_request con 0x7f80b009d930 auth_method 0 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.712+0000 7f80ae59c640 10 monclient(hunting): handle_auth_done global_id 164505 payload 824 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b0134580 msgr2=0x7f80b00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f80b0134580 0x7f80b00702b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b009d930 msgr2=0x7f80b00741e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b009d930 0x7f80b00741e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.713+0000 7f80ae59c640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80b0073520 con 0x7f80b009ab50 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80ae59c640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 msgr2=0x7f80b0098db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80ae59c640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009870 tx=0x7f80a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0041020 con 0x7f80b009ab50 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: dump: 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.730 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80a003d830 con 0x7f80b009ab50 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 5 monclient: authenticate success, global_id 164505 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 10 monclient: _renew_subs 2024-09-10T16:24:24.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80b0073800 con 0x7f80b009ab50 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80a0037040 con 0x7f80b009ab50 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80a6ffd640 10 monclient: dump: 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.732 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80ad59a640 4 set_mon_vals no callback set 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 10 monclient: _renew_subs 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.715+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80b0071e00 con 0x7f80b009ab50 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80b49ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80b49ad640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 0x7f80b002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f80b00a1e40 con 0x7f80b002d440 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f80a0044020 con 0x7f80b009ab50 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 0x7f80b002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient: get_auth_request con 0x7f80b002d440 auth_method 0 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a6ffd640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f809003e210 0x7f80900406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(622..622 src has 481..622) ==== 4755+0+0 (secure 0 0 0) 0x7f80a0073230 con 0x7f80b009ab50 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 10 monclient(hunting): handle_auth_done global_id 154528 payload 824 2024-09-10T16:24:24.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80a7fff640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f80b006dac0 con 0x7f80b002d440 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80aed9d640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f809003e210 0x7f80900406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.716+0000 7f80aed9d640 10 monclient: get_auth_request con 0x7f809003e210 auth_method 0 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a7fff640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 msgr2=0x7f80b002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a7fff640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 0x7f80b002d820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f80b0074900 tx=0x7f8098011d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f809800fe50 con 0x7f80b002d440 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 10 monclient: dump: 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.735 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80aed9d640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f809003e210 0x7f80900406d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f80a80055e0 tx=0x7f80a8003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f809800fe50 con 0x7f80b002d440 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 msgr2=0x7f80b002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 0x7f80b002d820 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f80b0074900 tx=0x7f8098011d40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 msgr2=0x7f80b002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.719+0000 7f80a6ffd640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002d440 0x7f80b002d820 secure :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f80b0074900 tx=0x7f8098011d40 comp rx=0 tx=0).stop 2024-09-10T16:24:24.737 INFO:tasks.rados.rados.0.smithi165.stdout:2793: expect (ObjNum 913 snap 280 seq_num 913) 2024-09-10T16:24:24.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80b49ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80b49ad640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 0x7f80b002e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80b49ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f80b00916a0 con 0x7f80b002dd60 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 0x7f80b002e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient: get_auth_request con 0x7f80b002dd60 auth_method 0 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 10 monclient(hunting): handle_auth_done global_id 154531 payload 824 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.725+0000 7f80a7fff640 1 -- 172.21.15.164:0/2685573446 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f80b0071070 con 0x7f80b002dd60 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a7fff640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 msgr2=0x7f80b002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a7fff640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 0x7f80b002e170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f80b0074900 tx=0x7f80980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098014850 con 0x7f80b002dd60 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 10 monclient: dump: 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f809801d400 con 0x7f80b002dd60 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 msgr2=0x7f80b002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 0x7f80b002e170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f80b0074900 tx=0x7f80980046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 msgr2=0x7f80b002e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80a6ffd640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f80b002dd60 0x7f80b002e170 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.741 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 176, 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 283279, 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.742 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 8, 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.743 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.744 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.745 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:24.746 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.747 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:24.748 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.749 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.750 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.751 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.752 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.753 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:24.754 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:24.755 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:24.756 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f809003e210 msgr2=0x7f80900406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f809003e210 0x7f80900406d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f80a80055e0 tx=0x7f80a8003d10 comp rx=0 tx=0).stop 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 10 monclient: shutdown 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 msgr2=0x7f80b0098db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80b49ad640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f80a0009870 tx=0x7f80a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.728+0000 7f80aed9d640 1 -- 172.21.15.164:0/2685573446 reap_dead start 2024-09-10T16:24:24.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.729+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 shutdown_connections 2024-09-10T16:24:24.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.729+0000 7f80b49ad640 1 --2- 172.21.15.164:0/2685573446 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f80b009ab50 0x7f80b0098db0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.729+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 >> 172.21.15.164:0/2685573446 conn(0x7f80b00a3110 msgr2=0x7f80b00997e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.729+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 shutdown_connections 2024-09-10T16:24:24.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.729+0000 7f80b49ad640 1 -- 172.21.15.164:0/2685573446 wait complete. 2024-09-10T16:24:24.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:24:24.758 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:24:24.758 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:24:24.759 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:24:24.759 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:24:24.759 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:24:24.759 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:24:24.768 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:24:24.768 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:24:24.768 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:24.804 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2131987 2024-09-10T16:24:24.892 INFO:tasks.rados.rados.0.smithi165.stdout:2789: done (6 left) 2024-09-10T16:24:24.892 INFO:tasks.rados.rados.0.smithi165.stdout:2790: done (5 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2791: done (4 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2792: done (3 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2793: done (2 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2794: done (1 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2795: done (0 left) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2796: read oid 15 snap -1 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2796: expect (ObjNum 976 snap 297 seq_num 976) 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2797: write oid 49 current snap is 297 2024-09-10T16:24:24.893 INFO:tasks.rados.rados.0.smithi165.stdout:2797: seq_num 979 ranges {454053=561955,1422342=483679,2368404=85649} 2024-09-10T16:24:24.897 INFO:tasks.rados.rados.0.smithi165.stdout:2797: writing smithi16534841-49 from 454053 to 1016008 tid 1 2024-09-10T16:24:24.902 INFO:tasks.rados.rados.0.smithi165.stdout:2797: writing smithi16534841-49 from 1422342 to 1906021 tid 2 2024-09-10T16:24:24.906 INFO:tasks.rados.rados.0.smithi165.stdout:2797: writing smithi16534841-49 from 2368404 to 2454053 tid 3 2024-09-10T16:24:24.906 INFO:tasks.rados.rados.0.smithi165.stdout:2798: delete oid 17 current snap is 297 2024-09-10T16:24:24.906 INFO:tasks.rados.rados.0.smithi165.stdout:2796: expect (ObjNum 976 snap 297 seq_num 976) 2024-09-10T16:24:24.906 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.904+0000 7f63f54bd640 10 monclient: get_monmap_and_config 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.904+0000 7f63f54bd640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.904+0000 7f63f54bd640 10 monclient: monmap: 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:24.905553+0000 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:24.905553+0000 2024-09-10T16:24:24.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.905+0000 7f63f54bd640 1 Processor -- start 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.905+0000 7f63f54bd640 1 -- start start 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.905+0000 7f63f54bd640 10 monclient: init 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 0x7f63f0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-b con 0x7f63f006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63f00a46a0 0x7f63f00a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-a con 0x7f63f00a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f63f0137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f63f009ce50 con 0x7f63f0137ca0 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f63f006db60 con 0x7f63f00a46a0 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63f006cf50 con 0x7f63f006cb70 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63f54bd640 10 monclient(hunting): authenticate will time out at 1843.393144s 2024-09-10T16:24:24.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63eeffd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 msgr2=0x7f63f0134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63eeffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 0x7f63f0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63f00a46a0 0x7f63f00a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63ef7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f63f0137ca0 0x7f63f0138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39704/0 (socket says 172.21.15.164:39704) 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63ef7fe640 1 -- 172.21.15.164:0/1457406164 learned_addr learned my addr 172.21.15.164:0/1457406164 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f63f00a46a0 0x7f63f00a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42902/0 (socket says 172.21.15.164:42902) 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): get_auth_request con 0x7f63f00a46a0 auth_method 0 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient(hunting): handle_auth_done global_id 164508 payload 274 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 -- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 msgr2=0x7f63f0134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 --2- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 0x7f63f0134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 -- 172.21.15.164:0/1457406164 >> v1:172.21.15.164:6789/0 conn(0x7f63f0137ca0 legacy=0x7f63f0138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.906+0000 7f63e7fff640 1 -- 172.21.15.164:0/1457406164 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f009cb70 con 0x7f63f00a46a0 2024-09-10T16:24:24.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63e7fff640 1 -- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f00a46a0 msgr2=0x7f63f00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63e7fff640 1 --2- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f00a46a0 0x7f63f00a4a80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d8029fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7b19cfc3cdb5ad server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 1 -- 172.21.15.164:0/1457406164 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8037070 con 0x7f63f00a46a0 2024-09-10T16:24:24.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: dump: 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 1 -- 172.21.15.164:0/1457406164 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63d8032640 con 0x7f63f00a46a0 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 1 -- 172.21.15.164:0/1457406164 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d8032a10 con 0x7f63f00a46a0 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63f54bd640 5 monclient: authenticate success, global_id 164508 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63f54bd640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63f54bd640 4 set_mon_vals no callback set 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.909+0000 7f63ee7fc640 10 monclient: dump: 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.920 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 10 monclient: shutdown 2024-09-10T16:24:24.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 -- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f00a46a0 msgr2=0x7f63f00a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 --2- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f00a46a0 0x7f63f00a4a80 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f63d80097a0 tx=0x7f63d8029fe0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 -- 172.21.15.164:0/1457406164 shutdown_connections 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 --2- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f00a46a0 0x7f63f00a4a80 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 --2- 172.21.15.164:0/1457406164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f006cb70 0x7f63f0134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 -- 172.21.15.164:0/1457406164 >> 172.21.15.164:0/1457406164 conn(0x7f63f00aa900 msgr2=0x7f63f00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 -- 172.21.15.164:0/1457406164 shutdown_connections 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.910+0000 7f63f54bd640 1 -- 172.21.15.164:0/1457406164 wait complete. 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient: build_initial_monmap 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient: monmap: 2024-09-10T16:24:24.922 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 1 Processor -- start 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 1 -- start start 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient: init 2024-09-10T16:24:24.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.911+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-a con 0x7f63f006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63f00a46a0 0x7f63f00a3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-b con 0x7f63f00a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 0x7f63f01c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): picked mon.noname-c con 0x7f63f0137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f63f01389c0 con 0x7f63f006cb70 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f63f0138840 con 0x7f63f00a46a0 2024-09-10T16:24:24.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f63f0140040 con 0x7f63f0137ca0 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63f54bd640 10 monclient(hunting): authenticate will time out at 1843.399055s 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63ef7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 msgr2=0x7f63f01c9f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63ef7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 0x7f63f01c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42904/0 (socket says 172.21.15.164:42904) 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 -- 172.21.15.164:0/534679606 learned_addr learned my addr 172.21.15.164:0/534679606 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): get_auth_request con 0x7f63f006cb70 auth_method 0 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63f00a46a0 0x7f63f00a3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): get_auth_request con 0x7f63f00a46a0 auth_method 0 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63e7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient(hunting): handle_auth_done global_id 164511 payload 824 2024-09-10T16:24:24.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 -- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 msgr2=0x7f63f01c9f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 0x7f63f01c9f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 -- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63f00a46a0 msgr2=0x7f63f00a3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63f00a46a0 0x7f63f00a3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient: _finish_hunting 0 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 monclient: found mon.noname-a 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.912+0000 7f63eeffd640 1 -- 172.21.15.164:0/534679606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63f006e720 con 0x7f63f006cb70 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63eeffd640 1 -- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 msgr2=0x7f63f0131780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63eeffd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63e000b550 tx=0x7f63e000ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e0003dd0 con 0x7f63f006cb70 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: dump: 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.928 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f63e0015070 con 0x7f63f006cb70 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 5 monclient: authenticate success, global_id 164511 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63e00052b0 con 0x7f63f006cb70 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 10 monclient: _renew_subs 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63f009ff40 con 0x7f63f006cb70 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:24.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63edffb640 4 set_mon_vals no callback set 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63e77fe640 10 monclient: dump: 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:24.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 10 monclient: _renew_subs 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.914+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63f009f200 con 0x7f63f006cb70 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f63e0005490 con 0x7f63f006cb70 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63f54bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63f54bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63f54bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63f0134f70 con 0x7f63f006cb70 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63e77fe640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 0x7f63d0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.915+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(622..622 src has 481..622) ==== 4755+0+0 (secure 0 0 0) 0x7f63e00526e0 con 0x7f63f006cb70 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.916+0000 7f63e7fff640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 0x7f63d0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:24.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.916+0000 7f63e7fff640 10 monclient: get_auth_request con 0x7f63d003e1c0 auth_method 0 2024-09-10T16:24:24.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.918+0000 7f63e7fff640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 0x7f63d0040680 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f63d8031000 tx=0x7f63d802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:24.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.920+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f63e00510f0 con 0x7f63f006cb70 2024-09-10T16:24:24.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.920+0000 7f63e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:24.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:24.920+0000 7f63e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:24.950 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: finishing write tid 1 to smithi16534841-49 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2796: done (2 left) 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2798: done (1 left) 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2799: delete oid 41 current snap is 297 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: finishing write tid 2 to smithi16534841-49 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: finishing write tid 3 to smithi16534841-49 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: finishing write tid 4 to smithi16534841-49 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: finishing write tid 5 to smithi16534841-49 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: oid 49 updating version 0 to 1554 2024-09-10T16:24:24.976 INFO:tasks.rados.rados.0.smithi165.stdout:2797: oid 49 updating version 1554 to 1555 2024-09-10T16:24:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:2797: oid 49 version 1555 is already newer than 1552 2024-09-10T16:24:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:2797: oid 49 version 1555 is already newer than 1553 2024-09-10T16:24:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1555 (ObjNum 979 snap 297 seq_num 979) dirty exists 2024-09-10T16:24:24.977 INFO:tasks.rados.rados.0.smithi165.stdout:2797: left oid 49 (ObjNum 979 snap 297 seq_num 979) 2024-09-10T16:24:24.978 INFO:tasks.rados.rados.0.smithi165.stdout:2797: done (1 left) 2024-09-10T16:24:24.978 INFO:tasks.rados.rados.0.smithi165.stdout:2799: done (0 left) 2024-09-10T16:24:24.978 INFO:tasks.rados.rados.0.smithi165.stdout:2800: write (excl) oid 11 current snap is 297 2024-09-10T16:24:24.978 INFO:tasks.rados.rados.0.smithi165.stdout:2800: seq_num 982 ranges {504474=504055,1714141=631006} 2024-09-10T16:24:24.983 INFO:tasks.rados.rados.0.smithi165.stdout:2800: writing smithi16534841-11 from 504474 to 1008529 tid 1 2024-09-10T16:24:24.991 INFO:tasks.rados.rados.0.smithi165.stdout:2800: writing smithi16534841-11 from 1714141 to 2345147 tid 2 2024-09-10T16:24:24.991 INFO:tasks.rados.rados.0.smithi165.stdout:2801: snap_create 2024-09-10T16:24:24.991 INFO:tasks.rados.rados.0.smithi165.stdout:2800: finishing write tid 1 to smithi16534841-11 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: finishing write tid 2 to smithi16534841-11 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: finishing write tid 3 to smithi16534841-11 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: finishing write tid 4 to smithi16534841-11 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: oid 11 updating version 0 to 1291 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: oid 11 updating version 1291 to 1292 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:2800: oid 11 version 1292 is already newer than 1290 2024-09-10T16:24:24.994 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1292 (ObjNum 982 snap 297 seq_num 982) dirty exists 2024-09-10T16:24:24.995 INFO:tasks.rados.rados.0.smithi165.stdout:2800: left oid 11 (ObjNum 982 snap 297 seq_num 982) 2024-09-10T16:24:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63f54bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:25.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63f54bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63f54bd640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f63f00a0320 con 0x7f63f006cb70 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63e77fe640 1 -- 172.21.15.164:0/534679606 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f63e00523e0 con 0x7f63f006cb70 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.160+0000 7f63e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:25.162 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 msgr2=0x7f63d0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 0x7f63d0040680 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f63d8031000 tx=0x7f63d802e000 comp rx=0 tx=0).stop 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 10 monclient: shutdown 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 msgr2=0x7f63f0131780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f63e000b550 tx=0x7f63e000ba10 comp rx=0 tx=0).stop 2024-09-10T16:24:25.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 shutdown_connections 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f63d003e1c0 0x7f63d0040680 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f63f0137ca0 0x7f63f01c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f63f00a46a0 0x7f63f00a3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 --2- 172.21.15.164:0/534679606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f63f006cb70 0x7f63f0131780 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 >> 172.21.15.164:0/534679606 conn(0x7f63f00aa900 msgr2=0x7f63f00a9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 shutdown_connections 2024-09-10T16:24:25.165 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:25.162+0000 7f63f54bd640 1 -- 172.21.15.164:0/534679606 wait complete. 2024-09-10T16:24:28.176 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.283+0000 7f2daefd3640 10 monclient: get_monmap_and_config 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.283+0000 7f2daefd3640 10 monclient: build_initial_monmap 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.283+0000 7f2daefd3640 10 monclient: monmap: 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:28.284281+0000 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:28.284281+0000 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:28.286 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:28.286 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 Processor -- start 2024-09-10T16:24:28.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 -- start start 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient: init 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient: _add_conns ranks=[2,0,3,1,4] 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 0x7f2da809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-b con 0x7f2da809d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2da809ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2da809e250 0x7f2da8134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-a con 0x7f2da809e250 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da80a0870 con 0x7f2da809ab50 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2da809d3e0 con 0x7f2da809e250 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:28.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2da8074c70 con 0x7f2da809d930 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2daefd3640 10 monclient(hunting): authenticate will time out at 1846.771694s 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dacd6f640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 msgr2=0x7f2da809dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dacd6f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 0x7f2da809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2da809e250 0x7f2da8134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2da7fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2da809ab50 0x7f2da8134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39708/0 (socket says 172.21.15.164:39708) 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f2da809e250 0x7f2da8134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42916/0 (socket says 172.21.15.164:42916) 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2da7fff640 1 -- 172.21.15.164:0/3125120481 learned_addr learned my addr 172.21.15.164:0/3125120481 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): get_auth_request con 0x7f2da809e250 auth_method 0 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:28.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.284+0000 7f2dad570640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 10 monclient(hunting): handle_auth_done global_id 164514 payload 274 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 1 -- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 msgr2=0x7f2da809dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 1 --2- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 0x7f2da809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 1 -- 172.21.15.164:0/3125120481 >> v1:172.21.15.164:6789/0 conn(0x7f2da809ab50 legacy=0x7f2da8134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 10 monclient: _finish_hunting 0 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 1 monclient: found mon.noname-a 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.285+0000 7f2dad570640 1 -- 172.21.15.164:0/3125120481 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da809ecb0 con 0x7f2da809e250 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.287+0000 7f2dad570640 1 -- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809e250 msgr2=0x7f2da8134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.287+0000 7f2dad570640 1 --2- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809e250 0x7f2da8134c90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009d90 tx=0x7f2da002a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58065c91b336d918 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 1 -- 172.21.15.164:0/3125120481 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0007a30 con 0x7f2da809e250 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: dump: 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.297 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: _finish_auth 0 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 1 -- 172.21.15.164:0/3125120481 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2da0007bd0 con 0x7f2da809e250 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 5 monclient: authenticate success, global_id 164514 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:28.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 4 set_mon_vals no callback set 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 1 -- 172.21.15.164:0/3125120481 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2da0031400 con 0x7f2da809e250 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2da77fe640 10 monclient: dump: 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.299 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 10 monclient: shutdown 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 -- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809e250 msgr2=0x7f2da8134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 --2- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809e250 0x7f2da8134c90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2da0009d90 tx=0x7f2da002a540 comp rx=0 tx=0).stop 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 -- 172.21.15.164:0/3125120481 shutdown_connections 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 --2- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809e250 0x7f2da8134c90 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 --2- 172.21.15.164:0/3125120481 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809d930 0x7f2da809dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 -- 172.21.15.164:0/3125120481 >> 172.21.15.164:0/3125120481 conn(0x7f2da80a3110 msgr2=0x7f2da80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 -- 172.21.15.164:0/3125120481 shutdown_connections 2024-09-10T16:24:28.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.288+0000 7f2daefd3640 1 -- 172.21.15.164:0/3125120481 wait complete. 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.289+0000 7f2daefd3640 10 monclient: build_initial_monmap 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.289+0000 7f2daefd3640 10 monclient: monmap: 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:28.301 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.289+0000 7f2daefd3640 1 Processor -- start 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.289+0000 7f2daefd3640 1 -- start start 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.289+0000 7f2daefd3640 10 monclient: init 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da809ab50 0x7f2da8073780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-b con 0x7f2da809ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-a con 0x7f2da809d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 0x7f2da81cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): picked mon.noname-c con 0x7f2da809e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2da809ef30 con 0x7f2da809d930 2024-09-10T16:24:28.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2da809e6e0 con 0x7f2da809ab50 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2da80a2940 con 0x7f2da809e250 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2daefd3640 10 monclient(hunting): authenticate will time out at 1846.777512s 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2dad570640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 msgr2=0x7f2da81cb960 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2dad570640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 0x7f2da81cb960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42922/0 (socket says 172.21.15.164:42922) 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 -- 172.21.15.164:0/1964705079 learned_addr learned my addr 172.21.15.164:0/1964705079 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): get_auth_request con 0x7f2da809d930 auth_method 0 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:28.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient(hunting): handle_auth_done global_id 164517 payload 824 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 -- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 msgr2=0x7f2da81cb960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 0x7f2da81cb960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 -- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da809ab50 msgr2=0x7f2da8073780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da809ab50 0x7f2da8073780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 monclient: found mon.noname-a 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:28.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.290+0000 7f2da7fff640 1 -- 172.21.15.164:0/1964705079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da8130000 con 0x7f2da809d930 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da7fff640 1 -- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 msgr2=0x7f2da81cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da7fff640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400b460 tx=0x7f2d9400b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94015020 con 0x7f2da809d930 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: dump: 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.305 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: _finish_auth 0 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d94011ac0 con 0x7f2da809d930 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 5 monclient: authenticate success, global_id 164517 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d940103e0 con 0x7f2da809d930 2024-09-10T16:24:28.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 10 monclient: _renew_subs 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da809d620 con 0x7f2da809d930 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da57fa640 10 monclient: dump: 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:28.307 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2da6ffd640 4 set_mon_vals no callback set 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 10 monclient: _renew_subs 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.293+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da812eae0 con 0x7f2da809d930 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2d94018020 con 0x7f2da809d930 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2daefd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2daefd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2daefd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da809dd10 con 0x7f2da809d930 2024-09-10T16:24:28.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2da57fa640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 0x7f2d8c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(622..622 src has 481..622) ==== 4755+0+0 (secure 0 0 0) 0x7f2d94052fe0 con 0x7f2da809d930 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2dacd6f640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 0x7f2d8c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.294+0000 7f2dacd6f640 10 monclient: get_auth_request con 0x7f2d8c03e1c0 auth_method 0 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.297+0000 7f2dacd6f640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 0x7f2d8c040680 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c003980 tx=0x7f2d9c006c90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.299+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2d9401f020 con 0x7f2da809d930 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.299+0000 7f2da57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:28.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.299+0000 7f2da57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.541+0000 7f2daefd3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.541+0000 7f2daefd3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.541+0000 7f2daefd3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.541+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2da806dac0 con 0x7f2da809d930 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.542+0000 7f2da57fa640 1 -- 172.21.15.164:0/1964705079 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f2d9400d040 con 0x7f2da809d930 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.542+0000 7f2da57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:28.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.542+0000 7f2da57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:28.544 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 msgr2=0x7f2d8c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:28.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 0x7f2d8c040680 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f2d9c003980 tx=0x7f2d9c006c90 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 10 monclient: shutdown 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 msgr2=0x7f2da81cb420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400b460 tx=0x7f2d9400b920 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 shutdown_connections 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2d8c03e1c0 0x7f2d8c040680 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2da809e250 0x7f2da81cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2da809d930 0x7f2da81cb420 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 --2- 172.21.15.164:0/1964705079 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2da809ab50 0x7f2da8073780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 >> 172.21.15.164:0/1964705079 conn(0x7f2da80a3110 msgr2=0x7f2da81bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 shutdown_connections 2024-09-10T16:24:28.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:28.544+0000 7f2daefd3640 1 -- 172.21.15.164:0/1964705079 wait complete. 2024-09-10T16:24:31.557 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.664+0000 7f5ab0ab8640 10 monclient: get_monmap_and_config 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.664+0000 7f5ab0ab8640 10 monclient: build_initial_monmap 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.665+0000 7f5ab0ab8640 10 monclient: monmap: 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:31.666 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:31.665886+0000 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:31.665886+0000 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:31.667 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.665+0000 7f5ab0ab8640 1 Processor -- start 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.665+0000 7f5ab0ab8640 1 -- start start 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.665+0000 7f5ab0ab8640 10 monclient: init 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f5aac074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5aac0a1090 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5aac0a14d0 0x7f5aac0a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-c con 0x7f5aac0a14d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:31.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac09ebc0 con 0x7f5aac0a1090 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5aac0698a0 con 0x7f5aac0a14d0 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5aac06bfb0 con 0x7f5aac074af0 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5ab0ab8640 10 monclient(hunting): authenticate will time out at 1850.153364s 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaa59c640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 msgr2=0x7f5aac137e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaa59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac137e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5aac0a14d0 0x7f5aac0a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5aac0a14d0 0x7f5aac0a35d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:47116/0 (socket says 172.21.15.164:47116) 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aa9d9b640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f5aac0a1090 0x7f5aac0a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:48842/0 (socket says 172.21.15.164:48842) 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 1 -- 172.21.15.164:0/4054310605 learned_addr learned my addr 172.21.15.164:0/4054310605 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): get_auth_request con 0x7f5aac0a14d0 auth_method 0 2024-09-10T16:24:31.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.666+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_done global_id 154537 payload 274 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 1 -- 172.21.15.164:0/4054310605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 msgr2=0x7f5aac137e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/4054310605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac137e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 1 -- 172.21.15.164:0/4054310605 >> v1:172.21.15.165:6789/0 conn(0x7f5aac0a1090 legacy=0x7f5aac0a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 1 monclient: found mon.noname-c 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:31.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:24:31.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.667+0000 7f5aaad9d640 1 -- 172.21.15.164:0/4054310605 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac0a1940 con 0x7f5aac0a14d0 2024-09-10T16:24:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.669+0000 7f5aaad9d640 1 -- 172.21.15.164:0/4054310605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a14d0 msgr2=0x7f5aac0a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.669+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/4054310605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a14d0 0x7f5aac0a35d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009870 tx=0x7f5aa402a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d64df68dd7b75b5c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 1 -- 172.21.15.164:0/4054310605 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4037070 con 0x7f5aac0a14d0 2024-09-10T16:24:31.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: dump: 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:31.679 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: _finish_auth 0 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: _check_auth_tickets 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 1 -- 172.21.15.164:0/4054310605 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa4032640 con 0x7f5aac0a14d0 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 5 monclient: authenticate success, global_id 154537 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 1 -- 172.21.15.164:0/4054310605 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4032a10 con 0x7f5aac0a14d0 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 4 set_mon_vals no callback set 2024-09-10T16:24:31.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5aa959a640 10 monclient: dump: 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 10 monclient: shutdown 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/4054310605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a14d0 msgr2=0x7f5aac0a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/4054310605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a14d0 0x7f5aac0a35d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f5aa4009870 tx=0x7f5aa402a0d0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/4054310605 shutdown_connections 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/4054310605 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a14d0 0x7f5aac0a35d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/4054310605 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/4054310605 >> 172.21.15.164:0/4054310605 conn(0x7f5aac0a3fb0 msgr2=0x7f5aac0a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/4054310605 shutdown_connections 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.670+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/4054310605 wait complete. 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.671+0000 7f5ab0ab8640 10 monclient: build_initial_monmap 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.671+0000 7f5ab0ab8640 10 monclient: monmap: 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:31.682 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 Processor -- start 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 -- start start 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient: init 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:24:31.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac09b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-c con 0x7f5aac074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 0x7f5aac09bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-b con 0x7f5aac0a1090 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): picked mon.noname-a con 0x7f5aac0a14d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac069370 con 0x7f5aac0a14d0 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5aac074ed0 con 0x7f5aac0a1090 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5aac0a3b10 con 0x7f5aac074af0 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:31.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5ab0ab8640 10 monclient(hunting): authenticate will time out at 1850.159592s 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaa59c640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 msgr2=0x7f5aac09b5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaa59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac09b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42932/0 (socket says 172.21.15.164:42932) 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 1 -- 172.21.15.164:0/804388387 learned_addr learned my addr 172.21.15.164:0/804388387 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): get_auth_request con 0x7f5aac0a14d0 auth_method 0 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:31.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aa9d9b640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 0x7f5aac09bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aa9d9b640 10 monclient(hunting): get_auth_request con 0x7f5aac0a1090 auth_method 0 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aa9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aa9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aa9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.672+0000 7f5aaad9d640 10 monclient(hunting): handle_auth_done global_id 164520 payload 824 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 -- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 msgr2=0x7f5aac09b5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac09b5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 -- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 msgr2=0x7f5aac09bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 0x7f5aac09bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 monclient: found mon.noname-a 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:31.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aaad9d640 1 -- 172.21.15.164:0/804388387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac12d140 con 0x7f5aac0a14d0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.673+0000 7f5aa9d9b640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 0x7f5aac09bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aaad9d640 1 -- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 msgr2=0x7f5aac09c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40090b0 tx=0x7f5aa40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aaad9d640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40090b0 tx=0x7f5aa40046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f5aa4041070 mon_map magic: 0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4041070 con 0x7f5aac0a14d0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: dump: 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 5 monclient: authenticate success, global_id 164520 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa403c880 con 0x7f5aac0a14d0 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 10 monclient: _renew_subs 2024-09-10T16:24:31.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac0a6f80 con 0x7f5aac0a14d0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 10 monclient: _renew_subs 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5aa3fff640 4 set_mon_vals no callback set 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.675+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aac12ce60 con 0x7f5aac0a14d0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5ab0ab8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5ab0ab8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5ab0ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aac12ca10 con 0x7f5aac0a14d0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4037040 con 0x7f5aac0a14d0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:31.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 10 monclient: dump: 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5aa4044070 con 0x7f5aac0a14d0 2024-09-10T16:24:31.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.676+0000 7f5aa2ffd640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 0x7f5a8c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.677+0000 7f5aaa59c640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 0x7f5a8c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.677+0000 7f5aaa59c640 10 monclient: get_auth_request con 0x7f5a8c03e1c0 auth_method 0 2024-09-10T16:24:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.679+0000 7f5aaa59c640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 0x7f5a8c040680 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c003d00 tx=0x7f5a9c009270 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:31.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:31.681+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(622..622 src has 481..622) ==== 4755+0+0 (secure 0 0 0) 0x7f5aa4073640 con 0x7f5aac0a14d0 2024-09-10T16:24:32.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.298+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5aac12ca10 con 0x7f5aac0a14d0 2024-09-10T16:24:32.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.298+0000 7f5aa2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:32.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.298+0000 7f5aa2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.543+0000 7f5ab0ab8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.543+0000 7f5ab0ab8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.543+0000 7f5ab0ab8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:32.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.543+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5aac137e70 con 0x7f5aac0a14d0 2024-09-10T16:24:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.543+0000 7f5aa2ffd640 1 -- 172.21.15.164:0/804388387 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f5aa4044c60 con 0x7f5aac0a14d0 2024-09-10T16:24:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.544+0000 7f5aa2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:32.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.544+0000 7f5aa2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:32.546 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":176,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:32.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 msgr2=0x7f5a8c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:32.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 0x7f5a8c040680 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f5a9c003d00 tx=0x7f5a9c009270 comp rx=0 tx=0).stop 2024-09-10T16:24:32.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 10 monclient: shutdown 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 msgr2=0x7f5aac09c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40090b0 tx=0x7f5aa40046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.545+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 shutdown_connections 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5a8c03e1c0 0x7f5a8c040680 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5aac0a14d0 0x7f5aac09c070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5aac0a1090 0x7f5aac09bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 --2- 172.21.15.164:0/804388387 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5aac074af0 0x7f5aac09b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:32.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 >> 172.21.15.164:0/804388387 conn(0x7f5aac0a3fb0 msgr2=0x7f5aac098230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:32.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 shutdown_connections 2024-09-10T16:24:32.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:32.546+0000 7f5ab0ab8640 1 -- 172.21.15.164:0/804388387 wait complete. 2024-09-10T16:24:35.558 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:24:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.666+0000 7f8783dbc640 10 monclient: get_monmap_and_config 2024-09-10T16:24:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.666+0000 7f8783dbc640 10 monclient: build_initial_monmap 2024-09-10T16:24:35.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.666+0000 7f8783dbc640 10 monclient: monmap: 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:35.667218+0000 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:35.667218+0000 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:35.668 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.666+0000 7f8783dbc640 1 Processor -- start 2024-09-10T16:24:35.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 -- start start 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient: init 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f877c09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f877c09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f877c09f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f877c09a430 con 0x7f877c09f4c0 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f877c0a24e0 con 0x7f877c09d160 2024-09-10T16:24:35.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f877c0a2bf0 con 0x7f877c09c180 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8783dbc640 10 monclient(hunting): authenticate will time out at 1854.154732s 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8781b58640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 msgr2=0x7f877c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.667+0000 7f8781b58640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8782359640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f877c09f4c0 0x7f877c13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:42376/0 (socket says 172.21.15.164:42376) 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 learned_addr learned my addr 172.21.15.164:0/1581884380 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 10 monclient(hunting): get_auth_request con 0x7f877c09d160 auth_method 0 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:35.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:24:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.668+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:35.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.670+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 >> v1:172.21.15.165:6789/0 conn(0x7f877c09f4c0 legacy=0x7f877c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:35.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.868+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 msgr2=0x7f877c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:35.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.868+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.869+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.869+0000 7f8781357640 10 monclient(hunting): get_auth_request con 0x7f877c09d160 auth_method 0 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.869+0000 7f8781357640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.869+0000 7f8781357640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.869+0000 7f8781357640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.870+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:35.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.870+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:35.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.870+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:35.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:35.870+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:24:36.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.269+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 msgr2=0x7f877c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:36.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.269+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:24:36.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:36.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 10 monclient(hunting): get_auth_request con 0x7f877c09d160 auth_method 0 2024-09-10T16:24:36.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:36.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:36.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:36.272+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:24:37.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.070+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 msgr2=0x7f877c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:37.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.070+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:24:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 10 monclient(hunting): get_auth_request con 0x7f877c09d160 auth_method 0 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:37.073+0000 7f8781357640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): tick 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 monclient(hunting): continuing hunt 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 msgr2=0x7f877c09ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f877c09c180 0x7f877c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 msgr2=0x7f877c09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:38.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.165:3300/0 conn(0x7f877c09d160 0x7f877c09ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 -- 172.21.15.164:0/1581884380 >> v1:172.21.15.165:6789/0 conn(0x7f877c09f4c0 legacy=0x7f877c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient: _add_conns ranks=[3,0] 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): picked mon.noname-a con 0x7f8768003760 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8768008440 addr v1:172.21.15.164:6789/0 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 -- 172.21.15.164:0/1581884380 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f877c09c560 con 0x7f8768008440 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 1 -- 172.21.15.164:0/1581884380 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f877c0a24e0 con 0x7f8768003760 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f877b7fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 10 monclient(hunting): get_auth_request con 0x7f8768003760 auth_method 0 2024-09-10T16:24:38.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.668+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:38.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.670+0000 7f8781357640 1 -- 172.21.15.164:0/1581884380 >> v1:172.21.15.164:6789/0 conn(0x7f8768008440 legacy=0x7f876800a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:38.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 10 monclient(hunting): get_auth_request con 0x7f8768003760 auth_method 0 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:38.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:38.869+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:24:39.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 10 monclient(hunting): get_auth_request con 0x7f8768003760 auth_method 0 2024-09-10T16:24:39.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:39.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:24:39.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:24:39.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:24:39.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.270+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:24:39.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2320669985 0 0) 0x7f877c09c560 con 0x7f8768008440 2024-09-10T16:24:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 10 monclient(hunting): _negotiate 2024-09-10T16:24:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 10 monclient(hunting): my global_id is 164523 2024-09-10T16:24:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.538+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8764003520 con 0x7f8768008440 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2705564584 0 0) 0x7f8764003520 con 0x7f8768008440 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 msgr2=0x7f8768005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 --2- 172.21.15.164:0/1581884380 >> v2:172.21.15.164:3300/0 conn(0x7f8768003760 0x7f8768005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _finish_hunting 0 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:24:39.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f876800d1d0 con 0x7f8768008440 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _finish_auth 0 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _check_auth_tickets 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f876c009d10 con 0x7f8768008440 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8783dbc640 5 monclient: authenticate success, global_id 164523 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8783dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:39.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: dump: 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 >> v1:172.21.15.164:6789/0 conn(0x7f8768008440 legacy=0x7f876800a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 reap_dead start 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 0x7f8764003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): picked mon.c con 0x7f8764003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:39.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8764004800 0x7f8764004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): picked mon.a con 0x7f8764004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 0x7f876400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): picked mon.b con 0x7f87640053c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f877c0a24e0 con 0x7f8764004800 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f876400c250 con 0x7f87640053c0 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f876400c450 con 0x7f8764003a20 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:39.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8780b56640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8783dbc640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8781b58640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 msgr2=0x7f8764003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8781b58640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 0x7f8764003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 0x7f876400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): get_auth_request con 0x7f87640053c0 auth_method 0 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:24:39.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.539+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 10 monclient(hunting): handle_auth_done global_id 164523 payload 293 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 msgr2=0x7f8764003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 0x7f8764003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8764004800 msgr2=0x7f8764004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8764004800 0x7f8764004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 10 monclient: _finish_hunting 0 2024-09-10T16:24:39.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 monclient: found mon.b 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8764003520 con 0x7f87640053c0 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 10 monclient: _finish_auth 0 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 10 monclient: _check_auth_tickets 2024-09-10T16:24:39.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.540+0000 7f8782359640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:39.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8782359640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 msgr2=0x7f876400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8782359640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 0x7f876400b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f877c09f8a0 tx=0x7f8764005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f5d026a89208a4a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774001620 con 0x7f87640053c0 2024-09-10T16:24:39.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: dump: 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f877402e070 con 0x7f87640053c0 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8783dbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8783dbc640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 1 -- 172.21.15.164:0/1581884380 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774005db0 con 0x7f87640053c0 2024-09-10T16:24:39.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8783dbc640 4 set_mon_vals no callback set 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.543+0000 7f8780b56640 10 monclient: dump: 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 10 monclient: shutdown 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 -- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 msgr2=0x7f876400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 0x7f876400b9c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f877c09f8a0 tx=0x7f8764005830 comp rx=0 tx=0).stop 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 -- 172.21.15.164:0/1581884380 shutdown_connections 2024-09-10T16:24:39.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87640053c0 0x7f876400b9c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8764004800 0x7f8764004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.544+0000 7f8783dbc640 1 --2- 172.21.15.164:0/1581884380 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764003a20 0x7f8764003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.545+0000 7f8783dbc640 1 -- 172.21.15.164:0/1581884380 >> 172.21.15.164:0/1581884380 conn(0x7f877c0a3110 msgr2=0x7f877c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.545+0000 7f8783dbc640 1 -- 172.21.15.164:0/1581884380 shutdown_connections 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.545+0000 7f8783dbc640 1 -- 172.21.15.164:0/1581884380 wait complete. 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.546+0000 7f8783dbc640 10 monclient: build_initial_monmap 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.546+0000 7f8783dbc640 10 monclient: monmap: 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:39.563 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.546+0000 7f8783dbc640 1 Processor -- start 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.546+0000 7f8783dbc640 1 -- start start 2024-09-10T16:24:39.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.546+0000 7f8783dbc640 10 monclient: init 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8764003a20 0x7f877c1ccba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-b con 0x7f8764003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 0x7f877c099330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-c con 0x7f8764004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): picked mon.noname-a con 0x7f87640053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f877c074f90 con 0x7f87640053c0 2024-09-10T16:24:39.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f877c074e10 con 0x7f8764003a20 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f877c09d930 con 0x7f8764004800 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8783dbc640 10 monclient(hunting): authenticate will time out at 1858.034481s 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781357640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 msgr2=0x7f877c099330 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781357640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 0x7f877c099330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49312/0 (socket says 172.21.15.164:49312) 2024-09-10T16:24:39.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 -- 172.21.15.164:0/944378276 learned_addr learned my addr 172.21.15.164:0/944378276 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): get_auth_request con 0x7f87640053c0 auth_method 0 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8764003a20 0x7f877c1ccba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): get_auth_request con 0x7f8764003a20 auth_method 0 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8781b58640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient(hunting): handle_auth_done global_id 164526 payload 824 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 -- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 msgr2=0x7f877c099330 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 0x7f877c099330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 -- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8764003a20 msgr2=0x7f877c1ccba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8764003a20 0x7f877c1ccba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient: _finish_hunting 0 2024-09-10T16:24:39.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 monclient: found mon.noname-a 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.547+0000 7f8782359640 1 -- 172.21.15.164:0/944378276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f877c09f050 con 0x7f87640053c0 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8782359640 1 -- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 msgr2=0x7f877c092da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8782359640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8774000c00 tx=0x7f8774001460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774001620 con 0x7f87640053c0 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: dump: 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.569 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87740049e0 con 0x7f87640053c0 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 5 monclient: authenticate success, global_id 164526 2024-09-10T16:24:39.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 10 monclient: _renew_subs 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f877c09d620 con 0x7f87640053c0 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8774004dd0 con 0x7f87640053c0 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877a7fc640 10 monclient: dump: 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 10 monclient: _renew_subs 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f877c096320 con 0x7f87640053c0 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.550+0000 7f877bfff640 4 set_mon_vals no callback set 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f8783dbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8774005030 con 0x7f87640053c0 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f8783dbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f8783dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f877c09dd10 con 0x7f87640053c0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.551+0000 7f877a7fc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 0x7f8768041010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.552+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(623..623 src has 481..623) ==== 4755+0+0 (secure 0 0 0) 0x7f8774072440 con 0x7f87640053c0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.552+0000 7f8781b58640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 0x7f8768041010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.552+0000 7f8781b58640 10 monclient: get_auth_request con 0x7f876803eb50 auth_method 0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.554+0000 7f8781b58640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 0x7f8768041010 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8770003ac0 tx=0x7f8770005040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.556+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8774070e50 con 0x7f87640053c0 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.556+0000 7f877a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:39.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.556+0000 7f877a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f8783dbc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f8783dbc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f8783dbc640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f877c0953c0 con 0x7f87640053c0 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f877a7fc640 1 -- 172.21.15.164:0/944378276 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f8774072130 con 0x7f87640053c0 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f877a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:24:39.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.802+0000 7f877a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:39.805 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":180,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:24:39.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 msgr2=0x7f8768041010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 0x7f8768041010 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f8770003ac0 tx=0x7f8770005040 comp rx=0 tx=0).stop 2024-09-10T16:24:39.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 10 monclient: shutdown 2024-09-10T16:24:39.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 msgr2=0x7f877c092da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8774000c00 tx=0x7f8774001460 comp rx=0 tx=0).stop 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.804+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 shutdown_connections 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f876803eb50 0x7f8768041010 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87640053c0 0x7f877c092da0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8764004800 0x7f877c099330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 --2- 172.21.15.164:0/944378276 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8764003a20 0x7f877c1ccba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 >> 172.21.15.164:0/944378276 conn(0x7f877c0a3110 msgr2=0x7f877c09df20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 shutdown_connections 2024-09-10T16:24:39.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.805+0000 7f8783dbc640 1 -- 172.21.15.164:0/944378276 wait complete. 2024-09-10T16:24:39.817 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:24:39.817 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:24:39.818 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:24:39.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.916+0000 7f5cca9f3640 10 monclient: get_monmap_and_config 2024-09-10T16:24:39.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.916+0000 7f5cca9f3640 10 monclient: build_initial_monmap 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.916+0000 7f5cca9f3640 10 monclient: monmap: 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:39.917206+0000 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:39.917206+0000 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:39.918 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:39.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:39.919 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:39.919 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:39.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 Processor -- start 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 -- start start 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient: init 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5cc409c180 0x7f5cc409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f5cc409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5cc409d160 0x7f5cc409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-c con 0x7f5cc409d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5cc409f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cc409a430 con 0x7f5cc409f4c0 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40a24e0 con 0x7f5cc409c180 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc40a2bf0 con 0x7f5cc409d160 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cca9f3640 10 monclient(hunting): authenticate will time out at 1858.404607s 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5cc409c180 0x7f5cc409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5cc409c180 0x7f5cc409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49320/0 (socket says 172.21.15.164:49320) 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3886896525 learned_addr learned my addr 172.21.15.164:0/3886896525 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:39.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): get_auth_request con 0x7f5cc409c180 auth_method 0 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_done global_id 164529 payload 274 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3886896525 >> v2:172.21.15.165:3300/0 conn(0x7f5cc409d160 msgr2=0x7f5cc409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3886896525 >> v2:172.21.15.165:3300/0 conn(0x7f5cc409d160 0x7f5cc409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3886896525 >> v1:172.21.15.165:6789/0 conn(0x7f5cc409f4c0 legacy=0x7f5cc413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:39.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 monclient: found mon.noname-a 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.917+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3886896525 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc409c560 con 0x7f5cc409c180 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3886896525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409c180 msgr2=0x7f5cc409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3886896525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409c180 0x7f5cc409ea40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400bb10 tx=0x7f5cb402c7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d1cf400bb3f01f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 1 -- 172.21.15.164:0/3886896525 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4031a00 con 0x7f5cc409c180 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:39.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: dump: 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:24:39.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cca9f3640 5 monclient: authenticate success, global_id 164529 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 1 -- 172.21.15.164:0/3886896525 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb4031ba0 con 0x7f5cc409c180 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cca9f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cca9f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cca9f3640 4 set_mon_vals no callback set 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 1 -- 172.21.15.164:0/3886896525 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4033400 con 0x7f5cc409c180 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.920+0000 7f5cc2ffd640 10 monclient: dump: 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.931 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 10 monclient: shutdown 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3886896525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409c180 msgr2=0x7f5cc409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3886896525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409c180 0x7f5cc409ea40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400bb10 tx=0x7f5cb402c7a0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3886896525 shutdown_connections 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3886896525 >> v2:172.21.15.165:3300/0 conn(0x7f5cc409d160 0x7f5cc409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3886896525 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409c180 0x7f5cc409ea40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3886896525 >> 172.21.15.164:0/3886896525 conn(0x7f5cc40a3110 msgr2=0x7f5cc40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3886896525 shutdown_connections 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3886896525 wait complete. 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 10 monclient: build_initial_monmap 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.921+0000 7f5cca9f3640 10 monclient: monmap: 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:39.933 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 Processor -- start 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 -- start start 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient: init 2024-09-10T16:24:39.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:39.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5cc409c180 0x7f5cc41c9030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-c con 0x7f5cc409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-a con 0x7f5cc409d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cc409f4c0 0x7f5cc41bad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): picked mon.noname-b con 0x7f5cc409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4074f90 con 0x7f5cc409d160 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4074e10 con 0x7f5cc409f4c0 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc409d930 con 0x7f5cc409c180 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.922+0000 7f5cca9f3640 10 monclient(hunting): authenticate will time out at 1858.409993s 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc3fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5cc409c180 msgr2=0x7f5cc41c9030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5cc409c180 0x7f5cc41c9030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49334/0 (socket says 172.21.15.164:49334) 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 -- 172.21.15.164:0/3816723109 learned_addr learned my addr 172.21.15.164:0/3816723109 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): get_auth_request con 0x7f5cc409d160 auth_method 0 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc8f90640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cc409f4c0 0x7f5cc41bad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc8f90640 10 monclient(hunting): get_auth_request con 0x7f5cc409f4c0 auth_method 0 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc8f90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc8f90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc8f90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient(hunting): handle_auth_done global_id 164532 payload 824 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5cc409c180 msgr2=0x7f5cc41c9030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5cc409c180 0x7f5cc41c9030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cc409f4c0 msgr2=0x7f5cc41bad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5cc409f4c0 0x7f5cc41bad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:24:39.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 monclient: found mon.noname-a 2024-09-10T16:24:39.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:39.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.923+0000 7f5cc37fe640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40a46f0 con 0x7f5cc409d160 2024-09-10T16:24:39.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc37fe640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 msgr2=0x7f5cc41ba7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc37fe640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b450 tx=0x7f5cb800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8015020 con 0x7f5cc409d160 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: dump: 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.945 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc37fe640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b450 tx=0x7f5cb800b910 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f5cb8011aa0 config(0 keys) 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 5 monclient: authenticate success, global_id 164532 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 10 monclient: _renew_subs 2024-09-10T16:24:39.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc40a4230 con 0x7f5cc409d160 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb8011aa0 con 0x7f5cc409d160 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 10 monclient: _renew_subs 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc41b3f70 con 0x7f5cc409d160 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.925+0000 7f5cc27fc640 4 set_mon_vals no callback set 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb80103e0 con 0x7f5cc409d160 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:39.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 10 monclient: dump: 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.948 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5cb8018020 con 0x7f5cc409d160 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cca9f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 0x7f5cc41b0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc409dcc0 con 0x7f5cc41b0640 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 0x7f5cc41b0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient: get_auth_request con 0x7f5cc41b0640 auth_method 0 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc0ff9640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cb003e1f0 0x7f5cb00406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.926+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.927+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_done global_id 164535 payload 824 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.927+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cc409eda0 con 0x7f5cc41b0640 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.927+0000 7f5cc8f90640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cb003e1f0 0x7f5cb00406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.927+0000 7f5cc8f90640 10 monclient: get_auth_request con 0x7f5cb003e1f0 auth_method 0 2024-09-10T16:24:39.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 msgr2=0x7f5cc41b0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 0x7f5cc41b0a20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4036000 tx=0x7f5cb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4042070 con 0x7f5cc41b0640 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 10 monclient: dump: 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.951 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5cb4034e30 con 0x7f5cc41b0640 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:39.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 msgr2=0x7f5cc41b0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 0x7f5cc41b0a20 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4036000 tx=0x7f5cb40046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 msgr2=0x7f5cc41b0a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc0ff9640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0640 0x7f5cc41b0a20 secure :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4036000 tx=0x7f5cb40046c0 comp rx=0 tx=0).stop 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.929+0000 7f5cc8f90640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cb003e1f0 0x7f5cb00406b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc007fd0 tx=0x7f5cbc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cca9f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 0x7f5cc40707e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cca9f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4077260 con 0x7f5cc41b0f60 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 0x7f5cc40707e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient: get_auth_request con 0x7f5cc41b0f60 auth_method 0 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:39.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.935+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.936+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.936+0000 7f5cc3fff640 10 monclient(hunting): handle_auth_done global_id 164538 payload 824 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.936+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3816723109 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5cc41b1370 con 0x7f5cc41b0f60 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc3fff640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 msgr2=0x7f5cc40707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc3fff640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 0x7f5cc40707e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000c00 tx=0x7f5cb4032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4047880 con 0x7f5cc41b0f60 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 10 monclient: dump: 2024-09-10T16:24:39.954 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:39.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f5cb4047860 con 0x7f5cc41b0f60 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 msgr2=0x7f5cc40707e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 0x7f5cc40707e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000c00 tx=0x7f5cb4032000 comp rx=0 tx=0).stop 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 msgr2=0x7f5cc40707e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:39.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.938+0000 7f5cc0ff9640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc41b0f60 0x7f5cc40707e0 secure :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4000c00 tx=0x7f5cb4032000 comp rx=0 tx=0).stop 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 180, 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 250640, 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:39.957 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:39.958 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:39.959 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:39.960 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:39.961 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:39.962 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:39.963 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:39.964 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.965 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:39.966 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:39.967 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:39.968 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:39.969 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:39.970 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cb003e1f0 msgr2=0x7f5cb00406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cb003e1f0 0x7f5cb00406b0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc007fd0 tx=0x7f5cbc009040 comp rx=0 tx=0).stop 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 10 monclient: shutdown 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 msgr2=0x7f5cc41ba7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:39.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 --2- 172.21.15.164:0/3816723109 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5cc409d160 0x7f5cc41ba7d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800b450 tx=0x7f5cb800b910 comp rx=0 tx=0).stop 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cc8f90640 1 -- 172.21.15.164:0/3816723109 reap_dead start 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cc0ff9640 1 -- 172.21.15.164:0/3816723109 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(623..623 src has 481..623) ==== 4755+0+0 (unknown 0 0 0) 0x7f5cb8052f30 con 0x7f5cc409d160 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cc8f90640 1 -- 172.21.15.164:0/3816723109 reap_dead start 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 shutdown_connections 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 >> 172.21.15.164:0/3816723109 conn(0x7f5cc40a3110 msgr2=0x7f5cc409ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 shutdown_connections 2024-09-10T16:24:39.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:39.939+0000 7f5cca9f3640 1 -- 172.21.15.164:0/3816723109 wait complete. 2024-09-10T16:24:39.972 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:24:39.972 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:24:39.973 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.071+0000 7f0c91910640 10 monclient: get_monmap_and_config 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.071+0000 7f0c91910640 10 monclient: build_initial_monmap 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.071+0000 7f0c91910640 10 monclient: monmap: 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.073 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:24:40.072813+0000 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:24:40.072813+0000 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:24:40.074 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.072+0000 7f0c91910640 1 Processor -- start 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.072+0000 7f0c91910640 1 -- start start 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.072+0000 7f0c91910640 10 monclient: init 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c8c09d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c8c09ab50 0x7f0c8c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-a con 0x7f0c8c09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0c8c09e420 0x7f0c8c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8c09e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c8c0a0870 con 0x7f0c8c09d930 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0c8c09d3e0 con 0x7f0c8c09ab50 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0c8c074c70 con 0x7f0c8c09e420 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c91910640 10 monclient(hunting): authenticate will time out at 1858.560142s 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c8c09ab50 0x7f0c8c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c8c09ab50 0x7f0c8c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49350/0 (socket says 172.21.15.164:49350) 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 -- 172.21.15.164:0/3469282095 learned_addr learned my addr 172.21.15.164:0/3469282095 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): get_auth_request con 0x7f0c8c09ab50 auth_method 0 2024-09-10T16:24:40.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 1 --2- 172.21.15.164:0/3469282095 >> v2:172.21.15.165:3300/0 conn(0x7f0c8c09e420 0x7f0c8c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c8c09e420 auth_method 0 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:40.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient(hunting): handle_auth_done global_id 164541 payload 274 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 -- 172.21.15.164:0/3469282095 >> v2:172.21.15.165:3300/0 conn(0x7f0c8c09e420 msgr2=0x7f0c8c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 --2- 172.21.15.164:0/3469282095 >> v2:172.21.15.165:3300/0 conn(0x7f0c8c09e420 0x7f0c8c13bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 -- 172.21.15.164:0/3469282095 >> v1:172.21.15.165:6789/0 conn(0x7f0c8c09d930 legacy=0x7f0c8c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 monclient: found mon.noname-a 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:40.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.073+0000 7f0c8a7fc640 1 -- 172.21.15.164:0/3469282095 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c09ecb0 con 0x7f0c8c09ab50 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c8a7fc640 1 -- 172.21.15.164:0/3469282095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09ab50 msgr2=0x7f0c8c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c8a7fc640 1 --2- 172.21.15.164:0/3469282095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09ab50 0x7f0c8c1397c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c09af60 tx=0x7f0c8002c770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=652f95c619674d45 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 1 -- 172.21.15.164:0/3469282095 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80038070 con 0x7f0c8c09ab50 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: dump: 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: _finish_auth 0 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 1 -- 172.21.15.164:0/3469282095 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c80031920 con 0x7f0c8c09ab50 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 5 monclient: authenticate success, global_id 164541 2024-09-10T16:24:40.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 10 monclient: get_monmap_and_config success 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 1 -- 172.21.15.164:0/3469282095 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80031cf0 con 0x7f0c8c09ab50 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 4 set_mon_vals no callback set 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c89ffb640 10 monclient: dump: 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.091 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 10 monclient: shutdown 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 -- 172.21.15.164:0/3469282095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09ab50 msgr2=0x7f0c8c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 --2- 172.21.15.164:0/3469282095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09ab50 0x7f0c8c1397c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c09af60 tx=0x7f0c8002c770 comp rx=0 tx=0).stop 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 -- 172.21.15.164:0/3469282095 shutdown_connections 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 --2- 172.21.15.164:0/3469282095 >> v2:172.21.15.165:3300/0 conn(0x7f0c8c09e420 0x7f0c8c13bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 --2- 172.21.15.164:0/3469282095 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09ab50 0x7f0c8c1397c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.076+0000 7f0c91910640 1 -- 172.21.15.164:0/3469282095 >> 172.21.15.164:0/3469282095 conn(0x7f0c8c0a3110 msgr2=0x7f0c8c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.077+0000 7f0c91910640 1 -- 172.21.15.164:0/3469282095 shutdown_connections 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.077+0000 7f0c91910640 1 -- 172.21.15.164:0/3469282095 wait complete. 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.077+0000 7f0c91910640 10 monclient: build_initial_monmap 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.077+0000 7f0c91910640 10 monclient: monmap: 2024-09-10T16:24:40.093 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:24:40.094 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.077+0000 7f0c91910640 1 Processor -- start 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 -- start start 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient: init 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient: _reopen_session rank -1 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c8c09ab50 0x7f0c8c073780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-c con 0x7f0c8c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c09d930 0x7f0c8c134a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-b con 0x7f0c8c09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 0x7f0c8c134fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): picked mon.noname-a con 0x7f0c8c09e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c0a2940 con 0x7f0c8c09e420 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c09ef30 con 0x7f0c8c09d930 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0c8c0a2ac0 con 0x7f0c8c09ab50 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): _renew_subs 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c91910640 10 monclient(hunting): authenticate will time out at 1858.565529s 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8affd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c8c09ab50 msgr2=0x7f0c8c073780 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c8c09ab50 0x7f0c8c073780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 0x7f0c8c134fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 0x7f0c8c134fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49354/0 (socket says 172.21.15.164:49354) 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 learned_addr learned my addr 172.21.15.164:0/1956612963 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): get_auth_request con 0x7f0c8c09e420 auth_method 0 2024-09-10T16:24:40.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient(hunting): handle_auth_done global_id 164544 payload 824 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c8c09ab50 msgr2=0x7f0c8c073780 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c8c09ab50 0x7f0c8c073780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8a7fc640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c09d930 0x7f0c8c134a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c09d930 msgr2=0x7f0c8c134a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c09d930 0x7f0c8c134a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:24:40.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 monclient: found mon.noname-a 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.078+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c8c132550 con 0x7f0c8c09e420 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 msgr2=0x7f0c8c134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c8b7fe640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 0x7f0c8c134fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400e930 tx=0x7f0c8400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84017070 con 0x7f0c8c09e420 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: dump: 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.099 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: _finish_auth 0 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: _check_auth_tickets 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 5 monclient: authenticate success, global_id 164544 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8400cb90 con 0x7f0c8c09e420 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:24:40.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 10 monclient: _renew_subs 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c84013470 con 0x7f0c8c09e420 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c8c09d620 con 0x7f0c8c09e420 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c73fff640 10 monclient: dump: 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c897fa640 4 set_mon_vals no callback set 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 10 monclient: _renew_subs 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.081+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c8c132270 con 0x7f0c8c09e420 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c91910640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c91910640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 0x7f0c8c12fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c131ef0 con 0x7f0c8c12f900 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0c8401a070 con 0x7f0c8c09e420 2024-09-10T16:24:40.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 0x7f0c8c12fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient: get_auth_request con 0x7f0c8c12f900 auth_method 0 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c73fff640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7803e210 0x7f0c780406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(623..623 src has 481..623) ==== 4755+0+0 (secure 0 0 0) 0x7f0c84052de0 con 0x7f0c8c09e420 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8a7fc640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7803e210 0x7f0c780406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8a7fc640 10 monclient: get_auth_request con 0x7f0c7803e210 auth_method 0 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 10 monclient(hunting): handle_auth_done global_id 154552 payload 824 2024-09-10T16:24:40.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.082+0000 7f0c8affd640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c8c09dd10 con 0x7f0c8c12f900 2024-09-10T16:24:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c8affd640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 msgr2=0x7f0c8c12fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c8affd640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 0x7f0c8c12fce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c6f0 tx=0x7f0c7c00cbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c017070 con 0x7f0c8c12f900 2024-09-10T16:24:40.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c8a7fc640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7803e210 0x7f0c780406d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c06d500 tx=0x7f0c80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 10 monclient: dump: 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.107 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c7c00a8b0 con 0x7f0c8c12f900 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 msgr2=0x7f0c8c12fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 0x7f0c8c12fce0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c6f0 tx=0x7f0c7c00cbb0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 msgr2=0x7f0c8c12fce0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.085+0000 7f0c73fff640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c12f900 0x7f0c8c12fce0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.091+0000 7f0c91910640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.091+0000 7f0c91910640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 0x7f0c8c06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.091+0000 7f0c91910640 10 monclient(hunting): start opening mon connection 2024-09-10T16:24:40.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.091+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c8c0a5e70 con 0x7f0c8c130220 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.091+0000 7f0c8affd640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 0x7f0c8c06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient: get_auth_request con 0x7f0c8c130220 auth_method 0 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 10 monclient(hunting): handle_auth_done global_id 154555 payload 824 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.092+0000 7f0c8affd640 1 -- 172.21.15.164:0/1956612963 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c8c06dac0 con 0x7f0c8c130220 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c8affd640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 msgr2=0x7f0c8c06f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c8affd640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 0x7f0c8c06f7e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c6f0 tx=0x7f0c7c003b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:24:40.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c7c017040 con 0x7f0c8c130220 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c73fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.094+0000 7f0c73fff640 10 monclient: dump: 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:24:40.110 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4877 (secure 0 0 0) 0x7f0c7c007070 con 0x7f0c8c130220 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 msgr2=0x7f0c8c06f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 0x7f0c8c06f7e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0c7c00c6f0 tx=0x7f0c7c003b60 comp rx=0 tx=0).stop 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 msgr2=0x7f0c8c06f7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c73fff640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c8c130220 0x7f0c8c06f7e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:24:40.111 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 180, 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 298645, 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:24:40.112 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:40.114 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:24:40.115 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:24:40.116 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.117 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:40.118 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:40.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:24:40.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:24:40.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:24:40.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:24:40.125 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7803e210 msgr2=0x7f0c780406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7803e210 0x7f0c780406d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c06d500 tx=0x7f0c80008040 comp rx=0 tx=0).stop 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 10 monclient: shutdown 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 msgr2=0x7f0c8c134fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 --2- 172.21.15.164:0/1956612963 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c8c09e420 0x7f0c8c134fd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0c8400e930 tx=0x7f0c8400edf0 comp rx=0 tx=0).stop 2024-09-10T16:24:40.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 reap_dead start 2024-09-10T16:24:40.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c8b7fe640 1 -- 172.21.15.164:0/1956612963 reap_dead start 2024-09-10T16:24:40.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 shutdown_connections 2024-09-10T16:24:40.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 >> 172.21.15.164:0/1956612963 conn(0x7f0c8c0a3110 msgr2=0x7f0c8c1bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:24:40.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 shutdown_connections 2024-09-10T16:24:40.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:24:40.095+0000 7f0c91910640 1 -- 172.21.15.164:0/1956612963 wait complete. 2024-09-10T16:24:40.128 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:24:40.129 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:24:40.543 INFO:tasks.rados.rados.0.smithi165.stdout:2800: done (1 left) 2024-09-10T16:24:40.543 INFO:tasks.rados.rados.0.smithi165.stdout:2801: done (0 left) 2024-09-10T16:24:40.544 INFO:tasks.rados.rados.0.smithi165.stdout:2802: rollback oid 42 current snap is 298 2024-09-10T16:24:40.544 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 290 2024-09-10T16:24:40.544 INFO:tasks.rados.rados.0.smithi165.stdout:2803: write oid 4 current snap is 298 2024-09-10T16:24:40.544 INFO:tasks.rados.rados.0.smithi165.stdout:2803: seq_num 983 ranges {521281=579304,1656270=774664} 2024-09-10T16:24:40.549 INFO:tasks.rados.rados.0.smithi165.stdout:2803: writing smithi16534841-4 from 521281 to 1100585 tid 1 2024-09-10T16:24:40.557 INFO:tasks.rados.rados.0.smithi165.stdout:2803: writing smithi16534841-4 from 1656270 to 2430934 tid 2 2024-09-10T16:24:40.557 INFO:tasks.rados.rados.0.smithi165.stdout:2804: snap_remove snap 294 2024-09-10T16:24:41.476 INFO:tasks.rados.rados.0.smithi165.stdout:2804: done (2 left) 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2805: copy_from oid 18 from oid 34 current snap is 298 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2802: finishing rollback tid 0 to smithi16534841-42 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2802: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2802: finishing rollback tid 2 to smithi16534841-42 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1295 (ObjNum 758 snap 230 seq_num 758) dirty exists 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: finishing write tid 1 to smithi16534841-4 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: finishing write tid 2 to smithi16534841-4 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: finishing write tid 3 to smithi16534841-4 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: finishing write tid 4 to smithi16534841-4 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: oid 4 updating version 0 to 1343 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: oid 4 updating version 1343 to 1344 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:2803: oid 4 version 1344 is already newer than 1342 2024-09-10T16:24:41.477 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1344 (ObjNum 983 snap 298 seq_num 983) dirty exists 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2803: left oid 4 (ObjNum 983 snap 298 seq_num 983) 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2802: done (2 left) 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2803: done (1 left) 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2806: read oid 31 snap -1 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2806: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2807: copy_from oid 27 from oid 32 current snap is 298 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2808: write (excl) oid 12 current snap is 298 2024-09-10T16:24:41.478 INFO:tasks.rados.rados.0.smithi165.stdout:2808: seq_num 984 ranges {538088=654553,1714752=634675} 2024-09-10T16:24:41.484 INFO:tasks.rados.rados.0.smithi165.stdout:2808: writing smithi16534841-12 from 538088 to 1192641 tid 1 2024-09-10T16:24:41.491 INFO:tasks.rados.rados.0.smithi165.stdout:2808: writing smithi16534841-12 from 1714752 to 2349427 tid 2 2024-09-10T16:24:41.492 INFO:tasks.rados.rados.0.smithi165.stdout:2809: write oid 17 current snap is 298 2024-09-10T16:24:41.492 INFO:tasks.rados.rados.0.smithi165.stdout:2807: finishing copy_from to smithi16534841-27 2024-09-10T16:24:41.492 INFO:tasks.rados.rados.0.smithi165.stdout:2807: got expected ENOENT (src dne) 2024-09-10T16:24:41.492 INFO:tasks.rados.rados.0.smithi165.stdout:2807: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:24:41.492 INFO:tasks.rados.rados.0.smithi165.stdout:2809: seq_num 985 ranges {554895=729802,1773234=778333} 2024-09-10T16:24:41.499 INFO:tasks.rados.rados.0.smithi165.stdout:2809: writing smithi16534841-17 from 554895 to 1284697 tid 1 2024-09-10T16:24:41.506 INFO:tasks.rados.rados.0.smithi165.stdout:2809: writing smithi16534841-17 from 1773234 to 2551567 tid 2 2024-09-10T16:24:41.506 INFO:tasks.rados.rados.0.smithi165.stdout:2807: done (4 left) 2024-09-10T16:24:41.506 INFO:tasks.rados.rados.0.smithi165.stdout:2810: write (excl) oid 49 current snap is 298 2024-09-10T16:24:41.507 INFO:tasks.rados.rados.0.smithi165.stdout:2810: seq_num 986 ranges {571702=405051,1431716=638344,2560365=11337} 2024-09-10T16:24:41.511 INFO:tasks.rados.rados.0.smithi165.stdout:2810: writing smithi16534841-49 from 571702 to 976753 tid 1 2024-09-10T16:24:41.517 INFO:tasks.rados.rados.0.smithi165.stdout:2810: writing smithi16534841-49 from 1431716 to 2070060 tid 2 2024-09-10T16:24:41.520 INFO:tasks.rados.rados.0.smithi165.stdout:2810: writing smithi16534841-49 from 2560365 to 2571702 tid 3 2024-09-10T16:24:41.521 INFO:tasks.rados.rados.0.smithi165.stdout:2811: delete oid 40 current snap is 298 2024-09-10T16:24:41.521 INFO:tasks.rados.rados.0.smithi165.stdout:2805: finishing copy_from racing read to smithi16534841-18 2024-09-10T16:24:41.521 INFO:tasks.rados.rados.0.smithi165.stdout:2806: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:41.557 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 507280 2024-09-10T16:24:41.634 INFO:tasks.rados.rados.0.smithi165.stdout:2805: finishing copy_from to smithi16534841-18 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: finishing write tid 1 to smithi16534841-12 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: finishing write tid 2 to smithi16534841-12 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: finishing write tid 3 to smithi16534841-12 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: finishing write tid 4 to smithi16534841-12 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: oid 12 updating version 0 to 1371 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: oid 12 version 1371 is already newer than 1370 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: oid 12 version 1371 is already newer than 1369 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1371 (ObjNum 984 snap 298 seq_num 984) dirty exists 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2808: left oid 12 (ObjNum 984 snap 298 seq_num 984) 2024-09-10T16:24:41.635 INFO:tasks.rados.rados.0.smithi165.stdout:2809: finishing write tid 1 to smithi16534841-17 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2805: done (5 left) 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2806: done (4 left) 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2808: done (3 left) 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2811: done (2 left) 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2812: rollback oid 14 current snap is 298 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 292 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: finishing write tid 2 to smithi16534841-17 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: finishing write tid 3 to smithi16534841-17 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: finishing write tid 4 to smithi16534841-17 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: oid 17 updating version 0 to 1556 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: oid 17 updating version 1556 to 1557 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: oid 17 updating version 1557 to 1558 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1558 (ObjNum 985 snap 298 seq_num 985) dirty exists 2024-09-10T16:24:41.636 INFO:tasks.rados.rados.0.smithi165.stdout:2809: left oid 17 (ObjNum 985 snap 298 seq_num 985) 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: finishing write tid 1 to smithi16534841-49 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: finishing write tid 2 to smithi16534841-49 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: finishing write tid 3 to smithi16534841-49 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: finishing write tid 4 to smithi16534841-49 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: finishing write tid 5 to smithi16534841-49 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: oid 49 updating version 0 to 1560 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: oid 49 updating version 1560 to 1562 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: oid 49 version 1562 is already newer than 1559 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: oid 49 version 1562 is already newer than 1561 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1562 (ObjNum 986 snap 298 seq_num 986) dirty exists 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: left oid 49 (ObjNum 986 snap 298 seq_num 986) 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2809: done (2 left) 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2810: done (1 left) 2024-09-10T16:24:41.637 INFO:tasks.rados.rados.0.smithi165.stdout:2813: snap_remove snap 293 2024-09-10T16:24:42.549 INFO:tasks.rados.rados.0.smithi165.stdout:2813: done (1 left) 2024-09-10T16:24:42.549 INFO:tasks.rados.rados.0.smithi165.stdout:2814: write oid 8 current snap is 298 2024-09-10T16:24:42.549 INFO:tasks.rados.rados.0.smithi165.stdout:2812: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:24:42.549 INFO:tasks.rados.rados.0.smithi165.stdout:2814: seq_num 988 ranges {605316=555549,1832327=642013} 2024-09-10T16:24:42.555 INFO:tasks.rados.rados.0.smithi165.stdout:2814: writing smithi16534841-8 from 605316 to 1160865 tid 1 2024-09-10T16:24:42.563 INFO:tasks.rados.rados.0.smithi165.stdout:2814: writing smithi16534841-8 from 1832327 to 2474340 tid 2 2024-09-10T16:24:42.563 INFO:tasks.rados.rados.0.smithi165.stdout:2815: snap_remove snap 296 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:2815: done (2 left) 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:2816: write (excl) oid 21 current snap is 298 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:2812: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:2812: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1241 (ObjNum 966 snap 292 seq_num 966) dirty exists 2024-09-10T16:24:44.415 INFO:tasks.rados.rados.0.smithi165.stdout:2816: seq_num 989 ranges {622123=747151,2007162=614961} 2024-09-10T16:24:44.422 INFO:tasks.rados.rados.0.smithi165.stdout:2816: writing smithi16534841-21 from 622123 to 1369274 tid 1 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2816: writing smithi16534841-21 from 2007162 to 2622123 tid 2 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2812: done (2 left) 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2817: copy_from oid 12 from oid 27 current snap is 298 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2814: finishing write tid 1 to smithi16534841-8 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2814: finishing write tid 2 to smithi16534841-8 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2814: finishing write tid 3 to smithi16534841-8 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2814: finishing write tid 4 to smithi16534841-8 2024-09-10T16:24:44.430 INFO:tasks.rados.rados.0.smithi165.stdout:2814: oid 8 updating version 0 to 1621 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2814: oid 8 version 1621 is already newer than 1619 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2814: oid 8 version 1621 is already newer than 1620 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1621 (ObjNum 988 snap 298 seq_num 988) dirty exists 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2814: left oid 8 (ObjNum 988 snap 298 seq_num 988) 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2816: finishing write tid 1 to smithi16534841-21 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2814: done (2 left) 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2818: read oid 4 snap -1 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2818: expect (ObjNum 983 snap 298 seq_num 983) 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2819: copy_from oid 16 from oid 23 current snap is 298 2024-09-10T16:24:44.431 INFO:tasks.rados.rados.0.smithi165.stdout:2820: delete oid 40 current snap is 298 2024-09-10T16:24:44.432 INFO:tasks.rados.rados.0.smithi165.stdout:2820: done (4 left) 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2821: read oid 43 snap -1 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2821: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2822: read oid 29 snap 292 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2822: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2823: copy_from oid 41 from oid 18 current snap is 298 2024-09-10T16:24:44.433 INFO:tasks.rados.rados.0.smithi165.stdout:2824: snap_create 2024-09-10T16:24:44.436 INFO:tasks.rados.rados.0.smithi165.stdout:2822: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:24:44.618 INFO:tasks.rados.rados.0.smithi165.stdout:2822: done (7 left) 2024-09-10T16:24:44.618 INFO:tasks.rados.rados.0.smithi165.stdout:2824: done (6 left) 2024-09-10T16:24:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:2825: read oid 3 snap -1 2024-09-10T16:24:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:2825: expect deleted 2024-09-10T16:24:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:2826: write oid 45 current snap is 299 2024-09-10T16:24:44.619 INFO:tasks.rados.rados.0.smithi165.stdout:2826: seq_num 991 ranges {655737=497649,1607773=789340} 2024-09-10T16:24:44.624 INFO:tasks.rados.rados.0.smithi165.stdout:2826: writing smithi16534841-45 from 655737 to 1153386 tid 1 2024-09-10T16:24:44.631 INFO:tasks.rados.rados.0.smithi165.stdout:2826: writing smithi16534841-45 from 1607773 to 2397113 tid 2 2024-09-10T16:24:44.632 INFO:tasks.rados.rados.0.smithi165.stdout:2827: write oid 22 current snap is 299 2024-09-10T16:24:44.632 INFO:tasks.rados.rados.0.smithi165.stdout:2818: expect (ObjNum 983 snap 298 seq_num 983) 2024-09-10T16:24:44.640 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 575092 2024-09-10T16:24:44.815 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2107052 2024-09-10T16:24:44.826 INFO:tasks.rados.rados.0.smithi165.stdout:2816: finishing write tid 2 to smithi16534841-21 2024-09-10T16:24:44.826 INFO:tasks.rados.rados.0.smithi165.stdout:2816: finishing write tid 3 to smithi16534841-21 2024-09-10T16:24:44.826 INFO:tasks.rados.rados.0.smithi165.stdout:2827: seq_num 992 ranges {672544=572898,1666255=532998} 2024-09-10T16:24:44.832 INFO:tasks.rados.rados.0.smithi165.stdout:2827: writing smithi16534841-22 from 672544 to 1245442 tid 1 2024-09-10T16:24:44.838 INFO:tasks.rados.rados.0.smithi165.stdout:2827: writing smithi16534841-22 from 1666255 to 2199253 tid 2 2024-09-10T16:24:44.838 INFO:tasks.rados.rados.0.smithi165.stdout:2818: done (8 left) 2024-09-10T16:24:44.838 INFO:tasks.rados.rados.0.smithi165.stdout:2828: read oid 26 snap -1 2024-09-10T16:24:44.838 INFO:tasks.rados.rados.0.smithi165.stdout:2828: expect (ObjNum 930 snap 285 seq_num 930) 2024-09-10T16:24:44.838 INFO:tasks.rados.rados.0.smithi165.stdout:2829: write (excl) oid 36 current snap is 299 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2816: finishing write tid 4 to smithi16534841-21 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2816: oid 21 updating version 0 to 1556 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2816: oid 21 updating version 1556 to 1557 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2816: oid 21 version 1557 is already newer than 1555 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1557 (ObjNum 989 snap 298 seq_num 989) dirty exists 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2816: left oid 21 (ObjNum 989 snap 298 seq_num 989) 2024-09-10T16:24:44.839 INFO:tasks.rados.rados.0.smithi165.stdout:2821: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:24:44.915 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2817: finishing copy_from racing read to smithi16534841-12 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2817: finishing copy_from to smithi16534841-12 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1618 (ObjNum 970 snap 294 seq_num 970) dirty exists 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2823: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2819: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2823: finishing copy_from to smithi16534841-41 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:24:45.074 INFO:tasks.rados.rados.0.smithi165.stdout:2819: finishing copy_from to smithi16534841-16 2024-09-10T16:24:45.075 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1413 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:24:45.075 INFO:tasks.rados.rados.0.smithi165.stdout:2829: seq_num 993 ranges {689351=648147,2124737=564614} 2024-09-10T16:24:45.080 INFO:tasks.rados.rados.0.smithi165.stdout:2829: writing smithi16534841-36 from 689351 to 1337498 tid 1 2024-09-10T16:24:45.088 INFO:tasks.rados.rados.0.smithi165.stdout:2829: writing smithi16534841-36 from 2124737 to 2689351 tid 2 2024-09-10T16:24:45.088 INFO:tasks.rados.rados.0.smithi165.stdout:2816: done (9 left) 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2817: done (8 left) 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2819: done (7 left) 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2821: done (6 left) 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2823: done (5 left) 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2830: write oid 46 current snap is 299 2024-09-10T16:24:45.089 INFO:tasks.rados.rados.0.smithi165.stdout:2830: seq_num 994 ranges {706158=723396,2183219=522939} 2024-09-10T16:24:45.095 INFO:tasks.rados.rados.0.smithi165.stdout:2830: writing smithi16534841-46 from 706158 to 1429554 tid 1 2024-09-10T16:24:45.102 INFO:tasks.rados.rados.0.smithi165.stdout:2830: writing smithi16534841-46 from 2183219 to 2706158 tid 2 2024-09-10T16:24:45.102 INFO:tasks.rados.rados.0.smithi165.stdout:2831: snap_remove snap 292 2024-09-10T16:24:45.561 INFO:tasks.rados.rados.0.smithi165.stdout:2831: done (6 left) 2024-09-10T16:24:45.561 INFO:tasks.rados.rados.0.smithi165.stdout:2832: read oid 27 snap -1 2024-09-10T16:24:45.561 INFO:tasks.rados.rados.0.smithi165.stdout:2832: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:24:45.561 INFO:tasks.rados.rados.0.smithi165.stdout:2833: copy_from oid 25 from oid 33 current snap is 299 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: finishing write tid 1 to smithi16534841-45 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: finishing write tid 2 to smithi16534841-45 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: finishing write tid 3 to smithi16534841-45 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: finishing write tid 4 to smithi16534841-45 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: oid 45 updating version 0 to 1669 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: oid 45 updating version 1669 to 1671 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: oid 45 version 1671 is already newer than 1670 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1671 (ObjNum 991 snap 299 seq_num 991) dirty exists 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2826: left oid 45 (ObjNum 991 snap 299 seq_num 991) 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2827: finishing write tid 1 to smithi16534841-22 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2827: finishing write tid 2 to smithi16534841-22 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2827: finishing write tid 3 to smithi16534841-22 2024-09-10T16:24:45.562 INFO:tasks.rados.rados.0.smithi165.stdout:2827: finishing write tid 4 to smithi16534841-22 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:2827: oid 22 updating version 0 to 1417 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:2827: oid 22 updating version 1417 to 1419 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:2827: oid 22 version 1419 is already newer than 1418 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1419 (ObjNum 992 snap 299 seq_num 992) dirty exists 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:2827: left oid 22 (ObjNum 992 snap 299 seq_num 992) 2024-09-10T16:24:45.563 INFO:tasks.rados.rados.0.smithi165.stdout:2828: expect (ObjNum 930 snap 285 seq_num 930) 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: finishing write tid 1 to smithi16534841-36 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: finishing write tid 2 to smithi16534841-36 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: finishing write tid 3 to smithi16534841-36 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: finishing write tid 4 to smithi16534841-36 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: oid 36 updating version 0 to 1387 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: oid 36 version 1387 is already newer than 1386 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: oid 36 version 1387 is already newer than 1385 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1387 (ObjNum 993 snap 299 seq_num 993) dirty exists 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2829: left oid 36 (ObjNum 993 snap 299 seq_num 993) 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2825: done (7 left) 2024-09-10T16:24:45.833 INFO:tasks.rados.rados.0.smithi165.stdout:2826: done (6 left) 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2827: done (5 left) 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2828: done (4 left) 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2829: done (3 left) 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2834: read oid 39 snap -1 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2834: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2835: read oid 32 snap 298 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2835: expect deleted 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2836: rollback oid 12 current snap is 299 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 295 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2830: finishing write tid 1 to smithi16534841-46 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2830: finishing write tid 2 to smithi16534841-46 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2830: finishing write tid 3 to smithi16534841-46 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2830: finishing write tid 4 to smithi16534841-46 2024-09-10T16:24:45.834 INFO:tasks.rados.rados.0.smithi165.stdout:2830: oid 46 updating version 0 to 1159 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2830: oid 46 version 1159 is already newer than 1158 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2830: oid 46 updating version 1159 to 1160 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1160 (ObjNum 994 snap 299 seq_num 994) dirty exists 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2830: left oid 46 (ObjNum 994 snap 299 seq_num 994) 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2833: finishing copy_from to smithi16534841-25 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2833: got expected ENOENT (src dne) 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2833: finishing copy_from racing read to smithi16534841-25 2024-09-10T16:24:45.835 INFO:tasks.rados.rados.0.smithi165.stdout:2832: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:24:45.845 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1532227 2024-09-10T16:24:45.857 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:45.892 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:45.926 INFO:tasks.rados.rados.0.smithi165.stdout:2836: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:24:45.926 INFO:tasks.rados.rados.0.smithi165.stdout:2836: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:24:45.926 INFO:tasks.rados.rados.0.smithi165.stdout:2834: expect (ObjNum 860 snap 263 seq_num 860) 2024-09-10T16:24:45.966 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1489989 2024-09-10T16:24:46.025 INFO:tasks.rados.rados.0.smithi165.stdout:2836: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1621 (ObjNum 944 snap 285 seq_num 944) dirty exists 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2830: done (5 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2832: done (4 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2833: done (3 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2834: done (2 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2835: done (1 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2836: done (0 left) 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2837: write (excl) oid 35 current snap is 299 2024-09-10T16:24:46.026 INFO:tasks.rados.rados.0.smithi165.stdout:2837: seq_num 995 ranges {722965=798645,2125348=597617} 2024-09-10T16:24:46.033 INFO:tasks.rados.rados.0.smithi165.stdout:2837: writing smithi16534841-35 from 722965 to 1521610 tid 1 2024-09-10T16:24:46.040 INFO:tasks.rados.rados.0.smithi165.stdout:2837: writing smithi16534841-35 from 2125348 to 2722965 tid 2 2024-09-10T16:24:46.040 INFO:tasks.rados.rados.0.smithi165.stdout:2838: delete oid 46 current snap is 299 2024-09-10T16:24:46.040 INFO:tasks.rados.rados.0.smithi165.stdout:2837: finishing write tid 1 to smithi16534841-35 2024-09-10T16:24:46.041 INFO:tasks.rados.rados.0.smithi165.stdout:2838: done (1 left) 2024-09-10T16:24:46.041 INFO:tasks.rados.rados.0.smithi165.stdout:2839: write oid 12 current snap is 299 2024-09-10T16:24:46.041 INFO:tasks.rados.rados.0.smithi165.stdout:2839: seq_num 997 ranges {756579=665496,1958665=400347} 2024-09-10T16:24:46.049 INFO:tasks.rados.rados.0.smithi165.stdout:2839: writing smithi16534841-12 from 756579 to 1422075 tid 1 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:2839: writing smithi16534841-12 from 1958665 to 2359012 tid 2 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:2840: rollback oid 33 current snap is 299 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 297 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:2837: finishing write tid 2 to smithi16534841-35 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:2837: finishing write tid 3 to smithi16534841-35 2024-09-10T16:24:46.055 INFO:tasks.rados.rados.0.smithi165.stdout:2837: finishing write tid 4 to smithi16534841-35 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2837: oid 35 updating version 0 to 1421 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2837: oid 35 updating version 1421 to 1422 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2837: oid 35 version 1422 is already newer than 1420 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1422 (ObjNum 995 snap 299 seq_num 995) dirty exists 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2837: left oid 35 (ObjNum 995 snap 299 seq_num 995) 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2839: finishing write tid 1 to smithi16534841-12 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2837: done (2 left) 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2841: copy_from oid 16 from oid 41 current snap is 299 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2842: read oid 31 snap -1 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2842: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2843: copy_from oid 7 from oid 26 current snap is 299 2024-09-10T16:24:46.056 INFO:tasks.rados.rados.0.smithi165.stdout:2844: snap_remove snap 295 2024-09-10T16:24:46.564 INFO:tasks.rados.rados.0.smithi165.stdout:2844: done (5 left) 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2845: read oid 32 snap -1 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2845: expect deleted 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2846: read oid 2 snap 298 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2846: expect (ObjNum 957 snap 289 seq_num 957) 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2847: write oid 15 current snap is 299 2024-09-10T16:24:46.565 INFO:tasks.rados.rados.0.smithi165.stdout:2847: seq_num 998 ranges {773386=740745,2017147=544005} 2024-09-10T16:24:46.573 INFO:tasks.rados.rados.0.smithi165.stdout:2847: writing smithi16534841-15 from 773386 to 1514131 tid 1 2024-09-10T16:24:46.580 INFO:tasks.rados.rados.0.smithi165.stdout:2847: writing smithi16534841-15 from 2017147 to 2561152 tid 2 2024-09-10T16:24:46.580 INFO:tasks.rados.rados.0.smithi165.stdout:2848: write oid 19 current snap is 299 2024-09-10T16:24:46.580 INFO:tasks.rados.rados.0.smithi165.stdout:2848: seq_num 999 ranges {790193=415994,1959276=404016} 2024-09-10T16:24:46.586 INFO:tasks.rados.rados.0.smithi165.stdout:2848: writing smithi16534841-19 from 790193 to 1206187 tid 1 2024-09-10T16:24:46.592 INFO:tasks.rados.rados.0.smithi165.stdout:2848: writing smithi16534841-19 from 1959276 to 2363292 tid 2 2024-09-10T16:24:46.593 INFO:tasks.rados.rados.0.smithi165.stdout:2849: snap_remove snap 290 2024-09-10T16:24:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:2849: done (9 left) 2024-09-10T16:24:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:2850: write oid 13 current snap is 299 2024-09-10T16:24:47.569 INFO:tasks.rados.rados.0.smithi165.stdout:2850: seq_num 1000 ranges {407000=491243,1617758=547674,2595228=211772} 2024-09-10T16:24:47.573 INFO:tasks.rados.rados.0.smithi165.stdout:2850: writing smithi16534841-13 from 407000 to 898243 tid 1 2024-09-10T16:24:47.580 INFO:tasks.rados.rados.0.smithi165.stdout:2850: writing smithi16534841-13 from 1617758 to 2165432 tid 2 2024-09-10T16:24:47.584 INFO:tasks.rados.rados.0.smithi165.stdout:2850: writing smithi16534841-13 from 2595228 to 2807000 tid 3 2024-09-10T16:24:47.584 INFO:tasks.rados.rados.0.smithi165.stdout:2851: write (excl) oid 48 current snap is 299 2024-09-10T16:24:47.584 INFO:tasks.rados.rados.0.smithi165.stdout:2840: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:24:47.585 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1546 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:24:47.585 INFO:tasks.rados.rados.0.smithi165.stdout:2851: seq_num 1001 ranges {423807=566492,1676240=407685,2739004=84803} 2024-09-10T16:24:47.589 INFO:tasks.rados.rados.0.smithi165.stdout:2851: writing smithi16534841-48 from 423807 to 990299 tid 1 2024-09-10T16:24:47.596 INFO:tasks.rados.rados.0.smithi165.stdout:2851: writing smithi16534841-48 from 1676240 to 2083925 tid 2 2024-09-10T16:24:47.600 INFO:tasks.rados.rados.0.smithi165.stdout:2851: writing smithi16534841-48 from 2739004 to 2823807 tid 3 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2840: done (10 left) 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2852: read oid 25 snap -1 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2852: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2853: delete oid 17 current snap is 299 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: finishing write tid 2 to smithi16534841-12 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: finishing write tid 3 to smithi16534841-12 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: finishing write tid 4 to smithi16534841-12 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: oid 12 updating version 0 to 1623 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: oid 12 updating version 1623 to 1624 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:2839: oid 12 version 1624 is already newer than 1622 2024-09-10T16:24:47.601 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1624 (ObjNum 997 snap 299 seq_num 997) dirty exists 2024-09-10T16:24:47.602 INFO:tasks.rados.rados.0.smithi165.stdout:2839: left oid 12 (ObjNum 997 snap 299 seq_num 997) 2024-09-10T16:24:47.602 INFO:tasks.rados.rados.0.smithi165.stdout:2842: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:47.652 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1094175 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2841: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2843: finishing copy_from to smithi16534841-7 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1662 (ObjNum 930 snap 285 seq_num 930) dirty exists 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2843: finishing copy_from racing read to smithi16534841-7 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2841: finishing copy_from to smithi16534841-16 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2839: done (11 left) 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2841: done (10 left) 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2842: done (9 left) 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2843: done (8 left) 2024-09-10T16:24:47.712 INFO:tasks.rados.rados.0.smithi165.stdout:2853: done (7 left) 2024-09-10T16:24:47.713 INFO:tasks.rados.rados.0.smithi165.stdout:2854: snap_create 2024-09-10T16:24:47.713 INFO:tasks.rados.rados.0.smithi165.stdout:2846: expect (ObjNum 957 snap 289 seq_num 957) 2024-09-10T16:24:47.734 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:47.771 INFO:tasks.rados.rados.0.smithi165.stdout:2847: finishing write tid 1 to smithi16534841-15 2024-09-10T16:24:47.771 INFO:tasks.rados.rados.0.smithi165.stdout:2847: finishing write tid 2 to smithi16534841-15 2024-09-10T16:24:47.771 INFO:tasks.rados.rados.0.smithi165.stdout:2847: finishing write tid 3 to smithi16534841-15 2024-09-10T16:24:47.771 INFO:tasks.rados.rados.0.smithi165.stdout:2847: finishing write tid 4 to smithi16534841-15 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2847: oid 15 updating version 0 to 1346 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2847: oid 15 updating version 1346 to 1347 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2847: oid 15 version 1347 is already newer than 1345 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1347 (ObjNum 998 snap 299 seq_num 998) dirty exists 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2847: left oid 15 (ObjNum 998 snap 299 seq_num 998) 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2848: finishing write tid 1 to smithi16534841-19 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2848: finishing write tid 2 to smithi16534841-19 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2848: finishing write tid 3 to smithi16534841-19 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2848: finishing write tid 4 to smithi16534841-19 2024-09-10T16:24:47.772 INFO:tasks.rados.rados.0.smithi165.stdout:2848: oid 19 updating version 0 to 1795 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2848: oid 19 version 1795 is already newer than 1793 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2848: oid 19 version 1795 is already newer than 1794 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1795 (ObjNum 999 snap 299 seq_num 999) dirty exists 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2848: left oid 19 (ObjNum 999 snap 299 seq_num 999) 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: finishing write tid 1 to smithi16534841-13 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: finishing write tid 2 to smithi16534841-13 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: finishing write tid 3 to smithi16534841-13 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: finishing write tid 4 to smithi16534841-13 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: finishing write tid 5 to smithi16534841-13 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: oid 13 updating version 0 to 1423 2024-09-10T16:24:47.773 INFO:tasks.rados.rados.0.smithi165.stdout:2850: oid 13 updating version 1423 to 1424 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2850: oid 13 updating version 1424 to 1425 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2850: oid 13 updating version 1425 to 1426 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1426 (ObjNum 1000 snap 299 seq_num 1000) dirty exists 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2850: left oid 13 (ObjNum 1000 snap 299 seq_num 1000) 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: finishing write tid 1 to smithi16534841-48 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: finishing write tid 2 to smithi16534841-48 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: finishing write tid 3 to smithi16534841-48 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: finishing write tid 4 to smithi16534841-48 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: finishing write tid 5 to smithi16534841-48 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: oid 48 updating version 0 to 1806 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: oid 48 updating version 1806 to 1808 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: oid 48 version 1808 is already newer than 1805 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:2851: oid 48 version 1808 is already newer than 1807 2024-09-10T16:24:47.774 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1808 (ObjNum 1001 snap 299 seq_num 1001) dirty exists 2024-09-10T16:24:47.775 INFO:tasks.rados.rados.0.smithi165.stdout:2851: left oid 48 (ObjNum 1001 snap 299 seq_num 1001) 2024-09-10T16:24:47.775 INFO:tasks.rados.rados.0.smithi165.stdout:2852: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2845: done (7 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2846: done (6 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2847: done (5 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2848: done (4 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2850: done (3 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2851: done (2 left) 2024-09-10T16:24:48.672 INFO:tasks.rados.rados.0.smithi165.stdout:2852: done (1 left) 2024-09-10T16:24:48.673 INFO:tasks.rados.rados.0.smithi165.stdout:2854: done (0 left) 2024-09-10T16:24:48.673 INFO:tasks.rados.rados.0.smithi165.stdout:2855: write oid 39 current snap is 300 2024-09-10T16:24:48.673 INFO:tasks.rados.rados.0.smithi165.stdout:2855: seq_num 1003 ranges {457421=716990,1676851=411354,2677497=179924} 2024-09-10T16:24:48.678 INFO:tasks.rados.rados.0.smithi165.stdout:2855: writing smithi16534841-39 from 457421 to 1174411 tid 1 2024-09-10T16:24:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:2855: writing smithi16534841-39 from 1676851 to 2088205 tid 2 2024-09-10T16:24:48.688 INFO:tasks.rados.rados.0.smithi165.stdout:2855: writing smithi16534841-39 from 2677497 to 2857421 tid 3 2024-09-10T16:24:48.688 INFO:tasks.rados.rados.0.smithi165.stdout:2856: read oid 8 snap 297 2024-09-10T16:24:48.688 INFO:tasks.rados.rados.0.smithi165.stdout:2856: expect (ObjNum 921 snap 281 seq_num 921) 2024-09-10T16:24:48.688 INFO:tasks.rados.rados.0.smithi165.stdout:2857: read oid 3 snap 297 2024-09-10T16:24:48.688 INFO:tasks.rados.rados.0.smithi165.stdout:2857: expect deleted 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2855: finishing write tid 1 to smithi16534841-39 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2855: finishing write tid 2 to smithi16534841-39 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2858: read oid 31 snap -1 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2858: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2859: write (excl) oid 16 current snap is 300 2024-09-10T16:24:48.689 INFO:tasks.rados.rados.0.smithi165.stdout:2859: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2024-09-10T16:24:48.694 INFO:tasks.rados.rados.0.smithi165.stdout:2859: writing smithi16534841-16 from 474228 to 982820 tid 1 2024-09-10T16:24:48.700 INFO:tasks.rados.rados.0.smithi165.stdout:2859: writing smithi16534841-16 from 1451686 to 2006698 tid 2 2024-09-10T16:24:48.705 INFO:tasks.rados.rados.0.smithi165.stdout:2859: writing smithi16534841-16 from 2704920 to 2874228 tid 3 2024-09-10T16:24:48.705 INFO:tasks.rados.rados.0.smithi165.stdout:2860: write (excl) oid 45 current snap is 300 2024-09-10T16:24:48.705 INFO:tasks.rados.rados.0.smithi165.stdout:2860: seq_num 1005 ranges {491035=583841,1510168=698670,2732343=158692} 2024-09-10T16:24:48.711 INFO:tasks.rados.rados.0.smithi165.stdout:2860: writing smithi16534841-45 from 491035 to 1074876 tid 1 2024-09-10T16:24:48.717 INFO:tasks.rados.rados.0.smithi165.stdout:2860: writing smithi16534841-45 from 1510168 to 2208838 tid 2 2024-09-10T16:24:48.721 INFO:tasks.rados.rados.0.smithi165.stdout:2860: writing smithi16534841-45 from 2732343 to 2891035 tid 3 2024-09-10T16:24:48.722 INFO:tasks.rados.rados.0.smithi165.stdout:2861: write oid 38 current snap is 300 2024-09-10T16:24:48.722 INFO:tasks.rados.rados.0.smithi165.stdout:2861: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2024-09-10T16:24:48.727 INFO:tasks.rados.rados.0.smithi165.stdout:2861: writing smithi16534841-38 from 507842 to 1166932 tid 1 2024-09-10T16:24:48.733 INFO:tasks.rados.rados.0.smithi165.stdout:2861: writing smithi16534841-38 from 1568650 to 2127331 tid 2 2024-09-10T16:24:48.737 INFO:tasks.rados.rados.0.smithi165.stdout:2861: writing smithi16534841-38 from 2759766 to 2907842 tid 3 2024-09-10T16:24:48.737 INFO:tasks.rados.rados.0.smithi165.stdout:2862: snap_remove snap 299 2024-09-10T16:24:48.737 INFO:tasks.rados.rados.0.smithi165.stdout:2855: finishing write tid 3 to smithi16534841-39 2024-09-10T16:24:48.738 INFO:tasks.rados.rados.0.smithi165.stdout:2858: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:24:48.738 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:48.848 INFO:tasks.rados.rados.0.smithi165.stdout:2855: finishing write tid 4 to smithi16534841-39 2024-09-10T16:24:48.848 INFO:tasks.rados.rados.0.smithi165.stdout:2855: finishing write tid 5 to smithi16534841-39 2024-09-10T16:24:48.848 INFO:tasks.rados.rados.0.smithi165.stdout:2855: oid 39 updating version 0 to 1424 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:2855: oid 39 version 1424 is already newer than 1423 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:2855: oid 39 version 1424 is already newer than 1422 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:2855: oid 39 updating version 1424 to 1425 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1425 (ObjNum 1003 snap 300 seq_num 1003) dirty exists 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:2855: left oid 39 (ObjNum 1003 snap 300 seq_num 1003) 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:2856: expect (ObjNum 921 snap 281 seq_num 921) 2024-09-10T16:24:48.849 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:49.039 INFO:tasks.rados.rados.0.smithi165.stdout:2859: finishing write tid 1 to smithi16534841-16 2024-09-10T16:24:49.039 INFO:tasks.rados.rados.0.smithi165.stdout:2859: finishing write tid 2 to smithi16534841-16 2024-09-10T16:24:49.039 INFO:tasks.rados.rados.0.smithi165.stdout:2859: finishing write tid 3 to smithi16534841-16 2024-09-10T16:24:49.039 INFO:tasks.rados.rados.0.smithi165.stdout:2859: finishing write tid 4 to smithi16534841-16 2024-09-10T16:24:49.040 INFO:tasks.rados.rados.0.smithi165.stdout:2859: finishing write tid 5 to smithi16534841-16 2024-09-10T16:24:49.040 INFO:tasks.rados.rados.0.smithi165.stdout:2859: oid 16 updating version 0 to 1811 2024-09-10T16:24:49.040 INFO:tasks.rados.rados.0.smithi165.stdout:2859: oid 16 updating version 1811 to 1813 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2859: oid 16 version 1813 is already newer than 1810 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2859: oid 16 version 1813 is already newer than 1812 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1813 (ObjNum 1004 snap 300 seq_num 1004) dirty exists 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2859: left oid 16 (ObjNum 1004 snap 300 seq_num 1004) 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2860: finishing write tid 1 to smithi16534841-45 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2860: finishing write tid 2 to smithi16534841-45 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2860: finishing write tid 3 to smithi16534841-45 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2860: finishing write tid 4 to smithi16534841-45 2024-09-10T16:24:49.041 INFO:tasks.rados.rados.0.smithi165.stdout:2860: finishing write tid 5 to smithi16534841-45 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2860: oid 45 updating version 0 to 1673 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2860: oid 45 updating version 1673 to 1675 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2860: oid 45 version 1675 is already newer than 1674 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2860: oid 45 version 1675 is already newer than 1672 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1675 (ObjNum 1005 snap 300 seq_num 1005) dirty exists 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2860: left oid 45 (ObjNum 1005 snap 300 seq_num 1005) 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2861: finishing write tid 1 to smithi16534841-38 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2861: finishing write tid 2 to smithi16534841-38 2024-09-10T16:24:49.042 INFO:tasks.rados.rados.0.smithi165.stdout:2861: finishing write tid 3 to smithi16534841-38 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2855: done (7 left) 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2856: done (6 left) 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2857: done (5 left) 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2858: done (4 left) 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2859: done (3 left) 2024-09-10T16:24:49.597 INFO:tasks.rados.rados.0.smithi165.stdout:2860: done (2 left) 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2862: done (1 left) 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2863: read oid 23 snap -1 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2863: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2864: snap_create 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2861: finishing write tid 4 to smithi16534841-38 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2861: finishing write tid 5 to smithi16534841-38 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2861: oid 38 updating version 0 to 1391 2024-09-10T16:24:49.598 INFO:tasks.rados.rados.0.smithi165.stdout:2861: oid 38 version 1391 is already newer than 1390 2024-09-10T16:24:49.599 INFO:tasks.rados.rados.0.smithi165.stdout:2861: oid 38 version 1391 is already newer than 1389 2024-09-10T16:24:49.599 INFO:tasks.rados.rados.0.smithi165.stdout:2861: oid 38 version 1391 is already newer than 1388 2024-09-10T16:24:49.599 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1391 (ObjNum 1006 snap 300 seq_num 1006) dirty exists 2024-09-10T16:24:49.599 INFO:tasks.rados.rados.0.smithi165.stdout:2861: left oid 38 (ObjNum 1006 snap 300 seq_num 1006) 2024-09-10T16:24:49.599 INFO:tasks.rados.rados.0.smithi165.stdout:2863: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:24:49.767 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1433894 2024-09-10T16:24:50.589 INFO:tasks.rados.rados.0.smithi165.stdout:2861: done (2 left) 2024-09-10T16:24:50.589 INFO:tasks.rados.rados.0.smithi165.stdout:2863: done (1 left) 2024-09-10T16:24:50.589 INFO:tasks.rados.rados.0.smithi165.stdout:2864: done (0 left) 2024-09-10T16:24:50.589 INFO:tasks.rados.rados.0.smithi165.stdout:2865: snap_remove snap 298 2024-09-10T16:24:51.479 INFO:tasks.rados.rados.0.smithi165.stdout:2865: done (0 left) 2024-09-10T16:24:51.479 INFO:tasks.rados.rados.0.smithi165.stdout:2866: snap_create 2024-09-10T16:24:52.487 INFO:tasks.rados.rados.0.smithi165.stdout:2866: done (0 left) 2024-09-10T16:24:52.487 INFO:tasks.rados.rados.0.smithi165.stdout:2867: snap_remove snap 297 2024-09-10T16:24:53.488 INFO:tasks.rados.rados.0.smithi165.stdout:2867: done (0 left) 2024-09-10T16:24:53.488 INFO:tasks.rados.rados.0.smithi165.stdout:2868: snap_create 2024-09-10T16:24:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:2868: done (0 left) 2024-09-10T16:24:54.497 INFO:tasks.rados.rados.0.smithi165.stdout:2869: snap_create 2024-09-10T16:24:55.494 INFO:tasks.rados.rados.0.smithi165.stdout:2869: done (0 left) 2024-09-10T16:24:55.495 INFO:tasks.rados.rados.0.smithi165.stdout:2870: snap_create 2024-09-10T16:24:56.480 INFO:tasks.rados.rados.0.smithi165.stdout:2870: done (0 left) 2024-09-10T16:24:56.480 INFO:tasks.rados.rados.0.smithi165.stdout:2871: rollback oid 2 current snap is 305 2024-09-10T16:24:56.480 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 303 2024-09-10T16:24:56.480 INFO:tasks.rados.rados.0.smithi165.stdout:2872: copy_from oid 39 from oid 7 current snap is 305 2024-09-10T16:24:56.480 INFO:tasks.rados.rados.0.smithi165.stdout:2873: read oid 46 snap -1 2024-09-10T16:24:56.481 INFO:tasks.rados.rados.0.smithi165.stdout:2873: expect deleted 2024-09-10T16:24:56.481 INFO:tasks.rados.rados.0.smithi165.stdout:2874: copy_from oid 38 from oid 11 current snap is 305 2024-09-10T16:24:56.481 INFO:tasks.rados.rados.0.smithi165.stdout:2875: write oid 35 current snap is 305 2024-09-10T16:24:56.481 INFO:tasks.rados.rados.0.smithi165.stdout:2875: seq_num 1007 ranges {524649=734339,1910779=702339} 2024-09-10T16:24:56.487 INFO:tasks.rados.rados.0.smithi165.stdout:2875: writing smithi16534841-35 from 524649 to 1258988 tid 1 2024-09-10T16:24:56.496 INFO:tasks.rados.rados.0.smithi165.stdout:2875: writing smithi16534841-35 from 1910779 to 2613118 tid 2 2024-09-10T16:24:56.496 INFO:tasks.rados.rados.0.smithi165.stdout:2876: copy_from oid 17 from oid 10 current snap is 305 2024-09-10T16:24:56.496 INFO:tasks.rados.rados.0.smithi165.stdout:2871: finishing rollback tid 0 to smithi16534841-2 2024-09-10T16:24:56.496 INFO:tasks.rados.rados.0.smithi165.stdout:2871: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:24:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2873: done (5 left) 2024-09-10T16:24:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2877: read oid 15 snap -1 2024-09-10T16:24:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2877: expect (ObjNum 998 snap 299 seq_num 998) 2024-09-10T16:24:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2878: write (excl) oid 12 current snap is 305 2024-09-10T16:24:56.497 INFO:tasks.rados.rados.0.smithi165.stdout:2878: seq_num 1008 ranges {541456=409588,1569261=562350,2814612=126844} 2024-09-10T16:24:56.502 INFO:tasks.rados.rados.0.smithi165.stdout:2878: writing smithi16534841-12 from 541456 to 951044 tid 1 2024-09-10T16:24:56.509 INFO:tasks.rados.rados.0.smithi165.stdout:2878: writing smithi16534841-12 from 1569261 to 2131611 tid 2 2024-09-10T16:24:56.514 INFO:tasks.rados.rados.0.smithi165.stdout:2878: writing smithi16534841-12 from 2814612 to 2941456 tid 3 2024-09-10T16:24:56.514 INFO:tasks.rados.rados.0.smithi165.stdout:2879: read oid 8 snap 304 2024-09-10T16:24:56.514 INFO:tasks.rados.rados.0.smithi165.stdout:2879: expect (ObjNum 988 snap 298 seq_num 988) 2024-09-10T16:24:56.514 INFO:tasks.rados.rados.0.smithi165.stdout:2880: snap_remove snap 300 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:2880: done (8 left) 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:2881: rollback oid 49 current snap is 305 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 304 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:2871: finishing rollback tid 2 to smithi16534841-2 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1624 (ObjNum 957 snap 289 seq_num 957) dirty exists 2024-09-10T16:24:57.520 INFO:tasks.rados.rados.0.smithi165.stdout:2875: finishing write tid 1 to smithi16534841-35 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2872: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: finishing write tid 2 to smithi16534841-35 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: finishing write tid 3 to smithi16534841-35 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: finishing write tid 4 to smithi16534841-35 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: oid 35 updating version 0 to 1429 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: oid 35 version 1429 is already newer than 1428 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: oid 35 version 1429 is already newer than 1427 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1429 (ObjNum 1007 snap 305 seq_num 1007) dirty exists 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2875: left oid 35 (ObjNum 1007 snap 305 seq_num 1007) 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2874: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:2877: expect (ObjNum 998 snap 299 seq_num 998) 2024-09-10T16:24:57.521 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:57.568 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1130241 2024-09-10T16:24:57.596 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:24:57.637 INFO:tasks.rados.rados.0.smithi165.stdout:2874: finishing copy_from to smithi16534841-38 2024-09-10T16:24:57.637 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1292 (ObjNum 982 snap 297 seq_num 982) dirty exists 2024-09-10T16:24:57.637 INFO:tasks.rados.rados.0.smithi165.stdout:2876: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:24:57.638 INFO:tasks.rados.rados.0.smithi165.stdout:2878: finishing write tid 1 to smithi16534841-12 2024-09-10T16:24:57.638 INFO:tasks.rados.rados.0.smithi165.stdout:2878: finishing write tid 2 to smithi16534841-12 2024-09-10T16:24:57.638 INFO:tasks.rados.rados.0.smithi165.stdout:2876: finishing copy_from to smithi16534841-17 2024-09-10T16:24:57.638 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1551 (ObjNum 977 snap 297 seq_num 977) dirty exists 2024-09-10T16:24:57.638 INFO:tasks.rados.rados.0.smithi165.stdout:2879: expect (ObjNum 988 snap 298 seq_num 988) 2024-09-10T16:24:57.784 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2141740 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2872: finishing copy_from to smithi16534841-39 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1662 (ObjNum 930 snap 285 seq_num 930) dirty exists 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: finishing write tid 3 to smithi16534841-12 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: finishing write tid 4 to smithi16534841-12 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: finishing write tid 5 to smithi16534841-12 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: oid 12 updating version 0 to 1627 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: oid 12 updating version 1627 to 1628 2024-09-10T16:24:57.796 INFO:tasks.rados.rados.0.smithi165.stdout:2878: oid 12 version 1628 is already newer than 1626 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2878: oid 12 version 1628 is already newer than 1625 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1628 (ObjNum 1008 snap 305 seq_num 1008) dirty exists 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2878: left oid 12 (ObjNum 1008 snap 305 seq_num 1008) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2881: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2881: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2871: done (8 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2872: done (7 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2874: done (6 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2875: done (5 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2876: done (4 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2877: done (3 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2878: done (2 left) 2024-09-10T16:24:57.797 INFO:tasks.rados.rados.0.smithi165.stdout:2879: done (1 left) 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:2882: rollback oid 35 current snap is 305 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 301 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:2881: finishing rollback tid 2 to smithi16534841-49 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1566 (ObjNum 986 snap 298 seq_num 986) dirty exists 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:2881: done (1 left) 2024-09-10T16:24:57.798 INFO:tasks.rados.rados.0.smithi165.stdout:2883: delete oid 24 current snap is 305 2024-09-10T16:24:57.799 INFO:tasks.rados.rados.0.smithi165.stdout:2882: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:24:57.799 INFO:tasks.rados.rados.0.smithi165.stdout:2882: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:24:57.799 INFO:tasks.rados.rados.0.smithi165.stdout:2882: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:24:57.799 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1432 (ObjNum 995 snap 299 seq_num 995) dirty exists 2024-09-10T16:24:57.817 INFO:tasks.rados.rados.0.smithi165.stdout:2882: done (1 left) 2024-09-10T16:24:57.817 INFO:tasks.rados.rados.0.smithi165.stdout:2883: done (0 left) 2024-09-10T16:24:57.817 INFO:tasks.rados.rados.0.smithi165.stdout:2884: snap_remove snap 301 2024-09-10T16:24:58.524 INFO:tasks.rados.rados.0.smithi165.stdout:2884: done (0 left) 2024-09-10T16:24:58.524 INFO:tasks.rados.rados.0.smithi165.stdout:2885: snap_create 2024-09-10T16:24:59.525 INFO:tasks.rados.rados.0.smithi165.stdout:2885: done (0 left) 2024-09-10T16:24:59.525 INFO:tasks.rados.rados.0.smithi165.stdout:2886: delete oid 7 current snap is 306 2024-09-10T16:24:59.533 INFO:tasks.rados.rados.0.smithi165.stdout:2886: done (0 left) 2024-09-10T16:24:59.533 INFO:tasks.rados.rados.0.smithi165.stdout:2887: read oid 7 snap -1 2024-09-10T16:24:59.533 INFO:tasks.rados.rados.0.smithi165.stdout:2887: expect deleted 2024-09-10T16:24:59.533 INFO:tasks.rados.rados.0.smithi165.stdout:2888: read oid 32 snap 304 2024-09-10T16:24:59.534 INFO:tasks.rados.rados.0.smithi165.stdout:2888: expect deleted 2024-09-10T16:24:59.534 INFO:tasks.rados.rados.0.smithi165.stdout:2889: read oid 12 snap -1 2024-09-10T16:24:59.534 INFO:tasks.rados.rados.0.smithi165.stdout:2889: expect (ObjNum 1008 snap 305 seq_num 1008) 2024-09-10T16:24:59.534 INFO:tasks.rados.rados.0.smithi165.stdout:2890: snap_remove snap 305 2024-09-10T16:25:00.129 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:25:00.129 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:25:00.130 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:25:00.133 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:25:00.133 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:25:00.133 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 10 monclient: get_monmap_and_config 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 10 monclient: build_initial_monmap 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 10 monclient: monmap: 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:00.242 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:00.242006+0000 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:00.242006+0000 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:00.243 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 1 Processor -- start 2024-09-10T16:25:00.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 1 -- start start 2024-09-10T16:25:00.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.241+0000 7f85a6ca9640 10 monclient: init 2024-09-10T16:25:00.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85a006cb70 0x7f85a0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f85a006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85a00a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f85a00a4ab0 0x7f85a01383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f85a00a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85a009ce50 con 0x7f85a00a46a0 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f85a006db60 con 0x7f85a006cb70 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f85a006cf50 con 0x7f85a00a4ab0 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a6ca9640 10 monclient(hunting): authenticate will time out at 1878.729522s 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85a006cb70 0x7f85a0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85a006cb70 0x7f85a0134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:32980/0 (socket says 172.21.15.164:32980) 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 -- 172.21.15.164:0/1645476415 learned_addr learned my addr 172.21.15.164:0/1645476415 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): get_auth_request con 0x7f85a006cb70 auth_method 0 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:00.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a5246640 1 --2- 172.21.15.164:0/1645476415 >> v2:172.21.15.165:3300/0 conn(0x7f85a00a4ab0 0x7f85a01383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient(hunting): handle_auth_done global_id 164547 payload 274 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 -- 172.21.15.164:0/1645476415 >> v2:172.21.15.165:3300/0 conn(0x7f85a00a4ab0 msgr2=0x7f85a01383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 --2- 172.21.15.164:0/1645476415 >> v2:172.21.15.165:3300/0 conn(0x7f85a00a4ab0 0x7f85a01383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 -- 172.21.15.164:0/1645476415 >> v1:172.21.15.165:6789/0 conn(0x7f85a00a46a0 legacy=0x7f85a0137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient: _finish_hunting 0 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 monclient: found mon.noname-a 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:00.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.242+0000 7f85a4a45640 1 -- 172.21.15.164:0/1645476415 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a009cb70 con 0x7f85a006cb70 2024-09-10T16:25:00.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.245+0000 7f85a4a45640 1 -- 172.21.15.164:0/1645476415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 msgr2=0x7f85a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:00.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.245+0000 7f85a4a45640 1 --2- 172.21.15.164:0/1645476415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a0134f70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f85940098a0 tx=0x7f859402a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a96028132a5bd54 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:00.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 1 -- 172.21.15.164:0/1645476415 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594037070 con 0x7f85a006cb70 2024-09-10T16:25:00.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: dump: 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:00.256 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 1 -- 172.21.15.164:0/1645476415 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8594032640 con 0x7f85a006cb70 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 5 monclient: authenticate success, global_id 164547 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 1 -- 172.21.15.164:0/1645476415 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594032a10 con 0x7f85a006cb70 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 4 set_mon_vals no callback set 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:00.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f859f7fe640 10 monclient: dump: 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 10 monclient: shutdown 2024-09-10T16:25:00.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 -- 172.21.15.164:0/1645476415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 msgr2=0x7f85a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/1645476415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a0134f70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f85940098a0 tx=0x7f859402a110 comp rx=0 tx=0).stop 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 -- 172.21.15.164:0/1645476415 shutdown_connections 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/1645476415 >> v2:172.21.15.165:3300/0 conn(0x7f85a00a4ab0 0x7f85a01383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/1645476415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a0134f70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 -- 172.21.15.164:0/1645476415 >> 172.21.15.164:0/1645476415 conn(0x7f85a00aa900 msgr2=0x7f85a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 -- 172.21.15.164:0/1645476415 shutdown_connections 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.246+0000 7f85a6ca9640 1 -- 172.21.15.164:0/1645476415 wait complete. 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.247+0000 7f85a6ca9640 10 monclient: build_initial_monmap 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.247+0000 7f85a6ca9640 10 monclient: monmap: 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:00.259 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.247+0000 7f85a6ca9640 1 Processor -- start 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.247+0000 7f85a6ca9640 1 -- start start 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.247+0000 7f85a6ca9640 10 monclient: init 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-a con 0x7f85a006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a00a46a0 0x7f85a00a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-b con 0x7f85a00a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): picked mon.noname-c con 0x7f85a00a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f85a0138a70 con 0x7f85a006cb70 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85a01388f0 con 0x7f85a00a46a0 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f85a0140040 con 0x7f85a00a4ab0 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a6ca9640 10 monclient(hunting): authenticate will time out at 1878.735319s 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:32992/0 (socket says 172.21.15.164:32992) 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 -- 172.21.15.164:0/925921841 learned_addr learned my addr 172.21.15.164:0/925921841 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): get_auth_request con 0x7f85a006cb70 auth_method 0 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 10 monclient(hunting): get_auth_request con 0x7f85a00a4ab0 auth_method 0 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 msgr2=0x7f85a00a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:00.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 msgr2=0x7f85a00a3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f859ffff640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a00a46a0 0x7f85a00a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a5246640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f859ffff640 10 monclient(hunting): get_auth_request con 0x7f85a00a46a0 auth_method 0 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f859ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f859ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f859ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient(hunting): handle_auth_done global_id 164550 payload 824 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 msgr2=0x7f85a00a3ea0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a00a46a0 msgr2=0x7f85a00a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a00a46a0 0x7f85a00a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient: _finish_hunting 0 2024-09-10T16:25:00.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 monclient: found mon.noname-a 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.248+0000 7f85a4a45640 1 -- 172.21.15.164:0/925921841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85a00a2150 con 0x7f85a006cb70 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.250+0000 7f85a4a45640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 msgr2=0x7f85a01cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.250+0000 7f85a4a45640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f85940090b0 tx=0x7f85940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594041070 con 0x7f85a006cb70 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: dump: 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:00.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8594004220 con 0x7f85a006cb70 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 5 monclient: authenticate success, global_id 164550 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 10 monclient: _renew_subs 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85a00a2430 con 0x7f85a006cb70 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:00.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8594037040 con 0x7f85a006cb70 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859d7fa640 10 monclient: dump: 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f859effd640 4 set_mon_vals no callback set 2024-09-10T16:25:00.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 10 monclient: _renew_subs 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.251+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85a00a00d0 con 0x7f85a006cb70 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f85a6ca9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f85a6ca9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f85a6ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85a0134f70 con 0x7f85a006cb70 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f859403d630 con 0x7f85a006cb70 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f859d7fa640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 0x7f8584040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(643..643 src has 494..643) ==== 4739+0+0 (secure 0 0 0) 0x7f85940731f0 con 0x7f85a006cb70 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f859ffff640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 0x7f8584040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:00.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.252+0000 7f859ffff640 10 monclient: get_auth_request con 0x7f858403e260 auth_method 0 2024-09-10T16:25:00.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.255+0000 7f859ffff640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 0x7f8584040720 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f85a00a0bc0 tx=0x7f858c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:00.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.257+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f85a009ce50 con 0x7f85a006cb70 2024-09-10T16:25:00.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.257+0000 7f859d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:00.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.257+0000 7f859d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:00.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f85a6ca9640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f85a6ca9640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f85a6ca9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85a01cd710 con 0x7f85a006cb70 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f859d7fa640 1 -- 172.21.15.164:0/925921841 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f8594003610 con 0x7f85a006cb70 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f859d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.498+0000 7f859d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:00.500 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":180,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 msgr2=0x7f8584040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 0x7f8584040720 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f85a00a0bc0 tx=0x7f858c009040 comp rx=0 tx=0).stop 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 10 monclient: shutdown 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 msgr2=0x7f85a01cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f85940090b0 tx=0x7f85940046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 shutdown_connections 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f858403e260 0x7f8584040720 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85a00a4ab0 0x7f85a00a3ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85a00a46a0 0x7f85a00a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 --2- 172.21.15.164:0/925921841 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85a006cb70 0x7f85a01cd1d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:00.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 >> 172.21.15.164:0/925921841 conn(0x7f85a00aa900 msgr2=0x7f85a00a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:00.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 shutdown_connections 2024-09-10T16:25:00.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:00.500+0000 7f85a6ca9640 1 -- 172.21.15.164:0/925921841 wait complete. 2024-09-10T16:25:00.537 INFO:tasks.rados.rados.0.smithi165.stdout:2890: done (3 left) 2024-09-10T16:25:00.538 INFO:tasks.rados.rados.0.smithi165.stdout:2891: read oid 46 snap 303 2024-09-10T16:25:00.538 INFO:tasks.rados.rados.0.smithi165.stdout:2891: expect deleted 2024-09-10T16:25:00.538 INFO:tasks.rados.rados.0.smithi165.stdout:2892: delete oid 17 current snap is 306 2024-09-10T16:25:00.538 INFO:tasks.rados.rados.0.smithi165.stdout:2889: expect (ObjNum 1008 snap 305 seq_num 1008) 2024-09-10T16:25:00.599 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 24165 2024-09-10T16:25:00.731 INFO:tasks.rados.rados.0.smithi165.stdout:2887: done (4 left) 2024-09-10T16:25:00.732 INFO:tasks.rados.rados.0.smithi165.stdout:2888: done (3 left) 2024-09-10T16:25:00.732 INFO:tasks.rados.rados.0.smithi165.stdout:2889: done (2 left) 2024-09-10T16:25:00.732 INFO:tasks.rados.rados.0.smithi165.stdout:2891: done (1 left) 2024-09-10T16:25:00.732 INFO:tasks.rados.rados.0.smithi165.stdout:2892: done (0 left) 2024-09-10T16:25:00.732 INFO:tasks.rados.rados.0.smithi165.stdout:2893: snap_remove snap 304 2024-09-10T16:25:03.513 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:03.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.621+0000 7f01d5d75640 10 monclient: get_monmap_and_config 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.621+0000 7f01d5d75640 10 monclient: build_initial_monmap 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.621+0000 7f01d5d75640 10 monclient: monmap: 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:03.622358+0000 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:03.622358+0000 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:03.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:03.624 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:03.624 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:03.624 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:03.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 Processor -- start 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 -- start start 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient: init 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f01d009c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f01d009d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-a con 0x7f01d013b5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d009fbc0 con 0x7f01d009d190 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f01d00a2a20 con 0x7f01d009c310 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f01d009f860 con 0x7f01d013b5d0 2024-09-10T16:25:03.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.622+0000 7f01d5d75640 10 monclient(hunting): authenticate will time out at 1882.109958s 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01ceffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f01d009d190 0x7f01d009f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36382/0 (socket says 172.21.15.164:36382) 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 learned_addr learned my addr 172.21.15.164:0/3752335515 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d013b5d0 auth_method 0 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:03.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.623+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:03.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.625+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 >> v1:172.21.15.164:6789/0 conn(0x7f01d009d190 legacy=0x7f01d009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:03.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.625+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3752335515 >> v1:172.21.15.165:6789/0 conn(0x7f01d009c310 legacy=0x7f01d009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:03.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:03.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d013b5d0 auth_method 0 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.824+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.825+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:03.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:03.825+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:25:04.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.226+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d013b5d0 auth_method 0 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:04.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:04.227+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d013b5d0 auth_method 0 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:05.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:05.028+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3341555205 0 0) 0x7f01d00a2a20 con 0x7f01d009c310 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 10 monclient(hunting): my global_id is 154561 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f01b8003520 con 0x7f01d009c310 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1915843909 0 0) 0x7f01b8003520 con 0x7f01d009c310 2024-09-10T16:25:06.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 msgr2=0x7f01d013d9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 --2- 172.21.15.164:0/3752335515 >> v2:172.21.15.164:3300/0 conn(0x7f01d013b5d0 0x7f01d013d9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.295+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 >> v1:172.21.15.164:6789/0 conn(0x7f01d009d190 legacy=0x7f01d009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d009c6f0 con 0x7f01d009c310 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f01c00019f0 con 0x7f01d009c310 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:06.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: dump: 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:25:06.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 >> v1:172.21.15.165:6789/0 conn(0x7f01d009c310 legacy=0x7f01d009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8003b10 0x7f01b8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): picked mon.a con 0x7f01b8003b10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 0x7f01b8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): picked mon.c con 0x7f01b8004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01b80099b0 0x7f01b800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): picked mon.b con 0x7f01b80099b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01d009f860 con 0x7f01b8003b10 2024-09-10T16:25:06.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01b800c3c0 con 0x7f01b80099b0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01b800c5c0 con 0x7f01b8004800 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ce7fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 0x7f01b8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): get_auth_request con 0x7f01b8004800 auth_method 0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8003b10 0x7f01b8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01b80099b0 0x7f01b800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): get_auth_request con 0x7f01b8003b10 auth_method 0 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01b80099b0 auth_method 0 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:06.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient(hunting): handle_auth_done global_id 154561 payload 293 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01b80099b0 msgr2=0x7f01b800bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01b80099b0 0x7f01b800bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8003b10 msgr2=0x7f01b8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8003b10 0x7f01b8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 monclient: found mon.c 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:06.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01b8003520 con 0x7f01b8004800 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01d5d75640 5 monclient: authenticate success, global_id 154561 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01d5d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cffff640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01b80099b0 0x7f01b800bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.296+0000 7f01cf7fe640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8003b10 0x7f01b8003f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:06.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.297+0000 7f01cffff640 1 -- 172.21.15.164:0/3752335515 reap_dead start 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.299+0000 7f01ceffd640 1 -- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 msgr2=0x7f01b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.299+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 0x7f01b8004c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01d006e780 tx=0x7f01b8005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87a6bb9588321067 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c4001760 con 0x7f01b8004800 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:06.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: dump: 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01d5d75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c402e070 con 0x7f01b8004800 2024-09-10T16:25:06.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01d5d75640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01d5d75640 4 set_mon_vals no callback set 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 1 -- 172.21.15.164:0/3752335515 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c40048a0 con 0x7f01b8004800 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.300+0000 7f01ce7fc640 10 monclient: dump: 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.315 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 10 monclient: shutdown 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 -- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 msgr2=0x7f01b8004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 0x7f01b8004c80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f01d006e780 tx=0x7f01b8005830 comp rx=0 tx=0).stop 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 -- 172.21.15.164:0/3752335515 shutdown_connections 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3752335515 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01b8004800 0x7f01b8004c80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 -- 172.21.15.164:0/3752335515 >> 172.21.15.164:0/3752335515 conn(0x7f01d00a3110 msgr2=0x7f01d00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 -- 172.21.15.164:0/3752335515 shutdown_connections 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.301+0000 7f01d5d75640 1 -- 172.21.15.164:0/3752335515 wait complete. 2024-09-10T16:25:06.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.302+0000 7f01d5d75640 10 monclient: build_initial_monmap 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.302+0000 7f01d5d75640 10 monclient: monmap: 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.302+0000 7f01d5d75640 1 Processor -- start 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 -- start start 2024-09-10T16:25:06.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient: init 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-a con 0x7f01b8004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 0x7f01d006f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-b con 0x7f01d006ed80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): picked mon.noname-c con 0x7f01d006f700 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f01d00a20b0 con 0x7f01b8004800 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f01d009a7b0 con 0x7f01d006ed80 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:06.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f01d00a2230 con 0x7f01d006f700 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01d5d75640 10 monclient(hunting): authenticate will time out at 1884.790660s 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38186/0 (socket says 172.21.15.164:38186) 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3186398431 learned_addr learned my addr 172.21.15.164:0/3186398431 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): get_auth_request con 0x7f01b8004800 auth_method 0 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 10 monclient(hunting): get_auth_request con 0x7f01d006f700 auth_method 0 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 0x7f01d006f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01ceffd640 10 monclient(hunting): get_auth_request con 0x7f01d006ed80 auth_method 0 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01ceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 msgr2=0x7f01d01d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 msgr2=0x7f01d01d37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.303+0000 7f01cffff640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cffff640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:06.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01ceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 10 monclient(hunting): handle_auth_done global_id 164553 payload 824 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 msgr2=0x7f01d01d37c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 msgr2=0x7f01d006f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 0x7f01d006f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 monclient: found mon.noname-a 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3186398431 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01d00a4e10 con 0x7f01b8004800 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.304+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 0x7f01d006f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:06.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01cf7fe640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 msgr2=0x7f01d006e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01cf7fe640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c900 tx=0x7f01c000cdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c0017020 con 0x7f01b8004800 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: dump: 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:06.322 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: _finish_auth 0 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f01c00075b0 con 0x7f01b8004800 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 5 monclient: authenticate success, global_id 164553 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 10 monclient: _renew_subs 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01d0070190 con 0x7f01b8004800 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01c000fa70 con 0x7f01b8004800 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01b7fff640 10 monclient: dump: 2024-09-10T16:25:06.323 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 10 monclient: _renew_subs 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:06.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01d0095ec0 con 0x7f01b8004800 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.306+0000 7f01cdffb640 4 set_mon_vals no callback set 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.307+0000 7f01d5d75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01d5d75640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01d5d75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f01c001a020 con 0x7f01b8004800 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01d009f620 con 0x7f01b8004800 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01b7fff640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 0x7f01bc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(644..644 src has 494..644) ==== 4783+0+0 (secure 0 0 0) 0x7f01c0052b90 con 0x7f01b8004800 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 0x7f01bc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.308+0000 7f01ceffd640 10 monclient: get_auth_request con 0x7f01bc03e210 auth_method 0 2024-09-10T16:25:06.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.311+0000 7f01ceffd640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 0x7f01bc0406d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005000 tx=0x7f01c4007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:06.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.313+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f01d009f620 con 0x7f01b8004800 2024-09-10T16:25:06.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.313+0000 7f01b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:06.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.313+0000 7f01b7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:06.356 INFO:tasks.rados.rados.0.smithi165.stdout:2893: done (0 left) 2024-09-10T16:25:06.356 INFO:tasks.rados.rados.0.smithi165.stdout:2894: snap_create 2024-09-10T16:25:06.481 INFO:tasks.rados.rados.0.smithi165.stdout:2894: done (0 left) 2024-09-10T16:25:06.481 INFO:tasks.rados.rados.0.smithi165.stdout:2895: snap_remove snap 302 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.566+0000 7f01d5d75640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.566+0000 7f01d5d75640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.566+0000 7f01d5d75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.566+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f01d00008d0 con 0x7f01b8004800 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.567+0000 7f01b7fff640 1 -- 172.21.15.164:0/3186398431 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f01c0021030 con 0x7f01b8004800 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.567+0000 7f01b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:06.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.567+0000 7f01b7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:06.569 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":184,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:06.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 msgr2=0x7f01bc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 0x7f01bc0406d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f01c4005000 tx=0x7f01c4007000 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 10 monclient: shutdown 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 msgr2=0x7f01d006e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f01c000c900 tx=0x7f01c000cdc0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 shutdown_connections 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f01bc03e210 0x7f01bc0406d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f01d006f700 0x7f01d01d37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f01d006ed80 0x7f01d006f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 --2- 172.21.15.164:0/3186398431 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f01b8004800 0x7f01d006e840 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 >> 172.21.15.164:0/3186398431 conn(0x7f01d00a3110 msgr2=0x7f01d013a880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:06.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 shutdown_connections 2024-09-10T16:25:06.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:06.569+0000 7f01d5d75640 1 -- 172.21.15.164:0/3186398431 wait complete. 2024-09-10T16:25:07.491 INFO:tasks.rados.rados.0.smithi165.stdout:2895: done (0 left) 2024-09-10T16:25:07.491 INFO:tasks.rados.rados.0.smithi165.stdout:2896: write oid 23 current snap is 307 2024-09-10T16:25:07.491 INFO:tasks.rados.rados.0.smithi165.stdout:2896: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2024-09-10T16:25:07.497 INFO:tasks.rados.rados.0.smithi165.stdout:2896: writing smithi16534841-23 from 608684 to 1035621 tid 1 2024-09-10T16:25:07.504 INFO:tasks.rados.rados.0.smithi165.stdout:2896: writing smithi16534841-23 from 1803189 to 2372877 tid 2 2024-09-10T16:25:07.508 INFO:tasks.rados.rados.0.smithi165.stdout:2896: writing smithi16534841-23 from 2924304 to 3008684 tid 3 2024-09-10T16:25:07.508 INFO:tasks.rados.rados.0.smithi165.stdout:2897: delete oid 22 current snap is 307 2024-09-10T16:25:07.513 INFO:tasks.rados.rados.0.smithi165.stdout:2896: finishing write tid 1 to smithi16534841-23 2024-09-10T16:25:07.514 INFO:tasks.rados.rados.0.smithi165.stdout:2896: finishing write tid 2 to smithi16534841-23 2024-09-10T16:25:07.514 INFO:tasks.rados.rados.0.smithi165.stdout:2897: done (1 left) 2024-09-10T16:25:07.514 INFO:tasks.rados.rados.0.smithi165.stdout:2898: write (excl) oid 3 current snap is 307 2024-09-10T16:25:07.514 INFO:tasks.rados.rados.0.smithi165.stdout:2898: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2024-09-10T16:25:07.520 INFO:tasks.rados.rados.0.smithi165.stdout:2898: writing smithi16534841-3 from 642298 to 1219733 tid 1 2024-09-10T16:25:07.527 INFO:tasks.rados.rados.0.smithi165.stdout:2898: writing smithi16534841-3 from 1920153 to 2493510 tid 2 2024-09-10T16:25:07.530 INFO:tasks.rados.rados.0.smithi165.stdout:2898: writing smithi16534841-3 from 2979150 to 3042298 tid 3 2024-09-10T16:25:07.530 INFO:tasks.rados.rados.0.smithi165.stdout:2899: write oid 42 current snap is 307 2024-09-10T16:25:07.530 INFO:tasks.rados.rados.0.smithi165.stdout:2896: finishing write tid 3 to smithi16534841-23 2024-09-10T16:25:07.530 INFO:tasks.rados.rados.0.smithi165.stdout:2899: seq_num 1015 ranges {659105=652684,1862282=717015} 2024-09-10T16:25:07.537 INFO:tasks.rados.rados.0.smithi165.stdout:2899: writing smithi16534841-42 from 659105 to 1311789 tid 1 2024-09-10T16:25:07.545 INFO:tasks.rados.rados.0.smithi165.stdout:2899: writing smithi16534841-42 from 1862282 to 2579297 tid 2 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2900: snap_create 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: finishing write tid 4 to smithi16534841-23 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: finishing write tid 5 to smithi16534841-23 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: oid 23 updating version 0 to 1665 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: oid 23 version 1665 is already newer than 1664 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: oid 23 version 1665 is already newer than 1663 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: oid 23 updating version 1665 to 1666 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1666 (ObjNum 1012 snap 307 seq_num 1012) dirty exists 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2896: left oid 23 (ObjNum 1012 snap 307 seq_num 1012) 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2898: finishing write tid 1 to smithi16534841-3 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2898: finishing write tid 2 to smithi16534841-3 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2898: finishing write tid 3 to smithi16534841-3 2024-09-10T16:25:07.546 INFO:tasks.rados.rados.0.smithi165.stdout:2898: finishing write tid 4 to smithi16534841-3 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: finishing write tid 5 to smithi16534841-3 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: oid 3 updating version 0 to 1572 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: oid 3 version 1572 is already newer than 1570 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: oid 3 version 1572 is already newer than 1571 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: oid 3 version 1572 is already newer than 1569 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1572 (ObjNum 1014 snap 307 seq_num 1014) dirty exists 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2898: left oid 3 (ObjNum 1014 snap 307 seq_num 1014) 2024-09-10T16:25:07.547 INFO:tasks.rados.rados.0.smithi165.stdout:2899: finishing write tid 1 to smithi16534841-42 2024-09-10T16:25:07.548 INFO:tasks.rados.rados.0.smithi165.stdout:2899: finishing write tid 2 to smithi16534841-42 2024-09-10T16:25:07.548 INFO:tasks.rados.rados.0.smithi165.stdout:2899: finishing write tid 3 to smithi16534841-42 2024-09-10T16:25:07.548 INFO:tasks.rados.rados.0.smithi165.stdout:2899: finishing write tid 4 to smithi16534841-42 2024-09-10T16:25:07.549 INFO:tasks.rados.rados.0.smithi165.stdout:2899: oid 42 updating version 0 to 1296 2024-09-10T16:25:07.549 INFO:tasks.rados.rados.0.smithi165.stdout:2899: oid 42 updating version 1296 to 1298 2024-09-10T16:25:07.549 INFO:tasks.rados.rados.0.smithi165.stdout:2899: oid 42 version 1298 is already newer than 1297 2024-09-10T16:25:07.549 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1298 (ObjNum 1015 snap 307 seq_num 1015) dirty exists 2024-09-10T16:25:07.549 INFO:tasks.rados.rados.0.smithi165.stdout:2899: left oid 42 (ObjNum 1015 snap 307 seq_num 1015) 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2896: done (3 left) 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2898: done (2 left) 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2899: done (1 left) 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2900: done (0 left) 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2901: read oid 24 snap -1 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2901: expect deleted 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:2902: rollback oid 45 current snap is 308 2024-09-10T16:25:08.505 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 45 to 306 2024-09-10T16:25:08.506 INFO:tasks.rados.rados.0.smithi165.stdout:2903: write (excl) oid 12 current snap is 308 2024-09-10T16:25:08.506 INFO:tasks.rados.rados.0.smithi165.stdout:2903: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2024-09-10T16:25:08.513 INFO:tasks.rados.rados.0.smithi165.stdout:2903: writing smithi16534841-12 from 675912 to 1403845 tid 1 2024-09-10T16:25:08.518 INFO:tasks.rados.rados.0.smithi165.stdout:2903: writing smithi16534841-12 from 1920764 to 2381437 tid 2 2024-09-10T16:25:08.522 INFO:tasks.rados.rados.0.smithi165.stdout:2903: writing smithi16534841-12 from 2801290 to 3075912 tid 3 2024-09-10T16:25:08.522 INFO:tasks.rados.rados.0.smithi165.stdout:2904: copy_from oid 30 from oid 10 current snap is 308 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2902: finishing rollback tid 0 to smithi16534841-45 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2902: finishing rollback tid 1 to smithi16534841-45 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2902: finishing rollback tid 2 to smithi16534841-45 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1678 (ObjNum 1005 snap 300 seq_num 1005) dirty exists 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2903: finishing write tid 1 to smithi16534841-12 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2903: finishing write tid 2 to smithi16534841-12 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2901: done (3 left) 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2902: done (2 left) 2024-09-10T16:25:08.523 INFO:tasks.rados.rados.0.smithi165.stdout:2905: delete oid 6 current snap is 308 2024-09-10T16:25:08.524 INFO:tasks.rados.rados.0.smithi165.stdout:2903: finishing write tid 3 to smithi16534841-12 2024-09-10T16:25:08.525 INFO:tasks.rados.rados.0.smithi165.stdout:2903: finishing write tid 4 to smithi16534841-12 2024-09-10T16:25:08.525 INFO:tasks.rados.rados.0.smithi165.stdout:2903: finishing write tid 5 to smithi16534841-12 2024-09-10T16:25:08.525 INFO:tasks.rados.rados.0.smithi165.stdout:2903: oid 12 updating version 0 to 1629 2024-09-10T16:25:08.526 INFO:tasks.rados.rados.0.smithi165.stdout:2903: oid 12 updating version 1629 to 1632 2024-09-10T16:25:08.526 INFO:tasks.rados.rados.0.smithi165.stdout:2903: oid 12 version 1632 is already newer than 1630 2024-09-10T16:25:08.526 INFO:tasks.rados.rados.0.smithi165.stdout:2903: oid 12 version 1632 is already newer than 1631 2024-09-10T16:25:08.526 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1632 (ObjNum 1016 snap 308 seq_num 1016) dirty exists 2024-09-10T16:25:08.526 INFO:tasks.rados.rados.0.smithi165.stdout:2903: left oid 12 (ObjNum 1016 snap 308 seq_num 1016) 2024-09-10T16:25:08.528 INFO:tasks.rados.rados.0.smithi165.stdout:2903: done (2 left) 2024-09-10T16:25:08.528 INFO:tasks.rados.rados.0.smithi165.stdout:2905: done (1 left) 2024-09-10T16:25:08.528 INFO:tasks.rados.rados.0.smithi165.stdout:2906: snap_remove snap 307 2024-09-10T16:25:09.582 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.690+0000 7fa4baa70640 10 monclient: get_monmap_and_config 2024-09-10T16:25:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.690+0000 7fa4baa70640 10 monclient: build_initial_monmap 2024-09-10T16:25:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.690+0000 7fa4baa70640 10 monclient: monmap: 2024-09-10T16:25:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:09.691 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:09.691040+0000 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:09.691040+0000 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:09.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.690+0000 7fa4baa70640 1 Processor -- start 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 1 -- start start 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient: init 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4b409c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa4b409d160 0x7fa4b409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b409d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa4b413a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4b409a430 con 0x7fa4b413a5c0 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4b40a24e0 con 0x7fa4b409c180 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa4b40a2bf0 con 0x7fa4b409d160 2024-09-10T16:25:09.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4baa70640 10 monclient(hunting): authenticate will time out at 1888.178631s 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa4b409d160 0x7fa4b409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b900d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa4b413a5c0 0x7fa4b413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45736/0 (socket says 172.21.15.164:45736) 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b900d640 1 -- 172.21.15.164:0/928633740 learned_addr learned my addr 172.21.15.164:0/928633740 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b3fff640 10 monclient(hunting): get_auth_request con 0x7fa4b409d160 auth_method 0 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.691+0000 7fa4b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:09.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_done global_id 164556 payload 274 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 1 -- 172.21.15.164:0/928633740 >> v1:172.21.15.165:6789/0 conn(0x7fa4b409c180 legacy=0x7fa4b409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 1 -- 172.21.15.164:0/928633740 >> v1:172.21.15.164:6789/0 conn(0x7fa4b413a5c0 legacy=0x7fa4b413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 1 monclient: found mon.noname-a 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.692+0000 7fa4b3fff640 1 -- 172.21.15.164:0/928633740 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b409c560 con 0x7fa4b409d160 2024-09-10T16:25:09.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.695+0000 7fa4b3fff640 1 -- 172.21.15.164:0/928633740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409d160 msgr2=0x7fa4b409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:09.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.695+0000 7fa4b3fff640 1 --2- 172.21.15.164:0/928633740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409d160 0x7fa4b409f150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009dc0 tx=0x7fa49c02a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d29e584de91d433 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 1 -- 172.21.15.164:0/928633740 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c007a30 con 0x7fa4b409d160 2024-09-10T16:25:09.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: dump: 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4baa70640 5 monclient: authenticate success, global_id 164556 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4baa70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 1 -- 172.21.15.164:0/928633740 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49c007bd0 con 0x7fa4b409d160 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 1 -- 172.21.15.164:0/928633740 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c031400 con 0x7fa4b409d160 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4baa70640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4baa70640 4 set_mon_vals no callback set 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.698+0000 7fa4b37fe640 10 monclient: dump: 2024-09-10T16:25:09.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:09.702 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2906: done (1 left) 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2907: copy_from oid 37 from oid 1 current snap is 308 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2904: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2904: finishing copy_from to smithi16534841-30 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1551 (ObjNum 977 snap 297 seq_num 977) dirty exists 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2904: done (1 left) 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2908: read oid 14 snap 303 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2908: expect (ObjNum 966 snap 292 seq_num 966) 2024-09-10T16:25:09.704 INFO:tasks.rados.rados.0.smithi165.stdout:2909: snap_remove snap 306 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 10 monclient: shutdown 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 -- 172.21.15.164:0/928633740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409d160 msgr2=0x7fa4b409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 --2- 172.21.15.164:0/928633740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409d160 0x7fa4b409f150 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fa49c009dc0 tx=0x7fa49c02a4d0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 -- 172.21.15.164:0/928633740 shutdown_connections 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 --2- 172.21.15.164:0/928633740 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409d160 0x7fa4b409f150 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 -- 172.21.15.164:0/928633740 >> 172.21.15.164:0/928633740 conn(0x7fa4b40a3110 msgr2=0x7fa4b40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 -- 172.21.15.164:0/928633740 shutdown_connections 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.699+0000 7fa4baa70640 1 -- 172.21.15.164:0/928633740 wait complete. 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.700+0000 7fa4baa70640 10 monclient: build_initial_monmap 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.700+0000 7fa4baa70640 10 monclient: monmap: 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:09.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.700+0000 7fa4baa70640 1 Processor -- start 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.700+0000 7fa4baa70640 1 -- start start 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.700+0000 7fa4baa70640 10 monclient: init 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-a con 0x7fa4b409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b409d160 0x7fa4b41cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-b con 0x7fa4b409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): picked mon.noname-c con 0x7fa4b413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:09.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4074f90 con 0x7fa4b409c180 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa4b4074e10 con 0x7fa4b409d160 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa4b409f690 con 0x7fa4b413a5c0 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4baa70640 10 monclient(hunting): authenticate will time out at 1888.188254s 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38202/0 (socket says 172.21.15.164:38202) 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 -- 172.21.15.164:0/3726752154 learned_addr learned my addr 172.21.15.164:0/3726752154 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): get_auth_request con 0x7fa4b409c180 auth_method 0 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:09.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 10 monclient(hunting): get_auth_request con 0x7fa4b413a5c0 auth_method 0 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b409d160 0x7fa4b41cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): get_auth_request con 0x7fa4b409d160 auth_method 0 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:09.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 msgr2=0x7fa4b406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 msgr2=0x7fa4b406e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b900d640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient(hunting): handle_auth_done global_id 164559 payload 824 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 msgr2=0x7fa4b406e800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b409d160 msgr2=0x7fa4b41cdf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b409d160 0x7fa4b41cdf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient: _finish_hunting 0 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 monclient: found mon.noname-a 2024-09-10T16:25:09.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.701+0000 7fa4b880c640 1 -- 172.21.15.164:0/3726752154 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4b4132670 con 0x7fa4b409c180 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b880c640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 msgr2=0x7fa4b41cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b880c640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c460 tx=0x7fa4a400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a40119a0 con 0x7fa4b409c180 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: dump: 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:09.715 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: _finish_auth 0 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4a4017070 con 0x7fa4b409c180 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 5 monclient: authenticate success, global_id 164559 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 10 monclient: _renew_subs 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4b4132950 con 0x7fa4b409c180 2024-09-10T16:25:09.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4a4011dc0 con 0x7fa4b409c180 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b17fa640 10 monclient: dump: 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:09.717 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4b2ffd640 4 set_mon_vals no callback set 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 10 monclient: _renew_subs 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.704+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4b40a5f50 con 0x7fa4b409c180 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa4a401c460 con 0x7fa4b409c180 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4baa70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4baa70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4baa70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4b409ea40 con 0x7fa4b409c180 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4b17fa640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 0x7fa4940406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(649..649 src has 494..649) ==== 4799+0+0 (secure 0 0 0) 0x7fa4a4052b70 con 0x7fa4b409c180 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4b3fff640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 0x7fa4940406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.705+0000 7fa4b3fff640 10 monclient: get_auth_request con 0x7fa49403e210 auth_method 0 2024-09-10T16:25:09.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.708+0000 7fa4b3fff640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 0x7fa4940406d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa4b409a720 tx=0x7fa49c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:09.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.710+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa4a401ac60 con 0x7fa4b409c180 2024-09-10T16:25:09.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.710+0000 7fa4b17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:09.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.710+0000 7fa4b17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.954+0000 7fa4baa70640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.954+0000 7fa4baa70640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.954+0000 7fa4baa70640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:09.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.954+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4b4077260 con 0x7fa4b409c180 2024-09-10T16:25:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.955+0000 7fa4b17fa640 1 -- 172.21.15.164:0/3726752154 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fa4a4051560 con 0x7fa4b409c180 2024-09-10T16:25:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.955+0000 7fa4b17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:09.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.955+0000 7fa4b17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:09.957 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":186,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 msgr2=0x7fa4940406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 0x7fa4940406d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fa4b409a720 tx=0x7fa49c02f000 comp rx=0 tx=0).stop 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 10 monclient: shutdown 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 msgr2=0x7fa4b41cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:09.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa4a400c460 tx=0x7fa4a400c920 comp rx=0 tx=0).stop 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 shutdown_connections 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa49403e210 0x7fa4940406d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa4b413a5c0 0x7fa4b406e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa4b409d160 0x7fa4b41cdf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 --2- 172.21.15.164:0/3726752154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa4b409c180 0x7fa4b41cda00 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 >> 172.21.15.164:0/3726752154 conn(0x7fa4b40a3110 msgr2=0x7fa4b4139bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 shutdown_connections 2024-09-10T16:25:09.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:09.957+0000 7fa4baa70640 1 -- 172.21.15.164:0/3726752154 wait complete. 2024-09-10T16:25:09.969 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:25:09.970 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:25:09.970 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:25:10.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.068+0000 7fb383d4b640 10 monclient: get_monmap_and_config 2024-09-10T16:25:10.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.068+0000 7fb383d4b640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.068+0000 7fb383d4b640 10 monclient: monmap: 2024-09-10T16:25:10.069 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:10.069055+0000 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:10.069055+0000 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:10.070 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.068+0000 7fb383d4b640 1 Processor -- start 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 -- start start 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient: init 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb37c06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:10.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb37c134f70 0x7fb37c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb37c134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb37c13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb37c09ce50 con 0x7fb37c06cb70 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb37c06db60 con 0x7fb37c134f70 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c06cf50 con 0x7fb37c13d560 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb383d4b640 10 monclient(hunting): authenticate will time out at 1888.556546s 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb381ae7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fb37c06cb70 0x7fb37c0a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45748/0 (socket says 172.21.15.164:45748) 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb381ae7640 1 -- 172.21.15.164:0/2058641247 learned_addr learned my addr 172.21.15.164:0/2058641247 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 10 monclient(hunting): get_auth_request con 0x7fb37c13d560 auth_method 0 2024-09-10T16:25:10.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 1 -- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 1 -- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3822e8640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3812e6640 1 --2- 172.21.15.164:0/2058641247 >> v2:172.21.15.165:3300/0 conn(0x7fb37c134f70 0x7fb37c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3812e6640 10 monclient(hunting): get_auth_request con 0x7fb37c134f70 auth_method 0 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3812e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3812e6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.069+0000 7fb3812e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient(hunting): handle_auth_done global_id 154570 payload 274 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 1 -- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c13f950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 1 -- 172.21.15.164:0/2058641247 >> v1:172.21.15.164:6789/0 conn(0x7fb37c06cb70 legacy=0x7fb37c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 1 monclient: found mon.noname-c 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:10.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.070+0000 7fb3812e6640 1 -- 172.21.15.164:0/2058641247 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c135380 con 0x7fb37c134f70 2024-09-10T16:25:10.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb3812e6640 1 -- 172.21.15.164:0/2058641247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c134f70 msgr2=0x7fb37c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb3812e6640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c134f70 0x7fb37c13d020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0097d0 tx=0x7fb36c02a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c18680d444d11047 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 1 -- 172.21.15.164:0/2058641247 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c037070 con 0x7fb37c134f70 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: dump: 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.084 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 1 -- 172.21.15.164:0/2058641247 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c032440 con 0x7fb37c134f70 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 1 -- 172.21.15.164:0/2058641247 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c032810 con 0x7fb37c134f70 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb380ae5640 10 monclient: dump: 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.085 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.086 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.087 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb383d4b640 5 monclient: authenticate success, global_id 154570 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb383d4b640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.073+0000 7fb383d4b640 4 set_mon_vals no callback set 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 10 monclient: shutdown 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 -- 172.21.15.164:0/2058641247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c134f70 msgr2=0x7fb37c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c134f70 0x7fb37c13d020 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0097d0 tx=0x7fb36c02a000 comp rx=0 tx=0).stop 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 -- 172.21.15.164:0/2058641247 shutdown_connections 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c13f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 --2- 172.21.15.164:0/2058641247 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c134f70 0x7fb37c13d020 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 -- 172.21.15.164:0/2058641247 >> 172.21.15.164:0/2058641247 conn(0x7fb37c0aa900 msgr2=0x7fb37c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 -- 172.21.15.164:0/2058641247 shutdown_connections 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 1 -- 172.21.15.164:0/2058641247 wait complete. 2024-09-10T16:25:10.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.074+0000 7fb383d4b640 10 monclient: monmap: 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 Processor -- start 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 -- start start 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient: init 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c06cb70 0x7fb37c12e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fb37c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 0x7fb37c12e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fb37c134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c138db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fb37c13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c0a4db0 con 0x7fb37c134f70 2024-09-10T16:25:10.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c09d130 con 0x7fb37c06cb70 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb37c13fe90 con 0x7fb37c13d560 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb383d4b640 10 monclient(hunting): authenticate will time out at 1888.562580s 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 0x7fb37c12e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c138db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 0x7fb37c12e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38212/0 (socket says 172.21.15.164:38212) 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 -- 172.21.15.164:0/3031734601 learned_addr learned my addr 172.21.15.164:0/3031734601 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): get_auth_request con 0x7fb37c134f70 auth_method 0 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 10 monclient(hunting): get_auth_request con 0x7fb37c13d560 auth_method 0 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c138db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3822e8640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c138db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c06cb70 0x7fb37c12e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient(hunting): handle_auth_done global_id 164562 payload 824 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 msgr2=0x7fb37c138db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb37c13d560 0x7fb37c138db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c06cb70 msgr2=0x7fb37c12e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c06cb70 0x7fb37c12e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 monclient: found mon.noname-a 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb3812e6640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb37c06e720 con 0x7fb37c134f70 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb381ae7640 10 monclient: get_auth_request con 0x7fb37c06cb70 auth_method 0 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.075+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb37c06cb70 0x7fb37c12e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3812e6640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 msgr2=0x7fb37c12e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3812e6640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 0x7fb37c12e9b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0098f0 tx=0x7fb36c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c041070 con 0x7fb37c134f70 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: dump: 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.094 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb36c032e50 con 0x7fb37c134f70 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 5 monclient: authenticate success, global_id 164562 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 10 monclient: _renew_subs 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb37c1385b0 con 0x7fb37c134f70 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb36c037040 con 0x7fb37c134f70 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb3727fc640 10 monclient: dump: 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 10 monclient: _renew_subs 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb373fff640 4 set_mon_vals no callback set 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.078+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb37c13b420 con 0x7fb37c134f70 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb383d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb383d4b640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 0x7fb37c1396d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c13b940 con 0x7fb37c1392f0 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 0x7fb37c1396d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient: get_auth_request con 0x7fb37c1392f0 auth_method 0 2024-09-10T16:25:10.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb36c044070 con 0x7fb37c134f70 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 10 monclient(hunting): handle_auth_done global_id 164565 payload 824 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb381ae7640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb37c139c10 con 0x7fb37c1392f0 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb3727fc640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb36003e240 0x7fb360040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.079+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(649..649 src has 494..649) ==== 4799+0+0 (secure 0 0 0) 0x7fb36c0736d0 con 0x7fb37c134f70 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.080+0000 7fb3822e8640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb36003e240 0x7fb360040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.080+0000 7fb3822e8640 10 monclient: get_auth_request con 0x7fb36003e240 auth_method 0 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb381ae7640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 msgr2=0x7fb37c1396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 0x7fb37c1396d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0aa110 tx=0x7fb37800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378015070 con 0x7fb37c1392f0 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb3727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb3727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.081+0000 7fb3727fc640 10 monclient: dump: 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.099 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb378011a10 con 0x7fb37c1392f0 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 msgr2=0x7fb37c1396d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 0x7fb37c1396d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0aa110 tx=0x7fb37800c950 comp rx=0 tx=0).stop 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 msgr2=0x7fb37c1396d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3727fc640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c1392f0 0x7fb37c1396d0 secure :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0aa110 tx=0x7fb37800c950 comp rx=0 tx=0).stop 2024-09-10T16:25:10.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.082+0000 7fb3822e8640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb36003e240 0x7fb360040700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb37c077650 tx=0x7fb37400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb383d4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb383d4b640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 0x7fb37c130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb383d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb37c0a46a0 con 0x7fb37c130250 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 0x7fb37c130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient: get_auth_request con 0x7fb37c130250 auth_method 0 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 10 monclient(hunting): handle_auth_done global_id 164568 payload 824 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.088+0000 7fb381ae7640 1 -- 172.21.15.164:0/3031734601 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb37c0770f0 con 0x7fb37c130250 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb381ae7640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 msgr2=0x7fb37c130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb381ae7640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 0x7fb37c130660 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb378017000 tx=0x7fb3780058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb378011e40 con 0x7fb37c130250 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb3727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb3727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.090+0000 7fb3727fc640 10 monclient: dump: 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7fb378011e40 con 0x7fb37c130250 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 msgr2=0x7fb37c130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 0x7fb37c130660 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb378017000 tx=0x7fb3780058d0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 msgr2=0x7fb37c130660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3727fc640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c130250 0x7fb37c130660 secure :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fb378017000 tx=0x7fb3780058d0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 280793, 2024-09-10T16:25:10.105 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.106 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.107 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:10.108 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.109 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:10.110 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:10.111 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:10.112 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:10.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:10.114 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.115 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:10.116 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.117 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:10.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.118 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.118 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:10.118 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:10.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb36003e240 msgr2=0x7fb360040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb36003e240 0x7fb360040700 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fb37c077650 tx=0x7fb37400b040 comp rx=0 tx=0).stop 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 10 monclient: shutdown 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 msgr2=0x7fb37c12e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 --2- 172.21.15.164:0/3031734601 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb37c134f70 0x7fb37c12e9b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fb36c0098f0 tx=0x7fb36c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3822e8640 1 -- 172.21.15.164:0/3031734601 reap_dead start 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb3822e8640 1 -- 172.21.15.164:0/3031734601 reap_dead start 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 shutdown_connections 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 >> 172.21.15.164:0/3031734601 conn(0x7fb37c0aa900 msgr2=0x7fb37c0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 shutdown_connections 2024-09-10T16:25:10.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.091+0000 7fb383d4b640 1 -- 172.21.15.164:0/3031734601 wait complete. 2024-09-10T16:25:10.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:10.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:25:10.121 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.217+0000 7f53b3c47640 10 monclient: get_monmap_and_config 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.217+0000 7f53b3c47640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.217+0000 7f53b3c47640 10 monclient: monmap: 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:10.218600+0000 2024-09-10T16:25:10.219 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:10.218600+0000 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:10.220 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.218+0000 7f53b3c47640 1 Processor -- start 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.218+0000 7f53b3c47640 1 -- start start 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.218+0000 7f53b3c47640 10 monclient: init 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53ac1329f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-b con 0x7f53ac0a2f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53ac13b2b0 0x7f53ac13d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-a con 0x7f53ac13b2b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53ac1334e0 con 0x7f53ac1329f0 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f53ac0a0730 con 0x7f53ac13b2b0 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac0a21a0 con 0x7f53ac0a2f10 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b3c47640 10 monclient(hunting): authenticate will time out at 1888.706151s 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53ac13b2b0 0x7f53ac13d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f53ac13b2b0 0x7f53ac13d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38242/0 (socket says 172.21.15.164:38242) 2024-09-10T16:25:10.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52072/0 (socket says 172.21.15.164:52072) 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 -- 172.21.15.164:0/59141760 learned_addr learned my addr 172.21.15.164:0/59141760 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): get_auth_request con 0x7f53ac13b2b0 auth_method 0 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 10 monclient(hunting): get_auth_request con 0x7f53ac0a2f10 auth_method 0 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 -- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 msgr2=0x7f53ac13ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:25:10.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 -- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 msgr2=0x7f53ac13ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b11e2640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient(hunting): handle_auth_done global_id 164571 payload 274 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 -- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 msgr2=0x7f53ac13ad70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 -- 172.21.15.164:0/59141760 >> v1:172.21.15.165:6789/0 conn(0x7f53ac1329f0 legacy=0x7f53ac132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 monclient: found mon.noname-a 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.219+0000 7f53b21e4640 1 -- 172.21.15.164:0/59141760 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac0a2480 con 0x7f53ac13b2b0 2024-09-10T16:25:10.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b21e4640 1 -- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac13b2b0 msgr2=0x7f53ac13d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b21e4640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac13b2b0 0x7f53ac13d6a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009e50 tx=0x7f53a402a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5da0a55045332e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 1 -- 172.21.15.164:0/59141760 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4036070 con 0x7f53ac13b2b0 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: dump: 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.234 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 1 -- 172.21.15.164:0/59141760 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53a4007800 con 0x7f53ac13b2b0 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b3c47640 5 monclient: authenticate success, global_id 164571 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b3c47640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 1 -- 172.21.15.164:0/59141760 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a4007bd0 con 0x7f53ac13b2b0 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b3c47640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:10.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b3c47640 4 set_mon_vals no callback set 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.222+0000 7f53b09e1640 10 monclient: dump: 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.236 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 10 monclient: shutdown 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 -- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac13b2b0 msgr2=0x7f53ac13d6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac13b2b0 0x7f53ac13d6a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f53a4009e50 tx=0x7f53a402a560 comp rx=0 tx=0).stop 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 -- 172.21.15.164:0/59141760 shutdown_connections 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac0a2f10 0x7f53ac13ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 --2- 172.21.15.164:0/59141760 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac13b2b0 0x7f53ac13d6a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 -- 172.21.15.164:0/59141760 >> 172.21.15.164:0/59141760 conn(0x7f53ac0a7320 msgr2=0x7f53ac0a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 -- 172.21.15.164:0/59141760 shutdown_connections 2024-09-10T16:25:10.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 1 -- 172.21.15.164:0/59141760 wait complete. 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.223+0000 7f53b3c47640 10 monclient: monmap: 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 Processor -- start 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 -- start start 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient: init 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 0x7f53ac139650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-a con 0x7f53ac0a2f10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53ac1329f0 0x7f53ac12de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-b con 0x7f53ac1329f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 0x7f53ac12e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): picked mon.noname-c con 0x7f53ac13b2b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac0a1a80 con 0x7f53ac0a2f10 2024-09-10T16:25:10.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f53ac0a1900 con 0x7f53ac1329f0 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac09e400 con 0x7f53ac13b2b0 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b3c47640 10 monclient(hunting): authenticate will time out at 1888.711682s 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 0x7f53ac139650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 0x7f53ac139650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38256/0 (socket says 172.21.15.164:38256) 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 1 -- 172.21.15.164:0/2332917568 learned_addr learned my addr 172.21.15.164:0/2332917568 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): get_auth_request con 0x7f53ac0a2f10 auth_method 0 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 0x7f53ac12e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 10 monclient(hunting): get_auth_request con 0x7f53ac13b2b0 auth_method 0 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 msgr2=0x7f53ac12e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 msgr2=0x7f53ac12e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 0x7f53ac12e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b21e4640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 0x7f53ac12e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 10 monclient(hunting): handle_auth_done global_id 164574 payload 824 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 msgr2=0x7f53ac12e360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.224+0000 7f53b19e3640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac13b2b0 0x7f53ac12e360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53ac1329f0 msgr2=0x7f53ac12de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53ac1329f0 0x7f53ac12de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 1 monclient: found mon.noname-a 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.225+0000 7f53b19e3640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53ac09fd10 con 0x7f53ac0a2f10 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b19e3640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 msgr2=0x7f53ac139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b19e3640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 0x7f53ac139650 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c460 tx=0x7f539c00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c016020 con 0x7f53ac0a2f10 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: dump: 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.243 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f539c012a80 con 0x7f53ac0a2f10 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 5 monclient: authenticate success, global_id 164574 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 10 monclient: _renew_subs 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53ac09e0f0 con 0x7f53ac0a2f10 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f539c011380 con 0x7f53ac0a2f10 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a27fc640 10 monclient: dump: 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 10 monclient: _renew_subs 2024-09-10T16:25:10.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53a3fff640 4 set_mon_vals no callback set 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.227+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53ac138640 con 0x7f53ac0a2f10 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b3c47640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b3c47640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac0a4960 con 0x7f53ac02e5c0 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 10 monclient: get_auth_request con 0x7f53ac02e5c0 auth_method 0 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 msgr2=0x7f53ac12e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 msgr2=0x7f53ac12e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:10.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f539c019020 con 0x7f53ac0a2f10 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53a27fc640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f539003e210 0x7f53900406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.228+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(649..649 src has 494..649) ==== 4799+0+0 (secure 0 0 0) 0x7f539c052b80 con 0x7f53ac0a2f10 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.229+0000 7f53b21e4640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f539003e210 0x7f53900406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.229+0000 7f53b21e4640 10 monclient: get_auth_request con 0x7f539003e210 auth_method 0 2024-09-10T16:25:10.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.231+0000 7f53b21e4640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f539003e210 0x7f53900406d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0a7f10 tx=0x7f53a4009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient: get_auth_request con 0x7f53ac02e5c0 auth_method 0 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 10 monclient(hunting): handle_auth_done global_id 214100 payload 824 2024-09-10T16:25:10.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.429+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53ac02e9a0 con 0x7f53ac02e5c0 2024-09-10T16:25:10.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 msgr2=0x7f53ac12e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a800b510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8016c60 con 0x7f53ac02e5c0 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.431+0000 7f53a27fc640 10 monclient: dump: 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.434 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.434 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f53a8016c60 con 0x7f53ac02e5c0 2024-09-10T16:25:10.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:10.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 msgr2=0x7f53ac12e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a800b510 comp rx=0 tx=0).stop 2024-09-10T16:25:10.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 msgr2=0x7f53ac12e8a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.432+0000 7f53a27fc640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac02e5c0 0x7f53ac12e8a0 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a800b510 comp rx=0 tx=0).stop 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b3c47640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b3c47640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 0x7f53ac064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b3c47640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f53ac132dd0 con 0x7f53ac063d50 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 0x7f53ac064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient: get_auth_request con 0x7f53ac063d50 auth_method 0 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 10 monclient(hunting): handle_auth_done global_id 214103 payload 824 2024-09-10T16:25:10.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.438+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53ac13b090 con 0x7f53ac063d50 2024-09-10T16:25:10.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.440+0000 7f53b11e2640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 msgr2=0x7f53ac064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.440+0000 7f53b11e2640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 0x7f53ac064160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a8006e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8026980 con 0x7f53ac063d50 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 10 monclient: dump: 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f53a8016e10 con 0x7f53ac063d50 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 msgr2=0x7f53ac064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 0x7f53ac064160 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a8006e10 comp rx=0 tx=0).stop 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 msgr2=0x7f53ac064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53a27fc640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f53ac063d50 0x7f53ac064160 secure :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f53ac139c20 tx=0x7f53a8006e10 comp rx=0 tx=0).stop 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:10.444 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10219, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:10.445 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.446 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.447 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:10.448 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.449 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.450 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.451 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.452 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.453 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.454 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:10.455 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.456 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:10.457 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f539003e210 msgr2=0x7f53900406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f539003e210 0x7f53900406d0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f53ac0a7f10 tx=0x7f53a4009000 comp rx=0 tx=0).stop 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 10 monclient: shutdown 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 msgr2=0x7f53ac139650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 --2- 172.21.15.164:0/2332917568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f53ac0a2f10 0x7f53ac139650 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f539c00c460 tx=0x7f539c00c920 comp rx=0 tx=0).stop 2024-09-10T16:25:10.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b21e4640 1 -- 172.21.15.164:0/2332917568 reap_dead start 2024-09-10T16:25:10.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b21e4640 1 -- 172.21.15.164:0/2332917568 reap_dead start 2024-09-10T16:25:10.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 shutdown_connections 2024-09-10T16:25:10.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 >> 172.21.15.164:0/2332917568 conn(0x7f53ac0a7320 msgr2=0x7f53ac131a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 shutdown_connections 2024-09-10T16:25:10.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.441+0000 7f53b3c47640 1 -- 172.21.15.164:0/2332917568 wait complete. 2024-09-10T16:25:10.461 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:10.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:25:10.461 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:25:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 10 monclient: get_monmap_and_config 2024-09-10T16:25:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 10 monclient: monmap: 2024-09-10T16:25:10.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:10.560029+0000 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:10.560029+0000 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:10.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 1 Processor -- start 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 1 -- start start 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.559+0000 7fe6cc389640 10 monclient: init 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe6c40a46a0 0x7fe6c40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c40a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:10.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6c4137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6c409ce50 con 0x7fe6c4137ca0 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe6c406db60 con 0x7fe6c40a46a0 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c406cf50 con 0x7fe6c406cb70 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6cc389640 10 monclient(hunting): authenticate will time out at 1889.047432s 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52132/0 (socket says 172.21.15.164:52132) 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 -- 172.21.15.164:0/3817382494 learned_addr learned my addr 172.21.15.164:0/3817382494 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 10 monclient(hunting): get_auth_request con 0x7fe6c406cb70 auth_method 0 2024-09-10T16:25:10.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 -- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 msgr2=0x7fe6c4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 -- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 msgr2=0x7fe6c4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6ca125640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 1 --2- 172.21.15.164:0/3817382494 >> v2:172.21.15.165:3300/0 conn(0x7fe6c40a46a0 0x7fe6c40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): get_auth_request con 0x7fe6c40a46a0 auth_method 0 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.560+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 10 monclient(hunting): handle_auth_done global_id 154573 payload 274 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 1 -- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 msgr2=0x7fe6c4134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 1 -- 172.21.15.164:0/3817382494 >> v1:172.21.15.165:6789/0 conn(0x7fe6c4137ca0 legacy=0x7fe6c4138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 1 monclient: found mon.noname-c 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:10.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.561+0000 7fe6c9924640 1 -- 172.21.15.164:0/3817382494 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c409cb70 con 0x7fe6c40a46a0 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9924640 1 -- 172.21.15.164:0/3817382494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a46a0 msgr2=0x7fe6c40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9924640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a46a0 0x7fe6c40a4a80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009d90 tx=0x7fe6b802a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8130532f12c6a57 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 1 -- 172.21.15.164:0/3817382494 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8007950 con 0x7fe6c40a46a0 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: dump: 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 5 monclient: authenticate success, global_id 154573 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:10.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 1 -- 172.21.15.164:0/3817382494 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6b8007af0 con 0x7fe6c40a46a0 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 1 -- 172.21.15.164:0/3817382494 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8031400 con 0x7fe6c40a46a0 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 4 set_mon_vals no callback set 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6c9123640 10 monclient: dump: 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.577 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 10 monclient: shutdown 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 -- 172.21.15.164:0/3817382494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a46a0 msgr2=0x7fe6c40a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a46a0 0x7fe6c40a4a80 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009d90 tx=0x7fe6b802a490 comp rx=0 tx=0).stop 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 -- 172.21.15.164:0/3817382494 shutdown_connections 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a46a0 0x7fe6c40a4a80 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 --2- 172.21.15.164:0/3817382494 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c4134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 -- 172.21.15.164:0/3817382494 >> 172.21.15.164:0/3817382494 conn(0x7fe6c40aa900 msgr2=0x7fe6c40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 -- 172.21.15.164:0/3817382494 shutdown_connections 2024-09-10T16:25:10.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.564+0000 7fe6cc389640 1 -- 172.21.15.164:0/3817382494 wait complete. 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.565+0000 7fe6cc389640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.565+0000 7fe6cc389640 10 monclient: monmap: 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.565+0000 7fe6cc389640 1 Processor -- start 2024-09-10T16:25:10.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.565+0000 7fe6cc389640 1 -- start start 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.565+0000 7fe6cc389640 10 monclient: init 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-c con 0x7fe6c406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 0x7fe6c413f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-a con 0x7fe6c40a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c4137ca0 0x7fe6c40a2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): picked mon.noname-b con 0x7fe6c4137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c41389c0 con 0x7fe6c40a46a0 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c4138840 con 0x7fe6c4137ca0 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c4140040 con 0x7fe6c406cb70 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6cc389640 10 monclient(hunting): authenticate will time out at 1889.053307s 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 0x7fe6c413f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52136/0 (socket says 172.21.15.164:52136) 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 -- 172.21.15.164:0/115391879 learned_addr learned my addr 172.21.15.164:0/115391879 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 0x7fe6c413f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38262/0 (socket says 172.21.15.164:38262) 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): get_auth_request con 0x7fe6c406cb70 auth_method 0 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): get_auth_request con 0x7fe6c40a46a0 auth_method 0 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c4137ca0 0x7fe6c40a2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): get_auth_request con 0x7fe6c4137ca0 auth_method 0 2024-09-10T16:25:10.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca926640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient(hunting): handle_auth_done global_id 214109 payload 824 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c4137ca0 msgr2=0x7fe6c40a2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c4137ca0 0x7fe6c40a2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 msgr2=0x7fe6c413f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 0x7fe6c413f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 monclient: found mon.noname-c 2024-09-10T16:25:10.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6ca125640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c413aab0 con 0x7fe6c406cb70 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.566+0000 7fe6c9924640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe6c40a46a0 0x7fe6c413f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.568+0000 7fe6ca125640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 msgr2=0x7fe6c41cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.568+0000 7fe6ca125640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e810 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc017070 con 0x7fe6c406cb70 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: dump: 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.584 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6bc004020 con 0x7fe6c406cb70 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 5 monclient: authenticate success, global_id 214109 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 10 monclient: _renew_subs 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc00f040 con 0x7fe6c406cb70 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6c41cd670 con 0x7fe6c406cb70 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c2ffd640 10 monclient: dump: 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6c8922640 4 set_mon_vals no callback set 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 10 monclient: _renew_subs 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.569+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6c4132c80 con 0x7fe6c406cb70 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6cc389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6cc389640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 0x7fe6c40a2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c4138ed0 con 0x7fe6c413f830 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6ca125640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e810 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7fe6bc052b20 osd_map(649..649 src has 494..649) 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe6bc00c8e0 con 0x7fe6c406cb70 2024-09-10T16:25:10.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 0x7fe6c40a2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient: get_auth_request con 0x7fe6c413f830 auth_method 0 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c2ffd640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe6b003e240 0x7fe6b0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 10 monclient(hunting): handle_auth_done global_id 154579 payload 824 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6c9924640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6c4133270 con 0x7fe6c413f830 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6ca926640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe6b003e240 0x7fe6b0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.570+0000 7fe6ca926640 10 monclient: get_auth_request con 0x7fe6b003e240 auth_method 0 2024-09-10T16:25:10.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.572+0000 7fe6c9924640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 msgr2=0x7fe6c40a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.572+0000 7fe6c9924640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 0x7fe6c40a2dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8034000 tx=0x7fe6b8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8007e30 con 0x7fe6c413f830 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 10 monclient: dump: 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6ca926640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe6b003e240 0x7fe6b0040700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe6c41397d0 tx=0x7fe6b4007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe6b8007e30 con 0x7fe6c413f830 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 msgr2=0x7fe6c40a2dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 0x7fe6c40a2dc0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8034000 tx=0x7fe6b8004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 msgr2=0x7fe6c40a2dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.573+0000 7fe6c2ffd640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c413f830 0x7fe6c40a2dc0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6cc389640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6cc389640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 0x7fe6c409d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6cc389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c40a3710 con 0x7fe6c40a3300 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 0x7fe6c409d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient: get_auth_request con 0x7fe6c40a3300 auth_method 0 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.579+0000 7fe6c9924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.580+0000 7fe6c9924640 10 monclient(hunting): handle_auth_done global_id 154582 payload 824 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.580+0000 7fe6c9924640 1 -- 172.21.15.164:0/115391879 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe6c4134f70 con 0x7fe6c40a3300 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.582+0000 7fe6c9924640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 msgr2=0x7fe6c409d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.582+0000 7fe6c9924640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 0x7fe6c409d630 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe6b802a4d0 tx=0x7fe6b8030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6b804a570 con 0x7fe6c40a3300 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 10 monclient: dump: 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fe6b803c420 con 0x7fe6c40a3300 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 msgr2=0x7fe6c409d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 0x7fe6c409d630 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe6b802a4d0 tx=0x7fe6b8030000 comp rx=0 tx=0).stop 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 msgr2=0x7fe6c409d630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe6c40a3300 0x7fe6c409d630 secure :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe6b802a4d0 tx=0x7fe6b8030000 comp rx=0 tx=0).stop 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 329134, 2024-09-10T16:25:10.595 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.596 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.597 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:10.598 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:10.599 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.600 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.602 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:10.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:10.604 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:10.605 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.606 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.607 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:10.608 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe6b003e240 msgr2=0x7fe6b0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe6b003e240 0x7fe6b0040700 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe6c41397d0 tx=0x7fe6b4007db0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 10 monclient: shutdown 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 msgr2=0x7fe6c41cd130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 --2- 172.21.15.164:0/115391879 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe6c406cb70 0x7fe6c41cd130 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe6bc00e810 tx=0x7fe6bc00ecd0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6ca926640 1 -- 172.21.15.164:0/115391879 reap_dead start 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6c2ffd640 1 -- 172.21.15.164:0/115391879 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(649..649 src has 494..649) ==== 4799+0+0 (unknown 0 0 0) 0x7fe6bc052b20 con 0x7fe6c406cb70 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6ca926640 1 -- 172.21.15.164:0/115391879 reap_dead start 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 shutdown_connections 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 >> 172.21.15.164:0/115391879 conn(0x7fe6c40aa900 msgr2=0x7fe6c40a9280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 shutdown_connections 2024-09-10T16:25:10.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.585+0000 7fe6cc389640 1 -- 172.21.15.164:0/115391879 wait complete. 2024-09-10T16:25:10.610 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:10.611 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:25:10.611 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:25:10.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.707+0000 7f40387c9640 10 monclient: get_monmap_and_config 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.707+0000 7f40387c9640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.707+0000 7f40387c9640 10 monclient: monmap: 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:10.708821+0000 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:10.708821+0000 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:10.710 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:10.711 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.708+0000 7f40387c9640 1 Processor -- start 2024-09-10T16:25:10.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.708+0000 7f40387c9640 1 -- start start 2024-09-10T16:25:10.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.708+0000 7f40387c9640 10 monclient: init 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f403009f930 0x7f40300a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-a con 0x7f403009f930 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-b con 0x7f40300a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f403013a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403009fd10 con 0x7f403013a5c0 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f403009cd20 con 0x7f403009f930 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f403009a0d0 con 0x7f40300a2380 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f40387c9640 10 monclient(hunting): authenticate will time out at 1889.196249s 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f403009f930 0x7f40300a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f403009f930 0x7f40300a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38264/0 (socket says 172.21.15.164:38264) 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52150/0 (socket says 172.21.15.164:52150) 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 -- 172.21.15.164:0/607765562 learned_addr learned my addr 172.21.15.164:0/607765562 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): get_auth_request con 0x7f403009f930 auth_method 0 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): get_auth_request con 0x7f40300a2380 auth_method 0 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient(hunting): handle_auth_done global_id 164580 payload 274 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 -- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 msgr2=0x7f40300a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 -- 172.21.15.164:0/607765562 >> v1:172.21.15.165:6789/0 conn(0x7f403013a5c0 legacy=0x7f403013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 monclient: found mon.noname-a 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4036565640 1 -- 172.21.15.164:0/607765562 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403009e840 con 0x7f403009f930 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.709+0000 7f4035d64640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4036565640 1 -- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 msgr2=0x7f40300a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4036565640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40300a1e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f402c009870 tx=0x7f402c02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b7470ae055300b4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 1 -- 172.21.15.164:0/607765562 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c037070 con 0x7f403009f930 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: dump: 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.719 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 1 -- 172.21.15.164:0/607765562 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c032440 con 0x7f403009f930 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 1 -- 172.21.15.164:0/607765562 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c032810 con 0x7f403009f930 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f4035563640 10 monclient: dump: 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.720 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f40387c9640 5 monclient: authenticate success, global_id 164580 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f40387c9640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.712+0000 7f40387c9640 4 set_mon_vals no callback set 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 10 monclient: shutdown 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 -- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 msgr2=0x7f40300a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40300a1e40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f402c009870 tx=0x7f402c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 -- 172.21.15.164:0/607765562 shutdown_connections 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40300a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 --2- 172.21.15.164:0/607765562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40300a1e40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 -- 172.21.15.164:0/607765562 >> 172.21.15.164:0/607765562 conn(0x7f40300a3110 msgr2=0x7f40300a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 -- 172.21.15.164:0/607765562 shutdown_connections 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 1 -- 172.21.15.164:0/607765562 wait complete. 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 10 monclient: build_initial_monmap 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.713+0000 7f40387c9640 10 monclient: monmap: 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:10.722 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 Processor -- start 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 -- start start 2024-09-10T16:25:10.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient: init 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-a con 0x7f403009f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40301c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-c con 0x7f40300a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 0x7f40301c8290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): picked mon.noname-b con 0x7f403013a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f403009d930 con 0x7f403009f930 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f403009ee00 con 0x7f403013a5c0 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f403009c8b0 con 0x7f40300a2380 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f40387c9640 10 monclient(hunting): authenticate will time out at 1889.201721s 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38266/0 (socket says 172.21.15.164:38266) 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40301c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 1 -- 172.21.15.164:0/1584847716 learned_addr learned my addr 172.21.15.164:0/1584847716 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): get_auth_request con 0x7f403009f930 auth_method 0 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): get_auth_request con 0x7f40300a2380 auth_method 0 2024-09-10T16:25:10.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036565640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4035d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.714+0000 7f4036d66640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 0x7f40301c8290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 10 monclient(hunting): handle_auth_done global_id 164583 payload 824 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 -- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 msgr2=0x7f40301c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40301c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 -- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 msgr2=0x7f40301c8290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 0x7f40301c8290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 10 monclient: _finish_hunting 0 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 monclient: found mon.noname-a 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036565640 1 -- 172.21.15.164:0/1584847716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403012f3d0 con 0x7f403009f930 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036d66640 10 monclient: get_auth_request con 0x7f403013a5c0 auth_method 0 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.715+0000 7f4036d66640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 0x7f40301c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4036565640 1 -- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 msgr2=0x7f40301c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4036565640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f402c0090b0 tx=0x7f402c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c041020 con 0x7f403009f930 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: dump: 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f402c03d630 con 0x7f403009f930 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 5 monclient: authenticate success, global_id 164583 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 10 monclient: _renew_subs 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403012f6b0 con 0x7f403009f930 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c037040 con 0x7f403009f930 2024-09-10T16:25:10.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4026ffd640 10 monclient: dump: 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:10.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f4034d62640 4 set_mon_vals no callback set 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 10 monclient: _renew_subs 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.717+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40300a1910 con 0x7f403009f930 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f402c044020 con 0x7f403009f930 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f40387c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f40387c9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f40387c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40300a0760 con 0x7f403009f930 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f4026ffd640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 0x7f40140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(649..649 src has 494..649) ==== 4799+0+0 (secure 0 0 0) 0x7f402c073080 con 0x7f403009f930 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f4035d64640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 0x7f40140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:10.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.718+0000 7f4035d64640 10 monclient: get_auth_request con 0x7f401403e210 auth_method 0 2024-09-10T16:25:10.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.721+0000 7f4035d64640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 0x7f40140406d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4020009560 tx=0x7f4020009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:10.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.723+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f402c04b030 con 0x7f403009f930 2024-09-10T16:25:10.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.723+0000 7f4026ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:10.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.723+0000 7f4026ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.965+0000 7f40387c9640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:25:10.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.965+0000 7f40387c9640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:25:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.966+0000 7f40387c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.966+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f403012f050 con 0x7f403009f930 2024-09-10T16:25:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.966+0000 7f4026ffd640 1 -- 172.21.15.164:0/1584847716 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f402c03b3f0 con 0x7f403009f930 2024-09-10T16:25:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.966+0000 7f4026ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:25:10.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.966+0000 7f4026ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 msgr2=0x7f40140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 0x7f40140406d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f4020009560 tx=0x7f4020009190 comp rx=0 tx=0).stop 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 10 monclient: shutdown 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 msgr2=0x7f40301c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:10.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f402c0090b0 tx=0x7f402c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 shutdown_connections 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f401403e210 0x7f40140406d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f403013a5c0 0x7f40301c8290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f40300a2380 0x7f40301c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 --2- 172.21.15.164:0/1584847716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f403009f930 0x7f40301c7810 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 >> 172.21.15.164:0/1584847716 conn(0x7f40300a3110 msgr2=0x7f4030138b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 shutdown_connections 2024-09-10T16:25:10.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f40387c9640 1 -- 172.21.15.164:0/1584847716 wait complete. 2024-09-10T16:25:10.973 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:10.973 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=12,logm=45} crc {auth=1668949082,config=2586702516,health=655917057,kv=805391194,logm=1030116544}) 2024-09-10T16:25:10.973 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=30,config=1,health=11,kv=12,logm=46} crc {auth=1847386585,config=1747058506,health=2633641155,kv=805391194,logm=194035471}) 2024-09-10T16:25:10.973 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.969+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:10.973 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.970+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=31,config=2,health=10,kv=12,logm=45} crc {auth=1668949082,config=2586702516,health=655917057,kv=805391194,logm=1030116544}) 2024-09-10T16:25:10.974 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.970+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=29,config=2,health=12,kv=12,logm=45} crc {auth=650175549,config=2586702516,health=86150175,kv=805391194,logm=2246185958}) 2024-09-10T16:25:10.979 INFO:tasks.rados.rados.0.smithi165.stdout:2909: done (2 left) 2024-09-10T16:25:10.980 INFO:tasks.rados.rados.0.smithi165.stdout:2910: read oid 27 snap 303 2024-09-10T16:25:10.980 INFO:tasks.rados.rados.0.smithi165.stdout:2910: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:25:10.980 INFO:tasks.rados.rados.0.smithi165.stdout:2911: delete oid 2 current snap is 308 2024-09-10T16:25:10.980 INFO:tasks.rados.rados.0.smithi165.stdout:2908: expect (ObjNum 966 snap 292 seq_num 966) 2024-09-10T16:25:10.980 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:10.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.977+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:10.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.977+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613947730}) 2024-09-10T16:25:10.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.978+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1061555395}) 2024-09-10T16:25:10.981 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.978+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:10.981 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.978+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=613947730}) 2024-09-10T16:25:10.981 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:10.978+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2810700559}) 2024-09-10T16:25:10.984 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:25:11.057 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 854643 2024-09-10T16:25:11.166 INFO:tasks.rados.rados.0.smithi165.stdout:2907: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:25:11.166 INFO:tasks.rados.rados.0.smithi165.stdout:2907: finishing copy_from to smithi16534841-37 2024-09-10T16:25:11.166 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1417 (ObjNum 941 snap 285 seq_num 941) dirty exists 2024-09-10T16:25:11.166 INFO:tasks.rados.rados.0.smithi165.stdout:2910: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444062112}) 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2005155277}) 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3444062112}) 2024-09-10T16:25:11.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.171+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1559024879}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3406954608}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532763689}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3406954608}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.172+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3791496718}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264643587}) 2024-09-10T16:25:11.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489405084}) 2024-09-10T16:25:11.176 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.176 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2264643587}) 2024-09-10T16:25:11.176 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.173+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3553486635}) 2024-09-10T16:25:11.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704461254}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148074874}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704461254}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1838826202}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301534230}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693685922}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.174+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=301534230}) 2024-09-10T16:25:11.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4157496672}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947472195}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793043541}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947472195}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.175+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3774324228}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511860175}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1906170352}) 2024-09-10T16:25:11.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=511860175}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1303673125}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465949544}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2176930339}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2465949544}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.176+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3476697627}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879224412}) 2024-09-10T16:25:11.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3081671029}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1879224412}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.177+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836401729}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763929524}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2467222043}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2763929524}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=169698022}) 2024-09-10T16:25:11.184 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351427663}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3427704882}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1351427663}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.178+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1453567631}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165753237}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=133733654}) 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.185 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3165753237}) 2024-09-10T16:25:11.186 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.179+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3747271820}) 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569363278}) 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=505294122}) 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=569363278}) 2024-09-10T16:25:11.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3182893386}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654723823}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316096185}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3654723823}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.180+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3664983695}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734572986}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3353168557}) 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=734572986}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1165514748}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256267285}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2073194417}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1256267285}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.181+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4118501160}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182071675}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596394512}) 2024-09-10T16:25:11.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=182071675}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.182+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1592170196}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030151588}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=480415553}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2030151588}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=125465521}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694450977}) 2024-09-10T16:25:11.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1805229349}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=694450977}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.183+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3471260395}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971150327}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4012911382}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=971150327}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.184+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3331531348}) 2024-09-10T16:25:11.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198825164}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3244089852}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=198825164}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2645354089}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951691529}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3161738528}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3951691529}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.185+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3194792827}) 2024-09-10T16:25:11.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209287241}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=682519676}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3209287241}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.186+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3802769571}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695814308}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1750003423}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1695814308}) 2024-09-10T16:25:11.203 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=592845203}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877335656}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3483980301}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877335656}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.187+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3839949319}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680570106}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4086975890}) 2024-09-10T16:25:11.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1680570106}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1716827387}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961106434}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=61415451}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961106434}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.188+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2688065991}) 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281155490}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=315128414}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3281155490}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.189+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335340553}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304452002}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2591257601}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1304452002}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=14165082}) 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.206 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642679193}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=662446256}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642679193}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.190+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844513104}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146948947}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2101693589}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3146948947}) 2024-09-10T16:25:11.207 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2717277578}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.191+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250384419}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=877769371}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250384419}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262921905}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974254862}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=373173245}) 2024-09-10T16:25:11.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=974254862}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.192+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1348044533}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769597437}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3671788481}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=769597437}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1456214997}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228829643}) 2024-09-10T16:25:11.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2391724542}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.193+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1228829643}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3789695510}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759207758}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=481511507}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759207758}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.194+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3286912547}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042662288}) 2024-09-10T16:25:11.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2857934694}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042662288}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1067185819}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431577063}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2887159258}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431577063}) 2024-09-10T16:25:11.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.195+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320919782}) 2024-09-10T16:25:11.212 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 995028 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045883108}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2068454351}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2045883108}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.196+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2433378432}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095113879}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489629593}) 2024-09-10T16:25:11.225 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095113879}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487459134}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178275303}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2365982024}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4178275303}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.197+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3021627508}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115180828}) 2024-09-10T16:25:11.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244510109}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=115180828}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.198+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1783485414}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880488757}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3009488976}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880488757}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3214872572}) 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503605307}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1012303676}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=503605307}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.199+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817698473}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521268369}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950235063}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=521268369}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2679115893}) 2024-09-10T16:25:11.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176014077}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=81746087}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176014077}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.200+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3870982315}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361144388}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3977281171}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=361144388}) 2024-09-10T16:25:11.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.201+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1527297510}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006852923}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2973091500}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4006852923}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024418923}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000059405}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1349520540}) 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4000059405}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.202+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571135114}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829978646}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745127051}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3829978646}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.203+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=237590250}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294973662}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1948288244}) 2024-09-10T16:25:11.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1294973662}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1777995366}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51913072}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=452897925}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=51913072}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.204+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=994003166}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760167580}) 2024-09-10T16:25:11.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=487623896}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1760167580}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.205+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1595408725}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528375229}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4176728926}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2528375229}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=642014995}) 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015942385}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3519730930}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4015942385}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.206+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4122472564}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728435412}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4255213469}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3728435412}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1458691988}) 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461459370}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=258557929}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2461459370}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.207+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=382706391}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520102401}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1254163978}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1520102401}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.208+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3149306683}) 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2431920715}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571470027}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2431920715}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2093477386}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250000941}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3440506795}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4250000941}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.209+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617958609}) 2024-09-10T16:25:11.236 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196946785}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1860596075}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=196946785}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.210+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2373791411}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084956672}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3947645522}) 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3084956672}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2429968579}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583163088}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311573107}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=583163088}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.211+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2266953698}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15} crc {logm=738182098,mdsmap=931935893,mgr=1898881612,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4240889745,mon_config_key=1765032840}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=57,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=2} crc {logm=3644340167,mdsmap=931935893,mgr=2441926604,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=302730370,mon_config_key=665552247,monmap=765578399}) 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15} crc {logm=738182098,mdsmap=931935893,mgr=1898881612,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4240889745,mon_config_key=1765032840}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.212+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=58,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=16,monmap=2} crc {logm=1712653465,mdsmap=931935893,mgr=3838082885,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4240889745,mon_config_key=3751376834,monmap=765578399}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=87} crc {monmap=1433040829,nvmeofgw=598337686,osd_metadata=263750264,osd_pg_creating=925513361,osd_snap=2449239365}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=87} crc {monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=925513361,osd_snap=1077235805}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=87} crc {monmap=1433040829,nvmeofgw=598337686,osd_metadata=263750264,osd_pg_creating=925513361,osd_snap=2449239365}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.213+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=88} crc {monmap=1433040829,nvmeofgw=1596038039,osd_metadata=4005186995,osd_pg_creating=925513361,osd_snap=2020733183}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=30,osdmap=70} crc {osd_snap=963608620,osdmap=3588003410}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=33,osdmap=67} crc {osd_snap=2235686604,osdmap=762716866}) 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=30,osdmap=70} crc {osd_snap=963608620,osdmap=3588003410}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=34,osdmap=66} crc {osd_snap=605067555,osdmap=572793399}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3268349445}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1883193720}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.214+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.215+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3268349445}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.215+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1021143527}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=3946634537}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=69} crc {osdmap=3866584387}) 2024-09-10T16:25:11.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:11.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=72} crc {osdmap=3946634537}) 2024-09-10T16:25:11.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:11.218+0000 7f5a79774640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=3326035889}) 2024-09-10T16:25:11.265 INFO:tasks.rados.rados.0.smithi165.stdout:2907: done (3 left) 2024-09-10T16:25:11.265 INFO:tasks.rados.rados.0.smithi165.stdout:2908: done (2 left) 2024-09-10T16:25:11.265 INFO:tasks.rados.rados.0.smithi165.stdout:2910: done (1 left) 2024-09-10T16:25:11.265 INFO:tasks.rados.rados.0.smithi165.stdout:2911: done (0 left) 2024-09-10T16:25:11.265 INFO:tasks.rados.rados.0.smithi165.stdout:2912: copy_from oid 20 from oid 50 current snap is 308 2024-09-10T16:25:11.266 INFO:tasks.rados.rados.0.smithi165.stdout:2913: snap_create 2024-09-10T16:25:11.273 INFO:tasks.rados.rados.0.smithi165.stdout:2912: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:25:11.274 INFO:tasks.rados.rados.0.smithi165.stdout:2912: finishing copy_from to smithi16534841-20 2024-09-10T16:25:11.274 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1289 (ObjNum 975 snap 297 seq_num 975) dirty exists 2024-09-10T16:25:11.484 INFO:tasks.rados.rados.0.smithi165.stdout:2912: done (1 left) 2024-09-10T16:25:11.484 INFO:tasks.rados.rados.0.smithi165.stdout:2913: done (0 left) 2024-09-10T16:25:11.484 INFO:tasks.rados.rados.0.smithi165.stdout:2914: snap_remove snap 308 2024-09-10T16:25:11.985 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:25:11.986 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.083+0000 7f6f7e4a3640 10 monclient: get_monmap_and_config 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.083+0000 7f6f7e4a3640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.083+0000 7f6f7e4a3640 10 monclient: monmap: 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:12.084681+0000 2024-09-10T16:25:12.085 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:12.084681+0000 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:12.086 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.084+0000 7f6f7e4a3640 1 Processor -- start 2024-09-10T16:25:12.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.084+0000 7f6f7e4a3640 1 -- start start 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.084+0000 7f6f7e4a3640 10 monclient: init 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.084+0000 7f6f7e4a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.084+0000 7f6f7e4a3640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6f7809c180 0x7f6f7809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f7809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6f7809f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7809a430 con 0x7f6f7809f4c0 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6f780a24e0 con 0x7f6f7809c180 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f780a2bf0 con 0x7f6f7809d160 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7e4a3640 10 monclient(hunting): authenticate will time out at 1890.572087s 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f7809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6f7809c180 0x7f6f7809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7ca40640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6f7809f4c0 0x7f6f7813b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45754/0 (socket says 172.21.15.164:45754) 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f7ca40640 1 -- 172.21.15.164:0/1263499303 learned_addr learned my addr 172.21.15.164:0/1263499303 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): get_auth_request con 0x7f6f7809d160 auth_method 0 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): get_auth_request con 0x7f6f7809c180 auth_method 0 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient(hunting): handle_auth_done global_id 164586 payload 274 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 -- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 msgr2=0x7f6f7809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f7809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 -- 172.21.15.164:0/1263499303 >> v1:172.21.15.164:6789/0 conn(0x7f6f7809f4c0 legacy=0x7f6f7813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 monclient: found mon.noname-a 2024-09-10T16:25:12.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f77fff640 1 -- 172.21.15.164:0/1263499303 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7809c560 con 0x7f6f7809c180 2024-09-10T16:25:12.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.085+0000 7f6f777fe640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f7809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:12.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f77fff640 1 -- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 msgr2=0x7f6f7809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f77fff640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f7809ea40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f680097a0 tx=0x7f6f6802a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbcf1c23c574c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 1 -- 172.21.15.164:0/1263499303 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68037070 con 0x7f6f7809c180 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: dump: 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.101 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 1 -- 172.21.15.164:0/1263499303 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f68032440 con 0x7f6f7809c180 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f7e4a3640 5 monclient: authenticate success, global_id 164586 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f7e4a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f7e4a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 1 -- 172.21.15.164:0/1263499303 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f68032810 con 0x7f6f7809c180 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f7e4a3640 4 set_mon_vals no callback set 2024-09-10T16:25:12.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.088+0000 7f6f76ffd640 10 monclient: dump: 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 10 monclient: shutdown 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 msgr2=0x7f6f7809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f7809ea40 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f6f680097a0 tx=0x7f6f6802a050 comp rx=0 tx=0).stop 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/1263499303 shutdown_connections 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f7809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/1263499303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f7809ea40 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/1263499303 >> 172.21.15.164:0/1263499303 conn(0x7f6f780a3110 msgr2=0x7f6f780a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/1263499303 shutdown_connections 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/1263499303 wait complete. 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.089+0000 7f6f7e4a3640 10 monclient: monmap: 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:12.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 Processor -- start 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 -- start start 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient: init 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 0x7f6f7809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7809f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78074f90 con 0x7f6f7809c180 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6f78074e10 con 0x7f6f7809f4c0 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7813cce0 con 0x7f6f7809d160 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f7e4a3640 10 monclient(hunting): authenticate will time out at 1890.577804s 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38286/0 (socket says 172.21.15.164:38286) 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52174/0 (socket says 172.21.15.164:52174) 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 1 -- 172.21.15.164:0/913333966 learned_addr learned my addr 172.21.15.164:0/913333966 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): get_auth_request con 0x7f6f7809c180 auth_method 0 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): get_auth_request con 0x7f6f7809d160 auth_method 0 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f77fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.090+0000 7f6f777fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f7ca40640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 0x7f6f7809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 10 monclient(hunting): handle_auth_done global_id 214121 payload 824 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 -- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 msgr2=0x7f6f7809d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 0x7f6f7809d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 -- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 msgr2=0x7f6f781cd270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 monclient: found mon.noname-c 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f777fe640 1 -- 172.21.15.164:0/913333966 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7809d620 con 0x7f6f7809d160 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f7ca40640 10 monclient: get_auth_request con 0x7f6f7809f4c0 auth_method 0 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f77fff640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.091+0000 7f6f7ca40640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 0x7f6f7809d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f777fe640 1 -- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 msgr2=0x7f6f781cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f777fe640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e900 tx=0x7f6f6c00edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c017070 con 0x7f6f7809d160 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: dump: 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.109 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f6c00cc10 con 0x7f6f7809d160 2024-09-10T16:25:12.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 5 monclient: authenticate success, global_id 214121 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 10 monclient: _renew_subs 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f7808e1e0 con 0x7f6f7809d160 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f6c013470 con 0x7f6f7809d160 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f74ff9640 10 monclient: dump: 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.111 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 10 monclient: _renew_subs 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f767fc640 4 set_mon_vals no callback set 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.093+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7808f700 con 0x7f6f7809d160 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6f6c01a070 con 0x7f6f7809d160 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f7e4a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f7e4a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f7e4a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7809ea40 con 0x7f6f7809d160 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f74ff9640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 0x7f6f640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(651..651 src has 494..651) ==== 4815+0+0 (secure 0 0 0) 0x7f6f6c052c20 con 0x7f6f7809d160 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.094+0000 7f6f77fff640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 0x7f6f640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.095+0000 7f6f77fff640 10 monclient: get_auth_request con 0x7f6f6403e210 auth_method 0 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.097+0000 7f6f77fff640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 0x7f6f640406d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802a090 tx=0x7f6f6802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.099+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6f6c051600 con 0x7f6f7809d160 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.099+0000 7f6f74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.099+0000 7f6f74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f7e4a3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f7e4a3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f7e4a3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6f7808f2b0 con 0x7f6f7809d160 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f74ff9640 1 -- 172.21.15.164:0/913333966 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6f7808f2b0 con 0x7f6f7809d160 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.341+0000 7f6f74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:12.343 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":186,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 msgr2=0x7f6f640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 0x7f6f640406d0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f6f6802a090 tx=0x7f6f6802e000 comp rx=0 tx=0).stop 2024-09-10T16:25:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 10 monclient: shutdown 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 msgr2=0x7f6f781cd7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f6f6c00e900 tx=0x7f6f6c00edc0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 shutdown_connections 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f6403e210 0x7f6f640406d0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7809f4c0 0x7f6f7809d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.343+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7809d160 0x7f6f781cd7b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.344+0000 7f6f7e4a3640 1 --2- 172.21.15.164:0/913333966 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7809c180 0x7f6f781cd270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.344+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 >> 172.21.15.164:0/913333966 conn(0x7f6f780a3110 msgr2=0x7f6f7813a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.344+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 shutdown_connections 2024-09-10T16:25:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.344+0000 7f6f7e4a3640 1 -- 172.21.15.164:0/913333966 wait complete. 2024-09-10T16:25:12.356 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:25:12.356 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:25:12.356 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:25:12.356 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.453+0000 7f2160616640 10 monclient: get_monmap_and_config 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.453+0000 7f2160616640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.453+0000 7f2160616640 10 monclient: monmap: 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:12.454592+0000 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:12.454592+0000 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.455 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:12.456 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:12.456 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:12.456 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:12.456 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:12.456 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 Processor -- start 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 -- start start 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient: init 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-10T16:25:12.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): picked mon.noname-b con 0x7f215809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f215809d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215813a5c0 0x7f215813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): picked mon.noname-a con 0x7f215813a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f215809fbc0 con 0x7f215809d190 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f21580a2a20 con 0x7f215813a5c0 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f215809c6f0 con 0x7f215809c310 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.454+0000 7f2160616640 10 monclient(hunting): authenticate will time out at 1890.941926s 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215813a5c0 0x7f215813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f215813a5c0 0x7f215813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38296/0 (socket says 172.21.15.164:38296) 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52176/0 (socket says 172.21.15.164:52176) 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 -- 172.21.15.164:0/3641435182 learned_addr learned my addr 172.21.15.164:0/3641435182 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): get_auth_request con 0x7f215813a5c0 auth_method 0 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): get_auth_request con 0x7f215809c310 auth_method 0 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient(hunting): handle_auth_done global_id 164592 payload 274 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 -- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 msgr2=0x7f215809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 -- 172.21.15.164:0/3641435182 >> v1:172.21.15.165:6789/0 conn(0x7f215809d190 legacy=0x7f215809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 monclient: found mon.noname-a 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.461 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215ebb3640 1 -- 172.21.15.164:0/3641435182 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f215809a4d0 con 0x7f215813a5c0 2024-09-10T16:25:12.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.455+0000 7f215e3b2640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215ebb3640 1 -- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 msgr2=0x7f215813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215ebb3640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215813c9b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265b74c66331bde0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 1 -- 172.21.15.164:0/3641435182 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150036070 con 0x7f215813a5c0 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:12.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: dump: 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 5 monclient: authenticate success, global_id 164592 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 1 -- 172.21.15.164:0/3641435182 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2150007800 con 0x7f215813a5c0 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 1 -- 172.21.15.164:0/3641435182 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2150007bd0 con 0x7f215813a5c0 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 4 set_mon_vals no callback set 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f215d3b0640 10 monclient: dump: 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.470 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 10 monclient: shutdown 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 -- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 msgr2=0x7f215813c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215813c9b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f2150009e50 tx=0x7f215002a5f0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 -- 172.21.15.164:0/3641435182 shutdown_connections 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215813c9b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 --2- 172.21.15.164:0/3641435182 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215809ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 -- 172.21.15.164:0/3641435182 >> 172.21.15.164:0/3641435182 conn(0x7f21580a3110 msgr2=0x7f21580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 -- 172.21.15.164:0/3641435182 shutdown_connections 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.458+0000 7f2160616640 1 -- 172.21.15.164:0/3641435182 wait complete. 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.459+0000 7f2160616640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.459+0000 7f2160616640 10 monclient: monmap: 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.459+0000 7f2160616640 1 Processor -- start 2024-09-10T16:25:12.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.459+0000 7f2160616640 1 -- start start 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.459+0000 7f2160616640 10 monclient: init 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215808efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): picked mon.noname-c con 0x7f215809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215808f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): picked mon.noname-a con 0x7f215813a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f215808fa50 0x7f21581375b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): picked mon.noname-b con 0x7f215808fa50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f21580a2230 con 0x7f215813a5c0 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f21580a20b0 con 0x7f215808fa50 2024-09-10T16:25:12.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2158074e00 con 0x7f215809c310 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f2160616640 10 monclient(hunting): authenticate will time out at 1890.947235s 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215808efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215808f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215808efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52184/0 (socket says 172.21.15.164:52184) 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 1 -- 172.21.15.164:0/4133945044 learned_addr learned my addr 172.21.15.164:0/4133945044 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): get_auth_request con 0x7f215809c310 auth_method 0 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): get_auth_request con 0x7f215813a5c0 auth_method 0 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f215808fa50 0x7f21581375b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 10 monclient(hunting): get_auth_request con 0x7f215808fa50 auth_method 0 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient(hunting): handle_auth_done global_id 164595 payload 824 2024-09-10T16:25:12.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 msgr2=0x7f215808efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f215809c310 0x7f215808efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f215808fa50 msgr2=0x7f21581375b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f215808fa50 0x7f21581375b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 monclient: found mon.noname-a 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215dbb1640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2158091ec0 con 0x7f215813a5c0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.460+0000 7f215ebb3640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f215808fa50 0x7f21581375b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f215dbb1640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 msgr2=0x7f215808f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f215dbb1640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215808f510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f214800c3e0 tx=0x7f214800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21480119a0 con 0x7f215813a5c0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: dump: 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2148017070 con 0x7f215813a5c0 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f2160616640 5 monclient: authenticate success, global_id 164595 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f2160616640 10 monclient: _renew_subs 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f2160616640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2158091900 con 0x7f215813a5c0 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2148011d80 con 0x7f215813a5c0 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f214effd640 10 monclient: dump: 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.462+0000 7f215cbaf640 4 set_mon_vals no callback set 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 10 monclient: _renew_subs 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2158073f10 con 0x7f215813a5c0 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 0x7f215802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f215808fe60 con 0x7f215809d190 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f214801c460 con 0x7f215813a5c0 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 0x7f215802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient: get_auth_request con 0x7f215809d190 auth_method 0 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.463+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f214effd640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f213c03e210 0x7f213c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f215e3b2640 10 monclient(hunting): handle_auth_done global_id 164598 payload 824 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f215e3b2640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f215809f720 con 0x7f215809d190 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(651..651 src has 494..651) ==== 4815+0+0 (secure 0 0 0) 0x7f2148052af0 con 0x7f215813a5c0 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f215ebb3640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f213c03e210 0x7f213c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.464+0000 7f215ebb3640 10 monclient: get_auth_request con 0x7f213c03e210 auth_method 0 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f215e3b2640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 msgr2=0x7f215802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f215e3b2640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 0x7f215802d440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2154011eb0 tx=0x7f2154010810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215401f070 con 0x7f215809d190 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 10 monclient: dump: 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f215400f7f0 con 0x7f215809d190 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:12.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 msgr2=0x7f215802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 0x7f215802d440 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2154011eb0 tx=0x7f2154010810 comp rx=0 tx=0).stop 2024-09-10T16:25:12.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 msgr2=0x7f215802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f214effd640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215809d190 0x7f215802d440 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2154011eb0 tx=0x7f2154010810 comp rx=0 tx=0).stop 2024-09-10T16:25:12.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.466+0000 7f215ebb3640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f213c03e210 0x7f213c0406d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2150033000 tx=0x7f2150009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f2160616640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f2160616640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 0x7f215802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f2160616640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f21580906c0 con 0x7f215802d980 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 0x7f215802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient: get_auth_request con 0x7f215802d980 auth_method 0 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.472+0000 7f215e3b2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.473+0000 7f215e3b2640 10 monclient(hunting): handle_auth_done global_id 164601 payload 824 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.473+0000 7f215e3b2640 1 -- 172.21.15.164:0/4133945044 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f215809d620 con 0x7f215802d980 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f215e3b2640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 msgr2=0x7f215802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f215e3b2640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 0x7f215802dd90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f215401c000 tx=0x7f21540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f215401f040 con 0x7f215802d980 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 10 monclient: dump: 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.486 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f2154021a90 con 0x7f215802d980 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 msgr2=0x7f215802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 0x7f215802dd90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f215401c000 tx=0x7f21540046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 msgr2=0x7f215802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f214effd640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215802d980 0x7f215802dd90 secure :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f215401c000 tx=0x7f21540046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.487 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 283178, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.488 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.489 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.490 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:12.491 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.492 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:12.493 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.495 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.496 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.497 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.498 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.499 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:12.500 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:12.501 INFO:tasks.rados.rados.0.smithi165.stdout:2914: done (0 left) 2024-09-10T16:25:12.504 INFO:tasks.rados.rados.0.smithi165.stdout:2915: snap_create 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f213c03e210 msgr2=0x7f213c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f213c03e210 0x7f213c0406d0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f2150033000 tx=0x7f2150009000 comp rx=0 tx=0).stop 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 10 monclient: shutdown 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 msgr2=0x7f215808f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f2160616640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215808f510 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f214800c3e0 tx=0x7f214800c8a0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.475+0000 7f215ebb3640 1 -- 172.21.15.164:0/4133945044 reap_dead start 2024-09-10T16:25:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.476+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 shutdown_connections 2024-09-10T16:25:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.476+0000 7f2160616640 1 --2- 172.21.15.164:0/4133945044 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f215813a5c0 0x7f215808f510 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.476+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 >> 172.21.15.164:0/4133945044 conn(0x7f21580a3110 msgr2=0x7f21581387d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.476+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 shutdown_connections 2024-09-10T16:25:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.476+0000 7f2160616640 1 -- 172.21.15.164:0/4133945044 wait complete. 2024-09-10T16:25:12.506 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:12.506 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:25:12.506 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:25:12.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.603+0000 7f05dc00c640 10 monclient: get_monmap_and_config 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.603+0000 7f05dc00c640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.603+0000 7f05dc00c640 10 monclient: monmap: 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:12.604304+0000 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:12.604304+0000 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:12.605 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:12.606 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.603+0000 7f05dc00c640 1 Processor -- start 2024-09-10T16:25:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 -- start start 2024-09-10T16:25:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient: init 2024-09-10T16:25:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient: _add_conns ranks=[4,2,1,0,3] 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f05d409d930 0x7f05d409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-c con 0x7f05d409d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-b con 0x7f05d409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f05d4134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05d40a0870 con 0x7f05d4134580 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f05d409d3e0 con 0x7f05d409d930 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05d409e790 con 0x7f05d409ab50 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05dc00c640 10 monclient(hunting): authenticate will time out at 1891.091775s 2024-09-10T16:25:12.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52196/0 (socket says 172.21.15.164:52196) 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 1 -- 172.21.15.164:0/1263322398 learned_addr learned my addr 172.21.15.164:0/1263322398 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): get_auth_request con 0x7f05d409ab50 auth_method 0 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.604+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d9da8640 1 --2- 172.21.15.164:0/1263322398 >> v2:172.21.15.165:3300/0 conn(0x7f05d409d930 0x7f05d409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d9da8640 10 monclient(hunting): get_auth_request con 0x7f05d409d930 auth_method 0 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d9da8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d9da8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d9da8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 10 monclient(hunting): handle_auth_done global_id 214130 payload 274 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 1 -- 172.21.15.164:0/1263322398 >> v2:172.21.15.165:3300/0 conn(0x7f05d409d930 msgr2=0x7f05d409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 1 --2- 172.21.15.164:0/1263322398 >> v2:172.21.15.165:3300/0 conn(0x7f05d409d930 0x7f05d409dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 1 -- 172.21.15.164:0/1263322398 >> v1:172.21.15.165:6789/0 conn(0x7f05d4134580 legacy=0x7f05d4134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 1 monclient: found mon.noname-b 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.605+0000 7f05d95a7640 1 -- 172.21.15.164:0/1263322398 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d4074c70 con 0x7f05d409ab50 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.607+0000 7f05d95a7640 1 -- 172.21.15.164:0/1263322398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 msgr2=0x7f05d409e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.607+0000 7f05d95a7640 1 --2- 172.21.15.164:0/1263322398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009870 tx=0x7f05d002a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61d0f899359fd3ce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 1 -- 172.21.15.164:0/1263322398 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0037070 con 0x7f05d409ab50 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: dump: 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 1 -- 172.21.15.164:0/1263322398 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05d0032640 con 0x7f05d409ab50 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 5 monclient: authenticate success, global_id 214130 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:12.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 1 -- 172.21.15.164:0/1263322398 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d0032a10 con 0x7f05d409ab50 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 4 set_mon_vals no callback set 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05d8da6640 10 monclient: dump: 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 10 monclient: shutdown 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 -- 172.21.15.164:0/1263322398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 msgr2=0x7f05d409e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 --2- 172.21.15.164:0/1263322398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f05d0009870 tx=0x7f05d002a130 comp rx=0 tx=0).stop 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 -- 172.21.15.164:0/1263322398 shutdown_connections 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 --2- 172.21.15.164:0/1263322398 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d409ab50 0x7f05d409e250 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 --2- 172.21.15.164:0/1263322398 >> v2:172.21.15.165:3300/0 conn(0x7f05d409d930 0x7f05d409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 -- 172.21.15.164:0/1263322398 >> 172.21.15.164:0/1263322398 conn(0x7f05d40a3110 msgr2=0x7f05d40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 -- 172.21.15.164:0/1263322398 shutdown_connections 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.608+0000 7f05dc00c640 1 -- 172.21.15.164:0/1263322398 wait complete. 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.609+0000 7f05dc00c640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.609+0000 7f05dc00c640 10 monclient: monmap: 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.609+0000 7f05dc00c640 1 Processor -- start 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.609+0000 7f05dc00c640 1 -- start start 2024-09-10T16:25:12.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.609+0000 7f05dc00c640 10 monclient: init 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05d409ab50 0x7f05d4071dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-b con 0x7f05d409ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 0x7f05d4072310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-a con 0x7f05d409d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): picked mon.noname-c con 0x7f05d4134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f05d409ef30 con 0x7f05d409d930 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05d40a02b0 con 0x7f05d409ab50 2024-09-10T16:25:12.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05d40a2940 con 0x7f05d4134580 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05dc00c640 10 monclient(hunting): authenticate will time out at 1891.097254s 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 0x7f05d4072310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 0x7f05d4072310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38324/0 (socket says 172.21.15.164:38324) 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52204/0 (socket says 172.21.15.164:52204) 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 1 -- 172.21.15.164:0/2192449346 learned_addr learned my addr 172.21.15.164:0/2192449346 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): get_auth_request con 0x7f05d409d930 auth_method 0 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): get_auth_request con 0x7f05d4134580 auth_method 0 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05d409ab50 0x7f05d4071dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): get_auth_request con 0x7f05d409ab50 auth_method 0 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient(hunting): handle_auth_done global_id 214133 payload 824 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05d409ab50 msgr2=0x7f05d4071dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05d409ab50 0x7f05d4071dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 msgr2=0x7f05d4072310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 0x7f05d4072310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 monclient: found mon.noname-c 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05da5a9640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05d4070470 con 0x7f05d4134580 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.610+0000 7f05d95a7640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05d409d930 0x7f05d4072310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.612+0000 7f05da5a9640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 msgr2=0x7f05d41cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.612+0000 7f05da5a9640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00c780 tx=0x7f05cc00cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.612+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc016020 con 0x7f05d4134580 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.612+0000 7f05ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: dump: 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.628 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 5 monclient: authenticate success, global_id 214133 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05cc00ac60 con 0x7f05d4134580 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 monclient: _renew_subs 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05d41342b0 con 0x7f05d4134580 2024-09-10T16:25:12.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05cc012660 con 0x7f05d4134580 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 10 monclient: dump: 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 monclient: _renew_subs 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05cbfff640 4 set_mon_vals no callback set 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05d406ec70 con 0x7f05d4134580 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 0x7f05d41cda10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05d40a1e40 con 0x7f05d41cd630 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.613+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f05cc019020 con 0x7f05d4134580 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 0x7f05d41cda10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient: get_auth_request con 0x7f05d41cd630 auth_method 0 2024-09-10T16:25:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05ca7fc640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05b803e210 0x7f05b80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 10 monclient(hunting): handle_auth_done global_id 214136 payload 824 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d9da8640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05d406dac0 con 0x7f05d41cd630 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f05cc052c60 con 0x7f05d4134580 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d95a7640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05b803e210 0x7f05b80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.614+0000 7f05d95a7640 10 monclient: get_auth_request con 0x7f05b803e210 auth_method 0 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05d9da8640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 msgr2=0x7f05d41cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05d9da8640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 0x7f05d41cda10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05c4010780 tx=0x7f05c4010c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4018020 con 0x7f05d41cd630 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 10 monclient: dump: 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.633 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f05c4004450 con 0x7f05d41cd630 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 msgr2=0x7f05d41cda10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 0x7f05d41cda10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05c4010780 tx=0x7f05c4010c40 comp rx=0 tx=0).stop 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 msgr2=0x7f05d41cda10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.616+0000 7f05ca7fc640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d41cd630 0x7f05d41cda10 secure :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f05c4010780 tx=0x7f05c4010c40 comp rx=0 tx=0).stop 2024-09-10T16:25:12.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.617+0000 7f05d95a7640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05b803e210 0x7f05b80406d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05d002a170 tx=0x7f05d002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.622+0000 7f05dc00c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.622+0000 7f05dc00c640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 0x7f05d4070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.622+0000 7f05dc00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.622+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05d4077260 con 0x7f05d40709b0 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 0x7f05d4070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient: get_auth_request con 0x7f05d40709b0 auth_method 0 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 10 monclient(hunting): handle_auth_done global_id 214139 payload 824 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.623+0000 7f05d9da8640 1 -- 172.21.15.164:0/2192449346 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05d41cdf50 con 0x7f05d40709b0 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05d9da8640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 msgr2=0x7f05d4070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05d9da8640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 0x7f05d4070dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f05c4019000 tx=0x7f05c40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05c4018070 con 0x7f05d40709b0 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 10 monclient: dump: 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f05c400ecb0 con 0x7f05d40709b0 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 msgr2=0x7f05d4070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 0x7f05d4070dc0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f05c4019000 tx=0x7f05c40046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 msgr2=0x7f05d4070dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.625+0000 7f05ca7fc640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d40709b0 0x7f05d4070dc0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12403, 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:12.639 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.640 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.641 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:12.642 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.643 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:12.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.646 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.647 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:12.648 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.649 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.650 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:12.651 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05b803e210 msgr2=0x7f05b80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05b803e210 0x7f05b80406d0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f05d002a170 tx=0x7f05d002e000 comp rx=0 tx=0).stop 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 10 monclient: shutdown 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 msgr2=0x7f05d41cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05da5a9640 1 -- 172.21.15.164:0/2192449346 reap_dead start 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f05cc00c780 tx=0x7f05cc00cc40 comp rx=0 tx=0).stop 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 shutdown_connections 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 --2- 172.21.15.164:0/2192449346 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f05d4134580 0x7f05d41cd0f0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 >> 172.21.15.164:0/2192449346 conn(0x7f05d40a3110 msgr2=0x7f05d409b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 shutdown_connections 2024-09-10T16:25:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.626+0000 7f05dc00c640 1 -- 172.21.15.164:0/2192449346 wait complete. 2024-09-10T16:25:12.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:12.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:25:12.654 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:25:12.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 10 monclient: get_monmap_and_config 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 10 monclient: monmap: 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:12.753062+0000 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:12.753062+0000 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:12.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 1 Processor -- start 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 1 -- start start 2024-09-10T16:25:12.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.752+0000 7f3ec95d3640 10 monclient: init 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ec406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3ec4134f70 0x7f3ec413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec4134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ec413d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:12.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec409ce50 con 0x7f3ec406cb70 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ec4135380 con 0x7f3ec413d560 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3ec406db60 con 0x7f3ec4134f70 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec95d3640 10 monclient(hunting): authenticate will time out at 1891.240338s 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3ec4134f70 0x7f3ec413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec2ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3ec406cb70 0x7f3ec40a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:45768/0 (socket says 172.21.15.164:45768) 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3ec4134f70 0x7f3ec413d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38330/0 (socket says 172.21.15.164:38330) 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/349204011 learned_addr learned my addr 172.21.15.164:0/349204011 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request con 0x7f3ec4134f70 auth_method 0 2024-09-10T16:25:12.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_done global_id 164607 payload 274 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 -- 172.21.15.164:0/349204011 >> v1:172.21.15.165:6789/0 conn(0x7f3ec413d560 legacy=0x7f3ec413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 -- 172.21.15.164:0/349204011 >> v1:172.21.15.164:6789/0 conn(0x7f3ec406cb70 legacy=0x7f3ec40a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 monclient: found mon.noname-a 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.753+0000 7f3ec27fc640 1 -- 172.21.15.164:0/349204011 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec409cb70 con 0x7f3ec4134f70 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec27fc640 1 -- 172.21.15.164:0/349204011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec4134f70 msgr2=0x7f3ec413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec27fc640 1 --2- 172.21.15.164:0/349204011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec4134f70 0x7f3ec413d020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097d0 tx=0x7f3eac02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=897cda4c21ce568a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 1 -- 172.21.15.164:0/349204011 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac037070 con 0x7f3ec4134f70 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: dump: 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.768 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec95d3640 5 monclient: authenticate success, global_id 164607 2024-09-10T16:25:12.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 1 -- 172.21.15.164:0/349204011 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eac032440 con 0x7f3ec4134f70 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec95d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec95d3640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec95d3640 4 set_mon_vals no callback set 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 1 -- 172.21.15.164:0/349204011 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac032810 con 0x7f3ec4134f70 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.756+0000 7f3ec1ffb640 10 monclient: dump: 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.770 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 10 monclient: shutdown 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 -- 172.21.15.164:0/349204011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec4134f70 msgr2=0x7f3ec413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/349204011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec4134f70 0x7f3ec413d020 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3eac0097d0 tx=0x7f3eac02a000 comp rx=0 tx=0).stop 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 -- 172.21.15.164:0/349204011 shutdown_connections 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/349204011 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec4134f70 0x7f3ec413d020 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 -- 172.21.15.164:0/349204011 >> 172.21.15.164:0/349204011 conn(0x7f3ec40aa900 msgr2=0x7f3ec40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 -- 172.21.15.164:0/349204011 shutdown_connections 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 1 -- 172.21.15.164:0/349204011 wait complete. 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.757+0000 7f3ec95d3640 10 monclient: monmap: 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:12.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 Processor -- start 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 -- start start 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient: init 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ec406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ec4134f70 0x7f3ec40a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ec4134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413d560 0x7f3ec40a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ec413d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec40a4db0 con 0x7f3ec406cb70 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec406cf50 con 0x7f3ec413d560 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3ec4141070 con 0x7f3ec4134f70 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec95d3640 10 monclient(hunting): authenticate will time out at 1891.245883s 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec27fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ec4134f70 0x7f3ec40a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38332/0 (socket says 172.21.15.164:38332) 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/3098829714 learned_addr learned my addr 172.21.15.164:0/3098829714 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 10 monclient(hunting): get_auth_request con 0x7f3ec406cb70 auth_method 0 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.758+0000 7f3ec2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request con 0x7f3ec4134f70 auth_method 0 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413d560 0x7f3ec40a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 10 monclient(hunting): get_auth_request con 0x7f3ec413d560 auth_method 0 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient(hunting): handle_auth_done global_id 164610 payload 824 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ec4134f70 msgr2=0x7f3ec40a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3ec4134f70 0x7f3ec40a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413d560 msgr2=0x7f3ec40a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413d560 0x7f3ec40a3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 monclient: found mon.noname-a 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec409dbe0 con 0x7f3ec406cb70 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.759+0000 7f3ec37fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413d560 0x7f3ec40a3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec2ffd640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 msgr2=0x7f3ec41cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec2ffd640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400ba50 tx=0x7f3eb400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4019070 con 0x7f3ec406cb70 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: dump: 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3eb400ce10 con 0x7f3ec406cb70 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 5 monclient: authenticate success, global_id 164610 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb4014680 con 0x7f3ec406cb70 2024-09-10T16:25:12.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 10 monclient: _renew_subs 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec406e720 con 0x7f3ec406cb70 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ebb7fe640 10 monclient: dump: 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec17fa640 4 set_mon_vals no callback set 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 10 monclient: _renew_subs 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:12.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.761+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec409e180 con 0x7f3ec406cb70 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec95d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 0x7f3ec413a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec40a44e0 con 0x7f3ec413a060 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3eb401c070 con 0x7f3ec406cb70 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec27fc640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 0x7f3ec413a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec27fc640 10 monclient: get_auth_request con 0x7f3ec413a060 auth_method 0 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.762+0000 7f3ebb7fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3ea403e210 0x7f3ea40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f3eb4052b80 con 0x7f3ec406cb70 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec37fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3ea403e210 0x7f3ea40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec37fe640 10 monclient: get_auth_request con 0x7f3ea403e210 auth_method 0 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_done global_id 154597 payload 824 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.763+0000 7f3ec27fc640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ec40a46a0 con 0x7f3ec413a060 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ec27fc640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 msgr2=0x7f3ec413a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ec27fc640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 0x7f3ec413a440 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3eac034000 tx=0x7f3eac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ec37fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3ea403e210 0x7f3ea40406d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406a520 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac037040 con 0x7f3ec413a060 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ebb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.765+0000 7f3ebb7fe640 10 monclient: dump: 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3eac03c620 con 0x7f3ec413a060 2024-09-10T16:25:12.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:12.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 msgr2=0x7f3ec413a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 0x7f3ec413a440 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3eac034000 tx=0x7f3eac0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 msgr2=0x7f3ec413a440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.766+0000 7f3ebb7fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a060 0x7f3ec413a440 secure :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3eac034000 tx=0x7f3eac0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec95d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 0x7f3ec413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec95d3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec409e520 con 0x7f3ec413a980 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 0x7f3ec413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient: get_auth_request con 0x7f3ec413a980 auth_method 0 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 10 monclient(hunting): handle_auth_done global_id 154600 payload 824 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.772+0000 7f3ec27fc640 1 -- 172.21.15.164:0/3098829714 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3ec40770f0 con 0x7f3ec413a980 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ec27fc640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 msgr2=0x7f3ec413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ec27fc640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 0x7f3ec413d940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3eac02a040 tx=0x7f3eac008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac04b040 con 0x7f3ec413a980 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 10 monclient: dump: 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.787 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f3eac046820 con 0x7f3ec413a980 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 msgr2=0x7f3ec413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 0x7f3ec413d940 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3eac02a040 tx=0x7f3eac008030 comp rx=0 tx=0).stop 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 msgr2=0x7f3ec413d940 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:12.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.775+0000 7f3ebb7fe640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3ec413a980 0x7f3ec413d940 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 186, 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 331325, 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.789 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.790 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.791 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:12.792 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:12.793 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.794 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:12.796 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.797 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:12.798 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:12.799 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:12.800 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.801 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:12.802 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:12.803 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:12.803 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:12.803 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:12.803 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3ea403e210 msgr2=0x7f3ea40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3ea403e210 0x7f3ea40406d0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f3ec406a520 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 10 monclient: shutdown 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 msgr2=0x7f3ec41cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400ba50 tx=0x7f3eb400bf10 comp rx=0 tx=0).stop 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec37fe640 1 -- 172.21.15.164:0/3098829714 reap_dead start 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 shutdown_connections 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 --2- 172.21.15.164:0/3098829714 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3ec406cb70 0x7f3ec41cd530 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 >> 172.21.15.164:0/3098829714 conn(0x7f3ec40aa900 msgr2=0x7f3ec41335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 shutdown_connections 2024-09-10T16:25:12.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.776+0000 7f3ec95d3640 1 -- 172.21.15.164:0/3098829714 wait complete. 2024-09-10T16:25:12.806 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:12.806 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:25:12.806 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:25:12.806 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:25:12.806 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:25:12.806 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:25:12.806 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:25:12.820 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:25:12.820 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:25:12.820 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:12.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 10 monclient: get_monmap_and_config 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 10 monclient: monmap: 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:12.957233+0000 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:12.957233+0000 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:12.958 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:12.959 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:12.959 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 1 Processor -- start 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 1 -- start start 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.956+0000 7fded5332640 10 monclient: init 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fded006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fded0134f70 0x7fded013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): picked mon.noname-a con 0x7fded0134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fded013d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded009ce50 con 0x7fded013d560 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fded0135380 con 0x7fded006cb70 2024-09-10T16:25:12.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fded006db60 con 0x7fded0134f70 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fded5332640 10 monclient(hunting): authenticate will time out at 1891.444515s 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fdece7fc640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fded0134f70 msgr2=0x7fded013d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fdecf7fe640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fded013d560 legacy=0x7fded013f950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fdece7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fded0134f70 0x7fded013d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fdeceffd640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fded006cb70 0x7fded00a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:45716/0 (socket says 172.21.15.164:45716) 2024-09-10T16:25:12.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.957+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 learned_addr learned my addr 172.21.15.164:0/2029354015 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 >> v1:172.21.15.165:6789/0 conn(0x7fded006cb70 legacy=0x7fded00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2177695307 0 0) 0x7fded0135380 con 0x7fded006cb70 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): _negotiate 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): my global_id is 154603 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdeb8003520 con 0x7fded006cb70 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3705500688 0 0) 0x7fdeb8003520 con 0x7fded006cb70 2024-09-10T16:25:12.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 >> v2:172.21.15.164:3300/0 conn(0x7fded0134f70 msgr2=0x7fded013d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 --2- 172.21.15.164:0/2029354015 >> v2:172.21.15.164:3300/0 conn(0x7fded0134f70 0x7fded013d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 >> v1:172.21.15.164:6789/0 conn(0x7fded013d560 legacy=0x7fded013f950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded009cb70 con 0x7fded006cb70 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fdec8001a60 con 0x7fded006cb70 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: dump: 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 >> v1:172.21.15.165:6789/0 conn(0x7fded006cb70 legacy=0x7fded00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 0x7fdeb8003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): picked mon.b con 0x7fdeb8003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8004800 0x7fdeb8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): picked mon.c con 0x7fdeb8004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdeb8005410 0x7fdeb800ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): picked mon.a con 0x7fdeb8005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb800c2f0 con 0x7fdeb8005410 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdeb800c4a0 con 0x7fdeb8003a20 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdeb800c6a0 con 0x7fdeb8004800 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecdffb640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecf7fe640 1 -- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdeb8005410 msgr2=0x7fdeb800ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdecf7fe640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdeb8005410 0x7fdeb800ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 0x7fdeb8003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8004800 0x7fdeb8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): get_auth_request con 0x7fdeb8003a20 auth_method 0 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): get_auth_request con 0x7fdeb8004800 auth_method 0 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.960+0000 7fdece7fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 10 monclient(hunting): handle_auth_done global_id 154603 payload 293 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8004800 msgr2=0x7fdeb8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8004800 0x7fdeb8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdeb8005410 msgr2=0x7fdeb800ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdeb8005410 0x7fdeb800ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 monclient: found mon.b 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdecf7fe640 1 -- 172.21.15.164:0/2029354015 reap_dead start 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdeb8003520 con 0x7fdeb8003a20 2024-09-10T16:25:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fdeceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fded5332640 5 monclient: authenticate success, global_id 154603 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.961+0000 7fded5332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.963+0000 7fdeceffd640 1 -- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 msgr2=0x7fdeb8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.963+0000 7fdeceffd640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 0x7fdeb8003e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c3b0 tx=0x7fdeb8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1fc9f96f8b3ce88 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8035070 con 0x7fdeb8003a20 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: dump: 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec8005ad0 con 0x7fdeb8003a20 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 1 -- 172.21.15.164:0/2029354015 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec800c0e0 con 0x7fdeb8003a20 2024-09-10T16:25:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 4 set_mon_vals no callback set 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fdecdffb640 10 monclient: dump: 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 10 monclient: shutdown 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 -- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 msgr2=0x7fdeb8003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 0x7fdeb8003e20 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fdec800c3b0 tx=0x7fdeb8005830 comp rx=0 tx=0).stop 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 -- 172.21.15.164:0/2029354015 shutdown_connections 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 --2- 172.21.15.164:0/2029354015 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdeb8003a20 0x7fdeb8003e20 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 -- 172.21.15.164:0/2029354015 >> 172.21.15.164:0/2029354015 conn(0x7fded00aa900 msgr2=0x7fded00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 -- 172.21.15.164:0/2029354015 shutdown_connections 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.964+0000 7fded5332640 1 -- 172.21.15.164:0/2029354015 wait complete. 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.965+0000 7fded5332640 10 monclient: build_initial_monmap 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.965+0000 7fded5332640 10 monclient: monmap: 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.965+0000 7fded5332640 1 Processor -- start 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 -- start start 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient: init 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): picked mon.noname-c con 0x7fdeb8003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 0x7fded013a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): picked mon.noname-a con 0x7fded013a030 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded013a9d0 0x7fded01d3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): picked mon.noname-b con 0x7fded013a9d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fded00a46a0 con 0x7fded013a030 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fded006cf50 con 0x7fded013a9d0 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fded00a4820 con 0x7fdeb8003a20 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fded5332640 10 monclient(hunting): authenticate will time out at 1891.453511s 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdece7fc640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 msgr2=0x7fded013a490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdece7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 0x7fded013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:52290/0 (socket says 172.21.15.164:52290) 2024-09-10T16:25:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 -- 172.21.15.164:0/1513787618 learned_addr learned my addr 172.21.15.164:0/1513787618 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): get_auth_request con 0x7fdeb8003a20 auth_method 0 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdecf7fe640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded013a9d0 0x7fded01d3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdecf7fe640 10 monclient(hunting): get_auth_request con 0x7fded013a9d0 auth_method 0 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdecf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdecf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdecf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient(hunting): handle_auth_done global_id 214145 payload 824 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 -- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded013a9d0 msgr2=0x7fded01d3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded013a9d0 0x7fded01d3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 -- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 msgr2=0x7fded013a490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 0x7fded013a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 monclient: found mon.noname-c 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.966+0000 7fdeceffd640 1 -- 172.21.15.164:0/1513787618 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fded006e720 con 0x7fdeb8003a20 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeceffd640 1 -- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 msgr2=0x7fded0139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeceffd640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdec8000c00 tx=0x7fded009cd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8035030 con 0x7fdeb8003a20 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: dump: 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 5 monclient: authenticate success, global_id 214145 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdec8041070 con 0x7fdeb8003a20 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 10 monclient: _renew_subs 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fded009d6d0 con 0x7fdeb8003a20 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdec8004e40 con 0x7fdeb8003a20 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdeb77fe640 10 monclient: dump: 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 10 monclient: _renew_subs 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fded013cc50 con 0x7fdeb8003a20 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.969+0000 7fdecd7fa640 4 set_mon_vals no callback set 2024-09-10T16:25:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdec8007070 con 0x7fdeb8003a20 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fded5332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fded5332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fded5332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fded00a9e50 con 0x7fdeb8003a20 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fdeb77fe640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 0x7fdebc040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7fdec8072ac0 con 0x7fdeb8003a20 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fdece7fc640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 0x7fdebc040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.970+0000 7fdece7fc640 10 monclient: get_auth_request con 0x7fdebc03e1c0 auth_method 0 2024-09-10T16:25:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.973+0000 7fdece7fc640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 0x7fdebc040680 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fded013bc80 tx=0x7fdec4009520 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.974+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fded00a9e50 con 0x7fdeb8003a20 2024-09-10T16:25:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.974+0000 7fdeb77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:12.974+0000 7fdeb77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.219+0000 7fded5332640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.219+0000 7fded5332640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.219+0000 7fded5332640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.219+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fded0135380 con 0x7fdeb8003a20 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.220+0000 7fdeb77fe640 1 -- 172.21.15.164:0/1513787618 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fdec804b060 con 0x7fdeb8003a20 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.220+0000 7fdeb77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.220+0000 7fdeb77fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:13.222 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":186,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 msgr2=0x7fdebc040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 0x7fdebc040680 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fded013bc80 tx=0x7fdec4009520 comp rx=0 tx=0).stop 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 10 monclient: shutdown 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 msgr2=0x7fded0139af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fdec8000c00 tx=0x7fded009cd40 comp rx=0 tx=0).stop 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 shutdown_connections 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdebc03e1c0 0x7fdebc040680 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fded013a9d0 0x7fded01d3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fded013a030 0x7fded013a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:13.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 --2- 172.21.15.164:0/1513787618 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdeb8003a20 0x7fded0139af0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 >> 172.21.15.164:0/1513787618 conn(0x7fded00aa900 msgr2=0x7fded0131840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 shutdown_connections 2024-09-10T16:25:13.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:13.222+0000 7fded5332640 1 -- 172.21.15.164:0/1513787618 wait complete. 2024-09-10T16:25:16.235 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:16.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.343+0000 7f3bb40c6640 10 monclient: get_monmap_and_config 2024-09-10T16:25:16.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.343+0000 7f3bb40c6640 10 monclient: build_initial_monmap 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.343+0000 7f3bb40c6640 10 monclient: monmap: 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:16.344144+0000 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:16.344144+0000 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:16.345 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.343+0000 7f3bb40c6640 1 Processor -- start 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 -- start start 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient: init 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-10T16:25:16.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 0x7f3bac134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3bac0a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3bac0a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3bac09ce50 con 0x7f3bac0a46a0 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3bac06db60 con 0x7f3bac06cb70 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3bac06cf50 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb40c6640 10 monclient(hunting): authenticate will time out at 1894.831619s 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb1e62640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 msgr2=0x7f3bac134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:16.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb1e62640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 0x7f3bac134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55496/0 (socket says 172.21.15.164:55496) 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 1 -- 172.21.15.164:0/1495131983 learned_addr learned my addr 172.21.15.164:0/1495131983 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): get_auth_request con 0x7f3bac0a4ab0 auth_method 0 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.344+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 10 monclient(hunting): handle_auth_done global_id 214148 payload 274 2024-09-10T16:25:16.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 1 -- 172.21.15.164:0/1495131983 >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 msgr2=0x7f3bac134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 1 --2- 172.21.15.164:0/1495131983 >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 0x7f3bac134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 1 -- 172.21.15.164:0/1495131983 >> v1:172.21.15.165:6789/0 conn(0x7f3bac0a46a0 legacy=0x7f3bac137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 10 monclient: _finish_hunting 0 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 1 monclient: found mon.noname-b 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:16.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.345+0000 7f3bb2663640 1 -- 172.21.15.164:0/1495131983 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bac09cb70 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.347+0000 7f3bb2663640 1 -- 172.21.15.164:0/1495131983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 msgr2=0x7f3bac1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.347+0000 7f3bb2663640 1 --2- 172.21.15.164:0/1495131983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009e80 tx=0x7f3b9c02a5b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ed898a320b8df97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 1 -- 172.21.15.164:0/1495131983 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c036070 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: dump: 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.357 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: _finish_auth 0 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: _check_auth_tickets 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 1 -- 172.21.15.164:0/1495131983 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b9c007800 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 5 monclient: authenticate success, global_id 214148 2024-09-10T16:25:16.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 1 -- 172.21.15.164:0/1495131983 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c007bd0 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 4 set_mon_vals no callback set 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb0e60640 10 monclient: dump: 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.359 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 10 monclient: shutdown 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 -- 172.21.15.164:0/1495131983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 msgr2=0x7f3bac1383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/1495131983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c009e80 tx=0x7f3b9c02a5b0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 -- 172.21.15.164:0/1495131983 shutdown_connections 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/1495131983 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1383b0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/1495131983 >> v2:172.21.15.164:3300/0 conn(0x7f3bac06cb70 0x7f3bac134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 -- 172.21.15.164:0/1495131983 >> 172.21.15.164:0/1495131983 conn(0x7f3bac0aa900 msgr2=0x7f3bac0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 -- 172.21.15.164:0/1495131983 shutdown_connections 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.348+0000 7f3bb40c6640 1 -- 172.21.15.164:0/1495131983 wait complete. 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 10 monclient: build_initial_monmap 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 10 monclient: monmap: 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:16.361 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 1 Processor -- start 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 1 -- start start 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 10 monclient: init 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.349+0000 7f3bb40c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 0x7f3bac1cd4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3bac06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 0x7f3bac0a3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3bac0a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3bac0a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3bac138a70 con 0x7f3bac0a46a0 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3bac1388f0 con 0x7f3bac06cb70 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3bac140040 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb40c6640 10 monclient(hunting): authenticate will time out at 1894.837066s 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1661640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 msgr2=0x7f3bac0a3d10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1661640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 0x7f3bac0a3d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55506/0 (socket says 172.21.15.164:55506) 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 -- 172.21.15.164:0/620727452 learned_addr learned my addr 172.21.15.164:0/620727452 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:16.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): get_auth_request con 0x7f3bac0a4ab0 auth_method 0 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 0x7f3bac1cd4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 10 monclient(hunting): get_auth_request con 0x7f3bac06cb70 auth_method 0 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient(hunting): handle_auth_done global_id 214151 payload 824 2024-09-10T16:25:16.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 -- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 msgr2=0x7f3bac1cd4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 0x7f3bac1cd4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 -- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 msgr2=0x7f3bac0a3d10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 0x7f3bac0a3d10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient: _finish_hunting 0 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 monclient: found mon.noname-c 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb2663640 1 -- 172.21.15.164:0/620727452 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3bac0a04a0 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.350+0000 7f3bb1e62640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 0x7f3bac1cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb2663640 1 -- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 msgr2=0x7f3bac1c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb2663640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c034040 tx=0x7f3b9c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb2663640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c034040 tx=0x7f3b9c004a40 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f3b9c041030 mon_map magic: 0 2024-09-10T16:25:16.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c041030 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: dump: 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:16.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b9c007600 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb40c6640 5 monclient: authenticate success, global_id 214151 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb40c6640 10 monclient: _renew_subs 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb40c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3bac0a2800 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3baa7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.352+0000 7f3babfff640 4 set_mon_vals no callback set 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3bb40c6640 10 monclient: _renew_subs 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3bb40c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3bac06e720 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9c036040 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3baa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3baa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3baa7fc640 10 monclient: dump: 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.353+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3b9c044070 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb40c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb40c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb40c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3bac134f70 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3baa7fc640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 0x7f3b98040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb1e62640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 0x7f3b98040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.354+0000 7f3bb1e62640 10 monclient: get_auth_request con 0x7f3b9803e1c0 auth_method 0 2024-09-10T16:25:16.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.357+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:25:16.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.357+0000 7f3bb1e62640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 0x7f3b98040680 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4003980 tx=0x7f3ba4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:16.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.359+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f3b9c073ac0 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.975+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3b9c04b070 con 0x7f3bac0a4ab0 2024-09-10T16:25:16.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.975+0000 7f3baa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:16.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:16.975+0000 7f3baa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.222+0000 7f3bb40c6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.222+0000 7f3bb40c6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.222+0000 7f3bb40c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.222+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3bac1386d0 con 0x7f3bac0a4ab0 2024-09-10T16:25:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.223+0000 7f3baa7fc640 1 -- 172.21.15.164:0/620727452 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f3bac09ce50 con 0x7f3bac0a4ab0 2024-09-10T16:25:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.223+0000 7f3baa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.223+0000 7f3baa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:17.225 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":186,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 msgr2=0x7f3b98040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 0x7f3b98040680 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4003980 tx=0x7f3ba4008040 comp rx=0 tx=0).stop 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 10 monclient: shutdown 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 msgr2=0x7f3bac1c9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3b9c034040 tx=0x7f3b9c004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 shutdown_connections 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b9803e1c0 0x7f3b98040680 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3bac0a4ab0 0x7f3bac1c9f90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:17.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3bac0a46a0 0x7f3bac0a3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:17.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 --2- 172.21.15.164:0/620727452 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3bac06cb70 0x7f3bac1cd4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:17.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 >> 172.21.15.164:0/620727452 conn(0x7f3bac0aa900 msgr2=0x7f3bac12e090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:17.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 shutdown_connections 2024-09-10T16:25:17.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:17.225+0000 7f3bb40c6640 1 -- 172.21.15.164:0/620727452 wait complete. 2024-09-10T16:25:20.238 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:20.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.345+0000 7f117a5a1640 10 monclient: get_monmap_and_config 2024-09-10T16:25:20.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.345+0000 7f117a5a1640 10 monclient: build_initial_monmap 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.345+0000 7f117a5a1640 10 monclient: monmap: 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:20.346235+0000 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:20.346235+0000 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:20.347 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.345+0000 7f117a5a1640 1 Processor -- start 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 -- start start 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient: init 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f117409ba80 0x7f11740a42d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-c con 0x7f117409ba80 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f117406cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:20.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 0x7f117413f590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-a con 0x7f117406d6b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f117406dbf0 con 0x7f117406cb70 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f11740a4810 con 0x7f117406d6b0 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f117406ea40 con 0x7f117409ba80 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f117a5a1640 10 monclient(hunting): authenticate will time out at 1898.833759s 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f1179da0640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 msgr2=0x7f117413f590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.346+0000 7f1179da0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 0x7f117413f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f117409ba80 0x7f11740a42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f117409ba80 0x7f11740a42d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:48608/0 (socket says 172.21.15.164:48608) 2024-09-10T16:25:20.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f1178d9e640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f117406cb70 0x7f117406cfa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:58062/0 (socket says 172.21.15.164:58062) 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 -- 172.21.15.164:0/691596386 learned_addr learned my addr 172.21.15.164:0/691596386 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): get_auth_request con 0x7f117409ba80 auth_method 0 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient(hunting): handle_auth_done global_id 154612 payload 274 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 -- 172.21.15.164:0/691596386 >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 msgr2=0x7f117413f590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 --2- 172.21.15.164:0/691596386 >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 0x7f117413f590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 -- 172.21.15.164:0/691596386 >> v1:172.21.15.165:6789/0 conn(0x7f117406cb70 legacy=0x7f117406cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:20.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient: _finish_hunting 0 2024-09-10T16:25:20.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 monclient: found mon.noname-c 2024-09-10T16:25:20.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:20.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:20.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.347+0000 7f117959f640 1 -- 172.21.15.164:0/691596386 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117406e130 con 0x7f117409ba80 2024-09-10T16:25:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117959f640 1 -- 172.21.15.164:0/691596386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117409ba80 msgr2=0x7f11740a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117959f640 1 --2- 172.21.15.164:0/691596386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117409ba80 0x7f11740a42d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1174070070 tx=0x7f117002a0d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c09bf36c7aa7f746 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 1 -- 172.21.15.164:0/691596386 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170037070 con 0x7f117409ba80 2024-09-10T16:25:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:20.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: dump: 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:20.358 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: _finish_auth 0 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 1 -- 172.21.15.164:0/691596386 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1170032640 con 0x7f117409ba80 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117a5a1640 5 monclient: authenticate success, global_id 154612 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117a5a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117a5a1640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 1 -- 172.21.15.164:0/691596386 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170032a10 con 0x7f117409ba80 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f117a5a1640 4 set_mon_vals no callback set 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:20.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.350+0000 7f116bfff640 10 monclient: dump: 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 10 monclient: shutdown 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 -- 172.21.15.164:0/691596386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117409ba80 msgr2=0x7f11740a42d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 --2- 172.21.15.164:0/691596386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117409ba80 0x7f11740a42d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f1174070070 tx=0x7f117002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 -- 172.21.15.164:0/691596386 shutdown_connections 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 --2- 172.21.15.164:0/691596386 >> v2:172.21.15.164:3300/0 conn(0x7f117406d6b0 0x7f117413f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 --2- 172.21.15.164:0/691596386 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117409ba80 0x7f11740a42d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 -- 172.21.15.164:0/691596386 >> 172.21.15.164:0/691596386 conn(0x7f11740a4ee0 msgr2=0x7f11740a52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 -- 172.21.15.164:0/691596386 shutdown_connections 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 1 -- 172.21.15.164:0/691596386 wait complete. 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.351+0000 7f117a5a1640 10 monclient: build_initial_monmap 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient: monmap: 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:20.361 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 Processor -- start 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 -- start start 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient: init 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:20.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-b con 0x7f117406cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 0x7f117413ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-a con 0x7f117406d6b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): picked mon.noname-c con 0x7f117409ba80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f117406e410 con 0x7f117406d6b0 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1174069e70 con 0x7f117406cb70 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f117413fce0 con 0x7f117409ba80 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f1178d9e640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 msgr2=0x7f117413ed60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:20.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f117a5a1640 10 monclient(hunting): authenticate will time out at 1898.839949s 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.352+0000 7f1178d9e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 0x7f117413ed60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55510/0 (socket says 172.21.15.164:55510) 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 -- 172.21.15.164:0/584121281 learned_addr learned my addr 172.21.15.164:0/584121281 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): get_auth_request con 0x7f117409ba80 auth_method 0 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f117959f640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f117959f640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 msgr2=0x7f117413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:25:20.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f117959f640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 msgr2=0x7f117413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f117959f640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f117959f640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient(hunting): handle_auth_done global_id 214154 payload 824 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 msgr2=0x7f117413e820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 msgr2=0x7f117413ed60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 0x7f117413ed60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient: _finish_hunting 0 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 monclient: found mon.noname-c 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.353+0000 7f1179da0640 1 -- 172.21.15.164:0/584121281 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117409f5d0 con 0x7f117409ba80 2024-09-10T16:25:20.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1179da0640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 msgr2=0x7f1174093420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1179da0640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f116c00be60 tx=0x7f116c00bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c016630 con 0x7f117409ba80 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: dump: 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:20.366 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1179da0640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f116c00be60 tx=0x7f116c00bf50 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f116c010030 config(0 keys) 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f117a5a1640 5 monclient: authenticate success, global_id 214154 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f117a5a1640 10 monclient: _renew_subs 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f117a5a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1174096e70 con 0x7f117409ba80 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116c010030 con 0x7f117409ba80 2024-09-10T16:25:20.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f1169ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:20.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.355+0000 7f116b7fe640 4 set_mon_vals no callback set 2024-09-10T16:25:20.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 10 monclient: _renew_subs 2024-09-10T16:25:20.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:20.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1174097b30 con 0x7f117409ba80 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c016a10 con 0x7f117409ba80 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 10 monclient: dump: 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:20.369 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f116c0153e0 con 0x7f117409ba80 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f1169ffb640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 0x7f11580406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:20.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.356+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11740a42d0 con 0x7f117409ba80 2024-09-10T16:25:20.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.357+0000 7f117959f640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 0x7f11580406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:20.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.357+0000 7f117959f640 10 monclient: get_auth_request con 0x7f115803e1f0 auth_method 0 2024-09-10T16:25:20.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.359+0000 7f117959f640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 0x7f11580406b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1170031000 tx=0x7f117002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:20.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.361+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f116c053900 con 0x7f117409ba80 2024-09-10T16:25:20.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.979+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f116c0522f0 con 0x7f117409ba80 2024-09-10T16:25:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.979+0000 7f1169ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:20.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:20.979+0000 7f1169ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f117a5a1640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f117a5a1640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f117a5a1640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1174076eb0 con 0x7f117409ba80 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f1169ffb640 1 -- 172.21.15.164:0/584121281 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f116c03e080 con 0x7f117409ba80 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f1169ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.223+0000 7f1169ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:21.225 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":186,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:21.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.225+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 msgr2=0x7f11580406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:21.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.225+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 0x7f11580406b0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f1170031000 tx=0x7f117002e000 comp rx=0 tx=0).stop 2024-09-10T16:25:21.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 10 monclient: shutdown 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 msgr2=0x7f1174093420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f116c00be60 tx=0x7f116c00bf50 comp rx=0 tx=0).stop 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 shutdown_connections 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f115803e1f0 0x7f11580406b0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f117409ba80 0x7f1174093420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f117406d6b0 0x7f117413ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 --2- 172.21.15.164:0/584121281 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f117406cb70 0x7f117413e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 >> 172.21.15.164:0/584121281 conn(0x7f11740a4ee0 msgr2=0x7f117409e0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 shutdown_connections 2024-09-10T16:25:21.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:21.226+0000 7f117a5a1640 1 -- 172.21.15.164:0/584121281 wait complete. 2024-09-10T16:25:24.239 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 10 monclient: get_monmap_and_config 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 10 monclient: build_initial_monmap 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 10 monclient: monmap: 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:24.350 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:24.349922+0000 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:24.349922+0000 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:24.351 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 1 Processor -- start 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 1 -- start start 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.349+0000 7fe33e905640 10 monclient: init 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-c con 0x7fe33806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe3380a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-b con 0x7fe3380a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:24.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33809ce50 con 0x7fe3380a46a0 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe33806db60 con 0x7fe33806cb70 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe33806cf50 con 0x7fe3380a4ab0 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33e905640 10 monclient(hunting): authenticate will time out at 1902.837492s 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe3377fe640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fe3380a46a0 legacy=0x7fe338137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60678/0 (socket says 172.21.15.164:60678) 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 learned_addr learned my addr 172.21.15.164:0/3118118606 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 10 monclient(hunting): get_auth_request con 0x7fe3380a4ab0 auth_method 0 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe33806cb70 auth_method 0 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.350+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe3377fe640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.164:6789/0 conn(0x7fe3380a46a0 legacy=0x7fe338137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 10 monclient(hunting): get_auth_request con 0x7fe3380a4ab0 auth_method 0 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:24.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe33806cb70 auth_method 0 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.551+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:25:24.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe3377fe640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.164:6789/0 conn(0x7fe3380a46a0 legacy=0x7fe338137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:24.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe33806cb70 auth_method 0 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe33cea2640 10 monclient(hunting): get_auth_request con 0x7fe3380a4ab0 auth_method 0 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe33cea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:24.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe33cea2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:24.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.952+0000 7fe33cea2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:24.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.953+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:24.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.953+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:24.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.953+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:24.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:24.953+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:25:25.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe3377fe640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.164:6789/0 conn(0x7fe3380a46a0 legacy=0x7fe338137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe33806cb70 auth_method 0 2024-09-10T16:25:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:25.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:25.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.753+0000 7fe337fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:25.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.754+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:25.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.754+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:25.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.754+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:25.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.754+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 10 monclient(hunting): get_auth_request con 0x7fe3380a4ab0 auth_method 0 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:25.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:25.755+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): tick 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 monclient(hunting): continuing hunt 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 msgr2=0x7fe3381383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe3380a4ab0 0x7fe3381383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 msgr2=0x7fe338134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.165:3300/0 conn(0x7fe33806cb70 0x7fe338134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.164:6789/0 conn(0x7fe3380a46a0 legacy=0x7fe338137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.164:3300/0 conn(0x7fe3240036d0 0x7fe324005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fe3240036d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe324006220 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 -- 172.21.15.164:0/3118118606 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe33809cb70 con 0x7fe324006220 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 1 -- 172.21.15.164:0/3118118606 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fe33806cf50 con 0x7fe3240036d0 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe335ffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.164:3300/0 conn(0x7fe3240036d0 msgr2=0x7fe324005b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:27.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.351+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.164:3300/0 conn(0x7fe3240036d0 0x7fe324005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:27.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.353+0000 7fe3377fe640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.165:6789/0 conn(0x7fe324006220 legacy=0x7fe3240086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2414966109 0 0) 0x7fe33809cb70 con 0x7fe324006220 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 10 monclient(hunting): _negotiate 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 10 monclient(hunting): my global_id is 154615 2024-09-10T16:25:27.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.496+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe320003520 con 0x7fe324006220 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.497+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 758311293 0 0) 0x7fe320003520 con 0x7fe324006220 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 >> v2:172.21.15.164:3300/0 conn(0x7fe3240036d0 msgr2=0x7fe324005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 --2- 172.21.15.164:0/3118118606 >> v2:172.21.15.164:3300/0 conn(0x7fe3240036d0 0x7fe324005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe32400aff0 con 0x7fe324006220 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:27.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fe32c004600 con 0x7fe324006220 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: dump: 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.503 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 >> v1:172.21.15.165:6789/0 conn(0x7fe324006220 legacy=0x7fe3240086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 1 -- 172.21.15.164:0/3118118606 reap_dead start 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 0x7fe320003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): picked mon.c con 0x7fe320003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 0x7fe320004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): picked mon.a con 0x7fe320004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320005410 0x7fe32000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): picked mon.b con 0x7fe320005410 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe33806cf50 con 0x7fe320004800 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe32000c320 con 0x7fe320005410 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe32000c520 con 0x7fe320003a20 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe336ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe3377fe640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 msgr2=0x7fe320004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe3377fe640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 0x7fe320004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 0x7fe320003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe320003a20 auth_method 0 2024-09-10T16:25:27.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320005410 0x7fe32000ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): get_auth_request con 0x7fe320005410 auth_method 0 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33cea2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:25:27.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient(hunting): handle_auth_done global_id 154615 payload 293 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320005410 msgr2=0x7fe32000ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320005410 0x7fe32000ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 msgr2=0x7fe320004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 0x7fe320004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 monclient: found mon.c 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe320003520 con 0x7fe320003a20 2024-09-10T16:25:27.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient: _finish_auth 0 2024-09-10T16:25:27.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:27.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe337fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:27.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33e905640 5 monclient: authenticate success, global_id 154615 2024-09-10T16:25:27.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.498+0000 7fe33e905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:27.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe337fff640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 msgr2=0x7fe320003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe337fff640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 0x7fe320003e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe328003a10 tx=0x7fe320005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fd6d45562c2ec97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328031070 con 0x7fe320003a20 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: dump: 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32802b7a0 con 0x7fe320003a20 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe33e905640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe33e905640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 1 -- 172.21.15.164:0/3118118606 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe328003610 con 0x7fe320003a20 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe33e905640 4 set_mon_vals no callback set 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.501+0000 7fe336ffd640 10 monclient: dump: 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.515 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 10 monclient: shutdown 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 -- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 msgr2=0x7fe320003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 0x7fe320003e20 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe328003a10 tx=0x7fe320005830 comp rx=0 tx=0).stop 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 -- 172.21.15.164:0/3118118606 shutdown_connections 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320005410 0x7fe32000ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320004800 0x7fe320004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 --2- 172.21.15.164:0/3118118606 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320003a20 0x7fe320003e20 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 -- 172.21.15.164:0/3118118606 >> 172.21.15.164:0/3118118606 conn(0x7fe3380aa900 msgr2=0x7fe3380aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 -- 172.21.15.164:0/3118118606 shutdown_connections 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.502+0000 7fe33e905640 1 -- 172.21.15.164:0/3118118606 wait complete. 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.503+0000 7fe33e905640 10 monclient: build_initial_monmap 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.503+0000 7fe33e905640 10 monclient: monmap: 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:27.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 Processor -- start 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 -- start start 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient: init 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a20 0x7fe338071d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-b con 0x7fe320003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-c con 0x7fe320004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 0x7fe3380a26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): picked mon.noname-a con 0x7fe320005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:27.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe3381350f0 con 0x7fe320005410 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe338134f70 con 0x7fe320003a20 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe338135270 con 0x7fe320004800 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.504+0000 7fe33e905640 10 monclient(hunting): authenticate will time out at 1905.991990s 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe33cea2640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 msgr2=0x7fe3380a26d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe33cea2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 0x7fe3380a26d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60724/0 (socket says 172.21.15.164:60724) 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 -- 172.21.15.164:0/393768657 learned_addr learned my addr 172.21.15.164:0/393768657 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:27.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): get_auth_request con 0x7fe320004800 auth_method 0 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a20 0x7fe338071d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): get_auth_request con 0x7fe320003a20 auth_method 0 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe337fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient(hunting): handle_auth_done global_id 214157 payload 824 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 -- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a20 msgr2=0x7fe338071d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a20 0x7fe338071d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 -- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 msgr2=0x7fe3380a26d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 0x7fe3380a26d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient: _finish_hunting 0 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 monclient: found mon.noname-c 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.505+0000 7fe3377fe640 1 -- 172.21.15.164:0/393768657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe338134350 con 0x7fe320004800 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.507+0000 7fe3377fe640 1 -- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 msgr2=0x7fe3380a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.507+0000 7fe3377fe640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c680 tx=0x7fe32c00cb40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c004ce0 con 0x7fe320004800 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: dump: 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.522 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: _finish_auth 0 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 5 monclient: authenticate success, global_id 214157 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe32c007110 con 0x7fe320004800 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 10 monclient: _renew_subs 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe338070430 con 0x7fe320004800 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0074e0 con 0x7fe320004800 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe334ff9640 10 monclient: dump: 2024-09-10T16:25:27.523 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 10 monclient: _renew_subs 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe3367fc640 4 set_mon_vals no callback set 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.508+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe338133e00 con 0x7fe320004800 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe32c011070 con 0x7fe320004800 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe334ff9640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 0x7fe324040f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe33e905640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe33e905640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe33e905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7fe32c0529e0 con 0x7fe320004800 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3380a6d30 con 0x7fe320004800 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe337fff640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 0x7fe324040f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.509+0000 7fe337fff640 10 monclient: get_auth_request con 0x7fe32403ea40 auth_method 0 2024-09-10T16:25:27.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.512+0000 7fe337fff640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 0x7fe324040f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe328006fd0 tx=0x7fe328031000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.514+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe3380a6d30 con 0x7fe320004800 2024-09-10T16:25:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.514+0000 7fe334ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:27.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.514+0000 7fe334ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:27.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.757+0000 7fe33e905640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.757+0000 7fe33e905640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.757+0000 7fe33e905640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.757+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe338137ca0 con 0x7fe320004800 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.758+0000 7fe334ff9640 1 -- 172.21.15.164:0/393768657 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fe338134350 con 0x7fe320004800 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.758+0000 7fe334ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.758+0000 7fe334ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:27.760 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":188,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:27.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 msgr2=0x7fe324040f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 0x7fe324040f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fe328006fd0 tx=0x7fe328031000 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 10 monclient: shutdown 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 msgr2=0x7fe3380a2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c680 tx=0x7fe32c00cb40 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 shutdown_connections 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe32403ea40 0x7fe324040f00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe320005410 0x7fe3380a26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe320004800 0x7fe3380a2190 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 --2- 172.21.15.164:0/393768657 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe320003a20 0x7fe338071d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 >> 172.21.15.164:0/393768657 conn(0x7fe3380aa900 msgr2=0x7fe338132fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 shutdown_connections 2024-09-10T16:25:27.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.760+0000 7fe33e905640 1 -- 172.21.15.164:0/393768657 wait complete. 2024-09-10T16:25:27.772 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:25:27.772 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:25:27.773 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.870+0000 7f8f8d6e6640 10 monclient: get_monmap_and_config 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.870+0000 7f8f8d6e6640 10 monclient: build_initial_monmap 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.870+0000 7f8f8d6e6640 10 monclient: monmap: 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:27.871481+0000 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:27.871481+0000 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:27.872 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:27.873 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:27.873 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 Processor -- start 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 -- start start 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient: init 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f880a4540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:27.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8f881385c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 0x7f8f88134cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f880a60b0 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f880a71c0 con 0x7f8f881385c0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f8f8809c9f0 con 0x7f8f880a60b0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8809ccd0 con 0x7f8f880a4540 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f8d6e6640 10 monclient(hunting): authenticate will time out at 1906.358718s 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f877fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 msgr2=0x7f8f88134cf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f867fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f8f881385c0 legacy=0x7f8f881345e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f877fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 0x7f8f88134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:27.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60738/0 (socket says 172.21.15.164:60738) 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 1 -- 172.21.15.164:0/1349099046 learned_addr learned my addr 172.21.15.164:0/1349099046 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request con 0x7f8f880a4540 auth_method 0 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.871+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_done global_id 214160 payload 274 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 1 -- 172.21.15.164:0/1349099046 >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 msgr2=0x7f8f88134cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/1349099046 >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 0x7f8f88134cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 1 -- 172.21.15.164:0/1349099046 >> v1:172.21.15.164:6789/0 conn(0x7f8f881385c0 legacy=0x7f8f881345e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 1 monclient: found mon.noname-b 2024-09-10T16:25:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.872+0000 7f8f86ffd640 1 -- 172.21.15.164:0/1349099046 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f8809cfb0 con 0x7f8f880a4540 2024-09-10T16:25:27.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.874+0000 7f8f86ffd640 1 -- 172.21.15.164:0/1349099046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 msgr2=0x7f8f880a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.874+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/1349099046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e50 tx=0x7f8f7802a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63a5dad4acedad62 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 1 -- 172.21.15.164:0/1349099046 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78036070 con 0x7f8f880a4540 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: dump: 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 5 monclient: authenticate success, global_id 214160 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 1 -- 172.21.15.164:0/1349099046 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f78007800 con 0x7f8f880a4540 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 4 set_mon_vals no callback set 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 1 -- 172.21.15.164:0/1349099046 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78007bd0 con 0x7f8f880a4540 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f85ffb640 10 monclient: dump: 2024-09-10T16:25:27.887 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 10 monclient: shutdown 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/1349099046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 msgr2=0x7f8f880a4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/1349099046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f8f78009e50 tx=0x7f8f7802a570 comp rx=0 tx=0).stop 2024-09-10T16:25:27.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/1349099046 shutdown_connections 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/1349099046 >> v2:172.21.15.164:3300/0 conn(0x7f8f880a60b0 0x7f8f88134cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/1349099046 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a4540 0x7f8f880a4920 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/1349099046 >> 172.21.15.164:0/1349099046 conn(0x7f8f880aaa00 msgr2=0x7f8f880aae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/1349099046 shutdown_connections 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.875+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/1349099046 wait complete. 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.876+0000 7f8f8d6e6640 10 monclient: build_initial_monmap 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.876+0000 7f8f8d6e6640 10 monclient: monmap: 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:27.889 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.876+0000 7f8f8d6e6640 1 Processor -- start 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.876+0000 7f8f8d6e6640 1 -- start start 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.876+0000 7f8f8d6e6640 10 monclient: init 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880a4540 0x7f8f8812e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f880a4540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f880a60b0 0x7f8f8812e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f880a60b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 0x7f8f881c9df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f880a3e60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88137f50 con 0x7f8f880a60b0 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f867fc640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f880a60b0 msgr2=0x7f8f8812e9b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88137dd0 con 0x7f8f880a4540 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8f88135230 con 0x7f8f880a3e60 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f8d6e6640 10 monclient(hunting): authenticate will time out at 1906.364314s 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f867fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f880a60b0 0x7f8f8812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 0x7f8f881c9df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 0x7f8f881c9df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60752/0 (socket says 172.21.15.164:60752) 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 learned_addr learned my addr 172.21.15.164:0/2274241001 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): get_auth_request con 0x7f8f880a3e60 auth_method 0 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880a4540 0x7f8f8812e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request con 0x7f8f880a4540 auth_method 0 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient(hunting): handle_auth_done global_id 214163 payload 824 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880a4540 msgr2=0x7f8f8812e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f880a4540 0x7f8f8812e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f880a60b0 msgr2=0x7f8f8812e9b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f880a60b0 0x7f8f8812e9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient: _finish_hunting 0 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 monclient: found mon.noname-c 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.877+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f88132c10 con 0x7f8f880a3e60 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.879+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 msgr2=0x7f8f881c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.879+0000 7f8f877fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 0x7f8f881c9df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000b440 tx=0x7f8f8000b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.879+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f80017070 con 0x7f8f880a3e60 2024-09-10T16:25:27.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.879+0000 7f8f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: dump: 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:27.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f800060d0 con 0x7f8f880a3e60 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 5 monclient: authenticate success, global_id 214163 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 monclient: _renew_subs 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f880a2e30 con 0x7f8f880a3e60 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f800064c0 con 0x7f8f880a3e60 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 10 monclient: dump: 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f857fa640 4 set_mon_vals no callback set 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 monclient: _renew_subs 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8812eef0 con 0x7f8f880a3e60 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 0x7f8f881ca330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8f880aa5e0 con 0x7f8f881385c0 2024-09-10T16:25:27.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8f8001a070 con 0x7f8f880a3e60 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 0x7f8f881ca330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f86ffd640 10 monclient: get_auth_request con 0x7f8f881385c0 auth_method 0 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.880+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f7f7fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f6803e1c0 0x7f8f68040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f8f80051f80 con 0x7f8f880a3e60 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_done global_id 214166 payload 824 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f86ffd640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f880a4920 con 0x7f8f881385c0 2024-09-10T16:25:27.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f867fc640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f6803e1c0 0x7f8f68040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.881+0000 7f8f867fc640 10 monclient: get_auth_request con 0x7f8f6803e1c0 auth_method 0 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f86ffd640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 msgr2=0x7f8f881ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 0x7f8f881ca330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f880a54f0 tx=0x7f8f78004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78042030 con 0x7f8f881385c0 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 10 monclient: dump: 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.898 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f867fc640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f6803e1c0 0x7f8f68040680 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8f881329c0 tx=0x7f8f70006d30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8f78037040 con 0x7f8f881385c0 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 msgr2=0x7f8f881ca330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 0x7f8f881ca330 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f880a54f0 tx=0x7f8f78004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 msgr2=0x7f8f881ca330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:27.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.883+0000 7f8f7f7fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881385c0 0x7f8f881ca330 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8f880a54f0 tx=0x7f8f78004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:27.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f8d6e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:27.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 0x7f8f88130350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:27.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f8d6e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8f881cac80 con 0x7f8f881ca870 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 0x7f8f88130350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f86ffd640 10 monclient: get_auth_request con 0x7f8f881ca870 auth_method 0 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.889+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.890+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.890+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.890+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.890+0000 7f8f86ffd640 10 monclient(hunting): handle_auth_done global_id 214169 payload 824 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.890+0000 7f8f86ffd640 1 -- 172.21.15.164:0/2274241001 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8f880a20f0 con 0x7f8f881ca870 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f86ffd640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 msgr2=0x7f8f88130350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:27.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f86ffd640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 0x7f8f88130350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f880aa760 tx=0x7f8f7802f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78042030 con 0x7f8f881ca870 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 10 monclient: dump: 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:27.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f8f78004250 con 0x7f8f881ca870 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 msgr2=0x7f8f88130350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 0x7f8f88130350 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f8f880aa760 tx=0x7f8f7802f000 comp rx=0 tx=0).stop 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 msgr2=0x7f8f88130350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f7f7fe640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f881ca870 0x7f8f88130350 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:27.904 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 188, 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27670, 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:27.905 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:27.906 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:27.907 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:27.908 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:27.909 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:27.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:27.911 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:27.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:27.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:27.915 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.916 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:27.917 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.892+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f6803e1c0 msgr2=0x7f8f68040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f6803e1c0 0x7f8f68040680 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f8f881329c0 tx=0x7f8f70006d30 comp rx=0 tx=0).stop 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 10 monclient: shutdown 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 msgr2=0x7f8f881c9df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 --2- 172.21.15.164:0/2274241001 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f880a3e60 0x7f8f881c9df0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8f8000b440 tx=0x7f8f8000b900 comp rx=0 tx=0).stop 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 reap_dead start 2024-09-10T16:25:27.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f877fe640 1 -- 172.21.15.164:0/2274241001 reap_dead start 2024-09-10T16:25:27.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 shutdown_connections 2024-09-10T16:25:27.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 >> 172.21.15.164:0/2274241001 conn(0x7f8f880aaa00 msgr2=0x7f8f880a74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:27.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 shutdown_connections 2024-09-10T16:25:27.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:27.893+0000 7f8f8d6e6640 1 -- 172.21.15.164:0/2274241001 wait complete. 2024-09-10T16:25:27.920 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:25:27.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:25:27.920 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:25:28.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 10 monclient: get_monmap_and_config 2024-09-10T16:25:28.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 10 monclient: build_initial_monmap 2024-09-10T16:25:28.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 10 monclient: monmap: 2024-09-10T16:25:28.020 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:28.019907+0000 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:28.019907+0000 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:28.021 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 1 Processor -- start 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 1 -- start start 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.019+0000 7f7c4e99f640 10 monclient: init 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7c481354c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c480a5ff0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:28.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 0x7f7c4813f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c4813d530 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4806e180 con 0x7f7c481354c0 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7c480a65d0 con 0x7f7c4813d530 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c480a63d0 con 0x7f7c480a5ff0 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e99f640 10 monclient(hunting): authenticate will time out at 1906.507300s 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e19e640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 msgr2=0x7f7c4813f920 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4e19e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 0x7f7c4813f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60776/0 (socket says 172.21.15.164:60776) 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 -- 172.21.15.164:0/124021524 learned_addr learned my addr 172.21.15.164:0/124021524 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): get_auth_request con 0x7f7c480a5ff0 auth_method 0 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_done global_id 214172 payload 274 2024-09-10T16:25:28.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 -- 172.21.15.164:0/124021524 >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 msgr2=0x7f7c4813f920 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 --2- 172.21.15.164:0/124021524 >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 0x7f7c4813f920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 -- 172.21.15.164:0/124021524 >> v1:172.21.15.165:6789/0 conn(0x7f7c481354c0 legacy=0x7f7c4809d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient: _finish_hunting 0 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 monclient: found mon.noname-b 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:28.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.020+0000 7f7c4d19c640 1 -- 172.21.15.164:0/124021524 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4806e460 con 0x7f7c480a5ff0 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4d19c640 1 -- 172.21.15.164:0/124021524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 msgr2=0x7f7c4813cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4d19c640 1 --2- 172.21.15.164:0/124021524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009e50 tx=0x7f7c3c02a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b70d847854b4bd4a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 1 -- 172.21.15.164:0/124021524 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c036070 con 0x7f7c480a5ff0 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: dump: 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.035 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: _finish_auth 0 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: _check_auth_tickets 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 1 -- 172.21.15.164:0/124021524 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c007800 con 0x7f7c480a5ff0 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4e99f640 5 monclient: authenticate success, global_id 214172 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4e99f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 1 -- 172.21.15.164:0/124021524 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c007bd0 con 0x7f7c480a5ff0 2024-09-10T16:25:28.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4e99f640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:28.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4e99f640 4 set_mon_vals no callback set 2024-09-10T16:25:28.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:28.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.023+0000 7f7c4c99b640 10 monclient: dump: 2024-09-10T16:25:28.040 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.040 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 10 monclient: shutdown 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:28.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 -- 172.21.15.164:0/124021524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 msgr2=0x7f7c4813cff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/124021524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009e50 tx=0x7f7c3c02a570 comp rx=0 tx=0).stop 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 -- 172.21.15.164:0/124021524 shutdown_connections 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/124021524 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c480a5ff0 0x7f7c4813cff0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/124021524 >> v2:172.21.15.164:3300/0 conn(0x7f7c4813d530 0x7f7c4813f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 -- 172.21.15.164:0/124021524 >> 172.21.15.164:0/124021524 conn(0x7f7c480a70e0 msgr2=0x7f7c480a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 -- 172.21.15.164:0/124021524 shutdown_connections 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 1 -- 172.21.15.164:0/124021524 wait complete. 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 10 monclient: build_initial_monmap 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.024+0000 7f7c4e99f640 10 monclient: monmap: 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:28.042 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 Processor -- start 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 -- start start 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient: init 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c480a5ff0 0x7f7c480a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-a con 0x7f7c480a5ff0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 0x7f7c480a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-c con 0x7f7c481354c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c4813d530 0x7f7c480a1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): picked mon.noname-b con 0x7f7c4813d530 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48134d80 con 0x7f7c480a5ff0 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4809db60 con 0x7f7c4813d530 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4809dec0 con 0x7f7c481354c0 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d99d640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c480a5ff0 msgr2=0x7f7c480a1020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:25:28.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4e99f640 10 monclient(hunting): authenticate will time out at 1906.512718s 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d99d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c480a5ff0 0x7f7c480a1020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 0x7f7c480a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 0x7f7c480a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60788/0 (socket says 172.21.15.164:60788) 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 1 -- 172.21.15.164:0/144970430 learned_addr learned my addr 172.21.15.164:0/144970430 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): get_auth_request con 0x7f7c481354c0 auth_method 0 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.025+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 10 monclient(hunting): handle_auth_done global_id 214175 payload 824 2024-09-10T16:25:28.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4e19e640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c4813d530 0x7f7c480a1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c4813d530 msgr2=0x7f7c480a1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c4813d530 0x7f7c480a1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c480a5ff0 msgr2=0x7f7c480a1020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7c480a5ff0 0x7f7c480a1020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 10 monclient: _finish_hunting 0 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 monclient: found mon.noname-c 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4d19c640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4813f930 con 0x7f7c481354c0 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4e19e640 10 monclient: get_auth_request con 0x7f7c4813d530 auth_method 0 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.026+0000 7f7c4e19e640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c4813d530 0x7f7c480a1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4d19c640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 msgr2=0x7f7c480a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:28.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4d19c640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 0x7f7c480a17d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009e50 tx=0x7f7c3c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c041020 con 0x7f7c481354c0 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: dump: 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.047 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c031b60 con 0x7f7c481354c0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 5 monclient: authenticate success, global_id 214175 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 10 monclient: _renew_subs 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c480a3eb0 con 0x7f7c481354c0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c036040 con 0x7f7c481354c0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:28.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c467fc640 10 monclient: dump: 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 10 monclient: _renew_subs 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:28.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c480a2dc0 con 0x7f7c481354c0 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.028+0000 7f7c47fff640 4 set_mon_vals no callback set 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e99f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 0x7f7c4809e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7c480a3990 con 0x7f7c480a2250 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7c3c044020 con 0x7f7c481354c0 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c467fc640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7c3403e1c0 0x7f7c34040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(652..652 src has 494..652) ==== 4799+0+0 (secure 0 0 0) 0x7f7c3c073ed0 con 0x7f7c481354c0 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 0x7f7c4809e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient: get_auth_request con 0x7f7c480a2250 auth_method 0 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:28.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e19e640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7c3403e1c0 0x7f7c34040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.029+0000 7f7c4e19e640 10 monclient: get_auth_request con 0x7f7c3403e1c0 auth_method 0 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.030+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_done global_id 154624 payload 824 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.030+0000 7f7c4d99d640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7c4809d450 con 0x7f7c480a2250 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c4d99d640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 msgr2=0x7f7c4809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c4d99d640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 0x7f7c4809e4e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c380 tx=0x7f7c4000c840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c4e19e640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7c3403e1c0 0x7f7c34040680 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7c38003930 tx=0x7f7c38008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40017070 con 0x7f7c480a2250 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 10 monclient: dump: 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7c400057c0 con 0x7f7c480a2250 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 msgr2=0x7f7c4809e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 0x7f7c4809e4e0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7c4000c380 tx=0x7f7c4000c840 comp rx=0 tx=0).stop 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 msgr2=0x7f7c4809e4e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:28.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.032+0000 7f7c467fc640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2250 0x7f7c4809e4e0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.038+0000 7f7c4e99f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.038+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 0x7f7c4809ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.038+0000 7f7c4e99f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.038+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7c48076eb0 con 0x7f7c480a2630 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 0x7f7c4809ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient: get_auth_request con 0x7f7c480a2630 auth_method 0 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:28.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:28.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:28.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:28.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:28.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 10 monclient(hunting): handle_auth_done global_id 154627 payload 824 2024-09-10T16:25:28.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.039+0000 7f7c4d99d640 1 -- 172.21.15.164:0/144970430 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7c4813cc50 con 0x7f7c480a2630 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c4d99d640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 msgr2=0x7f7c4809ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c4d99d640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 0x7f7c4809ea20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c40000c00 tx=0x7f7c400037e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c40017040 con 0x7f7c480a2630 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c467fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.041+0000 7f7c467fc640 10 monclient: dump: 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:28.057 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f7c40022400 con 0x7f7c480a2630 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 msgr2=0x7f7c4809ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 0x7f7c4809ea20 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c40000c00 tx=0x7f7c400037e0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 msgr2=0x7f7c4809ea20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:28.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c467fc640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7c480a2630 0x7f7c4809ea20 secure :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c40000c00 tx=0x7f7c400037e0 comp rx=0 tx=0).stop 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 188, 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 346592, 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:25:28.059 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:28.060 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:28.061 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:28.062 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:28.063 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.064 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:28.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:28.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:28.068 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:28.069 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.070 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:28.071 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:28.072 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:28.073 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:28.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7c3403e1c0 msgr2=0x7f7c34040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7c3403e1c0 0x7f7c34040680 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f7c38003930 tx=0x7f7c38008040 comp rx=0 tx=0).stop 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 10 monclient: shutdown 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 msgr2=0x7f7c480a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 --2- 172.21.15.164:0/144970430 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7c481354c0 0x7f7c480a17d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c009e50 tx=0x7f7c3c004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e19e640 1 -- 172.21.15.164:0/144970430 reap_dead start 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e19e640 1 -- 172.21.15.164:0/144970430 reap_dead start 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 shutdown_connections 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 >> 172.21.15.164:0/144970430 conn(0x7f7c480a70e0 msgr2=0x7f7c4813eba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 shutdown_connections 2024-09-10T16:25:28.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:28.042+0000 7f7c4e99f640 1 -- 172.21.15.164:0/144970430 wait complete. 2024-09-10T16:25:28.076 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:25:28.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:25:28.351 INFO:tasks.rados.rados.0.smithi165.stdout:2915: done (0 left) 2024-09-10T16:25:28.351 INFO:tasks.rados.rados.0.smithi165.stdout:2916: snap_remove snap 309 2024-09-10T16:25:29.359 INFO:tasks.rados.rados.0.smithi165.stdout:2916: done (0 left) 2024-09-10T16:25:29.359 INFO:tasks.rados.rados.0.smithi165.stdout:2917: delete oid 5 current snap is 310 2024-09-10T16:25:29.361 INFO:tasks.rados.rados.0.smithi165.stdout:2917: done (0 left) 2024-09-10T16:25:29.361 INFO:tasks.rados.rados.0.smithi165.stdout:2918: write (excl) oid 16 current snap is 310 2024-09-10T16:25:29.361 INFO:tasks.rados.rados.0.smithi165.stdout:2918: seq_num 1020 ranges {743140=745282,2154692=468011} 2024-09-10T16:25:29.369 INFO:tasks.rados.rados.0.smithi165.stdout:2918: writing smithi16534841-16 from 743140 to 1488422 tid 1 2024-09-10T16:25:29.375 INFO:tasks.rados.rados.0.smithi165.stdout:2918: writing smithi16534841-16 from 2154692 to 2622703 tid 2 2024-09-10T16:25:29.376 INFO:tasks.rados.rados.0.smithi165.stdout:2919: copy_from oid 36 from oid 50 current snap is 310 2024-09-10T16:25:29.376 INFO:tasks.rados.rados.0.smithi165.stdout:2918: finishing write tid 1 to smithi16534841-16 2024-09-10T16:25:29.376 INFO:tasks.rados.rados.0.smithi165.stdout:2920: snap_remove snap 303 2024-09-10T16:25:30.708 INFO:tasks.rados.rados.0.smithi165.stdout:2920: done (2 left) 2024-09-10T16:25:30.708 INFO:tasks.rados.rados.0.smithi165.stdout:2921: delete oid 28 current snap is 310 2024-09-10T16:25:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2918: finishing write tid 2 to smithi16534841-16 2024-09-10T16:25:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2918: finishing write tid 3 to smithi16534841-16 2024-09-10T16:25:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2918: finishing write tid 4 to smithi16534841-16 2024-09-10T16:25:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2918: oid 16 updating version 0 to 1843 2024-09-10T16:25:30.709 INFO:tasks.rados.rados.0.smithi165.stdout:2918: oid 16 updating version 1843 to 1844 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:2918: oid 16 updating version 1844 to 1845 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1845 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:2918: left oid 16 (ObjNum 1020 snap 310 seq_num 1020) 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:2919: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:2919: finishing copy_from to smithi16534841-36 2024-09-10T16:25:30.710 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1289 (ObjNum 975 snap 297 seq_num 975) dirty exists 2024-09-10T16:25:30.723 INFO:tasks.rados.rados.0.smithi165.stdout:2918: done (2 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2919: done (1 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2921: done (0 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2922: rollback oid 32 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2922: done (0 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2923: copy_from oid 21 from oid 50 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2924: copy_from oid 42 from oid 16 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2925: rollback oid 31 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2925: done (2 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2926: rollback oid 36 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2926: done (2 left) 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2927: write oid 7 current snap is 310 2024-09-10T16:25:30.724 INFO:tasks.rados.rados.0.smithi165.stdout:2927: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2024-09-10T16:25:30.731 INFO:tasks.rados.rados.0.smithi165.stdout:2927: writing smithi16534841-7 from 776754 to 1272534 tid 1 2024-09-10T16:25:30.737 INFO:tasks.rados.rados.0.smithi165.stdout:2927: writing smithi16534841-7 from 1871656 to 2343336 tid 2 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2927: writing smithi16534841-7 from 2965828 to 3176754 tid 3 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2928: copy_from oid 13 from oid 43 current snap is 310 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2923: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2923: finishing copy_from to smithi16534841-21 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1289 (ObjNum 975 snap 297 seq_num 975) dirty exists 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2927: finishing write tid 1 to smithi16534841-7 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2924: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2924: finishing copy_from to smithi16534841-42 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1845 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2927: finishing write tid 2 to smithi16534841-7 2024-09-10T16:25:30.743 INFO:tasks.rados.rados.0.smithi165.stdout:2923: done (3 left) 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2924: done (2 left) 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2929: rollback oid 26 current snap is 310 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2929: done (2 left) 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2930: rollback oid 31 current snap is 310 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2930: done (2 left) 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2931: read oid 41 snap -1 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2931: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:25:30.744 INFO:tasks.rados.rados.0.smithi165.stdout:2932: delete oid 26 current snap is 310 2024-09-10T16:25:30.745 INFO:tasks.rados.rados.0.smithi165.stdout:2931: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:25:30.745 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 275665 2024-09-10T16:25:30.842 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1974009 2024-09-10T16:25:30.844 INFO:tasks.rados.rados.0.smithi165.stdout:2927: finishing write tid 3 to smithi16534841-7 2024-09-10T16:25:30.844 INFO:tasks.rados.rados.0.smithi165.stdout:2927: finishing write tid 4 to smithi16534841-7 2024-09-10T16:25:30.844 INFO:tasks.rados.rados.0.smithi165.stdout:2931: done (3 left) 2024-09-10T16:25:30.844 INFO:tasks.rados.rados.0.smithi165.stdout:2932: done (2 left) 2024-09-10T16:25:30.844 INFO:tasks.rados.rados.0.smithi165.stdout:2933: read oid 5 snap -1 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2933: expect deleted 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2934: read oid 12 snap -1 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2934: expect (ObjNum 1016 snap 308 seq_num 1016) 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2935: copy_from oid 32 from oid 17 current snap is 310 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: finishing write tid 5 to smithi16534841-7 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: oid 7 updating version 0 to 1853 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: oid 7 updating version 1853 to 1855 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: oid 7 version 1855 is already newer than 1852 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: oid 7 version 1855 is already newer than 1854 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1855 (ObjNum 1022 snap 310 seq_num 1022) dirty exists 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2927: left oid 7 (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2928: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:25:30.845 INFO:tasks.rados.rados.0.smithi165.stdout:2928: finishing copy_from to smithi16534841-13 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1333 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2927: done (4 left) 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2928: done (3 left) 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2936: read oid 30 snap -1 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2936: expect (ObjNum 977 snap 297 seq_num 977) 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2937: snap_create 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2935: finishing copy_from to smithi16534841-32 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2935: got expected ENOENT (src dne) 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2935: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:25:30.846 INFO:tasks.rados.rados.0.smithi165.stdout:2934: expect (ObjNum 1016 snap 308 seq_num 1016) 2024-09-10T16:25:30.860 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1560591 2024-09-10T16:25:31.078 INFO:tasks.rados.rados.0.smithi165.stdout:2936: expect (ObjNum 977 snap 297 seq_num 977) 2024-09-10T16:25:31.112 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:31.491 INFO:tasks.rados.rados.0.smithi165.stdout:2933: done (4 left) 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2934: done (3 left) 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2935: done (2 left) 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2936: done (1 left) 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2937: done (0 left) 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2938: copy_from oid 5 from oid 47 current snap is 311 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2939: write oid 4 current snap is 311 2024-09-10T16:25:31.492 INFO:tasks.rados.rados.0.smithi165.stdout:2939: seq_num 1024 ranges {410368=646278,1472267=475349,2504321=405600} 2024-09-10T16:25:31.496 INFO:tasks.rados.rados.0.smithi165.stdout:2939: writing smithi16534841-4 from 410368 to 1056646 tid 1 2024-09-10T16:25:31.501 INFO:tasks.rados.rados.0.smithi165.stdout:2939: writing smithi16534841-4 from 1472267 to 1947616 tid 2 2024-09-10T16:25:31.507 INFO:tasks.rados.rados.0.smithi165.stdout:2939: writing smithi16534841-4 from 2504321 to 2909921 tid 3 2024-09-10T16:25:31.507 INFO:tasks.rados.rados.0.smithi165.stdout:2940: snap_create 2024-09-10T16:25:31.507 INFO:tasks.rados.rados.0.smithi165.stdout:2939: finishing write tid 1 to smithi16534841-4 2024-09-10T16:25:31.507 INFO:tasks.rados.rados.0.smithi165.stdout:2939: finishing write tid 2 to smithi16534841-4 2024-09-10T16:25:31.508 INFO:tasks.rados.rados.0.smithi165.stdout:2938: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:25:31.508 INFO:tasks.rados.rados.0.smithi165.stdout:2938: finishing copy_from to smithi16534841-5 2024-09-10T16:25:31.508 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1615 (ObjNum 967 snap 292 seq_num 967) dirty exists 2024-09-10T16:25:31.513 INFO:tasks.rados.rados.0.smithi165.stdout:2939: finishing write tid 3 to smithi16534841-4 2024-09-10T16:25:31.513 INFO:tasks.rados.rados.0.smithi165.stdout:2939: finishing write tid 4 to smithi16534841-4 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: finishing write tid 5 to smithi16534841-4 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: oid 4 updating version 0 to 1348 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: oid 4 updating version 1348 to 1351 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: oid 4 version 1351 is already newer than 1350 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: oid 4 version 1351 is already newer than 1349 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1351 (ObjNum 1024 snap 311 seq_num 1024) dirty exists 2024-09-10T16:25:31.514 INFO:tasks.rados.rados.0.smithi165.stdout:2939: left oid 4 (ObjNum 1024 snap 311 seq_num 1024) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2938: done (2 left) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2939: done (1 left) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2940: done (0 left) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2941: read oid 1 snap -1 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2941: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2942: read oid 32 snap -1 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2942: expect deleted 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2943: read oid 27 snap -1 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2943: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2944: read oid 11 snap -1 2024-09-10T16:25:32.504 INFO:tasks.rados.rados.0.smithi165.stdout:2944: expect (ObjNum 982 snap 297 seq_num 982) 2024-09-10T16:25:32.505 INFO:tasks.rados.rados.0.smithi165.stdout:2945: snap_create 2024-09-10T16:25:32.505 INFO:tasks.rados.rados.0.smithi165.stdout:2943: expect (ObjNum 970 snap 294 seq_num 970) 2024-09-10T16:25:32.518 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1540946 2024-09-10T16:25:32.593 INFO:tasks.rados.rados.0.smithi165.stdout:2941: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:25:32.756 INFO:tasks.rados.rados.0.smithi165.stdout:2944: expect (ObjNum 982 snap 297 seq_num 982) 2024-09-10T16:25:32.757 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2941: done (4 left) 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2942: done (3 left) 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2943: done (2 left) 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2944: done (1 left) 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2945: done (0 left) 2024-09-10T16:25:33.505 INFO:tasks.rados.rados.0.smithi165.stdout:2946: delete oid 11 current snap is 313 2024-09-10T16:25:33.508 INFO:tasks.rados.rados.0.smithi165.stdout:2946: done (0 left) 2024-09-10T16:25:33.508 INFO:tasks.rados.rados.0.smithi165.stdout:2947: write (excl) oid 36 current snap is 313 2024-09-10T16:25:33.508 INFO:tasks.rados.rados.0.smithi165.stdout:2947: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2024-09-10T16:25:33.514 INFO:tasks.rados.rados.0.smithi165.stdout:2947: writing smithi16534841-36 from 443982 to 1240758 tid 1 2024-09-10T16:25:33.521 INFO:tasks.rados.rados.0.smithi165.stdout:2947: writing smithi16534841-36 from 1989231 to 2468249 tid 2 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2947: writing smithi16534841-36 from 2959167 to 3243982 tid 3 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2948: copy_from oid 21 from oid 19 current snap is 313 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2947: finishing write tid 1 to smithi16534841-36 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2947: finishing write tid 2 to smithi16534841-36 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2949: copy_from oid 15 from oid 35 current snap is 313 2024-09-10T16:25:33.526 INFO:tasks.rados.rados.0.smithi165.stdout:2950: write (excl) oid 50 current snap is 313 2024-09-10T16:25:33.527 INFO:tasks.rados.rados.0.smithi165.stdout:2950: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2024-09-10T16:25:33.531 INFO:tasks.rados.rados.0.smithi165.stdout:2950: writing smithi16534841-50 from 460789 to 1049167 tid 1 2024-09-10T16:25:33.541 INFO:tasks.rados.rados.0.smithi165.stdout:2950: writing smithi16534841-50 from 1764066 to 2503095 tid 2 2024-09-10T16:25:33.545 INFO:tasks.rados.rados.0.smithi165.stdout:2950: writing smithi16534841-50 from 3219296 to 3260789 tid 3 2024-09-10T16:25:33.545 INFO:tasks.rados.rados.0.smithi165.stdout:2951: read oid 41 snap 312 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2951: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2952: write oid 37 current snap is 313 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: finishing write tid 3 to smithi16534841-36 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: finishing write tid 4 to smithi16534841-36 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: finishing write tid 5 to smithi16534841-36 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: oid 36 updating version 0 to 1394 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: oid 36 version 1394 is already newer than 1393 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: oid 36 updating version 1394 to 1395 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: oid 36 version 1395 is already newer than 1392 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1395 (ObjNum 1026 snap 313 seq_num 1026) dirty exists 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2947: left oid 36 (ObjNum 1026 snap 313 seq_num 1026) 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2949: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2950: finishing write tid 1 to smithi16534841-50 2024-09-10T16:25:33.546 INFO:tasks.rados.rados.0.smithi165.stdout:2949: finishing copy_from to smithi16534841-15 2024-09-10T16:25:33.547 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1432 (ObjNum 995 snap 299 seq_num 995) dirty exists 2024-09-10T16:25:33.547 INFO:tasks.rados.rados.0.smithi165.stdout:2948: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:25:33.547 INFO:tasks.rados.rados.0.smithi165.stdout:2952: seq_num 1028 ranges {477596=663627,1706195=482687,2614013=567041} 2024-09-10T16:25:33.552 INFO:tasks.rados.rados.0.smithi165.stdout:2952: writing smithi16534841-37 from 477596 to 1141223 tid 1 2024-09-10T16:25:33.558 INFO:tasks.rados.rados.0.smithi165.stdout:2952: writing smithi16534841-37 from 1706195 to 2188882 tid 2 2024-09-10T16:25:33.563 INFO:tasks.rados.rados.0.smithi165.stdout:2952: writing smithi16534841-37 from 2614013 to 3181054 tid 3 2024-09-10T16:25:33.563 INFO:tasks.rados.rados.0.smithi165.stdout:2947: done (5 left) 2024-09-10T16:25:33.564 INFO:tasks.rados.rados.0.smithi165.stdout:2949: done (4 left) 2024-09-10T16:25:33.564 INFO:tasks.rados.rados.0.smithi165.stdout:2953: write oid 39 current snap is 313 2024-09-10T16:25:33.564 INFO:tasks.rados.rados.0.smithi165.stdout:2953: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2024-09-10T16:25:33.570 INFO:tasks.rados.rados.0.smithi165.stdout:2953: writing smithi16534841-39 from 494403 to 1233279 tid 1 2024-09-10T16:25:33.576 INFO:tasks.rados.rados.0.smithi165.stdout:2953: writing smithi16534841-39 from 1764677 to 2391022 tid 2 2024-09-10T16:25:33.581 INFO:tasks.rados.rados.0.smithi165.stdout:2953: writing smithi16534841-39 from 3041436 to 3294403 tid 3 2024-09-10T16:25:33.581 INFO:tasks.rados.rados.0.smithi165.stdout:2954: snap_remove snap 310 2024-09-10T16:25:34.517 INFO:tasks.rados.rados.0.smithi165.stdout:2954: done (5 left) 2024-09-10T16:25:34.518 INFO:tasks.rados.rados.0.smithi165.stdout:2955: write oid 18 current snap is 313 2024-09-10T16:25:34.518 INFO:tasks.rados.rados.0.smithi165.stdout:2955: seq_num 1030 ranges {511210=414125,1423159=486356,2668859=589585} 2024-09-10T16:25:34.522 INFO:tasks.rados.rados.0.smithi165.stdout:2955: writing smithi16534841-18 from 511210 to 925335 tid 1 2024-09-10T16:25:34.527 INFO:tasks.rados.rados.0.smithi165.stdout:2955: writing smithi16534841-18 from 1423159 to 1909515 tid 2 2024-09-10T16:25:34.535 INFO:tasks.rados.rados.0.smithi165.stdout:2955: writing smithi16534841-18 from 2668859 to 3258444 tid 3 2024-09-10T16:25:34.535 INFO:tasks.rados.rados.0.smithi165.stdout:2956: snap_create 2024-09-10T16:25:34.536 INFO:tasks.rados.rados.0.smithi165.stdout:2951: expect (ObjNum 719 snap 221 seq_num 719) 2024-09-10T16:25:34.622 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2071967 2024-09-10T16:25:34.622 INFO:tasks.rados.rados.0.smithi165.stdout:2948: finishing copy_from to smithi16534841-21 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1795 (ObjNum 999 snap 299 seq_num 999) dirty exists 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: finishing write tid 2 to smithi16534841-50 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: finishing write tid 3 to smithi16534841-50 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: finishing write tid 4 to smithi16534841-50 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: finishing write tid 5 to smithi16534841-50 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: oid 50 updating version 0 to 1847 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: oid 50 updating version 1847 to 1848 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: oid 50 updating version 1848 to 1849 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: oid 50 updating version 1849 to 1850 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1850 (ObjNum 1027 snap 313 seq_num 1027) dirty exists 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2950: left oid 50 (ObjNum 1027 snap 313 seq_num 1027) 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2952: finishing write tid 1 to smithi16534841-37 2024-09-10T16:25:34.623 INFO:tasks.rados.rados.0.smithi165.stdout:2952: finishing write tid 2 to smithi16534841-37 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: finishing write tid 3 to smithi16534841-37 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: finishing write tid 4 to smithi16534841-37 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: finishing write tid 5 to smithi16534841-37 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: oid 37 updating version 0 to 1565 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: oid 37 updating version 1565 to 1568 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: oid 37 version 1568 is already newer than 1566 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: oid 37 version 1568 is already newer than 1567 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1568 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2952: left oid 37 (ObjNum 1028 snap 313 seq_num 1028) 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2953: finishing write tid 1 to smithi16534841-39 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2953: finishing write tid 2 to smithi16534841-39 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2953: finishing write tid 3 to smithi16534841-39 2024-09-10T16:25:34.624 INFO:tasks.rados.rados.0.smithi165.stdout:2953: finishing write tid 4 to smithi16534841-39 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: finishing write tid 5 to smithi16534841-39 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: oid 39 updating version 0 to 1667 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: oid 39 updating version 1667 to 1669 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: oid 39 updating version 1669 to 1670 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: oid 39 version 1670 is already newer than 1668 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1670 (ObjNum 1029 snap 313 seq_num 1029) dirty exists 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2953: left oid 39 (ObjNum 1029 snap 313 seq_num 1029) 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: finishing write tid 1 to smithi16534841-18 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: finishing write tid 2 to smithi16534841-18 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: finishing write tid 3 to smithi16534841-18 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: finishing write tid 4 to smithi16534841-18 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: finishing write tid 5 to smithi16534841-18 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: oid 18 updating version 0 to 1673 2024-09-10T16:25:34.625 INFO:tasks.rados.rados.0.smithi165.stdout:2955: oid 18 version 1673 is already newer than 1672 2024-09-10T16:25:34.626 INFO:tasks.rados.rados.0.smithi165.stdout:2955: oid 18 updating version 1673 to 1674 2024-09-10T16:25:34.626 INFO:tasks.rados.rados.0.smithi165.stdout:2955: oid 18 version 1674 is already newer than 1671 2024-09-10T16:25:34.626 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1674 (ObjNum 1030 snap 313 seq_num 1030) dirty exists 2024-09-10T16:25:34.626 INFO:tasks.rados.rados.0.smithi165.stdout:2955: left oid 18 (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2948: done (6 left) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2950: done (5 left) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2951: done (4 left) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2952: done (3 left) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2953: done (2 left) 2024-09-10T16:25:35.516 INFO:tasks.rados.rados.0.smithi165.stdout:2955: done (1 left) 2024-09-10T16:25:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2956: done (0 left) 2024-09-10T16:25:35.517 INFO:tasks.rados.rados.0.smithi165.stdout:2957: delete oid 47 current snap is 314 2024-09-10T16:25:35.523 INFO:tasks.rados.rados.0.smithi165.stdout:2957: done (0 left) 2024-09-10T16:25:35.523 INFO:tasks.rados.rados.0.smithi165.stdout:2958: delete oid 9 current snap is 314 2024-09-10T16:25:35.528 INFO:tasks.rados.rados.0.smithi165.stdout:2958: done (0 left) 2024-09-10T16:25:35.528 INFO:tasks.rados.rados.0.smithi165.stdout:2959: write oid 4 current snap is 314 2024-09-10T16:25:35.528 INFO:tasks.rados.rados.0.smithi165.stdout:2959: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2024-09-10T16:25:35.534 INFO:tasks.rados.rados.0.smithi165.stdout:2959: writing smithi16534841-4 from 561631 to 1201503 tid 1 2024-09-10T16:25:35.541 INFO:tasks.rados.rados.0.smithi165.stdout:2959: writing smithi16534841-4 from 1882252 to 2515935 tid 2 2024-09-10T16:25:35.546 INFO:tasks.rados.rados.0.smithi165.stdout:2959: writing smithi16534841-4 from 3034775 to 3361631 tid 3 2024-09-10T16:25:35.546 INFO:tasks.rados.rados.0.smithi165.stdout:2960: write (excl) oid 39 current snap is 314 2024-09-10T16:25:35.546 INFO:tasks.rados.rados.0.smithi165.stdout:2959: finishing write tid 1 to smithi16534841-4 2024-09-10T16:25:35.546 INFO:tasks.rados.rados.0.smithi165.stdout:2960: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2024-09-10T16:25:35.551 INFO:tasks.rados.rados.0.smithi165.stdout:2960: writing smithi16534841-39 from 578438 to 1009912 tid 1 2024-09-10T16:25:35.558 INFO:tasks.rados.rados.0.smithi165.stdout:2960: writing smithi16534841-39 from 1657087 to 2150781 tid 2 2024-09-10T16:25:35.565 INFO:tasks.rados.rados.0.smithi165.stdout:2960: writing smithi16534841-39 from 2778551 to 3378438 tid 3 2024-09-10T16:25:35.565 INFO:tasks.rados.rados.0.smithi165.stdout:2961: snap_create 2024-09-10T16:25:35.565 INFO:tasks.rados.rados.0.smithi165.stdout:2959: finishing write tid 2 to smithi16534841-4 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: finishing write tid 3 to smithi16534841-4 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: finishing write tid 4 to smithi16534841-4 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: finishing write tid 5 to smithi16534841-4 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: oid 4 updating version 0 to 1434 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: oid 4 updating version 1434 to 1435 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: oid 4 version 1435 is already newer than 1433 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: oid 4 updating version 1435 to 1436 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1436 (ObjNum 1033 snap 314 seq_num 1033) dirty exists 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2959: left oid 4 (ObjNum 1033 snap 314 seq_num 1033) 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2960: finishing write tid 1 to smithi16534841-39 2024-09-10T16:25:35.566 INFO:tasks.rados.rados.0.smithi165.stdout:2960: finishing write tid 2 to smithi16534841-39 2024-09-10T16:25:35.568 INFO:tasks.rados.rados.0.smithi165.stdout:2960: finishing write tid 3 to smithi16534841-39 2024-09-10T16:25:35.568 INFO:tasks.rados.rados.0.smithi165.stdout:2960: finishing write tid 4 to smithi16534841-39 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: finishing write tid 5 to smithi16534841-39 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: oid 39 updating version 0 to 1677 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: oid 39 updating version 1677 to 1678 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: oid 39 version 1678 is already newer than 1676 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: oid 39 version 1678 is already newer than 1675 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1678 (ObjNum 1034 snap 314 seq_num 1034) dirty exists 2024-09-10T16:25:35.569 INFO:tasks.rados.rados.0.smithi165.stdout:2960: left oid 39 (ObjNum 1034 snap 314 seq_num 1034) 2024-09-10T16:25:36.493 INFO:tasks.rados.rados.0.smithi165.stdout:2959: done (2 left) 2024-09-10T16:25:36.493 INFO:tasks.rados.rados.0.smithi165.stdout:2960: done (1 left) 2024-09-10T16:25:36.493 INFO:tasks.rados.rados.0.smithi165.stdout:2961: done (0 left) 2024-09-10T16:25:36.493 INFO:tasks.rados.rados.0.smithi165.stdout:2962: copy_from oid 22 from oid 46 current snap is 315 2024-09-10T16:25:36.493 INFO:tasks.rados.rados.0.smithi165.stdout:2963: snap_create 2024-09-10T16:25:36.499 INFO:tasks.rados.rados.0.smithi165.stdout:2962: finishing copy_from to smithi16534841-22 2024-09-10T16:25:36.499 INFO:tasks.rados.rados.0.smithi165.stdout:2962: got expected ENOENT (src dne) 2024-09-10T16:25:36.499 INFO:tasks.rados.rados.0.smithi165.stdout:2962: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:25:37.498 INFO:tasks.rados.rados.0.smithi165.stdout:2962: done (1 left) 2024-09-10T16:25:37.498 INFO:tasks.rados.rados.0.smithi165.stdout:2963: done (0 left) 2024-09-10T16:25:37.498 INFO:tasks.rados.rados.0.smithi165.stdout:2964: snap_create 2024-09-10T16:25:38.503 INFO:tasks.rados.rados.0.smithi165.stdout:2964: done (0 left) 2024-09-10T16:25:38.503 INFO:tasks.rados.rados.0.smithi165.stdout:2965: snap_create 2024-09-10T16:25:39.504 INFO:tasks.rados.rados.0.smithi165.stdout:2965: done (0 left) 2024-09-10T16:25:39.504 INFO:tasks.rados.rados.0.smithi165.stdout:2966: snap_remove snap 314 2024-09-10T16:25:40.506 INFO:tasks.rados.rados.0.smithi165.stdout:2966: done (0 left) 2024-09-10T16:25:40.506 INFO:tasks.rados.rados.0.smithi165.stdout:2967: snap_remove snap 316 2024-09-10T16:25:41.493 INFO:tasks.rados.rados.0.smithi165.stdout:2967: done (0 left) 2024-09-10T16:25:41.494 INFO:tasks.rados.rados.0.smithi165.stdout:2968: rollback oid 39 current snap is 318 2024-09-10T16:25:41.494 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 315 2024-09-10T16:25:41.494 INFO:tasks.rados.rados.0.smithi165.stdout:2969: write oid 38 current snap is 318 2024-09-10T16:25:41.494 INFO:tasks.rados.rados.0.smithi165.stdout:2969: seq_num 1035 ranges {595245=506723,1715569=637352,2805974=562298} 2024-09-10T16:25:41.499 INFO:tasks.rados.rados.0.smithi165.stdout:2969: writing smithi16534841-38 from 595245 to 1101968 tid 1 2024-09-10T16:25:41.506 INFO:tasks.rados.rados.0.smithi165.stdout:2969: writing smithi16534841-38 from 1715569 to 2352921 tid 2 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2969: writing smithi16534841-38 from 2805974 to 3368272 tid 3 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2970: read oid 33 snap -1 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2970: expect deleted 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2971: read oid 20 snap -1 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2971: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2972: write (excl) oid 10 current snap is 318 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2968: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:25:41.513 INFO:tasks.rados.rados.0.smithi165.stdout:2972: seq_num 1036 ranges {612052=581972,1657698=497363,2717044=695008} 2024-09-10T16:25:41.519 INFO:tasks.rados.rados.0.smithi165.stdout:2972: writing smithi16534841-10 from 612052 to 1194024 tid 1 2024-09-10T16:25:41.525 INFO:tasks.rados.rados.0.smithi165.stdout:2972: writing smithi16534841-10 from 1657698 to 2155061 tid 2 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2972: writing smithi16534841-10 from 2717044 to 3412052 tid 3 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2973: write (excl) oid 44 current snap is 318 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2968: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2968: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1681 (ObjNum 1034 snap 314 seq_num 1034) dirty exists 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2969: finishing write tid 1 to smithi16534841-38 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2969: finishing write tid 2 to smithi16534841-38 2024-09-10T16:25:41.532 INFO:tasks.rados.rados.0.smithi165.stdout:2973: seq_num 1037 ranges {628859=657221,1716180=641021,3144467=284392} 2024-09-10T16:25:41.539 INFO:tasks.rados.rados.0.smithi165.stdout:2973: writing smithi16534841-44 from 628859 to 1286080 tid 1 2024-09-10T16:25:41.545 INFO:tasks.rados.rados.0.smithi165.stdout:2973: writing smithi16534841-44 from 1716180 to 2357201 tid 2 2024-09-10T16:25:41.551 INFO:tasks.rados.rados.0.smithi165.stdout:2973: writing smithi16534841-44 from 3144467 to 3428859 tid 3 2024-09-10T16:25:41.551 INFO:tasks.rados.rados.0.smithi165.stdout:2968: done (5 left) 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2974: snap_create 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: finishing write tid 3 to smithi16534841-38 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: finishing write tid 4 to smithi16534841-38 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: finishing write tid 5 to smithi16534841-38 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: oid 38 updating version 0 to 1396 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: oid 38 updating version 1396 to 1398 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: oid 38 version 1398 is already newer than 1397 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: oid 38 updating version 1398 to 1399 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1399 (ObjNum 1035 snap 318 seq_num 1035) dirty exists 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2969: left oid 38 (ObjNum 1035 snap 318 seq_num 1035) 2024-09-10T16:25:41.552 INFO:tasks.rados.rados.0.smithi165.stdout:2971: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:25:41.599 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 706605 2024-09-10T16:25:41.645 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1690888 2024-09-10T16:25:41.671 INFO:tasks.rados.rados.0.smithi165.stdout:2972: finishing write tid 1 to smithi16534841-10 2024-09-10T16:25:41.671 INFO:tasks.rados.rados.0.smithi165.stdout:2972: finishing write tid 2 to smithi16534841-10 2024-09-10T16:25:41.671 INFO:tasks.rados.rados.0.smithi165.stdout:2972: finishing write tid 3 to smithi16534841-10 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: finishing write tid 4 to smithi16534841-10 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: finishing write tid 5 to smithi16534841-10 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: oid 10 updating version 0 to 1570 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: oid 10 version 1570 is already newer than 1568 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: oid 10 version 1570 is already newer than 1569 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: oid 10 updating version 1570 to 1571 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1571 (ObjNum 1036 snap 318 seq_num 1036) dirty exists 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2972: left oid 10 (ObjNum 1036 snap 318 seq_num 1036) 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2973: finishing write tid 1 to smithi16534841-44 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2973: finishing write tid 2 to smithi16534841-44 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2973: finishing write tid 3 to smithi16534841-44 2024-09-10T16:25:41.672 INFO:tasks.rados.rados.0.smithi165.stdout:2973: finishing write tid 4 to smithi16534841-44 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: finishing write tid 5 to smithi16534841-44 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: oid 44 updating version 0 to 1163 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: oid 44 updating version 1163 to 1165 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: oid 44 version 1165 is already newer than 1164 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: oid 44 version 1165 is already newer than 1162 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1165 (ObjNum 1037 snap 318 seq_num 1037) dirty exists 2024-09-10T16:25:41.673 INFO:tasks.rados.rados.0.smithi165.stdout:2973: left oid 44 (ObjNum 1037 snap 318 seq_num 1037) 2024-09-10T16:25:42.519 INFO:tasks.rados.rados.0.smithi165.stdout:2969: done (5 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2970: done (4 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2971: done (3 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2972: done (2 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2973: done (1 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2974: done (0 left) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2975: read oid 3 snap -1 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2975: expect (ObjNum 1014 snap 307 seq_num 1014) 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2976: write oid 11 current snap is 319 2024-09-10T16:25:42.520 INFO:tasks.rados.rados.0.smithi165.stdout:2976: seq_num 1038 ranges {645666=732470,2174662=501032,3288243=157423} 2024-09-10T16:25:42.527 INFO:tasks.rados.rados.0.smithi165.stdout:2976: writing smithi16534841-11 from 645666 to 1378136 tid 1 2024-09-10T16:25:42.534 INFO:tasks.rados.rados.0.smithi165.stdout:2976: writing smithi16534841-11 from 2174662 to 2675694 tid 2 2024-09-10T16:25:42.539 INFO:tasks.rados.rados.0.smithi165.stdout:2976: writing smithi16534841-11 from 3288243 to 3445666 tid 3 2024-09-10T16:25:42.539 INFO:tasks.rados.rados.0.smithi165.stdout:2977: snap_create 2024-09-10T16:25:42.539 INFO:tasks.rados.rados.0.smithi165.stdout:2975: expect (ObjNum 1014 snap 307 seq_num 1014) 2024-09-10T16:25:42.549 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2590068 2024-09-10T16:25:42.558 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:42.623 INFO:tasks.rados.rados.0.smithi165.stdout:2976: finishing write tid 1 to smithi16534841-11 2024-09-10T16:25:42.623 INFO:tasks.rados.rados.0.smithi165.stdout:2976: finishing write tid 2 to smithi16534841-11 2024-09-10T16:25:42.623 INFO:tasks.rados.rados.0.smithi165.stdout:2976: finishing write tid 3 to smithi16534841-11 2024-09-10T16:25:42.623 INFO:tasks.rados.rados.0.smithi165.stdout:2976: finishing write tid 4 to smithi16534841-11 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: finishing write tid 5 to smithi16534841-11 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: oid 11 updating version 0 to 1853 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: oid 11 version 1853 is already newer than 1851 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: oid 11 version 1853 is already newer than 1852 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: oid 11 updating version 1853 to 1854 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1854 (ObjNum 1038 snap 319 seq_num 1038) dirty exists 2024-09-10T16:25:42.624 INFO:tasks.rados.rados.0.smithi165.stdout:2976: left oid 11 (ObjNum 1038 snap 319 seq_num 1038) 2024-09-10T16:25:43.524 INFO:tasks.rados.rados.0.smithi165.stdout:2975: done (2 left) 2024-09-10T16:25:43.524 INFO:tasks.rados.rados.0.smithi165.stdout:2976: done (1 left) 2024-09-10T16:25:43.524 INFO:tasks.rados.rados.0.smithi165.stdout:2977: done (0 left) 2024-09-10T16:25:43.524 INFO:tasks.rados.rados.0.smithi165.stdout:2978: delete oid 28 current snap is 320 2024-09-10T16:25:43.526 INFO:tasks.rados.rados.0.smithi165.stdout:2978: done (0 left) 2024-09-10T16:25:43.526 INFO:tasks.rados.rados.0.smithi165.stdout:2979: write oid 35 current snap is 320 2024-09-10T16:25:43.526 INFO:tasks.rados.rados.0.smithi165.stdout:2979: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2024-09-10T16:25:43.532 INFO:tasks.rados.rados.0.smithi165.stdout:2979: writing smithi16534841-35 from 679280 to 1162248 tid 1 2024-09-10T16:25:43.540 INFO:tasks.rados.rados.0.smithi165.stdout:2979: writing smithi16534841-35 from 1775273 to 2563621 tid 2 2024-09-10T16:25:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:2979: writing smithi16534841-35 from 3110383 to 3479280 tid 3 2024-09-10T16:25:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:2980: delete oid 28 current snap is 320 2024-09-10T16:25:43.545 INFO:tasks.rados.rados.0.smithi165.stdout:2979: finishing write tid 1 to smithi16534841-35 2024-09-10T16:25:43.546 INFO:tasks.rados.rados.0.smithi165.stdout:2979: finishing write tid 2 to smithi16534841-35 2024-09-10T16:25:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:2980: done (1 left) 2024-09-10T16:25:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:2981: read oid 25 snap 312 2024-09-10T16:25:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:2981: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:25:43.547 INFO:tasks.rados.rados.0.smithi165.stdout:2982: snap_create 2024-09-10T16:25:43.548 INFO:tasks.rados.rados.0.smithi165.stdout:2979: finishing write tid 3 to smithi16534841-35 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: finishing write tid 4 to smithi16534841-35 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: finishing write tid 5 to smithi16534841-35 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: oid 35 updating version 0 to 1435 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: oid 35 version 1435 is already newer than 1434 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: oid 35 updating version 1435 to 1436 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: oid 35 updating version 1436 to 1437 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1437 (ObjNum 1040 snap 320 seq_num 1040) dirty exists 2024-09-10T16:25:43.551 INFO:tasks.rados.rados.0.smithi165.stdout:2979: left oid 35 (ObjNum 1040 snap 320 seq_num 1040) 2024-09-10T16:25:43.556 INFO:tasks.rados.rados.0.smithi165.stdout:2981: expect (ObjNum 926 snap 283 seq_num 926) 2024-09-10T16:25:43.603 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3016205 2024-09-10T16:25:43.615 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2979: done (2 left) 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2981: done (1 left) 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2982: done (0 left) 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2983: rollback oid 14 current snap is 321 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 319 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2984: write (excl) oid 5 current snap is 321 2024-09-10T16:25:44.534 INFO:tasks.rados.rados.0.smithi165.stdout:2984: seq_num 1042 ranges {712894=749819,2008590=792017,3281582=231312} 2024-09-10T16:25:44.542 INFO:tasks.rados.rados.0.smithi165.stdout:2984: writing smithi16534841-5 from 712894 to 1462713 tid 1 2024-09-10T16:25:44.549 INFO:tasks.rados.rados.0.smithi165.stdout:2984: writing smithi16534841-5 from 2008590 to 2800607 tid 2 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2984: writing smithi16534841-5 from 3281582 to 3512894 tid 3 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2985: rollback oid 43 current snap is 321 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 320 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2983: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2983: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2983: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1244 (ObjNum 966 snap 292 seq_num 966) dirty exists 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2984: finishing write tid 1 to smithi16534841-5 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2983: done (2 left) 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2986: read oid 44 snap -1 2024-09-10T16:25:44.555 INFO:tasks.rados.rados.0.smithi165.stdout:2986: expect (ObjNum 1037 snap 318 seq_num 1037) 2024-09-10T16:25:44.556 INFO:tasks.rados.rados.0.smithi165.stdout:2987: read oid 28 snap -1 2024-09-10T16:25:44.556 INFO:tasks.rados.rados.0.smithi165.stdout:2987: expect deleted 2024-09-10T16:25:44.556 INFO:tasks.rados.rados.0.smithi165.stdout:2988: rollback oid 25 current snap is 321 2024-09-10T16:25:44.556 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 318 2024-09-10T16:25:44.556 INFO:tasks.rados.rados.0.smithi165.stdout:2989: delete oid 20 current snap is 321 2024-09-10T16:25:44.557 INFO:tasks.rados.rados.0.smithi165.stdout:2984: finishing write tid 2 to smithi16534841-5 2024-09-10T16:25:44.557 INFO:tasks.rados.rados.0.smithi165.stdout:2984: finishing write tid 3 to smithi16534841-5 2024-09-10T16:25:44.557 INFO:tasks.rados.rados.0.smithi165.stdout:2984: finishing write tid 4 to smithi16534841-5 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: finishing write tid 5 to smithi16534841-5 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: oid 5 updating version 0 to 1619 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: oid 5 version 1619 is already newer than 1616 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: oid 5 version 1619 is already newer than 1618 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: oid 5 version 1619 is already newer than 1617 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1619 (ObjNum 1042 snap 321 seq_num 1042) dirty exists 2024-09-10T16:25:44.558 INFO:tasks.rados.rados.0.smithi165.stdout:2984: left oid 5 (ObjNum 1042 snap 321 seq_num 1042) 2024-09-10T16:25:44.560 INFO:tasks.rados.rados.0.smithi165.stdout:2986: expect (ObjNum 1037 snap 318 seq_num 1037) 2024-09-10T16:25:44.571 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 920041 2024-09-10T16:25:44.709 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3266141 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2985: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2988: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2988: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2985: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2988: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1798 (ObjNum 926 snap 283 seq_num 926) dirty exists 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2985: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1681 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2984: done (5 left) 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2985: done (4 left) 2024-09-10T16:25:44.711 INFO:tasks.rados.rados.0.smithi165.stdout:2986: done (3 left) 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2987: done (2 left) 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2988: done (1 left) 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2989: done (0 left) 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2990: read oid 4 snap -1 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2990: expect (ObjNum 1033 snap 314 seq_num 1033) 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2991: copy_from oid 21 from oid 41 current snap is 321 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2992: write oid 27 current snap is 321 2024-09-10T16:25:44.712 INFO:tasks.rados.rados.0.smithi165.stdout:2992: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2024-09-10T16:25:44.717 INFO:tasks.rados.rados.0.smithi165.stdout:2992: writing smithi16534841-27 from 746508 to 1246825 tid 1 2024-09-10T16:25:44.726 INFO:tasks.rados.rados.0.smithi165.stdout:2992: writing smithi16534841-27 from 2009201 to 2804887 tid 2 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:2992: writing smithi16534841-27 from 3220075 to 3546508 tid 3 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:2993: snap_create 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:2991: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:2991: finishing copy_from to smithi16534841-21 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:25:44.731 INFO:tasks.rados.rados.0.smithi165.stdout:2990: expect (ObjNum 1033 snap 314 seq_num 1033) 2024-09-10T16:25:44.885 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:44.962 INFO:tasks.rados.rados.0.smithi165.stdout:2992: finishing write tid 1 to smithi16534841-27 2024-09-10T16:25:44.962 INFO:tasks.rados.rados.0.smithi165.stdout:2992: finishing write tid 2 to smithi16534841-27 2024-09-10T16:25:44.962 INFO:tasks.rados.rados.0.smithi165.stdout:2992: finishing write tid 3 to smithi16534841-27 2024-09-10T16:25:44.962 INFO:tasks.rados.rados.0.smithi165.stdout:2992: finishing write tid 4 to smithi16534841-27 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: finishing write tid 5 to smithi16534841-27 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: oid 27 updating version 0 to 1635 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: oid 27 version 1635 is already newer than 1634 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: oid 27 updating version 1635 to 1637 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: oid 27 version 1637 is already newer than 1636 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1637 (ObjNum 1044 snap 321 seq_num 1044) dirty exists 2024-09-10T16:25:44.963 INFO:tasks.rados.rados.0.smithi165.stdout:2992: left oid 27 (ObjNum 1044 snap 321 seq_num 1044) 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2990: done (3 left) 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2991: done (2 left) 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2992: done (1 left) 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2993: done (0 left) 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2994: read oid 26 snap 319 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2994: expect deleted 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2995: copy_from oid 37 from oid 44 current snap is 322 2024-09-10T16:25:45.536 INFO:tasks.rados.rados.0.smithi165.stdout:2996: snap_remove snap 318 2024-09-10T16:25:47.231 INFO:tasks.rados.rados.0.smithi165.stdout:2996: done (2 left) 2024-09-10T16:25:47.232 INFO:tasks.rados.rados.0.smithi165.stdout:2997: write oid 33 current snap is 322 2024-09-10T16:25:47.232 INFO:tasks.rados.rados.0.smithi165.stdout:2997: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2024-09-10T16:25:47.239 INFO:tasks.rados.rados.0.smithi165.stdout:2997: writing smithi16534841-33 from 763315 to 1338881 tid 1 2024-09-10T16:25:47.248 INFO:tasks.rados.rados.0.smithi165.stdout:2997: writing smithi16534841-33 from 2067683 to 2723380 tid 2 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:2997: writing smithi16534841-33 from 3247498 to 3563315 tid 3 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:2998: delete oid 22 current snap is 322 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:2995: finishing copy_from racing read to smithi16534841-37 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:2995: finishing copy_from to smithi16534841-37 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1165 (ObjNum 1037 snap 318 seq_num 1037) dirty exists 2024-09-10T16:25:47.254 INFO:tasks.rados.rados.0.smithi165.stdout:2997: finishing write tid 1 to smithi16534841-33 2024-09-10T16:25:47.255 INFO:tasks.rados.rados.0.smithi165.stdout:2997: finishing write tid 2 to smithi16534841-33 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:2994: done (3 left) 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:2995: done (2 left) 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:2998: done (1 left) 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:2999: rollback oid 25 current snap is 322 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 25 to 317 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:3000: read oid 29 snap -1 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:3000: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:3001: write (excl) oid 40 current snap is 322 2024-09-10T16:25:47.263 INFO:tasks.rados.rados.0.smithi165.stdout:3001: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2024-09-10T16:25:47.271 INFO:tasks.rados.rados.0.smithi165.stdout:3001: writing smithi16534841-40 from 796929 to 1522993 tid 1 2024-09-10T16:25:47.279 INFO:tasks.rados.rados.0.smithi165.stdout:3001: writing smithi16534841-40 from 2184647 to 2844013 tid 2 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:3001: writing smithi16534841-40 from 3302344 to 3596929 tid 3 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:3002: snap_create 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: finishing write tid 3 to smithi16534841-33 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: finishing write tid 4 to smithi16534841-33 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: finishing write tid 5 to smithi16534841-33 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: oid 33 updating version 0 to 1803 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: oid 33 version 1803 is already newer than 1802 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: oid 33 version 1803 is already newer than 1800 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: oid 33 version 1803 is already newer than 1801 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1803 (ObjNum 1045 snap 322 seq_num 1045) dirty exists 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:2997: left oid 33 (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:25:47.284 INFO:tasks.rados.rados.0.smithi165.stdout:3000: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:25:47.318 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3208851 2024-09-10T16:25:47.451 INFO:tasks.rados.rados.0.smithi165.stdout:2999: finishing rollback tid 0 to smithi16534841-25 2024-09-10T16:25:47.451 INFO:tasks.rados.rados.0.smithi165.stdout:2999: finishing rollback tid 1 to smithi16534841-25 2024-09-10T16:25:47.451 INFO:tasks.rados.rados.0.smithi165.stdout:2999: finishing rollback tid 2 to smithi16534841-25 2024-09-10T16:25:47.451 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1806 (ObjNum 926 snap 283 seq_num 926) dirty exists 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: finishing write tid 1 to smithi16534841-40 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: finishing write tid 2 to smithi16534841-40 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: finishing write tid 3 to smithi16534841-40 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: finishing write tid 4 to smithi16534841-40 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: finishing write tid 5 to smithi16534841-40 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: oid 40 updating version 0 to 1248 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: oid 40 version 1248 is already newer than 1246 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: oid 40 version 1248 is already newer than 1247 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: oid 40 version 1248 is already newer than 1245 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1248 (ObjNum 1047 snap 322 seq_num 1047) dirty exists 2024-09-10T16:25:47.452 INFO:tasks.rados.rados.0.smithi165.stdout:3001: left oid 40 (ObjNum 1047 snap 322 seq_num 1047) 2024-09-10T16:25:48.077 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:25:48.077 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:25:48.077 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:25:48.080 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:25:48.081 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:25:48.081 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.188+0000 7fc5bee32640 10 monclient: get_monmap_and_config 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.188+0000 7fc5bee32640 10 monclient: build_initial_monmap 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.188+0000 7fc5bee32640 10 monclient: monmap: 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:48.191 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:48.190742+0000 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:48.190742+0000 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:48.192 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.189+0000 7fc5bee32640 1 Processor -- start 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.189+0000 7fc5bee32640 1 -- start start 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.189+0000 7fc5bee32640 10 monclient: init 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5b806d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc5b806d540 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 0x7fc5b8140460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b813e070 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b8135510 con 0x7fc5b806d160 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5b809cbe0 con 0x7fc5b806d540 2024-09-10T16:25:48.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b809cf10 con 0x7fc5b813e070 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bee32640 10 monclient(hunting): authenticate will time out at 1926.678229s 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bcbce640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc5b806d160 0x7fc5b8134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:42646/0 (socket says 172.21.15.164:42646) 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bcbce640 1 -- 172.21.15.164:0/4156017680 learned_addr learned my addr 172.21.15.164:0/4156017680 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 0x7fc5b8140460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): get_auth_request con 0x7fc5b813e070 auth_method 0 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:48.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient(hunting): handle_auth_done global_id 214178 payload 274 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/4156017680 >> v1:172.21.15.165:6789/0 conn(0x7fc5b806d540 legacy=0x7fc5b80a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/4156017680 >> v1:172.21.15.164:6789/0 conn(0x7fc5b806d160 legacy=0x7fc5b8134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient: _finish_hunting 0 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 1 monclient: found mon.noname-b 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.190+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/4156017680 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b809d1a0 con 0x7fc5b813e070 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/4156017680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 msgr2=0x7fc5b8140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bd3cf640 1 --2- 172.21.15.164:0/4156017680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 0x7fc5b8140460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009dc0 tx=0x7fc5b002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f13630717477dede server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 1 -- 172.21.15.164:0/4156017680 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0007a30 con 0x7fc5b813e070 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: dump: 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 1 -- 172.21.15.164:0/4156017680 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b0007bd0 con 0x7fc5b813e070 2024-09-10T16:25:48.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bee32640 5 monclient: authenticate success, global_id 214178 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bee32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 1 -- 172.21.15.164:0/4156017680 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0031400 con 0x7fc5b813e070 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bee32640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5bee32640 4 set_mon_vals no callback set 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.193+0000 7fc5b77fe640 10 monclient: dump: 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 10 monclient: shutdown 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 -- 172.21.15.164:0/4156017680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 msgr2=0x7fc5b8140460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 --2- 172.21.15.164:0/4156017680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 0x7fc5b8140460 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009dc0 tx=0x7fc5b002a480 comp rx=0 tx=0).stop 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 -- 172.21.15.164:0/4156017680 shutdown_connections 2024-09-10T16:25:48.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 --2- 172.21.15.164:0/4156017680 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b813e070 0x7fc5b8140460 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 -- 172.21.15.164:0/4156017680 >> 172.21.15.164:0/4156017680 conn(0x7fc5b80aacb0 msgr2=0x7fc5b80ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 -- 172.21.15.164:0/4156017680 shutdown_connections 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 1 -- 172.21.15.164:0/4156017680 wait complete. 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 10 monclient: build_initial_monmap 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.194+0000 7fc5bee32640 10 monclient: monmap: 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:48.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 Processor -- start 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 -- start start 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient: init 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b806d160 0x7fc5b809e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-b con 0x7fc5b806d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-c con 0x7fc5b806d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:48.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): picked mon.noname-a con 0x7fc5b813e070 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b8140d70 con 0x7fc5b813e070 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b8140bf0 con 0x7fc5b806d160 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc5b8140ef0 con 0x7fc5b806d540 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5bee32640 10 monclient(hunting): authenticate will time out at 1926.683912s 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.195+0000 7fc5b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:58126/0 (socket says 172.21.15.164:58126) 2024-09-10T16:25:48.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 -- 172.21.15.164:0/1885143902 learned_addr learned my addr 172.21.15.164:0/1885143902 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): get_auth_request con 0x7fc5b806d540 auth_method 0 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 10 monclient(hunting): get_auth_request con 0x7fc5b813e070 auth_method 0 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 msgr2=0x7fc5b809f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:25:48.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 msgr2=0x7fc5b809f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bcbce640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b806d160 0x7fc5b809e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bd3cf640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bcbce640 10 monclient(hunting): get_auth_request con 0x7fc5b806d160 auth_method 0 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bcbce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bcbce640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5bcbce640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient(hunting): handle_auth_done global_id 214181 payload 824 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b806d160 msgr2=0x7fc5b809e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b806d160 0x7fc5b809e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 msgr2=0x7fc5b809f3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:48.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 monclient: found mon.noname-c 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.196+0000 7fc5b7fff640 1 -- 172.21.15.164:0/1885143902 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b80a2180 con 0x7fc5b806d540 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b7fff640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 msgr2=0x7fc5b809eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b7fff640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400c430 tx=0x7fc5a400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a40119a0 con 0x7fc5b806d540 2024-09-10T16:25:48.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: dump: 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:48.214 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: _finish_auth 0 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: _check_auth_tickets 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5a4017070 con 0x7fc5b806d540 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 5 monclient: authenticate success, global_id 214181 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a4011d80 con 0x7fc5b806d540 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 10 monclient: _renew_subs 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b80a0120 con 0x7fc5b806d540 2024-09-10T16:25:48.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b6ffd640 4 set_mon_vals no callback set 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5b57fa640 10 monclient: dump: 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:48.216 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 10 monclient: _renew_subs 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.198+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b80a4890 con 0x7fc5b806d540 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc5a401c460 con 0x7fc5b806d540 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5bee32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5bee32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5bee32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b8134e00 con 0x7fc5b806d540 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5b57fa640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 0x7fc59c040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.199+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(671..671 src has 494..671) ==== 4827+0+0 (secure 0 0 0) 0x7fc5a4052c50 con 0x7fc5b806d540 2024-09-10T16:25:48.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.200+0000 7fc5bcbce640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 0x7fc59c040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:48.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.200+0000 7fc5bcbce640 10 monclient: get_auth_request con 0x7fc59c03e260 auth_method 0 2024-09-10T16:25:48.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.202+0000 7fc5bcbce640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 0x7fc59c040720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c7f0 tx=0x7fc5ac009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:48.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.204+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc5a4051620 con 0x7fc5b806d540 2024-09-10T16:25:48.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.204+0000 7fc5b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:48.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.204+0000 7fc5b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:48.240 INFO:tasks.rados.rados.0.smithi165.stdout:2997: done (4 left) 2024-09-10T16:25:48.240 INFO:tasks.rados.rados.0.smithi165.stdout:2999: done (3 left) 2024-09-10T16:25:48.240 INFO:tasks.rados.rados.0.smithi165.stdout:3000: done (2 left) 2024-09-10T16:25:48.241 INFO:tasks.rados.rados.0.smithi165.stdout:3001: done (1 left) 2024-09-10T16:25:48.241 INFO:tasks.rados.rados.0.smithi165.stdout:3002: done (0 left) 2024-09-10T16:25:48.241 INFO:tasks.rados.rados.0.smithi165.stdout:3003: write oid 28 current snap is 323 2024-09-10T16:25:48.241 INFO:tasks.rados.rados.0.smithi165.stdout:3003: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2024-09-10T16:25:48.244 INFO:tasks.rados.rados.0.smithi165.stdout:3003: writing smithi16534841-28 from 413736 to 815049 tid 1 2024-09-10T16:25:48.249 INFO:tasks.rados.rados.0.smithi165.stdout:3003: writing smithi16534841-28 from 1326776 to 1729800 tid 2 2024-09-10T16:25:48.255 INFO:tasks.rados.rados.0.smithi165.stdout:3003: writing smithi16534841-28 from 2413414 to 2871307 tid 3 2024-09-10T16:25:48.260 INFO:tasks.rados.rados.0.smithi165.stdout:3003: writing smithi16534841-28 from 3438716 to 3613736 tid 4 2024-09-10T16:25:48.260 INFO:tasks.rados.rados.0.smithi165.stdout:3004: delete oid 24 current snap is 323 2024-09-10T16:25:48.260 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 1 to smithi16534841-28 2024-09-10T16:25:48.260 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 2 to smithi16534841-28 2024-09-10T16:25:48.260 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 3 to smithi16534841-28 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 4 to smithi16534841-28 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 5 to smithi16534841-28 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: finishing write tid 6 to smithi16534841-28 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: oid 28 updating version 0 to 1279 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: oid 28 updating version 1279 to 1280 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: oid 28 version 1280 is already newer than 1276 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: oid 28 version 1280 is already newer than 1277 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: oid 28 version 1280 is already newer than 1278 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1280 (ObjNum 1048 snap 323 seq_num 1048) dirty exists 2024-09-10T16:25:48.262 INFO:tasks.rados.rados.0.smithi165.stdout:3003: left oid 28 (ObjNum 1048 snap 323 seq_num 1048) 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3003: done (1 left) 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3004: done (0 left) 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3005: read oid 18 snap 317 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3005: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3006: write (excl) oid 10 current snap is 323 2024-09-10T16:25:48.264 INFO:tasks.rados.rados.0.smithi165.stdout:3006: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2024-09-10T16:25:48.269 INFO:tasks.rados.rados.0.smithi165.stdout:3006: writing smithi16534841-10 from 447350 to 1115514 tid 1 2024-09-10T16:25:48.274 INFO:tasks.rados.rados.0.smithi165.stdout:3006: writing smithi16534841-10 from 1560093 to 1966786 tid 2 2024-09-10T16:25:48.280 INFO:tasks.rados.rados.0.smithi165.stdout:3006: writing smithi16534841-10 from 2584613 to 3065050 tid 3 2024-09-10T16:25:48.280 INFO:tasks.rados.rados.0.smithi165.stdout:3007: write (excl) oid 35 current snap is 323 2024-09-10T16:25:48.280 INFO:tasks.rados.rados.0.smithi165.stdout:3007: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2024-09-10T16:25:48.286 INFO:tasks.rados.rados.0.smithi165.stdout:3007: writing smithi16534841-35 from 464157 to 1207570 tid 1 2024-09-10T16:25:48.292 INFO:tasks.rados.rados.0.smithi165.stdout:3007: writing smithi16534841-35 from 1618575 to 2285279 tid 2 2024-09-10T16:25:48.299 INFO:tasks.rados.rados.0.smithi165.stdout:3007: writing smithi16534841-35 from 3012036 to 3420098 tid 3 2024-09-10T16:25:48.299 INFO:tasks.rados.rados.0.smithi165.stdout:3008: rollback oid 16 current snap is 323 2024-09-10T16:25:48.299 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 321 2024-09-10T16:25:48.299 INFO:tasks.rados.rados.0.smithi165.stdout:3005: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:25:48.322 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1841258 2024-09-10T16:25:48.377 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2365005 2024-09-10T16:25:48.440 INFO:tasks.rados.rados.0.smithi165.stdout:3006: finishing write tid 1 to smithi16534841-10 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: finishing write tid 2 to smithi16534841-10 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3005: done (3 left) 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3009: write oid 34 current snap is 323 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: finishing write tid 3 to smithi16534841-10 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: finishing write tid 4 to smithi16534841-10 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: finishing write tid 5 to smithi16534841-10 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: oid 10 updating version 0 to 1575 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: oid 10 version 1575 is already newer than 1574 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: oid 10 version 1575 is already newer than 1572 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: oid 10 version 1575 is already newer than 1573 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1575 (ObjNum 1050 snap 323 seq_num 1050) dirty exists 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3006: left oid 10 (ObjNum 1050 snap 323 seq_num 1050) 2024-09-10T16:25:48.441 INFO:tasks.rados.rados.0.smithi165.stdout:3007: finishing write tid 1 to smithi16534841-35 2024-09-10T16:25:48.442 INFO:tasks.rados.rados.0.smithi165.stdout:3009: seq_num 1052 ranges {480964=418662,1560704=410362,2523106=619334} 2024-09-10T16:25:48.445 INFO:tasks.rados.rados.0.smithi165.stdout:3009: writing smithi16534841-34 from 480964 to 899626 tid 1 2024-09-10T16:25:48.450 INFO:tasks.rados.rados.0.smithi165.stdout:3009: writing smithi16534841-34 from 1560704 to 1971066 tid 2 2024-09-10T16:25:48.457 INFO:tasks.rados.rados.0.smithi165.stdout:3009: writing smithi16534841-34 from 2523106 to 3142440 tid 3 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3006: done (3 left) 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3010: rollback oid 44 current snap is 323 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 317 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: finishing write tid 2 to smithi16534841-35 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: finishing write tid 3 to smithi16534841-35 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: finishing write tid 4 to smithi16534841-35 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: finishing write tid 5 to smithi16534841-35 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: oid 35 updating version 0 to 1440 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: oid 35 updating version 1440 to 1441 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: oid 35 version 1441 is already newer than 1439 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: oid 35 version 1441 is already newer than 1438 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1441 (ObjNum 1051 snap 323 seq_num 1051) dirty exists 2024-09-10T16:25:48.458 INFO:tasks.rados.rados.0.smithi165.stdout:3007: left oid 35 (ObjNum 1051 snap 323 seq_num 1051) 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3008: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3008: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3008: finishing rollback tid 2 to smithi16534841-16 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1860 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3009: finishing write tid 1 to smithi16534841-34 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3009: finishing write tid 2 to smithi16534841-34 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3007: done (3 left) 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3008: done (2 left) 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3011: read oid 36 snap 312 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3011: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:25:48.459 INFO:tasks.rados.rados.0.smithi165.stdout:3012: delete oid 10 current snap is 323 2024-09-10T16:25:48.461 INFO:tasks.rados.rados.0.smithi165.stdout:3012: done (3 left) 2024-09-10T16:25:48.461 INFO:tasks.rados.rados.0.smithi165.stdout:3013: rollback oid 6 current snap is 323 2024-09-10T16:25:48.461 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 317 2024-09-10T16:25:48.461 INFO:tasks.rados.rados.0.smithi165.stdout:3014: write (excl) oid 21 current snap is 323 2024-09-10T16:25:48.461 INFO:tasks.rados.rados.0.smithi165.stdout:3014: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2024-09-10T16:25:48.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.460+0000 7fc5bee32640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.460+0000 7fc5bee32640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.460+0000 7fc5bee32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.460+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc5b80aa1c0 con 0x7fc5b806d540 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.461+0000 7fc5b57fa640 1 -- 172.21.15.164:0/1885143902 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fc5a401c930 con 0x7fc5b806d540 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.461+0000 7fc5b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.461+0000 7fc5b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:48.463 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":188,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:48.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 msgr2=0x7fc59c040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 0x7fc59c040720 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7fc5ac00c7f0 tx=0x7fc5ac009040 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 10 monclient: shutdown 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 msgr2=0x7fc5b809eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc5a400c430 tx=0x7fc5a400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 shutdown_connections 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc59c03e260 0x7fc59c040720 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5b813e070 0x7fc5b809f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5b806d540 0x7fc5b809eeb0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 --2- 172.21.15.164:0/1885143902 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5b806d160 0x7fc5b809e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 >> 172.21.15.164:0/1885143902 conn(0x7fc5b80aacb0 msgr2=0x7fc5b8133ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 shutdown_connections 2024-09-10T16:25:48.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:48.463+0000 7fc5bee32640 1 -- 172.21.15.164:0/1885143902 wait complete. 2024-09-10T16:25:48.467 INFO:tasks.rados.rados.0.smithi165.stdout:3014: writing smithi16534841-21 from 514578 to 1083738 tid 1 2024-09-10T16:25:48.472 INFO:tasks.rados.rados.0.smithi165.stdout:3014: writing smithi16534841-21 from 1677668 to 2091699 tid 2 2024-09-10T16:25:48.479 INFO:tasks.rados.rados.0.smithi165.stdout:3014: writing smithi16534841-21 from 2577952 to 3219830 tid 3 2024-09-10T16:25:48.479 INFO:tasks.rados.rados.0.smithi165.stdout:3015: read oid 33 snap 319 2024-09-10T16:25:48.479 INFO:tasks.rados.rados.0.smithi165.stdout:3015: expect deleted 2024-09-10T16:25:48.479 INFO:tasks.rados.rados.0.smithi165.stdout:3016: write oid 50 current snap is 323 2024-09-10T16:25:48.479 INFO:tasks.rados.rados.0.smithi165.stdout:3016: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2024-09-10T16:25:48.485 INFO:tasks.rados.rados.0.smithi165.stdout:3016: writing smithi16534841-50 from 531385 to 1175794 tid 1 2024-09-10T16:25:48.491 INFO:tasks.rados.rados.0.smithi165.stdout:3016: writing smithi16534841-50 from 1736150 to 2293839 tid 2 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3016: writing smithi16534841-50 from 3005375 to 3574878 tid 3 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3017: read oid 2 snap -1 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3017: expect deleted 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3018: write (excl) oid 23 current snap is 323 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3009: finishing write tid 3 to smithi16534841-34 2024-09-10T16:25:48.499 INFO:tasks.rados.rados.0.smithi165.stdout:3018: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2024-09-10T16:25:48.505 INFO:tasks.rados.rados.0.smithi165.stdout:3018: writing smithi16534841-23 from 548192 to 1267850 tid 1 2024-09-10T16:25:48.510 INFO:tasks.rados.rados.0.smithi165.stdout:3018: writing smithi16534841-23 from 1678279 to 2095979 tid 2 2024-09-10T16:25:48.517 INFO:tasks.rados.rados.0.smithi165.stdout:3018: writing smithi16534841-23 from 2516445 to 3297220 tid 3 2024-09-10T16:25:48.517 INFO:tasks.rados.rados.0.smithi165.stdout:3019: snap_create 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: finishing write tid 4 to smithi16534841-34 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: finishing write tid 5 to smithi16534841-34 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: oid 34 updating version 0 to 1571 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: oid 34 updating version 1571 to 1572 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: oid 34 version 1572 is already newer than 1569 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: oid 34 version 1572 is already newer than 1570 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1572 (ObjNum 1052 snap 323 seq_num 1052) dirty exists 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3009: left oid 34 (ObjNum 1052 snap 323 seq_num 1052) 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3013: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3010: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:25:48.518 INFO:tasks.rados.rados.0.smithi165.stdout:3010: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:25:48.519 INFO:tasks.rados.rados.0.smithi165.stdout:3010: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:25:48.519 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1168 (ObjNum 962 snap 292 seq_num 962) dirty exists 2024-09-10T16:25:48.519 INFO:tasks.rados.rados.0.smithi165.stdout:3011: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:25:48.595 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 334298 2024-09-10T16:25:48.639 INFO:tasks.rados.rados.0.smithi165.stdout:3014: finishing write tid 1 to smithi16534841-21 2024-09-10T16:25:48.639 INFO:tasks.rados.rados.0.smithi165.stdout:3014: finishing write tid 2 to smithi16534841-21 2024-09-10T16:25:48.639 INFO:tasks.rados.rados.0.smithi165.stdout:3014: finishing write tid 3 to smithi16534841-21 2024-09-10T16:25:48.639 INFO:tasks.rados.rados.0.smithi165.stdout:3014: finishing write tid 4 to smithi16534841-21 2024-09-10T16:25:48.639 INFO:tasks.rados.rados.0.smithi165.stdout:3014: finishing write tid 5 to smithi16534841-21 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3014: oid 21 updating version 0 to 1808 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3014: oid 21 updating version 1808 to 1809 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3014: oid 21 version 1809 is already newer than 1807 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3014: oid 21 updating version 1809 to 1810 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1810 (ObjNum 1054 snap 323 seq_num 1054) dirty exists 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3014: left oid 21 (ObjNum 1054 snap 323 seq_num 1054) 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: finishing write tid 1 to smithi16534841-50 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: finishing write tid 2 to smithi16534841-50 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: finishing write tid 3 to smithi16534841-50 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: finishing write tid 4 to smithi16534841-50 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: finishing write tid 5 to smithi16534841-50 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: oid 50 updating version 0 to 1858 2024-09-10T16:25:48.640 INFO:tasks.rados.rados.0.smithi165.stdout:3016: oid 50 version 1858 is already newer than 1855 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3016: oid 50 version 1858 is already newer than 1856 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3016: oid 50 version 1858 is already newer than 1857 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1858 (ObjNum 1055 snap 323 seq_num 1055) dirty exists 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3016: left oid 50 (ObjNum 1055 snap 323 seq_num 1055) 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: finishing write tid 1 to smithi16534841-23 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: finishing write tid 2 to smithi16534841-23 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: finishing write tid 3 to smithi16534841-23 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: finishing write tid 4 to smithi16534841-23 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: finishing write tid 5 to smithi16534841-23 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: oid 23 updating version 0 to 1684 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: oid 23 updating version 1684 to 1685 2024-09-10T16:25:48.641 INFO:tasks.rados.rados.0.smithi165.stdout:3018: oid 23 version 1685 is already newer than 1682 2024-09-10T16:25:48.642 INFO:tasks.rados.rados.0.smithi165.stdout:3018: oid 23 version 1685 is already newer than 1683 2024-09-10T16:25:48.642 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1685 (ObjNum 1056 snap 323 seq_num 1056) dirty exists 2024-09-10T16:25:48.642 INFO:tasks.rados.rados.0.smithi165.stdout:3018: left oid 23 (ObjNum 1056 snap 323 seq_num 1056) 2024-09-10T16:25:49.242 INFO:tasks.rados.rados.0.smithi165.stdout:3009: done (9 left) 2024-09-10T16:25:49.242 INFO:tasks.rados.rados.0.smithi165.stdout:3010: done (8 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3011: done (7 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3013: done (6 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3014: done (5 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3015: done (4 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3016: done (3 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3017: done (2 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3018: done (1 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3019: done (0 left) 2024-09-10T16:25:49.243 INFO:tasks.rados.rados.0.smithi165.stdout:3020: snap_create 2024-09-10T16:25:50.252 INFO:tasks.rados.rados.0.smithi165.stdout:3020: done (0 left) 2024-09-10T16:25:50.252 INFO:tasks.rados.rados.0.smithi165.stdout:3021: delete oid 35 current snap is 325 2024-09-10T16:25:50.256 INFO:tasks.rados.rados.0.smithi165.stdout:3021: done (0 left) 2024-09-10T16:25:50.256 INFO:tasks.rados.rados.0.smithi165.stdout:3022: read oid 18 snap -1 2024-09-10T16:25:50.256 INFO:tasks.rados.rados.0.smithi165.stdout:3022: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:25:50.256 INFO:tasks.rados.rados.0.smithi165.stdout:3023: snap_remove snap 317 2024-09-10T16:25:51.260 INFO:tasks.rados.rados.0.smithi165.stdout:3023: done (1 left) 2024-09-10T16:25:51.260 INFO:tasks.rados.rados.0.smithi165.stdout:3024: read oid 37 snap 311 2024-09-10T16:25:51.260 INFO:tasks.rados.rados.0.smithi165.stdout:3024: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:25:51.260 INFO:tasks.rados.rados.0.smithi165.stdout:3025: snap_remove snap 320 2024-09-10T16:25:51.477 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:51.496 INFO:tasks.rados.rados.0.smithi165.stdout:3025: done (2 left) 2024-09-10T16:25:51.497 INFO:tasks.rados.rados.0.smithi165.stdout:3026: snap_create 2024-09-10T16:25:51.497 INFO:tasks.rados.rados.0.smithi165.stdout:3022: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:25:51.561 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:51.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.590+0000 7f6f78953640 10 monclient: get_monmap_and_config 2024-09-10T16:25:51.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.590+0000 7f6f78953640 10 monclient: build_initial_monmap 2024-09-10T16:25:51.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.590+0000 7f6f78953640 10 monclient: monmap: 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:51.591320+0000 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:51.591320+0000 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:51.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 Processor -- start 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 -- start start 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient: init 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6f7409d160 0x7f6f7409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7409d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:51.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6f7409f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6f7409a430 con 0x7f6f7409f4c0 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6f740a24e0 con 0x7f6f7409c180 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6f740a2bf0 con 0x7f6f7409d160 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f78953640 10 monclient(hunting): authenticate will time out at 1930.078750s 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55596/0 (socket says 172.21.15.164:55596) 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 1 -- 172.21.15.164:0/2647174383 learned_addr learned my addr 172.21.15.164:0/2647174383 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 10 monclient(hunting): get_auth_request con 0x7f6f7409c180 auth_method 0 2024-09-10T16:25:51.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 1 -- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 msgr2=0x7f6f7409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.591+0000 7f6f7259c640 1 -- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 msgr2=0x7f6f7409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f7259c640 1 --2- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/2647174383 >> v2:172.21.15.165:3300/0 conn(0x7f6f7409d160 0x7f6f7409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f7259c640 1 --2- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): get_auth_request con 0x7f6f7409d160 auth_method 0 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:51.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_done global_id 154633 payload 274 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 -- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 msgr2=0x7f6f7409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 -- 172.21.15.164:0/2647174383 >> v1:172.21.15.165:6789/0 conn(0x7f6f7409f4c0 legacy=0x7f6f7413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 monclient: found mon.noname-c 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:51.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.592+0000 7f6f71d9b640 1 -- 172.21.15.164:0/2647174383 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f7409c560 con 0x7f6f7409d160 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f71d9b640 1 -- 172.21.15.164:0/2647174383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409d160 msgr2=0x7f6f7409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/2647174383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409d160 0x7f6f7409ef80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097a0 tx=0x7f6f5c02a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2819ff48e55b22e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 1 -- 172.21.15.164:0/2647174383 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c037070 con 0x7f6f7409d160 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: dump: 2024-09-10T16:25:51.604 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: _finish_auth 0 2024-09-10T16:25:51.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: _check_auth_tickets 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 1 -- 172.21.15.164:0/2647174383 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c032440 con 0x7f6f7409d160 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 1 -- 172.21.15.164:0/2647174383 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c032810 con 0x7f6f7409d160 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f78953640 5 monclient: authenticate success, global_id 154633 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f78953640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f78953640 4 set_mon_vals no callback set 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:51.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.595+0000 7f6f7159a640 10 monclient: dump: 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 10 monclient: shutdown 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 -- 172.21.15.164:0/2647174383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409d160 msgr2=0x7f6f7409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 --2- 172.21.15.164:0/2647174383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409d160 0x7f6f7409ef80 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6f5c0097a0 tx=0x7f6f5c02a010 comp rx=0 tx=0).stop 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 -- 172.21.15.164:0/2647174383 shutdown_connections 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 --2- 172.21.15.164:0/2647174383 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409d160 0x7f6f7409ef80 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 --2- 172.21.15.164:0/2647174383 >> v2:172.21.15.164:3300/0 conn(0x7f6f7409c180 0x7f6f7409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 -- 172.21.15.164:0/2647174383 >> 172.21.15.164:0/2647174383 conn(0x7f6f740a3110 msgr2=0x7f6f740a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 -- 172.21.15.164:0/2647174383 shutdown_connections 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.596+0000 7f6f78953640 1 -- 172.21.15.164:0/2647174383 wait complete. 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient: build_initial_monmap 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient: monmap: 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:51.608 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 1 Processor -- start 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 1 -- start start 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient: init 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:51.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-a con 0x7f6f7409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 0x7f6f740999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-c con 0x7f6f7409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409f4c0 0x7f6f741cace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient(hunting): picked mon.noname-b con 0x7f6f7409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.597+0000 7f6f78953640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6f74074f90 con 0x7f6f7409c180 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6f74074e10 con 0x7f6f7409f4c0 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6f7413cce0 con 0x7f6f7409d160 2024-09-10T16:25:51.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f78953640 10 monclient(hunting): authenticate will time out at 1930.085023s 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55600/0 (socket says 172.21.15.164:55600) 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 -- 172.21.15.164:0/133020826 learned_addr learned my addr 172.21.15.164:0/133020826 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 10 monclient(hunting): get_auth_request con 0x7f6f7409c180 auth_method 0 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:51.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 0x7f6f740999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): get_auth_request con 0x7f6f7409d160 auth_method 0 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409f4c0 0x7f6f741cace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): get_auth_request con 0x7f6f7409f4c0 auth_method 0 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:51.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 msgr2=0x7f6f74099480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 msgr2=0x7f6f74099480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f7259c640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f72d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient(hunting): handle_auth_done global_id 214184 payload 824 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409f4c0 msgr2=0x7f6f741cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409f4c0 0x7f6f741cace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 msgr2=0x7f6f74099480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 monclient: found mon.noname-c 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.598+0000 7f6f71d9b640 1 -- 172.21.15.164:0/133020826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f740734a0 con 0x7f6f7409d160 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f71d9b640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 msgr2=0x7f6f740999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f71d9b640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 0x7f6f740999c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409abb0 tx=0x7f6f5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c040020 con 0x7f6f7409d160 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:51.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: dump: 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: _finish_auth 0 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:51.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 5 monclient: authenticate success, global_id 214184 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6f5c038040 con 0x7f6f7409d160 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 10 monclient: _renew_subs 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f74073780 con 0x7f6f7409d160 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5c037070 con 0x7f6f7409d160 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f6affd640 10 monclient: dump: 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:51.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 10 monclient: _renew_subs 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f7409d620 con 0x7f6f7409d160 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.600+0000 7f6f70d99640 4 set_mon_vals no callback set 2024-09-10T16:25:51.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.601+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6f5c043020 con 0x7f6f7409d160 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.601+0000 7f6f78953640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.601+0000 7f6f78953640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.601+0000 7f6f78953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.601+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f7413d080 con 0x7f6f7409d160 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.602+0000 7f6f6affd640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 0x7f6f540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.602+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(676..676 src has 494..676) ==== 4827+0+0 (secure 0 0 0) 0x7f6f5c072f50 con 0x7f6f7409d160 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.602+0000 7f6f7259c640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 0x7f6f540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.602+0000 7f6f7259c640 10 monclient: get_auth_request con 0x7f6f5403e210 auth_method 0 2024-09-10T16:25:51.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.605+0000 7f6f7259c640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 0x7f6f540406d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6f640091d0 tx=0x7f6f64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.606+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6f5c04a030 con 0x7f6f7409d160 2024-09-10T16:25:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.606+0000 7f6f6affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:51.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.606+0000 7f6f6affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:51.683 INFO:tasks.rados.rados.0.smithi165.stdout:3024: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:25:51.696 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1455065 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f78953640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f78953640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f78953640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6f74071b00 con 0x7f6f7409d160 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f6affd640 1 -- 172.21.15.164:0/133020826 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f6f5c071920 con 0x7f6f7409d160 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f6affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:51.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.848+0000 7f6f6affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:51.851 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":188,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:51.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 msgr2=0x7f6f540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:51.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 0x7f6f540406d0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f6f640091d0 tx=0x7f6f64008040 comp rx=0 tx=0).stop 2024-09-10T16:25:51.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 10 monclient: shutdown 2024-09-10T16:25:51.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 msgr2=0x7f6f740999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.850+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 0x7f6f740999c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6f7409abb0 tx=0x7f6f5c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 shutdown_connections 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6f5403e210 0x7f6f540406d0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6f7409f4c0 0x7f6f741cace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6f7409d160 0x7f6f740999c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 --2- 172.21.15.164:0/133020826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6f7409c180 0x7f6f74099480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 >> 172.21.15.164:0/133020826 conn(0x7f6f740a3110 msgr2=0x7f6f740a4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 shutdown_connections 2024-09-10T16:25:51.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:51.851+0000 7f6f78953640 1 -- 172.21.15.164:0/133020826 wait complete. 2024-09-10T16:25:52.501 INFO:tasks.rados.rados.0.smithi165.stdout:3022: done (2 left) 2024-09-10T16:25:52.501 INFO:tasks.rados.rados.0.smithi165.stdout:3024: done (1 left) 2024-09-10T16:25:52.501 INFO:tasks.rados.rados.0.smithi165.stdout:3026: done (0 left) 2024-09-10T16:25:52.501 INFO:tasks.rados.rados.0.smithi165.stdout:3027: write oid 3 current snap is 326 2024-09-10T16:25:52.501 INFO:tasks.rados.rados.0.smithi165.stdout:3027: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2024-09-10T16:25:52.507 INFO:tasks.rados.rados.0.smithi165.stdout:3027: writing smithi16534841-3 from 581806 to 1168315 tid 1 2024-09-10T16:25:52.513 INFO:tasks.rados.rados.0.smithi165.stdout:3027: writing smithi16534841-3 from 1911596 to 2332965 tid 2 2024-09-10T16:25:52.520 INFO:tasks.rados.rados.0.smithi165.stdout:3027: writing smithi16534841-3 from 3087644 to 3490963 tid 3 2024-09-10T16:25:52.520 INFO:tasks.rados.rados.0.smithi165.stdout:3028: snap_create 2024-09-10T16:25:52.520 INFO:tasks.rados.rados.0.smithi165.stdout:3027: finishing write tid 1 to smithi16534841-3 2024-09-10T16:25:52.520 INFO:tasks.rados.rados.0.smithi165.stdout:3027: finishing write tid 2 to smithi16534841-3 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: finishing write tid 3 to smithi16534841-3 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: finishing write tid 4 to smithi16534841-3 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: finishing write tid 5 to smithi16534841-3 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: oid 3 updating version 0 to 1813 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: oid 3 version 1813 is already newer than 1811 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: oid 3 version 1813 is already newer than 1812 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: oid 3 updating version 1813 to 1814 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1814 (ObjNum 1058 snap 326 seq_num 1058) dirty exists 2024-09-10T16:25:52.523 INFO:tasks.rados.rados.0.smithi165.stdout:3027: left oid 3 (ObjNum 1058 snap 326 seq_num 1058) 2024-09-10T16:25:53.506 INFO:tasks.rados.rados.0.smithi165.stdout:3027: done (1 left) 2024-09-10T16:25:53.506 INFO:tasks.rados.rados.0.smithi165.stdout:3028: done (0 left) 2024-09-10T16:25:53.506 INFO:tasks.rados.rados.0.smithi165.stdout:3029: read oid 43 snap -1 2024-09-10T16:25:53.506 INFO:tasks.rados.rados.0.smithi165.stdout:3029: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:25:53.506 INFO:tasks.rados.rados.0.smithi165.stdout:3030: snap_create 2024-09-10T16:25:53.515 INFO:tasks.rados.rados.0.smithi165.stdout:3029: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:25:53.515 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:25:53.597 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 365268 2024-09-10T16:25:54.518 INFO:tasks.rados.rados.0.smithi165.stdout:3029: done (1 left) 2024-09-10T16:25:54.518 INFO:tasks.rados.rados.0.smithi165.stdout:3030: done (0 left) 2024-09-10T16:25:54.518 INFO:tasks.rados.rados.0.smithi165.stdout:3031: snap_remove snap 315 2024-09-10T16:25:54.864 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.970+0000 7fa05762b640 10 monclient: get_monmap_and_config 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.970+0000 7fa05762b640 10 monclient: build_initial_monmap 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.970+0000 7fa05762b640 10 monclient: monmap: 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:54.971567+0000 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:54.971567+0000 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:54.972 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:54.973 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:54.973 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.971+0000 7fa05762b640 1 Processor -- start 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.971+0000 7fa05762b640 1 -- start start 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.971+0000 7fa05762b640 10 monclient: init 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.971+0000 7fa05762b640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.971+0000 7fa05762b640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa05009a430 0x7fa05009a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-c con 0x7fa05009a430 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:54.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa050074cb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa0500a2820 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05009cbd0 con 0x7fa050074cb0 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05013d420 con 0x7fa0500a2820 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa0500a2320 con 0x7fa05009a430 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa05762b640 10 monclient(hunting): authenticate will time out at 1933.459128s 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa054bc6640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa050074cb0 0x7fa05013c600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55586/0 (socket says 172.21.15.164:55586) 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa054bc6640 1 -- 172.21.15.164:0/2032505658 learned_addr learned my addr 172.21.15.164:0/2032505658 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 1 --2- 172.21.15.164:0/2032505658 >> v2:172.21.15.165:3300/0 conn(0x7fa05009a430 0x7fa05009a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): get_auth_request con 0x7fa05009a430 auth_method 0 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:54.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient(hunting): handle_auth_done global_id 154639 payload 274 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 1 -- 172.21.15.164:0/2032505658 >> v1:172.21.15.165:6789/0 conn(0x7fa0500a2820 legacy=0x7fa05013cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 1 -- 172.21.15.164:0/2032505658 >> v1:172.21.15.164:6789/0 conn(0x7fa050074cb0 legacy=0x7fa05013c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient: _finish_hunting 0 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 1 monclient: found mon.noname-c 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:54.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.972+0000 7fa0553c7640 1 -- 172.21.15.164:0/2032505658 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05009c2c0 con 0x7fa05009a430 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.975+0000 7fa0553c7640 1 -- 172.21.15.164:0/2032505658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa05009a430 msgr2=0x7fa05009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.975+0000 7fa0553c7640 1 --2- 172.21.15.164:0/2032505658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa05009a430 0x7fa05009a810 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097a0 tx=0x7fa04c029fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0d37d6d56540cb9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 1 -- 172.21.15.164:0/2032505658 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c037070 con 0x7fa05009a430 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: dump: 2024-09-10T16:25:54.981 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: _finish_auth 0 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:54.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 5 monclient: authenticate success, global_id 154639 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 1 -- 172.21.15.164:0/2032505658 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04c032640 con 0x7fa05009a430 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 1 -- 172.21.15.164:0/2032505658 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c032a10 con 0x7fa05009a430 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa047fff640 10 monclient: dump: 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.983 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 4 set_mon_vals no callback set 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 10 monclient: shutdown 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 -- 172.21.15.164:0/2032505658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa05009a430 msgr2=0x7fa05009a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 --2- 172.21.15.164:0/2032505658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa05009a430 0x7fa05009a810 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097a0 tx=0x7fa04c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 -- 172.21.15.164:0/2032505658 shutdown_connections 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 --2- 172.21.15.164:0/2032505658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa05009a430 0x7fa05009a810 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:54.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 -- 172.21.15.164:0/2032505658 >> 172.21.15.164:0/2032505658 conn(0x7fa0500a3110 msgr2=0x7fa0500a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 -- 172.21.15.164:0/2032505658 shutdown_connections 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.976+0000 7fa05762b640 1 -- 172.21.15.164:0/2032505658 wait complete. 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.977+0000 7fa05762b640 10 monclient: build_initial_monmap 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.977+0000 7fa05762b640 10 monclient: monmap: 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:54.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.977+0000 7fa05762b640 1 Processor -- start 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.977+0000 7fa05762b640 1 -- start start 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.977+0000 7fa05762b640 10 monclient: init 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa050074cb0 0x7fa0501d1890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-b con 0x7fa050074cb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-a con 0x7fa05009a430 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 0x7fa0500724d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:54.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): picked mon.noname-c con 0x7fa0500a2820 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa05009d450 con 0x7fa05009a430 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa05009d2d0 con 0x7fa050074cb0 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa05009ad50 con 0x7fa0500a2820 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa05762b640 10 monclient(hunting): authenticate will time out at 1933.465311s 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 0x7fa0500724d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39266/0 (socket says 172.21.15.164:39266) 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): get_auth_request con 0x7fa0500a2820 auth_method 0 2024-09-10T16:25:54.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 -- 172.21.15.164:0/664650592 learned_addr learned my addr 172.21.15.164:0/664650592 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 10 monclient(hunting): get_auth_request con 0x7fa05009a430 auth_method 0 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 msgr2=0x7fa0501d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 msgr2=0x7fa0501d1dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:54.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa050074cb0 0x7fa0501d1890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): get_auth_request con 0x7fa050074cb0 auth_method 0 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa054bc6640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa0553c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient(hunting): handle_auth_done global_id 214187 payload 824 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa050074cb0 msgr2=0x7fa0501d1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:54.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa050074cb0 0x7fa0501d1890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 msgr2=0x7fa0501d1dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient: _finish_hunting 0 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 monclient: found mon.noname-c 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:54.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.978+0000 7fa055bc8640 1 -- 172.21.15.164:0/664650592 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05012c9d0 con 0x7fa0500a2820 2024-09-10T16:25:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.980+0000 7fa055bc8640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 msgr2=0x7fa0500724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.980+0000 7fa055bc8640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 0x7fa0500724d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa04800c750 tx=0x7fa04800cc10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:54.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa048016020 con 0x7fa0500a2820 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: dump: 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:54.993 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04800aa60 con 0x7fa0500a2820 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 5 monclient: authenticate success, global_id 214187 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 10 monclient: _renew_subs 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05012ccb0 con 0x7fa0500a2820 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04800ae50 con 0x7fa0500a2820 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:54.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa045ffb640 10 monclient: dump: 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa0477fe640 4 set_mon_vals no callback set 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 10 monclient: _renew_subs 2024-09-10T16:25:54.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.981+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05012c410 con 0x7fa0500a2820 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa048019020 con 0x7fa0500a2820 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa05762b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa05762b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa05762b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa05009b1b0 con 0x7fa0500a2820 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa045ffb640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.982+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(679..679 src has 494..679) ==== 4755+0+0 (secure 0 0 0) 0x7fa048052b90 con 0x7fa0500a2820 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 10 monclient: get_auth_request con 0x7fa03403e3b0 auth_method 0 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 msgr2=0x7fa034040870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 msgr2=0x7fa034040870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:25:54.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.983+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:25:54.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.987+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa048020030 con 0x7fa0500a2820 2024-09-10T16:25:54.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.987+0000 7fa045ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:54.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:54.987+0000 7fa045ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:55.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.184+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.184+0000 7fa0553c7640 10 monclient: get_auth_request con 0x7fa03403e3b0 auth_method 0 2024-09-10T16:25:55.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.187+0000 7fa0553c7640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa05006e760 tx=0x7fa04c03e040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.231+0000 7fa05762b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.231+0000 7fa05762b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:55.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.231+0000 7fa05762b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.231+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa05012c610 con 0x7fa0500a2820 2024-09-10T16:25:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.232+0000 7fa045ffb640 1 -- 172.21.15.164:0/664650592 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fa0480515a0 con 0x7fa0500a2820 2024-09-10T16:25:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.232+0000 7fa045ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:55.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.232+0000 7fa045ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:55.234 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":192,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:55.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 msgr2=0x7fa034040870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fa05006e760 tx=0x7fa04c03e040 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 10 monclient: shutdown 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 msgr2=0x7fa0500724d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 0x7fa0500724d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa04800c750 tx=0x7fa04800cc10 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 shutdown_connections 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa03403e3b0 0x7fa034040870 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa0500a2820 0x7fa0500724d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa05009a430 0x7fa0501d1dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 --2- 172.21.15.164:0/664650592 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa050074cb0 0x7fa0501d1890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 >> 172.21.15.164:0/664650592 conn(0x7fa0500a3110 msgr2=0x7fa0500980e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 shutdown_connections 2024-09-10T16:25:55.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.234+0000 7fa05762b640 1 -- 172.21.15.164:0/664650592 wait complete. 2024-09-10T16:25:55.248 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:25:55.248 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:25:55.248 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:25:55.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.348+0000 7f3f4be2a640 10 monclient: get_monmap_and_config 2024-09-10T16:25:55.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.348+0000 7f3f4be2a640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.348+0000 7f3f4be2a640 10 monclient: monmap: 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:55.349193+0000 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:55.349193+0000 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:55.350 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:55.351 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.348+0000 7f3f4be2a640 1 Processor -- start 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 -- start start 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient: init 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3f4409c180 0x7f3f4409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4409c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:55.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f4409d160 0x7f3f4409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4409d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3f4409f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f4409a430 con 0x7f3f4409f4c0 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3f440a24e0 con 0x7f3f4409d160 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3f440a2bf0 con 0x7f3f4409c180 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4be2a640 10 monclient(hunting): authenticate will time out at 1933.836599s 2024-09-10T16:25:55.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3f4409d160 0x7f3f4409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4a3c7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3f4409f4c0 0x7f3f4413b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55602/0 (socket says 172.21.15.164:55602) 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f4a3c7640 1 -- 172.21.15.164:0/2805672793 learned_addr learned my addr 172.21.15.164:0/2805672793 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): get_auth_request con 0x7f3f4409d160 auth_method 0 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 1 --2- 172.21.15.164:0/2805672793 >> v2:172.21.15.165:3300/0 conn(0x7f3f4409c180 0x7f3f4409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): get_auth_request con 0x7f3f4409c180 auth_method 0 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 10 monclient(hunting): handle_auth_done global_id 224097 payload 274 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 1 -- 172.21.15.164:0/2805672793 >> v2:172.21.15.165:3300/0 conn(0x7f3f4409c180 msgr2=0x7f3f4409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 1 --2- 172.21.15.164:0/2805672793 >> v2:172.21.15.165:3300/0 conn(0x7f3f4409c180 0x7f3f4409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.349+0000 7f3f493c5640 1 -- 172.21.15.164:0/2805672793 >> v1:172.21.15.164:6789/0 conn(0x7f3f4409f4c0 legacy=0x7f3f4413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.350+0000 7f3f493c5640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.350+0000 7f3f493c5640 1 monclient: found mon.noname-a 2024-09-10T16:25:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.350+0000 7f3f493c5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.350+0000 7f3f493c5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.350+0000 7f3f493c5640 1 -- 172.21.15.164:0/2805672793 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4409c560 con 0x7f3f4409d160 2024-09-10T16:25:55.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f493c5640 1 -- 172.21.15.164:0/2805672793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409d160 msgr2=0x7f3f4409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f493c5640 1 --2- 172.21.15.164:0/2805672793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409d160 0x7f3f4409ef80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f340097a0 tx=0x7f3f3402a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585b00d39bbbd8dd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f48bc4640 1 -- 172.21.15.164:0/2805672793 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34037070 con 0x7f3f4409d160 2024-09-10T16:25:55.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f48bc4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f48bc4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.352+0000 7f3f48bc4640 10 monclient: dump: 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.366 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 1 -- 172.21.15.164:0/2805672793 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f34032440 con 0x7f3f4409d160 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 5 monclient: authenticate success, global_id 224097 2024-09-10T16:25:55.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 1 -- 172.21.15.164:0/2805672793 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34032810 con 0x7f3f4409d160 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 4 set_mon_vals no callback set 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f48bc4640 10 monclient: dump: 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 10 monclient: shutdown 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 -- 172.21.15.164:0/2805672793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409d160 msgr2=0x7f3f4409ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/2805672793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409d160 0x7f3f4409ef80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3f340097a0 tx=0x7f3f3402a010 comp rx=0 tx=0).stop 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 -- 172.21.15.164:0/2805672793 shutdown_connections 2024-09-10T16:25:55.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/2805672793 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409d160 0x7f3f4409ef80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/2805672793 >> v2:172.21.15.165:3300/0 conn(0x7f3f4409c180 0x7f3f4409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 -- 172.21.15.164:0/2805672793 >> 172.21.15.164:0/2805672793 conn(0x7f3f440a3110 msgr2=0x7f3f440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 -- 172.21.15.164:0/2805672793 shutdown_connections 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.353+0000 7f3f4be2a640 1 -- 172.21.15.164:0/2805672793 wait complete. 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 10 monclient: monmap: 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.370 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 1 Processor -- start 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 1 -- start start 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 10 monclient: init 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.354+0000 7f3f4be2a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 0x7f3f441cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-c con 0x7f3f4409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4409d160 0x7f3f44073fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-b con 0x7f3f4409d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409f4c0 0x7f3f44074520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): picked mon.noname-a con 0x7f3f4409f4c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f44074f90 con 0x7f3f4409f4c0 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3f44074e10 con 0x7f3f4409d160 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3f4413cce0 con 0x7f3f4409c180 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4be2a640 10 monclient(hunting): authenticate will time out at 1933.842059s 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 0x7f3f441cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409f4c0 0x7f3f44074520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 0x7f3f441cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34412/0 (socket says 172.21.15.164:34412) 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 -- 172.21.15.164:0/991681753 learned_addr learned my addr 172.21.15.164:0/991681753 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): get_auth_request con 0x7f3f4409c180 auth_method 0 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): get_auth_request con 0x7f3f4409f4c0 auth_method 0 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4409d160 0x7f3f44073fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): get_auth_request con 0x7f3f4409d160 auth_method 0 2024-09-10T16:25:55.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient(hunting): handle_auth_done global_id 214190 payload 824 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4409d160 msgr2=0x7f3f44073fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3f4409d160 0x7f3f44073fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409f4c0 msgr2=0x7f3f44074520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409f4c0 0x7f3f44074520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 monclient: found mon.noname-c 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f49bc6640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4413f110 con 0x7f3f4409c180 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.355+0000 7f3f4a3c7640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4409f4c0 0x7f3f44074520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f49bc6640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 msgr2=0x7f3f441cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f49bc6640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 0x7f3f441cd0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e9e0 tx=0x7f3f4000eea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40017070 con 0x7f3f4409c180 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:55.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: dump: 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f4000cbe0 con 0x7f3f4409c180 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 5 monclient: authenticate success, global_id 214190 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40013470 con 0x7f3f4409c180 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3a7fc640 10 monclient: dump: 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 10 monclient: _renew_subs 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f3bfff640 4 set_mon_vals no callback set 2024-09-10T16:25:55.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f441cd630 con 0x7f3f4409c180 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 10 monclient: _renew_subs 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.357+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4413c2f0 con 0x7f3f4409c180 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f4be2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 0x7f3f4413aed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4413f580 con 0x7f3f4413aaf0 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3f4001a070 con 0x7f3f4409c180 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 0x7f3f4413aed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient: get_auth_request con 0x7f3f4413aaf0 auth_method 0 2024-09-10T16:25:55.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 10 monclient(hunting): handle_auth_done global_id 224103 payload 824 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f3a7fc640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f2803e210 0x7f3f280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.358+0000 7f3f493c5640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f4413d080 con 0x7f3f4413aaf0 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.359+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(679..679 src has 494..679) ==== 4755+0+0 (secure 0 0 0) 0x7f3f40052d70 con 0x7f3f4409c180 2024-09-10T16:25:55.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.359+0000 7f3f4a3c7640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f2803e210 0x7f3f280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.359+0000 7f3f4a3c7640 10 monclient: get_auth_request con 0x7f3f2803e210 auth_method 0 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f493c5640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 msgr2=0x7f3f4413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f493c5640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 0x7f3f4413aed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3f34034000 tx=0x7f3f340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34043070 con 0x7f3f4413aaf0 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 10 monclient: dump: 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.382 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3f3403c850 con 0x7f3f4413aaf0 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 msgr2=0x7f3f4413aed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 0x7f3f4413aed0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3f34034000 tx=0x7f3f340046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 msgr2=0x7f3f4413aed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.361+0000 7f3f3a7fc640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413aaf0 0x7f3f4413aed0 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3f34034000 tx=0x7f3f340046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.362+0000 7f3f4a3c7640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f2803e210 0x7f3f280406d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005c00 tx=0x7f3f3c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.367+0000 7f3f4be2a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:55.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.367+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 0x7f3f44098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.367+0000 7f3f4be2a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3f44077260 con 0x7f3f4413b410 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 0x7f3f44098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient: get_auth_request con 0x7f3f4413b410 auth_method 0 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 10 monclient(hunting): handle_auth_done global_id 224106 payload 824 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.368+0000 7f3f493c5640 1 -- 172.21.15.164:0/991681753 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f4413e1b0 con 0x7f3f4413b410 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f493c5640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 msgr2=0x7f3f44098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f493c5640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 0x7f3f44098220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402a050 tx=0x7f3f34009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f34037040 con 0x7f3f4413b410 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 10 monclient: dump: 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.387 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f3f340513f0 con 0x7f3f4413b410 2024-09-10T16:25:55.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 msgr2=0x7f3f44098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 0x7f3f44098220 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402a050 tx=0x7f3f34009190 comp rx=0 tx=0).stop 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 msgr2=0x7f3f44098220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.370+0000 7f3f3a7fc640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3f4413b410 0x7f3f44098220 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3f3402a050 tx=0x7f3f34009190 comp rx=0 tx=0).stop 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:55.389 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7201, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.390 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.391 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.392 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:55.393 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:55.394 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.395 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.396 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.397 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:55.398 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.399 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:55.400 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.401 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.402 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.403 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:55.404 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:55.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f2803e210 msgr2=0x7f3f280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3f2803e210 0x7f3f280406d0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c005c00 tx=0x7f3f3c009040 comp rx=0 tx=0).stop 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 10 monclient: shutdown 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 msgr2=0x7f3f441cd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 --2- 172.21.15.164:0/991681753 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3f4409c180 0x7f3f441cd0f0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000e9e0 tx=0x7f3f4000eea0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4a3c7640 1 -- 172.21.15.164:0/991681753 reap_dead start 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4a3c7640 1 -- 172.21.15.164:0/991681753 reap_dead start 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 shutdown_connections 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 >> 172.21.15.164:0/991681753 conn(0x7f3f440a3110 msgr2=0x7f3f440a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 shutdown_connections 2024-09-10T16:25:55.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.371+0000 7f3f4be2a640 1 -- 172.21.15.164:0/991681753 wait complete. 2024-09-10T16:25:55.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:55.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:25:55.407 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:25:55.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.505+0000 7fad87d3d640 10 monclient: get_monmap_and_config 2024-09-10T16:25:55.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.505+0000 7fad87d3d640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.505+0000 7fad87d3d640 10 monclient: monmap: 2024-09-10T16:25:55.506 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:55.506049+0000 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:55.506049+0000 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:55.507 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.505+0000 7fad87d3d640 1 Processor -- start 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 -- start start 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient: init 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 0x7fad80134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-b con 0x7fad8006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 0x7fad800a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-c con 0x7fad800a46a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fad80137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad8009ce50 con 0x7fad80137ca0 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fad8006db60 con 0x7fad800a46a0 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad8006cf50 con 0x7fad8006cb70 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad87d3d640 10 monclient(hunting): authenticate will time out at 1933.993555s 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 0x7fad80134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad862da640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fad80137ca0 0x7fad80138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55610/0 (socket says 172.21.15.164:55610) 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad862da640 1 -- 172.21.15.164:0/3386589517 learned_addr learned my addr 172.21.15.164:0/3386589517 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:55.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): get_auth_request con 0x7fad8006cb70 auth_method 0 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad852d8640 1 --2- 172.21.15.164:0/3386589517 >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 0x7fad800a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient(hunting): handle_auth_done global_id 214193 payload 274 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 -- 172.21.15.164:0/3386589517 >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 msgr2=0x7fad800a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 --2- 172.21.15.164:0/3386589517 >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 0x7fad800a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 -- 172.21.15.164:0/3386589517 >> v1:172.21.15.164:6789/0 conn(0x7fad80137ca0 legacy=0x7fad80138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 monclient: found mon.noname-b 2024-09-10T16:25:55.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad85ad9640 1 -- 172.21.15.164:0/3386589517 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad8009cb70 con 0x7fad8006cb70 2024-09-10T16:25:55.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad852d8640 10 monclient: get_auth_request con 0x7fad800a46a0 auth_method 0 2024-09-10T16:25:55.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.506+0000 7fad852d8640 1 --2- 172.21.15.164:0/3386589517 >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 0x7fad800a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:55.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad85ad9640 1 -- 172.21.15.164:0/3386589517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 msgr2=0x7fad80134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad85ad9640 1 --2- 172.21.15.164:0/3386589517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 0x7fad80134f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0097a0 tx=0x7fad7c029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f539a0775fe171b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 1 -- 172.21.15.164:0/3386589517 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c037070 con 0x7fad8006cb70 2024-09-10T16:25:55.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 10 monclient: dump: 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad87d3d640 5 monclient: authenticate success, global_id 214193 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad84ad7640 1 -- 172.21.15.164:0/3386589517 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7c032640 con 0x7fad8006cb70 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.509+0000 7fad87d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad84ad7640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad84ad7640 1 -- 172.21.15.164:0/3386589517 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c032a10 con 0x7fad8006cb70 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad84ad7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 4 set_mon_vals no callback set 2024-09-10T16:25:55.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad84ad7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad84ad7640 10 monclient: dump: 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 10 monclient: shutdown 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 -- 172.21.15.164:0/3386589517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 msgr2=0x7fad80134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3386589517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 0x7fad80134f70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fad7c0097a0 tx=0x7fad7c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 -- 172.21.15.164:0/3386589517 shutdown_connections 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3386589517 >> v2:172.21.15.165:3300/0 conn(0x7fad800a46a0 0x7fad800a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3386589517 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad8006cb70 0x7fad80134f70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 -- 172.21.15.164:0/3386589517 >> 172.21.15.164:0/3386589517 conn(0x7fad800aa900 msgr2=0x7fad800aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 -- 172.21.15.164:0/3386589517 shutdown_connections 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.510+0000 7fad87d3d640 1 -- 172.21.15.164:0/3386589517 wait complete. 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient: monmap: 2024-09-10T16:25:55.524 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 1 Processor -- start 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 1 -- start start 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient: init 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 0x7fad800a27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-a con 0x7fad8006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.511+0000 7fad87d3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 0x7fad800a2d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-c con 0x7fad800a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad80137ca0 0x7fad800a3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): picked mon.noname-b con 0x7fad80137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fad801389c0 con 0x7fad8006cb70 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fad80138840 con 0x7fad80137ca0 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad80140040 con 0x7fad800a46a0 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad87d3d640 10 monclient(hunting): authenticate will time out at 1933.999032s 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 0x7fad800a27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 0x7fad800a2d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 0x7fad800a27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39336/0 (socket says 172.21.15.164:39336) 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 -- 172.21.15.164:0/3867676303 learned_addr learned my addr 172.21.15.164:0/3867676303 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): get_auth_request con 0x7fad8006cb70 auth_method 0 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 0x7fad800a2d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34430/0 (socket says 172.21.15.164:34430) 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): get_auth_request con 0x7fad800a46a0 auth_method 0 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient(hunting): handle_auth_done global_id 224109 payload 824 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad862da640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad80137ca0 0x7fad800a3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 msgr2=0x7fad800a2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 0x7fad800a2d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad80137ca0 msgr2=0x7fad800a3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fad80137ca0 0x7fad800a3240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 monclient: found mon.noname-a 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad85ad9640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad801390a0 con 0x7fad8006cb70 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.512+0000 7fad852d8640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad800a46a0 0x7fad800a2d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad85ad9640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 msgr2=0x7fad800a27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad85ad9640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 0x7fad800a27c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fad7c000c00 tx=0x7fad7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c037040 con 0x7fad8006cb70 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: dump: 2024-09-10T16:25:55.529 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad7c042070 con 0x7fad8006cb70 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 5 monclient: authenticate success, global_id 224109 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 10 monclient: _renew_subs 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad8013a760 con 0x7fad8006cb70 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7c030d40 con 0x7fad8006cb70 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad767fc640 10 monclient: dump: 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 10 monclient: _renew_subs 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad8013aa40 con 0x7fad8006cb70 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.514+0000 7fad77fff640 4 set_mon_vals no callback set 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad87d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 0x7fad801cd510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad80134f70 con 0x7fad801cd130 2024-09-10T16:25:55.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fad7c032500 con 0x7fad8006cb70 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 0x7fad801cd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient: get_auth_request con 0x7fad801cd130 auth_method 0 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.515+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad767fc640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad6403e240 0x7fad64040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad852d8640 10 monclient(hunting): handle_auth_done global_id 214199 payload 824 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad852d8640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fad800008d0 con 0x7fad801cd130 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(679..679 src has 494..679) ==== 4755+0+0 (secure 0 0 0) 0x7fad7c0735b0 con 0x7fad8006cb70 2024-09-10T16:25:55.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad862da640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad6403e240 0x7fad64040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.516+0000 7fad862da640 10 monclient: get_auth_request con 0x7fad6403e240 auth_method 0 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad852d8640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 msgr2=0x7fad801cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad852d8640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 0x7fad801cd510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad7000d590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70013850 con 0x7fad801cd130 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 10 monclient: dump: 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.534 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fad7000fe50 con 0x7fad801cd130 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 msgr2=0x7fad801cd510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 0x7fad801cd510 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad7000d590 comp rx=0 tx=0).stop 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 msgr2=0x7fad801cd510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad767fc640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cd130 0x7fad801cd510 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad7000d590 comp rx=0 tx=0).stop 2024-09-10T16:25:55.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.518+0000 7fad862da640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad6403e240 0x7fad64040700 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fad80139b40 tx=0x7fad78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad87d3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:55.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 0x7fad8009d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad87d3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fad8013a480 con 0x7fad801cda50 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 0x7fad8009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient: get_auth_request con 0x7fad801cda50 auth_method 0 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.524+0000 7fad852d8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.525+0000 7fad852d8640 10 monclient(hunting): handle_auth_done global_id 214202 payload 824 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.525+0000 7fad852d8640 1 -- 172.21.15.164:0/3867676303 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fad801cde60 con 0x7fad801cda50 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad852d8640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 msgr2=0x7fad8009d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad852d8640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 0x7fad8009d630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70020c70 con 0x7fad801cda50 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 10 monclient: dump: 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.539 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fad7001b400 con 0x7fad801cda50 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 msgr2=0x7fad8009d630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 0x7fad8009d630 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad700046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 msgr2=0x7fad8009d630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.527+0000 7fad767fc640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fad801cda50 0x7fad8009d630 secure :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fad8013ff10 tx=0x7fad700046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:55.540 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 55306, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:55.541 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:55.542 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.543 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:55.544 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.545 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:55.546 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.547 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.548 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:55.549 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.550 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.551 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.552 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:55.553 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:55.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad6403e240 msgr2=0x7fad64040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fad6403e240 0x7fad64040700 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fad80139b40 tx=0x7fad78009040 comp rx=0 tx=0).stop 2024-09-10T16:25:55.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 10 monclient: shutdown 2024-09-10T16:25:55.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 msgr2=0x7fad800a27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 --2- 172.21.15.164:0/3867676303 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fad8006cb70 0x7fad800a27c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fad7c000c00 tx=0x7fad7c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad862da640 1 -- 172.21.15.164:0/3867676303 reap_dead start 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad862da640 1 -- 172.21.15.164:0/3867676303 reap_dead start 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 shutdown_connections 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 >> 172.21.15.164:0/3867676303 conn(0x7fad800aa900 msgr2=0x7fad800a9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 shutdown_connections 2024-09-10T16:25:55.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.528+0000 7fad87d3d640 1 -- 172.21.15.164:0/3867676303 wait complete. 2024-09-10T16:25:55.555 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:55.556 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:25:55.556 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:25:55.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 10 monclient: get_monmap_and_config 2024-09-10T16:25:55.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 10 monclient: monmap: 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:55.656062+0000 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:55.656062+0000 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:55.657 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 1 Processor -- start 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 1 -- start start 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.655+0000 7f51f1964640 10 monclient: init 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51ec06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f51ec134f70 0x7f51ec13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-c con 0x7f51ec134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51ec13d560 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51ec09ce50 con 0x7f51ec13d560 2024-09-10T16:25:55.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51ec135380 con 0x7f51ec06cb70 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f51ec06db60 con 0x7f51ec134f70 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51f1964640 10 monclient(hunting): authenticate will time out at 1934.143413s 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51eb7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f51ec13d560 0x7f51ec13f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55626/0 (socket says 172.21.15.164:55626) 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51eb7fe640 1 -- 172.21.15.164:0/1666305873 learned_addr learned my addr 172.21.15.164:0/1666305873 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 1 --2- 172.21.15.164:0/1666305873 >> v2:172.21.15.165:3300/0 conn(0x7f51ec134f70 0x7f51ec13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): get_auth_request con 0x7f51ec134f70 auth_method 0 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_done global_id 154651 payload 274 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1666305873 >> v1:172.21.15.165:6789/0 conn(0x7f51ec06cb70 legacy=0x7f51ec0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1666305873 >> v1:172.21.15.164:6789/0 conn(0x7f51ec13d560 legacy=0x7f51ec13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 1 monclient: found mon.noname-c 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:25:55.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.656+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1666305873 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec09cb70 con 0x7f51ec134f70 2024-09-10T16:25:55.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.659+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1666305873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec134f70 msgr2=0x7f51ec13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.659+0000 7f51ea7fc640 1 --2- 172.21.15.164:0/1666305873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec134f70 0x7f51ec13d020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097d0 tx=0x7f51d402a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df67b1c223f35f5f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.659+0000 7f51e9ffb640 1 -- 172.21.15.164:0/1666305873 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4037070 con 0x7f51ec134f70 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.659+0000 7f51e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: dump: 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.673 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 1 -- 172.21.15.164:0/1666305873 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d4032440 con 0x7f51ec134f70 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 5 monclient: authenticate success, global_id 154651 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 1 -- 172.21.15.164:0/1666305873 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4032810 con 0x7f51ec134f70 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 4 set_mon_vals no callback set 2024-09-10T16:25:55.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51e9ffb640 10 monclient: dump: 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 10 monclient: shutdown 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 -- 172.21.15.164:0/1666305873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec134f70 msgr2=0x7f51ec13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 --2- 172.21.15.164:0/1666305873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec134f70 0x7f51ec13d020 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f51d40097d0 tx=0x7f51d402a000 comp rx=0 tx=0).stop 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 -- 172.21.15.164:0/1666305873 shutdown_connections 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 --2- 172.21.15.164:0/1666305873 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec134f70 0x7f51ec13d020 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 -- 172.21.15.164:0/1666305873 >> 172.21.15.164:0/1666305873 conn(0x7f51ec0aa900 msgr2=0x7f51ec0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 -- 172.21.15.164:0/1666305873 shutdown_connections 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.660+0000 7f51f1964640 1 -- 172.21.15.164:0/1666305873 wait complete. 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.661+0000 7f51f1964640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.661+0000 7f51f1964640 10 monclient: monmap: 2024-09-10T16:25:55.676 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.661+0000 7f51f1964640 1 Processor -- start 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.661+0000 7f51f1964640 1 -- start start 2024-09-10T16:25:55.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.661+0000 7f51f1964640 10 monclient: init 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec06cb70 0x7f51ec131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-b con 0x7f51ec06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-a con 0x7f51ec134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51ec13d560 0x7f51ec13a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): picked mon.noname-c con 0x7f51ec13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec0a4db0 con 0x7f51ec134f70 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec06cf50 con 0x7f51ec06cb70 2024-09-10T16:25:55.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f51ec141070 con 0x7f51ec13d560 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51f1964640 10 monclient(hunting): authenticate will time out at 1934.149092s 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39348/0 (socket says 172.21.15.164:39348) 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1047087038 learned_addr learned my addr 172.21.15.164:0/1047087038 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): get_auth_request con 0x7f51ec134f70 auth_method 0 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51ec13d560 0x7f51ec13a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): get_auth_request con 0x7f51ec13d560 auth_method 0 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec06cb70 0x7f51ec131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): get_auth_request con 0x7f51ec06cb70 auth_method 0 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient(hunting): handle_auth_done global_id 224112 payload 824 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51ec13d560 msgr2=0x7f51ec13a560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f51ec13d560 0x7f51ec13a560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec06cb70 msgr2=0x7f51ec131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec06cb70 0x7f51ec131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 monclient: found mon.noname-a 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.662+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51ec09ef10 con 0x7f51ec134f70 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51ea7fc640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 msgr2=0x7f51ec13a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51ea7fc640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009720 tx=0x7f51d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4041070 con 0x7f51ec134f70 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 10 monclient: dump: 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.682 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 5 monclient: authenticate success, global_id 224112 2024-09-10T16:25:55.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 10 monclient: _renew_subs 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51ec09e270 con 0x7f51ec134f70 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 10 monclient: _renew_subs 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.664+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51ec13b9a0 con 0x7f51ec134f70 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51d403c670 con 0x7f51ec134f70 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51d4037040 con 0x7f51ec134f70 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 10 monclient: dump: 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.684 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e97fa640 4 set_mon_vals no callback set 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51f1964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51f1964640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 0x7f51ec13aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec09def0 con 0x7f51ec131b50 2024-09-10T16:25:55.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.665+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f51d4044070 con 0x7f51ec134f70 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51e3fff640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f51cc03e210 0x7f51cc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 0x7f51ec13aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(679..679 src has 494..679) ==== 4755+0+0 (secure 0 0 0) 0x7f51d40733f0 con 0x7f51ec134f70 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient: get_auth_request con 0x7f51ec131b50 auth_method 0 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eb7fe640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f51cc03e210 0x7f51cc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eb7fe640 10 monclient: get_auth_request con 0x7f51cc03e210 auth_method 0 2024-09-10T16:25:55.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 10 monclient(hunting): handle_auth_done global_id 154657 payload 824 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.666+0000 7f51eaffd640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51ec0008d0 con 0x7f51ec131b50 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.668+0000 7f51eaffd640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 msgr2=0x7f51ec13aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.668+0000 7f51eaffd640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 0x7f51ec13aaa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00bea0 tx=0x7f51dc00bf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.668+0000 7f51eb7fe640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f51cc03e210 0x7f51cc0406d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f51e4007c90 tx=0x7f51e4007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc003970 con 0x7f51ec131b50 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 10 monclient: dump: 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.687 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51dc003970 con 0x7f51ec131b50 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 msgr2=0x7f51ec13aaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 0x7f51ec13aaa0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00bea0 tx=0x7f51dc00bf90 comp rx=0 tx=0).stop 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 msgr2=0x7f51ec13aaa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.669+0000 7f51e3fff640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec131b50 0x7f51ec13aaa0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51f1964640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:55.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51f1964640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 0x7f51ec135870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51f1964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f51ec13b3f0 con 0x7f51ec13afe0 2024-09-10T16:25:55.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 0x7f51ec135870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient: get_auth_request con 0x7f51ec13afe0 auth_method 0 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.675+0000 7f51eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.676+0000 7f51eaffd640 10 monclient(hunting): handle_auth_done global_id 154660 payload 824 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.676+0000 7f51eaffd640 1 -- 172.21.15.164:0/1047087038 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51ec0770f0 con 0x7f51ec13afe0 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51eaffd640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 msgr2=0x7f51ec135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51eaffd640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 0x7f51ec135870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00bea0 tx=0x7f51dc004a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51dc014420 con 0x7f51ec13afe0 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51e3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.678+0000 7f51e3fff640 10 monclient: dump: 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f51dc014420 con 0x7f51ec13afe0 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 msgr2=0x7f51ec135870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 0x7f51ec135870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00bea0 tx=0x7f51dc004a70 comp rx=0 tx=0).stop 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 msgr2=0x7f51ec135870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51e3fff640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f51ec13afe0 0x7f51ec135870 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f51dc00bea0 tx=0x7f51dc004a70 comp rx=0 tx=0).stop 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:55.693 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 374229, 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.694 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.695 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:55.696 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:55.697 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:55.698 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.699 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:55.700 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.701 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:55.702 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.703 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.704 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:25:55.705 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:55.706 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f51cc03e210 msgr2=0x7f51cc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f51cc03e210 0x7f51cc0406d0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f51e4007c90 tx=0x7f51e4007be0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 10 monclient: shutdown 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 msgr2=0x7f51ec13a020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f51d4009720 tx=0x7f51d40046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51eb7fe640 1 -- 172.21.15.164:0/1047087038 reap_dead start 2024-09-10T16:25:55.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 shutdown_connections 2024-09-10T16:25:55.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 --2- 172.21.15.164:0/1047087038 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f51ec134f70 0x7f51ec13a020 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 >> 172.21.15.164:0/1047087038 conn(0x7f51ec0aa900 msgr2=0x7f51ec134070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 shutdown_connections 2024-09-10T16:25:55.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.679+0000 7f51f1964640 1 -- 172.21.15.164:0/1047087038 wait complete. 2024-09-10T16:25:55.709 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:55.709 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:25:55.709 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:25:55.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.806+0000 7f8e72a62640 10 monclient: get_monmap_and_config 2024-09-10T16:25:55.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.806+0000 7f8e72a62640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.807+0000 7f8e72a62640 10 monclient: monmap: 2024-09-10T16:25:55.808 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:55.807869+0000 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:55.807869+0000 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:55.809 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.807+0000 7f8e72a62640 1 Processor -- start 2024-09-10T16:25:55.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.807+0000 7f8e72a62640 1 -- start start 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.807+0000 7f8e72a62640 10 monclient: init 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 0x7f8e6c09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8e6c09ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8e6c09e250 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c0a0870 con 0x7f8e6c09e250 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e6c1353a0 con 0x7f8e6c09ab50 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e6c09d3e0 con 0x7f8e6c09d930 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e72a62640 10 monclient(hunting): authenticate will time out at 1934.295424s 2024-09-10T16:25:55.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 0x7f8e6c09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e70fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f8e6c09e250 0x7f8e6c134c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55638/0 (socket says 172.21.15.164:55638) 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e70fff640 1 -- 172.21.15.164:0/4034945063 learned_addr learned my addr 172.21.15.164:0/4034945063 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): get_auth_request con 0x7f8e6c09d930 auth_method 0 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_done global_id 214208 payload 274 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4034945063 >> v1:172.21.15.165:6789/0 conn(0x7f8e6c09ab50 legacy=0x7f8e6c134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4034945063 >> v1:172.21.15.164:6789/0 conn(0x7f8e6c09e250 legacy=0x7f8e6c134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:55.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 1 monclient: found mon.noname-b 2024-09-10T16:25:55.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.808+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4034945063 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c074cc0 con 0x7f8e6c09d930 2024-09-10T16:25:55.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4034945063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 msgr2=0x7f8e6c09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6bfff640 1 --2- 172.21.15.164:0/4034945063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 0x7f8e6c09dd10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00a970 tx=0x7f8e5c02b3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3e11129aadafeca server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 1 -- 172.21.15.164:0/4034945063 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c00b040 con 0x7f8e6c09d930 2024-09-10T16:25:55.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: dump: 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e72a62640 5 monclient: authenticate success, global_id 214208 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e72a62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 1 -- 172.21.15.164:0/4034945063 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5c030a00 con 0x7f8e6c09d930 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e72a62640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e72a62640 4 set_mon_vals no callback set 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 1 -- 172.21.15.164:0/4034945063 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c030dd0 con 0x7f8e6c09d930 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.811+0000 7f8e6affd640 10 monclient: dump: 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 10 monclient: shutdown 2024-09-10T16:25:55.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 -- 172.21.15.164:0/4034945063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 msgr2=0x7f8e6c09dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4034945063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 0x7f8e6c09dd10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00a970 tx=0x7f8e5c02b3e0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 -- 172.21.15.164:0/4034945063 shutdown_connections 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4034945063 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09d930 0x7f8e6c09dd10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 -- 172.21.15.164:0/4034945063 >> 172.21.15.164:0/4034945063 conn(0x7f8e6c0a3110 msgr2=0x7f8e6c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 -- 172.21.15.164:0/4034945063 shutdown_connections 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.812+0000 7f8e72a62640 1 -- 172.21.15.164:0/4034945063 wait complete. 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.813+0000 7f8e72a62640 10 monclient: build_initial_monmap 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.813+0000 7f8e72a62640 10 monclient: monmap: 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:55.821 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.813+0000 7f8e72a62640 1 Processor -- start 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.813+0000 7f8e72a62640 1 -- start start 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.813+0000 7f8e72a62640 10 monclient: init 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-c con 0x7f8e6c09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6c09d930 0x7f8e6c12fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-a con 0x7f8e6c09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 0x7f8e6c1300a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): picked mon.noname-b con 0x7f8e6c09e250 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c09e6e0 con 0x7f8e6c09d930 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8e6c0a02b0 con 0x7f8e6c09e250 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8e6c09ef30 con 0x7f8e6c09ab50 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e72a62640 10 monclient(hunting): authenticate will time out at 1934.301246s 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6c09d930 0x7f8e6c12fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34450/0 (socket says 172.21.15.164:34450) 2024-09-10T16:25:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4147165116 learned_addr learned my addr 172.21.15.164:0/4147165116 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): get_auth_request con 0x7f8e6c09ab50 auth_method 0 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): get_auth_request con 0x7f8e6c09d930 auth_method 0 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e70fff640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 0x7f8e6c1300a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient(hunting): handle_auth_done global_id 214211 payload 824 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 msgr2=0x7f8e6c1300a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 0x7f8e6c1300a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6c09d930 msgr2=0x7f8e6c12fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6c09d930 0x7f8e6c12fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 monclient: found mon.noname-c 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4147165116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6c132f90 con 0x7f8e6c09ab50 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e70fff640 10 monclient: get_auth_request con 0x7f8e6c09e250 auth_method 0 2024-09-10T16:25:55.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.814+0000 7f8e70fff640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 0x7f8e6c1300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:55.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e6bfff640 1 -- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 msgr2=0x7f8e6c12f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:55.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e6bfff640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00a970 tx=0x7f8e5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c041070 con 0x7f8e6c09ab50 2024-09-10T16:25:55.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 10 monclient: dump: 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 10 monclient: _finish_auth 0 2024-09-10T16:25:55.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e5c03b430 con 0x7f8e6c09ab50 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e72a62640 5 monclient: authenticate success, global_id 214211 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e72a62640 10 monclient: _renew_subs 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e72a62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.816+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6c133270 con 0x7f8e6c09ab50 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e5c03b820 con 0x7f8e6c09ab50 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 10 monclient: dump: 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:55.829 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 10 monclient: _renew_subs 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e6c132710 con 0x7f8e6c09ab50 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e6a7fc640 4 set_mon_vals no callback set 2024-09-10T16:25:55.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8e5c044070 con 0x7f8e6c09ab50 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.817+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e6c09d620 con 0x7f8e6c09ab50 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.818+0000 7f8e68ff9640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 0x7f8e580406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.818+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(679..679 src has 494..679) ==== 4755+0+0 (secure 0 0 0) 0x7f8e5c073af0 con 0x7f8e6c09ab50 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.818+0000 7f8e6b7fe640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 0x7f8e580406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.818+0000 7f8e6b7fe640 10 monclient: get_auth_request con 0x7f8e5803e210 auth_method 0 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.821+0000 7f8e6b7fe640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 0x7f8e580406d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c09f2b0 tx=0x7f8e60007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.822+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8e5c072500 con 0x7f8e6c09ab50 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.822+0000 7f8e68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:55.822+0000 7f8e68ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:56.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.062+0000 7f8e72a62640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:25:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.062+0000 7f8e72a62640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:25:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.062+0000 7f8e72a62640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.062+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8e6c0a2a50 con 0x7f8e6c09ab50 2024-09-10T16:25:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.063+0000 7f8e68ff9640 1 -- 172.21.15.164:0/4147165116 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8e5c045dc0 con 0x7f8e6c09ab50 2024-09-10T16:25:56.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.063+0000 7f8e68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:25:56.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.063+0000 7f8e68ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:56.065 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.065 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=270388282,config=2586702516,health=3918307125,kv=3185102672,logm=3330321513}) 2024-09-10T16:25:56.065 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=10,logm=51} crc {auth=2901771645,config=2586702516,health=4045999481,kv=3502268455,logm=2455298823}) 2024-09-10T16:25:56.065 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=270388282,config=2586702516,health=3918307125,kv=3185102672,logm=3330321513}) 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.064+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=1464484872,config=2586702516,health=1602468074,kv=805391194,logm=3468684723}) 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827801201}) 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1986457479}) 2024-09-10T16:25:56.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2827801201}) 2024-09-10T16:25:56.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.065+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2836898997}) 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 msgr2=0x7f8e580406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 0x7f8e580406d0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c09f2b0 tx=0x7f8e60007e30 comp rx=0 tx=0).stop 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 10 monclient: shutdown 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 msgr2=0x7f8e6c12f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8e5c00a970 tx=0x7f8e5c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 shutdown_connections 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8e5803e210 0x7f8e580406d0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:56.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8e6c09e250 0x7f8e6c1300a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:56.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8e6c09d930 0x7f8e6c12fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:56.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 --2- 172.21.15.164:0/4147165116 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8e6c09ab50 0x7f8e6c12f620 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:56.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 >> 172.21.15.164:0/4147165116 conn(0x7f8e6c0a3110 msgr2=0x7f8e6c09b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:56.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 shutdown_connections 2024-09-10T16:25:56.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:56.066+0000 7f8e72a62640 1 -- 172.21.15.164:0/4147165116 wait complete. 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131801595}) 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960365266}) 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1131801595}) 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.066+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2630745731}) 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.072 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551848886}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=109039700}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1551848886}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.067+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3588666949}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617890817}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466210468}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=617890817}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.068+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3752896657}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907514139}) 2024-09-10T16:25:56.073 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=237696772}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907514139}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.069+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182456485}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012000345}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3312678307}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1012000345}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.070+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785657368}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252166427}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1105246188}) 2024-09-10T16:25:56.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252166427}) 2024-09-10T16:25:56.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=246627197}) 2024-09-10T16:25:56.077 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.077 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429229066}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4208934028}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429229066}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.071+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1951080261}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090826958}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2513807016}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2090826958}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.072+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2368344100}) 2024-09-10T16:25:56.078 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974215391}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4259952332}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2974215391}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715892507}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170337570}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1951582482}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170337570}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.073+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528312209}) 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.079 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102324985}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3176779330}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1102324985}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.074+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926451019}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497967086}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309916813}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3497967086}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2761944596}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093333403}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=717408120}) 2024-09-10T16:25:56.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1093333403}) 2024-09-10T16:25:56.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.075+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=730224927}) 2024-09-10T16:25:56.081 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:25:56.084 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.084 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117326695}) 2024-09-10T16:25:56.084 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529540543}) 2024-09-10T16:25:56.084 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2117326695}) 2024-09-10T16:25:56.085 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.083+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1125960599}) 2024-09-10T16:25:56.086 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704655517}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2885853162}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=704655517}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2798888427}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219220679}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2649877667}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2219220679}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.084+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3322549899}) 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502514228}) 2024-09-10T16:25:56.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2082261810}) 2024-09-10T16:25:56.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1502514228}) 2024-09-10T16:25:56.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.085+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3047379697}) 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747233803}) 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2528748775}) 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2747233803}) 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.086+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239853882}) 2024-09-10T16:25:56.091 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577363400}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075249337}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=577363400}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1062067564}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108706868}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2279847585}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108706868}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.087+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1670883363}) 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.092 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031299428}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3926246113}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4031299428}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.088+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967201657}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481322049}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2485391214}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2481322049}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3288764753}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285382000}) 2024-09-10T16:25:56.093 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1500705105}) 2024-09-10T16:25:56.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285382000}) 2024-09-10T16:25:56.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.089+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2612553509}) 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188358952}) 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2080896229}) 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3188358952}) 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.090+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=111823964}) 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181087921}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2601485638}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1181087921}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.091+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893270712}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517557057}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1645575144}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1517557057}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515921936}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247808253}) 2024-09-10T16:25:56.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=15943022}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3247808253}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.092+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1778884048}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853080045}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=846034353}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853080045}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.093+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3250505031}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784730851}) 2024-09-10T16:25:56.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2501210944}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1784730851}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.094+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=182182869}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222171970}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3649230400}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3222171970}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1403969336}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644951977}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2196099902}) 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.102 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.095+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2644951977}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713614316}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429851680}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925448802}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1429851680}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.096+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2559155540}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486006413}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2312984392}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1486006413}) 2024-09-10T16:25:56.103 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.097+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1815899870}) 2024-09-10T16:25:56.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188429110}) 2024-09-10T16:25:56.113 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1975651433}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2188429110}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4007428023}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141396406}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2829087108}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=141396406}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1356057499}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149636599}) 2024-09-10T16:25:56.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1832450809}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1149636599}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1812555571}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285525478}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2102121499}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3285525478}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2421686332}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193853770}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011515912}) 2024-09-10T16:25:56.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=193853770}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=479971868}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552372207}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2708282938}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552372207}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2268368632}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469039210}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=881222519}) 2024-09-10T16:25:56.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3469039210}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2048352903}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156861782}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=46530194}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156861782}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988046087}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793560527}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=476792108}) 2024-09-10T16:25:56.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=793560527}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1280086113}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204058667}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1596161267}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4204058667}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958389069}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007180833}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1158512300}) 2024-09-10T16:25:56.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2007180833}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.106+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3068504359}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264810167}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1692091807}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1264810167}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=194768301}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828973686}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4018159781}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2828973686}) 2024-09-10T16:25:56.119 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.107+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=126605180}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388863060}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3337244268}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388863060}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.108+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3426870481}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596147385}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2356826810}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3596147385}) 2024-09-10T16:25:56.120 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3877530876}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722355464}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=42302981}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1722355464}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.109+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3244363022}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408917907}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740593936}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=408917907}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.110+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3962343890}) 2024-09-10T16:25:56.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752915128}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892556911}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752915128}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.111+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=277637422}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793211340}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880507492}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2793211340}) 2024-09-10T16:25:56.122 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4271910032}) 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875753320}) 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925232347}) 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2875753320}) 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.112+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=363474243}) 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490275629}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1360277702}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1490275629}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.113+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=362197129}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507908256}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3788184767}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3507908256}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.114+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3455609576}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772443093}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2497367519}) 2024-09-10T16:25:56.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=772443093}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1533382931}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.115+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832073546}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491521612}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=832073546}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=144371234}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679044606}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=732165327}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2679044606}) 2024-09-10T16:25:56.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.116+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1927792215}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976194568}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1824973845}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2976194568}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300616039}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.117+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706967944}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3808774103}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=706967944}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=955425417}) 2024-09-10T16:25:56.137 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.118+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854324181}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=485372748}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2854324181}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1926039210}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939642705}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453793158}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1939642705}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.119+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618546496}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036855726}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3212955437}) 2024-09-10T16:25:56.138 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036855726}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4178880935}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.120+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370202999}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1761289985}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370202999}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224927620}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732378203}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1817150064}) 2024-09-10T16:25:56.139 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2732378203}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.121+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=625700293}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774862132}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3850431783}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3774862132}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.122+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940261695}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062427451}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3186316762}) 2024-09-10T16:25:56.140 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3062427451}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=573919934}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370883243}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3844820475}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370883243}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.123+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213429968}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=15} crc {logm=1095913654,mdsmap=204415920,mgr=1154067016,mgr_metadata=2169216343,mgrstat=3023926736,mon_config_key=686230354}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=59,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=15} crc {logm=703897957,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=789928432,mon_config_key=2475304868}) 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.141 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=62,mdsmap=2,mgr=8,mgr_metadata=1,mgrstat=12,mon_config_key=15} crc {logm=1095913654,mdsmap=204415920,mgr=1154067016,mgr_metadata=2169216343,mgrstat=3023926736,mon_config_key=686230354}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.124+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=60,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=13,mon_config_key=14} crc {logm=4233970997,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=789928432,mon_config_key=3849472225}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=86} crc {mon_config_key=559485441,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=407488823,osd_snap=2862877126}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=1,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=86} crc {mon_config_key=559485441,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=407488823,osd_snap=1547784037}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=1,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=86} crc {mon_config_key=559485441,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=407488823,osd_snap=2862877126}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.125+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=1,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=87} crc {mon_config_key=559485441,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=4005186995,osd_pg_creating=407488823,osd_snap=1973909146}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=3341583502,osdmap=254780060}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=36,osdmap=64} crc {osd_snap=2152863515,osdmap=3605091728}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=35,osdmap=65} crc {osd_snap=3341583502,osdmap=254780060}) 2024-09-10T16:25:56.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.126+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=38,osdmap=62} crc {osd_snap=907092671,osdmap=3934126096}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3261440104}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1354084259}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3261440104}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.127+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=91795561}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614786030}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=204034250}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2614786030}) 2024-09-10T16:25:56.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.128+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2131527391}) 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=322655647}) 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=1529469013}) 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=20} crc {osdmap=322655647}) 2024-09-10T16:25:56.144 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:25:56.132+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=19} crc {osdmap=1622449008}) 2024-09-10T16:25:56.245 INFO:tasks.rados.rados.0.smithi165.stdout:3031: done (0 left) 2024-09-10T16:25:56.245 INFO:tasks.rados.rados.0.smithi165.stdout:3032: snap_remove snap 313 2024-09-10T16:25:57.082 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:25:57.083 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 10 monclient: get_monmap_and_config 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 10 monclient: monmap: 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.191 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:57.190910+0000 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:57.190910+0000 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:57.192 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 1 Processor -- start 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 1 -- start start 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.190+0000 7f0aa0ec6640 10 monclient: init 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a9c09cb10 0x7f0a9c09cef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c09cb10 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c138480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c138040 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a9c1346f0 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:57.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a9c09d430 con 0x7f0a9c1346f0 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0a9c0a2820 con 0x7f0a9c09cb10 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c074a70 con 0x7f0a9c138040 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0aa0ec6640 10 monclient(hunting): authenticate will time out at 1935.678452s 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a9c09cb10 0x7f0a9c09cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c138480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9ad9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0a9c1346f0 0x7f0a9c134ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55646/0 (socket says 172.21.15.164:55646) 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9ad9d640 1 -- 172.21.15.164:0/1324289119 learned_addr learned my addr 172.21.15.164:0/1324289119 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): get_auth_request con 0x7f0a9c09cb10 auth_method 0 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): get_auth_request con 0x7f0a9c138040 auth_method 0 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient(hunting): handle_auth_done global_id 224118 payload 274 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 -- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 msgr2=0x7f0a9c138480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c138480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 -- 172.21.15.164:0/1324289119 >> v1:172.21.15.164:6789/0 conn(0x7f0a9c1346f0 legacy=0x7f0a9c134ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 monclient: found mon.noname-a 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a9a59c640 1 -- 172.21.15.164:0/1324289119 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c074d50 con 0x7f0a9c09cb10 2024-09-10T16:25:57.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.191+0000 7f0a99d9b640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c138480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:57.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9a59c640 1 -- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c09cb10 msgr2=0x7f0a9c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9a59c640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097a0 tx=0x7f0a8c02a050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3844d65b7da9bc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 1 -- 172.21.15.164:0/1324289119 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c037070 con 0x7f0a9c09cb10 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: dump: 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 1 -- 172.21.15.164:0/1324289119 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a8c032440 con 0x7f0a9c09cb10 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0aa0ec6640 5 monclient: authenticate success, global_id 224118 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0aa0ec6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0aa0ec6640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0aa0ec6640 4 set_mon_vals no callback set 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 1 -- 172.21.15.164:0/1324289119 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a8c032810 con 0x7f0a9c09cb10 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.194+0000 7f0a9959a640 10 monclient: dump: 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 10 monclient: shutdown 2024-09-10T16:25:57.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c09cb10 msgr2=0x7f0a9c09cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09cef0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c0097a0 tx=0x7f0a8c02a050 comp rx=0 tx=0).stop 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1324289119 shutdown_connections 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c138480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1324289119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09cef0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1324289119 >> 172.21.15.164:0/1324289119 conn(0x7f0a9c0a3110 msgr2=0x7f0a9c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1324289119 shutdown_connections 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.195+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1324289119 wait complete. 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.196+0000 7f0aa0ec6640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.196+0000 7f0aa0ec6640 10 monclient: monmap: 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:57.207 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.196+0000 7f0aa0ec6640 1 Processor -- start 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.196+0000 7f0aa0ec6640 1 -- start start 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.196+0000 7f0aa0ec6640 10 monclient: init 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a9c09cb10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a9c1346f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c1cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a9c138040 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0a2230 con 0x7f0a9c1346f0 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0a9c0a20b0 con 0x7f0a9c09cb10 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0a9c1352b0 con 0x7f0a9c138040 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0aa0ec6640 10 monclient(hunting): authenticate will time out at 1935.684335s 2024-09-10T16:25:57.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c1cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39380/0 (socket says 172.21.15.164:39380) 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 -- 172.21.15.164:0/1949356308 learned_addr learned my addr 172.21.15.164:0/1949356308 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): get_auth_request con 0x7f0a9c1346f0 auth_method 0 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): get_auth_request con 0x7f0a9c138040 auth_method 0 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9a59c640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9a59c640 10 monclient(hunting): get_auth_request con 0x7f0a9c09cb10 auth_method 0 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient(hunting): handle_auth_done global_id 224121 payload 824 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 -- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 msgr2=0x7f0a9c1cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c1cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 -- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a9c09cb10 msgr2=0x7f0a9c09e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 monclient: found mon.noname-a 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a99d9b640 1 -- 172.21.15.164:0/1949356308 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a9c132280 con 0x7f0a9c1346f0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.197+0000 7f0a9ad9d640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c1cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a99d9b640 1 -- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 msgr2=0x7f0a9c09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a99d9b640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000ec10 tx=0x7f0a9000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90017070 con 0x7f0a9c1346f0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: dump: 2024-09-10T16:25:57.212 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 5 monclient: authenticate success, global_id 224121 2024-09-10T16:25:57.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a9000ce40 con 0x7f0a9c1346f0 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 10 monclient: _renew_subs 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a9c09f550 con 0x7f0a9c1346f0 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a90013690 con 0x7f0a9c1346f0 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a8affd640 10 monclient: dump: 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.214 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 10 monclient: _renew_subs 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0a98d99640 4 set_mon_vals no callback set 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.200+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a9c0a0890 con 0x7f0a9c1346f0 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0a9001a070 con 0x7f0a9c1346f0 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0a8affd640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 0x7f0a7c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0aa0ec6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0aa0ec6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0aa0ec6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a9c09f0f0 con 0x7f0a9c1346f0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(680..680 src has 494..680) ==== 4799+0+0 (secure 0 0 0) 0x7f0a90052d10 con 0x7f0a9c1346f0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0a9a59c640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 0x7f0a7c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.201+0000 7f0a9a59c640 10 monclient: get_auth_request con 0x7f0a7c03e210 auth_method 0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.204+0000 7f0a9a59c640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 0x7f0a7c0406d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c09a4c0 tx=0x7f0a8c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.206+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0a900123d0 con 0x7f0a9c1346f0 2024-09-10T16:25:57.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.206+0000 7f0a8affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.206+0000 7f0a8affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:3032: done (0 left) 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:3033: copy_from oid 30 from oid 19 current snap is 328 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:3034: rollback oid 35 current snap is 328 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 327 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:3035: write (excl) oid 34 current snap is 328 2024-09-10T16:25:57.247 INFO:tasks.rados.rados.0.smithi165.stdout:3035: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2024-09-10T16:25:57.254 INFO:tasks.rados.rados.0.smithi165.stdout:3035: writing smithi16534841-34 from 598613 to 1260371 tid 1 2024-09-10T16:25:57.261 INFO:tasks.rados.rados.0.smithi165.stdout:3035: writing smithi16534841-34 from 1970078 to 2535105 tid 2 2024-09-10T16:25:57.268 INFO:tasks.rados.rados.0.smithi165.stdout:3035: writing smithi16534841-34 from 3115067 to 3729658 tid 3 2024-09-10T16:25:57.268 INFO:tasks.rados.rados.0.smithi165.stdout:3036: snap_remove snap 322 2024-09-10T16:25:57.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0aa0ec6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0aa0ec6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0aa0ec6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0a9c09cef0 con 0x7f0a9c1346f0 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0a8affd640 1 -- 172.21.15.164:0/1949356308 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f0a900232f0 con 0x7f0a9c1346f0 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0a8affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.461+0000 7f0a8affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:57.463 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":192,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:25:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 msgr2=0x7f0a7c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 0x7f0a7c0406d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f0a9c09a4c0 tx=0x7f0a8c02e000 comp rx=0 tx=0).stop 2024-09-10T16:25:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 10 monclient: shutdown 2024-09-10T16:25:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 msgr2=0x7f0a9c09e5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f0a9000ec10 tx=0x7f0a9000c520 comp rx=0 tx=0).stop 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 shutdown_connections 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a7c03e210 0x7f0a7c0406d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a9c138040 0x7f0a9c1cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a9c1346f0 0x7f0a9c09e5d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 --2- 172.21.15.164:0/1949356308 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a9c09cb10 0x7f0a9c09e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.463+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 >> 172.21.15.164:0/1949356308 conn(0x7f0a9c0a3110 msgr2=0x7f0a9c133370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.464+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 shutdown_connections 2024-09-10T16:25:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.464+0000 7f0aa0ec6640 1 -- 172.21.15.164:0/1949356308 wait complete. 2024-09-10T16:25:57.475 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:25:57.476 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:25:57.476 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:25:57.476 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.572+0000 7fbdcf5a0640 10 monclient: get_monmap_and_config 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.572+0000 7fbdcf5a0640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.572+0000 7fbdcf5a0640 10 monclient: monmap: 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:57.573818+0000 2024-09-10T16:25:57.574 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:57.573818+0000 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:57.575 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.573+0000 7fbdcf5a0640 1 Processor -- start 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.573+0000 7fbdcf5a0640 1 -- start start 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.573+0000 7fbdcf5a0640 10 monclient: init 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 0x7fbdc80a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-c con 0x7fbdc809f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbdc80a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc80a27b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbdc809fd10 con 0x7fbdc80a2380 2024-09-10T16:25:57.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbdc80a2c60 con 0x7fbdc809f930 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc809cd20 con 0x7fbdc80a27b0 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcf5a0640 10 monclient(hunting): authenticate will time out at 1936.061311s 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdccb3b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbdc80a2380 0x7fbdc813a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55648/0 (socket says 172.21.15.164:55648) 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdccb3b640 1 -- 172.21.15.164:0/1632324923 learned_addr learned my addr 172.21.15.164:0/1632324923 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34476/0 (socket says 172.21.15.164:34476) 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): get_auth_request con 0x7fbdc80a27b0 auth_method 0 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_done global_id 214220 payload 274 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/1632324923 >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 msgr2=0x7fbdc80a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcd33c640 1 --2- 172.21.15.164:0/1632324923 >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 0x7fbdc80a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/1632324923 >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 0x7fbdc80a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/1632324923 >> v1:172.21.15.164:6789/0 conn(0x7fbdc80a2380 legacy=0x7fbdc813a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 monclient: found mon.noname-b 2024-09-10T16:25:57.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/1632324923 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc809a0d0 con 0x7fbdc80a27b0 2024-09-10T16:25:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcd33c640 10 monclient: get_auth_request con 0x7fbdc809f930 auth_method 0 2024-09-10T16:25:57.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.574+0000 7fbdcd33c640 1 --2- 172.21.15.164:0/1632324923 >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 0x7fbdc80a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 msgr2=0x7fbdc813dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00097a0 tx=0x7fbdc0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4159b0fac97ea18a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 1 -- 172.21.15.164:0/1632324923 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0037070 con 0x7fbdc80a27b0 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: dump: 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdcf5a0640 5 monclient: authenticate success, global_id 214220 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 1 -- 172.21.15.164:0/1632324923 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdc0032640 con 0x7fbdc80a27b0 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 1 -- 172.21.15.164:0/1632324923 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdc0032a10 con 0x7fbdc80a27b0 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdcf5a0640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdcf5a0640 4 set_mon_vals no callback set 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.577+0000 7fbdc7fff640 10 monclient: dump: 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 10 monclient: shutdown 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 msgr2=0x7fbdc813dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fbdc00097a0 tx=0x7fbdc0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/1632324923 shutdown_connections 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/1632324923 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc80a27b0 0x7fbdc813dcf0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/1632324923 >> v2:172.21.15.165:3300/0 conn(0x7fbdc809f930 0x7fbdc80a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/1632324923 >> 172.21.15.164:0/1632324923 conn(0x7fbdc80a3110 msgr2=0x7fbdc80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/1632324923 shutdown_connections 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.578+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/1632324923 wait complete. 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient: monmap: 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:57.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 1 Processor -- start 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 1 -- start start 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient: init 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 0x7fbdc81c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-c con 0x7fbdc809f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.579+0000 7fbdcf5a0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc80a2380 0x7fbdc81baaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-a con 0x7fbdc80a2380 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdc80a27b0 0x7fbdc81cab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): picked mon.noname-b con 0x7fbdc80a27b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc809d930 con 0x7fbdc80a2380 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc809ee00 con 0x7fbdc80a27b0 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbdc809c8b0 con 0x7fbdc809f930 2024-09-10T16:25:57.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcf5a0640 10 monclient(hunting): authenticate will time out at 1936.067046s 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 0x7fbdc81c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 0x7fbdc81c7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34490/0 (socket says 172.21.15.164:34490) 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc80a2380 0x7fbdc81baaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 -- 172.21.15.164:0/2849711927 learned_addr learned my addr 172.21.15.164:0/2849711927 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): get_auth_request con 0x7fbdc809f930 auth_method 0 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): get_auth_request con 0x7fbdc80a2380 auth_method 0 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdc80a27b0 0x7fbdc81cab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): get_auth_request con 0x7fbdc80a27b0 auth_method 0 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcdb3d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient(hunting): handle_auth_done global_id 214223 payload 824 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdc80a27b0 msgr2=0x7fbdc81cab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbdc80a27b0 0x7fbdc81cab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc80a2380 msgr2=0x7fbdc81baaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc80a2380 0x7fbdc81baaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 monclient: found mon.noname-c 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.580+0000 7fbdcd33c640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc81b5250 con 0x7fbdc809f930 2024-09-10T16:25:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcd33c640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 msgr2=0x7fbdc81c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcd33c640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 0x7fbdc81c7670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800cf90 tx=0x7fbdb800a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb801a070 con 0x7fbdc809f930 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdb800da50 con 0x7fbdc809f930 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcf5a0640 5 monclient: authenticate success, global_id 214223 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcf5a0640 10 monclient: _renew_subs 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcf5a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc806ee20 con 0x7fbdc809f930 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdb800de20 con 0x7fbdc809f930 2024-09-10T16:25:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 10 monclient: _renew_subs 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc81b4f70 con 0x7fbdc809f930 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.582+0000 7fbdc77fe640 4 set_mon_vals no callback set 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 0x7fbdc81cb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc80a0760 con 0x7fbdc81bafe0 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbdb801d070 con 0x7fbdc809f930 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.583+0000 7fbdccb3b640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 0x7fbdc81cb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient: get_auth_request con 0x7fbdc81bafe0 auth_method 0 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdc5ffb640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbdb403e210 0x7fbdb40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(681..681 src has 494..681) ==== 4815+0+0 (secure 0 0 0) 0x7fbdb80535d0 con 0x7fbdc809f930 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_done global_id 224127 payload 824 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdccb3b640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbdc81b38b0 con 0x7fbdc81bafe0 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbdb403e210 0x7fbdb40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.584+0000 7fbdcdb3d640 10 monclient: get_auth_request con 0x7fbdb403e210 auth_method 0 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdccb3b640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 msgr2=0x7fbdc81cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdccb3b640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 0x7fbdc81cb0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00ea70 tx=0x7fbdbc00ef30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc01a070 con 0x7fbdc81bafe0 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbdbc00cc50 con 0x7fbdc81bafe0 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 msgr2=0x7fbdc81cb0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 0x7fbdc81cb0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00ea70 tx=0x7fbdbc00ef30 comp rx=0 tx=0).stop 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 msgr2=0x7fbdc81cb0d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdc5ffb640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81bafe0 0x7fbdc81cb0d0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc00ea70 tx=0x7fbdbc00ef30 comp rx=0 tx=0).stop 2024-09-10T16:25:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.586+0000 7fbdcdb3d640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbdb403e210 0x7fbdb40406d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbdc809f500 tx=0x7fbdc002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.592+0000 7fbdcf5a0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.592+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 0x7fbdc8072120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.592+0000 7fbdcf5a0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.592+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc81cb9f0 con 0x7fbdc81cb610 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.592+0000 7fbdccb3b640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 0x7fbdc8072120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient: get_auth_request con 0x7fbdc81cb610 auth_method 0 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 10 monclient(hunting): handle_auth_done global_id 224130 payload 824 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.593+0000 7fbdccb3b640 1 -- 172.21.15.164:0/2849711927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbdc8077260 con 0x7fbdc81cb610 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdccb3b640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 msgr2=0x7fbdc8072120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdccb3b640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 0x7fbdc8072120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc017000 tx=0x7fbdbc004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdbc01a070 con 0x7fbdc81cb610 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 10 monclient: dump: 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7fbdbc01b040 con 0x7fbdc81cb610 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 msgr2=0x7fbdc8072120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 0x7fbdc8072120 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc017000 tx=0x7fbdbc004940 comp rx=0 tx=0).stop 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 msgr2=0x7fbdc8072120 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.595+0000 7fbdc5ffb640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbdc81cb610 0x7fbdc8072120 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbdbc017000 tx=0x7fbdbc004940 comp rx=0 tx=0).stop 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 9426, 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:57.609 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.610 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.611 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.612 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.613 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.614 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.617 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.618 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.620 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.621 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:57.621 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:57.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbdb403e210 msgr2=0x7fbdb40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbdb403e210 0x7fbdb40406d0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fbdc809f500 tx=0x7fbdc002e000 comp rx=0 tx=0).stop 2024-09-10T16:25:57.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 10 monclient: shutdown 2024-09-10T16:25:57.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 msgr2=0x7fbdc81c7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 --2- 172.21.15.164:0/2849711927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbdc809f930 0x7fbdc81c7670 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbdb800cf90 tx=0x7fbdb800a7d0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/2849711927 reap_dead start 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcdb3d640 1 -- 172.21.15.164:0/2849711927 reap_dead start 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 shutdown_connections 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 >> 172.21.15.164:0/2849711927 conn(0x7fbdc80a3110 msgr2=0x7fbdc81bb4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 shutdown_connections 2024-09-10T16:25:57.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.596+0000 7fbdcf5a0640 1 -- 172.21.15.164:0/2849711927 wait complete. 2024-09-10T16:25:57.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:57.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:25:57.624 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.721+0000 7efe0c447640 10 monclient: get_monmap_and_config 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.721+0000 7efe0c447640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.721+0000 7efe0c447640 10 monclient: monmap: 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:57.722708+0000 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:57.722708+0000 2024-09-10T16:25:57.723 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:57.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.722+0000 7efe0c447640 1 Processor -- start 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.722+0000 7efe0c447640 1 -- start start 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.722+0000 7efe0c447640 10 monclient: init 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efe0409c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-b con 0x7efe0409d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efe0413a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe0409fbc0 con 0x7efe0409c310 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe040a2a20 con 0x7efe0413a5c0 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efe0409c6f0 con 0x7efe0409d190 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0c447640 10 monclient(hunting): authenticate will time out at 1936.210138s 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0a1e3640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efe0409c310 0x7efe0409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:55656/0 (socket says 172.21.15.164:55656) 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe0a1e3640 1 -- 172.21.15.164:0/4262745200 learned_addr learned my addr 172.21.15.164:0/4262745200 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): get_auth_request con 0x7efe0409d190 auth_method 0 2024-09-10T16:25:57.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient(hunting): handle_auth_done global_id 214226 payload 274 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 1 -- 172.21.15.164:0/4262745200 >> v1:172.21.15.165:6789/0 conn(0x7efe0413a5c0 legacy=0x7efe0413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 1 -- 172.21.15.164:0/4262745200 >> v1:172.21.15.164:6789/0 conn(0x7efe0409c310 legacy=0x7efe0409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 1 monclient: found mon.noname-b 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.723+0000 7efe099e2640 1 -- 172.21.15.164:0/4262745200 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0409a4d0 con 0x7efe0409d190 2024-09-10T16:25:57.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe099e2640 1 -- 172.21.15.164:0/4262745200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 msgr2=0x7efe0409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe099e2640 1 --2- 172.21.15.164:0/4262745200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0409f150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdf80097d0 tx=0x7efdf802a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=726c89b21dd7a111 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 1 -- 172.21.15.164:0/4262745200 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8037070 con 0x7efe0409d190 2024-09-10T16:25:57.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: dump: 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 1 -- 172.21.15.164:0/4262745200 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdf8032640 con 0x7efe0409d190 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 1 -- 172.21.15.164:0/4262745200 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf8032a10 con 0x7efe0409d190 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe0c447640 5 monclient: authenticate success, global_id 214226 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe0c447640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe0c447640 4 set_mon_vals no callback set 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.726+0000 7efe091e1640 10 monclient: dump: 2024-09-10T16:25:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 10 monclient: shutdown 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 -- 172.21.15.164:0/4262745200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 msgr2=0x7efe0409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 --2- 172.21.15.164:0/4262745200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0409f150 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efdf80097d0 tx=0x7efdf802a030 comp rx=0 tx=0).stop 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 -- 172.21.15.164:0/4262745200 shutdown_connections 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 --2- 172.21.15.164:0/4262745200 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0409f150 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 -- 172.21.15.164:0/4262745200 >> 172.21.15.164:0/4262745200 conn(0x7efe040a3110 msgr2=0x7efe040a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 -- 172.21.15.164:0/4262745200 shutdown_connections 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 1 -- 172.21.15.164:0/4262745200 wait complete. 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.727+0000 7efe0c447640 10 monclient: monmap: 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 Processor -- start 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 -- start start 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient: init 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe0409c310 0x7efe04073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-b con 0x7efe0409c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 0x7efe0409d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-a con 0x7efe0413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 0x7efe0409de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): picked mon.noname-c con 0x7efe0412bb00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efe040a2230 con 0x7efe0413a5c0 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efe040a20b0 con 0x7efe0409c310 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efe04074e00 con 0x7efe0412bb00 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0c447640 10 monclient(hunting): authenticate will time out at 1936.215703s 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 0x7efe0409d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 0x7efe0409de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 0x7efe0409d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39422/0 (socket says 172.21.15.164:39422) 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 1 -- 172.21.15.164:0/2130505858 learned_addr learned my addr 172.21.15.164:0/2130505858 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): get_auth_request con 0x7efe0413a5c0 auth_method 0 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): get_auth_request con 0x7efe0412bb00 auth_method 0 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe0a9e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.728+0000 7efe099e2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a1e3640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe0409c310 0x7efe04073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 10 monclient(hunting): handle_auth_done global_id 214229 payload 824 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe0409c310 msgr2=0x7efe04073a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efe0409c310 0x7efe04073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 msgr2=0x7efe0409d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 0x7efe0409d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 monclient: found mon.noname-c 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe0a9e4640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe040a4da0 con 0x7efe0412bb00 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.729+0000 7efe099e2640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efe0413a5c0 0x7efe0409d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0a9e4640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 msgr2=0x7efe0409de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0a9e4640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 0x7efe0409de70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00c3e0 tx=0x7efdfc00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc0119a0 con 0x7efe0412bb00 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: dump: 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.744 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efdfc017070 con 0x7efe0412bb00 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 5 monclient: authenticate success, global_id 214229 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 10 monclient: _renew_subs 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0412d1c0 con 0x7efe0412bb00 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc011d80 con 0x7efe0412bb00 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe02ffd640 10 monclient: dump: 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.746 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 10 monclient: _renew_subs 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0409e3b0 con 0x7efe0412bb00 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.731+0000 7efe089e0640 4 set_mon_vals no callback set 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0c447640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0c447640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0406f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efe0412cee0 con 0x7efe0409d190 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efdfc01c460 con 0x7efe0412bb00 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0406f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient: get_auth_request con 0x7efe0409d190 auth_method 0 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe02ffd640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdf003e290 0x7efdf0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(681..681 src has 494..681) ==== 4815+0+0 (secure 0 0 0) 0x7efdfc052b20 con 0x7efe0412bb00 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_done global_id 214232 payload 824 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe0a1e3640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe0409f800 con 0x7efe0409d190 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe099e2640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdf003e290 0x7efdf0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.732+0000 7efe099e2640 10 monclient: get_auth_request con 0x7efdf003e290 auth_method 0 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.734+0000 7efe0a1e3640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 msgr2=0x7efe0406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe0a1e3640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0406f090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf400b0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf4012410 con 0x7efe0409d190 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 10 monclient: dump: 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.751 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efdf4017070 con 0x7efe0409d190 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 msgr2=0x7efe0406f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0406f090 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf400b0d0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 msgr2=0x7efe0406f090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe02ffd640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0409d190 0x7efe0406f090 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf400b0d0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.735+0000 7efe099e2640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdf003e290 0x7efdf0040750 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efdf802a070 tx=0x7efdf802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0c447640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0c447640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 0x7efe0406f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0c447640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efe0412d6e0 con 0x7efe0406f5d0 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 0x7efe0406f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient: get_auth_request con 0x7efe0406f5d0 auth_method 0 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 10 monclient(hunting): handle_auth_done global_id 214235 payload 824 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.741+0000 7efe0a1e3640 1 -- 172.21.15.164:0/2130505858 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efe04077260 con 0x7efe0406f5d0 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0a1e3640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 msgr2=0x7efe0406f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0a1e3640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 0x7efe0406f9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf4004cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdf4012a30 con 0x7efe0406f5d0 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 10 monclient: dump: 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7efdf4012a30 con 0x7efe0406f5d0 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 msgr2=0x7efe0406f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 0x7efe0406f9e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf4004cc0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 msgr2=0x7efe0406f9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe02ffd640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0406f5d0 0x7efe0406f9e0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efdf400db00 tx=0x7efdf4004cc0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 57523, 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:57.756 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.757 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.758 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:57.759 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.760 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:57.761 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.762 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.763 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.764 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.765 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.766 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.767 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:57.768 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdf003e290 msgr2=0x7efdf0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efdf003e290 0x7efdf0040750 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7efdf802a070 tx=0x7efdf802e000 comp rx=0 tx=0).stop 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 10 monclient: shutdown 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 msgr2=0x7efe0409de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0a9e4640 1 -- 172.21.15.164:0/2130505858 reap_dead start 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 0x7efe0409de70 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00c3e0 tx=0x7efdfc00c8a0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 shutdown_connections 2024-09-10T16:25:57.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 --2- 172.21.15.164:0/2130505858 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efe0412bb00 0x7efe0409de70 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 >> 172.21.15.164:0/2130505858 conn(0x7efe040a3110 msgr2=0x7efe04137ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 shutdown_connections 2024-09-10T16:25:57.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.744+0000 7efe0c447640 1 -- 172.21.15.164:0/2130505858 wait complete. 2024-09-10T16:25:57.771 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:57.771 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:25:57.771 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:25:57.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.868+0000 7f5c3d7f8640 10 monclient: get_monmap_and_config 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.868+0000 7f5c3d7f8640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.868+0000 7f5c3d7f8640 10 monclient: monmap: 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:25:57.869345+0000 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:25:57.869345+0000 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:25:57.870 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:25:57.871 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 Processor -- start 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 -- start start 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient: init 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c380a4520 0x7f5c380a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-a con 0x7f5c380a4520 addr v2:172.21.15.164:3300/0 2024-09-10T16:25:57.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-b con 0x7f5c380a66d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c38138140 addr v1:172.21.15.165:6789/0 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c380a64c0 con 0x7f5c38138140 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5c3809ccb0 con 0x7f5c380a4520 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c3809c9f0 con 0x7f5c380a66d0 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c3d7f8640 10 monclient(hunting): authenticate will time out at 1936.356590s 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c380a4520 0x7f5c380a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5c380a4520 0x7f5c380a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39424/0 (socket says 172.21.15.164:39424) 2024-09-10T16:25:57.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34534/0 (socket says 172.21.15.164:34534) 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3860216977 learned_addr learned my addr 172.21.15.164:0/3860216977 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request con 0x7f5c380a4520 auth_method 0 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): get_auth_request con 0x7f5c380a66d0 auth_method 0 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_done global_id 224136 payload 274 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 msgr2=0x7f5c38137c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3860216977 >> v1:172.21.15.165:6789/0 conn(0x7f5c38138140 legacy=0x7f5c3813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.869+0000 7f5c36ffd640 1 monclient: found mon.noname-a 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.870+0000 7f5c36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.870+0000 7f5c36ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.870+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3860216977 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c380a7200 con 0x7f5c380a4520 2024-09-10T16:25:57.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.870+0000 7f5c367fc640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:25:57.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a4520 msgr2=0x7f5c380a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a4520 0x7f5c380a4900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009e50 tx=0x7f5c3002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d8b4570af06306 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 1 -- 172.21.15.164:0/3860216977 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30036070 con 0x7f5c380a4520 2024-09-10T16:25:57.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: dump: 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 1 -- 172.21.15.164:0/3860216977 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c30007800 con 0x7f5c380a4520 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c3d7f8640 5 monclient: authenticate success, global_id 224136 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c3d7f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c3d7f8640 10 monclient: get_monmap_and_config success 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 1 -- 172.21.15.164:0/3860216977 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30007bd0 con 0x7f5c380a4520 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c3d7f8640 4 set_mon_vals no callback set 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.872+0000 7f5c35ffb640 10 monclient: dump: 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.886 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 10 monclient: shutdown 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a4520 msgr2=0x7f5c380a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a4520 0x7f5c380a4900 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009e50 tx=0x7f5c3002a560 comp rx=0 tx=0).stop 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3860216977 shutdown_connections 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a66d0 0x7f5c38137c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3860216977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a4520 0x7f5c380a4900 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3860216977 >> 172.21.15.164:0/3860216977 conn(0x7f5c380aac50 msgr2=0x7f5c380ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3860216977 shutdown_connections 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.873+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3860216977 wait complete. 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 10 monclient: build_initial_monmap 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 10 monclient: monmap: 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:25:57.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 1 Processor -- start 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 1 -- start start 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 10 monclient: init 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.874+0000 7f5c3d7f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a4520 0x7f5c381210c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-c con 0x7f5c380a4520 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 0x7f5c38121600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-a con 0x7f5c380a66d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c38138140 0x7f5c38121b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): picked mon.noname-b con 0x7f5c38138140 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c380a4fc0 con 0x7f5c380a66d0 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c380a4e40 con 0x7f5c38138140 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c38134930 con 0x7f5c380a4520 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c3d7f8640 10 monclient(hunting): authenticate will time out at 1936.362088s 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a4520 0x7f5c381210c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 0x7f5c38121600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a4520 0x7f5c381210c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34550/0 (socket says 172.21.15.164:34550) 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3081413470 learned_addr learned my addr 172.21.15.164:0/3081413470 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request con 0x7f5c380a4520 auth_method 0 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): get_auth_request con 0x7f5c380a66d0 auth_method 0 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c377fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c38138140 0x7f5c38121b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient(hunting): handle_auth_done global_id 224139 payload 824 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a4520 msgr2=0x7f5c381210c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c380a4520 0x7f5c381210c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c38138140 msgr2=0x7f5c38121b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c38138140 0x7f5c38121b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient: _finish_hunting 0 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 monclient: found mon.noname-a 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c367fc640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c381357d0 con 0x7f5c380a66d0 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c377fe640 10 monclient: get_auth_request con 0x7f5c38138140 auth_method 0 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.875+0000 7f5c377fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c38138140 0x7f5c38121b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c367fc640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 msgr2=0x7f5c38121600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c367fc640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 0x7f5c38121600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e900 tx=0x7f5c2000edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20017070 con 0x7f5c380a66d0 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: dump: 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 5 monclient: authenticate success, global_id 224139 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c2000cc10 con 0x7f5c380a66d0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 10 monclient: _renew_subs 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c38123430 con 0x7f5c380a66d0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c20013470 con 0x7f5c380a66d0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c2f7fe640 10 monclient: dump: 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c357fa640 4 set_mon_vals no callback set 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 10 monclient: _renew_subs 2024-09-10T16:25:57.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.877+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c381221f0 con 0x7f5c380a66d0 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c3d7f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 0x7f5c381cd830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c38050fa0 con 0x7f5c381cd450 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5c2001a070 con 0x7f5c380a66d0 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.878+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 0x7f5c381cd830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c2f7fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5c1803e240 0x7f5c18040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient: get_auth_request con 0x7f5c381cd450 auth_method 0 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(681..681 src has 494..681) ==== 4815+0+0 (secure 0 0 0) 0x7f5c20052c00 con 0x7f5c380a66d0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c377fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5c1803e240 0x7f5c18040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c377fe640 10 monclient: get_auth_request con 0x7f5c1803e240 auth_method 0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_done global_id 154669 payload 824 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.879+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c3806e8b0 con 0x7f5c381cd450 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 msgr2=0x7f5c381cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 0x7f5c381cd830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5c3002a5a0 tx=0x7f5c30004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c377fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5c1803e240 0x7f5c18040700 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5c28007fd0 tx=0x7f5c28009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30042070 con 0x7f5c381cd450 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.881+0000 7f5c2f7fe640 10 monclient: dump: 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c30037040 con 0x7f5c381cd450 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:25:57.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:25:57.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 msgr2=0x7f5c381cd830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 0x7f5c381cd830 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5c3002a5a0 tx=0x7f5c30004a40 comp rx=0 tx=0).stop 2024-09-10T16:25:57.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 msgr2=0x7f5c381cd830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.882+0000 7f5c2f7fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cd450 0x7f5c381cd830 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c3d7f8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 0x7f5c3809f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c3d7f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c38123b50 con 0x7f5c381cdd70 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 0x7f5c3809f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient: get_auth_request con 0x7f5c381cdd70 auth_method 0 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:25:57.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 10 monclient(hunting): handle_auth_done global_id 154672 payload 824 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.888+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3081413470 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c380a4900 con 0x7f5c381cdd70 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c36ffd640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 msgr2=0x7f5c3809f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c36ffd640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 0x7f5c3809f480 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5c3806ebe0 tx=0x7f5c3002f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c30036040 con 0x7f5c381cdd70 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 10 monclient: dump: 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:25:57.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f5c30009af0 con 0x7f5c381cdd70 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 msgr2=0x7f5c3809f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 0x7f5c3809f480 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5c3806ebe0 tx=0x7f5c3002f000 comp rx=0 tx=0).stop 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 msgr2=0x7f5c3809f480 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:25:57.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c2f7fe640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c381cdd70 0x7f5c3809f480 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5c3806ebe0 tx=0x7f5c3002f000 comp rx=0 tx=0).stop 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 192, 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 376442, 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:25:57.904 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.905 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.906 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:25:57.907 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:25:57.908 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:25:57.909 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:25:57.912 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.913 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.914 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.915 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:25:57.916 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5c1803e240 msgr2=0x7f5c18040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5c1803e240 0x7f5c18040700 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f5c28007fd0 tx=0x7f5c28009040 comp rx=0 tx=0).stop 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 10 monclient: shutdown 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 msgr2=0x7f5c38121600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 --2- 172.21.15.164:0/3081413470 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c380a66d0 0x7f5c38121600 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e900 tx=0x7f5c2000edc0 comp rx=0 tx=0).stop 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c377fe640 1 -- 172.21.15.164:0/3081413470 reap_dead start 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c377fe640 1 -- 172.21.15.164:0/3081413470 reap_dead start 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 shutdown_connections 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 >> 172.21.15.164:0/3081413470 conn(0x7f5c380aac50 msgr2=0x7f5c381325c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.891+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 shutdown_connections 2024-09-10T16:25:57.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:25:57.892+0000 7f5c3d7f8640 1 -- 172.21.15.164:0/3081413470 wait complete. 2024-09-10T16:25:57.919 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:25:57.919 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:25:57.919 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-10T16:25:57.919 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:25:57.919 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:25:57.919 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:25:57.919 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:25:57.937 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:25:57.937 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:25:57.938 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:25:57.938 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:26:12.939 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:26:12.940 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:26:12.940 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:26:12.940 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:26:13.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.048+0000 7fdf83789640 10 monclient: get_monmap_and_config 2024-09-10T16:26:13.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.048+0000 7fdf83789640 10 monclient: build_initial_monmap 2024-09-10T16:26:13.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.048+0000 7fdf83789640 10 monclient: monmap: 2024-09-10T16:26:13.049 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:13.049 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:26:13.049101+0000 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:26:13.049101+0000 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:26:13.050 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:13.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.048+0000 7fdf83789640 1 Processor -- start 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 1 -- start start 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient: init 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-c con 0x7fdf7c06cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdf7c0a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf7c0a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c09ce50 con 0x7fdf7c0a4ab0 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c138ac0 con 0x7fdf7c0a46a0 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:13.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdf7c06db60 con 0x7fdf7c06cb70 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf83789640 10 monclient(hunting): authenticate will time out at 1951.536680s 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf81d26640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fdf7c0a4ab0 0x7fdf7c1383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44694/0 (socket says 172.21.15.164:44694) 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf81525640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 msgr2=0x7fdf7c134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf80d24640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fdf7c0a46a0 legacy=0x7fdf7c137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf81d26640 1 -- 172.21.15.164:0/4069705047 learned_addr learned my addr 172.21.15.164:0/4069705047 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.049+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:13.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.052+0000 7fdf81d26640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.164:6789/0 conn(0x7fdf7c0a4ab0 legacy=0x7fdf7c1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:13.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.250+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.165:6789/0 conn(0x7fdf7c0a46a0 legacy=0x7fdf7c137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:13.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.250+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 msgr2=0x7fdf7c134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:13.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.250+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:13.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.652+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 msgr2=0x7fdf7c134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:13.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.652+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.165:6789/0 conn(0x7fdf7c0a46a0 legacy=0x7fdf7c137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:13.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:13.652+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:14.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:14.453+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.165:6789/0 conn(0x7fdf7c0a46a0 legacy=0x7fdf7c137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:14.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:14.453+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 msgr2=0x7fdf7c134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:14.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:14.453+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): tick 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 msgr2=0x7fdf7c134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf7c06cb70 0x7fdf7c134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.165:6789/0 conn(0x7fdf7c0a46a0 legacy=0x7fdf7c137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.164:6789/0 conn(0x7fdf7c0a4ab0 legacy=0x7fdf7c1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:16.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient: _add_conns ranks=[3,2] 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fdf60003770 addr v2:172.21.15.164:3300/0 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): picked mon.noname-b con 0x7fdf60006280 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdf60008df0 con 0x7fdf60003770 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdf60008fa0 con 0x7fdf60006280 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.049+0000 7fdf72ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 10 monclient(hunting): get_auth_request con 0x7fdf60006280 auth_method 0 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60003770 auth_method 0 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:16.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.050+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:16.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60003770 auth_method 0 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 10 monclient(hunting): get_auth_request con 0x7fdf60006280 auth_method 0 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:16.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:16.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.251+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60003770 auth_method 0 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 10 monclient(hunting): get_auth_request con 0x7fdf60006280 auth_method 0 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:16.652+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:17.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 10 monclient(hunting): get_auth_request con 0x7fdf60006280 auth_method 0 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60003770 auth_method 0 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:17.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:17.454+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:18.961 INFO:tasks.rados.rados.0.smithi165.stdout:3036: done (3 left) 2024-09-10T16:26:18.962 INFO:tasks.rados.rados.0.smithi165.stdout:3037: write (excl) oid 46 current snap is 328 2024-09-10T16:26:18.962 INFO:tasks.rados.rados.0.smithi165.stdout:3037: seq_num 1060 ranges {615420=737007,1912207=425038,3026137=542216} 2024-09-10T16:26:18.969 INFO:tasks.rados.rados.0.smithi165.stdout:3037: writing smithi16534841-46 from 615420 to 1352427 tid 1 2024-09-10T16:26:18.974 INFO:tasks.rados.rados.0.smithi165.stdout:3037: writing smithi16534841-46 from 1912207 to 2337245 tid 2 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3037: writing smithi16534841-46 from 3026137 to 3568353 tid 3 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3038: write oid 45 current snap is 328 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3034: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1442 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3033: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3035: finishing write tid 1 to smithi16534841-34 2024-09-10T16:26:18.982 INFO:tasks.rados.rados.0.smithi165.stdout:3038: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2024-09-10T16:26:18.987 INFO:tasks.rados.rados.0.smithi165.stdout:3038: writing smithi16534841-45 from 632227 to 1044483 tid 1 2024-09-10T16:26:18.993 INFO:tasks.rados.rados.0.smithi165.stdout:3038: writing smithi16534841-45 from 1570689 to 2139385 tid 2 2024-09-10T16:26:19.001 INFO:tasks.rados.rados.0.smithi165.stdout:3038: writing smithi16534841-45 from 2653560 to 3407048 tid 3 2024-09-10T16:26:19.001 INFO:tasks.rados.rados.0.smithi165.stdout:3034: done (4 left) 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3039: read oid 33 snap -1 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3039: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3040: copy_from oid 50 from oid 28 current snap is 328 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3033: finishing copy_from to smithi16534841-30 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1795 (ObjNum 999 snap 299 seq_num 999) dirty exists 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: finishing write tid 2 to smithi16534841-34 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: finishing write tid 3 to smithi16534841-34 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: finishing write tid 4 to smithi16534841-34 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: finishing write tid 5 to smithi16534841-34 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: oid 34 updating version 0 to 1574 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: oid 34 updating version 1574 to 1576 2024-09-10T16:26:19.002 INFO:tasks.rados.rados.0.smithi165.stdout:3035: oid 34 version 1576 is already newer than 1573 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3035: oid 34 version 1576 is already newer than 1575 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1576 (ObjNum 1059 snap 328 seq_num 1059) dirty exists 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3035: left oid 34 (ObjNum 1059 snap 328 seq_num 1059) 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3037: finishing write tid 1 to smithi16534841-46 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3037: finishing write tid 2 to smithi16534841-46 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3033: done (5 left) 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3035: done (4 left) 2024-09-10T16:26:19.003 INFO:tasks.rados.rados.0.smithi165.stdout:3041: snap_remove snap 321 2024-09-10T16:26:19.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): tick 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 msgr2=0x7fdf60008740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60006280 0x7fdf60008740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 msgr2=0x7fdf60005bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60003770 0x7fdf60005bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81d26640 1 -- 172.21.15.164:0/4069705047 reap_dead start 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60009400 0x7fdf6000b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): picked mon.noname-a con 0x7fdf60009400 addr v2:172.21.15.164:3300/0 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 0x7fdf60006730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): picked mon.noname-c con 0x7fdf60003c10 addr v2:172.21.15.165:3300/0 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdf60006c70 addr v1:172.21.15.164:6789/0 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7c09cb70 con 0x7fdf60006c70 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fdf60008df0 con 0x7fdf60009400 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 1 -- 172.21.15.164:0/4069705047 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fdf60008fa0 con 0x7fdf60003c10 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf72ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf80d24640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 msgr2=0x7fdf60006730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf80d24640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 0x7fdf60006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.164:3300/0 conn(0x7fdf60009400 0x7fdf6000b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60009400 auth_method 0 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient(hunting): handle_auth_done global_id 224142 payload 274 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 msgr2=0x7fdf60006730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 0x7fdf60006730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> v1:172.21.15.164:6789/0 conn(0x7fdf60006c70 legacy=0x7fdf60007120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.050+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c138ac0 con 0x7fdf60009400 2024-09-10T16:26:19.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.053+0000 7fdf81525640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60009400 msgr2=0x7fdf6000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.053+0000 7fdf81525640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60009400 0x7fdf6000b7f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdf60003be0 tx=0x7fdf6c02b380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df3f00b1c5abd35 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 1 -- 172.21.15.164:0/4069705047 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02be20 con 0x7fdf60009400 2024-09-10T16:26:19.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: dump: 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 1 -- 172.21.15.164:0/4069705047 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf6c02f600 con 0x7fdf60009400 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 5 monclient: authenticate success, global_id 224142 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 10 monclient: get_monmap_and_config success 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 1 -- 172.21.15.164:0/4069705047 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf6c02f9f0 con 0x7fdf60009400 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 4 set_mon_vals no callback set 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf73fff640 10 monclient: dump: 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 10 monclient: shutdown 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 1 -- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60009400 msgr2=0x7fdf6000b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.054+0000 7fdf83789640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60009400 0x7fdf6000b7f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdf60003be0 tx=0x7fdf6c02b380 comp rx=0 tx=0).stop 2024-09-10T16:26:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 -- 172.21.15.164:0/4069705047 shutdown_connections 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 --2- 172.21.15.164:0/4069705047 >> v2:172.21.15.165:3300/0 conn(0x7fdf60003c10 0x7fdf60006730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 --2- 172.21.15.164:0/4069705047 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60009400 0x7fdf6000b7f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 -- 172.21.15.164:0/4069705047 >> 172.21.15.164:0/4069705047 conn(0x7fdf7c0aa900 msgr2=0x7fdf7c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 -- 172.21.15.164:0/4069705047 shutdown_connections 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 1 -- 172.21.15.164:0/4069705047 wait complete. 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 10 monclient: build_initial_monmap 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.055+0000 7fdf83789640 10 monclient: monmap: 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:19.061 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 Processor -- start 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 -- start start 2024-09-10T16:26:19.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient: init 2024-09-10T16:26:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:26:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 0x7fdf7c1cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-c con 0x7fdf60003c10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-a con 0x7fdf60006c70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 0x7fdf7c0a3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): picked mon.noname-b con 0x7fdf60009400 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c134f70 con 0x7fdf60006c70 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fdf7c06cf50 con 0x7fdf60009400 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fdf7c1350f0 con 0x7fdf60003c10 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.056+0000 7fdf83789640 10 monclient(hunting): authenticate will time out at 1957.543960s 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 0x7fdf7c1cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59798/0 (socket says 172.21.15.164:59798) 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 -- 172.21.15.164:0/1947805945 learned_addr learned my addr 172.21.15.164:0/1947805945 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): get_auth_request con 0x7fdf60006c70 auth_method 0 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 0x7fdf7c1cd590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56300/0 (socket says 172.21.15.164:56300) 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): get_auth_request con 0x7fdf60003c10 auth_method 0 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81d26640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 msgr2=0x7fdf7c0a3fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81d26640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 0x7fdf7c0a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf81525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient(hunting): handle_auth_done global_id 224145 payload 824 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 msgr2=0x7fdf7c1cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 0x7fdf7c1cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 msgr2=0x7fdf7c0a3fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:26:19.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 0x7fdf7c0a3fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.057+0000 7fdf80d24640 1 -- 172.21.15.164:0/1947805945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf7c0a13e0 con 0x7fdf60006c70 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.059+0000 7fdf80d24640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 msgr2=0x7fdf7c0a3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.059+0000 7fdf80d24640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009af0 tx=0x7fdf78009fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.059+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf780173f0 con 0x7fdf60006c70 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.059+0000 7fdf71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: dump: 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.070 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 5 monclient: authenticate success, global_id 224145 2024-09-10T16:26:19.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf78017590 con 0x7fdf60006c70 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf78017940 con 0x7fdf60006c70 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 10 monclient: _renew_subs 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf7c06e720 con 0x7fdf60006c70 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf71ffb640 10 monclient: dump: 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf737fe640 4 set_mon_vals no callback set 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 10 monclient: _renew_subs 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.060+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf7c0a2570 con 0x7fdf60006c70 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fdf78017b60 con 0x7fdf60006c70 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf83789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf83789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf83789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf7c134070 con 0x7fdf60006c70 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf71ffb640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 0x7fdf60042450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(682..682 src has 494..682) ==== 4831+0+0 (secure 0 0 0) 0x7fdf78053a70 con 0x7fdf60006c70 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf81525640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 0x7fdf60042450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.061+0000 7fdf81525640 10 monclient: get_auth_request con 0x7fdf6003ff90 auth_method 0 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.064+0000 7fdf81525640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 0x7fdf60042450 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdf60006100 tx=0x7fdf6c02d000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.066+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fdf7c134070 con 0x7fdf60006c70 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.066+0000 7fdf71ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.066+0000 7fdf71ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:26:19.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.309+0000 7fdf83789640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:26:19.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.309+0000 7fdf83789640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.309+0000 7fdf83789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.309+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdf7c137ca0 con 0x7fdf60006c70 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.310+0000 7fdf71ffb640 1 -- 172.21.15.164:0/1947805945 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fdf78052440 con 0x7fdf60006c70 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.310+0000 7fdf71ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.310+0000 7fdf71ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:26:19.312 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":196,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 msgr2=0x7fdf60042450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 0x7fdf60042450 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fdf60006100 tx=0x7fdf6c02d000 comp rx=0 tx=0).stop 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 10 monclient: shutdown 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 msgr2=0x7fdf7c0a3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009af0 tx=0x7fdf78009fb0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 shutdown_connections 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fdf6003ff90 0x7fdf60042450 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fdf60009400 0x7fdf7c0a3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fdf60006c70 0x7fdf7c0a3aa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 --2- 172.21.15.164:0/1947805945 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fdf60003c10 0x7fdf7c1cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 >> 172.21.15.164:0/1947805945 conn(0x7fdf7c0aa900 msgr2=0x7fdf7c0a7bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 shutdown_connections 2024-09-10T16:26:19.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.312+0000 7fdf83789640 1 -- 172.21.15.164:0/1947805945 wait complete. 2024-09-10T16:26:19.324 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:26:19.324 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:26:19.325 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:26:19.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.422+0000 7f9bac933640 10 monclient: get_monmap_and_config 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.422+0000 7f9bac933640 10 monclient: build_initial_monmap 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.422+0000 7f9bac933640 10 monclient: monmap: 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:26:19.423419+0000 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:26:19.423419+0000 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:26:19.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:26:19.425 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:26:19.425 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:26:19.425 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 Processor -- start 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 -- start start 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient: init 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient: _add_conns ranks=[1,0,3,2,4] 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ba809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ba809d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9ba813b5d0 0x7f9ba813d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba813b5d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:26:19.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ba809fbc0 con 0x7f9ba809d190 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ba80a2a20 con 0x7f9ba809c310 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9ba809f860 con 0x7f9ba813b5d0 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9bac933640 10 monclient(hunting): authenticate will time out at 1957.910620s 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba659c640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f9ba809c310 legacy=0x7f9ba809ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9ba813b5d0 0x7f9ba813d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9b9dd9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9ba809d190 0x7f9ba809f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44776/0 (socket says 172.21.15.164:44776) 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/652726227 learned_addr learned my addr 172.21.15.164:0/652726227 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): get_auth_request con 0x7f9ba813b5d0 auth_method 0 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.423+0000 7f9ba6d9d640 10 monclient(hunting): handle_auth_done global_id 224148 payload 274 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/652726227 >> v1:172.21.15.165:6789/0 conn(0x7f9ba809c310 legacy=0x7f9ba809ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/652726227 >> v1:172.21.15.164:6789/0 conn(0x7f9ba809d190 legacy=0x7f9ba809f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.424+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/652726227 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba809c6f0 con 0x7f9ba813b5d0 2024-09-10T16:26:19.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.426+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/652726227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 msgr2=0x7f9ba813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.426+0000 7f9ba6d9d640 1 --2- 172.21.15.164:0/652726227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6490c47c2665ef3e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 1 -- 172.21.15.164:0/652726227 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0007a30 con 0x7f9ba813b5d0 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: dump: 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.436 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 1 -- 172.21.15.164:0/652726227 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ba0007bd0 con 0x7f9ba813b5d0 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 5 monclient: authenticate success, global_id 224148 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 1 -- 172.21.15.164:0/652726227 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ba0031400 con 0x7f9ba813b5d0 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 10 monclient: get_monmap_and_config success 2024-09-10T16:26:19.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 4 set_mon_vals no callback set 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9ba5d9b640 10 monclient: dump: 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.438 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 10 monclient: shutdown 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 -- 172.21.15.164:0/652726227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 msgr2=0x7f9ba813d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 --2- 172.21.15.164:0/652726227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813d9c0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0009d90 tx=0x7f9ba002a490 comp rx=0 tx=0).stop 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 -- 172.21.15.164:0/652726227 shutdown_connections 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 --2- 172.21.15.164:0/652726227 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813d9c0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 -- 172.21.15.164:0/652726227 >> 172.21.15.164:0/652726227 conn(0x7f9ba80a3110 msgr2=0x7f9ba80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 -- 172.21.15.164:0/652726227 shutdown_connections 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.427+0000 7f9bac933640 1 -- 172.21.15.164:0/652726227 wait complete. 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.428+0000 7f9bac933640 10 monclient: build_initial_monmap 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.428+0000 7f9bac933640 10 monclient: monmap: 2024-09-10T16:26:19.439 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.428+0000 7f9bac933640 1 Processor -- start 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.428+0000 7f9bac933640 1 -- start start 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.428+0000 7f9bac933640 10 monclient: init 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:19.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ba809c310 0x7f9ba813ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba813b5d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba808df80 0x7f9ba808e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba808df80 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba80a20b0 con 0x7f9ba813b5d0 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba809a530 con 0x7f9ba808df80 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9ba80a2230 con 0x7f9ba809c310 2024-09-10T16:26:19.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9bac933640 10 monclient(hunting): authenticate will time out at 1957.916319s 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ba809c310 0x7f9ba813ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59814/0 (socket says 172.21.15.164:59814) 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ba809c310 0x7f9ba813ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:56312/0 (socket says 172.21.15.164:56312) 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/1076813106 learned_addr learned my addr 172.21.15.164:0/1076813106 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): get_auth_request con 0x7f9ba813b5d0 auth_method 0 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): get_auth_request con 0x7f9ba809c310 auth_method 0 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba808df80 msgr2=0x7f9ba808e410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba6d9d640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba808df80 0x7f9ba808e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient(hunting): handle_auth_done global_id 224151 payload 824 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ba809c310 msgr2=0x7f9ba813ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ba809c310 0x7f9ba813ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba808df80 msgr2=0x7f9ba808e410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ba808df80 0x7f9ba808e410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.429+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ba8091810 con 0x7f9ba813b5d0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.431+0000 7f9b9dd9b640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 msgr2=0x7f9ba813b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.431+0000 7f9b9dd9b640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813b050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c430 tx=0x7f9b9000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b90017070 con 0x7f9ba813b5d0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: dump: 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b900070d0 con 0x7f9ba813b5d0 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 5 monclient: authenticate success, global_id 224151 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 monclient: _renew_subs 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ba813db60 con 0x7f9ba813b5d0 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b900074c0 con 0x7f9ba813b5d0 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 10 monclient: dump: 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.446 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9ba559a640 4 set_mon_vals no callback set 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 monclient: _renew_subs 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ba8091530 con 0x7f9ba813b5d0 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 0x7f9ba808e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba809d930 con 0x7f9ba809d190 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.432+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9b9001a070 con 0x7f9ba813b5d0 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 0x7f9ba808e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient: get_auth_request con 0x7f9ba809d190 auth_method 0 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9b9f7fe640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b8803e1c0 0x7f9b88040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 10 monclient(hunting): handle_auth_done global_id 224154 payload 824 2024-09-10T16:26:19.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba659c640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9ba809d620 con 0x7f9ba809d190 2024-09-10T16:26:19.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(682..682 src has 494..682) ==== 4831+0+0 (secure 0 0 0) 0x7f9b90052200 con 0x7f9ba813b5d0 2024-09-10T16:26:19.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba6d9d640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b8803e1c0 0x7f9b88040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.433+0000 7f9ba6d9d640 10 monclient: get_auth_request con 0x7f9b8803e1c0 auth_method 0 2024-09-10T16:26:19.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9ba659c640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 msgr2=0x7f9ba808e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9ba659c640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 0x7f9ba808e950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b98003930 tx=0x7f9b98009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98016070 con 0x7f9ba809d190 2024-09-10T16:26:19.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 10 monclient: dump: 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9b980128f0 con 0x7f9ba809d190 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 msgr2=0x7f9ba808e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 0x7f9ba808e950 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f9b98003930 tx=0x7f9b98009b00 comp rx=0 tx=0).stop 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 msgr2=0x7f9ba808e950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.435+0000 7f9b9f7fe640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba809d190 0x7f9ba808e950 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.436+0000 7f9ba6d9d640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b8803e1c0 0x7f9b88040680 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0034000 tx=0x7f9ba002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9bac933640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9bac933640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 0x7f9ba813b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9bac933640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9ba8077260 con 0x7f9ba808ee90 2024-09-10T16:26:19.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9ba659c640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 0x7f9ba813b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.441+0000 7f9ba659c640 10 monclient: get_auth_request con 0x7f9ba808ee90 auth_method 0 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 10 monclient(hunting): handle_auth_done global_id 224157 payload 824 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.442+0000 7f9ba659c640 1 -- 172.21.15.164:0/1076813106 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9ba808f2a0 con 0x7f9ba808ee90 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9ba659c640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 msgr2=0x7f9ba813b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9ba659c640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 0x7f9ba813b9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b98003930 tx=0x7f9b980059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b98016040 con 0x7f9ba808ee90 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 10 monclient: dump: 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f9b9801b450 con 0x7f9ba808ee90 2024-09-10T16:26:19.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 msgr2=0x7f9ba813b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 0x7f9ba813b9b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b98003930 tx=0x7f9b980059d0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 msgr2=0x7f9ba813b9b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9b9f7fe640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba808ee90 0x7f9ba813b9b0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f9b98003930 tx=0x7f9b980059d0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:26:19.456 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 196, 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 31275, 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.457 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.458 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:26:19.459 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:26:19.460 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.461 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.462 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.463 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.464 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:26:19.465 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.466 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.467 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.468 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:26:19.469 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.469 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.469 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.469 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:26:19.469 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:26:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b8803e1c0 msgr2=0x7f9b88040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9bac933640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9b8803e1c0 0x7f9b88040680 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9ba0034000 tx=0x7f9ba002f000 comp rx=0 tx=0).stop 2024-09-10T16:26:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9bac933640 10 monclient: shutdown 2024-09-10T16:26:19.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9bac933640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.444+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 msgr2=0x7f9ba813b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9bac933640 1 --2- 172.21.15.164:0/1076813106 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ba813b5d0 0x7f9ba813b050 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9b9000c430 tx=0x7f9b9000c8f0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/1076813106 reap_dead start 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9ba6d9d640 1 -- 172.21.15.164:0/1076813106 reap_dead start 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 shutdown_connections 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 >> 172.21.15.164:0/1076813106 conn(0x7f9ba80a3110 msgr2=0x7f9ba809f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 shutdown_connections 2024-09-10T16:26:19.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.445+0000 7f9bac933640 1 -- 172.21.15.164:0/1076813106 wait complete. 2024-09-10T16:26:19.471 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:26:19.471 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:26:19.472 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.568+0000 7fa9dbcc6640 10 monclient: get_monmap_and_config 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.568+0000 7fa9dbcc6640 10 monclient: build_initial_monmap 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.569+0000 7fa9dbcc6640 10 monclient: monmap: 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:26:19.569864+0000 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:26:19.569864+0000 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:26:19.571 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.569+0000 7fa9dbcc6640 1 Processor -- start 2024-09-10T16:26:19.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.569+0000 7fa9dbcc6640 1 -- start start 2024-09-10T16:26:19.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.569+0000 7fa9dbcc6640 10 monclient: init 2024-09-10T16:26:19.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa9d40a3b60 addr v1:172.21.15.165:6789/0 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa9d40a0530 0x7fa9d40a0980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d40a0530 addr v2:172.21.15.164:3300/0 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 0x7fa9d413f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d413cf20 addr v2:172.21.15.165:3300/0 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9d4138390 con 0x7fa9d40a3b60 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d40a3f40 con 0x7fa9d40a0530 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa9d409cfe0 con 0x7fa9d413cf20 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9dbcc6640 10 monclient(hunting): authenticate will time out at 1958.057290s 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9a62640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fa9d40a3b60 legacy=0x7fa9d4137c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa9d40a0530 0x7fa9d40a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9da263640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 msgr2=0x7fa9d413f310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9da263640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 0x7fa9d413f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa9d40a0530 0x7fa9d40a0980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59842/0 (socket says 172.21.15.164:59842) 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 -- 172.21.15.164:0/4051332731 learned_addr learned my addr 172.21.15.164:0/4051332731 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): get_auth_request con 0x7fa9d40a0530 auth_method 0 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient(hunting): handle_auth_done global_id 224160 payload 274 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 -- 172.21.15.164:0/4051332731 >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 msgr2=0x7fa9d413f310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 --2- 172.21.15.164:0/4051332731 >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 0x7fa9d413f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 -- 172.21.15.164:0/4051332731 >> v1:172.21.15.165:6789/0 conn(0x7fa9d40a3b60 legacy=0x7fa9d4137c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.570+0000 7fa9d9261640 1 -- 172.21.15.164:0/4051332731 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d4074520 con 0x7fa9d40a0530 2024-09-10T16:26:19.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d9261640 1 -- 172.21.15.164:0/4051332731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a0530 msgr2=0x7fa9d40a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d9261640 1 --2- 172.21.15.164:0/4051332731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a0530 0x7fa9d40a0980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4073040 tx=0x7fa9c402a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b226f5776ee5347 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 1 -- 172.21.15.164:0/4051332731 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4037070 con 0x7fa9d40a0530 2024-09-10T16:26:19.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: dump: 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.583 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 1 -- 172.21.15.164:0/4051332731 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c4032640 con 0x7fa9d40a0530 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9dbcc6640 5 monclient: authenticate success, global_id 224160 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9dbcc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9dbcc6640 10 monclient: get_monmap_and_config success 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 1 -- 172.21.15.164:0/4051332731 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4032a10 con 0x7fa9d40a0530 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9dbcc6640 4 set_mon_vals no callback set 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.573+0000 7fa9d8a60640 10 monclient: dump: 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 10 monclient: shutdown 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/4051332731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a0530 msgr2=0x7fa9d40a0980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/4051332731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a0530 0x7fa9d40a0980 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4073040 tx=0x7fa9c402a0d0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/4051332731 shutdown_connections 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/4051332731 >> v2:172.21.15.165:3300/0 conn(0x7fa9d413cf20 0x7fa9d413f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/4051332731 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a0530 0x7fa9d40a0980 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/4051332731 >> 172.21.15.164:0/4051332731 conn(0x7fa9d40a6f80 msgr2=0x7fa9d40a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/4051332731 shutdown_connections 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/4051332731 wait complete. 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 10 monclient: build_initial_monmap 2024-09-10T16:26:19.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.574+0000 7fa9dbcc6640 10 monclient: monmap: 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 Processor -- start 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 -- start start 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient: init 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9d40a0530 0x7fa9d4131530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-b con 0x7fa9d40a0530 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-a con 0x7fa9d40a3b60 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d413cf20 0x7fa9d41ccd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): picked mon.noname-c con 0x7fa9d413cf20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d413f9d0 con 0x7fa9d40a3b60 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa9d413f850 con 0x7fa9d40a0530 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d413fb50 con 0x7fa9d413cf20 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9dbcc6640 10 monclient(hunting): authenticate will time out at 1958.062617s 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9a62640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9d40a0530 msgr2=0x7fa9d4131530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d413cf20 0x7fa9d41ccd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:59844/0 (socket says 172.21.15.164:59844) 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9a62640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9d40a0530 0x7fa9d4131530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 -- 172.21.15.164:0/461452016 learned_addr learned my addr 172.21.15.164:0/461452016 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:26:19.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): get_auth_request con 0x7fa9d40a3b60 auth_method 0 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): get_auth_request con 0x7fa9d413cf20 auth_method 0 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9da263640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient(hunting): handle_auth_done global_id 224163 payload 824 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d413cf20 msgr2=0x7fa9d41ccd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d413cf20 0x7fa9d41ccd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9d40a0530 msgr2=0x7fa9d4131530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa9d40a0530 0x7fa9d4131530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient: _finish_hunting 0 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 monclient: found mon.noname-a 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.575+0000 7fa9d9261640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9d40a3510 con 0x7fa9d40a3b60 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9d9261640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 msgr2=0x7fa9d4131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9d9261640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009960 tx=0x7fa9c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4041070 con 0x7fa9d40a3b60 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: dump: 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: _finish_auth 0 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 5 monclient: authenticate success, global_id 224163 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9c403c880 con 0x7fa9d40a3b60 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 10 monclient: _renew_subs 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9d4137170 con 0x7fa9d40a3b60 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9c4037040 con 0x7fa9d40a3b60 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9ca7fc640 10 monclient: dump: 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9cbfff640 4 set_mon_vals no callback set 2024-09-10T16:26:19.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 10 monclient: _renew_subs 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.578+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9d413a2f0 con 0x7fa9d40a3b60 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9dbcc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 0x7fa9d402d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d4131fb0 con 0x7fa9d402d440 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 0x7fa9d402d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient: get_auth_request con 0x7fa9d402d440 auth_method 0 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa9c4044070 con 0x7fa9d40a3b60 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_done global_id 214253 payload 824 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9d9a62640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9d40a0980 con 0x7fa9d402d440 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9ca7fc640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa9b803e240 0x7fa9b8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(682..682 src has 494..682) ==== 4831+0+0 (secure 0 0 0) 0x7fa9c40731e0 con 0x7fa9d40a3b60 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9da263640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa9b803e240 0x7fa9b8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.579+0000 7fa9da263640 10 monclient: get_auth_request con 0x7fa9b803e240 auth_method 0 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9d9a62640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 msgr2=0x7fa9d402d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9d9a62640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 0x7fa9d402d820 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000bcc0 tx=0x7fa9d0009f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0005560 con 0x7fa9d402d440 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.581+0000 7fa9ca7fc640 10 monclient: dump: 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa9d0005560 con 0x7fa9d402d440 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:26:19.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 msgr2=0x7fa9d402d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 0x7fa9d402d820 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa9d000bcc0 tx=0x7fa9d0009f30 comp rx=0 tx=0).stop 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 msgr2=0x7fa9d402d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9ca7fc640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402d440 0x7fa9d402d820 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.582+0000 7fa9da263640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa9b803e240 0x7fa9b8040700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4139e00 tx=0x7fa9cc009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9dbcc6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 0x7fa9d402e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9dbcc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa9d409fc20 con 0x7fa9d402dd60 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 0x7fa9d402e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient: get_auth_request con 0x7fa9d402dd60 auth_method 0 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:19.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 10 monclient(hunting): handle_auth_done global_id 214256 payload 824 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.588+0000 7fa9d9a62640 1 -- 172.21.15.164:0/461452016 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa9d406cb70 con 0x7fa9d402dd60 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9d9a62640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 msgr2=0x7fa9d402e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9d9a62640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 0x7fa9d402e170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0014000 tx=0x7fa9d0005dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0019040 con 0x7fa9d402dd60 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9ca7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9ca7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.590+0000 7fa9ca7fc640 10 monclient: dump: 2024-09-10T16:26:19.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fa9d0014430 con 0x7fa9d402dd60 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:26:19.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 msgr2=0x7fa9d402e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 0x7fa9d402e170 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0014000 tx=0x7fa9d0005dc0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 msgr2=0x7fa9d402e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9ca7fc640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa9d402dd60 0x7fa9d402e170 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fa9d0014000 tx=0x7fa9d0005dc0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 196, 2024-09-10T16:26:19.604 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 79368, 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.605 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.606 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:26:19.607 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:26:19.608 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:26:19.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:26:19.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:26:19.613 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:26:19.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.615 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:26:19.616 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:26:19.617 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:26:19.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa9b803e240 msgr2=0x7fa9b8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa9b803e240 0x7fa9b8040700 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4139e00 tx=0x7fa9cc009060 comp rx=0 tx=0).stop 2024-09-10T16:26:19.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 10 monclient: shutdown 2024-09-10T16:26:19.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:26:19.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 msgr2=0x7fa9d4131a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa9c4009960 tx=0x7fa9c40046c0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9da263640 1 -- 172.21.15.164:0/461452016 reap_dead start 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 shutdown_connections 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 --2- 172.21.15.164:0/461452016 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa9d40a3b60 0x7fa9d4131a70 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 >> 172.21.15.164:0/461452016 conn(0x7fa9d40a6f80 msgr2=0x7fa9d413e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 shutdown_connections 2024-09-10T16:26:19.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:19.591+0000 7fa9dbcc6640 1 -- 172.21.15.164:0/461452016 wait complete. 2024-09-10T16:26:19.619 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:26:19.619 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:26:19.963 INFO:tasks.rados.rados.0.smithi165.stdout:3041: done (4 left) 2024-09-10T16:26:19.963 INFO:tasks.rados.rados.0.smithi165.stdout:3042: rollback oid 48 current snap is 328 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 324 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: finishing write tid 3 to smithi16534841-46 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: finishing write tid 4 to smithi16534841-46 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: finishing write tid 5 to smithi16534841-46 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: oid 46 updating version 0 to 1169 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: oid 46 updating version 1169 to 1172 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: oid 46 version 1172 is already newer than 1171 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: oid 46 version 1172 is already newer than 1170 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1172 (ObjNum 1060 snap 328 seq_num 1060) dirty exists 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3037: left oid 46 (ObjNum 1060 snap 328 seq_num 1060) 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3038: finishing write tid 1 to smithi16534841-45 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3038: finishing write tid 2 to smithi16534841-45 2024-09-10T16:26:19.964 INFO:tasks.rados.rados.0.smithi165.stdout:3039: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:26:19.974 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2760923 2024-09-10T16:26:20.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:20.070 INFO:tasks.rados.rados.0.smithi165.stdout:3038: finishing write tid 3 to smithi16534841-45 2024-09-10T16:26:20.070 INFO:tasks.rados.rados.0.smithi165.stdout:3038: finishing write tid 4 to smithi16534841-45 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: finishing write tid 5 to smithi16534841-45 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: oid 45 updating version 0 to 1685 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: oid 45 version 1685 is already newer than 1684 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: oid 45 version 1685 is already newer than 1682 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: oid 45 version 1685 is already newer than 1683 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1685 (ObjNum 1061 snap 328 seq_num 1061) dirty exists 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: left oid 45 (ObjNum 1061 snap 328 seq_num 1061) 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3040: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3040: finishing copy_from to smithi16534841-50 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1280 (ObjNum 1048 snap 323 seq_num 1048) dirty exists 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3037: done (4 left) 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3038: done (3 left) 2024-09-10T16:26:20.071 INFO:tasks.rados.rados.0.smithi165.stdout:3039: done (2 left) 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3040: done (1 left) 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3043: read oid 17 snap -1 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3043: expect deleted 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3044: rollback oid 34 current snap is 328 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 323 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3042: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3042: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3042: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1882 (ObjNum 1001 snap 299 seq_num 1001) dirty exists 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3042: done (2 left) 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3045: copy_from oid 8 from oid 41 current snap is 328 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:3046: rollback oid 39 current snap is 328 2024-09-10T16:26:20.072 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 311 2024-09-10T16:26:20.073 INFO:tasks.rados.rados.0.smithi165.stdout:3047: write oid 45 current snap is 328 2024-09-10T16:26:20.073 INFO:tasks.rados.rados.0.smithi165.stdout:3047: seq_num 1062 ranges {649034=487505,1629171=428707,2680983=564760,3798768=50266} 2024-09-10T16:26:20.075 INFO:tasks.rados.rados.0.smithi165.stdout:3047: writing smithi16534841-45 from 649034 to 1136539 tid 1 2024-09-10T16:26:20.081 INFO:tasks.rados.rados.0.smithi165.stdout:3047: writing smithi16534841-45 from 1629171 to 2057878 tid 2 2024-09-10T16:26:20.088 INFO:tasks.rados.rados.0.smithi165.stdout:3047: writing smithi16534841-45 from 2680983 to 3245743 tid 3 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:3047: writing smithi16534841-45 from 3798768 to 3849034 tid 4 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:3048: rollback oid 27 current snap is 328 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 27 to 323 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:3044: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:3044: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:26:20.092 INFO:tasks.rados.rados.0.smithi165.stdout:3044: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1579 (ObjNum 1052 snap 323 seq_num 1052) dirty exists 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3046: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3046: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3046: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1688 (ObjNum 930 snap 285 seq_num 930) dirty exists 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 1 to smithi16534841-45 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 2 to smithi16534841-45 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3045: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3045: finishing copy_from to smithi16534841-8 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1390 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 3 to smithi16534841-45 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3043: done (5 left) 2024-09-10T16:26:20.093 INFO:tasks.rados.rados.0.smithi165.stdout:3044: done (4 left) 2024-09-10T16:26:20.094 INFO:tasks.rados.rados.0.smithi165.stdout:3045: done (3 left) 2024-09-10T16:26:20.094 INFO:tasks.rados.rados.0.smithi165.stdout:3046: done (2 left) 2024-09-10T16:26:20.094 INFO:tasks.rados.rados.0.smithi165.stdout:3049: delete oid 30 current snap is 328 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 4 to smithi16534841-45 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 5 to smithi16534841-45 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: finishing write tid 6 to smithi16534841-45 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: oid 45 updating version 0 to 1689 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: oid 45 version 1689 is already newer than 1686 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: oid 45 version 1689 is already newer than 1688 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: oid 45 version 1689 is already newer than 1687 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: oid 45 updating version 1689 to 1690 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1690 (ObjNum 1062 snap 328 seq_num 1062) dirty exists 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: left oid 45 (ObjNum 1062 snap 328 seq_num 1062) 2024-09-10T16:26:20.095 INFO:tasks.rados.rados.0.smithi165.stdout:3047: done (2 left) 2024-09-10T16:26:20.096 INFO:tasks.rados.rados.0.smithi165.stdout:3049: done (1 left) 2024-09-10T16:26:20.096 INFO:tasks.rados.rados.0.smithi165.stdout:3050: read oid 11 snap 319 2024-09-10T16:26:20.096 INFO:tasks.rados.rados.0.smithi165.stdout:3050: expect (ObjNum 1038 snap 319 seq_num 1038) 2024-09-10T16:26:20.096 INFO:tasks.rados.rados.0.smithi165.stdout:3051: write (excl) oid 48 current snap is 328 2024-09-10T16:26:20.096 INFO:tasks.rados.rados.0.smithi165.stdout:3051: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2024-09-10T16:26:20.101 INFO:tasks.rados.rados.0.smithi165.stdout:3051: writing smithi16534841-48 from 682648 to 1320651 tid 1 2024-09-10T16:26:20.107 INFO:tasks.rados.rados.0.smithi165.stdout:3051: writing smithi16534841-48 from 2029782 to 2462158 tid 2 2024-09-10T16:26:20.114 INFO:tasks.rados.rados.0.smithi165.stdout:3051: writing smithi16534841-48 from 3019476 to 3606780 tid 3 2024-09-10T16:26:20.114 INFO:tasks.rados.rados.0.smithi165.stdout:3052: rollback oid 47 current snap is 328 2024-09-10T16:26:20.114 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 323 2024-09-10T16:26:20.114 INFO:tasks.rados.rados.0.smithi165.stdout:3048: finishing rollback tid 0 to smithi16534841-27 2024-09-10T16:26:20.114 INFO:tasks.rados.rados.0.smithi165.stdout:3048: finishing rollback tid 1 to smithi16534841-27 2024-09-10T16:26:20.115 INFO:tasks.rados.rados.0.smithi165.stdout:3048: finishing rollback tid 2 to smithi16534841-27 2024-09-10T16:26:20.115 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1654 (ObjNum 1044 snap 321 seq_num 1044) dirty exists 2024-09-10T16:26:20.115 INFO:tasks.rados.rados.0.smithi165.stdout:3050: expect (ObjNum 1038 snap 319 seq_num 1038) 2024-09-10T16:26:20.159 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1299520 2024-09-10T16:26:20.195 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3226571 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3051: finishing write tid 1 to smithi16534841-48 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3051: finishing write tid 2 to smithi16534841-48 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3052: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1632 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3051: finishing write tid 3 to smithi16534841-48 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3051: finishing write tid 4 to smithi16534841-48 2024-09-10T16:26:20.205 INFO:tasks.rados.rados.0.smithi165.stdout:3051: finishing write tid 5 to smithi16534841-48 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: oid 48 updating version 0 to 1884 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: oid 48 updating version 1884 to 1885 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: oid 48 updating version 1885 to 1886 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: oid 48 updating version 1886 to 1887 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1887 (ObjNum 1064 snap 328 seq_num 1064) dirty exists 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: left oid 48 (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3048: done (3 left) 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3050: done (2 left) 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3051: done (1 left) 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3052: done (0 left) 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3053: read oid 42 snap 325 2024-09-10T16:26:20.206 INFO:tasks.rados.rados.0.smithi165.stdout:3053: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-10T16:26:20.207 INFO:tasks.rados.rados.0.smithi165.stdout:3054: read oid 22 snap -1 2024-09-10T16:26:20.207 INFO:tasks.rados.rados.0.smithi165.stdout:3054: expect deleted 2024-09-10T16:26:20.207 INFO:tasks.rados.rados.0.smithi165.stdout:3055: read oid 21 snap -1 2024-09-10T16:26:20.207 INFO:tasks.rados.rados.0.smithi165.stdout:3055: expect (ObjNum 1054 snap 323 seq_num 1054) 2024-09-10T16:26:20.207 INFO:tasks.rados.rados.0.smithi165.stdout:3056: snap_create 2024-09-10T16:26:20.208 INFO:tasks.rados.rados.0.smithi165.stdout:3053: expect (ObjNum 1020 snap 310 seq_num 1020) 2024-09-10T16:26:20.382 INFO:tasks.rados.rados.0.smithi165.stdout:3055: expect (ObjNum 1054 snap 323 seq_num 1054) 2024-09-10T16:26:20.520 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 439591 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3053: done (3 left) 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3054: done (2 left) 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3055: done (1 left) 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3056: done (0 left) 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3057: write (excl) oid 50 current snap is 329 2024-09-10T16:26:20.974 INFO:tasks.rados.rados.0.smithi165.stdout:3057: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2024-09-10T16:26:20.980 INFO:tasks.rados.rados.0.smithi165.stdout:3057: writing smithi16534841-50 from 699455 to 1129060 tid 1 2024-09-10T16:26:20.986 INFO:tasks.rados.rados.0.smithi165.stdout:3057: writing smithi16534841-50 from 1804617 to 2380651 tid 2 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3057: writing smithi16534841-50 from 3163252 to 3678181 tid 3 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3058: read oid 21 snap -1 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3058: expect (ObjNum 1054 snap 323 seq_num 1054) 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3059: snap_create 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3057: finishing write tid 1 to smithi16534841-50 2024-09-10T16:26:20.994 INFO:tasks.rados.rados.0.smithi165.stdout:3057: finishing write tid 2 to smithi16534841-50 2024-09-10T16:26:20.997 INFO:tasks.rados.rados.0.smithi165.stdout:3057: finishing write tid 3 to smithi16534841-50 2024-09-10T16:26:20.997 INFO:tasks.rados.rados.0.smithi165.stdout:3057: finishing write tid 4 to smithi16534841-50 2024-09-10T16:26:20.997 INFO:tasks.rados.rados.0.smithi165.stdout:3057: finishing write tid 5 to smithi16534841-50 2024-09-10T16:26:20.997 INFO:tasks.rados.rados.0.smithi165.stdout:3057: oid 50 updating version 0 to 1860 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:3057: oid 50 updating version 1860 to 1861 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:3057: oid 50 updating version 1861 to 1862 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:3057: oid 50 version 1862 is already newer than 1859 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1862 (ObjNum 1065 snap 329 seq_num 1065) dirty exists 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:3057: left oid 50 (ObjNum 1065 snap 329 seq_num 1065) 2024-09-10T16:26:20.998 INFO:tasks.rados.rados.0.smithi165.stdout:3058: expect (ObjNum 1054 snap 323 seq_num 1054) 2024-09-10T16:26:21.983 INFO:tasks.rados.rados.0.smithi165.stdout:3057: done (2 left) 2024-09-10T16:26:21.983 INFO:tasks.rados.rados.0.smithi165.stdout:3058: done (1 left) 2024-09-10T16:26:21.983 INFO:tasks.rados.rados.0.smithi165.stdout:3059: done (0 left) 2024-09-10T16:26:21.984 INFO:tasks.rados.rados.0.smithi165.stdout:3060: read oid 6 snap -1 2024-09-10T16:26:21.984 INFO:tasks.rados.rados.0.smithi165.stdout:3060: expect deleted 2024-09-10T16:26:21.984 INFO:tasks.rados.rados.0.smithi165.stdout:3061: delete oid 9 current snap is 330 2024-09-10T16:26:21.985 INFO:tasks.rados.rados.0.smithi165.stdout:3061: done (1 left) 2024-09-10T16:26:21.985 INFO:tasks.rados.rados.0.smithi165.stdout:3062: read oid 29 snap -1 2024-09-10T16:26:21.985 INFO:tasks.rados.rados.0.smithi165.stdout:3062: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:26:21.985 INFO:tasks.rados.rados.0.smithi165.stdout:3063: delete oid 20 current snap is 330 2024-09-10T16:26:21.989 INFO:tasks.rados.rados.0.smithi165.stdout:3062: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:26:22.012 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1931858 2024-09-10T16:26:22.048 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 328426 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3060: done (2 left) 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3062: done (1 left) 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3063: done (0 left) 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3064: read oid 18 snap -1 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3064: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:26:22.159 INFO:tasks.rados.rados.0.smithi165.stdout:3065: read oid 12 snap -1 2024-09-10T16:26:22.160 INFO:tasks.rados.rados.0.smithi165.stdout:3065: expect (ObjNum 1016 snap 308 seq_num 1016) 2024-09-10T16:26:22.160 INFO:tasks.rados.rados.0.smithi165.stdout:3066: delete oid 30 current snap is 330 2024-09-10T16:26:22.161 INFO:tasks.rados.rados.0.smithi165.stdout:3064: expect (ObjNum 1030 snap 313 seq_num 1030) 2024-09-10T16:26:22.188 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2218532 2024-09-10T16:26:22.313 INFO:tasks.rados.rados.0.smithi165.stdout:3065: expect (ObjNum 1016 snap 308 seq_num 1016) 2024-09-10T16:26:22.386 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:22.551 INFO:tasks.rados.rados.0.smithi165.stdout:3064: done (2 left) 2024-09-10T16:26:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:3065: done (1 left) 2024-09-10T16:26:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:3066: done (0 left) 2024-09-10T16:26:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:3067: read oid 6 snap 327 2024-09-10T16:26:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:3067: expect deleted 2024-09-10T16:26:22.552 INFO:tasks.rados.rados.0.smithi165.stdout:3068: delete oid 24 current snap is 330 2024-09-10T16:26:22.554 INFO:tasks.rados.rados.0.smithi165.stdout:3067: done (1 left) 2024-09-10T16:26:22.554 INFO:tasks.rados.rados.0.smithi165.stdout:3068: done (0 left) 2024-09-10T16:26:22.554 INFO:tasks.rados.rados.0.smithi165.stdout:3069: snap_remove snap 326 2024-09-10T16:26:22.948 INFO:tasks.rados.rados.0.smithi165.stdout:3069: done (0 left) 2024-09-10T16:26:22.948 INFO:tasks.rados.rados.0.smithi165.stdout:3070: rollback oid 16 current snap is 330 2024-09-10T16:26:22.948 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 319 2024-09-10T16:26:22.948 INFO:tasks.rados.rados.0.smithi165.stdout:3071: delete oid 46 current snap is 330 2024-09-10T16:26:22.957 INFO:tasks.rados.rados.0.smithi165.stdout:3071: done (1 left) 2024-09-10T16:26:22.957 INFO:tasks.rados.rados.0.smithi165.stdout:3072: delete oid 17 current snap is 330 2024-09-10T16:26:22.960 INFO:tasks.rados.rados.0.smithi165.stdout:3072: done (1 left) 2024-09-10T16:26:22.960 INFO:tasks.rados.rados.0.smithi165.stdout:3073: delete oid 34 current snap is 330 2024-09-10T16:26:22.965 INFO:tasks.rados.rados.0.smithi165.stdout:3070: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:26:22.965 INFO:tasks.rados.rados.0.smithi165.stdout:3070: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:26:22.968 INFO:tasks.rados.rados.0.smithi165.stdout:3070: finishing rollback tid 2 to smithi16534841-16 2024-09-10T16:26:22.968 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 1892 (ObjNum 1020 snap 310 seq_num 1020) dirty exists 2024-09-10T16:26:22.970 INFO:tasks.rados.rados.0.smithi165.stdout:3070: done (1 left) 2024-09-10T16:26:22.970 INFO:tasks.rados.rados.0.smithi165.stdout:3073: done (0 left) 2024-09-10T16:26:22.970 INFO:tasks.rados.rados.0.smithi165.stdout:3074: write (excl) oid 36 current snap is 330 2024-09-10T16:26:22.970 INFO:tasks.rados.rados.0.smithi165.stdout:3074: seq_num 1073 ranges {433911=747950,1756120=277791} 2024-09-10T16:26:22.976 INFO:tasks.rados.rados.0.smithi165.stdout:3074: writing smithi16534841-36 from 433911 to 1181861 tid 1 2024-09-10T16:26:22.981 INFO:tasks.rados.rados.0.smithi165.stdout:3074: writing smithi16534841-36 from 1756120 to 2033911 tid 2 2024-09-10T16:26:22.981 INFO:tasks.rados.rados.0.smithi165.stdout:3075: write oid 8 current snap is 330 2024-09-10T16:26:22.981 INFO:tasks.rados.rados.0.smithi165.stdout:3075: seq_num 1074 ranges {450718=423199,1414602=636116} 2024-09-10T16:26:22.985 INFO:tasks.rados.rados.0.smithi165.stdout:3075: writing smithi16534841-8 from 450718 to 873917 tid 1 2024-09-10T16:26:22.992 INFO:tasks.rados.rados.0.smithi165.stdout:3075: writing smithi16534841-8 from 1414602 to 2050718 tid 2 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:3076: rollback oid 15 current snap is 330 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 325 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:3077: read oid 4 snap -1 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:3077: expect (ObjNum 1033 snap 314 seq_num 1033) 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:3078: write oid 44 current snap is 330 2024-09-10T16:26:22.993 INFO:tasks.rados.rados.0.smithi165.stdout:3078: seq_num 1075 ranges {467525=498448,1473084=594379} 2024-09-10T16:26:22.998 INFO:tasks.rados.rados.0.smithi165.stdout:3078: writing smithi16534841-44 from 467525 to 965973 tid 1 2024-09-10T16:26:23.005 INFO:tasks.rados.rados.0.smithi165.stdout:3078: writing smithi16534841-44 from 1473084 to 2067463 tid 2 2024-09-10T16:26:23.005 INFO:tasks.rados.rados.0.smithi165.stdout:3079: delete oid 39 current snap is 330 2024-09-10T16:26:23.005 INFO:tasks.rados.rados.0.smithi165.stdout:3074: finishing write tid 1 to smithi16534841-36 2024-09-10T16:26:23.005 INFO:tasks.rados.rados.0.smithi165.stdout:3074: finishing write tid 2 to smithi16534841-36 2024-09-10T16:26:23.005 INFO:tasks.rados.rados.0.smithi165.stdout:3074: finishing write tid 3 to smithi16534841-36 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3074: finishing write tid 4 to smithi16534841-36 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3074: oid 36 updating version 0 to 1400 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3074: oid 36 updating version 1400 to 1401 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3074: oid 36 updating version 1401 to 1402 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1402 (ObjNum 1073 snap 330 seq_num 1073) dirty exists 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3074: left oid 36 (ObjNum 1073 snap 330 seq_num 1073) 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: finishing write tid 1 to smithi16534841-8 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: finishing write tid 2 to smithi16534841-8 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: finishing write tid 3 to smithi16534841-8 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: finishing write tid 4 to smithi16534841-8 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: oid 8 updating version 0 to 1664 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: oid 8 version 1664 is already newer than 1662 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:3075: oid 8 version 1664 is already newer than 1663 2024-09-10T16:26:23.006 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1664 (ObjNum 1074 snap 330 seq_num 1074) dirty exists 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3075: left oid 8 (ObjNum 1074 snap 330 seq_num 1074) 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3076: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3076: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3076: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1439 (ObjNum 995 snap 299 seq_num 995) dirty exists 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3078: finishing write tid 1 to smithi16534841-44 2024-09-10T16:26:23.007 INFO:tasks.rados.rados.0.smithi165.stdout:3077: expect (ObjNum 1033 snap 314 seq_num 1033) 2024-09-10T16:26:23.119 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2530866 2024-09-10T16:26:23.147 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 11356 2024-09-10T16:26:23.225 INFO:tasks.rados.rados.0.smithi165.stdout:3078: finishing write tid 2 to smithi16534841-44 2024-09-10T16:26:23.225 INFO:tasks.rados.rados.0.smithi165.stdout:3078: finishing write tid 3 to smithi16534841-44 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3078: finishing write tid 4 to smithi16534841-44 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3078: oid 44 updating version 0 to 1176 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3078: oid 44 version 1176 is already newer than 1175 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3078: oid 44 version 1176 is already newer than 1174 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1176 (ObjNum 1075 snap 330 seq_num 1075) dirty exists 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3078: left oid 44 (ObjNum 1075 snap 330 seq_num 1075) 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3074: done (5 left) 2024-09-10T16:26:23.226 INFO:tasks.rados.rados.0.smithi165.stdout:3075: done (4 left) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3076: done (3 left) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3077: done (2 left) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3078: done (1 left) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3079: done (0 left) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3080: read oid 50 snap -1 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3080: expect (ObjNum 1065 snap 329 seq_num 1065) 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:3081: rollback oid 10 current snap is 330 2024-09-10T16:26:23.227 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 325 2024-09-10T16:26:23.228 INFO:tasks.rados.rados.0.smithi165.stdout:3082: snap_remove snap 328 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3082: done (2 left) 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3083: read oid 3 snap -1 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3083: expect (ObjNum 1058 snap 326 seq_num 1058) 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3081: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1576 (ObjNum 257 snap 0 seq_num 1162160480) dirty dne 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3084: read oid 39 snap 323 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3084: expect (ObjNum 1034 snap 314 seq_num 1034) 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3081: done (3 left) 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:3085: rollback oid 44 current snap is 330 2024-09-10T16:26:24.591 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 327 2024-09-10T16:26:24.592 INFO:tasks.rados.rados.0.smithi165.stdout:3080: expect (ObjNum 1065 snap 329 seq_num 1065) 2024-09-10T16:26:24.592 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:24.653 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:24.769 INFO:tasks.rados.rados.0.smithi165.stdout:3080: done (3 left) 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:3086: copy_from oid 9 from oid 27 current snap is 330 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:3085: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:3085: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:3085: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1179 (ObjNum 962 snap 292 seq_num 962) dirty exists 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:3083: expect (ObjNum 1058 snap 326 seq_num 1058) 2024-09-10T16:26:24.770 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:24.949 INFO:tasks.rados.rados.0.smithi165.stdout:3084: expect (ObjNum 1034 snap 314 seq_num 1034) 2024-09-10T16:26:25.057 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:25.182 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1318987 2024-09-10T16:26:25.267 INFO:tasks.rados.rados.0.smithi165.stdout:3086: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:26:25.267 INFO:tasks.rados.rados.0.smithi165.stdout:3086: finishing copy_from to smithi16534841-9 2024-09-10T16:26:25.267 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1654 (ObjNum 1044 snap 321 seq_num 1044) dirty exists 2024-09-10T16:26:25.267 INFO:tasks.rados.rados.0.smithi165.stdout:3083: done (3 left) 2024-09-10T16:26:25.268 INFO:tasks.rados.rados.0.smithi165.stdout:3084: done (2 left) 2024-09-10T16:26:25.268 INFO:tasks.rados.rados.0.smithi165.stdout:3085: done (1 left) 2024-09-10T16:26:25.268 INFO:tasks.rados.rados.0.smithi165.stdout:3086: done (0 left) 2024-09-10T16:26:25.268 INFO:tasks.rados.rados.0.smithi165.stdout:3087: copy_from oid 39 from oid 23 current snap is 330 2024-09-10T16:26:25.268 INFO:tasks.rados.rados.0.smithi165.stdout:3088: snap_remove snap 329 2024-09-10T16:26:25.592 INFO:tasks.rados.rados.0.smithi165.stdout:3088: done (1 left) 2024-09-10T16:26:25.592 INFO:tasks.rados.rados.0.smithi165.stdout:3089: write (excl) oid 22 current snap is 330 2024-09-10T16:26:25.593 INFO:tasks.rados.rados.0.smithi165.stdout:3087: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:26:25.593 INFO:tasks.rados.rados.0.smithi165.stdout:3089: seq_num 1077 ranges {501139=648946,1873695=227444} 2024-09-10T16:26:25.599 INFO:tasks.rados.rados.0.smithi165.stdout:3089: writing smithi16534841-22 from 501139 to 1150085 tid 1 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:3089: writing smithi16534841-22 from 1873695 to 2101139 tid 2 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:3090: rollback oid 17 current snap is 330 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 312 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:3087: finishing copy_from to smithi16534841-39 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1685 (ObjNum 1056 snap 323 seq_num 1056) dirty exists 2024-09-10T16:26:25.605 INFO:tasks.rados.rados.0.smithi165.stdout:3089: finishing write tid 1 to smithi16534841-22 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3087: done (2 left) 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3091: read oid 13 snap 325 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3091: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3092: read oid 36 snap -1 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3092: expect (ObjNum 1073 snap 330 seq_num 1073) 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3093: delete oid 38 current snap is 330 2024-09-10T16:26:25.606 INFO:tasks.rados.rados.0.smithi165.stdout:3090: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:26:25.607 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 0 (ObjNum 257 snap 0 seq_num 1162188688) dirty dne 2024-09-10T16:26:25.613 INFO:tasks.rados.rados.0.smithi165.stdout:3092: expect (ObjNum 1073 snap 330 seq_num 1073) 2024-09-10T16:26:25.649 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:25.686 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 499369 2024-09-10T16:26:25.715 INFO:tasks.rados.rados.0.smithi165.stdout:3089: finishing write tid 2 to smithi16534841-22 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: finishing write tid 3 to smithi16534841-22 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: finishing write tid 4 to smithi16534841-22 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: oid 22 updating version 0 to 1443 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: oid 22 updating version 1443 to 1445 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: oid 22 version 1445 is already newer than 1444 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1445 (ObjNum 1077 snap 330 seq_num 1077) dirty exists 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3089: left oid 22 (ObjNum 1077 snap 330 seq_num 1077) 2024-09-10T16:26:25.716 INFO:tasks.rados.rados.0.smithi165.stdout:3091: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:26:25.959 INFO:tasks.rados.rados.0.smithi165.stdout:3089: done (4 left) 2024-09-10T16:26:25.959 INFO:tasks.rados.rados.0.smithi165.stdout:3090: done (3 left) 2024-09-10T16:26:25.959 INFO:tasks.rados.rados.0.smithi165.stdout:3091: done (2 left) 2024-09-10T16:26:25.960 INFO:tasks.rados.rados.0.smithi165.stdout:3092: done (1 left) 2024-09-10T16:26:25.960 INFO:tasks.rados.rados.0.smithi165.stdout:3093: done (0 left) 2024-09-10T16:26:25.960 INFO:tasks.rados.rados.0.smithi165.stdout:3094: write oid 47 current snap is 330 2024-09-10T16:26:25.960 INFO:tasks.rados.rados.0.smithi165.stdout:3094: seq_num 1079 ranges {534753=799444,1990659=144094} 2024-09-10T16:26:25.965 INFO:tasks.rados.rados.0.smithi165.stdout:3094: writing smithi16534841-47 from 534753 to 1334197 tid 1 2024-09-10T16:26:25.970 INFO:tasks.rados.rados.0.smithi165.stdout:3094: writing smithi16534841-47 from 1990659 to 2134753 tid 2 2024-09-10T16:26:25.970 INFO:tasks.rados.rados.0.smithi165.stdout:3095: copy_from oid 14 from oid 29 current snap is 330 2024-09-10T16:26:25.970 INFO:tasks.rados.rados.0.smithi165.stdout:3096: rollback oid 43 current snap is 330 2024-09-10T16:26:25.970 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 323 2024-09-10T16:26:25.970 INFO:tasks.rados.rados.0.smithi165.stdout:3097: copy_from oid 32 from oid 38 current snap is 330 2024-09-10T16:26:25.971 INFO:tasks.rados.rados.0.smithi165.stdout:3094: finishing write tid 1 to smithi16534841-47 2024-09-10T16:26:25.971 INFO:tasks.rados.rados.0.smithi165.stdout:3098: snap_remove snap 324 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3098: done (4 left) 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3099: read oid 6 snap -1 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3099: expect deleted 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3100: write oid 21 current snap is 330 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3097: finishing copy_from to smithi16534841-32 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3097: got expected ENOENT (src dne) 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3097: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3096: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:26:26.604 INFO:tasks.rados.rados.0.smithi165.stdout:3100: seq_num 1080 ranges {551560=591046,1765494=386066} 2024-09-10T16:26:26.609 INFO:tasks.rados.rados.0.smithi165.stdout:3100: writing smithi16534841-21 from 551560 to 1142606 tid 1 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3100: writing smithi16534841-21 from 1765494 to 2151560 tid 2 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3097: done (5 left) 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3101: rollback oid 37 current snap is 330 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 319 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3102: snap_create 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3096: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3096: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1693 (ObjNum 939 snap 285 seq_num 939) dirty exists 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3094: finishing write tid 2 to smithi16534841-47 2024-09-10T16:26:26.616 INFO:tasks.rados.rados.0.smithi165.stdout:3094: finishing write tid 3 to smithi16534841-47 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3094: finishing write tid 4 to smithi16534841-47 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3094: oid 47 updating version 0 to 1671 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3094: oid 47 version 1671 is already newer than 1669 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3094: oid 47 version 1671 is already newer than 1670 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1671 (ObjNum 1079 snap 330 seq_num 1079) dirty exists 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3094: left oid 47 (ObjNum 1079 snap 330 seq_num 1079) 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3095: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:3095: finishing copy_from to smithi16534841-14 2024-09-10T16:26:26.618 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1576 (ObjNum 946 snap 285 seq_num 946) dirty exists 2024-09-10T16:26:26.621 INFO:tasks.rados.rados.0.smithi165.stdout:3100: finishing write tid 1 to smithi16534841-21 2024-09-10T16:26:26.621 INFO:tasks.rados.rados.0.smithi165.stdout:3100: finishing write tid 2 to smithi16534841-21 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: finishing write tid 3 to smithi16534841-21 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: finishing write tid 4 to smithi16534841-21 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: oid 21 updating version 0 to 1816 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: oid 21 updating version 1816 to 1817 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: oid 21 version 1817 is already newer than 1815 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1817 (ObjNum 1080 snap 330 seq_num 1080) dirty exists 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3100: left oid 21 (ObjNum 1080 snap 330 seq_num 1080) 2024-09-10T16:26:26.622 INFO:tasks.rados.rados.0.smithi165.stdout:3101: finishing rollback tid 0 to smithi16534841-37 2024-09-10T16:26:26.623 INFO:tasks.rados.rados.0.smithi165.stdout:3101: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:26:26.623 INFO:tasks.rados.rados.0.smithi165.stdout:3101: finishing rollback tid 2 to smithi16534841-37 2024-09-10T16:26:26.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1583 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:26:27.602 INFO:tasks.rados.rados.0.smithi165.stdout:3094: done (6 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3095: done (5 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3096: done (4 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3099: done (3 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3100: done (2 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3101: done (1 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3102: done (0 left) 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3103: rollback oid 19 current snap is 331 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 19 to 330 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3104: write (excl) oid 41 current snap is 331 2024-09-10T16:26:27.603 INFO:tasks.rados.rados.0.smithi165.stdout:3104: seq_num 1081 ranges {568367=666295,1707623=460744} 2024-09-10T16:26:27.609 INFO:tasks.rados.rados.0.smithi165.stdout:3104: writing smithi16534841-41 from 568367 to 1234662 tid 1 2024-09-10T16:26:27.615 INFO:tasks.rados.rados.0.smithi165.stdout:3104: writing smithi16534841-41 from 1707623 to 2168367 tid 2 2024-09-10T16:26:27.615 INFO:tasks.rados.rados.0.smithi165.stdout:3105: read oid 29 snap 327 2024-09-10T16:26:27.615 INFO:tasks.rados.rados.0.smithi165.stdout:3105: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:26:27.615 INFO:tasks.rados.rados.0.smithi165.stdout:3106: snap_remove snap 325 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3106: done (3 left) 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3107: read oid 38 snap -1 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3107: expect deleted 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3108: copy_from oid 27 from oid 28 current snap is 331 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3103: finishing rollback tid 0 to smithi16534841-19 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3103: finishing rollback tid 1 to smithi16534841-19 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3103: finishing rollback tid 2 to smithi16534841-19 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 1910 (ObjNum 999 snap 299 seq_num 999) dirty exists 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3104: finishing write tid 1 to smithi16534841-41 2024-09-10T16:26:27.951 INFO:tasks.rados.rados.0.smithi165.stdout:3103: done (4 left) 2024-09-10T16:26:27.952 INFO:tasks.rados.rados.0.smithi165.stdout:3109: snap_remove snap 323 2024-09-10T16:26:28.957 INFO:tasks.rados.rados.0.smithi165.stdout:3109: done (4 left) 2024-09-10T16:26:28.957 INFO:tasks.rados.rados.0.smithi165.stdout:3110: write oid 11 current snap is 331 2024-09-10T16:26:28.957 INFO:tasks.rados.rados.0.smithi165.stdout:3110: seq_num 1082 ranges {585174=741544,1766105=419069} 2024-09-10T16:26:28.964 INFO:tasks.rados.rados.0.smithi165.stdout:3110: writing smithi16534841-11 from 585174 to 1326718 tid 1 2024-09-10T16:26:28.970 INFO:tasks.rados.rados.0.smithi165.stdout:3110: writing smithi16534841-11 from 1766105 to 2185174 tid 2 2024-09-10T16:26:28.970 INFO:tasks.rados.rados.0.smithi165.stdout:3111: write (excl) oid 13 current snap is 331 2024-09-10T16:26:28.970 INFO:tasks.rados.rados.0.smithi165.stdout:3111: seq_num 1083 ranges {601981=416793,1424587=492702} 2024-09-10T16:26:28.975 INFO:tasks.rados.rados.0.smithi165.stdout:3111: writing smithi16534841-13 from 601981 to 1018774 tid 1 2024-09-10T16:26:28.981 INFO:tasks.rados.rados.0.smithi165.stdout:3111: writing smithi16534841-13 from 1424587 to 1917289 tid 2 2024-09-10T16:26:28.981 INFO:tasks.rados.rados.0.smithi165.stdout:3112: write oid 2 current snap is 331 2024-09-10T16:26:28.981 INFO:tasks.rados.rados.0.smithi165.stdout:3104: finishing write tid 2 to smithi16534841-41 2024-09-10T16:26:28.981 INFO:tasks.rados.rados.0.smithi165.stdout:3105: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:26:29.111 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 732786 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: finishing write tid 3 to smithi16534841-41 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: finishing write tid 4 to smithi16534841-41 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: oid 41 updating version 0 to 1692 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: oid 41 version 1692 is already newer than 1691 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: oid 41 version 1692 is already newer than 1690 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1692 (ObjNum 1081 snap 331 seq_num 1081) dirty exists 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3104: left oid 41 (ObjNum 1081 snap 331 seq_num 1081) 2024-09-10T16:26:29.168 INFO:tasks.rados.rados.0.smithi165.stdout:3112: seq_num 1084 ranges {618788=492042,1883069=335719} 2024-09-10T16:26:29.173 INFO:tasks.rados.rados.0.smithi165.stdout:3112: writing smithi16534841-2 from 618788 to 1110830 tid 1 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3112: writing smithi16534841-2 from 1883069 to 2218788 tid 2 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3104: done (6 left) 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3105: done (5 left) 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3113: snap_create 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3108: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3108: finishing copy_from to smithi16534841-27 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1280 (ObjNum 1048 snap 323 seq_num 1048) dirty exists 2024-09-10T16:26:29.180 INFO:tasks.rados.rados.0.smithi165.stdout:3110: finishing write tid 1 to smithi16534841-11 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: finishing write tid 2 to smithi16534841-11 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: finishing write tid 3 to smithi16534841-11 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: finishing write tid 4 to smithi16534841-11 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: oid 11 updating version 0 to 1863 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: oid 11 updating version 1863 to 1864 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: oid 11 updating version 1864 to 1865 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1865 (ObjNum 1082 snap 331 seq_num 1082) dirty exists 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3110: left oid 11 (ObjNum 1082 snap 331 seq_num 1082) 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3111: finishing write tid 1 to smithi16534841-13 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3111: finishing write tid 2 to smithi16534841-13 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3111: finishing write tid 3 to smithi16534841-13 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3111: finishing write tid 4 to smithi16534841-13 2024-09-10T16:26:29.181 INFO:tasks.rados.rados.0.smithi165.stdout:3111: oid 13 updating version 0 to 1448 2024-09-10T16:26:29.182 INFO:tasks.rados.rados.0.smithi165.stdout:3111: oid 13 version 1448 is already newer than 1447 2024-09-10T16:26:29.182 INFO:tasks.rados.rados.0.smithi165.stdout:3111: oid 13 version 1448 is already newer than 1446 2024-09-10T16:26:29.182 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1448 (ObjNum 1083 snap 331 seq_num 1083) dirty exists 2024-09-10T16:26:29.182 INFO:tasks.rados.rados.0.smithi165.stdout:3111: left oid 13 (ObjNum 1083 snap 331 seq_num 1083) 2024-09-10T16:26:29.182 INFO:tasks.rados.rados.0.smithi165.stdout:3112: finishing write tid 1 to smithi16534841-2 2024-09-10T16:26:29.184 INFO:tasks.rados.rados.0.smithi165.stdout:3112: finishing write tid 2 to smithi16534841-2 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: finishing write tid 3 to smithi16534841-2 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: finishing write tid 4 to smithi16534841-2 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: oid 2 updating version 0 to 1686 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: oid 2 updating version 1686 to 1687 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: oid 2 updating version 1687 to 1688 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1688 (ObjNum 1084 snap 331 seq_num 1084) dirty exists 2024-09-10T16:26:29.186 INFO:tasks.rados.rados.0.smithi165.stdout:3112: left oid 2 (ObjNum 1084 snap 331 seq_num 1084) 2024-09-10T16:26:29.959 INFO:tasks.rados.rados.0.smithi165.stdout:3107: done (5 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3108: done (4 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3110: done (3 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3111: done (2 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3112: done (1 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3113: done (0 left) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3114: read oid 15 snap 311 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3114: expect (ObjNum 998 snap 299 seq_num 998) 2024-09-10T16:26:29.960 INFO:tasks.rados.rados.0.smithi165.stdout:3115: snap_remove snap 327 2024-09-10T16:26:30.961 INFO:tasks.rados.rados.0.smithi165.stdout:3115: done (1 left) 2024-09-10T16:26:30.962 INFO:tasks.rados.rados.0.smithi165.stdout:3116: read oid 20 snap -1 2024-09-10T16:26:30.962 INFO:tasks.rados.rados.0.smithi165.stdout:3116: expect deleted 2024-09-10T16:26:30.962 INFO:tasks.rados.rados.0.smithi165.stdout:3117: read oid 33 snap 311 2024-09-10T16:26:30.962 INFO:tasks.rados.rados.0.smithi165.stdout:3117: expect deleted 2024-09-10T16:26:30.962 INFO:tasks.rados.rados.0.smithi165.stdout:3118: delete oid 12 current snap is 332 2024-09-10T16:26:30.963 INFO:tasks.rados.rados.0.smithi165.stdout:3114: expect (ObjNum 998 snap 299 seq_num 998) 2024-09-10T16:26:30.967 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1561331 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3114: done (3 left) 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3116: done (2 left) 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3118: done (1 left) 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3119: copy_from oid 44 from oid 18 current snap is 332 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3117: done (1 left) 2024-09-10T16:26:31.068 INFO:tasks.rados.rados.0.smithi165.stdout:3120: write (excl) oid 35 current snap is 332 2024-09-10T16:26:31.069 INFO:tasks.rados.rados.0.smithi165.stdout:3120: seq_num 1086 ranges {652402=642540,1883680=368722} 2024-09-10T16:26:31.075 INFO:tasks.rados.rados.0.smithi165.stdout:3120: writing smithi16534841-35 from 652402 to 1294942 tid 1 2024-09-10T16:26:31.080 INFO:tasks.rados.rados.0.smithi165.stdout:3120: writing smithi16534841-35 from 1883680 to 2252402 tid 2 2024-09-10T16:26:31.080 INFO:tasks.rados.rados.0.smithi165.stdout:3121: write oid 28 current snap is 332 2024-09-10T16:26:31.081 INFO:tasks.rados.rados.0.smithi165.stdout:3119: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:26:31.081 INFO:tasks.rados.rados.0.smithi165.stdout:3121: seq_num 1087 ranges {669209=717789,1942162=327047} 2024-09-10T16:26:31.088 INFO:tasks.rados.rados.0.smithi165.stdout:3121: writing smithi16534841-28 from 669209 to 1386998 tid 1 2024-09-10T16:26:31.093 INFO:tasks.rados.rados.0.smithi165.stdout:3121: writing smithi16534841-28 from 1942162 to 2269209 tid 2 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3122: read oid 26 snap -1 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3122: expect deleted 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3123: write oid 40 current snap is 332 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: finishing write tid 1 to smithi16534841-35 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3119: finishing copy_from to smithi16534841-44 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1674 (ObjNum 1030 snap 313 seq_num 1030) dirty exists 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: finishing write tid 2 to smithi16534841-35 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: finishing write tid 3 to smithi16534841-35 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: finishing write tid 4 to smithi16534841-35 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: oid 35 updating version 0 to 1449 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: oid 35 updating version 1449 to 1450 2024-09-10T16:26:31.094 INFO:tasks.rados.rados.0.smithi165.stdout:3120: oid 35 updating version 1450 to 1451 2024-09-10T16:26:31.095 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1451 (ObjNum 1086 snap 332 seq_num 1086) dirty exists 2024-09-10T16:26:31.095 INFO:tasks.rados.rados.0.smithi165.stdout:3120: left oid 35 (ObjNum 1086 snap 332 seq_num 1086) 2024-09-10T16:26:31.095 INFO:tasks.rados.rados.0.smithi165.stdout:3121: finishing write tid 1 to smithi16534841-28 2024-09-10T16:26:31.095 INFO:tasks.rados.rados.0.smithi165.stdout:3123: seq_num 1088 ranges {686016=509391,1716997=569019} 2024-09-10T16:26:31.099 INFO:tasks.rados.rados.0.smithi165.stdout:3123: writing smithi16534841-40 from 686016 to 1195407 tid 1 2024-09-10T16:26:31.106 INFO:tasks.rados.rados.0.smithi165.stdout:3123: writing smithi16534841-40 from 1716997 to 2286016 tid 2 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3119: done (4 left) 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3120: done (3 left) 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3124: read oid 39 snap -1 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3124: expect (ObjNum 1056 snap 323 seq_num 1056) 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3125: read oid 50 snap 312 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3125: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3126: delete oid 11 current snap is 332 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3121: finishing write tid 2 to smithi16534841-28 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3121: finishing write tid 3 to smithi16534841-28 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3121: finishing write tid 4 to smithi16534841-28 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3121: oid 28 updating version 0 to 1283 2024-09-10T16:26:31.107 INFO:tasks.rados.rados.0.smithi165.stdout:3121: oid 28 version 1283 is already newer than 1282 2024-09-10T16:26:31.108 INFO:tasks.rados.rados.0.smithi165.stdout:3121: oid 28 version 1283 is already newer than 1281 2024-09-10T16:26:31.108 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1283 (ObjNum 1087 snap 332 seq_num 1087) dirty exists 2024-09-10T16:26:31.108 INFO:tasks.rados.rados.0.smithi165.stdout:3121: left oid 28 (ObjNum 1087 snap 332 seq_num 1087) 2024-09-10T16:26:31.108 INFO:tasks.rados.rados.0.smithi165.stdout:3123: finishing write tid 1 to smithi16534841-40 2024-09-10T16:26:31.109 INFO:tasks.rados.rados.0.smithi165.stdout:3123: finishing write tid 2 to smithi16534841-40 2024-09-10T16:26:31.109 INFO:tasks.rados.rados.0.smithi165.stdout:3123: finishing write tid 3 to smithi16534841-40 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:3123: finishing write tid 4 to smithi16534841-40 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:3123: oid 40 updating version 0 to 1657 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:3123: oid 40 version 1657 is already newer than 1656 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:3123: oid 40 version 1657 is already newer than 1655 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1657 (ObjNum 1088 snap 332 seq_num 1088) dirty exists 2024-09-10T16:26:31.110 INFO:tasks.rados.rados.0.smithi165.stdout:3123: left oid 40 (ObjNum 1088 snap 332 seq_num 1088) 2024-09-10T16:26:31.111 INFO:tasks.rados.rados.0.smithi165.stdout:3125: expect (ObjNum 975 snap 297 seq_num 975) 2024-09-10T16:26:31.111 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:31.207 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1573795 2024-09-10T16:26:31.234 INFO:tasks.rados.rados.0.smithi165.stdout:3124: expect (ObjNum 1056 snap 323 seq_num 1056) 2024-09-10T16:26:31.326 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:31.428 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:31.530 INFO:tasks.rados.rados.0.smithi165.stdout:3121: done (5 left) 2024-09-10T16:26:31.530 INFO:tasks.rados.rados.0.smithi165.stdout:3122: done (4 left) 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3123: done (3 left) 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3124: done (2 left) 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3125: done (1 left) 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3126: done (0 left) 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3127: rollback oid 15 current snap is 332 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 311 2024-09-10T16:26:31.531 INFO:tasks.rados.rados.0.smithi165.stdout:3128: snap_remove snap 330 2024-09-10T16:26:31.970 INFO:tasks.rados.rados.0.smithi165.stdout:3128: done (1 left) 2024-09-10T16:26:31.970 INFO:tasks.rados.rados.0.smithi165.stdout:3129: write (excl) oid 29 current snap is 332 2024-09-10T16:26:31.970 INFO:tasks.rados.rados.0.smithi165.stdout:3129: seq_num 1090 ranges {719630=659889,2117608=202022} 2024-09-10T16:26:31.977 INFO:tasks.rados.rados.0.smithi165.stdout:3129: writing smithi16534841-29 from 719630 to 1379519 tid 1 2024-09-10T16:26:31.983 INFO:tasks.rados.rados.0.smithi165.stdout:3129: writing smithi16534841-29 from 2117608 to 2319630 tid 2 2024-09-10T16:26:31.983 INFO:tasks.rados.rados.0.smithi165.stdout:3130: write oid 23 current snap is 332 2024-09-10T16:26:31.983 INFO:tasks.rados.rados.0.smithi165.stdout:3127: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:26:31.983 INFO:tasks.rados.rados.0.smithi165.stdout:3130: seq_num 1091 ranges {736437=735138,2176090=160347} 2024-09-10T16:26:31.990 INFO:tasks.rados.rados.0.smithi165.stdout:3130: writing smithi16534841-23 from 736437 to 1471575 tid 1 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:3130: writing smithi16534841-23 from 2176090 to 2336437 tid 2 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:3131: rollback oid 18 current snap is 332 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 18 to 311 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:3127: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:3127: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:26:31.996 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1442 (ObjNum 998 snap 299 seq_num 998) dirty exists 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: finishing write tid 1 to smithi16534841-29 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: finishing write tid 2 to smithi16534841-29 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: finishing write tid 3 to smithi16534841-29 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: finishing write tid 4 to smithi16534841-29 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: oid 29 updating version 0 to 1697 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: oid 29 version 1697 is already newer than 1696 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: oid 29 updating version 1697 to 1698 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1698 (ObjNum 1090 snap 332 seq_num 1090) dirty exists 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: left oid 29 (ObjNum 1090 snap 332 seq_num 1090) 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3127: done (3 left) 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3129: done (2 left) 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3132: copy_from oid 8 from oid 6 current snap is 332 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3133: write oid 24 current snap is 332 2024-09-10T16:26:31.997 INFO:tasks.rados.rados.0.smithi165.stdout:3133: seq_num 1092 ranges {753244=410387,1834572=518672} 2024-09-10T16:26:32.002 INFO:tasks.rados.rados.0.smithi165.stdout:3133: writing smithi16534841-24 from 753244 to 1163631 tid 1 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3133: writing smithi16534841-24 from 1834572 to 2353244 tid 2 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3134: copy_from oid 21 from oid 14 current snap is 332 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3132: finishing copy_from to smithi16534841-8 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3132: got expected ENOENT (src dne) 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3132: finishing copy_from racing read to smithi16534841-8 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: finishing write tid 1 to smithi16534841-23 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: finishing write tid 2 to smithi16534841-23 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: finishing write tid 3 to smithi16534841-23 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: finishing write tid 4 to smithi16534841-23 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: oid 23 updating version 0 to 1695 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: oid 23 version 1695 is already newer than 1693 2024-09-10T16:26:32.010 INFO:tasks.rados.rados.0.smithi165.stdout:3130: oid 23 version 1695 is already newer than 1694 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1695 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3130: left oid 23 (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3131: finishing rollback tid 0 to smithi16534841-18 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3131: finishing rollback tid 1 to smithi16534841-18 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3131: finishing rollback tid 2 to smithi16534841-18 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1698 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3133: finishing write tid 1 to smithi16534841-24 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3130: done (4 left) 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3131: done (3 left) 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3132: done (2 left) 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3135: write oid 38 current snap is 332 2024-09-10T16:26:32.011 INFO:tasks.rados.rados.0.smithi165.stdout:3135: seq_num 1093 ranges {770051=485636,1776701=511047} 2024-09-10T16:26:32.016 INFO:tasks.rados.rados.0.smithi165.stdout:3135: writing smithi16534841-38 from 770051 to 1255687 tid 1 2024-09-10T16:26:32.023 INFO:tasks.rados.rados.0.smithi165.stdout:3135: writing smithi16534841-38 from 1776701 to 2287748 tid 2 2024-09-10T16:26:32.023 INFO:tasks.rados.rados.0.smithi165.stdout:3136: read oid 31 snap -1 2024-09-10T16:26:32.023 INFO:tasks.rados.rados.0.smithi165.stdout:3136: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:26:32.023 INFO:tasks.rados.rados.0.smithi165.stdout:3137: write oid 47 current snap is 332 2024-09-10T16:26:32.023 INFO:tasks.rados.rados.0.smithi165.stdout:3133: finishing write tid 2 to smithi16534841-24 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: finishing write tid 3 to smithi16534841-24 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: finishing write tid 4 to smithi16534841-24 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: oid 24 updating version 0 to 1699 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: oid 24 updating version 1699 to 1700 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: oid 24 updating version 1700 to 1701 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1701 (ObjNum 1092 snap 332 seq_num 1092) dirty exists 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3133: left oid 24 (ObjNum 1092 snap 332 seq_num 1092) 2024-09-10T16:26:32.024 INFO:tasks.rados.rados.0.smithi165.stdout:3137: seq_num 1094 ranges {786858=560885,1835183=551675} 2024-09-10T16:26:32.030 INFO:tasks.rados.rados.0.smithi165.stdout:3137: writing smithi16534841-47 from 786858 to 1347743 tid 1 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3137: writing smithi16534841-47 from 1835183 to 2386858 tid 2 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3133: done (4 left) 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3138: snap_create 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3135: finishing write tid 1 to smithi16534841-38 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3135: finishing write tid 2 to smithi16534841-38 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3135: finishing write tid 3 to smithi16534841-38 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3135: finishing write tid 4 to smithi16534841-38 2024-09-10T16:26:32.037 INFO:tasks.rados.rados.0.smithi165.stdout:3135: oid 38 updating version 0 to 1405 2024-09-10T16:26:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:3135: oid 38 version 1405 is already newer than 1404 2024-09-10T16:26:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:3135: oid 38 updating version 1405 to 1406 2024-09-10T16:26:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1406 (ObjNum 1093 snap 332 seq_num 1093) dirty exists 2024-09-10T16:26:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:3135: left oid 38 (ObjNum 1093 snap 332 seq_num 1093) 2024-09-10T16:26:32.038 INFO:tasks.rados.rados.0.smithi165.stdout:3136: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:26:32.138 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2428981 2024-09-10T16:26:32.149 INFO:tasks.rados.rados.0.smithi165.stdout:3134: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:26:32.149 INFO:tasks.rados.rados.0.smithi165.stdout:3134: finishing copy_from to smithi16534841-21 2024-09-10T16:26:32.149 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1576 (ObjNum 946 snap 285 seq_num 946) dirty exists 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: finishing write tid 1 to smithi16534841-47 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: finishing write tid 2 to smithi16534841-47 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: finishing write tid 3 to smithi16534841-47 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: finishing write tid 4 to smithi16534841-47 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: oid 47 updating version 0 to 1705 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: oid 47 version 1705 is already newer than 1703 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: oid 47 version 1705 is already newer than 1704 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1705 (ObjNum 1094 snap 332 seq_num 1094) dirty exists 2024-09-10T16:26:32.150 INFO:tasks.rados.rados.0.smithi165.stdout:3137: left oid 47 (ObjNum 1094 snap 332 seq_num 1094) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3134: done (4 left) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3135: done (3 left) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3136: done (2 left) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3137: done (1 left) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3138: done (0 left) 2024-09-10T16:26:32.951 INFO:tasks.rados.rados.0.smithi165.stdout:3139: delete oid 16 current snap is 333 2024-09-10T16:26:32.955 INFO:tasks.rados.rados.0.smithi165.stdout:3139: done (0 left) 2024-09-10T16:26:32.955 INFO:tasks.rados.rados.0.smithi165.stdout:3140: read oid 50 snap -1 2024-09-10T16:26:32.955 INFO:tasks.rados.rados.0.smithi165.stdout:3140: expect (ObjNum 1065 snap 329 seq_num 1065) 2024-09-10T16:26:32.955 INFO:tasks.rados.rados.0.smithi165.stdout:3141: snap_remove snap 312 2024-09-10T16:26:33.955 INFO:tasks.rados.rados.0.smithi165.stdout:3141: done (1 left) 2024-09-10T16:26:33.956 INFO:tasks.rados.rados.0.smithi165.stdout:3142: copy_from oid 35 from oid 38 current snap is 333 2024-09-10T16:26:33.956 INFO:tasks.rados.rados.0.smithi165.stdout:3140: expect (ObjNum 1065 snap 329 seq_num 1065) 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:3142: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:3142: finishing copy_from to smithi16534841-35 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1406 (ObjNum 1093 snap 332 seq_num 1093) dirty exists 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:3140: done (1 left) 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:3142: done (0 left) 2024-09-10T16:26:34.107 INFO:tasks.rados.rados.0.smithi165.stdout:3143: read oid 16 snap -1 2024-09-10T16:26:34.108 INFO:tasks.rados.rados.0.smithi165.stdout:3143: expect deleted 2024-09-10T16:26:34.108 INFO:tasks.rados.rados.0.smithi165.stdout:3144: read oid 13 snap 319 2024-09-10T16:26:34.108 INFO:tasks.rados.rados.0.smithi165.stdout:3144: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:26:34.108 INFO:tasks.rados.rados.0.smithi165.stdout:3145: write oid 25 current snap is 333 2024-09-10T16:26:34.108 INFO:tasks.rados.rados.0.smithi165.stdout:3145: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2024-09-10T16:26:34.111 INFO:tasks.rados.rados.0.smithi165.stdout:3145: writing smithi16534841-25 from 420472 to 848208 tid 1 2024-09-10T16:26:34.118 INFO:tasks.rados.rados.0.smithi165.stdout:3145: writing smithi16534841-25 from 1268500 to 1926874 tid 2 2024-09-10T16:26:34.121 INFO:tasks.rados.rados.0.smithi165.stdout:3145: writing smithi16534841-25 from 2347953 to 2420472 tid 3 2024-09-10T16:26:34.121 INFO:tasks.rados.rados.0.smithi165.stdout:3146: read oid 35 snap -1 2024-09-10T16:26:34.121 INFO:tasks.rados.rados.0.smithi165.stdout:3146: expect (ObjNum 1093 snap 332 seq_num 1093) 2024-09-10T16:26:34.121 INFO:tasks.rados.rados.0.smithi165.stdout:3147: copy_from oid 3 from oid 4 current snap is 333 2024-09-10T16:26:34.122 INFO:tasks.rados.rados.0.smithi165.stdout:3144: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:26:34.263 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3752751 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: finishing write tid 1 to smithi16534841-25 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: finishing write tid 2 to smithi16534841-25 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: finishing write tid 3 to smithi16534841-25 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: finishing write tid 4 to smithi16534841-25 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: finishing write tid 5 to smithi16534841-25 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: oid 25 updating version 0 to 1818 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: oid 25 updating version 1818 to 1820 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: oid 25 updating version 1820 to 1821 2024-09-10T16:26:34.350 INFO:tasks.rados.rados.0.smithi165.stdout:3145: oid 25 version 1821 is already newer than 1819 2024-09-10T16:26:34.351 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1821 (ObjNum 1096 snap 333 seq_num 1096) dirty exists 2024-09-10T16:26:34.351 INFO:tasks.rados.rados.0.smithi165.stdout:3145: left oid 25 (ObjNum 1096 snap 333 seq_num 1096) 2024-09-10T16:26:34.351 INFO:tasks.rados.rados.0.smithi165.stdout:3146: expect (ObjNum 1093 snap 332 seq_num 1093) 2024-09-10T16:26:34.351 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:34.428 INFO:tasks.rados.rados.0.smithi165.stdout:3147: finishing copy_from racing read to smithi16534841-3 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3147: finishing copy_from to smithi16534841-3 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1436 (ObjNum 1033 snap 314 seq_num 1033) dirty exists 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3143: done (4 left) 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3144: done (3 left) 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3145: done (2 left) 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3146: done (1 left) 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3147: done (0 left) 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3148: write (excl) oid 21 current snap is 333 2024-09-10T16:26:34.429 INFO:tasks.rados.rados.0.smithi165.stdout:3148: seq_num 1097 ranges {437279=502985,1610629=518385} 2024-09-10T16:26:34.433 INFO:tasks.rados.rados.0.smithi165.stdout:3148: writing smithi16534841-21 from 437279 to 940264 tid 1 2024-09-10T16:26:34.440 INFO:tasks.rados.rados.0.smithi165.stdout:3148: writing smithi16534841-21 from 1610629 to 2129014 tid 2 2024-09-10T16:26:34.440 INFO:tasks.rados.rados.0.smithi165.stdout:3149: rollback oid 38 current snap is 333 2024-09-10T16:26:34.440 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 311 2024-09-10T16:26:34.440 INFO:tasks.rados.rados.0.smithi165.stdout:3148: finishing write tid 1 to smithi16534841-21 2024-09-10T16:26:34.440 INFO:tasks.rados.rados.0.smithi165.stdout:3150: delete oid 24 current snap is 333 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: finishing write tid 2 to smithi16534841-21 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: finishing write tid 3 to smithi16534841-21 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: finishing write tid 4 to smithi16534841-21 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: oid 21 updating version 0 to 1823 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: oid 21 version 1823 is already newer than 1822 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: oid 21 updating version 1823 to 1824 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1824 (ObjNum 1097 snap 333 seq_num 1097) dirty exists 2024-09-10T16:26:34.444 INFO:tasks.rados.rados.0.smithi165.stdout:3148: left oid 21 (ObjNum 1097 snap 333 seq_num 1097) 2024-09-10T16:26:34.445 INFO:tasks.rados.rados.0.smithi165.stdout:3149: finishing rollback tid 0 to smithi16534841-38 2024-09-10T16:26:34.445 INFO:tasks.rados.rados.0.smithi165.stdout:3149: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:26:34.445 INFO:tasks.rados.rados.0.smithi165.stdout:3149: finishing rollback tid 2 to smithi16534841-38 2024-09-10T16:26:34.445 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1409 (ObjNum 982 snap 297 seq_num 982) dirty exists 2024-09-10T16:26:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:3148: done (2 left) 2024-09-10T16:26:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:3149: done (1 left) 2024-09-10T16:26:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:3150: done (0 left) 2024-09-10T16:26:34.448 INFO:tasks.rados.rados.0.smithi165.stdout:3151: snap_create 2024-09-10T16:26:34.963 INFO:tasks.rados.rados.0.smithi165.stdout:3151: done (0 left) 2024-09-10T16:26:34.963 INFO:tasks.rados.rados.0.smithi165.stdout:3152: rollback oid 29 current snap is 334 2024-09-10T16:26:34.963 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 311 2024-09-10T16:26:34.963 INFO:tasks.rados.rados.0.smithi165.stdout:3153: write oid 28 current snap is 334 2024-09-10T16:26:34.963 INFO:tasks.rados.rados.0.smithi165.stdout:3153: seq_num 1099 ranges {470893=653483,1727593=522054} 2024-09-10T16:26:34.969 INFO:tasks.rados.rados.0.smithi165.stdout:3153: writing smithi16534841-28 from 470893 to 1124376 tid 1 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3153: writing smithi16534841-28 from 1727593 to 2249647 tid 2 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3154: rollback oid 12 current snap is 334 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 319 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3152: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3152: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3152: finishing rollback tid 2 to smithi16534841-29 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1719 (ObjNum 946 snap 285 seq_num 946) dirty exists 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3152: done (2 left) 2024-09-10T16:26:34.976 INFO:tasks.rados.rados.0.smithi165.stdout:3155: snap_create 2024-09-10T16:26:34.978 INFO:tasks.rados.rados.0.smithi165.stdout:3153: finishing write tid 1 to smithi16534841-28 2024-09-10T16:26:34.979 INFO:tasks.rados.rados.0.smithi165.stdout:3153: finishing write tid 2 to smithi16534841-28 2024-09-10T16:26:34.979 INFO:tasks.rados.rados.0.smithi165.stdout:3153: finishing write tid 3 to smithi16534841-28 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3153: finishing write tid 4 to smithi16534841-28 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3153: oid 28 updating version 0 to 1285 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3153: oid 28 updating version 1285 to 1286 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3153: oid 28 version 1286 is already newer than 1284 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1286 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3153: left oid 28 (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3154: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:3154: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:26:34.980 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1635 (ObjNum 1016 snap 308 seq_num 1016) dirty exists 2024-09-10T16:26:35.966 INFO:tasks.rados.rados.0.smithi165.stdout:3153: done (2 left) 2024-09-10T16:26:35.966 INFO:tasks.rados.rados.0.smithi165.stdout:3154: done (1 left) 2024-09-10T16:26:35.967 INFO:tasks.rados.rados.0.smithi165.stdout:3155: done (0 left) 2024-09-10T16:26:35.967 INFO:tasks.rados.rados.0.smithi165.stdout:3156: write oid 12 current snap is 335 2024-09-10T16:26:35.967 INFO:tasks.rados.rados.0.smithi165.stdout:3156: seq_num 1100 ranges {487700=728732,1786075=665712} 2024-09-10T16:26:35.972 INFO:tasks.rados.rados.0.smithi165.stdout:3156: writing smithi16534841-12 from 487700 to 1216432 tid 1 2024-09-10T16:26:35.980 INFO:tasks.rados.rados.0.smithi165.stdout:3156: writing smithi16534841-12 from 1786075 to 2451787 tid 2 2024-09-10T16:26:35.980 INFO:tasks.rados.rados.0.smithi165.stdout:3157: snap_create 2024-09-10T16:26:35.980 INFO:tasks.rados.rados.0.smithi165.stdout:3156: finishing write tid 1 to smithi16534841-12 2024-09-10T16:26:35.984 INFO:tasks.rados.rados.0.smithi165.stdout:3156: finishing write tid 2 to smithi16534841-12 2024-09-10T16:26:35.984 INFO:tasks.rados.rados.0.smithi165.stdout:3156: finishing write tid 3 to smithi16534841-12 2024-09-10T16:26:35.984 INFO:tasks.rados.rados.0.smithi165.stdout:3156: finishing write tid 4 to smithi16534841-12 2024-09-10T16:26:35.984 INFO:tasks.rados.rados.0.smithi165.stdout:3156: oid 12 updating version 0 to 1638 2024-09-10T16:26:35.984 INFO:tasks.rados.rados.0.smithi165.stdout:3156: oid 12 version 1638 is already newer than 1637 2024-09-10T16:26:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:3156: oid 12 version 1638 is already newer than 1636 2024-09-10T16:26:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1638 (ObjNum 1100 snap 335 seq_num 1100) dirty exists 2024-09-10T16:26:35.985 INFO:tasks.rados.rados.0.smithi165.stdout:3156: left oid 12 (ObjNum 1100 snap 335 seq_num 1100) 2024-09-10T16:26:36.968 INFO:tasks.rados.rados.0.smithi165.stdout:3156: done (1 left) 2024-09-10T16:26:36.968 INFO:tasks.rados.rados.0.smithi165.stdout:3157: done (0 left) 2024-09-10T16:26:36.968 INFO:tasks.rados.rados.0.smithi165.stdout:3158: delete oid 45 current snap is 336 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3158: done (0 left) 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3159: read oid 7 snap -1 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3159: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3160: rollback oid 31 current snap is 336 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 332 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3161: rollback oid 32 current snap is 336 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 32 to 335 2024-09-10T16:26:36.974 INFO:tasks.rados.rados.0.smithi165.stdout:3162: write oid 30 current snap is 336 2024-09-10T16:26:36.975 INFO:tasks.rados.rados.0.smithi165.stdout:3162: seq_num 1102 ranges {521314=479230,1786686=669381} 2024-09-10T16:26:36.979 INFO:tasks.rados.rados.0.smithi165.stdout:3162: writing smithi16534841-30 from 521314 to 1000544 tid 1 2024-09-10T16:26:36.987 INFO:tasks.rados.rados.0.smithi165.stdout:3162: writing smithi16534841-30 from 1786686 to 2456067 tid 2 2024-09-10T16:26:36.987 INFO:tasks.rados.rados.0.smithi165.stdout:3163: write oid 12 current snap is 336 2024-09-10T16:26:36.987 INFO:tasks.rados.rados.0.smithi165.stdout:3163: seq_num 1103 ranges {538121=670832,1961521=413039} 2024-09-10T16:26:36.994 INFO:tasks.rados.rados.0.smithi165.stdout:3163: writing smithi16534841-12 from 538121 to 1208953 tid 1 2024-09-10T16:26:37.001 INFO:tasks.rados.rados.0.smithi165.stdout:3163: writing smithi16534841-12 from 1961521 to 2374560 tid 2 2024-09-10T16:26:37.001 INFO:tasks.rados.rados.0.smithi165.stdout:3164: write (excl) oid 35 current snap is 336 2024-09-10T16:26:37.001 INFO:tasks.rados.rados.0.smithi165.stdout:3164: seq_num 1104 ranges {554928=746081,2020003=534925} 2024-09-10T16:26:37.008 INFO:tasks.rados.rados.0.smithi165.stdout:3164: writing smithi16534841-35 from 554928 to 1301009 tid 1 2024-09-10T16:26:37.016 INFO:tasks.rados.rados.0.smithi165.stdout:3164: writing smithi16534841-35 from 2020003 to 2554928 tid 2 2024-09-10T16:26:37.016 INFO:tasks.rados.rados.0.smithi165.stdout:3165: delete oid 18 current snap is 336 2024-09-10T16:26:37.016 INFO:tasks.rados.rados.0.smithi165.stdout:3161: finishing rollback tid 1 to smithi16534841-32 2024-09-10T16:26:37.016 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 0 (ObjNum 225 snap 0 seq_num 1162195056) dirty dne 2024-09-10T16:26:37.016 INFO:tasks.rados.rados.0.smithi165.stdout:3159: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:26:37.037 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:3160: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:26:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:3160: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:26:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:3160: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:26:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 1950 (ObjNum 881 snap 269 seq_num 881) dirty exists 2024-09-10T16:26:37.110 INFO:tasks.rados.rados.0.smithi165.stdout:3162: finishing write tid 1 to smithi16534841-30 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: finishing write tid 2 to smithi16534841-30 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: finishing write tid 3 to smithi16534841-30 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: finishing write tid 4 to smithi16534841-30 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: oid 30 updating version 0 to 1953 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: oid 30 version 1953 is already newer than 1951 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: oid 30 version 1953 is already newer than 1952 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1953 (ObjNum 1102 snap 336 seq_num 1102) dirty exists 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3162: left oid 30 (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3163: finishing write tid 1 to smithi16534841-12 2024-09-10T16:26:37.111 INFO:tasks.rados.rados.0.smithi165.stdout:3159: done (6 left) 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3160: done (5 left) 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3161: done (4 left) 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3162: done (3 left) 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3165: done (2 left) 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3166: delete oid 4 current snap is 336 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: finishing write tid 2 to smithi16534841-12 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: finishing write tid 3 to smithi16534841-12 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: finishing write tid 4 to smithi16534841-12 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: oid 12 updating version 0 to 1639 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: oid 12 updating version 1639 to 1640 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:3163: oid 12 updating version 1640 to 1641 2024-09-10T16:26:37.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1641 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3163: left oid 12 (ObjNum 1103 snap 336 seq_num 1103) 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: finishing write tid 1 to smithi16534841-35 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: finishing write tid 2 to smithi16534841-35 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: finishing write tid 3 to smithi16534841-35 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: finishing write tid 4 to smithi16534841-35 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: oid 35 updating version 0 to 1454 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: oid 35 version 1454 is already newer than 1453 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: oid 35 version 1454 is already newer than 1452 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1454 (ObjNum 1104 snap 336 seq_num 1104) dirty exists 2024-09-10T16:26:37.113 INFO:tasks.rados.rados.0.smithi165.stdout:3164: left oid 35 (ObjNum 1104 snap 336 seq_num 1104) 2024-09-10T16:26:37.114 INFO:tasks.rados.rados.0.smithi165.stdout:3163: done (2 left) 2024-09-10T16:26:37.114 INFO:tasks.rados.rados.0.smithi165.stdout:3164: done (1 left) 2024-09-10T16:26:37.114 INFO:tasks.rados.rados.0.smithi165.stdout:3166: done (0 left) 2024-09-10T16:26:37.114 INFO:tasks.rados.rados.0.smithi165.stdout:3167: read oid 18 snap -1 2024-09-10T16:26:37.114 INFO:tasks.rados.rados.0.smithi165.stdout:3167: expect deleted 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3168: copy_from oid 20 from oid 46 current snap is 336 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3169: read oid 22 snap 331 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3169: expect (ObjNum 1077 snap 330 seq_num 1077) 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3170: snap_create 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3168: finishing copy_from to smithi16534841-20 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3168: got expected ENOENT (src dne) 2024-09-10T16:26:37.115 INFO:tasks.rados.rados.0.smithi165.stdout:3168: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:26:37.118 INFO:tasks.rados.rados.0.smithi165.stdout:3169: expect (ObjNum 1077 snap 330 seq_num 1077) 2024-09-10T16:26:37.143 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1641161 2024-09-10T16:26:37.154 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:26:37.953 INFO:tasks.rados.rados.0.smithi165.stdout:3167: done (3 left) 2024-09-10T16:26:37.953 INFO:tasks.rados.rados.0.smithi165.stdout:3168: done (2 left) 2024-09-10T16:26:37.953 INFO:tasks.rados.rados.0.smithi165.stdout:3169: done (1 left) 2024-09-10T16:26:37.953 INFO:tasks.rados.rados.0.smithi165.stdout:3170: done (0 left) 2024-09-10T16:26:37.953 INFO:tasks.rados.rados.0.smithi165.stdout:3171: snap_create 2024-09-10T16:26:38.953 INFO:tasks.rados.rados.0.smithi165.stdout:3171: done (0 left) 2024-09-10T16:26:38.953 INFO:tasks.rados.rados.0.smithi165.stdout:3172: write oid 48 current snap is 338 2024-09-10T16:26:38.953 INFO:tasks.rados.rados.0.smithi165.stdout:3172: seq_num 1107 ranges {605349=571828,1679096=420377} 2024-09-10T16:26:38.959 INFO:tasks.rados.rados.0.smithi165.stdout:3172: writing smithi16534841-48 from 605349 to 1177177 tid 1 2024-09-10T16:26:38.964 INFO:tasks.rados.rados.0.smithi165.stdout:3172: writing smithi16534841-48 from 1679096 to 2099473 tid 2 2024-09-10T16:26:38.964 INFO:tasks.rados.rados.0.smithi165.stdout:3173: snap_remove snap 337 2024-09-10T16:26:39.621 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:26:39.621 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:26:39.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:26:39.625 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:26:39.626 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T16:26:39.626 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T16:26:39.626 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T16:26:54.627 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T16:26:54.628 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T16:26:54.628 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:26:54.628 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.734+0000 7f5898989640 10 monclient: get_monmap_and_config 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.734+0000 7f5898989640 10 monclient: build_initial_monmap 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.734+0000 7f5898989640 10 monclient: monmap: 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:26:54.735529+0000 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:26:54.735529+0000 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:26:54.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:26:54.737 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:26:54.737 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:26:54.737 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:26:54.737 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 Processor -- start 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 -- start start 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient: init 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient: _reopen_session rank -1 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-10T16:26:54.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): picked mon.noname-a con 0x7f589409c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): picked mon.noname-b con 0x7f589409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f589409f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589409a430 con 0x7f589409f4c0 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f58940a24e0 con 0x7f589409c180 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f58940a2bf0 con 0x7f589409d160 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:54.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5898989640 10 monclient(hunting): authenticate will time out at 1993.222773s 2024-09-10T16:26:54.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:54.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:54.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46232/0 (socket says 172.21.15.164:46232) 2024-09-10T16:26:54.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5892d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f589409f4c0 0x7f589413b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:40588/0 (socket says 172.21.15.164:40588) 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 learned_addr learned my addr 172.21.15.164:0/22981639 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51854/0 (socket says 172.21.15.164:51854) 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f589409c180 auth_method 0 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f589259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f589409d160 auth_method 0 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.735+0000 7f5889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:54.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.736+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:54.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.738+0000 7f5892d9d640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.164:6789/0 conn(0x7f589409f4c0 legacy=0x7f589413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:54.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f589409d160 auth_method 0 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f589409c180 auth_method 0 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:54.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:54.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:54.937+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:55.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:55.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:55.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f589409d160 auth_method 0 2024-09-10T16:26:55.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f589409c180 auth_method 0 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:55.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:55.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:55.338+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:56.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:56.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:56.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f589409c180 auth_method 0 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f589409d160 auth_method 0 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.139+0000 7f5889d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:56.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:56.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:56.140+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:57.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): tick 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 monclient(hunting): continuing hunt 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 msgr2=0x7f589409ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f589409d160 0x7f589409ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 msgr2=0x7f589409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f589409c180 0x7f589409ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.164:6789/0 conn(0x7f589409f4c0 legacy=0x7f589413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient: _add_conns ranks=[4,1] 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:26:57.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): picked mon.noname-c con 0x7f5878003740 addr v2:172.21.15.165:3300/0 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58780061f0 addr v1:172.21.15.165:6789/0 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589409c560 con 0x7f58780061f0 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): start opening mon connection 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f58940a24e0 con 0x7f5878003740 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.735+0000 7f5890d99640 10 monclient(hunting): _renew_subs 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f5878003740 auth_method 0 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:57.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:26:57.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.738+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 legacy=0x7f58780086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:26:57.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f5878003740 auth_method 0 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:57.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:57.938+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:26:58.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f5878003740 auth_method 0 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:58.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:58.340+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:26:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:26:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f5878003740 auth_method 0 2024-09-10T16:26:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:26:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:26:59.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:26:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:26:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:26:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:26:59.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.141+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:26:59.647 INFO:tasks.rados.rados.0.smithi165.stdout:3173: done (1 left) 2024-09-10T16:26:59.647 INFO:tasks.rados.rados.0.smithi165.stdout:3174: snap_create 2024-09-10T16:26:59.647 INFO:tasks.rados.rados.0.smithi165.stdout:3172: finishing write tid 1 to smithi16534841-48 2024-09-10T16:26:59.647 INFO:tasks.rados.rados.0.smithi165.stdout:3172: finishing write tid 2 to smithi16534841-48 2024-09-10T16:26:59.647 INFO:tasks.rados.rados.0.smithi165.stdout:3172: finishing write tid 3 to smithi16534841-48 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:3172: finishing write tid 4 to smithi16534841-48 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:3172: oid 48 updating version 0 to 1956 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:3172: oid 48 updating version 1956 to 1957 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:3172: oid 48 version 1957 is already newer than 1955 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1957 (ObjNum 1107 snap 338 seq_num 1107) dirty exists 2024-09-10T16:26:59.648 INFO:tasks.rados.rados.0.smithi165.stdout:3172: left oid 48 (ObjNum 1107 snap 338 seq_num 1107) 2024-09-10T16:26:59.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.723+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 legacy=0x7f58780086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T16:26:59.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.723+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 legacy=0x7f58780086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:26:59.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.723+0000 7f5889d9b640 1 --1- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 0x7f58780086b0 :-1 s=OPENED pgs=20 cs=1 l=1).handle_message read tag failed 2024-09-10T16:26:59.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.723+0000 7f5889d9b640 1 --1- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 0x7f58780086b0 :-1 s=OPENED pgs=20 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:26:59.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:26:59.723+0000 7f5891d9b640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:27:00.662 INFO:tasks.rados.rados.0.smithi165.stdout:3172: done (1 left) 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3174: done (0 left) 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3175: read oid 33 snap -1 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3175: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3176: read oid 26 snap -1 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3176: expect deleted 2024-09-10T16:27:00.663 INFO:tasks.rados.rados.0.smithi165.stdout:3177: snap_create 2024-09-10T16:27:00.670 INFO:tasks.rados.rados.0.smithi165.stdout:3175: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:27:00.675 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 956527 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): tick 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 monclient(hunting): continuing hunt 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 msgr2=0x7f5878005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878003740 0x7f5878005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.165:6789/0 conn(0x7f58780061f0 legacy=0x7f58780086b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:00.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5892d9d640 1 -- 172.21.15.164:0/22981639 reap_dead start 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient: _add_conns ranks=[4,3,0,1,2] 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): picked mon.noname-c con 0x7f5878009fa0 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f587800a450 0x7f587800a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): picked mon.noname-a con 0x7f587800a450 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58780086b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f587800afc0 con 0x7f58780086b0 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f58940a24e0 con 0x7f587800a450 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:00.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 1 -- 172.21.15.164:0/22981639 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f58940a2bf0 con 0x7f5878009fa0 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5890d99640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.164:3300/0 conn(0x7f587800a450 0x7f587800a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f587800a450 auth_method 0 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f5878009fa0 auth_method 0 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 msgr2=0x7f5878005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 msgr2=0x7f5878005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient(hunting): handle_auth_done global_id 224166 payload 274 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f589259c640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 msgr2=0x7f5878005b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> v1:172.21.15.164:6789/0 conn(0x7f58780086b0 legacy=0x7f5878008e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 monclient: found mon.noname-a 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.736+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5878009660 con 0x7f587800a450 2024-09-10T16:27:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.739+0000 7f5889d9b640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 msgr2=0x7f587800a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.739+0000 7f5889d9b640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f587800a8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5878006190 tx=0x7f587c02aea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88fbc8edf09c36da server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 1 -- 172.21.15.164:0/22981639 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c031070 con 0x7f587800a450 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: dump: 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: _finish_auth 0 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 1 -- 172.21.15.164:0/22981639 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f587c02b9b0 con 0x7f587800a450 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 1 -- 172.21.15.164:0/22981639 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f587c02bd80 con 0x7f587800a450 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 5 monclient: authenticate success, global_id 224166 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 4 set_mon_vals no callback set 2024-09-10T16:27:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5891d9b640 10 monclient: dump: 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 10 monclient: shutdown 2024-09-10T16:27:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 1 -- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 msgr2=0x7f587800a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.740+0000 7f5898989640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f587800a8e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5878006190 tx=0x7f587c02aea0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 -- 172.21.15.164:0/22981639 shutdown_connections 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 --2- 172.21.15.164:0/22981639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f587800a8e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 --2- 172.21.15.164:0/22981639 >> v2:172.21.15.165:3300/0 conn(0x7f5878009fa0 0x7f5878005b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 -- 172.21.15.164:0/22981639 >> 172.21.15.164:0/22981639 conn(0x7f58940a3110 msgr2=0x7f58940a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 -- 172.21.15.164:0/22981639 shutdown_connections 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.741+0000 7f5898989640 1 -- 172.21.15.164:0/22981639 wait complete. 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.742+0000 7f5898989640 10 monclient: build_initial_monmap 2024-09-10T16:27:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.742+0000 7f5898989640 10 monclient: monmap: 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 Processor -- start 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 -- start start 2024-09-10T16:27:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient: init 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): picked mon.noname-c con 0x7f58780086b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): picked mon.noname-b con 0x7f5878009fa0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f5894071840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): picked mon.noname-a con 0x7f587800a450 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5894074f90 con 0x7f587800a450 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5894074e10 con 0x7f5878009fa0 2024-09-10T16:27:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f589409d930 con 0x7f58780086b0 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5898989640 10 monclient(hunting): authenticate will time out at 1999.230656s 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51878/0 (socket says 172.21.15.164:51878) 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 1 -- 172.21.15.164:0/2068323817 learned_addr learned my addr 172.21.15.164:0/2068323817 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f5894071840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): get_auth_request con 0x7f58780086b0 auth_method 0 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): get_auth_request con 0x7f587800a450 auth_method 0 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 10 monclient(hunting): get_auth_request con 0x7f5878009fa0 auth_method 0 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:00.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f589259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5892d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 msgr2=0x7f5894071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.743+0000 7f5889d9b640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 msgr2=0x7f5894071300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5889d9b640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5889d9b640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 10 monclient(hunting): handle_auth_done global_id 224169 payload 824 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 msgr2=0x7f5894073800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 msgr2=0x7f5894071300 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 monclient: found mon.noname-a 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f5892d9d640 1 -- 172.21.15.164:0/2068323817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58941384c0 con 0x7f587800a450 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.744+0000 7f589259c640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5892d9d640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 msgr2=0x7f5894071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5892d9d640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f5894071840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f589409f8a0 tx=0x7f588c00cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c017070 con 0x7f587800a450 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: dump: 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c00ce50 con 0x7f587800a450 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 5 monclient: authenticate success, global_id 224169 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 10 monclient: _renew_subs 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f589409d620 con 0x7f587800a450 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c00a7b0 con 0x7f587800a450 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f588b7fe640 10 monclient: dump: 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:00.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 10 monclient: _renew_subs 2024-09-10T16:27:00.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f589159a640 4 set_mon_vals no callback set 2024-09-10T16:27:00.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.746+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5894137ae0 con 0x7f587800a450 2024-09-10T16:27:00.763 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.747+0000 7f5898989640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.747+0000 7f5898989640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.747+0000 7f5898989640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.747+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f588c01a070 con 0x7f587800a450 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.747+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f589409dcd0 con 0x7f587800a450 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.748+0000 7f588b7fe640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 0x7f58780424d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.748+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(704..704 src has 494..704) ==== 4755+0+0 (secure 0 0 0) 0x7f588c051f20 con 0x7f587800a450 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.748+0000 7f589259c640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 0x7f58780424d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.748+0000 7f589259c640 10 monclient: get_auth_request con 0x7f5878040010 auth_method 0 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.751+0000 7f589259c640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 0x7f58780424d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5884006a90 tx=0x7f588401c040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:00.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.753+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f58941384c0 con 0x7f587800a450 2024-09-10T16:27:00.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.753+0000 7f588b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:00.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.753+0000 7f588b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:00.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f5898989640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f5898989640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f5898989640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5894077260 con 0x7f587800a450 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f588b7fe640 1 -- 172.21.15.164:0/2068323817 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f588c010300 con 0x7f587800a450 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f588b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.995+0000 7f588b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:00.997 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":202,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 msgr2=0x7f58780424d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 0x7f58780424d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5884006a90 tx=0x7f588401c040 comp rx=0 tx=0).stop 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 10 monclient: shutdown 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 msgr2=0x7f5894071840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:00.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f5894071840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f589409f8a0 tx=0x7f588c00cb10 comp rx=0 tx=0).stop 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 shutdown_connections 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5878040010 0x7f58780424d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f587800a450 0x7f5894071840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5878009fa0 0x7f5894071300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 --2- 172.21.15.164:0/2068323817 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58780086b0 0x7f5894073800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 >> 172.21.15.164:0/2068323817 conn(0x7f58940a3110 msgr2=0x7f589409dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 shutdown_connections 2024-09-10T16:27:01.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:00.997+0000 7f5898989640 1 -- 172.21.15.164:0/2068323817 wait complete. 2024-09-10T16:27:01.009 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:27:01.010 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:27:01.010 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:27:01.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.106+0000 7f4e901a3640 10 monclient: get_monmap_and_config 2024-09-10T16:27:01.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.106+0000 7f4e901a3640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.106+0000 7f4e901a3640 10 monclient: monmap: 2024-09-10T16:27:01.108 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.108 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:01.107482+0000 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:01.107482+0000 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 Processor -- start 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 -- start start 2024-09-10T16:27:01.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient: init 2024-09-10T16:27:01.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e8809d930 0x7f4e8809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8809d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8809ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4e88134580 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e880a0870 con 0x7f4e88134580 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f4e8809d3e0 con 0x7f4e8809d930 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e8809e790 con 0x7f4e8809ab50 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e901a3640 10 monclient(hunting): authenticate will time out at 1999.594837s 2024-09-10T16:27:01.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e8d73e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e8df3f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4e8809d930 0x7f4e8809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e8d73e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51882/0 (socket says 172.21.15.164:51882) 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e8d73e640 1 -- 172.21.15.164:0/1199621326 learned_addr learned my addr 172.21.15.164:0/1199621326 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.107+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request con 0x7f4e8809d930 auth_method 0 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): get_auth_request con 0x7f4e8809ab50 auth_method 0 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_done global_id 224172 payload 274 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 1 -- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 msgr2=0x7f4e8809e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 1 -- 172.21.15.164:0/1199621326 >> v1:172.21.15.164:6789/0 conn(0x7f4e88134580 legacy=0x7f4e88134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 1 monclient: found mon.noname-a 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8df3f640 1 -- 172.21.15.164:0/1199621326 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e88074c70 con 0x7f4e8809d930 2024-09-10T16:27:01.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.108+0000 7f4e8d73e640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:01.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8df3f640 1 -- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 msgr2=0x7f4e8809dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e8809dd10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097a0 tx=0x7f4e8402a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eafc0d99e7d57fb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 1 -- 172.21.15.164:0/1199621326 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84037070 con 0x7f4e8809d930 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: dump: 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.122 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 1 -- 172.21.15.164:0/1199621326 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e84032440 con 0x7f4e8809d930 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 5 monclient: authenticate success, global_id 224172 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 1 -- 172.21.15.164:0/1199621326 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84032810 con 0x7f4e8809d930 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 4 set_mon_vals no callback set 2024-09-10T16:27:01.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e8cf3d640 10 monclient: dump: 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 10 monclient: shutdown 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 -- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 msgr2=0x7f4e8809dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e8809dd10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f4e840097a0 tx=0x7f4e8402a040 comp rx=0 tx=0).stop 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 -- 172.21.15.164:0/1199621326 shutdown_connections 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e8809ab50 0x7f4e8809e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 --2- 172.21.15.164:0/1199621326 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e8809dd10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.111+0000 7f4e901a3640 1 -- 172.21.15.164:0/1199621326 >> 172.21.15.164:0/1199621326 conn(0x7f4e880a3110 msgr2=0x7f4e880a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 1 -- 172.21.15.164:0/1199621326 shutdown_connections 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 1 -- 172.21.15.164:0/1199621326 wait complete. 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 10 monclient: monmap: 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.125 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 1 Processor -- start 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 1 -- start start 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.112+0000 7f4e901a3640 10 monclient: init 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 0x7f4e8812ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-b con 0x7f4e8809ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e881ca120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-a con 0x7f4e8809d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e88134580 0x7f4e88133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): picked mon.noname-c con 0x7f4e88134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8809ef30 con 0x7f4e8809d930 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880a02b0 con 0x7f4e8809ab50 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4e880a2940 con 0x7f4e88134580 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e901a3640 10 monclient(hunting): authenticate will time out at 1999.600419s 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e881ca120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e88134580 0x7f4e88133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e881ca120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46290/0 (socket says 172.21.15.164:46290) 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 -- 172.21.15.164:0/2122164333 learned_addr learned my addr 172.21.15.164:0/2122164333 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): get_auth_request con 0x7f4e8809d930 auth_method 0 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): get_auth_request con 0x7f4e88134580 auth_method 0 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 0x7f4e8812ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request con 0x7f4e8809ab50 auth_method 0 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 msgr2=0x7f4e8812ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:01.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient(hunting): handle_auth_done global_id 224175 payload 824 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 msgr2=0x7f4e8812ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 0x7f4e8812ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 0x7f4e8812ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e88134580 msgr2=0x7f4e88133c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e88134580 0x7f4e88133c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 msgr2=0x7f4e8812ba90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4e8809ab50 0x7f4e8812ba90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 monclient: found mon.noname-a 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8d73e640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e88074360 con 0x7f4e8809d930 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.113+0000 7f4e8e740640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4e88134580 0x7f4e88133c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:01.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e8d73e640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 msgr2=0x7f4e881ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e8d73e640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e881ca120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800e950 tx=0x7f4e7800ee10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e7800b040 con 0x7f4e8809d930 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: dump: 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.131 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e78004030 con 0x7f4e8809d930 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 5 monclient: authenticate success, global_id 224175 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 10 monclient: _renew_subs 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8812c9f0 con 0x7f4e8809d930 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e780043e0 con 0x7f4e8809d930 2024-09-10T16:27:01.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7e7fc640 10 monclient: dump: 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e7ffff640 4 set_mon_vals no callback set 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 10 monclient: _renew_subs 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.116+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e881ca660 con 0x7f4e8809d930 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4e78003610 con 0x7f4e8809d930 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e901a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e901a3640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 0x7f4e881cd270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e880a1e40 con 0x7f4e88134170 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 0x7f4e881cd270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e7e7fc640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e6c03e210 0x7f4e6c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient: get_auth_request con 0x7f4e88134170 auth_method 0 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(704..704 src has 494..704) ==== 4755+0+0 (secure 0 0 0) 0x7f4e78052f00 con 0x7f4e8809d930 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_done global_id 224178 payload 824 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8e740640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e6c03e210 0x7f4e6c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e8806dac0 con 0x7f4e88134170 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.117+0000 7f4e8e740640 10 monclient: get_auth_request con 0x7f4e6c03e210 auth_method 0 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 msgr2=0x7f4e881cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 0x7f4e881cd270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809a180 tx=0x7f4e840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84042070 con 0x7f4e88134170 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e7e7fc640 10 monclient: dump: 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.119+0000 7f4e8e740640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e6c03e210 0x7f4e6c0406d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4e88131be0 tx=0x7f4e80003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4e8403d670 con 0x7f4e88134170 2024-09-10T16:27:01.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:01.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:01.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 msgr2=0x7f4e881cd270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 0x7f4e881cd270 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809a180 tx=0x7f4e840046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 msgr2=0x7f4e881cd270 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:01.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.120+0000 7f4e7e7fc640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e88134170 0x7f4e881cd270 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4e8809a180 tx=0x7f4e840046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e901a3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e901a3640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 0x7f4e881cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e901a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4e881ce0d0 con 0x7f4e881cd7b0 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 0x7f4e881cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient: get_auth_request con 0x7f4e881cd7b0 auth_method 0 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 10 monclient(hunting): handle_auth_done global_id 224181 payload 824 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.126+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e88077260 con 0x7f4e881cd7b0 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.128+0000 7f4e8df3f640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 msgr2=0x7f4e881cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.128+0000 7f4e8df3f640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 0x7f4e881cdb90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e84008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e84037040 con 0x7f4e881cd7b0 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 10 monclient: dump: 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.140 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f4e84051460 con 0x7f4e881cd7b0 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 msgr2=0x7f4e881cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 0x7f4e881cdb90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4e84000c00 tx=0x7f4e84008030 comp rx=0 tx=0).stop 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 msgr2=0x7f4e881cdb90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:01.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e7e7fc640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e881cd7b0 0x7f4e881cdb90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 72960, 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:01.142 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.143 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.144 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:01.145 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.146 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:01.147 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.148 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.149 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.150 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.151 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.152 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.153 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:01.154 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e6c03e210 msgr2=0x7f4e6c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4e6c03e210 0x7f4e6c0406d0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f4e88131be0 tx=0x7f4e80003d10 comp rx=0 tx=0).stop 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 10 monclient: shutdown 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 msgr2=0x7f4e881ca120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 --2- 172.21.15.164:0/2122164333 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4e8809d930 0x7f4e881ca120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800e950 tx=0x7f4e7800ee10 comp rx=0 tx=0).stop 2024-09-10T16:27:01.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e8e740640 1 -- 172.21.15.164:0/2122164333 reap_dead start 2024-09-10T16:27:01.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e8e740640 1 -- 172.21.15.164:0/2122164333 reap_dead start 2024-09-10T16:27:01.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 shutdown_connections 2024-09-10T16:27:01.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 >> 172.21.15.164:0/2122164333 conn(0x7f4e880a3110 msgr2=0x7f4e880a4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 shutdown_connections 2024-09-10T16:27:01.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.129+0000 7f4e901a3640 1 -- 172.21.15.164:0/2122164333 wait complete. 2024-09-10T16:27:01.157 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:01.157 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:27:01.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.254+0000 7faa1c4f6640 10 monclient: get_monmap_and_config 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.254+0000 7faa1c4f6640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.254+0000 7faa1c4f6640 10 monclient: monmap: 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:01.255697+0000 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:01.255697+0000 2024-09-10T16:27:01.256 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:01.257 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 1 Processor -- start 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 1 -- start start 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 10 monclient: init 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.255+0000 7faa1c4f6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 0x7faa140a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-c con 0x7faa1409f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-a con 0x7faa140a2380 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 0x7faa1413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-b con 0x7faa1413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7faa1409fd10 con 0x7faa140a2380 2024-09-10T16:27:01.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faa1409cd20 con 0x7faa1409f930 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa1409a0d0 con 0x7faa1413a5c0 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1c4f6640 10 monclient(hunting): authenticate will time out at 1999.743055s 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 0x7faa1413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46318/0 (socket says 172.21.15.164:46318) 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 1 -- 172.21.15.164:0/4233940553 learned_addr learned my addr 172.21.15.164:0/4233940553 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): get_auth_request con 0x7faa140a2380 auth_method 0 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): get_auth_request con 0x7faa1413a5c0 auth_method 0 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1a292640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 0x7faa140a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient(hunting): handle_auth_done global_id 214268 payload 274 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 -- 172.21.15.164:0/4233940553 >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 msgr2=0x7faa140a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 0x7faa140a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 -- 172.21.15.164:0/4233940553 >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 msgr2=0x7faa140a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 monclient: found mon.noname-b 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1aa93640 1 -- 172.21.15.164:0/4233940553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1409e840 con 0x7faa1413a5c0 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1a292640 10 monclient: get_auth_request con 0x7faa1409f930 auth_method 0 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa19a91640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:01.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.256+0000 7faa1a292640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 0x7faa140a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:01.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1aa93640 1 -- 172.21.15.164:0/4233940553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 msgr2=0x7faa1413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4233940553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 0x7faa1413c9b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa1406d470 tx=0x7faa04029fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9760c2e445b5aa5f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 1 -- 172.21.15.164:0/4233940553 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04037070 con 0x7faa1413a5c0 2024-09-10T16:27:01.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: dump: 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.271 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 1 -- 172.21.15.164:0/4233940553 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa04032440 con 0x7faa1413a5c0 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 5 monclient: authenticate success, global_id 214268 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 1 -- 172.21.15.164:0/4233940553 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04032810 con 0x7faa1413a5c0 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 4 set_mon_vals no callback set 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:01.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa19290640 10 monclient: dump: 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 10 monclient: shutdown 2024-09-10T16:27:01.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4233940553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 msgr2=0x7faa1413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.259+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4233940553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 0x7faa1413c9b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7faa1406d470 tx=0x7faa04029fe0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4233940553 shutdown_connections 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4233940553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa1413a5c0 0x7faa1413c9b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.164:3300/0 conn(0x7faa140a2380 0x7faa140a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4233940553 >> v2:172.21.15.165:3300/0 conn(0x7faa1409f930 0x7faa140a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4233940553 >> 172.21.15.164:0/4233940553 conn(0x7faa140a3110 msgr2=0x7faa140a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4233940553 shutdown_connections 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4233940553 wait complete. 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 10 monclient: monmap: 2024-09-10T16:27:01.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.260+0000 7faa1c4f6640 1 Processor -- start 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 -- start start 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient: init 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-a con 0x7faa1409f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 0x7faa141cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-c con 0x7faa140a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa1413a5c0 0x7faa141cd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): picked mon.noname-b con 0x7faa1413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faa1409d930 con 0x7faa1409f930 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faa1409ee00 con 0x7faa1413a5c0 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa1409c8b0 con 0x7faa140a2380 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1c4f6640 10 monclient(hunting): authenticate will time out at 1999.748541s 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 0x7faa141cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 0x7faa141cd240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51916/0 (socket says 172.21.15.164:51916) 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 1 -- 172.21.15.164:0/4141899880 learned_addr learned my addr 172.21.15.164:0/4141899880 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): get_auth_request con 0x7faa140a2380 auth_method 0 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46334/0 (socket says 172.21.15.164:46334) 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): get_auth_request con 0x7faa1409f930 auth_method 0 2024-09-10T16:27:01.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient(hunting): handle_auth_done global_id 224187 payload 824 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa1413a5c0 0x7faa141cd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 msgr2=0x7faa141cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 0x7faa141cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa1413a5c0 msgr2=0x7faa141cd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa1413a5c0 0x7faa141cd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 monclient: found mon.noname-a 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1a292640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa140a1c70 con 0x7faa1409f930 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1aa93640 10 monclient: get_auth_request con 0x7faa1413a5c0 auth_method 0 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa19a91640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa140a2380 0x7faa141cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.261+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faa1413a5c0 0x7faa141cd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1a292640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 msgr2=0x7faa141ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1a292640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c7f0 tx=0x7faa0c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c015070 con 0x7faa1409f930 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: dump: 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.280 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa0c004220 con 0x7faa1409f930 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 5 monclient: authenticate success, global_id 224187 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 10 monclient: _renew_subs 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa14074470 con 0x7faa1409f930 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0c007830 con 0x7faa1409f930 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa12ffd640 10 monclient: dump: 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa18a8f640 4 set_mon_vals no callback set 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 10 monclient: _renew_subs 2024-09-10T16:27:01.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.264+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa14096200 con 0x7faa1409f930 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1c4f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 0x7faa141c9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa140959d0 con 0x7faa141c97c0 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faa0c0079d0 con 0x7faa1409f930 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 0x7faa141c9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient: get_auth_request con 0x7faa141c97c0 auth_method 0 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 10 monclient(hunting): handle_auth_done global_id 214274 payload 824 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa12ffd640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa0003e210 0x7faa000406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa19a91640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faa14096030 con 0x7faa141c97c0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(704..704 src has 494..704) ==== 4755+0+0 (secure 0 0 0) 0x7faa0c052d20 con 0x7faa1409f930 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa0003e210 0x7faa000406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.265+0000 7faa1aa93640 10 monclient: get_auth_request con 0x7faa0003e210 auth_method 0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa19a91640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 msgr2=0x7faa141c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa19a91640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 0x7faa141c9ba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa08004980 tx=0x7faa0800e8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08019020 con 0x7faa141c97c0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.267+0000 7faa12ffd640 10 monclient: dump: 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faa08015870 con 0x7faa141c97c0 2024-09-10T16:27:01.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 msgr2=0x7faa141c9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 0x7faa141c9ba0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa08004980 tx=0x7faa0800e8c0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 msgr2=0x7faa141c9ba0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa12ffd640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141c97c0 0x7faa141c9ba0 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7faa08004980 tx=0x7faa0800e8c0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.268+0000 7faa1aa93640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa0003e210 0x7faa000406d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa140a2cf0 tx=0x7faa0402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa1c4f6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 0x7faa14071fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa1c4f6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faa140a0760 con 0x7faa141ca0e0 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 0x7faa14071fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient: get_auth_request con 0x7faa141ca0e0 auth_method 0 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 10 monclient(hunting): handle_auth_done global_id 214277 payload 824 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.274+0000 7faa19a91640 1 -- 172.21.15.164:0/4141899880 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faa14077260 con 0x7faa141ca0e0 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa19a91640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 msgr2=0x7faa14071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa19a91640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 0x7faa14071fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa0801a000 tx=0x7faa08004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08020220 con 0x7faa141ca0e0 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 10 monclient: dump: 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.289 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7faa08020220 con 0x7faa141ca0e0 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 msgr2=0x7faa14071fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 0x7faa14071fb0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa0801a000 tx=0x7faa08004a40 comp rx=0 tx=0).stop 2024-09-10T16:27:01.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 msgr2=0x7faa14071fb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.276+0000 7faa12ffd640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faa141ca0e0 0x7faa14071fb0 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faa0801a000 tx=0x7faa08004a40 comp rx=0 tx=0).stop 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 121054, 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:01.291 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.292 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.293 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:01.294 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:01.295 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:01.296 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.298 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:01.299 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.301 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:27:01.302 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:01.303 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:01.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:01.304 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:01.304 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:01.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa0003e210 msgr2=0x7faa000406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faa0003e210 0x7faa000406d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7faa140a2cf0 tx=0x7faa0402e000 comp rx=0 tx=0).stop 2024-09-10T16:27:01.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 10 monclient: shutdown 2024-09-10T16:27:01.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 msgr2=0x7faa141ccd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7faa0c00c7f0 tx=0x7faa0c00ccb0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1aa93640 1 -- 172.21.15.164:0/4141899880 reap_dead start 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 shutdown_connections 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 --2- 172.21.15.164:0/4141899880 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faa1409f930 0x7faa141ccd00 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 >> 172.21.15.164:0/4141899880 conn(0x7faa140a3110 msgr2=0x7faa141388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 shutdown_connections 2024-09-10T16:27:01.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.277+0000 7faa1c4f6640 1 -- 172.21.15.164:0/4141899880 wait complete. 2024-09-10T16:27:01.306 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:01.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:27:01.307 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.404+0000 7f8729b3d640 10 monclient: get_monmap_and_config 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.404+0000 7f8729b3d640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.404+0000 7f8729b3d640 10 monclient: monmap: 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:01.405747+0000 2024-09-10T16:27:01.406 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:01.405747+0000 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:01.407 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.405+0000 7f8729b3d640 1 Processor -- start 2024-09-10T16:27:01.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.405+0000 7f8729b3d640 1 -- start start 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.405+0000 7f8729b3d640 10 monclient: init 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f872409c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 0x7f872409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f872409d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f872413a5c0 0x7f872413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f872413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f872409fbc0 con 0x7f872409c310 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f87240a2a20 con 0x7f872413a5c0 2024-09-10T16:27:01.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f872409c6f0 con 0x7f872409d190 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8729b3d640 10 monclient(hunting): authenticate will time out at 1999.893200s 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 0x7f872409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f87237fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f872409c310 0x7f872409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:40608/0 (socket says 172.21.15.164:40608) 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f87237fe640 1 -- 172.21.15.164:0/3744421064 learned_addr learned my addr 172.21.15.164:0/3744421064 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 --2- 172.21.15.164:0/3744421064 >> v2:172.21.15.164:3300/0 conn(0x7f872413a5c0 0x7f872413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): get_auth_request con 0x7f872413a5c0 auth_method 0 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient(hunting): handle_auth_done global_id 224190 payload 274 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 -- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 msgr2=0x7f872409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 0x7f872409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 -- 172.21.15.164:0/3744421064 >> v1:172.21.15.164:6789/0 conn(0x7f872409c310 legacy=0x7f872409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f8723fff640 1 -- 172.21.15.164:0/3744421064 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f872409a4d0 con 0x7f872413a5c0 2024-09-10T16:27:01.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.406+0000 7f871affd640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 0x7f872409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8723fff640 1 -- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 msgr2=0x7f872413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8723fff640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f872413c9b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f871c009d90 tx=0x7f871c02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20e4d22199ac8739 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 1 -- 172.21.15.164:0/3744421064 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c007950 con 0x7f872413a5c0 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: dump: 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 1 -- 172.21.15.164:0/3744421064 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f871c007af0 con 0x7f872413a5c0 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8729b3d640 5 monclient: authenticate success, global_id 224190 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8729b3d640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8729b3d640 4 set_mon_vals no callback set 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 1 -- 172.21.15.164:0/3744421064 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871c031400 con 0x7f872413a5c0 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.409+0000 7f8722ffd640 10 monclient: dump: 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.421 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 10 monclient: shutdown 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 -- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 msgr2=0x7f872413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f872413c9b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f871c009d90 tx=0x7f871c02a530 comp rx=0 tx=0).stop 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 -- 172.21.15.164:0/3744421064 shutdown_connections 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f872413c9b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 --2- 172.21.15.164:0/3744421064 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f872409d190 0x7f872409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 -- 172.21.15.164:0/3744421064 >> 172.21.15.164:0/3744421064 conn(0x7f87240a3110 msgr2=0x7f87240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:01.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 -- 172.21.15.164:0/3744421064 shutdown_connections 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 -- 172.21.15.164:0/3744421064 wait complete. 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 10 monclient: build_initial_monmap 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 10 monclient: monmap: 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:01.423 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.410+0000 7f8729b3d640 1 Processor -- start 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 -- start start 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient: init 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409c310 0x7f8724139dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-b con 0x7f872409c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-a con 0x7f872413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724093800 0x7f8724093c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): picked mon.noname-c con 0x7f8724093800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87240a2230 con 0x7f872413a5c0 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87240a20b0 con 0x7f872409c310 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8724074e00 con 0x7f8724093800 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8729b3d640 10 monclient(hunting): authenticate will time out at 1999.898663s 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724093800 0x7f8724093c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46354/0 (socket says 172.21.15.164:46354) 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 1 -- 172.21.15.164:0/447432249 learned_addr learned my addr 172.21.15.164:0/447432249 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724093800 0x7f8724093c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:51950/0 (socket says 172.21.15.164:51950) 2024-09-10T16:27:01.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): get_auth_request con 0x7f872413a5c0 auth_method 0 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): get_auth_request con 0x7f8724093800 auth_method 0 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:01.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f8723fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409c310 0x7f8724139dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.411+0000 7f871affd640 10 monclient(hunting): handle_auth_done global_id 224193 payload 824 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724093800 msgr2=0x7f8724093c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8724093800 0x7f8724093c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409c310 msgr2=0x7f8724139dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409c310 0x7f8724139dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 10 monclient: _finish_hunting 0 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 monclient: found mon.noname-a 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.412+0000 7f871affd640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8724095b50 con 0x7f872413a5c0 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f871affd640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 msgr2=0x7f87240932c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:01.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f871affd640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f870c00ebe0 tx=0x7f870c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c017070 con 0x7f872413a5c0 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: dump: 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.428 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f870c00ce40 con 0x7f872413a5c0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 5 monclient: authenticate success, global_id 224193 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 10 monclient: _renew_subs 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f872409d620 con 0x7f872413a5c0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f870c013690 con 0x7f872413a5c0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:01.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8720ff9640 10 monclient: dump: 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f87227fc640 4 set_mon_vals no callback set 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 10 monclient: _renew_subs 2024-09-10T16:27:01.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.414+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8724094630 con 0x7f872413a5c0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8729b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f872409f720 con 0x7f872409d190 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f870c01a070 con 0x7f872413a5c0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8720ff9640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 0x7f87040406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(704..704 src has 494..704) ==== 4755+0+0 (secure 0 0 0) 0x7f870c052cd0 con 0x7f872413a5c0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8723fff640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 0x7f87040406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f8723fff640 10 monclient: get_auth_request con 0x7f870403e210 auth_method 0 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.415+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:01.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.418+0000 7f8723fff640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 0x7f87040406d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f872406d4d0 tx=0x7f871c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:01.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:01.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:01.616+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:27:01.665 INFO:tasks.rados.rados.0.smithi165.stdout:3175: done (2 left) 2024-09-10T16:27:01.665 INFO:tasks.rados.rados.0.smithi165.stdout:3176: done (1 left) 2024-09-10T16:27:01.665 INFO:tasks.rados.rados.0.smithi165.stdout:3177: done (0 left) 2024-09-10T16:27:01.665 INFO:tasks.rados.rados.0.smithi165.stdout:3178: write oid 42 current snap is 340 2024-09-10T16:27:01.666 INFO:tasks.rados.rados.0.smithi165.stdout:3178: seq_num 1108 ranges {622156=647077,1737578=680388} 2024-09-10T16:27:01.672 INFO:tasks.rados.rados.0.smithi165.stdout:3178: writing smithi16534841-42 from 622156 to 1269233 tid 1 2024-09-10T16:27:01.679 INFO:tasks.rados.rados.0.smithi165.stdout:3178: writing smithi16534841-42 from 1737578 to 2417966 tid 2 2024-09-10T16:27:01.679 INFO:tasks.rados.rados.0.smithi165.stdout:3179: copy_from oid 30 from oid 46 current snap is 340 2024-09-10T16:27:01.679 INFO:tasks.rados.rados.0.smithi165.stdout:3178: finishing write tid 1 to smithi16534841-42 2024-09-10T16:27:01.679 INFO:tasks.rados.rados.0.smithi165.stdout:3180: delete oid 36 current snap is 340 2024-09-10T16:27:01.682 INFO:tasks.rados.rados.0.smithi165.stdout:3180: done (2 left) 2024-09-10T16:27:01.682 INFO:tasks.rados.rados.0.smithi165.stdout:3181: snap_create 2024-09-10T16:27:01.683 INFO:tasks.rados.rados.0.smithi165.stdout:3178: finishing write tid 2 to smithi16534841-42 2024-09-10T16:27:01.683 INFO:tasks.rados.rados.0.smithi165.stdout:3178: finishing write tid 3 to smithi16534841-42 2024-09-10T16:27:01.683 INFO:tasks.rados.rados.0.smithi165.stdout:3178: finishing write tid 4 to smithi16534841-42 2024-09-10T16:27:01.684 INFO:tasks.rados.rados.0.smithi165.stdout:3178: oid 42 updating version 0 to 1869 2024-09-10T16:27:01.684 INFO:tasks.rados.rados.0.smithi165.stdout:3178: oid 42 version 1869 is already newer than 1867 2024-09-10T16:27:01.684 INFO:tasks.rados.rados.0.smithi165.stdout:3178: oid 42 version 1869 is already newer than 1868 2024-09-10T16:27:01.684 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1869 (ObjNum 1108 snap 340 seq_num 1108) dirty exists 2024-09-10T16:27:01.684 INFO:tasks.rados.rados.0.smithi165.stdout:3178: left oid 42 (ObjNum 1108 snap 340 seq_num 1108) 2024-09-10T16:27:01.685 INFO:tasks.rados.rados.0.smithi165.stdout:3179: finishing copy_from to smithi16534841-30 2024-09-10T16:27:01.685 INFO:tasks.rados.rados.0.smithi165.stdout:3179: got expected ENOENT (src dne) 2024-09-10T16:27:01.685 INFO:tasks.rados.rados.0.smithi165.stdout:3179: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:02.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:02.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.017+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:27:02.743 INFO:tasks.rados.rados.0.smithi165.stdout:3178: done (2 left) 2024-09-10T16:27:02.744 INFO:tasks.rados.rados.0.smithi165.stdout:3179: done (1 left) 2024-09-10T16:27:02.744 INFO:tasks.rados.rados.0.smithi165.stdout:3181: done (0 left) 2024-09-10T16:27:02.744 INFO:tasks.rados.rados.0.smithi165.stdout:3182: copy_from oid 26 from oid 28 current snap is 341 2024-09-10T16:27:02.744 INFO:tasks.rados.rados.0.smithi165.stdout:3183: delete oid 15 current snap is 341 2024-09-10T16:27:02.749 INFO:tasks.rados.rados.0.smithi165.stdout:3183: done (1 left) 2024-09-10T16:27:02.749 INFO:tasks.rados.rados.0.smithi165.stdout:3184: delete oid 34 current snap is 341 2024-09-10T16:27:02.754 INFO:tasks.rados.rados.0.smithi165.stdout:3184: done (1 left) 2024-09-10T16:27:02.754 INFO:tasks.rados.rados.0.smithi165.stdout:3185: write (excl) oid 48 current snap is 341 2024-09-10T16:27:02.754 INFO:tasks.rados.rados.0.smithi165.stdout:3185: seq_num 1112 ranges {689384=664426,1971506=687726} 2024-09-10T16:27:02.762 INFO:tasks.rados.rados.0.smithi165.stdout:3185: writing smithi16534841-48 from 689384 to 1353810 tid 1 2024-09-10T16:27:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:3185: writing smithi16534841-48 from 1971506 to 2659232 tid 2 2024-09-10T16:27:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:3186: delete oid 29 current snap is 341 2024-09-10T16:27:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:3182: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:27:02.769 INFO:tasks.rados.rados.0.smithi165.stdout:3182: finishing copy_from to smithi16534841-26 2024-09-10T16:27:02.770 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1286 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:27:02.770 INFO:tasks.rados.rados.0.smithi165.stdout:3185: finishing write tid 1 to smithi16534841-48 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3182: done (2 left) 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3186: done (1 left) 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3187: read oid 7 snap -1 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3187: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3188: read oid 31 snap -1 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3188: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:27:02.773 INFO:tasks.rados.rados.0.smithi165.stdout:3189: snap_remove snap 339 2024-09-10T16:27:02.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.818+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:02.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:02.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:02.819+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:27:03.749 INFO:tasks.rados.rados.0.smithi165.stdout:3189: done (3 left) 2024-09-10T16:27:03.750 INFO:tasks.rados.rados.0.smithi165.stdout:3190: write oid 3 current snap is 341 2024-09-10T16:27:03.750 INFO:tasks.rados.rados.0.smithi165.stdout:3190: seq_num 1114 ranges {722998=414924,1572117=575042} 2024-09-10T16:27:03.756 INFO:tasks.rados.rados.0.smithi165.stdout:3190: writing smithi16534841-3 from 722998 to 1137922 tid 1 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3190: writing smithi16534841-3 from 1572117 to 2147159 tid 2 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3191: delete oid 50 current snap is 341 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: finishing write tid 2 to smithi16534841-48 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: finishing write tid 3 to smithi16534841-48 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: finishing write tid 4 to smithi16534841-48 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: oid 48 updating version 0 to 1962 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: oid 48 updating version 1962 to 1963 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: oid 48 updating version 1963 to 1964 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1964 (ObjNum 1112 snap 341 seq_num 1112) dirty exists 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3185: left oid 48 (ObjNum 1112 snap 341 seq_num 1112) 2024-09-10T16:27:03.763 INFO:tasks.rados.rados.0.smithi165.stdout:3187: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:27:03.862 INFO:tasks.rados.rados.0.smithi165.stdout:3188: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:27:03.881 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2281596 2024-09-10T16:27:03.935 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: finishing write tid 1 to smithi16534841-3 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: finishing write tid 2 to smithi16534841-3 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: finishing write tid 3 to smithi16534841-3 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: finishing write tid 4 to smithi16534841-3 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: oid 3 updating version 0 to 1827 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: oid 3 version 1827 is already newer than 1825 2024-09-10T16:27:03.977 INFO:tasks.rados.rados.0.smithi165.stdout:3190: oid 3 version 1827 is already newer than 1826 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1827 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3190: left oid 3 (ObjNum 1114 snap 341 seq_num 1114) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3185: done (4 left) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3187: done (3 left) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3188: done (2 left) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3190: done (1 left) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3191: done (0 left) 2024-09-10T16:27:03.978 INFO:tasks.rados.rados.0.smithi165.stdout:3192: snap_remove snap 319 2024-09-10T16:27:04.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.411+0000 7f871b7fe640 10 monclient: tick 2024-09-10T16:27:04.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.411+0000 7f871b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:04.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.411+0000 7f871b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:04.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.411+0000 7f871b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872409d190 auth_method 0 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:04.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:04.421+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-09-10T16:27:04.645 INFO:tasks.rados.rados.0.smithi165.stdout:3192: done (0 left) 2024-09-10T16:27:04.645 INFO:tasks.rados.rados.0.smithi165.stdout:3193: read oid 47 snap -1 2024-09-10T16:27:04.645 INFO:tasks.rados.rados.0.smithi165.stdout:3193: expect (ObjNum 1094 snap 332 seq_num 1094) 2024-09-10T16:27:04.645 INFO:tasks.rados.rados.0.smithi165.stdout:3194: snap_create 2024-09-10T16:27:04.655 INFO:tasks.rados.rados.0.smithi165.stdout:3193: expect (ObjNum 1094 snap 332 seq_num 1094) 2024-09-10T16:27:04.711 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 10 monclient: tick 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 5 monclient: _check_tell_commands timeout tell command 1 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 0x7f86fc005a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 msgr2=0x7f872408df80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:05.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872409d190 0x7f872408df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f871b7fe640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f872409f720 con 0x7f86fc003540 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 0x7f86fc005a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f86fc003540 auth_method 0 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 10 monclient(hunting): handle_auth_done global_id 234100 payload 824 2024-09-10T16:27:05.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.412+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86fc0060e0 con 0x7f86fc003540 2024-09-10T16:27:05.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 msgr2=0x7f86fc005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:05.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 0x7f86fc005a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f872409ea10 tx=0x7f8714013b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:05.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f871401d470 con 0x7f86fc003540 2024-09-10T16:27:05.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 10 monclient: dump: 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8714018070 con 0x7f86fc003540 2024-09-10T16:27:05.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:05.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:05.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 msgr2=0x7f86fc005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 0x7f86fc005a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f872409ea10 tx=0x7f8714013b90 comp rx=0 tx=0).stop 2024-09-10T16:27:05.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 msgr2=0x7f86fc005a00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:05.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.415+0000 7f8720ff9640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f86fc003540 0x7f86fc005a00 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f872409ea10 tx=0x7f8714013b90 comp rx=0 tx=0).stop 2024-09-10T16:27:05.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.422+0000 7f8729b3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:05.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.422+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 0x7f872408e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.422+0000 7f8729b3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.422+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f872413a310 con 0x7f872408e4c0 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 0x7f872408e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient: get_auth_request con 0x7f872408e4c0 auth_method 0 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 10 monclient(hunting): handle_auth_done global_id 234103 payload 824 2024-09-10T16:27:05.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.423+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f872408ee10 con 0x7f872408e4c0 2024-09-10T16:27:05.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f87237fe640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 msgr2=0x7f872408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:05.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f87237fe640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 0x7f872408e8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f872409ea10 tx=0x7f8714007910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:05.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8714022770 con 0x7f872408e4c0 2024-09-10T16:27:05.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:05.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 10 monclient: dump: 2024-09-10T16:27:05.428 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f8714014400 con 0x7f872408e4c0 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:05.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 msgr2=0x7f872408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 0x7f872408e8d0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f872409ea10 tx=0x7f8714007910 comp rx=0 tx=0).stop 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 msgr2=0x7f872408e8d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8720ff9640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f872408e4c0 0x7f872408e8d0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.426+0000 7f8723fff640 1 -- 172.21.15.164:0/447432249 reap_dead start 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:05.430 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 25710, 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:27:05.431 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:05.432 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:05.433 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:05.434 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:05.435 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:05.436 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:05.437 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:05.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:05.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:05.441 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:05.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:05.443 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:05.443 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 msgr2=0x7f87040406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 0x7f87040406d0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f872406d4d0 tx=0x7f871c02f000 comp rx=0 tx=0).stop 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 10 monclient: shutdown 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 msgr2=0x7f87240932c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f870c00ebe0 tx=0x7f870c00c520 comp rx=0 tx=0).stop 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 shutdown_connections 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f870403e210 0x7f87040406d0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 --2- 172.21.15.164:0/447432249 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f872413a5c0 0x7f87240932c0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 >> 172.21.15.164:0/447432249 conn(0x7f87240a3110 msgr2=0x7f8724070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 shutdown_connections 2024-09-10T16:27:05.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.427+0000 7f8729b3d640 1 -- 172.21.15.164:0/447432249 wait complete. 2024-09-10T16:27:05.445 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:05.445 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:27:05.446 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.550+0000 7fc28f0be640 10 monclient: get_monmap_and_config 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.550+0000 7fc28f0be640 10 monclient: build_initial_monmap 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.550+0000 7fc28f0be640 10 monclient: monmap: 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:05.551647+0000 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:05.551647+0000 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:05.553 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:05.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:05.553 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:05.553 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:05.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 1 Processor -- start 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 1 -- start start 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient: init 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T16:27:05.555 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc28809c180 0x7fc28809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-a con 0x7fc28809c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809d160 0x7fc28809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-b con 0x7fc28809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc28809f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.551+0000 7fc28f0be640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc28809a430 con 0x7fc28809f4c0 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc2880a24e0 con 0x7fc28809c180 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc2880a2bf0 con 0x7fc28809d160 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28f0be640 10 monclient(hunting): authenticate will time out at 2004.039017s 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc28809c180 0x7fc28809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc28809c180 0x7fc28809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38648/0 (socket says 172.21.15.164:38648) 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809d160 0x7fc28809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 -- 172.21.15.164:0/1408768747 learned_addr learned my addr 172.21.15.164:0/1408768747 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): get_auth_request con 0x7fc28809c180 auth_method 0 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): get_auth_request con 0x7fc28809d160 auth_method 0 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_done global_id 224196 payload 274 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 -- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809d160 msgr2=0x7fc28809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 --2- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809d160 0x7fc28809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 -- 172.21.15.164:0/1408768747 >> v1:172.21.15.165:6789/0 conn(0x7fc28809f4c0 legacy=0x7fc28813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient: _finish_hunting 0 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 monclient: found mon.noname-a 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:05.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.552+0000 7fc28ce5a640 1 -- 172.21.15.164:0/1408768747 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc28809c560 con 0x7fc28809c180 2024-09-10T16:27:05.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28ce5a640 1 -- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809c180 msgr2=0x7fc28809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:05.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28ce5a640 1 --2- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809c180 0x7fc28809ea40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc284009e50 tx=0x7fc28402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86fa54d61dc85ddb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:05.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 1 -- 172.21.15.164:0/1408768747 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284036070 con 0x7fc28809c180 2024-09-10T16:27:05.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: dump: 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 5 monclient: authenticate success, global_id 224196 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 1 -- 172.21.15.164:0/1408768747 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc284007800 con 0x7fc28809c180 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 4 set_mon_vals no callback set 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 1 -- 172.21.15.164:0/1408768747 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284007bd0 con 0x7fc28809c180 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc27f7fe640 10 monclient: dump: 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.566 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 10 monclient: shutdown 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 -- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809c180 msgr2=0x7fc28809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 --2- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809c180 0x7fc28809ea40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc284009e50 tx=0x7fc28402a5f0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 -- 172.21.15.164:0/1408768747 shutdown_connections 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 --2- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809d160 0x7fc28809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 --2- 172.21.15.164:0/1408768747 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809c180 0x7fc28809ea40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 -- 172.21.15.164:0/1408768747 >> 172.21.15.164:0/1408768747 conn(0x7fc2880a3110 msgr2=0x7fc2880a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 -- 172.21.15.164:0/1408768747 shutdown_connections 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.555+0000 7fc28f0be640 1 -- 172.21.15.164:0/1408768747 wait complete. 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.556+0000 7fc28f0be640 10 monclient: build_initial_monmap 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.556+0000 7fc28f0be640 10 monclient: monmap: 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:05.568 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.556+0000 7fc28f0be640 1 Processor -- start 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.556+0000 7fc28f0be640 1 -- start start 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.556+0000 7fc28f0be640 10 monclient: init 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-c con 0x7fc28809c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 0x7fc288136d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-a con 0x7fc28809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc28809f4c0 0x7fc2881ccf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): picked mon.noname-b con 0x7fc28809f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc288074f90 con 0x7fc28809d160 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc288074e10 con 0x7fc28809f4c0 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc28813cce0 con 0x7fc28809c180 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28f0be640 10 monclient(hunting): authenticate will time out at 2004.044588s 2024-09-10T16:27:05.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34608/0 (socket says 172.21.15.164:34608) 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 0x7fc288136d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 -- 172.21.15.164:0/3095804521 learned_addr learned my addr 172.21.15.164:0/3095804521 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): get_auth_request con 0x7fc28809c180 auth_method 0 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): get_auth_request con 0x7fc28809d160 auth_method 0 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28d65b640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc28809f4c0 0x7fc2881ccf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28d65b640 10 monclient(hunting): get_auth_request con 0x7fc28809f4c0 auth_method 0 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28d65b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28d65b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28d65b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient(hunting): handle_auth_done global_id 214289 payload 824 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 -- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc28809f4c0 msgr2=0x7fc2881ccf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc28809f4c0 0x7fc2881ccf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 -- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 msgr2=0x7fc288136d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 0x7fc288136d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient: _finish_hunting 0 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 monclient: found mon.noname-c 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc28ce5a640 1 -- 172.21.15.164:0/3095804521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2880a6070 con 0x7fc28809c180 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.557+0000 7fc27ffff640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 0x7fc288136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28ce5a640 1 -- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 msgr2=0x7fc288136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28ce5a640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc284009e50 tx=0x7fc284004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284036050 con 0x7fc28809c180 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: dump: 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.573 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc284042070 con 0x7fc28809c180 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 5 monclient: authenticate success, global_id 214289 2024-09-10T16:27:05.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 10 monclient: _renew_subs 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc28809d620 con 0x7fc28809c180 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc284031e10 con 0x7fc28809c180 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27d7fa640 10 monclient: dump: 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:05.575 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 10 monclient: _renew_subs 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc28806fbb0 con 0x7fc28809c180 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.560+0000 7fc27effd640 4 set_mon_vals no callback set 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc284047460 con 0x7fc28809c180 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc28f0be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc28f0be640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc28f0be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc28809ea40 con 0x7fc28809c180 2024-09-10T16:27:05.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc27d7fa640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 0x7fc2740406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(708..708 src has 494..708) ==== 4827+0+0 (secure 0 0 0) 0x7fc284073c40 con 0x7fc28809c180 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc27ffff640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 0x7fc2740406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.561+0000 7fc27ffff640 10 monclient: get_auth_request con 0x7fc27403e210 auth_method 0 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.564+0000 7fc27ffff640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 0x7fc2740406d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc28806fe30 tx=0x7fc278009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.566+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc284072610 con 0x7fc28809c180 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.566+0000 7fc27d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:05.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.566+0000 7fc27d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:05.648 INFO:tasks.rados.rados.0.smithi165.stdout:3193: done (1 left) 2024-09-10T16:27:05.648 INFO:tasks.rados.rados.0.smithi165.stdout:3194: done (0 left) 2024-09-10T16:27:05.648 INFO:tasks.rados.rados.0.smithi165.stdout:3195: write oid 14 current snap is 342 2024-09-10T16:27:05.648 INFO:tasks.rados.rados.0.smithi165.stdout:3195: seq_num 1116 ranges {756612=565422,2089081=578711} 2024-09-10T16:27:05.655 INFO:tasks.rados.rados.0.smithi165.stdout:3195: writing smithi16534841-14 from 756612 to 1322034 tid 1 2024-09-10T16:27:05.662 INFO:tasks.rados.rados.0.smithi165.stdout:3195: writing smithi16534841-14 from 2089081 to 2667792 tid 2 2024-09-10T16:27:05.662 INFO:tasks.rados.rados.0.smithi165.stdout:3196: snap_remove snap 335 2024-09-10T16:27:05.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.812+0000 7fc28f0be640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:27:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.812+0000 7fc28f0be640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:27:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.812+0000 7fc28f0be640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.812+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fc288077260 con 0x7fc28809c180 2024-09-10T16:27:05.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.813+0000 7fc27d7fa640 1 -- 172.21.15.164:0/3095804521 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fc284046ce0 con 0x7fc28809c180 2024-09-10T16:27:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.813+0000 7fc27d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:27:05.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.813+0000 7fc27d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:05.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=3673248752,config=2586702516,health=849142597,kv=3005215917,logm=3388968963}) 2024-09-10T16:27:05.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=28,config=2,health=10,kv=12,logm=48} crc {auth=3596146809,config=2586702516,health=2738224559,kv=805391194,logm=3282090168}) 2024-09-10T16:27:05.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=25,config=2,health=11,kv=10,logm=52} crc {auth=3673248752,config=2586702516,health=849142597,kv=3005215917,logm=3388968963}) 2024-09-10T16:27:05.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.814+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=12,kv=12,logm=47} crc {auth=35694553,config=2586702516,health=2383050609,kv=603532201,logm=1756779292}) 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141053552}) 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1894414670}) 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2141053552}) 2024-09-10T16:27:05.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.815+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=669492064}) 2024-09-10T16:27:05.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 msgr2=0x7fc2740406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 0x7fc2740406d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fc28806fe30 tx=0x7fc278009190 comp rx=0 tx=0).stop 2024-09-10T16:27:05.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 10 monclient: shutdown 2024-09-10T16:27:05.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:05.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 msgr2=0x7fc288136850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fc284009e50 tx=0x7fc284004a40 comp rx=0 tx=0).stop 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 shutdown_connections 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc27403e210 0x7fc2740406d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc28809f4c0 0x7fc2881ccf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc28809d160 0x7fc288136d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 --2- 172.21.15.164:0/3095804521 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc28809c180 0x7fc288136850 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 >> 172.21.15.164:0/3095804521 conn(0x7fc2880a3110 msgr2=0x7fc28809af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 shutdown_connections 2024-09-10T16:27:05.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:05.816+0000 7fc28f0be640 1 -- 172.21.15.164:0/3095804521 wait complete. 2024-09-10T16:27:05.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219850126}) 2024-09-10T16:27:05.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1867734376}) 2024-09-10T16:27:05.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4219850126}) 2024-09-10T16:27:05.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.822+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2212130641}) 2024-09-10T16:27:05.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201358847}) 2024-09-10T16:27:05.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=216047626}) 2024-09-10T16:27:05.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3201358847}) 2024-09-10T16:27:05.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.823+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640030653}) 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329387921}) 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376471131}) 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3329387921}) 2024-09-10T16:27:05.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.824+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940194146}) 2024-09-10T16:27:05.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789422498}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2479044563}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=789422498}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.825+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4162586880}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496076846}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612019545}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2496076846}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213176853}) 2024-09-10T16:27:05.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025989260}) 2024-09-10T16:27:05.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3648909954}) 2024-09-10T16:27:05.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3025989260}) 2024-09-10T16:27:05.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.826+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3583401069}) 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091336521}) 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823244193}) 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4091336521}) 2024-09-10T16:27:05.830 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.827+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2788908667}) 2024-09-10T16:27:05.831 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533265145}) 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1509921077}) 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533265145}) 2024-09-10T16:27:05.832 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.828+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120541462}) 2024-09-10T16:27:05.836 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.837 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109643611}) 2024-09-10T16:27:05.837 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1375718873}) 2024-09-10T16:27:05.837 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.837 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109643611}) 2024-09-10T16:27:05.837 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.835+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3074246552}) 2024-09-10T16:27:05.839 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.839 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938100783}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637886977}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=938100783}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.836+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1454526852}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338726644}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3381712822}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338726644}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1597901507}) 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.840 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275372032}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2622300267}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2275372032}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.837+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153842795}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261510693}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3396511200}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4261510693}) 2024-09-10T16:27:05.841 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.838+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453213957}) 2024-09-10T16:27:05.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087357828}) 2024-09-10T16:27:05.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2559589541}) 2024-09-10T16:27:05.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2087357828}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=402599391}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151201774}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532977309}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4151201774}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.839+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635330124}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018391731}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1001971876}) 2024-09-10T16:27:05.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4018391731}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.840+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3279047787}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392051492}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3306685361}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=392051492}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.841+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=198034134}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552910249}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3605511624}) 2024-09-10T16:27:05.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552910249}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2867500829}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235569376}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2167179938}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3235569376}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.842+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1402923417}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891057646}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1346831028}) 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891057646}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1133380064}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.843+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875730352}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2481394557}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1875730352}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2670341440}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258194402}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660097269}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3258194402}) 2024-09-10T16:27:05.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.844+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3401495329}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003101776}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575067561}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3003101776}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.845+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3632643044}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159292016}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899040498}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4159292016}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3705075}) 2024-09-10T16:27:05.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769640888}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1563038578}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3769640888}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.846+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608311831}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610369038}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1582242909}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2610369038}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.847+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3027883132}) 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.855 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416757156}) 2024-09-10T16:27:05.856 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992975260}) 2024-09-10T16:27:05.856 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.856 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1416757156}) 2024-09-10T16:27:05.856 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1785555226}) 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488974618}) 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2364530707}) 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488974618}) 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.848+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547089839}) 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.860 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973217705}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=927429814}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1973217705}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.849+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1253424393}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142624709}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290799535}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2142624709}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.850+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1430325725}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347458879}) 2024-09-10T16:27:05.861 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3622378783}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=347458879}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.851+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3903161596}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009702487}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2100667335}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3009702487}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155657201}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947908728}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3539803980}) 2024-09-10T16:27:05.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1947908728}) 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.852+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4158035819}) 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168829421}) 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=941173275}) 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3168829421}) 2024-09-10T16:27:05.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:05.853+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3174843090}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391877054}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2543370436}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391877054}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1362685381}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249334418}) 2024-09-10T16:27:06.100 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2327191490}) 2024-09-10T16:27:06.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249334418}) 2024-09-10T16:27:06.101 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.098+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636522740}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239113202}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=132403931}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239113202}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.099+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777148559}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017238209}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3867575705}) 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.105 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4017238209}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=302718568}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598611371}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981215407}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=598611371}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.100+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=944147338}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631009247}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904881972}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2631009247}) 2024-09-10T16:27:06.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.101+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=178295054}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712711512}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2027106732}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2712711512}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2439532666}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.102+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804456232}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1589015980}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3804456232}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2686764342}) 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563279952}) 2024-09-10T16:27:06.108 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1053119035}) 2024-09-10T16:27:06.108 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.108 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3563279952}) 2024-09-10T16:27:06.108 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.103+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3232441315}) 2024-09-10T16:27:06.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773169503}) 2024-09-10T16:27:06.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523314277}) 2024-09-10T16:27:06.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=773169503}) 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.104+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1569822866}) 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302280080}) 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3750661725}) 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2302280080}) 2024-09-10T16:27:06.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.105+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3656356831}) 2024-09-10T16:27:06.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123520323}) 2024-09-10T16:27:06.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2879457887}) 2024-09-10T16:27:06.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=123520323}) 2024-09-10T16:27:06.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.207+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1692765055}) 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157195798}) 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3189419948}) 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2157195798}) 2024-09-10T16:27:06.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.208+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4019362954}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258660558}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3368954636}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2258660558}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.209+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=30411339}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231972143}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=521722179}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2231972143}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1336222066}) 2024-09-10T16:27:06.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571327474}) 2024-09-10T16:27:06.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2973776568}) 2024-09-10T16:27:06.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=571327474}) 2024-09-10T16:27:06.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867119317}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554363556}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3600365046}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2554363556}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.211+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1372349207}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461283868}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3938202135}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461283868}) 2024-09-10T16:27:06.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4033229091}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431553983}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3809495835}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431553983}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2569990062}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112478947}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=168510344}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3112478947}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076677466}) 2024-09-10T16:27:06.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863259971}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678657045}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2863259971}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.214+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3899470866}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686792783}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=641730427}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1686792783}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.215+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2312304194}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432328373}) 2024-09-10T16:27:06.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3116883874}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432328373}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051817414}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454153734}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3521277390}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3454153734}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.216+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606732489}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454762104}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3540594624}) 2024-09-10T16:27:06.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=454762104}) 2024-09-10T16:27:06.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.217+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2119908003}) 2024-09-10T16:27:06.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258842068}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1912311298}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1258842068}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.218+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2915842577}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780342050}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1617249255}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3780342050}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=757540261}) 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543044459}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3945069897}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3543044459}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.219+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2528491944}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515535576}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569001200}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515535576}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.220+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2504004577}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=322308723}) 2024-09-10T16:27:06.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3709398313}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=322308723}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3631216622}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080915705}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687078123}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1080915705}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.221+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1188892308}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822249342}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1513669003}) 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2822249342}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.222+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2864255508}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812896592}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4194456847}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2812896592}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.223+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3889604141}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775070941}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970830203}) 2024-09-10T16:27:06.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775070941}) 2024-09-10T16:27:06.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=38067912}) 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150706571}) 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1132568262}) 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4150706571}) 2024-09-10T16:27:06.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1193170246}) 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416415548}) 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251054923}) 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=416415548}) 2024-09-10T16:27:06.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.645+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812677895}) 2024-09-10T16:27:06.651 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552872330}) 2024-09-10T16:27:06.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3872734931}) 2024-09-10T16:27:06.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3552872330}) 2024-09-10T16:27:06.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.650+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=348578276}) 2024-09-10T16:27:06.652 INFO:tasks.rados.rados.0.smithi165.stdout:3196: done (1 left) 2024-09-10T16:27:06.653 INFO:tasks.rados.rados.0.smithi165.stdout:3197: write (excl) oid 22 current snap is 342 2024-09-10T16:27:06.653 INFO:tasks.rados.rados.0.smithi165.stdout:3195: finishing write tid 1 to smithi16534841-14 2024-09-10T16:27:06.653 INFO:tasks.rados.rados.0.smithi165.stdout:3197: seq_num 1117 ranges {773419=640671,2147563=438722} 2024-09-10T16:27:06.660 INFO:tasks.rados.rados.0.smithi165.stdout:3197: writing smithi16534841-22 from 773419 to 1414090 tid 1 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3197: writing smithi16534841-22 from 2147563 to 2586285 tid 2 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3198: rollback oid 12 current snap is 342 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 338 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: finishing write tid 2 to smithi16534841-14 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: finishing write tid 3 to smithi16534841-14 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: finishing write tid 4 to smithi16534841-14 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: oid 14 updating version 0 to 1660 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: oid 14 version 1660 is already newer than 1659 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:3195: oid 14 version 1660 is already newer than 1658 2024-09-10T16:27:06.668 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1660 (ObjNum 1116 snap 342 seq_num 1116) dirty exists 2024-09-10T16:27:06.669 INFO:tasks.rados.rados.0.smithi165.stdout:3195: left oid 14 (ObjNum 1116 snap 342 seq_num 1116) 2024-09-10T16:27:06.669 INFO:tasks.rados.rados.0.smithi165.stdout:3197: finishing write tid 1 to smithi16534841-22 2024-09-10T16:27:06.669 INFO:tasks.rados.rados.0.smithi165.stdout:3195: done (2 left) 2024-09-10T16:27:06.669 INFO:tasks.rados.rados.0.smithi165.stdout:3199: snap_remove snap 341 2024-09-10T16:27:06.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886851158}) 2024-09-10T16:27:06.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2963788094}) 2024-09-10T16:27:06.757 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3886851158}) 2024-09-10T16:27:06.758 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.756+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2197214215}) 2024-09-10T16:27:06.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592898310}) 2024-09-10T16:27:06.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2633727726}) 2024-09-10T16:27:06.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:06.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=592898310}) 2024-09-10T16:27:06.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:06.771+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2668227813}) 2024-09-10T16:27:06.832 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:27:06.833 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:06.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 10 monclient: get_monmap_and_config 2024-09-10T16:27:06.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 10 monclient: build_initial_monmap 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 10 monclient: monmap: 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:06.932073+0000 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:06.932073+0000 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:06.933 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:06.934 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 1 Processor -- start 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 1 -- start start 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.931+0000 7f6b4a6b6640 10 monclient: init 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-10T16:27:06.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6b4409d190 0x7f6b4409ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6b4409d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 0x7f6b4413b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4409f4c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6b4409fbc0 con 0x7f6b4409f4c0 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6b440a2a20 con 0x7f6b4409d190 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6b4409c6f0 con 0x7f6b4409c310 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:06.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b4a6b6640 10 monclient(hunting): authenticate will time out at 2005.419487s 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 0x7f6b4413b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34638/0 (socket says 172.21.15.164:34638) 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 -- 172.21.15.164:0/442395157 learned_addr learned my addr 172.21.15.164:0/442395157 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): get_auth_request con 0x7f6b4409c310 auth_method 0 2024-09-10T16:27:06.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:06.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:06.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:06.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): get_auth_request con 0x7f6b4409f4c0 auth_method 0 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient(hunting): handle_auth_done global_id 214292 payload 274 2024-09-10T16:27:06.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 -- 172.21.15.164:0/442395157 >> v2:172.21.15.165:3300/0 conn(0x7f6b4409d190 msgr2=0x7f6b4409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 --2- 172.21.15.164:0/442395157 >> v2:172.21.15.165:3300/0 conn(0x7f6b4409d190 0x7f6b4409ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 -- 172.21.15.164:0/442395157 >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 msgr2=0x7f6b4413b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 --2- 172.21.15.164:0/442395157 >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 0x7f6b4413b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 monclient: found mon.noname-b 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:06.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:27:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b43fff640 1 -- 172.21.15.164:0/442395157 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4409a4d0 con 0x7f6b4409c310 2024-09-10T16:27:06.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.932+0000 7f6b48c53640 1 --2- 172.21.15.164:0/442395157 >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 0x7f6b4413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b43fff640 1 -- 172.21.15.164:0/442395157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 msgr2=0x7f6b4409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b43fff640 1 --2- 172.21.15.164:0/442395157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b4406d5b0 tx=0x7f6b3402a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3ea4f21885df5a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 1 -- 172.21.15.164:0/442395157 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34037070 con 0x7f6b4409c310 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: dump: 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.948 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 1 -- 172.21.15.164:0/442395157 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b34032640 con 0x7f6b4409c310 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b4a6b6640 5 monclient: authenticate success, global_id 214292 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b4a6b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:06.949 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b4a6b6640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 1 -- 172.21.15.164:0/442395157 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34032a10 con 0x7f6b4409c310 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b4a6b6640 4 set_mon_vals no callback set 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.935+0000 7f6b42ffd640 10 monclient: dump: 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.950 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 10 monclient: shutdown 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/442395157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 msgr2=0x7f6b4409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/442395157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6b4406d5b0 tx=0x7f6b3402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/442395157 shutdown_connections 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/442395157 >> v2:172.21.15.164:3300/0 conn(0x7f6b4409f4c0 0x7f6b4413b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/442395157 >> v2:172.21.15.165:3300/0 conn(0x7f6b4409d190 0x7f6b4409ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/442395157 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b4409c310 0x7f6b4409ea40 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/442395157 >> 172.21.15.164:0/442395157 conn(0x7f6b440a3110 msgr2=0x7f6b440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:06.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/442395157 shutdown_connections 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/442395157 wait complete. 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 10 monclient: build_initial_monmap 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.936+0000 7f6b4a6b6640 10 monclient: monmap: 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:06.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 Processor -- start 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 -- start start 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient: init 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:27:06.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-a con 0x7f6b4409c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 0x7f6b440966b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-b con 0x7f6b4409f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 0x7f6b441ccd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): picked mon.noname-c con 0x7f6b44096bf0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6b440a2230 con 0x7f6b4409c310 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6b440a20b0 con 0x7f6b4409f4c0 2024-09-10T16:27:06.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6b44074e00 con 0x7f6b44096bf0 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.937+0000 7f6b4a6b6640 10 monclient(hunting): authenticate will time out at 2005.424975s 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 0x7f6b441ccd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38678/0 (socket says 172.21.15.164:38678) 2024-09-10T16:27:06.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 -- 172.21.15.164:0/1532098141 learned_addr learned my addr 172.21.15.164:0/1532098141 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): get_auth_request con 0x7f6b4409c310 auth_method 0 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): get_auth_request con 0x7f6b44096bf0 auth_method 0 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:06.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b437fe640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 0x7f6b440966b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient(hunting): handle_auth_done global_id 224205 payload 824 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 -- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 msgr2=0x7f6b441ccd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 0x7f6b441ccd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 -- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 msgr2=0x7f6b440966b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 0x7f6b440966b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:06.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b43fff640 1 -- 172.21.15.164:0/1532098141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4413d580 con 0x7f6b4409c310 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b437fe640 10 monclient: get_auth_request con 0x7f6b4409f4c0 auth_method 0 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b48c53640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 0x7f6b441ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.938+0000 7f6b437fe640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 0x7f6b440966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b43fff640 1 -- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 msgr2=0x7f6b44073e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b43fff640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b340090b0 tx=0x7f6b340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34041020 con 0x7f6b4409c310 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: dump: 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:06.958 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6b3403d850 con 0x7f6b4409c310 2024-09-10T16:27:06.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 5 monclient: authenticate success, global_id 224205 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b34037040 con 0x7f6b4409c310 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 10 monclient: _renew_subs 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4409eee0 con 0x7f6b4409c310 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b427fc640 4 set_mon_vals no callback set 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b40ff9640 10 monclient: dump: 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:06.960 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 10 monclient: _renew_subs 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.940+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4408ea30 con 0x7f6b4409c310 2024-09-10T16:27:06.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b4a6b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b4a6b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b4a6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6b34044020 con 0x7f6b4409c310 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b4409d930 con 0x7f6b4409c310 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.941+0000 7f6b40ff9640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 0x7f6b24040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.942+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(710..710 src has 494..710) ==== 4815+0+0 (secure 0 0 0) 0x7f6b34073290 con 0x7f6b4409c310 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.942+0000 7f6b437fe640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 0x7f6b24040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.942+0000 7f6b437fe640 10 monclient: get_auth_request con 0x7f6b2403e240 auth_method 0 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.944+0000 7f6b437fe640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 0x7f6b24040700 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.946+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6b3404b030 con 0x7f6b4409c310 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.946+0000 7f6b40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:06.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:06.946+0000 7f6b40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:07.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b4a6b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:07.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b4a6b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b4a6b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6b44077260 con 0x7f6b4409c310 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b40ff9640 1 -- 172.21.15.164:0/1532098141 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6b3403a350 con 0x7f6b4409c310 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.186+0000 7f6b40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:07.188 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":202,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 msgr2=0x7f6b24040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 0x7f6b24040700 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f6b2c006fd0 tx=0x7f6b2c008040 comp rx=0 tx=0).stop 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 10 monclient: shutdown 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 msgr2=0x7f6b44073e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f6b340090b0 tx=0x7f6b340046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 shutdown_connections 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6b2403e240 0x7f6b24040700 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6b44096bf0 0x7f6b441ccd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6b4409f4c0 0x7f6b440966b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 --2- 172.21.15.164:0/1532098141 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6b4409c310 0x7f6b44073e80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 >> 172.21.15.164:0/1532098141 conn(0x7f6b440a3110 msgr2=0x7f6b4409db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 shutdown_connections 2024-09-10T16:27:07.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.188+0000 7f6b4a6b6640 1 -- 172.21.15.164:0/1532098141 wait complete. 2024-09-10T16:27:07.200 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:27:07.200 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:27:07.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:27:07.201 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:27:07.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.298+0000 7f499adf3640 10 monclient: get_monmap_and_config 2024-09-10T16:27:07.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.298+0000 7f499adf3640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.299 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.298+0000 7f499adf3640 10 monclient: monmap: 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:07.299054+0000 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:07.299054+0000 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:07.300 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.298+0000 7f499adf3640 1 Processor -- start 2024-09-10T16:27:07.302 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 1 -- start start 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient: init 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f499406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4994134f70 0x7f499413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-a con 0x7f4994134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f499413d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f499409ce50 con 0x7f499406cb70 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4994135380 con 0x7f499413d560 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f499406db60 con 0x7f4994134f70 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f499adf3640 10 monclient(hunting): authenticate will time out at 2005.786567s 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4994134f70 0x7f499413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f4994134f70 0x7f499413d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38694/0 (socket says 172.21.15.164:38694) 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 -- 172.21.15.164:0/1997501850 learned_addr learned my addr 172.21.15.164:0/1997501850 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): get_auth_request con 0x7f4994134f70 auth_method 0 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4998b8f640 1 --1- 172.21.15.164:0/1997501850 >> v1:172.21.15.164:6789/0 conn(0x7f499406cb70 0x7f49940a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47102/0 (socket says 172.21.15.164:47102) 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient(hunting): handle_auth_done global_id 224208 payload 274 2024-09-10T16:27:07.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 -- 172.21.15.164:0/1997501850 >> v1:172.21.15.165:6789/0 conn(0x7f499413d560 legacy=0x7f499413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 -- 172.21.15.164:0/1997501850 >> v1:172.21.15.164:6789/0 conn(0x7f499406cb70 legacy=0x7f49940a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.299+0000 7f4993fff640 1 -- 172.21.15.164:0/1997501850 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499409cb70 con 0x7f4994134f70 2024-09-10T16:27:07.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.302+0000 7f4993fff640 1 -- 172.21.15.164:0/1997501850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4994134f70 msgr2=0x7f499413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.302+0000 7f4993fff640 1 --2- 172.21.15.164:0/1997501850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4994134f70 0x7f499413d020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f49880098a0 tx=0x7f498802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbe86ad6ffde5935 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 1 -- 172.21.15.164:0/1997501850 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988037070 con 0x7f4994134f70 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: dump: 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.310 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 1 -- 172.21.15.164:0/1997501850 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4988032640 con 0x7f4994134f70 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 5 monclient: authenticate success, global_id 224208 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 1 -- 172.21.15.164:0/1997501850 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988032a10 con 0x7f4994134f70 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 4 set_mon_vals no callback set 2024-09-10T16:27:07.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f49937fe640 10 monclient: dump: 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 10 monclient: shutdown 2024-09-10T16:27:07.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 -- 172.21.15.164:0/1997501850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4994134f70 msgr2=0x7f499413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 --2- 172.21.15.164:0/1997501850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4994134f70 0x7f499413d020 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f49880098a0 tx=0x7f498802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 -- 172.21.15.164:0/1997501850 shutdown_connections 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 --2- 172.21.15.164:0/1997501850 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4994134f70 0x7f499413d020 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 -- 172.21.15.164:0/1997501850 >> 172.21.15.164:0/1997501850 conn(0x7f49940aa900 msgr2=0x7f49940aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 -- 172.21.15.164:0/1997501850 shutdown_connections 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.303+0000 7f499adf3640 1 -- 172.21.15.164:0/1997501850 wait complete. 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.304+0000 7f499adf3640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.304+0000 7f499adf3640 10 monclient: monmap: 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:07.313 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.304+0000 7f499adf3640 1 Processor -- start 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.304+0000 7f499adf3640 1 -- start start 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.304+0000 7f499adf3640 10 monclient: init 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 0x7f49941cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-a con 0x7f499406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:07.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4994134f70 0x7f49940a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-c con 0x7f4994134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499413d560 0x7f49940a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): picked mon.noname-b con 0x7f499413d560 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49940a4db0 con 0x7f499406cb70 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f499406cf50 con 0x7f499413d560 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4994141070 con 0x7f4994134f70 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f499adf3640 10 monclient(hunting): authenticate will time out at 2005.792327s 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 0x7f49941cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 0x7f49941cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38706/0 (socket says 172.21.15.164:38706) 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4994134f70 0x7f49940a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 -- 172.21.15.164:0/2393008462 learned_addr learned my addr 172.21.15.164:0/2393008462 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): get_auth_request con 0x7f499406cb70 auth_method 0 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): get_auth_request con 0x7f4994134f70 auth_method 0 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient(hunting): handle_auth_done global_id 224211 payload 824 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4999390640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499413d560 0x7f49940a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4994134f70 msgr2=0x7f49940a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4994134f70 0x7f49940a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499413d560 msgr2=0x7f49940a3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f499413d560 0x7f49940a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.305+0000 7f4998b8f640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f499409dbe0 con 0x7f499406cb70 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.307+0000 7f4998b8f640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 msgr2=0x7f49941cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.307+0000 7f4998b8f640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 0x7f49941cd530 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f498400b410 tx=0x7f498400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984015020 con 0x7f499406cb70 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: dump: 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.318 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4984011a40 con 0x7f499406cb70 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 5 monclient: authenticate success, global_id 224211 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4984010380 con 0x7f499406cb70 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 10 monclient: _renew_subs 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f499406e720 con 0x7f499406cb70 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f49917fa640 10 monclient: dump: 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.320 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.321 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f4992ffd640 4 set_mon_vals no callback set 2024-09-10T16:27:07.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 10 monclient: _renew_subs 2024-09-10T16:27:07.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.308+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f499409e180 con 0x7f499406cb70 2024-09-10T16:27:07.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f499adf3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f499adf3640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 0x7f499413a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f49940a44e0 con 0x7f499413a060 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 0x7f499413a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient: get_auth_request con 0x7f499413a060 auth_method 0 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4984018020 con 0x7f499406cb70 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 10 monclient(hunting): handle_auth_done global_id 224214 payload 824 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f49917fa640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f498003e210 0x7f49800406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4993fff640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f49940a46a0 con 0x7f499413a060 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(710..710 src has 494..710) ==== 4815+0+0 (secure 0 0 0) 0x7f4984052c90 con 0x7f499406cb70 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4999390640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f498003e210 0x7f49800406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.309+0000 7f4999390640 10 monclient: get_auth_request con 0x7f498003e210 auth_method 0 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f4993fff640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 msgr2=0x7f499413a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f4993fff640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 0x7f499413a440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4988034000 tx=0x7f49880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988037040 con 0x7f499413a060 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f49917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f49917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.311+0000 7f49917fa640 10 monclient: dump: 2024-09-10T16:27:07.326 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f498803c820 con 0x7f499413a060 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:07.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 msgr2=0x7f499413a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 0x7f499413a440 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4988034000 tx=0x7f49880046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 msgr2=0x7f499413a440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f49917fa640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a060 0x7f499413a440 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4988034000 tx=0x7f49880046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.312+0000 7f4999390640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f498003e210 0x7f49800406d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f498c007fd0 tx=0x7f498c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f499adf3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f499adf3640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 0x7f499413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f499adf3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f499409e520 con 0x7f499413a980 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 0x7f499413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient: get_auth_request con 0x7f499413a980 auth_method 0 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 10 monclient(hunting): handle_auth_done global_id 224217 payload 824 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.318+0000 7f4993fff640 1 -- 172.21.15.164:0/2393008462 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f49940770f0 con 0x7f499413a980 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f4993fff640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 msgr2=0x7f499413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f4993fff640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 0x7f499413d940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4988000c00 tx=0x7f4988008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4988053070 con 0x7f499413a980 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f49917fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f49917fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.320+0000 7f49917fa640 10 monclient: dump: 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.329 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f4988046a40 con 0x7f499413a980 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 msgr2=0x7f499413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 0x7f499413d940 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4988000c00 tx=0x7f4988008030 comp rx=0 tx=0).stop 2024-09-10T16:27:07.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 msgr2=0x7f499413d940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f49917fa640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499413a980 0x7f499413d940 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4988000c00 tx=0x7f4988008030 comp rx=0 tx=0).stop 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 79151, 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:07.331 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.332 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.333 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:07.334 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.335 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.336 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.337 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.338 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.339 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.340 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.341 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.342 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.343 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:07.344 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f498003e210 msgr2=0x7f49800406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f498003e210 0x7f49800406d0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f498c007fd0 tx=0x7f498c009040 comp rx=0 tx=0).stop 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 10 monclient: shutdown 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 msgr2=0x7f49941cd530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 --2- 172.21.15.164:0/2393008462 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f499406cb70 0x7f49941cd530 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f498400b410 tx=0x7f498400b8d0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f4999390640 1 -- 172.21.15.164:0/2393008462 reap_dead start 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f4999390640 1 -- 172.21.15.164:0/2393008462 reap_dead start 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 shutdown_connections 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 >> 172.21.15.164:0/2393008462 conn(0x7f49940aa900 msgr2=0x7f49941335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 shutdown_connections 2024-09-10T16:27:07.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.321+0000 7f499adf3640 1 -- 172.21.15.164:0/2393008462 wait complete. 2024-09-10T16:27:07.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:07.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:27:07.347 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:27:07.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.444+0000 7f5c14cb6640 10 monclient: get_monmap_and_config 2024-09-10T16:27:07.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.444+0000 7f5c14cb6640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.444+0000 7f5c14cb6640 10 monclient: monmap: 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:07.445082+0000 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:07.445082+0000 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:07.446 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.444+0000 7f5c14cb6640 1 Processor -- start 2024-09-10T16:27:07.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 1 -- start start 2024-09-10T16:27:07.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient: init 2024-09-10T16:27:07.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5c1006cb70 0x7f5c10134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c1006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5c100a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5c100a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c1009ce50 con 0x7f5c100a4ab0 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c10138ac0 con 0x7f5c100a46a0 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5c1006db60 con 0x7f5c1006cb70 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c14cb6640 10 monclient(hunting): authenticate will time out at 2005.932503s 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0ed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5c100a4ab0 0x7f5c101383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47112/0 (socket says 172.21.15.164:47112) 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3938856397 learned_addr learned my addr 172.21.15.164:0/3938856397 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3938856397 >> v2:172.21.15.165:3300/0 conn(0x7f5c1006cb70 0x7f5c10134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request con 0x7f5c1006cb70 auth_method 0 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.450 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.445+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_done global_id 234109 payload 274 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3938856397 >> v1:172.21.15.165:6789/0 conn(0x7f5c100a46a0 legacy=0x7f5c10137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3938856397 >> v1:172.21.15.164:6789/0 conn(0x7f5c100a4ab0 legacy=0x7f5c101383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 1 monclient: found mon.noname-c 2024-09-10T16:27:07.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:27:07.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.446+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3938856397 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c1009cb70 con 0x7f5c1006cb70 2024-09-10T16:27:07.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.448+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3938856397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 msgr2=0x7f5c10134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.462 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.448+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3938856397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c10134f70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097d0 tx=0x7f5c0002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=359cb4152f73c902 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 1 -- 172.21.15.164:0/3938856397 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00037070 con 0x7f5c1006cb70 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: dump: 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.463 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 1 -- 172.21.15.164:0/3938856397 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c00032440 con 0x7f5c1006cb70 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 5 monclient: authenticate success, global_id 234109 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.464 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 4 set_mon_vals no callback set 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 1 -- 172.21.15.164:0/3938856397 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00032810 con 0x7f5c1006cb70 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c0dd9b640 10 monclient: dump: 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.465 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 10 monclient: shutdown 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3938856397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 msgr2=0x7f5c10134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3938856397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c10134f70 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f5c000097d0 tx=0x7f5c0002a000 comp rx=0 tx=0).stop 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3938856397 shutdown_connections 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3938856397 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c10134f70 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3938856397 >> 172.21.15.164:0/3938856397 conn(0x7f5c100aa900 msgr2=0x7f5c100aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3938856397 shutdown_connections 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.449+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3938856397 wait complete. 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.450+0000 7f5c14cb6640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.450+0000 7f5c14cb6640 10 monclient: monmap: 2024-09-10T16:27:07.466 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.450+0000 7f5c14cb6640 1 Processor -- start 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 -- start start 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient: init 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.467 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c101cd680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-b con 0x7f5c1006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c100a46a0 0x7f5c100a3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-c con 0x7f5c100a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): picked mon.noname-a con 0x7f5c100a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5c10141050 con 0x7f5c100a4ab0 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1006cf50 con 0x7f5c1006cb70 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c101411d0 con 0x7f5c100a46a0 2024-09-10T16:27:07.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c14cb6640 10 monclient(hunting): authenticate will time out at 2005.938483s 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c100a46a0 0x7f5c100a3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38738/0 (socket says 172.21.15.164:38738) 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 learned_addr learned my addr 172.21.15.164:0/3534576536 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): get_auth_request con 0x7f5c100a46a0 auth_method 0 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): get_auth_request con 0x7f5c100a4ab0 auth_method 0 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c101cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request con 0x7f5c1006cb70 auth_method 0 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient(hunting): handle_auth_done global_id 224220 payload 824 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c100a46a0 msgr2=0x7f5c100a3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c100a46a0 0x7f5c100a3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 msgr2=0x7f5c101cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c101cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c100a00d0 con 0x7f5c100a4ab0 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c07fff640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c100a46a0 0x7f5c100a3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.451+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5c1006cb70 0x7f5c101cd680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 msgr2=0x7f5c100a3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c0ed9d640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800caf0 tx=0x7f5c0800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c08012400 con 0x7f5c100a4ab0 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: dump: 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.472 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c080125c0 con 0x7f5c100a4ab0 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 5 monclient: authenticate success, global_id 224220 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 10 monclient: _renew_subs 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c10134030 con 0x7f5c100a4ab0 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c080129b0 con 0x7f5c100a4ab0 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c06ffd640 10 monclient: dump: 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.474 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c0d59a640 4 set_mon_vals no callback set 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 10 monclient: _renew_subs 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.454+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c100a2430 con 0x7f5c100a4ab0 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c14cb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 0x7f5c1009ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c10132900 con 0x7f5c1009e8e0 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5c08012b90 con 0x7f5c100a4ab0 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 0x7f5c1009ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient: get_auth_request con 0x7f5c1009e8e0 auth_method 0 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c06ffd640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_done global_id 214304 payload 824 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(710..710 src has 494..710) ==== 4815+0+0 (secure 0 0 0) 0x7f5c080533c0 con 0x7f5c100a4ab0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c10134f70 con 0x7f5c1009e8e0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c07fff640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.455+0000 7f5c07fff640 10 monclient: get_auth_request con 0x7f5bf003e210 auth_method 0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 msgr2=0x7f5c1009ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 0x7f5c1009ecc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c00034000 tx=0x7f5c000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00043070 con 0x7f5c1009e8e0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.457+0000 7f5c06ffd640 10 monclient: dump: 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c0003c650 con 0x7f5c1009e8e0 2024-09-10T16:27:07.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 msgr2=0x7f5c1009ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 0x7f5c1009ecc0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c00034000 tx=0x7f5c000046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 msgr2=0x7f5c1009ecc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c06ffd640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009e8e0 0x7f5c1009ecc0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f5c00034000 tx=0x7f5c000046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.458+0000 7f5c07fff640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5c100a0eb0 tx=0x7f5bf8007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c14cb6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 0x7f5c1012f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c14cb6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5c100a0c30 con 0x7f5c1009f200 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 0x7f5c1012f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient: get_auth_request con 0x7f5c1009f200 auth_method 0 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 10 monclient(hunting): handle_auth_done global_id 214307 payload 824 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.464+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3534576536 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c100a1bf0 con 0x7f5c1009f200 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c0e59c640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 msgr2=0x7f5c1012f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c0e59c640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 0x7f5c1012f240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5c0002a040 tx=0x7f5c00009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00054020 con 0x7f5c1009f200 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 10 monclient: dump: 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f5c00003810 con 0x7f5c1009f200 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 msgr2=0x7f5c1012f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 0x7f5c1012f240 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5c0002a040 tx=0x7f5c00009190 comp rx=0 tx=0).stop 2024-09-10T16:27:07.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 msgr2=0x7f5c1012f240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.466+0000 7f5c06ffd640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5c1009f200 0x7f5c1012f240 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5c0002a040 tx=0x7f5c00009190 comp rx=0 tx=0).stop 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 127244, 2024-09-10T16:27:07.484 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.485 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.486 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:07.487 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:07.488 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.489 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.490 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.491 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.492 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:07.493 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.494 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:07.495 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.496 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.496 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.496 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:07.496 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.497 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.498 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.498 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.498 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.498 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.498 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.499 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.500 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.501 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.502 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.503 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:27:07.504 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.505 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:27:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:27:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.506 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.507 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:07.508 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:07.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 msgr2=0x7f5bf00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5bf003e210 0x7f5bf00406d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5c100a0eb0 tx=0x7f5bf8007e30 comp rx=0 tx=0).stop 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 10 monclient: shutdown 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 msgr2=0x7f5c100a3ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5c0800caf0 tx=0x7f5c0800a060 comp rx=0 tx=0).stop 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c0ed9d640 1 -- 172.21.15.164:0/3534576536 reap_dead start 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 shutdown_connections 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 --2- 172.21.15.164:0/3534576536 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5c100a4ab0 0x7f5c100a3ea0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 >> 172.21.15.164:0/3534576536 conn(0x7f5c100aa900 msgr2=0x7f5c100a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 shutdown_connections 2024-09-10T16:27:07.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.467+0000 7f5c14cb6640 1 -- 172.21.15.164:0/3534576536 wait complete. 2024-09-10T16:27:07.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:07.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:27:07.511 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.608+0000 7f6efdaef640 10 monclient: get_monmap_and_config 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.608+0000 7f6efdaef640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.608+0000 7f6efdaef640 10 monclient: monmap: 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:07.609822+0000 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:07.609822+0000 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:07.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.609+0000 7f6efdaef640 1 Processor -- start 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.609+0000 7f6efdaef640 1 -- start start 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.609+0000 7f6efdaef640 10 monclient: init 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ef809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ef809d160 0x7f6ef809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6ef813a5c0 0x7f6ef813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ef809a430 con 0x7f6ef809c180 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef80a24e0 con 0x7f6ef809d160 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6ef80a2bf0 con 0x7f6ef813a5c0 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6efdaef640 10 monclient(hunting): authenticate will time out at 2006.097273s 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ef809d160 0x7f6ef809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6ef809d160 0x7f6ef809f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38752/0 (socket says 172.21.15.164:38752) 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/4007167783 learned_addr learned my addr 172.21.15.164:0/4007167783 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f6ef809d160 auth_method 0 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef77fe640 1 --1- 172.21.15.164:0/4007167783 >> v1:172.21.15.164:6789/0 conn(0x7f6ef809c180 0x7f6ef809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47118/0 (socket says 172.21.15.164:47118) 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef7fff640 1 --2- 172.21.15.164:0/4007167783 >> v2:172.21.15.165:3300/0 conn(0x7f6ef813a5c0 0x7f6ef813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_done global_id 224223 payload 274 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/4007167783 >> v2:172.21.15.165:3300/0 conn(0x7f6ef813a5c0 msgr2=0x7f6ef813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/4007167783 >> v2:172.21.15.165:3300/0 conn(0x7f6ef813a5c0 0x7f6ef813c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/4007167783 >> v1:172.21.15.164:6789/0 conn(0x7f6ef809c180 legacy=0x7f6ef809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.610+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/4007167783 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef809c560 con 0x7f6ef809d160 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/4007167783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809d160 msgr2=0x7f6ef809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/4007167783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809d160 0x7f6ef809f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6ea02fa7f4a7cf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 1 -- 172.21.15.164:0/4007167783 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec037070 con 0x7f6ef809d160 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: dump: 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 1 -- 172.21.15.164:0/4007167783 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6eec032640 con 0x7f6ef809d160 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6efdaef640 5 monclient: authenticate success, global_id 224223 2024-09-10T16:27:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6efdaef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 1 -- 172.21.15.164:0/4007167783 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec032a10 con 0x7f6ef809d160 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6efdaef640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6efdaef640 4 set_mon_vals no callback set 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.613+0000 7f6ef67fc640 10 monclient: dump: 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 10 monclient: shutdown 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 -- 172.21.15.164:0/4007167783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809d160 msgr2=0x7f6ef809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 --2- 172.21.15.164:0/4007167783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809d160 0x7f6ef809f150 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6eec009870 tx=0x7f6eec02a100 comp rx=0 tx=0).stop 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 -- 172.21.15.164:0/4007167783 shutdown_connections 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 --2- 172.21.15.164:0/4007167783 >> v2:172.21.15.165:3300/0 conn(0x7f6ef813a5c0 0x7f6ef813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 --2- 172.21.15.164:0/4007167783 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809d160 0x7f6ef809f150 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 -- 172.21.15.164:0/4007167783 >> 172.21.15.164:0/4007167783 conn(0x7f6ef80a3110 msgr2=0x7f6ef80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 -- 172.21.15.164:0/4007167783 shutdown_connections 2024-09-10T16:27:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 1 -- 172.21.15.164:0/4007167783 wait complete. 2024-09-10T16:27:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.614+0000 7f6efdaef640 10 monclient: monmap: 2024-09-10T16:27:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 Processor -- start 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 -- start start 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient: init 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 0x7f6ef809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-a con 0x7f6ef809c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ef809d160 0x7f6ef808f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-c con 0x7f6ef809d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef813a5c0 0x7f6ef808fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): picked mon.noname-b con 0x7f6ef813a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8074f90 con 0x7f6ef809c180 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8074e10 con 0x7f6ef813a5c0 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ef809f690 con 0x7f6ef809d160 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6efdaef640 10 monclient(hunting): authenticate will time out at 2006.102726s 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 0x7f6ef809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ef809d160 0x7f6ef808f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 0x7f6ef809d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38762/0 (socket says 172.21.15.164:38762) 2024-09-10T16:27:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 1 -- 172.21.15.164:0/3165465501 learned_addr learned my addr 172.21.15.164:0/3165465501 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): get_auth_request con 0x7f6ef809c180 auth_method 0 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request con 0x7f6ef809d160 auth_method 0 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.615+0000 7f6ef7fff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef813a5c0 0x7f6ef808fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef7fff640 10 monclient(hunting): get_auth_request con 0x7f6ef813a5c0 auth_method 0 2024-09-10T16:27:07.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 10 monclient(hunting): handle_auth_done global_id 224226 payload 824 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ef809d160 msgr2=0x7f6ef808f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ef809d160 0x7f6ef808f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef813a5c0 msgr2=0x7f6ef808fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef813a5c0 0x7f6ef808fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.616+0000 7f6ef77fe640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ef80992d0 con 0x7f6ef809c180 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6ef77fe640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 msgr2=0x7f6ef809d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6ef77fe640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 0x7f6ef809d930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800c450 tx=0x7f6ee800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8017070 con 0x7f6ef809c180 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: dump: 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.635 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ee8005810 con 0x7f6ef809c180 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 5 monclient: authenticate success, global_id 224226 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 10 monclient: _renew_subs 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ef80960c0 con 0x7f6ef809c180 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ee8005c00 con 0x7f6ef809c180 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6edffff640 10 monclient: dump: 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.637 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.638 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6ef5ffb640 4 set_mon_vals no callback set 2024-09-10T16:27:07.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 10 monclient: _renew_subs 2024-09-10T16:27:07.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.618+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ef80a5120 con 0x7f6ef809c180 2024-09-10T16:27:07.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6efdaef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6efdaef640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 0x7f6ef8090020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef809d680 con 0x7f6ef802e5c0 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6ee801a070 con 0x7f6ef809c180 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 0x7f6ef8090020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient: get_auth_request con 0x7f6ef802e5c0 auth_method 0 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6edffff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e240 0x7f6ee4040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(710..710 src has 494..710) ==== 4815+0+0 (secure 0 0 0) 0x7f6ee8052240 con 0x7f6ef809c180 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_done global_id 234118 payload 824 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ef802e9a0 con 0x7f6ef802e5c0 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.619+0000 7f6ef7fff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e240 0x7f6ee4040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.620+0000 7f6ef7fff640 10 monclient: get_auth_request con 0x7f6ee403e240 auth_method 0 2024-09-10T16:27:07.643 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.621+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.644 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.621+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844045410}) 2024-09-10T16:27:07.644 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.621+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=153596729}) 2024-09-10T16:27:07.644 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.621+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.644 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.621+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844045410}) 2024-09-10T16:27:07.644 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.622+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=122083138}) 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 msgr2=0x7f6ef8090020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 0x7f6ef8090020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6eec034000 tx=0x7f6eec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec032e60 con 0x7f6ef802e5c0 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6ef7fff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e240 0x7f6ee4040700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8090bd0 tx=0x7f6ef00074d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:07.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 10 monclient: dump: 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.647 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6eec03c850 con 0x7f6ef802e5c0 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 msgr2=0x7f6ef8090020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 0x7f6ef8090020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6eec034000 tx=0x7f6eec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 msgr2=0x7f6ef8090020 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.622+0000 7f6edffff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef802e5c0 0x7f6ef8090020 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6eec034000 tx=0x7f6eec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.628+0000 7f6efdaef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.628+0000 7f6efdaef640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 0x7f6ef8064160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.628+0000 7f6efdaef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.628+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6ef8090560 con 0x7f6ef8063d50 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 0x7f6ef8064160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient: get_auth_request con 0x7f6ef8063d50 auth_method 0 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 10 monclient(hunting): handle_auth_done global_id 234121 payload 824 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.629+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/3165465501 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ef8072760 con 0x7f6ef8063d50 2024-09-10T16:27:07.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.631+0000 7f6ef6ffd640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 msgr2=0x7f6ef8064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.631+0000 7f6ef6ffd640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 0x7f6ef8064160 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6eec02a140 tx=0x7f6eec009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eec032e60 con 0x7f6ef8063d50 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 10 monclient: dump: 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.650 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.651 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070134510}) 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8552144}) 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4070134510}) 2024-09-10T16:27:07.652 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.632+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1414642268}) 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f6eec051400 con 0x7f6ef8063d50 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 msgr2=0x7f6ef8064160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 0x7f6ef8064160 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6eec02a140 tx=0x7f6eec009240 comp rx=0 tx=0).stop 2024-09-10T16:27:07.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 msgr2=0x7f6ef8064160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6edffff640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ef8063d50 0x7f6ef8064160 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6eec02a140 tx=0x7f6eec009240 comp rx=0 tx=0).stop 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 202, 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27916, 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:07.654 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.655 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.656 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.657 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:07.658 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:07.659 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:07.660 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.661 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.662 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.663 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.664 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:07.665 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.666 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.667 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:07.668 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:07.669 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:07.669 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:07.670 INFO:tasks.rados.rados.0.smithi165.stdout:3199: done (2 left) 2024-09-10T16:27:07.670 INFO:tasks.rados.rados.0.smithi165.stdout:3200: delete oid 38 current snap is 342 2024-09-10T16:27:07.670 INFO:tasks.rados.rados.0.smithi165.stdout:3197: finishing write tid 2 to smithi16534841-22 2024-09-10T16:27:07.670 INFO:tasks.rados.rados.0.smithi165.stdout:3197: finishing write tid 3 to smithi16534841-22 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3197: finishing write tid 4 to smithi16534841-22 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3197: oid 22 updating version 0 to 1455 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3197: oid 22 updating version 1455 to 1456 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3197: oid 22 updating version 1456 to 1457 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1457 (ObjNum 1117 snap 342 seq_num 1117) dirty exists 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3197: left oid 22 (ObjNum 1117 snap 342 seq_num 1117) 2024-09-10T16:27:07.671 INFO:tasks.rados.rados.0.smithi165.stdout:3198: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3198: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3198: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1644 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3197: done (2 left) 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3198: done (1 left) 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3200: done (0 left) 2024-09-10T16:27:07.672 INFO:tasks.rados.rados.0.smithi165.stdout:3201: snap_create 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e240 msgr2=0x7f6ee4040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6ee403e240 0x7f6ee4040700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8090bd0 tx=0x7f6ef00074d0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 10 monclient: shutdown 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 msgr2=0x7f6ef809d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 --2- 172.21.15.164:0/3165465501 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ef809c180 0x7f6ef809d930 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ee800c450 tx=0x7f6ee800c910 comp rx=0 tx=0).stop 2024-09-10T16:27:07.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6ef7fff640 1 -- 172.21.15.164:0/3165465501 reap_dead start 2024-09-10T16:27:07.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6ef7fff640 1 -- 172.21.15.164:0/3165465501 reap_dead start 2024-09-10T16:27:07.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 shutdown_connections 2024-09-10T16:27:07.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 >> 172.21.15.164:0/3165465501 conn(0x7f6ef80a3110 msgr2=0x7f6ef8139a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 shutdown_connections 2024-09-10T16:27:07.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.632+0000 7f6efdaef640 1 -- 172.21.15.164:0/3165465501 wait complete. 2024-09-10T16:27:07.675 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:07.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:27:07.675 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:27:07.675 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:27:07.675 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:27:07.675 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:27:07.675 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524686283}) 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=306864774}) 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3524686283}) 2024-09-10T16:27:07.681 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.647+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353423169}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042050210}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2517225689}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1042050210}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.649+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1668184600}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2309740068,mdsmap=931935893}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=2142472737,mdsmap=2674069449}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=2309740068,mdsmap=931935893}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.665+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=97,mdsmap=3} crc {logm=383197211,mdsmap=931935893}) 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.682 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {mgr=2188831455,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2588574553,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1427281147,osd_snap=1566514970}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=53} crc {mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2588574553,mon_config_key=647691077,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1427281147,osd_snap=1441729534}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=54} crc {mgr=2188831455,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2588574553,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1427281147,osd_snap=1566514970}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=53} crc {mgr=3329267774,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2588574553,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1427281147,osd_snap=1253556771}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=82,osdmap=18} crc {osd_snap=3872914551,osdmap=2278047712}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=79,osdmap=21} crc {osd_snap=2571585171,osdmap=378569244}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=82,osdmap=18} crc {osd_snap=3872914551,osdmap=2278047712}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.667+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=78,osdmap=22} crc {osd_snap=2435299583,osdmap=1412368240}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=601623543}) 2024-09-10T16:27:07.683 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1746745559}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=601623543}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.668+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2847527990}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3406796134}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2545279476}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3406796134}) 2024-09-10T16:27:07.684 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.669+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1428747330}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=601420251}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2259447972}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=601420251}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.670+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3032184007}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=2004318701}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=16} crc {osdmap=548573507}) 2024-09-10T16:27:07.685 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:07.686 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=15} crc {osdmap=2004318701}) 2024-09-10T16:27:07.686 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:07.675+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=16} crc {osdmap=548573507}) 2024-09-10T16:27:07.688 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:27:07.688 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:27:07.688 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.823+0000 7f38b9c8c640 10 monclient: get_monmap_and_config 2024-09-10T16:27:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.823+0000 7f38b9c8c640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.823+0000 7f38b9c8c640 10 monclient: monmap: 2024-09-10T16:27:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:07.824801+0000 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:07.824801+0000 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:07.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.824+0000 7f38b9c8c640 1 Processor -- start 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.824+0000 7f38b9c8c640 1 -- start start 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.824+0000 7f38b9c8c640 10 monclient: init 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f38b409c180 0x7f38b409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-c con 0x7f38b409c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f38b409d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38b413a5c0 0x7f38b413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f38b413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38b409a430 con 0x7f38b409d160 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f38b40a24e0 con 0x7f38b413a5c0 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f38b40a2bf0 con 0x7f38b409c180 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b9c8c640 10 monclient(hunting): authenticate will time out at 2006.312322s 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38b413a5c0 0x7f38b413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f38b413a5c0 0x7f38b413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38794/0 (socket says 172.21.15.164:38794) 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38aaffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f38b409d160 0x7f38b409ef80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47132/0 (socket says 172.21.15.164:47132) 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 -- 172.21.15.164:0/1904289103 learned_addr learned my addr 172.21.15.164:0/1904289103 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): get_auth_request con 0x7f38b413a5c0 auth_method 0 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b37fe640 1 --2- 172.21.15.164:0/1904289103 >> v2:172.21.15.165:3300/0 conn(0x7f38b409c180 0x7f38b409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b37fe640 10 monclient(hunting): get_auth_request con 0x7f38b409c180 auth_method 0 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient(hunting): handle_auth_done global_id 224229 payload 274 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 -- 172.21.15.164:0/1904289103 >> v2:172.21.15.165:3300/0 conn(0x7f38b409c180 msgr2=0x7f38b409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 --2- 172.21.15.164:0/1904289103 >> v2:172.21.15.165:3300/0 conn(0x7f38b409c180 0x7f38b409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 -- 172.21.15.164:0/1904289103 >> v1:172.21.15.164:6789/0 conn(0x7f38b409d160 legacy=0x7f38b409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.825+0000 7f38b3fff640 1 -- 172.21.15.164:0/1904289103 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b409c560 con 0x7f38b413a5c0 2024-09-10T16:27:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b3fff640 1 -- 172.21.15.164:0/1904289103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b413a5c0 msgr2=0x7f38b413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b3fff640 1 --2- 172.21.15.164:0/1904289103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b413a5c0 0x7f38b413c9b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009e50 tx=0x7f38ac02a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20fc28ea11d436c6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 1 -- 172.21.15.164:0/1904289103 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac036070 con 0x7f38b413a5c0 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: dump: 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.838 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b9c8c640 5 monclient: authenticate success, global_id 224229 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b9c8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 1 -- 172.21.15.164:0/1904289103 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38ac007800 con 0x7f38b413a5c0 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b9c8c640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b9c8c640 4 set_mon_vals no callback set 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 1 -- 172.21.15.164:0/1904289103 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38ac007bd0 con 0x7f38b413a5c0 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.828+0000 7f38b2ffd640 10 monclient: dump: 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 10 monclient: shutdown 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:07.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 -- 172.21.15.164:0/1904289103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b413a5c0 msgr2=0x7f38b413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/1904289103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b413a5c0 0x7f38b413c9b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f38ac009e50 tx=0x7f38ac02a5b0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 -- 172.21.15.164:0/1904289103 shutdown_connections 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/1904289103 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b413a5c0 0x7f38b413c9b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/1904289103 >> v2:172.21.15.165:3300/0 conn(0x7f38b409c180 0x7f38b409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 -- 172.21.15.164:0/1904289103 >> 172.21.15.164:0/1904289103 conn(0x7f38b40a3110 msgr2=0x7f38b40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 -- 172.21.15.164:0/1904289103 shutdown_connections 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 1 -- 172.21.15.164:0/1904289103 wait complete. 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 10 monclient: build_initial_monmap 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.829+0000 7f38b9c8c640 10 monclient: monmap: 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:07.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 Processor -- start 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 -- start start 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient: init 2024-09-10T16:27:07.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-a con 0x7f38b409c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 0x7f38b4091e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-c con 0x7f38b409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38b413a5c0 0x7f38b40923d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): picked mon.noname-b con 0x7f38b413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f38b4074f90 con 0x7f38b409c180 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f38b4074e10 con 0x7f38b413a5c0 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f38b409f690 con 0x7f38b409d160 2024-09-10T16:27:07.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38b9c8c640 10 monclient(hunting): authenticate will time out at 2006.317954s 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38aaffd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 msgr2=0x7f38b4091e90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.830+0000 7f38aaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 0x7f38b4091e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38802/0 (socket says 172.21.15.164:38802) 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 -- 172.21.15.164:0/4108452482 learned_addr learned my addr 172.21.15.164:0/4108452482 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): get_auth_request con 0x7f38b409c180 auth_method 0 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:07.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b3fff640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38b413a5c0 0x7f38b40923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient(hunting): handle_auth_done global_id 224232 payload 824 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 -- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 msgr2=0x7f38b4091e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 0x7f38b4091e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 -- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38b413a5c0 msgr2=0x7f38b40923d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38b413a5c0 0x7f38b40923d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:07.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.831+0000 7f38b37fe640 1 -- 172.21.15.164:0/4108452482 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b4095480 con 0x7f38b409c180 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b37fe640 1 -- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 msgr2=0x7f38b4091950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:07.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b37fe640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f38a400d800 tx=0x7f38a400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a400b8d0 con 0x7f38b409c180 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: dump: 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 5 monclient: authenticate success, global_id 224232 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38a4019070 con 0x7f38b409c180 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 10 monclient: _renew_subs 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a400bc90 con 0x7f38b409c180 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b409d620 con 0x7f38b409c180 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b0ff9640 10 monclient: dump: 2024-09-10T16:27:07.850 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b27fc640 4 set_mon_vals no callback set 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 10 monclient: _renew_subs 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.833+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b40a5df0 con 0x7f38b409c180 2024-09-10T16:27:07.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b9c8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b9c8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b9c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f38a401e460 con 0x7f38b409c180 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b409ea40 con 0x7f38b409c180 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b0ff9640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 0x7f38940406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.834+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(711..711 src has 494..711) ==== 4831+0+0 (secure 0 0 0) 0x7f38a4053260 con 0x7f38b409c180 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.835+0000 7f38aaffd640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 0x7f38940406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:07.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.835+0000 7f38aaffd640 10 monclient: get_auth_request con 0x7f389403e1f0 auth_method 0 2024-09-10T16:27:07.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.837+0000 7f38aaffd640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 0x7f38940406b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f38b4093790 tx=0x7f389c007a60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:07.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.839+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f38a4051c30 con 0x7f38b409c180 2024-09-10T16:27:07.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.839+0000 7f38b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:07.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:07.839+0000 7f38b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:08.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b9c8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b9c8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b9c8c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f38b4093f50 con 0x7f38b409c180 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b0ff9640 1 -- 172.21.15.164:0/4108452482 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f38a401da00 con 0x7f38b409c180 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.082+0000 7f38b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:08.084 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":202,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 msgr2=0x7f38940406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 0x7f38940406b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f38b4093790 tx=0x7f389c007a60 comp rx=0 tx=0).stop 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 10 monclient: shutdown 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 msgr2=0x7f38b4091950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:08.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f38a400d800 tx=0x7f38a400dcc0 comp rx=0 tx=0).stop 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 shutdown_connections 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f389403e1f0 0x7f38940406b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f38b413a5c0 0x7f38b40923d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38b409d160 0x7f38b4091e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 --2- 172.21.15.164:0/4108452482 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f38b409c180 0x7f38b4091950 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 >> 172.21.15.164:0/4108452482 conn(0x7f38b40a3110 msgr2=0x7f38b4139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 shutdown_connections 2024-09-10T16:27:08.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:08.084+0000 7f38b9c8c640 1 -- 172.21.15.164:0/4108452482 wait complete. 2024-09-10T16:27:11.097 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.205+0000 7f4b7d839640 10 monclient: get_monmap_and_config 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.205+0000 7f4b7d839640 10 monclient: build_initial_monmap 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.205+0000 7f4b7d839640 10 monclient: monmap: 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:11.206648+0000 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:11.206648+0000 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:11.207 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:11.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 Processor -- start 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 -- start start 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient: init 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b7806cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4b78134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b78140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7813e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b7809ce50 con 0x7f4b7806cb70 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b78135380 con 0x7f4b78134f70 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7806db60 con 0x7f4b7813e740 2024-09-10T16:27:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b7d839640 10 monclient(hunting): authenticate will time out at 2009.693978s 2024-09-10T16:27:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.206+0000 7f4b777fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 msgr2=0x7f4b78140b30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.207+0000 7f4b777fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b78140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.207+0000 7f4b76ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4b7806cb70 0x7f4b780a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:47136/0 (socket says 172.21.15.164:47136) 2024-09-10T16:27:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.207+0000 7f4b76ffd640 1 -- 172.21.15.164:0/3562825528 learned_addr learned my addr 172.21.15.164:0/3562825528 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b76ffd640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.164:6789/0 conn(0x7f4b7806cb70 legacy=0x7f4b780a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 legacy=0x7f4b7813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645955459 0 0) 0x7f4b68001450 con 0x7f4b7806cb70 2024-09-10T16:27:11.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): _negotiate 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): my global_id is 224235 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b64003520 con 0x7f4b7806cb70 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 legacy=0x7f4b7813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 legacy=0x7f4b7813d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 --1- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 0x7f4b7813d020 :-1 s=OPENED pgs=43 cs=1 l=1).handle_message read tag failed 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 --1- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 0x7f4b7813d020 :-1 s=OPENED pgs=43 cs=1 l=1).fault on lossy channel, failing 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.165:6789/0 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4282536810 0 0) 0x7f4b64003520 con 0x7f4b7806cb70 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 msgr2=0x7f4b78140b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b78140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.165:6789/0 conn(0x7f4b78134f70 legacy=0x7f4b7813d020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _finish_hunting 0 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7809cb70 con 0x7f4b7806cb70 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f4b68001b70 con 0x7f4b7806cb70 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: dump: 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:27:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 >> v1:172.21.15.164:6789/0 conn(0x7f4b7806cb70 legacy=0x7f4b780a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b64003a20 0x7f4b64003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): picked mon.b con 0x7f4b64003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b64004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): picked mon.c con 0x7f4b7813e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 0x7f4b6400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): picked mon.a con 0x7f4b64004ef0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6400bf00 con 0x7f4b64004ef0 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b6400c080 con 0x7f4b64003a20 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4b6400c200 con 0x7f4b7813e740 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b767fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 msgr2=0x7f4b64004800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b6e7fc640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b64004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b777fe640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 0x7f4b6400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b777fe640 10 monclient(hunting): get_auth_request con 0x7f4b64004ef0 auth_method 0 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b777fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.209+0000 7f4b76ffd640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b64003a20 0x7f4b64003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): get_auth_request con 0x7f4b64003a20 auth_method 0 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b76ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient(hunting): handle_auth_done global_id 224235 payload 293 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 msgr2=0x7f4b64004800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b7813e740 0x7f4b64004800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b64003a20 msgr2=0x7f4b64003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b64003a20 0x7f4b64003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:11.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 monclient: found mon.a 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 1 -- 172.21.15.164:0/3562825528 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b64003520 con 0x7f4b64004ef0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b7d839640 5 monclient: authenticate success, global_id 224235 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.210+0000 7f4b7d839640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.212+0000 7f4b777fe640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 msgr2=0x7f4b6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.212+0000 7f4b777fe640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 0x7f4b6400b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4b70003d00 tx=0x7f4b64005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a0c3e46eba4c41 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70009a50 con 0x7f4b64004ef0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:11.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: dump: 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b70034070 con 0x7f4b64004ef0 2024-09-10T16:27:11.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 4 set_mon_vals no callback set 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 1 -- 172.21.15.164:0/3562825528 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70005700 con 0x7f4b64004ef0 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b767fc640 10 monclient: dump: 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:11.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 10 monclient: shutdown 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 -- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 msgr2=0x7f4b6400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3562825528 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b64004ef0 0x7f4b6400b9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f4b70003d00 tx=0x7f4b64005830 comp rx=0 tx=0).stop 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b777fe640 1 -- 172.21.15.164:0/3562825528 reap_dead start 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 -- 172.21.15.164:0/3562825528 shutdown_connections 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 -- 172.21.15.164:0/3562825528 >> 172.21.15.164:0/3562825528 conn(0x7f4b780aa900 msgr2=0x7f4b780aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 -- 172.21.15.164:0/3562825528 shutdown_connections 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.213+0000 7f4b7d839640 1 -- 172.21.15.164:0/3562825528 wait complete. 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.214+0000 7f4b7d839640 10 monclient: build_initial_monmap 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.214+0000 7f4b7d839640 10 monclient: monmap: 2024-09-10T16:27:11.225 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.214+0000 7f4b7d839640 1 Processor -- start 2024-09-10T16:27:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.214+0000 7f4b7d839640 1 -- start start 2024-09-10T16:27:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.214+0000 7f4b7d839640 10 monclient: init 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 0x7f4b7813d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-c con 0x7f4b64004ef0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-a con 0x7f4b7809d5f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7809dfb0 0x7f4b781d3ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): picked mon.noname-b con 0x7f4b7809dfb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4b780a4db0 con 0x7f4b7809d5f0 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7806cf50 con 0x7f4b7809dfb0 2024-09-10T16:27:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4b7813eb20 con 0x7f4b64004ef0 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b7d839640 10 monclient(hunting): authenticate will time out at 2009.702445s 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b76ffd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 msgr2=0x7f4b7813d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b76ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 0x7f4b7813d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:38820/0 (socket says 172.21.15.164:38820) 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3693440217 learned_addr learned my addr 172.21.15.164:0/3693440217 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): get_auth_request con 0x7f4b7809d5f0 auth_method 0 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b777fe640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7809dfb0 0x7f4b781d3ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b777fe640 10 monclient(hunting): get_auth_request con 0x7f4b7809dfb0 auth_method 0 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b777fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b777fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient(hunting): handle_auth_done global_id 224238 payload 824 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 msgr2=0x7f4b7813d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 0x7f4b7813d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7809dfb0 msgr2=0x7f4b781d3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7809dfb0 0x7f4b781d3ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 monclient: found mon.noname-a 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.215+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3693440217 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7806e720 con 0x7f4b7809d5f0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b6e7fc640 1 -- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 msgr2=0x7f4b7809da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b6e7fc640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c730 tx=0x7f4b6000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60007480 con 0x7f4b7809d5f0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: dump: 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b60007620 con 0x7f4b7809d5f0 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b60005a00 con 0x7f4b7809d5f0 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 5 monclient: authenticate success, global_id 224238 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 10 monclient: _renew_subs 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b780a0a30 con 0x7f4b7809d5f0 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b74ff9640 10 monclient: dump: 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b75ffb640 4 set_mon_vals no callback set 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 10 monclient: _renew_subs 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.218+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7809edb0 con 0x7f4b7809d5f0 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4b60005c60 con 0x7f4b7809d5f0 2024-09-10T16:27:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b74ff9640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 0x7f4b58040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b7d839640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b7d839640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b7d839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b780a46a0 con 0x7f4b7809d5f0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(711..711 src has 494..711) ==== 4831+0+0 (secure 0 0 0) 0x7f4b60052b20 con 0x7f4b7809d5f0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b76ffd640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 0x7f4b58040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.219+0000 7f4b76ffd640 10 monclient: get_auth_request con 0x7f4b5803e1c0 auth_method 0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.222+0000 7f4b76ffd640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 0x7f4b58040680 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b68000c00 tx=0x7f4b68004bb0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.224+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f4b7806e720 con 0x7f4b7809d5f0 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.224+0000 7f4b74ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.224+0000 7f4b74ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.467+0000 7f4b7d839640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.467+0000 7f4b7d839640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:11.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.467+0000 7f4b7d839640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.467+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b7813df70 con 0x7f4b7809d5f0 2024-09-10T16:27:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.468+0000 7f4b74ff9640 1 -- 172.21.15.164:0/3693440217 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f4b6001b400 con 0x7f4b7809d5f0 2024-09-10T16:27:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.468+0000 7f4b74ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:11.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.468+0000 7f4b74ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:11.470 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":202,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 msgr2=0x7f4b58040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 0x7f4b58040680 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f4b68000c00 tx=0x7f4b68004bb0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 10 monclient: shutdown 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 msgr2=0x7f4b7809da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:11.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f4b6000c730 tx=0x7f4b6000cbf0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 shutdown_connections 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4b5803e1c0 0x7f4b58040680 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4b7809dfb0 0x7f4b781d3ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4b7809d5f0 0x7f4b7809da70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 --2- 172.21.15.164:0/3693440217 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4b64004ef0 0x7f4b7813d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 >> 172.21.15.164:0/3693440217 conn(0x7f4b780aa900 msgr2=0x7f4b78132770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 shutdown_connections 2024-09-10T16:27:11.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:11.470+0000 7f4b7d839640 1 -- 172.21.15.164:0/3693440217 wait complete. 2024-09-10T16:27:14.484 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:14.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.593+0000 7f50d9b9d640 10 monclient: get_monmap_and_config 2024-09-10T16:27:14.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.593+0000 7f50d9b9d640 10 monclient: build_initial_monmap 2024-09-10T16:27:14.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.593+0000 7f50d9b9d640 10 monclient: monmap: 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:14.594136+0000 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:14.594136+0000 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:14.595 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:14.596 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:14.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 Processor -- start 2024-09-10T16:27:14.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 -- start start 2024-09-10T16:27:14.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient: init 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f50d41354c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f50d40a5ff0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f50d413d530 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d406e180 con 0x7f50d413d530 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50d4135090 con 0x7f50d41354c0 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f50d40a63d0 con 0x7f50d40a5ff0 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d3fff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 msgr2=0x7f50d413cff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d9b9d640 10 monclient(hunting): authenticate will time out at 2013.081637s 2024-09-10T16:27:14.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:14.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d939c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f50d413d530 0x7f50d413f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53800/0 (socket says 172.21.15.164:53800) 2024-09-10T16:27:14.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.594+0000 7f50d939c640 1 -- 172.21.15.164:0/3665242172 learned_addr learned my addr 172.21.15.164:0/3665242172 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:14.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.596+0000 7f50d939c640 1 -- 172.21.15.164:0/3665242172 >> v1:172.21.15.164:6789/0 conn(0x7f50d413d530 legacy=0x7f50d413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:14.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.597+0000 7f50d8b9b640 1 -- 172.21.15.164:0/3665242172 >> v1:172.21.15.165:6789/0 conn(0x7f50d41354c0 legacy=0x7f50d409d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:14.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.795+0000 7f50d3fff640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 msgr2=0x7f50d413cff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:14.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:14.795+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:27:15.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:15.196+0000 7f50d3fff640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 msgr2=0x7f50d413cff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:15.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:15.196+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:27:15.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:15.997+0000 7f50d3fff640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 msgr2=0x7f50d413cff0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:15.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:15.997+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:27:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): tick 2024-09-10T16:27:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 monclient(hunting): continuing hunt 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 msgr2=0x7f50d413cff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50d40a5ff0 0x7f50d413cff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 -- 172.21.15.164:0/3665242172 >> v1:172.21.15.165:6789/0 conn(0x7f50d41354c0 legacy=0x7f50d409d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 -- 172.21.15.164:0/3665242172 >> v1:172.21.15.164:6789/0 conn(0x7f50d413d530 legacy=0x7f50d413f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 --2- 172.21.15.164:0/3665242172 >> v2:172.21.15.164:3300/0 conn(0x7f50b8003810 0x7f50b8005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f50b8003810 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 --2- 172.21.15.164:0/3665242172 >> v2:172.21.15.165:3300/0 conn(0x7f50b8006380 0x7f50b8008800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f50b8006380 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 -- 172.21.15.164:0/3665242172 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f50d40a63d0 con 0x7f50b8003810 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 1 -- 172.21.15.164:0/3665242172 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f50b8008ec0 con 0x7f50b8006380 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.594+0000 7f50d27fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 1 --2- 172.21.15.164:0/3665242172 >> v2:172.21.15.164:3300/0 conn(0x7f50b8003810 0x7f50b8005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): get_auth_request con 0x7f50b8003810 auth_method 0 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3665242172 >> v2:172.21.15.165:3300/0 conn(0x7f50b8006380 0x7f50b8008800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): get_auth_request con 0x7f50b8006380 auth_method 0 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient(hunting): handle_auth_done global_id 224241 payload 274 2024-09-10T16:27:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 1 -- 172.21.15.164:0/3665242172 >> v2:172.21.15.165:3300/0 conn(0x7f50b8006380 msgr2=0x7f50b8008800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 1 --2- 172.21.15.164:0/3665242172 >> v2:172.21.15.165:3300/0 conn(0x7f50b8006380 0x7f50b8008800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient: _finish_hunting 0 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 1 monclient: found mon.noname-a 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.595+0000 7f50d8b9b640 1 -- 172.21.15.164:0/3665242172 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d406e460 con 0x7f50b8003810 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d8b9b640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8003810 msgr2=0x7f50b8005c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d8b9b640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8003810 0x7f50b8005c90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b8006350 tx=0x7f50c802a2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c22609d6160ab17f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 1 -- 172.21.15.164:0/3665242172 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8007070 con 0x7f50b8003810 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: dump: 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d9b9d640 5 monclient: authenticate success, global_id 224241 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 1 -- 172.21.15.164:0/3665242172 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c802e070 con 0x7f50b8003810 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d9b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:17.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d9b9d640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d9b9d640 4 set_mon_vals no callback set 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 1 -- 172.21.15.164:0/3665242172 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c8007450 con 0x7f50b8003810 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.598+0000 7f50d37fe640 10 monclient: dump: 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:17.607 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 10 monclient: shutdown 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8003810 msgr2=0x7f50b8005c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8003810 0x7f50b8005c90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f50b8006350 tx=0x7f50c802a2f0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d939c640 1 -- 172.21.15.164:0/3665242172 reap_dead start 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3665242172 shutdown_connections 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3665242172 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50b8003810 0x7f50b8005c90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3665242172 >> 172.21.15.164:0/3665242172 conn(0x7f50d40a70e0 msgr2=0x7f50d40a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:17.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3665242172 shutdown_connections 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.599+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3665242172 wait complete. 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.600+0000 7f50d9b9d640 10 monclient: build_initial_monmap 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.600+0000 7f50d9b9d640 10 monclient: monmap: 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.600+0000 7f50d9b9d640 1 Processor -- start 2024-09-10T16:27:17.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.600+0000 7f50d9b9d640 1 -- start start 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.600+0000 7f50d9b9d640 10 monclient: init 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 0x7f50d40a4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f50b8003810 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f50d409a500 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d41cd270 0x7f50d41cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f50d41cd270 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f50d4134d80 con 0x7f50d409a500 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f50d409dec0 con 0x7f50d41cd270 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:17.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f50d409e040 con 0x7f50b8003810 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d9b9d640 10 monclient(hunting): authenticate will time out at 2016.088690s 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d8b9b640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 msgr2=0x7f50d40a4480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d8b9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 0x7f50d40a4480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49754/0 (socket says 172.21.15.164:49754) 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 1 -- 172.21.15.164:0/3637081630 learned_addr learned my addr 172.21.15.164:0/3637081630 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): get_auth_request con 0x7f50d409a500 auth_method 0 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:17.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.601+0000 7f50d939c640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d41cd270 0x7f50d41cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): get_auth_request con 0x7f50d41cd270 auth_method 0 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d939c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 10 monclient(hunting): handle_auth_done global_id 224244 payload 824 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 -- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 msgr2=0x7f50d40a4480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:17.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 0x7f50d40a4480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 -- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d41cd270 msgr2=0x7f50d41cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d41cd270 0x7f50d41cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:17.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.602+0000 7f50d3fff640 1 -- 172.21.15.164:0/3637081630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50d4140940 con 0x7f50d409a500 2024-09-10T16:27:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d3fff640 1 -- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 msgr2=0x7f50d409a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50c000bdd0 tx=0x7f50c0007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d3fff640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50c000bdd0 tx=0x7f50c0007660 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f50c00054e0 config(0 keys) 2024-09-10T16:27:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c001a070 con 0x7f50d409a500 2024-09-10T16:27:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: dump: 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: _finish_auth 0 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: _check_auth_tickets 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50c00054e0 con 0x7f50d409a500 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 5 monclient: authenticate success, global_id 224244 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 10 monclient: _renew_subs 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50d40a3470 con 0x7f50d409a500 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d2ffd640 4 set_mon_vals no callback set 2024-09-10T16:27:17.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 10 monclient: _renew_subs 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.604+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50d409aea0 con 0x7f50d409a500 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50c0009e60 con 0x7f50d409a500 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d17fa640 10 monclient: dump: 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:17.618 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f50c00058d0 con 0x7f50d409a500 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d9b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d9b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d9b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.605+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50d40a3860 con 0x7f50d409a500 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.606+0000 7f50d17fa640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 0x7f50b8041720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.606+0000 7f50d8b9b640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 0x7f50b8041720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.606+0000 7f50d8b9b640 10 monclient: get_auth_request con 0x7f50b803f260 auth_method 0 2024-09-10T16:27:17.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.608+0000 7f50d8b9b640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 0x7f50b8041720 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f50b80037a0 tx=0x7f50c8007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:17.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:17.611+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(711..711 src has 494..711) ==== 4831+0+0 (secure 0 0 0) 0x7f50c00534b0 con 0x7f50d409a500 2024-09-10T16:27:18.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.432+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f50c0005e20 con 0x7f50d409a500 2024-09-10T16:27:18.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.432+0000 7f50d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:18.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.432+0000 7f50d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:18.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.684+0000 7f50d9b9d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:18.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.684+0000 7f50d9b9d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:18.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.684+0000 7f50d9b9d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:18.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:18.684+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f50d40a3640 con 0x7f50d409a500 2024-09-10T16:27:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:20.601+0000 7f50bffff640 10 monclient: tick 2024-09-10T16:27:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:20.601+0000 7f50bffff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:20.601+0000 7f50bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:20.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:20.601+0000 7f50bffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:21.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:21.602+0000 7f50bffff640 10 monclient: tick 2024-09-10T16:27:21.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:21.602+0000 7f50bffff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:21.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:21.602+0000 7f50bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.602+0000 7f50bffff640 10 monclient: tick 2024-09-10T16:27:22.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.602+0000 7f50bffff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.602+0000 7f50bffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.656+0000 7f50d17fa640 1 -- 172.21.15.164:0/3637081630 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f50c0051e80 con 0x7f50d409a500 2024-09-10T16:27:22.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.656+0000 7f50d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:22.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.656+0000 7f50d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:22.658 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":206,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:22.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 msgr2=0x7f50b8041720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 0x7f50b8041720 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f50b80037a0 tx=0x7f50c8007000 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 10 monclient: shutdown 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 msgr2=0x7f50d409a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f50c000bdd0 tx=0x7f50c0007660 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 shutdown_connections 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f50b803f260 0x7f50b8041720 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f50d41cd270 0x7f50d41cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f50d409a500 0x7f50d409a960 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 --2- 172.21.15.164:0/3637081630 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f50b8003810 0x7f50d40a4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 >> 172.21.15.164:0/3637081630 conn(0x7f50d40a70e0 msgr2=0x7f50d413f920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:22.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 shutdown_connections 2024-09-10T16:27:22.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.660+0000 7f50d9b9d640 1 -- 172.21.15.164:0/3637081630 wait complete. 2024-09-10T16:27:22.678 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:27:22.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:27:22.678 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.783+0000 7fca924d4640 10 monclient: get_monmap_and_config 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.783+0000 7fca924d4640 10 monclient: build_initial_monmap 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.784+0000 7fca924d4640 10 monclient: monmap: 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:22.784883+0000 2024-09-10T16:27:22.785 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:22.784883+0000 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:22.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.784+0000 7fca924d4640 1 Processor -- start 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.784+0000 7fca924d4640 1 -- start start 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.784+0000 7fca924d4640 10 monclient: init 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fca8c09d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fca8c09ab50 0x7fca8c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-c con 0x7fca8c09ab50 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fca8c09e420 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8c0a0870 con 0x7fca8c09d930 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca8c074c70 con 0x7fca8c09e420 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fca8c09d3e0 con 0x7fca8c09ab50 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca924d4640 10 monclient(hunting): authenticate will time out at 2021.272279s 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fca8c09d930 0x7fca8c09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53814/0 (socket says 172.21.15.164:53814) 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8bfff640 1 -- 172.21.15.164:0/4294590976 learned_addr learned my addr 172.21.15.164:0/4294590976 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 1 --2- 172.21.15.164:0/4294590976 >> v2:172.21.15.165:3300/0 conn(0x7fca8c09ab50 0x7fca8c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): get_auth_request con 0x7fca8c09ab50 auth_method 0 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.785+0000 7fca8b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8bfff640 1 -- 172.21.15.164:0/4294590976 >> v1:172.21.15.164:6789/0 conn(0x7fca8c09d930 legacy=0x7fca8c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 1 -- 172.21.15.164:0/4294590976 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2227297178 0 0) 0x7fca8c0a0870 con 0x7fca8c09d930 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 10 monclient(hunting): _negotiate 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 10 monclient(hunting): my global_id is 224247 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8affd640 1 -- 172.21.15.164:0/4294590976 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca74003520 con 0x7fca8c09d930 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca90a71640 1 -- 172.21.15.164:0/4294590976 >> v1:172.21.15.165:6789/0 conn(0x7fca8c09e420 legacy=0x7fca8c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 10 monclient(hunting): handle_auth_done global_id 234136 payload 274 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 1 -- 172.21.15.164:0/4294590976 >> v1:172.21.15.165:6789/0 conn(0x7fca8c09e420 legacy=0x7fca8c13bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 1 -- 172.21.15.164:0/4294590976 >> v1:172.21.15.164:6789/0 conn(0x7fca8c09d930 legacy=0x7fca8c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 1 monclient: found mon.noname-c 2024-09-10T16:27:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:27:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.787+0000 7fca8b7fe640 1 -- 172.21.15.164:0/4294590976 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8c09ecb0 con 0x7fca8c09ab50 2024-09-10T16:27:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.790+0000 7fca8b7fe640 1 -- 172.21.15.164:0/4294590976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 msgr2=0x7fca8c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8b7fe640 1 --2- 172.21.15.164:0/4294590976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c1397c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca800098a0 tx=0x7fca8002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cfaf783ad3632aa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 1 -- 172.21.15.164:0/4294590976 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80037070 con 0x7fca8c09ab50 2024-09-10T16:27:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: dump: 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: _finish_auth 0 2024-09-10T16:27:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 1 -- 172.21.15.164:0/4294590976 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca80032640 con 0x7fca8c09ab50 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 5 monclient: authenticate success, global_id 234136 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 1 -- 172.21.15.164:0/4294590976 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca80032a10 con 0x7fca8c09ab50 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 4 set_mon_vals no callback set 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca8affd640 10 monclient: dump: 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 10 monclient: shutdown 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 -- 172.21.15.164:0/4294590976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 msgr2=0x7fca8c1397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 --2- 172.21.15.164:0/4294590976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c1397c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fca800098a0 tx=0x7fca8002a210 comp rx=0 tx=0).stop 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 -- 172.21.15.164:0/4294590976 shutdown_connections 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 --2- 172.21.15.164:0/4294590976 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c1397c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 -- 172.21.15.164:0/4294590976 >> 172.21.15.164:0/4294590976 conn(0x7fca8c0a3110 msgr2=0x7fca8c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 -- 172.21.15.164:0/4294590976 shutdown_connections 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.791+0000 7fca924d4640 1 -- 172.21.15.164:0/4294590976 wait complete. 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.792+0000 7fca924d4640 10 monclient: build_initial_monmap 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.792+0000 7fca924d4640 10 monclient: monmap: 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:22.798 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.792+0000 7fca924d4640 1 Processor -- start 2024-09-10T16:27:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 -- start start 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient: init 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c073a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-b con 0x7fca8c09ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca8c09d930 0x7fca8c1cd500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-c con 0x7fca8c09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): picked mon.noname-a con 0x7fca8c09e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c0a2940 con 0x7fca8c09e420 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c09ef30 con 0x7fca8c09ab50 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fca8c0a2ac0 con 0x7fca8c09d930 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca924d4640 10 monclient(hunting): authenticate will time out at 2021.280493s 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8b7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca8c09d930 msgr2=0x7fca8c1cd500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca8c09d930 0x7fca8c1cd500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49756/0 (socket says 172.21.15.164:49756) 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 learned_addr learned my addr 172.21.15.164:0/2301070901 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): get_auth_request con 0x7fca8c09e420 auth_method 0 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c073a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): get_auth_request con 0x7fca8c09ab50 auth_method 0 2024-09-10T16:27:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient(hunting): handle_auth_done global_id 224250 payload 824 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca8c09d930 msgr2=0x7fca8c1cd500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fca8c09d930 0x7fca8c1cd500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 msgr2=0x7fca8c073a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fca8c09ab50 0x7fca8c073a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient: _finish_hunting 0 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 monclient: found mon.noname-a 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.793+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca8c1cdf80 con 0x7fca8c09e420 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 msgr2=0x7fca8c1cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca90a71640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fca8400c790 tx=0x7fca8400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca84017070 con 0x7fca8c09e420 2024-09-10T16:27:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: dump: 2024-09-10T16:27:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.811 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca8400a7e0 con 0x7fca8c09e420 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 5 monclient: authenticate success, global_id 224250 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 10 monclient: _renew_subs 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca8c1326b0 con 0x7fca8c09e420 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca8400ab90 con 0x7fca8c09e420 2024-09-10T16:27:22.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca88ff9640 10 monclient: dump: 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 10 monclient: _renew_subs 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca8c074340 con 0x7fca8c09e420 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.796+0000 7fca89ffb640 4 set_mon_vals no callback set 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca924d4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca924d4640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 0x7fca8c12fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c09dd90 con 0x7fca8c12f900 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 0x7fca8c12fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient: get_auth_request con 0x7fca8c12f900 auth_method 0 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fca8401a070 con 0x7fca8c09e420 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 10 monclient(hunting): handle_auth_done global_id 224253 payload 824 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8bfff640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fca8c06dac0 con 0x7fca8c12f900 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca88ff9640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fca7803e1c0 0x7fca78040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(711..711 src has 494..711) ==== 4831+0+0 (secure 0 0 0) 0x7fca840522c0 con 0x7fca8c09e420 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8b7fe640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fca7803e1c0 0x7fca78040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.797+0000 7fca8b7fe640 10 monclient: get_auth_request con 0x7fca7803e1c0 auth_method 0 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca8bfff640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 msgr2=0x7fca8c12fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca8bfff640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 0x7fca8c12fce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0014d0 tx=0x7fca7c007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c011870 con 0x7fca8c12f900 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.799+0000 7fca88ff9640 10 monclient: dump: 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fca7c018070 con 0x7fca8c12f900 2024-09-10T16:27:22.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 msgr2=0x7fca8c12fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 0x7fca8c12fce0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0014d0 tx=0x7fca7c007410 comp rx=0 tx=0).stop 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 msgr2=0x7fca8c12fce0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca88ff9640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c12f900 0x7fca8c12fce0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.800+0000 7fca8b7fe640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fca7803e1c0 0x7fca78040680 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca8c06d500 tx=0x7fca8002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca924d4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca924d4640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 0x7fca8c06f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca924d4640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fca8c132430 con 0x7fca8c130220 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 0x7fca8c06f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient: get_auth_request con 0x7fca8c130220 auth_method 0 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 10 monclient(hunting): handle_auth_done global_id 224256 payload 824 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.806+0000 7fca8bfff640 1 -- 172.21.15.164:0/2301070901 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fca8c077260 con 0x7fca8c130220 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca8bfff640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 msgr2=0x7fca8c06f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca8bfff640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 0x7fca8c06f7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0014d0 tx=0x7fca7c001500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca7c01de20 con 0x7fca8c130220 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca88ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca88ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.808+0000 7fca88ff9640 10 monclient: dump: 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.820 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fca7c013810 con 0x7fca8c130220 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 msgr2=0x7fca8c06f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 0x7fca8c06f7e0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fca7c0014d0 tx=0x7fca7c001500 comp rx=0 tx=0).stop 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 msgr2=0x7fca8c06f7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:22.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca88ff9640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c130220 0x7fca8c06f7e0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 206, 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 94639, 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.822 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.823 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.824 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:22.825 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.826 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.827 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.828 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.829 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:22.830 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.832 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:27:22.833 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.834 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.834 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:22.834 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fca7803e1c0 msgr2=0x7fca78040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fca7803e1c0 0x7fca78040680 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fca8c06d500 tx=0x7fca8002e000 comp rx=0 tx=0).stop 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 10 monclient: shutdown 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 msgr2=0x7fca8c1cda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fca8400c790 tx=0x7fca8400cc50 comp rx=0 tx=0).stop 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca90a71640 1 -- 172.21.15.164:0/2301070901 reap_dead start 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 shutdown_connections 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 --2- 172.21.15.164:0/2301070901 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fca8c09e420 0x7fca8c1cda40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 >> 172.21.15.164:0/2301070901 conn(0x7fca8c0a3110 msgr2=0x7fca8c1bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 shutdown_connections 2024-09-10T16:27:22.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.809+0000 7fca924d4640 1 -- 172.21.15.164:0/2301070901 wait complete. 2024-09-10T16:27:22.836 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:27:22.836 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:27:22.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:27:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.933+0000 7f6525b75640 10 monclient: get_monmap_and_config 2024-09-10T16:27:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.933+0000 7f6525b75640 10 monclient: build_initial_monmap 2024-09-10T16:27:22.934 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.933+0000 7f6525b75640 10 monclient: monmap: 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:22.934227+0000 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:22.934227+0000 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:22.935 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 Processor -- start 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 -- start start 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient: init 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f652009c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f652009d160 0x7f652009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-c con 0x7f652009d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f652013a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:22.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652009a430 con 0x7f652013a5c0 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65200a24e0 con 0x7f652009c180 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f65200a2bf0 con 0x7f652009d160 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6525b75640 10 monclient(hunting): authenticate will time out at 2021.421697s 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f651ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f652013a5c0 0x7f652013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53830/0 (socket says 172.21.15.164:53830) 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 learned_addr learned my addr 172.21.15.164:0/643435273 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:22.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6516ffd640 1 --2- 172.21.15.164:0/643435273 >> v2:172.21.15.165:3300/0 conn(0x7f652009d160 0x7f652009f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6516ffd640 10 monclient(hunting): get_auth_request con 0x7f652009d160 auth_method 0 2024-09-10T16:27:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6516ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6516ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.934+0000 7f6516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.936+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 >> v1:172.21.15.164:6789/0 conn(0x7f652013a5c0 legacy=0x7f652013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22745125 0 0) 0x7f652009a430 con 0x7f652013a5c0 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): _negotiate 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): my global_id is 224259 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651f7fe640 1 -- 172.21.15.164:0/643435273 >> v1:172.21.15.165:6789/0 conn(0x7f652009c180 legacy=0x7f652009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f650c003520 con 0x7f652013a5c0 2024-09-10T16:27:22.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 991146547 0 0) 0x7f650c003520 con 0x7f652013a5c0 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 >> v2:172.21.15.165:3300/0 conn(0x7f652009d160 msgr2=0x7f652009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 --2- 172.21.15.164:0/643435273 >> v2:172.21.15.165:3300/0 conn(0x7f652009d160 0x7f652009f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 >> v1:172.21.15.165:6789/0 conn(0x7f652009c180 legacy=0x7f652009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _finish_hunting 0 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:27:22.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652009c560 con 0x7f652013a5c0 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _finish_auth 0 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f6518001d20 con 0x7f652013a5c0 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6525b75640 5 monclient: authenticate success, global_id 224259 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6525b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 1 --2- 172.21.15.164:0/643435273 >> v2:172.21.15.165:3300/0 conn(0x7f652009d160 0x7f652009f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:27:22.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 >> v1:172.21.15.164:6789/0 conn(0x7f652013a5c0 legacy=0x7f652013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f650c003b40 0x7f650c003f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): picked mon.c con 0x7f650c003b40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c004800 0x7f650c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): picked mon.b con 0x7f650c004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 0x7f650c00ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): picked mon.a con 0x7f650c005410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f65200a2bf0 con 0x7f650c005410 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f650c00c320 con 0x7f650c004800 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f650c00c520 con 0x7f650c003b40 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651f7fe640 1 -- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f650c003b40 msgr2=0x7f650c003f60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:22.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651effd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651f7fe640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f650c003b40 0x7f650c003f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6525b75640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 0x7f650c00ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c004800 0x7f650c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): get_auth_request con 0x7f650c005410 auth_method 0 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): get_auth_request con 0x7f650c004800 auth_method 0 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient(hunting): handle_auth_done global_id 224259 payload 293 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f650c003b40 msgr2=0x7f650c003f60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f650c003b40 0x7f650c003f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c004800 msgr2=0x7f650c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c004800 0x7f650c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:22.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 monclient: found mon.a 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f650c003520 con 0x7f650c005410 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient: _finish_auth 0 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f651ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.937+0000 7f6516ffd640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c004800 0x7f650c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.938+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 reap_dead start 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.940+0000 7f651ffff640 1 -- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 msgr2=0x7f650c00ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.940+0000 7f651ffff640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 0x7f650c00ba40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6518001f40 tx=0x7f650c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c736822d7a62467c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6508009860 con 0x7f650c005410 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f651802f070 con 0x7f650c005410 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:22.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 1 -- 172.21.15.164:0/643435273 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f651800a7e0 con 0x7f650c005410 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 4 set_mon_vals no callback set 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f651effd640 10 monclient: dump: 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.963 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 10 monclient: shutdown 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 -- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 msgr2=0x7f650c00ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 0x7f650c00ba40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6518001f40 tx=0x7f650c005830 comp rx=0 tx=0).stop 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 -- 172.21.15.164:0/643435273 shutdown_connections 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 --2- 172.21.15.164:0/643435273 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f650c005410 0x7f650c00ba40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 -- 172.21.15.164:0/643435273 >> 172.21.15.164:0/643435273 conn(0x7f65200a3110 msgr2=0x7f65200a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:22.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 -- 172.21.15.164:0/643435273 shutdown_connections 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.941+0000 7f6525b75640 1 -- 172.21.15.164:0/643435273 wait complete. 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.942+0000 7f6525b75640 10 monclient: build_initial_monmap 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.942+0000 7f6525b75640 10 monclient: monmap: 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:22.965 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.942+0000 7f6525b75640 1 Processor -- start 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.942+0000 7f6525b75640 1 -- start start 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.942+0000 7f6525b75640 10 monclient: init 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c005410 0x7f65201395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-b con 0x7f650c005410 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 0x7f6520139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-a con 0x7f6520098b70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652013a050 0x7f65200949b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): picked mon.noname-c con 0x7f652013a050 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6520074f90 con 0x7f6520098b70 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6520074e10 con 0x7f650c005410 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f652009d930 con 0x7f652013a050 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6525b75640 10 monclient(hunting): authenticate will time out at 2021.430288s 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651ffff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652013a050 msgr2=0x7f65200949b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652013a050 0x7f65200949b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 0x7f6520139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 0x7f6520139b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:49798/0 (socket says 172.21.15.164:49798) 2024-09-10T16:27:22.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 -- 172.21.15.164:0/548904392 learned_addr learned my addr 172.21.15.164:0/548904392 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): get_auth_request con 0x7f6520098b70 auth_method 0 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c005410 0x7f65201395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): get_auth_request con 0x7f650c005410 auth_method 0 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient(hunting): handle_auth_done global_id 224262 payload 824 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652013a050 msgr2=0x7f65200949b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652013a050 0x7f65200949b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c005410 msgr2=0x7f65201395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f650c005410 0x7f65201395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 monclient: found mon.noname-a 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.943+0000 7f6516ffd640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6520098450 con 0x7f6520098b70 2024-09-10T16:27:22.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.945+0000 7f6516ffd640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 msgr2=0x7f6520139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.945+0000 7f6516ffd640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 0x7f6520139b10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6508009fd0 tx=0x7f6508011f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.945+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f650800fdc0 con 0x7f6520098b70 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: dump: 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.970 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 5 monclient: authenticate success, global_id 224262 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6508014440 con 0x7f6520098b70 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65080147f0 con 0x7f6520098b70 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 monclient: _renew_subs 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6520098170 con 0x7f6520098b70 2024-09-10T16:27:22.971 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651cff9640 10 monclient: dump: 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.972 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f651e7fc640 4 set_mon_vals no callback set 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 monclient: _renew_subs 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6520096c50 con 0x7f6520098b70 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 0x7f652002d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.946+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f652009d620 con 0x7f652002d440 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6508012040 con 0x7f6520098b70 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 0x7f652002d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient: get_auth_request con 0x7f652002d440 auth_method 0 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651cff9640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650003e270 0x7f6500040730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(711..711 src has 494..711) ==== 4831+0+0 (secure 0 0 0) 0x7f6508053430 con 0x7f6520098b70 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 10 monclient(hunting): handle_auth_done global_id 234154 payload 824 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651f7fe640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f652009f380 con 0x7f652002d440 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651ffff640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650003e270 0x7f6500040730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.947+0000 7f651ffff640 10 monclient: get_auth_request con 0x7f650003e270 auth_method 0 2024-09-10T16:27:22.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651f7fe640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 msgr2=0x7f652002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651f7fe640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 0x7f652002d820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65100014f0 tx=0x7f651000fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65100072e0 con 0x7f652002d440 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651cff9640 10 monclient: dump: 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.975 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.949+0000 7f651ffff640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650003e270 0x7f6500040730 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f652009ab50 tx=0x7f6518009030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6510018070 con 0x7f652002d440 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 msgr2=0x7f652002d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 0x7f652002d820 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f65100014f0 tx=0x7f651000fc00 comp rx=0 tx=0).stop 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 msgr2=0x7f652002d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:22.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.950+0000 7f651cff9640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002d440 0x7f652002d820 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f6525b75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:22.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f6525b75640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 0x7f652002e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:22.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f6525b75640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:22.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f65200a4230 con 0x7f652002dd60 2024-09-10T16:27:22.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 0x7f652002e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient: get_auth_request con 0x7f652002dd60 auth_method 0 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 10 monclient(hunting): handle_auth_done global_id 234157 payload 824 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.956+0000 7f651f7fe640 1 -- 172.21.15.164:0/548904392 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6520097f70 con 0x7f652002dd60 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.958+0000 7f651f7fe640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 msgr2=0x7f652002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.958+0000 7f651f7fe640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 0x7f652002e170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65100014f0 tx=0x7f6510001590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6510005be0 con 0x7f652002dd60 2024-09-10T16:27:22.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 10 monclient: dump: 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:22.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f6510005be0 con 0x7f652002dd60 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 msgr2=0x7f652002e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 0x7f652002e170 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f65100014f0 tx=0x7f6510001590 comp rx=0 tx=0).stop 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 msgr2=0x7f652002e170 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651cff9640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652002dd60 0x7f652002e170 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:22.981 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 206, 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 43243, 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.982 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.983 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:22.984 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:22.985 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:22.986 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.987 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.988 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.989 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:22.990 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:22.991 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.992 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.993 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:22.994 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:22.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650003e270 msgr2=0x7f6500040730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f650003e270 0x7f6500040730 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f652009ab50 tx=0x7f6518009030 comp rx=0 tx=0).stop 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 10 monclient: shutdown 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 msgr2=0x7f6520139b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 --2- 172.21.15.164:0/548904392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6520098b70 0x7f6520139b10 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6508009fd0 tx=0x7f6508011f00 comp rx=0 tx=0).stop 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651ffff640 1 -- 172.21.15.164:0/548904392 reap_dead start 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f651ffff640 1 -- 172.21.15.164:0/548904392 reap_dead start 2024-09-10T16:27:22.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 shutdown_connections 2024-09-10T16:27:22.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 >> 172.21.15.164:0/548904392 conn(0x7f65200a3110 msgr2=0x7f65200707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:22.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 shutdown_connections 2024-09-10T16:27:22.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:22.959+0000 7f6525b75640 1 -- 172.21.15.164:0/548904392 wait complete. 2024-09-10T16:27:22.997 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:27:22.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:27:23.671 INFO:tasks.rados.rados.0.smithi165.stdout:3201: done (0 left) 2024-09-10T16:27:23.672 INFO:tasks.rados.rados.0.smithi165.stdout:3202: read oid 1 snap -1 2024-09-10T16:27:23.672 INFO:tasks.rados.rados.0.smithi165.stdout:3202: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:27:23.672 INFO:tasks.rados.rados.0.smithi165.stdout:3203: snap_create 2024-09-10T16:27:23.681 INFO:tasks.rados.rados.0.smithi165.stdout:3202: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:27:23.797 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3202: done (1 left) 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3203: done (0 left) 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3204: read oid 46 snap -1 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3204: expect deleted 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3205: rollback oid 10 current snap is 344 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 331 2024-09-10T16:27:24.641 INFO:tasks.rados.rados.0.smithi165.stdout:3206: snap_create 2024-09-10T16:27:24.645 INFO:tasks.rados.rados.0.smithi165.stdout:3205: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:27:24.645 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1576 (ObjNum 65 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:27:25.642 INFO:tasks.rados.rados.0.smithi165.stdout:3204: done (2 left) 2024-09-10T16:27:25.642 INFO:tasks.rados.rados.0.smithi165.stdout:3205: done (1 left) 2024-09-10T16:27:25.642 INFO:tasks.rados.rados.0.smithi165.stdout:3206: done (0 left) 2024-09-10T16:27:25.642 INFO:tasks.rados.rados.0.smithi165.stdout:3207: snap_create 2024-09-10T16:27:26.644 INFO:tasks.rados.rados.0.smithi165.stdout:3207: done (0 left) 2024-09-10T16:27:26.644 INFO:tasks.rados.rados.0.smithi165.stdout:3208: snap_create 2024-09-10T16:27:27.645 INFO:tasks.rados.rados.0.smithi165.stdout:3208: done (0 left) 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3209: read oid 31 snap -1 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3209: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3210: read oid 30 snap 334 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3210: expect deleted 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3211: read oid 50 snap -1 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3211: expect deleted 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3212: read oid 17 snap -1 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3212: expect deleted 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3213: copy_from oid 49 from oid 10 current snap is 347 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3214: read oid 8 snap -1 2024-09-10T16:27:27.646 INFO:tasks.rados.rados.0.smithi165.stdout:3214: expect (ObjNum 1074 snap 330 seq_num 1074) 2024-09-10T16:27:27.647 INFO:tasks.rados.rados.0.smithi165.stdout:3215: copy_from oid 7 from oid 19 current snap is 347 2024-09-10T16:27:27.647 INFO:tasks.rados.rados.0.smithi165.stdout:3216: read oid 13 snap -1 2024-09-10T16:27:27.647 INFO:tasks.rados.rados.0.smithi165.stdout:3216: expect (ObjNum 1083 snap 331 seq_num 1083) 2024-09-10T16:27:27.647 INFO:tasks.rados.rados.0.smithi165.stdout:3217: snap_remove snap 343 2024-09-10T16:27:28.644 INFO:tasks.rados.rados.0.smithi165.stdout:3213: finishing copy_from to smithi16534841-49 2024-09-10T16:27:28.645 INFO:tasks.rados.rados.0.smithi165.stdout:3213: got expected ENOENT (src dne) 2024-09-10T16:27:28.645 INFO:tasks.rados.rados.0.smithi165.stdout:3213: finishing copy_from racing read to smithi16534841-49 2024-09-10T16:27:28.645 INFO:tasks.rados.rados.0.smithi165.stdout:3214: expect (ObjNum 1074 snap 330 seq_num 1074) 2024-09-10T16:27:28.645 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:28.727 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1113880 2024-09-10T16:27:28.746 INFO:tasks.rados.rados.0.smithi165.stdout:3209: expect (ObjNum 881 snap 269 seq_num 881) 2024-09-10T16:27:28.746 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 105411 2024-09-10T16:27:28.833 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 717271 2024-09-10T16:27:28.866 INFO:tasks.rados.rados.0.smithi165.stdout:3215: finishing copy_from racing read to smithi16534841-7 2024-09-10T16:27:28.866 INFO:tasks.rados.rados.0.smithi165.stdout:3216: expect (ObjNum 1083 snap 331 seq_num 1083) 2024-09-10T16:27:28.866 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3215: finishing copy_from to smithi16534841-7 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 1910 (ObjNum 999 snap 299 seq_num 999) dirty exists 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3209: done (8 left) 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3210: done (7 left) 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3211: done (6 left) 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3212: done (5 left) 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3213: done (4 left) 2024-09-10T16:27:29.007 INFO:tasks.rados.rados.0.smithi165.stdout:3214: done (3 left) 2024-09-10T16:27:29.008 INFO:tasks.rados.rados.0.smithi165.stdout:3215: done (2 left) 2024-09-10T16:27:29.008 INFO:tasks.rados.rados.0.smithi165.stdout:3216: done (1 left) 2024-09-10T16:27:29.008 INFO:tasks.rados.rados.0.smithi165.stdout:3217: done (0 left) 2024-09-10T16:27:29.008 INFO:tasks.rados.rados.0.smithi165.stdout:3218: delete oid 27 current snap is 347 2024-09-10T16:27:29.010 INFO:tasks.rados.rados.0.smithi165.stdout:3218: done (0 left) 2024-09-10T16:27:29.011 INFO:tasks.rados.rados.0.smithi165.stdout:3219: write oid 1 current snap is 347 2024-09-10T16:27:29.011 INFO:tasks.rados.rados.0.smithi165.stdout:3219: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2024-09-10T16:27:29.016 INFO:tasks.rados.rados.0.smithi165.stdout:3219: writing smithi16534841-1 from 423840 to 1006611 tid 1 2024-09-10T16:27:29.022 INFO:tasks.rados.rados.0.smithi165.stdout:3219: writing smithi16534841-1 from 1523009 to 2109058 tid 2 2024-09-10T16:27:29.026 INFO:tasks.rados.rados.0.smithi165.stdout:3219: writing smithi16534841-1 from 2657046 to 2823840 tid 3 2024-09-10T16:27:29.026 INFO:tasks.rados.rados.0.smithi165.stdout:3220: write (excl) oid 47 current snap is 347 2024-09-10T16:27:29.026 INFO:tasks.rados.rados.0.smithi165.stdout:3219: finishing write tid 1 to smithi16534841-1 2024-09-10T16:27:29.026 INFO:tasks.rados.rados.0.smithi165.stdout:3219: finishing write tid 2 to smithi16534841-1 2024-09-10T16:27:29.026 INFO:tasks.rados.rados.0.smithi165.stdout:3220: seq_num 1121 ranges {440647=658020,1581491=446060,2800822=39825} 2024-09-10T16:27:29.032 INFO:tasks.rados.rados.0.smithi165.stdout:3220: writing smithi16534841-47 from 440647 to 1098667 tid 1 2024-09-10T16:27:29.037 INFO:tasks.rados.rados.0.smithi165.stdout:3220: writing smithi16534841-47 from 1581491 to 2027551 tid 2 2024-09-10T16:27:29.042 INFO:tasks.rados.rados.0.smithi165.stdout:3220: writing smithi16534841-47 from 2800822 to 2840647 tid 3 2024-09-10T16:27:29.042 INFO:tasks.rados.rados.0.smithi165.stdout:3221: read oid 17 snap 333 2024-09-10T16:27:29.042 INFO:tasks.rados.rados.0.smithi165.stdout:3221: expect deleted 2024-09-10T16:27:29.042 INFO:tasks.rados.rados.0.smithi165.stdout:3222: copy_from oid 15 from oid 10 current snap is 347 2024-09-10T16:27:29.042 INFO:tasks.rados.rados.0.smithi165.stdout:3219: finishing write tid 3 to smithi16534841-1 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: finishing write tid 4 to smithi16534841-1 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: finishing write tid 5 to smithi16534841-1 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: oid 1 updating version 0 to 1701 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: oid 1 version 1701 is already newer than 1700 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: oid 1 updating version 1701 to 1703 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: oid 1 version 1703 is already newer than 1702 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1703 (ObjNum 1120 snap 347 seq_num 1120) dirty exists 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: left oid 1 (ObjNum 1120 snap 347 seq_num 1120) 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3220: finishing write tid 1 to smithi16534841-47 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3220: finishing write tid 2 to smithi16534841-47 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3219: done (3 left) 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3223: read oid 27 snap -1 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3223: expect deleted 2024-09-10T16:27:29.043 INFO:tasks.rados.rados.0.smithi165.stdout:3224: snap_remove snap 338 2024-09-10T16:27:29.668 INFO:tasks.rados.rados.0.smithi165.stdout:3224: done (4 left) 2024-09-10T16:27:29.668 INFO:tasks.rados.rados.0.smithi165.stdout:3225: read oid 26 snap 333 2024-09-10T16:27:29.668 INFO:tasks.rados.rados.0.smithi165.stdout:3225: expect deleted 2024-09-10T16:27:29.668 INFO:tasks.rados.rados.0.smithi165.stdout:3226: write (excl) oid 18 current snap is 347 2024-09-10T16:27:29.668 INFO:tasks.rados.rados.0.smithi165.stdout:3226: seq_num 1122 ranges {457454=733269,1923620=589718} 2024-09-10T16:27:29.674 INFO:tasks.rados.rados.0.smithi165.stdout:3226: writing smithi16534841-18 from 457454 to 1190723 tid 1 2024-09-10T16:27:29.682 INFO:tasks.rados.rados.0.smithi165.stdout:3226: writing smithi16534841-18 from 1923620 to 2513338 tid 2 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3227: read oid 5 snap 311 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3227: expect (ObjNum 967 snap 292 seq_num 967) 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3228: delete oid 34 current snap is 347 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3222: finishing copy_from to smithi16534841-15 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3222: got expected ENOENT (src dne) 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3222: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: finishing write tid 3 to smithi16534841-47 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: finishing write tid 4 to smithi16534841-47 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: finishing write tid 5 to smithi16534841-47 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: oid 47 updating version 0 to 1736 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: oid 47 updating version 1736 to 1737 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: oid 47 updating version 1737 to 1738 2024-09-10T16:27:29.683 INFO:tasks.rados.rados.0.smithi165.stdout:3220: oid 47 version 1738 is already newer than 1735 2024-09-10T16:27:29.684 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1738 (ObjNum 1121 snap 347 seq_num 1121) dirty exists 2024-09-10T16:27:29.684 INFO:tasks.rados.rados.0.smithi165.stdout:3220: left oid 47 (ObjNum 1121 snap 347 seq_num 1121) 2024-09-10T16:27:29.684 INFO:tasks.rados.rados.0.smithi165.stdout:3226: finishing write tid 1 to smithi16534841-18 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3226: finishing write tid 2 to smithi16534841-18 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3226: finishing write tid 3 to smithi16534841-18 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3220: done (7 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3221: done (6 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3222: done (5 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3223: done (4 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3225: done (3 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3228: done (2 left) 2024-09-10T16:27:29.688 INFO:tasks.rados.rados.0.smithi165.stdout:3229: snap_remove snap 346 2024-09-10T16:27:30.671 INFO:tasks.rados.rados.0.smithi165.stdout:3229: done (2 left) 2024-09-10T16:27:30.671 INFO:tasks.rados.rados.0.smithi165.stdout:3226: finishing write tid 4 to smithi16534841-18 2024-09-10T16:27:30.671 INFO:tasks.rados.rados.0.smithi165.stdout:3226: oid 18 updating version 0 to 1705 2024-09-10T16:27:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:3226: oid 18 updating version 1705 to 1706 2024-09-10T16:27:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:3226: oid 18 version 1706 is already newer than 1704 2024-09-10T16:27:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1706 (ObjNum 1122 snap 347 seq_num 1122) dirty exists 2024-09-10T16:27:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:3226: left oid 18 (ObjNum 1122 snap 347 seq_num 1122) 2024-09-10T16:27:30.672 INFO:tasks.rados.rados.0.smithi165.stdout:3227: expect (ObjNum 967 snap 292 seq_num 967) 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3230: read oid 20 snap 340 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3230: expect deleted 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3226: done (2 left) 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3227: done (1 left) 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3231: read oid 23 snap -1 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3231: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:27:30.775 INFO:tasks.rados.rados.0.smithi165.stdout:3232: rollback oid 11 current snap is 347 2024-09-10T16:27:30.776 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 311 2024-09-10T16:27:30.776 INFO:tasks.rados.rados.0.smithi165.stdout:3233: delete oid 25 current snap is 347 2024-09-10T16:27:30.779 INFO:tasks.rados.rados.0.smithi165.stdout:3230: done (3 left) 2024-09-10T16:27:30.779 INFO:tasks.rados.rados.0.smithi165.stdout:3233: done (2 left) 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3234: read oid 8 snap -1 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3234: expect (ObjNum 1074 snap 330 seq_num 1074) 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3235: read oid 30 snap -1 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3235: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3236: read oid 28 snap -1 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3236: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3237: copy_from oid 44 from oid 3 current snap is 347 2024-09-10T16:27:30.780 INFO:tasks.rados.rados.0.smithi165.stdout:3238: delete oid 45 current snap is 347 2024-09-10T16:27:30.781 INFO:tasks.rados.rados.0.smithi165.stdout:3232: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:27:30.781 INFO:tasks.rados.rados.0.smithi165.stdout:3232: finishing rollback tid 2 to smithi16534841-11 2024-09-10T16:27:30.781 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1872 (ObjNum 982 snap 297 seq_num 982) dirty exists 2024-09-10T16:27:30.782 INFO:tasks.rados.rados.0.smithi165.stdout:3232: done (6 left) 2024-09-10T16:27:30.782 INFO:tasks.rados.rados.0.smithi165.stdout:3238: done (5 left) 2024-09-10T16:27:30.782 INFO:tasks.rados.rados.0.smithi165.stdout:3239: write (excl) oid 45 current snap is 347 2024-09-10T16:27:30.782 INFO:tasks.rados.rados.0.smithi165.stdout:3239: seq_num 1126 ranges {524682=750618,1757548=597056} 2024-09-10T16:27:30.790 INFO:tasks.rados.rados.0.smithi165.stdout:3239: writing smithi16534841-45 from 524682 to 1275300 tid 1 2024-09-10T16:27:30.796 INFO:tasks.rados.rados.0.smithi165.stdout:3239: writing smithi16534841-45 from 1757548 to 2354604 tid 2 2024-09-10T16:27:30.796 INFO:tasks.rados.rados.0.smithi165.stdout:3240: write oid 48 current snap is 347 2024-09-10T16:27:30.796 INFO:tasks.rados.rados.0.smithi165.stdout:3240: seq_num 1127 ranges {541489=425867,1416030=740714,2732654=208835} 2024-09-10T16:27:30.801 INFO:tasks.rados.rados.0.smithi165.stdout:3240: writing smithi16534841-48 from 541489 to 967356 tid 1 2024-09-10T16:27:30.808 INFO:tasks.rados.rados.0.smithi165.stdout:3240: writing smithi16534841-48 from 1416030 to 2156744 tid 2 2024-09-10T16:27:30.813 INFO:tasks.rados.rados.0.smithi165.stdout:3240: writing smithi16534841-48 from 2732654 to 2941489 tid 3 2024-09-10T16:27:30.813 INFO:tasks.rados.rados.0.smithi165.stdout:3241: snap_remove snap 345 2024-09-10T16:27:31.684 INFO:tasks.rados.rados.0.smithi165.stdout:3241: done (7 left) 2024-09-10T16:27:31.684 INFO:tasks.rados.rados.0.smithi165.stdout:3242: delete oid 43 current snap is 347 2024-09-10T16:27:31.684 INFO:tasks.rados.rados.0.smithi165.stdout:3231: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:27:31.684 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:31.846 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 61322 2024-09-10T16:27:31.926 INFO:tasks.rados.rados.0.smithi165.stdout:3234: expect (ObjNum 1074 snap 330 seq_num 1074) 2024-09-10T16:27:32.018 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1735608 2024-09-10T16:27:32.020 INFO:tasks.rados.rados.0.smithi165.stdout:3236: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:27:32.020 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 385804 2024-09-10T16:27:32.065 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 461087 2024-09-10T16:27:32.104 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:32.148 INFO:tasks.rados.rados.0.smithi165.stdout:3235: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:27:32.173 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1080969 2024-09-10T16:27:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:3239: finishing write tid 1 to smithi16534841-45 2024-09-10T16:27:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:3237: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:27:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:3237: finishing copy_from to smithi16534841-44 2024-09-10T16:27:32.215 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1827 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: finishing write tid 2 to smithi16534841-45 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: finishing write tid 3 to smithi16534841-45 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: finishing write tid 4 to smithi16534841-45 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: oid 45 updating version 0 to 1696 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: oid 45 version 1696 is already newer than 1695 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: oid 45 updating version 1696 to 1697 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1697 (ObjNum 1126 snap 347 seq_num 1126) dirty exists 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3239: left oid 45 (ObjNum 1126 snap 347 seq_num 1126) 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3240: finishing write tid 1 to smithi16534841-48 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3231: done (7 left) 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3234: done (6 left) 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3235: done (5 left) 2024-09-10T16:27:32.216 INFO:tasks.rados.rados.0.smithi165.stdout:3236: done (4 left) 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3237: done (3 left) 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3239: done (2 left) 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3242: done (1 left) 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3243: read oid 43 snap -1 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3243: expect deleted 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3244: read oid 33 snap 333 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3244: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3245: delete oid 49 current snap is 347 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3240: finishing write tid 2 to smithi16534841-48 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3240: finishing write tid 3 to smithi16534841-48 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3240: finishing write tid 4 to smithi16534841-48 2024-09-10T16:27:32.217 INFO:tasks.rados.rados.0.smithi165.stdout:3240: finishing write tid 5 to smithi16534841-48 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3240: oid 48 updating version 0 to 1994 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3240: oid 48 version 1994 is already newer than 1992 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3240: oid 48 version 1994 is already newer than 1993 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3240: oid 48 updating version 1994 to 1995 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1995 (ObjNum 1127 snap 347 seq_num 1127) dirty exists 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3240: left oid 48 (ObjNum 1127 snap 347 seq_num 1127) 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:3244: expect (ObjNum 1045 snap 322 seq_num 1045) 2024-09-10T16:27:32.218 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:32.337 INFO:tasks.rados.rados.0.smithi165.stdout:3240: done (3 left) 2024-09-10T16:27:32.337 INFO:tasks.rados.rados.0.smithi165.stdout:3243: done (2 left) 2024-09-10T16:27:32.337 INFO:tasks.rados.rados.0.smithi165.stdout:3244: done (1 left) 2024-09-10T16:27:32.337 INFO:tasks.rados.rados.0.smithi165.stdout:3245: done (0 left) 2024-09-10T16:27:32.337 INFO:tasks.rados.rados.0.smithi165.stdout:3246: snap_create 2024-09-10T16:27:32.682 INFO:tasks.rados.rados.0.smithi165.stdout:3246: done (0 left) 2024-09-10T16:27:32.682 INFO:tasks.rados.rados.0.smithi165.stdout:3247: copy_from oid 21 from oid 25 current snap is 348 2024-09-10T16:27:32.682 INFO:tasks.rados.rados.0.smithi165.stdout:3248: snap_remove snap 332 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3248: done (1 left) 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3249: copy_from oid 39 from oid 48 current snap is 348 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3247: finishing copy_from to smithi16534841-21 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3247: got expected ENOENT (src dne) 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3247: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3247: done (1 left) 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3250: write (excl) oid 3 current snap is 348 2024-09-10T16:27:33.685 INFO:tasks.rados.rados.0.smithi165.stdout:3250: seq_num 1130 ranges {591910=651614,1875123=604394} 2024-09-10T16:27:33.692 INFO:tasks.rados.rados.0.smithi165.stdout:3250: writing smithi16534841-3 from 591910 to 1243524 tid 1 2024-09-10T16:27:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:3250: writing smithi16534841-3 from 1875123 to 2479517 tid 2 2024-09-10T16:27:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:3251: read oid 38 snap 347 2024-09-10T16:27:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:3251: expect deleted 2024-09-10T16:27:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:3252: write (excl) oid 47 current snap is 348 2024-09-10T16:27:33.700 INFO:tasks.rados.rados.0.smithi165.stdout:3252: seq_num 1131 ranges {608717=726863,1933605=748052} 2024-09-10T16:27:33.709 INFO:tasks.rados.rados.0.smithi165.stdout:3252: writing smithi16534841-47 from 608717 to 1335580 tid 1 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3252: writing smithi16534841-47 from 1933605 to 2681657 tid 2 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3253: copy_from oid 40 from oid 29 current snap is 348 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3249: finishing copy_from racing read to smithi16534841-39 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: finishing write tid 1 to smithi16534841-3 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3249: finishing copy_from to smithi16534841-39 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 1995 (ObjNum 1127 snap 347 seq_num 1127) dirty exists 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: finishing write tid 2 to smithi16534841-3 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: finishing write tid 3 to smithi16534841-3 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: finishing write tid 4 to smithi16534841-3 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: oid 3 updating version 0 to 1830 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: oid 3 version 1830 is already newer than 1829 2024-09-10T16:27:33.718 INFO:tasks.rados.rados.0.smithi165.stdout:3250: oid 3 updating version 1830 to 1831 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1831 (ObjNum 1130 snap 348 seq_num 1130) dirty exists 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:3250: left oid 3 (ObjNum 1130 snap 348 seq_num 1130) 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:3249: done (4 left) 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:3250: done (3 left) 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:3251: done (2 left) 2024-09-10T16:27:33.719 INFO:tasks.rados.rados.0.smithi165.stdout:3254: delete oid 50 current snap is 348 2024-09-10T16:27:33.720 INFO:tasks.rados.rados.0.smithi165.stdout:3254: done (2 left) 2024-09-10T16:27:33.720 INFO:tasks.rados.rados.0.smithi165.stdout:3255: delete oid 7 current snap is 348 2024-09-10T16:27:33.720 INFO:tasks.rados.rados.0.smithi165.stdout:3252: finishing write tid 1 to smithi16534841-47 2024-09-10T16:27:33.723 INFO:tasks.rados.rados.0.smithi165.stdout:3255: done (2 left) 2024-09-10T16:27:33.723 INFO:tasks.rados.rados.0.smithi165.stdout:3256: write (excl) oid 6 current snap is 348 2024-09-10T16:27:33.723 INFO:tasks.rados.rados.0.smithi165.stdout:3256: seq_num 1134 ranges {659138=668963,2109051=611732} 2024-09-10T16:27:33.730 INFO:tasks.rados.rados.0.smithi165.stdout:3256: writing smithi16534841-6 from 659138 to 1328101 tid 1 2024-09-10T16:27:33.738 INFO:tasks.rados.rados.0.smithi165.stdout:3256: writing smithi16534841-6 from 2109051 to 2720783 tid 2 2024-09-10T16:27:33.738 INFO:tasks.rados.rados.0.smithi165.stdout:3257: read oid 20 snap 334 2024-09-10T16:27:33.738 INFO:tasks.rados.rados.0.smithi165.stdout:3257: expect deleted 2024-09-10T16:27:33.739 INFO:tasks.rados.rados.0.smithi165.stdout:3258: write oid 8 current snap is 348 2024-09-10T16:27:33.739 INFO:tasks.rados.rados.0.smithi165.stdout:3252: finishing write tid 2 to smithi16534841-47 2024-09-10T16:27:33.739 INFO:tasks.rados.rados.0.smithi165.stdout:3258: seq_num 1135 ranges {675945=744212,2167533=755390} 2024-09-10T16:27:33.746 INFO:tasks.rados.rados.0.smithi165.stdout:3258: writing smithi16534841-8 from 675945 to 1420157 tid 1 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3258: writing smithi16534841-8 from 2167533 to 2922923 tid 2 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3259: snap_create 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3253: finishing copy_from to smithi16534841-40 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3253: got expected ENOENT (src dne) 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3253: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3252: finishing write tid 3 to smithi16534841-47 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3252: finishing write tid 4 to smithi16534841-47 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3252: oid 47 updating version 0 to 1758 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3252: oid 47 updating version 1758 to 1764 2024-09-10T16:27:33.755 INFO:tasks.rados.rados.0.smithi165.stdout:3252: oid 47 version 1764 is already newer than 1763 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1764 (ObjNum 1131 snap 348 seq_num 1131) dirty exists 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3252: left oid 47 (ObjNum 1131 snap 348 seq_num 1131) 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: finishing write tid 1 to smithi16534841-6 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: finishing write tid 2 to smithi16534841-6 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: finishing write tid 3 to smithi16534841-6 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: finishing write tid 4 to smithi16534841-6 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: oid 6 updating version 0 to 1622 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: oid 6 version 1622 is already newer than 1620 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: oid 6 version 1622 is already newer than 1621 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1622 (ObjNum 1134 snap 348 seq_num 1134) dirty exists 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3256: left oid 6 (ObjNum 1134 snap 348 seq_num 1134) 2024-09-10T16:27:33.756 INFO:tasks.rados.rados.0.smithi165.stdout:3258: finishing write tid 1 to smithi16534841-8 2024-09-10T16:27:33.759 INFO:tasks.rados.rados.0.smithi165.stdout:3258: finishing write tid 2 to smithi16534841-8 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: finishing write tid 3 to smithi16534841-8 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: finishing write tid 4 to smithi16534841-8 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: oid 8 updating version 0 to 1768 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: oid 8 version 1768 is already newer than 1767 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: oid 8 version 1768 is already newer than 1766 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1768 (ObjNum 1135 snap 348 seq_num 1135) dirty exists 2024-09-10T16:27:33.760 INFO:tasks.rados.rados.0.smithi165.stdout:3258: left oid 8 (ObjNum 1135 snap 348 seq_num 1135) 2024-09-10T16:27:34.641 INFO:tasks.rados.rados.0.smithi165.stdout:3252: done (5 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3253: done (4 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3256: done (3 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3257: done (2 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3258: done (1 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3259: done (0 left) 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3260: copy_from oid 41 from oid 39 current snap is 349 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3261: write oid 49 current snap is 349 2024-09-10T16:27:34.642 INFO:tasks.rados.rados.0.smithi165.stdout:3261: seq_num 1136 ranges {692752=419461,1826015=615401,2979461=113291} 2024-09-10T16:27:34.649 INFO:tasks.rados.rados.0.smithi165.stdout:3261: writing smithi16534841-49 from 692752 to 1112213 tid 1 2024-09-10T16:27:34.655 INFO:tasks.rados.rados.0.smithi165.stdout:3261: writing smithi16534841-49 from 1826015 to 2441416 tid 2 2024-09-10T16:27:34.659 INFO:tasks.rados.rados.0.smithi165.stdout:3261: writing smithi16534841-49 from 2979461 to 3092752 tid 3 2024-09-10T16:27:34.660 INFO:tasks.rados.rados.0.smithi165.stdout:3262: snap_create 2024-09-10T16:27:34.660 INFO:tasks.rados.rados.0.smithi165.stdout:3261: finishing write tid 1 to smithi16534841-49 2024-09-10T16:27:34.660 INFO:tasks.rados.rados.0.smithi165.stdout:3260: finishing copy_from racing read to smithi16534841-41 2024-09-10T16:27:34.660 INFO:tasks.rados.rados.0.smithi165.stdout:3261: finishing write tid 2 to smithi16534841-49 2024-09-10T16:27:34.661 INFO:tasks.rados.rados.0.smithi165.stdout:3260: finishing copy_from to smithi16534841-41 2024-09-10T16:27:34.661 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1995 (ObjNum 1127 snap 347 seq_num 1127) dirty exists 2024-09-10T16:27:34.662 INFO:tasks.rados.rados.0.smithi165.stdout:3261: finishing write tid 3 to smithi16534841-49 2024-09-10T16:27:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:3261: finishing write tid 4 to smithi16534841-49 2024-09-10T16:27:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:3261: finishing write tid 5 to smithi16534841-49 2024-09-10T16:27:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:3261: oid 49 updating version 0 to 1581 2024-09-10T16:27:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:3261: oid 49 version 1581 is already newer than 1578 2024-09-10T16:27:34.663 INFO:tasks.rados.rados.0.smithi165.stdout:3261: oid 49 version 1581 is already newer than 1579 2024-09-10T16:27:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:3261: oid 49 version 1581 is already newer than 1580 2024-09-10T16:27:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1581 (ObjNum 1136 snap 349 seq_num 1136) dirty exists 2024-09-10T16:27:34.664 INFO:tasks.rados.rados.0.smithi165.stdout:3261: left oid 49 (ObjNum 1136 snap 349 seq_num 1136) 2024-09-10T16:27:35.644 INFO:tasks.rados.rados.0.smithi165.stdout:3260: done (2 left) 2024-09-10T16:27:35.644 INFO:tasks.rados.rados.0.smithi165.stdout:3261: done (1 left) 2024-09-10T16:27:35.644 INFO:tasks.rados.rados.0.smithi165.stdout:3262: done (0 left) 2024-09-10T16:27:35.644 INFO:tasks.rados.rados.0.smithi165.stdout:3263: copy_from oid 14 from oid 44 current snap is 350 2024-09-10T16:27:35.644 INFO:tasks.rados.rados.0.smithi165.stdout:3264: write oid 4 current snap is 350 2024-09-10T16:27:35.645 INFO:tasks.rados.rados.0.smithi165.stdout:3264: seq_num 1137 ranges {709559=494710,1884497=759059} 2024-09-10T16:27:35.650 INFO:tasks.rados.rados.0.smithi165.stdout:3264: writing smithi16534841-4 from 709559 to 1204269 tid 1 2024-09-10T16:27:35.659 INFO:tasks.rados.rados.0.smithi165.stdout:3264: writing smithi16534841-4 from 1884497 to 2643556 tid 2 2024-09-10T16:27:35.659 INFO:tasks.rados.rados.0.smithi165.stdout:3265: snap_create 2024-09-10T16:27:35.659 INFO:tasks.rados.rados.0.smithi165.stdout:3264: finishing write tid 1 to smithi16534841-4 2024-09-10T16:27:35.660 INFO:tasks.rados.rados.0.smithi165.stdout:3263: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:27:35.664 INFO:tasks.rados.rados.0.smithi165.stdout:3263: finishing copy_from to smithi16534841-14 2024-09-10T16:27:35.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1827 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: finishing write tid 2 to smithi16534841-4 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: finishing write tid 3 to smithi16534841-4 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: finishing write tid 4 to smithi16534841-4 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: oid 4 updating version 0 to 1445 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: oid 4 updating version 1445 to 1446 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:3264: oid 4 updating version 1446 to 1447 2024-09-10T16:27:35.665 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1447 (ObjNum 1137 snap 350 seq_num 1137) dirty exists 2024-09-10T16:27:35.666 INFO:tasks.rados.rados.0.smithi165.stdout:3264: left oid 4 (ObjNum 1137 snap 350 seq_num 1137) 2024-09-10T16:27:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:3263: done (2 left) 2024-09-10T16:27:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:3264: done (1 left) 2024-09-10T16:27:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:3265: done (0 left) 2024-09-10T16:27:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:3266: write (excl) oid 47 current snap is 351 2024-09-10T16:27:36.650 INFO:tasks.rados.rados.0.smithi165.stdout:3266: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2024-09-10T16:27:36.657 INFO:tasks.rados.rados.0.smithi165.stdout:3266: writing smithi16534841-47 from 726366 to 1296325 tid 1 2024-09-10T16:27:36.662 INFO:tasks.rados.rados.0.smithi165.stdout:3266: writing smithi16534841-47 from 1826626 to 2329343 tid 2 2024-09-10T16:27:36.667 INFO:tasks.rados.rados.0.smithi165.stdout:3266: writing smithi16534841-47 from 2801601 to 3126366 tid 3 2024-09-10T16:27:36.667 INFO:tasks.rados.rados.0.smithi165.stdout:3267: copy_from oid 40 from oid 50 current snap is 351 2024-09-10T16:27:36.667 INFO:tasks.rados.rados.0.smithi165.stdout:3266: finishing write tid 1 to smithi16534841-47 2024-09-10T16:27:36.668 INFO:tasks.rados.rados.0.smithi165.stdout:3266: finishing write tid 2 to smithi16534841-47 2024-09-10T16:27:36.668 INFO:tasks.rados.rados.0.smithi165.stdout:3268: snap_create 2024-09-10T16:27:36.673 INFO:tasks.rados.rados.0.smithi165.stdout:3266: finishing write tid 3 to smithi16534841-47 2024-09-10T16:27:36.673 INFO:tasks.rados.rados.0.smithi165.stdout:3266: finishing write tid 4 to smithi16534841-47 2024-09-10T16:27:36.673 INFO:tasks.rados.rados.0.smithi165.stdout:3266: finishing write tid 5 to smithi16534841-47 2024-09-10T16:27:36.673 INFO:tasks.rados.rados.0.smithi165.stdout:3266: oid 47 updating version 0 to 1773 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3266: oid 47 version 1773 is already newer than 1772 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3266: oid 47 version 1773 is already newer than 1770 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3266: oid 47 version 1773 is already newer than 1771 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1773 (ObjNum 1138 snap 351 seq_num 1138) dirty exists 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3266: left oid 47 (ObjNum 1138 snap 351 seq_num 1138) 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3267: finishing copy_from to smithi16534841-40 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3267: got expected ENOENT (src dne) 2024-09-10T16:27:36.674 INFO:tasks.rados.rados.0.smithi165.stdout:3267: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:27:37.653 INFO:tasks.rados.rados.0.smithi165.stdout:3266: done (2 left) 2024-09-10T16:27:37.654 INFO:tasks.rados.rados.0.smithi165.stdout:3267: done (1 left) 2024-09-10T16:27:37.654 INFO:tasks.rados.rados.0.smithi165.stdout:3268: done (0 left) 2024-09-10T16:27:37.654 INFO:tasks.rados.rados.0.smithi165.stdout:3269: snap_create 2024-09-10T16:27:38.656 INFO:tasks.rados.rados.0.smithi165.stdout:3269: done (0 left) 2024-09-10T16:27:38.656 INFO:tasks.rados.rados.0.smithi165.stdout:3270: copy_from oid 17 from oid 21 current snap is 353 2024-09-10T16:27:38.656 INFO:tasks.rados.rados.0.smithi165.stdout:3271: read oid 27 snap 349 2024-09-10T16:27:38.657 INFO:tasks.rados.rados.0.smithi165.stdout:3271: expect deleted 2024-09-10T16:27:38.657 INFO:tasks.rados.rados.0.smithi165.stdout:3272: write oid 41 current snap is 353 2024-09-10T16:27:38.657 INFO:tasks.rados.rados.0.smithi165.stdout:3272: seq_num 1139 ranges {743173=645208,1885108=762728} 2024-09-10T16:27:38.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: writing smithi16534841-41 from 743173 to 1388381 tid 1 2024-09-10T16:27:38.671 INFO:tasks.rados.rados.0.smithi165.stdout:3272: writing smithi16534841-41 from 1885108 to 2647836 tid 2 2024-09-10T16:27:38.671 INFO:tasks.rados.rados.0.smithi165.stdout:3273: rollback oid 44 current snap is 353 2024-09-10T16:27:38.671 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 351 2024-09-10T16:27:38.672 INFO:tasks.rados.rados.0.smithi165.stdout:3270: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:27:38.672 INFO:tasks.rados.rados.0.smithi165.stdout:3271: done (3 left) 2024-09-10T16:27:38.672 INFO:tasks.rados.rados.0.smithi165.stdout:3274: write oid 35 current snap is 353 2024-09-10T16:27:38.672 INFO:tasks.rados.rados.0.smithi165.stdout:3274: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2024-09-10T16:27:38.679 INFO:tasks.rados.rados.0.smithi165.stdout:3274: writing smithi16534841-35 from 759980 to 1480437 tid 1 2024-09-10T16:27:38.685 INFO:tasks.rados.rados.0.smithi165.stdout:3274: writing smithi16534841-35 from 1943590 to 2449976 tid 2 2024-09-10T16:27:38.689 INFO:tasks.rados.rados.0.smithi165.stdout:3274: writing smithi16534841-35 from 2856447 to 3159980 tid 3 2024-09-10T16:27:38.689 INFO:tasks.rados.rados.0.smithi165.stdout:3275: snap_remove snap 340 2024-09-10T16:27:39.642 INFO:tasks.rados.rados.0.smithi165.stdout:3275: done (4 left) 2024-09-10T16:27:39.643 INFO:tasks.rados.rados.0.smithi165.stdout:3276: write (excl) oid 48 current snap is 353 2024-09-10T16:27:39.643 INFO:tasks.rados.rados.0.smithi165.stdout:3270: finishing copy_from to smithi16534841-17 2024-09-10T16:27:39.643 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1824 (ObjNum 1097 snap 333 seq_num 1097) dirty exists 2024-09-10T16:27:39.643 INFO:tasks.rados.rados.0.smithi165.stdout:3276: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2024-09-10T16:27:39.650 INFO:tasks.rados.rados.0.smithi165.stdout:3276: writing smithi16534841-48 from 776787 to 1288846 tid 1 2024-09-10T16:27:39.658 INFO:tasks.rados.rados.0.smithi165.stdout:3276: writing smithi16534841-48 from 1718425 to 2484822 tid 2 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3276: writing smithi16534841-48 from 3000223 to 3176787 tid 3 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3270: done (4 left) 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3277: read oid 8 snap -1 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3277: expect (ObjNum 1135 snap 348 seq_num 1135) 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3278: rollback oid 32 current snap is 353 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 32 to 311 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3273: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:27:39.663 INFO:tasks.rados.rados.0.smithi165.stdout:3273: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: finishing write tid 1 to smithi16534841-41 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: finishing write tid 2 to smithi16534841-41 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3273: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1830 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: finishing write tid 3 to smithi16534841-41 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: finishing write tid 4 to smithi16534841-41 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: oid 41 updating version 0 to 1997 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: oid 41 version 1997 is already newer than 1996 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: oid 41 updating version 1997 to 1998 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 1998 (ObjNum 1139 snap 353 seq_num 1139) dirty exists 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3272: left oid 41 (ObjNum 1139 snap 353 seq_num 1139) 2024-09-10T16:27:39.664 INFO:tasks.rados.rados.0.smithi165.stdout:3274: finishing write tid 1 to smithi16534841-35 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: finishing write tid 2 to smithi16534841-35 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: finishing write tid 3 to smithi16534841-35 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: finishing write tid 4 to smithi16534841-35 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: finishing write tid 5 to smithi16534841-35 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: oid 35 updating version 0 to 1458 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: oid 35 updating version 1458 to 1461 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: oid 35 version 1461 is already newer than 1460 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: oid 35 version 1461 is already newer than 1459 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1461 (ObjNum 1140 snap 353 seq_num 1140) dirty exists 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: left oid 35 (ObjNum 1140 snap 353 seq_num 1140) 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3272: done (5 left) 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3273: done (4 left) 2024-09-10T16:27:39.665 INFO:tasks.rados.rados.0.smithi165.stdout:3274: done (3 left) 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:3279: read oid 13 snap 311 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:3279: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:3280: rollback oid 49 current snap is 353 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 49 to 348 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:3281: write oid 19 current snap is 353 2024-09-10T16:27:39.666 INFO:tasks.rados.rados.0.smithi165.stdout:3281: seq_num 1142 ranges {793594=587308,2060554=510055} 2024-09-10T16:27:39.670 INFO:tasks.rados.rados.0.smithi165.stdout:3281: writing smithi16534841-19 from 793594 to 1380902 tid 1 2024-09-10T16:27:39.678 INFO:tasks.rados.rados.0.smithi165.stdout:3281: writing smithi16534841-19 from 2060554 to 2570609 tid 2 2024-09-10T16:27:39.678 INFO:tasks.rados.rados.0.smithi165.stdout:3282: delete oid 33 current snap is 353 2024-09-10T16:27:39.678 INFO:tasks.rados.rados.0.smithi165.stdout:3278: finishing rollback tid 1 to smithi16534841-32 2024-09-10T16:27:39.678 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 0 (ObjNum 129 snap 0 seq_num 1162193296) dirty dne 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3280: finishing rollback tid 0 to smithi16534841-49 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: finishing write tid 1 to smithi16534841-48 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: finishing write tid 2 to smithi16534841-48 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: finishing write tid 3 to smithi16534841-48 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3280: finishing rollback tid 1 to smithi16534841-49 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1826 (ObjNum 257 snap 0 seq_num 4062158288) dirty dne 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: finishing write tid 4 to smithi16534841-48 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: finishing write tid 5 to smithi16534841-48 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: oid 48 updating version 0 to 2017 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: oid 48 updating version 2017 to 2018 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: oid 48 version 2018 is already newer than 2015 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:3276: oid 48 version 2018 is already newer than 2016 2024-09-10T16:27:39.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 2018 (ObjNum 1141 snap 353 seq_num 1141) dirty exists 2024-09-10T16:27:39.680 INFO:tasks.rados.rados.0.smithi165.stdout:3276: left oid 48 (ObjNum 1141 snap 353 seq_num 1141) 2024-09-10T16:27:39.680 INFO:tasks.rados.rados.0.smithi165.stdout:3277: expect (ObjNum 1135 snap 348 seq_num 1135) 2024-09-10T16:27:39.688 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 921660 2024-09-10T16:27:39.771 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:39.818 INFO:tasks.rados.rados.0.smithi165.stdout:3279: expect (ObjNum 939 snap 285 seq_num 939) 2024-09-10T16:27:39.818 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:39.967 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: finishing write tid 1 to smithi16534841-19 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: finishing write tid 2 to smithi16534841-19 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: finishing write tid 3 to smithi16534841-19 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: finishing write tid 4 to smithi16534841-19 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: oid 19 updating version 0 to 2022 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: oid 19 version 2022 is already newer than 2021 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: oid 19 version 2022 is already newer than 2020 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2022 (ObjNum 1142 snap 353 seq_num 1142) dirty exists 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3281: left oid 19 (ObjNum 1142 snap 353 seq_num 1142) 2024-09-10T16:27:40.049 INFO:tasks.rados.rados.0.smithi165.stdout:3276: done (6 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3277: done (5 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3278: done (4 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3279: done (3 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3280: done (2 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3281: done (1 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3282: done (0 left) 2024-09-10T16:27:40.050 INFO:tasks.rados.rados.0.smithi165.stdout:3283: delete oid 18 current snap is 353 2024-09-10T16:27:40.052 INFO:tasks.rados.rados.0.smithi165.stdout:3283: done (0 left) 2024-09-10T16:27:40.052 INFO:tasks.rados.rados.0.smithi165.stdout:3284: copy_from oid 47 from oid 18 current snap is 353 2024-09-10T16:27:40.052 INFO:tasks.rados.rados.0.smithi165.stdout:3285: snap_create 2024-09-10T16:27:40.053 INFO:tasks.rados.rados.0.smithi165.stdout:3284: finishing copy_from to smithi16534841-47 2024-09-10T16:27:40.053 INFO:tasks.rados.rados.0.smithi165.stdout:3284: got expected ENOENT (src dne) 2024-09-10T16:27:40.053 INFO:tasks.rados.rados.0.smithi165.stdout:3284: finishing copy_from racing read to smithi16534841-47 2024-09-10T16:27:40.673 INFO:tasks.rados.rados.0.smithi165.stdout:3284: done (1 left) 2024-09-10T16:27:40.673 INFO:tasks.rados.rados.0.smithi165.stdout:3285: done (0 left) 2024-09-10T16:27:40.673 INFO:tasks.rados.rados.0.smithi165.stdout:3286: rollback oid 14 current snap is 354 2024-09-10T16:27:40.673 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 348 2024-09-10T16:27:40.673 INFO:tasks.rados.rados.0.smithi165.stdout:3287: copy_from oid 34 from oid 37 current snap is 354 2024-09-10T16:27:40.674 INFO:tasks.rados.rados.0.smithi165.stdout:3288: delete oid 4 current snap is 354 2024-09-10T16:27:40.677 INFO:tasks.rados.rados.0.smithi165.stdout:3288: done (2 left) 2024-09-10T16:27:40.677 INFO:tasks.rados.rados.0.smithi165.stdout:3289: read oid 50 snap -1 2024-09-10T16:27:40.677 INFO:tasks.rados.rados.0.smithi165.stdout:3289: expect deleted 2024-09-10T16:27:40.677 INFO:tasks.rados.rados.0.smithi165.stdout:3290: rollback oid 26 current snap is 354 2024-09-10T16:27:40.678 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 26 to 347 2024-09-10T16:27:40.678 INFO:tasks.rados.rados.0.smithi165.stdout:3291: read oid 35 snap -1 2024-09-10T16:27:40.678 INFO:tasks.rados.rados.0.smithi165.stdout:3291: expect (ObjNum 1140 snap 353 seq_num 1140) 2024-09-10T16:27:40.678 INFO:tasks.rados.rados.0.smithi165.stdout:3292: snap_create 2024-09-10T16:27:40.679 INFO:tasks.rados.rados.0.smithi165.stdout:3286: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:27:40.679 INFO:tasks.rados.rados.0.smithi165.stdout:3286: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:27:40.680 INFO:tasks.rados.rados.0.smithi165.stdout:3286: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:27:40.680 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1830 (ObjNum 1116 snap 342 seq_num 1116) dirty exists 2024-09-10T16:27:40.683 INFO:tasks.rados.rados.0.smithi165.stdout:3290: finishing rollback tid 0 to smithi16534841-26 2024-09-10T16:27:40.683 INFO:tasks.rados.rados.0.smithi165.stdout:3290: finishing rollback tid 1 to smithi16534841-26 2024-09-10T16:27:40.683 INFO:tasks.rados.rados.0.smithi165.stdout:3290: finishing rollback tid 2 to smithi16534841-26 2024-09-10T16:27:40.684 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2026 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:27:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:3291: expect (ObjNum 1140 snap 353 seq_num 1140) 2024-09-10T16:27:40.689 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:40.767 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1974728 2024-09-10T16:27:40.843 INFO:tasks.rados.rados.0.smithi165.stdout:3287: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:27:40.843 INFO:tasks.rados.rados.0.smithi165.stdout:3287: finishing copy_from to smithi16534841-34 2024-09-10T16:27:40.843 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1583 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3286: done (5 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3287: done (4 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3289: done (3 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3290: done (2 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3291: done (1 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3292: done (0 left) 2024-09-10T16:27:41.746 INFO:tasks.rados.rados.0.smithi165.stdout:3293: rollback oid 38 current snap is 355 2024-09-10T16:27:41.747 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 38 to 351 2024-09-10T16:27:41.747 INFO:tasks.rados.rados.0.smithi165.stdout:3294: write oid 8 current snap is 355 2024-09-10T16:27:41.747 INFO:tasks.rados.rados.0.smithi165.stdout:3294: seq_num 1146 ranges {460822=488304,1378129=517393,2504632=405962} 2024-09-10T16:27:41.751 INFO:tasks.rados.rados.0.smithi165.stdout:3294: writing smithi16534841-8 from 460822 to 949126 tid 1 2024-09-10T16:27:41.756 INFO:tasks.rados.rados.0.smithi165.stdout:3294: writing smithi16534841-8 from 1378129 to 1895522 tid 2 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3294: writing smithi16534841-8 from 2504632 to 2910594 tid 3 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3295: copy_from oid 15 from oid 25 current snap is 355 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3293: finishing rollback tid 1 to smithi16534841-38 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1411 (ObjNum 1162324992 snap 21859 seq_num 304) dirty dne 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3294: finishing write tid 1 to smithi16534841-8 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3294: finishing write tid 2 to smithi16534841-8 2024-09-10T16:27:41.762 INFO:tasks.rados.rados.0.smithi165.stdout:3293: done (2 left) 2024-09-10T16:27:41.763 INFO:tasks.rados.rados.0.smithi165.stdout:3296: snap_create 2024-09-10T16:27:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:3295: finishing copy_from to smithi16534841-15 2024-09-10T16:27:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:3295: got expected ENOENT (src dne) 2024-09-10T16:27:41.765 INFO:tasks.rados.rados.0.smithi165.stdout:3295: finishing copy_from racing read to smithi16534841-15 2024-09-10T16:27:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:3294: finishing write tid 3 to smithi16534841-8 2024-09-10T16:27:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:3294: finishing write tid 4 to smithi16534841-8 2024-09-10T16:27:41.766 INFO:tasks.rados.rados.0.smithi165.stdout:3294: finishing write tid 5 to smithi16534841-8 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:3294: oid 8 updating version 0 to 1784 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:3294: oid 8 version 1784 is already newer than 1783 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:3294: oid 8 updating version 1784 to 1785 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:3294: oid 8 updating version 1785 to 1786 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1786 (ObjNum 1146 snap 355 seq_num 1146) dirty exists 2024-09-10T16:27:41.767 INFO:tasks.rados.rados.0.smithi165.stdout:3294: left oid 8 (ObjNum 1146 snap 355 seq_num 1146) 2024-09-10T16:27:42.755 INFO:tasks.rados.rados.0.smithi165.stdout:3294: done (2 left) 2024-09-10T16:27:42.755 INFO:tasks.rados.rados.0.smithi165.stdout:3295: done (1 left) 2024-09-10T16:27:42.756 INFO:tasks.rados.rados.0.smithi165.stdout:3296: done (0 left) 2024-09-10T16:27:42.756 INFO:tasks.rados.rados.0.smithi165.stdout:3297: snap_create 2024-09-10T16:27:42.997 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:27:42.998 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:27:42.998 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:27:43.001 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:27:43.001 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:27:43.001 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 10 monclient: get_monmap_and_config 2024-09-10T16:27:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 10 monclient: build_initial_monmap 2024-09-10T16:27:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 10 monclient: monmap: 2024-09-10T16:27:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:43.111 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:43.110998+0000 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:43.110998+0000 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:43.112 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 1 Processor -- start 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 1 -- start start 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.109+0000 7f32e4233640 10 monclient: init 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f32dc06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f32dc134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:43.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f32dc13e740 0x7f32dc140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-c con 0x7f32dc13e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32dc09ce50 con 0x7f32dc06cb70 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32dc135380 con 0x7f32dc134f70 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f32dc06db60 con 0x7f32dc13e740 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e4233640 10 monclient(hunting): authenticate will time out at 2041.598547s 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e1fcf640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f32dc06cb70 0x7f32dc0a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49668/0 (socket says 172.21.15.164:49668) 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 learned_addr learned my addr 172.21.15.164:0/1706770545 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 1 --2- 172.21.15.164:0/1706770545 >> v2:172.21.15.165:3300/0 conn(0x7f32dc13e740 0x7f32dc140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): get_auth_request con 0x7f32dc13e740 auth_method 0 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.110+0000 7f32e27d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 >> v1:172.21.15.164:6789/0 conn(0x7f32dc06cb70 legacy=0x7f32dc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4201868247 0 0) 0x7f32d0001450 con 0x7f32dc06cb70 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 10 monclient(hunting): _negotiate 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 10 monclient(hunting): my global_id is 224265 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.112+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f32c8003520 con 0x7f32dc06cb70 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 1 -- 172.21.15.164:0/1706770545 >> v1:172.21.15.165:6789/0 conn(0x7f32dc134f70 legacy=0x7f32dc13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:43.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2709318566 0 0) 0x7f32c8003520 con 0x7f32dc06cb70 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 >> v2:172.21.15.165:3300/0 conn(0x7f32dc13e740 msgr2=0x7f32dc140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 --2- 172.21.15.164:0/1706770545 >> v2:172.21.15.165:3300/0 conn(0x7f32dc13e740 0x7f32dc140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 >> v1:172.21.15.165:6789/0 conn(0x7f32dc134f70 legacy=0x7f32dc13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _finish_hunting 0 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc09cb70 con 0x7f32dc06cb70 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _finish_auth 0 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f32d0001b70 con 0x7f32dc06cb70 2024-09-10T16:27:43.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: dump: 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:27:43.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 >> v1:172.21.15.164:6789/0 conn(0x7f32dc06cb70 legacy=0x7f32dc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 0x7f32c8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): picked mon.a con 0x7f32c8003b10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8004800 0x7f32c8004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): picked mon.b con 0x7f32c8004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32c80053c0 0x7f32c800ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): picked mon.c con 0x7f32c80053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f32c800c2d0 con 0x7f32c8003b10 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32c800c480 con 0x7f32c8004800 2024-09-10T16:27:43.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f32c800c680 con 0x7f32c80053c0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e0fcd640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 0x7f32c8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): get_auth_request con 0x7f32c8003b10 auth_method 0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32c80053c0 0x7f32c800ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8004800 0x7f32c8004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): get_auth_request con 0x7f32c8004800 auth_method 0 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 10 monclient(hunting): get_auth_request con 0x7f32c80053c0 auth_method 0 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:43.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient(hunting): handle_auth_done global_id 224265 payload 293 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32c80053c0 msgr2=0x7f32c800ba20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32c80053c0 0x7f32c800ba20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8004800 msgr2=0x7f32c8004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8004800 0x7f32c8004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient: _finish_hunting 0 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 monclient: found mon.a 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32c8003520 con 0x7f32c8003b10 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient: _finish_auth 0 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 10 monclient: _check_auth_tickets 2024-09-10T16:27:43.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e1fcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e4233640 5 monclient: authenticate success, global_id 224265 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e4233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32c80053c0 0x7f32c800ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.113+0000 7f32e27d0640 1 -- 172.21.15.164:0/1706770545 reap_dead start 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e1fcf640 1 -- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 msgr2=0x7f32c8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 0x7f32c8003f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f32d00016a0 tx=0x7f32c8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55d9fcfedd7afd5f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d000d200 con 0x7f32c8003b10 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: dump: 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.129 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e4233640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d0033070 con 0x7f32c8003b10 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e4233640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e4233640 4 set_mon_vals no callback set 2024-09-10T16:27:43.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 1 -- 172.21.15.164:0/1706770545 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d0005db0 con 0x7f32c8003b10 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.116+0000 7f32e0fcd640 10 monclient: dump: 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 10 monclient: shutdown 2024-09-10T16:27:43.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 -- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 msgr2=0x7f32c8003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 0x7f32c8003f30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f32d00016a0 tx=0x7f32c8005830 comp rx=0 tx=0).stop 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 -- 172.21.15.164:0/1706770545 shutdown_connections 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 --2- 172.21.15.164:0/1706770545 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32c8003b10 0x7f32c8003f30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 -- 172.21.15.164:0/1706770545 >> 172.21.15.164:0/1706770545 conn(0x7f32dc0aa900 msgr2=0x7f32dc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 -- 172.21.15.164:0/1706770545 shutdown_connections 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 1 -- 172.21.15.164:0/1706770545 wait complete. 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 10 monclient: build_initial_monmap 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.117+0000 7f32e4233640 10 monclient: monmap: 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:43.132 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:43.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 Processor -- start 2024-09-10T16:27:43.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 -- start start 2024-09-10T16:27:43.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient: init 2024-09-10T16:27:43.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 0x7f32dc13d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-b con 0x7f32c8003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-a con 0x7f32dc09d630 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): picked mon.noname-c con 0x7f32dc09dff0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc0a46a0 con 0x7f32dc09d630 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc06cf50 con 0x7f32c8003b10 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f32dc0a4820 con 0x7f32dc09dff0 2024-09-10T16:27:43.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e4233640 10 monclient(hunting): authenticate will time out at 2041.606783s 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:44304/0 (socket says 172.21.15.164:44304) 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e27d0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 1 -- 172.21.15.164:0/2697365990 learned_addr learned my addr 172.21.15.164:0/2697365990 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): get_auth_request con 0x7f32dc09d630 auth_method 0 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e27d0640 10 monclient(hunting): get_auth_request con 0x7f32dc09dff0 auth_method 0 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e27d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e27d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:43.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e27d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.118+0000 7f32e17ce640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e27d0640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 msgr2=0x7f32dc1d3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e27d0640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 msgr2=0x7f32dc1d3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e27d0640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e27d0640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 0x7f32dc13d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 10 monclient(hunting): handle_auth_done global_id 224268 payload 824 2024-09-10T16:27:43.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 msgr2=0x7f32dc1d3900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 msgr2=0x7f32dc13d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 0x7f32dc13d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 10 monclient: _finish_hunting 0 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 monclient: found mon.noname-a 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e17ce640 1 -- 172.21.15.164:0/2697365990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc0a0b40 con 0x7f32dc09d630 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e1fcf640 10 monclient: get_auth_request con 0x7f32c8003b10 auth_method 0 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.119+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 0x7f32dc13d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e17ce640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 msgr2=0x7f32dc09dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e17ce640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006e60 tx=0x7f32d80019c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:43.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8017020 con 0x7f32dc09d630 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: dump: 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:43.142 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d800f870 con 0x7f32dc09d630 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 5 monclient: authenticate success, global_id 224268 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 10 monclient: _renew_subs 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32dc133ea0 con 0x7f32dc09d630 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d800fc20 con 0x7f32dc09d630 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32ce7fc640 10 monclient: dump: 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:43.143 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32cffff640 4 set_mon_vals no callback set 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 10 monclient: _renew_subs 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.121+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32dc0a0860 con 0x7f32dc09d630 2024-09-10T16:27:43.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f32d801a020 con 0x7f32dc09d630 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32e4233640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32e4233640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32e4233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32dc0a4bb0 con 0x7f32dc09d630 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32ce7fc640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 0x7f32c0040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.122+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(731..731 src has 677..731) ==== 4755+0+0 (secure 0 0 0) 0x7f32d8052c90 con 0x7f32dc09d630 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.123+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 0x7f32c0040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:43.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.123+0000 7f32e1fcf640 10 monclient: get_auth_request con 0x7f32c003e290 auth_method 0 2024-09-10T16:27:43.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.125+0000 7f32e1fcf640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 0x7f32c0040750 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f32d0001d90 tx=0x7f32d002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:43.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.127+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f32d8059080 con 0x7f32dc09d630 2024-09-10T16:27:43.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.127+0000 7f32ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:43.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.127+0000 7f32ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32e4233640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32e4233640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32e4233640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f32dc09e420 con 0x7f32dc09d630 2024-09-10T16:27:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32ce7fc640 1 -- 172.21.15.164:0/2697365990 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f32d8021020 con 0x7f32dc09d630 2024-09-10T16:27:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.367+0000 7f32ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:43.370 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":206,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 msgr2=0x7f32c0040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 0x7f32c0040750 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f32d0001d90 tx=0x7f32d002e000 comp rx=0 tx=0).stop 2024-09-10T16:27:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 10 monclient: shutdown 2024-09-10T16:27:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 msgr2=0x7f32dc09dab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006e60 tx=0x7f32d80019c0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 shutdown_connections 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f32c003e290 0x7f32c0040750 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f32dc09dff0 0x7f32dc1d3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f32dc09d630 0x7f32dc09dab0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 --2- 172.21.15.164:0/2697365990 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f32c8003b10 0x7f32dc13d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 >> 172.21.15.164:0/2697365990 conn(0x7f32dc0aa900 msgr2=0x7f32dc13e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 shutdown_connections 2024-09-10T16:27:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:43.369+0000 7f32e4233640 1 -- 172.21.15.164:0/2697365990 wait complete. 2024-09-10T16:27:43.757 INFO:tasks.rados.rados.0.smithi165.stdout:3297: done (0 left) 2024-09-10T16:27:43.758 INFO:tasks.rados.rados.0.smithi165.stdout:3298: write (excl) oid 21 current snap is 357 2024-09-10T16:27:43.758 INFO:tasks.rados.rados.0.smithi165.stdout:3298: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2024-09-10T16:27:43.763 INFO:tasks.rados.rados.0.smithi165.stdout:3298: writing smithi16534841-21 from 477629 to 1041182 tid 1 2024-09-10T16:27:43.772 INFO:tasks.rados.rados.0.smithi165.stdout:3298: writing smithi16534841-21 from 1836611 to 2614015 tid 2 2024-09-10T16:27:43.776 INFO:tasks.rados.rados.0.smithi165.stdout:3298: writing smithi16534841-21 from 3048408 to 3277629 tid 3 2024-09-10T16:27:43.776 INFO:tasks.rados.rados.0.smithi165.stdout:3299: snap_create 2024-09-10T16:27:43.776 INFO:tasks.rados.rados.0.smithi165.stdout:3298: finishing write tid 1 to smithi16534841-21 2024-09-10T16:27:43.779 INFO:tasks.rados.rados.0.smithi165.stdout:3298: finishing write tid 2 to smithi16534841-21 2024-09-10T16:27:43.779 INFO:tasks.rados.rados.0.smithi165.stdout:3298: finishing write tid 3 to smithi16534841-21 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: finishing write tid 4 to smithi16534841-21 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: finishing write tid 5 to smithi16534841-21 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: oid 21 updating version 0 to 1833 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: oid 21 updating version 1833 to 1836 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: oid 21 version 1836 is already newer than 1835 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: oid 21 version 1836 is already newer than 1834 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1836 (ObjNum 1147 snap 357 seq_num 1147) dirty exists 2024-09-10T16:27:43.781 INFO:tasks.rados.rados.0.smithi165.stdout:3298: left oid 21 (ObjNum 1147 snap 357 seq_num 1147) 2024-09-10T16:27:44.643 INFO:tasks.rados.rados.0.smithi165.stdout:3298: done (1 left) 2024-09-10T16:27:44.643 INFO:tasks.rados.rados.0.smithi165.stdout:3299: done (0 left) 2024-09-10T16:27:44.643 INFO:tasks.rados.rados.0.smithi165.stdout:3300: snap_create 2024-09-10T16:27:45.645 INFO:tasks.rados.rados.0.smithi165.stdout:3300: done (0 left) 2024-09-10T16:27:45.645 INFO:tasks.rados.rados.0.smithi165.stdout:3301: rollback oid 22 current snap is 359 2024-09-10T16:27:45.645 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 22 to 353 2024-09-10T16:27:45.646 INFO:tasks.rados.rados.0.smithi165.stdout:3302: snap_remove snap 333 2024-09-10T16:27:46.383 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.490+0000 7fc4789ad640 10 monclient: get_monmap_and_config 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.490+0000 7fc4789ad640 10 monclient: build_initial_monmap 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.490+0000 7fc4789ad640 10 monclient: monmap: 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:46.491612+0000 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:46.491612+0000 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.492 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:46.493 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:46.493 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:46.493 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:46.493 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:46.493 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.491+0000 7fc4789ad640 1 Processor -- start 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.491+0000 7fc4789ad640 1 -- start start 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.491+0000 7fc4789ad640 10 monclient: init 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc4741329f0 0x7fc474132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-c con 0x7fc4741329f0 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:46.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc4740a2f10 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc47409d520 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4740a0730 con 0x7fc4740a2f10 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4740a21a0 con 0x7fc47409d520 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc4740a2480 con 0x7fc4741329f0 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc4789ad640 10 monclient(hunting): authenticate will time out at 2044.979199s 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc46bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc4740a2f10 0x7fc47409ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59068/0 (socket says 172.21.15.164:59068) 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc46bfff640 1 -- 172.21.15.164:0/2418941883 learned_addr learned my addr 172.21.15.164:0/2418941883 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:46.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 1 --2- 172.21.15.164:0/2418941883 >> v2:172.21.15.165:3300/0 conn(0x7fc4741329f0 0x7fc474132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): get_auth_request con 0x7fc4741329f0 auth_method 0 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:46.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.492+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:46.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc46bfff640 1 -- 172.21.15.164:0/2418941883 >> v1:172.21.15.164:6789/0 conn(0x7fc4740a2f10 legacy=0x7fc47409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:46.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031037872 0 0) 0x7fc4740a0730 con 0x7fc4740a2f10 2024-09-10T16:27:46.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient(hunting): _negotiate 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient(hunting): my global_id is 224271 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc460003520 con 0x7fc4740a2f10 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 >> v1:172.21.15.165:6789/0 conn(0x7fc47409d520 legacy=0x7fc47413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1277288598 0 0) 0x7fc460003520 con 0x7fc4740a2f10 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 >> v2:172.21.15.165:3300/0 conn(0x7fc4741329f0 msgr2=0x7fc474132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 --2- 172.21.15.164:0/2418941883 >> v2:172.21.15.165:3300/0 conn(0x7fc4741329f0 0x7fc474132dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 >> v1:172.21.15.165:6789/0 conn(0x7fc47409d520 legacy=0x7fc47413dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4740a1060 con 0x7fc4740a2f10 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _finish_auth 0 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fc45c001990 con 0x7fc4740a2f10 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc4789ad640 5 monclient: authenticate success, global_id 224271 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc4789ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:46.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: dump: 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:46.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 >> v1:172.21.15.164:6789/0 conn(0x7fc4740a2f10 legacy=0x7fc47409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.494+0000 7fc471d9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc460003b10 0x7fc460003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): picked mon.b con 0x7fc460003b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc460004800 0x7fc460004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): picked mon.a con 0x7fc460004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc46000ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): picked mon.c con 0x7fc4600053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc4740a2480 con 0x7fc460004800 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc46000c300 con 0x7fc460003b10 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc46000c500 con 0x7fc4600053c0 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc471d9b640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc4789ad640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc460004800 0x7fc460004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): get_auth_request con 0x7fc460004800 auth_method 0 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc46000ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): get_auth_request con 0x7fc4600053c0 auth_method 0 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:46.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc460003b10 0x7fc460003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): get_auth_request con 0x7fc460003b10 auth_method 0 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:46.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc47259c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient(hunting): handle_auth_done global_id 224271 payload 293 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc460003b10 msgr2=0x7fc460003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc460003b10 0x7fc460003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc460004800 msgr2=0x7fc460004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc460004800 0x7fc460004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 monclient: found mon.c 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:46.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc460003520 con 0x7fc4600053c0 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient: _finish_auth 0 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 10 monclient: _check_auth_tickets 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc472d9d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.495+0000 7fc46bfff640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc460004800 0x7fc460004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.496+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 reap_dead start 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc472d9d640 1 -- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 msgr2=0x7fc46000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc472d9d640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc46000ba20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc46c002090 tx=0x7fc460005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2937ab8be6e0e80c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc45c00c0e0 con 0x7fc4600053c0 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:46.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: dump: 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc46c005d40 con 0x7fc4600053c0 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 1 -- 172.21.15.164:0/2418941883 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c001d90 con 0x7fc4600053c0 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 4 set_mon_vals no callback set 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc471d9b640 10 monclient: dump: 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.521 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 10 monclient: shutdown 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 1 -- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 msgr2=0x7fc46000ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc46000ba20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc46c002090 tx=0x7fc460005830 comp rx=0 tx=0).stop 2024-09-10T16:27:46.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 1 -- 172.21.15.164:0/2418941883 shutdown_connections 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 1 --2- 172.21.15.164:0/2418941883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc46000ba20 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.498+0000 7fc4789ad640 1 -- 172.21.15.164:0/2418941883 >> 172.21.15.164:0/2418941883 conn(0x7fc4740a7320 msgr2=0x7fc4740a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.499+0000 7fc4789ad640 1 -- 172.21.15.164:0/2418941883 shutdown_connections 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.499+0000 7fc4789ad640 1 -- 172.21.15.164:0/2418941883 wait complete. 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.499+0000 7fc4789ad640 10 monclient: build_initial_monmap 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.499+0000 7fc4789ad640 10 monclient: monmap: 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:46.523 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 Processor -- start 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 -- start start 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient: init 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:46.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-c con 0x7fc4600053c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc474131730 0x7fc474131b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-b con 0x7fc474131730 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 0x7fc4741bdb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): picked mon.noname-a con 0x7fc4741bd6d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc47409da80 con 0x7fc4741bd6d0 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc47409d900 con 0x7fc474131730 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:46.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc47409dc00 con 0x7fc4600053c0 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc4789ad640 10 monclient(hunting): authenticate will time out at 2044.987647s 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 0x7fc4741bdb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49470/0 (socket says 172.21.15.164:49470) 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 -- 172.21.15.164:0/3405990918 learned_addr learned my addr 172.21.15.164:0/3405990918 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 10 monclient(hunting): get_auth_request con 0x7fc4600053c0 auth_method 0 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:46.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): get_auth_request con 0x7fc4741bd6d0 auth_method 0 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 msgr2=0x7fc4741cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 msgr2=0x7fc4741cac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc47259c640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc472d9d640 10 monclient(hunting): handle_auth_done global_id 224274 payload 824 2024-09-10T16:27:46.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.500+0000 7fc46bfff640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc474131730 0x7fc474131b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 msgr2=0x7fc4741cac80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc474131730 msgr2=0x7fc474131b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc474131730 0x7fc474131b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 monclient: found mon.noname-a 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.501+0000 7fc472d9d640 1 -- 172.21.15.164:0/3405990918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc47409e0f0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc472d9d640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 msgr2=0x7fc4741bdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:46.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc472d9d640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 0x7fc4741bdb50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc474074bd0 tx=0x7fc46c001830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c037040 con 0x7fc4741bd6d0 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: dump: 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: _finish_auth 0 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc46c041070 con 0x7fc4741bd6d0 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 5 monclient: authenticate success, global_id 224274 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 10 monclient: _renew_subs 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc47411ef10 con 0x7fc4741bd6d0 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:46.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc46c0076c0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc46affd640 10 monclient: dump: 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:46.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 10 monclient: _renew_subs 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc47159a640 4 set_mon_vals no callback set 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.503+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4740a6820 con 0x7fc4741bd6d0 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc46c007860 con 0x7fc4741bd6d0 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc4789ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc4789ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:46.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc4789ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4741bf1e0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc46affd640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 0x7fc4540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.504+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(734..734 src has 677..734) ==== 4711+0+0 (secure 0 0 0) 0x7fc46c072ca0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.505+0000 7fc47259c640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 0x7fc4540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.505+0000 7fc47259c640 10 monclient: get_auth_request con 0x7fc45403e210 auth_method 0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.507+0000 7fc47259c640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 0x7fc4540406d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc464006b10 tx=0x7fc464008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.509+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc47409e0f0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.509+0000 7fc46affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:46.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.509+0000 7fc46affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:46.650 INFO:tasks.rados.rados.0.smithi165.stdout:3302: done (1 left) 2024-09-10T16:27:46.651 INFO:tasks.rados.rados.0.smithi165.stdout:3303: write oid 18 current snap is 359 2024-09-10T16:27:46.651 INFO:tasks.rados.rados.0.smithi165.stdout:3301: finishing rollback tid 0 to smithi16534841-22 2024-09-10T16:27:46.651 INFO:tasks.rados.rados.0.smithi165.stdout:3303: seq_num 1148 ranges {494436=755155,2011446=521062,3075831=218605} 2024-09-10T16:27:46.658 INFO:tasks.rados.rados.0.smithi165.stdout:3303: writing smithi16534841-18 from 494436 to 1249591 tid 1 2024-09-10T16:27:46.666 INFO:tasks.rados.rados.0.smithi165.stdout:3303: writing smithi16534841-18 from 2011446 to 2532508 tid 2 2024-09-10T16:27:46.671 INFO:tasks.rados.rados.0.smithi165.stdout:3303: writing smithi16534841-18 from 3075831 to 3294436 tid 3 2024-09-10T16:27:46.671 INFO:tasks.rados.rados.0.smithi165.stdout:3304: rollback oid 5 current snap is 359 2024-09-10T16:27:46.671 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 5 to 342 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3301: finishing rollback tid 1 to smithi16534841-22 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3301: finishing rollback tid 2 to smithi16534841-22 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1464 (ObjNum 1117 snap 342 seq_num 1117) dirty exists 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3303: finishing write tid 1 to smithi16534841-18 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3303: finishing write tid 2 to smithi16534841-18 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3301: done (2 left) 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3305: read oid 23 snap -1 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3305: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3306: read oid 30 snap -1 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3306: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:27:46.672 INFO:tasks.rados.rados.0.smithi165.stdout:3307: snap_remove snap 331 2024-09-10T16:27:46.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.763+0000 7fc4789ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:46.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.763+0000 7fc4789ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:46.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.763+0000 7fc4789ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:46.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.763+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc4740733d0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.764+0000 7fc46affd640 1 -- 172.21.15.164:0/3405990918 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fc4740733d0 con 0x7fc4741bd6d0 2024-09-10T16:27:46.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.764+0000 7fc46affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:46.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.764+0000 7fc46affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:46.766 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":206,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 msgr2=0x7fc4540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 0x7fc4540406d0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fc464006b10 tx=0x7fc464008040 comp rx=0 tx=0).stop 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 10 monclient: shutdown 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 msgr2=0x7fc4741bdb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 0x7fc4741bdb50 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fc474074bd0 tx=0x7fc46c001830 comp rx=0 tx=0).stop 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 shutdown_connections 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc45403e210 0x7fc4540406d0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc4741bd6d0 0x7fc4741bdb50 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc474131730 0x7fc474131b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 --2- 172.21.15.164:0/3405990918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc4600053c0 0x7fc4741cac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:46.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 >> 172.21.15.164:0/3405990918 conn(0x7fc4740a7320 msgr2=0x7fc474072cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:46.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 shutdown_connections 2024-09-10T16:27:46.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:46.766+0000 7fc4789ad640 1 -- 172.21.15.164:0/3405990918 wait complete. 2024-09-10T16:27:47.662 INFO:tasks.rados.rados.0.smithi165.stdout:3307: done (4 left) 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3308: snap_create 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: finishing write tid 3 to smithi16534841-18 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: finishing write tid 4 to smithi16534841-18 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: finishing write tid 5 to smithi16534841-18 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: oid 18 updating version 0 to 2001 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: oid 18 updating version 2001 to 2003 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: oid 18 version 2003 is already newer than 2000 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: oid 18 version 2003 is already newer than 2002 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 2003 (ObjNum 1148 snap 359 seq_num 1148) dirty exists 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3303: left oid 18 (ObjNum 1148 snap 359 seq_num 1148) 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:3305: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:27:47.663 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:47.816 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:47.895 INFO:tasks.rados.rados.0.smithi165.stdout:3306: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:27:47.898 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1330613 2024-09-10T16:27:47.932 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:47.957 INFO:tasks.rados.rados.0.smithi165.stdout:3304: finishing rollback tid 0 to smithi16534841-5 2024-09-10T16:27:47.957 INFO:tasks.rados.rados.0.smithi165.stdout:3304: finishing rollback tid 1 to smithi16534841-5 2024-09-10T16:27:47.957 INFO:tasks.rados.rados.0.smithi165.stdout:3304: finishing rollback tid 2 to smithi16534841-5 2024-09-10T16:27:47.958 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1625 (ObjNum 1042 snap 321 seq_num 1042) dirty exists 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3303: done (4 left) 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3304: done (3 left) 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3305: done (2 left) 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3306: done (1 left) 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3308: done (0 left) 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3309: write (excl) oid 20 current snap is 360 2024-09-10T16:27:48.665 INFO:tasks.rados.rados.0.smithi165.stdout:3309: seq_num 1149 ranges {511243=430404,1669928=664720,3103254=207989} 2024-09-10T16:27:48.670 INFO:tasks.rados.rados.0.smithi165.stdout:3309: writing smithi16534841-20 from 511243 to 941647 tid 1 2024-09-10T16:27:48.677 INFO:tasks.rados.rados.0.smithi165.stdout:3309: writing smithi16534841-20 from 1669928 to 2334648 tid 2 2024-09-10T16:27:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:3309: writing smithi16534841-20 from 3103254 to 3311243 tid 3 2024-09-10T16:27:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:3310: delete oid 11 current snap is 360 2024-09-10T16:27:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:3309: finishing write tid 1 to smithi16534841-20 2024-09-10T16:27:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:3309: finishing write tid 2 to smithi16534841-20 2024-09-10T16:27:48.686 INFO:tasks.rados.rados.0.smithi165.stdout:3309: finishing write tid 3 to smithi16534841-20 2024-09-10T16:27:48.686 INFO:tasks.rados.rados.0.smithi165.stdout:3309: finishing write tid 4 to smithi16534841-20 2024-09-10T16:27:48.686 INFO:tasks.rados.rados.0.smithi165.stdout:3309: finishing write tid 5 to smithi16534841-20 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: oid 20 updating version 0 to 1838 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: oid 20 updating version 1838 to 1840 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: oid 20 version 1840 is already newer than 1839 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: oid 20 version 1840 is already newer than 1837 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1840 (ObjNum 1149 snap 360 seq_num 1149) dirty exists 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: left oid 20 (ObjNum 1149 snap 360 seq_num 1149) 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3309: done (1 left) 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3310: done (0 left) 2024-09-10T16:27:48.687 INFO:tasks.rados.rados.0.smithi165.stdout:3311: snap_create 2024-09-10T16:27:49.779 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 10 monclient: get_monmap_and_config 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 10 monclient: build_initial_monmap 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 10 monclient: monmap: 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.883 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:49.882899+0000 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:49.882899+0000 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:49.884 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 1 Processor -- start 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 1 -- start start 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.882+0000 7f061840f640 10 monclient: init 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f061009f930 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f061013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): picked mon.noname-b con 0x7f06100a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06100a2980 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061009fd10 con 0x7f06100a2980 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061009cd20 con 0x7f061009f930 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f061009a0d0 con 0x7f06100a2550 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f061840f640 10 monclient(hunting): authenticate will time out at 2048.370365s 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06169ac640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f06100a2980 0x7f061013db20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59078/0 (socket says 172.21.15.164:59078) 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06169ac640 1 -- 172.21.15.164:0/1854693414 learned_addr learned my addr 172.21.15.164:0/1854693414 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f061013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 10 monclient(hunting): get_auth_request con 0x7f06100a2550 auth_method 0 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:49.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 msgr2=0x7f061013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:27:49.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 msgr2=0x7f061013b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:27:49.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f061013b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:49.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.883+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f061013b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.885+0000 7f06169ac640 1 -- 172.21.15.164:0/1854693414 >> v1:172.21.15.164:6789/0 conn(0x7f06100a2980 legacy=0x7f061013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06161ab640 1 -- 172.21.15.164:0/1854693414 >> v1:172.21.15.165:6789/0 conn(0x7f061009f930 legacy=0x7f06100a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2637053728 0 0) 0x7f061009fd10 con 0x7f06100a2980 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): _negotiate 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): my global_id is 224277 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f05fc003520 con 0x7f06100a2980 2024-09-10T16:27:49.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3686797356 0 0) 0x7f05fc003520 con 0x7f06100a2980 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 msgr2=0x7f061013b5d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f061013b5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 >> v1:172.21.15.165:6789/0 conn(0x7f061009f930 legacy=0x7f06100a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _finish_hunting 0 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061009e840 con 0x7f06100a2980 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _finish_auth 0 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:49.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f061840f640 5 monclient: authenticate success, global_id 224277 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f0608001d20 con 0x7f06100a2980 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f061840f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: dump: 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:49.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 >> v1:172.21.15.164:6789/0 conn(0x7f06100a2980 legacy=0x7f061013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05fc003af0 0x7f05fc003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): picked mon.b con 0x7f05fc003af0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f05fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): picked mon.c con 0x7f06100a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05fc004f40 0x7f05fc00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): picked mon.a con 0x7f05fc004f40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f061009a0d0 con 0x7f05fc004f40 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f05fc00bf00 con 0x7f05fc003af0 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f05fc00c080 con 0x7f06100a2550 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06151a9640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f061840f640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f05fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05fc004f40 0x7f05fc00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): get_auth_request con 0x7f06100a2550 auth_method 0 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): get_auth_request con 0x7f05fc004f40 auth_method 0 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:49.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06161ab640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05fc003af0 0x7f05fc003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient(hunting): handle_auth_done global_id 224277 payload 293 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05fc003af0 msgr2=0x7f05fc003f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05fc003af0 0x7f05fc003f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05fc004f40 msgr2=0x7f05fc00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05fc004f40 0x7f05fc00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient: _finish_hunting 0 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 monclient: found mon.c 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:49.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05fc003520 con 0x7f06100a2550 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient: _finish_auth 0 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 10 monclient: _check_auth_tickets 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06159aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06161ab640 10 monclient: get_auth_request con 0x7f05fc003af0 auth_method 0 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06169ac640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f05fc004f40 0x7f05fc00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.886+0000 7f06161ab640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f05fc003af0 0x7f05fc003f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06159aa640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 msgr2=0x7f05fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06159aa640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f05fc004800 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f060000b9e0 tx=0x7f05fc005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60307de39e71792c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0600030070 con 0x7f06100a2550 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: dump: 2024-09-10T16:27:49.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f061840f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0600034400 con 0x7f06100a2550 2024-09-10T16:27:49.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f061840f640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 1 -- 172.21.15.164:0/1854693414 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060000b5e0 con 0x7f06100a2550 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f061840f640 4 set_mon_vals no callback set 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.889+0000 7f06151a9640 10 monclient: dump: 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:49.904 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 10 monclient: shutdown 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 -- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 msgr2=0x7f05fc004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 --2- 172.21.15.164:0/1854693414 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f06100a2550 0x7f05fc004800 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f060000b9e0 tx=0x7f05fc005830 comp rx=0 tx=0).stop 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f06169ac640 1 -- 172.21.15.164:0/1854693414 reap_dead start 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 -- 172.21.15.164:0/1854693414 shutdown_connections 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 -- 172.21.15.164:0/1854693414 >> 172.21.15.164:0/1854693414 conn(0x7f06100a3110 msgr2=0x7f06100a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 -- 172.21.15.164:0/1854693414 shutdown_connections 2024-09-10T16:27:49.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 1 -- 172.21.15.164:0/1854693414 wait complete. 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 10 monclient: build_initial_monmap 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.890+0000 7f061840f640 10 monclient: monmap: 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 Processor -- start 2024-09-10T16:27:49.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 -- start start 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient: init 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): picked mon.noname-a con 0x7f06100a2550 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): picked mon.noname-c con 0x7f0610091b00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06101cd220 0x7f06101cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): picked mon.noname-b con 0x7f06101cd220 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f061009c8b0 con 0x7f06100a2550 2024-09-10T16:27:49.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f061009d930 con 0x7f06101cd220 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f061009ca30 con 0x7f0610091b00 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f061840f640 10 monclient(hunting): authenticate will time out at 2048.378782s 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48520/0 (socket says 172.21.15.164:48520) 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 10 monclient(hunting): get_auth_request con 0x7f0610091b00 auth_method 0 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 1 -- 172.21.15.164:0/3356270092 learned_addr learned my addr 172.21.15.164:0/3356270092 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): get_auth_request con 0x7f06100a2550 auth_method 0 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 msgr2=0x7f0610091f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 msgr2=0x7f0610091f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06159aa640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.891+0000 7f06161ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06169ac640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06101cd220 0x7f06101cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06169ac640 10 monclient(hunting): get_auth_request con 0x7f06101cd220 auth_method 0 2024-09-10T16:27:49.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06169ac640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06169ac640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06169ac640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 10 monclient(hunting): handle_auth_done global_id 224280 payload 824 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 msgr2=0x7f0610091f20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06101cd220 msgr2=0x7f06101cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06101cd220 0x7f06101cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 10 monclient: _finish_hunting 0 2024-09-10T16:27:49.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 monclient: found mon.noname-a 2024-09-10T16:27:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:49.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.892+0000 7f06161ab640 1 -- 172.21.15.164:0/3356270092 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0610093620 con 0x7f06100a2550 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f06161ab640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 msgr2=0x7f06100915c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f06161ab640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f060c00d7a0 tx=0x7f060c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c018070 con 0x7f06100a2550 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:49.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: dump: 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:49.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f060c007480 con 0x7f06100a2550 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 5 monclient: authenticate success, global_id 224280 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f060c0057c0 con 0x7f06100a2550 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 10 monclient: _renew_subs 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0610094f50 con 0x7f06100a2550 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f0606ffd640 10 monclient: dump: 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.915 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f06149a8640 4 set_mon_vals no callback set 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 10 monclient: _renew_subs 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.894+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061009d620 con 0x7f06100a2550 2024-09-10T16:27:49.916 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f061840f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f060c01b070 con 0x7f06100a2550 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f061840f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f061840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06100a0840 con 0x7f06100a2550 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f0606ffd640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 0x7f05f40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.895+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(737..737 src has 677..737) ==== 4755+0+0 (secure 0 0 0) 0x7f060c0526c0 con 0x7f06100a2550 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.896+0000 7f06159aa640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 0x7f05f40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.896+0000 7f06159aa640 10 monclient: get_auth_request con 0x7f05f403e210 auth_method 0 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.898+0000 7f06159aa640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 0x7f05f40406d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0600006fd0 tx=0x7f0600030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.900+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f060c0510d0 con 0x7f06100a2550 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.900+0000 7f0606ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:49.917 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:49.900+0000 7f0606ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:50.027 INFO:tasks.rados.rados.0.smithi165.stdout:3311: done (0 left) 2024-09-10T16:27:50.028 INFO:tasks.rados.rados.0.smithi165.stdout:3312: write oid 49 current snap is 361 2024-09-10T16:27:50.028 INFO:tasks.rados.rados.0.smithi165.stdout:3312: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2024-09-10T16:27:50.034 INFO:tasks.rados.rados.0.smithi165.stdout:3312: writing smithi16534841-49 from 544857 to 1125759 tid 1 2024-09-10T16:27:50.040 INFO:tasks.rados.rados.0.smithi165.stdout:3312: writing smithi16534841-49 from 1670539 to 2338928 tid 2 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3312: writing smithi16534841-49 from 3041747 to 3344857 tid 3 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3313: read oid 5 snap -1 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3313: expect (ObjNum 1042 snap 321 seq_num 1042) 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3314: rollback oid 4 current snap is 361 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 4 to 347 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3312: finishing write tid 1 to smithi16534841-49 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3312: finishing write tid 2 to smithi16534841-49 2024-09-10T16:27:50.046 INFO:tasks.rados.rados.0.smithi165.stdout:3315: snap_create 2024-09-10T16:27:50.048 INFO:tasks.rados.rados.0.smithi165.stdout:3314: finishing rollback tid 1 to smithi16534841-4 2024-09-10T16:27:50.048 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1448 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:27:50.049 INFO:tasks.rados.rados.0.smithi165.stdout:3312: finishing write tid 3 to smithi16534841-49 2024-09-10T16:27:50.051 INFO:tasks.rados.rados.0.smithi165.stdout:3312: finishing write tid 4 to smithi16534841-49 2024-09-10T16:27:50.051 INFO:tasks.rados.rados.0.smithi165.stdout:3312: finishing write tid 5 to smithi16534841-49 2024-09-10T16:27:50.051 INFO:tasks.rados.rados.0.smithi165.stdout:3312: oid 49 updating version 0 to 1827 2024-09-10T16:27:50.052 INFO:tasks.rados.rados.0.smithi165.stdout:3312: oid 49 updating version 1827 to 1828 2024-09-10T16:27:50.052 INFO:tasks.rados.rados.0.smithi165.stdout:3312: oid 49 updating version 1828 to 1830 2024-09-10T16:27:50.052 INFO:tasks.rados.rados.0.smithi165.stdout:3312: oid 49 version 1830 is already newer than 1829 2024-09-10T16:27:50.052 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1830 (ObjNum 1151 snap 361 seq_num 1151) dirty exists 2024-09-10T16:27:50.052 INFO:tasks.rados.rados.0.smithi165.stdout:3312: left oid 49 (ObjNum 1151 snap 361 seq_num 1151) 2024-09-10T16:27:50.053 INFO:tasks.rados.rados.0.smithi165.stdout:3313: expect (ObjNum 1042 snap 321 seq_num 1042) 2024-09-10T16:27:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.148+0000 7f061840f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.148+0000 7f061840f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.148+0000 7f061840f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.148+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0610093a30 con 0x7f06100a2550 2024-09-10T16:27:50.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.149+0000 7f0606ffd640 1 -- 172.21.15.164:0/3356270092 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f060c005a20 con 0x7f06100a2550 2024-09-10T16:27:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.149+0000 7f0606ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:50.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.149+0000 7f0606ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:50.151 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:50.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 msgr2=0x7f05f40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 0x7f05f40406d0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f0600006fd0 tx=0x7f0600030000 comp rx=0 tx=0).stop 2024-09-10T16:27:50.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 10 monclient: shutdown 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 msgr2=0x7f06100915c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f060c00d7a0 tx=0x7f060c00dc60 comp rx=0 tx=0).stop 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 shutdown_connections 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f05f403e210 0x7f05f40406d0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f06101cd220 0x7f06101cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0610091b00 0x7f0610091f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 --2- 172.21.15.164:0/3356270092 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f06100a2550 0x7f06100915c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 >> 172.21.15.164:0/3356270092 conn(0x7f06100a3110 msgr2=0x7f06101c0940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:50.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 shutdown_connections 2024-09-10T16:27:50.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.151+0000 7f061840f640 1 -- 172.21.15.164:0/3356270092 wait complete. 2024-09-10T16:27:50.163 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:27:50.163 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:27:50.163 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:27:50.171 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1110534 2024-09-10T16:27:50.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.260+0000 7f0f29b2b640 10 monclient: get_monmap_and_config 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.260+0000 7f0f29b2b640 10 monclient: build_initial_monmap 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.260+0000 7f0f29b2b640 10 monclient: monmap: 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:50.261272+0000 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:50.261272+0000 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:50.262 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:50.263 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 Processor -- start 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 -- start start 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient: init 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f2409c310 0x7f0f2409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f2409c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f2409d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f2413a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f2409fbc0 con 0x7f0f2413a5c0 2024-09-10T16:27:50.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f240a2a20 con 0x7f0f2409d190 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0f2409c6f0 con 0x7f0f2409c310 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f29b2b640 10 monclient(hunting): authenticate will time out at 2048.748671s 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f2409c310 0x7f0f2409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f23fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0f2413a5c0 0x7f0f2413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59080/0 (socket says 172.21.15.164:59080) 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f23fff640 1 -- 172.21.15.164:0/1076015514 learned_addr learned my addr 172.21.15.164:0/1076015514 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): get_auth_request con 0x7f0f2409c310 auth_method 0 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.261+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 10 monclient(hunting): handle_auth_done global_id 224283 payload 274 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 1 -- 172.21.15.164:0/1076015514 >> v1:172.21.15.165:6789/0 conn(0x7f0f2409d190 legacy=0x7f0f2409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 1 -- 172.21.15.164:0/1076015514 >> v1:172.21.15.164:6789/0 conn(0x7f0f2413a5c0 legacy=0x7f0f2413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.262+0000 7f0f237fe640 1 -- 172.21.15.164:0/1076015514 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f2409a4d0 con 0x7f0f2409c310 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.264+0000 7f0f237fe640 1 -- 172.21.15.164:0/1076015514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409c310 msgr2=0x7f0f2409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.264+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1076015514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409c310 0x7f0f2409ea40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f14029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fda4a23293323b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 1 -- 172.21.15.164:0/1076015514 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14037070 con 0x7f0f2409c310 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: dump: 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.274 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 1 -- 172.21.15.164:0/1076015514 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f14032640 con 0x7f0f2409c310 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 5 monclient: authenticate success, global_id 224283 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:50.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 1 -- 172.21.15.164:0/1076015514 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14032a10 con 0x7f0f2409c310 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 4 set_mon_vals no callback set 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f227fc640 10 monclient: dump: 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.276 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 10 monclient: shutdown 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1076015514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409c310 msgr2=0x7f0f2409ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1076015514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409c310 0x7f0f2409ea40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f0f140097a0 tx=0x7f0f14029fd0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1076015514 shutdown_connections 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1076015514 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409c310 0x7f0f2409ea40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1076015514 >> 172.21.15.164:0/1076015514 conn(0x7f0f240a3110 msgr2=0x7f0f240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1076015514 shutdown_connections 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.265+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1076015514 wait complete. 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.266+0000 7f0f29b2b640 10 monclient: build_initial_monmap 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.266+0000 7f0f29b2b640 10 monclient: monmap: 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:50.277 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.266+0000 7f0f29b2b640 1 Processor -- start 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.266+0000 7f0f29b2b640 1 -- start start 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.266+0000 7f0f29b2b640 10 monclient: init 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:50.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f2409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f2413a5c0 0x7f0f24091700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f2413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 0x7f0f240920f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f24091c40 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0f240a2230 con 0x7f0f24091c40 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f240a20b0 con 0x7f0f2413a5c0 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0f24074e00 con 0x7f0f2409c310 2024-09-10T16:27:50.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f29b2b640 10 monclient(hunting): authenticate will time out at 2048.754326s 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 0x7f0f240920f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49516/0 (socket says 172.21.15.164:49516) 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 learned_addr learned my addr 172.21.15.164:0/1591029445 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 10 monclient(hunting): get_auth_request con 0x7f0f2409c310 auth_method 0 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): get_auth_request con 0x7f0f24091c40 auth_method 0 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 msgr2=0x7f0f24093b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 msgr2=0x7f0f24093b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f22ffd640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f2413a5c0 0x7f0f24091700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f22ffd640 10 monclient(hunting): get_auth_request con 0x7f0f2413a5c0 auth_method 0 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f22ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient(hunting): handle_auth_done global_id 224286 payload 824 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 msgr2=0x7f0f24093b80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f2409c310 0x7f0f24093b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f2413a5c0 msgr2=0x7f0f24091700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f2413a5c0 0x7f0f24091700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 monclient: found mon.noname-a 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.267+0000 7f0f23fff640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f24099cb0 con 0x7f0f24091c40 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.269+0000 7f0f23fff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 msgr2=0x7f0f240920f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.269+0000 7f0f23fff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 0x7f0f240920f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00caf0 tx=0x7f0f1c00cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c00ade0 con 0x7f0f24091c40 2024-09-10T16:27:50.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: dump: 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: _finish_auth 0 2024-09-10T16:27:50.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1c012400 con 0x7f0f24091c40 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 5 monclient: authenticate success, global_id 224286 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 monclient: _renew_subs 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f24094dc0 con 0x7f0f24091c40 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c0127f0 con 0x7f0f24091c40 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 10 monclient: dump: 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.284 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 monclient: _renew_subs 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f24094510 con 0x7f0f24091c40 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f21ffb640 4 set_mon_vals no callback set 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:50.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 0x7f0f2408df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24094ae0 con 0x7f0f2409d190 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0f1c0129d0 con 0x7f0f24091c40 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.270+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 0x7f0f2408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient: get_auth_request con 0x7f0f2409d190 auth_method 0 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f0bfff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f1003e240 0x7f0f10040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 10 monclient(hunting): handle_auth_done global_id 224289 payload 824 2024-09-10T16:27:50.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(738..738 src has 677..738) ==== 4755+0+0 (secure 0 0 0) 0x7f0f1c053400 con 0x7f0f24091c40 2024-09-10T16:27:50.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f2409f800 con 0x7f0f2409d190 2024-09-10T16:27:50.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f22ffd640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f1003e240 0x7f0f10040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.271+0000 7f0f22ffd640 10 monclient: get_auth_request con 0x7f0f1003e240 auth_method 0 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 msgr2=0x7f0f2408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 0x7f0f2408df80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f1402a010 tx=0x7f0f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14043070 con 0x7f0f2409d190 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 10 monclient: dump: 2024-09-10T16:27:50.289 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0f14032e40 con 0x7f0f2409d190 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:50.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 msgr2=0x7f0f2408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 0x7f0f2408df80 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f1402a010 tx=0x7f0f140046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 msgr2=0x7f0f2408df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f0bfff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2409d190 0x7f0f2408df80 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f0f1402a010 tx=0x7f0f140046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.273+0000 7f0f22ffd640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f1003e240 0x7f0f10040700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0f240999b0 tx=0x7f0f18009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.279+0000 7f0f29b2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.279+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 0x7f0f2408e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.279+0000 7f0f29b2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.279+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0f24093020 con 0x7f0f2408e4c0 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 0x7f0f2408e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient: get_auth_request con 0x7f0f2408e4c0 auth_method 0 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 10 monclient(hunting): handle_auth_done global_id 224292 payload 824 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.280+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0f24077260 con 0x7f0f2408e4c0 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f237fe640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 msgr2=0x7f0f2408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f237fe640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 0x7f0f2408e8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f24094c60 tx=0x7f0f14008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14037040 con 0x7f0f2408e4c0 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 10 monclient: dump: 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.292 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f0f14003610 con 0x7f0f2408e4c0 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 msgr2=0x7f0f2408e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 0x7f0f2408e8d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f24094c60 tx=0x7f0f14008030 comp rx=0 tx=0).stop 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 msgr2=0x7f0f2408e8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:50.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.282+0000 7f0f0bfff640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f2408e4c0 0x7f0f2408e8d0 secure :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f0f24094c60 tx=0x7f0f14008030 comp rx=0 tx=0).stop 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 122113, 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:50.294 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:50.295 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:50.296 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:50.297 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:50.298 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.299 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.300 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:50.301 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.302 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.303 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.304 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:50.305 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:50.306 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:50.307 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:50.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f1003e240 msgr2=0x7f0f10040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f1003e240 0x7f0f10040700 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f0f240999b0 tx=0x7f0f18009040 comp rx=0 tx=0).stop 2024-09-10T16:27:50.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 10 monclient: shutdown 2024-09-10T16:27:50.308 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 msgr2=0x7f0f240920f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f23fff640 1 -- 172.21.15.164:0/1591029445 reap_dead start 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 0x7f0f240920f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00caf0 tx=0x7f0f1c00cef0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 shutdown_connections 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 --2- 172.21.15.164:0/1591029445 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f24091c40 0x7f0f240920f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 >> 172.21.15.164:0/1591029445 conn(0x7f0f240a3110 msgr2=0x7f0f241397e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 shutdown_connections 2024-09-10T16:27:50.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.283+0000 7f0f29b2b640 1 -- 172.21.15.164:0/1591029445 wait complete. 2024-09-10T16:27:50.309 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:50.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:27:50.310 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:27:50.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.407+0000 7f3860564640 10 monclient: get_monmap_and_config 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.407+0000 7f3860564640 10 monclient: build_initial_monmap 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.407+0000 7f3860564640 10 monclient: monmap: 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:50.408302+0000 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:50.408302+0000 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:50.409 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:50.410 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:50.410 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:50.410 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.407+0000 7f3860564640 1 Processor -- start 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 -- start start 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient: init 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f385809c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f385809d160 0x7f385809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): picked mon.noname-a con 0x7f385809d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f385813a5c0 0x7f385813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): picked mon.noname-c con 0x7f385813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f385809a430 con 0x7f385809c180 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f38580a24e0 con 0x7f385809d160 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f38580a2bf0 con 0x7f385813a5c0 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f3860564640 10 monclient(hunting): authenticate will time out at 2048.895630s 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f385809d160 0x7f385809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385e300640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f385809c180 0x7f385809ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:59090/0 (socket says 172.21.15.164:59090) 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385e300640 1 -- 172.21.15.164:0/1415679819 learned_addr learned my addr 172.21.15.164:0/1415679819 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): get_auth_request con 0x7f385809d160 auth_method 0 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385eb01640 1 --2- 172.21.15.164:0/1415679819 >> v2:172.21.15.165:3300/0 conn(0x7f385813a5c0 0x7f385813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385eb01640 10 monclient(hunting): get_auth_request con 0x7f385813a5c0 auth_method 0 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385eb01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385eb01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.408+0000 7f385eb01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385eb01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385eb01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385eb01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 10 monclient(hunting): handle_auth_done global_id 224295 payload 274 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 1 -- 172.21.15.164:0/1415679819 >> v2:172.21.15.165:3300/0 conn(0x7f385813a5c0 msgr2=0x7f385813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 1 --2- 172.21.15.164:0/1415679819 >> v2:172.21.15.165:3300/0 conn(0x7f385813a5c0 0x7f385813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 1 -- 172.21.15.164:0/1415679819 >> v1:172.21.15.164:6789/0 conn(0x7f385809c180 legacy=0x7f385809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:50.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 1 monclient: found mon.noname-a 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.409+0000 7f385daff640 1 -- 172.21.15.164:0/1415679819 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385809c560 con 0x7f385809d160 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.411+0000 7f385daff640 1 -- 172.21.15.164:0/1415679819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 msgr2=0x7f385809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.411+0000 7f385daff640 1 --2- 172.21.15.164:0/1415679819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f385809f150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f38480097a0 tx=0x7f384802a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3127241c3ccbbb75 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 1 -- 172.21.15.164:0/1415679819 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037070 con 0x7f385809d160 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: dump: 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.423 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 1 -- 172.21.15.164:0/1415679819 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3848032440 con 0x7f385809d160 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 5 monclient: authenticate success, global_id 224295 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 1 -- 172.21.15.164:0/1415679819 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848032810 con 0x7f385809d160 2024-09-10T16:27:50.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 4 set_mon_vals no callback set 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f385d2fe640 10 monclient: dump: 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 10 monclient: shutdown 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 -- 172.21.15.164:0/1415679819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 msgr2=0x7f385809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 --2- 172.21.15.164:0/1415679819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f385809f150 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f38480097a0 tx=0x7f384802a010 comp rx=0 tx=0).stop 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 -- 172.21.15.164:0/1415679819 shutdown_connections 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 --2- 172.21.15.164:0/1415679819 >> v2:172.21.15.165:3300/0 conn(0x7f385813a5c0 0x7f385813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 --2- 172.21.15.164:0/1415679819 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f385809f150 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 -- 172.21.15.164:0/1415679819 >> 172.21.15.164:0/1415679819 conn(0x7f38580a3110 msgr2=0x7f38580a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 -- 172.21.15.164:0/1415679819 shutdown_connections 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.412+0000 7f3860564640 1 -- 172.21.15.164:0/1415679819 wait complete. 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.413+0000 7f3860564640 10 monclient: build_initial_monmap 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.413+0000 7f3860564640 10 monclient: monmap: 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:50.426 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.413+0000 7f3860564640 1 Processor -- start 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.413+0000 7f3860564640 1 -- start start 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.413+0000 7f3860564640 10 monclient: init 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:50.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f385809c180 0x7f3858073e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): picked mon.noname-b con 0x7f385809c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f38581cd790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): picked mon.noname-a con 0x7f385809d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 0x7f38581cdcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): picked mon.noname-c con 0x7f385813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3858074f90 con 0x7f385809d160 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3858074e10 con 0x7f385809c180 2024-09-10T16:27:50.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f385809f690 con 0x7f385813a5c0 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f3860564640 10 monclient(hunting): authenticate will time out at 2048.901187s 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f38581cd790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f38581cd790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48564/0 (socket says 172.21.15.164:48564) 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 -- 172.21.15.164:0/650291700 learned_addr learned my addr 172.21.15.164:0/650291700 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): get_auth_request con 0x7f385809d160 auth_method 0 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 0x7f38581cdcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f385809c180 0x7f3858073e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 10 monclient(hunting): get_auth_request con 0x7f385813a5c0 auth_method 0 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): get_auth_request con 0x7f385809c180 auth_method 0 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 msgr2=0x7f38581cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 msgr2=0x7f38581cdcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:50.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 0x7f38581cdcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 0x7f38581cdcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient(hunting): handle_auth_done global_id 224298 payload 824 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 msgr2=0x7f38581cdcd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f385813a5c0 0x7f38581cdcd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f385809c180 msgr2=0x7f3858073e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f385809c180 0x7f3858073e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 monclient: found mon.noname-a 2024-09-10T16:27:50.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.414+0000 7f385daff640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f385809d620 con 0x7f385809d160 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f385daff640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 msgr2=0x7f38581cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f385daff640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f38581cd790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38480098c0 tx=0x7f38480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3848037040 con 0x7f385809d160 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: dump: 2024-09-10T16:27:50.432 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: _finish_auth 0 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:50.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 5 monclient: authenticate success, global_id 224298 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3848042070 con 0x7f385809d160 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 10 monclient: _renew_subs 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38580962f0 con 0x7f385809d160 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f384803bb00 con 0x7f385809d160 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f384effd640 10 monclient: dump: 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:50.434 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f385cafd640 4 set_mon_vals no callback set 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 10 monclient: _renew_subs 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.416+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3858095130 con 0x7f385809d160 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f3860564640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f3860564640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f38581ce5f0 con 0x7f38581ce210 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f385e300640 10 monclient: get_auth_request con 0x7f38581ce210 auth_method 0 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3848047460 con 0x7f385809d160 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.417+0000 7f385e300640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f384effd640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:50.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(738..738 src has 677..738) ==== 4755+0+0 (secure 0 0 0) 0x7f38480734c0 con 0x7f385809d160 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 10 monclient: get_auth_request con 0x7f383c03e210 auth_method 0 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 msgr2=0x7f383c0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 msgr2=0x7f383c0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:50.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.418+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:50.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 10 monclient: get_auth_request con 0x7f38581ce210 auth_method 0 2024-09-10T16:27:50.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.619+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.620+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:50.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.620+0000 7f385eb01640 10 monclient: get_auth_request con 0x7f383c03e210 auth_method 0 2024-09-10T16:27:50.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:50.622+0000 7f385eb01640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f38580a1980 tx=0x7f385000b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 10 monclient: get_auth_request con 0x7f38581ce210 auth_method 0 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:27:51.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:51.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:51.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.020+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:27:51.183 INFO:tasks.rados.rados.0.smithi165.stdout:3312: done (3 left) 2024-09-10T16:27:51.184 INFO:tasks.rados.rados.0.smithi165.stdout:3313: done (2 left) 2024-09-10T16:27:51.184 INFO:tasks.rados.rados.0.smithi165.stdout:3314: done (1 left) 2024-09-10T16:27:51.184 INFO:tasks.rados.rados.0.smithi165.stdout:3315: done (0 left) 2024-09-10T16:27:51.184 INFO:tasks.rados.rados.0.smithi165.stdout:3316: write oid 43 current snap is 362 2024-09-10T16:27:51.184 INFO:tasks.rados.rados.0.smithi165.stdout:3316: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2024-09-10T16:27:51.190 INFO:tasks.rados.rados.0.smithi165.stdout:3316: writing smithi16534841-43 from 561664 to 1217815 tid 1 2024-09-10T16:27:51.195 INFO:tasks.rados.rados.0.smithi165.stdout:3316: writing smithi16534841-43 from 1729021 to 2257421 tid 2 2024-09-10T16:27:51.202 INFO:tasks.rados.rados.0.smithi165.stdout:3316: writing smithi16534841-43 from 2669170 to 3361664 tid 3 2024-09-10T16:27:51.202 INFO:tasks.rados.rados.0.smithi165.stdout:3317: delete oid 48 current snap is 362 2024-09-10T16:27:51.202 INFO:tasks.rados.rados.0.smithi165.stdout:3316: finishing write tid 1 to smithi16534841-43 2024-09-10T16:27:51.202 INFO:tasks.rados.rados.0.smithi165.stdout:3316: finishing write tid 2 to smithi16534841-43 2024-09-10T16:27:51.205 INFO:tasks.rados.rados.0.smithi165.stdout:3316: finishing write tid 3 to smithi16534841-43 2024-09-10T16:27:51.205 INFO:tasks.rados.rados.0.smithi165.stdout:3317: done (1 left) 2024-09-10T16:27:51.206 INFO:tasks.rados.rados.0.smithi165.stdout:3318: rollback oid 29 current snap is 362 2024-09-10T16:27:51.206 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 360 2024-09-10T16:27:51.206 INFO:tasks.rados.rados.0.smithi165.stdout:3319: snap_remove snap 356 2024-09-10T16:27:51.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:51.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 10 monclient: get_auth_request con 0x7f38581ce210 auth_method 0 2024-09-10T16:27:51.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:51.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:51.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:51.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:27:51.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:51.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:51.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:51.821+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3319: done (2 left) 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3320: rollback oid 14 current snap is 362 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 360 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3316: finishing write tid 4 to smithi16534841-43 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3316: finishing write tid 5 to smithi16534841-43 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3316: oid 43 updating version 0 to 1702 2024-09-10T16:27:52.189 INFO:tasks.rados.rados.0.smithi165.stdout:3316: oid 43 version 1702 is already newer than 1701 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3316: oid 43 version 1702 is already newer than 1700 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3316: oid 43 version 1702 is already newer than 1699 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1702 (ObjNum 1152 snap 362 seq_num 1152) dirty exists 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3316: left oid 43 (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3318: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1721 (ObjNum 129 snap 0 seq_num 1162800400) dirty dne 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3316: done (2 left) 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3318: done (1 left) 2024-09-10T16:27:52.190 INFO:tasks.rados.rados.0.smithi165.stdout:3321: read oid 7 snap 342 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3321: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3322: read oid 3 snap 311 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3322: expect (ObjNum 1014 snap 307 seq_num 1014) 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3323: copy_from oid 50 from oid 16 current snap is 362 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3324: write oid 34 current snap is 362 2024-09-10T16:27:52.191 INFO:tasks.rados.rados.0.smithi165.stdout:3324: seq_num 1154 ranges {595278=406649,1729632=532069,3007663=387615} 2024-09-10T16:27:52.194 INFO:tasks.rados.rados.0.smithi165.stdout:3324: writing smithi16534841-34 from 595278 to 1001927 tid 1 2024-09-10T16:27:52.201 INFO:tasks.rados.rados.0.smithi165.stdout:3324: writing smithi16534841-34 from 1729632 to 2261701 tid 2 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3324: writing smithi16534841-34 from 3007663 to 3395278 tid 3 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3325: snap_create 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3320: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3320: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3320: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1833 (ObjNum 1116 snap 342 seq_num 1116) dirty exists 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3323: finishing copy_from to smithi16534841-50 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3323: got expected ENOENT (src dne) 2024-09-10T16:27:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3323: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:27:52.209 INFO:tasks.rados.rados.0.smithi165.stdout:3322: expect (ObjNum 1014 snap 307 seq_num 1014) 2024-09-10T16:27:52.242 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 180496 2024-09-10T16:27:52.275 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:3324: finishing write tid 1 to smithi16534841-34 2024-09-10T16:27:52.310 INFO:tasks.rados.rados.0.smithi165.stdout:3321: expect (ObjNum 1022 snap 310 seq_num 1022) 2024-09-10T16:27:52.337 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 511481 2024-09-10T16:27:52.369 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: finishing write tid 2 to smithi16534841-34 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: finishing write tid 3 to smithi16534841-34 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: finishing write tid 4 to smithi16534841-34 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: finishing write tid 5 to smithi16534841-34 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: oid 34 updating version 0 to 1586 2024-09-10T16:27:52.406 INFO:tasks.rados.rados.0.smithi165.stdout:3324: oid 34 version 1586 is already newer than 1584 2024-09-10T16:27:52.407 INFO:tasks.rados.rados.0.smithi165.stdout:3324: oid 34 updating version 1586 to 1587 2024-09-10T16:27:52.407 INFO:tasks.rados.rados.0.smithi165.stdout:3324: oid 34 version 1587 is already newer than 1585 2024-09-10T16:27:52.407 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1587 (ObjNum 1154 snap 362 seq_num 1154) dirty exists 2024-09-10T16:27:52.407 INFO:tasks.rados.rados.0.smithi165.stdout:3324: left oid 34 (ObjNum 1154 snap 362 seq_num 1154) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3320: done (5 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3321: done (4 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3322: done (3 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3323: done (2 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3324: done (1 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3325: done (0 left) 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3326: read oid 25 snap -1 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3326: expect deleted 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3327: write (excl) oid 42 current snap is 363 2024-09-10T16:27:53.195 INFO:tasks.rados.rados.0.smithi165.stdout:3327: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2024-09-10T16:27:53.200 INFO:tasks.rados.rados.0.smithi165.stdout:3327: writing smithi16534841-42 from 612085 to 1093983 tid 1 2024-09-10T16:27:53.208 INFO:tasks.rados.rados.0.smithi165.stdout:3327: writing smithi16534841-42 from 1788114 to 2463841 tid 2 2024-09-10T16:27:53.213 INFO:tasks.rados.rados.0.smithi165.stdout:3327: writing smithi16534841-42 from 3035086 to 3412085 tid 3 2024-09-10T16:27:53.213 INFO:tasks.rados.rados.0.smithi165.stdout:3328: snap_create 2024-09-10T16:27:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3327: finishing write tid 1 to smithi16534841-42 2024-09-10T16:27:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3327: finishing write tid 2 to smithi16534841-42 2024-09-10T16:27:53.216 INFO:tasks.rados.rados.0.smithi165.stdout:3327: finishing write tid 3 to smithi16534841-42 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: finishing write tid 4 to smithi16534841-42 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: finishing write tid 5 to smithi16534841-42 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: oid 42 updating version 0 to 1877 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: oid 42 version 1877 is already newer than 1874 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: oid 42 version 1877 is already newer than 1876 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:3327: oid 42 version 1877 is already newer than 1875 2024-09-10T16:27:53.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1877 (ObjNum 1155 snap 363 seq_num 1155) dirty exists 2024-09-10T16:27:53.219 INFO:tasks.rados.rados.0.smithi165.stdout:3327: left oid 42 (ObjNum 1155 snap 363 seq_num 1155) 2024-09-10T16:27:53.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.414+0000 7f384dffb640 10 monclient: tick 2024-09-10T16:27:53.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.414+0000 7f384dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:27:53.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.414+0000 7f384dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:53.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.414+0000 7f384dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:53.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient: get_auth_request con 0x7f38581ce210 auth_method 0 2024-09-10T16:27:53.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.423+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.424+0000 7f385e300640 10 monclient(hunting): handle_auth_done global_id 244100 payload 824 2024-09-10T16:27:53.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.424+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f385809ef80 con 0x7f38581ce210 2024-09-10T16:27:53.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854016f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854018440 con 0x7f38581ce210 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f384effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.426+0000 7f384effd640 10 monclient: dump: 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.428 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3854018440 con 0x7f38581ce210 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854016f00 comp rx=0 tx=0).stop 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 msgr2=0x7f3858063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:53.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.427+0000 7f384effd640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f38581ce210 0x7f3858063d50 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854016f00 comp rx=0 tx=0).stop 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f3860564640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f3860564640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 0x7f385802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f3860564640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f385809d930 con 0x7f3858064290 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 0x7f385802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient: get_auth_request con 0x7f3858064290 auth_method 0 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 10 monclient(hunting): handle_auth_done global_id 244103 payload 824 2024-09-10T16:27:53.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.434+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3858071ef0 con 0x7f3858064290 2024-09-10T16:27:53.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.436+0000 7f385e300640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 msgr2=0x7f385802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.436+0000 7f385e300640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 0x7f385802d440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854007f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854022a20 con 0x7f3858064290 2024-09-10T16:27:53.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 10 monclient: dump: 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f38540173d0 con 0x7f3858064290 2024-09-10T16:27:53.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 msgr2=0x7f385802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 0x7f385802d440 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854007f20 comp rx=0 tx=0).stop 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 msgr2=0x7f385802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f384effd640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3858064290 0x7f385802d440 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3854016a40 tx=0x7f3854007f20 comp rx=0 tx=0).stop 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:53.440 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10345, 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:53.441 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.442 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.443 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:53.444 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.445 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.446 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.447 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.448 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.449 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.450 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.451 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.452 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.453 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.454 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:53.454 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.454 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.454 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.456 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:53.456 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:53.457 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 msgr2=0x7f383c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f383c03e210 0x7f383c0406d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f38580a1980 tx=0x7f385000b040 comp rx=0 tx=0).stop 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 10 monclient: shutdown 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 msgr2=0x7f38581cd790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 --2- 172.21.15.164:0/650291700 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f385809d160 0x7f38581cd790 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f38480098c0 tx=0x7f38480046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 reap_dead start 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f385eb01640 1 -- 172.21.15.164:0/650291700 reap_dead start 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 shutdown_connections 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 >> 172.21.15.164:0/650291700 conn(0x7f38580a3110 msgr2=0x7f3858138860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 shutdown_connections 2024-09-10T16:27:53.458 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.437+0000 7f3860564640 1 -- 172.21.15.164:0/650291700 wait complete. 2024-09-10T16:27:53.459 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:53.459 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:27:53.459 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:27:53.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.565+0000 7fc3d6ee6640 10 monclient: get_monmap_and_config 2024-09-10T16:27:53.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.565+0000 7fc3d6ee6640 10 monclient: build_initial_monmap 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.565+0000 7fc3d6ee6640 10 monclient: monmap: 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:53.566341+0000 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:53.566341+0000 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:53.567 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:53.568 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 Processor -- start 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 -- start start 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient: init 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d009c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d009d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3d009f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:53.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3d009a430 con 0x7fc3d009f4c0 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc3d00a24e0 con 0x7fc3d009c180 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d00a2bf0 con 0x7fc3d009d160 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d6ee6640 10 monclient(hunting): authenticate will time out at 2052.053763s 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48580/0 (socket says 172.21.15.164:48580) 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49610/0 (socket says 172.21.15.164:49610) 2024-09-10T16:27:53.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 1 -- 172.21.15.164:0/2334710014 learned_addr learned my addr 172.21.15.164:0/2334710014 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 10 monclient(hunting): get_auth_request con 0x7fc3d009c180 auth_method 0 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3d4c82640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 10 monclient(hunting): get_auth_request con 0x7fc3d009d160 auth_method 0 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.566+0000 7fc3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient(hunting): handle_auth_done global_id 244106 payload 274 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 1 -- 172.21.15.164:0/2334710014 >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 msgr2=0x7fc3d009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 1 --2- 172.21.15.164:0/2334710014 >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 1 -- 172.21.15.164:0/2334710014 >> v1:172.21.15.165:6789/0 conn(0x7fc3d009f4c0 legacy=0x7fc3d013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:53.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 1 monclient: found mon.noname-b 2024-09-10T16:27:53.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:53.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3cffff640 1 -- 172.21.15.164:0/2334710014 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d009c560 con 0x7fc3d009d160 2024-09-10T16:27:53.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.567+0000 7fc3d4c82640 1 --2- 172.21.15.164:0/2334710014 >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:53.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.569+0000 7fc3cffff640 1 -- 172.21.15.164:0/2334710014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 msgr2=0x7fc3d009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.569+0000 7fc3cffff640 1 --2- 172.21.15.164:0/2334710014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0097a0 tx=0x7fc3bc02a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f420353d0ca5a97 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 1 -- 172.21.15.164:0/2334710014 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc037070 con 0x7fc3d009d160 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: dump: 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.582 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 1 -- 172.21.15.164:0/2334710014 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3bc032440 con 0x7fc3d009d160 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 5 monclient: authenticate success, global_id 244106 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 1 -- 172.21.15.164:0/2334710014 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc032810 con 0x7fc3d009d160 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 4 set_mon_vals no callback set 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3cf7fe640 10 monclient: dump: 2024-09-10T16:27:53.583 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 10 monclient: shutdown 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/2334710014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 msgr2=0x7fc3d009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/2334710014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0097a0 tx=0x7fc3bc02a050 comp rx=0 tx=0).stop 2024-09-10T16:27:53.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/2334710014 shutdown_connections 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/2334710014 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009d160 0x7fc3d009ef80 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/2334710014 >> v2:172.21.15.164:3300/0 conn(0x7fc3d009c180 0x7fc3d009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/2334710014 >> 172.21.15.164:0/2334710014 conn(0x7fc3d00a3110 msgr2=0x7fc3d00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/2334710014 shutdown_connections 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.570+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/2334710014 wait complete. 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.571+0000 7fc3d6ee6640 10 monclient: build_initial_monmap 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.571+0000 7fc3d6ee6640 10 monclient: monmap: 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.585 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.571+0000 7fc3d6ee6640 1 Processor -- start 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.571+0000 7fc3d6ee6640 1 -- start start 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.571+0000 7fc3d6ee6640 10 monclient: init 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 0x7fc3d008eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-a con 0x7fc3d009c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d009d160 0x7fc3d008f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-b con 0x7fc3d009d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:53.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009f4c0 0x7fc3d008f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): picked mon.noname-c con 0x7fc3d009f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0074f90 con 0x7fc3d009c180 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d0074e10 con 0x7fc3d009d160 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3d009d930 con 0x7fc3d009f4c0 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d6ee6640 10 monclient(hunting): authenticate will time out at 2052.059478s 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 0x7fc3d008eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 0x7fc3d008eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48592/0 (socket says 172.21.15.164:48592) 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 -- 172.21.15.164:0/1823707307 learned_addr learned my addr 172.21.15.164:0/1823707307 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): get_auth_request con 0x7fc3d009c180 auth_method 0 2024-09-10T16:27:53.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009f4c0 0x7fc3d008f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): get_auth_request con 0x7fc3d009f4c0 auth_method 0 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d5483640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient(hunting): handle_auth_done global_id 224304 payload 824 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d009d160 0x7fc3d008f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009f4c0 msgr2=0x7fc3d008f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3d009f4c0 0x7fc3d008f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d009d160 msgr2=0x7fc3d008f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d009d160 0x7fc3d008f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient: _finish_hunting 0 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 monclient: found mon.noname-a 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3d4c82640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3d0092350 con 0x7fc3d009c180 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3cffff640 10 monclient: get_auth_request con 0x7fc3d009d160 auth_method 0 2024-09-10T16:27:53.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.572+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d009d160 0x7fc3d008f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d4c82640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 msgr2=0x7fc3d008eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d4c82640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 0x7fc3d008eaf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4010f10 tx=0x7fc3c400e730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c401e070 con 0x7fc3d009c180 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: dump: 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3c4011a70 con 0x7fc3d009c180 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 5 monclient: authenticate success, global_id 224304 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 10 monclient: _renew_subs 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3d0090e90 con 0x7fc3d009c180 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3c4011e20 con 0x7fc3d009c180 2024-09-10T16:27:53.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3cd7fa640 10 monclient: dump: 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3ceffd640 4 set_mon_vals no callback set 2024-09-10T16:27:53.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 10 monclient: _renew_subs 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3d0090b50 con 0x7fc3d009c180 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.575+0000 7fc3d6ee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 0x7fc3d0063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d002e9a0 con 0x7fc3d002e5c0 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc3c4021070 con 0x7fc3d009c180 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 0x7fc3d0063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient: get_auth_request con 0x7fc3d002e5c0 auth_method 0 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cd7fa640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3b403e210 0x7fc3b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(741..741 src has 677..741) ==== 4771+0+0 (secure 0 0 0) 0x7fc3c40558d0 con 0x7fc3d009c180 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 10 monclient(hunting): handle_auth_done global_id 234178 payload 824 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3d5483640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3b403e210 0x7fc3b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3cffff640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3d0091bc0 con 0x7fc3d002e5c0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.576+0000 7fc3d5483640 10 monclient: get_auth_request con 0x7fc3b403e210 auth_method 0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.578+0000 7fc3cffff640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 msgr2=0x7fc3d0063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.578+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 0x7fc3d0063d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc034000 tx=0x7fc3bc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc041070 con 0x7fc3d002e5c0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:53.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 10 monclient: dump: 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3d5483640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3b403e210 0x7fc3b40406d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80055e0 tx=0x7fc3c8003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc3bc03c660 con 0x7fc3d002e5c0 2024-09-10T16:27:53.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:53.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 msgr2=0x7fc3d0063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 0x7fc3d0063d50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc034000 tx=0x7fc3bc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 msgr2=0x7fc3d0063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:53.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.579+0000 7fc3cd7fa640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d002e5c0 0x7fc3d0063d50 secure :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc034000 tx=0x7fc3bc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3d6ee6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 0x7fc3d01d3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3d6ee6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3d009dd60 con 0x7fc3d0064290 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 0x7fc3d01d3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient: get_auth_request con 0x7fc3d0064290 auth_method 0 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.585+0000 7fc3cffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.586+0000 7fc3cffff640 10 monclient(hunting): handle_auth_done global_id 234181 payload 824 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.586+0000 7fc3cffff640 1 -- 172.21.15.164:0/1823707307 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3d0071ef0 con 0x7fc3d0064290 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cffff640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 msgr2=0x7fc3d01d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cffff640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 0x7fc3d01d3570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0097a0 tx=0x7fc3bc009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3bc052020 con 0x7fc3d0064290 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.588+0000 7fc3cd7fa640 10 monclient: dump: 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fc3bc03ce10 con 0x7fc3d0064290 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 msgr2=0x7fc3d01d3570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 0x7fc3d01d3570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0097a0 tx=0x7fc3bc009190 comp rx=0 tx=0).stop 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 msgr2=0x7fc3d01d3570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:53.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3cd7fa640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3d0064290 0x7fc3d01d3570 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0097a0 tx=0x7fc3bc009190 comp rx=0 tx=0).stop 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 73872, 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.601 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 5, 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.602 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.603 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:53.604 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:53.605 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.606 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.608 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.609 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:53.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:53.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.612 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:53.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:53.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:53.615 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:53.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:53.615 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:53.615 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3b403e210 msgr2=0x7fc3b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3b403e210 0x7fc3b40406d0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc3c80055e0 tx=0x7fc3c8003d10 comp rx=0 tx=0).stop 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 10 monclient: shutdown 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 msgr2=0x7fc3d008eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 --2- 172.21.15.164:0/1823707307 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3d009c180 0x7fc3d008eaf0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc3c4010f10 tx=0x7fc3c400e730 comp rx=0 tx=0).stop 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d5483640 1 -- 172.21.15.164:0/1823707307 reap_dead start 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d5483640 1 -- 172.21.15.164:0/1823707307 reap_dead start 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 shutdown_connections 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 >> 172.21.15.164:0/1823707307 conn(0x7fc3d00a3110 msgr2=0x7fc3d0099690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 shutdown_connections 2024-09-10T16:27:53.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.589+0000 7fc3d6ee6640 1 -- 172.21.15.164:0/1823707307 wait complete. 2024-09-10T16:27:53.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:53.617 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:27:53.617 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.713+0000 7f934c433640 10 monclient: get_monmap_and_config 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.713+0000 7f934c433640 10 monclient: build_initial_monmap 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.713+0000 7f934c433640 10 monclient: monmap: 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.715 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:53.714404+0000 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:53.714404+0000 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 Processor -- start 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 -- start start 2024-09-10T16:27:53.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient: init 2024-09-10T16:27:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:27:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f934409d930 0x7f934409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): picked mon.noname-c con 0x7f934409d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:53.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f934409ab50 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f934409e250 0x7f9344134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): picked mon.noname-a con 0x7f934409e250 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93440a0870 con 0x7f934409ab50 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f934409d3e0 con 0x7f934409e250 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9344074c70 con 0x7f934409d930 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934c433640 10 monclient(hunting): authenticate will time out at 2052.201771s 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f934409e250 0x7f9344134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f934409e250 0x7f9344134c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48600/0 (socket says 172.21.15.164:48600) 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 1 -- 172.21.15.164:0/704796637 learned_addr learned my addr 172.21.15.164:0/704796637 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 10 monclient(hunting): get_auth_request con 0x7f934409e250 auth_method 0 2024-09-10T16:27:53.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.714+0000 7f934a1cf640 1 --2- 172.21.15.164:0/704796637 >> v2:172.21.15.165:3300/0 conn(0x7f934409d930 0x7f934409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a1cf640 10 monclient(hunting): get_auth_request con 0x7f934409d930 auth_method 0 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a1cf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient(hunting): handle_auth_done global_id 224307 payload 274 2024-09-10T16:27:53.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 1 -- 172.21.15.164:0/704796637 >> v2:172.21.15.165:3300/0 conn(0x7f934409d930 msgr2=0x7f934409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 1 --2- 172.21.15.164:0/704796637 >> v2:172.21.15.165:3300/0 conn(0x7f934409d930 0x7f934409dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 1 -- 172.21.15.164:0/704796637 >> v1:172.21.15.165:6789/0 conn(0x7f934409ab50 legacy=0x7f9344134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient: _finish_hunting 0 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 1 monclient: found mon.noname-a 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:53.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.715+0000 7f934a9d0640 1 -- 172.21.15.164:0/704796637 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934409ecb0 con 0x7f934409e250 2024-09-10T16:27:53.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.717+0000 7f934a9d0640 1 -- 172.21.15.164:0/704796637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 msgr2=0x7f9344134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934a9d0640 1 --2- 172.21.15.164:0/704796637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f9344134c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f934406d500 tx=0x7f933c02a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f8ff1204801c41c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 1 -- 172.21.15.164:0/704796637 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c036070 con 0x7f934409e250 2024-09-10T16:27:53.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: dump: 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.729 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: _finish_auth 0 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 1 -- 172.21.15.164:0/704796637 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f933c007800 con 0x7f934409e250 2024-09-10T16:27:53.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 5 monclient: authenticate success, global_id 224307 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 4 set_mon_vals no callback set 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 1 -- 172.21.15.164:0/704796637 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933c007bd0 con 0x7f934409e250 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f93491cd640 10 monclient: dump: 2024-09-10T16:27:53.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.732 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 10 monclient: shutdown 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 -- 172.21.15.164:0/704796637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 msgr2=0x7f9344134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 --2- 172.21.15.164:0/704796637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f9344134c90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f934406d500 tx=0x7f933c02a5b0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 -- 172.21.15.164:0/704796637 shutdown_connections 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 --2- 172.21.15.164:0/704796637 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f9344134c90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 --2- 172.21.15.164:0/704796637 >> v2:172.21.15.165:3300/0 conn(0x7f934409d930 0x7f934409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 -- 172.21.15.164:0/704796637 >> 172.21.15.164:0/704796637 conn(0x7f93440a3110 msgr2=0x7f93440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 -- 172.21.15.164:0/704796637 shutdown_connections 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.718+0000 7f934c433640 1 -- 172.21.15.164:0/704796637 wait complete. 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.719+0000 7f934c433640 10 monclient: build_initial_monmap 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.719+0000 7f934c433640 10 monclient: monmap: 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:53.733 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.719+0000 7f934c433640 1 Processor -- start 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.719+0000 7f934c433640 1 -- start start 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.719+0000 7f934c433640 10 monclient: init 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:53.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): picked mon.noname-c con 0x7f934409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934409d930 0x7f9344139460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): picked mon.noname-b con 0x7f934409d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): picked mon.noname-a con 0x7f934409e250 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f934409ef30 con 0x7f934409e250 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f934409e6e0 con 0x7f934409d930 2024-09-10T16:27:53.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f93440a2940 con 0x7f934409ab50 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934c433640 10 monclient(hunting): authenticate will time out at 2052.207283s 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:49624/0 (socket says 172.21.15.164:49624) 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:48608/0 (socket says 172.21.15.164:48608) 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 -- 172.21.15.164:0/1011787429 learned_addr learned my addr 172.21.15.164:0/1011787429 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): get_auth_request con 0x7f934409ab50 auth_method 0 2024-09-10T16:27:53.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): get_auth_request con 0x7f934409e250 auth_method 0 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f93499ce640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934409d930 0x7f9344139460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f93499ce640 10 monclient(hunting): get_auth_request con 0x7f934409d930 auth_method 0 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f93499ce640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f93499ce640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f93499ce640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient(hunting): handle_auth_done global_id 244112 payload 824 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 -- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934409d930 msgr2=0x7f9344139460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934409d930 0x7f9344139460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 -- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 msgr2=0x7f93441370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient: _finish_hunting 0 2024-09-10T16:27:53.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 monclient: found mon.noname-c 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a1cf640 1 -- 172.21.15.164:0/1011787429 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934413d990 con 0x7f934409ab50 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.720+0000 7f934a9d0640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.722+0000 7f934a1cf640 1 -- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 msgr2=0x7f934413e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.722+0000 7f934a1cf640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f934000d830 tx=0x7f934000dcf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000b8d0 con 0x7f934409ab50 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: dump: 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.739 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: _finish_auth 0 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 5 monclient: authenticate success, global_id 244112 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9340019070 con 0x7f934409ab50 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000bc90 con 0x7f934409ab50 2024-09-10T16:27:53.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 10 monclient: _renew_subs 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9344138900 con 0x7f934409ab50 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f933affd640 10 monclient: dump: 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:53.741 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f93489cc640 4 set_mon_vals no callback set 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 10 monclient: _renew_subs 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.723+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f93441377f0 con 0x7f934409ab50 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f934001e460 con 0x7f934409ab50 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f934c433640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f934c433640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f934c433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f934409dd10 con 0x7f934409ab50 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f933affd640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 0x7f93280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(741..741 src has 677..741) ==== 4771+0+0 (secure 0 0 0) 0x7f93400531b0 con 0x7f934409ab50 2024-09-10T16:27:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f93499ce640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 0x7f93280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:53.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.724+0000 7f93499ce640 10 monclient: get_auth_request con 0x7f932803e210 auth_method 0 2024-09-10T16:27:53.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.727+0000 7f93499ce640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 0x7f93280406d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9334006fd0 tx=0x7f9334008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:53.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.729+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9340051bb0 con 0x7f934409ab50 2024-09-10T16:27:53.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.729+0000 7f933affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:53.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.729+0000 7f933affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:53.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.975+0000 7f934c433640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:27:53.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.975+0000 7f934c433640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:27:53.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.975+0000 7f934c433640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:53.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.975+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f934413a830 con 0x7f934409ab50 2024-09-10T16:27:53.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.977+0000 7f933affd640 1 -- 172.21.15.164:0/1011787429 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9340014430 con 0x7f934409ab50 2024-09-10T16:27:53.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.977+0000 7f933affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:27:53.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.977+0000 7f933affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=2486323861,config=2586702516,health=3566308711,kv=805391194,logm=2919658352}) 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=29,config=2,health=11,kv=12,logm=46} crc {auth=1261777780,config=2586702516,health=517958292,kv=805391194,logm=1097066912}) 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=30,config=2,health=12,kv=12,logm=44} crc {auth=2486323861,config=2586702516,health=3566308711,kv=805391194,logm=2919658352}) 2024-09-10T16:27:53.979 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.977+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=30,config=2,health=10,kv=12,logm=46} crc {auth=2486323861,config=2586702516,health=1812244325,kv=805391194,logm=924402269}) 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1678682726}) 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747881157}) 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1678682726}) 2024-09-10T16:27:53.980 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.978+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1320762177}) 2024-09-10T16:27:53.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 msgr2=0x7f93280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 0x7f93280406d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f9334006fd0 tx=0x7f9334008040 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 10 monclient: shutdown 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 msgr2=0x7f934413e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f934000d830 tx=0x7f934000dcf0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 shutdown_connections 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f932803e210 0x7f93280406d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f934409e250 0x7f93441370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f934409d930 0x7f9344139460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 --2- 172.21.15.164:0/1011787429 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f934409ab50 0x7f934413e300 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 >> 172.21.15.164:0/1011787429 conn(0x7f93440a3110 msgr2=0x7f9344099450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 shutdown_connections 2024-09-10T16:27:53.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:53.979+0000 7f934c433640 1 -- 172.21.15.164:0/1011787429 wait complete. 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469721677}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3661128736}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2469721677}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.979+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1273733047}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093450389}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978162136}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3093450389}) 2024-09-10T16:27:53.985 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.981+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2366156873}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059426993}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159511513}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3059426993}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.982+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3475351836}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176579564}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=529818476}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176579564}) 2024-09-10T16:27:53.986 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.983+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1604609649}) 2024-09-10T16:27:53.990 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.990 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181407450}) 2024-09-10T16:27:53.990 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946658904}) 2024-09-10T16:27:53.990 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.990 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=181407450}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3348399358}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.984+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040236096}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2104190204}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1040236096}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1721381675}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709676265}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4166917669}) 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.991 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=709676265}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.985+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3658932}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176669275}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117851335}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176669275}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.986+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53958305}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461112430}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3405812071}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3461112430}) 2024-09-10T16:27:53.992 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.987+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2028584271}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022982925}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=858675264}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1022982925}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.988+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2352544999}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280866951}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3316881132}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=280866951}) 2024-09-10T16:27:53.993 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4042913752}) 2024-09-10T16:27:53.994 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.994 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3518867473}) 2024-09-10T16:27:53.994 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2143413523}) 2024-09-10T16:27:53.995 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:53.995 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3518867473}) 2024-09-10T16:27:53.995 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.989+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2141119450}) 2024-09-10T16:27:53.996 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110810634}) 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2613029998}) 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1110810634}) 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.990+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932162645}) 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364921967}) 2024-09-10T16:27:54.002 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777214291}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=364921967}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.991+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1469553606}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344337235}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3557221379}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3344337235}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.992+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1305888488}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424352848}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4268484028}) 2024-09-10T16:27:54.003 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1424352848}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.993+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=254871775}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556235596}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2630068432}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556235596}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2165070452}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535117523}) 2024-09-10T16:27:54.004 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=779194378}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.994+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1535117523}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=753778655}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108431048}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3665749369}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2108431048}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.995+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1034406534}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415318901}) 2024-09-10T16:27:54.005 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1942487847}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2415318901}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.996+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2729158253}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086864414}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3190445373}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086864414}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.997+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487797231}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809611345}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1993406428}) 2024-09-10T16:27:54.006 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2809611345}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403994863}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.998+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840294994}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=900055312}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840294994}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3687599702}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264789217}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3292919039}) 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.007 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=264789217}) 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:53.999+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078448865}) 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055663212}) 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4040801001}) 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2055663212}) 2024-09-10T16:27:54.008 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.000+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2690131342}) 2024-09-10T16:27:54.015 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.015 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562617987}) 2024-09-10T16:27:54.015 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2098081059}) 2024-09-10T16:27:54.015 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562617987}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.001+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=504577652}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770297423}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3672582719}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=770297423}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.002+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3446685222}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752346083}) 2024-09-10T16:27:54.016 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319526760}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3752346083}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448352749}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295325815}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2865736138}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.003+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=295325815}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518600305}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300256525}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2972896497}) 2024-09-10T16:27:54.017 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=300256525}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.004+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=221474052}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801935450}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=717823683}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1801935450}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.005+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2755147766}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560827496}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133934875}) 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.018 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560827496}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3353579462}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137224932}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3460928587}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2137224932}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.006+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1257215065}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226835491}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=551589290}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4226835491}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.007+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2617226507}) 2024-09-10T16:27:54.019 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567219746}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1768522116}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=567219746}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.008+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=585603932}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420960526}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3348487618}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3420960526}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.009+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2049121944}) 2024-09-10T16:27:54.020 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050118439}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2439617354}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4050118439}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3449084017}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313831865}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994447093}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.010+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313831865}) 2024-09-10T16:27:54.021 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.011+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279462322}) 2024-09-10T16:27:54.023 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636048451}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1062369391}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=636048451}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.018+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3241795841}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247092425}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3240566451}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=247092425}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3795909877}) 2024-09-10T16:27:54.024 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221067293}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1265713842}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.019+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4221067293}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024353306}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565405146}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=607337138}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565405146}) 2024-09-10T16:27:54.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.020+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465352705}) 2024-09-10T16:27:54.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775252147}) 2024-09-10T16:27:54.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1975769029}) 2024-09-10T16:27:54.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=775252147}) 2024-09-10T16:27:54.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.200+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2042077746}) 2024-09-10T16:27:54.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861271218}) 2024-09-10T16:27:54.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2164730931}) 2024-09-10T16:27:54.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861271218}) 2024-09-10T16:27:54.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.210+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2101542222}) 2024-09-10T16:27:54.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.212+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539216728}) 2024-09-10T16:27:54.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007052240}) 2024-09-10T16:27:54.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3539216728}) 2024-09-10T16:27:54.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.213+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3407762049}) 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801188995}) 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=746474845}) 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801188995}) 2024-09-10T16:27:54.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.224+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792884698}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120858623}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1373943685}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1120858623}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53441384}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911184967}) 2024-09-10T16:27:54.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646325654}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3911184967}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.225+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=499994895}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249311103}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1584765712}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=249311103}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.226+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3861535243}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206411111}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26038608}) 2024-09-10T16:27:54.230 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=206411111}) 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1832576276}) 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903689711}) 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2655117196}) 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903689711}) 2024-09-10T16:27:54.231 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.227+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1068304863}) 2024-09-10T16:27:54.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.240 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785152140}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3658861678}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=785152140}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.228+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3300297298}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707114661}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2482338686}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3707114661}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2282513509}) 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301593539}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=806281482}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301593539}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.229+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642254961}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248541463}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2765678350}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248541463}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.230+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=53684434}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862350498}) 2024-09-10T16:27:54.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3384351780}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3862350498}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3922240707}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488142705}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3768818492}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3488142705}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.231+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1906379644}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888679219}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948202835}) 2024-09-10T16:27:54.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3888679219}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.232+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959931745}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805438694}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=175282939}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805438694}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1850942252}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759776506}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3960466842}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1759776506}) 2024-09-10T16:27:54.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.233+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2724687126}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403304912}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908674631}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2403304912}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.234+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2857462668}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547158516}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581870808}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=547158516}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=855064724}) 2024-09-10T16:27:54.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307413914}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=936075790}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3307413914}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.235+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=212901859}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216468738}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2509622369}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=216468738}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.236+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=893681932}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429461758}) 2024-09-10T16:27:54.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=345121301}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=429461758}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.237+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909192259}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972326360}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2596299269}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1972326360}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=19007668}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633988173}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3974075784}) 2024-09-10T16:27:54.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=633988173}) 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.238+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3898142784}) 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329079647}) 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4151078363}) 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1329079647}) 2024-09-10T16:27:54.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4226641462}) 2024-09-10T16:27:54.249 INFO:tasks.rados.rados.0.smithi165.stdout:3326: done (2 left) 2024-09-10T16:27:54.249 INFO:tasks.rados.rados.0.smithi165.stdout:3327: done (1 left) 2024-09-10T16:27:54.249 INFO:tasks.rados.rados.0.smithi165.stdout:3328: done (0 left) 2024-09-10T16:27:54.250 INFO:tasks.rados.rados.0.smithi165.stdout:3329: snap_create 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3570027684}) 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3468104128}) 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3570027684}) 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.239+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1525613453}) 2024-09-10T16:27:54.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537791498}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3909297848}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537791498}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.240+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3762690923}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635281174}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=896533889}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635281174}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1975382601}) 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.259 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.241+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759160422}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=823661787}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759160422}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3566217123}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487721342}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2970510708}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=487721342}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.242+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1156718686}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590472161}) 2024-09-10T16:27:54.260 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4290944808}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3590472161}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=464877058}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380611033}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=960621591}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.243+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1380611033}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1683291852}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398421150}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4004179577}) 2024-09-10T16:27:54.261 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1398421150}) 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.244+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1106750494}) 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818169824}) 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=118299943}) 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.262 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1818169824}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.245+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2468364691}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549967786}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2178270358}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1549967786}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3208721888}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967450017}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1184475025}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967450017}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.246+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=671809330}) 2024-09-10T16:27:54.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607756155}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204577941}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2607756155}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=908449311}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145387222}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2330453884}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145387222}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.247+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1503846139}) 2024-09-10T16:27:54.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=2,nvmeofgw=2} crc {logm=1286135698,mdsmap=931935893,mgr=3838082885,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2903209570,mon_config_key=472225872,monmap=765578399,nvmeofgw=1596038039}) 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=58,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=1} crc {logm=3995158263,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2903209570,mon_config_key=665552247,monmap=846217120}) 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=58,mdsmap=3,mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=2,nvmeofgw=2} crc {logm=1286135698,mdsmap=931935893,mgr=3838082885,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2903209570,mon_config_key=472225872,monmap=765578399,nvmeofgw=1596038039}) 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.248+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=57,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=1} crc {logm=2885237862,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=15573522,mon_config_key=4125614794,monmap=1433040829,nvmeofgw=598337686,osd_metadata=3368639193}) 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=92} crc {nvmeofgw=1228700967,osd_metadata=4005186995,osd_pg_creating=4046207900,osd_snap=673044462}) 2024-09-10T16:27:54.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=92} crc {nvmeofgw=1228700967,osd_metadata=4005186995,osd_pg_creating=4046207900,osd_snap=829956550}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {nvmeofgw=1,osd_metadata=6,osd_pg_creating=1,osd_snap=92} crc {nvmeofgw=1228700967,osd_metadata=4005186995,osd_pg_creating=4046207900,osd_snap=673044462}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.249+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {nvmeofgw=1,osd_metadata=4,osd_pg_creating=1,osd_snap=94} crc {nvmeofgw=1228700967,osd_metadata=3492384962,osd_pg_creating=4046207900,osd_snap=2160321070}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=3057494235,osdmap=3712812351}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=48,osdmap=52} crc {osd_snap=2647620440,osdmap=1796494039}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=3057494235,osdmap=3712812351}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.250+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=52,osdmap=48} crc {osd_snap=71337734,osdmap=4156583260}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=523753842}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=62} crc {osdmap=2571077166}) 2024-09-10T16:27:54.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:27:54.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=62} crc {osdmap=523753842}) 2024-09-10T16:27:54.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:27:54.255+0000 7ff2dedb1640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=62} crc {osdmap=1821076633}) 2024-09-10T16:27:54.648 INFO:tasks.rados.rados.0.smithi165.stdout:3329: done (0 left) 2024-09-10T16:27:54.648 INFO:tasks.rados.rados.0.smithi165.stdout:3330: delete oid 50 current snap is 365 2024-09-10T16:27:54.651 INFO:tasks.rados.rados.0.smithi165.stdout:3330: done (0 left) 2024-09-10T16:27:54.651 INFO:tasks.rados.rados.0.smithi165.stdout:3331: copy_from oid 30 from oid 26 current snap is 365 2024-09-10T16:27:54.652 INFO:tasks.rados.rados.0.smithi165.stdout:3332: read oid 5 snap -1 2024-09-10T16:27:54.652 INFO:tasks.rados.rados.0.smithi165.stdout:3332: expect (ObjNum 1042 snap 321 seq_num 1042) 2024-09-10T16:27:54.652 INFO:tasks.rados.rados.0.smithi165.stdout:3333: delete oid 29 current snap is 365 2024-09-10T16:27:54.659 INFO:tasks.rados.rados.0.smithi165.stdout:3333: done (2 left) 2024-09-10T16:27:54.659 INFO:tasks.rados.rados.0.smithi165.stdout:3334: rollback oid 20 current snap is 365 2024-09-10T16:27:54.659 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 20 to 352 2024-09-10T16:27:54.659 INFO:tasks.rados.rados.0.smithi165.stdout:3335: snap_remove snap 358 2024-09-10T16:27:54.996 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:27:54.997 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.096+0000 7fc3b695d640 10 monclient: get_monmap_and_config 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.096+0000 7fc3b695d640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.096+0000 7fc3b695d640 10 monclient: monmap: 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:55.097555+0000 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:55.097555+0000 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.098 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:55.099 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 Processor -- start 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 -- start start 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient: init 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient: _add_conns ranks=[0,4,2,1,3] 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3b009c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3b009d160 0x7fc3b009f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3b009d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 0x7fc3b013c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3b013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b009a430 con 0x7fc3b009c180 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc3b00a24e0 con 0x7fc3b009d160 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.097+0000 7fc3b695d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b00a2bf0 con 0x7fc3b013a5c0 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b695d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b695d640 10 monclient(hunting): authenticate will time out at 2053.585013s 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3affff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc3b009c180 0x7fc3b009ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:58632/0 (socket says 172.21.15.164:58632) 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3affff640 1 -- 172.21.15.164:0/2648394037 learned_addr learned my addr 172.21.15.164:0/2648394037 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 --2- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 0x7fc3b013c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): get_auth_request con 0x7fc3b013a5c0 auth_method 0 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 1 --2- 172.21.15.164:0/2648394037 >> v2:172.21.15.165:3300/0 conn(0x7fc3b009d160 0x7fc3b009f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): get_auth_request con 0x7fc3b009d160 auth_method 0 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient(hunting): handle_auth_done global_id 244115 payload 274 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 -- 172.21.15.164:0/2648394037 >> v2:172.21.15.165:3300/0 conn(0x7fc3b009d160 msgr2=0x7fc3b009f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 --2- 172.21.15.164:0/2648394037 >> v2:172.21.15.165:3300/0 conn(0x7fc3b009d160 0x7fc3b009f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 -- 172.21.15.164:0/2648394037 >> v1:172.21.15.164:6789/0 conn(0x7fc3b009c180 legacy=0x7fc3b009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 monclient: found mon.noname-b 2024-09-10T16:27:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:27:55.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.098+0000 7fc3b4efa640 1 -- 172.21.15.164:0/2648394037 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b009c560 con 0x7fc3b013a5c0 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b4efa640 1 -- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 msgr2=0x7fc3b013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b4efa640 1 --2- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 0x7fc3b013c9b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009dc0 tx=0x7fc3a802a4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e7e29f3fff1b61d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 1 -- 172.21.15.164:0/2648394037 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8007950 con 0x7fc3b013a5c0 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:27:55.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: dump: 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b695d640 5 monclient: authenticate success, global_id 244115 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 1 -- 172.21.15.164:0/2648394037 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a8007af0 con 0x7fc3b013a5c0 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b695d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 1 -- 172.21.15.164:0/2648394037 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a8031400 con 0x7fc3b013a5c0 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b695d640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3b695d640 4 set_mon_vals no callback set 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.101+0000 7fc3aeffd640 10 monclient: dump: 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.112 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 10 monclient: shutdown 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 -- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 msgr2=0x7fc3b013c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 --2- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 0x7fc3b013c9b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3a8009dc0 tx=0x7fc3a802a4d0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 -- 172.21.15.164:0/2648394037 shutdown_connections 2024-09-10T16:27:55.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 --2- 172.21.15.164:0/2648394037 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b013a5c0 0x7fc3b013c9b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 --2- 172.21.15.164:0/2648394037 >> v2:172.21.15.165:3300/0 conn(0x7fc3b009d160 0x7fc3b009f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 -- 172.21.15.164:0/2648394037 >> 172.21.15.164:0/2648394037 conn(0x7fc3b00a3110 msgr2=0x7fc3b00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 -- 172.21.15.164:0/2648394037 shutdown_connections 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 1 -- 172.21.15.164:0/2648394037 wait complete. 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.102+0000 7fc3b695d640 10 monclient: monmap: 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.114 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 Processor -- start 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 -- start start 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient: init 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-c con 0x7fc3b009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-a con 0x7fc3b009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3b013a5c0 0x7fc3b0139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): picked mon.noname-b con 0x7fc3b013a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b0074f90 con 0x7fc3b009d160 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b0074e10 con 0x7fc3b013a5c0 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b009f690 con 0x7fc3b009c180 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3b695d640 10 monclient(hunting): authenticate will time out at 2053.590775s 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.116 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34844/0 (socket says 172.21.15.164:34844) 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 1 -- 172.21.15.164:0/3397205695 learned_addr learned my addr 172.21.15.164:0/3397205695 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37004/0 (socket says 172.21.15.164:37004) 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): get_auth_request con 0x7fc3b009c180 auth_method 0 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): get_auth_request con 0x7fc3b009d160 auth_method 0 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.117 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.103+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3b4efa640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3b013a5c0 0x7fc3b0139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3b4efa640 10 monclient(hunting): get_auth_request con 0x7fc3b013a5c0 auth_method 0 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3b4efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3b4efa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3b4efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 10 monclient(hunting): handle_auth_done global_id 224313 payload 824 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 -- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 msgr2=0x7fc3b00960a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 -- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3b013a5c0 msgr2=0x7fc3b0139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3b013a5c0 0x7fc3b0139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3af7fe640 1 -- 172.21.15.164:0/3397205695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b009d620 con 0x7fc3b009d160 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.104+0000 7fc3affff640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3af7fe640 1 -- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 msgr2=0x7fc3b01395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3af7fe640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400e820 tx=0x7fc3a400ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4017070 con 0x7fc3b009d160 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:55.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: dump: 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a400caf0 con 0x7fc3b009d160 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 5 monclient: authenticate success, global_id 224313 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 10 monclient: _renew_subs 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b0097000 con 0x7fc3b009d160 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4013470 con 0x7fc3b009d160 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3acff9640 10 monclient: dump: 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.121 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 10 monclient: _renew_subs 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b0099120 con 0x7fc3b009d160 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.106+0000 7fc3ae7fc640 4 set_mon_vals no callback set 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3b695d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3b695d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3b695d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b009ea40 con 0x7fc3b009d160 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc3a401a070 con 0x7fc3b009d160 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3acff9640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 0x7fc39c040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.107+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(743..743 src has 677..743) ==== 4783+0+0 (secure 0 0 0) 0x7fc3a4052cb0 con 0x7fc3b009d160 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.108+0000 7fc3affff640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 0x7fc39c040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.108+0000 7fc3affff640 10 monclient: get_auth_request con 0x7fc39c03e240 auth_method 0 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.110+0000 7fc3affff640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 0x7fc39c040700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0007d50 tx=0x7fc3a0007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.123 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.112+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc3a4023d00 con 0x7fc3b009d160 2024-09-10T16:27:55.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.112+0000 7fc3acff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.112+0000 7fc3acff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3b695d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3b695d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:55.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3b695d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3b0077260 con 0x7fc3b009d160 2024-09-10T16:27:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3acff9640 1 -- 172.21.15.164:0/3397205695 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fc3a40123d0 con 0x7fc3b009d160 2024-09-10T16:27:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3acff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:55.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.356+0000 7fc3acff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:55.358 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 msgr2=0x7fc39c040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 0x7fc39c040700 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0007d50 tx=0x7fc3a0007ca0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 10 monclient: shutdown 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 msgr2=0x7fc3b01395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc3a400e820 tx=0x7fc3a400ece0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 shutdown_connections 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc39c03e240 0x7fc39c040700 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3b013a5c0 0x7fc3b0139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3b009d160 0x7fc3b01395d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 --2- 172.21.15.164:0/3397205695 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3b009c180 0x7fc3b00960a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 >> 172.21.15.164:0/3397205695 conn(0x7fc3b00a3110 msgr2=0x7fc3b0099830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:55.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 shutdown_connections 2024-09-10T16:27:55.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.358+0000 7fc3b695d640 1 -- 172.21.15.164:0/3397205695 wait complete. 2024-09-10T16:27:55.370 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:27:55.370 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:27:55.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:27:55.370 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:27:55.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 10 monclient: get_monmap_and_config 2024-09-10T16:27:55.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 10 monclient: monmap: 2024-09-10T16:27:55.468 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.468 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:55.467952+0000 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:55.467952+0000 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:55.469 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 1 Processor -- start 2024-09-10T16:27:55.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 1 -- start start 2024-09-10T16:27:55.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.467+0000 7faaf4189640 10 monclient: init 2024-09-10T16:27:55.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 0x7faaec09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-c con 0x7faaec09d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faaec09ab50 0x7faaec09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-a con 0x7faaec09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faaec134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faaec0a0870 con 0x7faaec134580 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7faaec09d3e0 con 0x7faaec09ab50 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faaec09e790 con 0x7faaec09d930 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf4189640 10 monclient(hunting): authenticate will time out at 2053.955367s 2024-09-10T16:27:55.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faaec09ab50 0x7faaec09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faaec09ab50 0x7faaec09e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37016/0 (socket says 172.21.15.164:37016) 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 -- 172.21.15.164:0/3869265846 learned_addr learned my addr 172.21.15.164:0/3869265846 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): get_auth_request con 0x7faaec09ab50 auth_method 0 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient(hunting): handle_auth_done global_id 224316 payload 274 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 -- 172.21.15.164:0/3869265846 >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 msgr2=0x7faaec09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1f25640 1 --2- 172.21.15.164:0/3869265846 >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 0x7faaec09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 --2- 172.21.15.164:0/3869265846 >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 0x7faaec09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 -- 172.21.15.164:0/3869265846 >> v1:172.21.15.165:6789/0 conn(0x7faaec134580 legacy=0x7faaec134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 monclient: found mon.noname-a 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1724640 1 -- 172.21.15.164:0/3869265846 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec074c70 con 0x7faaec09ab50 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1f25640 10 monclient: get_auth_request con 0x7faaec09d930 auth_method 0 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.468+0000 7faaf1f25640 1 --2- 172.21.15.164:0/3869265846 >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 0x7faaec09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf1724640 1 -- 172.21.15.164:0/3869265846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09ab50 msgr2=0x7faaec09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf1724640 1 --2- 172.21.15.164:0/3869265846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09ab50 0x7faaec09e250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faaec06d500 tx=0x7faadc02a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81afaf546382fad9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 1 -- 172.21.15.164:0/3869265846 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc037070 con 0x7faaec09ab50 2024-09-10T16:27:55.477 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: dump: 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 1 -- 172.21.15.164:0/3869265846 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc032640 con 0x7faaec09ab50 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf4189640 5 monclient: authenticate success, global_id 224316 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf4189640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 1 -- 172.21.15.164:0/3869265846 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc032a10 con 0x7faaec09ab50 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf4189640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf4189640 4 set_mon_vals no callback set 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.471+0000 7faaf0f23640 10 monclient: dump: 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.479 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 10 monclient: shutdown 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 -- 172.21.15.164:0/3869265846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09ab50 msgr2=0x7faaec09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 --2- 172.21.15.164:0/3869265846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09ab50 0x7faaec09e250 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7faaec06d500 tx=0x7faadc02a0b0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 -- 172.21.15.164:0/3869265846 shutdown_connections 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 --2- 172.21.15.164:0/3869265846 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09ab50 0x7faaec09e250 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 --2- 172.21.15.164:0/3869265846 >> v2:172.21.15.165:3300/0 conn(0x7faaec09d930 0x7faaec09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 -- 172.21.15.164:0/3869265846 >> 172.21.15.164:0/3869265846 conn(0x7faaec0a3110 msgr2=0x7faaec0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 -- 172.21.15.164:0/3869265846 shutdown_connections 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 1 -- 172.21.15.164:0/3869265846 wait complete. 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.472+0000 7faaf4189640 10 monclient: monmap: 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 Processor -- start 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 -- start start 2024-09-10T16:27:55.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient: init 2024-09-10T16:27:55.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:55.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 0x7faaec1cb690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-c con 0x7faaec09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 0x7faaec1cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-a con 0x7faaec09d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec134580 0x7faaec1cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): picked mon.noname-b con 0x7faaec134580 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faaec09ef30 con 0x7faaec09d930 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faaec0a02b0 con 0x7faaec134580 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faaec0a2940 con 0x7faaec09ab50 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf4189640 10 monclient(hunting): authenticate will time out at 2053.960766s 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 0x7faaec1cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 0x7faaec1cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 0x7faaec1cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34850/0 (socket says 172.21.15.164:34850) 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 1 -- 172.21.15.164:0/2534514416 learned_addr learned my addr 172.21.15.164:0/2534514416 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): get_auth_request con 0x7faaec09d930 auth_method 0 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): get_auth_request con 0x7faaec09ab50 auth_method 0 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1724640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.473+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf2726640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec134580 0x7faaec1cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf2726640 10 monclient(hunting): get_auth_request con 0x7faaec134580 auth_method 0 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf2726640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf2726640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf2726640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 10 monclient(hunting): handle_auth_done global_id 224319 payload 824 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 msgr2=0x7faaec1cb690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 0x7faaec1cb690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec134580 msgr2=0x7faaec1cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faaec134580 0x7faaec1cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 monclient: found mon.noname-a 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1724640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faaec098040 con 0x7faaec09d930 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.474+0000 7faaf1f25640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faaec09ab50 0x7faaec1cb690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf1724640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 msgr2=0x7faaec1cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf1724640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 0x7faaec1cbbd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faadc031000 tx=0x7faadc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc041020 con 0x7faaec09d930 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: dump: 2024-09-10T16:27:55.493 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faadc03d930 con 0x7faaec09d930 2024-09-10T16:27:55.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 5 monclient: authenticate success, global_id 224319 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faadc037040 con 0x7faaec09d930 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 10 monclient: _renew_subs 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faaec099e80 con 0x7faaec09d930 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae3fff640 4 set_mon_vals no callback set 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faae27fc640 10 monclient: dump: 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.495 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 10 monclient: _renew_subs 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.476+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faaec0a53b0 con 0x7faaec09d930 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf4189640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf4189640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 0x7faaec070dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faaec09d620 con 0x7faaec0709e0 2024-09-10T16:27:55.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 0x7faaec070dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient: get_auth_request con 0x7faaec0709e0 auth_method 0 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faadc044020 con 0x7faaec09d930 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 10 monclient(hunting): handle_auth_done global_id 224322 payload 824 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faaf1f25640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaec0a1e40 con 0x7faaec0709e0 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faae27fc640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad003e240 0x7faad0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.477+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(743..743 src has 677..743) ==== 4783+0+0 (secure 0 0 0) 0x7faadc073330 con 0x7faaec09d930 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.478+0000 7faaf2726640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad003e240 0x7faad0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.478+0000 7faaf2726640 10 monclient: get_auth_request con 0x7faad003e240 auth_method 0 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faaf1f25640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 msgr2=0x7faaec070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faaf1f25640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 0x7faaec070dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faae800fd10 tx=0x7faae800d590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8013850 con 0x7faaec0709e0 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 10 monclient: dump: 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.498 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faae8013850 con 0x7faaec0709e0 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 msgr2=0x7faaec070dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 0x7faaec070dc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faae800fd10 tx=0x7faae800d590 comp rx=0 tx=0).stop 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 msgr2=0x7faaec070dc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faae27fc640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec0709e0 0x7faaec070dc0 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7faae800fd10 tx=0x7faae800d590 comp rx=0 tx=0).stop 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.480+0000 7faaf2726640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad003e240 0x7faad0040700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7faae4007fd0 tx=0x7faae4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf4189640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf4189640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 0x7faaec096660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf4189640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faaec06dac0 con 0x7faaec071300 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 0x7faaec096660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient: get_auth_request con 0x7faaec071300 auth_method 0 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 10 monclient(hunting): handle_auth_done global_id 224325 payload 824 2024-09-10T16:27:55.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.486+0000 7faaf1f25640 1 -- 172.21.15.164:0/2534514416 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faaec077260 con 0x7faaec071300 2024-09-10T16:27:55.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.488+0000 7faaf1f25640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 msgr2=0x7faaec096660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.488+0000 7faaf1f25640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 0x7faaec096660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae8008010 tx=0x7faae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faae8006240 con 0x7faaec071300 2024-09-10T16:27:55.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 10 monclient: dump: 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7faae8006240 con 0x7faaec071300 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 msgr2=0x7faaec096660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 0x7faaec096660 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7faae8008010 tx=0x7faae80046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 msgr2=0x7faaec096660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faae27fc640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec071300 0x7faaec096660 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:55.503 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 127320, 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:55.504 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:55.505 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:55.506 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:55.507 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:55.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:55.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:55.510 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.512 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:55.513 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.514 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:55.515 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:55.516 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:55.516 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:55.516 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad003e240 msgr2=0x7faad0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faad003e240 0x7faad0040700 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7faae4007fd0 tx=0x7faae4009040 comp rx=0 tx=0).stop 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 10 monclient: shutdown 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 msgr2=0x7faaec1cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 --2- 172.21.15.164:0/2534514416 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faaec09d930 0x7faaec1cbbd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7faadc031000 tx=0x7faadc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf2726640 1 -- 172.21.15.164:0/2534514416 reap_dead start 2024-09-10T16:27:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf2726640 1 -- 172.21.15.164:0/2534514416 reap_dead start 2024-09-10T16:27:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 shutdown_connections 2024-09-10T16:27:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 >> 172.21.15.164:0/2534514416 conn(0x7faaec0a3110 msgr2=0x7faaec0a4330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 shutdown_connections 2024-09-10T16:27:55.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.489+0000 7faaf4189640 1 -- 172.21.15.164:0/2534514416 wait complete. 2024-09-10T16:27:55.518 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:55.518 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:27:55.518 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.616+0000 7fb6b2662640 10 monclient: get_monmap_and_config 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.616+0000 7fb6b2662640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.616+0000 7fb6b2662640 10 monclient: monmap: 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:55.617397+0000 2024-09-10T16:27:55.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:55.617397+0000 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:55.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 Processor -- start 2024-09-10T16:27:55.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 -- start start 2024-09-10T16:27:55.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient: init 2024-09-10T16:27:55.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6ac09c730 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ac09cb10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6ac139e50 0x7fb6ac13c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-c con 0x7fb6ac139e50 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6ac0a00a0 con 0x7fb6ac09c730 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ac09d460 con 0x7fb6ac139e50 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac132350 con 0x7fb6ac09cb10 2024-09-10T16:27:55.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.617+0000 7fb6b2662640 10 monclient(hunting): authenticate will time out at 2054.104920s 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34864/0 (socket says 172.21.15.164:34864) 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 learned_addr learned my addr 172.21.15.164:0/2130373782 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb6ac09cb10 auth_method 0 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6b0bff640 1 --2- 172.21.15.164:0/2130373782 >> v2:172.21.15.165:3300/0 conn(0x7fb6ac139e50 0x7fb6ac13c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_done global_id 244124 payload 274 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> v2:172.21.15.165:3300/0 conn(0x7fb6ac139e50 msgr2=0x7fb6ac13c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> v2:172.21.15.165:3300/0 conn(0x7fb6ac139e50 0x7fb6ac13c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> v1:172.21.15.165:6789/0 conn(0x7fb6ac09c730 legacy=0x7fb6ac131c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 monclient: found mon.noname-b 2024-09-10T16:27:55.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac0a0310 con 0x7fb6ac09cb10 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6b0bff640 10 monclient: get_auth_request con 0x7fb6ac139e50 auth_method 0 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6b0bff640 1 --2- 172.21.15.164:0/2130373782 >> v2:172.21.15.165:3300/0 conn(0x7fb6ac139e50 0x7fb6ac13c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 msgr2=0x7fb6ac09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 msgr2=0x7fb6ac09cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7fb6a000bb10 tx=0x7fb6a002c750 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:55.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.618+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:3335: done (3 left) 2024-09-10T16:27:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:3336: delete oid 3 current snap is 365 2024-09-10T16:27:55.655 INFO:tasks.rados.rados.0.smithi165.stdout:3332: expect (ObjNum 1042 snap 321 seq_num 1042) 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3334: finishing rollback tid 0 to smithi16534841-20 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3334: finishing rollback tid 1 to smithi16534841-20 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1842 (ObjNum 481 snap 0 seq_num 1162108896) dirty dne 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3331: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3331: finishing copy_from to smithi16534841-30 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2026 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3331: done (3 left) 2024-09-10T16:27:55.812 INFO:tasks.rados.rados.0.smithi165.stdout:3332: done (2 left) 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3334: done (1 left) 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3336: done (0 left) 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3337: read oid 43 snap -1 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3337: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3338: write oid 25 current snap is 365 2024-09-10T16:27:55.813 INFO:tasks.rados.rados.0.smithi165.stdout:3338: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2024-09-10T16:27:55.818 INFO:tasks.rados.rados.0.smithi165.stdout:3338: writing smithi16534841-25 from 679313 to 1178560 tid 1 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient: get_auth_request con 0x7fb6ac09cb10 auth_method 0 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 0 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 msgr2=0x7fb6ac09cf20 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-10T16:27:55.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac09cb10 0x7fb6ac09cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient: _add_conns ranks=[0,3] 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb694003500 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 1 --2- 172.21.15.164:0/2130373782 >> v2:172.21.15.164:3300/0 conn(0x7fb694007490 0x7fb694009950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fb694007490 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb69400a260 con 0x7fb694003500 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb69400b460 con 0x7fb694007490 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6aaffd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> v2:172.21.15.164:3300/0 conn(0x7fb694007490 0x7fb694009950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb694007490 auth_method 0 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_done global_id 244124 payload 293 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> v1:172.21.15.164:6789/0 conn(0x7fb694003500 legacy=0x7fb694005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.819+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb69400b720 con 0x7fb694007490 2024-09-10T16:27:55.825 INFO:tasks.rados.rados.0.smithi165.stdout:3338: writing smithi16534841-25 from 1622042 to 2305107 tid 2 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb694007490 msgr2=0x7fb694009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb694007490 0x7fb694009950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0a2550 tx=0x7fb6a000b450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3866b5235c6243ea server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a003a070 con 0x7fb694007490 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: dump: 2024-09-10T16:27:55.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.832 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a0004230 con 0x7fb694007490 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 1 -- 172.21.15.164:0/2130373782 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a0009870 con 0x7fb694007490 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6b2662640 5 monclient: authenticate success, global_id 244124 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6b2662640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6b2662640 4 set_mon_vals no callback set 2024-09-10T16:27:55.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.822+0000 7fb6aaffd640 10 monclient: dump: 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 10 monclient: shutdown 2024-09-10T16:27:55.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 -- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb694007490 msgr2=0x7fb694009950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb694007490 0x7fb694009950 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac0a2550 tx=0x7fb6a000b450 comp rx=0 tx=0).stop 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b0bff640 1 -- 172.21.15.164:0/2130373782 reap_dead start 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 -- 172.21.15.164:0/2130373782 shutdown_connections 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 --2- 172.21.15.164:0/2130373782 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb694007490 0x7fb694009950 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 -- 172.21.15.164:0/2130373782 >> 172.21.15.164:0/2130373782 conn(0x7fb6ac0a2f30 msgr2=0x7fb6ac0a3340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 -- 172.21.15.164:0/2130373782 shutdown_connections 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 1 -- 172.21.15.164:0/2130373782 wait complete. 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 10 monclient: build_initial_monmap 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.823+0000 7fb6b2662640 10 monclient: monmap: 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:55.835 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 Processor -- start 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 -- start start 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient: init 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:55.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb694007490 0x7fb6ac12c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-c con 0x7fb694007490 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 0x7fb6ac12cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ac12ca80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6ac099030 0x7fb6ac0994c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ac099030 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac0a3e40 con 0x7fb6ac12ca80 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ac0a3cc0 con 0x7fb6ac099030 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac09d930 con 0x7fb694007490 2024-09-10T16:27:55.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.824+0000 7fb6b2662640 10 monclient(hunting): authenticate will time out at 2054.311928s 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 0x7fb6ac12cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 0x7fb6ac12cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37070/0 (socket says 172.21.15.164:37070) 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/3505343864 learned_addr learned my addr 172.21.15.164:0/3505343864 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request con 0x7fb6ac12ca80 auth_method 0 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb694007490 0x7fb6ac12c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): get_auth_request con 0x7fb694007490 auth_method 0 2024-09-10T16:27:55.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6ac099030 0x7fb6ac0994c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): get_auth_request con 0x7fb6ac099030 auth_method 0 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6b0bff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient(hunting): handle_auth_done global_id 224328 payload 824 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb694007490 msgr2=0x7fb6ac12c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb694007490 0x7fb6ac12c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6ac099030 msgr2=0x7fb6ac0994c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6ac099030 0x7fb6ac0994c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:55.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.825+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ac12ffe0 con 0x7fb6ac12ca80 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6ab7fe640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 msgr2=0x7fb6ac12cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6ab7fe640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 0x7fb6ac12cec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a003a040 con 0x7fb6ac12ca80 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: dump: 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.841 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6a0003e40 con 0x7fb6ac12ca80 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6a002cdd0 con 0x7fb6ac12ca80 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6a8ff9640 10 monclient: dump: 2024-09-10T16:27:55.842 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 5 monclient: authenticate success, global_id 224328 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 10 monclient: _renew_subs 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ac12f740 con 0x7fb6ac12ca80 2024-09-10T16:27:55.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6aa7fc640 4 set_mon_vals no callback set 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 10 monclient: _renew_subs 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.827+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ac12dd00 con 0x7fb6ac12ca80 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6b2662640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6b2662640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 0x7fb6ac071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac0a59c0 con 0x7fb6ac099a00 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb6a0036440 con 0x7fb6ac12ca80 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 0x7fb6ac071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient: get_auth_request con 0x7fb6ac099a00 auth_method 0 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.828+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6a8ff9640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb69803e290 0x7fb698040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7fb6a00769c0 con 0x7fb6ac12ca80 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6abfff640 10 monclient(hunting): handle_auth_done global_id 244130 payload 824 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6ac09cf20 con 0x7fb6ac099a00 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6b0bff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb69803e290 0x7fb698040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.829+0000 7fb6b0bff640 10 monclient: get_auth_request con 0x7fb69803e290 auth_method 0 2024-09-10T16:27:55.846 INFO:tasks.rados.rados.0.smithi165.stdout:3338: writing smithi16534841-25 from 2744778 to 3446335 tid 3 2024-09-10T16:27:55.847 INFO:tasks.rados.rados.0.smithi165.stdout:3339: delete oid 31 current snap is 365 2024-09-10T16:27:55.847 INFO:tasks.rados.rados.0.smithi165.stdout:3337: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:27:55.847 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2229823 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 msgr2=0x7fb6ac071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 0x7fb6ac071ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c00fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c0035f0 con 0x7fb6ac099a00 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 10 monclient: dump: 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.848 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6b0bff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb69803e290 0x7fb698040750 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac06cc60 tx=0x7fb6a4007540 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb69c018070 con 0x7fb6ac099a00 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 msgr2=0x7fb6ac071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 0x7fb6ac071ff0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c00fa30 comp rx=0 tx=0).stop 2024-09-10T16:27:55.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 msgr2=0x7fb6ac071ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.831+0000 7fb6a8ff9640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac099a00 0x7fb6ac071ff0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c00fa30 comp rx=0 tx=0).stop 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6b2662640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6b2662640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6b2662640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ac130f80 con 0x7fb6ac072530 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient: get_auth_request con 0x7fb6ac072530 auth_method 0 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:55.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:55.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 msgr2=0x7fb6ac072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:27:55.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 msgr2=0x7fb6ac072940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:27:55.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:27:55.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:55.838+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: finishing write tid 1 to smithi16534841-25 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: finishing write tid 2 to smithi16534841-25 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: finishing write tid 3 to smithi16534841-25 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: finishing write tid 4 to smithi16534841-25 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: finishing write tid 5 to smithi16534841-25 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: oid 25 updating version 0 to 1847 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: oid 25 version 1847 is already newer than 1846 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: oid 25 version 1847 is already newer than 1844 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: oid 25 version 1847 is already newer than 1845 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1847 (ObjNum 1159 snap 365 seq_num 1159) dirty exists 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3338: left oid 25 (ObjNum 1159 snap 365 seq_num 1159) 2024-09-10T16:27:55.922 INFO:tasks.rados.rados.0.smithi165.stdout:3337: done (2 left) 2024-09-10T16:27:55.923 INFO:tasks.rados.rados.0.smithi165.stdout:3338: done (1 left) 2024-09-10T16:27:55.923 INFO:tasks.rados.rados.0.smithi165.stdout:3339: done (0 left) 2024-09-10T16:27:55.923 INFO:tasks.rados.rados.0.smithi165.stdout:3340: write (excl) oid 4 current snap is 365 2024-09-10T16:27:55.923 INFO:tasks.rados.rados.0.smithi165.stdout:3340: seq_num 1161 ranges {712927=649745,2139006=686734} 2024-09-10T16:27:55.928 INFO:tasks.rados.rados.0.smithi165.stdout:3340: writing smithi16534841-4 from 712927 to 1362672 tid 1 2024-09-10T16:27:55.937 INFO:tasks.rados.rados.0.smithi165.stdout:3340: writing smithi16534841-4 from 2139006 to 2825740 tid 2 2024-09-10T16:27:55.937 INFO:tasks.rados.rados.0.smithi165.stdout:3341: write oid 9 current snap is 365 2024-09-10T16:27:55.937 INFO:tasks.rados.rados.0.smithi165.stdout:3340: finishing write tid 1 to smithi16534841-4 2024-09-10T16:27:55.937 INFO:tasks.rados.rados.0.smithi165.stdout:3341: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2024-09-10T16:27:55.944 INFO:tasks.rados.rados.0.smithi165.stdout:3341: writing smithi16534841-9 from 729734 to 1454728 tid 1 2024-09-10T16:27:55.951 INFO:tasks.rados.rados.0.smithi165.stdout:3341: writing smithi16534841-9 from 2197488 to 2627880 tid 2 2024-09-10T16:27:55.956 INFO:tasks.rados.rados.0.smithi165.stdout:3341: writing smithi16534841-9 from 3227047 to 3529734 tid 3 2024-09-10T16:27:55.956 INFO:tasks.rados.rados.0.smithi165.stdout:3342: read oid 44 snap 353 2024-09-10T16:27:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:3342: expect (ObjNum 1114 snap 341 seq_num 1114) 2024-09-10T16:27:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:3343: write oid 13 current snap is 365 2024-09-10T16:27:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:3340: finishing write tid 2 to smithi16534841-4 2024-09-10T16:27:55.957 INFO:tasks.rados.rados.0.smithi165.stdout:3343: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2024-09-10T16:27:55.962 INFO:tasks.rados.rados.0.smithi165.stdout:3343: writing smithi16534841-13 from 746541 to 1146784 tid 1 2024-09-10T16:27:55.969 INFO:tasks.rados.rados.0.smithi165.stdout:3343: writing smithi16534841-13 from 1739617 to 2430020 tid 2 2024-09-10T16:27:55.976 INFO:tasks.rados.rados.0.smithi165.stdout:3343: writing smithi16534841-13 from 3138117 to 3546541 tid 3 2024-09-10T16:27:55.976 INFO:tasks.rados.rados.0.smithi165.stdout:3344: copy_from oid 20 from oid 12 current snap is 365 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: finishing write tid 3 to smithi16534841-4 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: finishing write tid 4 to smithi16534841-4 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: oid 4 updating version 0 to 1450 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: oid 4 updating version 1450 to 1451 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: oid 4 version 1451 is already newer than 1449 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1451 (ObjNum 1161 snap 365 seq_num 1161) dirty exists 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3340: left oid 4 (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3341: finishing write tid 1 to smithi16534841-9 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3341: finishing write tid 2 to smithi16534841-9 2024-09-10T16:27:55.977 INFO:tasks.rados.rados.0.smithi165.stdout:3342: expect (ObjNum 1114 snap 341 seq_num 1114) 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient: get_auth_request con 0x7fb6ac072530 auth_method 0 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 10 monclient(hunting): handle_auth_done global_id 244136 payload 824 2024-09-10T16:27:56.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.040+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6ac09d620 con 0x7fb6ac072530 2024-09-10T16:27:56.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6abfff640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 msgr2=0x7fb6ac072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6abfff640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c001020 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb69c007050 con 0x7fb6ac072530 2024-09-10T16:27:56.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.042+0000 7fb6a8ff9640 10 monclient: dump: 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fb69c007050 con 0x7fb6ac072530 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 msgr2=0x7fb6ac072940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c001020 comp rx=0 tx=0).stop 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 msgr2=0x7fb6ac072940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6a8ff9640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ac072530 0x7fb6ac072940 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb69c00f570 tx=0x7fb69c001020 comp rx=0 tx=0).stop 2024-09-10T16:27:56.046 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12950, 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:56.047 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.048 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.049 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:56.050 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.051 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.052 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.053 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:56.054 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.055 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.056 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.057 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.058 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:56.059 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb69803e290 msgr2=0x7fb698040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb69803e290 0x7fb698040750 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fb6ac06cc60 tx=0x7fb6a4007540 comp rx=0 tx=0).stop 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 10 monclient: shutdown 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 msgr2=0x7fb6ac12cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 --2- 172.21.15.164:0/3505343864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ac12ca80 0x7fb6ac12cec0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fb6a0000c00 tx=0x7fb6a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b0bff640 1 -- 172.21.15.164:0/3505343864 reap_dead start 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b0bff640 1 -- 172.21.15.164:0/3505343864 reap_dead start 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 shutdown_connections 2024-09-10T16:27:56.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 >> 172.21.15.164:0/3505343864 conn(0x7fb6ac0a2f30 msgr2=0x7fb6ac0a4650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 shutdown_connections 2024-09-10T16:27:56.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.043+0000 7fb6b2662640 1 -- 172.21.15.164:0/3505343864 wait complete. 2024-09-10T16:27:56.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:56.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:27:56.062 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:27:56.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.158+0000 7fed16917640 10 monclient: get_monmap_and_config 2024-09-10T16:27:56.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.158+0000 7fed16917640 10 monclient: build_initial_monmap 2024-09-10T16:27:56.160 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.158+0000 7fed16917640 10 monclient: monmap: 2024-09-10T16:27:56.160 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:56.160 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:56.159819+0000 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:56.159819+0000 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:56.161 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.159+0000 7fed16917640 1 Processor -- start 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.159+0000 7fed16917640 1 -- start start 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.159+0000 7fed16917640 10 monclient: init 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed1009c180 0x7fed1009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): picked mon.noname-a con 0x7fed1009c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fed1009d160 0x7fed1009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): picked mon.noname-c con 0x7fed1009d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed1013b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.166 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): picked mon.noname-b con 0x7fed1009f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fed1009a430 con 0x7fed1009c180 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fed100a24e0 con 0x7fed1009d160 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed100a2bf0 con 0x7fed1009f4c0 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed16917640 10 monclient(hunting): authenticate will time out at 2054.647308s 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed1009c180 0x7fed1009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed1013b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fed1009c180 0x7fed1009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37080/0 (socket says 172.21.15.164:37080) 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 -- 172.21.15.164:0/3842941821 learned_addr learned my addr 172.21.15.164:0/3842941821 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:56.167 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): get_auth_request con 0x7fed1009c180 auth_method 0 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): get_auth_request con 0x7fed1009f4c0 auth_method 0 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 1 --2- 172.21.15.164:0/3842941821 >> v2:172.21.15.165:3300/0 conn(0x7fed1009d160 0x7fed1009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request con 0x7fed1009d160 auth_method 0 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.168 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient(hunting): handle_auth_done global_id 224331 payload 274 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 -- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 msgr2=0x7fed1013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed1013b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 -- 172.21.15.164:0/3842941821 >> v2:172.21.15.165:3300/0 conn(0x7fed1009d160 msgr2=0x7fed1009ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 --2- 172.21.15.164:0/3842941821 >> v2:172.21.15.165:3300/0 conn(0x7fed1009d160 0x7fed1009ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.169 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 monclient: found mon.noname-a 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed0ffff640 1 -- 172.21.15.164:0/3842941821 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1009c560 con 0x7fed1009c180 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.160+0000 7fed14eb4640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed1013b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0ffff640 1 -- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009c180 msgr2=0x7fed1009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0ffff640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009c180 0x7fed1009ea40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002a0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51cb0dc98cc1733e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 1 -- 172.21.15.164:0/3842941821 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00037070 con 0x7fed1009c180 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: dump: 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: _finish_auth 0 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 1 -- 172.21.15.164:0/3842941821 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed00032640 con 0x7fed1009c180 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed16917640 5 monclient: authenticate success, global_id 224331 2024-09-10T16:27:56.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed16917640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed16917640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 1 -- 172.21.15.164:0/3842941821 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00032a10 con 0x7fed1009c180 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed16917640 4 set_mon_vals no callback set 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.163+0000 7fed0effd640 10 monclient: dump: 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 10 monclient: shutdown 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 -- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009c180 msgr2=0x7fed1009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009c180 0x7fed1009ea40 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002a0f0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 -- 172.21.15.164:0/3842941821 shutdown_connections 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed1013b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 --2- 172.21.15.164:0/3842941821 >> v2:172.21.15.165:3300/0 conn(0x7fed1009d160 0x7fed1009ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 --2- 172.21.15.164:0/3842941821 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009c180 0x7fed1009ea40 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 -- 172.21.15.164:0/3842941821 >> 172.21.15.164:0/3842941821 conn(0x7fed100a3110 msgr2=0x7fed100a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:56.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 -- 172.21.15.164:0/3842941821 shutdown_connections 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 1 -- 172.21.15.164:0/3842941821 wait complete. 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 10 monclient: build_initial_monmap 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.164+0000 7fed16917640 10 monclient: monmap: 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 Processor -- start 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 -- start start 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient: init 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed1009c180 0x7fed101caed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): picked mon.noname-b con 0x7fed1009c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:56.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 0x7fed10136ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): picked mon.noname-a con 0x7fed1009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed10137010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): picked mon.noname-c con 0x7fed1009f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fed10074e10 con 0x7fed1009d160 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed1009f8d0 con 0x7fed1009c180 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fed10074f90 con 0x7fed1009f4c0 2024-09-10T16:27:56.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed16917640 10 monclient(hunting): authenticate will time out at 2054.652797s 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 0x7fed10136ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed10137010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 0x7fed10136ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37096/0 (socket says 172.21.15.164:37096) 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 1 -- 172.21.15.164:0/1693570314 learned_addr learned my addr 172.21.15.164:0/1693570314 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request con 0x7fed1009d160 auth_method 0 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): get_auth_request con 0x7fed1009f4c0 auth_method 0 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0ffff640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed1009c180 0x7fed101caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0ffff640 10 monclient(hunting): get_auth_request con 0x7fed1009c180 auth_method 0 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.165+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed14eb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 10 monclient(hunting): handle_auth_done global_id 224334 payload 824 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 msgr2=0x7fed10137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed10137010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed1009c180 msgr2=0x7fed101caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed1009c180 0x7fed101caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed0f7fe640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1012d260 con 0x7fed1009d160 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.166+0000 7fed14eb4640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fed1009f4c0 0x7fed10137010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0f7fe640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 msgr2=0x7fed10136ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0f7fe640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 0x7fed10136ad0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed0400e980 tx=0x7fed0400ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04017070 con 0x7fed1009d160 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: dump: 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.186 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed0400cb50 con 0x7fed1009d160 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 5 monclient: authenticate success, global_id 224334 2024-09-10T16:27:56.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 10 monclient: _renew_subs 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1012d540 con 0x7fed1009d160 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed04013470 con 0x7fed1009d160 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0cff9640 10 monclient: dump: 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 10 monclient: _renew_subs 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed1012cca0 con 0x7fed1009d160 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.168+0000 7fed0e7fc640 4 set_mon_vals no callback set 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed16917640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed16917640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 0x7fed10072ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed100a57c0 con 0x7fed10072af0 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fed0401a070 con 0x7fed1009d160 2024-09-10T16:27:56.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 0x7fed10072ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient: get_auth_request con 0x7fed10072af0 auth_method 0 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0cff9640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecfc03e210 0x7fecfc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7fed04052cf0 con 0x7fed1009d160 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 10 monclient(hunting): handle_auth_done global_id 234208 payload 824 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed0ffff640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fed1009d930 con 0x7fed10072af0 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed14eb4640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecfc03e210 0x7fecfc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.169+0000 7fed14eb4640 10 monclient: get_auth_request con 0x7fecfc03e210 auth_method 0 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0ffff640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 msgr2=0x7fed10072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0ffff640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 0x7fed10072ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed00034000 tx=0x7fed000046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00037040 con 0x7fed10072af0 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed14eb4640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecfc03e210 0x7fecfc0406d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fed1009bec0 tx=0x7fed0800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 10 monclient: dump: 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.191 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fed00009490 con 0x7fed10072af0 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 msgr2=0x7fed10072ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 0x7fed10072ed0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed00034000 tx=0x7fed000046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 msgr2=0x7fed10072ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.172+0000 7fed0cff9640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10072af0 0x7fed10072ed0 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fed00034000 tx=0x7fed000046c0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.178+0000 7fed16917640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.178+0000 7fed16917640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 0x7fed1006e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.178+0000 7fed16917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.178+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fed101cb410 con 0x7fed10073410 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 0x7fed1006e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient: get_auth_request con 0x7fed10073410 auth_method 0 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 10 monclient(hunting): handle_auth_done global_id 234211 payload 824 2024-09-10T16:27:56.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.179+0000 7fed0ffff640 1 -- 172.21.15.164:0/1693570314 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fed10077260 con 0x7fed10073410 2024-09-10T16:27:56.195 INFO:tasks.rados.rados.0.smithi165.stdout:3341: finishing write tid 3 to smithi16534841-9 2024-09-10T16:27:56.195 INFO:tasks.rados.rados.0.smithi165.stdout:3341: finishing write tid 4 to smithi16534841-9 2024-09-10T16:27:56.195 INFO:tasks.rados.rados.0.smithi165.stdout:3341: finishing write tid 5 to smithi16534841-9 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: oid 9 updating version 0 to 1837 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: oid 9 version 1837 is already newer than 1835 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: oid 9 version 1837 is already newer than 1834 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: oid 9 version 1837 is already newer than 1836 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1837 (ObjNum 1162 snap 365 seq_num 1162) dirty exists 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: left oid 9 (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3343: finishing write tid 1 to smithi16534841-13 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3340: done (4 left) 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3341: done (3 left) 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3342: done (2 left) 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3345: write oid 3 current snap is 365 2024-09-10T16:27:56.196 INFO:tasks.rados.rados.0.smithi165.stdout:3343: finishing write tid 2 to smithi16534841-13 2024-09-10T16:27:56.197 INFO:tasks.rados.rados.0.smithi165.stdout:3345: seq_num 1164 ranges {763348=591845,1914452=434061,2881893=674270} 2024-09-10T16:27:56.197 INFO:tasks.rados.rados.0.smithi165.stdout:3345: writing smithi16534841-3 from 763348 to 1355193 tid 1 2024-09-10T16:27:56.197 INFO:tasks.rados.rados.0.smithi165.stdout:3345: writing smithi16534841-3 from 1914452 to 2348513 tid 2 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0ffff640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 msgr2=0x7fed1006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0ffff640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 0x7fed1006e7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed0002a130 tx=0x7fed00009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00037020 con 0x7fed10073410 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.181+0000 7fed0cff9640 10 monclient: dump: 2024-09-10T16:27:56.197 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fed00004230 con 0x7fed10073410 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 msgr2=0x7fed1006e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 0x7fed1006e7b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed0002a130 tx=0x7fed00009240 comp rx=0 tx=0).stop 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 msgr2=0x7fed1006e7b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed0cff9640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fed10073410 0x7fed1006e7b0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fed0002a130 tx=0x7fed00009240 comp rx=0 tx=0).stop 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 208, 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 76465, 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:27:56.199 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.200 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.201 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.202 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:27:56.203 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:27:56.204 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.205 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.206 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.207 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:27:56.208 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:27:56.209 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:27:56.211 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:27:56.212 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecfc03e210 msgr2=0x7fecfc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fecfc03e210 0x7fecfc0406d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fed1009bec0 tx=0x7fed0800b040 comp rx=0 tx=0).stop 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 10 monclient: shutdown 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 msgr2=0x7fed10136ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 --2- 172.21.15.164:0/1693570314 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fed1009d160 0x7fed10136ad0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fed0400e980 tx=0x7fed0400ed80 comp rx=0 tx=0).stop 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed14eb4640 1 -- 172.21.15.164:0/1693570314 reap_dead start 2024-09-10T16:27:56.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed14eb4640 1 -- 172.21.15.164:0/1693570314 reap_dead start 2024-09-10T16:27:56.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 shutdown_connections 2024-09-10T16:27:56.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 >> 172.21.15.164:0/1693570314 conn(0x7fed100a3110 msgr2=0x7fed10139670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:56.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 shutdown_connections 2024-09-10T16:27:56.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.182+0000 7fed16917640 1 -- 172.21.15.164:0/1693570314 wait complete. 2024-09-10T16:27:56.215 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:27:56.215 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:27:56.215 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:27:56.216 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:27:56.216 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:27:56.216 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:27:56.216 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:27:56.217 INFO:tasks.rados.rados.0.smithi165.stdout:3345: writing smithi16534841-3 from 2881893 to 3556163 tid 3 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3346: write oid 49 current snap is 365 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: finishing write tid 3 to smithi16534841-13 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: finishing write tid 4 to smithi16534841-13 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: finishing write tid 5 to smithi16534841-13 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: oid 13 updating version 0 to 1466 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: oid 13 updating version 1466 to 1467 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: oid 13 version 1467 is already newer than 1465 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: oid 13 updating version 1467 to 1468 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1468 (ObjNum 1163 snap 365 seq_num 1163) dirty exists 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3343: left oid 13 (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3344: finishing copy_from racing read to smithi16534841-20 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3344: finishing copy_from to smithi16534841-20 2024-09-10T16:27:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1644 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3346: seq_num 1165 ranges {780155=667094,1972934=694072,3309316=270839} 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3346: writing smithi16534841-49 from 780155 to 1447249 tid 1 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3346: writing smithi16534841-49 from 1972934 to 2667006 tid 2 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3346: writing smithi16534841-49 from 3309316 to 3580155 tid 3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3343: done (3 left) 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3344: done (2 left) 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3347: snap_create 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: finishing write tid 1 to smithi16534841-3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: finishing write tid 2 to smithi16534841-3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: finishing write tid 3 to smithi16534841-3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: finishing write tid 4 to smithi16534841-3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: finishing write tid 5 to smithi16534841-3 2024-09-10T16:27:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3345: oid 3 updating version 0 to 1851 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3345: oid 3 version 1851 is already newer than 1848 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3345: oid 3 version 1851 is already newer than 1849 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3345: oid 3 version 1851 is already newer than 1850 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3345: left oid 3 (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3346: finishing write tid 1 to smithi16534841-49 2024-09-10T16:27:56.220 INFO:tasks.rados.rados.0.smithi165.stdout:3346: finishing write tid 2 to smithi16534841-49 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: finishing write tid 3 to smithi16534841-49 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: finishing write tid 4 to smithi16534841-49 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: finishing write tid 5 to smithi16534841-49 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: oid 49 updating version 0 to 1833 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: oid 49 updating version 1833 to 1834 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: oid 49 version 1834 is already newer than 1832 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: oid 49 version 1834 is already newer than 1831 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1834 (ObjNum 1165 snap 365 seq_num 1165) dirty exists 2024-09-10T16:27:56.221 INFO:tasks.rados.rados.0.smithi165.stdout:3346: left oid 49 (ObjNum 1165 snap 365 seq_num 1165) 2024-09-10T16:27:56.229 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:27:56.229 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:27:56.229 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.365+0000 7fe7a65e8640 10 monclient: get_monmap_and_config 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.365+0000 7fe7a65e8640 10 monclient: build_initial_monmap 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.365+0000 7fe7a65e8640 10 monclient: monmap: 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:56.366532+0000 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:56.366532+0000 2024-09-10T16:27:56.367 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:56.368 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 Processor -- start 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 -- start start 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient: init 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient: _add_conns ranks=[1,4,0,2,3] 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe7a0137bb0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe7a01386a0 0x7fe7a00a2e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a01386a0 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe7a013d140 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a0073820 con 0x7fe7a013d140 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7a00a1c90 con 0x7fe7a0137bb0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fe7a00a1f70 con 0x7fe7a01386a0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.366+0000 7fe7a65e8640 10 monclient(hunting): authenticate will time out at 2054.853990s 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe7a4b85640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fe7a013d140 legacy=0x7fe7a013f530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe7a01386a0 0x7fe7a00a2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fe7a01386a0 0x7fe7a00a2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:35802/0 (socket says 172.21.15.164:35802) 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 1 -- 172.21.15.164:0/1990799523 learned_addr learned my addr 172.21.15.164:0/1990799523 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79ffff640 1 --1- 172.21.15.164:0/1990799523 >> v1:172.21.15.165:6789/0 conn(0x7fe7a0137bb0 0x7fe7a0137f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:37620/0 (socket says 172.21.15.164:37620) 2024-09-10T16:27:56.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): get_auth_request con 0x7fe7a01386a0 auth_method 0 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.367+0000 7fe79f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79ffff640 1 -- 172.21.15.164:0/1990799523 >> v1:172.21.15.165:6789/0 conn(0x7fe7a0137bb0 legacy=0x7fe7a0137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 10 monclient(hunting): handle_auth_done global_id 234214 payload 274 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 1 -- 172.21.15.164:0/1990799523 >> v1:172.21.15.165:6789/0 conn(0x7fe7a0137bb0 legacy=0x7fe7a0137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 1 -- 172.21.15.164:0/1990799523 >> v1:172.21.15.164:6789/0 conn(0x7fe7a013d140 legacy=0x7fe7a013f530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:27:56.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.369+0000 7fe79f7fe640 1 -- 172.21.15.164:0/1990799523 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a00a2250 con 0x7fe7a01386a0 2024-09-10T16:27:56.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.372+0000 7fe79f7fe640 1 -- 172.21.15.164:0/1990799523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 msgr2=0x7fe7a00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.372+0000 7fe79f7fe640 1 --2- 172.21.15.164:0/1990799523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a00a2e80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00a7ed0 tx=0x7fe79402a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fffbc648d54284d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 1 -- 172.21.15.164:0/1990799523 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe794037070 con 0x7fe7a01386a0 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: dump: 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.378 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: _finish_auth 0 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 1 -- 172.21.15.164:0/1990799523 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe794032640 con 0x7fe7a01386a0 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 5 monclient: authenticate success, global_id 234214 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 4 set_mon_vals no callback set 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 1 -- 172.21.15.164:0/1990799523 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7940329f0 con 0x7fe7a01386a0 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe79effd640 10 monclient: dump: 2024-09-10T16:27:56.379 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 10 monclient: shutdown 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 -- 172.21.15.164:0/1990799523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 msgr2=0x7fe7a00a2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.380 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/1990799523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a00a2e80 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00a7ed0 tx=0x7fe79402a1e0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 -- 172.21.15.164:0/1990799523 shutdown_connections 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/1990799523 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a00a2e80 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 -- 172.21.15.164:0/1990799523 >> 172.21.15.164:0/1990799523 conn(0x7fe7a00a7320 msgr2=0x7fe7a00a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 -- 172.21.15.164:0/1990799523 shutdown_connections 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.373+0000 7fe7a65e8640 1 -- 172.21.15.164:0/1990799523 wait complete. 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.374+0000 7fe7a65e8640 10 monclient: build_initial_monmap 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.374+0000 7fe7a65e8640 10 monclient: monmap: 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.381 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.374+0000 7fe7a65e8640 1 Processor -- start 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.374+0000 7fe7a65e8640 1 -- start start 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.374+0000 7fe7a65e8640 10 monclient: init 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-c con 0x7fe7a0137bb0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a0134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-b con 0x7fe7a01386a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:56.382 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 0x7fe7a0134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): picked mon.noname-a con 0x7fe7a013d140 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0077060 con 0x7fe7a013d140 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fe7a0076ee0 con 0x7fe7a01386a0 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fe7a0140c50 con 0x7fe7a0137bb0 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a65e8640 10 monclient(hunting): authenticate will time out at 2054.862210s 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a4b85640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 msgr2=0x7fe7a0134f40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe7a4b85640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 0x7fe7a0134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.383 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34948/0 (socket says 172.21.15.164:34948) 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 -- 172.21.15.164:0/3246721531 learned_addr learned my addr 172.21.15.164:0/3246721531 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): get_auth_request con 0x7fe7a0137bb0 auth_method 0 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79f7fe640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a0134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79f7fe640 10 monclient(hunting): get_auth_request con 0x7fe7a01386a0 auth_method 0 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:56.384 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient(hunting): handle_auth_done global_id 244145 payload 824 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 -- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 msgr2=0x7fe7a0134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a0134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 -- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 msgr2=0x7fe7a0134f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 0x7fe7a0134f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 monclient: found mon.noname-c 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:27:56.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.375+0000 7fe79ffff640 1 -- 172.21.15.164:0/3246721531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7a013bc30 con 0x7fe7a0137bb0 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.377+0000 7fe79ffff640 1 -- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 msgr2=0x7fe7a01cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.377+0000 7fe79ffff640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c500 tx=0x7fe79000c9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7900047e0 con 0x7fe7a0137bb0 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: dump: 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.388 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: _finish_auth 0 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe790004980 con 0x7fe7a0137bb0 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 5 monclient: authenticate success, global_id 244145 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 10 monclient: _renew_subs 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:56.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7a01cd750 con 0x7fe7a0137bb0 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe790004d50 con 0x7fe7a0137bb0 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79cff9640 10 monclient: dump: 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:56.390 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 10 monclient: _renew_subs 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7a011dde0 con 0x7fe7a0137bb0 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.378+0000 7fe79e7fc640 4 set_mon_vals no callback set 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe7a65e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe7a65e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe7a65e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7a00a2f00 con 0x7fe7a0137bb0 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fe79001a020 con 0x7fe7a0137bb0 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe79cff9640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 0x7fe78c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:56.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7fe790052550 con 0x7fe7a0137bb0 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe79f7fe640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 0x7fe78c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.379+0000 7fe79f7fe640 10 monclient: get_auth_request con 0x7fe78c03e1c0 auth_method 0 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.382+0000 7fe79f7fe640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 0x7fe78c040680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00a29c0 tx=0x7fe79402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.383+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fe790050f50 con 0x7fe7a0137bb0 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.383+0000 7fe79cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:56.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.383+0000 7fe79cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:27:56.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.624+0000 7fe7a65e8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:27:56.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.624+0000 7fe7a65e8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.624+0000 7fe7a65e8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.624+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe7a006d1c0 con 0x7fe7a0137bb0 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.625+0000 7fe79cff9640 1 -- 172.21.15.164:0/3246721531 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fe790052230 con 0x7fe7a0137bb0 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.625+0000 7fe79cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.625+0000 7fe79cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:27:56.627 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:27:56.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 msgr2=0x7fe78c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 0x7fe78c040680 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fe7a00a29c0 tx=0x7fe79402e000 comp rx=0 tx=0).stop 2024-09-10T16:27:56.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 10 monclient: shutdown 2024-09-10T16:27:56.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:56.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 msgr2=0x7fe7a01cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe79000c500 tx=0x7fe79000c9c0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 shutdown_connections 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fe78c03e1c0 0x7fe78c040680 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fe7a013d140 0x7fe7a0134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fe7a01386a0 0x7fe7a0134a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 --2- 172.21.15.164:0/3246721531 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fe7a0137bb0 0x7fe7a01cd210 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 >> 172.21.15.164:0/3246721531 conn(0x7fe7a00a7320 msgr2=0x7fe7a0136b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 shutdown_connections 2024-09-10T16:27:56.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:56.627+0000 7fe7a65e8640 1 -- 172.21.15.164:0/3246721531 wait complete. 2024-09-10T16:27:59.640 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.748+0000 7f0e5b59e640 10 monclient: get_monmap_and_config 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.748+0000 7f0e5b59e640 10 monclient: build_initial_monmap 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.748+0000 7f0e5b59e640 10 monclient: monmap: 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:27:59.749785+0000 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:27:59.749785+0000 2024-09-10T16:27:59.750 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:27:59.751 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.749+0000 7f0e5b59e640 1 Processor -- start 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.749+0000 7f0e5b59e640 1 -- start start 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.749+0000 7f0e5b59e640 10 monclient: init 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0e5c1354c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:27:59.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0e5c0a5ff0 addr v1:172.21.15.165:6789/0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0e5c13e710 0x7f0e5c140b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c13e710 addr v2:172.21.15.165:3300/0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e5c06e180 con 0x7f0e5c1354c0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e5c135090 con 0x7f0e5c0a5ff0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0e5c0a63d0 con 0x7f0e5c13e710 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5b59e640 10 monclient(hunting): authenticate will time out at 2058.237246s 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5a59c640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f0e5c1354c0 legacy=0x7f0e5c09d450 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0e5c13e710 0x7f0e5c140b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0e5c13e710 0x7f0e5c140b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:35826/0 (socket says 172.21.15.164:35826) 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 1 -- 172.21.15.164:0/2806015197 learned_addr learned my addr 172.21.15.164:0/2806015197 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:59.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): get_auth_request con 0x7f0e5c13e710 auth_method 0 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:59.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.750+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e59d9b640 1 -- 172.21.15.164:0/2806015197 >> v1:172.21.15.165:6789/0 conn(0x7f0e5c0a5ff0 legacy=0x7f0e5c13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_done global_id 234220 payload 274 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e5ad9d640 1 -- 172.21.15.164:0/2806015197 >> v1:172.21.15.165:6789/0 conn(0x7f0e5c0a5ff0 legacy=0x7f0e5c13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e5ad9d640 1 -- 172.21.15.164:0/2806015197 >> v1:172.21.15.164:6789/0 conn(0x7f0e5c1354c0 legacy=0x7f0e5c09d450 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e5ad9d640 10 monclient: _finish_hunting 0 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.752+0000 7f0e5ad9d640 1 monclient: found mon.noname-c 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.753+0000 7f0e5ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.753+0000 7f0e5ad9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:27:59.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.753+0000 7f0e5ad9d640 1 -- 172.21.15.164:0/2806015197 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c06e460 con 0x7f0e5c13e710 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.755+0000 7f0e5ad9d640 1 -- 172.21.15.164:0/2806015197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 msgr2=0x7f0e5c140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.755+0000 7f0e5ad9d640 1 --2- 172.21.15.164:0/2806015197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c140b00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009ed0 tx=0x7f0e5402a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c14e2892df5e089e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 1 -- 172.21.15.164:0/2806015197 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54036070 con 0x7f0e5c13e710 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: dump: 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:59.758 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: _finish_auth 0 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: _check_auth_tickets 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 1 -- 172.21.15.164:0/2806015197 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e54007800 con 0x7f0e5c13e710 2024-09-10T16:27:59.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 5 monclient: authenticate success, global_id 234220 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 10 monclient: get_monmap_and_config success 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 1 -- 172.21.15.164:0/2806015197 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e54007bd0 con 0x7f0e5c13e710 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 4 set_mon_vals no callback set 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5959a640 10 monclient: dump: 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 10 monclient: shutdown 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 -- 172.21.15.164:0/2806015197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 msgr2=0x7f0e5c140b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/2806015197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c140b00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0e54009ed0 tx=0x7f0e5402a6b0 comp rx=0 tx=0).stop 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 -- 172.21.15.164:0/2806015197 shutdown_connections 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/2806015197 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c140b00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 -- 172.21.15.164:0/2806015197 >> 172.21.15.164:0/2806015197 conn(0x7f0e5c0a70e0 msgr2=0x7f0e5c0a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:27:59.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 -- 172.21.15.164:0/2806015197 shutdown_connections 2024-09-10T16:27:59.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.756+0000 7f0e5b59e640 1 -- 172.21.15.164:0/2806015197 wait complete. 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.757+0000 7f0e5b59e640 10 monclient: build_initial_monmap 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.757+0000 7f0e5b59e640 10 monclient: monmap: 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:27:59.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.757+0000 7f0e5b59e640 1 Processor -- start 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.757+0000 7f0e5b59e640 1 -- start start 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.757+0000 7f0e5b59e640 10 monclient: init 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient: _reopen_session rank -1 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 0x7f0e5c137dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-a con 0x7f0e5c0a5ff0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-c con 0x7f0e5c1354c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c0a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): picked mon.noname-b con 0x7f0e5c13e710 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:27:59.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c134d80 con 0x7f0e5c0a5ff0 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0e5c06e7d0 con 0x7f0e5c13e710 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0e5c09dec0 con 0x7f0e5c1354c0 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): _renew_subs 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5b59e640 10 monclient(hunting): authenticate will time out at 2058.245413s 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5a59c640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 msgr2=0x7f0e5c137dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5a59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 0x7f0e5c137dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:59.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34952/0 (socket says 172.21.15.164:34952) 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 -- 172.21.15.164:0/1030175190 learned_addr learned my addr 172.21.15.164:0/1030175190 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): get_auth_request con 0x7f0e5c1354c0 auth_method 0 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c0a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): get_auth_request con 0x7f0e5c13e710 auth_method 0 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:27:59.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e5ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient(hunting): handle_auth_done global_id 244148 payload 824 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 -- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 msgr2=0x7f0e5c0a2330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c0a2330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 -- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 msgr2=0x7f0e5c137dd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 0x7f0e5c137dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 monclient: found mon.noname-c 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:27:59.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.758+0000 7f0e59d9b640 1 -- 172.21.15.164:0/1030175190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5c138dc0 con 0x7f0e5c1354c0 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e59d9b640 1 -- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 msgr2=0x7f0e5c0a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e59d9b640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c3a0 tx=0x7f0e4400c860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44017070 con 0x7f0e5c1354c0 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: dump: 2024-09-10T16:27:59.773 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: _finish_auth 0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e440057c0 con 0x7f0e5c1354c0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 5 monclient: authenticate success, global_id 244148 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e44005bb0 con 0x7f0e5c1354c0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 10 monclient: _renew_subs 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e5c0a4090 con 0x7f0e5c1354c0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e52ffd640 10 monclient: dump: 2024-09-10T16:27:59.775 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e53fff640 4 set_mon_vals no callback set 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 10 monclient: _renew_subs 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:59.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.761+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e5c0a3410 con 0x7f0e5c1354c0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0e4401a070 con 0x7f0e5c1354c0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5b59e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5b59e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e5c051000 con 0x7f0e5c1354c0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e52ffd640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 0x7f0e3c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7f0e44052040 con 0x7f0e5c1354c0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5a59c640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 0x7f0e3c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.762+0000 7f0e5a59c640 10 monclient: get_auth_request con 0x7f0e3c03e1c0 auth_method 0 2024-09-10T16:27:59.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.765+0000 7f0e5a59c640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 0x7f0e3c040680 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c006fd0 tx=0x7f0e4c006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:27:59.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.767+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0e5c051000 con 0x7f0e5c1354c0 2024-09-10T16:27:59.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.767+0000 7f0e52ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:27:59.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:27:59.767+0000 7f0e52ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:00.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e5b59e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:00.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e5b59e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e5b59e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0e5c0a47c0 con 0x7f0e5c1354c0 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e52ffd640 1 -- 172.21.15.164:0/1030175190 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f0e5c0a47c0 con 0x7f0e5c1354c0 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e52ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.009+0000 7f0e52ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:00.011 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 msgr2=0x7f0e3c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 0x7f0e3c040680 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c006fd0 tx=0x7f0e4c006d80 comp rx=0 tx=0).stop 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 10 monclient: shutdown 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 msgr2=0x7f0e5c0a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0e4400c3a0 tx=0x7f0e4400c860 comp rx=0 tx=0).stop 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 shutdown_connections 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0e3c03e1c0 0x7f0e3c040680 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:00.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0e5c13e710 0x7f0e5c0a2330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:00.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0e5c1354c0 0x7f0e5c0a1df0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:00.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 --2- 172.21.15.164:0/1030175190 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0e5c0a5ff0 0x7f0e5c137dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:00.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 >> 172.21.15.164:0/1030175190 conn(0x7f0e5c0a70e0 msgr2=0x7f0e5c0a04f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:00.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 shutdown_connections 2024-09-10T16:28:00.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:00.011+0000 7f0e5b59e640 1 -- 172.21.15.164:0/1030175190 wait complete. 2024-09-10T16:28:03.025 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.132+0000 7fb36daed640 10 monclient: get_monmap_and_config 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.132+0000 7fb36daed640 10 monclient: build_initial_monmap 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.132+0000 7fb36daed640 10 monclient: monmap: 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:03.133541+0000 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:03.133541+0000 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:03.134 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:03.135 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:03.135 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:03.135 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 1 Processor -- start 2024-09-10T16:28:03.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 1 -- start start 2024-09-10T16:28:03.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient: init 2024-09-10T16:28:03.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:03.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-b con 0x7fb36809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb36809d190 0x7fb36809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-c con 0x7fb36809d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.133+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb36809f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb36809fbc0 con 0x7fb36809f4c0 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb3680a2a20 con 0x7fb36809d190 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb36809c6f0 con 0x7fb36809c310 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb36daed640 10 monclient(hunting): authenticate will time out at 2061.621038s 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb367fff640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fb36809f4c0 legacy=0x7fb36813b5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:03.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34964/0 (socket says 172.21.15.164:34964) 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 -- 172.21.15.164:0/1501270933 learned_addr learned my addr 172.21.15.164:0/1501270933 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): get_auth_request con 0x7fb36809c310 auth_method 0 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb366ffd640 1 --2- 172.21.15.164:0/1501270933 >> v2:172.21.15.165:3300/0 conn(0x7fb36809d190 0x7fb36809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb366ffd640 10 monclient(hunting): get_auth_request con 0x7fb36809d190 auth_method 0 2024-09-10T16:28:03.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient(hunting): handle_auth_done global_id 244151 payload 274 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 -- 172.21.15.164:0/1501270933 >> v2:172.21.15.165:3300/0 conn(0x7fb36809d190 msgr2=0x7fb36809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 --2- 172.21.15.164:0/1501270933 >> v2:172.21.15.165:3300/0 conn(0x7fb36809d190 0x7fb36809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 -- 172.21.15.164:0/1501270933 >> v1:172.21.15.164:6789/0 conn(0x7fb36809f4c0 legacy=0x7fb36813b5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient: _finish_hunting 0 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 monclient: found mon.noname-b 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.134+0000 7fb3677fe640 1 -- 172.21.15.164:0/1501270933 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb36809a4d0 con 0x7fb36809c310 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3677fe640 1 -- 172.21.15.164:0/1501270933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 msgr2=0x7fb36809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3677fe640 1 --2- 172.21.15.164:0/1501270933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb35802a560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b49dfabb0d10ff5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 1 -- 172.21.15.164:0/1501270933 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358036070 con 0x7fb36809c310 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: dump: 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:03.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: _finish_auth 0 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: _check_auth_tickets 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 5 monclient: authenticate success, global_id 244151 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 1 -- 172.21.15.164:0/1501270933 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb358007800 con 0x7fb36809c310 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:03.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 4 set_mon_vals no callback set 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 1 -- 172.21.15.164:0/1501270933 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb358007bd0 con 0x7fb36809c310 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb3667fc640 10 monclient: dump: 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:03.148 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 10 monclient: shutdown 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 -- 172.21.15.164:0/1501270933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 msgr2=0x7fb36809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 --2- 172.21.15.164:0/1501270933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb358009e50 tx=0x7fb35802a560 comp rx=0 tx=0).stop 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 -- 172.21.15.164:0/1501270933 shutdown_connections 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 --2- 172.21.15.164:0/1501270933 >> v2:172.21.15.165:3300/0 conn(0x7fb36809d190 0x7fb36809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 --2- 172.21.15.164:0/1501270933 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809c310 0x7fb36809ea40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.137+0000 7fb36daed640 1 -- 172.21.15.164:0/1501270933 >> 172.21.15.164:0/1501270933 conn(0x7fb3680a3110 msgr2=0x7fb3680a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.138+0000 7fb36daed640 1 -- 172.21.15.164:0/1501270933 shutdown_connections 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.138+0000 7fb36daed640 1 -- 172.21.15.164:0/1501270933 wait complete. 2024-09-10T16:28:03.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.138+0000 7fb36daed640 10 monclient: build_initial_monmap 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.138+0000 7fb36daed640 10 monclient: monmap: 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.138+0000 7fb36daed640 1 Processor -- start 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 -- start start 2024-09-10T16:28:03.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient: init 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 0x7fb3681cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-a con 0x7fb36809c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-c con 0x7fb36809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb368074240 0x7fb36813baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): picked mon.noname-b con 0x7fb368074240 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:03.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb3680a2230 con 0x7fb36809c310 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb3680a20b0 con 0x7fb368074240 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb368074e00 con 0x7fb36809f4c0 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb36daed640 10 monclient(hunting): authenticate will time out at 2061.626620s 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb3677fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 msgr2=0x7fb3681cd0f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb3677fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 0x7fb3681cd0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:34972/0 (socket says 172.21.15.164:34972) 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 1 -- 172.21.15.164:0/2974257060 learned_addr learned my addr 172.21.15.164:0/2974257060 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): get_auth_request con 0x7fb36809f4c0 auth_method 0 2024-09-10T16:28:03.152 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb367fff640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb368074240 0x7fb36813baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb367fff640 10 monclient(hunting): get_auth_request con 0x7fb368074240 auth_method 0 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb367fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb367fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb367fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.139+0000 7fb366ffd640 10 monclient(hunting): handle_auth_done global_id 244154 payload 824 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 -- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb368074240 msgr2=0x7fb36813baf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb368074240 0x7fb36813baf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.153 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 -- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 msgr2=0x7fb3681cd0f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 0x7fb3681cd0f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 monclient: found mon.noname-c 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.140+0000 7fb366ffd640 1 -- 172.21.15.164:0/2974257060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3680a4780 con 0x7fb36809f4c0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb366ffd640 1 -- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 msgr2=0x7fb368073d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb366ffd640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00d800 tx=0x7fb35c00dcc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00b8d0 con 0x7fb36809f4c0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: dump: 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:03.154 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: _finish_auth 0 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 5 monclient: authenticate success, global_id 244154 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb35c019070 con 0x7fb36809f4c0 2024-09-10T16:28:03.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 10 monclient: _renew_subs 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb36813e220 con 0x7fb36809f4c0 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb35c00bc90 con 0x7fb36809f4c0 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb34ffff640 10 monclient: dump: 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:03.156 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 10 monclient: _renew_subs 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb36813f710 con 0x7fb36809f4c0 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.142+0000 7fb365ffb640 4 set_mon_vals no callback set 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb35c01e460 con 0x7fb36809f4c0 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb36daed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb36daed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb36daed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb36809d930 con 0x7fb36809f4c0 2024-09-10T16:28:03.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb34ffff640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 0x7fb3540406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7fb35c0531e0 con 0x7fb36809f4c0 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb3677fe640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 0x7fb3540406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.143+0000 7fb3677fe640 10 monclient: get_auth_request con 0x7fb35403e210 auth_method 0 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.146+0000 7fb3677fe640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 0x7fb3540406d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb368074db0 tx=0x7fb358009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.148+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb35c051be0 con 0x7fb36809f4c0 2024-09-10T16:28:03.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.148+0000 7fb34ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:03.159 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.148+0000 7fb34ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb36daed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb36daed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb36daed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb3680a53e0 con 0x7fb36809f4c0 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb34ffff640 1 -- 172.21.15.164:0/2974257060 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7fb35c01da20 con 0x7fb36809f4c0 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb34ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.392+0000 7fb34ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:03.394 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":208,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 msgr2=0x7fb3540406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 0x7fb3540406d0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fb368074db0 tx=0x7fb358009000 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 10 monclient: shutdown 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 msgr2=0x7fb368073d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb35c00d800 tx=0x7fb35c00dcc0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 shutdown_connections 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb35403e210 0x7fb3540406d0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb368074240 0x7fb36813baf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb36809f4c0 0x7fb368073d00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 --2- 172.21.15.164:0/2974257060 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb36809c310 0x7fb3681cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:03.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 >> 172.21.15.164:0/2974257060 conn(0x7fb3680a3110 msgr2=0x7fb368097210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:03.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 shutdown_connections 2024-09-10T16:28:03.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:03.395+0000 7fb36daed640 1 -- 172.21.15.164:0/2974257060 wait complete. 2024-09-10T16:28:06.408 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:06.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 10 monclient: get_monmap_and_config 2024-09-10T16:28:06.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 10 monclient: build_initial_monmap 2024-09-10T16:28:06.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 10 monclient: monmap: 2024-09-10T16:28:06.516 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:06.516054+0000 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:06.516054+0000 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:06.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:06.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 1 Processor -- start 2024-09-10T16:28:06.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 1 -- start start 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.515+0000 7f1acd647640 10 monclient: init 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-c con 0x7f1ac806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1ac80a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-a con 0x7f1ac80a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac809ce50 con 0x7f1ac80a46a0 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac806db60 con 0x7f1ac80a4ab0 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:06.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1ac806cf50 con 0x7f1ac806cb70 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1acd647640 10 monclient(hunting): authenticate will time out at 2065.003548s 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac67fc640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f1ac80a46a0 legacy=0x7f1ac8137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac77fe640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 msgr2=0x7f1ac81383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:48842/0 (socket says 172.21.15.164:48842) 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 learned_addr learned my addr 172.21.15.164:0/1348302019 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ac806cb70 auth_method 0 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.516+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.517+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:06.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.517+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:06.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.517+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:06.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.517+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:06.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.717+0000 7f1ac67fc640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.164:6789/0 conn(0x7f1ac80a46a0 legacy=0x7f1ac8137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:06.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.717+0000 7f1ac77fe640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 msgr2=0x7f1ac81383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:06.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.717+0000 7f1ac77fe640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ac806cb70 auth_method 0 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:06.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:06.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:07.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.118+0000 7f1ac67fc640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.164:6789/0 conn(0x7f1ac80a46a0 legacy=0x7f1ac8137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:07.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.118+0000 7f1ac77fe640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 msgr2=0x7f1ac81383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:07.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.118+0000 7f1ac77fe640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ac806cb70 auth_method 0 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:07.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:07.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:07.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:07.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:07.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.919+0000 7f1ac77fe640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 msgr2=0x7f1ac81383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:07.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.919+0000 7f1ac67fc640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.164:6789/0 conn(0x7f1ac80a46a0 legacy=0x7f1ac8137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:07.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.919+0000 7f1ac77fe640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:07.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:07.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ac806cb70 auth_method 0 2024-09-10T16:28:07.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:07.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:07.920+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): tick 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 msgr2=0x7f1ac8134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.165:3300/0 conn(0x7f1ac806cb70 0x7f1ac8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 -- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 msgr2=0x7f1ac81383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:09.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 --2- 172.21.15.164:0/1348302019 >> v2:172.21.15.164:3300/0 conn(0x7f1ac80a4ab0 0x7f1ac81383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.164:6789/0 conn(0x7f1ac80a46a0 legacy=0x7f1ac8137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient: _add_conns ranks=[2,1] 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1aa80036d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1aa80061b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 -- 172.21.15.164:0/1348302019 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ac809cb70 con 0x7f1aa80061b0 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 1 -- 172.21.15.164:0/1348302019 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac806cf50 con 0x7f1aa80036d0 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac4ff9640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.516+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1aa80036d0 auth_method 0 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:09.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.517+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:09.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.519+0000 7f1ac67fc640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.165:6789/0 conn(0x7f1aa80061b0 legacy=0x7f1aa8008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1aa80036d0 auth_method 0 2024-09-10T16:28:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:09.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:09.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:09.718+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1aa80036d0 auth_method 0 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:10.121 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:10.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:10.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.119+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55993973 0 0) 0x7f1ac809cb70 con 0x7f1aa80061b0 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 10 monclient(hunting): _negotiate 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 10 monclient(hunting): my global_id is 234235 2024-09-10T16:28:10.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.663+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ab4003520 con 0x7f1aa80061b0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2237405529 0 0) 0x7f1ab4003520 con 0x7f1aa80061b0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 msgr2=0x7f1aa8005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1aa80036d0 0x7f1aa8005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _finish_hunting 0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa800b020 con 0x7f1aa80061b0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:28:10.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f1ab0004470 con 0x7f1aa80061b0 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1acd647640 5 monclient: authenticate success, global_id 234235 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1acd647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: dump: 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.672 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 >> v1:172.21.15.165:6789/0 conn(0x7f1aa80061b0 legacy=0x7f1aa8008670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac77fe640 1 -- 172.21.15.164:0/1348302019 reap_dead start 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 0x7f1ab4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): picked mon.b con 0x7f1ab4003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4004800 0x7f1ab4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): picked mon.c con 0x7f1ab4004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:10.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ab400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): picked mon.a con 0x7f1ab40053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac806cf50 con 0x7f1ab40053c0 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab400c250 con 0x7f1ab4003a20 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab400c450 con 0x7f1ab4004800 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:28:10.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1acd647640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac77fe640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 msgr2=0x7f1ab400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac77fe640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ab400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 0x7f1ab4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4004800 0x7f1ab4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ab4003a20 auth_method 0 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): get_auth_request con 0x7f1ab4004800 auth_method 0 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.667+0000 7f1ac67fc640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_done global_id 234235 payload 293 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4004800 msgr2=0x7f1ab4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4004800 0x7f1ab4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 msgr2=0x7f1ab400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ab400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 monclient: found mon.b 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab4003520 con 0x7f1ab4003a20 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:28:10.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.668+0000 7f1ac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 msgr2=0x7f1ab4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 0x7f1ab4003e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80096d0 tx=0x7f1ab4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85b4afb31c25b76f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8032070 con 0x7f1ab4003a20 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: dump: 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.683 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1acd647640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab802cc60 con 0x7f1ab4003a20 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1acd647640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 1 -- 172.21.15.164:0/1348302019 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab80098a0 con 0x7f1ab4003a20 2024-09-10T16:28:10.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1acd647640 4 set_mon_vals no callback set 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.671+0000 7f1ac5ffb640 10 monclient: dump: 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 10 monclient: shutdown 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 -- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 msgr2=0x7f1ab4003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 0x7f1ab4003e20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80096d0 tx=0x7f1ab4005830 comp rx=0 tx=0).stop 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 -- 172.21.15.164:0/1348302019 shutdown_connections 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ab400b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4004800 0x7f1ab4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 --2- 172.21.15.164:0/1348302019 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4003a20 0x7f1ab4003e20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 -- 172.21.15.164:0/1348302019 >> 172.21.15.164:0/1348302019 conn(0x7f1ac80aa900 msgr2=0x7f1ac80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 -- 172.21.15.164:0/1348302019 shutdown_connections 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.672+0000 7f1acd647640 1 -- 172.21.15.164:0/1348302019 wait complete. 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.673+0000 7f1acd647640 10 monclient: build_initial_monmap 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.673+0000 7f1acd647640 10 monclient: monmap: 2024-09-10T16:28:10.686 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.673+0000 7f1acd647640 1 Processor -- start 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 -- start start 2024-09-10T16:28:10.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient: init 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-c con 0x7f1ab4003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 0x7f1ac80a2230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-b con 0x7f1ab4004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ac80a2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): picked mon.noname-a con 0x7f1ab40053c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac81350f0 con 0x7f1ab40053c0 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1ac8134f70 con 0x7f1ab4004800 2024-09-10T16:28:10.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1ac8135270 con 0x7f1ab4003a20 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1acd647640 10 monclient(hunting): authenticate will time out at 2069.161573s 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac77fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 msgr2=0x7f1ac80a2770 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac77fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ac80a2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:45572/0 (socket says 172.21.15.164:45572) 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1465144154 learned_addr learned my addr 172.21.15.164:0/1465144154 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request con 0x7f1ab4003a20 auth_method 0 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac67fc640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 0x7f1ac80a2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac67fc640 10 monclient(hunting): get_auth_request con 0x7f1ab4004800 auth_method 0 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.674+0000 7f1ac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 10 monclient(hunting): handle_auth_done global_id 244157 payload 824 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 msgr2=0x7f1ac80a2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 0x7f1ac80a2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 msgr2=0x7f1ac80a2770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ac80a2770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 monclient: found mon.noname-c 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1465144154 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ac8122870 con 0x7f1ab4003a20 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.675+0000 7f1ac67fc640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 0x7f1ac80a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1ac6ffd640 1 -- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 msgr2=0x7f1ac81cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1ac6ffd640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8038040 tx=0x7f1ab8004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8045070 con 0x7f1ab4003a20 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:10.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: dump: 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:10.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ab8009880 con 0x7f1ab4003a20 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 5 monclient: authenticate success, global_id 244157 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8041850 con 0x7f1ab4003a20 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 10 monclient: _renew_subs 2024-09-10T16:28:10.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ac8133ec0 con 0x7f1ab4003a20 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1ac57fa640 4 set_mon_vals no callback set 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1abf7fe640 10 monclient: dump: 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:10.694 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 10 monclient: _renew_subs 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.677+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ac80a3c00 con 0x7f1ab4003a20 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.678+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1ab8048070 con 0x7f1ab4003a20 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1acd647640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1acd647640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1acd647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ac8137ca0 con 0x7f1ab4003a20 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1abf7fe640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 0x7f1aa8041020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(744..744 src has 677..744) ==== 4783+0+0 (secure 0 0 0) 0x7f1ab8074fb0 con 0x7f1ab4003a20 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1ac67fc640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 0x7f1aa8041020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:10.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.679+0000 7f1ac67fc640 10 monclient: get_auth_request con 0x7f1aa803eb60 auth_method 0 2024-09-10T16:28:10.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.682+0000 7f1ac67fc640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 0x7f1aa8041020 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0003b90 tx=0x7f1ab0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:10.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.684+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1ab80739b0 con 0x7f1ab4003a20 2024-09-10T16:28:10.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.684+0000 7f1abf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:10.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.684+0000 7f1abf7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:10.722 INFO:tasks.rados.rados.0.smithi165.stdout:3345: done (2 left) 2024-09-10T16:28:10.722 INFO:tasks.rados.rados.0.smithi165.stdout:3346: done (1 left) 2024-09-10T16:28:10.722 INFO:tasks.rados.rados.0.smithi165.stdout:3347: done (0 left) 2024-09-10T16:28:10.722 INFO:tasks.rados.rados.0.smithi165.stdout:3348: write oid 31 current snap is 366 2024-09-10T16:28:10.722 INFO:tasks.rados.rados.0.smithi165.stdout:3348: seq_num 1166 ranges {796962=742343,2031416=437730,2936739=413167} 2024-09-10T16:28:10.730 INFO:tasks.rados.rados.0.smithi165.stdout:3348: writing smithi16534841-31 from 796962 to 1539305 tid 1 2024-09-10T16:28:10.736 INFO:tasks.rados.rados.0.smithi165.stdout:3348: writing smithi16534841-31 from 2031416 to 2469146 tid 2 2024-09-10T16:28:10.741 INFO:tasks.rados.rados.0.smithi165.stdout:3348: writing smithi16534841-31 from 2936739 to 3349906 tid 3 2024-09-10T16:28:10.741 INFO:tasks.rados.rados.0.smithi165.stdout:3349: delete oid 22 current snap is 366 2024-09-10T16:28:10.741 INFO:tasks.rados.rados.0.smithi165.stdout:3348: finishing write tid 1 to smithi16534841-31 2024-09-10T16:28:10.741 INFO:tasks.rados.rados.0.smithi165.stdout:3348: finishing write tid 2 to smithi16534841-31 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: finishing write tid 3 to smithi16534841-31 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: finishing write tid 4 to smithi16534841-31 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: finishing write tid 5 to smithi16534841-31 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: oid 31 updating version 0 to 2057 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: oid 31 updating version 2057 to 2058 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: oid 31 updating version 2058 to 2060 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:3348: oid 31 version 2060 is already newer than 2059 2024-09-10T16:28:10.745 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 2060 (ObjNum 1166 snap 366 seq_num 1166) dirty exists 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3348: left oid 31 (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3348: done (1 left) 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3349: done (0 left) 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3350: read oid 30 snap -1 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3350: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3351: write (excl) oid 12 current snap is 366 2024-09-10T16:28:10.746 INFO:tasks.rados.rados.0.smithi165.stdout:3351: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2024-09-10T16:28:10.751 INFO:tasks.rados.rados.0.smithi165.stdout:3351: writing smithi16534841-12 from 430576 to 923417 tid 1 2024-09-10T16:28:10.758 INFO:tasks.rados.rados.0.smithi165.stdout:3351: writing smithi16534841-12 from 1632027 to 2073426 tid 2 2024-09-10T16:28:10.762 INFO:tasks.rados.rados.0.smithi165.stdout:3351: writing smithi16534841-12 from 2475232 to 2910943 tid 3 2024-09-10T16:28:10.767 INFO:tasks.rados.rados.0.smithi165.stdout:3351: writing smithi16534841-12 from 3365397 to 3630576 tid 4 2024-09-10T16:28:10.767 INFO:tasks.rados.rados.0.smithi165.stdout:3352: snap_create 2024-09-10T16:28:10.767 INFO:tasks.rados.rados.0.smithi165.stdout:3350: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:28:10.767 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 1 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 2 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 3 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 4 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 5 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: finishing write tid 6 to smithi16534841-12 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: oid 12 updating version 0 to 1648 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: oid 12 version 1648 is already newer than 1645 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: oid 12 version 1648 is already newer than 1646 2024-09-10T16:28:10.886 INFO:tasks.rados.rados.0.smithi165.stdout:3351: oid 12 updating version 1648 to 1649 2024-09-10T16:28:10.887 INFO:tasks.rados.rados.0.smithi165.stdout:3351: oid 12 version 1649 is already newer than 1647 2024-09-10T16:28:10.887 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1649 (ObjNum 1168 snap 366 seq_num 1168) dirty exists 2024-09-10T16:28:10.887 INFO:tasks.rados.rados.0.smithi165.stdout:3351: left oid 12 (ObjNum 1168 snap 366 seq_num 1168) 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.933+0000 7f1acd647640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.933+0000 7f1acd647640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.933+0000 7f1acd647640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.933+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1ac8121070 con 0x7f1ab4003a20 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.934+0000 7f1abf7fe640 1 -- 172.21.15.164:0/1465144154 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f1ab803f410 con 0x7f1ab4003a20 2024-09-10T16:28:10.935 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.934+0000 7f1abf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:10.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.934+0000 7f1abf7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:10.936 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 msgr2=0x7f1aa8041020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 0x7f1aa8041020 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f1ab0003b90 tx=0x7f1ab0009040 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 10 monclient: shutdown 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 msgr2=0x7f1ac81cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f1ab8038040 tx=0x7f1ab8004ab0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 shutdown_connections 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1aa803eb60 0x7f1aa8041020 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1ab40053c0 0x7f1ac80a2770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1ab4004800 0x7f1ac80a2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 --2- 172.21.15.164:0/1465144154 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1ab4003a20 0x7f1ac81cd1e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:10.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 >> 172.21.15.164:0/1465144154 conn(0x7f1ac80aa900 msgr2=0x7f1ac8132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:10.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 shutdown_connections 2024-09-10T16:28:10.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:10.936+0000 7f1acd647640 1 -- 172.21.15.164:0/1465144154 wait complete. 2024-09-10T16:28:10.949 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:28:10.949 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:28:10.949 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:28:11.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 10 monclient: get_monmap_and_config 2024-09-10T16:28:11.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 10 monclient: build_initial_monmap 2024-09-10T16:28:11.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 10 monclient: monmap: 2024-09-10T16:28:11.047 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:11.047 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:11.047069+0000 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:11.047069+0000 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:11.048 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:11.049 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:11.049 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 1 Processor -- start 2024-09-10T16:28:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 1 -- start start 2024-09-10T16:28:11.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.046+0000 7fb6ed09b640 10 monclient: init 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6e806cb70 0x7fb6e8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-c con 0x7fb6e806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6e80a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6e80a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:11.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e809ce50 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6e8138ac0 con 0x7fb6e80a46a0 2024-09-10T16:28:11.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb6e806db60 con 0x7fb6e806cb70 2024-09-10T16:28:11.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:11.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6ed09b640 10 monclient(hunting): authenticate will time out at 2069.534408s 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e759e640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fb6e80a4ab0 legacy=0x7fb6e81383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6e806cb70 0x7fb6e8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6e806cb70 0x7fb6e8134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:48902/0 (socket says 172.21.15.164:48902) 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e659c640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fb6e80a46a0 0x7fb6e8137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:39854/0 (socket says 172.21.15.164:39854) 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2394507117 learned_addr learned my addr 172.21.15.164:0/2394507117 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request con 0x7fb6e806cb70 auth_method 0 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.047+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.049+0000 7fb6e659c640 1 -- 172.21.15.164:0/2394507117 >> v1:172.21.15.165:6789/0 conn(0x7fb6e80a46a0 legacy=0x7fb6e8137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_done global_id 234241 payload 274 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2394507117 >> v1:172.21.15.165:6789/0 conn(0x7fb6e80a46a0 legacy=0x7fb6e8137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2394507117 >> v1:172.21.15.164:6789/0 conn(0x7fb6e80a4ab0 legacy=0x7fb6e81383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 1 monclient: found mon.noname-c 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:28:11.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.050+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2394507117 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e809cb70 con 0x7fb6e806cb70 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2394507117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 msgr2=0x7fb6e8134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2394507117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e8134f70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009f00 tx=0x7fb6d802a6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e7f7002f72821c4e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 1 -- 172.21.15.164:0/2394507117 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8036070 con 0x7fb6e806cb70 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:11.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: dump: 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: _finish_auth 0 2024-09-10T16:28:11.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 1 -- 172.21.15.164:0/2394507117 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d8007800 con 0x7fb6e806cb70 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 5 monclient: authenticate success, global_id 234241 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 4 set_mon_vals no callback set 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 1 -- 172.21.15.164:0/2394507117 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8007bd0 con 0x7fb6e806cb70 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6e5d9b640 10 monclient: dump: 2024-09-10T16:28:11.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 10 monclient: shutdown 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2394507117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 msgr2=0x7fb6e8134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2394507117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e8134f70 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8009f00 tx=0x7fb6d802a6c0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2394507117 shutdown_connections 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2394507117 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e8134f70 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2394507117 >> 172.21.15.164:0/2394507117 conn(0x7fb6e80aa900 msgr2=0x7fb6e80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2394507117 shutdown_connections 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.053+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2394507117 wait complete. 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.054+0000 7fb6ed09b640 10 monclient: build_initial_monmap 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.054+0000 7fb6ed09b640 10 monclient: monmap: 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:11.068 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.054+0000 7fb6ed09b640 1 Processor -- start 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.054+0000 7fb6ed09b640 1 -- start start 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.054+0000 7fb6ed09b640 10 monclient: init 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e812d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-b con 0x7fb6e806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6e80a46a0 0x7fb6e812d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-a con 0x7fb6e80a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 0x7fb6e809e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): picked mon.noname-c con 0x7fb6e80a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e8141050 con 0x7fb6e80a46a0 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6e806cf50 con 0x7fb6e806cb70 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e81411d0 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6ed09b640 10 monclient(hunting): authenticate will time out at 2069.542422s 2024-09-10T16:28:11.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e659c640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6e80a46a0 msgr2=0x7fb6e812d7c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6e80a46a0 0x7fb6e812d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 0x7fb6e809e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 0x7fb6e809e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:45578/0 (socket says 172.21.15.164:45578) 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 learned_addr learned my addr 172.21.15.164:0/2579099477 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): get_auth_request con 0x7fb6e80a4ab0 auth_method 0 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e812d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request con 0x7fb6e806cb70 auth_method 0 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient(hunting): handle_auth_done global_id 244160 payload 824 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 msgr2=0x7fb6e812d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6e806cb70 0x7fb6e812d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6e80a46a0 msgr2=0x7fb6e812d7c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6e80a46a0 0x7fb6e812d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient: _finish_hunting 0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 monclient: found mon.noname-c 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.055+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6e80a1f50 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 msgr2=0x7fb6e809e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6e759e640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 0x7fb6e809e6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000b3e0 tx=0x7fb6e000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e0017070 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: dump: 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.073 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: _finish_auth 0 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e00060d0 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 5 monclient: authenticate success, global_id 244160 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 monclient: _renew_subs 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6e809fed0 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e00064e0 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6df7fe640 10 monclient: dump: 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6e559a640 4 set_mon_vals no callback set 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 monclient: _renew_subs 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6e80a0a30 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 0x7fb6e809eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.058+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e80a2340 con 0x7fb6e809ec10 2024-09-10T16:28:11.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 0x7fb6e809eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient: get_auth_request con 0x7fb6e809ec10 auth_method 0 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb6e001a070 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_done global_id 244163 payload 824 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6e8134f70 con 0x7fb6e809ec10 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6df7fe640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6c803e1c0 0x7fb6c8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(745..745 src has 677..745) ==== 4755+0+0 (secure 0 0 0) 0x7fb6e0051f00 con 0x7fb6e80a4ab0 2024-09-10T16:28:11.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e659c640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6c803e1c0 0x7fb6c8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.059+0000 7fb6e659c640 10 monclient: get_auth_request con 0x7fb6c803e1c0 auth_method 0 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 msgr2=0x7fb6e809eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 0x7fb6e809eff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8033000 tx=0x7fb6d8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8043070 con 0x7fb6e809ec10 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.061+0000 7fb6df7fe640 10 monclient: dump: 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.078 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6e659c640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6c803e1c0 0x7fb6c8040680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80a0540 tx=0x7fb6d0007a00 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6d803d880 con 0x7fb6e809ec10 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 msgr2=0x7fb6e809eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 0x7fb6e809eff0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8033000 tx=0x7fb6d8004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 msgr2=0x7fb6e809eff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:11.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.062+0000 7fb6df7fe640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809ec10 0x7fb6e809eff0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fb6d8033000 tx=0x7fb6d8004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:11.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6ed09b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 0x7fb6e812f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6ed09b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6e80770f0 con 0x7fb6e809f530 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 0x7fb6e812f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient: get_auth_request con 0x7fb6e809f530 auth_method 0 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 10 monclient(hunting): handle_auth_done global_id 244166 payload 824 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.068+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2579099477 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6e812dd00 con 0x7fb6e809f530 2024-09-10T16:28:11.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6e6d9d640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 msgr2=0x7fb6e812f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6e6d9d640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 0x7fb6e812f240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80093e0 tx=0x7fb6d802f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d8036040 con 0x7fb6e809f530 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.070+0000 7fb6df7fe640 10 monclient: dump: 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fb6d8003e80 con 0x7fb6e809f530 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 msgr2=0x7fb6e812f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 0x7fb6e812f240 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80093e0 tx=0x7fb6d802f000 comp rx=0 tx=0).stop 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 msgr2=0x7fb6e812f240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6df7fe640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e809f530 0x7fb6e812f240 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb6d80093e0 tx=0x7fb6d802f000 comp rx=0 tx=0).stop 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:11.084 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 210, 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27978, 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:28:11.085 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:28:11.086 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:28:11.087 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:28:11.088 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.089 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.090 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.091 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.092 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:11.093 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.094 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.095 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.096 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:28:11.097 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.097 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.097 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.097 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:28:11.097 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:28:11.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6c803e1c0 msgr2=0x7fb6c8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6c803e1c0 0x7fb6c8040680 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fb6e80a0540 tx=0x7fb6d0007a00 comp rx=0 tx=0).stop 2024-09-10T16:28:11.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 10 monclient: shutdown 2024-09-10T16:28:11.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:11.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 msgr2=0x7fb6e809e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 --2- 172.21.15.164:0/2579099477 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6e80a4ab0 0x7fb6e809e6d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb6e000b3e0 tx=0x7fb6e000b8a0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 reap_dead start 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6e759e640 1 -- 172.21.15.164:0/2579099477 reap_dead start 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 shutdown_connections 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 >> 172.21.15.164:0/2579099477 conn(0x7fb6e80aa900 msgr2=0x7fb6e80a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 shutdown_connections 2024-09-10T16:28:11.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.071+0000 7fb6ed09b640 1 -- 172.21.15.164:0/2579099477 wait complete. 2024-09-10T16:28:11.099 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:28:11.099 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:28:11.100 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:28:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.196+0000 7f83a606e640 10 monclient: get_monmap_and_config 2024-09-10T16:28:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.196+0000 7f83a606e640 10 monclient: build_initial_monmap 2024-09-10T16:28:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.196+0000 7f83a606e640 10 monclient: monmap: 2024-09-10T16:28:11.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:11.197837+0000 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:11.197837+0000 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:11.199 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.197+0000 7f83a606e640 1 Processor -- start 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.197+0000 7f83a606e640 1 -- start start 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.197+0000 7f83a606e640 10 monclient: init 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient: _add_conns ranks=[4,0,1,3,2] 2024-09-10T16:28:11.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f83a009c310 0x7f83a009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-c con 0x7f83a009c310 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f83a009d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f83a013a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a009fbc0 con 0x7f83a009d190 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83a00a2a20 con 0x7f83a013a5c0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f83a009c6f0 con 0x7f83a009c310 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839effd640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f83a009d190 legacy=0x7f83a009ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f83a606e640 10 monclient(hunting): authenticate will time out at 2069.685344s 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f83a009c310 0x7f83a009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f83a009c310 0x7f83a009ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:48914/0 (socket says 172.21.15.164:48914) 2024-09-10T16:28:11.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 1 -- 172.21.15.164:0/252571218 learned_addr learned my addr 172.21.15.164:0/252571218 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): get_auth_request con 0x7f83a009c310 auth_method 0 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839ffff640 1 --1- 172.21.15.164:0/252571218 >> v1:172.21.15.165:6789/0 conn(0x7f83a013a5c0 0x7f83a013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:39866/0 (socket says 172.21.15.164:39866) 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.198+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839ffff640 1 -- 172.21.15.164:0/252571218 >> v1:172.21.15.165:6789/0 conn(0x7f83a013a5c0 legacy=0x7f83a013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 10 monclient(hunting): handle_auth_done global_id 234250 payload 274 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 1 -- 172.21.15.164:0/252571218 >> v1:172.21.15.165:6789/0 conn(0x7f83a013a5c0 legacy=0x7f83a013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 1 -- 172.21.15.164:0/252571218 >> v1:172.21.15.164:6789/0 conn(0x7f83a009d190 legacy=0x7f83a009ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:28:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:28:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:28:11.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.200+0000 7f839f7fe640 1 -- 172.21.15.164:0/252571218 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a009a4d0 con 0x7f83a009c310 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839f7fe640 1 -- 172.21.15.164:0/252571218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009c310 msgr2=0x7f83a009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839f7fe640 1 --2- 172.21.15.164:0/252571218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009c310 0x7f83a009ea40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83900098f0 tx=0x7f839002a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a7a1f67e584b72 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 1 -- 172.21.15.164:0/252571218 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390037070 con 0x7f83a009c310 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: dump: 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.209 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 1 -- 172.21.15.164:0/252571218 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8390032640 con 0x7f83a009c310 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f83a606e640 5 monclient: authenticate success, global_id 234250 2024-09-10T16:28:11.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f83a606e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.203+0000 7f839e7fc640 1 -- 172.21.15.164:0/252571218 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390032a10 con 0x7f83a009c310 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 4 set_mon_vals no callback set 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f839e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f839e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f839e7fc640 10 monclient: dump: 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.211 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 10 monclient: shutdown 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 -- 172.21.15.164:0/252571218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009c310 msgr2=0x7f83a009ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 --2- 172.21.15.164:0/252571218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009c310 0x7f83a009ea40 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f83900098f0 tx=0x7f839002a1c0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 -- 172.21.15.164:0/252571218 shutdown_connections 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 --2- 172.21.15.164:0/252571218 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009c310 0x7f83a009ea40 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 -- 172.21.15.164:0/252571218 >> 172.21.15.164:0/252571218 conn(0x7f83a00a3110 msgr2=0x7f83a00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 -- 172.21.15.164:0/252571218 shutdown_connections 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.204+0000 7f83a606e640 1 -- 172.21.15.164:0/252571218 wait complete. 2024-09-10T16:28:11.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.205+0000 7f83a606e640 10 monclient: build_initial_monmap 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.205+0000 7f83a606e640 10 monclient: monmap: 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:11.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.205+0000 7f83a606e640 1 Processor -- start 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.205+0000 7f83a606e640 1 -- start start 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.205+0000 7f83a606e640 10 monclient: init 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:28:11.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83a009c310 0x7f83a0090750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-a con 0x7f83a009c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a013a5c0 0x7f83a0090c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-b con 0x7f83a013a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 0x7f83a01d17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): picked mon.noname-c con 0x7f83a00911d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839f7fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83a009c310 msgr2=0x7f83a0090750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00a2230 con 0x7f83a009c310 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83a009c310 0x7f83a0090750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f83a00a20b0 con 0x7f83a013a5c0 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f83a0074e00 con 0x7f83a00911d0 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f83a606e640 10 monclient(hunting): authenticate will time out at 2069.693319s 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 0x7f83a01d17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 0x7f83a01d17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:45622/0 (socket says 172.21.15.164:45622) 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 learned_addr learned my addr 172.21.15.164:0/1618416864 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): get_auth_request con 0x7f83a00911d0 auth_method 0 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a013a5c0 0x7f83a0090c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 10 monclient(hunting): get_auth_request con 0x7f83a013a5c0 auth_method 0 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient(hunting): handle_auth_done global_id 244169 payload 824 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a013a5c0 msgr2=0x7f83a0090c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a013a5c0 0x7f83a0090c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83a009c310 msgr2=0x7f83a0090750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:11.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f83a009c310 0x7f83a0090750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 monclient: found mon.noname-c 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83a00a5170 con 0x7f83a00911d0 2024-09-10T16:28:11.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.206+0000 7f839effd640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a013a5c0 0x7f83a0090c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:28:11.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.208+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 msgr2=0x7f83a01d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.208+0000 7f839ffff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 0x7f83a01d17a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f839800c7a0 tx=0x7f839800cc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8398017070 con 0x7f83a00911d0 2024-09-10T16:28:11.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: dump: 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.228 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: _finish_auth 0 2024-09-10T16:28:11.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: _check_auth_tickets 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f839800a840 con 0x7f83a00911d0 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 5 monclient: authenticate success, global_id 244169 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f839800abf0 con 0x7f83a00911d0 2024-09-10T16:28:11.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 10 monclient: _renew_subs 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83a00930c0 con 0x7f83a00911d0 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f8387fff640 10 monclient: dump: 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.231 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f839dffb640 4 set_mon_vals no callback set 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 10 monclient: _renew_subs 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f83a00a4c90 con 0x7f83a00911d0 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.209+0000 7f83a606e640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 0x7f83a0071050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f83a009f7b0 con 0x7f83a009d190 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f839801a070 con 0x7f83a00911d0 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f8387fff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838c03e1c0 0x7f838c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(745..745 src has 677..745) ==== 4755+0+0 (secure 0 0 0) 0x7f8398052200 con 0x7f83a00911d0 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 0x7f83a0071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient: get_auth_request con 0x7f83a009d190 auth_method 0 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839effd640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838c03e1c0 0x7f838c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839effd640 10 monclient: get_auth_request con 0x7f838c03e1c0 auth_method 0 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 10 monclient(hunting): handle_auth_done global_id 234259 payload 824 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.210+0000 7f839f7fe640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f83a0091f50 con 0x7f83a009d190 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.212+0000 7f839f7fe640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 msgr2=0x7f83a0071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.212+0000 7f839f7fe640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 0x7f83a0071050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8390034000 tx=0x7f83900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.212+0000 7f839effd640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838c03e1c0 0x7f838c040680 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f83a006e780 tx=0x7f8394008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390037040 con 0x7f83a009d190 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:11.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 10 monclient: dump: 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8390009490 con 0x7f83a009d190 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:11.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 msgr2=0x7f83a0071050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 0x7f83a0071050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8390034000 tx=0x7f83900046c0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 msgr2=0x7f83a0071050 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.213+0000 7f8387fff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a009d190 0x7f83a0071050 secure :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8390034000 tx=0x7f83900046c0 comp rx=0 tx=0).stop 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f83a606e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f83a606e640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 0x7f83a0071970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f83a606e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f83a0077260 con 0x7f83a0071590 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f839f7fe640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 0x7f83a0071970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f839f7fe640 10 monclient: get_auth_request con 0x7f83a0071590 auth_method 0 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f839f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f839f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:11.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.219+0000 7f839f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.220+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.220+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.220+0000 7f839f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.220+0000 7f839f7fe640 10 monclient(hunting): handle_auth_done global_id 234262 payload 824 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.220+0000 7f839f7fe640 1 -- 172.21.15.164:0/1618416864 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f83a0071eb0 con 0x7f83a0071590 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f839f7fe640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 msgr2=0x7f83a0071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f839f7fe640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 0x7f83a0071970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f8390009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8390037040 con 0x7f83a0071590 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f8387fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f8387fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.222+0000 7f8387fff640 10 monclient: dump: 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:11.238 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4878 (secure 0 0 0) 0x7f839004c070 con 0x7f83a0071590 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 msgr2=0x7f83a0071970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 0x7f83a0071970 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f8390009240 comp rx=0 tx=0).stop 2024-09-10T16:28:11.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 msgr2=0x7f83a0071970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:11.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f8387fff640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f83a0071590 0x7f83a0071970 secure :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f8390000c00 tx=0x7f8390009240 comp rx=0 tx=0).stop 2024-09-10T16:28:11.240 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:28:11.240 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:11.240 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:11.240 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 210, 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 91506, 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.241 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:28:11.242 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.243 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:11.244 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:11.245 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.246 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.247 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:11.248 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.249 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:11.250 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:11.251 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.252 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:11.253 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:28:11.254 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:28:11.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838c03e1c0 msgr2=0x7f838c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f838c03e1c0 0x7f838c040680 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f83a006e780 tx=0x7f8394008040 comp rx=0 tx=0).stop 2024-09-10T16:28:11.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 10 monclient: shutdown 2024-09-10T16:28:11.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:11.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 msgr2=0x7f83a01d17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 --2- 172.21.15.164:0/1618416864 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f83a00911d0 0x7f83a01d17a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f839800c7a0 tx=0x7f839800cc60 comp rx=0 tx=0).stop 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 reap_dead start 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f839ffff640 1 -- 172.21.15.164:0/1618416864 reap_dead start 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 shutdown_connections 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 >> 172.21.15.164:0/1618416864 conn(0x7f83a00a3110 msgr2=0x7f83a009af10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 shutdown_connections 2024-09-10T16:28:11.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:11.223+0000 7f83a606e640 1 -- 172.21.15.164:0/1618416864 wait complete. 2024-09-10T16:28:11.256 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:28:11.256 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:28:11.727 INFO:tasks.rados.rados.0.smithi165.stdout:3350: done (2 left) 2024-09-10T16:28:11.727 INFO:tasks.rados.rados.0.smithi165.stdout:3351: done (1 left) 2024-09-10T16:28:11.727 INFO:tasks.rados.rados.0.smithi165.stdout:3352: done (0 left) 2024-09-10T16:28:11.727 INFO:tasks.rados.rados.0.smithi165.stdout:3353: delete oid 19 current snap is 367 2024-09-10T16:28:11.735 INFO:tasks.rados.rados.0.smithi165.stdout:3353: done (0 left) 2024-09-10T16:28:11.735 INFO:tasks.rados.rados.0.smithi165.stdout:3354: snap_create 2024-09-10T16:28:12.729 INFO:tasks.rados.rados.0.smithi165.stdout:3354: done (0 left) 2024-09-10T16:28:12.729 INFO:tasks.rados.rados.0.smithi165.stdout:3355: read oid 35 snap 363 2024-09-10T16:28:12.729 INFO:tasks.rados.rados.0.smithi165.stdout:3355: expect (ObjNum 1140 snap 353 seq_num 1140) 2024-09-10T16:28:12.729 INFO:tasks.rados.rados.0.smithi165.stdout:3356: write (excl) oid 1 current snap is 368 2024-09-10T16:28:12.730 INFO:tasks.rados.rados.0.smithi165.stdout:3356: seq_num 1170 ranges {464190=643339,1748991=445068,2930078=574608} 2024-09-10T16:28:12.735 INFO:tasks.rados.rados.0.smithi165.stdout:3356: writing smithi16534841-1 from 464190 to 1107529 tid 1 2024-09-10T16:28:12.742 INFO:tasks.rados.rados.0.smithi165.stdout:3356: writing smithi16534841-1 from 1748991 to 2194059 tid 2 2024-09-10T16:28:12.750 INFO:tasks.rados.rados.0.smithi165.stdout:3356: writing smithi16534841-1 from 2930078 to 3504686 tid 3 2024-09-10T16:28:12.750 INFO:tasks.rados.rados.0.smithi165.stdout:3357: read oid 6 snap 342 2024-09-10T16:28:12.750 INFO:tasks.rados.rados.0.smithi165.stdout:3357: expect deleted 2024-09-10T16:28:12.750 INFO:tasks.rados.rados.0.smithi165.stdout:3358: snap_create 2024-09-10T16:28:12.750 INFO:tasks.rados.rados.0.smithi165.stdout:3355: expect (ObjNum 1140 snap 353 seq_num 1140) 2024-09-10T16:28:12.863 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 89138 2024-09-10T16:28:12.918 INFO:tasks.rados.rados.0.smithi165.stdout:3356: finishing write tid 1 to smithi16534841-1 2024-09-10T16:28:12.918 INFO:tasks.rados.rados.0.smithi165.stdout:3356: finishing write tid 2 to smithi16534841-1 2024-09-10T16:28:12.918 INFO:tasks.rados.rados.0.smithi165.stdout:3356: finishing write tid 3 to smithi16534841-1 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: finishing write tid 4 to smithi16534841-1 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: finishing write tid 5 to smithi16534841-1 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: oid 1 updating version 0 to 2004 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: oid 1 updating version 2004 to 2005 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: oid 1 updating version 2005 to 2006 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: oid 1 updating version 2006 to 2007 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2007 (ObjNum 1170 snap 368 seq_num 1170) dirty exists 2024-09-10T16:28:12.919 INFO:tasks.rados.rados.0.smithi165.stdout:3356: left oid 1 (ObjNum 1170 snap 368 seq_num 1170) 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3355: done (3 left) 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3356: done (2 left) 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3357: done (1 left) 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3358: done (0 left) 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3359: copy_from oid 21 from oid 13 current snap is 369 2024-09-10T16:28:13.745 INFO:tasks.rados.rados.0.smithi165.stdout:3360: snap_create 2024-09-10T16:28:13.767 INFO:tasks.rados.rados.0.smithi165.stdout:3359: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:28:13.770 INFO:tasks.rados.rados.0.smithi165.stdout:3359: finishing copy_from to smithi16534841-21 2024-09-10T16:28:13.770 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1468 (ObjNum 1163 snap 365 seq_num 1163) dirty exists 2024-09-10T16:28:14.736 INFO:tasks.rados.rados.0.smithi165.stdout:3359: done (1 left) 2024-09-10T16:28:14.736 INFO:tasks.rados.rados.0.smithi165.stdout:3360: done (0 left) 2024-09-10T16:28:14.736 INFO:tasks.rados.rados.0.smithi165.stdout:3361: snap_remove snap 336 2024-09-10T16:28:15.737 INFO:tasks.rados.rados.0.smithi165.stdout:3361: done (0 left) 2024-09-10T16:28:15.737 INFO:tasks.rados.rados.0.smithi165.stdout:3362: write oid 35 current snap is 370 2024-09-10T16:28:15.737 INFO:tasks.rados.rados.0.smithi165.stdout:3362: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2024-09-10T16:28:15.742 INFO:tasks.rados.rados.0.smithi165.stdout:3362: writing smithi16534841-35 from 480997 to 915938 tid 1 2024-09-10T16:28:15.749 INFO:tasks.rados.rados.0.smithi165.stdout:3362: writing smithi16534841-35 from 1407473 to 2112552 tid 2 2024-09-10T16:28:15.756 INFO:tasks.rados.rados.0.smithi165.stdout:3362: writing smithi16534841-35 from 2557501 to 3343381 tid 3 2024-09-10T16:28:15.756 INFO:tasks.rados.rados.0.smithi165.stdout:3363: rollback oid 44 current snap is 370 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 369 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:3362: finishing write tid 1 to smithi16534841-35 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:3362: finishing write tid 2 to smithi16534841-35 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:3364: read oid 50 snap 362 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:3364: expect deleted 2024-09-10T16:28:15.757 INFO:tasks.rados.rados.0.smithi165.stdout:3365: delete oid 40 current snap is 370 2024-09-10T16:28:15.766 INFO:tasks.rados.rados.0.smithi165.stdout:3362: finishing write tid 3 to smithi16534841-35 2024-09-10T16:28:15.766 INFO:tasks.rados.rados.0.smithi165.stdout:3363: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:28:15.767 INFO:tasks.rados.rados.0.smithi165.stdout:3364: done (3 left) 2024-09-10T16:28:15.767 INFO:tasks.rados.rados.0.smithi165.stdout:3365: done (2 left) 2024-09-10T16:28:15.767 INFO:tasks.rados.rados.0.smithi165.stdout:3366: write oid 22 current snap is 370 2024-09-10T16:28:15.767 INFO:tasks.rados.rados.0.smithi165.stdout:3366: seq_num 1173 ranges {514611=585439,1524437=592395,2895994=408424} 2024-09-10T16:28:15.774 INFO:tasks.rados.rados.0.smithi165.stdout:3366: writing smithi16534841-22 from 514611 to 1100050 tid 1 2024-09-10T16:28:15.780 INFO:tasks.rados.rados.0.smithi165.stdout:3366: writing smithi16534841-22 from 1524437 to 2116832 tid 2 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:3366: writing smithi16534841-22 from 2895994 to 3304418 tid 3 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:3367: delete oid 7 current snap is 370 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:3363: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:3362: finishing write tid 4 to smithi16534841-35 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:3363: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:28:15.787 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1833 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: finishing write tid 5 to smithi16534841-35 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: oid 35 updating version 0 to 1470 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: oid 35 updating version 1470 to 1471 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: oid 35 updating version 1471 to 1473 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: oid 35 version 1473 is already newer than 1472 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1473 (ObjNum 1171 snap 370 seq_num 1171) dirty exists 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3362: left oid 35 (ObjNum 1171 snap 370 seq_num 1171) 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3366: finishing write tid 1 to smithi16534841-22 2024-09-10T16:28:15.788 INFO:tasks.rados.rados.0.smithi165.stdout:3366: finishing write tid 2 to smithi16534841-22 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3366: finishing write tid 3 to smithi16534841-22 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3362: done (3 left) 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3363: done (2 left) 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3367: done (1 left) 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3368: delete oid 21 current snap is 370 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3366: finishing write tid 4 to smithi16534841-22 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3366: finishing write tid 5 to smithi16534841-22 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3366: oid 22 updating version 0 to 1476 2024-09-10T16:28:15.790 INFO:tasks.rados.rados.0.smithi165.stdout:3366: oid 22 version 1476 is already newer than 1475 2024-09-10T16:28:15.791 INFO:tasks.rados.rados.0.smithi165.stdout:3366: oid 22 version 1476 is already newer than 1474 2024-09-10T16:28:15.791 INFO:tasks.rados.rados.0.smithi165.stdout:3366: oid 22 updating version 1476 to 1477 2024-09-10T16:28:15.791 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1477 (ObjNum 1173 snap 370 seq_num 1173) dirty exists 2024-09-10T16:28:15.791 INFO:tasks.rados.rados.0.smithi165.stdout:3366: left oid 22 (ObjNum 1173 snap 370 seq_num 1173) 2024-09-10T16:28:15.793 INFO:tasks.rados.rados.0.smithi165.stdout:3366: done (1 left) 2024-09-10T16:28:15.793 INFO:tasks.rados.rados.0.smithi165.stdout:3368: done (0 left) 2024-09-10T16:28:15.793 INFO:tasks.rados.rados.0.smithi165.stdout:3369: copy_from oid 5 from oid 13 current snap is 370 2024-09-10T16:28:15.793 INFO:tasks.rados.rados.0.smithi165.stdout:3370: write oid 33 current snap is 370 2024-09-10T16:28:15.794 INFO:tasks.rados.rados.0.smithi165.stdout:3370: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2024-09-10T16:28:15.798 INFO:tasks.rados.rados.0.smithi165.stdout:3370: writing smithi16534841-33 from 565032 to 976218 tid 1 2024-09-10T16:28:15.804 INFO:tasks.rados.rados.0.smithi165.stdout:3370: writing smithi16534841-33 from 1583530 to 2039605 tid 2 2024-09-10T16:28:15.811 INFO:tasks.rados.rados.0.smithi165.stdout:3370: writing smithi16534841-33 from 2578263 to 3336856 tid 3 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:3371: delete oid 15 current snap is 370 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:3370: finishing write tid 1 to smithi16534841-33 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:3369: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:3370: finishing write tid 2 to smithi16534841-33 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:3369: finishing copy_from to smithi16534841-5 2024-09-10T16:28:15.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1468 (ObjNum 1163 snap 365 seq_num 1163) dirty exists 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3369: done (2 left) 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3371: done (1 left) 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3372: read oid 29 snap -1 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3372: expect deleted 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3373: read oid 37 snap -1 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3373: expect (ObjNum 1028 snap 313 seq_num 1028) 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3374: rollback oid 12 current snap is 370 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 363 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3375: rollback oid 39 current snap is 370 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 347 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3376: read oid 32 snap -1 2024-09-10T16:28:15.814 INFO:tasks.rados.rados.0.smithi165.stdout:3376: expect deleted 2024-09-10T16:28:15.815 INFO:tasks.rados.rados.0.smithi165.stdout:3377: snap_create 2024-09-10T16:28:15.818 INFO:tasks.rados.rados.0.smithi165.stdout:3370: finishing write tid 3 to smithi16534841-33 2024-09-10T16:28:15.818 INFO:tasks.rados.rados.0.smithi165.stdout:3373: expect (ObjNum 1028 snap 313 seq_num 1028) 2024-09-10T16:28:15.865 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2809881 2024-09-10T16:28:15.920 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2779254 2024-09-10T16:28:15.930 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: finishing write tid 4 to smithi16534841-33 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: finishing write tid 5 to smithi16534841-33 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: oid 33 updating version 0 to 1854 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: oid 33 updating version 1854 to 1855 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: oid 33 version 1855 is already newer than 1853 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:3370: oid 33 updating version 1855 to 1856 2024-09-10T16:28:16.001 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1856 (ObjNum 1176 snap 370 seq_num 1176) dirty exists 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3370: left oid 33 (ObjNum 1176 snap 370 seq_num 1176) 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3374: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3374: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3374: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1652 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3375: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3375: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:3375: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:28:16.002 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2010 (ObjNum 1056 snap 323 seq_num 1056) dirty exists 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3370: done (6 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3372: done (5 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3373: done (4 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3374: done (3 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3375: done (2 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3376: done (1 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3377: done (0 left) 2024-09-10T16:28:16.753 INFO:tasks.rados.rados.0.smithi165.stdout:3378: snap_create 2024-09-10T16:28:17.759 INFO:tasks.rados.rados.0.smithi165.stdout:3378: done (0 left) 2024-09-10T16:28:17.759 INFO:tasks.rados.rados.0.smithi165.stdout:3379: write (excl) oid 21 current snap is 372 2024-09-10T16:28:17.759 INFO:tasks.rados.rados.0.smithi165.stdout:3379: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2024-09-10T16:28:17.765 INFO:tasks.rados.rados.0.smithi165.stdout:3379: writing smithi16534841-21 from 598646 to 1160330 tid 1 2024-09-10T16:28:17.771 INFO:tasks.rados.rados.0.smithi165.stdout:3379: writing smithi16534841-21 from 1700494 to 2160238 tid 2 2024-09-10T16:28:17.778 INFO:tasks.rados.rados.0.smithi165.stdout:3379: writing smithi16534841-21 from 2633109 to 3414246 tid 3 2024-09-10T16:28:17.778 INFO:tasks.rados.rados.0.smithi165.stdout:3380: snap_create 2024-09-10T16:28:17.778 INFO:tasks.rados.rados.0.smithi165.stdout:3379: finishing write tid 1 to smithi16534841-21 2024-09-10T16:28:17.778 INFO:tasks.rados.rados.0.smithi165.stdout:3379: finishing write tid 2 to smithi16534841-21 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: finishing write tid 3 to smithi16534841-21 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: finishing write tid 4 to smithi16534841-21 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: finishing write tid 5 to smithi16534841-21 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: oid 21 updating version 0 to 1858 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: oid 21 updating version 1858 to 1860 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: oid 21 version 1860 is already newer than 1859 2024-09-10T16:28:17.783 INFO:tasks.rados.rados.0.smithi165.stdout:3379: oid 21 version 1860 is already newer than 1857 2024-09-10T16:28:17.784 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1860 (ObjNum 1178 snap 372 seq_num 1178) dirty exists 2024-09-10T16:28:17.784 INFO:tasks.rados.rados.0.smithi165.stdout:3379: left oid 21 (ObjNum 1178 snap 372 seq_num 1178) 2024-09-10T16:28:18.763 INFO:tasks.rados.rados.0.smithi165.stdout:3379: done (1 left) 2024-09-10T16:28:18.763 INFO:tasks.rados.rados.0.smithi165.stdout:3380: done (0 left) 2024-09-10T16:28:18.763 INFO:tasks.rados.rados.0.smithi165.stdout:3381: snap_remove snap 347 2024-09-10T16:28:19.783 INFO:tasks.rados.rados.0.smithi165.stdout:3381: done (0 left) 2024-09-10T16:28:19.783 INFO:tasks.rados.rados.0.smithi165.stdout:3382: rollback oid 14 current snap is 373 2024-09-10T16:28:19.783 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 351 2024-09-10T16:28:19.783 INFO:tasks.rados.rados.0.smithi165.stdout:3383: write oid 26 current snap is 373 2024-09-10T16:28:19.784 INFO:tasks.rados.rados.0.smithi165.stdout:3383: seq_num 1179 ranges {615453=753286,2158976=603402,3460532=354921} 2024-09-10T16:28:19.791 INFO:tasks.rados.rados.0.smithi165.stdout:3383: writing smithi16534841-26 from 615453 to 1368739 tid 1 2024-09-10T16:28:19.800 INFO:tasks.rados.rados.0.smithi165.stdout:3383: writing smithi16534841-26 from 2158976 to 2762378 tid 2 2024-09-10T16:28:19.806 INFO:tasks.rados.rados.0.smithi165.stdout:3383: writing smithi16534841-26 from 3460532 to 3815453 tid 3 2024-09-10T16:28:19.806 INFO:tasks.rados.rados.0.smithi165.stdout:3384: delete oid 27 current snap is 373 2024-09-10T16:28:19.806 INFO:tasks.rados.rados.0.smithi165.stdout:3382: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:28:19.806 INFO:tasks.rados.rados.0.smithi165.stdout:3382: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:28:19.807 INFO:tasks.rados.rados.0.smithi165.stdout:3382: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:28:19.807 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1841 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:28:19.807 INFO:tasks.rados.rados.0.smithi165.stdout:3383: finishing write tid 1 to smithi16534841-26 2024-09-10T16:28:19.807 INFO:tasks.rados.rados.0.smithi165.stdout:3383: finishing write tid 2 to smithi16534841-26 2024-09-10T16:28:19.809 INFO:tasks.rados.rados.0.smithi165.stdout:3382: done (2 left) 2024-09-10T16:28:19.809 INFO:tasks.rados.rados.0.smithi165.stdout:3384: done (1 left) 2024-09-10T16:28:19.809 INFO:tasks.rados.rados.0.smithi165.stdout:3385: write oid 38 current snap is 373 2024-09-10T16:28:19.809 INFO:tasks.rados.rados.0.smithi165.stdout:3385: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2024-09-10T16:28:19.815 INFO:tasks.rados.rados.0.smithi165.stdout:3385: writing smithi16534841-38 from 649067 to 1152851 tid 1 2024-09-10T16:28:19.822 INFO:tasks.rados.rados.0.smithi165.stdout:3385: writing smithi16534841-38 from 1875940 to 2483011 tid 2 2024-09-10T16:28:19.831 INFO:tasks.rados.rados.0.smithi165.stdout:3385: writing smithi16534841-38 from 3115378 to 3846684 tid 3 2024-09-10T16:28:19.831 INFO:tasks.rados.rados.0.smithi165.stdout:3386: snap_create 2024-09-10T16:28:19.831 INFO:tasks.rados.rados.0.smithi165.stdout:3383: finishing write tid 3 to smithi16534841-26 2024-09-10T16:28:19.831 INFO:tasks.rados.rados.0.smithi165.stdout:3383: finishing write tid 4 to smithi16534841-26 2024-09-10T16:28:19.831 INFO:tasks.rados.rados.0.smithi165.stdout:3383: finishing write tid 5 to smithi16534841-26 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:3383: oid 26 updating version 0 to 2088 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:3383: oid 26 updating version 2088 to 2089 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:3383: oid 26 version 2089 is already newer than 2083 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:3383: oid 26 updating version 2089 to 2090 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2090 (ObjNum 1179 snap 373 seq_num 1179) dirty exists 2024-09-10T16:28:19.832 INFO:tasks.rados.rados.0.smithi165.stdout:3383: left oid 26 (ObjNum 1179 snap 373 seq_num 1179) 2024-09-10T16:28:19.833 INFO:tasks.rados.rados.0.smithi165.stdout:3385: finishing write tid 1 to smithi16534841-38 2024-09-10T16:28:19.833 INFO:tasks.rados.rados.0.smithi165.stdout:3385: finishing write tid 2 to smithi16534841-38 2024-09-10T16:28:19.834 INFO:tasks.rados.rados.0.smithi165.stdout:3385: finishing write tid 3 to smithi16534841-38 2024-09-10T16:28:19.834 INFO:tasks.rados.rados.0.smithi165.stdout:3385: finishing write tid 4 to smithi16534841-38 2024-09-10T16:28:19.834 INFO:tasks.rados.rados.0.smithi165.stdout:3385: finishing write tid 5 to smithi16534841-38 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:3385: oid 38 updating version 0 to 1415 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:3385: oid 38 version 1415 is already newer than 1413 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:3385: oid 38 version 1415 is already newer than 1414 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:3385: oid 38 version 1415 is already newer than 1412 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1415 (ObjNum 1181 snap 373 seq_num 1181) dirty exists 2024-09-10T16:28:19.835 INFO:tasks.rados.rados.0.smithi165.stdout:3385: left oid 38 (ObjNum 1181 snap 373 seq_num 1181) 2024-09-10T16:28:20.785 INFO:tasks.rados.rados.0.smithi165.stdout:3383: done (2 left) 2024-09-10T16:28:20.785 INFO:tasks.rados.rados.0.smithi165.stdout:3385: done (1 left) 2024-09-10T16:28:20.785 INFO:tasks.rados.rados.0.smithi165.stdout:3386: done (0 left) 2024-09-10T16:28:20.785 INFO:tasks.rados.rados.0.smithi165.stdout:3387: write oid 16 current snap is 374 2024-09-10T16:28:20.785 INFO:tasks.rados.rados.0.smithi165.stdout:3387: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2024-09-10T16:28:20.791 INFO:tasks.rados.rados.0.smithi165.stdout:3387: writing smithi16534841-16 from 665874 to 1244907 tid 1 2024-09-10T16:28:20.798 INFO:tasks.rados.rados.0.smithi165.stdout:3387: writing smithi16534841-16 from 1934422 to 2401504 tid 2 2024-09-10T16:28:20.805 INFO:tasks.rados.rados.0.smithi165.stdout:3387: writing smithi16534841-16 from 3142801 to 3685379 tid 3 2024-09-10T16:28:20.806 INFO:tasks.rados.rados.0.smithi165.stdout:3388: delete oid 34 current snap is 374 2024-09-10T16:28:20.806 INFO:tasks.rados.rados.0.smithi165.stdout:3387: finishing write tid 1 to smithi16534841-16 2024-09-10T16:28:20.806 INFO:tasks.rados.rados.0.smithi165.stdout:3387: finishing write tid 2 to smithi16534841-16 2024-09-10T16:28:20.809 INFO:tasks.rados.rados.0.smithi165.stdout:3387: finishing write tid 3 to smithi16534841-16 2024-09-10T16:28:20.809 INFO:tasks.rados.rados.0.smithi165.stdout:3387: finishing write tid 4 to smithi16534841-16 2024-09-10T16:28:20.809 INFO:tasks.rados.rados.0.smithi165.stdout:3387: finishing write tid 5 to smithi16534841-16 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:3387: oid 16 updating version 0 to 2093 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:3387: oid 16 version 2093 is already newer than 2091 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:3387: oid 16 updating version 2093 to 2094 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:3387: oid 16 version 2094 is already newer than 2092 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2094 (ObjNum 1182 snap 374 seq_num 1182) dirty exists 2024-09-10T16:28:20.810 INFO:tasks.rados.rados.0.smithi165.stdout:3387: left oid 16 (ObjNum 1182 snap 374 seq_num 1182) 2024-09-10T16:28:20.812 INFO:tasks.rados.rados.0.smithi165.stdout:3387: done (1 left) 2024-09-10T16:28:20.812 INFO:tasks.rados.rados.0.smithi165.stdout:3388: done (0 left) 2024-09-10T16:28:20.812 INFO:tasks.rados.rados.0.smithi165.stdout:3389: snap_remove snap 352 2024-09-10T16:28:21.793 INFO:tasks.rados.rados.0.smithi165.stdout:3389: done (0 left) 2024-09-10T16:28:21.794 INFO:tasks.rados.rados.0.smithi165.stdout:3390: copy_from oid 50 from oid 49 current snap is 374 2024-09-10T16:28:21.794 INFO:tasks.rados.rados.0.smithi165.stdout:3391: delete oid 33 current snap is 374 2024-09-10T16:28:21.807 INFO:tasks.rados.rados.0.smithi165.stdout:3391: done (1 left) 2024-09-10T16:28:21.807 INFO:tasks.rados.rados.0.smithi165.stdout:3392: read oid 18 snap 359 2024-09-10T16:28:21.808 INFO:tasks.rados.rados.0.smithi165.stdout:3392: expect (ObjNum 1148 snap 359 seq_num 1148) 2024-09-10T16:28:21.808 INFO:tasks.rados.rados.0.smithi165.stdout:3393: delete oid 15 current snap is 374 2024-09-10T16:28:21.811 INFO:tasks.rados.rados.0.smithi165.stdout:3393: done (2 left) 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:3394: rollback oid 24 current snap is 374 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 350 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:3395: read oid 40 snap -1 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:3395: expect deleted 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:3396: delete oid 48 current snap is 374 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:3394: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:28:21.812 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-10T16:28:21.814 INFO:tasks.rados.rados.0.smithi165.stdout:3390: finishing copy_from racing read to smithi16534841-50 2024-09-10T16:28:21.819 INFO:tasks.rados.rados.0.smithi165.stdout:3394: done (4 left) 2024-09-10T16:28:21.819 INFO:tasks.rados.rados.0.smithi165.stdout:3395: done (3 left) 2024-09-10T16:28:21.819 INFO:tasks.rados.rados.0.smithi165.stdout:3396: done (2 left) 2024-09-10T16:28:21.819 INFO:tasks.rados.rados.0.smithi165.stdout:3397: write oid 11 current snap is 374 2024-09-10T16:28:21.819 INFO:tasks.rados.rados.0.smithi165.stdout:3397: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2024-09-10T16:28:21.827 INFO:tasks.rados.rados.0.smithi165.stdout:3397: writing smithi16534841-11 from 749909 to 1421540 tid 1 2024-09-10T16:28:21.834 INFO:tasks.rados.rados.0.smithi165.stdout:3397: writing smithi16534841-11 from 2110479 to 2728557 tid 2 2024-09-10T16:28:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:3397: writing smithi16534841-11 from 3163563 to 3678854 tid 3 2024-09-10T16:28:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:3398: rollback oid 14 current snap is 374 2024-09-10T16:28:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 366 2024-09-10T16:28:21.840 INFO:tasks.rados.rados.0.smithi165.stdout:3392: expect (ObjNum 1148 snap 359 seq_num 1148) 2024-09-10T16:28:21.867 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:21.906 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1668981 2024-09-10T16:28:21.930 INFO:tasks.rados.rados.0.smithi165.stdout:3390: finishing copy_from to smithi16534841-50 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1834 (ObjNum 1165 snap 365 seq_num 1165) dirty exists 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: finishing write tid 1 to smithi16534841-11 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: finishing write tid 2 to smithi16534841-11 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3398: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: finishing write tid 3 to smithi16534841-11 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3398: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: finishing write tid 4 to smithi16534841-11 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3398: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1844 (ObjNum 1116 snap 342 seq_num 1116) dirty exists 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: finishing write tid 5 to smithi16534841-11 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: oid 11 updating version 0 to 1880 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: oid 11 version 1880 is already newer than 1879 2024-09-10T16:28:21.931 INFO:tasks.rados.rados.0.smithi165.stdout:3397: oid 11 updating version 1880 to 1881 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3397: oid 11 version 1881 is already newer than 1878 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1881 (ObjNum 1187 snap 374 seq_num 1187) dirty exists 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3397: left oid 11 (ObjNum 1187 snap 374 seq_num 1187) 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3390: done (3 left) 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3392: done (2 left) 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3397: done (1 left) 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3398: done (0 left) 2024-09-10T16:28:21.932 INFO:tasks.rados.rados.0.smithi165.stdout:3399: snap_create 2024-09-10T16:28:22.801 INFO:tasks.rados.rados.0.smithi165.stdout:3399: done (0 left) 2024-09-10T16:28:22.801 INFO:tasks.rados.rados.0.smithi165.stdout:3400: delete oid 47 current snap is 375 2024-09-10T16:28:22.807 INFO:tasks.rados.rados.0.smithi165.stdout:3400: done (0 left) 2024-09-10T16:28:22.807 INFO:tasks.rados.rados.0.smithi165.stdout:3401: snap_create 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3401: done (0 left) 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3402: read oid 33 snap -1 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3402: expect deleted 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3403: read oid 11 snap -1 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3403: expect (ObjNum 1187 snap 374 seq_num 1187) 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3404: copy_from oid 10 from oid 23 current snap is 376 2024-09-10T16:28:23.807 INFO:tasks.rados.rados.0.smithi165.stdout:3405: snap_remove snap 375 2024-09-10T16:28:24.738 INFO:tasks.rados.rados.0.smithi165.stdout:3405: done (3 left) 2024-09-10T16:28:24.738 INFO:tasks.rados.rados.0.smithi165.stdout:3406: delete oid 34 current snap is 376 2024-09-10T16:28:24.739 INFO:tasks.rados.rados.0.smithi165.stdout:3404: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:28:24.739 INFO:tasks.rados.rados.0.smithi165.stdout:3403: expect (ObjNum 1187 snap 374 seq_num 1187) 2024-09-10T16:28:24.739 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:24.787 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1550032 2024-09-10T16:28:24.861 INFO:tasks.rados.rados.0.smithi165.stdout:3404: finishing copy_from to smithi16534841-10 2024-09-10T16:28:24.861 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1695 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3402: done (3 left) 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3403: done (2 left) 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3404: done (1 left) 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3406: done (0 left) 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3407: read oid 5 snap -1 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3407: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3408: write oid 29 current snap is 376 2024-09-10T16:28:24.862 INFO:tasks.rados.rados.0.smithi165.stdout:3408: seq_num 1190 ranges {400330=497378,1485925=514405} 2024-09-10T16:28:24.866 INFO:tasks.rados.rados.0.smithi165.stdout:3408: writing smithi16534841-29 from 400330 to 897708 tid 1 2024-09-10T16:28:24.872 INFO:tasks.rados.rados.0.smithi165.stdout:3408: writing smithi16534841-29 from 1485925 to 2000330 tid 2 2024-09-10T16:28:24.872 INFO:tasks.rados.rados.0.smithi165.stdout:3409: read oid 40 snap -1 2024-09-10T16:28:24.873 INFO:tasks.rados.rados.0.smithi165.stdout:3409: expect deleted 2024-09-10T16:28:24.873 INFO:tasks.rados.rados.0.smithi165.stdout:3410: write oid 6 current snap is 376 2024-09-10T16:28:24.873 INFO:tasks.rados.rados.0.smithi165.stdout:3410: seq_num 1191 ranges {417137=572627,1428054=589083} 2024-09-10T16:28:24.878 INFO:tasks.rados.rados.0.smithi165.stdout:3410: writing smithi16534841-6 from 417137 to 989764 tid 1 2024-09-10T16:28:24.884 INFO:tasks.rados.rados.0.smithi165.stdout:3410: writing smithi16534841-6 from 1428054 to 2017137 tid 2 2024-09-10T16:28:24.884 INFO:tasks.rados.rados.0.smithi165.stdout:3411: read oid 31 snap -1 2024-09-10T16:28:24.884 INFO:tasks.rados.rados.0.smithi165.stdout:3411: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:28:24.885 INFO:tasks.rados.rados.0.smithi165.stdout:3412: snap_create 2024-09-10T16:28:24.885 INFO:tasks.rados.rados.0.smithi165.stdout:3408: finishing write tid 1 to smithi16534841-29 2024-09-10T16:28:24.885 INFO:tasks.rados.rados.0.smithi165.stdout:3407: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:24.885 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: finishing write tid 2 to smithi16534841-29 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: finishing write tid 3 to smithi16534841-29 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: finishing write tid 4 to smithi16534841-29 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: oid 29 updating version 0 to 1822 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: oid 29 updating version 1822 to 1824 2024-09-10T16:28:25.085 INFO:tasks.rados.rados.0.smithi165.stdout:3408: oid 29 version 1824 is already newer than 1823 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1824 (ObjNum 1190 snap 376 seq_num 1190) dirty exists 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3408: left oid 29 (ObjNum 1190 snap 376 seq_num 1190) 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: finishing write tid 1 to smithi16534841-6 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: finishing write tid 2 to smithi16534841-6 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: finishing write tid 3 to smithi16534841-6 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: finishing write tid 4 to smithi16534841-6 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: oid 6 updating version 0 to 1628 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: oid 6 version 1628 is already newer than 1626 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: oid 6 version 1628 is already newer than 1627 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1628 (ObjNum 1191 snap 376 seq_num 1191) dirty exists 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3410: left oid 6 (ObjNum 1191 snap 376 seq_num 1191) 2024-09-10T16:28:25.086 INFO:tasks.rados.rados.0.smithi165.stdout:3411: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:28:25.088 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1790530 2024-09-10T16:28:25.113 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:25.812 INFO:tasks.rados.rados.0.smithi165.stdout:3407: done (5 left) 2024-09-10T16:28:25.812 INFO:tasks.rados.rados.0.smithi165.stdout:3408: done (4 left) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3409: done (3 left) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3410: done (2 left) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3411: done (1 left) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3412: done (0 left) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3413: read oid 21 snap 362 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3413: expect (ObjNum 1147 snap 357 seq_num 1147) 2024-09-10T16:28:25.813 INFO:tasks.rados.rados.0.smithi165.stdout:3414: read oid 12 snap -1 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3414: expect (ObjNum 1103 snap 336 seq_num 1103) 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3415: rollback oid 15 current snap is 377 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 374 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3416: read oid 20 snap 351 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3416: expect deleted 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3417: write oid 25 current snap is 377 2024-09-10T16:28:25.814 INFO:tasks.rados.rados.0.smithi165.stdout:3417: seq_num 1192 ranges {433944=647876,1486536=547408} 2024-09-10T16:28:25.818 INFO:tasks.rados.rados.0.smithi165.stdout:3417: writing smithi16534841-25 from 433944 to 1081820 tid 1 2024-09-10T16:28:25.825 INFO:tasks.rados.rados.0.smithi165.stdout:3417: writing smithi16534841-25 from 1486536 to 2033944 tid 2 2024-09-10T16:28:25.825 INFO:tasks.rados.rados.0.smithi165.stdout:3418: read oid 29 snap -1 2024-09-10T16:28:25.826 INFO:tasks.rados.rados.0.smithi165.stdout:3418: expect (ObjNum 1190 snap 376 seq_num 1190) 2024-09-10T16:28:25.826 INFO:tasks.rados.rados.0.smithi165.stdout:3419: read oid 38 snap -1 2024-09-10T16:28:25.826 INFO:tasks.rados.rados.0.smithi165.stdout:3419: expect (ObjNum 1181 snap 373 seq_num 1181) 2024-09-10T16:28:25.826 INFO:tasks.rados.rados.0.smithi165.stdout:3415: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:28:25.826 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1444 (ObjNum 49 snap 0 seq_num 1127808944) dirty dne 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3420: read oid 31 snap -1 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3420: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3415: done (7 left) 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3421: read oid 24 snap -1 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3421: expect deleted 2024-09-10T16:28:25.827 INFO:tasks.rados.rados.0.smithi165.stdout:3422: delete oid 35 current snap is 377 2024-09-10T16:28:25.828 INFO:tasks.rados.rados.0.smithi165.stdout:3413: expect (ObjNum 1147 snap 357 seq_num 1147) 2024-09-10T16:28:25.924 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2589051 2024-09-10T16:28:25.981 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2807138 2024-09-10T16:28:25.991 INFO:tasks.rados.rados.0.smithi165.stdout:3414: expect (ObjNum 1103 snap 336 seq_num 1103) 2024-09-10T16:28:26.000 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1130697 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: finishing write tid 1 to smithi16534841-25 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: finishing write tid 2 to smithi16534841-25 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: finishing write tid 3 to smithi16534841-25 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: finishing write tid 4 to smithi16534841-25 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: oid 25 updating version 0 to 1863 2024-09-10T16:28:26.205 INFO:tasks.rados.rados.0.smithi165.stdout:3417: oid 25 version 1863 is already newer than 1862 2024-09-10T16:28:26.206 INFO:tasks.rados.rados.0.smithi165.stdout:3417: oid 25 updating version 1863 to 1864 2024-09-10T16:28:26.206 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1864 (ObjNum 1192 snap 377 seq_num 1192) dirty exists 2024-09-10T16:28:26.206 INFO:tasks.rados.rados.0.smithi165.stdout:3417: left oid 25 (ObjNum 1192 snap 377 seq_num 1192) 2024-09-10T16:28:26.206 INFO:tasks.rados.rados.0.smithi165.stdout:3418: expect (ObjNum 1190 snap 376 seq_num 1190) 2024-09-10T16:28:26.206 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 83315 2024-09-10T16:28:26.268 INFO:tasks.rados.rados.0.smithi165.stdout:3420: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:28:26.274 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1122347 2024-09-10T16:28:26.311 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2595860 2024-09-10T16:28:26.364 INFO:tasks.rados.rados.0.smithi165.stdout:3419: expect (ObjNum 1181 snap 373 seq_num 1181) 2024-09-10T16:28:26.396 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1560665 2024-09-10T16:28:26.424 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:3413: done (8 left) 2024-09-10T16:28:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:3414: done (7 left) 2024-09-10T16:28:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:3416: done (6 left) 2024-09-10T16:28:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:3417: done (5 left) 2024-09-10T16:28:26.467 INFO:tasks.rados.rados.0.smithi165.stdout:3418: done (4 left) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3419: done (3 left) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3420: done (2 left) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3421: done (1 left) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3422: done (0 left) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3423: read oid 17 snap -1 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3423: expect (ObjNum 1097 snap 333 seq_num 1097) 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3424: copy_from oid 36 from oid 33 current snap is 377 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3425: copy_from oid 45 from oid 20 current snap is 377 2024-09-10T16:28:26.468 INFO:tasks.rados.rados.0.smithi165.stdout:3426: snap_remove snap 367 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3426: done (3 left) 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3427: copy_from oid 43 from oid 34 current snap is 377 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3424: finishing copy_from to smithi16534841-36 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3424: got expected ENOENT (src dne) 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3424: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:28:26.816 INFO:tasks.rados.rados.0.smithi165.stdout:3423: expect (ObjNum 1097 snap 333 seq_num 1097) 2024-09-10T16:28:26.817 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:26.873 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:26.973 INFO:tasks.rados.rados.0.smithi165.stdout:3425: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:28:26.973 INFO:tasks.rados.rados.0.smithi165.stdout:3425: finishing copy_from to smithi16534841-45 2024-09-10T16:28:26.973 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1644 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:28:26.973 INFO:tasks.rados.rados.0.smithi165.stdout:3427: finishing copy_from to smithi16534841-43 2024-09-10T16:28:26.973 INFO:tasks.rados.rados.0.smithi165.stdout:3427: got expected ENOENT (src dne) 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3427: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3423: done (3 left) 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3424: done (2 left) 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3425: done (1 left) 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3427: done (0 left) 2024-09-10T16:28:26.974 INFO:tasks.rados.rados.0.smithi165.stdout:3428: delete oid 13 current snap is 377 2024-09-10T16:28:26.976 INFO:tasks.rados.rados.0.smithi165.stdout:3428: done (0 left) 2024-09-10T16:28:26.976 INFO:tasks.rados.rados.0.smithi165.stdout:3429: read oid 9 snap -1 2024-09-10T16:28:26.976 INFO:tasks.rados.rados.0.smithi165.stdout:3429: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:28:26.976 INFO:tasks.rados.rados.0.smithi165.stdout:3430: delete oid 36 current snap is 377 2024-09-10T16:28:26.978 INFO:tasks.rados.rados.0.smithi165.stdout:3430: done (1 left) 2024-09-10T16:28:26.978 INFO:tasks.rados.rados.0.smithi165.stdout:3431: read oid 5 snap -1 2024-09-10T16:28:26.978 INFO:tasks.rados.rados.0.smithi165.stdout:3431: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:26.978 INFO:tasks.rados.rados.0.smithi165.stdout:3432: snap_remove snap 353 2024-09-10T16:28:27.821 INFO:tasks.rados.rados.0.smithi165.stdout:3432: done (2 left) 2024-09-10T16:28:27.821 INFO:tasks.rados.rados.0.smithi165.stdout:3433: snap_remove snap 361 2024-09-10T16:28:28.826 INFO:tasks.rados.rados.0.smithi165.stdout:3433: done (2 left) 2024-09-10T16:28:28.826 INFO:tasks.rados.rados.0.smithi165.stdout:3434: read oid 43 snap 373 2024-09-10T16:28:28.826 INFO:tasks.rados.rados.0.smithi165.stdout:3434: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:28:28.826 INFO:tasks.rados.rados.0.smithi165.stdout:3435: snap_create 2024-09-10T16:28:28.826 INFO:tasks.rados.rados.0.smithi165.stdout:3429: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:28:28.892 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 504647 2024-09-10T16:28:28.969 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1266925 2024-09-10T16:28:29.019 INFO:tasks.rados.rados.0.smithi165.stdout:3431: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:29.020 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:29.228 INFO:tasks.rados.rados.0.smithi165.stdout:3434: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:28:29.228 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:29.263 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:29.299 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:29.741 INFO:tasks.rados.rados.0.smithi165.stdout:3429: done (3 left) 2024-09-10T16:28:29.742 INFO:tasks.rados.rados.0.smithi165.stdout:3431: done (2 left) 2024-09-10T16:28:29.742 INFO:tasks.rados.rados.0.smithi165.stdout:3434: done (1 left) 2024-09-10T16:28:29.742 INFO:tasks.rados.rados.0.smithi165.stdout:3435: done (0 left) 2024-09-10T16:28:29.742 INFO:tasks.rados.rados.0.smithi165.stdout:3436: snap_create 2024-09-10T16:28:30.741 INFO:tasks.rados.rados.0.smithi165.stdout:3436: done (0 left) 2024-09-10T16:28:30.741 INFO:tasks.rados.rados.0.smithi165.stdout:3437: write (excl) oid 33 current snap is 379 2024-09-10T16:28:30.741 INFO:tasks.rados.rados.0.smithi165.stdout:3437: seq_num 1196 ranges {501172=665225,1720464=380708} 2024-09-10T16:28:30.747 INFO:tasks.rados.rados.0.smithi165.stdout:3437: writing smithi16534841-33 from 501172 to 1166397 tid 1 2024-09-10T16:28:30.753 INFO:tasks.rados.rados.0.smithi165.stdout:3437: writing smithi16534841-33 from 1720464 to 2101172 tid 2 2024-09-10T16:28:30.753 INFO:tasks.rados.rados.0.smithi165.stdout:3438: snap_create 2024-09-10T16:28:30.757 INFO:tasks.rados.rados.0.smithi165.stdout:3437: finishing write tid 1 to smithi16534841-33 2024-09-10T16:28:30.757 INFO:tasks.rados.rados.0.smithi165.stdout:3437: finishing write tid 2 to smithi16534841-33 2024-09-10T16:28:30.757 INFO:tasks.rados.rados.0.smithi165.stdout:3437: finishing write tid 3 to smithi16534841-33 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:3437: finishing write tid 4 to smithi16534841-33 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:3437: oid 33 updating version 0 to 1866 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:3437: oid 33 updating version 1866 to 1867 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:3437: oid 33 version 1867 is already newer than 1865 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1867 (ObjNum 1196 snap 379 seq_num 1196) dirty exists 2024-09-10T16:28:30.758 INFO:tasks.rados.rados.0.smithi165.stdout:3437: left oid 33 (ObjNum 1196 snap 379 seq_num 1196) 2024-09-10T16:28:31.257 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:28:31.257 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:28:31.257 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:28:31.261 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:28:31.261 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:28:31.261 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:31.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.368+0000 7f1f39bc6640 10 monclient: get_monmap_and_config 2024-09-10T16:28:31.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.368+0000 7f1f39bc6640 10 monclient: build_initial_monmap 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.368+0000 7f1f39bc6640 10 monclient: monmap: 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:31.369231+0000 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:31.369231+0000 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:31.370 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:31.371 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 Processor -- start 2024-09-10T16:28:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 -- start start 2024-09-10T16:28:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient: init 2024-09-10T16:28:31.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-b con 0x7f1f34074af0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f340a1090 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f1f340a14d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f3409ebc0 con 0x7f1f340a1090 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1f340698a0 con 0x7f1f340a14d0 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1f3406bfb0 con 0x7f1f34074af0 2024-09-10T16:28:31.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f39bc6640 10 monclient(hunting): authenticate will time out at 2089.856790s 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f33fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33310/0 (socket says 172.21.15.164:33310) 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 1 -- 172.21.15.164:0/2958137732 learned_addr learned my addr 172.21.15.164:0/2958137732 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 10 monclient(hunting): get_auth_request con 0x7f1f34074af0 auth_method 0 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f33fff640 10 monclient(hunting): get_auth_request con 0x7f1f340a14d0 auth_method 0 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:31.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.369+0000 7f1f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f33fff640 1 -- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 msgr2=0x7f1f340a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f33fff640 1 -- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 msgr2=0x7f1f340a35d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f33fff640 1 --2- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f33fff640 1 --2- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient(hunting): handle_auth_done global_id 244172 payload 274 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 1 -- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 msgr2=0x7f1f340a35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 1 --2- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 1 -- 172.21.15.164:0/2958137732 >> v1:172.21.15.165:6789/0 conn(0x7f1f340a1090 legacy=0x7f1f340a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient: _finish_hunting 0 2024-09-10T16:28:31.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 1 monclient: found mon.noname-b 2024-09-10T16:28:31.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:31.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.377 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.370+0000 7f1f337fe640 1 -- 172.21.15.164:0/2958137732 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f340a1940 con 0x7f1f34074af0 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.372+0000 7f1f337fe640 1 -- 172.21.15.164:0/2958137732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 msgr2=0x7f1f34137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.372+0000 7f1f337fe640 1 --2- 172.21.15.164:0/2958137732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009870 tx=0x7f1f2402a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbc13bf0a78a7472 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 1 -- 172.21.15.164:0/2958137732 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24037070 con 0x7f1f34074af0 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: dump: 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:31.385 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: _finish_auth 0 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: _check_auth_tickets 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 1 -- 172.21.15.164:0/2958137732 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f24032440 con 0x7f1f34074af0 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 5 monclient: authenticate success, global_id 244172 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 1 -- 172.21.15.164:0/2958137732 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f24032810 con 0x7f1f34074af0 2024-09-10T16:28:31.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 4 set_mon_vals no callback set 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f327fc640 10 monclient: dump: 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.388 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 10 monclient: shutdown 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:31.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 -- 172.21.15.164:0/2958137732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 msgr2=0x7f1f34137e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/2958137732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1f24009870 tx=0x7f1f2402a0a0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 -- 172.21.15.164:0/2958137732 shutdown_connections 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/2958137732 >> v2:172.21.15.164:3300/0 conn(0x7f1f340a14d0 0x7f1f340a35d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/2958137732 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f34074af0 0x7f1f34137e70 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 -- 172.21.15.164:0/2958137732 >> 172.21.15.164:0/2958137732 conn(0x7f1f340a3fb0 msgr2=0x7f1f340a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 -- 172.21.15.164:0/2958137732 shutdown_connections 2024-09-10T16:28:31.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.373+0000 7f1f39bc6640 1 -- 172.21.15.164:0/2958137732 wait complete. 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.374+0000 7f1f39bc6640 10 monclient: build_initial_monmap 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.374+0000 7f1f39bc6640 10 monclient: monmap: 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:31.391 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.374+0000 7f1f39bc6640 1 Processor -- start 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.374+0000 7f1f39bc6640 1 -- start start 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.374+0000 7f1f39bc6640 10 monclient: init 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:28:31.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 0x7f1f3409b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-b con 0x7f1f34074af0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-a con 0x7f1f340a1090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): picked mon.noname-c con 0x7f1f340a14d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1f34069370 con 0x7f1f340a1090 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1f34074ed0 con 0x7f1f34074af0 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1f340a3b10 con 0x7f1f340a14d0 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f39bc6640 10 monclient(hunting): authenticate will time out at 2089.862407s 2024-09-10T16:28:31.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33320/0 (socket says 172.21.15.164:33320) 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 -- 172.21.15.164:0/496650392 learned_addr learned my addr 172.21.15.164:0/496650392 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): get_auth_request con 0x7f1f340a14d0 auth_method 0 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 10 monclient(hunting): get_auth_request con 0x7f1f340a1090 auth_method 0 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:31.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 msgr2=0x7f1f3409bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 msgr2=0x7f1f3409bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f32ffd640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 0x7f1f3409b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 10 monclient(hunting): get_auth_request con 0x7f1f34074af0 auth_method 0 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient(hunting): handle_auth_done global_id 244175 payload 824 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 msgr2=0x7f1f3409b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 0x7f1f3409b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 msgr2=0x7f1f3409bd70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient: _finish_hunting 0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 monclient: found mon.noname-c 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f33fff640 1 -- 172.21.15.164:0/496650392 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f34132390 con 0x7f1f340a14d0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.375+0000 7f1f337fe640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 0x7f1f3409b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f33fff640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 msgr2=0x7f1f3409c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f33fff640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c007e60 tx=0x7f1f2c00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c013790 con 0x7f1f340a14d0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:31.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: dump: 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: _finish_auth 0 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:31.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f2c019070 con 0x7f1f340a14d0 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 5 monclient: authenticate success, global_id 244175 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 10 monclient: _renew_subs 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f3412d3c0 con 0x7f1f340a14d0 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c013b70 con 0x7f1f340a14d0 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f1bfff640 10 monclient: dump: 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.398 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 10 monclient: _renew_subs 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f31ffb640 4 set_mon_vals no callback set 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.378+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f3409e6a0 con 0x7f1f340a14d0 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f39bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f39bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f39bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1f2c01e460 con 0x7f1f340a14d0 2024-09-10T16:28:31.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f3409d040 con 0x7f1f340a14d0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f1bfff640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 0x7f1f200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(765..765 src has 677..765) ==== 4787+0+0 (secure 0 0 0) 0x7f1f2c053660 con 0x7f1f340a14d0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f337fe640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 0x7f1f200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.379+0000 7f1f337fe640 10 monclient: get_auth_request con 0x7f1f2003e210 auth_method 0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.382+0000 7f1f337fe640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 0x7f1f200406d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406f200 tx=0x7f1f2402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.384+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1f2c052050 con 0x7f1f340a14d0 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.384+0000 7f1f1bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:31.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.384+0000 7f1f1bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f39bc6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f39bc6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f39bc6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1f34137df0 con 0x7f1f340a14d0 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f1bfff640 1 -- 172.21.15.164:0/496650392 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f1f2c013d90 con 0x7f1f340a14d0 2024-09-10T16:28:31.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f1bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:31.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.627+0000 7f1f1bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:31.630 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:31.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 msgr2=0x7f1f200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:31.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 0x7f1f200406d0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1f3406f200 tx=0x7f1f2402e000 comp rx=0 tx=0).stop 2024-09-10T16:28:31.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 10 monclient: shutdown 2024-09-10T16:28:31.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:31.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 msgr2=0x7f1f3409c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:31.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.629+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c007e60 tx=0x7f1f2c00e4b0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 shutdown_connections 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1f2003e210 0x7f1f200406d0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1f340a14d0 0x7f1f3409c2b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1f340a1090 0x7f1f3409bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 --2- 172.21.15.164:0/496650392 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1f34074af0 0x7f1f3409b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 >> 172.21.15.164:0/496650392 conn(0x7f1f340a3fb0 msgr2=0x7f1f34098230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 shutdown_connections 2024-09-10T16:28:31.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:31.630+0000 7f1f39bc6640 1 -- 172.21.15.164:0/496650392 wait complete. 2024-09-10T16:28:31.743 INFO:tasks.rados.rados.0.smithi165.stdout:3437: done (1 left) 2024-09-10T16:28:31.743 INFO:tasks.rados.rados.0.smithi165.stdout:3438: done (0 left) 2024-09-10T16:28:31.743 INFO:tasks.rados.rados.0.smithi165.stdout:3439: copy_from oid 35 from oid 1 current snap is 380 2024-09-10T16:28:31.744 INFO:tasks.rados.rados.0.smithi165.stdout:3440: copy_from oid 34 from oid 4 current snap is 380 2024-09-10T16:28:31.744 INFO:tasks.rados.rados.0.smithi165.stdout:3441: delete oid 6 current snap is 380 2024-09-10T16:28:31.747 INFO:tasks.rados.rados.0.smithi165.stdout:3441: done (2 left) 2024-09-10T16:28:31.747 INFO:tasks.rados.rados.0.smithi165.stdout:3442: delete oid 17 current snap is 380 2024-09-10T16:28:31.750 INFO:tasks.rados.rados.0.smithi165.stdout:3442: done (2 left) 2024-09-10T16:28:31.750 INFO:tasks.rados.rados.0.smithi165.stdout:3443: write (excl) oid 33 current snap is 380 2024-09-10T16:28:31.751 INFO:tasks.rados.rados.0.smithi165.stdout:3443: seq_num 1199 ranges {551593=490972,1779557=372036} 2024-09-10T16:28:31.757 INFO:tasks.rados.rados.0.smithi165.stdout:3443: writing smithi16534841-33 from 551593 to 1042565 tid 1 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3443: writing smithi16534841-33 from 1779557 to 2151593 tid 2 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3444: copy_from oid 30 from oid 3 current snap is 380 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3440: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3440: finishing copy_from to smithi16534841-34 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1451 (ObjNum 1161 snap 365 seq_num 1161) dirty exists 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3439: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3440: done (3 left) 2024-09-10T16:28:31.764 INFO:tasks.rados.rados.0.smithi165.stdout:3445: snap_remove snap 363 2024-09-10T16:28:32.745 INFO:tasks.rados.rados.0.smithi165.stdout:3445: done (3 left) 2024-09-10T16:28:32.745 INFO:tasks.rados.rados.0.smithi165.stdout:3446: write oid 7 current snap is 380 2024-09-10T16:28:32.745 INFO:tasks.rados.rados.0.smithi165.stdout:3446: seq_num 1200 ranges {568400=566221,1838039=330361} 2024-09-10T16:28:32.751 INFO:tasks.rados.rados.0.smithi165.stdout:3446: writing smithi16534841-7 from 568400 to 1134621 tid 1 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3446: writing smithi16534841-7 from 1838039 to 2168400 tid 2 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3447: snap_create 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3439: finishing copy_from to smithi16534841-35 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2007 (ObjNum 1170 snap 368 seq_num 1170) dirty exists 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3443: finishing write tid 1 to smithi16534841-33 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3443: finishing write tid 2 to smithi16534841-33 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3443: finishing write tid 3 to smithi16534841-33 2024-09-10T16:28:32.758 INFO:tasks.rados.rados.0.smithi165.stdout:3443: finishing write tid 4 to smithi16534841-33 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3443: oid 33 updating version 0 to 1870 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3443: oid 33 version 1870 is already newer than 1868 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3443: oid 33 version 1870 is already newer than 1869 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1870 (ObjNum 1199 snap 380 seq_num 1199) dirty exists 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3443: left oid 33 (ObjNum 1199 snap 380 seq_num 1199) 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3444: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3444: finishing copy_from to smithi16534841-30 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:28:32.759 INFO:tasks.rados.rados.0.smithi165.stdout:3446: finishing write tid 1 to smithi16534841-7 2024-09-10T16:28:32.763 INFO:tasks.rados.rados.0.smithi165.stdout:3446: finishing write tid 2 to smithi16534841-7 2024-09-10T16:28:32.763 INFO:tasks.rados.rados.0.smithi165.stdout:3446: finishing write tid 3 to smithi16534841-7 2024-09-10T16:28:32.763 INFO:tasks.rados.rados.0.smithi165.stdout:3446: finishing write tid 4 to smithi16534841-7 2024-09-10T16:28:32.764 INFO:tasks.rados.rados.0.smithi165.stdout:3446: oid 7 updating version 0 to 2130 2024-09-10T16:28:32.764 INFO:tasks.rados.rados.0.smithi165.stdout:3446: oid 7 updating version 2130 to 2136 2024-09-10T16:28:32.764 INFO:tasks.rados.rados.0.smithi165.stdout:3446: oid 7 version 2136 is already newer than 2135 2024-09-10T16:28:32.764 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2136 (ObjNum 1200 snap 380 seq_num 1200) dirty exists 2024-09-10T16:28:32.764 INFO:tasks.rados.rados.0.smithi165.stdout:3446: left oid 7 (ObjNum 1200 snap 380 seq_num 1200) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3439: done (4 left) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3443: done (3 left) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3444: done (2 left) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3446: done (1 left) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3447: done (0 left) 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3448: rollback oid 42 current snap is 381 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 42 to 374 2024-09-10T16:28:33.745 INFO:tasks.rados.rados.0.smithi165.stdout:3449: snap_create 2024-09-10T16:28:33.890 INFO:tasks.rados.rados.0.smithi165.stdout:3448: finishing rollback tid 0 to smithi16534841-42 2024-09-10T16:28:33.890 INFO:tasks.rados.rados.0.smithi165.stdout:3448: finishing rollback tid 1 to smithi16534841-42 2024-09-10T16:28:33.890 INFO:tasks.rados.rados.0.smithi165.stdout:3448: finishing rollback tid 2 to smithi16534841-42 2024-09-10T16:28:33.890 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1884 (ObjNum 1155 snap 363 seq_num 1155) dirty exists 2024-09-10T16:28:34.642 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3448: done (1 left) 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3449: done (0 left) 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3450: read oid 13 snap 372 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3450: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3451: write (excl) oid 27 current snap is 382 2024-09-10T16:28:34.743 INFO:tasks.rados.rados.0.smithi165.stdout:3451: seq_num 1201 ranges {585207=641470,1896521=288686} 2024-09-10T16:28:34.749 INFO:tasks.rados.rados.0.smithi165.stdout:3451: writing smithi16534841-27 from 585207 to 1226677 tid 1 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.752+0000 7fc5a6d24640 10 monclient: get_monmap_and_config 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.752+0000 7fc5a6d24640 10 monclient: build_initial_monmap 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.752+0000 7fc5a6d24640 10 monclient: monmap: 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:34.753770+0000 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:34.753770+0000 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:34.754 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:34.755 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:34.755 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.756 INFO:tasks.rados.rados.0.smithi165.stdout:3451: writing smithi16534841-27 from 1896521 to 2185207 tid 2 2024-09-10T16:28:34.756 INFO:tasks.rados.rados.0.smithi165.stdout:3452: snap_remove snap 351 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.753+0000 7fc5a6d24640 1 Processor -- start 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.753+0000 7fc5a6d24640 1 -- start start 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.753+0000 7fc5a6d24640 10 monclient: init 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 0x7fc5a0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc5a00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc5a00a4a80 0x7fc5a01383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a00a4a80 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5a009ce50 con 0x7fc5a00a46a0 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc5a006db60 con 0x7fc5a00a4a80 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a00a4f10 con 0x7fc5a006cb70 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a6d24640 10 monclient(hunting): authenticate will time out at 2093.241808s 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc59ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc5a00a46a0 0x7fc5a0137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39212/0 (socket says 172.21.15.164:39212) 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc59ffff640 1 -- 172.21.15.164:0/4243493668 learned_addr learned my addr 172.21.15.164:0/4243493668 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a4ac0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 0x7fc5a0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a4ac0640 10 monclient(hunting): get_auth_request con 0x7fc5a006cb70 auth_method 0 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a4ac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:34.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a4ac0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.754+0000 7fc5a4ac0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a52c1640 1 --2- 172.21.15.164:0/4243493668 >> v2:172.21.15.165:3300/0 conn(0x7fc5a00a4a80 0x7fc5a01383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_done global_id 244178 payload 274 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 1 -- 172.21.15.164:0/4243493668 >> v2:172.21.15.165:3300/0 conn(0x7fc5a00a4a80 msgr2=0x7fc5a01383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 1 --2- 172.21.15.164:0/4243493668 >> v2:172.21.15.165:3300/0 conn(0x7fc5a00a4a80 0x7fc5a01383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 1 -- 172.21.15.164:0/4243493668 >> v1:172.21.15.164:6789/0 conn(0x7fc5a00a46a0 legacy=0x7fc5a0137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient: _finish_hunting 0 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 1 monclient: found mon.noname-b 2024-09-10T16:28:34.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:34.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:34.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.755+0000 7fc5a4ac0640 1 -- 172.21.15.164:0/4243493668 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a009cb70 con 0x7fc5a006cb70 2024-09-10T16:28:34.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a4ac0640 1 -- 172.21.15.164:0/4243493668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 msgr2=0x7fc5a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:34.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a4ac0640 1 --2- 172.21.15.164:0/4243493668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 0x7fc5a0134f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc588029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=362a0e32ad6cee9a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 1 -- 172.21.15.164:0/4243493668 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588037070 con 0x7fc5a006cb70 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: dump: 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:34.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a6d24640 5 monclient: authenticate success, global_id 244178 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 1 -- 172.21.15.164:0/4243493668 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc588032640 con 0x7fc5a006cb70 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a6d24640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a6d24640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc5a6d24640 4 set_mon_vals no callback set 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 1 -- 172.21.15.164:0/4243493668 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc588032a10 con 0x7fc5a006cb70 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:34.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.758+0000 7fc59f7fe640 10 monclient: dump: 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 10 monclient: shutdown 2024-09-10T16:28:34.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 -- 172.21.15.164:0/4243493668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 msgr2=0x7fc5a0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/4243493668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 0x7fc5a0134f70 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc5880097a0 tx=0x7fc588029fd0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 -- 172.21.15.164:0/4243493668 shutdown_connections 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/4243493668 >> v2:172.21.15.165:3300/0 conn(0x7fc5a00a4a80 0x7fc5a01383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/4243493668 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a006cb70 0x7fc5a0134f70 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 -- 172.21.15.164:0/4243493668 >> 172.21.15.164:0/4243493668 conn(0x7fc5a00aa900 msgr2=0x7fc5a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 -- 172.21.15.164:0/4243493668 shutdown_connections 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.759+0000 7fc5a6d24640 1 -- 172.21.15.164:0/4243493668 wait complete. 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.760+0000 7fc5a6d24640 10 monclient: build_initial_monmap 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.760+0000 7fc5a6d24640 10 monclient: monmap: 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:34.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.760+0000 7fc5a6d24640 1 Processor -- start 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.760+0000 7fc5a6d24640 1 -- start start 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.760+0000 7fc5a6d24640 10 monclient: init 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:28:34.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a006cb70 0x7fc5a0131760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-b con 0x7fc5a006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-c con 0x7fc5a00a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): picked mon.noname-a con 0x7fc5a00a4a80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a01388f0 con 0x7fc5a00a4a80 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc5a006cf50 con 0x7fc5a006cb70 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc5a0138a70 con 0x7fc5a00a46a0 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:34.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a6d24640 10 monclient(hunting): authenticate will time out at 2093.248474s 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46968/0 (socket says 172.21.15.164:46968) 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 -- 172.21.15.164:0/2093597689 learned_addr learned my addr 172.21.15.164:0/2093597689 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): get_auth_request con 0x7fc5a00a46a0 auth_method 0 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 10 monclient(hunting): get_auth_request con 0x7fc5a00a4a80 auth_method 0 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:34.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a006cb70 0x7fc5a0131760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): get_auth_request con 0x7fc5a006cb70 auth_method 0 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 msgr2=0x7fc5a009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 msgr2=0x7fc5a009d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a52c1640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:34.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc5a4ac0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient(hunting): handle_auth_done global_id 244181 payload 824 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a006cb70 msgr2=0x7fc5a0131760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a006cb70 0x7fc5a0131760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 msgr2=0x7fc5a009d630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.761+0000 7fc59ffff640 1 monclient: found mon.noname-c 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.762+0000 7fc59ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.762+0000 7fc59ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.762+0000 7fc59ffff640 1 -- 172.21.15.164:0/2093597689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5a009f570 con 0x7fc5a00a46a0 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59ffff640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 msgr2=0x7fc5a00a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:34.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59ffff640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc59000cc10 tx=0x7fc59000a430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc590012400 con 0x7fc5a00a46a0 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: dump: 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:34.777 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5900125e0 con 0x7fc5a00a46a0 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5900129d0 con 0x7fc5a00a46a0 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 5 monclient: authenticate success, global_id 244181 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 10 monclient: _renew_subs 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:34.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5a009fdf0 con 0x7fc5a00a46a0 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59d7fa640 10 monclient: dump: 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:34.779 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc59effd640 4 set_mon_vals no callback set 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 10 monclient: _renew_subs 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.764+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5a01cd830 con 0x7fc5a00a46a0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc5a6d24640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc5a6d24640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc5a6d24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5a0134f70 con 0x7fc5a00a46a0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc590012bf0 con 0x7fc5a00a46a0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc59d7fa640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 0x7fc580040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.765+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(769..769 src has 677..769) ==== 4815+0+0 (secure 0 0 0) 0x7fc590053490 con 0x7fc5a00a46a0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.766+0000 7fc5a4ac0640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 0x7fc580040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:34.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.766+0000 7fc5a4ac0640 10 monclient: get_auth_request con 0x7fc58003e260 auth_method 0 2024-09-10T16:28:34.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.768+0000 7fc5a4ac0640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 0x7fc580040720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc5a006d1b0 tx=0x7fc58802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:34.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.770+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc5a009fdf0 con 0x7fc5a00a46a0 2024-09-10T16:28:34.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.770+0000 7fc59d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:34.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:34.770+0000 7fc59d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:35.010 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc5a6d24640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc5a6d24640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc5a6d24640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc5a00a0750 con 0x7fc5a00a46a0 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc59d7fa640 1 -- 172.21.15.164:0/2093597689 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fc59001bc00 con 0x7fc5a00a46a0 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc59d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.009+0000 7fc59d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:35.011 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":210,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 msgr2=0x7fc580040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 0x7fc580040720 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fc5a006d1b0 tx=0x7fc58802e000 comp rx=0 tx=0).stop 2024-09-10T16:28:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 10 monclient: shutdown 2024-09-10T16:28:35.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 msgr2=0x7fc5a00a3be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.011+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc59000cc10 tx=0x7fc59000a430 comp rx=0 tx=0).stop 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 shutdown_connections 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc58003e260 0x7fc580040720 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc5a00a4a80 0x7fc5a009d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc5a00a46a0 0x7fc5a00a3be0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 --2- 172.21.15.164:0/2093597689 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc5a006cb70 0x7fc5a0131760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 >> 172.21.15.164:0/2093597689 conn(0x7fc5a00aa900 msgr2=0x7fc5a0132620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 shutdown_connections 2024-09-10T16:28:35.014 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:35.012+0000 7fc5a6d24640 1 -- 172.21.15.164:0/2093597689 wait complete. 2024-09-10T16:28:35.756 INFO:tasks.rados.rados.0.smithi165.stdout:3452: done (2 left) 2024-09-10T16:28:35.756 INFO:tasks.rados.rados.0.smithi165.stdout:3453: write (excl) oid 19 current snap is 382 2024-09-10T16:28:35.756 INFO:tasks.rados.rados.0.smithi165.stdout:3453: seq_num 1202 ranges {602014=433072,1671356=530658} 2024-09-10T16:28:35.761 INFO:tasks.rados.rados.0.smithi165.stdout:3453: writing smithi16534841-19 from 602014 to 1035086 tid 1 2024-09-10T16:28:35.769 INFO:tasks.rados.rados.0.smithi165.stdout:3453: writing smithi16534841-19 from 1671356 to 2202014 tid 2 2024-09-10T16:28:35.769 INFO:tasks.rados.rados.0.smithi165.stdout:3454: write (excl) oid 7 current snap is 382 2024-09-10T16:28:35.769 INFO:tasks.rados.rados.0.smithi165.stdout:3454: seq_num 1203 ranges {618821=508321,1613485=531077} 2024-09-10T16:28:35.775 INFO:tasks.rados.rados.0.smithi165.stdout:3454: writing smithi16534841-7 from 618821 to 1127142 tid 1 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:3454: writing smithi16534841-7 from 1613485 to 2144562 tid 2 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:3455: read oid 34 snap -1 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:3455: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:3456: copy_from oid 48 from oid 4 current snap is 382 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:3450: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:28:35.782 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:35.909 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:35.977 INFO:tasks.rados.rados.0.smithi165.stdout:3451: finishing write tid 1 to smithi16534841-27 2024-09-10T16:28:35.977 INFO:tasks.rados.rados.0.smithi165.stdout:3451: finishing write tid 2 to smithi16534841-27 2024-09-10T16:28:35.977 INFO:tasks.rados.rados.0.smithi165.stdout:3451: finishing write tid 3 to smithi16534841-27 2024-09-10T16:28:35.977 INFO:tasks.rados.rados.0.smithi165.stdout:3451: finishing write tid 4 to smithi16534841-27 2024-09-10T16:28:35.977 INFO:tasks.rados.rados.0.smithi165.stdout:3451: oid 27 updating version 0 to 1836 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3451: oid 27 version 1836 is already newer than 1835 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3451: oid 27 updating version 1836 to 1837 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1837 (ObjNum 1201 snap 382 seq_num 1201) dirty exists 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3451: left oid 27 (ObjNum 1201 snap 382 seq_num 1201) 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3450: done (5 left) 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3451: done (4 left) 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3457: snap_create 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: finishing write tid 1 to smithi16534841-19 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: finishing write tid 2 to smithi16534841-19 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: finishing write tid 3 to smithi16534841-19 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: finishing write tid 4 to smithi16534841-19 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: oid 19 updating version 0 to 2141 2024-09-10T16:28:35.978 INFO:tasks.rados.rados.0.smithi165.stdout:3453: oid 19 updating version 2141 to 2147 2024-09-10T16:28:35.979 INFO:tasks.rados.rados.0.smithi165.stdout:3453: oid 19 version 2147 is already newer than 2146 2024-09-10T16:28:35.979 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2147 (ObjNum 1202 snap 382 seq_num 1202) dirty exists 2024-09-10T16:28:35.979 INFO:tasks.rados.rados.0.smithi165.stdout:3453: left oid 19 (ObjNum 1202 snap 382 seq_num 1202) 2024-09-10T16:28:35.979 INFO:tasks.rados.rados.0.smithi165.stdout:3454: finishing write tid 1 to smithi16534841-7 2024-09-10T16:28:35.979 INFO:tasks.rados.rados.0.smithi165.stdout:3455: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:28:36.025 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 991822 2024-09-10T16:28:36.053 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 356967 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3456: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: finishing write tid 2 to smithi16534841-7 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: finishing write tid 3 to smithi16534841-7 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: finishing write tid 4 to smithi16534841-7 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: oid 7 updating version 0 to 2157 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: oid 7 version 2157 is already newer than 2153 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: oid 7 version 2157 is already newer than 2156 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2157 (ObjNum 1203 snap 382 seq_num 1203) dirty exists 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3454: left oid 7 (ObjNum 1203 snap 382 seq_num 1203) 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:3456: finishing copy_from to smithi16534841-48 2024-09-10T16:28:36.090 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 1451 (ObjNum 1161 snap 365 seq_num 1161) dirty exists 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3453: done (4 left) 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3454: done (3 left) 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3455: done (2 left) 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3456: done (1 left) 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3457: done (0 left) 2024-09-10T16:28:36.762 INFO:tasks.rados.rados.0.smithi165.stdout:3458: copy_from oid 45 from oid 46 current snap is 383 2024-09-10T16:28:36.763 INFO:tasks.rados.rados.0.smithi165.stdout:3459: read oid 43 snap -1 2024-09-10T16:28:36.763 INFO:tasks.rados.rados.0.smithi165.stdout:3459: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:28:36.763 INFO:tasks.rados.rados.0.smithi165.stdout:3460: delete oid 23 current snap is 383 2024-09-10T16:28:36.770 INFO:tasks.rados.rados.0.smithi165.stdout:3460: done (2 left) 2024-09-10T16:28:36.770 INFO:tasks.rados.rados.0.smithi165.stdout:3461: rollback oid 14 current snap is 383 2024-09-10T16:28:36.770 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 14 to 334 2024-09-10T16:28:36.770 INFO:tasks.rados.rados.0.smithi165.stdout:3458: finishing copy_from to smithi16534841-45 2024-09-10T16:28:36.771 INFO:tasks.rados.rados.0.smithi165.stdout:3458: got expected ENOENT (src dne) 2024-09-10T16:28:36.771 INFO:tasks.rados.rados.0.smithi165.stdout:3462: write oid 25 current snap is 383 2024-09-10T16:28:36.771 INFO:tasks.rados.rados.0.smithi165.stdout:3462: seq_num 1205 ranges {652435=658819,1730449=521986} 2024-09-10T16:28:36.776 INFO:tasks.rados.rados.0.smithi165.stdout:3462: writing smithi16534841-25 from 652435 to 1311254 tid 1 2024-09-10T16:28:36.782 INFO:tasks.rados.rados.0.smithi165.stdout:3462: writing smithi16534841-25 from 1730449 to 2252435 tid 2 2024-09-10T16:28:36.783 INFO:tasks.rados.rados.0.smithi165.stdout:3463: rollback oid 1 current snap is 383 2024-09-10T16:28:36.783 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 344 2024-09-10T16:28:36.783 INFO:tasks.rados.rados.0.smithi165.stdout:3458: finishing copy_from racing read to smithi16534841-45 2024-09-10T16:28:36.783 INFO:tasks.rados.rados.0.smithi165.stdout:3459: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:28:36.783 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:36.819 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:28:36.857 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1459990 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:3461: finishing rollback tid 0 to smithi16534841-14 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:3461: finishing rollback tid 1 to smithi16534841-14 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:3461: finishing rollback tid 2 to smithi16534841-14 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1847 (ObjNum 946 snap 285 seq_num 946) dirty exists 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:3462: finishing write tid 1 to smithi16534841-25 2024-09-10T16:28:36.882 INFO:tasks.rados.rados.0.smithi165.stdout:3462: finishing write tid 2 to smithi16534841-25 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: finishing write tid 3 to smithi16534841-25 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: finishing write tid 4 to smithi16534841-25 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: oid 25 updating version 0 to 1873 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: oid 25 version 1873 is already newer than 1871 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: oid 25 version 1873 is already newer than 1872 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1873 (ObjNum 1205 snap 383 seq_num 1205) dirty exists 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3462: left oid 25 (ObjNum 1205 snap 383 seq_num 1205) 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3463: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3463: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3463: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2014 (ObjNum 941 snap 285 seq_num 941) dirty exists 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3458: done (4 left) 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3459: done (3 left) 2024-09-10T16:28:36.883 INFO:tasks.rados.rados.0.smithi165.stdout:3461: done (2 left) 2024-09-10T16:28:36.884 INFO:tasks.rados.rados.0.smithi165.stdout:3462: done (1 left) 2024-09-10T16:28:36.884 INFO:tasks.rados.rados.0.smithi165.stdout:3463: done (0 left) 2024-09-10T16:28:36.884 INFO:tasks.rados.rados.0.smithi165.stdout:3464: write (excl) oid 8 current snap is 383 2024-09-10T16:28:36.884 INFO:tasks.rados.rados.0.smithi165.stdout:3464: seq_num 1206 ranges {669242=734068,2188931=80311} 2024-09-10T16:28:36.889 INFO:tasks.rados.rados.0.smithi165.stdout:3464: writing smithi16534841-8 from 669242 to 1403310 tid 1 2024-09-10T16:28:36.893 INFO:tasks.rados.rados.0.smithi165.stdout:3464: writing smithi16534841-8 from 2188931 to 2269242 tid 2 2024-09-10T16:28:36.894 INFO:tasks.rados.rados.0.smithi165.stdout:3465: snap_create 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: finishing write tid 1 to smithi16534841-8 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: finishing write tid 2 to smithi16534841-8 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: finishing write tid 3 to smithi16534841-8 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: finishing write tid 4 to smithi16534841-8 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: oid 8 updating version 0 to 1845 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: oid 8 version 1845 is already newer than 1843 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:3464: oid 8 version 1845 is already newer than 1844 2024-09-10T16:28:36.906 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1845 (ObjNum 1206 snap 383 seq_num 1206) dirty exists 2024-09-10T16:28:36.907 INFO:tasks.rados.rados.0.smithi165.stdout:3464: left oid 8 (ObjNum 1206 snap 383 seq_num 1206) 2024-09-10T16:28:38.025 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.131+0000 7f532cf65640 10 monclient: get_monmap_and_config 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.131+0000 7f532cf65640 10 monclient: build_initial_monmap 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.131+0000 7f532cf65640 10 monclient: monmap: 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:38.134 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:38.133729+0000 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:38.133729+0000 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:38.135 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.132+0000 7f532cf65640 1 Processor -- start 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.132+0000 7f532cf65640 1 -- start start 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.132+0000 7f532cf65640 10 monclient: init 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-b con 0x7f532809d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f532809ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:38.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f532809e250 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53280a0870 con 0x7f532809ab50 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53281353a0 con 0x7f532809e250 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f532809d930 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532cf65640 10 monclient(hunting): authenticate will time out at 2096.621285s 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f5325d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f532809ab50 0x7f5328134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:39216/0 (socket says 172.21.15.164:39216) 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 learned_addr learned my addr 172.21.15.164:0/2824321994 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f532809d930 auth_method 0 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:38.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.133+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.135+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f532809ab50 legacy=0x7f5328134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:38.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.135+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f532809e250 legacy=0x7f5328134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f532809d930 auth_method 0 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:38.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.334+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:38.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f532809d930 auth_method 0 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:38.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:38.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f532809d930 auth_method 0 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:39.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:39.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:39.538+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): tick 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 msgr2=0x7f532809dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f532809d930 0x7f532809dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f532809e250 legacy=0x7f5328134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f532809ab50 legacy=0x7f5328134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:41.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-a con 0x7f5308003810 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-c con 0x7f53080062c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f5308003810 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5308008e40 con 0x7f53080062c0 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5317fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308003810 auth_method 0 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:41.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f53080062c0 auth_method 0 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.134+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:41.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308003810 auth_method 0 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f53080062c0 auth_method 0 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:41.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:41.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.335+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308003810 auth_method 0 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:41.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f53080062c0 auth_method 0 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:41.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:41.736+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:42.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308003810 auth_method 0 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f53080062c0 auth_method 0 2024-09-10T16:28:42.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:42.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:42.537+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:42.680 INFO:tasks.rados.rados.0.smithi165.stdout:3464: done (1 left) 2024-09-10T16:28:42.680 INFO:tasks.rados.rados.0.smithi165.stdout:3465: done (0 left) 2024-09-10T16:28:42.680 INFO:tasks.rados.rados.0.smithi165.stdout:3466: snap_create 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): tick 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 msgr2=0x7f5308008780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f53080062c0 0x7f5308008780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 msgr2=0x7f5308005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:44.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308003810 0x7f5308005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 reap_dead start 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f532809e250 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5308006770 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-b con 0x7f5308005310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5328074cc0 con 0x7f5308006770 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308005f50 con 0x7f532809e250 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f5308005310 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5317fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:44.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.134+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.136+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f5308006770 legacy=0x7f5308003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:44.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.136+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f532809e250 legacy=0x7f530800d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:44.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:44.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.335+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:44.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:44.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:44.736+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:45.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:45.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:45.538+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:47.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): tick 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 msgr2=0x7f53080057c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308005310 0x7f53080057c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f532809e250 legacy=0x7f530800d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f5308006770 legacy=0x7f5308003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient: _add_conns ranks=[4,3] 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-c con 0x7f5308010e30 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-a con 0x7f5308011910 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:47.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f5308011910 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5308008e40 con 0x7f5308010e30 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5317fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5308011910 auth_method 0 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308010e30 auth_method 0 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:47.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:47.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.135+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5308011910 auth_method 0 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308010e30 auth_method 0 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:47.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308010e30 auth_method 0 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5308011910 auth_method 0 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:47.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:47.739+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5308011910 auth_method 0 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:48.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308010e30 auth_method 0 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:48.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:48.540+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:48.678 INFO:tasks.rados.rados.0.smithi165.stdout:3466: done (0 left) 2024-09-10T16:28:48.678 INFO:tasks.rados.rados.0.smithi165.stdout:3467: write (excl) oid 26 current snap is 385 2024-09-10T16:28:48.678 INFO:tasks.rados.rados.0.smithi165.stdout:3467: seq_num 1207 ranges {686049=409317,1847413=438636} 2024-09-10T16:28:48.683 INFO:tasks.rados.rados.0.smithi165.stdout:3467: writing smithi16534841-26 from 686049 to 1095366 tid 1 2024-09-10T16:28:48.690 INFO:tasks.rados.rados.0.smithi165.stdout:3467: writing smithi16534841-26 from 1847413 to 2286049 tid 2 2024-09-10T16:28:48.690 INFO:tasks.rados.rados.0.smithi165.stdout:3468: write oid 45 current snap is 385 2024-09-10T16:28:48.690 INFO:tasks.rados.rados.0.smithi165.stdout:3468: seq_num 1208 ranges {702856=484566,1789542=513314} 2024-09-10T16:28:48.696 INFO:tasks.rados.rados.0.smithi165.stdout:3468: writing smithi16534841-45 from 702856 to 1187422 tid 1 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:3468: writing smithi16534841-45 from 1789542 to 2302856 tid 2 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:3469: copy_from oid 18 from oid 30 current snap is 385 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:3470: rollback oid 17 current snap is 385 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 360 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:3471: read oid 6 snap -1 2024-09-10T16:28:48.703 INFO:tasks.rados.rados.0.smithi165.stdout:3471: expect deleted 2024-09-10T16:28:48.704 INFO:tasks.rados.rados.0.smithi165.stdout:3472: delete oid 40 current snap is 385 2024-09-10T16:28:48.704 INFO:tasks.rados.rados.0.smithi165.stdout:3468: finishing write tid 1 to smithi16534841-45 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3472: done (5 left) 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3473: copy_from oid 2 from oid 28 current snap is 385 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3474: snap_create 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3468: finishing write tid 2 to smithi16534841-45 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3468: finishing write tid 3 to smithi16534841-45 2024-09-10T16:28:48.707 INFO:tasks.rados.rados.0.smithi165.stdout:3468: finishing write tid 4 to smithi16534841-45 2024-09-10T16:28:48.708 INFO:tasks.rados.rados.0.smithi165.stdout:3468: oid 45 updating version 0 to 1705 2024-09-10T16:28:48.708 INFO:tasks.rados.rados.0.smithi165.stdout:3468: oid 45 version 1705 is already newer than 1704 2024-09-10T16:28:48.708 INFO:tasks.rados.rados.0.smithi165.stdout:3468: oid 45 version 1705 is already newer than 1703 2024-09-10T16:28:48.708 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1705 (ObjNum 1208 snap 385 seq_num 1208) dirty exists 2024-09-10T16:28:48.708 INFO:tasks.rados.rados.0.smithi165.stdout:3468: left oid 45 (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): tick 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 msgr2=0x7f5308011240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308010e30 0x7f5308011240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:50.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 msgr2=0x7f5308011d20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f5308011910 0x7f5308011d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 reap_dead start 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-b con 0x7f5308006770 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5308011210 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-c con 0x7f5308005310 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308012ba0 con 0x7f5308011210 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f5308005310 2024-09-10T16:28:50.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5308008e40 con 0x7f5308006770 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f5317fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308006770 auth_method 0 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.135+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:50.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.136+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:50.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.138+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f5308011210 legacy=0x7f5308003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:50.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308006770 auth_method 0 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:50.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:50.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.337+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:50.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308006770 auth_method 0 2024-09-10T16:28:50.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:50.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:50.738+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:51.007 INFO:tasks.rados.rados.0.smithi165.stdout:3470: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:28:51.007 INFO:tasks.rados.rados.0.smithi165.stdout:3470: finishing rollback tid 2 to smithi16534841-17 2024-09-10T16:28:51.007 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1837 (ObjNum 1097 snap 333 seq_num 1097) dirty exists 2024-09-10T16:28:51.012 INFO:tasks.rados.rados.0.smithi165.stdout:3467: finishing write tid 1 to smithi16534841-26 2024-09-10T16:28:51.014 INFO:tasks.rados.rados.0.smithi165.stdout:3473: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:28:51.015 INFO:tasks.rados.rados.0.smithi165.stdout:3467: finishing write tid 2 to smithi16534841-26 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: finishing write tid 3 to smithi16534841-26 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: finishing write tid 4 to smithi16534841-26 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: oid 26 updating version 0 to 2160 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: oid 26 updating version 2160 to 2161 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: oid 26 updating version 2161 to 2162 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2162 (ObjNum 1207 snap 385 seq_num 1207) dirty exists 2024-09-10T16:28:51.016 INFO:tasks.rados.rados.0.smithi165.stdout:3467: left oid 26 (ObjNum 1207 snap 385 seq_num 1207) 2024-09-10T16:28:51.017 INFO:tasks.rados.rados.0.smithi165.stdout:3473: finishing copy_from to smithi16534841-2 2024-09-10T16:28:51.017 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1286 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:28:51.027 INFO:tasks.rados.rados.0.smithi165.stdout:3469: finishing copy_from racing read to smithi16534841-18 2024-09-10T16:28:51.029 INFO:tasks.rados.rados.0.smithi165.stdout:3469: finishing copy_from to smithi16534841-18 2024-09-10T16:28:51.029 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:28:51.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:51.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:51.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5308006770 auth_method 0 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f5308005310 auth_method 0 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:51.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.539+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:28:51.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.540+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:51.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.540+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:51.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.540+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:51.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:51.540+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): tick 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 monclient(hunting): continuing hunt 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 msgr2=0x7f5308011cf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5308006770 0x7f5308011cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 msgr2=0x7f5308005760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.165:3300/0 conn(0x7f5308005310 0x7f5308005760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.165:6789/0 conn(0x7f5308011210 legacy=0x7f5308003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient: _add_conns ranks=[0,3] 2024-09-10T16:28:53.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f530800f460 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): picked mon.noname-a con 0x7f530800ff80 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308012590 con 0x7f530800f460 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.135+0000 7f5317fff640 1 -- 172.21.15.164:0/2824321994 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f530800ff80 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5317fff640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f530800ff80 auth_method 0 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:53.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:53.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:53.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.136+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:53.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.138+0000 7f532659c640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f530800f460 legacy=0x7f530800f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:53.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:53.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f530800ff80 auth_method 0 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:53.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.338+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f530800ff80 auth_method 0 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:53.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:53.740+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f530800ff80 auth_method 0 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:28:54.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:54.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.541+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:28:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2756671075 0 0) 0x7f5308012590 con 0x7f530800f460 2024-09-10T16:28:54.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 10 monclient(hunting): _negotiate 2024-09-10T16:28:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 10 monclient(hunting): my global_id is 254097 2024-09-10T16:28:54.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.616+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5310003520 con 0x7f530800f460 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1888283712 0 0) 0x7f5310003520 con 0x7f530800f460 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 msgr2=0x7f5308010390 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 --2- 172.21.15.164:0/2824321994 >> v2:172.21.15.164:3300/0 conn(0x7f530800ff80 0x7f5308010390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _finish_hunting 0 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:28:54.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5308010900 con 0x7f530800f460 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _finish_auth 0 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _check_auth_tickets 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f531800f040 con 0x7f530800f460 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: dump: 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:54.622 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:54.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 >> v1:172.21.15.164:6789/0 conn(0x7f530800f460 legacy=0x7f530800f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f5326d9d640 1 -- 172.21.15.164:0/2824321994 reap_dead start 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f5310003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): picked mon.a con 0x7f5310003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): picked mon.c con 0x7f5310004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f531000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): picked mon.b con 0x7f53100053c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f532809d3e0 con 0x7f5310003a20 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f531000c250 con 0x7f53100053c0 2024-09-10T16:28:54.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f531000c450 con 0x7f5310004800 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.617+0000 7f532559a640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f5310003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5310004800 auth_method 0 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5310003a20 auth_method 0 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f531000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f53100053c0 auth_method 0 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:28:54.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient(hunting): handle_auth_done global_id 254097 payload 293 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 msgr2=0x7f531000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f531000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 msgr2=0x7f5310003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f5310003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 monclient: found mon.c 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5310003520 con 0x7f5310004800 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient: _finish_auth 0 2024-09-10T16:28:54.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:28:54.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f5325d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:54.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532cf65640 5 monclient: authenticate success, global_id 254097 2024-09-10T16:28:54.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532cf65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:54.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.618+0000 7f532659c640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f5310003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:28:54.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f5325d9b640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 msgr2=0x7f5310004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f5325d9b640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f5310004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f531c007c00 tx=0x7f5310005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c764ec44481674ac server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c038070 con 0x7f5310004800 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: dump: 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:54.634 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532cf65640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f531c033400 con 0x7f5310004800 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 1 -- 172.21.15.164:0/2824321994 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f531c005ba0 con 0x7f5310004800 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532cf65640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532cf65640 4 set_mon_vals no callback set 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.621+0000 7f532559a640 10 monclient: dump: 2024-09-10T16:28:54.635 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 10 monclient: shutdown 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 -- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 msgr2=0x7f5310004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:54.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f5310004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f531c007c00 tx=0x7f5310005830 comp rx=0 tx=0).stop 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 -- 172.21.15.164:0/2824321994 shutdown_connections 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f531000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f5310004c80 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 --2- 172.21.15.164:0/2824321994 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f5310003e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 -- 172.21.15.164:0/2824321994 >> 172.21.15.164:0/2824321994 conn(0x7f53280a3110 msgr2=0x7f53280a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 -- 172.21.15.164:0/2824321994 shutdown_connections 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.622+0000 7f532cf65640 1 -- 172.21.15.164:0/2824321994 wait complete. 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.623+0000 7f532cf65640 10 monclient: build_initial_monmap 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.623+0000 7f532cf65640 10 monclient: monmap: 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:54.637 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 Processor -- start 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 -- start start 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient: init 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:28:54.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-a con 0x7f5310003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-c con 0x7f5310004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f532812d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): picked mon.noname-b con 0x7f53100053c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f532809e6e0 con 0x7f5310003a20 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f53280a02b0 con 0x7f53100053c0 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f532809ef30 con 0x7f5310004800 2024-09-10T16:28:54.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.624+0000 7f532cf65640 10 monclient(hunting): authenticate will time out at 2113.111926s 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56642/0 (socket says 172.21.15.164:56642) 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 -- 172.21.15.164:0/3704960699 learned_addr learned my addr 172.21.15.164:0/3704960699 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 10 monclient(hunting): get_auth_request con 0x7f5310004800 auth_method 0 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): get_auth_request con 0x7f5310003a20 auth_method 0 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f532812d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): get_auth_request con 0x7f53100053c0 auth_method 0 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 msgr2=0x7f532812cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 msgr2=0x7f532812cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:54.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5325d9b640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f5326d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient(hunting): handle_auth_done global_id 254100 payload 824 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 msgr2=0x7f532812cf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 msgr2=0x7f532812d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f532812d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient: _finish_hunting 0 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 monclient: found mon.noname-a 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:28:54.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.625+0000 7f532659c640 1 -- 172.21.15.164:0/3704960699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f532812f900 con 0x7f5310003a20 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f532659c640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 msgr2=0x7f532812c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f532659c640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5318006a50 tx=0x7f5318018820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318018e50 con 0x7f5310003a20 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 10 monclient: dump: 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.643 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 10 monclient: _finish_auth 0 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.627+0000 7f5316ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 5 monclient: authenticate success, global_id 254100 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53180167b0 con 0x7f5310003a20 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 10 monclient: _renew_subs 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5328130770 con 0x7f5310003a20 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:54.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318024800 con 0x7f5310003a20 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5316ffd640 10 monclient: dump: 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:54.645 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 10 monclient: _renew_subs 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f532812e410 con 0x7f5310003a20 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.628+0000 7f5317fff640 4 set_mon_vals no callback set 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f532cf65640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f532cf65640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f532cf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f53180249a0 con 0x7f5310003a20 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5328130040 con 0x7f5310003a20 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f5316ffd640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 0x7f5308044ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(773..773 src has 677..773) ==== 4755+0+0 (secure 0 0 0) 0x7f5318055820 con 0x7f5310003a20 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f5325d9b640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 0x7f5308044ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.629+0000 7f5325d9b640 10 monclient: get_auth_request con 0x7f53080425f0 auth_method 0 2024-09-10T16:28:54.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.632+0000 7f5325d9b640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 0x7f5308044ab0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f531c007fd0 tx=0x7f531c01b030 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:54.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.634+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5328130040 con 0x7f5310003a20 2024-09-10T16:28:54.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.634+0000 7f5316ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:54.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.634+0000 7f5316ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:54.675 INFO:tasks.rados.rados.0.smithi165.stdout:3467: done (6 left) 2024-09-10T16:28:54.675 INFO:tasks.rados.rados.0.smithi165.stdout:3468: done (5 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3469: done (4 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3470: done (3 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3471: done (2 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3473: done (1 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3474: done (0 left) 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3475: copy_from oid 21 from oid 38 current snap is 386 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3476: copy_from oid 1 from oid 37 current snap is 386 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3477: read oid 46 snap -1 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3477: expect deleted 2024-09-10T16:28:54.676 INFO:tasks.rados.rados.0.smithi165.stdout:3478: snap_remove snap 364 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.884+0000 7f532cf65640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.884+0000 7f532cf65640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.884+0000 7f532cf65640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.884+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f53280a2940 con 0x7f5310003a20 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.885+0000 7f5316ffd640 1 -- 172.21.15.164:0/3704960699 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f5318031060 con 0x7f5310003a20 2024-09-10T16:28:54.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.885+0000 7f5316ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:54.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.885+0000 7f5316ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:54.887 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":222,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 msgr2=0x7f5308044ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 0x7f5308044ab0 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f531c007fd0 tx=0x7f531c01b030 comp rx=0 tx=0).stop 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 10 monclient: shutdown 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 msgr2=0x7f532812c9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5318006a50 tx=0x7f5318018820 comp rx=0 tx=0).stop 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 shutdown_connections 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f53080425f0 0x7f5308044ab0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f53100053c0 0x7f532812d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5310004800 0x7f532812cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 --2- 172.21.15.164:0/3704960699 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5310003a20 0x7f532812c9d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:54.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 >> 172.21.15.164:0/3704960699 conn(0x7f53280a3110 msgr2=0x7f5328134790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:54.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 shutdown_connections 2024-09-10T16:28:54.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:54.887+0000 7f532cf65640 1 -- 172.21.15.164:0/3704960699 wait complete. 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3478: done (3 left) 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3479: rollback oid 48 current snap is 386 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 334 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3476: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3475: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3476: finishing copy_from to smithi16534841-1 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1583 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:3475: finishing copy_from to smithi16534841-21 2024-09-10T16:28:56.866 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1415 (ObjNum 1181 snap 373 seq_num 1181) dirty exists 2024-09-10T16:28:56.867 INFO:tasks.rados.rados.0.smithi165.stdout:3475: done (3 left) 2024-09-10T16:28:56.867 INFO:tasks.rados.rados.0.smithi165.stdout:3476: done (2 left) 2024-09-10T16:28:56.867 INFO:tasks.rados.rados.0.smithi165.stdout:3477: done (1 left) 2024-09-10T16:28:56.867 INFO:tasks.rados.rados.0.smithi165.stdout:3480: snap_create 2024-09-10T16:28:56.873 INFO:tasks.rados.rados.0.smithi165.stdout:3479: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:28:56.874 INFO:tasks.rados.rados.0.smithi165.stdout:3479: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:28:56.875 INFO:tasks.rados.rados.0.smithi165.stdout:3479: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:28:56.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 2166 (ObjNum 1064 snap 328 seq_num 1064) dirty exists 2024-09-10T16:28:57.877 INFO:tasks.rados.rados.0.smithi165.stdout:3479: done (1 left) 2024-09-10T16:28:57.877 INFO:tasks.rados.rados.0.smithi165.stdout:3480: done (0 left) 2024-09-10T16:28:57.877 INFO:tasks.rados.rados.0.smithi165.stdout:3481: snap_create 2024-09-10T16:28:57.900 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:28:58.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 10 monclient: get_monmap_and_config 2024-09-10T16:28:58.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.007 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 10 monclient: monmap: 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:58.006997+0000 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:58.006997+0000 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.008 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:58.009 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:58.009 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:58.009 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:58.009 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:58.009 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 1 Processor -- start 2024-09-10T16:28:58.019 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 1 -- start start 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.006+0000 7fee48382640 10 monclient: init 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee4009f930 0x7fee400a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): picked mon.noname-c con 0x7fee4009f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fee400a2380 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4013dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): picked mon.noname-b con 0x7fee400a27b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee4009fd10 con 0x7fee400a2380 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.020 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fee400a2c60 con 0x7fee4009f930 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fee4009cd20 con 0x7fee400a27b0 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee48382640 10 monclient(hunting): authenticate will time out at 2116.494536s 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4591d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fee400a2380 0x7fee4013a5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53574/0 (socket says 172.21.15.164:53574) 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4591d640 1 -- 172.21.15.164:0/3754541008 learned_addr learned my addr 172.21.15.164:0/3754541008 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4013dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): get_auth_request con 0x7fee400a27b0 auth_method 0 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4611e640 1 --2- 172.21.15.164:0/3754541008 >> v2:172.21.15.165:3300/0 conn(0x7fee4009f930 0x7fee400a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4611e640 10 monclient(hunting): get_auth_request con 0x7fee4009f930 auth_method 0 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4611e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4611e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4611e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient(hunting): handle_auth_done global_id 244184 payload 274 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 -- 172.21.15.164:0/3754541008 >> v2:172.21.15.165:3300/0 conn(0x7fee4009f930 msgr2=0x7fee400a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 --2- 172.21.15.164:0/3754541008 >> v2:172.21.15.165:3300/0 conn(0x7fee4009f930 0x7fee400a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 -- 172.21.15.164:0/3754541008 >> v1:172.21.15.164:6789/0 conn(0x7fee400a2380 legacy=0x7fee4013a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 monclient: found mon.noname-b 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.007+0000 7fee4691f640 1 -- 172.21.15.164:0/3754541008 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee4009a0d0 con 0x7fee400a27b0 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.010+0000 7fee4691f640 1 -- 172.21.15.164:0/3754541008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 msgr2=0x7fee4013dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.010+0000 7fee4691f640 1 --2- 172.21.15.164:0/3754541008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4013dcf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee38009d90 tx=0x7fee3802a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=30cac3aedcbfd2ef server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.010+0000 7fee4511c640 1 -- 172.21.15.164:0/3754541008 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38007950 con 0x7fee400a27b0 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: dump: 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.023 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 1 -- 172.21.15.164:0/3754541008 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee38007af0 con 0x7fee400a27b0 2024-09-10T16:28:58.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 5 monclient: authenticate success, global_id 244184 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 1 -- 172.21.15.164:0/3754541008 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee38031400 con 0x7fee400a27b0 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 4 set_mon_vals no callback set 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee4511c640 10 monclient: dump: 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.025 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 10 monclient: shutdown 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 -- 172.21.15.164:0/3754541008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 msgr2=0x7fee4013dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 --2- 172.21.15.164:0/3754541008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4013dcf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fee38009d90 tx=0x7fee3802a490 comp rx=0 tx=0).stop 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 -- 172.21.15.164:0/3754541008 shutdown_connections 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 --2- 172.21.15.164:0/3754541008 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4013dcf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 --2- 172.21.15.164:0/3754541008 >> v2:172.21.15.165:3300/0 conn(0x7fee4009f930 0x7fee400a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 -- 172.21.15.164:0/3754541008 >> 172.21.15.164:0/3754541008 conn(0x7fee400a3110 msgr2=0x7fee400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 -- 172.21.15.164:0/3754541008 shutdown_connections 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.011+0000 7fee48382640 1 -- 172.21.15.164:0/3754541008 wait complete. 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.012+0000 7fee48382640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.012+0000 7fee48382640 10 monclient: monmap: 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:58.027 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.012+0000 7fee48382640 1 Processor -- start 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.012+0000 7fee48382640 1 -- start start 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.012+0000 7fee48382640 10 monclient: init 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:28:58.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): picked mon.noname-a con 0x7fee4009f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee400a2380 0x7fee40099610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): picked mon.noname-b con 0x7fee400a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4009a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): picked mon.noname-c con 0x7fee400a27b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee4009d930 con 0x7fee4009f930 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fee4009ee00 con 0x7fee400a2380 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fee4009c8b0 con 0x7fee400a27b0 2024-09-10T16:28:58.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee48382640 10 monclient(hunting): authenticate will time out at 2116.500243s 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56650/0 (socket says 172.21.15.164:56650) 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 -- 172.21.15.164:0/3720829565 learned_addr learned my addr 172.21.15.164:0/3720829565 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): get_auth_request con 0x7fee4009f930 auth_method 0 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4009a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): get_auth_request con 0x7fee400a27b0 auth_method 0 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee400a2380 0x7fee40099610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): get_auth_request con 0x7fee400a2380 auth_method 0 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4591d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient(hunting): handle_auth_done global_id 254103 payload 824 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 -- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 msgr2=0x7fee4009a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4009a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 -- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee400a2380 msgr2=0x7fee40099610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee400a2380 0x7fee40099610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 monclient: found mon.noname-a 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4611e640 1 -- 172.21.15.164:0/3720829565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee40095e00 con 0x7fee4009f930 2024-09-10T16:28:58.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.013+0000 7fee4691f640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4009a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.015+0000 7fee4611e640 1 -- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 msgr2=0x7fee401ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.015+0000 7fee4611e640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fee3000d830 tx=0x7fee3000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30018020 con 0x7fee4009f930 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: dump: 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.033 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee300098b0 con 0x7fee4009f930 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 5 monclient: authenticate success, global_id 254103 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee30009c80 con 0x7fee4009f930 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 10 monclient: _renew_subs 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee401cd3e0 con 0x7fee4009f930 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 10 monclient: dump: 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee4491b640 4 set_mon_vals no callback set 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 10 monclient: _renew_subs 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee4006e860 con 0x7fee4009f930 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.016+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fee3001b020 con 0x7fee4009f930 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee36ffd640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 0x7fee240406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee48382640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee48382640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee48382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(776..776 src has 677..776) ==== 4755+0+0 (secure 0 0 0) 0x7fee30053200 con 0x7fee4009f930 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee400a0760 con 0x7fee4009f930 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee4591d640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 0x7fee240406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.017+0000 7fee4591d640 10 monclient: get_auth_request con 0x7fee2403e210 auth_method 0 2024-09-10T16:28:58.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.019+0000 7fee4591d640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 0x7fee240406d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fee3c007c90 tx=0x7fee3c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.022+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fee30022020 con 0x7fee4009f930 2024-09-10T16:28:58.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.022+0000 7fee36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.022+0000 7fee36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:58.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee48382640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:28:58.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee48382640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee48382640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fee40095a80 con 0x7fee4009f930 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee36ffd640 1 -- 172.21.15.164:0/3720829565 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fee30009e20 con 0x7fee4009f930 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.263+0000 7fee36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:58.265 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 msgr2=0x7fee240406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 0x7fee240406d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fee3c007c90 tx=0x7fee3c007be0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 10 monclient: shutdown 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 msgr2=0x7fee401ccea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fee3000d830 tx=0x7fee3000dcf0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 shutdown_connections 2024-09-10T16:28:58.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fee2403e210 0x7fee240406d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee400a27b0 0x7fee4009a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee400a2380 0x7fee40099610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 --2- 172.21.15.164:0/3720829565 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee4009f930 0x7fee401ccea0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 >> 172.21.15.164:0/3720829565 conn(0x7fee400a3110 msgr2=0x7fee400a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 shutdown_connections 2024-09-10T16:28:58.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.265+0000 7fee48382640 1 -- 172.21.15.164:0/3720829565 wait complete. 2024-09-10T16:28:58.277 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:28:58.277 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:28:58.278 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.374+0000 7fccdf671640 10 monclient: get_monmap_and_config 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.374+0000 7fccdf671640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.374+0000 7fccdf671640 10 monclient: monmap: 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.376 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:58.375685+0000 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:58.375685+0000 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:58.377 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.375+0000 7fccdf671640 1 Processor -- start 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.375+0000 7fccdf671640 1 -- start start 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.375+0000 7fccdf671640 10 monclient: init 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:28:58.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fccd809c310 0x7fccd809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-a con 0x7fccd809c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-b con 0x7fccd809d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fccd809f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccd809fbc0 con 0x7fccd809f4c0 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fccd80a2a20 con 0x7fccd809c310 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fccd809c6f0 con 0x7fccd809d190 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdf671640 10 monclient(hunting): authenticate will time out at 2116.863196s 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fccd809c310 0x7fccd809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53400/0 (socket says 172.21.15.164:53400) 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fccd809c310 0x7fccd809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56660/0 (socket says 172.21.15.164:56660) 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 1 -- 172.21.15.164:0/1829125583 learned_addr learned my addr 172.21.15.164:0/1829125583 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): get_auth_request con 0x7fccd809d190 auth_method 0 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): get_auth_request con 0x7fccd809c310 auth_method 0 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient(hunting): handle_auth_done global_id 254106 payload 274 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 -- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 msgr2=0x7fccd809ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 -- 172.21.15.164:0/1829125583 >> v1:172.21.15.165:6789/0 conn(0x7fccd809f4c0 legacy=0x7fccd813b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.391 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 monclient: found mon.noname-a 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdd40d640 1 -- 172.21.15.164:0/1829125583 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd809a4d0 con 0x7fccd809c310 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.376+0000 7fccdcc0c640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdd40d640 1 -- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 msgr2=0x7fccd809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdd40d640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd809ea40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009d90 tx=0x7fccc802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e5b6b5aae5e111a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 1 -- 172.21.15.164:0/1829125583 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8007950 con 0x7fccd809c310 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: dump: 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.392 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 1 -- 172.21.15.164:0/1829125583 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccc8007af0 con 0x7fccd809c310 2024-09-10T16:28:58.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdf671640 5 monclient: authenticate success, global_id 254106 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdf671640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 1 -- 172.21.15.164:0/1829125583 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8031400 con 0x7fccd809c310 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccd7fff640 10 monclient: dump: 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.394 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdf671640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.379+0000 7fccdf671640 4 set_mon_vals no callback set 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 10 monclient: shutdown 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.395 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 -- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 msgr2=0x7fccd809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd809ea40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fccc8009d90 tx=0x7fccc802a490 comp rx=0 tx=0).stop 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 -- 172.21.15.164:0/1829125583 shutdown_connections 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809d190 0x7fccd809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 --2- 172.21.15.164:0/1829125583 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd809ea40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 -- 172.21.15.164:0/1829125583 >> 172.21.15.164:0/1829125583 conn(0x7fccd80a3110 msgr2=0x7fccd80a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 -- 172.21.15.164:0/1829125583 shutdown_connections 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 1 -- 172.21.15.164:0/1829125583 wait complete. 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.380+0000 7fccdf671640 10 monclient: monmap: 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:58.396 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 Processor -- start 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 -- start start 2024-09-10T16:28:58.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient: init 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd80992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-a con 0x7fccd809c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 0x7fccd8099830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-c con 0x7fccd809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd81c8bf0 0x7fccd80902a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): picked mon.noname-b con 0x7fccd81c8bf0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fccd80a2230 con 0x7fccd809c310 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fccd80a20b0 con 0x7fccd81c8bf0 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fccd8074e00 con 0x7fccd809f4c0 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdf671640 10 monclient(hunting): authenticate will time out at 2116.868672s 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd80992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 0x7fccd8099830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd80992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56666/0 (socket says 172.21.15.164:56666) 2024-09-10T16:28:58.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 1 -- 172.21.15.164:0/2075934534 learned_addr learned my addr 172.21.15.164:0/2075934534 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): get_auth_request con 0x7fccd809c310 auth_method 0 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): get_auth_request con 0x7fccd809f4c0 auth_method 0 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccddc0e640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd81c8bf0 0x7fccd80902a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccddc0e640 10 monclient(hunting): get_auth_request con 0x7fccd81c8bf0 auth_method 0 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccddc0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccddc0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccddc0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 10 monclient(hunting): handle_auth_done global_id 244193 payload 824 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd81c8bf0 msgr2=0x7fccd80902a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.381+0000 7fccdcc0c640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fccd81c8bf0 0x7fccd80902a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 msgr2=0x7fccd80992f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809c310 0x7fccd80992f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 1 monclient: found mon.noname-c 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.382+0000 7fccdcc0c640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccd8093e80 con 0x7fccd809f4c0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdcc0c640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 msgr2=0x7fccd8099830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdcc0c640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 0x7fccd8099830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcccc010dc0 tx=0x7fcccc00e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc004230 con 0x7fccd809f4c0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: dump: 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 5 monclient: authenticate success, global_id 244193 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcccc0043d0 con 0x7fccd809f4c0 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 10 monclient: _renew_subs 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccd809d620 con 0x7fccd809f4c0 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcccc011880 con 0x7fccd809f4c0 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd5ffb640 10 monclient: dump: 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.404 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 10 monclient: _renew_subs 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fccd8093ba0 con 0x7fccd809f4c0 2024-09-10T16:28:58.405 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.384+0000 7fccd77fe640 4 set_mon_vals no callback set 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdf671640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdf671640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 0x7fccd802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fccd809ed50 con 0x7fccd809d190 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 0x7fccd802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient: get_auth_request con 0x7fccd809d190 auth_method 0 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcccc00b5f0 con 0x7fccd809f4c0 2024-09-10T16:28:58.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 10 monclient(hunting): handle_auth_done global_id 254112 payload 824 2024-09-10T16:28:58.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccdd40d640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fccd809d930 con 0x7fccd809d190 2024-09-10T16:28:58.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccd5ffb640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fccc403e210 0x7fccc40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.385+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(776..776 src has 677..776) ==== 4755+0+0 (secure 0 0 0) 0x7fcccc055aa0 con 0x7fccd809f4c0 2024-09-10T16:28:58.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.386+0000 7fccddc0e640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fccc403e210 0x7fccc40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.386+0000 7fccddc0e640 10 monclient: get_auth_request con 0x7fccc403e210 auth_method 0 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.387+0000 7fccdd40d640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 msgr2=0x7fccd802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.387+0000 7fccdd40d640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 0x7fccd802d440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fccc802a4d0 tx=0x7fccc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc8031ac0 con 0x7fccd809d190 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 10 monclient: dump: 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.410 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fccc803c420 con 0x7fccd809d190 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 msgr2=0x7fccd802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 0x7fccd802d440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fccc802a4d0 tx=0x7fccc8004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 msgr2=0x7fccd802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccd5ffb640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd809d190 0x7fccd802d440 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.388+0000 7fccddc0e640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fccc403e210 0x7fccc40406d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fccd0007fd0 tx=0x7fccd0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdf671640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdf671640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 0x7fccd802dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdf671640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fccd8099d70 con 0x7fccd802d980 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 0x7fccd802dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient: get_auth_request con 0x7fccd802d980 auth_method 0 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 10 monclient(hunting): handle_auth_done global_id 254115 payload 824 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.394+0000 7fccdd40d640 1 -- 172.21.15.164:0/2075934534 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fccd8073610 con 0x7fccd802d980 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.396+0000 7fccdd40d640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 msgr2=0x7fccd802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.396+0000 7fccdd40d640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 0x7fccd802dd90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fccc8043000 tx=0x7fccc8030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc804a040 con 0x7fccd802d980 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 10 monclient: dump: 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.413 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7fccc8004450 con 0x7fccd802d980 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:58.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 msgr2=0x7fccd802dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 0x7fccd802dd90 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fccc8043000 tx=0x7fccc8030000 comp rx=0 tx=0).stop 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 msgr2=0x7fccd802dd90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccd5ffb640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fccd802d980 0x7fccd802dd90 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:28:58.415 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 27048, 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.416 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.417 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:28:58.418 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.419 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:28:58.420 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.421 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.422 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:28:58.423 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:58.424 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.425 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:28:58.426 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.427 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.428 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.429 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.429 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:28:58.429 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:28:58.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fccc403e210 msgr2=0x7fccc40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fccc403e210 0x7fccc40406d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fccd0007fd0 tx=0x7fccd0009040 comp rx=0 tx=0).stop 2024-09-10T16:28:58.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 10 monclient: shutdown 2024-09-10T16:28:58.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 msgr2=0x7fccd8099830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 --2- 172.21.15.164:0/2075934534 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fccd809f4c0 0x7fccd8099830 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fcccc010dc0 tx=0x7fcccc00e730 comp rx=0 tx=0).stop 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccddc0e640 1 -- 172.21.15.164:0/2075934534 reap_dead start 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccddc0e640 1 -- 172.21.15.164:0/2075934534 reap_dead start 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 shutdown_connections 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 >> 172.21.15.164:0/2075934534 conn(0x7fccd80a3110 msgr2=0x7fccd80707e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 shutdown_connections 2024-09-10T16:28:58.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.397+0000 7fccdf671640 1 -- 172.21.15.164:0/2075934534 wait complete. 2024-09-10T16:28:58.431 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:28:58.431 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:28:58.432 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.529+0000 7f72d2f52640 10 monclient: get_monmap_and_config 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.529+0000 7f72d2f52640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.529+0000 7f72d2f52640 10 monclient: monmap: 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.531 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:58.530530+0000 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:58.530530+0000 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:58.532 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 1 Processor -- start 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 1 -- start start 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient: init 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:28:58.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-b con 0x7f72cc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f72cc09d160 0x7f72cc09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-c con 0x7f72cc09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f72cc09f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.530+0000 7f72d2f52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f72cc09a430 con 0x7f72cc09f4c0 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f72cc0a24e0 con 0x7f72cc09d160 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc0a2bf0 con 0x7f72cc09c180 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d2f52640 10 monclient(hunting): authenticate will time out at 2117.018017s 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53412/0 (socket says 172.21.15.164:53412) 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 -- 172.21.15.164:0/3147820497 learned_addr learned my addr 172.21.15.164:0/3147820497 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d14ef640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f72cc09f4c0 0x7f72cc13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53582/0 (socket says 172.21.15.164:53582) 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): get_auth_request con 0x7f72cc09c180 auth_method 0 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient(hunting): handle_auth_done global_id 244196 payload 274 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72cbfff640 1 --2- 172.21.15.164:0/3147820497 >> v2:172.21.15.165:3300/0 conn(0x7f72cc09d160 0x7f72cc09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 -- 172.21.15.164:0/3147820497 >> v2:172.21.15.165:3300/0 conn(0x7f72cc09d160 msgr2=0x7f72cc09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 --2- 172.21.15.164:0/3147820497 >> v2:172.21.15.165:3300/0 conn(0x7f72cc09d160 0x7f72cc09ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 -- 172.21.15.164:0/3147820497 >> v1:172.21.15.164:6789/0 conn(0x7f72cc09f4c0 legacy=0x7f72cc13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 monclient: found mon.noname-b 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.531+0000 7f72d0cee640 1 -- 172.21.15.164:0/3147820497 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc09c560 con 0x7f72cc09c180 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d0cee640 1 -- 172.21.15.164:0/3147820497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 msgr2=0x7f72cc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d0cee640 1 --2- 172.21.15.164:0/3147820497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00bb10 tx=0x7f72bc02c7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e988d9a08907634 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 1 -- 172.21.15.164:0/3147820497 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc031a00 con 0x7f72cc09c180 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: dump: 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 1 -- 172.21.15.164:0/3147820497 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc031ba0 con 0x7f72cc09c180 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 5 monclient: authenticate success, global_id 244196 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 1 -- 172.21.15.164:0/3147820497 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc033400 con 0x7f72cc09c180 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 4 set_mon_vals no callback set 2024-09-10T16:28:58.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72cb7fe640 10 monclient: dump: 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 10 monclient: shutdown 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 -- 172.21.15.164:0/3147820497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 msgr2=0x7f72cc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 --2- 172.21.15.164:0/3147820497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00bb10 tx=0x7f72bc02c7e0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 -- 172.21.15.164:0/3147820497 shutdown_connections 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 --2- 172.21.15.164:0/3147820497 >> v2:172.21.15.165:3300/0 conn(0x7f72cc09d160 0x7f72cc09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 --2- 172.21.15.164:0/3147820497 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc09ea40 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 -- 172.21.15.164:0/3147820497 >> 172.21.15.164:0/3147820497 conn(0x7f72cc0a3110 msgr2=0x7f72cc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 -- 172.21.15.164:0/3147820497 shutdown_connections 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.534+0000 7f72d2f52640 1 -- 172.21.15.164:0/3147820497 wait complete. 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.535+0000 7f72d2f52640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.535+0000 7f72d2f52640 10 monclient: monmap: 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.546 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.535+0000 7f72d2f52640 1 Processor -- start 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.535+0000 7f72d2f52640 1 -- start start 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.535+0000 7f72d2f52640 10 monclient: init 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-c con 0x7f72cc09c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72cc09d160 0x7f72cc097a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-b con 0x7f72cc09d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72cc09f4c0 0x7f72cc1c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): picked mon.noname-a con 0x7f72cc09f4c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc074f90 con 0x7f72cc09f4c0 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f72cc074e10 con 0x7f72cc09d160 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc09d930 con 0x7f72cc09c180 2024-09-10T16:28:58.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d2f52640 10 monclient(hunting): authenticate will time out at 2117.023449s 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72cc09f4c0 0x7f72cc1c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53416/0 (socket says 172.21.15.164:53416) 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72cc09f4c0 0x7f72cc1c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56688/0 (socket says 172.21.15.164:56688) 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 -- 172.21.15.164:0/674702204 learned_addr learned my addr 172.21.15.164:0/674702204 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): get_auth_request con 0x7f72cc09c180 auth_method 0 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): get_auth_request con 0x7f72cc09f4c0 auth_method 0 2024-09-10T16:28:58.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d14ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72cc09d160 0x7f72cc097a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient(hunting): handle_auth_done global_id 244199 payload 824 2024-09-10T16:28:58.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72cc09d160 msgr2=0x7f72cc097a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72cc09d160 0x7f72cc097a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72cc09f4c0 msgr2=0x7f72cc1c5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72cc09f4c0 0x7f72cc1c5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 monclient: found mon.noname-c 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72d0cee640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72cc08fd50 con 0x7f72cc09c180 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72cbfff640 10 monclient: get_auth_request con 0x7f72cc09d160 auth_method 0 2024-09-10T16:28:58.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.536+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f72cc09d160 0x7f72cc097a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d0cee640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 msgr2=0x7f72cc097520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d0cee640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00b2c0 tx=0x7f72bc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc041070 con 0x7f72cc09c180 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: dump: 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.557 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f72bc046660 con 0x7f72cc09c180 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 5 monclient: authenticate success, global_id 244199 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 10 monclient: _renew_subs 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72cc0a4550 con 0x7f72cc09c180 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc046a50 con 0x7f72cc09c180 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72c97fa640 10 monclient: dump: 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72ca7fc640 4 set_mon_vals no callback set 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 10 monclient: _renew_subs 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.539+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f72cc0a17c0 con 0x7f72cc09c180 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d2f52640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d2f52640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 0x7f72cc1c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc09d620 con 0x7f72cc091d30 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 0x7f72cc1c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f72bc031a10 con 0x7f72cc09c180 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient: get_auth_request con 0x7f72cc091d30 auth_method 0 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 10 monclient(hunting): handle_auth_done global_id 244202 payload 824 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72cbfff640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72cc09e4d0 con 0x7f72cc091d30 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72c97fa640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f72b803e210 0x7f72b80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(776..776 src has 677..776) ==== 4755+0+0 (secure 0 0 0) 0x7f72bc076650 con 0x7f72cc09c180 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d14ef640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f72b803e210 0x7f72b80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.540+0000 7f72d14ef640 10 monclient: get_auth_request con 0x7f72b803e210 auth_method 0 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72cbfff640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 msgr2=0x7f72cc1c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 0x7f72cc1c59b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c000df20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c001b070 con 0x7f72cc091d30 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.542+0000 7f72c97fa640 10 monclient: dump: 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.562 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f72c0016640 con 0x7f72cc091d30 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 msgr2=0x7f72cc1c59b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 0x7f72cc1c59b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c000df20 comp rx=0 tx=0).stop 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 msgr2=0x7f72cc1c59b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72c97fa640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc091d30 0x7f72cc1c59b0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c000df20 comp rx=0 tx=0).stop 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.543+0000 7f72d14ef640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f72b803e210 0x7f72b80406d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f72c4007fd0 tx=0x7f72c4007de0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72d2f52640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:28:58.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72d2f52640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 0x7f72cc0712c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72d2f52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f72cc1c6300 con 0x7f72cc1c5ef0 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 0x7f72cc0712c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient: get_auth_request con 0x7f72cc1c5ef0 auth_method 0 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 10 monclient(hunting): handle_auth_done global_id 244205 payload 824 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.549+0000 7f72cbfff640 1 -- 172.21.15.164:0/674702204 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72cc092110 con 0x7f72cc1c5ef0 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72cbfff640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 msgr2=0x7f72cc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72cbfff640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 0x7f72cc0712c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c001b040 con 0x7f72cc1c5ef0 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 10 monclient: dump: 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.565 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f72c0026480 con 0x7f72cc1c5ef0 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 msgr2=0x7f72cc0712c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 0x7f72cc0712c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c0004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 msgr2=0x7f72cc0712c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.551+0000 7f72c97fa640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc1c5ef0 0x7f72cc0712c0 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f72c000da60 tx=0x7f72c0004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 75459, 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:28:58.567 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.568 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.569 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:28:58.570 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.571 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.572 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.573 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.574 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.575 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.576 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:28:58.577 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.578 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.579 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:28:58.580 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f72b803e210 msgr2=0x7f72b80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f72b803e210 0x7f72b80406d0 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f72c4007fd0 tx=0x7f72c4007de0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 10 monclient: shutdown 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 msgr2=0x7f72cc097520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f72bc00b2c0 tx=0x7f72bc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d14ef640 1 -- 172.21.15.164:0/674702204 reap_dead start 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 shutdown_connections 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 --2- 172.21.15.164:0/674702204 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f72cc09c180 0x7f72cc097520 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 >> 172.21.15.164:0/674702204 conn(0x7f72cc0a3110 msgr2=0x7f72cc099600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 shutdown_connections 2024-09-10T16:28:58.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.552+0000 7f72d2f52640 1 -- 172.21.15.164:0/674702204 wait complete. 2024-09-10T16:28:58.583 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:28:58.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:28:58.584 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.680+0000 7f168329b640 10 monclient: get_monmap_and_config 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.680+0000 7f168329b640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.680+0000 7f168329b640 10 monclient: monmap: 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.685 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:58.681783+0000 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:58.681783+0000 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.681+0000 7f168329b640 1 Processor -- start 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.681+0000 7f168329b640 1 -- start start 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.681+0000 7f168329b640 10 monclient: init 2024-09-10T16:28:58.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f167c09d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): picked mon.noname-a con 0x7f167c09ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c13bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): picked mon.noname-b con 0x7f167c09e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f167c0a0870 con 0x7f167c09d930 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f167c09d3e0 con 0x7f167c09ab50 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f167c074c70 con 0x7f167c09e420 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f168329b640 10 monclient(hunting): authenticate will time out at 2117.169167s 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c13bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56708/0 (socket says 172.21.15.164:56708) 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 1 -- 172.21.15.164:0/1365117883 learned_addr learned my addr 172.21.15.164:0/1365117883 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): get_auth_request con 0x7f167c09ab50 auth_method 0 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681037640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f167c09d930 0x7f167c09dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53586/0 (socket says 172.21.15.164:53586) 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): get_auth_request con 0x7f167c09e420 auth_method 0 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient(hunting): handle_auth_done global_id 244208 payload 274 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 -- 172.21.15.164:0/1365117883 >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 msgr2=0x7f167c1397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 --2- 172.21.15.164:0/1365117883 >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 -- 172.21.15.164:0/1365117883 >> v1:172.21.15.164:6789/0 conn(0x7f167c09d930 legacy=0x7f167c09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 monclient: found mon.noname-b 2024-09-10T16:28:58.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1681838640 1 -- 172.21.15.164:0/1365117883 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c09ecb0 con 0x7f167c09e420 2024-09-10T16:28:58.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.682+0000 7f1680836640 1 --2- 172.21.15.164:0/1365117883 >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:28:58.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f1681838640 1 -- 172.21.15.164:0/1365117883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 msgr2=0x7f167c13bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f1681838640 1 --2- 172.21.15.164:0/1365117883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c13bd10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f166c009d90 tx=0x7f166c02a490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c06d27697ad2065 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 1 -- 172.21.15.164:0/1365117883 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c007950 con 0x7f167c09e420 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: dump: 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 5 monclient: authenticate success, global_id 244208 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 1 -- 172.21.15.164:0/1365117883 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f166c007af0 con 0x7f167c09e420 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 1 -- 172.21.15.164:0/1365117883 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f166c031400 con 0x7f167c09e420 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 4 set_mon_vals no callback set 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f167bfff640 10 monclient: dump: 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 10 monclient: shutdown 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 1 -- 172.21.15.164:0/1365117883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 msgr2=0x7f167c13bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.685+0000 7f168329b640 1 --2- 172.21.15.164:0/1365117883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c13bd10 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f166c009d90 tx=0x7f166c02a490 comp rx=0 tx=0).stop 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 -- 172.21.15.164:0/1365117883 shutdown_connections 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 --2- 172.21.15.164:0/1365117883 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c13bd10 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 --2- 172.21.15.164:0/1365117883 >> v2:172.21.15.164:3300/0 conn(0x7f167c09ab50 0x7f167c1397c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 -- 172.21.15.164:0/1365117883 >> 172.21.15.164:0/1365117883 conn(0x7f167c0a3110 msgr2=0x7f167c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 -- 172.21.15.164:0/1365117883 shutdown_connections 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 -- 172.21.15.164:0/1365117883 wait complete. 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 10 monclient: monmap: 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:58.701 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.686+0000 7f168329b640 1 Processor -- start 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 -- start start 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient: init 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): picked mon.noname-a con 0x7f167c09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c09d930 0x7f167c06ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): picked mon.noname-b con 0x7f167c09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c06f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): picked mon.noname-c con 0x7f167c09e420 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f167c0a2940 con 0x7f167c09ab50 2024-09-10T16:28:58.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f167c09ef30 con 0x7f167c09d930 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f167c0a2ac0 con 0x7f167c09e420 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f168329b640 10 monclient(hunting): authenticate will time out at 2117.174457s 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c06f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56712/0 (socket says 172.21.15.164:56712) 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 -- 172.21.15.164:0/2662755530 learned_addr learned my addr 172.21.15.164:0/2662755530 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): get_auth_request con 0x7f167c09ab50 auth_method 0 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): get_auth_request con 0x7f167c09e420 auth_method 0 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681838640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c09d930 0x7f167c06ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient(hunting): handle_auth_done global_id 254124 payload 824 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 msgr2=0x7f167c06f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f167c09e420 0x7f167c06f250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c09d930 msgr2=0x7f167c06ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c09d930 0x7f167c06ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 monclient: found mon.noname-a 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1681037640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c070200 con 0x7f167c09ab50 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1680836640 10 monclient: get_auth_request con 0x7f167c09d930 auth_method 0 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.687+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c09d930 0x7f167c06ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:28:58.705 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1681037640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 msgr2=0x7f167c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1681037640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f167400c450 tx=0x7f167400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674017070 con 0x7f167c09ab50 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: dump: 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.706 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674005810 con 0x7f167c09ab50 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 5 monclient: authenticate success, global_id 254124 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 10 monclient: _renew_subs 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c12d0c0 con 0x7f167c09ab50 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674007690 con 0x7f167c09ab50 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f1679ffb640 10 monclient: dump: 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.708 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 10 monclient: _renew_subs 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f167b7fe640 4 set_mon_vals no callback set 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.690+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c1cd190 con 0x7f167c09ab50 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f168329b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f168329b640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 0x7f167c1cd890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f167c09b820 con 0x7f167c1cd4b0 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f167401a070 con 0x7f167c09ab50 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1679ffb640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f166803e210 0x7f16680406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 0x7f167c1cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient: get_auth_request con 0x7f167c1cd4b0 auth_method 0 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(776..776 src has 677..776) ==== 4755+0+0 (secure 0 0 0) 0x7f1674052110 con 0x7f167c09ab50 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1681838640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f166803e210 0x7f16680406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1681838640 10 monclient: get_auth_request con 0x7f166803e210 auth_method 0 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 10 monclient(hunting): handle_auth_done global_id 234283 payload 824 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.691+0000 7f1680836640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f167c09dd10 con 0x7f167c1cd4b0 2024-09-10T16:28:58.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1680836640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 msgr2=0x7f167c1cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 0x7f167c1cd890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f1670010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1681838640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f166803e210 0x7f16680406d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f166c034000 tx=0x7f166c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f167000f040 con 0x7f167c1cd4b0 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 10 monclient: dump: 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.711 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1670010e90 con 0x7f167c1cd4b0 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 msgr2=0x7f167c1cd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 0x7f167c1cd890 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f1670010c80 comp rx=0 tx=0).stop 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 msgr2=0x7f167c1cd890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.694+0000 7f1679ffb640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cd4b0 0x7f167c1cd890 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f1670010c80 comp rx=0 tx=0).stop 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.700+0000 7f168329b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.700+0000 7f168329b640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 0x7f167c1357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.700+0000 7f168329b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.700+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f167c09d620 con 0x7f167c1cddd0 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 0x7f167c1357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient: get_auth_request con 0x7f167c1cddd0 auth_method 0 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 10 monclient(hunting): handle_auth_done global_id 234286 payload 824 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.701+0000 7f1680836640 1 -- 172.21.15.164:0/2662755530 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f167c0a44a0 con 0x7f167c1cddd0 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.703+0000 7f1680836640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 msgr2=0x7f167c1357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.703+0000 7f1680836640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 0x7f167c1357a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f16700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1670004240 con 0x7f167c1cddd0 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 10 monclient: dump: 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f1670004220 con 0x7f167c1cddd0 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 msgr2=0x7f167c1357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 0x7f167c1357a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f16700046c0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 msgr2=0x7f167c1357a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1679ffb640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f167c1cddd0 0x7f167c1357a0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f16700107c0 tx=0x7f16700046c0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.717 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 138988, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:28:58.718 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:28:58.719 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.720 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:28:58.721 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:28:58.722 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.723 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:28:58.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.726 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.727 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.729 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:28:58.730 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:28:58.731 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:28:58.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f166803e210 msgr2=0x7f16680406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f166803e210 0x7f16680406d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f166c034000 tx=0x7f166c02f000 comp rx=0 tx=0).stop 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 10 monclient: shutdown 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 msgr2=0x7f167c06e7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f167400c450 tx=0x7f167400c910 comp rx=0 tx=0).stop 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f1681838640 1 -- 172.21.15.164:0/2662755530 reap_dead start 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 shutdown_connections 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 --2- 172.21.15.164:0/2662755530 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f167c09ab50 0x7f167c06e7d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.704+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 >> 172.21.15.164:0/2662755530 conn(0x7f167c0a3110 msgr2=0x7f167c13c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.705+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 shutdown_connections 2024-09-10T16:28:58.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.705+0000 7f168329b640 1 -- 172.21.15.164:0/2662755530 wait complete. 2024-09-10T16:28:58.733 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:28:58.734 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:28:58.734 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:28:58.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 10 monclient: get_monmap_and_config 2024-09-10T16:28:58.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 10 monclient: monmap: 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:28:58.831030+0000 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:28:58.831030+0000 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:28:58.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 1 Processor -- start 2024-09-10T16:28:58.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 1 -- start start 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.830+0000 7f69731a3640 10 monclient: init 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient: _add_conns ranks=[1,0,2,4,3] 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f696c06cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f696c134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 0x7f696c140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-b con 0x7f696c13e740 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c09ce50 con 0x7f696c134f70 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f696c135380 con 0x7f696c06cb70 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f696c06db60 con 0x7f696c13e740 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f69731a3640 10 monclient(hunting): authenticate will time out at 2117.318441s 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6963fff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f696c134f70 0x7f696c13d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53594/0 (socket says 172.21.15.164:53594) 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6963fff640 1 -- 172.21.15.164:0/2188841876 learned_addr learned my addr 172.21.15.164:0/2188841876 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 1 --2- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 0x7f696c140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): get_auth_request con 0x7f696c13e740 auth_method 0 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient(hunting): handle_auth_done global_id 244214 payload 274 2024-09-10T16:28:58.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 1 -- 172.21.15.164:0/2188841876 >> v1:172.21.15.165:6789/0 conn(0x7f696c06cb70 legacy=0x7f696c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 1 -- 172.21.15.164:0/2188841876 >> v1:172.21.15.164:6789/0 conn(0x7f696c134f70 legacy=0x7f696c13d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 1 monclient: found mon.noname-b 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:28:58.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.831+0000 7f6971740640 1 -- 172.21.15.164:0/2188841876 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c09cb70 con 0x7f696c13e740 2024-09-10T16:28:58.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f6971740640 1 -- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 msgr2=0x7f696c140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f6971740640 1 --2- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 0x7f696c140b30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6964009df0 tx=0x7f696402a4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=901a2157869fc926 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 1 -- 172.21.15.164:0/2188841876 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964007a30 con 0x7f696c13e740 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 10 monclient: dump: 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69637fe640 1 -- 172.21.15.164:0/2188841876 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6964007bd0 con 0x7f696c13e740 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69731a3640 5 monclient: authenticate success, global_id 244214 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.834+0000 7f69731a3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69637fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 10 monclient: get_monmap_and_config success 2024-09-10T16:28:58.844 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69637fe640 1 -- 172.21.15.164:0/2188841876 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964031400 con 0x7f696c13e740 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69637fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 4 set_mon_vals no callback set 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69637fe640 10 monclient: dump: 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.845 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 10 monclient: shutdown 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 -- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 msgr2=0x7f696c140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 --2- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 0x7f696c140b30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6964009df0 tx=0x7f696402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 -- 172.21.15.164:0/2188841876 shutdown_connections 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 --2- 172.21.15.164:0/2188841876 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c13e740 0x7f696c140b30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 -- 172.21.15.164:0/2188841876 >> 172.21.15.164:0/2188841876 conn(0x7f696c0aa900 msgr2=0x7f696c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 -- 172.21.15.164:0/2188841876 shutdown_connections 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.835+0000 7f69731a3640 1 -- 172.21.15.164:0/2188841876 wait complete. 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.836+0000 7f69731a3640 10 monclient: build_initial_monmap 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.836+0000 7f69731a3640 10 monclient: monmap: 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:28:58.846 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.836+0000 7f69731a3640 1 Processor -- start 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.836+0000 7f69731a3640 1 -- start start 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.836+0000 7f69731a3640 10 monclient: init 2024-09-10T16:28:58.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient: _reopen_session rank -1 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 0x7f696c13acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-b con 0x7f696c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 0x7f696c13b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-c con 0x7f696c134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): picked mon.noname-a con 0x7f696c13e740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f696c0a4db0 con 0x7f696c13e740 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f696c06cf50 con 0x7f696c06cb70 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:28:58.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f696c141070 con 0x7f696c134f70 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): _renew_subs 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f69731a3640 10 monclient(hunting): authenticate will time out at 2117.324120s 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 0x7f696c13b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56720/0 (socket says 172.21.15.164:56720) 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 0x7f696c13b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53482/0 (socket says 172.21.15.164:53482) 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 -- 172.21.15.164:0/3087545313 learned_addr learned my addr 172.21.15.164:0/3087545313 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): get_auth_request con 0x7f696c13e740 auth_method 0 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): get_auth_request con 0x7f696c134f70 auth_method 0 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6963fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 0x7f696c13acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 10 monclient(hunting): get_auth_request con 0x7f696c06cb70 auth_method 0 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:28:58.850 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient(hunting): handle_auth_done global_id 254127 payload 824 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 -- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 msgr2=0x7f696c13b1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 0x7f696c13b1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 -- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 msgr2=0x7f696c13acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 0x7f696c13acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient: _finish_hunting 0 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 monclient: found mon.noname-a 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6971740640 1 -- 172.21.15.164:0/3087545313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f696c06e720 con 0x7f696c13e740 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.837+0000 7f6970f3f640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 0x7f696c13acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:28:58.851 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f6971740640 1 -- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 msgr2=0x7f696c13b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f6971740640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6964009d90 tx=0x7f6964004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6964040070 con 0x7f696c13e740 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: dump: 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.852 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: _finish_auth 0 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: _check_auth_tickets 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6964045660 con 0x7f696c13e740 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69731a3640 5 monclient: authenticate success, global_id 254127 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69731a3640 10 monclient: _renew_subs 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69731a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f696c09dfb0 con 0x7f696c13e740 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69640314c0 con 0x7f696c13e740 2024-09-10T16:28:58.853 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f69617fa640 10 monclient: dump: 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:28:58.854 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.840+0000 7f69731a3640 10 monclient: _renew_subs 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.839+0000 7f6962ffd640 4 set_mon_vals no callback set 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.840+0000 7f69731a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.840+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f696c09d610 con 0x7f696c13e740 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.840+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6964039040 con 0x7f696c13e740 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69617fa640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 0x7f6958040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69731a3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69731a3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69731a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f696c0a46a0 con 0x7f696c13e740 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(776..776 src has 677..776) ==== 4755+0+0 (secure 0 0 0) 0x7f69640747a0 con 0x7f696c13e740 2024-09-10T16:28:58.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f6970f3f640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 0x7f6958040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:28:58.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.841+0000 7f6970f3f640 10 monclient: get_auth_request con 0x7f695803e240 auth_method 0 2024-09-10T16:28:58.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.843+0000 7f6970f3f640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 0x7f6958040700 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6968006fd0 tx=0x7f6968008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:28:58.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.845+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6964060070 con 0x7f696c13e740 2024-09-10T16:28:58.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.845+0000 7f69617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:28:58.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:58.845+0000 7f69617fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:28:58.909 INFO:tasks.rados.rados.0.smithi165.stdout:3481: done (0 left) 2024-09-10T16:28:58.910 INFO:tasks.rados.rados.0.smithi165.stdout:3482: write oid 25 current snap is 388 2024-09-10T16:28:58.910 INFO:tasks.rados.rados.0.smithi165.stdout:3482: seq_num 1210 ranges {736470=751417,2022859=313611} 2024-09-10T16:28:58.918 INFO:tasks.rados.rados.0.smithi165.stdout:3482: writing smithi16534841-25 from 736470 to 1487887 tid 1 2024-09-10T16:28:58.923 INFO:tasks.rados.rados.0.smithi165.stdout:3482: writing smithi16534841-25 from 2022859 to 2336470 tid 2 2024-09-10T16:28:58.923 INFO:tasks.rados.rados.0.smithi165.stdout:3483: snap_remove snap 384 2024-09-10T16:28:59.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69731a3640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:28:59.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69731a3640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:28:59.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69731a3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:28:59.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f696c140bb0 con 0x7f696c13e740 2024-09-10T16:28:59.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69617fa640 1 -- 172.21.15.164:0/3087545313 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f6964074020 con 0x7f696c13e740 2024-09-10T16:28:59.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:28:59.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.091+0000 7f69617fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=12,logm=48} crc {auth=3134712501,config=1228700967,health=1544899556,kv=805391194,logm=2711884740}) 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=3134712501,config=2586702516,health=3105631864,kv=805391194,logm=2611644033}) 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=12,kv=12,logm=48} crc {auth=3134712501,config=1228700967,health=1544899556,kv=805391194,logm=2711884740}) 2024-09-10T16:28:59.094 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.092+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=10,kv=11,logm=56} crc {auth=4132798231,config=2586702516,health=367856401,kv=2518733882,logm=2775968860}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175761186}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217737844}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175761186}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.093+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=817164651}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236553872}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4263651712}) 2024-09-10T16:28:59.096 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.097 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3236553872}) 2024-09-10T16:28:59.097 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.094+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3793223199}) 2024-09-10T16:28:59.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 msgr2=0x7f6958040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 0x7f6958040700 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7f6968006fd0 tx=0x7f6968008040 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 10 monclient: shutdown 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 msgr2=0x7f696c13b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6964009d90 tx=0x7f6964004a40 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 shutdown_connections 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f695803e240 0x7f6958040700 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f696c13e740 0x7f696c13b730 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f696c134f70 0x7f696c13b1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 --2- 172.21.15.164:0/3087545313 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f696c06cb70 0x7f696c13acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:28:59.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 >> 172.21.15.164:0/3087545313 conn(0x7f696c0aa900 msgr2=0x7f696c1327b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:28:59.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 shutdown_connections 2024-09-10T16:28:59.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:28:59.095+0000 7f69731a3640 1 -- 172.21.15.164:0/3087545313 wait complete. 2024-09-10T16:28:59.106 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905980642}) 2024-09-10T16:28:59.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1502654293}) 2024-09-10T16:28:59.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=905980642}) 2024-09-10T16:28:59.107 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.105+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465792591}) 2024-09-10T16:28:59.108 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:28:59.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.106+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.109 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748582711}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4218430136}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2748582711}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=327241184}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16595110}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=245404959}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=16595110}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.107+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=905247941}) 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.110 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979630228}) 2024-09-10T16:28:59.111 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2547678392}) 2024-09-10T16:28:59.111 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.111 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2979630228}) 2024-09-10T16:28:59.111 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.108+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1945395340}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652998624}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4099934478}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652998624}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4098834213}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2622662355}) 2024-09-10T16:28:59.114 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=408016460}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2622662355}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.109+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181160804}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948266829}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4037050272}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2948266829}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.110+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1589211213}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344990777}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=135695859}) 2024-09-10T16:28:59.115 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1344990777}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.111+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1908905279}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941312250}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26387231}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=941312250}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=723233497}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426582743}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3946420822}) 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.116 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426582743}) 2024-09-10T16:28:59.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.112+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3657967216}) 2024-09-10T16:28:59.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833453977}) 2024-09-10T16:28:59.117 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1936601254}) 2024-09-10T16:28:59.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833453977}) 2024-09-10T16:28:59.118 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.113+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1751891557}) 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673336244}) 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4108446637}) 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1673336244}) 2024-09-10T16:28:59.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4241851008}) 2024-09-10T16:28:59.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771091012}) 2024-09-10T16:28:59.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1252133530}) 2024-09-10T16:28:59.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1771091012}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2697698101}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762052472}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2211627054}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1762052472}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2298908006}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947723612}) 2024-09-10T16:28:59.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4148666343}) 2024-09-10T16:28:59.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947723612}) 2024-09-10T16:28:59.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2573472672}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179087867}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3743499446}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4179087867}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=429199369}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790768911}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3536039451}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790768911}) 2024-09-10T16:28:59.196 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2590863787}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250741093}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1572005915}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3250741093}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3735277646}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551047152}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4278798238}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=551047152}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341441383}) 2024-09-10T16:28:59.197 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913633757}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581421992}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913633757}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762188612}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3220854852}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729192254}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3220854852}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504953974}) 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556670455}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2923769247}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2556670455}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2728066712}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391871629}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=443205927}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391871629}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2959102391}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049201325}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=683225779}) 2024-09-10T16:28:59.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4049201325}) 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=718409784}) 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448788204}) 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2059742751}) 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448788204}) 2024-09-10T16:28:59.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=359462293}) 2024-09-10T16:28:59.211 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503057219}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=943237180}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1503057219}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760109489}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840210278}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274818126}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3840210278}) 2024-09-10T16:28:59.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237674351}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615981006}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723348247}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1615981006}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2278615504}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955460121}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3874038400}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1955460121}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=812689720}) 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125615935}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1034157071}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1125615935}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153036349}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259719040}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104634600}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3259719040}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847118949}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444104170}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3566256520}) 2024-09-10T16:28:59.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2444104170}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2988956763}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482965228}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3261451182}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1482965228}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1250221421}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424999563}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=956142851}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424999563}) 2024-09-10T16:28:59.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3699588066}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791185423}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1998979622}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3791185423}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892165920}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715790512}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160058919}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=715790512}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3556525800}) 2024-09-10T16:28:59.216 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704341095}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=987120511}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1704341095}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3107378790}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759482872}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1685986586}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2759482872}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2589471228}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298305567}) 2024-09-10T16:28:59.217 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=827223488}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=298305567}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2984901759}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548600606}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2413213096}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3548600606}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1278960190}) 2024-09-10T16:28:59.218 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211125767}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=669024338}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3211125767}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2314811564}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337645269}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2626647966}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2337645269}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3817969753}) 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.219 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817004098}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1569582923}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2817004098}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=471939418}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658728138}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3606796399}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3658728138}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2950449003}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345975594}) 2024-09-10T16:28:59.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=12763478}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3345975594}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2567080310}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925102434}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2896330672}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=925102434}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1743359668}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589241160}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2527249680}) 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.221 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1589241160}) 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961003309}) 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731848643}) 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4129008065}) 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1731848643}) 2024-09-10T16:28:59.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1972472450}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008291338}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162820631}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4008291338}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252567285}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849860216}) 2024-09-10T16:28:59.226 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2386558380}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3849860216}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1544901413}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542951933}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3830258257}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=542951933}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1259675219}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265731963}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3150577455}) 2024-09-10T16:28:59.227 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4265731963}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2727376459}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892437123}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1060357700}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3892437123}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2130258621}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446740898}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1440597974}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446740898}) 2024-09-10T16:28:59.228 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.215+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2480164303}) 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432641592}) 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=509993018}) 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432641592}) 2024-09-10T16:28:59.229 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.216+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1703597259}) 2024-09-10T16:28:59.594 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.594 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913613555}) 2024-09-10T16:28:59.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2164832183}) 2024-09-10T16:28:59.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=913613555}) 2024-09-10T16:28:59.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2497432811}) 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141360310}) 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=929217260}) 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3141360310}) 2024-09-10T16:28:59.649 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.647+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3053707887}) 2024-09-10T16:28:59.765 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877597184}) 2024-09-10T16:28:59.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1947251836}) 2024-09-10T16:28:59.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2877597184}) 2024-09-10T16:28:59.766 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446656528}) 2024-09-10T16:28:59.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.858 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240554291}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=498937567}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240554291}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.855+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=336993929}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.856+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.856+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1761756381}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.856+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=899124558}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.857+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.857+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1761756381}) 2024-09-10T16:28:59.859 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.857+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3211642298}) 2024-09-10T16:28:59.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.857+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339825453}) 2024-09-10T16:28:59.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1297532730}) 2024-09-10T16:28:59.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.862 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1339825453}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=383810493}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558701376}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3213278865}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3558701376}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.858+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1288054059}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498092797}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2290234013}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498092797}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.859+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1427933883}) 2024-09-10T16:28:59.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332022599}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4213517534}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332022599}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1300796218}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620263549}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3248544192}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620263549}) 2024-09-10T16:28:59.864 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.860+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3220110967}) 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617060503}) 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1320850885}) 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2617060503}) 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=548687288}) 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390459320}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=410576776}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=390459320}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3071168483}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645080596}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1005823460}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=645080596}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90355265}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178815713}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019241589}) 2024-09-10T16:28:59.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178815713}) 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1394098732}) 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472953840}) 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=437604454}) 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=472953840}) 2024-09-10T16:28:59.869 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1937942023}) 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550969024}) 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2291705802}) 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=550969024}) 2024-09-10T16:28:59.917 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.915+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2344082524}) 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191252832}) 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3664374878}) 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4191252832}) 2024-09-10T16:28:59.932 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.928+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1941225611}) 2024-09-10T16:28:59.936 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328760368}) 2024-09-10T16:28:59.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3252420947}) 2024-09-10T16:28:59.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:28:59.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3328760368}) 2024-09-10T16:28:59.937 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:28:59.933+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134031036}) 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3483: done (1 left) 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3484: snap_create 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: finishing write tid 1 to smithi16534841-25 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: finishing write tid 2 to smithi16534841-25 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: finishing write tid 3 to smithi16534841-25 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: finishing write tid 4 to smithi16534841-25 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: oid 25 updating version 0 to 1874 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: oid 25 updating version 1874 to 1875 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: oid 25 updating version 1875 to 1876 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1876 (ObjNum 1210 snap 388 seq_num 1210) dirty exists 2024-09-10T16:28:59.938 INFO:tasks.rados.rados.0.smithi165.stdout:3482: left oid 25 (ObjNum 1210 snap 388 seq_num 1210) 2024-09-10T16:29:00.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.025 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694106236}) 2024-09-10T16:29:00.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2262949437}) 2024-09-10T16:29:00.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2694106236}) 2024-09-10T16:29:00.026 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.023+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3508239422}) 2024-09-10T16:29:00.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533050903}) 2024-09-10T16:29:00.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=680925954}) 2024-09-10T16:29:00.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.029 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=533050903}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.024+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2666586846}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092089059}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693089806}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1092089059}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.025+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1775024515}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156350123}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3414582468}) 2024-09-10T16:29:00.030 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=156350123}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.026+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=811492386}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906771196}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4091040681}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=906771196}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3396616944}) 2024-09-10T16:29:00.031 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644641519}) 2024-09-10T16:29:00.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2209696772}) 2024-09-10T16:29:00.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1644641519}) 2024-09-10T16:29:00.032 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.027+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3872550939}) 2024-09-10T16:29:00.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956787040}) 2024-09-10T16:29:00.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4133527759}) 2024-09-10T16:29:00.033 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1956787040}) 2024-09-10T16:29:00.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.028+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3684568931}) 2024-09-10T16:29:00.108 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:29:00.109 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.208+0000 7f7188d4b640 10 monclient: get_monmap_and_config 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.208+0000 7f7188d4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.208+0000 7f7188d4b640 10 monclient: monmap: 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:00.209448+0000 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:00.209448+0000 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:00.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:00.211 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:00.211 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:00.211 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:00.211 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 Processor -- start 2024-09-10T16:29:00.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 -- start start 2024-09-10T16:29:00.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient: init 2024-09-10T16:29:00.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 0x7f718409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f718409c310 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f718409d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f718413a5c0 0x7f718413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f718413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f718409fbc0 con 0x7f718409d190 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f71840a2a20 con 0x7f718413a5c0 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f718409c6f0 con 0x7f718409c310 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.209+0000 7f7188d4b640 10 monclient(hunting): authenticate will time out at 2118.696918s 2024-09-10T16:29:00.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f718413a5c0 0x7f718413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f718413a5c0 0x7f718413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56724/0 (socket says 172.21.15.164:56724) 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 -- 172.21.15.164:0/2711957977 learned_addr learned my addr 172.21.15.164:0/2711957977 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): get_auth_request con 0x7f718413a5c0 auth_method 0 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f718259c640 1 --2- 172.21.15.164:0/2711957977 >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 0x7f718409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient(hunting): handle_auth_done global_id 254130 payload 274 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 -- 172.21.15.164:0/2711957977 >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 msgr2=0x7f718409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 --2- 172.21.15.164:0/2711957977 >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 0x7f718409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 -- 172.21.15.164:0/2711957977 >> v1:172.21.15.165:6789/0 conn(0x7f718409d190 legacy=0x7f718409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 monclient: found mon.noname-a 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f7182d9d640 1 -- 172.21.15.164:0/2711957977 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f718409a4d0 con 0x7f718413a5c0 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f718259c640 10 monclient: get_auth_request con 0x7f718409c310 auth_method 0 2024-09-10T16:29:00.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.210+0000 7f718259c640 1 --2- 172.21.15.164:0/2711957977 >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 0x7f718409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:00.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7182d9d640 1 -- 172.21.15.164:0/2711957977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 msgr2=0x7f718413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7182d9d640 1 --2- 172.21.15.164:0/2711957977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f718413c9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f718406d4d0 tx=0x7f717c02a570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5638fe5616cd017 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 1 -- 172.21.15.164:0/2711957977 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c036070 con 0x7f718413a5c0 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: dump: 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.222 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 5 monclient: authenticate success, global_id 254130 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 1 -- 172.21.15.164:0/2711957977 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f717c007800 con 0x7f718413a5c0 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 1 -- 172.21.15.164:0/2711957977 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f717c007bd0 con 0x7f718413a5c0 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 4 set_mon_vals no callback set 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7181d9b640 10 monclient: dump: 2024-09-10T16:29:00.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 10 monclient: shutdown 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 -- 172.21.15.164:0/2711957977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 msgr2=0x7f718413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 --2- 172.21.15.164:0/2711957977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f718413c9b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f718406d4d0 tx=0x7f717c02a570 comp rx=0 tx=0).stop 2024-09-10T16:29:00.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 -- 172.21.15.164:0/2711957977 shutdown_connections 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 --2- 172.21.15.164:0/2711957977 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f718413c9b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 --2- 172.21.15.164:0/2711957977 >> v2:172.21.15.165:3300/0 conn(0x7f718409c310 0x7f718409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 -- 172.21.15.164:0/2711957977 >> 172.21.15.164:0/2711957977 conn(0x7f71840a3110 msgr2=0x7f71840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.213+0000 7f7188d4b640 1 -- 172.21.15.164:0/2711957977 shutdown_connections 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.214+0000 7f7188d4b640 1 -- 172.21.15.164:0/2711957977 wait complete. 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.214+0000 7f7188d4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.214+0000 7f7188d4b640 10 monclient: monmap: 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:00.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 Processor -- start 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 -- start start 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient: init 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 0x7f7184095fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f718409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f718413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7184139b10 0x7f7184139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7184139b10 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f71840a2230 con 0x7f718413a5c0 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f71840a20b0 con 0x7f7184139b10 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7184074e00 con 0x7f718409c310 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7188d4b640 10 monclient(hunting): authenticate will time out at 2118.702590s 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56734/0 (socket says 172.21.15.164:56734) 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 0x7f7184095fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 1 -- 172.21.15.164:0/3653381335 learned_addr learned my addr 172.21.15.164:0/3653381335 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): get_auth_request con 0x7f718413a5c0 auth_method 0 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): get_auth_request con 0x7f718409c310 auth_method 0 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7184139b10 0x7f7184139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): get_auth_request con 0x7f7184139b10 auth_method 0 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f718259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f7182d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.215+0000 7f717bfff640 10 monclient(hunting): handle_auth_done global_id 254133 payload 824 2024-09-10T16:29:00.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 -- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 msgr2=0x7f7184095fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 0x7f7184095fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 -- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7184139b10 msgr2=0x7f7184139fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7184139b10 0x7f7184139fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 monclient: found mon.noname-a 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f717bfff640 1 -- 172.21.15.164:0/3653381335 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71840977e0 con 0x7f718413a5c0 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.216+0000 7f718259c640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 0x7f7184095fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717bfff640 1 -- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 msgr2=0x7f71841395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717bfff640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c400 tx=0x7f716c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c017070 con 0x7f718413a5c0 2024-09-10T16:29:00.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: dump: 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 5 monclient: authenticate success, global_id 254133 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f716c0037f0 con 0x7f718413a5c0 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 10 monclient: _renew_subs 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716c003be0 con 0x7f718413a5c0 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f718409d620 con 0x7f718413a5c0 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7180d99640 4 set_mon_vals no callback set 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f717affd640 10 monclient: dump: 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.232 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 10 monclient: _renew_subs 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.218+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7184097500 con 0x7f718413a5c0 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f716c01a070 con 0x7f718413a5c0 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f7188d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f7188d4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f7188d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f718409f720 con 0x7f718413a5c0 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f717affd640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 0x7f71640406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.219+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7f716c052240 con 0x7f718413a5c0 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.220+0000 7f718259c640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 0x7f71640406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.220+0000 7f718259c640 10 monclient: get_auth_request con 0x7f716403e210 auth_method 0 2024-09-10T16:29:00.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.222+0000 7f718259c640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 0x7f71640406d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f717400af30 tx=0x7f7174008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.224+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f716c050c30 con 0x7f718413a5c0 2024-09-10T16:29:00.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.224+0000 7f717affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.224+0000 7f717affd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.464+0000 7f7188d4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.464+0000 7f7188d4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.464+0000 7f7188d4b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.464+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7184077260 con 0x7f718413a5c0 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.464+0000 7f717affd640 1 -- 172.21.15.164:0/3653381335 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f716c012c50 con 0x7f718413a5c0 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.465+0000 7f717affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:00.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.465+0000 7f717affd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:00.467 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:00.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 msgr2=0x7f71640406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 0x7f71640406d0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f717400af30 tx=0x7f7174008040 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 10 monclient: shutdown 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 msgr2=0x7f71841395d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f716c00c400 tx=0x7f716c00c8c0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 shutdown_connections 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f716403e210 0x7f71640406d0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7184139b10 0x7f7184139fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f718413a5c0 0x7f71841395d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 --2- 172.21.15.164:0/3653381335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f718409c310 0x7f7184095fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 >> 172.21.15.164:0/3653381335 conn(0x7f71840a3110 msgr2=0x7f7184099730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 shutdown_connections 2024-09-10T16:29:00.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.467+0000 7f7188d4b640 1 -- 172.21.15.164:0/3653381335 wait complete. 2024-09-10T16:29:00.478 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:29:00.479 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:29:00.479 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:29:00.479 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.575+0000 7f95e2fd3640 10 monclient: get_monmap_and_config 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.575+0000 7f95e2fd3640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.575+0000 7f95e2fd3640 10 monclient: monmap: 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:00.576782+0000 2024-09-10T16:29:00.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:00.576782+0000 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:00.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.576+0000 7f95e2fd3640 1 Processor -- start 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.576+0000 7f95e2fd3640 1 -- start start 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.576+0000 7f95e2fd3640 10 monclient: init 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient: _add_conns ranks=[2,1,4,3,0] 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-b con 0x7f95dc09c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f95dc09d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:00.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 0x7f95dc13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-c con 0x7f95dc13a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95dc09fbc0 con 0x7f95dc09d190 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f95dc0a2a20 con 0x7f95dc13a5c0 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95dc09c6f0 con 0x7f95dc09c310 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e2fd3640 10 monclient(hunting): authenticate will time out at 2119.064309s 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53498/0 (socket says 172.21.15.164:53498) 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1849415781 learned_addr learned my addr 172.21.15.164:0/1849415781 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request con 0x7f95dc09c310 auth_method 0 2024-09-10T16:29:00.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_done global_id 244223 payload 274 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1849415781 >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 msgr2=0x7f95dc13c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e1570640 1 --2- 172.21.15.164:0/1849415781 >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 0x7f95dc13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1849415781 >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 0x7f95dc13c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1849415781 >> v1:172.21.15.165:6789/0 conn(0x7f95dc09d190 legacy=0x7f95dc09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 monclient: found mon.noname-b 2024-09-10T16:29:00.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1849415781 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc09a4d0 con 0x7f95dc09c310 2024-09-10T16:29:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e1570640 10 monclient: get_auth_request con 0x7f95dc13a5c0 auth_method 0 2024-09-10T16:29:00.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.577+0000 7f95e1570640 1 --2- 172.21.15.164:0/1849415781 >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 0x7f95dc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1849415781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 msgr2=0x7f95dc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1849415781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009870 tx=0x7f95cc02a130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71004acad05d3bd6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 1 -- 172.21.15.164:0/1849415781 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc037070 con 0x7f95dc09c310 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: dump: 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.591 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e2fd3640 5 monclient: authenticate success, global_id 244223 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 1 -- 172.21.15.164:0/1849415781 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95cc032640 con 0x7f95dc09c310 2024-09-10T16:29:00.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e2fd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e2fd3640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 1 -- 172.21.15.164:0/1849415781 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc032a10 con 0x7f95dc09c310 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95e2fd3640 4 set_mon_vals no callback set 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.580+0000 7f95db7fe640 10 monclient: dump: 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.593 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 10 monclient: shutdown 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1849415781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 msgr2=0x7f95dc09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1849415781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009870 tx=0x7f95cc02a130 comp rx=0 tx=0).stop 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1849415781 shutdown_connections 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1849415781 >> v2:172.21.15.165:3300/0 conn(0x7f95dc13a5c0 0x7f95dc13c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1849415781 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc09c310 0x7f95dc09ea40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1849415781 >> 172.21.15.164:0/1849415781 conn(0x7f95dc0a3110 msgr2=0x7f95dc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1849415781 shutdown_connections 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1849415781 wait complete. 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.581+0000 7f95e2fd3640 10 monclient: monmap: 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:00.595 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 1 Processor -- start 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 1 -- start start 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient: init 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95dc09c310 0x7f95dc09d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-b con 0x7f95dc09c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc13a5c0 0x7f95dc1395d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-a con 0x7f95dc13a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 0x7f95dc139fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.582+0000 7f95e2fd3640 10 monclient(hunting): picked mon.noname-c con 0x7f95dc139b10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc0a2230 con 0x7f95dc13a5c0 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc0a20b0 con 0x7f95dc09c310 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f95dc074e00 con 0x7f95dc139b10 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e2fd3640 10 monclient(hunting): authenticate will time out at 2119.070023s 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 0x7f95dc139fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 0x7f95dc139fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53508/0 (socket says 172.21.15.164:53508) 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 learned_addr learned my addr 172.21.15.164:0/1066460788 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): get_auth_request con 0x7f95dc139b10 auth_method 0 2024-09-10T16:29:00.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc13a5c0 0x7f95dc1395d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): get_auth_request con 0x7f95dc13a5c0 auth_method 0 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95dc09c310 0x7f95dc09d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request con 0x7f95dc09c310 auth_method 0 2024-09-10T16:29:00.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient(hunting): handle_auth_done global_id 244226 payload 824 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95dc09c310 msgr2=0x7f95dc09d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f95dc09c310 0x7f95dc09d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc13a5c0 msgr2=0x7f95dc1395d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc13a5c0 0x7f95dc1395d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 monclient: found mon.noname-c 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.583+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95dc09e270 con 0x7f95dc139b10 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 msgr2=0x7f95dc139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e1570640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 0x7f95dc139fa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95d400cf20 tx=0x7f95d400a7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d401a070 con 0x7f95dc139b10 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: dump: 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95d400da50 con 0x7f95dc139b10 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 5 monclient: authenticate success, global_id 244226 2024-09-10T16:29:00.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 10 monclient: _renew_subs 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95dc098400 con 0x7f95dc139b10 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95d400de40 con 0x7f95dc139b10 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95d97fa640 10 monclient: dump: 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.602 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95daffd640 4 set_mon_vals no callback set 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 10 monclient: _renew_subs 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.585+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f95dc092a70 con 0x7f95dc139b10 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e2fd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 0x7f95dc091400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc09f330 con 0x7f95dc09d190 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f95d401d070 con 0x7f95dc139b10 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 0x7f95dc091400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient: get_auth_request con 0x7f95dc09d190 auth_method 0 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.586+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_done global_id 254139 payload 824 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95d97fa640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95c803e290 0x7f95c8040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95dc09f720 con 0x7f95dc09d190 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7f95d4053630 con 0x7f95dc139b10 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95dbfff640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95c803e290 0x7f95c8040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.587+0000 7f95dbfff640 10 monclient: get_auth_request con 0x7f95c803e290 auth_method 0 2024-09-10T16:29:00.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 msgr2=0x7f95dc091400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 0x7f95dc091400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95cc034000 tx=0x7f95cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc032e60 con 0x7f95dc09d190 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 10 monclient: dump: 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.605 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95cc03c850 con 0x7f95dc09d190 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 msgr2=0x7f95dc091400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 0x7f95dc091400 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95cc034000 tx=0x7f95cc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 msgr2=0x7f95dc091400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95d97fa640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc09d190 0x7f95dc091400 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.589+0000 7f95dbfff640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95c803e290 0x7f95c8040750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95dc09e8e0 tx=0x7f95d0007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611197537}) 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3154350799}) 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1611197537}) 2024-09-10T16:29:00.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1282224572}) 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e2fd3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 0x7f95dc091d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e2fd3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f95dc09d620 con 0x7f95dc091940 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 0x7f95dc091d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient: get_auth_request con 0x7f95dc091940 auth_method 0 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 10 monclient(hunting): handle_auth_done global_id 254142 payload 824 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.596+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1066460788 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95dc092290 con 0x7f95dc091940 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e0d6f640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 msgr2=0x7f95dc091d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e0d6f640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 0x7f95dc091d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95cc02a170 tx=0x7f95cc008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95cc032e60 con 0x7f95dc091940 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 10 monclient: dump: 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f95cc003c30 con 0x7f95dc091940 2024-09-10T16:29:00.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 msgr2=0x7f95dc091d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 0x7f95dc091d50 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95cc02a170 tx=0x7f95cc008030 comp rx=0 tx=0).stop 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 msgr2=0x7f95dc091d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95d97fa640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f95dc091940 0x7f95dc091d50 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f95cc02a170 tx=0x7f95cc008030 comp rx=0 tx=0).stop 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:29:00.612 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 29250, 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.613 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.614 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.615 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:00.616 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.617 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.618 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.619 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.620 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.621 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:00.622 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.623 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.624 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:00.625 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:00.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95c803e290 msgr2=0x7f95c8040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f95c803e290 0x7f95c8040750 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f95dc09e8e0 tx=0x7f95d0007e30 comp rx=0 tx=0).stop 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 10 monclient: shutdown 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 msgr2=0x7f95dc139fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 --2- 172.21.15.164:0/1066460788 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f95dc139b10 0x7f95dc139fa0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f95d400cf20 tx=0x7f95d400a7d0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 reap_dead start 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e1570640 1 -- 172.21.15.164:0/1066460788 reap_dead start 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 shutdown_connections 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 >> 172.21.15.164:0/1066460788 conn(0x7f95dc0a3110 msgr2=0x7f95dc070d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 shutdown_connections 2024-09-10T16:29:00.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.599+0000 7f95e2fd3640 1 -- 172.21.15.164:0/1066460788 wait complete. 2024-09-10T16:29:00.628 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:00.628 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:29:00.628 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:29:00.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 10 monclient: get_monmap_and_config 2024-09-10T16:29:00.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 10 monclient: monmap: 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:00.727074+0000 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:00.727074+0000 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:00.728 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 1 Processor -- start 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 1 -- start start 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.726+0000 7efd2d039640 10 monclient: init 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 0x7efd28134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-b con 0x7efd2806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efd280a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd280a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2809ce50 con 0x7efd280a46a0 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd28138ac0 con 0x7efd280a4a80 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd2806db60 con 0x7efd2806cb70 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2d039640 10 monclient(hunting): authenticate will time out at 2119.214389s 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 0x7efd28134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2659c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7efd280a46a0 0x7efd28137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53610/0 (socket says 172.21.15.164:53610) 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd2659c640 1 -- 172.21.15.164:0/1524267806 learned_addr learned my addr 172.21.15.164:0/1524267806 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): get_auth_request con 0x7efd2806cb70 auth_method 0 2024-09-10T16:29:00.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient(hunting): handle_auth_done global_id 244229 payload 274 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 1 -- 172.21.15.164:0/1524267806 >> v1:172.21.15.165:6789/0 conn(0x7efd280a4a80 legacy=0x7efd281383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 1 -- 172.21.15.164:0/1524267806 >> v1:172.21.15.164:6789/0 conn(0x7efd280a46a0 legacy=0x7efd28137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 1 monclient: found mon.noname-b 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.727+0000 7efd26d9d640 1 -- 172.21.15.164:0/1524267806 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2809cb70 con 0x7efd2806cb70 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd26d9d640 1 -- 172.21.15.164:0/1524267806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 msgr2=0x7efd28134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd26d9d640 1 --2- 172.21.15.164:0/1524267806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 0x7efd28134f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009dc0 tx=0x7efd1c02a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24aa85f0dbd571ed server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 1 -- 172.21.15.164:0/1524267806 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c007a30 con 0x7efd2806cb70 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: dump: 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.741 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 1 -- 172.21.15.164:0/1524267806 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1c007bd0 con 0x7efd2806cb70 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd2d039640 5 monclient: authenticate success, global_id 244229 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd2d039640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:00.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd2d039640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 1 -- 172.21.15.164:0/1524267806 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c031400 con 0x7efd2806cb70 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd2d039640 4 set_mon_vals no callback set 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.730+0000 7efd25d9b640 10 monclient: dump: 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.743 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 10 monclient: shutdown 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 -- 172.21.15.164:0/1524267806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 msgr2=0x7efd28134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 --2- 172.21.15.164:0/1524267806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 0x7efd28134f70 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efd1c009dc0 tx=0x7efd1c02a480 comp rx=0 tx=0).stop 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 -- 172.21.15.164:0/1524267806 shutdown_connections 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 --2- 172.21.15.164:0/1524267806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd2806cb70 0x7efd28134f70 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 -- 172.21.15.164:0/1524267806 >> 172.21.15.164:0/1524267806 conn(0x7efd280aa900 msgr2=0x7efd280aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 -- 172.21.15.164:0/1524267806 shutdown_connections 2024-09-10T16:29:00.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.731+0000 7efd2d039640 1 -- 172.21.15.164:0/1524267806 wait complete. 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient: monmap: 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 1 Processor -- start 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 1 -- start start 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient: init 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd2806cb70 0x7efd281cd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-b con 0x7efd2806cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.732+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-a con 0x7efd280a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a4a80 0x7efd28140570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): picked mon.noname-c con 0x7efd280a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7efd2806cf50 con 0x7efd280a46a0 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7efd28141050 con 0x7efd2806cb70 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd281411d0 con 0x7efd280a4a80 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2d039640 10 monclient(hunting): authenticate will time out at 2119.220034s 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56790/0 (socket says 172.21.15.164:56790) 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 -- 172.21.15.164:0/805111263 learned_addr learned my addr 172.21.15.164:0/805111263 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): get_auth_request con 0x7efd280a46a0 auth_method 0 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a4a80 0x7efd28140570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): get_auth_request con 0x7efd280a4a80 auth_method 0 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd2806cb70 0x7efd281cd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): get_auth_request con 0x7efd2806cb70 auth_method 0 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient(hunting): handle_auth_done global_id 254145 payload 824 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a4a80 msgr2=0x7efd28140570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a4a80 0x7efd28140570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd2806cb70 msgr2=0x7efd281cd540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7efd2806cb70 0x7efd281cd540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 monclient: found mon.noname-a 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2659c640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2813a220 con 0x7efd280a46a0 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.733+0000 7efd2759e640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a4a80 0x7efd28140570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2659c640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 msgr2=0x7efd28140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2659c640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd1800c460 tx=0x7efd1800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18007070 con 0x7efd280a46a0 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: dump: 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.750 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 5 monclient: authenticate success, global_id 254145 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd18007230 con 0x7efd280a46a0 2024-09-10T16:29:00.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18007620 con 0x7efd280a46a0 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 10 monclient: _renew_subs 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd280a8a20 con 0x7efd280a46a0 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd0f7fe640 10 monclient: dump: 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.752 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2559a640 4 set_mon_vals no callback set 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 10 monclient: _renew_subs 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.735+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd280a4050 con 0x7efd280a46a0 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd2d039640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd2d039640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 0x7efd2802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd280a4f10 con 0x7efd280a2f00 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 0x7efd2802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient: get_auth_request con 0x7efd280a2f00 auth_method 0 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd2659c640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd1800c460 tx=0x7efd1800c920 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7efd18052990 osd_map(778..778 src has 677..778) 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7efd1801b400 con 0x7efd280a46a0 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 10 monclient(hunting): handle_auth_done global_id 244235 payload 824 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.736+0000 7efd26d9d640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd28134f70 con 0x7efd280a2f00 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.737+0000 7efd0f7fe640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efd1403e210 0x7efd140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.737+0000 7efd2759e640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efd1403e210 0x7efd140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.737+0000 7efd2759e640 10 monclient: get_auth_request con 0x7efd1403e210 auth_method 0 2024-09-10T16:29:00.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd26d9d640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 msgr2=0x7efd2802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd26d9d640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 0x7efd2802d440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd1c034000 tx=0x7efd1c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c031cd0 con 0x7efd280a2f00 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 10 monclient: dump: 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd1c045430 con 0x7efd280a2f00 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 msgr2=0x7efd2802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 0x7efd2802d440 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7efd1c034000 tx=0x7efd1c004a40 comp rx=0 tx=0).stop 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 msgr2=0x7efd2802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd0f7fe640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a2f00 0x7efd2802d440 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.739+0000 7efd2759e640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efd1403e210 0x7efd140406d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efd281ce390 tx=0x7efd20007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd2d039640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd2d039640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 0x7efd2802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd2d039640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7efd281392c0 con 0x7efd280a32e0 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 0x7efd2802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient: get_auth_request con 0x7efd280a32e0 auth_method 0 2024-09-10T16:29:00.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 10 monclient(hunting): handle_auth_done global_id 244238 payload 824 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.745+0000 7efd26d9d640 1 -- 172.21.15.164:0/805111263 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd2809ed00 con 0x7efd280a32e0 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd26d9d640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 msgr2=0x7efd2802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd26d9d640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 0x7efd2802d980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd2802eb00 tx=0x7efd1c030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd1c004450 con 0x7efd280a32e0 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 10 monclient: dump: 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7efd1c004450 con 0x7efd280a32e0 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:00.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 msgr2=0x7efd2802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 0x7efd2802d980 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd2802eb00 tx=0x7efd1c030000 comp rx=0 tx=0).stop 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 msgr2=0x7efd2802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7efd280a32e0 0x7efd2802d980 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7efd2802eb00 tx=0x7efd1c030000 comp rx=0 tx=0).stop 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:29:00.761 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 77656, 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.762 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.763 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:00.764 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.765 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.766 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.767 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.768 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.769 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.770 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.771 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.772 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:00.773 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:00.774 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:00.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efd1403e210 msgr2=0x7efd140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7efd1403e210 0x7efd140406d0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7efd281ce390 tx=0x7efd20007db0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 10 monclient: shutdown 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 msgr2=0x7efd28140030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efd1800c460 tx=0x7efd1800c920 comp rx=0 tx=0).stop 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2759e640 1 -- 172.21.15.164:0/805111263 reap_dead start 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd0f7fe640 1 -- 172.21.15.164:0/805111263 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (unknown 0 0 0) 0x7efd18052990 con 0x7efd280a46a0 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 shutdown_connections 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 --2- 172.21.15.164:0/805111263 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7efd280a46a0 0x7efd28140030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 >> 172.21.15.164:0/805111263 conn(0x7efd280aa900 msgr2=0x7efd28137880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 shutdown_connections 2024-09-10T16:29:00.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.748+0000 7efd2d039640 1 -- 172.21.15.164:0/805111263 wait complete. 2024-09-10T16:29:00.777 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:00.777 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:29:00.777 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881633865}) 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3238890339}) 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2881633865}) 2024-09-10T16:29:00.863 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.861+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4212928868}) 2024-09-10T16:29:00.865 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160913829}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2072722823}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2160913829}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.862+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334271715}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770013733}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=252332180}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1770013733}) 2024-09-10T16:29:00.866 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115479016}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118421966}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2336448771}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2118421966}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777043041}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263716839}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2966228436}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1263716839}) 2024-09-10T16:29:00.867 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.863+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3516781660}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950809074}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2354194204}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3950809074}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=708399275}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199324169}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3666626101}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4199324169}) 2024-09-10T16:29:00.868 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.864+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988695756}) 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2990588726,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3843541130}) 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2748362100,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2633766621}) 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12} crc {logm=2990588726,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3843541130}) 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.865+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=77,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=934968265,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2633766621}) 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.870 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=68} crc {mgrstat=2191700149,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=128661328,osd_snap=264153757}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=3,mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=71} crc {mgrstat=2191700149,mon_config_key=3292199900,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=128661328,osd_snap=645299436}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=3,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=68} crc {mgrstat=2191700149,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=128661328,osd_snap=264153757}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=3,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=68} crc {mgrstat=2191700149,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_snap=846093792}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=3532992505,osdmap=3376960448}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=86,osdmap=14} crc {osd_snap=2757608417,osdmap=439861797}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=76,osdmap=24} crc {osd_snap=3532992505,osdmap=3376960448}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.866+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=84,osdmap=16} crc {osd_snap=1897775744,osdmap=1779138857}) 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.871 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=220034446}) 2024-09-10T16:29:00.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=477423271}) 2024-09-10T16:29:00.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=220034446}) 2024-09-10T16:29:00.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.867+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3183037175}) 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=1702346495}) 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=38} crc {osdmap=534633199}) 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=35} crc {osdmap=1702346495}) 2024-09-10T16:29:00.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:00.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=36} crc {osdmap=465364963}) 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 10 monclient: get_monmap_and_config 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 10 monclient: monmap: 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:00.876906+0000 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:00.876906+0000 2024-09-10T16:29:00.877 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:00.878 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 1 Processor -- start 2024-09-10T16:29:00.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 1 -- start start 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.876+0000 7effc9389640 10 monclient: init 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient: _add_conns ranks=[2,4,0,1,3] 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 0x7effc40a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): picked mon.noname-b con 0x7effc409f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 0x7effc40a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): picked mon.noname-c con 0x7effc40a2380 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7effc413a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effc409fd10 con 0x7effc413a5c0 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7effc40a2cc0 con 0x7effc40a2380 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7effc409cd70 con 0x7effc409f930 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc9389640 10 monclient(hunting): authenticate will time out at 2119.364291s 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 0x7effc40a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc37fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7effc413a5c0 0x7effc413c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53624/0 (socket says 172.21.15.164:53624) 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc37fe640 1 -- 172.21.15.164:0/416252798 learned_addr learned my addr 172.21.15.164:0/416252798 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): get_auth_request con 0x7effc409f930 auth_method 0 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effbbfff640 1 --2- 172.21.15.164:0/416252798 >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 0x7effc40a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient(hunting): handle_auth_done global_id 244241 payload 274 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 -- 172.21.15.164:0/416252798 >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 msgr2=0x7effc40a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 --2- 172.21.15.164:0/416252798 >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 0x7effc40a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 -- 172.21.15.164:0/416252798 >> v1:172.21.15.164:6789/0 conn(0x7effc413a5c0 legacy=0x7effc413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 monclient: found mon.noname-b 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effc2ffd640 1 -- 172.21.15.164:0/416252798 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc409a120 con 0x7effc409f930 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effbbfff640 10 monclient: get_auth_request con 0x7effc40a2380 auth_method 0 2024-09-10T16:29:00.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.877+0000 7effbbfff640 1 --2- 172.21.15.164:0/416252798 >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 0x7effc40a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc2ffd640 1 -- 172.21.15.164:0/416252798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 msgr2=0x7effc40a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc2ffd640 1 --2- 172.21.15.164:0/416252798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 0x7effc40a1e40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7effb40097a0 tx=0x7effb4029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acba12e02574d31d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 1 -- 172.21.15.164:0/416252798 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4037070 con 0x7effc409f930 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: dump: 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.891 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 1 -- 172.21.15.164:0/416252798 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effb4032640 con 0x7effc409f930 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc9389640 5 monclient: authenticate success, global_id 244241 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc9389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:00.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc9389640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 1 -- 172.21.15.164:0/416252798 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4032a10 con 0x7effc409f930 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc9389640 4 set_mon_vals no callback set 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.880+0000 7effc27fc640 10 monclient: dump: 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 10 monclient: shutdown 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 -- 172.21.15.164:0/416252798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 msgr2=0x7effc40a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 --2- 172.21.15.164:0/416252798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 0x7effc40a1e40 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7effb40097a0 tx=0x7effb4029fd0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 -- 172.21.15.164:0/416252798 shutdown_connections 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 --2- 172.21.15.164:0/416252798 >> v2:172.21.15.165:3300/0 conn(0x7effc40a2380 0x7effc40a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 --2- 172.21.15.164:0/416252798 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc409f930 0x7effc40a1e40 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 -- 172.21.15.164:0/416252798 >> 172.21.15.164:0/416252798 conn(0x7effc40a3110 msgr2=0x7effc40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 -- 172.21.15.164:0/416252798 shutdown_connections 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 1 -- 172.21.15.164:0/416252798 wait complete. 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 10 monclient: build_initial_monmap 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.881+0000 7effc9389640 10 monclient: monmap: 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 Processor -- start 2024-09-10T16:29:00.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 -- start start 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient: init 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409f930 0x7effc409a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): picked mon.noname-b con 0x7effc409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 0x7effc409a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): picked mon.noname-c con 0x7effc40a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 0x7effc409af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): picked mon.noname-a con 0x7effc413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7effc409d930 con 0x7effc413a5c0 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7effc409ee00 con 0x7effc409f930 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7effc409c8b0 con 0x7effc40a2380 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc9389640 10 monclient(hunting): authenticate will time out at 2119.369682s 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 0x7effc409a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 0x7effc409af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 0x7effc409a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53568/0 (socket says 172.21.15.164:53568) 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 1 -- 172.21.15.164:0/3799644768 learned_addr learned my addr 172.21.15.164:0/3799644768 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): get_auth_request con 0x7effc40a2380 auth_method 0 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): get_auth_request con 0x7effc413a5c0 auth_method 0 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effbbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409f930 0x7effc409a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc2ffd640 10 monclient(hunting): get_auth_request con 0x7effc409f930 auth_method 0 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.882+0000 7effc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 10 monclient(hunting): handle_auth_done global_id 254148 payload 824 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 msgr2=0x7effc409a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 0x7effc409a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409f930 msgr2=0x7effc409a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409f930 0x7effc409a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 monclient: found mon.noname-a 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc37fe640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effc4074480 con 0x7effc413a5c0 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effbbfff640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7effc40a2380 0x7effc409a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:00.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.883+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409f930 0x7effc409a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc37fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 msgr2=0x7effc409af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc37fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 0x7effc409af20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effbc00cb20 tx=0x7effbc00cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc017070 con 0x7effc413a5c0 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: dump: 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.900 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effbc00ade0 con 0x7effc413a5c0 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 5 monclient: authenticate success, global_id 254148 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effbc012640 con 0x7effc413a5c0 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:00.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effbb7fe640 10 monclient: dump: 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 10 monclient: _renew_subs 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc1ffb640 4 set_mon_vals no callback set 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effc409d620 con 0x7effc413a5c0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 10 monclient: _renew_subs 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.885+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effc4099d00 con 0x7effc413a5c0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7effbc01a070 con 0x7effc413a5c0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc9389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc9389640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 0x7effc4071e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7effc409c3f0 con 0x7effc409b460 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effbb7fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effa403e210 0x7effa40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7effbc052ad0 con 0x7effc413a5c0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 0x7effc4071e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient: get_auth_request con 0x7effc409b460 auth_method 0 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effbbfff640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effa403e210 0x7effa40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effbbfff640 10 monclient: get_auth_request con 0x7effa403e210 auth_method 0 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.886+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.887+0000 7effc2ffd640 10 monclient(hunting): handle_auth_done global_id 234304 payload 824 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.887+0000 7effc2ffd640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7effc406fae0 con 0x7effc409b460 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbbfff640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effa403e210 0x7effa40406d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7effc4095b90 tx=0x7effac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effc2ffd640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 msgr2=0x7effc4071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 0x7effc4071e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effc409f5c0 tx=0x7effb40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4042070 con 0x7effc409b460 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 10 monclient: dump: 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.905 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7effb403e870 con 0x7effc409b460 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 msgr2=0x7effc4071e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 0x7effc4071e00 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effc409f5c0 tx=0x7effb40046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 msgr2=0x7effc4071e00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.889+0000 7effbb7fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc409b460 0x7effc4071e00 secure :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7effc409f5c0 tx=0x7effb40046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.895+0000 7effc9389640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.895+0000 7effc9389640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 0x7effc4072750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.895+0000 7effc9389640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.895+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7effc409b840 con 0x7effc4072340 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 0x7effc4072750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient: get_auth_request con 0x7effc4072340 auth_method 0 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:00.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 10 monclient(hunting): handle_auth_done global_id 234307 payload 824 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.896+0000 7effc2ffd640 1 -- 172.21.15.164:0/3799644768 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7effc4077260 con 0x7effc4072340 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effc2ffd640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 msgr2=0x7effc4072750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effc2ffd640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 0x7effc4072750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7effb40097a0 tx=0x7effb4008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effb4037040 con 0x7effc4072340 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effbb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effbb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.899+0000 7effbb7fe640 10 monclient: dump: 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:00.909 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7effb4003820 con 0x7effc4072340 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 msgr2=0x7effc4072750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 0x7effc4072750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7effb40097a0 tx=0x7effb4008030 comp rx=0 tx=0).stop 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 msgr2=0x7effc4072750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:00.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effbb7fe640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7effc4072340 0x7effc4072750 secure :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7effb40097a0 tx=0x7effb4008030 comp rx=0 tx=0).stop 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 226, 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 141183, 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.911 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.912 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.913 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:00.914 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:00.915 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.916 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.917 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.918 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.919 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:00.920 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:00.921 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:00.922 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:29:00.923 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:00.924 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effa403e210 msgr2=0x7effa40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7effa403e210 0x7effa40406d0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7effc4095b90 tx=0x7effac008040 comp rx=0 tx=0).stop 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 10 monclient: shutdown 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 msgr2=0x7effc409af20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc37fe640 1 -- 172.21.15.164:0/3799644768 reap_dead start 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 0x7effc409af20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effbc00cb20 tx=0x7effbc00cf20 comp rx=0 tx=0).stop 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 shutdown_connections 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 --2- 172.21.15.164:0/3799644768 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7effc413a5c0 0x7effc409af20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 >> 172.21.15.164:0/3799644768 conn(0x7effc40a3110 msgr2=0x7effc41388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 shutdown_connections 2024-09-10T16:29:00.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:00.900+0000 7effc9389640 1 -- 172.21.15.164:0/3799644768 wait complete. 2024-09-10T16:29:00.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:00.927 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:29:00.927 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:29:00.927 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:29:00.927 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:29:00.928 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:29:00.928 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:29:00.943 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:29:00.943 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:29:00.944 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.039+0000 7fec9749b640 10 monclient: get_monmap_and_config 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.039+0000 7fec9749b640 10 monclient: build_initial_monmap 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.039+0000 7fec9749b640 10 monclient: monmap: 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:01.040123+0000 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:01.040123+0000 2024-09-10T16:29:01.041 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.039+0000 7fec9749b640 1 Processor -- start 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 -- start start 2024-09-10T16:29:01.042 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient: init 2024-09-10T16:29:01.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:01.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-10T16:29:01.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec90134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-b con 0x7fec9006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec900a46a0 0x7fec900a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-a con 0x7fec900a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fec90137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec9009ce50 con 0x7fec90137ca0 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fec9006db60 con 0x7fec900a46a0 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec9006cf50 con 0x7fec9006cb70 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec9749b640 10 monclient(hunting): authenticate will time out at 2119.527611s 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fec900a46a0 0x7fec900a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:01.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95a38640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fec90137ca0 0x7fec90138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53640/0 (socket says 172.21.15.164:53640) 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec90134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95a38640 1 -- 172.21.15.164:0/3662286135 learned_addr learned my addr 172.21.15.164:0/3662286135 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): get_auth_request con 0x7fec900a46a0 auth_method 0 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): get_auth_request con 0x7fec9006cb70 auth_method 0 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:01.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec94a36640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.040+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 10 monclient(hunting): handle_auth_done global_id 254151 payload 274 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 1 -- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 msgr2=0x7fec90134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec90134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 1 -- 172.21.15.164:0/3662286135 >> v1:172.21.15.164:6789/0 conn(0x7fec90137ca0 legacy=0x7fec90138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 10 monclient: _finish_hunting 0 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 1 monclient: found mon.noname-a 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec94a36640 1 -- 172.21.15.164:0/3662286135 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9009cb70 con 0x7fec900a46a0 2024-09-10T16:29:01.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.041+0000 7fec95237640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec90134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec94a36640 1 -- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec900a46a0 msgr2=0x7fec900a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec94a36640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec900a46a0 0x7fec900a4a80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0097a0 tx=0x7fec8c029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e1a9a0cf0a7bb0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 1 -- 172.21.15.164:0/3662286135 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c037070 con 0x7fec900a46a0 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: dump: 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.050 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: _finish_auth 0 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 1 -- 172.21.15.164:0/3662286135 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8c032440 con 0x7fec900a46a0 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec9749b640 5 monclient: authenticate success, global_id 254151 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec9749b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec9749b640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:01.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec9749b640 4 set_mon_vals no callback set 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.043+0000 7fec83fff640 1 -- 172.21.15.164:0/3662286135 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8c032810 con 0x7fec900a46a0 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec83fff640 10 monclient: dump: 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:01.052 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 10 monclient: shutdown 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 -- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec900a46a0 msgr2=0x7fec900a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec900a46a0 0x7fec900a4a80 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fec8c0097a0 tx=0x7fec8c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 -- 172.21.15.164:0/3662286135 shutdown_connections 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec900a46a0 0x7fec900a4a80 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 --2- 172.21.15.164:0/3662286135 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec90134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 -- 172.21.15.164:0/3662286135 >> 172.21.15.164:0/3662286135 conn(0x7fec900aa900 msgr2=0x7fec900aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 -- 172.21.15.164:0/3662286135 shutdown_connections 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.044+0000 7fec9749b640 1 -- 172.21.15.164:0/3662286135 wait complete. 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 10 monclient: build_initial_monmap 2024-09-10T16:29:01.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 10 monclient: monmap: 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 1 Processor -- start 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 1 -- start start 2024-09-10T16:29:01.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 10 monclient: init 2024-09-10T16:29:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.045+0000 7fec9749b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-c con 0x7fec9006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 0x7fec9009e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-b con 0x7fec900a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 0x7fec901cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): picked mon.noname-a con 0x7fec90137ca0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fec901389c0 con 0x7fec90137ca0 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fec90138840 con 0x7fec900a46a0 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fec90140040 con 0x7fec9006cb70 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:01.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec9749b640 10 monclient(hunting): authenticate will time out at 2119.533095s 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 0x7fec901cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53590/0 (socket says 172.21.15.164:53590) 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 -- 172.21.15.164:0/2114534088 learned_addr learned my addr 172.21.15.164:0/2114534088 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): get_auth_request con 0x7fec9006cb70 auth_method 0 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): get_auth_request con 0x7fec90137ca0 auth_method 0 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:01.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec94a36640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 msgr2=0x7fec9009e030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec94a36640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 0x7fec9009e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient(hunting): handle_auth_done global_id 244250 payload 824 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 msgr2=0x7fec9009e030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 0x7fec9009e030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 msgr2=0x7fec901cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 0x7fec901cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient: _finish_hunting 0 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 monclient: found mon.noname-c 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95237640 1 -- 172.21.15.164:0/2114534088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec9006e720 con 0x7fec9006cb70 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.046+0000 7fec95a38640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 0x7fec901cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec95237640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 msgr2=0x7fec9009daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec95237640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e960 tx=0x7fec8400ee20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84017070 con 0x7fec9006cb70 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: dump: 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:01.064 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:01.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec9749b640 5 monclient: authenticate success, global_id 244250 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8400cc10 con 0x7fec9006cb70 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec84013470 con 0x7fec9006cb70 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec9749b640 10 monclient: _renew_subs 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec9749b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec9009fc60 con 0x7fec9006cb70 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec837fe640 4 set_mon_vals no callback set 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.048+0000 7fec81ffb640 10 monclient: dump: 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.066 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 10 monclient: _renew_subs 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fec9009ed00 con 0x7fec9006cb70 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fec8401a070 con 0x7fec9006cb70 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:01.067 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.049+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec90134f70 con 0x7fec9006cb70 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.050+0000 7fec81ffb640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 0x7fec74040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.050+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7fec84052c00 con 0x7fec9006cb70 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.050+0000 7fec94a36640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 0x7fec74040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.050+0000 7fec94a36640 10 monclient: get_auth_request con 0x7fec7403e1c0 auth_method 0 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.053+0000 7fec94a36640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 0x7fec74040680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fec900a6cc0 tx=0x7fec8c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.054+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fec840515f0 con 0x7fec9006cb70 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.054+0000 7fec81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:01.068 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.054+0000 7fec81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:01.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.295+0000 7fec9749b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:01.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.295+0000 7fec9749b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:01.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.295+0000 7fec9749b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:01.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.295+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fec90131790 con 0x7fec9006cb70 2024-09-10T16:29:01.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.296+0000 7fec81ffb640 1 -- 172.21.15.164:0/2114534088 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fec840093e0 con 0x7fec9006cb70 2024-09-10T16:29:01.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.296+0000 7fec81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:01.298 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.296+0000 7fec81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:01.298 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 msgr2=0x7fec74040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 0x7fec74040680 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7fec900a6cc0 tx=0x7fec8c02e000 comp rx=0 tx=0).stop 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 10 monclient: shutdown 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 msgr2=0x7fec9009daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fec8400e960 tx=0x7fec8400ee20 comp rx=0 tx=0).stop 2024-09-10T16:29:01.300 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 shutdown_connections 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fec7403e1c0 0x7fec74040680 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fec90137ca0 0x7fec901cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fec900a46a0 0x7fec9009e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 --2- 172.21.15.164:0/2114534088 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fec9006cb70 0x7fec9009daf0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 >> 172.21.15.164:0/2114534088 conn(0x7fec900aa900 msgr2=0x7fec900a9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 shutdown_connections 2024-09-10T16:29:01.301 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:01.298+0000 7fec9749b640 1 -- 172.21.15.164:0/2114534088 wait complete. 2024-09-10T16:29:04.312 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.419+0000 7f0a5a4b6640 10 monclient: get_monmap_and_config 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.419+0000 7f0a5a4b6640 10 monclient: build_initial_monmap 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.419+0000 7f0a5a4b6640 10 monclient: monmap: 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:04.420588+0000 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:04.420588+0000 2024-09-10T16:29:04.421 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:04.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.420+0000 7f0a5a4b6640 1 Processor -- start 2024-09-10T16:29:04.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.420+0000 7f0a5a4b6640 1 -- start start 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.420+0000 7f0a5a4b6640 10 monclient: init 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.420+0000 7f0a5a4b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.420+0000 7f0a5a4b6640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0a5409c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a5409d160 0x7f0a5409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5409d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 0x7f0a5413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a5413a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a5409a430 con 0x7f0a5409c180 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0a540a24e0 con 0x7f0a5409d160 2024-09-10T16:29:04.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0a540a2bf0 con 0x7f0a5413a5c0 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a5a4b6640 10 monclient(hunting): authenticate will time out at 2122.908083s 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a53fff640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f0a5409c180 legacy=0x7f0a5409ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a58a53640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 msgr2=0x7f0a5413c9b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a58a53640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 0x7f0a5413c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a5409d160 0x7f0a5409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0a5409d160 0x7f0a5409f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:41724/0 (socket says 172.21.15.164:41724) 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 -- 172.21.15.164:0/3481794192 learned_addr learned my addr 172.21.15.164:0/3481794192 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): get_auth_request con 0x7f0a5409d160 auth_method 0 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:04.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient(hunting): handle_auth_done global_id 254157 payload 274 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 -- 172.21.15.164:0/3481794192 >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 msgr2=0x7f0a5413c9b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 --2- 172.21.15.164:0/3481794192 >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 0x7f0a5413c9b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 -- 172.21.15.164:0/3481794192 >> v1:172.21.15.165:6789/0 conn(0x7f0a5409c180 legacy=0x7f0a5409ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient: _finish_hunting 0 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 monclient: found mon.noname-a 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.421+0000 7f0a537fe640 1 -- 172.21.15.164:0/3481794192 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a5409c560 con 0x7f0a5409d160 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a537fe640 1 -- 172.21.15.164:0/3481794192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5409d160 msgr2=0x7f0a5409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a537fe640 1 --2- 172.21.15.164:0/3481794192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5409d160 0x7f0a5409f150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebacb95c02cb6c79 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 1 -- 172.21.15.164:0/3481794192 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c037070 con 0x7f0a5409d160 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: dump: 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:04.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 5 monclient: authenticate success, global_id 254157 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:04.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 1 -- 172.21.15.164:0/3481794192 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c032640 con 0x7f0a5409d160 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 4 set_mon_vals no callback set 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 1 -- 172.21.15.164:0/3481794192 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c032a10 con 0x7f0a5409d160 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a52ffd640 10 monclient: dump: 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.435 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 10 monclient: shutdown 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/3481794192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5409d160 msgr2=0x7f0a5409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/3481794192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5409d160 0x7f0a5409f150 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c009870 tx=0x7f0a3c02a110 comp rx=0 tx=0).stop 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/3481794192 shutdown_connections 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.424+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/3481794192 >> v2:172.21.15.165:3300/0 conn(0x7f0a5413a5c0 0x7f0a5413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/3481794192 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5409d160 0x7f0a5409f150 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/3481794192 >> 172.21.15.164:0/3481794192 conn(0x7f0a540a3110 msgr2=0x7f0a540a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/3481794192 shutdown_connections 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/3481794192 wait complete. 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 10 monclient: build_initial_monmap 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 10 monclient: monmap: 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:04.437 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.425+0000 7f0a5a4b6640 1 Processor -- start 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 -- start start 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient: init 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 0x7f0a5406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-b con 0x7f0a5409c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-c con 0x7f0a5409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 0x7f0a5406f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): picked mon.noname-a con 0x7f0a5413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:04.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54074f90 con 0x7f0a5413a5c0 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0a54074e10 con 0x7f0a5409c180 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0a5409f690 con 0x7f0a5409d160 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a5a4b6640 10 monclient(hunting): authenticate will time out at 2122.913639s 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 0x7f0a5406f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a53fff640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 msgr2=0x7f0a5406e800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:43852/0 (socket says 172.21.15.164:43852) 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 1 -- 172.21.15.164:0/2139193111 learned_addr learned my addr 172.21.15.164:0/2139193111 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a53fff640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 0x7f0a5406e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): get_auth_request con 0x7f0a5409d160 auth_method 0 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): get_auth_request con 0x7f0a5413a5c0 auth_method 0 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a537fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:04.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.426+0000 7f0a58a53640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 10 monclient(hunting): handle_auth_done global_id 254160 payload 824 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 -- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 msgr2=0x7f0a5406ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 -- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 msgr2=0x7f0a5406e800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 0x7f0a5406e800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 10 monclient: _finish_hunting 0 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 monclient: found mon.noname-a 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a58a53640 1 -- 172.21.15.164:0/2139193111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a540a5f00 con 0x7f0a5413a5c0 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.427+0000 7f0a537fe640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:04.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a58a53640 1 -- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 msgr2=0x7f0a5406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a58a53640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 0x7f0a5406f280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0a5409a7c0 tx=0x7f0a4c009ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c015470 con 0x7f0a5413a5c0 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: dump: 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:04.442 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: _finish_auth 0 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a4c010070 con 0x7f0a5413a5c0 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 5 monclient: authenticate success, global_id 254160 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 10 monclient: _renew_subs 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a54096b80 con 0x7f0a5413a5c0 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a4c015890 con 0x7f0a5413a5c0 2024-09-10T16:29:04.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a50ff9640 10 monclient: dump: 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a527fc640 4 set_mon_vals no callback set 2024-09-10T16:29:04.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 10 monclient: _renew_subs 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.429+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a54070420 con 0x7f0a5413a5c0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a5a4b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a5a4b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a5a4b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0a4c01e3f0 con 0x7f0a5413a5c0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a5409ea40 con 0x7f0a5413a5c0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a50ff9640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 0x7f0a34040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.430+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7f0a4c053120 con 0x7f0a5413a5c0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.431+0000 7f0a53fff640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 0x7f0a34040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:04.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.431+0000 7f0a53fff640 10 monclient: get_auth_request con 0x7f0a3403e1c0 auth_method 0 2024-09-10T16:29:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.433+0000 7f0a53fff640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 0x7f0a34040680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0a44006fd0 tx=0x7f0a44006d80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.435+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0a540a5f00 con 0x7f0a5413a5c0 2024-09-10T16:29:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.435+0000 7f0a50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:04.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.435+0000 7f0a50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:04.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.673+0000 7f0a5a4b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.673+0000 7f0a5a4b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.673+0000 7f0a5a4b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.673+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0a540701c0 con 0x7f0a5413a5c0 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.674+0000 7f0a50ff9640 1 -- 172.21.15.164:0/2139193111 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f0a4c051b10 con 0x7f0a5413a5c0 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.674+0000 7f0a50ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.674+0000 7f0a50ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:04.676 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:04.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 msgr2=0x7f0a34040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 0x7f0a34040680 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f0a44006fd0 tx=0x7f0a44006d80 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 10 monclient: shutdown 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 msgr2=0x7f0a5406f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 0x7f0a5406f280 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0a5409a7c0 tx=0x7f0a4c009ed0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 shutdown_connections 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0a3403e1c0 0x7f0a34040680 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0a5413a5c0 0x7f0a5406f280 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0a5409d160 0x7f0a5406ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 --2- 172.21.15.164:0/2139193111 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0a5409c180 0x7f0a5406e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 >> 172.21.15.164:0/2139193111 conn(0x7f0a540a3110 msgr2=0x7f0a54139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:04.678 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 shutdown_connections 2024-09-10T16:29:04.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:04.676+0000 7f0a5a4b6640 1 -- 172.21.15.164:0/2139193111 wait complete. 2024-09-10T16:29:07.689 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:07.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.798+0000 7f7f473ad640 10 monclient: get_monmap_and_config 2024-09-10T16:29:07.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.798+0000 7f7f473ad640 10 monclient: build_initial_monmap 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.798+0000 7f7f473ad640 10 monclient: monmap: 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:07.799299+0000 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:07.799299+0000 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:07.800 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:07.801 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:07.801 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.798+0000 7f7f473ad640 1 Processor -- start 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 -- start start 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient: init 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f4009d930 0x7f7f4009dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4009d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7f4009ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:07.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 0x7f7f40134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4009e250 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f400a0870 con 0x7f7f4009ab50 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7f4009d3e0 con 0x7f7f4009d930 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f7f40074c70 con 0x7f7f4009e250 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f473ad640 10 monclient(hunting): authenticate will time out at 2126.286747s 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f4009d930 0x7f7f4009dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f4594a640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 msgr2=0x7f7f40134c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:07.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f4594a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 0x7f7f40134c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f44948640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7f4009ab50 0x7f7f40134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44620/0 (socket says 172.21.15.164:44620) 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7f4009d930 0x7f7f4009dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:41744/0 (socket says 172.21.15.164:41744) 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 1 -- 172.21.15.164:0/2909212914 learned_addr learned my addr 172.21.15.164:0/2909212914 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 10 monclient(hunting): get_auth_request con 0x7f7f4009d930 auth_method 0 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.799+0000 7f7f45149640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient(hunting): handle_auth_done global_id 254163 payload 274 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 1 -- 172.21.15.164:0/2909212914 >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 msgr2=0x7f7f40134c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 1 --2- 172.21.15.164:0/2909212914 >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 0x7f7f40134c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:07.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 1 -- 172.21.15.164:0/2909212914 >> v1:172.21.15.164:6789/0 conn(0x7f7f4009ab50 legacy=0x7f7f40134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient: _finish_hunting 0 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 1 monclient: found mon.noname-a 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.800+0000 7f7f45149640 1 -- 172.21.15.164:0/2909212914 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f4009ecb0 con 0x7f7f4009d930 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f45149640 1 -- 172.21.15.164:0/2909212914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009d930 msgr2=0x7f7f4009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f45149640 1 --2- 172.21.15.164:0/2909212914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009d930 0x7f7f4009dd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f4009a5a0 tx=0x7f7f3c02c570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9799b7c5fb7bd848 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 1 -- 172.21.15.164:0/2909212914 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c03a070 con 0x7f7f4009d930 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: dump: 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:07.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: _finish_auth 0 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 1 -- 172.21.15.164:0/2909212914 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f3c032770 con 0x7f7f4009d930 2024-09-10T16:29:07.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 5 monclient: authenticate success, global_id 254163 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 1 -- 172.21.15.164:0/2909212914 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f3c032b40 con 0x7f7f4009d930 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 4 set_mon_vals no callback set 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f37fff640 10 monclient: dump: 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:07.815 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 10 monclient: shutdown 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 -- 172.21.15.164:0/2909212914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009d930 msgr2=0x7f7f4009dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 --2- 172.21.15.164:0/2909212914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009d930 0x7f7f4009dd10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f7f4009a5a0 tx=0x7f7f3c02c570 comp rx=0 tx=0).stop 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 -- 172.21.15.164:0/2909212914 shutdown_connections 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 --2- 172.21.15.164:0/2909212914 >> v2:172.21.15.165:3300/0 conn(0x7f7f4009e250 0x7f7f40134c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 --2- 172.21.15.164:0/2909212914 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009d930 0x7f7f4009dd10 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 -- 172.21.15.164:0/2909212914 >> 172.21.15.164:0/2909212914 conn(0x7f7f400a3110 msgr2=0x7f7f400a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 -- 172.21.15.164:0/2909212914 shutdown_connections 2024-09-10T16:29:07.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.803+0000 7f7f473ad640 1 -- 172.21.15.164:0/2909212914 wait complete. 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.804+0000 7f7f473ad640 10 monclient: build_initial_monmap 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.804+0000 7f7f473ad640 10 monclient: monmap: 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:07.817 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.804+0000 7f7f473ad640 1 Processor -- start 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.804+0000 7f7f473ad640 1 -- start start 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.804+0000 7f7f473ad640 10 monclient: init 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 0x7f7f400739f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4009ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 0x7f7f4012f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-b con 0x7f7f4009d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4009e250 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:07.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4009ef30 con 0x7f7f4009e250 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7f4009e6e0 con 0x7f7f4009d930 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7f400a2940 con 0x7f7f4009ab50 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f473ad640 10 monclient(hunting): authenticate will time out at 2126.292361s 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f44948640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 msgr2=0x7f7f4012f900 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f44948640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 0x7f7f4012f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 0x7f7f400739f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:41752/0 (socket says 172.21.15.164:41752) 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 -- 172.21.15.164:0/1944037378 learned_addr learned my addr 172.21.15.164:0/1944037378 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): get_auth_request con 0x7f7f4009e250 auth_method 0 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:07.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): get_auth_request con 0x7f7f4009ab50 auth_method 0 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient(hunting): handle_auth_done global_id 254166 payload 824 2024-09-10T16:29:07.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 -- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 msgr2=0x7f7f400739f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 0x7f7f400739f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 -- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 msgr2=0x7f7f4012f900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 0x7f7f4012f900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient: _finish_hunting 0 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 monclient: found mon.noname-a 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f4594a640 1 -- 172.21.15.164:0/1944037378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f401324b0 con 0x7f7f4009e250 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.805+0000 7f7f45149640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 0x7f7f400739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f4594a640 1 -- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 msgr2=0x7f7f4012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f4594a640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800be50 tx=0x7f7f3800be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38016630 con 0x7f7f4009e250 2024-09-10T16:29:07.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: dump: 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:07.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 5 monclient: authenticate success, global_id 254166 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f38010030 con 0x7f7f4009e250 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 10 monclient: _renew_subs 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f38016a10 con 0x7f7f4009e250 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f4009b1c0 con 0x7f7f4009e250 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f35ffb640 10 monclient: dump: 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.823 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f377fe640 4 set_mon_vals no callback set 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 10 monclient: _renew_subs 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:07.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.808+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f4009d620 con 0x7f7f4009e250 2024-09-10T16:29:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7f380153e0 con 0x7f7f4009e250 2024-09-10T16:29:07.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f473ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f473ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f473ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f4009dd10 con 0x7f7f4009e250 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f35ffb640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 0x7f7f24040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.809+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(778..778 src has 677..778) ==== 4799+0+0 (secure 0 0 0) 0x7f7f38053850 con 0x7f7f4009e250 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.810+0000 7f7f45149640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 0x7f7f24040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.810+0000 7f7f45149640 10 monclient: get_auth_request con 0x7f7f2403e1c0 auth_method 0 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.812+0000 7f7f45149640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 0x7f7f24040680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c037000 tx=0x7f7f3c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.814+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7f3801b020 con 0x7f7f4009e250 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.814+0000 7f7f35ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:07.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:07.814+0000 7f7f35ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:08.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.059+0000 7f7f473ad640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:08.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.059+0000 7f7f473ad640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:08.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.059+0000 7f7f473ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:08.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.059+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7f40131360 con 0x7f7f4009e250 2024-09-10T16:29:08.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.060+0000 7f7f35ffb640 1 -- 172.21.15.164:0/1944037378 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f7f38052240 con 0x7f7f4009e250 2024-09-10T16:29:08.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.060+0000 7f7f35ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:08.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.060+0000 7f7f35ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:08.062 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":226,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:08.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 msgr2=0x7f7f24040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 0x7f7f24040680 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f7f3c037000 tx=0x7f7f3c030000 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 10 monclient: shutdown 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 msgr2=0x7f7f4012fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7f3800be50 tx=0x7f7f3800be80 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 shutdown_connections 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7f2403e1c0 0x7f7f24040680 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7f4009e250 0x7f7f4012fe40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7f4009d930 0x7f7f4012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 --2- 172.21.15.164:0/1944037378 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7f4009ab50 0x7f7f400739f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 >> 172.21.15.164:0/1944037378 conn(0x7f7f400a3110 msgr2=0x7f7f401bc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:08.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 shutdown_connections 2024-09-10T16:29:08.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:08.062+0000 7f7f473ad640 1 -- 172.21.15.164:0/1944037378 wait complete. 2024-09-10T16:29:11.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.183+0000 7f386b6ed640 10 monclient: get_monmap_and_config 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.183+0000 7f386b6ed640 10 monclient: build_initial_monmap 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.183+0000 7f386b6ed640 10 monclient: monmap: 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:11.185 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:11.184632+0000 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:11.184632+0000 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:11.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 1 Processor -- start 2024-09-10T16:29:11.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 1 -- start start 2024-09-10T16:29:11.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 10 monclient: init 2024-09-10T16:29:11.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f386409d930 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.184+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-b con 0x7f386409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f386409e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f38640a0870 con 0x7f386409e420 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3864074c70 con 0x7f386409d930 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f386409d3e0 con 0x7f386409ab50 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f386b6ed640 10 monclient(hunting): authenticate will time out at 2129.672071s 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3869489640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f386409d930 legacy=0x7f386409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:11.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3869c8a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f386409e420 0x7f386413bd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44632/0 (socket says 172.21.15.164:44632) 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3869c8a640 1 -- 172.21.15.164:0/3145152960 learned_addr learned my addr 172.21.15.164:0/3145152960 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f385bfff640 1 -- 172.21.15.164:0/3145152960 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.185+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:11.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.187+0000 7f3869c8a640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.164:6789/0 conn(0x7f386409e420 legacy=0x7f386413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:11.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.165:6789/0 conn(0x7f386409d930 legacy=0x7f386409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:11.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:11.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.386+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.165:6789/0 conn(0x7f386409d930 legacy=0x7f386409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:11.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:11.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:11.787+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:12.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.165:6789/0 conn(0x7f386409d930 legacy=0x7f386409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:12.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:12.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:12.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:12.588+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:29:14.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): tick 2024-09-10T16:29:14.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:29:14.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f38641397c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f38641397c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.165:6789/0 conn(0x7f386409d930 legacy=0x7f386409dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.164:6789/0 conn(0x7f386409e420 legacy=0x7f386413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient: _add_conns ranks=[3,4] 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): picked mon.noname-a con 0x7f3848003750 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): picked mon.noname-c con 0x7f3848006220 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f386409d3e0 con 0x7f3848003750 2024-09-10T16:29:14.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3848008da0 con 0x7f3848006220 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f385affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f38480086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 10 monclient(hunting): get_auth_request con 0x7f3848003750 auth_method 0 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:14.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.185+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f38480086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 10 monclient(hunting): get_auth_request con 0x7f3848003750 auth_method 0 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:29:14.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:14.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:14.390 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.387+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:14.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f38480086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 10 monclient(hunting): get_auth_request con 0x7f3848003750 auth_method 0 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:14.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:14.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:14.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:14.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:14.788+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:15.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f38480086e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 10 monclient(hunting): get_auth_request con 0x7f3848003750 auth_method 0 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:15.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:15.589+0000 7f3869489640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:29:16.372 INFO:tasks.rados.rados.0.smithi165.stdout:3482: done (1 left) 2024-09-10T16:29:16.373 INFO:tasks.rados.rados.0.smithi165.stdout:3484: done (0 left) 2024-09-10T16:29:16.373 INFO:tasks.rados.rados.0.smithi165.stdout:3485: read oid 44 snap 387 2024-09-10T16:29:16.373 INFO:tasks.rados.rados.0.smithi165.stdout:3485: expect (ObjNum 1114 snap 341 seq_num 1114) 2024-09-10T16:29:16.373 INFO:tasks.rados.rados.0.smithi165.stdout:3486: snap_create 2024-09-10T16:29:16.378 INFO:tasks.rados.rados.0.smithi165.stdout:3485: expect (ObjNum 1114 snap 341 seq_num 1114) 2024-09-10T16:29:16.383 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 609229 2024-09-10T16:29:16.442 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 289555 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): tick 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 monclient(hunting): continuing hunt 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f38480086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f38480086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 msgr2=0x7f3848005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.164:3300/0 conn(0x7f3848003750 0x7f3848005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f386409e420 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3869c8a640 1 -- 172.21.15.164:0/3145152960 reap_dead start 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f384800e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): picked mon.noname-b con 0x7f386409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f384800ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): picked mon.noname-c con 0x7f3848006220 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f386409ecb0 con 0x7f386409e420 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f386409d3e0 con 0x7f3848006220 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 1 -- 172.21.15.164:0/3145152960 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3848008da0 con 0x7f386409ab50 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f385affd640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3869c8a640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f384800ee90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f384800e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3869c8a640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f384800ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.185+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 10 monclient(hunting): handle_auth_done global_id 244259 payload 274 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 msgr2=0x7f384800ee90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f384800ee90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> v1:172.21.15.164:6789/0 conn(0x7f386409e420 legacy=0x7f384800b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 1 monclient: found mon.noname-b 2024-09-10T16:29:17.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.186+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3864074c70 con 0x7f386409ab50 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f3868c88640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f384800e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f3868c88640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f384800e950 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3854006fd0 tx=0x7f385402ac70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1dab877559e6a88d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 1 -- 172.21.15.164:0/3145152960 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854031070 con 0x7f386409ab50 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: dump: 2024-09-10T16:29:17.197 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 1 -- 172.21.15.164:0/3145152960 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385402baa0 con 0x7f386409ab50 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f386b6ed640 5 monclient: authenticate success, global_id 244259 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f386b6ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f386b6ed640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 1 -- 172.21.15.164:0/3145152960 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854035400 con 0x7f386409ab50 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f386b6ed640 4 set_mon_vals no callback set 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.189+0000 7f385bfff640 10 monclient: dump: 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 10 monclient: shutdown 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 -- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f384800e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f384800e950 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3854006fd0 tx=0x7f385402ac70 comp rx=0 tx=0).stop 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 -- 172.21.15.164:0/3145152960 shutdown_connections 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3145152960 >> v2:172.21.15.165:3300/0 conn(0x7f3848006220 0x7f384800ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3145152960 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f384800e950 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 -- 172.21.15.164:0/3145152960 >> 172.21.15.164:0/3145152960 conn(0x7f38640a3110 msgr2=0x7f38640a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 -- 172.21.15.164:0/3145152960 shutdown_connections 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.190+0000 7f386b6ed640 1 -- 172.21.15.164:0/3145152960 wait complete. 2024-09-10T16:29:17.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.191+0000 7f386b6ed640 10 monclient: build_initial_monmap 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.191+0000 7f386b6ed640 10 monclient: monmap: 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.191+0000 7f386b6ed640 1 Processor -- start 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.191+0000 7f386b6ed640 1 -- start start 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.191+0000 7f386b6ed640 10 monclient: init 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 0x7f3864073d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-b con 0x7f3848006220 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-c con 0x7f386409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 0x7f386412bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): picked mon.noname-a con 0x7f386409e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f38640a2940 con 0x7f386409e420 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f386409ef30 con 0x7f3848006220 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f38640a2ac0 con 0x7f386409ab50 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f386b6ed640 10 monclient(hunting): authenticate will time out at 2135.679115s 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 0x7f386412bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869489640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 msgr2=0x7f3864073d00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50504/0 (socket says 172.21.15.164:50504) 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869489640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 0x7f3864073d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 -- 172.21.15.164:0/3015550658 learned_addr learned my addr 172.21.15.164:0/3015550658 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): get_auth_request con 0x7f386409ab50 auth_method 0 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): get_auth_request con 0x7f386409e420 auth_method 0 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient(hunting): handle_auth_done global_id 244262 payload 824 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 -- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 msgr2=0x7f3864073d00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 0x7f3864073d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 -- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 msgr2=0x7f386412bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 0x7f386412bef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 monclient: found mon.noname-c 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3868c88640 1 -- 172.21.15.164:0/3015550658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f386412dcc0 con 0x7f386409ab50 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.192+0000 7f3869c8a640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 0x7f386412bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3868c88640 1 -- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f386412b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3868c88640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f385402acb0 tx=0x7f38540046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854043020 con 0x7f386409ab50 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:17.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: dump: 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f385402b8a0 con 0x7f386409ab50 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3854000e60 con 0x7f386409ab50 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 5 monclient: authenticate success, global_id 244262 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 10 monclient: _renew_subs 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38640a4550 con 0x7f386409ab50 2024-09-10T16:29:17.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f3859ffb640 10 monclient: dump: 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.211 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f385b7fe640 4 set_mon_vals no callback set 2024-09-10T16:29:17.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 10 monclient: _renew_subs 2024-09-10T16:29:17.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.195+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f38640746b0 con 0x7f386409ab50 2024-09-10T16:29:17.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.196+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3854046020 con 0x7f386409ab50 2024-09-10T16:29:17.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.196+0000 7f386b6ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.196+0000 7f386b6ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.196+0000 7f386b6ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.196+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f386406dac0 con 0x7f386409ab50 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.197+0000 7f3859ffb640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 0x7f3848043750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.197+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(780..780 src has 677..780) ==== 4799+0+0 (secure 0 0 0) 0x7f3854073d40 con 0x7f386409ab50 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.197+0000 7f3869489640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 0x7f3848043750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.197+0000 7f3869489640 10 monclient: get_auth_request con 0x7f3848041290 auth_method 0 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.200+0000 7f3869489640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 0x7f3848043750 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3860003930 tx=0x7f386000d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.201+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f385404d070 con 0x7f386409ab50 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.201+0000 7f3859ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.201+0000 7f3859ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:17.378 INFO:tasks.rados.rados.0.smithi165.stdout:3485: done (1 left) 2024-09-10T16:29:17.378 INFO:tasks.rados.rados.0.smithi165.stdout:3486: done (0 left) 2024-09-10T16:29:17.378 INFO:tasks.rados.rados.0.smithi165.stdout:3487: copy_from oid 26 from oid 30 current snap is 390 2024-09-10T16:29:17.378 INFO:tasks.rados.rados.0.smithi165.stdout:3488: delete oid 38 current snap is 390 2024-09-10T16:29:17.383 INFO:tasks.rados.rados.0.smithi165.stdout:3488: done (1 left) 2024-09-10T16:29:17.383 INFO:tasks.rados.rados.0.smithi165.stdout:3489: snap_remove snap 366 2024-09-10T16:29:17.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f386b6ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:17.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f386b6ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:17.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f386b6ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f386409dda0 con 0x7f386409ab50 2024-09-10T16:29:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f3859ffb640 1 -- 172.21.15.164:0/3015550658 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f385403fc70 con 0x7f386409ab50 2024-09-10T16:29:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f3859ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:17.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.454+0000 7f3859ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:17.456 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":230,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.456+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 msgr2=0x7f3848043750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 0x7f3848043750 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3860003930 tx=0x7f386000d040 comp rx=0 tx=0).stop 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 10 monclient: shutdown 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 msgr2=0x7f386412b9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f385402acb0 tx=0x7f38540046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 shutdown_connections 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3848041290 0x7f3848043750 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f386409e420 0x7f386412bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f386409ab50 0x7f386412b9b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 --2- 172.21.15.164:0/3015550658 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3848006220 0x7f3864073d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 >> 172.21.15.164:0/3015550658 conn(0x7f38640a3110 msgr2=0x7f38641bc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 shutdown_connections 2024-09-10T16:29:17.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.457+0000 7f386b6ed640 1 -- 172.21.15.164:0/3015550658 wait complete. 2024-09-10T16:29:17.469 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:29:17.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:29:17.469 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:29:17.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.568+0000 7fee145f3640 10 monclient: get_monmap_and_config 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.568+0000 7fee145f3640 10 monclient: build_initial_monmap 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.568+0000 7fee145f3640 10 monclient: monmap: 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:17.569339+0000 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:17.569339+0000 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:17.570 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 Processor -- start 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 -- start start 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient: init 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 0x7fee0c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c09c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fee0c09d160 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee0c13a5c0 0x7fee0c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c13a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee0c09a430 con 0x7fee0c09d160 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fee0c0a24e0 con 0x7fee0c13a5c0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fee0c0a2bf0 con 0x7fee0c09c180 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee145f3640 10 monclient(hunting): authenticate will time out at 2136.056762s 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee11b8e640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fee0c09d160 legacy=0x7fee0c09ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee1238f640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 msgr2=0x7fee0c09ea40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee0c13a5c0 0x7fee0c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee1238f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 0x7fee0c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fee0c13a5c0 0x7fee0c13c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:51924/0 (socket says 172.21.15.164:51924) 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 1 -- 172.21.15.164:0/943414079 learned_addr learned my addr 172.21.15.164:0/943414079 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 10 monclient(hunting): get_auth_request con 0x7fee0c13a5c0 auth_method 0 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.569+0000 7fee12b90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient(hunting): handle_auth_done global_id 254172 payload 274 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 1 -- 172.21.15.164:0/943414079 >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 msgr2=0x7fee0c09ea40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 1 --2- 172.21.15.164:0/943414079 >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 0x7fee0c09ea40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 1 -- 172.21.15.164:0/943414079 >> v1:172.21.15.165:6789/0 conn(0x7fee0c09d160 legacy=0x7fee0c09ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 1 monclient: found mon.noname-a 2024-09-10T16:29:17.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:17.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.570+0000 7fee12b90640 1 -- 172.21.15.164:0/943414079 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c09c560 con 0x7fee0c13a5c0 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee12b90640 1 -- 172.21.15.164:0/943414079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 msgr2=0x7fee0c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee12b90640 1 --2- 172.21.15.164:0/943414079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c13c9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fee04009e50 tx=0x7fee0402a5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd332c73a08da8eb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 1 -- 172.21.15.164:0/943414079 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04036070 con 0x7fee0c13a5c0 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:17.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: dump: 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 5 monclient: authenticate success, global_id 254172 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 1 -- 172.21.15.164:0/943414079 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04007800 con 0x7fee0c13a5c0 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 4 set_mon_vals no callback set 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 1 -- 172.21.15.164:0/943414079 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04007bd0 con 0x7fee0c13a5c0 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee1138d640 10 monclient: dump: 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.583 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 10 monclient: shutdown 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 -- 172.21.15.164:0/943414079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 msgr2=0x7fee0c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 --2- 172.21.15.164:0/943414079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c13c9b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fee04009e50 tx=0x7fee0402a5d0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 -- 172.21.15.164:0/943414079 shutdown_connections 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 --2- 172.21.15.164:0/943414079 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c13c9b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 --2- 172.21.15.164:0/943414079 >> v2:172.21.15.165:3300/0 conn(0x7fee0c09c180 0x7fee0c09ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 -- 172.21.15.164:0/943414079 >> 172.21.15.164:0/943414079 conn(0x7fee0c0a3110 msgr2=0x7fee0c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 -- 172.21.15.164:0/943414079 shutdown_connections 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.573+0000 7fee145f3640 1 -- 172.21.15.164:0/943414079 wait complete. 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.574+0000 7fee145f3640 10 monclient: build_initial_monmap 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.574+0000 7fee145f3640 10 monclient: monmap: 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:17.585 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.574+0000 7fee145f3640 1 Processor -- start 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.574+0000 7fee145f3640 1 -- start start 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.574+0000 7fee145f3640 10 monclient: init 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0c09c180 0x7fee0c092200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-b con 0x7fee0c09c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee0c09d160 0x7fee0c090a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-c con 0x7fee0c09d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:17.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c090fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): picked mon.noname-a con 0x7fee0c13a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c074f90 con 0x7fee0c13a5c0 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c074e10 con 0x7fee0c09c180 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fee0c09f690 con 0x7fee0c09d160 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee145f3640 10 monclient(hunting): authenticate will time out at 2136.062236s 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c090fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee1238f640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0c09c180 msgr2=0x7fee0c092200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee0c09d160 0x7fee0c090a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee1238f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0c09c180 0x7fee0c092200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c090fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:51936/0 (socket says 172.21.15.164:51936) 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 learned_addr learned my addr 172.21.15.164:0/1224220961 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): get_auth_request con 0x7fee0c13a5c0 auth_method 0 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): get_auth_request con 0x7fee0c09d160 auth_method 0 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee11b8e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient(hunting): handle_auth_done global_id 254175 payload 824 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee0c09d160 msgr2=0x7fee0c090a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fee0c09d160 0x7fee0c090a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0c09c180 msgr2=0x7fee0c092200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fee0c09c180 0x7fee0c092200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 monclient: found mon.noname-a 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.575+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0c0944c0 con 0x7fee0c13a5c0 2024-09-10T16:29:17.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 msgr2=0x7fee0c090fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee12b90640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c090fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee04034040 tx=0x7fee04004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04041020 con 0x7fee0c13a5c0 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 10 monclient: dump: 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.590 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.577+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04031940 con 0x7fee0c13a5c0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 5 monclient: authenticate success, global_id 254175 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 monclient: _renew_subs 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee0c093f00 con 0x7fee0c13a5c0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04036040 con 0x7fee0c13a5c0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 10 monclient: dump: 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee10b8c640 4 set_mon_vals no callback set 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 monclient: _renew_subs 2024-09-10T16:29:17.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee0c139660 con 0x7fee0c13a5c0 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 0x7fee0c063d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c09eab0 con 0x7fee0c091510 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fee04044020 con 0x7fee0c13a5c0 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 0x7fee0c063d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient: get_auth_request con 0x7fee0c091510 auth_method 0 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.578+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee1238f640 10 monclient(hunting): handle_auth_done global_id 254178 payload 824 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee1238f640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fee0c0918f0 con 0x7fee0c091510 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee02ffd640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fedf003e1c0 0x7fedf0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(781..781 src has 677..781) ==== 4755+0+0 (secure 0 0 0) 0x7fee04073ad0 con 0x7fee0c13a5c0 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee11b8e640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fedf003e1c0 0x7fedf0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.579+0000 7fee11b8e640 10 monclient: get_auth_request con 0x7fedf003e1c0 auth_method 0 2024-09-10T16:29:17.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee1238f640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 msgr2=0x7fee0c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee1238f640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 0x7fee0c063d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee08003930 tx=0x7fee0800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08005270 con 0x7fee0c091510 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 10 monclient: dump: 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.597 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fee08015070 con 0x7fee0c091510 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 msgr2=0x7fee0c063d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 0x7fee0c063d50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fee08003930 tx=0x7fee0800b8c0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 msgr2=0x7fee0c063d50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.581+0000 7fee02ffd640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c091510 0x7fee0c063d50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.582+0000 7fee11b8e640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fedf003e1c0 0x7fedf0040680 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b630 tx=0x7fedfc00b5c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee145f3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee145f3640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 0x7fee0c02d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee145f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fee0c073650 con 0x7fee0c064290 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee1238f640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 0x7fee0c02d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee1238f640 10 monclient: get_auth_request con 0x7fee0c064290 auth_method 0 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee1238f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee1238f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.587+0000 7fee1238f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.588+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.588+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.588+0000 7fee1238f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.588+0000 7fee1238f640 10 monclient(hunting): handle_auth_done global_id 254181 payload 824 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.588+0000 7fee1238f640 1 -- 172.21.15.164:0/1224220961 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fee0c077260 con 0x7fee0c064290 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee1238f640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 msgr2=0x7fee0c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee1238f640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 0x7fee0c02d440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee08003930 tx=0x7fee08005a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee08010610 con 0x7fee0c064290 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 10 monclient: dump: 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7fee08010610 con 0x7fee0c064290 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 msgr2=0x7fee0c02d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 0x7fee0c02d440 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fee08003930 tx=0x7fee08005a20 comp rx=0 tx=0).stop 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 msgr2=0x7fee0c02d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.590+0000 7fee02ffd640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c064290 0x7fee0c02d440 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:17.601 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 230, 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 46241, 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:17.602 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:17.603 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.604 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:17.605 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.606 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.607 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.608 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.609 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.610 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.611 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.612 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.613 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:17.614 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fedf003e1c0 msgr2=0x7fedf0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fedf003e1c0 0x7fedf0040680 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fedfc00b630 tx=0x7fedfc00b5c0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 10 monclient: shutdown 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 msgr2=0x7fee0c090fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 --2- 172.21.15.164:0/1224220961 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fee0c13a5c0 0x7fee0c090fd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fee04034040 tx=0x7fee04004a40 comp rx=0 tx=0).stop 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 reap_dead start 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee12b90640 1 -- 172.21.15.164:0/1224220961 reap_dead start 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 shutdown_connections 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 >> 172.21.15.164:0/1224220961 conn(0x7fee0c0a3110 msgr2=0x7fee0c139c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 shutdown_connections 2024-09-10T16:29:17.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.591+0000 7fee145f3640 1 -- 172.21.15.164:0/1224220961 wait complete. 2024-09-10T16:29:17.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:29:17.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:29:17.617 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:29:17.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.715+0000 7f6acb608640 10 monclient: get_monmap_and_config 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.715+0000 7f6acb608640 10 monclient: build_initial_monmap 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.715+0000 7f6acb608640 10 monclient: monmap: 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:17.716374+0000 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:17.716374+0000 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:17.717 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:17.718 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:17.718 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:17.718 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 Processor -- start 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 -- start start 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient: init 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 0x7f6ac409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac409d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6ac409ab50 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 0x7f6ac4134c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac409e250 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac40a0870 con 0x7f6ac409ab50 2024-09-10T16:29:17.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac409d3e0 con 0x7f6ac409d930 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac4074c70 con 0x7f6ac409e250 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6acb608640 10 monclient(hunting): authenticate will time out at 2136.203708s 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 0x7f6ac4134c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac93a4640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 msgr2=0x7f6ac409dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac8ba3640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6ac409ab50 0x7f6ac4134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:53960/0 (socket says 172.21.15.164:53960) 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/1309571762 learned_addr learned my addr 172.21.15.164:0/1309571762 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac93a4640 1 --2- 172.21.15.164:0/1309571762 >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 0x7f6ac409dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): get_auth_request con 0x7f6ac409e250 auth_method 0 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.716+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_done global_id 244268 payload 274 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 1 -- 172.21.15.164:0/1309571762 >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 msgr2=0x7f6ac409dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 1 --2- 172.21.15.164:0/1309571762 >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 0x7f6ac409dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 1 -- 172.21.15.164:0/1309571762 >> v1:172.21.15.164:6789/0 conn(0x7f6ac409ab50 legacy=0x7f6ac4134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 1 monclient: found mon.noname-b 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.717+0000 7f6ac9ba5640 1 -- 172.21.15.164:0/1309571762 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac409ecb0 con 0x7f6ac409e250 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.719+0000 7f6ac9ba5640 1 -- 172.21.15.164:0/1309571762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 msgr2=0x7f6ac4134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.719+0000 7f6ac9ba5640 1 --2- 172.21.15.164:0/1309571762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 0x7f6ac4134c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00aea0 tx=0x7f6abc02b600 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9748ad3c6baa6fb6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 1 -- 172.21.15.164:0/1309571762 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc037070 con 0x7f6ac409e250 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:17.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: dump: 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 1 -- 172.21.15.164:0/1309571762 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6abc008850 con 0x7f6ac409e250 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 5 monclient: authenticate success, global_id 244268 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 1 -- 172.21.15.164:0/1309571762 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6abc008c20 con 0x7f6ac409e250 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 4 set_mon_vals no callback set 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6abbfff640 10 monclient: dump: 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.732 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 10 monclient: shutdown 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 -- 172.21.15.164:0/1309571762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 msgr2=0x7f6ac4134c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 --2- 172.21.15.164:0/1309571762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 0x7f6ac4134c90 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6abc00aea0 tx=0x7f6abc02b600 comp rx=0 tx=0).stop 2024-09-10T16:29:17.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 -- 172.21.15.164:0/1309571762 shutdown_connections 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 --2- 172.21.15.164:0/1309571762 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409e250 0x7f6ac4134c90 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 --2- 172.21.15.164:0/1309571762 >> v2:172.21.15.165:3300/0 conn(0x7f6ac409d930 0x7f6ac409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 -- 172.21.15.164:0/1309571762 >> 172.21.15.164:0/1309571762 conn(0x7f6ac40a3110 msgr2=0x7f6ac40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 -- 172.21.15.164:0/1309571762 shutdown_connections 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.720+0000 7f6acb608640 1 -- 172.21.15.164:0/1309571762 wait complete. 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient: build_initial_monmap 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient: monmap: 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:17.734 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 1 Processor -- start 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 1 -- start start 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient: init 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac409ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ac409d930 0x7f6ac412ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.721+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac409d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ac409e250 0x7f6ac412bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac409e250 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:17.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac409ef30 con 0x7f6ac409e250 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac409e6e0 con 0x7f6ac409d930 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac40a2940 con 0x7f6ac409ab50 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6acb608640 10 monclient(hunting): authenticate will time out at 2136.209059s 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50532/0 (socket says 172.21.15.164:50532) 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 -- 172.21.15.164:0/2879041538 learned_addr learned my addr 172.21.15.164:0/2879041538 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): get_auth_request con 0x7f6ac409ab50 auth_method 0 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ac409e250 0x7f6ac412bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): get_auth_request con 0x7f6ac409e250 auth_method 0 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ac409d930 msgr2=0x7f6ac412ba30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac8ba3640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ac409d930 0x7f6ac412ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac9ba5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient(hunting): handle_auth_done global_id 244271 payload 824 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ac409d930 msgr2=0x7f6ac412ba30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6ac409d930 0x7f6ac412ba30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ac409e250 msgr2=0x7f6ac412bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6ac409e250 0x7f6ac412bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient: _finish_hunting 0 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 monclient: found mon.noname-c 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.722+0000 7f6ac93a4640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac409d620 con 0x7f6ac409ab50 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ac93a4640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 msgr2=0x7f6ac412b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ac93a4640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b550 tx=0x7f6ac000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0013020 con 0x7f6ac409ab50 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.739 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6acb608640 5 monclient: authenticate success, global_id 244271 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac0005070 con 0x7f6ac409ab50 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac0005480 con 0x7f6ac409ab50 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.740 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6acb608640 10 monclient: _renew_subs 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6abb7fe640 4 set_mon_vals no callback set 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6acb608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.724+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac412e640 con 0x7f6ac409ab50 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 10 monclient: _renew_subs 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac40746e0 con 0x7f6ac409ab50 2024-09-10T16:29:17.741 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6ac0016020 con 0x7f6ac409ab50 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 0x7f6ac41c5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac409ba30 con 0x7f6ac41c5b10 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.725+0000 7f6ac8ba3640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 0x7f6ac41c5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ab9ffb640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6aa803e1c0 0x7f6aa8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient: get_auth_request con 0x7f6ac41c5b10 auth_method 0 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(781..781 src has 677..781) ==== 4755+0+0 (secure 0 0 0) 0x7f6ac0052700 con 0x7f6ac409ab50 2024-09-10T16:29:17.742 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_done global_id 244274 payload 824 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ac409dd10 con 0x7f6ac41c5b10 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac9ba5640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6aa803e1c0 0x7f6aa8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.726+0000 7f6ac9ba5640 10 monclient: get_auth_request con 0x7f6aa803e1c0 auth_method 0 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 msgr2=0x7f6ac41c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ac8ba3640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 0x7f6ac41c5ef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab400c810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4017070 con 0x7f6ac41c5b10 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-10T16:29:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6ab40070d0 con 0x7f6ac41c5b10 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:17.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 msgr2=0x7f6ac41c5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 0x7f6ac41c5ef0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab400c810 comp rx=0 tx=0).stop 2024-09-10T16:29:17.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 msgr2=0x7f6ac41c5ef0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:17.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ab9ffb640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c5b10 0x7f6ac41c5ef0 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4007fd0 tx=0x7f6ab400c810 comp rx=0 tx=0).stop 2024-09-10T16:29:17.745 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.728+0000 7f6ac9ba5640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6aa803e1c0 0x7f6aa8040680 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406de20 tx=0x7f6abc00a000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.746 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.734+0000 7f6acb608640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.734+0000 7f6acb608640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 0x7f6ac406f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.734+0000 7f6acb608640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.734+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac406dac0 con 0x7f6ac41c6430 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.734+0000 7f6ac8ba3640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 0x7f6ac406f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient: get_auth_request con 0x7f6ac41c6430 auth_method 0 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 10 monclient(hunting): handle_auth_done global_id 244277 payload 824 2024-09-10T16:29:17.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.735+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/2879041538 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ac412e160 con 0x7f6ac41c6430 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ac8ba3640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 msgr2=0x7f6ac406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ac8ba3640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 0x7f6ac406f7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a5390 tx=0x7f6ab40049e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4017030 con 0x7f6ac41c6430 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 10 monclient: dump: 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:17.748 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f6ab4022440 con 0x7f6ac41c6430 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 msgr2=0x7f6ac406f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 0x7f6ac406f7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a5390 tx=0x7f6ab40049e0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 msgr2=0x7f6ac406f7e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.737+0000 7f6ab9ffb640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac41c6430 0x7f6ac406f7e0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6ac40a5390 tx=0x7f6ab40049e0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:17.749 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 230, 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 94645, 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.750 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.751 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:17.752 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:17.753 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:17.754 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.755 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:17.756 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:17.758 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:17.759 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:17.760 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:29:17.761 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:17.762 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6aa803e1c0 msgr2=0x7f6aa8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6aa803e1c0 0x7f6aa8040680 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f6ac406de20 tx=0x7f6abc00a000 comp rx=0 tx=0).stop 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 10 monclient: shutdown 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 msgr2=0x7f6ac412b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6ac9ba5640 1 -- 172.21.15.164:0/2879041538 reap_dead start 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f6ac000b550 tx=0x7f6ac000ba10 comp rx=0 tx=0).stop 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 shutdown_connections 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 --2- 172.21.15.164:0/2879041538 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6ac409ab50 0x7f6ac412b4f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:17.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 >> 172.21.15.164:0/2879041538 conn(0x7f6ac40a3110 msgr2=0x7f6ac41bc580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:17.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 shutdown_connections 2024-09-10T16:29:17.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:17.738+0000 7f6acb608640 1 -- 172.21.15.164:0/2879041538 wait complete. 2024-09-10T16:29:17.765 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:29:17.765 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:29:18.380 INFO:tasks.rados.rados.0.smithi165.stdout:3489: done (1 left) 2024-09-10T16:29:18.380 INFO:tasks.rados.rados.0.smithi165.stdout:3490: write oid 8 current snap is 390 2024-09-10T16:29:18.380 INFO:tasks.rados.rados.0.smithi165.stdout:3487: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:29:18.380 INFO:tasks.rados.rados.0.smithi165.stdout:3490: seq_num 1212 ranges {770084=501915,2023470=346614} 2024-09-10T16:29:18.386 INFO:tasks.rados.rados.0.smithi165.stdout:3490: writing smithi16534841-8 from 770084 to 1271999 tid 1 2024-09-10T16:29:18.393 INFO:tasks.rados.rados.0.smithi165.stdout:3490: writing smithi16534841-8 from 2023470 to 2370084 tid 2 2024-09-10T16:29:18.393 INFO:tasks.rados.rados.0.smithi165.stdout:3491: snap_create 2024-09-10T16:29:18.393 INFO:tasks.rados.rados.0.smithi165.stdout:3487: finishing copy_from to smithi16534841-26 2024-09-10T16:29:18.393 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:29:18.397 INFO:tasks.rados.rados.0.smithi165.stdout:3490: finishing write tid 1 to smithi16534841-8 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: finishing write tid 2 to smithi16534841-8 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: finishing write tid 3 to smithi16534841-8 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: finishing write tid 4 to smithi16534841-8 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: oid 8 updating version 0 to 1857 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: oid 8 updating version 1857 to 1863 2024-09-10T16:29:18.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: oid 8 version 1863 is already newer than 1862 2024-09-10T16:29:18.400 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1863 (ObjNum 1212 snap 390 seq_num 1212) dirty exists 2024-09-10T16:29:18.400 INFO:tasks.rados.rados.0.smithi165.stdout:3490: left oid 8 (ObjNum 1212 snap 390 seq_num 1212) 2024-09-10T16:29:19.399 INFO:tasks.rados.rados.0.smithi165.stdout:3487: done (2 left) 2024-09-10T16:29:19.399 INFO:tasks.rados.rados.0.smithi165.stdout:3490: done (1 left) 2024-09-10T16:29:19.399 INFO:tasks.rados.rados.0.smithi165.stdout:3491: done (0 left) 2024-09-10T16:29:19.399 INFO:tasks.rados.rados.0.smithi165.stdout:3492: read oid 50 snap -1 2024-09-10T16:29:19.400 INFO:tasks.rados.rados.0.smithi165.stdout:3492: expect (ObjNum 1165 snap 365 seq_num 1165) 2024-09-10T16:29:19.400 INFO:tasks.rados.rados.0.smithi165.stdout:3493: snap_remove snap 377 2024-09-10T16:29:20.403 INFO:tasks.rados.rados.0.smithi165.stdout:3493: done (1 left) 2024-09-10T16:29:20.403 INFO:tasks.rados.rados.0.smithi165.stdout:3494: delete oid 37 current snap is 391 2024-09-10T16:29:20.403 INFO:tasks.rados.rados.0.smithi165.stdout:3492: expect (ObjNum 1165 snap 365 seq_num 1165) 2024-09-10T16:29:20.478 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2322638 2024-09-10T16:29:20.500 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:20.553 INFO:tasks.rados.rados.0.smithi165.stdout:3492: done (1 left) 2024-09-10T16:29:20.553 INFO:tasks.rados.rados.0.smithi165.stdout:3494: done (0 left) 2024-09-10T16:29:20.553 INFO:tasks.rados.rados.0.smithi165.stdout:3495: delete oid 20 current snap is 391 2024-09-10T16:29:20.559 INFO:tasks.rados.rados.0.smithi165.stdout:3495: done (0 left) 2024-09-10T16:29:20.560 INFO:tasks.rados.rados.0.smithi165.stdout:3496: snap_remove snap 378 2024-09-10T16:29:21.371 INFO:tasks.rados.rados.0.smithi165.stdout:3496: done (0 left) 2024-09-10T16:29:21.372 INFO:tasks.rados.rados.0.smithi165.stdout:3497: delete oid 23 current snap is 391 2024-09-10T16:29:21.377 INFO:tasks.rados.rados.0.smithi165.stdout:3497: done (0 left) 2024-09-10T16:29:21.377 INFO:tasks.rados.rados.0.smithi165.stdout:3498: snap_create 2024-09-10T16:29:22.367 INFO:tasks.rados.rados.0.smithi165.stdout:3498: done (0 left) 2024-09-10T16:29:22.368 INFO:tasks.rados.rados.0.smithi165.stdout:3499: delete oid 32 current snap is 392 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3499: done (0 left) 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3500: read oid 22 snap 376 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3500: expect (ObjNum 1173 snap 370 seq_num 1173) 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3501: read oid 38 snap -1 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3501: expect deleted 2024-09-10T16:29:22.372 INFO:tasks.rados.rados.0.smithi165.stdout:3502: delete oid 36 current snap is 392 2024-09-10T16:29:22.376 INFO:tasks.rados.rados.0.smithi165.stdout:3501: done (2 left) 2024-09-10T16:29:22.376 INFO:tasks.rados.rados.0.smithi165.stdout:3502: done (1 left) 2024-09-10T16:29:22.376 INFO:tasks.rados.rados.0.smithi165.stdout:3503: write oid 38 current snap is 392 2024-09-10T16:29:22.376 INFO:tasks.rados.rados.0.smithi165.stdout:3503: seq_num 1218 ranges {470926=669762,1574362=700418} 2024-09-10T16:29:22.382 INFO:tasks.rados.rados.0.smithi165.stdout:3503: writing smithi16534841-38 from 470926 to 1140688 tid 1 2024-09-10T16:29:22.389 INFO:tasks.rados.rados.0.smithi165.stdout:3503: writing smithi16534841-38 from 1574362 to 2274780 tid 2 2024-09-10T16:29:22.389 INFO:tasks.rados.rados.0.smithi165.stdout:3504: snap_create 2024-09-10T16:29:22.396 INFO:tasks.rados.rados.0.smithi165.stdout:3500: expect (ObjNum 1173 snap 370 seq_num 1173) 2024-09-10T16:29:22.396 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:22.437 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: finishing write tid 1 to smithi16534841-38 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: finishing write tid 2 to smithi16534841-38 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: finishing write tid 3 to smithi16534841-38 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: finishing write tid 4 to smithi16534841-38 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: oid 38 updating version 0 to 1419 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: oid 38 version 1419 is already newer than 1417 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: oid 38 version 1419 is already newer than 1418 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1419 (ObjNum 1218 snap 392 seq_num 1218) dirty exists 2024-09-10T16:29:22.499 INFO:tasks.rados.rados.0.smithi165.stdout:3503: left oid 38 (ObjNum 1218 snap 392 seq_num 1218) 2024-09-10T16:29:23.369 INFO:tasks.rados.rados.0.smithi165.stdout:3500: done (2 left) 2024-09-10T16:29:23.369 INFO:tasks.rados.rados.0.smithi165.stdout:3503: done (1 left) 2024-09-10T16:29:23.369 INFO:tasks.rados.rados.0.smithi165.stdout:3504: done (0 left) 2024-09-10T16:29:23.369 INFO:tasks.rados.rados.0.smithi165.stdout:3505: delete oid 13 current snap is 393 2024-09-10T16:29:23.374 INFO:tasks.rados.rados.0.smithi165.stdout:3505: done (0 left) 2024-09-10T16:29:23.374 INFO:tasks.rados.rados.0.smithi165.stdout:3506: write (excl) oid 46 current snap is 393 2024-09-10T16:29:23.374 INFO:tasks.rados.rados.0.smithi165.stdout:3506: seq_num 1220 ranges {504540=420260,1574973=704087} 2024-09-10T16:29:23.378 INFO:tasks.rados.rados.0.smithi165.stdout:3506: writing smithi16534841-46 from 504540 to 924800 tid 1 2024-09-10T16:29:23.387 INFO:tasks.rados.rados.0.smithi165.stdout:3506: writing smithi16534841-46 from 1574973 to 2279060 tid 2 2024-09-10T16:29:23.387 INFO:tasks.rados.rados.0.smithi165.stdout:3507: write oid 24 current snap is 393 2024-09-10T16:29:23.387 INFO:tasks.rados.rados.0.smithi165.stdout:3506: finishing write tid 1 to smithi16534841-46 2024-09-10T16:29:23.387 INFO:tasks.rados.rados.0.smithi165.stdout:3507: seq_num 1221 ranges {521347=495509,1633455=564098} 2024-09-10T16:29:23.392 INFO:tasks.rados.rados.0.smithi165.stdout:3507: writing smithi16534841-24 from 521347 to 1016856 tid 1 2024-09-10T16:29:23.399 INFO:tasks.rados.rados.0.smithi165.stdout:3507: writing smithi16534841-24 from 1633455 to 2197553 tid 2 2024-09-10T16:29:23.399 INFO:tasks.rados.rados.0.smithi165.stdout:3508: write oid 21 current snap is 393 2024-09-10T16:29:23.399 INFO:tasks.rados.rados.0.smithi165.stdout:3506: finishing write tid 2 to smithi16534841-46 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: finishing write tid 3 to smithi16534841-46 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: finishing write tid 4 to smithi16534841-46 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: oid 46 updating version 0 to 1836 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: oid 46 version 1836 is already newer than 1835 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: oid 46 version 1836 is already newer than 1834 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1836 (ObjNum 1220 snap 393 seq_num 1220) dirty exists 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3506: left oid 46 (ObjNum 1220 snap 393 seq_num 1220) 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3507: finishing write tid 1 to smithi16534841-24 2024-09-10T16:29:23.400 INFO:tasks.rados.rados.0.smithi165.stdout:3508: seq_num 1222 ranges {538154=570758,1691937=707756} 2024-09-10T16:29:23.405 INFO:tasks.rados.rados.0.smithi165.stdout:3508: writing smithi16534841-21 from 538154 to 1108912 tid 1 2024-09-10T16:29:23.413 INFO:tasks.rados.rados.0.smithi165.stdout:3508: writing smithi16534841-21 from 1691937 to 2399693 tid 2 2024-09-10T16:29:23.413 INFO:tasks.rados.rados.0.smithi165.stdout:3506: done (2 left) 2024-09-10T16:29:23.413 INFO:tasks.rados.rados.0.smithi165.stdout:3509: snap_remove snap 360 2024-09-10T16:29:24.373 INFO:tasks.rados.rados.0.smithi165.stdout:3509: done (2 left) 2024-09-10T16:29:24.373 INFO:tasks.rados.rados.0.smithi165.stdout:3510: read oid 2 snap -1 2024-09-10T16:29:24.373 INFO:tasks.rados.rados.0.smithi165.stdout:3510: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:29:24.373 INFO:tasks.rados.rados.0.smithi165.stdout:3511: snap_remove snap 348 2024-09-10T16:29:25.383 INFO:tasks.rados.rados.0.smithi165.stdout:3511: done (3 left) 2024-09-10T16:29:25.383 INFO:tasks.rados.rados.0.smithi165.stdout:3512: snap_remove snap 379 2024-09-10T16:29:26.370 INFO:tasks.rados.rados.0.smithi165.stdout:3512: done (3 left) 2024-09-10T16:29:26.371 INFO:tasks.rados.rados.0.smithi165.stdout:3513: write oid 35 current snap is 393 2024-09-10T16:29:26.371 INFO:tasks.rados.rados.0.smithi165.stdout:3507: finishing write tid 2 to smithi16534841-24 2024-09-10T16:29:26.371 INFO:tasks.rados.rados.0.smithi165.stdout:3513: seq_num 1223 ranges {554961=646007,1750419=451414} 2024-09-10T16:29:26.376 INFO:tasks.rados.rados.0.smithi165.stdout:3513: writing smithi16534841-35 from 554961 to 1200968 tid 1 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3513: writing smithi16534841-35 from 1750419 to 2201833 tid 2 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3514: snap_create 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: finishing write tid 3 to smithi16534841-24 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: finishing write tid 4 to smithi16534841-24 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: oid 24 updating version 0 to 1874 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: oid 24 updating version 1874 to 1875 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: oid 24 updating version 1875 to 1876 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1876 (ObjNum 1221 snap 393 seq_num 1221) dirty exists 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3507: left oid 24 (ObjNum 1221 snap 393 seq_num 1221) 2024-09-10T16:29:26.384 INFO:tasks.rados.rados.0.smithi165.stdout:3508: finishing write tid 1 to smithi16534841-21 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: finishing write tid 2 to smithi16534841-21 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: finishing write tid 3 to smithi16534841-21 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: finishing write tid 4 to smithi16534841-21 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: oid 21 updating version 0 to 1878 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: oid 21 updating version 1878 to 1880 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: oid 21 version 1880 is already newer than 1879 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1880 (ObjNum 1222 snap 393 seq_num 1222) dirty exists 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3508: left oid 21 (ObjNum 1222 snap 393 seq_num 1222) 2024-09-10T16:29:26.385 INFO:tasks.rados.rados.0.smithi165.stdout:3510: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:29:26.470 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:26.515 INFO:tasks.rados.rados.0.smithi165.stdout:3513: finishing write tid 1 to smithi16534841-35 2024-09-10T16:29:26.515 INFO:tasks.rados.rados.0.smithi165.stdout:3513: finishing write tid 2 to smithi16534841-35 2024-09-10T16:29:26.515 INFO:tasks.rados.rados.0.smithi165.stdout:3513: finishing write tid 3 to smithi16534841-35 2024-09-10T16:29:26.515 INFO:tasks.rados.rados.0.smithi165.stdout:3513: finishing write tid 4 to smithi16534841-35 2024-09-10T16:29:26.515 INFO:tasks.rados.rados.0.smithi165.stdout:3513: oid 35 updating version 0 to 2010 2024-09-10T16:29:26.516 INFO:tasks.rados.rados.0.smithi165.stdout:3513: oid 35 version 2010 is already newer than 2009 2024-09-10T16:29:26.516 INFO:tasks.rados.rados.0.smithi165.stdout:3513: oid 35 version 2010 is already newer than 2008 2024-09-10T16:29:26.516 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2010 (ObjNum 1223 snap 393 seq_num 1223) dirty exists 2024-09-10T16:29:26.516 INFO:tasks.rados.rados.0.smithi165.stdout:3513: left oid 35 (ObjNum 1223 snap 393 seq_num 1223) 2024-09-10T16:29:27.388 INFO:tasks.rados.rados.0.smithi165.stdout:3507: done (4 left) 2024-09-10T16:29:27.388 INFO:tasks.rados.rados.0.smithi165.stdout:3508: done (3 left) 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3510: done (2 left) 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3513: done (1 left) 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3514: done (0 left) 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3515: read oid 45 snap -1 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3515: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:29:27.389 INFO:tasks.rados.rados.0.smithi165.stdout:3516: snap_remove snap 393 2024-09-10T16:29:28.395 INFO:tasks.rados.rados.0.smithi165.stdout:3516: done (1 left) 2024-09-10T16:29:28.395 INFO:tasks.rados.rados.0.smithi165.stdout:3517: read oid 36 snap -1 2024-09-10T16:29:28.395 INFO:tasks.rados.rados.0.smithi165.stdout:3517: expect deleted 2024-09-10T16:29:28.395 INFO:tasks.rados.rados.0.smithi165.stdout:3518: write oid 47 current snap is 394 2024-09-10T16:29:28.395 INFO:tasks.rados.rados.0.smithi165.stdout:3518: seq_num 1224 ranges {571768=437609,1808901=711425} 2024-09-10T16:29:28.400 INFO:tasks.rados.rados.0.smithi165.stdout:3518: writing smithi16534841-47 from 571768 to 1009377 tid 1 2024-09-10T16:29:28.408 INFO:tasks.rados.rados.0.smithi165.stdout:3518: writing smithi16534841-47 from 1808901 to 2520326 tid 2 2024-09-10T16:29:28.408 INFO:tasks.rados.rados.0.smithi165.stdout:3519: read oid 9 snap -1 2024-09-10T16:29:28.409 INFO:tasks.rados.rados.0.smithi165.stdout:3519: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:29:28.409 INFO:tasks.rados.rados.0.smithi165.stdout:3520: read oid 20 snap -1 2024-09-10T16:29:28.409 INFO:tasks.rados.rados.0.smithi165.stdout:3520: expect deleted 2024-09-10T16:29:28.409 INFO:tasks.rados.rados.0.smithi165.stdout:3521: snap_remove snap 357 2024-09-10T16:29:29.398 INFO:tasks.rados.rados.0.smithi165.stdout:3521: done (5 left) 2024-09-10T16:29:29.398 INFO:tasks.rados.rados.0.smithi165.stdout:3522: read oid 34 snap -1 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3522: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3523: read oid 23 snap -1 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3523: expect deleted 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3524: read oid 18 snap -1 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3524: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3525: read oid 8 snap -1 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3525: expect (ObjNum 1212 snap 390 seq_num 1212) 2024-09-10T16:29:29.399 INFO:tasks.rados.rados.0.smithi165.stdout:3526: snap_remove snap 349 2024-09-10T16:29:30.405 INFO:tasks.rados.rados.0.smithi165.stdout:3526: done (9 left) 2024-09-10T16:29:30.405 INFO:tasks.rados.rados.0.smithi165.stdout:3527: rollback oid 15 current snap is 394 2024-09-10T16:29:30.405 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 354 2024-09-10T16:29:30.405 INFO:tasks.rados.rados.0.smithi165.stdout:3515: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:29:30.519 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:30.582 INFO:tasks.rados.rados.0.smithi165.stdout:3518: finishing write tid 1 to smithi16534841-47 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: finishing write tid 2 to smithi16534841-47 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: finishing write tid 3 to smithi16534841-47 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: finishing write tid 4 to smithi16534841-47 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: oid 47 updating version 0 to 1893 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: oid 47 updating version 1893 to 1895 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: oid 47 version 1895 is already newer than 1894 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1895 (ObjNum 1224 snap 394 seq_num 1224) dirty exists 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3518: left oid 47 (ObjNum 1224 snap 394 seq_num 1224) 2024-09-10T16:29:30.583 INFO:tasks.rados.rados.0.smithi165.stdout:3519: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:29:30.656 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:30.801 INFO:tasks.rados.rados.0.smithi165.stdout:3525: expect (ObjNum 1212 snap 390 seq_num 1212) 2024-09-10T16:29:30.903 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:30.956 INFO:tasks.rados.rados.0.smithi165.stdout:3522: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:29:30.959 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1630972 2024-09-10T16:29:31.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1496678 2024-09-10T16:29:31.056 INFO:tasks.rados.rados.0.smithi165.stdout:3524: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:31.080 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:31.119 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:31.158 INFO:tasks.rados.rados.0.smithi165.stdout:3527: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1444 (ObjNum 1233 snap 0 seq_num 4062159104) dirty dne 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3515: done (9 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3517: done (8 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3518: done (7 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3519: done (6 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3520: done (5 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3522: done (4 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3523: done (3 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3524: done (2 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3525: done (1 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3527: done (0 left) 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3528: copy_from oid 40 from oid 38 current snap is 394 2024-09-10T16:29:31.159 INFO:tasks.rados.rados.0.smithi165.stdout:3529: snap_create 2024-09-10T16:29:31.167 INFO:tasks.rados.rados.0.smithi165.stdout:3528: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:29:31.170 INFO:tasks.rados.rados.0.smithi165.stdout:3528: finishing copy_from to smithi16534841-40 2024-09-10T16:29:31.170 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1419 (ObjNum 1218 snap 392 seq_num 1218) dirty exists 2024-09-10T16:29:31.372 INFO:tasks.rados.rados.0.smithi165.stdout:3528: done (1 left) 2024-09-10T16:29:31.372 INFO:tasks.rados.rados.0.smithi165.stdout:3529: done (0 left) 2024-09-10T16:29:31.372 INFO:tasks.rados.rados.0.smithi165.stdout:3530: write oid 21 current snap is 395 2024-09-10T16:29:31.372 INFO:tasks.rados.rados.0.smithi165.stdout:3530: seq_num 1225 ranges {588575=512858,1867383=455083} 2024-09-10T16:29:31.378 INFO:tasks.rados.rados.0.smithi165.stdout:3530: writing smithi16534841-21 from 588575 to 1101433 tid 1 2024-09-10T16:29:31.385 INFO:tasks.rados.rados.0.smithi165.stdout:3530: writing smithi16534841-21 from 1867383 to 2322466 tid 2 2024-09-10T16:29:31.385 INFO:tasks.rados.rados.0.smithi165.stdout:3531: read oid 13 snap 392 2024-09-10T16:29:31.386 INFO:tasks.rados.rados.0.smithi165.stdout:3531: expect deleted 2024-09-10T16:29:31.386 INFO:tasks.rados.rados.0.smithi165.stdout:3532: write oid 20 current snap is 395 2024-09-10T16:29:31.386 INFO:tasks.rados.rados.0.smithi165.stdout:3530: finishing write tid 1 to smithi16534841-21 2024-09-10T16:29:31.386 INFO:tasks.rados.rados.0.smithi165.stdout:3532: seq_num 1226 ranges {605382=588107,1925865=679517} 2024-09-10T16:29:31.392 INFO:tasks.rados.rados.0.smithi165.stdout:3532: writing smithi16534841-20 from 605382 to 1193489 tid 1 2024-09-10T16:29:31.400 INFO:tasks.rados.rados.0.smithi165.stdout:3532: writing smithi16534841-20 from 1925865 to 2605382 tid 2 2024-09-10T16:29:31.400 INFO:tasks.rados.rados.0.smithi165.stdout:3533: delete oid 7 current snap is 395 2024-09-10T16:29:31.400 INFO:tasks.rados.rados.0.smithi165.stdout:3530: finishing write tid 2 to smithi16534841-21 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: finishing write tid 3 to smithi16534841-21 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: finishing write tid 4 to smithi16534841-21 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: oid 21 updating version 0 to 1898 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: oid 21 updating version 1898 to 1899 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: oid 21 updating version 1899 to 1900 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1900 (ObjNum 1225 snap 395 seq_num 1225) dirty exists 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3530: left oid 21 (ObjNum 1225 snap 395 seq_num 1225) 2024-09-10T16:29:31.401 INFO:tasks.rados.rados.0.smithi165.stdout:3532: finishing write tid 1 to smithi16534841-20 2024-09-10T16:29:31.403 INFO:tasks.rados.rados.0.smithi165.stdout:3532: finishing write tid 2 to smithi16534841-20 2024-09-10T16:29:31.403 INFO:tasks.rados.rados.0.smithi165.stdout:3532: finishing write tid 3 to smithi16534841-20 2024-09-10T16:29:31.403 INFO:tasks.rados.rados.0.smithi165.stdout:3532: finishing write tid 4 to smithi16534841-20 2024-09-10T16:29:31.404 INFO:tasks.rados.rados.0.smithi165.stdout:3532: oid 20 updating version 0 to 1902 2024-09-10T16:29:31.404 INFO:tasks.rados.rados.0.smithi165.stdout:3532: oid 20 version 1902 is already newer than 1901 2024-09-10T16:29:31.404 INFO:tasks.rados.rados.0.smithi165.stdout:3532: oid 20 updating version 1902 to 1903 2024-09-10T16:29:31.404 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 1903 (ObjNum 1226 snap 395 seq_num 1226) dirty exists 2024-09-10T16:29:31.404 INFO:tasks.rados.rados.0.smithi165.stdout:3532: left oid 20 (ObjNum 1226 snap 395 seq_num 1226) 2024-09-10T16:29:31.408 INFO:tasks.rados.rados.0.smithi165.stdout:3530: done (3 left) 2024-09-10T16:29:31.408 INFO:tasks.rados.rados.0.smithi165.stdout:3531: done (2 left) 2024-09-10T16:29:31.408 INFO:tasks.rados.rados.0.smithi165.stdout:3532: done (1 left) 2024-09-10T16:29:31.408 INFO:tasks.rados.rados.0.smithi165.stdout:3533: done (0 left) 2024-09-10T16:29:31.408 INFO:tasks.rados.rados.0.smithi165.stdout:3534: snap_create 2024-09-10T16:29:32.416 INFO:tasks.rados.rados.0.smithi165.stdout:3534: done (0 left) 2024-09-10T16:29:32.416 INFO:tasks.rados.rados.0.smithi165.stdout:3535: read oid 1 snap 334 2024-09-10T16:29:32.416 INFO:tasks.rados.rados.0.smithi165.stdout:3535: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:29:32.416 INFO:tasks.rados.rados.0.smithi165.stdout:3536: snap_create 2024-09-10T16:29:32.426 INFO:tasks.rados.rados.0.smithi165.stdout:3535: expect (ObjNum 941 snap 285 seq_num 941) 2024-09-10T16:29:32.439 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1423424 2024-09-10T16:29:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:3535: done (1 left) 2024-09-10T16:29:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:3536: done (0 left) 2024-09-10T16:29:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:3537: write oid 49 current snap is 397 2024-09-10T16:29:33.420 INFO:tasks.rados.rados.0.smithi165.stdout:3537: seq_num 1228 ranges {638996=738605,1926476=712520} 2024-09-10T16:29:33.427 INFO:tasks.rados.rados.0.smithi165.stdout:3537: writing smithi16534841-49 from 638996 to 1377601 tid 1 2024-09-10T16:29:33.434 INFO:tasks.rados.rados.0.smithi165.stdout:3537: writing smithi16534841-49 from 1926476 to 2638996 tid 2 2024-09-10T16:29:33.434 INFO:tasks.rados.rados.0.smithi165.stdout:3538: read oid 14 snap -1 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:3538: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:3539: read oid 32 snap -1 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:3539: expect deleted 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:3540: rollback oid 43 current snap is 397 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 359 2024-09-10T16:29:33.435 INFO:tasks.rados.rados.0.smithi165.stdout:3541: snap_create 2024-09-10T16:29:33.439 INFO:tasks.rados.rados.0.smithi165.stdout:3537: finishing write tid 1 to smithi16534841-49 2024-09-10T16:29:33.439 INFO:tasks.rados.rados.0.smithi165.stdout:3537: finishing write tid 2 to smithi16534841-49 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:3537: finishing write tid 3 to smithi16534841-49 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:3537: finishing write tid 4 to smithi16534841-49 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:3537: oid 49 updating version 0 to 1839 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:3537: oid 49 updating version 1839 to 1840 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:3537: oid 49 version 1840 is already newer than 1838 2024-09-10T16:29:33.440 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1840 (ObjNum 1228 snap 397 seq_num 1228) dirty exists 2024-09-10T16:29:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:3537: left oid 49 (ObjNum 1228 snap 397 seq_num 1228) 2024-09-10T16:29:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:3540: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:29:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:3540: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:29:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1707 (ObjNum 129 snap 0 seq_num 1162386424) dirty dne 2024-09-10T16:29:33.441 INFO:tasks.rados.rados.0.smithi165.stdout:3538: expect (ObjNum 946 snap 285 seq_num 946) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3537: done (4 left) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3538: done (3 left) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3539: done (2 left) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3540: done (1 left) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3541: done (0 left) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3542: read oid 11 snap -1 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3542: expect (ObjNum 1187 snap 374 seq_num 1187) 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3543: copy_from oid 42 from oid 38 current snap is 398 2024-09-10T16:29:34.812 INFO:tasks.rados.rados.0.smithi165.stdout:3544: write (excl) oid 16 current snap is 398 2024-09-10T16:29:34.813 INFO:tasks.rados.rados.0.smithi165.stdout:3544: seq_num 1229 ranges {655803=413854,1584958=462421} 2024-09-10T16:29:34.820 INFO:tasks.rados.rados.0.smithi165.stdout:3544: writing smithi16534841-16 from 655803 to 1069657 tid 1 2024-09-10T16:29:34.824 INFO:tasks.rados.rados.0.smithi165.stdout:3544: writing smithi16534841-16 from 1584958 to 2047379 tid 2 2024-09-10T16:29:34.824 INFO:tasks.rados.rados.0.smithi165.stdout:3545: rollback oid 39 current snap is 398 2024-09-10T16:29:34.824 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 359 2024-09-10T16:29:34.824 INFO:tasks.rados.rados.0.smithi165.stdout:3542: expect (ObjNum 1187 snap 374 seq_num 1187) 2024-09-10T16:29:34.828 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1749418 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3544: finishing write tid 1 to smithi16534841-16 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3545: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3544: finishing write tid 2 to smithi16534841-16 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3543: finishing copy_from racing read to smithi16534841-42 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3545: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3545: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2017 (ObjNum 1127 snap 347 seq_num 1127) dirty exists 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3544: finishing write tid 3 to smithi16534841-16 2024-09-10T16:29:34.932 INFO:tasks.rados.rados.0.smithi165.stdout:3542: done (3 left) 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3545: done (2 left) 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3546: write (excl) oid 12 current snap is 398 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3543: finishing copy_from to smithi16534841-42 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 42 v 1419 (ObjNum 1218 snap 392 seq_num 1218) dirty exists 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3544: finishing write tid 4 to smithi16534841-16 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3544: oid 16 updating version 0 to 2242 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3544: oid 16 updating version 2242 to 2244 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3544: oid 16 version 2244 is already newer than 2243 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2244 (ObjNum 1229 snap 398 seq_num 1229) dirty exists 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3544: left oid 16 (ObjNum 1229 snap 398 seq_num 1229) 2024-09-10T16:29:34.933 INFO:tasks.rados.rados.0.smithi165.stdout:3546: seq_num 1230 ranges {672610=489103,1643440=722432} 2024-09-10T16:29:34.937 INFO:tasks.rados.rados.0.smithi165.stdout:3546: writing smithi16534841-12 from 672610 to 1161713 tid 1 2024-09-10T16:29:34.945 INFO:tasks.rados.rados.0.smithi165.stdout:3546: writing smithi16534841-12 from 1643440 to 2365872 tid 2 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3543: done (2 left) 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3544: done (1 left) 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3547: read oid 34 snap -1 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3547: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3548: copy_from oid 27 from oid 3 current snap is 398 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3546: finishing write tid 1 to smithi16534841-12 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3549: read oid 31 snap -1 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3549: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3550: read oid 28 snap -1 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3550: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3551: copy_from oid 21 from oid 4 current snap is 398 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3552: read oid 16 snap -1 2024-09-10T16:29:34.946 INFO:tasks.rados.rados.0.smithi165.stdout:3552: expect (ObjNum 1229 snap 398 seq_num 1229) 2024-09-10T16:29:34.947 INFO:tasks.rados.rados.0.smithi165.stdout:3553: delete oid 14 current snap is 398 2024-09-10T16:29:34.949 INFO:tasks.rados.rados.0.smithi165.stdout:3546: finishing write tid 2 to smithi16534841-12 2024-09-10T16:29:34.949 INFO:tasks.rados.rados.0.smithi165.stdout:3546: finishing write tid 3 to smithi16534841-12 2024-09-10T16:29:34.950 INFO:tasks.rados.rados.0.smithi165.stdout:3546: finishing write tid 4 to smithi16534841-12 2024-09-10T16:29:34.950 INFO:tasks.rados.rados.0.smithi165.stdout:3546: oid 12 updating version 0 to 1655 2024-09-10T16:29:34.950 INFO:tasks.rados.rados.0.smithi165.stdout:3546: oid 12 version 1655 is already newer than 1653 2024-09-10T16:29:34.950 INFO:tasks.rados.rados.0.smithi165.stdout:3546: oid 12 version 1655 is already newer than 1654 2024-09-10T16:29:34.950 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1655 (ObjNum 1230 snap 398 seq_num 1230) dirty exists 2024-09-10T16:29:34.951 INFO:tasks.rados.rados.0.smithi165.stdout:3546: left oid 12 (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:29:34.951 INFO:tasks.rados.rados.0.smithi165.stdout:3546: done (7 left) 2024-09-10T16:29:34.951 INFO:tasks.rados.rados.0.smithi165.stdout:3553: done (6 left) 2024-09-10T16:29:34.951 INFO:tasks.rados.rados.0.smithi165.stdout:3554: snap_remove snap 385 2024-09-10T16:29:35.826 INFO:tasks.rados.rados.0.smithi165.stdout:3554: done (6 left) 2024-09-10T16:29:35.826 INFO:tasks.rados.rados.0.smithi165.stdout:3555: read oid 12 snap -1 2024-09-10T16:29:35.826 INFO:tasks.rados.rados.0.smithi165.stdout:3555: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:3556: delete oid 8 current snap is 398 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:3548: finishing copy_from racing read to smithi16534841-27 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:3548: finishing copy_from to smithi16534841-27 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:3550: expect (ObjNum 1099 snap 334 seq_num 1099) 2024-09-10T16:29:35.827 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 234896 2024-09-10T16:29:35.872 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:35.959 INFO:tasks.rados.rados.0.smithi165.stdout:3551: finishing copy_from racing read to smithi16534841-21 2024-09-10T16:29:35.960 INFO:tasks.rados.rados.0.smithi165.stdout:3551: finishing copy_from to smithi16534841-21 2024-09-10T16:29:35.960 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 1451 (ObjNum 1161 snap 365 seq_num 1161) dirty exists 2024-09-10T16:29:35.960 INFO:tasks.rados.rados.0.smithi165.stdout:3547: expect (ObjNum 1161 snap 365 seq_num 1161) 2024-09-10T16:29:35.963 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1612952 2024-09-10T16:29:36.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:36.069 INFO:tasks.rados.rados.0.smithi165.stdout:3549: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:29:36.069 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 80293 2024-09-10T16:29:36.115 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1338411 2024-09-10T16:29:36.139 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:36.179 INFO:tasks.rados.rados.0.smithi165.stdout:3552: expect (ObjNum 1229 snap 398 seq_num 1229) 2024-09-10T16:29:36.285 INFO:tasks.rados.rados.0.smithi165.stdout:3555: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:29:36.369 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:36.503 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2213319 2024-09-10T16:29:36.535 INFO:tasks.rados.rados.0.smithi165.stdout:3547: done (7 left) 2024-09-10T16:29:36.535 INFO:tasks.rados.rados.0.smithi165.stdout:3548: done (6 left) 2024-09-10T16:29:36.535 INFO:tasks.rados.rados.0.smithi165.stdout:3549: done (5 left) 2024-09-10T16:29:36.535 INFO:tasks.rados.rados.0.smithi165.stdout:3550: done (4 left) 2024-09-10T16:29:36.536 INFO:tasks.rados.rados.0.smithi165.stdout:3551: done (3 left) 2024-09-10T16:29:36.536 INFO:tasks.rados.rados.0.smithi165.stdout:3552: done (2 left) 2024-09-10T16:29:36.536 INFO:tasks.rados.rados.0.smithi165.stdout:3555: done (1 left) 2024-09-10T16:29:36.536 INFO:tasks.rados.rados.0.smithi165.stdout:3556: done (0 left) 2024-09-10T16:29:36.536 INFO:tasks.rados.rados.0.smithi165.stdout:3557: delete oid 36 current snap is 398 2024-09-10T16:29:36.537 INFO:tasks.rados.rados.0.smithi165.stdout:3557: done (0 left) 2024-09-10T16:29:36.537 INFO:tasks.rados.rados.0.smithi165.stdout:3558: rollback oid 13 current snap is 398 2024-09-10T16:29:36.537 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 371 2024-09-10T16:29:36.537 INFO:tasks.rados.rados.0.smithi165.stdout:3559: read oid 36 snap -1 2024-09-10T16:29:36.538 INFO:tasks.rados.rados.0.smithi165.stdout:3559: expect deleted 2024-09-10T16:29:36.538 INFO:tasks.rados.rados.0.smithi165.stdout:3560: snap_create 2024-09-10T16:29:36.540 INFO:tasks.rados.rados.0.smithi165.stdout:3558: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:29:36.541 INFO:tasks.rados.rados.0.smithi165.stdout:3558: finishing rollback tid 2 to smithi16534841-13 2024-09-10T16:29:36.541 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 2012 (ObjNum 1163 snap 365 seq_num 1163) dirty exists 2024-09-10T16:29:37.377 INFO:tasks.rados.rados.0.smithi165.stdout:3558: done (2 left) 2024-09-10T16:29:37.377 INFO:tasks.rados.rados.0.smithi165.stdout:3559: done (1 left) 2024-09-10T16:29:37.377 INFO:tasks.rados.rados.0.smithi165.stdout:3560: done (0 left) 2024-09-10T16:29:37.377 INFO:tasks.rados.rados.0.smithi165.stdout:3561: write oid 33 current snap is 399 2024-09-10T16:29:37.377 INFO:tasks.rados.rados.0.smithi165.stdout:3561: seq_num 1234 ranges {739838=506452,1877368=613417} 2024-09-10T16:29:37.383 INFO:tasks.rados.rados.0.smithi165.stdout:3561: writing smithi16534841-33 from 739838 to 1246290 tid 1 2024-09-10T16:29:37.391 INFO:tasks.rados.rados.0.smithi165.stdout:3561: writing smithi16534841-33 from 1877368 to 2490785 tid 2 2024-09-10T16:29:37.391 INFO:tasks.rados.rados.0.smithi165.stdout:3562: snap_create 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: finishing write tid 1 to smithi16534841-33 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: finishing write tid 2 to smithi16534841-33 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: finishing write tid 3 to smithi16534841-33 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: finishing write tid 4 to smithi16534841-33 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: oid 33 updating version 0 to 1913 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: oid 33 updating version 1913 to 1914 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: oid 33 updating version 1914 to 1915 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 1915 (ObjNum 1234 snap 399 seq_num 1234) dirty exists 2024-09-10T16:29:37.396 INFO:tasks.rados.rados.0.smithi165.stdout:3561: left oid 33 (ObjNum 1234 snap 399 seq_num 1234) 2024-09-10T16:29:37.765 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:29:37.766 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:29:37.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:29:37.769 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:29:37.769 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:29:37.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:37.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.876+0000 7fc9a60ef640 10 monclient: get_monmap_and_config 2024-09-10T16:29:37.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.876+0000 7fc9a60ef640 10 monclient: build_initial_monmap 2024-09-10T16:29:37.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.876+0000 7fc9a60ef640 10 monclient: monmap: 2024-09-10T16:29:37.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:37.878010+0000 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:37.878010+0000 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:37.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.876+0000 7fc9a60ef640 1 Processor -- start 2024-09-10T16:29:37.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 1 -- start start 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient: init 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc9a006cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc9a0134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc9a013e740 0x7fc9a0140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a013e740 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a009ce50 con 0x7fc9a006cb70 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a0135380 con 0x7fc9a0134f70 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a006db60 con 0x7fc9a013e740 2024-09-10T16:29:37.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc9a60ef640 10 monclient(hunting): authenticate will time out at 2156.365596s 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc9a006cb70 0x7fc9a00a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54484/0 (socket says 172.21.15.164:54484) 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99f7fe640 1 -- 172.21.15.164:0/3582912119 learned_addr learned my addr 172.21.15.164:0/3582912119 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 1 --2- 172.21.15.164:0/3582912119 >> v2:172.21.15.164:3300/0 conn(0x7fc9a013e740 0x7fc9a0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): get_auth_request con 0x7fc9a013e740 auth_method 0 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient(hunting): handle_auth_done global_id 254187 payload 274 2024-09-10T16:29:37.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 1 -- 172.21.15.164:0/3582912119 >> v1:172.21.15.165:6789/0 conn(0x7fc9a0134f70 legacy=0x7fc9a013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 1 -- 172.21.15.164:0/3582912119 >> v1:172.21.15.164:6789/0 conn(0x7fc9a006cb70 legacy=0x7fc9a00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 1 monclient: found mon.noname-a 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:37.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.877+0000 7fc99ffff640 1 -- 172.21.15.164:0/3582912119 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a009cb70 con 0x7fc9a013e740 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99ffff640 1 -- 172.21.15.164:0/3582912119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 msgr2=0x7fc9a0140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99ffff640 1 --2- 172.21.15.164:0/3582912119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a0140b30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc998009dc0 tx=0x7fc99802a4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fc0649bc11aa3f8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 1 -- 172.21.15.164:0/3582912119 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998007a30 con 0x7fc9a013e740 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: dump: 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:37.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: _finish_auth 0 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 1 -- 172.21.15.164:0/3582912119 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc998007bd0 con 0x7fc9a013e740 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc9a60ef640 5 monclient: authenticate success, global_id 254187 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc9a60ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:37.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 1 -- 172.21.15.164:0/3582912119 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc998031400 con 0x7fc9a013e740 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc9a60ef640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc9a60ef640 4 set_mon_vals no callback set 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.880+0000 7fc99effd640 10 monclient: dump: 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:37.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 10 monclient: shutdown 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 -- 172.21.15.164:0/3582912119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 msgr2=0x7fc9a0140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/3582912119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a0140b30 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc998009dc0 tx=0x7fc99802a4d0 comp rx=0 tx=0).stop 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 -- 172.21.15.164:0/3582912119 shutdown_connections 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/3582912119 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a0140b30 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 -- 172.21.15.164:0/3582912119 >> 172.21.15.164:0/3582912119 conn(0x7fc9a00aa900 msgr2=0x7fc9a00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 -- 172.21.15.164:0/3582912119 shutdown_connections 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.881+0000 7fc9a60ef640 1 -- 172.21.15.164:0/3582912119 wait complete. 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.882+0000 7fc9a60ef640 10 monclient: build_initial_monmap 2024-09-10T16:29:37.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.882+0000 7fc9a60ef640 10 monclient: monmap: 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.882+0000 7fc9a60ef640 1 Processor -- start 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.882+0000 7fc9a60ef640 1 -- start start 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.882+0000 7fc9a60ef640 10 monclient: init 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:37.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-c con 0x7fc9a006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-b con 0x7fc9a0134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a01300e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): picked mon.noname-a con 0x7fc9a013e740 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a00a4db0 con 0x7fc9a013e740 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a006cf50 con 0x7fc9a0134f70 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a0141070 con 0x7fc9a006cb70 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:37.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc9a60ef640 10 monclient(hunting): authenticate will time out at 2156.371256s 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a01300e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36660/0 (socket says 172.21.15.164:36660) 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 -- 172.21.15.164:0/4260176083 learned_addr learned my addr 172.21.15.164:0/4260176083 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): get_auth_request con 0x7fc9a006cb70 auth_method 0 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:37.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): get_auth_request con 0x7fc9a013e740 auth_method 0 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:37.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 10 monclient(hunting): get_auth_request con 0x7fc9a0134f70 auth_method 0 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient(hunting): handle_auth_done global_id 244280 payload 824 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 msgr2=0x7fc9a012d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 msgr2=0x7fc9a012d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 msgr2=0x7fc9a012d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc996ffd640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:37.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 msgr2=0x7fc9a01300e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a01300e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 monclient: found mon.noname-c 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99f7fe640 1 -- 172.21.15.164:0/4260176083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a0131a60 con 0x7fc9a006cb70 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.883+0000 7fc99ffff640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a01300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.885+0000 7fc99f7fe640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 msgr2=0x7fc9a012d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.885+0000 7fc99f7fe640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc99000b430 tx=0x7fc99000b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990015020 con 0x7fc9a006cb70 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:37.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: dump: 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: _finish_auth 0 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:29:37.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc990011a60 con 0x7fc9a006cb70 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 5 monclient: authenticate success, global_id 244280 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 10 monclient: _renew_subs 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a013a060 con 0x7fc9a006cb70 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc990011e50 con 0x7fc9a006cb70 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99cff9640 10 monclient: dump: 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.906 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc99e7fc640 4 set_mon_vals no callback set 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 10 monclient: _renew_subs 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.886+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a012dd00 con 0x7fc9a006cb70 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc9a60ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc9a60ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc9a60ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:37.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc990018020 con 0x7fc9a006cb70 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a00a46a0 con 0x7fc9a006cb70 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc99cff9640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 0x7fc9800408a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(801..801 src has 677..801) ==== 4739+0+0 (secure 0 0 0) 0x7fc990052c80 con 0x7fc9a006cb70 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc996ffd640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 0x7fc9800408a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.887+0000 7fc996ffd640 10 monclient: get_auth_request con 0x7fc98003e3e0 auth_method 0 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.890+0000 7fc996ffd640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 0x7fc9800408a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0139030 tx=0x7fc988009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.891+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc99001f020 con 0x7fc9a006cb70 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.891+0000 7fc99cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:37.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:37.891+0000 7fc99cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.140+0000 7fc9a60ef640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.140+0000 7fc9a60ef640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.140+0000 7fc9a60ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.140+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc9a013a340 con 0x7fc9a006cb70 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.141+0000 7fc99cff9640 1 -- 172.21.15.164:0/4260176083 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fc9a013a060 con 0x7fc9a006cb70 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.141+0000 7fc99cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:38.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.141+0000 7fc99cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:38.144 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":230,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:38.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 msgr2=0x7fc9800408a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:38.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 0x7fc9800408a0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0139030 tx=0x7fc988009040 comp rx=0 tx=0).stop 2024-09-10T16:29:38.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 10 monclient: shutdown 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 msgr2=0x7fc9a012d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc99000b430 tx=0x7fc99000b8f0 comp rx=0 tx=0).stop 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 shutdown_connections 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc98003e3e0 0x7fc9800408a0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9a013e740 0x7fc9a01300e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc9a0134f70 0x7fc9a012d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 --2- 172.21.15.164:0/4260176083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc9a006cb70 0x7fc9a012d280 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 >> 172.21.15.164:0/4260176083 conn(0x7fc9a00aa900 msgr2=0x7fc9a00a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 shutdown_connections 2024-09-10T16:29:38.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:38.143+0000 7fc9a60ef640 1 -- 172.21.15.164:0/4260176083 wait complete. 2024-09-10T16:29:38.378 INFO:tasks.rados.rados.0.smithi165.stdout:3561: done (1 left) 2024-09-10T16:29:38.378 INFO:tasks.rados.rados.0.smithi165.stdout:3562: done (0 left) 2024-09-10T16:29:38.378 INFO:tasks.rados.rados.0.smithi165.stdout:3563: snap_remove snap 376 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3563: done (0 left) 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3564: rollback oid 1 current snap is 400 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 380 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3565: read oid 6 snap -1 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3565: expect deleted 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3566: copy_from oid 4 from oid 26 current snap is 400 2024-09-10T16:29:39.382 INFO:tasks.rados.rados.0.smithi165.stdout:3567: snap_remove snap 392 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3567: done (3 left) 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3568: delete oid 38 current snap is 400 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3564: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3564: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3564: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2020 (ObjNum 1170 snap 368 seq_num 1170) dirty exists 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3566: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:3566: finishing copy_from to smithi16534841-4 2024-09-10T16:29:40.397 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:29:40.401 INFO:tasks.rados.rados.0.smithi165.stdout:3564: done (3 left) 2024-09-10T16:29:40.401 INFO:tasks.rados.rados.0.smithi165.stdout:3565: done (2 left) 2024-09-10T16:29:40.401 INFO:tasks.rados.rados.0.smithi165.stdout:3566: done (1 left) 2024-09-10T16:29:40.401 INFO:tasks.rados.rados.0.smithi165.stdout:3568: done (0 left) 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3569: read oid 35 snap -1 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3569: expect (ObjNum 1223 snap 393 seq_num 1223) 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3570: read oid 25 snap -1 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3570: expect (ObjNum 1210 snap 388 seq_num 1210) 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3571: write (excl) oid 32 current snap is 400 2024-09-10T16:29:40.402 INFO:tasks.rados.rados.0.smithi165.stdout:3571: seq_num 1236 ranges {773452=656950,1877979=617086} 2024-09-10T16:29:40.409 INFO:tasks.rados.rados.0.smithi165.stdout:3571: writing smithi16534841-32 from 773452 to 1430402 tid 1 2024-09-10T16:29:40.416 INFO:tasks.rados.rados.0.smithi165.stdout:3571: writing smithi16534841-32 from 1877979 to 2495065 tid 2 2024-09-10T16:29:40.416 INFO:tasks.rados.rados.0.smithi165.stdout:3572: read oid 23 snap -1 2024-09-10T16:29:40.416 INFO:tasks.rados.rados.0.smithi165.stdout:3572: expect deleted 2024-09-10T16:29:40.417 INFO:tasks.rados.rados.0.smithi165.stdout:3573: read oid 26 snap -1 2024-09-10T16:29:40.417 INFO:tasks.rados.rados.0.smithi165.stdout:3573: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:40.417 INFO:tasks.rados.rados.0.smithi165.stdout:3574: copy_from oid 13 from oid 24 current snap is 400 2024-09-10T16:29:40.418 INFO:tasks.rados.rados.0.smithi165.stdout:3575: copy_from oid 40 from oid 47 current snap is 400 2024-09-10T16:29:40.418 INFO:tasks.rados.rados.0.smithi165.stdout:3570: expect (ObjNum 1210 snap 388 seq_num 1210) 2024-09-10T16:29:40.498 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 243589 2024-09-10T16:29:40.538 INFO:tasks.rados.rados.0.smithi165.stdout:3569: expect (ObjNum 1223 snap 393 seq_num 1223) 2024-09-10T16:29:40.610 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:40.759 INFO:tasks.rados.rados.0.smithi165.stdout:3571: finishing write tid 1 to smithi16534841-32 2024-09-10T16:29:40.759 INFO:tasks.rados.rados.0.smithi165.stdout:3575: finishing copy_from racing read to smithi16534841-40 2024-09-10T16:29:40.759 INFO:tasks.rados.rados.0.smithi165.stdout:3573: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:40.759 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:40.797 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: finishing write tid 2 to smithi16534841-32 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: finishing write tid 3 to smithi16534841-32 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: finishing write tid 4 to smithi16534841-32 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: oid 32 updating version 0 to 1630 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: oid 32 updating version 1630 to 1632 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:3571: oid 32 version 1632 is already newer than 1631 2024-09-10T16:29:40.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1632 (ObjNum 1236 snap 400 seq_num 1236) dirty exists 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3571: left oid 32 (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3574: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3569: done (6 left) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3570: done (5 left) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3571: done (4 left) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3572: done (3 left) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3573: done (2 left) 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3576: delete oid 41 current snap is 400 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3575: finishing copy_from to smithi16534841-40 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1895 (ObjNum 1224 snap 394 seq_num 1224) dirty exists 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:3574: finishing copy_from to smithi16534841-13 2024-09-10T16:29:40.876 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 1876 (ObjNum 1221 snap 393 seq_num 1221) dirty exists 2024-09-10T16:29:40.879 INFO:tasks.rados.rados.0.smithi165.stdout:3574: done (2 left) 2024-09-10T16:29:40.879 INFO:tasks.rados.rados.0.smithi165.stdout:3575: done (1 left) 2024-09-10T16:29:40.879 INFO:tasks.rados.rados.0.smithi165.stdout:3576: done (0 left) 2024-09-10T16:29:40.879 INFO:tasks.rados.rados.0.smithi165.stdout:3577: snap_create 2024-09-10T16:29:41.157 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.264+0000 7fbb5ba57640 10 monclient: get_monmap_and_config 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.264+0000 7fbb5ba57640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.264+0000 7fbb5ba57640 10 monclient: monmap: 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:41.266280+0000 2024-09-10T16:29:41.267 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:41.266280+0000 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 Processor -- start 2024-09-10T16:29:41.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 -- start start 2024-09-10T16:29:41.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient: init 2024-09-10T16:29:41.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-10T16:29:41.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5409d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5409dd10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 0x7fbb54134a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-c con 0x7fbb54134580 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbb5409a540 con 0x7fbb5409d930 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbb54074c70 con 0x7fbb54134580 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbb540a0060 con 0x7fbb5409dd10 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb5ba57640 10 monclient(hunting): authenticate will time out at 2159.753868s 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb58ff2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.265+0000 7fbb597f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36664/0 (socket says 172.21.15.164:36664) 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 -- 172.21.15.164:0/1226985474 learned_addr learned my addr 172.21.15.164:0/1226985474 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46968/0 (socket says 172.21.15.164:46968) 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): get_auth_request con 0x7fbb5409dd10 auth_method 0 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): get_auth_request con 0x7fbb5409d930 auth_method 0 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb59ff4640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 0x7fbb54134a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_done global_id 244283 payload 274 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 -- 172.21.15.164:0/1226985474 >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 msgr2=0x7fbb54134a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 0x7fbb54134a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 -- 172.21.15.164:0/1226985474 >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 msgr2=0x7fbb5409ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 monclient: found mon.noname-b 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb58ff2640 1 -- 172.21.15.164:0/1226985474 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb5409ed40 con 0x7fbb5409dd10 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb59ff4640 10 monclient: get_auth_request con 0x7fbb54134580 auth_method 0 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb59ff4640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 0x7fbb54134a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:41.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.266+0000 7fbb597f3640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb58ff2640 1 -- 172.21.15.164:0/1226985474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 msgr2=0x7fbb5409e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/1226985474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409f490 tx=0x7fbb4402c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=63199acb3f9e4d93 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 1 -- 172.21.15.164:0/1226985474 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44038070 con 0x7fbb5409dd10 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: dump: 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.280 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 5 monclient: authenticate success, global_id 244283 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 1 -- 172.21.15.164:0/1226985474 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb44031920 con 0x7fbb5409dd10 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:41.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 4 set_mon_vals no callback set 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 1 -- 172.21.15.164:0/1226985474 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44031cf0 con 0x7fbb5409dd10 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb4bfff640 10 monclient: dump: 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 10 monclient: shutdown 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 -- 172.21.15.164:0/1226985474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 msgr2=0x7fbb5409e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/1226985474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fbb5409f490 tx=0x7fbb4402c4d0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 -- 172.21.15.164:0/1226985474 shutdown_connections 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.165:3300/0 conn(0x7fbb54134580 0x7fbb54134a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/1226985474 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb5409dd10 0x7fbb5409e130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/1226985474 >> v2:172.21.15.164:3300/0 conn(0x7fbb5409d930 0x7fbb5409ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 -- 172.21.15.164:0/1226985474 >> 172.21.15.164:0/1226985474 conn(0x7fbb540a3110 msgr2=0x7fbb540a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 -- 172.21.15.164:0/1226985474 shutdown_connections 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.269+0000 7fbb5ba57640 1 -- 172.21.15.164:0/1226985474 wait complete. 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.270+0000 7fbb5ba57640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.270+0000 7fbb5ba57640 10 monclient: monmap: 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.270+0000 7fbb5ba57640 1 Processor -- start 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.270+0000 7fbb5ba57640 1 -- start start 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.270+0000 7fbb5ba57640 10 monclient: init 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:29:41.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-b con 0x7fbb5409d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-a con 0x7fbb5409dd10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb54134580 0x7fbb5412d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): picked mon.noname-c con 0x7fbb54134580 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbb540a2940 con 0x7fbb5409dd10 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbb5409e670 con 0x7fbb5409d930 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbb540a2ac0 con 0x7fbb54134580 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb5ba57640 10 monclient(hunting): authenticate will time out at 2159.759392s 2024-09-10T16:29:41.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb54134580 0x7fbb5412d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46982/0 (socket says 172.21.15.164:46982) 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 -- 172.21.15.164:0/4030319219 learned_addr learned my addr 172.21.15.164:0/4030319219 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): get_auth_request con 0x7fbb5409dd10 auth_method 0 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): get_auth_request con 0x7fbb54134580 auth_method 0 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb59ff4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 10 monclient(hunting): get_auth_request con 0x7fbb5409d930 auth_method 0 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 msgr2=0x7fbb5412e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 msgr2=0x7fbb5412e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient(hunting): handle_auth_done global_id 254196 payload 824 2024-09-10T16:29:41.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb597f3640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb54134580 msgr2=0x7fbb5412d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb54134580 0x7fbb5412d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 msgr2=0x7fbb5412e640 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 monclient: found mon.noname-a 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.271+0000 7fbb58ff2640 1 -- 172.21.15.164:0/4030319219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb54130920 con 0x7fbb5409dd10 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb58ff2640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 msgr2=0x7fbb5412cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb58ff2640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000c00 tx=0x7fbb440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44042020 con 0x7fbb5409dd10 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: dump: 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4403daa0 con 0x7fbb5409dd10 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 5 monclient: authenticate success, global_id 254196 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb44038040 con 0x7fbb5409dd10 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb49ffb640 10 monclient: dump: 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 10 monclient: _renew_subs 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb54134290 con 0x7fbb5409dd10 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb4affd640 4 set_mon_vals no callback set 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 10 monclient: _renew_subs 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.274+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb54130080 con 0x7fbb5409dd10 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbb44045020 con 0x7fbb5409dd10 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb5ba57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb5ba57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb5ba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb5409a0b0 con 0x7fbb5409dd10 2024-09-10T16:29:41.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb49ffb640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 0x7fbb380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(804..804 src has 677..804) ==== 4799+0+0 (secure 0 0 0) 0x7fbb440754f0 con 0x7fbb5409dd10 2024-09-10T16:29:41.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.275+0000 7fbb597f3640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 0x7fbb380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.276+0000 7fbb597f3640 10 monclient: get_auth_request con 0x7fbb3803e210 auth_method 0 2024-09-10T16:29:41.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.278+0000 7fbb597f3640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 0x7fbb380406d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000a610 tx=0x7fbb5000a460 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.280+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fbb5409a0b0 con 0x7fbb5409dd10 2024-09-10T16:29:41.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.280+0000 7fbb49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.280+0000 7fbb49ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3577: done (0 left) 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3578: rollback oid 17 current snap is 401 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 17 to 381 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3579: read oid 45 snap -1 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3579: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3580: read oid 48 snap -1 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3580: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3581: copy_from oid 6 from oid 38 current snap is 401 2024-09-10T16:29:41.401 INFO:tasks.rados.rados.0.smithi165.stdout:3582: read oid 7 snap -1 2024-09-10T16:29:41.402 INFO:tasks.rados.rados.0.smithi165.stdout:3582: expect deleted 2024-09-10T16:29:41.402 INFO:tasks.rados.rados.0.smithi165.stdout:3583: write oid 2 current snap is 401 2024-09-10T16:29:41.402 INFO:tasks.rados.rados.0.smithi165.stdout:3583: seq_num 1238 ranges {407066=407448,1594943=620755} 2024-09-10T16:29:41.405 INFO:tasks.rados.rados.0.smithi165.stdout:3583: writing smithi16534841-2 from 407066 to 814514 tid 1 2024-09-10T16:29:41.413 INFO:tasks.rados.rados.0.smithi165.stdout:3583: writing smithi16534841-2 from 1594943 to 2215698 tid 2 2024-09-10T16:29:41.413 INFO:tasks.rados.rados.0.smithi165.stdout:3584: snap_create 2024-09-10T16:29:41.413 INFO:tasks.rados.rados.0.smithi165.stdout:3579: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:29:41.413 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:41.485 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 713785 2024-09-10T16:29:41.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb5ba57640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb5ba57640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb5ba57640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbb540a5570 con 0x7fbb5409dd10 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb49ffb640 1 -- 172.21.15.164:0/4030319219 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fbb540a5570 con 0x7fbb5409dd10 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.531+0000 7fbb49ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:41.534 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 msgr2=0x7fbb380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 0x7fbb380406d0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fbb5000a610 tx=0x7fbb5000a460 comp rx=0 tx=0).stop 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 10 monclient: shutdown 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 msgr2=0x7fbb5412cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbb44000c00 tx=0x7fbb440046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 shutdown_connections 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbb3803e210 0x7fbb380406d0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbb54134580 0x7fbb5412d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbb5409dd10 0x7fbb5412cc80 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 --2- 172.21.15.164:0/4030319219 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbb5409d930 0x7fbb5412e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 >> 172.21.15.164:0/4030319219 conn(0x7fbb540a3110 msgr2=0x7fbb5409b110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 shutdown_connections 2024-09-10T16:29:41.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.533+0000 7fbb5ba57640 1 -- 172.21.15.164:0/4030319219 wait complete. 2024-09-10T16:29:41.546 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:29:41.546 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:29:41.546 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:29:41.581 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2066670 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3578: finishing rollback tid 0 to smithi16534841-17 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3581: finishing copy_from to smithi16534841-6 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3581: got expected ENOENT (src dne) 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3581: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3578: finishing rollback tid 1 to smithi16534841-17 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1842 (ObjNum 33 snap 0 seq_num 1162407920) dirty dne 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3583: finishing write tid 1 to smithi16534841-2 2024-09-10T16:29:41.583 INFO:tasks.rados.rados.0.smithi165.stdout:3580: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:29:41.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 10 monclient: get_monmap_and_config 2024-09-10T16:29:41.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 10 monclient: monmap: 2024-09-10T16:29:41.644 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.644 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:41.643966+0000 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:41.643966+0000 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:41.645 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 1 Processor -- start 2024-09-10T16:29:41.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 1 -- start start 2024-09-10T16:29:41.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.642+0000 7faae6a6e640 10 monclient: init 2024-09-10T16:29:41.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient: _add_conns ranks=[1,4,3,2,0] 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faae006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7faae0134f70 0x7faae013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-c con 0x7faae0134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faae013d560 0x7faae013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-a con 0x7faae013d560 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faae009ce50 con 0x7faae006cb70 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7faae006db60 con 0x7faae013d560 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7faae006cf50 con 0x7faae0134f70 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae6a6e640 10 monclient(hunting): authenticate will time out at 2160.131454s 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faae013d560 0x7faae013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7faae013d560 0x7faae013f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46992/0 (socket says 172.21.15.164:46992) 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 -- 172.21.15.164:0/1240677936 learned_addr learned my addr 172.21.15.164:0/1240677936 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): get_auth_request con 0x7faae013d560 auth_method 0 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faadffff640 1 --2- 172.21.15.164:0/1240677936 >> v2:172.21.15.165:3300/0 conn(0x7faae0134f70 0x7faae013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faadffff640 10 monclient(hunting): get_auth_request con 0x7faae0134f70 auth_method 0 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faadffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faadffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faadffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient(hunting): handle_auth_done global_id 254199 payload 274 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 -- 172.21.15.164:0/1240677936 >> v2:172.21.15.165:3300/0 conn(0x7faae0134f70 msgr2=0x7faae013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 --2- 172.21.15.164:0/1240677936 >> v2:172.21.15.165:3300/0 conn(0x7faae0134f70 0x7faae013d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 -- 172.21.15.164:0/1240677936 >> v1:172.21.15.165:6789/0 conn(0x7faae006cb70 legacy=0x7faae00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 monclient: found mon.noname-a 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.643+0000 7faae500b640 1 -- 172.21.15.164:0/1240677936 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae0135380 con 0x7faae013d560 2024-09-10T16:29:41.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.646+0000 7faae500b640 1 -- 172.21.15.164:0/1240677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013d560 msgr2=0x7faae013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.646+0000 7faae500b640 1 --2- 172.21.15.164:0/1240677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013d560 0x7faae013f950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faad0009e50 tx=0x7faad002a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17da42d39549e5c1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 1 -- 172.21.15.164:0/1240677936 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0036070 con 0x7faae013d560 2024-09-10T16:29:41.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: dump: 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 1 -- 172.21.15.164:0/1240677936 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faad0007800 con 0x7faae013d560 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 5 monclient: authenticate success, global_id 254199 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 1 -- 172.21.15.164:0/1240677936 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad0007bd0 con 0x7faae013d560 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 4 set_mon_vals no callback set 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faadf7fe640 10 monclient: dump: 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.656 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 10 monclient: shutdown 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 -- 172.21.15.164:0/1240677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013d560 msgr2=0x7faae013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1240677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013d560 0x7faae013f950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7faad0009e50 tx=0x7faad002a5b0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 -- 172.21.15.164:0/1240677936 shutdown_connections 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1240677936 >> v2:172.21.15.165:3300/0 conn(0x7faae0134f70 0x7faae013d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1240677936 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013d560 0x7faae013f950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 -- 172.21.15.164:0/1240677936 >> 172.21.15.164:0/1240677936 conn(0x7faae00aa900 msgr2=0x7faae00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 -- 172.21.15.164:0/1240677936 shutdown_connections 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.647+0000 7faae6a6e640 1 -- 172.21.15.164:0/1240677936 wait complete. 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.648+0000 7faae6a6e640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.648+0000 7faae6a6e640 10 monclient: monmap: 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:41.658 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.648+0000 7faae6a6e640 1 Processor -- start 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.648+0000 7faae6a6e640 1 -- start start 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.648+0000 7faae6a6e640 10 monclient: init 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 0x7faae01cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-b con 0x7faae006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 0x7faae01cdb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-a con 0x7faae0134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faae013d560 0x7faae013b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): picked mon.noname-c con 0x7faae013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faae00a4db0 con 0x7faae0134f70 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7faae009d130 con 0x7faae006cb70 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7faae013fe90 con 0x7faae013d560 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae6a6e640 10 monclient(hunting): authenticate will time out at 2160.137252s 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 0x7faae01cdb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 0x7faae01cdb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46996/0 (socket says 172.21.15.164:46996) 2024-09-10T16:29:41.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 0x7faae01cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faae013d560 0x7faae013b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 -- 172.21.15.164:0/1473274627 learned_addr learned my addr 172.21.15.164:0/1473274627 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): get_auth_request con 0x7faae0134f70 auth_method 0 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): get_auth_request con 0x7faae013d560 auth_method 0 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 10 monclient(hunting): get_auth_request con 0x7faae006cb70 auth_method 0 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae500b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 msgr2=0x7faae01cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 msgr2=0x7faae01cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 0x7faae01cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 0x7faae01cd5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient(hunting): handle_auth_done global_id 254202 payload 824 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faae013d560 msgr2=0x7faae013b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7faae013d560 0x7faae013b320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 msgr2=0x7faae01cd5d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:41.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7faae006cb70 0x7faae01cd5d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 monclient: found mon.noname-a 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.649+0000 7faadffff640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faae013fa60 con 0x7faae0134f70 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.651+0000 7faadffff640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 msgr2=0x7faae01cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.651+0000 7faadffff640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 0x7faae01cdb10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faac800d800 tx=0x7faac800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.651+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800b8d0 con 0x7faae0134f70 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: dump: 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.663 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 5 monclient: authenticate success, global_id 254202 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faac8019070 con 0x7faae0134f70 2024-09-10T16:29:41.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 monclient: _renew_subs 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faae009e050 con 0x7faae0134f70 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faac800bc90 con 0x7faae0134f70 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 10 monclient: dump: 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.665 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadeffd640 4 set_mon_vals no callback set 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 monclient: _renew_subs 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faae013cc50 con 0x7faae0134f70 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 0x7faae013b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faae0134750 con 0x7faae01ce050 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7faac801e460 con 0x7faae0134f70 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.652+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 0x7faae013b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient: get_auth_request con 0x7faae01ce050 auth_method 0 2024-09-10T16:29:41.670 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faadd7fa640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faac003e210 0x7faac00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 10 monclient(hunting): handle_auth_done global_id 254205 payload 824 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faae00a6120 con 0x7faae01ce050 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(805..805 src has 677..805) ==== 4771+0+0 (secure 0 0 0) 0x7faac80531b0 con 0x7faae0134f70 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae500b640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faac003e210 0x7faac00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.653+0000 7faae500b640 10 monclient: get_auth_request con 0x7faac003e210 auth_method 0 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 msgr2=0x7faae013b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 0x7faae013b860 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.671 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8018020 con 0x7faae01ce050 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 10 monclient: dump: 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.672 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7faad8014910 con 0x7faae01ce050 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 msgr2=0x7faae013b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faae500b640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faac003e210 0x7faac00406d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faae006e110 tx=0x7faad0009000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 0x7faae013b860 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad800d960 comp rx=0 tx=0).stop 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 msgr2=0x7faae013b860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.655+0000 7faadd7fa640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae01ce050 0x7faae013b860 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad800d960 comp rx=0 tx=0).stop 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.661+0000 7faae6a6e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.661+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 0x7faae013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.661+0000 7faae6a6e640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.661+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7faae013c1b0 con 0x7faae013bda0 2024-09-10T16:29:41.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 0x7faae013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient: get_auth_request con 0x7faae013bda0 auth_method 0 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 10 monclient(hunting): handle_auth_done global_id 254208 payload 824 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.662+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faae013ceb0 con 0x7faae013bda0 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faae480a640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 msgr2=0x7faae013d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faae480a640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 0x7faae013d940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad80046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faad8018040 con 0x7faae013bda0 2024-09-10T16:29:41.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 10 monclient: dump: 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7faad801f220 con 0x7faae013bda0 2024-09-10T16:29:41.675 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 msgr2=0x7faae013d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 0x7faae013d940 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad80046f0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 msgr2=0x7faae013d940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.664+0000 7faadd7fa640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae013bda0 0x7faae013d940 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7faad8006df0 tx=0x7faad80046f0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:41.676 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 70315, 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.677 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:41.678 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.679 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:41.680 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.681 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.682 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.683 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.684 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:41.685 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:41.686 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:41.687 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:41.688 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faac003e210 msgr2=0x7faac00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7faac003e210 0x7faac00406d0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7faae006e110 tx=0x7faad0009000 comp rx=0 tx=0).stop 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 10 monclient: shutdown 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 msgr2=0x7faae01cdb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 --2- 172.21.15.164:0/1473274627 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7faae0134f70 0x7faae01cdb10 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7faac800d800 tx=0x7faac800dcc0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae500b640 1 -- 172.21.15.164:0/1473274627 reap_dead start 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae500b640 1 -- 172.21.15.164:0/1473274627 reap_dead start 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 shutdown_connections 2024-09-10T16:29:41.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 >> 172.21.15.164:0/1473274627 conn(0x7faae00aa900 msgr2=0x7faae0134040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 shutdown_connections 2024-09-10T16:29:41.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.665+0000 7faae6a6e640 1 -- 172.21.15.164:0/1473274627 wait complete. 2024-09-10T16:29:41.691 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:41.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:29:41.691 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:29:41.697 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2364236 2024-09-10T16:29:41.728 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.787+0000 7fa7acb2b640 10 monclient: get_monmap_and_config 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.787+0000 7fa7acb2b640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.787+0000 7fa7acb2b640 10 monclient: monmap: 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:41.789721+0000 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:41.789721+0000 2024-09-10T16:29:41.790 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:41.791 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.788+0000 7fa7acb2b640 1 Processor -- start 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.788+0000 7fa7acb2b640 1 -- start start 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.788+0000 7fa7acb2b640 10 monclient: init 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a8133a00 0x7fa7a80a2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a8133a00 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7a8133de0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:41.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7a8134270 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a809dcf0 con 0x7fa7a8133de0 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a80a1c90 con 0x7fa7a8134270 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa7a8073ee0 con 0x7fa7a8133a00 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7acb2b640 10 monclient(hunting): authenticate will time out at 2160.277174s 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a8133a00 0x7fa7a80a2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa7a8133a00 0x7fa7a80a2ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47020/0 (socket says 172.21.15.164:47020) 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 -- 172.21.15.164:0/1994822360 learned_addr learned my addr 172.21.15.164:0/1994822360 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a5d9b640 1 --1- 172.21.15.164:0/1994822360 >> v1:172.21.15.164:6789/0 conn(0x7fa7a8133de0 0x7fa7a809ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54496/0 (socket says 172.21.15.164:54496) 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): get_auth_request con 0x7fa7a8133a00 auth_method 0 2024-09-10T16:29:41.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient(hunting): handle_auth_done global_id 254211 payload 274 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 -- 172.21.15.164:0/1994822360 >> v1:172.21.15.165:6789/0 conn(0x7fa7a8134270 legacy=0x7fa7a809d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 -- 172.21.15.164:0/1994822360 >> v1:172.21.15.164:6789/0 conn(0x7fa7a8133de0 legacy=0x7fa7a809ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 monclient: found mon.noname-a 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:41.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.789+0000 7fa7a659c640 1 -- 172.21.15.164:0/1994822360 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a80a1f70 con 0x7fa7a8133a00 2024-09-10T16:29:41.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a659c640 1 -- 172.21.15.164:0/1994822360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8133a00 msgr2=0x7fa7a80a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a659c640 1 --2- 172.21.15.164:0/1994822360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8133a00 0x7fa7a80a2ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009e50 tx=0x7fa7a002a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c42266204c5aa27 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 1 -- 172.21.15.164:0/1994822360 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0036070 con 0x7fa7a8133a00 2024-09-10T16:29:41.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: dump: 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 1 -- 172.21.15.164:0/1994822360 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a0007800 con 0x7fa7a8133a00 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7acb2b640 5 monclient: authenticate success, global_id 254211 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7acb2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 1 -- 172.21.15.164:0/1994822360 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0007bd0 con 0x7fa7a8133a00 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7acb2b640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7acb2b640 4 set_mon_vals no callback set 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.792+0000 7fa7a559a640 10 monclient: dump: 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.804 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 10 monclient: shutdown 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1994822360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8133a00 msgr2=0x7fa7a80a2ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1994822360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8133a00 0x7fa7a80a2ff0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa7a0009e50 tx=0x7fa7a002a560 comp rx=0 tx=0).stop 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1994822360 shutdown_connections 2024-09-10T16:29:41.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1994822360 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8133a00 0x7fa7a80a2ff0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1994822360 >> 172.21.15.164:0/1994822360 conn(0x7fa7a80a7320 msgr2=0x7fa7a80a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1994822360 shutdown_connections 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1994822360 wait complete. 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.793+0000 7fa7acb2b640 10 monclient: monmap: 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:41.806 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 Processor -- start 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 -- start start 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient: init 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 0x7fa7a813fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a8133a00 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 0x7fa7a8132af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a8133de0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8134270 0x7fa7a8133030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a8134270 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:41.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a80a36b0 con 0x7fa7a8134270 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a80a3530 con 0x7fa7a8133de0 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a8140fe0 con 0x7fa7a8133a00 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7acb2b640 10 monclient(hunting): authenticate will time out at 2160.282849s 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 0x7fa7a813fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 0x7fa7a813fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36690/0 (socket says 172.21.15.164:36690) 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 1 -- 172.21.15.164:0/1011855761 learned_addr learned my addr 172.21.15.164:0/1011855761 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 10 monclient(hunting): get_auth_request con 0x7fa7a8133a00 auth_method 0 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a6d9d640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8134270 0x7fa7a8133030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 0x7fa7a8132af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a6d9d640 10 monclient(hunting): get_auth_request con 0x7fa7a8134270 auth_method 0 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.794+0000 7fa7a6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 10 monclient(hunting): get_auth_request con 0x7fa7a8133de0 auth_method 0 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 msgr2=0x7fa7a8132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 msgr2=0x7fa7a8132af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 0x7fa7a8132af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 0x7fa7a8132af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient(hunting): handle_auth_done global_id 244292 payload 824 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 msgr2=0x7fa7a8132af0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7a8133de0 0x7fa7a8132af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8134270 msgr2=0x7fa7a8133030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8134270 0x7fa7a8133030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 monclient: found mon.noname-c 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a659c640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a813dbc0 con 0x7fa7a8133a00 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.795+0000 7fa7a6d9d640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7a8134270 0x7fa7a8133030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7a659c640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 msgr2=0x7fa7a813fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7a659c640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 0x7fa7a813fb50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00093e0 tx=0x7fa7a0004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0041020 con 0x7fa7a8133a00 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: dump: 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.811 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7a00319c0 con 0x7fa7a8133a00 2024-09-10T16:29:41.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 5 monclient: authenticate success, global_id 244292 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a0036040 con 0x7fa7a8133a00 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa79effd640 10 monclient: dump: 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.813 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 10 monclient: _renew_subs 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7a4d99640 4 set_mon_vals no callback set 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a809e0f0 con 0x7fa7a8133a00 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 10 monclient: _renew_subs 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.797+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a813eb20 con 0x7fa7a8133a00 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7acb2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 0x7fa7a802d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a81407e0 con 0x7fa7a813bd60 2024-09-10T16:29:41.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa7a0044020 con 0x7fa7a8133a00 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 0x7fa7a802d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient: get_auth_request con 0x7fa7a813bd60 auth_method 0 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa79effd640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78803e210 0x7fa7880406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_done global_id 244295 payload 824 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(805..805 src has 677..805) ==== 4771+0+0 (secure 0 0 0) 0x7fa7a0073df0 con 0x7fa7a8133a00 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.798+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa7a80a2ff0 con 0x7fa7a813bd60 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.799+0000 7fa7a6d9d640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78803e210 0x7fa7880406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.799+0000 7fa7a6d9d640 10 monclient: get_auth_request con 0x7fa78803e210 auth_method 0 2024-09-10T16:29:41.816 INFO:tasks.rados.rados.0.smithi165.stdout:3583: finishing write tid 2 to smithi16534841-2 2024-09-10T16:29:41.816 INFO:tasks.rados.rados.0.smithi165.stdout:3583: finishing write tid 3 to smithi16534841-2 2024-09-10T16:29:41.816 INFO:tasks.rados.rados.0.smithi165.stdout:3583: finishing write tid 4 to smithi16534841-2 2024-09-10T16:29:41.816 INFO:tasks.rados.rados.0.smithi165.stdout:3583: oid 2 updating version 0 to 1927 2024-09-10T16:29:41.817 INFO:tasks.rados.rados.0.smithi165.stdout:3583: oid 2 version 1927 is already newer than 1926 2024-09-10T16:29:41.817 INFO:tasks.rados.rados.0.smithi165.stdout:3583: oid 2 version 1927 is already newer than 1925 2024-09-10T16:29:41.817 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1927 (ObjNum 1238 snap 401 seq_num 1238) dirty exists 2024-09-10T16:29:41.817 INFO:tasks.rados.rados.0.smithi165.stdout:3583: left oid 2 (ObjNum 1238 snap 401 seq_num 1238) 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 msgr2=0x7fa7a802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 0x7fa7a802d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa7a813db50 tx=0x7fa79000e960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790004e20 con 0x7fa7a813bd60 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 10 monclient: dump: 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.819 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa7a6d9d640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78803e210 0x7fa7880406d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7a813d780 tx=0x7fa798007db0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa790018070 con 0x7fa7a813bd60 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 msgr2=0x7fa7a802d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 0x7fa7a802d440 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa7a813db50 tx=0x7fa79000e960 comp rx=0 tx=0).stop 2024-09-10T16:29:41.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 msgr2=0x7fa7a802d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.801+0000 7fa79effd640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813bd60 0x7fa7a802d440 secure :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa7a813db50 tx=0x7fa79000e960 comp rx=0 tx=0).stop 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7acb2b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 0x7fa7a802d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7acb2b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7a80a0a80 con 0x7fa7a813c140 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 0x7fa7a802d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient: get_auth_request con 0x7fa7a813c140 auth_method 0 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.807+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.808+0000 7fa7a5d9b640 10 monclient(hunting): handle_auth_done global_id 244298 payload 824 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.808+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa7a8133570 con 0x7fa7a813c140 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa7a5d9b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 msgr2=0x7fa7a802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa7a5d9b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 0x7fa7a802d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa79000efd0 tx=0x7fa790004930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7900062e0 con 0x7fa7a813c140 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 10 monclient: dump: 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.822 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7fa7900062e0 con 0x7fa7a813c140 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 msgr2=0x7fa7a802d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 0x7fa7a802d980 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa79000efd0 tx=0x7fa790004930 comp rx=0 tx=0).stop 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 msgr2=0x7fa7a802d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:41.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.810+0000 7fa79effd640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a813c140 0x7fa7a802d980 secure :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fa79000efd0 tx=0x7fa790004930 comp rx=0 tx=0).stop 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 118718, 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.824 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.825 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.826 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:41.827 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:41.828 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.829 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:41.830 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:41.831 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.832 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:41.833 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.834 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:41.835 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:41.836 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:41.837 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78803e210 msgr2=0x7fa7880406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa78803e210 0x7fa7880406d0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fa7a813d780 tx=0x7fa798007db0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 10 monclient: shutdown 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 msgr2=0x7fa7a813fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 --2- 172.21.15.164:0/1011855761 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7a8133a00 0x7fa7a813fb50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa7a00093e0 tx=0x7fa7a0004a40 comp rx=0 tx=0).stop 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7a6d9d640 1 -- 172.21.15.164:0/1011855761 reap_dead start 2024-09-10T16:29:41.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7a6d9d640 1 -- 172.21.15.164:0/1011855761 reap_dead start 2024-09-10T16:29:41.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 shutdown_connections 2024-09-10T16:29:41.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 >> 172.21.15.164:0/1011855761 conn(0x7fa7a80a7320 msgr2=0x7fa7a809e610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 shutdown_connections 2024-09-10T16:29:41.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.811+0000 7fa7acb2b640 1 -- 172.21.15.164:0/1011855761 wait complete. 2024-09-10T16:29:41.839 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:41.839 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:29:41.839 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:29:41.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.936+0000 7f97fc2f7640 10 monclient: get_monmap_and_config 2024-09-10T16:29:41.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.936+0000 7f97fc2f7640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.936+0000 7f97fc2f7640 10 monclient: monmap: 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:41.938565+0000 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:41.938565+0000 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.940 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:41.941 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 Processor -- start 2024-09-10T16:29:41.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 -- start start 2024-09-10T16:29:41.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient: init 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f97f409f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97f40a2550 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 0x7f97f413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-b con 0x7f97f40a2980 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f409fd10 con 0x7f97f409f930 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97f409cd20 con 0x7f97f40a2550 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f97f409a120 con 0x7f97f40a2980 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.937+0000 7f97fc2f7640 10 monclient(hunting): authenticate will time out at 2160.425964s 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa093640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f97f409f930 0x7f97f40a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54502/0 (socket says 172.21.15.164:54502) 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 0x7f97f413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa093640 1 -- 172.21.15.164:0/2757444340 learned_addr learned my addr 172.21.15.164:0/2757444340 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): get_auth_request con 0x7f97f40a2980 auth_method 0 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient(hunting): handle_auth_done global_id 244301 payload 274 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 1 -- 172.21.15.164:0/2757444340 >> v1:172.21.15.165:6789/0 conn(0x7f97f40a2550 legacy=0x7f97f413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 1 -- 172.21.15.164:0/2757444340 >> v1:172.21.15.164:6789/0 conn(0x7f97f409f930 legacy=0x7f97f40a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 1 monclient: found mon.noname-b 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.938+0000 7f97fa894640 1 -- 172.21.15.164:0/2757444340 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f409e840 con 0x7f97f40a2980 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fa894640 1 -- 172.21.15.164:0/2757444340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 msgr2=0x7f97f413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fa894640 1 --2- 172.21.15.164:0/2757444340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 0x7f97f413ed00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009d90 tx=0x7f97ec02a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9db6aac690339fd8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 1 -- 172.21.15.164:0/2757444340 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec007a30 con 0x7f97f40a2980 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: dump: 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 5 monclient: authenticate success, global_id 244301 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 1 -- 172.21.15.164:0/2757444340 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97ec007bd0 con 0x7f97f40a2980 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 4 set_mon_vals no callback set 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 1 -- 172.21.15.164:0/2757444340 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97ec031400 con 0x7f97f40a2980 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97f9091640 10 monclient: dump: 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.954 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 10 monclient: shutdown 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 -- 172.21.15.164:0/2757444340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 msgr2=0x7f97f413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/2757444340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 0x7f97f413ed00 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009d90 tx=0x7f97ec02a440 comp rx=0 tx=0).stop 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 -- 172.21.15.164:0/2757444340 shutdown_connections 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/2757444340 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2980 0x7f97f413ed00 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 -- 172.21.15.164:0/2757444340 >> 172.21.15.164:0/2757444340 conn(0x7f97f40a3110 msgr2=0x7f97f40a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 -- 172.21.15.164:0/2757444340 shutdown_connections 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.941+0000 7f97fc2f7640 1 -- 172.21.15.164:0/2757444340 wait complete. 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.942+0000 7f97fc2f7640 10 monclient: build_initial_monmap 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.942+0000 7f97fc2f7640 10 monclient: monmap: 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:41.956 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.942+0000 7f97fc2f7640 1 Processor -- start 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.942+0000 7f97fc2f7640 1 -- start start 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.942+0000 7f97fc2f7640 10 monclient: init 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:41.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 0x7f97f41ba400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-b con 0x7f97f409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-c con 0x7f97f40a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f40a2980 0x7f97f41b0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): picked mon.noname-a con 0x7f97f40a2980 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f97f409c8b0 con 0x7f97f40a2980 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f97f409d930 con 0x7f97f409f930 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f97f409ca30 con 0x7f97f40a2550 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fc2f7640 10 monclient(hunting): authenticate will time out at 2160.431290s 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f40a2980 0x7f97f41b0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36724/0 (socket says 172.21.15.164:36724) 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 -- 172.21.15.164:0/4113994927 learned_addr learned my addr 172.21.15.164:0/4113994927 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): get_auth_request con 0x7f97f40a2550 auth_method 0 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): get_auth_request con 0x7f97f40a2980 auth_method 0 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 0x7f97f41ba400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 10 monclient(hunting): get_auth_request con 0x7f97f409f930 auth_method 0 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa894640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:41.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 msgr2=0x7f97f41ba400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 msgr2=0x7f97f41ba400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 0x7f97f41ba400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 0x7f97f41ba400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient(hunting): handle_auth_done global_id 244304 payload 824 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 msgr2=0x7f97f41ba400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f409f930 0x7f97f41ba400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f40a2980 msgr2=0x7f97f41b0470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f97f40a2980 0x7f97f41b0470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient: _finish_hunting 0 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 monclient: found mon.noname-c 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:41.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.943+0000 7f97f9892640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97f41bb290 con 0x7f97f40a2550 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.945+0000 7f97f9892640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 msgr2=0x7f97f41ba940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.945+0000 7f97f9892640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c430 tx=0x7f97e400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e40119a0 con 0x7f97f40a2550 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: dump: 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 5 monclient: authenticate success, global_id 244304 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97e4017070 con 0x7f97f40a2550 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 monclient: _renew_subs 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4011d80 con 0x7f97f40a2550 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97f41bae80 con 0x7f97f40a2550 2024-09-10T16:29:41.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 10 monclient: dump: 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97f8890640 4 set_mon_vals no callback set 2024-09-10T16:29:41.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 monclient: _renew_subs 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f97f41b39c0 con 0x7f97f40a2550 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f97f41b2890 con 0x7f97f41cafe0 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.946+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f97e401c460 con 0x7f97f40a2550 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 0x7f97d80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97f41cafe0 auth_method 0 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(805..805 src has 677..805) ==== 4771+0+0 (secure 0 0 0) 0x7f97e4052ba0 con 0x7f97f40a2550 2024-09-10T16:29:41.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa894640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 0x7f97d80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.947+0000 7f97fa894640 10 monclient: get_auth_request con 0x7f97d803e210 auth_method 0 2024-09-10T16:29:41.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:41.949+0000 7f97fa894640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 0x7f97d80406d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f97f406d470 tx=0x7f97ec02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:42.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:42.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97f41cafe0 auth_method 0 2024-09-10T16:29:42.150 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:42.151 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.148+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:42.406 INFO:tasks.rados.rados.0.smithi165.stdout:3578: done (6 left) 2024-09-10T16:29:42.406 INFO:tasks.rados.rados.0.smithi165.stdout:3579: done (5 left) 2024-09-10T16:29:42.406 INFO:tasks.rados.rados.0.smithi165.stdout:3580: done (4 left) 2024-09-10T16:29:42.407 INFO:tasks.rados.rados.0.smithi165.stdout:3581: done (3 left) 2024-09-10T16:29:42.407 INFO:tasks.rados.rados.0.smithi165.stdout:3582: done (2 left) 2024-09-10T16:29:42.407 INFO:tasks.rados.rados.0.smithi165.stdout:3583: done (1 left) 2024-09-10T16:29:42.407 INFO:tasks.rados.rados.0.smithi165.stdout:3584: done (0 left) 2024-09-10T16:29:42.407 INFO:tasks.rados.rados.0.smithi165.stdout:3585: delete oid 27 current snap is 402 2024-09-10T16:29:42.410 INFO:tasks.rados.rados.0.smithi165.stdout:3585: done (0 left) 2024-09-10T16:29:42.410 INFO:tasks.rados.rados.0.smithi165.stdout:3586: write (excl) oid 37 current snap is 402 2024-09-10T16:29:42.411 INFO:tasks.rados.rados.0.smithi165.stdout:3586: seq_num 1240 ranges {440680=674299,1711907=624424} 2024-09-10T16:29:42.416 INFO:tasks.rados.rados.0.smithi165.stdout:3586: writing smithi16534841-37 from 440680 to 1114979 tid 1 2024-09-10T16:29:42.423 INFO:tasks.rados.rados.0.smithi165.stdout:3586: writing smithi16534841-37 from 1711907 to 2336331 tid 2 2024-09-10T16:29:42.423 INFO:tasks.rados.rados.0.smithi165.stdout:3587: delete oid 2 current snap is 402 2024-09-10T16:29:42.423 INFO:tasks.rados.rados.0.smithi165.stdout:3586: finishing write tid 1 to smithi16534841-37 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: finishing write tid 2 to smithi16534841-37 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: finishing write tid 3 to smithi16534841-37 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3587: done (1 left) 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3588: copy_from oid 13 from oid 27 current snap is 402 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: finishing write tid 4 to smithi16534841-37 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: oid 37 updating version 0 to 1592 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: oid 37 version 1592 is already newer than 1590 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:3586: oid 37 version 1592 is already newer than 1591 2024-09-10T16:29:42.427 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1592 (ObjNum 1240 snap 402 seq_num 1240) dirty exists 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3586: left oid 37 (ObjNum 1240 snap 402 seq_num 1240) 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3586: done (1 left) 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3589: delete oid 49 current snap is 402 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3588: finishing copy_from to smithi16534841-13 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3588: got expected ENOENT (src dne) 2024-09-10T16:29:42.428 INFO:tasks.rados.rados.0.smithi165.stdout:3588: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:29:42.430 INFO:tasks.rados.rados.0.smithi165.stdout:3588: done (1 left) 2024-09-10T16:29:42.430 INFO:tasks.rados.rados.0.smithi165.stdout:3589: done (0 left) 2024-09-10T16:29:42.430 INFO:tasks.rados.rados.0.smithi165.stdout:3590: snap_remove snap 374 2024-09-10T16:29:42.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97f41cafe0 auth_method 0 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:42.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:42.549+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97f41cafe0 auth_method 0 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:43.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 10 monclient(hunting): handle_auth_done global_id 264100 payload 824 2024-09-10T16:29:43.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.351+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97f409c440 con 0x7f97f41cafe0 2024-09-10T16:29:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:43.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97f0007c90 tx=0x7f97f0011cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0018070 con 0x7f97f41cafe0 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.354+0000 7f97eaffd640 10 monclient: dump: 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.356 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97f000fc60 con 0x7f97f41cafe0 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97f0007c90 tx=0x7f97f0011cc0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 msgr2=0x7f97f41b09b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:43.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.355+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41cafe0 0x7f97f41b09b0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97f0007c90 tx=0x7f97f0011cc0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.361+0000 7f97fc2f7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:43.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.361+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.364 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fc2f7640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f97f41b1300 con 0x7f97f41b0ef0 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97f41b0ef0 auth_method 0 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 10 monclient(hunting): handle_auth_done global_id 264103 payload 824 2024-09-10T16:29:43.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.362+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97f40a0880 con 0x7f97f41b0ef0 2024-09-10T16:29:43.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 msgr2=0x7f97f4070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:43.366 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f001a000 tx=0x7f97f0007c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f0018050 con 0x7f97f41b0ef0 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 10 monclient: dump: 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.368 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 msgr2=0x7f97f4070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 msgr2=0x7f97f4070540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f001a000 tx=0x7f97f0007c20 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97f001a000 tx=0x7f97f0007c20 comp rx=0 tx=0).stop 2024-09-10T16:29:43.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 msgr2=0x7f97f4070540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 0x7f97d8046810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 msgr2=0x7f97f4070540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97f41b0ef0 0x7f97f4070540 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f97d8046e20 con 0x7f97d8044420 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 0x7f97d8046810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.365+0000 7f97fa093640 10 monclient: get_auth_request con 0x7f97d8044420 auth_method 0 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 10 monclient(hunting): handle_auth_done global_id 264106 payload 824 2024-09-10T16:29:43.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.366+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97f000fc60 con 0x7f97d8044420 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97fa093640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 msgr2=0x7f97d8046810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97fa093640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 0x7f97d8046810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97f41b1480 tx=0x7f97f0009040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97f00135a0 con 0x7f97d8044420 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97eaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97eaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:43.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.368+0000 7f97eaffd640 10 monclient: dump: 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f97f00135a0 con 0x7f97d8044420 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:43.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 msgr2=0x7f97d8046810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 0x7f97d8046810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f97f41b1480 tx=0x7f97f0009040 comp rx=0 tx=0).stop 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 msgr2=0x7f97d8046810 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97eaffd640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f97d8044420 0x7f97d8046810 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fa894640 1 -- 172.21.15.164:0/4113994927 reap_dead start 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 5509, 2024-09-10T16:29:43.374 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:43.375 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:43.376 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:43.377 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:43.378 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.379 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.380 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:43.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:43.382 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:43.383 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.384 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:43.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:43.386 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:43.386 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 msgr2=0x7f97d80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 0x7f97d80406d0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7f97f406d470 tx=0x7f97ec02f000 comp rx=0 tx=0).stop 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 10 monclient: shutdown 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 msgr2=0x7f97f41ba940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f97e400c430 tx=0x7f97e400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 shutdown_connections 2024-09-10T16:29:43.387 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f97d803e210 0x7f97d80406d0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 --2- 172.21.15.164:0/4113994927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f97f40a2550 0x7f97f41ba940 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 >> 172.21.15.164:0/4113994927 conn(0x7f97f40a3110 msgr2=0x7f97f40a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:43.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 shutdown_connections 2024-09-10T16:29:43.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.369+0000 7f97fc2f7640 1 -- 172.21.15.164:0/4113994927 wait complete. 2024-09-10T16:29:43.388 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:43.388 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:29:43.388 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3590: done (0 left) 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3591: read oid 31 snap -1 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3591: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3592: read oid 30 snap -1 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3592: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3593: write oid 15 current snap is 402 2024-09-10T16:29:43.410 INFO:tasks.rados.rados.0.smithi165.stdout:3593: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2024-09-10T16:29:43.415 INFO:tasks.rados.rados.0.smithi165.stdout:3593: writing smithi16534841-15 from 491101 to 991147 tid 1 2024-09-10T16:29:43.421 INFO:tasks.rados.rados.0.smithi165.stdout:3593: writing smithi16534841-15 from 1487353 to 1975457 tid 2 2024-09-10T16:29:43.427 INFO:tasks.rados.rados.0.smithi165.stdout:3593: writing smithi16534841-15 from 2401133 to 2891101 tid 3 2024-09-10T16:29:43.427 INFO:tasks.rados.rados.0.smithi165.stdout:3594: read oid 5 snap -1 2024-09-10T16:29:43.427 INFO:tasks.rados.rados.0.smithi165.stdout:3594: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:29:43.427 INFO:tasks.rados.rados.0.smithi165.stdout:3595: write oid 11 current snap is 402 2024-09-10T16:29:43.428 INFO:tasks.rados.rados.0.smithi165.stdout:3595: seq_num 1244 ranges {507908=575295,1829482=631762} 2024-09-10T16:29:43.433 INFO:tasks.rados.rados.0.smithi165.stdout:3595: writing smithi16534841-11 from 507908 to 1083203 tid 1 2024-09-10T16:29:43.441 INFO:tasks.rados.rados.0.smithi165.stdout:3595: writing smithi16534841-11 from 1829482 to 2461244 tid 2 2024-09-10T16:29:43.441 INFO:tasks.rados.rados.0.smithi165.stdout:3596: write oid 1 current snap is 402 2024-09-10T16:29:43.441 INFO:tasks.rados.rados.0.smithi165.stdout:3596: seq_num 1245 ranges {524715=650544,1887964=491773} 2024-09-10T16:29:43.448 INFO:tasks.rados.rados.0.smithi165.stdout:3596: writing smithi16534841-1 from 524715 to 1175259 tid 1 2024-09-10T16:29:43.455 INFO:tasks.rados.rados.0.smithi165.stdout:3596: writing smithi16534841-1 from 1887964 to 2379737 tid 2 2024-09-10T16:29:43.455 INFO:tasks.rados.rados.0.smithi165.stdout:3591: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:29:43.455 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:43.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 10 monclient: get_monmap_and_config 2024-09-10T16:29:43.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 10 monclient: build_initial_monmap 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 10 monclient: monmap: 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:43.503086+0000 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:43.503086+0000 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.504 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:43.505 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:43.505 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:43.505 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:43.505 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:43.505 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 1 Processor -- start 2024-09-10T16:29:43.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 1 -- start start 2024-09-10T16:29:43.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.502+0000 7f231f495640 10 monclient: init 2024-09-10T16:29:43.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:43.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f231806cb70 0x7f2318134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): picked mon.noname-a con 0x7f231806cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23180a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f23180a4ab0 0x7f23181383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): picked mon.noname-c con 0x7f23180a4ab0 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f231809ce50 con 0x7f23180a46a0 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f231806db60 con 0x7f231806cb70 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f231806cf50 con 0x7f23180a4ab0 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:43.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231f495640 10 monclient(hunting): authenticate will time out at 2161.990473s 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f231806cb70 0x7f2318134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231ca30640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f23180a46a0 0x7f2318137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:54514/0 (socket says 172.21.15.164:54514) 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f231806cb70 0x7f2318134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47052/0 (socket says 172.21.15.164:47052) 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231ca30640 1 -- 172.21.15.164:0/1379474236 learned_addr learned my addr 172.21.15.164:0/1379474236 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): get_auth_request con 0x7f231806cb70 auth_method 0 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231da32640 1 --2- 172.21.15.164:0/1379474236 >> v2:172.21.15.165:3300/0 conn(0x7f23180a4ab0 0x7f23181383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231da32640 10 monclient(hunting): get_auth_request con 0x7f23180a4ab0 auth_method 0 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231da32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231da32640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231da32640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient(hunting): handle_auth_done global_id 254220 payload 274 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 -- 172.21.15.164:0/1379474236 >> v2:172.21.15.165:3300/0 conn(0x7f23180a4ab0 msgr2=0x7f23181383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 --2- 172.21.15.164:0/1379474236 >> v2:172.21.15.165:3300/0 conn(0x7f23180a4ab0 0x7f23181383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 -- 172.21.15.164:0/1379474236 >> v1:172.21.15.164:6789/0 conn(0x7f23180a46a0 legacy=0x7f2318137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient: _finish_hunting 0 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 monclient: found mon.noname-a 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.503+0000 7f231d231640 1 -- 172.21.15.164:0/1379474236 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231809cb70 con 0x7f231806cb70 2024-09-10T16:29:43.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f231d231640 1 -- 172.21.15.164:0/1379474236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f231806cb70 msgr2=0x7f2318134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f231d231640 1 --2- 172.21.15.164:0/1379474236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f231806cb70 0x7f2318134f70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23140097d0 tx=0x7f231402a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23048098ed1bdccc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 1 -- 172.21.15.164:0/1379474236 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314037070 con 0x7f231806cb70 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: dump: 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.518 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: _finish_auth 0 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f231f495640 5 monclient: authenticate success, global_id 254220 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 1 -- 172.21.15.164:0/1379474236 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2314032440 con 0x7f231806cb70 2024-09-10T16:29:43.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f231f495640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f231f495640 4 set_mon_vals no callback set 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 1 -- 172.21.15.164:0/1379474236 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314032810 con 0x7f231806cb70 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.506+0000 7f230ffff640 10 monclient: dump: 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.520 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 10 monclient: shutdown 2024-09-10T16:29:43.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 -- 172.21.15.164:0/1379474236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f231806cb70 msgr2=0x7f2318134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 --2- 172.21.15.164:0/1379474236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f231806cb70 0x7f2318134f70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f23140097d0 tx=0x7f231402a040 comp rx=0 tx=0).stop 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 -- 172.21.15.164:0/1379474236 shutdown_connections 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 --2- 172.21.15.164:0/1379474236 >> v2:172.21.15.165:3300/0 conn(0x7f23180a4ab0 0x7f23181383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 --2- 172.21.15.164:0/1379474236 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f231806cb70 0x7f2318134f70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 -- 172.21.15.164:0/1379474236 >> 172.21.15.164:0/1379474236 conn(0x7f23180aa900 msgr2=0x7f23180aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 -- 172.21.15.164:0/1379474236 shutdown_connections 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 1 -- 172.21.15.164:0/1379474236 wait complete. 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 10 monclient: build_initial_monmap 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.507+0000 7f231f495640 10 monclient: monmap: 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:43.522 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 Processor -- start 2024-09-10T16:29:43.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 -- start start 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient: init 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): picked mon.noname-c con 0x7f231806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): picked mon.noname-a con 0x7f23180a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23180a4ab0 0x7f23181cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): picked mon.noname-b con 0x7f23180a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2318138a70 con 0x7f23180a46a0 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f23181388f0 con 0x7f23180a4ab0 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2318140040 con 0x7f231806cb70 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231f495640 10 monclient(hunting): authenticate will time out at 2161.995762s 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47062/0 (socket says 172.21.15.164:47062) 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:36726/0 (socket says 172.21.15.164:36726) 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 1 -- 172.21.15.164:0/565411083 learned_addr learned my addr 172.21.15.164:0/565411083 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): get_auth_request con 0x7f23180a46a0 auth_method 0 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): get_auth_request con 0x7f231806cb70 auth_method 0 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231da32640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23180a4ab0 0x7f23181cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231da32640 10 monclient(hunting): get_auth_request con 0x7f23180a4ab0 auth_method 0 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231da32640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231da32640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231da32640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.526 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231ca30640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.508+0000 7f231d231640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231da32640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231da32640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231da32640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 10 monclient(hunting): handle_auth_done global_id 244307 payload 824 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 -- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23180a4ab0 msgr2=0x7f23181cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23180a4ab0 0x7f23181cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 -- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 msgr2=0x7f231813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 10 monclient: _finish_hunting 0 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 monclient: found mon.noname-c 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231d231640 1 -- 172.21.15.164:0/565411083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f231806e720 con 0x7f231806cb70 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.509+0000 7f231ca30640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231d231640 1 -- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 msgr2=0x7f231813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231d231640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2314031000 tx=0x7f23140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314037040 con 0x7f231806cb70 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: dump: 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.528 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: _finish_auth 0 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 5 monclient: authenticate success, global_id 244307 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 10 monclient: _renew_subs 2024-09-10T16:29:43.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f231813fba0 con 0x7f231806cb70 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2314004250 con 0x7f231806cb70 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231403c670 con 0x7f231806cb70 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230dffb640 10 monclient: dump: 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:43.530 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 10 monclient: _renew_subs 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f230f7fe640 4 set_mon_vals no callback set 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.511+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f231813b330 con 0x7f231806cb70 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f231403a070 con 0x7f231806cb70 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f231f495640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f231f495640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f231f495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f231813b660 con 0x7f231806cb70 2024-09-10T16:29:43.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.512+0000 7f230dffb640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 0x7f22fc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.513+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(807..807 src has 677..807) ==== 4815+0+0 (secure 0 0 0) 0x7f2314072ae0 con 0x7f231806cb70 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.513+0000 7f231ca30640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 0x7f22fc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.513+0000 7f231ca30640 10 monclient: get_auth_request con 0x7f22fc03e210 auth_method 0 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.515+0000 7f231ca30640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 0x7f22fc0406d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2308007d50 tx=0x7f2308007ca0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.517+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f231404a020 con 0x7f231806cb70 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.517+0000 7f230dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:43.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.517+0000 7f230dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3593: finishing write tid 1 to smithi16534841-15 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3597: read oid 27 snap -1 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3597: expect deleted 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3591: done (6 left) 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3598: snap_create 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3593: finishing write tid 2 to smithi16534841-15 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:3592: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:29:43.549 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 180923 2024-09-10T16:29:43.586 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:43.664 INFO:tasks.rados.rados.0.smithi165.stdout:3594: expect (ObjNum 1163 snap 365 seq_num 1163) 2024-09-10T16:29:43.723 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:43.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.758+0000 7f231f495640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:29:43.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.758+0000 7f231f495640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:29:43.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.758+0000 7f231f495640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:43.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.758+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2318134f70 con 0x7f231806cb70 2024-09-10T16:29:43.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f230dffb640 1 -- 172.21.15.164:0/565411083 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2314041020 con 0x7f231806cb70 2024-09-10T16:29:43.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f230dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:29:43.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f230dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:43.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.761 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=2661826565,config=2586702516,health=885635461,kv=1176791497,logm=427978343}) 2024-09-10T16:29:43.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=330405600,config=2586702516,health=885635461,kv=3293307581,logm=544245675}) 2024-09-10T16:29:43.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=2,health=12,kv=11,logm=48} crc {auth=2661826565,config=2586702516,health=885635461,kv=1176791497,logm=427978343}) 2024-09-10T16:29:43.762 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.759+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=27,config=2,health=11,kv=12,logm=48} crc {auth=3709592353,config=2586702516,health=3650641517,kv=805391194,logm=2312641130}) 2024-09-10T16:29:43.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267566004}) 2024-09-10T16:29:43.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627915779}) 2024-09-10T16:29:43.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.763 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267566004}) 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.761+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4014632897}) 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212149281}) 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929947559}) 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1212149281}) 2024-09-10T16:29:43.764 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.762+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=215624309}) 2024-09-10T16:29:43.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 msgr2=0x7f22fc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 0x7f22fc0406d0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7f2308007d50 tx=0x7f2308007ca0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 10 monclient: shutdown 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 msgr2=0x7f231813f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f2314031000 tx=0x7f23140046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 shutdown_connections 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f22fc03e210 0x7f22fc0406d0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f23180a4ab0 0x7f23181cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f23180a46a0 0x7f231813f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 --2- 172.21.15.164:0/565411083 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f231806cb70 0x7f231813f120 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 >> 172.21.15.164:0/565411083 conn(0x7f23180aa900 msgr2=0x7f23180a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 shutdown_connections 2024-09-10T16:29:43.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:43.763+0000 7f231f495640 1 -- 172.21.15.164:0/565411083 wait complete. 2024-09-10T16:29:43.767 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715083328}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3289764753}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2715083328}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.764+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1484875008}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300644194}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1253272740}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1300644194}) 2024-09-10T16:29:43.768 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.766+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=688955834}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301227202}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1284885087}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2301227202}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.767+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=968505572}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512041830}) 2024-09-10T16:29:43.770 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3964479293}) 2024-09-10T16:29:43.771 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.771 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=512041830}) 2024-09-10T16:29:43.771 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.768+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=763645244}) 2024-09-10T16:29:43.771 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664830578}) 2024-09-10T16:29:43.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4225591616}) 2024-09-10T16:29:43.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=664830578}) 2024-09-10T16:29:43.772 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.769+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1081133279}) 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631865601}) 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3456838712}) 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3631865601}) 2024-09-10T16:29:43.773 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.771+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720622735}) 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1130385991}) 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1599876724}) 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1130385991}) 2024-09-10T16:29:43.774 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.773+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2255211823}) 2024-09-10T16:29:43.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.774+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.774+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393481577}) 2024-09-10T16:29:43.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.774+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=181163790}) 2024-09-10T16:29:43.776 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.774+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.774+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2393481577}) 2024-09-10T16:29:43.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.775+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=685143326}) 2024-09-10T16:29:43.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.777 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135847981}) 2024-09-10T16:29:43.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3435279284}) 2024-09-10T16:29:43.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3135847981}) 2024-09-10T16:29:43.778 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.776+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2385041286}) 2024-09-10T16:29:43.779 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:29:43.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508662721}) 2024-09-10T16:29:43.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2077475413}) 2024-09-10T16:29:43.801 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3508662721}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=824953938}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=968586086}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1273796769}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=968586086}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.777+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1092377931}) 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.802 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965052486}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3067432909}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3965052486}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815660572}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211757516}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=583651849}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2211757516}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435983018}) 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.803 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797123842}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1285797513}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=797123842}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.779+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1953932367}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342204421}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2103951714}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=342204421}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2201397595}) 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.804 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252923108}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2050745510}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.780+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=252923108}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2179624604}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105623048}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2669302536}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1105623048}) 2024-09-10T16:29:43.805 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.781+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2823571451}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865973604}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2249537877}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865973604}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=783565125}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382445889}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2042280962}) 2024-09-10T16:29:43.806 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1382445889}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.782+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4174471980}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074283424}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1665750527}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4074283424}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.783+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=87165623}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821341649}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075410766}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3821341649}) 2024-09-10T16:29:43.807 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2548989200}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684430704}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3218740532}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=684430704}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.784+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1863544876}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838847108}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2399760949}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3838847108}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.785+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3335194339}) 2024-09-10T16:29:43.808 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824971240}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3114754874}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3824971240}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2132118633}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333073247}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4241735036}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2333073247}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.786+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2154075539}) 2024-09-10T16:29:43.809 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541095576}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2824315748}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=541095576}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.787+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036783490}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352861004}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2353280986}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3352861004}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3787401928}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422581612}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4258013258}) 2024-09-10T16:29:43.810 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422581612}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.788+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853517854}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004502399}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3934747181}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004502399}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3814916303}) 2024-09-10T16:29:43.811 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431273063}) 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2833469584}) 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1431273063}) 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.789+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3339932000}) 2024-09-10T16:29:43.812 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.813 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850956502}) 2024-09-10T16:29:43.813 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2221469644}) 2024-09-10T16:29:43.813 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.813 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1850956502}) 2024-09-10T16:29:43.813 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.790+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4004139389}) 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3854812181}) 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3013433209}) 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3854812181}) 2024-09-10T16:29:43.814 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094322526}) 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336918315}) 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2124102422}) 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2336918315}) 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.791+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1244856575}) 2024-09-10T16:29:43.815 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116048060}) 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1311855340}) 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116048060}) 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.792+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2131045497}) 2024-09-10T16:29:43.816 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6155580}) 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1481101376}) 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=6155580}) 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1706756193}) 2024-09-10T16:29:43.817 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236188458}) 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793099773}) 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236188458}) 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.793+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285237917}) 2024-09-10T16:29:43.818 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338840611}) 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3184346373}) 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1338840611}) 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.794+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2710642919}) 2024-09-10T16:29:43.819 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492635690}) 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=645477522}) 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1492635690}) 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1033427841}) 2024-09-10T16:29:43.820 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.821 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930436074}) 2024-09-10T16:29:43.821 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=176495855}) 2024-09-10T16:29:43.821 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.821 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=930436074}) 2024-09-10T16:29:43.821 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.795+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3772926097}) 2024-09-10T16:29:43.822 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.822 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624684475}) 2024-09-10T16:29:43.822 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828354628}) 2024-09-10T16:29:43.822 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.822 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2624684475}) 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4197601472}) 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722108497}) 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3109935420}) 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.823 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3722108497}) 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.796+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3924833575}) 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355503383}) 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=812790190}) 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.824 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2355503383}) 2024-09-10T16:29:43.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.797+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178985409}) 2024-09-10T16:29:43.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372430943}) 2024-09-10T16:29:43.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1216629099}) 2024-09-10T16:29:43.825 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2372430943}) 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955069327}) 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349770698}) 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2014400280}) 2024-09-10T16:29:43.826 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=349770698}) 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.798+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2547069061}) 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137899675}) 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2367689035}) 2024-09-10T16:29:43.827 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=137899675}) 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.799+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3218575531}) 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324823128}) 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=355510366}) 2024-09-10T16:29:43.828 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324823128}) 2024-09-10T16:29:43.829 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814470478}) 2024-09-10T16:29:43.846 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.846 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049689254}) 2024-09-10T16:29:43.846 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3691939219}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049689254}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.800+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2692684163}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387668647}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1165841612}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=387668647}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.801+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2169639594}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422724055}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=281450881}) 2024-09-10T16:29:43.847 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=422724055}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3704337211}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903081309}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=10224581}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=903081309}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.802+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3932280485}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899453287}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236756103}) 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.848 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3899453287}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1330448274}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.803+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610594553}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3529430085}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1610594553}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4016855869}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142841614}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2148731621}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=142841614}) 2024-09-10T16:29:43.849 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.804+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1617019158}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2500068248}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3478166774}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2500068248}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157999613}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965111380}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=911072753}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2965111380}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.805+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3582961825}) 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.850 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464886428}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1829466189}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3464886428}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.806+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3557643409}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973978483}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766788796}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2973978483}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=982684320}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476074825}) 2024-09-10T16:29:43.851 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966339085}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1476074825}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.807+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=240181097}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596901851}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133668498}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=596901851}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.808+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3876229858}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291954696}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678251089}) 2024-09-10T16:29:43.852 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2291954696}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1022993067}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.809+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100213685}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4283141298}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1100213685}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3042412446}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000528134}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4076046072}) 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.853 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2000528134}) 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.810+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=727656313}) 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122399259}) 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2772197743}) 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3122399259}) 2024-09-10T16:29:43.854 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.811+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1069414384}) 2024-09-10T16:29:43.854 INFO:tasks.rados.rados.0.smithi165.stdout:3593: finishing write tid 3 to smithi16534841-15 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: finishing write tid 4 to smithi16534841-15 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: finishing write tid 5 to smithi16534841-15 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: oid 15 updating version 0 to 1855 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: oid 15 version 1855 is already newer than 1853 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: oid 15 version 1855 is already newer than 1852 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: oid 15 version 1855 is already newer than 1854 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1855 (ObjNum 1243 snap 402 seq_num 1243) dirty exists 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3593: left oid 15 (ObjNum 1243 snap 402 seq_num 1243) 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3595: finishing write tid 1 to smithi16534841-11 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3595: finishing write tid 2 to smithi16534841-11 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3595: finishing write tid 3 to smithi16534841-11 2024-09-10T16:29:43.855 INFO:tasks.rados.rados.0.smithi165.stdout:3595: finishing write tid 4 to smithi16534841-11 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3595: oid 11 updating version 0 to 1887 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3595: oid 11 version 1887 is already newer than 1885 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3595: oid 11 version 1887 is already newer than 1886 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3595: left oid 11 (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: finishing write tid 1 to smithi16534841-1 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: finishing write tid 2 to smithi16534841-1 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: finishing write tid 3 to smithi16534841-1 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: finishing write tid 4 to smithi16534841-1 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: oid 1 updating version 0 to 2024 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: oid 1 version 2024 is already newer than 2023 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: oid 1 version 2024 is already newer than 2022 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2024 (ObjNum 1245 snap 402 seq_num 1245) dirty exists 2024-09-10T16:29:43.856 INFO:tasks.rados.rados.0.smithi165.stdout:3596: left oid 1 (ObjNum 1245 snap 402 seq_num 1245) 2024-09-10T16:29:43.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062077549}) 2024-09-10T16:29:43.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794983001}) 2024-09-10T16:29:43.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062077549}) 2024-09-10T16:29:43.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.871+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=607879887}) 2024-09-10T16:29:43.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432248785}) 2024-09-10T16:29:43.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=553345171}) 2024-09-10T16:29:43.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3432248785}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1645235930}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062749672}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1280746921}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1062749672}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.872+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1573943532}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731582664}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3389304770}) 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2731582664}) 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.873+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2309238625}) 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170491739}) 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2957833571}) 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3170491739}) 2024-09-10T16:29:43.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881161971}) 2024-09-10T16:29:43.878 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.878 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498844452}) 2024-09-10T16:29:43.878 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.874+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3906248941}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=498844452}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3114870188}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166681031}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839851398}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1166681031}) 2024-09-10T16:29:43.879 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:43.875+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881145126}) 2024-09-10T16:29:44.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949173392}) 2024-09-10T16:29:44.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3807697791}) 2024-09-10T16:29:44.087 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949173392}) 2024-09-10T16:29:44.088 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.086+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3075284608}) 2024-09-10T16:29:44.232 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447869591}) 2024-09-10T16:29:44.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3074132087}) 2024-09-10T16:29:44.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2447869591}) 2024-09-10T16:29:44.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=380758040}) 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425138527}) 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1827333126}) 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425138527}) 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=315037722}) 2024-09-10T16:29:44.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511892771}) 2024-09-10T16:29:44.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3749591553}) 2024-09-10T16:29:44.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3511892771}) 2024-09-10T16:29:44.235 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.232+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3678908513}) 2024-09-10T16:29:44.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269558429}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3429273842}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1269558429}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.233+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435389428}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907503467}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4137072139}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2907503467}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.234+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1389329702}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566840333}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3944321980}) 2024-09-10T16:29:44.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2566840333}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135534793}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575220253}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2763413859}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575220253}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.235+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=761387095}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962168628}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2663596163}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1962168628}) 2024-09-10T16:29:44.239 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274271480}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730008655}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=322725640}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2730008655}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1167945496}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498669939}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=287334401}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2498669939}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.237+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2024417541}) 2024-09-10T16:29:44.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372382839}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3522419285}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1372382839}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.238+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2264558779}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656241772}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1670911972}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=656241772}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.239+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2541953793}) 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014517988}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190506132}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4014517988}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4021584181}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158079200}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1830194930}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4158079200}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.240+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1545584375}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824561708}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3170469348}) 2024-09-10T16:29:44.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=824561708}) 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.241+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3296617612}) 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504984649}) 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2728970285}) 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504984649}) 2024-09-10T16:29:44.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.242+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017615003}) 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583265976}) 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2137349388}) 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1583265976}) 2024-09-10T16:29:44.251 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1963471892}) 2024-09-10T16:29:44.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3570247663}) 2024-09-10T16:29:44.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2828700612}) 2024-09-10T16:29:44.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3570247663}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.250+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3016542383}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1897119880}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2195097430}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1897119880}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.251+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3422191496}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071146261}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2978679411}) 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3071146261}) 2024-09-10T16:29:44.255 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.252+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2921424372}) 2024-09-10T16:29:44.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=11} crc {logm=2510495598,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1511762882,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_snap=3934746537}) 2024-09-10T16:29:44.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=2021116115,mdsmap=931935893,mgr=2441926604,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1437047297,mon_config_key=2124418738,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=684247286,osd_snap=3695905822}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=37,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_snap=11} crc {logm=2510495598,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1511762882,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_snap=3934746537}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=39,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=6,osd_snap=13} crc {logm=3952323246,mdsmap=931935893,mgr=639680456,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2028837457,mon_config_key=3798575338,monmap=765578399,nvmeofgw=598337686,osd_metadata=4005186995,osd_snap=2374498728}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4166834851}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=3960792306}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=4166834851}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.253+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=3633534812}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=48,osdmap=52} crc {osd_snap=1526698714,osdmap=3052998488}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=98530339,osdmap=1889180635}) 2024-09-10T16:29:44.257 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=48,osdmap=52} crc {osd_snap=1526698714,osdmap=3052998488}) 2024-09-10T16:29:44.258 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.254+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=49,osdmap=51} crc {osd_snap=98530339,osdmap=2979519319}) 2024-09-10T16:29:44.352 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.352 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=794181456}) 2024-09-10T16:29:44.353 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1960233444}) 2024-09-10T16:29:44.353 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.353 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=794181456}) 2024-09-10T16:29:44.353 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=936709803}) 2024-09-10T16:29:44.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=1564804527}) 2024-09-10T16:29:44.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=1998814283}) 2024-09-10T16:29:44.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:29:44.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=51} crc {osdmap=1564804527}) 2024-09-10T16:29:44.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:29:44.406+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=2921593742}) 2024-09-10T16:29:44.456 INFO:tasks.rados.rados.0.smithi165.stdout:3592: done (6 left) 2024-09-10T16:29:44.456 INFO:tasks.rados.rados.0.smithi165.stdout:3593: done (5 left) 2024-09-10T16:29:44.456 INFO:tasks.rados.rados.0.smithi165.stdout:3594: done (4 left) 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3595: done (3 left) 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3596: done (2 left) 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3597: done (1 left) 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3598: done (0 left) 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3599: write (excl) oid 16 current snap is 403 2024-09-10T16:29:44.457 INFO:tasks.rados.rados.0.smithi165.stdout:3599: seq_num 1246 ranges {541522=725793,1946446=635431} 2024-09-10T16:29:44.463 INFO:tasks.rados.rados.0.smithi165.stdout:3599: writing smithi16534841-16 from 541522 to 1267315 tid 1 2024-09-10T16:29:44.471 INFO:tasks.rados.rados.0.smithi165.stdout:3599: writing smithi16534841-16 from 1946446 to 2581877 tid 2 2024-09-10T16:29:44.471 INFO:tasks.rados.rados.0.smithi165.stdout:3600: read oid 32 snap -1 2024-09-10T16:29:44.471 INFO:tasks.rados.rados.0.smithi165.stdout:3600: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:29:44.471 INFO:tasks.rados.rados.0.smithi165.stdout:3601: write oid 29 current snap is 403 2024-09-10T16:29:44.471 INFO:tasks.rados.rados.0.smithi165.stdout:3601: seq_num 1247 ranges {558329=517395,1721281=779089} 2024-09-10T16:29:44.477 INFO:tasks.rados.rados.0.smithi165.stdout:3601: writing smithi16534841-29 from 558329 to 1075724 tid 1 2024-09-10T16:29:44.486 INFO:tasks.rados.rados.0.smithi165.stdout:3601: writing smithi16534841-29 from 1721281 to 2500370 tid 2 2024-09-10T16:29:44.486 INFO:tasks.rados.rados.0.smithi165.stdout:3602: delete oid 21 current snap is 403 2024-09-10T16:29:44.486 INFO:tasks.rados.rados.0.smithi165.stdout:3600: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:29:44.522 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1437348 2024-09-10T16:29:44.541 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 74012 2024-09-10T16:29:44.571 INFO:tasks.rados.rados.0.smithi165.stdout:3599: finishing write tid 1 to smithi16534841-16 2024-09-10T16:29:44.571 INFO:tasks.rados.rados.0.smithi165.stdout:3599: finishing write tid 2 to smithi16534841-16 2024-09-10T16:29:44.571 INFO:tasks.rados.rados.0.smithi165.stdout:3599: finishing write tid 3 to smithi16534841-16 2024-09-10T16:29:44.571 INFO:tasks.rados.rados.0.smithi165.stdout:3599: finishing write tid 4 to smithi16534841-16 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3599: oid 16 updating version 0 to 2270 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3599: oid 16 updating version 2270 to 2271 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3599: oid 16 updating version 2271 to 2272 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2272 (ObjNum 1246 snap 403 seq_num 1246) dirty exists 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3599: left oid 16 (ObjNum 1246 snap 403 seq_num 1246) 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3601: finishing write tid 1 to smithi16534841-29 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3599: done (3 left) 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3600: done (2 left) 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3602: done (1 left) 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3603: delete oid 22 current snap is 403 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3601: finishing write tid 2 to smithi16534841-29 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3601: finishing write tid 3 to smithi16534841-29 2024-09-10T16:29:44.572 INFO:tasks.rados.rados.0.smithi165.stdout:3601: finishing write tid 4 to smithi16534841-29 2024-09-10T16:29:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:3601: oid 29 updating version 0 to 1939 2024-09-10T16:29:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:3601: oid 29 updating version 1939 to 1941 2024-09-10T16:29:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:3601: oid 29 version 1941 is already newer than 1940 2024-09-10T16:29:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1941 (ObjNum 1247 snap 403 seq_num 1247) dirty exists 2024-09-10T16:29:44.573 INFO:tasks.rados.rados.0.smithi165.stdout:3601: left oid 29 (ObjNum 1247 snap 403 seq_num 1247) 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3601: done (1 left) 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3603: done (0 left) 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3604: copy_from oid 26 from oid 11 current snap is 403 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3605: read oid 40 snap -1 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3605: expect (ObjNum 1224 snap 394 seq_num 1224) 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:3606: rollback oid 29 current snap is 403 2024-09-10T16:29:44.574 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 29 to 365 2024-09-10T16:29:44.575 INFO:tasks.rados.rados.0.smithi165.stdout:3607: delete oid 3 current snap is 403 2024-09-10T16:29:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:3606: finishing rollback tid 0 to smithi16534841-29 2024-09-10T16:29:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:3606: finishing rollback tid 1 to smithi16534841-29 2024-09-10T16:29:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 1943 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:29:44.578 INFO:tasks.rados.rados.0.smithi165.stdout:3606: done (3 left) 2024-09-10T16:29:44.579 INFO:tasks.rados.rados.0.smithi165.stdout:3607: done (2 left) 2024-09-10T16:29:44.579 INFO:tasks.rados.rados.0.smithi165.stdout:3608: snap_create 2024-09-10T16:29:44.579 INFO:tasks.rados.rados.0.smithi165.stdout:3605: expect (ObjNum 1224 snap 394 seq_num 1224) 2024-09-10T16:29:44.579 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:44.608 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:29:44.665 INFO:tasks.rados.rados.0.smithi165.stdout:3604: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:29:44.666 INFO:tasks.rados.rados.0.smithi165.stdout:3604: finishing copy_from to smithi16534841-26 2024-09-10T16:29:44.666 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:29:44.780 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:29:44.780 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.876+0000 7f9fc4b4b640 10 monclient: get_monmap_and_config 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.876+0000 7f9fc4b4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.876+0000 7f9fc4b4b640 10 monclient: monmap: 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:44.877406+0000 2024-09-10T16:29:44.878 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:44.877406+0000 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 Processor -- start 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 -- start start 2024-09-10T16:29:44.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient: init 2024-09-10T16:29:44.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:44.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient: _add_conns ranks=[3,2,0,4,1] 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fc009f930 0x7f9fc00a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc009f930 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc00a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc00a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9fc013a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9fc009fd10 con 0x7f9fc013a5c0 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9fc009cd20 con 0x7f9fc009f930 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc009a0d0 con 0x7f9fc00a2380 2024-09-10T16:29:44.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fc4b4b640 10 monclient(hunting): authenticate will time out at 2163.364783s 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbe59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9fc009f930 0x7f9fc00a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbdd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc00a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9fc013a5c0 0x7f9fc013c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36574/0 (socket says 172.21.15.164:36574) 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbed9d640 1 -- 172.21.15.164:0/2814570459 learned_addr learned my addr 172.21.15.164:0/2814570459 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbe59c640 10 monclient(hunting): get_auth_request con 0x7f9fc009f930 auth_method 0 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbdd9b640 10 monclient(hunting): get_auth_request con 0x7f9fc00a2380 auth_method 0 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:44.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.877+0000 7f9fbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient(hunting): handle_auth_done global_id 254226 payload 274 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 1 -- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 msgr2=0x7f9fc00a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc00a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 1 -- 172.21.15.164:0/2814570459 >> v1:172.21.15.164:6789/0 conn(0x7f9fc013a5c0 legacy=0x7f9fc013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient: _finish_hunting 0 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 1 monclient: found mon.noname-a 2024-09-10T16:29:44.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:44.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:44.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbe59c640 1 -- 172.21.15.164:0/2814570459 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc009e840 con 0x7f9fc009f930 2024-09-10T16:29:44.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.878+0000 7f9fbdd9b640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.880+0000 7f9fbe59c640 1 -- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc009f930 msgr2=0x7f9fc00a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.880+0000 7f9fbe59c640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc009f930 0x7f9fc00a1e40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb802a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d21082472b7b9a4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 1 -- 172.21.15.164:0/2814570459 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8007950 con 0x7f9fc009f930 2024-09-10T16:29:44.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: dump: 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: _finish_auth 0 2024-09-10T16:29:44.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: _check_auth_tickets 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 1 -- 172.21.15.164:0/2814570459 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fb8007af0 con 0x7f9fc009f930 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 5 monclient: authenticate success, global_id 254226 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 1 -- 172.21.15.164:0/2814570459 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fb8031400 con 0x7f9fc009f930 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 4 set_mon_vals no callback set 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fbd59a640 10 monclient: dump: 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:44.892 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 10 monclient: shutdown 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc009f930 msgr2=0x7f9fc00a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:44.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc009f930 0x7f9fc00a1e40 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f9fb8009d90 tx=0x7f9fb802a490 comp rx=0 tx=0).stop 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2814570459 shutdown_connections 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc00a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2814570459 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc009f930 0x7f9fc00a1e40 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2814570459 >> 172.21.15.164:0/2814570459 conn(0x7f9fc00a3110 msgr2=0x7f9fc00a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2814570459 shutdown_connections 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.881+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2814570459 wait complete. 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.882+0000 7f9fc4b4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.882+0000 7f9fc4b4b640 10 monclient: monmap: 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:44.894 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.882+0000 7f9fc4b4b640 1 Processor -- start 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.882+0000 7f9fc4b4b640 1 -- start start 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.882+0000 7f9fc4b4b640 10 monclient: init 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fc009f930 0x7f9fc01cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f9fc009f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:44.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f9fc00a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 0x7f9fc009a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f9fc013a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc009d930 con 0x7f9fc013a5c0 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9fc009ee00 con 0x7f9fc009f930 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9fc009c8b0 con 0x7f9fc00a2380 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fc4b4b640 10 monclient(hunting): authenticate will time out at 2163.370225s 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57402/0 (socket says 172.21.15.164:57402) 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 0x7f9fc009a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 -- 172.21.15.164:0/2628692438 learned_addr learned my addr 172.21.15.164:0/2628692438 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): get_auth_request con 0x7f9fc00a2380 auth_method 0 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): get_auth_request con 0x7f9fc013a5c0 auth_method 0 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:44.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbe59c640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fc009f930 0x7f9fc01cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbe59c640 10 monclient(hunting): get_auth_request con 0x7f9fc009f930 auth_method 0 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbe59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbe59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbe59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient(hunting): handle_auth_done global_id 244313 payload 824 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 -- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fc009f930 msgr2=0x7f9fc01cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fc009f930 0x7f9fc01cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 -- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 msgr2=0x7f9fc009a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:44.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 0x7f9fc009a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 monclient: found mon.noname-c 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbdd9b640 1 -- 172.21.15.164:0/2628692438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9fc006e840 con 0x7f9fc00a2380 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.883+0000 7f9fbed9d640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 0x7f9fc009a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fbdd9b640 1 -- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 msgr2=0x7f9fc01cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fbdd9b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800ec10 tx=0x7f9fa800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8017070 con 0x7f9fc00a2380 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fb6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.885+0000 7f9fb6ffd640 10 monclient: dump: 2024-09-10T16:29:44.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:44.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9fa800ce40 con 0x7f9fc00a2380 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 5 monclient: authenticate success, global_id 244313 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 10 monclient: _renew_subs 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9fa8013690 con 0x7f9fc00a2380 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9fc00a18a0 con 0x7f9fc00a2380 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 10 monclient: dump: 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:44.901 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fbcd99640 4 set_mon_vals no callback set 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 10 monclient: _renew_subs 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9fc0131b40 con 0x7f9fc00a2380 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.886+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9fa801a070 con 0x7f9fc00a2380 2024-09-10T16:29:44.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fc4b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fc4b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fc4b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9fc00a16c0 con 0x7f9fc00a2380 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fb6ffd640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 0x7f9fa00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(808..808 src has 677..808) ==== 4787+0+0 (secure 0 0 0) 0x7f9fa8052ce0 con 0x7f9fc00a2380 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fbe59c640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 0x7f9fa00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.887+0000 7f9fbe59c640 10 monclient: get_auth_request con 0x7f9fa003e210 auth_method 0 2024-09-10T16:29:44.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.889+0000 7f9fbe59c640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 0x7f9fa00406d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00a2de0 tx=0x7f9fb802f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:44.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.891+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9fc00a16c0 con 0x7f9fc00a2380 2024-09-10T16:29:44.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.891+0000 7f9fb6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:44.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:44.891+0000 7f9fb6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:45.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.132+0000 7f9fc4b4b640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:45.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fc4b4b640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fc4b4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9fc0077260 con 0x7f9fc00a2380 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fb6ffd640 1 -- 172.21.15.164:0/2628692438 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f9fc006e840 con 0x7f9fc00a2380 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fb6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.133+0000 7f9fb6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:45.135 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 msgr2=0x7f9fa00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 0x7f9fa00406d0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7f9fc00a2de0 tx=0x7f9fb802f000 comp rx=0 tx=0).stop 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 10 monclient: shutdown 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 msgr2=0x7f9fc01cdf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9fa800ec10 tx=0x7f9fa800c520 comp rx=0 tx=0).stop 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 shutdown_connections 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9fa003e210 0x7f9fa00406d0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9fc013a5c0 0x7f9fc009a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9fc00a2380 0x7f9fc01cdf40 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 --2- 172.21.15.164:0/2628692438 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9fc009f930 0x7f9fc01cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 >> 172.21.15.164:0/2628692438 conn(0x7f9fc00a3110 msgr2=0x7f9fc0138bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 shutdown_connections 2024-09-10T16:29:45.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.135+0000 7f9fc4b4b640 1 -- 172.21.15.164:0/2628692438 wait complete. 2024-09-10T16:29:45.147 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:29:45.147 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:29:45.147 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:29:45.147 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:29:45.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 10 monclient: get_monmap_and_config 2024-09-10T16:29:45.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 10 monclient: monmap: 2024-09-10T16:29:45.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:45.245959+0000 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:45.245959+0000 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:45.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 1 Processor -- start 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 1 -- start start 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.245+0000 7f5fa8c08640 10 monclient: init 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient: _add_conns ranks=[3,0,2,1,4] 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 0x7f5fa4134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-a con 0x7f5fa406cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5fa40a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 0x7f5fa41383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa40a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fa409ce50 con 0x7f5fa40a46a0 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5fa406db60 con 0x7f5fa406cb70 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa406cf50 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa8c08640 10 monclient(hunting): authenticate will time out at 2163.733426s 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 0x7f5fa4134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa1d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f5fa40a46a0 0x7f5fa4137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36584/0 (socket says 172.21.15.164:36584) 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 0x7f5fa41383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa1d9b640 1 -- 172.21.15.164:0/684811972 learned_addr learned my addr 172.21.15.164:0/684811972 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): get_auth_request con 0x7f5fa406cb70 auth_method 0 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): get_auth_request con 0x7f5fa40a4ab0 auth_method 0 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_done global_id 244316 payload 274 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/684811972 >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 msgr2=0x7f5fa4134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 --2- 172.21.15.164:0/684811972 >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 0x7f5fa4134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/684811972 >> v1:172.21.15.164:6789/0 conn(0x7f5fa40a46a0 legacy=0x7f5fa4137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 monclient: found mon.noname-b 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/684811972 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa409cb70 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.246+0000 7f5fa259c640 1 --2- 172.21.15.164:0/684811972 >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 0x7f5fa4134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:45.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/684811972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 msgr2=0x7f5fa41383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa2d9d640 1 --2- 172.21.15.164:0/684811972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 0x7f5fa41383b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097d0 tx=0x7f5f9c02a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=263eca49afaac046 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 1 -- 172.21.15.164:0/684811972 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c037070 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: dump: 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 1 -- 172.21.15.164:0/684811972 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9c032440 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa8c08640 5 monclient: authenticate success, global_id 244316 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa8c08640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa8c08640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa8c08640 4 set_mon_vals no callback set 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 1 -- 172.21.15.164:0/684811972 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c032810 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.249+0000 7f5fa159a640 10 monclient: dump: 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 10 monclient: shutdown 2024-09-10T16:29:45.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 -- 172.21.15.164:0/684811972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 msgr2=0x7f5fa41383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/684811972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 0x7f5fa41383b0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0097d0 tx=0x7f5f9c02a000 comp rx=0 tx=0).stop 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 -- 172.21.15.164:0/684811972 shutdown_connections 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/684811972 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a4ab0 0x7f5fa41383b0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/684811972 >> v2:172.21.15.164:3300/0 conn(0x7f5fa406cb70 0x7f5fa4134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 -- 172.21.15.164:0/684811972 >> 172.21.15.164:0/684811972 conn(0x7f5fa40aa900 msgr2=0x7f5fa40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 -- 172.21.15.164:0/684811972 shutdown_connections 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.250+0000 7f5fa8c08640 1 -- 172.21.15.164:0/684811972 wait complete. 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient: monmap: 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:45.265 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 Processor -- start 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 -- start start 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient: init 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fa406cb70 0x7f5fa412e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-b con 0x7f5fa406cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 0x7f5fa412e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-c con 0x7f5fa40a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient(hunting): picked mon.noname-a con 0x7f5fa40a4ab0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.251+0000 7f5fa8c08640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa4138a70 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41388f0 con 0x7f5fa406cb70 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5fa4140040 con 0x7f5fa40a46a0 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa8c08640 10 monclient(hunting): authenticate will time out at 2163.739019s 2024-09-10T16:29:45.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 0x7f5fa412e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46304/0 (socket says 172.21.15.164:46304) 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 0x7f5fa412e940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57420/0 (socket says 172.21.15.164:57420) 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 learned_addr learned my addr 172.21.15.164:0/2555554626 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): get_auth_request con 0x7f5fa40a4ab0 auth_method 0 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): get_auth_request con 0x7f5fa40a46a0 auth_method 0 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fa406cb70 0x7f5fa412e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): get_auth_request con 0x7f5fa406cb70 auth_method 0 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient(hunting): handle_auth_done global_id 254235 payload 824 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 msgr2=0x7f5fa412e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 0x7f5fa412e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fa406cb70 msgr2=0x7f5fa412e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5fa406cb70 0x7f5fa412e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 monclient: found mon.noname-a 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fa41344a0 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.252+0000 7f5fa1d9b640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5fa40a46a0 0x7f5fa412e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 msgr2=0x7f5fa41cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa2d9d640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098f0 tx=0x7f5f9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c037040 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: dump: 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.271 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 5 monclient: authenticate success, global_id 254235 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f9c042070 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 10 monclient: _renew_subs 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fa4133f50 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c03bb00 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5f92ffd640 10 monclient: dump: 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa0d99640 4 set_mon_vals no callback set 2024-09-10T16:29:45.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 10 monclient: _renew_subs 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.254+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fa40a2b70 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa8c08640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 0x7f5fa41cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa41cdf20 con 0x7f5fa41cd600 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 0x7f5fa41cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient: get_auth_request con 0x7f5fa41cd600 auth_method 0 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5f9c03bca0 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 10 monclient(hunting): handle_auth_done global_id 254238 payload 824 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5fa259c640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fa4134f70 con 0x7f5fa41cd600 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.255+0000 7f5f92ffd640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f8403e210 0x7f5f840406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.256+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(808..808 src has 677..808) ==== 4787+0+0 (secure 0 0 0) 0x7f5f9c073510 con 0x7f5fa40a4ab0 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.256+0000 7f5fa1d9b640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f8403e210 0x7f5f840406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.256+0000 7f5fa1d9b640 10 monclient: get_auth_request con 0x7f5f8403e210 auth_method 0 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5fa259c640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 msgr2=0x7f5fa41cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5fa259c640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 0x7f5fa41cd9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f940099d0 tx=0x7f5f9400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f94019070 con 0x7f5fa41cd600 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 10 monclient: dump: 2024-09-10T16:29:45.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5f94003da0 con 0x7f5fa41cd600 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:45.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 msgr2=0x7f5fa41cd9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 0x7f5fa41cd9e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f940099d0 tx=0x7f5f9400f580 comp rx=0 tx=0).stop 2024-09-10T16:29:45.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 msgr2=0x7f5fa41cd9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5f92ffd640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa41cd600 0x7f5fa41cd9e0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f5f940099d0 tx=0x7f5f9400f580 comp rx=0 tx=0).stop 2024-09-10T16:29:45.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.258+0000 7f5fa1d9b640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f8403e210 0x7f5f840406d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40a2170 tx=0x7f5f980098a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa8c08640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 0x7f5fa4130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa8c08640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5fa40a3370 con 0x7f5fa4130250 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 0x7f5fa4130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient: get_auth_request con 0x7f5fa4130250 auth_method 0 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.264+0000 7f5fa259c640 10 monclient(hunting): handle_auth_done global_id 254241 payload 824 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.265+0000 7f5fa259c640 1 -- 172.21.15.164:0/2555554626 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5fa40770f0 con 0x7f5fa4130250 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa259c640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 msgr2=0x7f5fa4130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa259c640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 0x7f5fa4130660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b010 tx=0x7f5f94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f9401eaf0 con 0x7f5fa4130250 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 10 monclient: dump: 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.280 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f5f94015430 con 0x7f5fa4130250 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 msgr2=0x7f5fa4130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 0x7f5fa4130660 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5f9400b010 tx=0x7f5f94004a40 comp rx=0 tx=0).stop 2024-09-10T16:29:45.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 msgr2=0x7f5fa4130660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5f92ffd640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa4130250 0x7f5fa4130660 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 73918, 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:45.282 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 3, 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.283 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.284 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:45.285 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.286 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.287 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.288 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.289 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.290 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.292 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.293 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.294 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.295 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:45.295 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:45.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f8403e210 msgr2=0x7f5f840406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5f8403e210 0x7f5f840406d0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7f5fa40a2170 tx=0x7f5f980098a0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 10 monclient: shutdown 2024-09-10T16:29:45.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 msgr2=0x7f5fa41cd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c0098f0 tx=0x7f5f9c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa2d9d640 1 -- 172.21.15.164:0/2555554626 reap_dead start 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 shutdown_connections 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 --2- 172.21.15.164:0/2555554626 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5fa40a4ab0 0x7f5fa41cd0c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 >> 172.21.15.164:0/2555554626 conn(0x7f5fa40aa900 msgr2=0x7f5fa40a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.267+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 shutdown_connections 2024-09-10T16:29:45.297 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.268+0000 7f5fa8c08640 1 -- 172.21.15.164:0/2555554626 wait complete. 2024-09-10T16:29:45.297 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:45.297 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:29:45.298 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.394+0000 7f1782d4b640 10 monclient: get_monmap_and_config 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.394+0000 7f1782d4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.394+0000 7f1782d4b640 10 monclient: monmap: 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:45.395576+0000 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:45.395576+0000 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.396 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:45.397 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 Processor -- start 2024-09-10T16:29:45.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 -- start start 2024-09-10T16:29:45.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient: init 2024-09-10T16:29:45.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.399 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f177c09c310 0x7f177c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f177c09c310 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f177c09d190 0x7f177c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f177c09d190 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f177c09f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f177c09fbc0 con 0x7f177c09f4c0 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f177c0a2a20 con 0x7f177c09c310 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f177c09c6f0 con 0x7f177c09d190 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.400 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.395+0000 7f1782d4b640 10 monclient(hunting): authenticate will time out at 2163.882982s 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f177c09c310 0x7f177c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f17812e8640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f177c09f4c0 0x7f177c13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36592/0 (socket says 172.21.15.164:36592) 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f177c09c310 0x7f177c09ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46330/0 (socket says 172.21.15.164:46330) 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f17812e8640 1 -- 172.21.15.164:0/3084809527 learned_addr learned my addr 172.21.15.164:0/3084809527 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): get_auth_request con 0x7f177c09c310 auth_method 0 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f177bfff640 1 --2- 172.21.15.164:0/3084809527 >> v2:172.21.15.165:3300/0 conn(0x7f177c09d190 0x7f177c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.401 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f177bfff640 10 monclient(hunting): get_auth_request con 0x7f177c09d190 auth_method 0 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f177bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f177bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f177bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient(hunting): handle_auth_done global_id 254244 payload 274 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 -- 172.21.15.164:0/3084809527 >> v2:172.21.15.165:3300/0 conn(0x7f177c09d190 msgr2=0x7f177c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 --2- 172.21.15.164:0/3084809527 >> v2:172.21.15.165:3300/0 conn(0x7f177c09d190 0x7f177c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 -- 172.21.15.164:0/3084809527 >> v1:172.21.15.164:6789/0 conn(0x7f177c09f4c0 legacy=0x7f177c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 monclient: found mon.noname-a 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.402 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.396+0000 7f1780ae7640 1 -- 172.21.15.164:0/3084809527 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c09a4d0 con 0x7f177c09c310 2024-09-10T16:29:45.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1780ae7640 1 -- 172.21.15.164:0/3084809527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c09c310 msgr2=0x7f177c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1780ae7640 1 --2- 172.21.15.164:0/3084809527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c09c310 0x7f177c09ea40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f177c0a0280 tx=0x7f177002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=157532112c7fc539 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 1 -- 172.21.15.164:0/3084809527 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037070 con 0x7f177c09c310 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: dump: 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.410 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 1 -- 172.21.15.164:0/3084809527 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1770032440 con 0x7f177c09c310 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 5 monclient: authenticate success, global_id 254244 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 1 -- 172.21.15.164:0/3084809527 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770032810 con 0x7f177c09c310 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 4 set_mon_vals no callback set 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f177b7fe640 10 monclient: dump: 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 10 monclient: shutdown 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 -- 172.21.15.164:0/3084809527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c09c310 msgr2=0x7f177c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 --2- 172.21.15.164:0/3084809527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c09c310 0x7f177c09ea40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f177c0a0280 tx=0x7f177002a000 comp rx=0 tx=0).stop 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 -- 172.21.15.164:0/3084809527 shutdown_connections 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 --2- 172.21.15.164:0/3084809527 >> v2:172.21.15.165:3300/0 conn(0x7f177c09d190 0x7f177c09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 --2- 172.21.15.164:0/3084809527 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c09c310 0x7f177c09ea40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 -- 172.21.15.164:0/3084809527 >> 172.21.15.164:0/3084809527 conn(0x7f177c0a3110 msgr2=0x7f177c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 -- 172.21.15.164:0/3084809527 shutdown_connections 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.399+0000 7f1782d4b640 1 -- 172.21.15.164:0/3084809527 wait complete. 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.400+0000 7f1782d4b640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.400+0000 7f1782d4b640 10 monclient: monmap: 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.413 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.400+0000 7f1782d4b640 1 Processor -- start 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.400+0000 7f1782d4b640 1 -- start start 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.400+0000 7f1782d4b640 10 monclient: init 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:45.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-c con 0x7f177c09c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177c09f4c0 0x7f177c092690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-b con 0x7f177c09f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c092bd0 0x7f177c093040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): picked mon.noname-a con 0x7f177c092bd0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f177c0a2230 con 0x7f177c092bd0 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f177c0a20b0 con 0x7f177c09f4c0 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f177c074e00 con 0x7f177c09c310 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1782d4b640 10 monclient(hunting): authenticate will time out at 2163.888488s 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c092bd0 0x7f177c093040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57424/0 (socket says 172.21.15.164:57424) 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 -- 172.21.15.164:0/6635372 learned_addr learned my addr 172.21.15.164:0/6635372 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): get_auth_request con 0x7f177c09c310 auth_method 0 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): get_auth_request con 0x7f177c092bd0 auth_method 0 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177c09f4c0 0x7f177c092690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 10 monclient(hunting): get_auth_request con 0x7f177c09f4c0 auth_method 0 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient(hunting): handle_auth_done global_id 244322 payload 824 2024-09-10T16:29:45.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177c09f4c0 msgr2=0x7f177c092690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177c09f4c0 0x7f177c092690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c092bd0 msgr2=0x7f177c093040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c092bd0 0x7f177c093040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 monclient: found mon.noname-c 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f1780ae7640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f177c0964b0 con 0x7f177c09c310 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f177c09f4c0 0x7f177c092690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.401+0000 7f17812e8640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f177c092bd0 0x7f177c093040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1780ae7640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 msgr2=0x7f177c094cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1780ae7640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1770000c00 tx=0x7f17700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770041070 con 0x7f177c09c310 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: dump: 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f177003c890 con 0x7f177c09c310 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 5 monclient: authenticate success, global_id 244322 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037040 con 0x7f177c09c310 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f17797fa640 10 monclient: dump: 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 10 monclient: _renew_subs 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177c099eb0 con 0x7f177c09c310 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f177affd640 4 set_mon_vals no callback set 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 10 monclient: _renew_subs 2024-09-10T16:29:45.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.404+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f177c09d620 con 0x7f177c09c310 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f1782d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f1782d4b640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 0x7f177c1cb3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f177c09ea40 con 0x7f177c09d190 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1770044070 con 0x7f177c09c310 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 0x7f177c1cb3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient: get_auth_request con 0x7f177c09d190 auth_method 0 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f17797fa640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f176003e210 0x7f17600406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 10 monclient(hunting): handle_auth_done global_id 244325 payload 824 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f177bfff640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f177c095640 con 0x7f177c09d190 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(808..808 src has 677..808) ==== 4787+0+0 (secure 0 0 0) 0x7f1770073450 con 0x7f177c09c310 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f17812e8640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f176003e210 0x7f17600406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.405+0000 7f17812e8640 10 monclient: get_auth_request con 0x7f176003e210 auth_method 0 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.407+0000 7f177bfff640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 msgr2=0x7f177c1cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.407+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 0x7f177c1cb3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f177c094c40 tx=0x7f176800f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768004230 con 0x7f177c09d190 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 10 monclient: dump: 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.423 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1768004230 con 0x7f177c09d190 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 msgr2=0x7f177c1cb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 0x7f177c1cb3b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f177c094c40 tx=0x7f176800f9e0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 msgr2=0x7f177c1cb3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17797fa640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09d190 0x7f177c1cb3b0 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f177c094c40 tx=0x7f176800f9e0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.408+0000 7f17812e8640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f176003e210 0x7f17600406d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1774005c00 tx=0x7f1774009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f1782d4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f1782d4b640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 0x7f177c1cbcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f1782d4b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f177c1cc210 con 0x7f177c1cb8f0 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 0x7f177c1cbcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient: get_auth_request con 0x7f177c1cb8f0 auth_method 0 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 10 monclient(hunting): handle_auth_done global_id 244328 payload 824 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.414+0000 7f177bfff640 1 -- 172.21.15.164:0/6635372 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f177c094150 con 0x7f177c1cb8f0 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f177bfff640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 msgr2=0x7f177c1cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f177bfff640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 0x7f177c1cbcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1768019000 tx=0x7f17680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1768003810 con 0x7f177c1cb8f0 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f17797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f17797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.416+0000 7f17797fa640 10 monclient: dump: 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.428 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f1768003810 con 0x7f177c1cb8f0 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 msgr2=0x7f177c1cbcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 0x7f177c1cbcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1768019000 tx=0x7f17680046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 msgr2=0x7f177c1cbcd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17797fa640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c1cb8f0 0x7f177c1cbcd0 secure :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1768019000 tx=0x7f17680046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.429 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 122324, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:29:45.430 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.431 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.432 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:45.433 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.434 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.435 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.436 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.437 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.438 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.439 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.440 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.441 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:45.442 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f176003e210 msgr2=0x7f17600406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f176003e210 0x7f17600406d0 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f1774005c00 tx=0x7f1774009040 comp rx=0 tx=0).stop 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 10 monclient: shutdown 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 msgr2=0x7f177c094cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1770000c00 tx=0x7f17700046c0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f17812e8640 1 -- 172.21.15.164:0/6635372 reap_dead start 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 shutdown_connections 2024-09-10T16:29:45.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 --2- 172.21.15.164:0/6635372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f177c09c310 0x7f177c094cb0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 >> 172.21.15.164:0/6635372 conn(0x7f177c0a3110 msgr2=0x7f177c0a48a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 shutdown_connections 2024-09-10T16:29:45.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.417+0000 7f1782d4b640 1 -- 172.21.15.164:0/6635372 wait complete. 2024-09-10T16:29:45.445 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:45.445 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:29:45.445 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:29:45.446 INFO:tasks.rados.rados.0.smithi165.stdout:3604: done (2 left) 2024-09-10T16:29:45.446 INFO:tasks.rados.rados.0.smithi165.stdout:3605: done (1 left) 2024-09-10T16:29:45.446 INFO:tasks.rados.rados.0.smithi165.stdout:3608: done (0 left) 2024-09-10T16:29:45.446 INFO:tasks.rados.rados.0.smithi165.stdout:3609: snap_create 2024-09-10T16:29:45.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.543+0000 7f409eb3d640 10 monclient: get_monmap_and_config 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.543+0000 7f409eb3d640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.543+0000 7f409eb3d640 10 monclient: monmap: 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:45.544217+0000 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:45.544217+0000 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:45.545 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:45.546 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 Processor -- start 2024-09-10T16:29:45.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 -- start start 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient: init 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f40980a0bc0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4098136260 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f409809e070 0x7f409813e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-c con 0x7f409809e070 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40980a1740 con 0x7f40980a0bc0 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40980690a0 con 0x7f4098136260 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f409809e500 con 0x7f409809e070 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409eb3d640 10 monclient(hunting): authenticate will time out at 2164.031591s 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409c8d9640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f40980a0bc0 0x7f409809d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36604/0 (socket says 172.21.15.164:36604) 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409c8d9640 1 -- 172.21.15.164:0/2661609456 learned_addr learned my addr 172.21.15.164:0/2661609456 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409d0da640 1 --2- 172.21.15.164:0/2661609456 >> v2:172.21.15.165:3300/0 conn(0x7f409809e070 0x7f409813e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409d0da640 10 monclient(hunting): get_auth_request con 0x7f409809e070 auth_method 0 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409d0da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409d0da640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.544+0000 7f409d0da640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient(hunting): handle_auth_done global_id 264127 payload 274 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 1 -- 172.21.15.164:0/2661609456 >> v1:172.21.15.165:6789/0 conn(0x7f4098136260 legacy=0x7f409813b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 1 -- 172.21.15.164:0/2661609456 >> v1:172.21.15.164:6789/0 conn(0x7f40980a0bc0 legacy=0x7f409809d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 1 monclient: found mon.noname-c 2024-09-10T16:29:45.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:29:45.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.545+0000 7f409d0da640 1 -- 172.21.15.164:0/2661609456 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4098069380 con 0x7f409809e070 2024-09-10T16:29:45.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409d0da640 1 -- 172.21.15.164:0/2661609456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409809e070 msgr2=0x7f409813e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409d0da640 1 --2- 172.21.15.164:0/2661609456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409809e070 0x7f409813e7a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4090009d90 tx=0x7f409002a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf4f9919285d715d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 1 -- 172.21.15.164:0/2661609456 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090007a30 con 0x7f409809e070 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: dump: 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.557 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 1 -- 172.21.15.164:0/2661609456 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4090007bd0 con 0x7f409809e070 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 5 monclient: authenticate success, global_id 264127 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 1 -- 172.21.15.164:0/2661609456 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4090031400 con 0x7f409809e070 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 4 set_mon_vals no callback set 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:45.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f408f7fe640 10 monclient: dump: 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 10 monclient: shutdown 2024-09-10T16:29:45.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 1 -- 172.21.15.164:0/2661609456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409809e070 msgr2=0x7f409813e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 1 --2- 172.21.15.164:0/2661609456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409809e070 0x7f409813e7a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4090009d90 tx=0x7f409002a490 comp rx=0 tx=0).stop 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 1 -- 172.21.15.164:0/2661609456 shutdown_connections 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 1 --2- 172.21.15.164:0/2661609456 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409809e070 0x7f409813e7a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.548+0000 7f409eb3d640 1 -- 172.21.15.164:0/2661609456 >> 172.21.15.164:0/2661609456 conn(0x7f40980a3fb0 msgr2=0x7f40980a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 1 -- 172.21.15.164:0/2661609456 shutdown_connections 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 1 -- 172.21.15.164:0/2661609456 wait complete. 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 10 monclient: monmap: 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:45.560 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 1 Processor -- start 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 1 -- start start 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.549+0000 7f409eb3d640 10 monclient: init 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 0x7f409809a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-c con 0x7f409809e070 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 0x7f409809a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-a con 0x7f40980a0bc0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4098136260 0x7f40981cce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): picked mon.noname-b con 0x7f4098136260 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f40980a0fa0 con 0x7f40980a0bc0 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4098069660 con 0x7f4098136260 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f409813ece0 con 0x7f409809e070 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409eb3d640 10 monclient(hunting): authenticate will time out at 2164.037483s 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 0x7f409809a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 0x7f409809a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 0x7f409809a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57440/0 (socket says 172.21.15.164:57440) 2024-09-10T16:29:45.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): get_auth_request con 0x7f40980a0bc0 auth_method 0 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 learned_addr learned my addr 172.21.15.164:0/4668411 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): get_auth_request con 0x7f409809e070 auth_method 0 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409d0da640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4098136260 0x7f40981cce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient(hunting): handle_auth_done global_id 254250 payload 824 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 msgr2=0x7f409809a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 0x7f409809a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4098136260 msgr2=0x7f40981cce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4098136260 0x7f40981cce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 monclient: found mon.noname-a 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f408ffff640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409812cd10 con 0x7f40980a0bc0 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409d0da640 10 monclient: get_auth_request con 0x7f4098136260 auth_method 0 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f409809e070 0x7f409809a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:29:45.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.550+0000 7f409d0da640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4098136260 0x7f40981cce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408ffff640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 msgr2=0x7f409809a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408ffff640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 0x7f409809a940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f408800c410 tx=0x7f408800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088017070 con 0x7f40980a0bc0 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: dump: 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.565 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 5 monclient: authenticate success, global_id 254250 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4088005230 con 0x7f40980a0bc0 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 10 monclient: _renew_subs 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40880117a0 con 0x7f40980a0bc0 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f409812cff0 con 0x7f40980a0bc0 2024-09-10T16:29:45.566 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408d7fa640 10 monclient: dump: 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f408effd640 4 set_mon_vals no callback set 2024-09-10T16:29:45.567 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 10 monclient: _renew_subs 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.553+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40980a7150 con 0x7f40980a0bc0 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409eb3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409eb3d640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f409809fef0 con 0x7f409802d440 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f408801a070 con 0x7f40980a0bc0 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 10 monclient: get_auth_request con 0x7f409802d440 auth_method 0 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 msgr2=0x7f409802d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:29:45.568 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 msgr2=0x7f409802d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f408d7fa640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e240 0x7f4084040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(809..809 src has 677..809) ==== 4787+0+0 (secure 0 0 0) 0x7f4088051fb0 con 0x7f40980a0bc0 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409d0da640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e240 0x7f4084040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.554+0000 7f409d0da640 10 monclient: get_auth_request con 0x7f408403e240 auth_method 0 2024-09-10T16:29:45.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.557+0000 7f409d0da640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e240 0x7f4084040700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4090031000 tx=0x7f409002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient: get_auth_request con 0x7f409802d440 auth_method 0 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 10 monclient(hunting): handle_auth_done global_id 264130 payload 824 2024-09-10T16:29:45.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.756+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f409812d390 con 0x7f409802d440 2024-09-10T16:29:45.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 msgr2=0x7f409802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094007770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094019020 con 0x7f409802d440 2024-09-10T16:29:45.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 10 monclient: dump: 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.761 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4094011d00 con 0x7f409802d440 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 msgr2=0x7f409802d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094007770 comp rx=0 tx=0).stop 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 msgr2=0x7f409802d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.759+0000 7f408d7fa640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802d440 0x7f409802d820 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094007770 comp rx=0 tx=0).stop 2024-09-10T16:29:45.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.765+0000 7f409eb3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.765+0000 7f409eb3d640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 0x7f409802e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.765+0000 7f409eb3d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.765+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f40980a6b70 con 0x7f409802dd60 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 0x7f409802e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient: get_auth_request con 0x7f409802dd60 auth_method 0 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 10 monclient(hunting): handle_auth_done global_id 264133 payload 824 2024-09-10T16:29:45.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.766+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4098074e60 con 0x7f409802dd60 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f409c8d9640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 msgr2=0x7f409802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f409c8d9640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 0x7f409802e170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4094019030 con 0x7f409802dd60 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 10 monclient: dump: 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.771 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f4094015b80 con 0x7f409802dd60 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 msgr2=0x7f409802e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 0x7f409802e170 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094004940 comp rx=0 tx=0).stop 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 msgr2=0x7f409802e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:29:45.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.769+0000 7f408d7fa640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f409802dd60 0x7f409802e170 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f4094006fd0 tx=0x7f4094004940 comp rx=0 tx=0).stop 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 232, 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 7911, 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:29:45.773 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.774 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.775 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:29:45.776 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:29:45.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:29:45.778 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.779 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:29:45.780 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:29:45.782 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.783 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:29:45.784 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e240 msgr2=0x7f4084040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f408403e240 0x7f4084040700 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4090031000 tx=0x7f409002f000 comp rx=0 tx=0).stop 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 10 monclient: shutdown 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 msgr2=0x7f409809a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 --2- 172.21.15.164:0/4668411 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f40980a0bc0 0x7f409809a940 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f408800c410 tx=0x7f408800c8d0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409d0da640 1 -- 172.21.15.164:0/4668411 reap_dead start 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409d0da640 1 -- 172.21.15.164:0/4668411 reap_dead start 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 shutdown_connections 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 >> 172.21.15.164:0/4668411 conn(0x7f40980a3fb0 msgr2=0x7f4098072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 shutdown_connections 2024-09-10T16:29:45.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.770+0000 7f409eb3d640 1 -- 172.21.15.164:0/4668411 wait complete. 2024-09-10T16:29:45.787 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:29:45.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T16:29:45.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:29:45.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T16:29:45.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T16:29:45.788 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:29:45.788 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:29:45.800 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:29:45.800 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:29:45.800 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:45.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.935+0000 7fc3f1e39640 10 monclient: get_monmap_and_config 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.935+0000 7fc3f1e39640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.935+0000 7fc3f1e39640 10 monclient: monmap: 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:45.936329+0000 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:45.936329+0000 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:45.937 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:45.938 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:45.938 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 Processor -- start 2024-09-10T16:29:45.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 -- start start 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient: init 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 0x7fc3ec09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ec09d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3ec09ab50 0x7fc3ec09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ec09ab50 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc3ec0a1e40 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3ec0a0870 con 0x7fc3ec0a1e40 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc3ec09d3e0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec09e790 con 0x7fc3ec09d930 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3f1e39640 10 monclient(hunting): authenticate will time out at 2164.423759s 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3eb7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 msgr2=0x7fc3ec09dd10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.936+0000 7fc3eb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 0x7fc3ec09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3ec09ab50 0x7fc3ec09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3ebfff640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7fc3ec0a1e40 0x7fc3ec134580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:45042/0 (socket says 172.21.15.164:45042) 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc3ec09ab50 0x7fc3ec09e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:52790/0 (socket says 172.21.15.164:52790) 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3ebfff640 1 -- 172.21.15.164:0/1517790108 learned_addr learned my addr 172.21.15.164:0/1517790108 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): get_auth_request con 0x7fc3ec09ab50 auth_method 0 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_done global_id 264136 payload 274 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 -- 172.21.15.164:0/1517790108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 msgr2=0x7fc3ec09dd10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 --2- 172.21.15.164:0/1517790108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 0x7fc3ec09dd10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 -- 172.21.15.164:0/1517790108 >> v1:172.21.15.165:6789/0 conn(0x7fc3ec0a1e40 legacy=0x7fc3ec134580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 monclient: found mon.noname-c 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:29:45.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.937+0000 7fc3eaffd640 1 -- 172.21.15.164:0/1517790108 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec074c70 con 0x7fc3ec09ab50 2024-09-10T16:29:45.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3eaffd640 1 -- 172.21.15.164:0/1517790108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09ab50 msgr2=0x7fc3ec09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.950 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3eaffd640 1 --2- 172.21.15.164:0/1517790108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec09e250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40097a0 tx=0x7fc3d402a070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9afb1826c9f35aca server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 1 -- 172.21.15.164:0/1517790108 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4037070 con 0x7fc3ec09ab50 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: dump: 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.951 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.952 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 1 -- 172.21.15.164:0/1517790108 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d4032640 con 0x7fc3ec09ab50 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3f1e39640 5 monclient: authenticate success, global_id 264136 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3f1e39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3f1e39640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 1 -- 172.21.15.164:0/1517790108 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d4032a10 con 0x7fc3ec09ab50 2024-09-10T16:29:45.953 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3f1e39640 4 set_mon_vals no callback set 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.940+0000 7fc3ea7fc640 10 monclient: dump: 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.954 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 10 monclient: shutdown 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1517790108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09ab50 msgr2=0x7fc3ec09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1517790108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec09e250 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3d40097a0 tx=0x7fc3d402a070 comp rx=0 tx=0).stop 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1517790108 shutdown_connections 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1517790108 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec09e250 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1517790108 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec09d930 0x7fc3ec09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1517790108 >> 172.21.15.164:0/1517790108 conn(0x7fc3ec0a3110 msgr2=0x7fc3ec0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1517790108 shutdown_connections 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1517790108 wait complete. 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 10 monclient: build_initial_monmap 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.941+0000 7fc3f1e39640 10 monclient: monmap: 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.956 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 Processor -- start 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 -- start start 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient: init 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-a con 0x7fc3ec09ab50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09d930 0x7fc3ec070c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-b con 0x7fc3ec09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:45.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 0x7fc3ec06eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): picked mon.noname-c con 0x7fc3ec0a1e40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec09ef30 con 0x7fc3ec09ab50 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc3ec0a02b0 con 0x7fc3ec09d930 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc3ec0a2940 con 0x7fc3ec0a1e40 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3f1e39640 10 monclient(hunting): authenticate will time out at 2164.429816s 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3ebfff640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 msgr2=0x7fc3ec06eec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3ebfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 0x7fc3ec06eec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:45.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46370/0 (socket says 172.21.15.164:46370) 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 1 -- 172.21.15.164:0/1625441311 learned_addr learned my addr 172.21.15.164:0/1625441311 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 10 monclient(hunting): get_auth_request con 0x7fc3ec09ab50 auth_method 0 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.942+0000 7fc3eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09d930 0x7fc3ec070c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): get_auth_request con 0x7fc3ec09d930 auth_method 0 2024-09-10T16:29:45.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient(hunting): handle_auth_done global_id 254253 payload 824 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 -- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 msgr2=0x7fc3ec06eec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:45.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 0x7fc3ec06eec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 -- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09d930 msgr2=0x7fc3ec070c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09d930 0x7fc3ec070c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 monclient: found mon.noname-a 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.943+0000 7fc3eb7fe640 1 -- 172.21.15.164:0/1625441311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3ec072200 con 0x7fc3ec09ab50 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3eb7fe640 1 -- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 msgr2=0x7fc3ec099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3eb7fe640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b9e0 tx=0x7fc3dc00bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc00cdb0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: dump: 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.962 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: _finish_auth 0 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3dc010070 con 0x7fc3ec09ab50 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 5 monclient: authenticate success, global_id 254253 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 10 monclient: _renew_subs 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3ec0724e0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:45.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3dc0156b0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e3fff640 10 monclient: dump: 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:45.964 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3e9ffb640 4 set_mon_vals no callback set 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 10 monclient: _renew_subs 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.945+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3ec070910 con 0x7fc3ec09ab50 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc3dc01e3f0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3f1e39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3f1e39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3f1e39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3ec0a22d0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3e3fff640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 0x7fc3cc040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:45.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.946+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(809..809 src has 677..809) ==== 4787+0+0 (secure 0 0 0) 0x7fc3dc052eb0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.947+0000 7fc3eaffd640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 0x7fc3cc040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.947+0000 7fc3eaffd640 10 monclient: get_auth_request con 0x7fc3cc03e1c0 auth_method 0 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.949+0000 7fc3eaffd640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 0x7fc3cc040680 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4031000 tx=0x7fc3d402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.951+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc3dc0518a0 con 0x7fc3ec09ab50 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.951+0000 7fc3e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:45.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:45.951+0000 7fc3e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:46.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.197+0000 7fc3f1e39640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:46.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.197+0000 7fc3f1e39640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:46.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.197+0000 7fc3f1e39640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:46.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.197+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3ec06dac0 con 0x7fc3ec09ab50 2024-09-10T16:29:46.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.198+0000 7fc3e3fff640 1 -- 172.21.15.164:0/1625441311 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fc3dc012d30 con 0x7fc3ec09ab50 2024-09-10T16:29:46.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.198+0000 7fc3e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:46.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.198+0000 7fc3e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:46.200 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 msgr2=0x7fc3cc040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 0x7fc3cc040680 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7fc3d4031000 tx=0x7fc3d402e000 comp rx=0 tx=0).stop 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 10 monclient: shutdown 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 msgr2=0x7fc3ec099ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc3dc00b9e0 tx=0x7fc3dc00bea0 comp rx=0 tx=0).stop 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 shutdown_connections 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc3cc03e1c0 0x7fc3cc040680 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:46.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc3ec0a1e40 0x7fc3ec06eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc3ec09d930 0x7fc3ec070c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 --2- 172.21.15.164:0/1625441311 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc3ec09ab50 0x7fc3ec099ae0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 >> 172.21.15.164:0/1625441311 conn(0x7fc3ec0a3110 msgr2=0x7fc3ec09b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 shutdown_connections 2024-09-10T16:29:46.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:46.200+0000 7fc3f1e39640 1 -- 172.21.15.164:0/1625441311 wait complete. 2024-09-10T16:29:49.213 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.321+0000 7f0285bec640 10 monclient: get_monmap_and_config 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.321+0000 7f0285bec640 10 monclient: build_initial_monmap 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.321+0000 7f0285bec640 10 monclient: monmap: 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:49.322631+0000 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:49.322631+0000 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.323 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:49.324 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:49.324 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:49.324 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:49.324 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:49.324 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.322+0000 7f0285bec640 1 Processor -- start 2024-09-10T16:29:49.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.322+0000 7f0285bec640 1 -- start start 2024-09-10T16:29:49.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.322+0000 7f0285bec640 10 monclient: init 2024-09-10T16:29:49.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:49.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient: _add_conns ranks=[0,4,1,3,2] 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f028009f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f02800a2550 0x7f028013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-c con 0x7f02800a2550 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02800a2980 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f028009fd10 con 0x7f028009f930 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f028009cd20 con 0x7f02800a2980 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f028009a0d0 con 0x7f02800a2550 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f0285bec640 10 monclient(hunting): authenticate will time out at 2167.810116s 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027f7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f028009f930 0x7f02800a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36620/0 (socket says 172.21.15.164:36620) 2024-09-10T16:29:49.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027f7fe640 1 -- 172.21.15.164:0/2889872018 learned_addr learned my addr 172.21.15.164:0/2889872018 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 1 --2- 172.21.15.164:0/2889872018 >> v2:172.21.15.165:3300/0 conn(0x7f02800a2550 0x7f028013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): get_auth_request con 0x7f02800a2550 auth_method 0 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient(hunting): handle_auth_done global_id 264142 payload 274 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 1 -- 172.21.15.164:0/2889872018 >> v1:172.21.15.165:6789/0 conn(0x7f02800a2980 legacy=0x7f028013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 1 -- 172.21.15.164:0/2889872018 >> v1:172.21.15.164:6789/0 conn(0x7f028009f930 legacy=0x7f02800a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient: _finish_hunting 0 2024-09-10T16:29:49.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 1 monclient: found mon.noname-c 2024-09-10T16:29:49.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:49.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:29:49.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.323+0000 7f027effd640 1 -- 172.21.15.164:0/2889872018 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028009e840 con 0x7f02800a2550 2024-09-10T16:29:49.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.326+0000 7f027effd640 1 -- 172.21.15.164:0/2889872018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2550 msgr2=0x7f028013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.326+0000 7f027effd640 1 --2- 172.21.15.164:0/2889872018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2550 0x7f028013b5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f028009cef0 tx=0x7f0274029fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=213aa2531c040a60 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 1 -- 172.21.15.164:0/2889872018 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274037070 con 0x7f02800a2550 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: dump: 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.334 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 1 -- 172.21.15.164:0/2889872018 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0274032440 con 0x7f02800a2550 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 5 monclient: authenticate success, global_id 264142 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 1 -- 172.21.15.164:0/2889872018 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0274032810 con 0x7f02800a2550 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 4 set_mon_vals no callback set 2024-09-10T16:29:49.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f027e7fc640 10 monclient: dump: 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:49.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 10 monclient: shutdown 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 -- 172.21.15.164:0/2889872018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2550 msgr2=0x7f028013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 --2- 172.21.15.164:0/2889872018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2550 0x7f028013b5d0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f028009cef0 tx=0x7f0274029fb0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 -- 172.21.15.164:0/2889872018 shutdown_connections 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 --2- 172.21.15.164:0/2889872018 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2550 0x7f028013b5d0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 -- 172.21.15.164:0/2889872018 >> 172.21.15.164:0/2889872018 conn(0x7f02800a3110 msgr2=0x7f02800a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 -- 172.21.15.164:0/2889872018 shutdown_connections 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.327+0000 7f0285bec640 1 -- 172.21.15.164:0/2889872018 wait complete. 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.328+0000 7f0285bec640 10 monclient: build_initial_monmap 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.328+0000 7f0285bec640 10 monclient: monmap: 2024-09-10T16:29:49.337 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.328+0000 7f0285bec640 1 Processor -- start 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 -- start start 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient: init 2024-09-10T16:29:49.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-a con 0x7f028009f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 0x7f02801bac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-c con 0x7f02800a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 0x7f028009a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): picked mon.noname-b con 0x7f02800a2980 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f028009d930 con 0x7f028009f930 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f028009ebe0 con 0x7f02800a2980 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:49.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f028009c8b0 con 0x7f02800a2550 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f0285bec640 10 monclient(hunting): authenticate will time out at 2167.816576s 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027effd640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 msgr2=0x7f02801bac30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 0x7f02801bac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46384/0 (socket says 172.21.15.164:46384) 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 -- 172.21.15.164:0/1615101454 learned_addr learned my addr 172.21.15.164:0/1615101454 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): get_auth_request con 0x7f028009f930 auth_method 0 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:49.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027ffff640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 0x7f028009a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 10 monclient(hunting): handle_auth_done global_id 254256 payload 824 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 -- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 msgr2=0x7f02801bac30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 0x7f02801bac30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 -- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 msgr2=0x7f028009a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.329+0000 7f027f7fe640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 0x7f028009a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027f7fe640 1 monclient: found mon.noname-a 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027f7fe640 1 -- 172.21.15.164:0/1615101454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f028009bc30 con 0x7f028009f930 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027ffff640 10 monclient: get_auth_request con 0x7f02800a2980 auth_method 0 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.330+0000 7f027ffff640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 0x7f028009a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f027f7fe640 1 -- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 msgr2=0x7f02801ba6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f027f7fe640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f027000cc10 tx=0x7f027000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0270012400 con 0x7f028009f930 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:49.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: dump: 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: _finish_auth 0 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: _check_auth_tickets 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:49.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02700125e0 con 0x7f028009f930 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 5 monclient: authenticate success, global_id 254256 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 10 monclient: _renew_subs 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f028006e9d0 con 0x7f028009f930 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02700129d0 con 0x7f028009f930 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0267fff640 10 monclient: dump: 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.344 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 10 monclient: _renew_subs 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02801b58f0 con 0x7f028009f930 2024-09-10T16:29:49.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.332+0000 7f027d7fa640 4 set_mon_vals no callback set 2024-09-10T16:29:49.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0285bec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:49.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0285bec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0285bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0270012b70 con 0x7f028009f930 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02800a0760 con 0x7f028009f930 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0267fff640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 0x7f026c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(809..809 src has 677..809) ==== 4787+0+0 (secure 0 0 0) 0x7f0270053600 con 0x7f028009f930 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.333+0000 7f027effd640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 0x7f026c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.334+0000 7f027effd640 10 monclient: get_auth_request con 0x7f026c03e1c0 auth_method 0 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.336+0000 7f027effd640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 0x7f026c040680 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0274034000 tx=0x7f027402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.338+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0270051ff0 con 0x7f028009f930 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.338+0000 7f0267fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:49.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.338+0000 7f0267fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.579+0000 7f0285bec640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.579+0000 7f0285bec640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.579+0000 7f0285bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:49.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.579+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f02801b5c90 con 0x7f028009f930 2024-09-10T16:29:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.580+0000 7f0267fff640 1 -- 172.21.15.164:0/1615101454 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f02700532d0 con 0x7f028009f930 2024-09-10T16:29:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.580+0000 7f0267fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:49.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.580+0000 7f0267fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:49.582 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:49.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 msgr2=0x7f026c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 0x7f026c040680 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f0274034000 tx=0x7f027402e000 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 10 monclient: shutdown 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 msgr2=0x7f02801ba6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f027000cc10 tx=0x7f027000a430 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 shutdown_connections 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f026c03e1c0 0x7f026c040680 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f02800a2980 0x7f028009a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f02800a2550 0x7f02801bac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 --2- 172.21.15.164:0/1615101454 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f028009f930 0x7f02801ba6f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 >> 172.21.15.164:0/1615101454 conn(0x7f02800a3110 msgr2=0x7f02800a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:49.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 shutdown_connections 2024-09-10T16:29:49.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:49.582+0000 7f0285bec640 1 -- 172.21.15.164:0/1615101454 wait complete. 2024-09-10T16:29:52.595 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:52.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 10 monclient: get_monmap_and_config 2024-09-10T16:29:52.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 10 monclient: build_initial_monmap 2024-09-10T16:29:52.704 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 10 monclient: monmap: 2024-09-10T16:29:52.704 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:52.703986+0000 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:52.703986+0000 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:52.705 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:52.706 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:52.706 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 1 Processor -- start 2024-09-10T16:29:52.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 1 -- start start 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.703+0000 7fefd47c2640 10 monclient: init 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefcc06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fefcc134f70 0x7fefcc13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-a con 0x7fefcc134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 0x7fefcc13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-b con 0x7fefcc13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefcc09ce50 con 0x7fefcc06cb70 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:52.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fefcc06db60 con 0x7fefcc134f70 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc06cf50 con 0x7fefcc13d560 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd47c2640 10 monclient(hunting): authenticate will time out at 2171.191606s 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd2d5f640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 msgr2=0x7fefcc13f950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd2d5f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 0x7fefcc13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fefcc134f70 0x7fefcc13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd255e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fefcc06cb70 0x7fefcc0a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:36624/0 (socket says 172.21.15.164:36624) 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd255e640 1 -- 172.21.15.164:0/3238105406 learned_addr learned my addr 172.21.15.164:0/3238105406 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): get_auth_request con 0x7fefcc134f70 auth_method 0 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:52.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.704+0000 7fefd1d5d640 10 monclient(hunting): handle_auth_done global_id 254259 payload 274 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 1 -- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 msgr2=0x7fefcc13f950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 1 --2- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 0x7fefcc13f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 1 -- 172.21.15.164:0/3238105406 >> v1:172.21.15.164:6789/0 conn(0x7fefcc06cb70 legacy=0x7fefcc0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 10 monclient: _finish_hunting 0 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 1 monclient: found mon.noname-a 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.705+0000 7fefd1d5d640 1 -- 172.21.15.164:0/3238105406 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc135380 con 0x7fefcc134f70 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd1d5d640 1 -- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc134f70 msgr2=0x7fefcc13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd1d5d640 1 --2- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc134f70 0x7fefcc13d020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0097d0 tx=0x7fefbc02a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d31af271cffa095 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 1 -- 172.21.15.164:0/3238105406 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc037070 con 0x7fefcc134f70 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 10 monclient: dump: 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.714 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 10 monclient: _finish_auth 0 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 10 monclient: _check_auth_tickets 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 5 monclient: authenticate success, global_id 254259 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.707+0000 7fefd155c640 1 -- 172.21.15.164:0/3238105406 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefbc032640 con 0x7fefcc134f70 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd155c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 10 monclient: get_monmap_and_config success 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd155c640 1 -- 172.21.15.164:0/3238105406 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefbc032a10 con 0x7fefcc134f70 2024-09-10T16:29:52.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 4 set_mon_vals no callback set 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd155c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd155c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd155c640 10 monclient: dump: 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:52.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 10 monclient: shutdown 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 -- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc134f70 msgr2=0x7fefcc13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 --2- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc134f70 0x7fefcc13d020 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fefbc0097d0 tx=0x7fefbc02a010 comp rx=0 tx=0).stop 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 -- 172.21.15.164:0/3238105406 shutdown_connections 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 --2- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc13d560 0x7fefcc13f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 --2- 172.21.15.164:0/3238105406 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc134f70 0x7fefcc13d020 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 -- 172.21.15.164:0/3238105406 >> 172.21.15.164:0/3238105406 conn(0x7fefcc0aa900 msgr2=0x7fefcc0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 -- 172.21.15.164:0/3238105406 shutdown_connections 2024-09-10T16:29:52.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.708+0000 7fefd47c2640 1 -- 172.21.15.164:0/3238105406 wait complete. 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.709+0000 7fefd47c2640 10 monclient: build_initial_monmap 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.709+0000 7fefd47c2640 10 monclient: monmap: 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:29:52.718 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:29:52.719 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.709+0000 7fefd47c2640 1 Processor -- start 2024-09-10T16:29:52.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.709+0000 7fefd47c2640 1 -- start start 2024-09-10T16:29:52.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.709+0000 7fefd47c2640 10 monclient: init 2024-09-10T16:29:52.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 0x7fefcc13b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-c con 0x7fefcc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefcc134f70 0x7fefcc13b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-b con 0x7fefcc134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): picked mon.noname-a con 0x7fefcc13d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc0a4db0 con 0x7fefcc13d560 2024-09-10T16:29:52.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fefcc09d130 con 0x7fefcc134f70 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefcc13fe90 con 0x7fefcc06cb70 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd47c2640 10 monclient(hunting): authenticate will time out at 2171.197126s 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd255e640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 msgr2=0x7fefcc13b230 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd255e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 0x7fefcc13b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:46394/0 (socket says 172.21.15.164:46394) 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 -- 172.21.15.164:0/1244869623 learned_addr learned my addr 172.21.15.164:0/1244869623 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): get_auth_request con 0x7fefcc13d560 auth_method 0 2024-09-10T16:29:52.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd1d5d640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefcc134f70 0x7fefcc13b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd1d5d640 10 monclient(hunting): get_auth_request con 0x7fefcc134f70 auth_method 0 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd1d5d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd1d5d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd1d5d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:52.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient(hunting): handle_auth_done global_id 254262 payload 824 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 -- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 msgr2=0x7fefcc13b230 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 0x7fefcc13b230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 -- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefcc134f70 msgr2=0x7fefcc13b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefcc134f70 0x7fefcc13b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient: _finish_hunting 0 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 monclient: found mon.noname-a 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.710+0000 7fefd2d5f640 1 -- 172.21.15.164:0/1244869623 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefcc132820 con 0x7fefcc13d560 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd2d5f640 1 -- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 msgr2=0x7fefcc13bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd2d5f640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fefc400be30 tx=0x7fefc400bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:52.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc400cca0 con 0x7fefcc13d560 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: dump: 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:52.728 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 5 monclient: authenticate success, global_id 254262 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefc4010030 con 0x7fefcc13d560 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 10 monclient: _renew_subs 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefcc09f4c0 con 0x7fefcc13d560 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefc4016670 con 0x7fefcc13d560 2024-09-10T16:29:52.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefc2ffd640 10 monclient: dump: 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:52.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd0d5b640 4 set_mon_vals no callback set 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 10 monclient: _renew_subs 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.712+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefcc09e1c0 con 0x7fefcc13d560 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.713+0000 7fefd47c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.713+0000 7fefd47c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.713+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fefc4008070 con 0x7fefcc13d560 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.713+0000 7fefd47c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.713+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefcc0a46a0 con 0x7fefcc13d560 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.714+0000 7fefc2ffd640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 0x7fefb0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.714+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(809..809 src has 677..809) ==== 4787+0+0 (secure 0 0 0) 0x7fefc4052b10 con 0x7fefcc13d560 2024-09-10T16:29:52.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.714+0000 7fefd255e640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 0x7fefb0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:52.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.714+0000 7fefd255e640 10 monclient: get_auth_request con 0x7fefb003e1c0 auth_method 0 2024-09-10T16:29:52.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.717+0000 7fefd255e640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 0x7fefb0040680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefc8007fd0 tx=0x7fefc8007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:29:52.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.718+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fefcc132820 con 0x7fefcc13d560 2024-09-10T16:29:52.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.718+0000 7fefc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:29:52.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.718+0000 7fefc2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:29:52.960 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.958+0000 7fefd47c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.958+0000 7fefd47c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.958+0000 7fefd47c2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.958+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fefcc09f240 con 0x7fefcc13d560 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.959+0000 7fefc2ffd640 1 -- 172.21.15.164:0/1244869623 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7fefc401d300 con 0x7fefcc13d560 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.959+0000 7fefc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.959+0000 7fefc2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:29:52.961 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":232,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 msgr2=0x7fefb0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 0x7fefb0040680 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fefc8007fd0 tx=0x7fefc8007b20 comp rx=0 tx=0).stop 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 10 monclient: shutdown 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 msgr2=0x7fefcc13bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fefc400be30 tx=0x7fefc400bf20 comp rx=0 tx=0).stop 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 shutdown_connections 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefb003e1c0 0x7fefb0040680 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefcc13d560 0x7fefcc13bcb0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefcc134f70 0x7fefcc13b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.963 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 --2- 172.21.15.164:0/1244869623 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefcc06cb70 0x7fefcc13b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:52.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 >> 172.21.15.164:0/1244869623 conn(0x7fefcc0aa900 msgr2=0x7fefcc0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:29:52.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 shutdown_connections 2024-09-10T16:29:52.964 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:52.961+0000 7fefd47c2640 1 -- 172.21.15.164:0/1244869623 wait complete. 2024-09-10T16:29:55.975 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.082+0000 7fefeb623640 10 monclient: get_monmap_and_config 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.082+0000 7fefeb623640 10 monclient: build_initial_monmap 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.082+0000 7fefeb623640 10 monclient: monmap: 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:29:56.083772+0000 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:29:56.083772+0000 2024-09-10T16:29:56.084 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:29:56.085 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.083+0000 7fefeb623640 1 Processor -- start 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.083+0000 7fefeb623640 1 -- start start 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.083+0000 7fefeb623640 10 monclient: init 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient: _reopen_session rank -1 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefe406cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:29:56.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fefe409bcc0 addr v1:172.21.15.164:6789/0 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-b con 0x7fefe40a40d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe406e900 con 0x7fefe409bcc0 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe406e340 con 0x7fefe406cb70 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefe40a4600 con 0x7fefe40a40d0 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefeb623640 10 monclient(hunting): authenticate will time out at 2174.571303s 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefe9bc0640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 msgr2=0x7fefe406d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefe9bc0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefe8bbe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fefe409bcc0 0x7fefe413d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51836/0 (socket says 172.21.15.164:51836) 2024-09-10T16:29:56.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.084+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 learned_addr learned my addr 172.21.15.164:0/1536182207 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:29:56.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.086+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v1:172.21.15.164:6789/0 conn(0x7fefe409bcc0 legacy=0x7fefe413d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:56.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.086+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v1:172.21.15.165:6789/0 conn(0x7fefe406cb70 legacy=0x7fefe406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:29:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.285+0000 7fefe9bc0640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 msgr2=0x7fefe406d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:56.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.285+0000 7fefe9bc0640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:56.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.686+0000 7fefe9bc0640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 msgr2=0x7fefe406d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:56.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:56.686+0000 7fefe9bc0640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:57.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:57.487+0000 7fefe9bc0640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 msgr2=0x7fefe406d660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:29:57.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:57.487+0000 7fefe9bc0640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): tick 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 msgr2=0x7fefe406d660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefe40a40d0 0x7fefe406d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 >> v1:172.21.15.165:6789/0 conn(0x7fefe406cb70 legacy=0x7fefe406cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 >> v1:172.21.15.164:6789/0 conn(0x7fefe409bcc0 legacy=0x7fefe413d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient: _add_conns ranks=[4,3] 2024-09-10T16:29:59.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fefc8003810 addr v2:172.21.15.165:3300/0 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): picked mon.noname-a con 0x7fefc8006380 addr v2:172.21.15.164:3300/0 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fefc8008ed0 con 0x7fefc8006380 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fefc8009080 con 0x7fefc8003810 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefdaffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request con 0x7fefc8006380 auth_method 0 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003810 auth_method 0 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.084+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:59.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.085+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request con 0x7fefc8006380 auth_method 0 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.285+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:59.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003810 auth_method 0 2024-09-10T16:29:59.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.286+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request con 0x7fefc8006380 auth_method 0 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.686+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:29:59.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:29:59.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003810 auth_method 0 2024-09-10T16:29:59.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:29:59.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:29:59.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:29:59.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:29:59.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:29:59.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:29:59.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:29:59.687+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request con 0x7fefc8006380 auth_method 0 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:00.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:00.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:00.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:00.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.487+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:30:00.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:00.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003810 auth_method 0 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:00.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:00.489+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:30:01.373 INFO:tasks.rados.rados.0.smithi165.stdout:3609: done (0 left) 2024-09-10T16:30:01.373 INFO:tasks.rados.rados.0.smithi165.stdout:3610: write (excl) oid 45 current snap is 405 2024-09-10T16:30:01.373 INFO:tasks.rados.rados.0.smithi165.stdout:3610: seq_num 1251 ranges {625557=418391,1838856=786427} 2024-09-10T16:30:01.378 INFO:tasks.rados.rados.0.smithi165.stdout:3610: writing smithi16534841-45 from 625557 to 1043948 tid 1 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3610: writing smithi16534841-45 from 1838856 to 2625283 tid 2 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3611: read oid 27 snap -1 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3611: expect deleted 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3612: read oid 12 snap -1 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3612: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3613: write oid 9 current snap is 405 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3610: finishing write tid 1 to smithi16534841-45 2024-09-10T16:30:01.390 INFO:tasks.rados.rados.0.smithi165.stdout:3613: seq_num 1252 ranges {642364=493640,1897338=646438} 2024-09-10T16:30:01.396 INFO:tasks.rados.rados.0.smithi165.stdout:3613: writing smithi16534841-9 from 642364 to 1136004 tid 1 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:3613: writing smithi16534841-9 from 1897338 to 2543776 tid 2 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:3614: read oid 7 snap -1 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:3614: expect deleted 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:3615: delete oid 15 current snap is 405 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:3612: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:30:01.405 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:01.583 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:01.672 INFO:tasks.rados.rados.0.smithi165.stdout:3610: finishing write tid 2 to smithi16534841-45 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: finishing write tid 3 to smithi16534841-45 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: finishing write tid 4 to smithi16534841-45 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: oid 45 updating version 0 to 1710 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: oid 45 version 1710 is already newer than 1708 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: oid 45 version 1710 is already newer than 1709 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1710 (ObjNum 1251 snap 405 seq_num 1251) dirty exists 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: left oid 45 (ObjNum 1251 snap 405 seq_num 1251) 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3610: done (5 left) 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3611: done (4 left) 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3612: done (3 left) 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3615: done (2 left) 2024-09-10T16:30:01.673 INFO:tasks.rados.rados.0.smithi165.stdout:3616: snap_remove snap 355 2024-09-10T16:30:02.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): tick 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 msgr2=0x7fefc8005c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003810 0x7fefc8005c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 msgr2=0x7fefc8008840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.164:3300/0 conn(0x7fefc8006380 0x7fefc8008840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefe9bc0640 1 -- 172.21.15.164:0/1536182207 reap_dead start 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:30:02.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003c80 0x7fefc8006830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): picked mon.noname-c con 0x7fefc8003c80 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefc8007150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): picked mon.noname-b con 0x7fefc8006d70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fefc8007690 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefe40a01e0 con 0x7fefc8007690 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fefc8008ed0 con 0x7fefc8003c80 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 1 -- 172.21.15.164:0/1536182207 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefc8009080 con 0x7fefc8006d70 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.084+0000 7fefdaffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe8bbe640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 msgr2=0x7fefc8007150 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefc8007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> v2:172.21.15.165:3300/0 conn(0x7fefc8003c80 0x7fefc8006830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003c80 auth_method 0 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient(hunting): handle_auth_done global_id 264148 payload 274 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 msgr2=0x7fefc8007150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefc8007150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> v1:172.21.15.165:6789/0 conn(0x7fefc8007690 legacy=0x7fefc8007af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 monclient: found mon.noname-c 2024-09-10T16:30:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:30:02.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.085+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc80084d0 con 0x7fefc8003c80 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefe93bf640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8003c80 msgr2=0x7fefc8006830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefe93bf640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8003c80 0x7fefc8006830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefe0007af0 tx=0x7fefe002ab40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e540f17a805b3a34 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 1 -- 172.21.15.164:0/1536182207 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0034070 con 0x7fefc8003c80 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: dump: 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.095 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 1 -- 172.21.15.164:0/1536182207 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe002c740 con 0x7fefc8003c80 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefeb623640 5 monclient: authenticate success, global_id 264148 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefeb623640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:02.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefeb623640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefeb623640 4 set_mon_vals no callback set 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 1 -- 172.21.15.164:0/1536182207 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe002cb10 con 0x7fefc8003c80 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.088+0000 7fefdbfff640 10 monclient: dump: 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.097 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 10 monclient: shutdown 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 -- 172.21.15.164:0/1536182207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8003c80 msgr2=0x7fefc8006830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8003c80 0x7fefc8006830 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefe0007af0 tx=0x7fefe002ab40 comp rx=0 tx=0).stop 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 -- 172.21.15.164:0/1536182207 shutdown_connections 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefc8007150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 --2- 172.21.15.164:0/1536182207 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8003c80 0x7fefc8006830 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.089+0000 7fefeb623640 1 -- 172.21.15.164:0/1536182207 >> 172.21.15.164:0/1536182207 conn(0x7fefe40a4f50 msgr2=0x7fefe40a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.090+0000 7fefeb623640 1 -- 172.21.15.164:0/1536182207 shutdown_connections 2024-09-10T16:30:02.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.090+0000 7fefeb623640 1 -- 172.21.15.164:0/1536182207 wait complete. 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.090+0000 7fefeb623640 10 monclient: build_initial_monmap 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.090+0000 7fefeb623640 10 monclient: monmap: 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 Processor -- start 2024-09-10T16:30:02.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 -- start start 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient: init 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-a con 0x7fefc8003c80 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefe409a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-c con 0x7fefc8006d70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8007690 0x7fefe409aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): picked mon.noname-b con 0x7fefc8007690 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fefe406d360 con 0x7fefc8003c80 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fefe406d1e0 con 0x7fefc8007690 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fefe406d4e0 con 0x7fefc8006d70 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefeb623640 10 monclient(hunting): authenticate will time out at 2180.578842s 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe8bbe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 msgr2=0x7fefe409a570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe8bbe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefe409a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34082/0 (socket says 172.21.15.164:34082) 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 1 -- 172.21.15.164:0/2536768696 learned_addr learned my addr 172.21.15.164:0/2536768696 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 10 monclient(hunting): get_auth_request con 0x7fefc8003c80 auth_method 0 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.091+0000 7fefe93bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8007690 0x7fefe409aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): get_auth_request con 0x7fefc8007690 auth_method 0 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe9bc0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient(hunting): handle_auth_done global_id 254265 payload 824 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 -- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 msgr2=0x7fefe409a570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefe409a570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 -- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8007690 msgr2=0x7fefe409aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8007690 0x7fefe409aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 monclient: found mon.noname-a 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.092+0000 7fefe93bf640 1 -- 172.21.15.164:0/2536768696 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefe4095cd0 con 0x7fefc8003c80 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefe93bf640 1 -- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 msgr2=0x7fefe41cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefe93bf640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefe002b130 tx=0x7fefe0001520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe0034030 con 0x7fefc8003c80 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: dump: 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.105 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefe0039da0 con 0x7fefc8003c80 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefeb623640 5 monclient: authenticate success, global_id 254265 2024-09-10T16:30:02.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefeb623640 10 monclient: _renew_subs 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefeb623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefe409c140 con 0x7fefc8003c80 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefe00110b0 con 0x7fefc8003c80 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefd9ffb640 10 monclient: dump: 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.107 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.095+0000 7fefeb623640 10 monclient: _renew_subs 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.094+0000 7fefdb7fe640 4 set_mon_vals no callback set 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.095+0000 7fefeb623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.095+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefe4098030 con 0x7fefc8003c80 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefeb623640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefeb623640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefeb623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fefe0037070 con 0x7fefc8003c80 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefe406a2b0 con 0x7fefc8003c80 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefd9ffb640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 0x7fefc8042890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(811..811 src has 677..811) ==== 4739+0+0 (secure 0 0 0) 0x7fefe0073670 con 0x7fefc8003c80 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 0x7fefc8042890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.096+0000 7fefe8bbe640 10 monclient: get_auth_request con 0x7fefc80403d0 auth_method 0 2024-09-10T16:30:02.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.099+0000 7fefe8bbe640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 0x7fefc8042890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fefd4007fd0 tx=0x7fefd4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.101+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fefe0072090 con 0x7fefc8003c80 2024-09-10T16:30:02.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.101+0000 7fefd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.101+0000 7fefd9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:02.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefeb623640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:02.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefeb623640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefeb623640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fefe4097c00 con 0x7fefc8003c80 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefd9ffb640 1 -- 172.21.15.164:0/2536768696 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fefe00733c0 con 0x7fefc8003c80 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.345+0000 7fefd9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:02.347 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":236,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:02.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 msgr2=0x7fefc8042890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 0x7fefc8042890 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fefd4007fd0 tx=0x7fefd4009040 comp rx=0 tx=0).stop 2024-09-10T16:30:02.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 10 monclient: shutdown 2024-09-10T16:30:02.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 msgr2=0x7fefe41cd720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fefe002b130 tx=0x7fefe0001520 comp rx=0 tx=0).stop 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 shutdown_connections 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefc80403d0 0x7fefc8042890 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fefc8007690 0x7fefe409aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fefc8006d70 0x7fefe409a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 --2- 172.21.15.164:0/2536768696 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fefc8003c80 0x7fefe41cd720 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 >> 172.21.15.164:0/2536768696 conn(0x7fefe40a4f50 msgr2=0x7fefe409eb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 shutdown_connections 2024-09-10T16:30:02.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.347+0000 7fefeb623640 1 -- 172.21.15.164:0/2536768696 wait complete. 2024-09-10T16:30:02.360 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:30:02.360 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:30:02.360 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:30:02.381 INFO:tasks.rados.rados.0.smithi165.stdout:3616: done (2 left) 2024-09-10T16:30:02.381 INFO:tasks.rados.rados.0.smithi165.stdout:3617: snap_remove snap 396 2024-09-10T16:30:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 10 monclient: get_monmap_and_config 2024-09-10T16:30:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 10 monclient: build_initial_monmap 2024-09-10T16:30:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 10 monclient: monmap: 2024-09-10T16:30:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:02.466 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:02.466018+0000 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:02.466018+0000 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:02.467 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 1 Processor -- start 2024-09-10T16:30:02.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 1 -- start start 2024-09-10T16:30:02.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.465+0000 7f72058e6640 10 monclient: init 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient: _add_conns ranks=[1,3,2,4,0] 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f720006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7200134f70 0x7f720013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7200134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f720013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-b con 0x7f720013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f720009ce50 con 0x7f720006cb70 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f720006db60 con 0x7f7200134f70 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f720006cf50 con 0x7f720013d560 2024-09-10T16:30:02.470 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f72058e6640 10 monclient(hunting): authenticate will time out at 2180.953268s 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71ff7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 msgr2=0x7f720013f950 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71ff7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f720013f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7200134f70 0x7f720013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7200134f70 0x7f720013d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34088/0 (socket says 172.21.15.164:34088) 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 -- 172.21.15.164:0/434637111 learned_addr learned my addr 172.21.15.164:0/434637111 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): get_auth_request con 0x7f7200134f70 auth_method 0 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_done global_id 254268 payload 274 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 -- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 msgr2=0x7f720013f950 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 --2- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f720013f950 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 -- 172.21.15.164:0/434637111 >> v1:172.21.15.165:6789/0 conn(0x7f720006cb70 legacy=0x7f72000a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 monclient: found mon.noname-a 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.466+0000 7f71fe7fc640 1 -- 172.21.15.164:0/434637111 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7200135380 con 0x7f7200134f70 2024-09-10T16:30:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fe7fc640 1 -- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 msgr2=0x7f720013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fe7fc640 1 --2- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f720013d020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71e80098a0 tx=0x7f71e802a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a72c69dc0d6760 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 1 -- 172.21.15.164:0/434637111 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8037070 con 0x7f7200134f70 2024-09-10T16:30:02.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: dump: 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f72058e6640 5 monclient: authenticate success, global_id 254268 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 1 -- 172.21.15.164:0/434637111 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e8032640 con 0x7f7200134f70 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f72058e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f72058e6640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 1 -- 172.21.15.164:0/434637111 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8032a10 con 0x7f7200134f70 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f72058e6640 4 set_mon_vals no callback set 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.469+0000 7f71fdffb640 10 monclient: dump: 2024-09-10T16:30:02.480 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 10 monclient: shutdown 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 -- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 msgr2=0x7f720013d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 --2- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f720013d020 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f71e80098a0 tx=0x7f71e802a0e0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 -- 172.21.15.164:0/434637111 shutdown_connections 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 --2- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f720013f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 --2- 172.21.15.164:0/434637111 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f720013d020 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 -- 172.21.15.164:0/434637111 >> 172.21.15.164:0/434637111 conn(0x7f72000aa900 msgr2=0x7f72000aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 -- 172.21.15.164:0/434637111 shutdown_connections 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 1 -- 172.21.15.164:0/434637111 wait complete. 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 10 monclient: build_initial_monmap 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.470+0000 7f72058e6640 10 monclient: monmap: 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:02.482 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 1 Processor -- start 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 1 -- start start 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 10 monclient: init 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f720006cb70 0x7f72001cd440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-b con 0x7f720006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f72001cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.471+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-a con 0x7f7200134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:02.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f72001cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): picked mon.noname-c con 0x7f720013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f72000a4db0 con 0x7f7200134f70 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f720009d130 con 0x7f720006cb70 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f720013fe90 con 0x7f720013d560 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f72058e6640 10 monclient(hunting): authenticate will time out at 2180.959038s 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71ff7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 msgr2=0x7f72001cdec0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71ff7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f72001cdec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f72001cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f72001cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34096/0 (socket says 172.21.15.164:34096) 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 -- 172.21.15.164:0/4263783210 learned_addr learned my addr 172.21.15.164:0/4263783210 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): get_auth_request con 0x7f7200134f70 auth_method 0 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f720006cb70 0x7f72001cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): get_auth_request con 0x7f720006cb70 auth_method 0 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient(hunting): handle_auth_done global_id 254271 payload 824 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 msgr2=0x7f72001cdec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f720013d560 0x7f72001cdec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f720006cb70 msgr2=0x7f72001cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f720006cb70 0x7f72001cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 monclient: found mon.noname-a 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.472+0000 7f71fe7fc640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f720009f4e0 con 0x7f7200134f70 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71fe7fc640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 msgr2=0x7f72001cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71fe7fc640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f72001cd980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71e80090b0 tx=0x7f71e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8041070 con 0x7f7200134f70 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: dump: 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.487 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e803c880 con 0x7f7200134f70 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 5 monclient: authenticate success, global_id 254271 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8037040 con 0x7f7200134f70 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71f7fff640 10 monclient: dump: 2024-09-10T16:30:02.488 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 10 monclient: _renew_subs 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f71fd7fa640 4 set_mon_vals no callback set 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f720009f7c0 con 0x7f7200134f70 2024-09-10T16:30:02.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 10 monclient: _renew_subs 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.474+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f720006e720 con 0x7f7200134f70 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f72058e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f72058e6640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 0x7f72001c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f720009e2f0 con 0x7f720009ef20 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f71e8044070 con 0x7f7200134f70 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 0x7f72001c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient: get_auth_request con 0x7f720009ef20 auth_method 0 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71f7fff640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71e003e1c0 0x7f71e0040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 10 monclient(hunting): handle_auth_done global_id 254274 payload 824 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.475+0000 7f71feffd640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f72000a46a0 con 0x7f720009ef20 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.476+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(812..812 src has 677..812) ==== 4783+0+0 (secure 0 0 0) 0x7f71e8073500 con 0x7f7200134f70 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.476+0000 7f71ff7fe640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71e003e1c0 0x7f71e0040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.476+0000 7f71ff7fe640 10 monclient: get_auth_request con 0x7f71e003e1c0 auth_method 0 2024-09-10T16:30:02.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71feffd640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 msgr2=0x7f72001c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71feffd640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 0x7f72001c9f00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71f000bea0 tx=0x7f71f000bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f0003970 con 0x7f720009ef20 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 10 monclient: dump: 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.496 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71f0003970 con 0x7f720009ef20 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 msgr2=0x7f72001c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 0x7f72001c9f00 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f71f000bea0 tx=0x7f71f000bf90 comp rx=0 tx=0).stop 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 msgr2=0x7f72001c9f00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71f7fff640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f720009ef20 0x7f72001c9f00 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.478+0000 7f71ff7fe640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71e003e1c0 0x7f71e0040680 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f71f8003930 tx=0x7f71f8006c40 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f72058e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f72058e6640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 0x7f72001ca850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f72058e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f72001cad90 con 0x7f72001ca440 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 0x7f72001ca850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient: get_auth_request con 0x7f72001ca440 auth_method 0 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.484+0000 7f71feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.485+0000 7f71feffd640 10 monclient(hunting): handle_auth_done global_id 254277 payload 824 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.485+0000 7f71feffd640 1 -- 172.21.15.164:0/4263783210 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f72000770f0 con 0x7f72001ca440 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71feffd640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 msgr2=0x7f72001ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71feffd640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 0x7f72001ca850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f71f000bea0 tx=0x7f71f0004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71f0014420 con 0x7f72001ca440 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 10 monclient: dump: 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.499 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f71f0014420 con 0x7f72001ca440 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 msgr2=0x7f72001ca850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 0x7f72001ca850 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f71f000bea0 tx=0x7f71f0004a70 comp rx=0 tx=0).stop 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 msgr2=0x7f72001ca850 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:02.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71f7fff640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f72001ca440 0x7f72001ca850 secure :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f71f000bea0 tx=0x7f71f0004a70 comp rx=0 tx=0).stop 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 236, 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 91138, 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:02.501 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.502 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:02.503 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:02.504 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.505 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.506 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.507 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.508 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.509 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:02.510 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.511 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.512 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.513 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.513 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:02.513 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71e003e1c0 msgr2=0x7f71e0040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f71e003e1c0 0x7f71e0040680 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f71f8003930 tx=0x7f71f8006c40 comp rx=0 tx=0).stop 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 10 monclient: shutdown 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 msgr2=0x7f72001cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 --2- 172.21.15.164:0/4263783210 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7200134f70 0x7f72001cd980 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f71e80090b0 tx=0x7f71e80046c0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71ff7fe640 1 -- 172.21.15.164:0/4263783210 reap_dead start 2024-09-10T16:30:02.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f71ff7fe640 1 -- 172.21.15.164:0/4263783210 reap_dead start 2024-09-10T16:30:02.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 shutdown_connections 2024-09-10T16:30:02.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.487+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 >> 172.21.15.164:0/4263783210 conn(0x7f72000aa900 msgr2=0x7f72000a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.488+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 shutdown_connections 2024-09-10T16:30:02.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.488+0000 7f72058e6640 1 -- 172.21.15.164:0/4263783210 wait complete. 2024-09-10T16:30:02.515 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:30:02.515 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:30:02.515 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.613+0000 7f2f416be640 10 monclient: get_monmap_and_config 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.613+0000 7f2f416be640 10 monclient: build_initial_monmap 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.613+0000 7f2f416be640 10 monclient: monmap: 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:02.614252+0000 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:02.614252+0000 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:02.615 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 Processor -- start 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 -- start start 2024-09-10T16:30:02.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient: init 2024-09-10T16:30:02.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:02.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-10T16:30:02.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2f3c06cb70 0x7f2f3c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3c06cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 0x7f2f3c0a4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3c0a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2f3c137ca0 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f3c09ce50 con 0x7f2f3c137ca0 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2f3c06db60 con 0x7f2f3c06cb70 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c06cf50 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f416be640 10 monclient(hunting): authenticate will time out at 2181.101660s 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3a7fc640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 msgr2=0x7f2f3c0a4ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3a7fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 0x7f2f3c0a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3b7fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f2f3c137ca0 0x7f2f3c138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:51840/0 (socket says 172.21.15.164:51840) 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3b7fe640 1 -- 172.21.15.164:0/703989670 learned_addr learned my addr 172.21.15.164:0/703989670 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 1 --2- 172.21.15.164:0/703989670 >> v2:172.21.15.165:3300/0 conn(0x7f2f3c06cb70 0x7f2f3c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): get_auth_request con 0x7f2f3c06cb70 auth_method 0 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.614+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 10 monclient(hunting): handle_auth_done global_id 264157 payload 274 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 1 -- 172.21.15.164:0/703989670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 msgr2=0x7f2f3c0a4ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 1 --2- 172.21.15.164:0/703989670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 0x7f2f3c0a4ab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 1 -- 172.21.15.164:0/703989670 >> v1:172.21.15.164:6789/0 conn(0x7f2f3c137ca0 legacy=0x7f2f3c138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 1 monclient: found mon.noname-c 2024-09-10T16:30:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:30:02.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.615+0000 7f2f3affd640 1 -- 172.21.15.164:0/703989670 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c09cb70 con 0x7f2f3c06cb70 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f3affd640 1 -- 172.21.15.164:0/703989670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 msgr2=0x7f2f3c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f3affd640 1 --2- 172.21.15.164:0/703989670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c134f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009d90 tx=0x7f2f3402a4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2b189b180ca664b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 1 -- 172.21.15.164:0/703989670 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34007a30 con 0x7f2f3c06cb70 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: dump: 2024-09-10T16:30:02.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 1 -- 172.21.15.164:0/703989670 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f34007bd0 con 0x7f2f3c06cb70 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 1 -- 172.21.15.164:0/703989670 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031400 con 0x7f2f3c06cb70 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f39ffb640 10 monclient: dump: 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 5 monclient: authenticate success, global_id 264157 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 4 set_mon_vals no callback set 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 10 monclient: shutdown 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 -- 172.21.15.164:0/703989670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 msgr2=0x7f2f3c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 --2- 172.21.15.164:0/703989670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c134f70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2f34009d90 tx=0x7f2f3402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 -- 172.21.15.164:0/703989670 shutdown_connections 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 --2- 172.21.15.164:0/703989670 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c0a46a0 0x7f2f3c0a4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 --2- 172.21.15.164:0/703989670 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c134f70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 -- 172.21.15.164:0/703989670 >> 172.21.15.164:0/703989670 conn(0x7f2f3c0aa900 msgr2=0x7f2f3c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 -- 172.21.15.164:0/703989670 shutdown_connections 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.618+0000 7f2f416be640 1 -- 172.21.15.164:0/703989670 wait complete. 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.619+0000 7f2f416be640 10 monclient: build_initial_monmap 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.619+0000 7f2f416be640 10 monclient: monmap: 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:02.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.619+0000 7f2f416be640 1 Processor -- start 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.619+0000 7f2f416be640 1 -- start start 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.619+0000 7f2f416be640 10 monclient: init 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c1315a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-b con 0x7f2f3c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-a con 0x7f2f3c0a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c137ca0 0x7f2f3c09e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): picked mon.noname-c con 0x7f2f3c137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c1389c0 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c138840 con 0x7f2f3c06cb70 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2f3c140040 con 0x7f2f3c137ca0 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f416be640 10 monclient(hunting): authenticate will time out at 2181.107383s 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3b7fe640 1 -- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c137ca0 msgr2=0x7f2f3c09e050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3301/0 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3b7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c137ca0 0x7f2f3c09e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:34128/0 (socket says 172.21.15.164:34128) 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 -- 172.21.15.164:0/3959639036 learned_addr learned my addr 172.21.15.164:0/3959639036 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): get_auth_request con 0x7f2f3c0a46a0 auth_method 0 2024-09-10T16:30:02.633 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c1315a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): get_auth_request con 0x7f2f3c06cb70 auth_method 0 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.634 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient(hunting): handle_auth_done global_id 254280 payload 824 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c137ca0 msgr2=0x7f2f3c09e050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2f3c137ca0 0x7f2f3c09e050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 msgr2=0x7f2f3c1315a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c06cb70 0x7f2f3c1315a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 monclient: found mon.noname-a 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.620+0000 7f2f3a7fc640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f3c0a07b0 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f3a7fc640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 msgr2=0x7f2f3c09db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f3a7fc640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400b550 tx=0x7f2f2400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24005070 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: dump: 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: _finish_auth 0 2024-09-10T16:30:02.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: _check_auth_tickets 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f24015070 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 5 monclient: authenticate success, global_id 254280 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f24005450 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f337fe640 10 monclient: dump: 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.637 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 10 monclient: _renew_subs 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f397fa640 4 set_mon_vals no callback set 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f3c06e720 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 10 monclient: _renew_subs 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.623+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f3c09eb30 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2f240055f0 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f416be640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f416be640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 0x7f2f3c1cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c0a0050 con 0x7f2f3c1cd2d0 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f337fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2f1c03e1c0 0x7f2f1c040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(812..812 src has 677..812) ==== 4783+0+0 (secure 0 0 0) 0x7f2f24052ab0 con 0x7f2f3c0a46a0 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 0x7f2f3c1cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient: get_auth_request con 0x7f2f3c1cd2d0 auth_method 0 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3b7fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2f1c03e1c0 0x7f2f1c040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3b7fe640 10 monclient: get_auth_request con 0x7f2f1c03e1c0 auth_method 0 2024-09-10T16:30:02.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 10 monclient(hunting): handle_auth_done global_id 264163 payload 824 2024-09-10T16:30:02.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.624+0000 7f2f3affd640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f3c134f70 con 0x7f2f3c1cd2d0 2024-09-10T16:30:02.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f3affd640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 msgr2=0x7f2f3c1cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f3affd640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 0x7f2f3c1cd6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2f34034000 tx=0x7f2f34004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f3b7fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2f1c03e1c0 0x7f2f1c040680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c007fd0 tx=0x7f2f2c007aa0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34031400 con 0x7f2f3c1cd2d0 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 10 monclient: dump: 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2f34031400 con 0x7f2f3c1cd2d0 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 msgr2=0x7f2f3c1cd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 0x7f2f3c1cd6b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2f34034000 tx=0x7f2f34004a40 comp rx=0 tx=0).stop 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 msgr2=0x7f2f3c1cd6b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.627+0000 7f2f337fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cd2d0 0x7f2f3c1cd6b0 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2f34034000 tx=0x7f2f34004a40 comp rx=0 tx=0).stop 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f416be640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f416be640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 0x7f2f3c12d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f416be640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2f3c09efd0 con 0x7f2f3c1cdbf0 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 0x7f2f3c12d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient: get_auth_request con 0x7f2f3c1cdbf0 auth_method 0 2024-09-10T16:30:02.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.633+0000 7f2f3affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.634+0000 7f2f3affd640 10 monclient(hunting): handle_auth_done global_id 264166 payload 824 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.634+0000 7f2f3affd640 1 -- 172.21.15.164:0/3959639036 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f3c0770f0 con 0x7f2f3c1cdbf0 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f3affd640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 msgr2=0x7f2f3c12d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f3affd640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 0x7f2f3c12d340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c069c20 tx=0x7f2f34030000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f34004540 con 0x7f2f3c1cdbf0 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 10 monclient: dump: 2024-09-10T16:30:02.645 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7f2f34004540 con 0x7f2f3c1cdbf0 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:02.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 msgr2=0x7f2f3c12d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 0x7f2f3c12d340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c069c20 tx=0x7f2f34030000 comp rx=0 tx=0).stop 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 msgr2=0x7f2f3c12d340 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.636+0000 7f2f337fe640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2f3c1cdbf0 0x7f2f3c12d340 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2f3c069c20 tx=0x7f2f34030000 comp rx=0 tx=0).stop 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 236, 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 24778, 2024-09-10T16:30:02.647 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: 1 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.648 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.649 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:02.650 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:02.651 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.652 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:02.653 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.654 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:02.655 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.656 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.657 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.658 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.659 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:02.660 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2f1c03e1c0 msgr2=0x7f2f1c040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2f1c03e1c0 0x7f2f1c040680 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7f2f2c007fd0 tx=0x7f2f2c007aa0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 10 monclient: shutdown 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 msgr2=0x7f2f3c09db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2f2400b550 tx=0x7f2f2400ba10 comp rx=0 tx=0).stop 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f3b7fe640 1 -- 172.21.15.164:0/3959639036 reap_dead start 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 shutdown_connections 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 --2- 172.21.15.164:0/3959639036 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2f3c0a46a0 0x7f2f3c09db10 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:02.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 >> 172.21.15.164:0/3959639036 conn(0x7f2f3c0aa900 msgr2=0x7f2f3c133050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:02.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 shutdown_connections 2024-09-10T16:30:02.663 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:02.637+0000 7f2f416be640 1 -- 172.21.15.164:0/3959639036 wait complete. 2024-09-10T16:30:02.663 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:30:02.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:30:03.383 INFO:tasks.rados.rados.0.smithi165.stdout:3617: done (2 left) 2024-09-10T16:30:03.383 INFO:tasks.rados.rados.0.smithi165.stdout:3618: write (excl) oid 40 current snap is 405 2024-09-10T16:30:03.383 INFO:tasks.rados.rados.0.smithi165.stdout:3618: seq_num 1254 ranges {675978=644138,1897949=650107} 2024-09-10T16:30:03.390 INFO:tasks.rados.rados.0.smithi165.stdout:3618: writing smithi16534841-40 from 675978 to 1320116 tid 1 2024-09-10T16:30:03.397 INFO:tasks.rados.rados.0.smithi165.stdout:3618: writing smithi16534841-40 from 1897949 to 2548056 tid 2 2024-09-10T16:30:03.397 INFO:tasks.rados.rados.0.smithi165.stdout:3619: snap_remove snap 371 2024-09-10T16:30:03.403 INFO:tasks.ceph.osd.4.smithi165.stderr:2024-09-10T16:30:03.402+0000 7f1d7173e640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 197 after trimming was started 2024-09-10T16:30:04.386 INFO:tasks.rados.rados.0.smithi165.stdout:3619: done (3 left) 2024-09-10T16:30:04.386 INFO:tasks.rados.rados.0.smithi165.stdout:3620: snap_remove snap 400 2024-09-10T16:30:05.390 INFO:tasks.rados.rados.0.smithi165.stdout:3620: done (3 left) 2024-09-10T16:30:05.390 INFO:tasks.rados.rados.0.smithi165.stdout:3621: write (excl) oid 4 current snap is 405 2024-09-10T16:30:05.390 INFO:tasks.rados.rados.0.smithi165.stdout:3621: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2024-09-10T16:30:05.395 INFO:tasks.rados.rados.0.smithi165.stdout:3621: writing smithi16534841-4 from 692785 to 1128525 tid 1 2024-09-10T16:30:05.403 INFO:tasks.rados.rados.0.smithi165.stdout:3621: writing smithi16534841-4 from 1672784 to 2466549 tid 2 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3621: writing smithi16534841-4 from 2897503 to 3092785 tid 3 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3622: read oid 32 snap 383 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3622: expect deleted 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3623: copy_from oid 43 from oid 42 current snap is 405 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: finishing write tid 1 to smithi16534841-9 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: finishing write tid 2 to smithi16534841-9 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: finishing write tid 3 to smithi16534841-9 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: finishing write tid 4 to smithi16534841-9 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: oid 9 updating version 0 to 1897 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: oid 9 updating version 1897 to 1898 2024-09-10T16:30:05.409 INFO:tasks.rados.rados.0.smithi165.stdout:3613: oid 9 version 1898 is already newer than 1896 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1898 (ObjNum 1252 snap 405 seq_num 1252) dirty exists 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3613: left oid 9 (ObjNum 1252 snap 405 seq_num 1252) 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: finishing write tid 1 to smithi16534841-40 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: finishing write tid 2 to smithi16534841-40 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3613: done (5 left) 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3614: done (4 left) 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3624: rollback oid 35 current snap is 405 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 35 to 365 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: finishing write tid 3 to smithi16534841-40 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: finishing write tid 4 to smithi16534841-40 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: oid 40 updating version 0 to 1900 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: oid 40 updating version 1900 to 1901 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3618: oid 40 version 1901 is already newer than 1899 2024-09-10T16:30:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1901 (ObjNum 1254 snap 405 seq_num 1254) dirty exists 2024-09-10T16:30:05.411 INFO:tasks.rados.rados.0.smithi165.stdout:3618: left oid 40 (ObjNum 1254 snap 405 seq_num 1254) 2024-09-10T16:30:05.411 INFO:tasks.rados.rados.0.smithi165.stdout:3621: finishing write tid 1 to smithi16534841-4 2024-09-10T16:30:05.411 INFO:tasks.rados.rados.0.smithi165.stdout:3618: done (4 left) 2024-09-10T16:30:05.411 INFO:tasks.rados.rados.0.smithi165.stdout:3625: write (excl) oid 30 current snap is 405 2024-09-10T16:30:05.411 INFO:tasks.rados.rados.0.smithi165.stdout:3625: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2024-09-10T16:30:05.415 INFO:tasks.rados.rados.0.smithi165.stdout:3625: writing smithi16534841-30 from 709592 to 1220581 tid 1 2024-09-10T16:30:05.422 INFO:tasks.rados.rados.0.smithi165.stdout:3625: writing smithi16534841-30 from 1731266 to 2385042 tid 2 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3625: writing smithi16534841-30 from 3041279 to 3109592 tid 3 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3626: read oid 49 snap -1 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3626: expect deleted 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3627: read oid 16 snap -1 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3627: expect (ObjNum 1246 snap 403 seq_num 1246) 2024-09-10T16:30:05.427 INFO:tasks.rados.rados.0.smithi165.stdout:3628: rollback oid 47 current snap is 405 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 47 to 365 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: finishing write tid 2 to smithi16534841-4 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: finishing write tid 3 to smithi16534841-4 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: finishing write tid 4 to smithi16534841-4 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: finishing write tid 5 to smithi16534841-4 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: oid 4 updating version 0 to 1859 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: oid 4 version 1859 is already newer than 1858 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: oid 4 updating version 1859 to 1860 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: oid 4 version 1860 is already newer than 1857 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1860 (ObjNum 1255 snap 405 seq_num 1255) dirty exists 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3621: left oid 4 (ObjNum 1255 snap 405 seq_num 1255) 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3624: finishing rollback tid 0 to smithi16534841-35 2024-09-10T16:30:05.428 INFO:tasks.rados.rados.0.smithi165.stdout:3624: finishing rollback tid 1 to smithi16534841-35 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3624: finishing rollback tid 2 to smithi16534841-35 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2016 (ObjNum 1140 snap 353 seq_num 1140) dirty exists 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3623: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3625: finishing write tid 1 to smithi16534841-30 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3621: done (7 left) 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3622: done (6 left) 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3624: done (5 left) 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3629: read oid 12 snap -1 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3629: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3630: read oid 32 snap -1 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3630: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3631: write (excl) oid 15 current snap is 405 2024-09-10T16:30:05.429 INFO:tasks.rados.rados.0.smithi165.stdout:3631: seq_num 1257 ranges {726399=586238,2073395=797434} 2024-09-10T16:30:05.434 INFO:tasks.rados.rados.0.smithi165.stdout:3631: writing smithi16534841-15 from 726399 to 1312637 tid 1 2024-09-10T16:30:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:3631: writing smithi16534841-15 from 2073395 to 2870829 tid 2 2024-09-10T16:30:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:3632: copy_from oid 2 from oid 28 current snap is 405 2024-09-10T16:30:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:3623: finishing copy_from to smithi16534841-43 2024-09-10T16:30:05.443 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1419 (ObjNum 1218 snap 392 seq_num 1218) dirty exists 2024-09-10T16:30:05.444 INFO:tasks.rados.rados.0.smithi165.stdout:3629: expect (ObjNum 1230 snap 398 seq_num 1230) 2024-09-10T16:30:05.654 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1785954 2024-09-10T16:30:05.690 INFO:tasks.rados.rados.0.smithi165.stdout:3630: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:30:05.694 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1778866 2024-09-10T16:30:05.750 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2610586 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:3625: finishing write tid 2 to smithi16534841-30 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:3628: finishing rollback tid 0 to smithi16534841-47 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:3628: finishing rollback tid 1 to smithi16534841-47 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:3628: finishing rollback tid 2 to smithi16534841-47 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 1967 (ObjNum 1138 snap 351 seq_num 1138) dirty exists 2024-09-10T16:30:05.756 INFO:tasks.rados.rados.0.smithi165.stdout:3627: expect (ObjNum 1246 snap 403 seq_num 1246) 2024-09-10T16:30:05.774 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2641592 2024-09-10T16:30:05.786 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:05.835 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 667875 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: finishing write tid 3 to smithi16534841-30 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: finishing write tid 4 to smithi16534841-30 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: finishing write tid 5 to smithi16534841-30 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: oid 30 updating version 0 to 2301 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: oid 30 version 2301 is already newer than 2298 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: oid 30 version 2301 is already newer than 2299 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: oid 30 version 2301 is already newer than 2300 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2301 (ObjNum 1256 snap 405 seq_num 1256) dirty exists 2024-09-10T16:30:05.873 INFO:tasks.rados.rados.0.smithi165.stdout:3625: left oid 30 (ObjNum 1256 snap 405 seq_num 1256) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3631: finishing write tid 1 to smithi16534841-15 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3623: done (8 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3625: done (7 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3626: done (6 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3627: done (5 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3628: done (4 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3629: done (3 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3630: done (2 left) 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3633: read oid 23 snap -1 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3633: expect deleted 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3634: copy_from oid 22 from oid 32 current snap is 405 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3631: finishing write tid 2 to smithi16534841-15 2024-09-10T16:30:05.874 INFO:tasks.rados.rados.0.smithi165.stdout:3631: finishing write tid 3 to smithi16534841-15 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: finishing write tid 4 to smithi16534841-15 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: oid 15 updating version 0 to 1861 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: oid 15 updating version 1861 to 1863 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: oid 15 version 1863 is already newer than 1862 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1863 (ObjNum 1257 snap 405 seq_num 1257) dirty exists 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: left oid 15 (ObjNum 1257 snap 405 seq_num 1257) 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3632: finishing copy_from racing read to smithi16534841-2 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3632: finishing copy_from to smithi16534841-2 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 1286 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3631: done (3 left) 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3632: done (2 left) 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3635: write oid 24 current snap is 405 2024-09-10T16:30:05.875 INFO:tasks.rados.rados.0.smithi165.stdout:3635: seq_num 1258 ranges {743206=661487,2131877=541092} 2024-09-10T16:30:05.879 INFO:tasks.rados.rados.0.smithi165.stdout:3635: writing smithi16534841-24 from 743206 to 1404693 tid 1 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3635: writing smithi16534841-24 from 2131877 to 2672969 tid 2 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3636: copy_from oid 26 from oid 18 current snap is 405 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3634: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3634: finishing copy_from to smithi16534841-22 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 1632 (ObjNum 1236 snap 400 seq_num 1236) dirty exists 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3635: finishing write tid 1 to smithi16534841-24 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3633: done (3 left) 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3634: done (2 left) 2024-09-10T16:30:05.887 INFO:tasks.rados.rados.0.smithi165.stdout:3637: snap_remove snap 354 2024-09-10T16:30:06.375 INFO:tasks.rados.rados.0.smithi165.stdout:3637: done (2 left) 2024-09-10T16:30:06.375 INFO:tasks.rados.rados.0.smithi165.stdout:3638: snap_remove snap 380 2024-09-10T16:30:07.375 INFO:tasks.rados.rados.0.smithi165.stdout:3638: done (2 left) 2024-09-10T16:30:07.375 INFO:tasks.rados.rados.0.smithi165.stdout:3639: snap_remove snap 395 2024-09-10T16:30:08.380 INFO:tasks.rados.rados.0.smithi165.stdout:3639: done (2 left) 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3640: delete oid 3 current snap is 405 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: finishing write tid 2 to smithi16534841-24 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: finishing write tid 3 to smithi16534841-24 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: finishing write tid 4 to smithi16534841-24 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: oid 24 updating version 0 to 1978 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: oid 24 version 1978 is already newer than 1976 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: oid 24 version 1978 is already newer than 1977 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1978 (ObjNum 1258 snap 405 seq_num 1258) dirty exists 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3635: left oid 24 (ObjNum 1258 snap 405 seq_num 1258) 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3636: finishing copy_from racing read to smithi16534841-26 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:3636: finishing copy_from to smithi16534841-26 2024-09-10T16:30:08.381 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 1851 (ObjNum 1164 snap 365 seq_num 1164) dirty exists 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3635: done (2 left) 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3636: done (1 left) 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3640: done (0 left) 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3641: read oid 20 snap -1 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3641: expect (ObjNum 1226 snap 395 seq_num 1226) 2024-09-10T16:30:08.389 INFO:tasks.rados.rados.0.smithi165.stdout:3642: delete oid 44 current snap is 405 2024-09-10T16:30:08.393 INFO:tasks.rados.rados.0.smithi165.stdout:3642: done (1 left) 2024-09-10T16:30:08.393 INFO:tasks.rados.rados.0.smithi165.stdout:3643: write (excl) oid 25 current snap is 405 2024-09-10T16:30:08.394 INFO:tasks.rados.rados.0.smithi165.stdout:3643: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2024-09-10T16:30:08.401 INFO:tasks.rados.rados.0.smithi165.stdout:3643: writing smithi16534841-25 from 793627 to 1280861 tid 1 2024-09-10T16:30:08.406 INFO:tasks.rados.rados.0.smithi165.stdout:3643: writing smithi16534841-25 from 1790970 to 2195742 tid 2 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:3643: writing smithi16534841-25 from 2829335 to 3193627 tid 3 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:3644: read oid 42 snap -1 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:3644: expect (ObjNum 1218 snap 392 seq_num 1218) 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:3645: rollback oid 46 current snap is 405 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 381 2024-09-10T16:30:08.412 INFO:tasks.rados.rados.0.smithi165.stdout:3641: expect (ObjNum 1226 snap 395 seq_num 1226) 2024-09-10T16:30:08.425 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2308036 2024-09-10T16:30:08.427 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3643: finishing write tid 1 to smithi16534841-25 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3643: finishing write tid 2 to smithi16534841-25 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3641: done (3 left) 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3646: read oid 8 snap -1 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3646: expect deleted 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3647: rollback oid 43 current snap is 405 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 43 to 389 2024-09-10T16:30:08.473 INFO:tasks.rados.rados.0.smithi165.stdout:3645: finishing rollback tid 0 to smithi16534841-46 2024-09-10T16:30:08.474 INFO:tasks.rados.rados.0.smithi165.stdout:3643: finishing write tid 3 to smithi16534841-25 2024-09-10T16:30:08.474 INFO:tasks.rados.rados.0.smithi165.stdout:3645: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:30:08.474 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1839 (ObjNum 129 snap 0 seq_num 1163092192) dirty dne 2024-09-10T16:30:08.474 INFO:tasks.rados.rados.0.smithi165.stdout:3644: expect (ObjNum 1218 snap 392 seq_num 1218) 2024-09-10T16:30:08.474 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:08.499 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: finishing write tid 4 to smithi16534841-25 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: finishing write tid 5 to smithi16534841-25 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: oid 25 updating version 0 to 1983 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: oid 25 version 1983 is already newer than 1980 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: oid 25 version 1983 is already newer than 1981 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:3643: oid 25 version 1983 is already newer than 1982 2024-09-10T16:30:08.552 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 1983 (ObjNum 1261 snap 405 seq_num 1261) dirty exists 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3643: left oid 25 (ObjNum 1261 snap 405 seq_num 1261) 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3643: done (4 left) 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3644: done (3 left) 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3645: done (2 left) 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3648: read oid 24 snap -1 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3648: expect (ObjNum 1258 snap 405 seq_num 1258) 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3649: copy_from oid 36 from oid 10 current snap is 405 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3647: finishing rollback tid 0 to smithi16534841-43 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3647: finishing rollback tid 1 to smithi16534841-43 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3647: finishing rollback tid 2 to smithi16534841-43 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1713 (ObjNum 1152 snap 362 seq_num 1152) dirty exists 2024-09-10T16:30:08.553 INFO:tasks.rados.rados.0.smithi165.stdout:3646: done (3 left) 2024-09-10T16:30:08.554 INFO:tasks.rados.rados.0.smithi165.stdout:3647: done (2 left) 2024-09-10T16:30:08.554 INFO:tasks.rados.rados.0.smithi165.stdout:3650: snap_remove snap 362 2024-09-10T16:30:09.386 INFO:tasks.rados.rados.0.smithi165.stdout:3650: done (2 left) 2024-09-10T16:30:09.387 INFO:tasks.rados.rados.0.smithi165.stdout:3651: delete oid 13 current snap is 405 2024-09-10T16:30:09.387 INFO:tasks.rados.rados.0.smithi165.stdout:3648: expect (ObjNum 1258 snap 405 seq_num 1258) 2024-09-10T16:30:09.407 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2195085 2024-09-10T16:30:09.423 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 371524 2024-09-10T16:30:09.487 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2254836 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3649: finishing copy_from racing read to smithi16534841-36 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3649: finishing copy_from to smithi16534841-36 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1695 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3648: done (2 left) 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3649: done (1 left) 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3651: done (0 left) 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3652: read oid 32 snap -1 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3652: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:30:09.503 INFO:tasks.rados.rados.0.smithi165.stdout:3653: snap_create 2024-09-10T16:30:09.509 INFO:tasks.rados.rados.0.smithi165.stdout:3652: expect (ObjNum 1236 snap 400 seq_num 1236) 2024-09-10T16:30:09.533 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:09.567 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1589790 2024-09-10T16:30:10.393 INFO:tasks.rados.rados.0.smithi165.stdout:3652: done (1 left) 2024-09-10T16:30:10.393 INFO:tasks.rados.rados.0.smithi165.stdout:3653: done (0 left) 2024-09-10T16:30:10.393 INFO:tasks.rados.rados.0.smithi165.stdout:3654: write oid 17 current snap is 406 2024-09-10T16:30:10.393 INFO:tasks.rados.rados.0.smithi165.stdout:3654: seq_num 1263 ranges {427241=754085,1624287=408441,2600534=549964} 2024-09-10T16:30:10.399 INFO:tasks.rados.rados.0.smithi165.stdout:3654: writing smithi16534841-17 from 427241 to 1181326 tid 1 2024-09-10T16:30:10.403 INFO:tasks.rados.rados.0.smithi165.stdout:3654: writing smithi16534841-17 from 1624287 to 2032728 tid 2 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3654: writing smithi16534841-17 from 2600534 to 3150498 tid 3 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3655: read oid 38 snap -1 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3655: expect deleted 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3656: read oid 45 snap -1 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3656: expect (ObjNum 1251 snap 405 seq_num 1251) 2024-09-10T16:30:10.410 INFO:tasks.rados.rados.0.smithi165.stdout:3657: snap_remove snap 383 2024-09-10T16:30:11.377 INFO:tasks.rados.rados.0.smithi165.stdout:3657: done (3 left) 2024-09-10T16:30:11.377 INFO:tasks.rados.rados.0.smithi165.stdout:3658: delete oid 5 current snap is 406 2024-09-10T16:30:11.377 INFO:tasks.rados.rados.0.smithi165.stdout:3654: finishing write tid 1 to smithi16534841-17 2024-09-10T16:30:11.377 INFO:tasks.rados.rados.0.smithi165.stdout:3654: finishing write tid 2 to smithi16534841-17 2024-09-10T16:30:11.377 INFO:tasks.rados.rados.0.smithi165.stdout:3654: finishing write tid 3 to smithi16534841-17 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: finishing write tid 4 to smithi16534841-17 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: finishing write tid 5 to smithi16534841-17 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: oid 17 updating version 0 to 1845 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: oid 17 updating version 1845 to 1846 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: oid 17 version 1846 is already newer than 1844 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: oid 17 updating version 1846 to 1847 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1847 (ObjNum 1263 snap 406 seq_num 1263) dirty exists 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3654: left oid 17 (ObjNum 1263 snap 406 seq_num 1263) 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:3656: expect (ObjNum 1251 snap 405 seq_num 1251) 2024-09-10T16:30:11.378 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 142016 2024-09-10T16:30:11.622 INFO:tasks.rados.rados.0.smithi165.stdout:3654: done (3 left) 2024-09-10T16:30:11.623 INFO:tasks.rados.rados.0.smithi165.stdout:3655: done (2 left) 2024-09-10T16:30:11.623 INFO:tasks.rados.rados.0.smithi165.stdout:3656: done (1 left) 2024-09-10T16:30:11.623 INFO:tasks.rados.rados.0.smithi165.stdout:3658: done (0 left) 2024-09-10T16:30:11.623 INFO:tasks.rados.rados.0.smithi165.stdout:3659: write (excl) oid 21 current snap is 406 2024-09-10T16:30:11.623 INFO:tasks.rados.rados.0.smithi165.stdout:3659: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2024-09-10T16:30:11.627 INFO:tasks.rados.rados.0.smithi165.stdout:3659: writing smithi16534841-21 from 460855 to 965438 tid 1 2024-09-10T16:30:11.633 INFO:tasks.rados.rados.0.smithi165.stdout:3659: writing smithi16534841-21 from 1624898 to 2037008 tid 2 2024-09-10T16:30:11.639 INFO:tasks.rados.rados.0.smithi165.stdout:3659: writing smithi16534841-21 from 2539027 to 3111535 tid 3 2024-09-10T16:30:11.639 INFO:tasks.rados.rados.0.smithi165.stdout:3660: write (excl) oid 26 current snap is 406 2024-09-10T16:30:11.639 INFO:tasks.rados.rados.0.smithi165.stdout:3659: finishing write tid 1 to smithi16534841-21 2024-09-10T16:30:11.640 INFO:tasks.rados.rados.0.smithi165.stdout:3660: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2024-09-10T16:30:11.645 INFO:tasks.rados.rados.0.smithi165.stdout:3660: writing smithi16534841-26 from 477662 to 1057494 tid 1 2024-09-10T16:30:11.651 INFO:tasks.rados.rados.0.smithi165.stdout:3660: writing smithi16534841-26 from 1683380 to 2239148 tid 2 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3660: writing smithi16534841-26 from 2966450 to 3277662 tid 3 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3661: copy_from oid 30 from oid 47 current snap is 406 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: finishing write tid 2 to smithi16534841-21 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: finishing write tid 3 to smithi16534841-21 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: finishing write tid 4 to smithi16534841-21 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: finishing write tid 5 to smithi16534841-21 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: oid 21 updating version 0 to 2001 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: oid 21 updating version 2001 to 2003 2024-09-10T16:30:11.658 INFO:tasks.rados.rados.0.smithi165.stdout:3659: oid 21 version 2003 is already newer than 2002 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3659: oid 21 version 2003 is already newer than 2000 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 2003 (ObjNum 1265 snap 406 seq_num 1265) dirty exists 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3659: left oid 21 (ObjNum 1265 snap 406 seq_num 1265) 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3660: finishing write tid 1 to smithi16534841-26 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3660: finishing write tid 2 to smithi16534841-26 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3659: done (2 left) 2024-09-10T16:30:11.659 INFO:tasks.rados.rados.0.smithi165.stdout:3662: delete oid 42 current snap is 406 2024-09-10T16:30:11.661 INFO:tasks.rados.rados.0.smithi165.stdout:3662: done (2 left) 2024-09-10T16:30:11.661 INFO:tasks.rados.rados.0.smithi165.stdout:3663: snap_remove snap 368 2024-09-10T16:30:12.397 INFO:tasks.rados.rados.0.smithi165.stdout:3663: done (2 left) 2024-09-10T16:30:12.397 INFO:tasks.rados.rados.0.smithi165.stdout:3664: rollback oid 28 current snap is 406 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 28 to 405 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: finishing write tid 3 to smithi16534841-26 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: finishing write tid 4 to smithi16534841-26 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: finishing write tid 5 to smithi16534841-26 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: oid 26 updating version 0 to 2349 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: oid 26 updating version 2349 to 2350 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: oid 26 version 2350 is already newer than 2348 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: oid 26 version 2350 is already newer than 2347 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2350 (ObjNum 1266 snap 406 seq_num 1266) dirty exists 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3660: left oid 26 (ObjNum 1266 snap 406 seq_num 1266) 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3661: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:3661: finishing copy_from to smithi16534841-30 2024-09-10T16:30:12.398 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1967 (ObjNum 1138 snap 351 seq_num 1138) dirty exists 2024-09-10T16:30:12.399 INFO:tasks.rados.rados.0.smithi165.stdout:3660: done (2 left) 2024-09-10T16:30:12.399 INFO:tasks.rados.rados.0.smithi165.stdout:3661: done (1 left) 2024-09-10T16:30:12.399 INFO:tasks.rados.rados.0.smithi165.stdout:3665: snap_remove snap 369 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3665: done (1 left) 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3666: read oid 48 snap -1 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3666: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3667: write oid 34 current snap is 406 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3664: finishing rollback tid 0 to smithi16534841-28 2024-09-10T16:30:13.402 INFO:tasks.rados.rados.0.smithi165.stdout:3667: seq_num 1268 ranges {511276=730330,1800344=559437,3021296=289980} 2024-09-10T16:30:13.408 INFO:tasks.rados.rados.0.smithi165.stdout:3667: writing smithi16534841-34 from 511276 to 1241606 tid 1 2024-09-10T16:30:13.415 INFO:tasks.rados.rados.0.smithi165.stdout:3667: writing smithi16534841-34 from 1800344 to 2359781 tid 2 2024-09-10T16:30:13.420 INFO:tasks.rados.rados.0.smithi165.stdout:3667: writing smithi16534841-34 from 3021296 to 3311276 tid 3 2024-09-10T16:30:13.421 INFO:tasks.rados.rados.0.smithi165.stdout:3668: snap_remove snap 370 2024-09-10T16:30:14.407 INFO:tasks.rados.rados.0.smithi165.stdout:3668: done (3 left) 2024-09-10T16:30:14.408 INFO:tasks.rados.rados.0.smithi165.stdout:3669: delete oid 30 current snap is 406 2024-09-10T16:30:14.408 INFO:tasks.rados.rados.0.smithi165.stdout:3664: finishing rollback tid 1 to smithi16534841-28 2024-09-10T16:30:14.408 INFO:tasks.rados.rados.0.smithi165.stdout:3664: finishing rollback tid 2 to smithi16534841-28 2024-09-10T16:30:14.408 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1289 (ObjNum 1099 snap 334 seq_num 1099) dirty exists 2024-09-10T16:30:14.408 INFO:tasks.rados.rados.0.smithi165.stdout:3666: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:30:14.417 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 927839 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: finishing write tid 1 to smithi16534841-34 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: finishing write tid 2 to smithi16534841-34 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: finishing write tid 3 to smithi16534841-34 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: finishing write tid 4 to smithi16534841-34 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: finishing write tid 5 to smithi16534841-34 2024-09-10T16:30:14.635 INFO:tasks.rados.rados.0.smithi165.stdout:3667: oid 34 updating version 0 to 1594 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3667: oid 34 updating version 1594 to 1596 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3667: oid 34 version 1596 is already newer than 1595 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3667: oid 34 version 1596 is already newer than 1593 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1596 (ObjNum 1268 snap 406 seq_num 1268) dirty exists 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3667: left oid 34 (ObjNum 1268 snap 406 seq_num 1268) 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3664: done (3 left) 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3666: done (2 left) 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3667: done (1 left) 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3669: done (0 left) 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3670: copy_from oid 24 from oid 40 current snap is 406 2024-09-10T16:30:14.636 INFO:tasks.rados.rados.0.smithi165.stdout:3671: snap_create 2024-09-10T16:30:14.643 INFO:tasks.rados.rados.0.smithi165.stdout:3670: finishing copy_from racing read to smithi16534841-24 2024-09-10T16:30:14.647 INFO:tasks.rados.rados.0.smithi165.stdout:3670: finishing copy_from to smithi16534841-24 2024-09-10T16:30:14.647 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 1901 (ObjNum 1254 snap 405 seq_num 1254) dirty exists 2024-09-10T16:30:15.410 INFO:tasks.rados.rados.0.smithi165.stdout:3670: done (1 left) 2024-09-10T16:30:15.411 INFO:tasks.rados.rados.0.smithi165.stdout:3671: done (0 left) 2024-09-10T16:30:15.411 INFO:tasks.rados.rados.0.smithi165.stdout:3672: snap_remove snap 388 2024-09-10T16:30:16.378 INFO:tasks.rados.rados.0.smithi165.stdout:3672: done (0 left) 2024-09-10T16:30:16.378 INFO:tasks.rados.rados.0.smithi165.stdout:3673: write oid 19 current snap is 407 2024-09-10T16:30:16.378 INFO:tasks.rados.rados.0.smithi165.stdout:3673: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2024-09-10T16:30:16.384 INFO:tasks.rados.rados.0.smithi165.stdout:3673: writing smithi16534841-19 from 544890 to 1142071 tid 1 2024-09-10T16:30:16.391 INFO:tasks.rados.rados.0.smithi165.stdout:3673: writing smithi16534841-19 from 1917308 to 2480414 tid 2 2024-09-10T16:30:16.396 INFO:tasks.rados.rados.0.smithi165.stdout:3673: writing smithi16534841-19 from 3076142 to 3344890 tid 3 2024-09-10T16:30:16.397 INFO:tasks.rados.rados.0.smithi165.stdout:3674: write (excl) oid 33 current snap is 407 2024-09-10T16:30:16.397 INFO:tasks.rados.rados.0.smithi165.stdout:3674: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2024-09-10T16:30:16.403 INFO:tasks.rados.rados.0.smithi165.stdout:3674: writing smithi16534841-33 from 561697 to 1234127 tid 1 2024-09-10T16:30:16.412 INFO:tasks.rados.rados.0.smithi165.stdout:3674: writing smithi16534841-33 from 1975790 to 2682554 tid 2 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3674: writing smithi16534841-33 from 3103565 to 3361697 tid 3 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3675: snap_create 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: finishing write tid 1 to smithi16534841-19 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: finishing write tid 2 to smithi16534841-19 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: finishing write tid 3 to smithi16534841-19 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: finishing write tid 4 to smithi16534841-19 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: finishing write tid 5 to smithi16534841-19 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: oid 19 updating version 0 to 2372 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: oid 19 version 2372 is already newer than 2371 2024-09-10T16:30:16.416 INFO:tasks.rados.rados.0.smithi165.stdout:3673: oid 19 updating version 2372 to 2377 2024-09-10T16:30:16.417 INFO:tasks.rados.rados.0.smithi165.stdout:3673: oid 19 updating version 2377 to 2378 2024-09-10T16:30:16.417 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2378 (ObjNum 1270 snap 407 seq_num 1270) dirty exists 2024-09-10T16:30:16.417 INFO:tasks.rados.rados.0.smithi165.stdout:3673: left oid 19 (ObjNum 1270 snap 407 seq_num 1270) 2024-09-10T16:30:16.425 INFO:tasks.rados.rados.0.smithi165.stdout:3674: finishing write tid 1 to smithi16534841-33 2024-09-10T16:30:16.425 INFO:tasks.rados.rados.0.smithi165.stdout:3674: finishing write tid 2 to smithi16534841-33 2024-09-10T16:30:16.425 INFO:tasks.rados.rados.0.smithi165.stdout:3674: finishing write tid 3 to smithi16534841-33 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: finishing write tid 4 to smithi16534841-33 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: finishing write tid 5 to smithi16534841-33 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: oid 33 updating version 0 to 2025 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: oid 33 version 2025 is already newer than 2023 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: oid 33 updating version 2025 to 2026 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:3674: oid 33 version 2026 is already newer than 2024 2024-09-10T16:30:16.426 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 2026 (ObjNum 1271 snap 407 seq_num 1271) dirty exists 2024-09-10T16:30:16.427 INFO:tasks.rados.rados.0.smithi165.stdout:3674: left oid 33 (ObjNum 1271 snap 407 seq_num 1271) 2024-09-10T16:30:17.417 INFO:tasks.rados.rados.0.smithi165.stdout:3673: done (2 left) 2024-09-10T16:30:17.418 INFO:tasks.rados.rados.0.smithi165.stdout:3674: done (1 left) 2024-09-10T16:30:17.418 INFO:tasks.rados.rados.0.smithi165.stdout:3675: done (0 left) 2024-09-10T16:30:17.418 INFO:tasks.rados.rados.0.smithi165.stdout:3676: write (excl) oid 14 current snap is 408 2024-09-10T16:30:17.418 INFO:tasks.rados.rados.0.smithi165.stdout:3676: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2024-09-10T16:30:17.424 INFO:tasks.rados.rados.0.smithi165.stdout:3676: writing smithi16534841-14 from 578504 to 1326183 tid 1 2024-09-10T16:30:17.432 INFO:tasks.rados.rados.0.smithi165.stdout:3676: writing smithi16534841-14 from 2034272 to 2601047 tid 2 2024-09-10T16:30:17.436 INFO:tasks.rados.rados.0.smithi165.stdout:3676: writing smithi16534841-14 from 3130988 to 3378504 tid 3 2024-09-10T16:30:17.436 INFO:tasks.rados.rados.0.smithi165.stdout:3677: delete oid 1 current snap is 408 2024-09-10T16:30:17.437 INFO:tasks.rados.rados.0.smithi165.stdout:3676: finishing write tid 1 to smithi16534841-14 2024-09-10T16:30:17.437 INFO:tasks.rados.rados.0.smithi165.stdout:3676: finishing write tid 2 to smithi16534841-14 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: finishing write tid 3 to smithi16534841-14 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: finishing write tid 4 to smithi16534841-14 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: finishing write tid 5 to smithi16534841-14 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: oid 14 updating version 0 to 1903 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: oid 14 updating version 1903 to 1904 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: oid 14 updating version 1904 to 1905 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:3676: oid 14 version 1905 is already newer than 1902 2024-09-10T16:30:17.440 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1905 (ObjNum 1272 snap 408 seq_num 1272) dirty exists 2024-09-10T16:30:17.441 INFO:tasks.rados.rados.0.smithi165.stdout:3676: left oid 14 (ObjNum 1272 snap 408 seq_num 1272) 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3676: done (1 left) 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3677: done (0 left) 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3678: read oid 4 snap -1 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3678: expect (ObjNum 1255 snap 405 seq_num 1255) 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3679: write (excl) oid 14 current snap is 408 2024-09-10T16:30:17.443 INFO:tasks.rados.rados.0.smithi165.stdout:3679: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2024-09-10T16:30:17.448 INFO:tasks.rados.rados.0.smithi165.stdout:3679: writing smithi16534841-14 from 612118 to 1110295 tid 1 2024-09-10T16:30:17.454 INFO:tasks.rados.rados.0.smithi165.stdout:3679: writing smithi16534841-14 from 1634883 to 2205327 tid 2 2024-09-10T16:30:17.462 INFO:tasks.rados.rados.0.smithi165.stdout:3679: writing smithi16534841-14 from 2669481 to 3376143 tid 3 2024-09-10T16:30:17.462 INFO:tasks.rados.rados.0.smithi165.stdout:3680: copy_from oid 9 from oid 32 current snap is 408 2024-09-10T16:30:17.462 INFO:tasks.rados.rados.0.smithi165.stdout:3678: expect (ObjNum 1255 snap 405 seq_num 1255) 2024-09-10T16:30:17.526 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1738630 2024-09-10T16:30:17.591 INFO:tasks.rados.rados.0.smithi165.stdout:3679: finishing write tid 1 to smithi16534841-14 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: finishing write tid 2 to smithi16534841-14 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: finishing write tid 3 to smithi16534841-14 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: finishing write tid 4 to smithi16534841-14 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: finishing write tid 5 to smithi16534841-14 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: oid 14 updating version 0 to 1909 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: oid 14 version 1909 is already newer than 1907 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: oid 14 version 1909 is already newer than 1906 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: oid 14 version 1909 is already newer than 1908 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1909 (ObjNum 1274 snap 408 seq_num 1274) dirty exists 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3679: left oid 14 (ObjNum 1274 snap 408 seq_num 1274) 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3680: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:30:17.592 INFO:tasks.rados.rados.0.smithi165.stdout:3680: finishing copy_from to smithi16534841-9 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1632 (ObjNum 1236 snap 400 seq_num 1236) dirty exists 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3678: done (2 left) 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3679: done (1 left) 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3680: done (0 left) 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3681: rollback oid 39 current snap is 408 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 39 to 386 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3682: write oid 50 current snap is 408 2024-09-10T16:30:17.593 INFO:tasks.rados.rados.0.smithi165.stdout:3682: seq_num 1275 ranges {628925=573426,1693365=714102,3096904=332021} 2024-09-10T16:30:17.597 INFO:tasks.rados.rados.0.smithi165.stdout:3682: writing smithi16534841-50 from 628925 to 1202351 tid 1 2024-09-10T16:30:17.604 INFO:tasks.rados.rados.0.smithi165.stdout:3682: writing smithi16534841-50 from 1693365 to 2407467 tid 2 2024-09-10T16:30:17.610 INFO:tasks.rados.rados.0.smithi165.stdout:3682: writing smithi16534841-50 from 3096904 to 3428925 tid 3 2024-09-10T16:30:17.610 INFO:tasks.rados.rados.0.smithi165.stdout:3683: write oid 32 current snap is 408 2024-09-10T16:30:17.610 INFO:tasks.rados.rados.0.smithi165.stdout:3681: finishing rollback tid 0 to smithi16534841-39 2024-09-10T16:30:17.610 INFO:tasks.rados.rados.0.smithi165.stdout:3683: seq_num 1276 ranges {645732=648675,1751847=574113,3124327=321405} 2024-09-10T16:30:17.616 INFO:tasks.rados.rados.0.smithi165.stdout:3683: writing smithi16534841-32 from 645732 to 1294407 tid 1 2024-09-10T16:30:17.622 INFO:tasks.rados.rados.0.smithi165.stdout:3683: writing smithi16534841-32 from 1751847 to 2325960 tid 2 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3683: writing smithi16534841-32 from 3124327 to 3445732 tid 3 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3684: snap_create 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3681: finishing rollback tid 1 to smithi16534841-39 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3681: finishing rollback tid 2 to smithi16534841-39 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2028 (ObjNum 1056 snap 323 seq_num 1056) dirty exists 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: finishing write tid 1 to smithi16534841-50 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: finishing write tid 2 to smithi16534841-50 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: finishing write tid 3 to smithi16534841-50 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: finishing write tid 4 to smithi16534841-50 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: finishing write tid 5 to smithi16534841-50 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: oid 50 updating version 0 to 1889 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: oid 50 updating version 1889 to 1891 2024-09-10T16:30:17.629 INFO:tasks.rados.rados.0.smithi165.stdout:3682: oid 50 updating version 1891 to 1892 2024-09-10T16:30:17.630 INFO:tasks.rados.rados.0.smithi165.stdout:3682: oid 50 version 1892 is already newer than 1890 2024-09-10T16:30:17.630 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1892 (ObjNum 1275 snap 408 seq_num 1275) dirty exists 2024-09-10T16:30:17.630 INFO:tasks.rados.rados.0.smithi165.stdout:3682: left oid 50 (ObjNum 1275 snap 408 seq_num 1275) 2024-09-10T16:30:17.630 INFO:tasks.rados.rados.0.smithi165.stdout:3683: finishing write tid 1 to smithi16534841-32 2024-09-10T16:30:17.630 INFO:tasks.rados.rados.0.smithi165.stdout:3683: finishing write tid 2 to smithi16534841-32 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: finishing write tid 3 to smithi16534841-32 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: finishing write tid 4 to smithi16534841-32 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: finishing write tid 5 to smithi16534841-32 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: oid 32 updating version 0 to 1634 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: oid 32 updating version 1634 to 1637 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: oid 32 version 1637 is already newer than 1635 2024-09-10T16:30:17.631 INFO:tasks.rados.rados.0.smithi165.stdout:3683: oid 32 version 1637 is already newer than 1636 2024-09-10T16:30:17.632 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1637 (ObjNum 1276 snap 408 seq_num 1276) dirty exists 2024-09-10T16:30:17.632 INFO:tasks.rados.rados.0.smithi165.stdout:3683: left oid 32 (ObjNum 1276 snap 408 seq_num 1276) 2024-09-10T16:30:18.424 INFO:tasks.rados.rados.0.smithi165.stdout:3681: done (3 left) 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3682: done (2 left) 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3683: done (1 left) 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3684: done (0 left) 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3685: rollback oid 2 current snap is 409 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 2 to 372 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3686: rollback oid 12 current snap is 409 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 382 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3687: write oid 18 current snap is 409 2024-09-10T16:30:18.425 INFO:tasks.rados.rados.0.smithi165.stdout:3687: seq_num 1277 ranges {662539=723924,2093976=717771,3435397=27142} 2024-09-10T16:30:18.430 INFO:tasks.rados.rados.0.smithi165.stdout:3687: writing smithi16534841-18 from 662539 to 1386463 tid 1 2024-09-10T16:30:18.438 INFO:tasks.rados.rados.0.smithi165.stdout:3687: writing smithi16534841-18 from 2093976 to 2811747 tid 2 2024-09-10T16:30:18.442 INFO:tasks.rados.rados.0.smithi165.stdout:3687: writing smithi16534841-18 from 3435397 to 3462539 tid 3 2024-09-10T16:30:18.442 INFO:tasks.rados.rados.0.smithi165.stdout:3688: read oid 29 snap -1 2024-09-10T16:30:18.442 INFO:tasks.rados.rados.0.smithi165.stdout:3688: expect deleted 2024-09-10T16:30:18.442 INFO:tasks.rados.rados.0.smithi165.stdout:3689: snap_create 2024-09-10T16:30:18.442 INFO:tasks.rados.rados.0.smithi165.stdout:3685: finishing rollback tid 0 to smithi16534841-2 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3685: finishing rollback tid 1 to smithi16534841-2 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3685: finishing rollback tid 2 to smithi16534841-2 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 2042 (ObjNum 1084 snap 331 seq_num 1084) dirty exists 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3686: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3686: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3686: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1658 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3687: finishing write tid 1 to smithi16534841-18 2024-09-10T16:30:18.443 INFO:tasks.rados.rados.0.smithi165.stdout:3687: finishing write tid 2 to smithi16534841-18 2024-09-10T16:30:18.444 INFO:tasks.rados.rados.0.smithi165.stdout:3687: finishing write tid 3 to smithi16534841-18 2024-09-10T16:30:18.444 INFO:tasks.rados.rados.0.smithi165.stdout:3687: finishing write tid 4 to smithi16534841-18 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: finishing write tid 5 to smithi16534841-18 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: oid 18 updating version 0 to 2030 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: oid 18 updating version 2030 to 2032 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: oid 18 version 2032 is already newer than 2029 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: oid 18 version 2032 is already newer than 2031 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 2032 (ObjNum 1277 snap 409 seq_num 1277) dirty exists 2024-09-10T16:30:18.445 INFO:tasks.rados.rados.0.smithi165.stdout:3687: left oid 18 (ObjNum 1277 snap 409 seq_num 1277) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3685: done (4 left) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3686: done (3 left) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3687: done (2 left) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3688: done (1 left) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3689: done (0 left) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3690: read oid 40 snap 406 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3690: expect (ObjNum 1254 snap 405 seq_num 1254) 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3691: write (excl) oid 39 current snap is 410 2024-09-10T16:30:19.429 INFO:tasks.rados.rados.0.smithi165.stdout:3691: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2024-09-10T16:30:19.435 INFO:tasks.rados.rados.0.smithi165.stdout:3691: writing smithi16534841-39 from 679346 to 1194872 tid 1 2024-09-10T16:30:19.442 INFO:tasks.rados.rados.0.smithi165.stdout:3691: writing smithi16534841-39 from 1868811 to 2446593 tid 2 2024-09-10T16:30:19.448 INFO:tasks.rados.rados.0.smithi165.stdout:3691: writing smithi16534841-39 from 3179173 to 3479346 tid 3 2024-09-10T16:30:19.448 INFO:tasks.rados.rados.0.smithi165.stdout:3692: write (excl) oid 45 current snap is 410 2024-09-10T16:30:19.448 INFO:tasks.rados.rados.0.smithi165.stdout:3692: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2024-09-10T16:30:19.455 INFO:tasks.rados.rados.0.smithi165.stdout:3692: writing smithi16534841-45 from 696153 to 1286928 tid 1 2024-09-10T16:30:19.462 INFO:tasks.rados.rados.0.smithi165.stdout:3692: writing smithi16534841-45 from 1927293 to 2648733 tid 2 2024-09-10T16:30:19.468 INFO:tasks.rados.rados.0.smithi165.stdout:3692: writing smithi16534841-45 from 3206596 to 3496153 tid 3 2024-09-10T16:30:19.468 INFO:tasks.rados.rados.0.smithi165.stdout:3693: read oid 26 snap 406 2024-09-10T16:30:19.468 INFO:tasks.rados.rados.0.smithi165.stdout:3693: expect (ObjNum 1266 snap 406 seq_num 1266) 2024-09-10T16:30:19.468 INFO:tasks.rados.rados.0.smithi165.stdout:3694: snap_create 2024-09-10T16:30:19.468 INFO:tasks.rados.rados.0.smithi165.stdout:3690: expect (ObjNum 1254 snap 405 seq_num 1254) 2024-09-10T16:30:19.580 INFO:tasks.rados.rados.0.smithi165.stdout:3691: finishing write tid 1 to smithi16534841-39 2024-09-10T16:30:19.580 INFO:tasks.rados.rados.0.smithi165.stdout:3691: finishing write tid 2 to smithi16534841-39 2024-09-10T16:30:19.580 INFO:tasks.rados.rados.0.smithi165.stdout:3691: finishing write tid 3 to smithi16534841-39 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: finishing write tid 4 to smithi16534841-39 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: finishing write tid 5 to smithi16534841-39 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: oid 39 updating version 0 to 2035 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: oid 39 updating version 2035 to 2036 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: oid 39 version 2036 is already newer than 2033 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: oid 39 version 2036 is already newer than 2034 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2036 (ObjNum 1278 snap 410 seq_num 1278) dirty exists 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3691: left oid 39 (ObjNum 1278 snap 410 seq_num 1278) 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3692: finishing write tid 1 to smithi16534841-45 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3692: finishing write tid 2 to smithi16534841-45 2024-09-10T16:30:19.581 INFO:tasks.rados.rados.0.smithi165.stdout:3692: finishing write tid 3 to smithi16534841-45 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: finishing write tid 4 to smithi16534841-45 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: finishing write tid 5 to smithi16534841-45 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: oid 45 updating version 0 to 1715 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: oid 45 version 1715 is already newer than 1714 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: oid 45 updating version 1715 to 1716 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: oid 45 updating version 1716 to 1717 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1717 (ObjNum 1279 snap 410 seq_num 1279) dirty exists 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3692: left oid 45 (ObjNum 1279 snap 410 seq_num 1279) 2024-09-10T16:30:19.582 INFO:tasks.rados.rados.0.smithi165.stdout:3693: expect (ObjNum 1266 snap 406 seq_num 1266) 2024-09-10T16:30:19.672 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1454018 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3690: done (4 left) 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3691: done (3 left) 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3692: done (2 left) 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3693: done (1 left) 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3694: done (0 left) 2024-09-10T16:30:20.433 INFO:tasks.rados.rados.0.smithi165.stdout:3695: read oid 3 snap 397 2024-09-10T16:30:20.434 INFO:tasks.rados.rados.0.smithi165.stdout:3695: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:30:20.434 INFO:tasks.rados.rados.0.smithi165.stdout:3696: snap_create 2024-09-10T16:30:20.438 INFO:tasks.rados.rados.0.smithi165.stdout:3695: expect (ObjNum 1164 snap 365 seq_num 1164) 2024-09-10T16:30:21.379 INFO:tasks.rados.rados.0.smithi165.stdout:3695: done (1 left) 2024-09-10T16:30:21.379 INFO:tasks.rados.rados.0.smithi165.stdout:3696: done (0 left) 2024-09-10T16:30:21.379 INFO:tasks.rados.rados.0.smithi165.stdout:3697: rollback oid 40 current snap is 412 2024-09-10T16:30:21.379 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 40 to 365 2024-09-10T16:30:21.379 INFO:tasks.rados.rados.0.smithi165.stdout:3698: snap_remove snap 342 2024-09-10T16:30:22.382 INFO:tasks.rados.rados.0.smithi165.stdout:3698: done (1 left) 2024-09-10T16:30:22.382 INFO:tasks.rados.rados.0.smithi165.stdout:3699: snap_remove snap 350 2024-09-10T16:30:22.664 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T16:30:22.664 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T16:30:22.664 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T16:30:22.667 INFO:tasks.ceph.mon.c:Started 2024-09-10T16:30:22.667 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:30:22.667 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.775+0000 7ff958e7c640 10 monclient: get_monmap_and_config 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.775+0000 7ff958e7c640 10 monclient: build_initial_monmap 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.775+0000 7ff958e7c640 10 monclient: monmap: 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:22.776528+0000 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:22.776528+0000 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:22.777 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:22.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:22.778 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:22.778 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:22.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 Processor -- start 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 -- start start 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient: init 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff95409c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:22.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff95409d160 0x7ff95409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff95409d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff95413a5c0 0x7ff95413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-a con 0x7ff95413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff95409a430 con 0x7ff95409c180 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff9540a24e0 con 0x7ff95413a5c0 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff9540a2bf0 con 0x7ff95409d160 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.776+0000 7ff958e7c640 10 monclient(hunting): authenticate will time out at 2201.263956s 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff95413a5c0 0x7ff95413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff95259c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff95409c180 0x7ff95409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:56998/0 (socket says 172.21.15.164:56998) 2024-09-10T16:30:22.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff95413a5c0 0x7ff95413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54380/0 (socket says 172.21.15.164:54380) 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff95259c640 1 -- 172.21.15.164:0/489168069 learned_addr learned my addr 172.21.15.164:0/489168069 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): get_auth_request con 0x7ff95413a5c0 auth_method 0 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 1 --2- 172.21.15.164:0/489168069 >> v2:172.21.15.165:3300/0 conn(0x7ff95409d160 0x7ff95409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): get_auth_request con 0x7ff95409d160 auth_method 0 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:22.782 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient(hunting): handle_auth_done global_id 254283 payload 274 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 -- 172.21.15.164:0/489168069 >> v2:172.21.15.165:3300/0 conn(0x7ff95409d160 msgr2=0x7ff95409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 --2- 172.21.15.164:0/489168069 >> v2:172.21.15.165:3300/0 conn(0x7ff95409d160 0x7ff95409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 -- 172.21.15.164:0/489168069 >> v1:172.21.15.164:6789/0 conn(0x7ff95409c180 legacy=0x7ff95409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 monclient: found mon.noname-a 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:22.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.777+0000 7ff952d9d640 1 -- 172.21.15.164:0/489168069 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff95409c560 con 0x7ff95413a5c0 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff952d9d640 1 -- 172.21.15.164:0/489168069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95413a5c0 msgr2=0x7ff95413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff952d9d640 1 --2- 172.21.15.164:0/489168069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95413a5c0 0x7ff95413c9b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00bfa0 tx=0x7ff94c02c6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2aafce03cd3e0698 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 1 -- 172.21.15.164:0/489168069 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c038070 con 0x7ff95413a5c0 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: dump: 2024-09-10T16:30:22.791 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: _finish_auth 0 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: _check_auth_tickets 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:22.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 1 -- 172.21.15.164:0/489168069 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff94c009740 con 0x7ff95413a5c0 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 5 monclient: authenticate success, global_id 254283 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 4 set_mon_vals no callback set 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 1 -- 172.21.15.164:0/489168069 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff94c009b10 con 0x7ff95413a5c0 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff95159a640 10 monclient: dump: 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.793 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 10 monclient: shutdown 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 -- 172.21.15.164:0/489168069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95413a5c0 msgr2=0x7ff95413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 --2- 172.21.15.164:0/489168069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95413a5c0 0x7ff95413c9b0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff94c00bfa0 tx=0x7ff94c02c6f0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 -- 172.21.15.164:0/489168069 shutdown_connections 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 --2- 172.21.15.164:0/489168069 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95413a5c0 0x7ff95413c9b0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 --2- 172.21.15.164:0/489168069 >> v2:172.21.15.165:3300/0 conn(0x7ff95409d160 0x7ff95409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 -- 172.21.15.164:0/489168069 >> 172.21.15.164:0/489168069 conn(0x7ff9540a3110 msgr2=0x7ff9540a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:22.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.780+0000 7ff958e7c640 1 -- 172.21.15.164:0/489168069 shutdown_connections 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.781+0000 7ff958e7c640 1 -- 172.21.15.164:0/489168069 wait complete. 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.781+0000 7ff958e7c640 10 monclient: build_initial_monmap 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.781+0000 7ff958e7c640 10 monclient: monmap: 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 Processor -- start 2024-09-10T16:30:22.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 -- start start 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient: init 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-c con 0x7ff95409c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-a con 0x7ff95409d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff95413a5c0 0x7ff9540932c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): picked mon.noname-b con 0x7ff95413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff954074f90 con 0x7ff95409d160 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:22.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff954074e10 con 0x7ff95413a5c0 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff95409f690 con 0x7ff95409c180 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff958e7c640 10 monclient(hunting): authenticate will time out at 2201.269576s 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54382/0 (socket says 172.21.15.164:54382) 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 -- 172.21.15.164:0/3974209084 learned_addr learned my addr 172.21.15.164:0/3974209084 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50124/0 (socket says 172.21.15.164:50124) 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): get_auth_request con 0x7ff95409d160 auth_method 0 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:22.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 10 monclient(hunting): get_auth_request con 0x7ff95409c180 auth_method 0 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:22.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 msgr2=0x7ff954092840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 msgr2=0x7ff954092840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff95259c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff95413a5c0 0x7ff9540932c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): get_auth_request con 0x7ff95413a5c0 auth_method 0 2024-09-10T16:30:22.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:22.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff952d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient(hunting): handle_auth_done global_id 254286 payload 824 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 msgr2=0x7ff954092840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff95413a5c0 msgr2=0x7ff9540932c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff95413a5c0 0x7ff9540932c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:30:22.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 1 monclient: found mon.noname-a 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.782+0000 7ff951d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.783+0000 7ff951d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.783+0000 7ff951d9b640 1 -- 172.21.15.164:0/3974209084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9540965c0 con 0x7ff95409d160 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff951d9b640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 msgr2=0x7ff954092d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff951d9b640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e980 tx=0x7ff94800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff948017070 con 0x7ff95409d160 2024-09-10T16:30:22.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: dump: 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.803 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:22.804 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: _finish_auth 0 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff94800cc10 con 0x7ff95409d160 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 5 monclient: authenticate success, global_id 254286 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 10 monclient: _renew_subs 2024-09-10T16:30:22.805 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff95409d620 con 0x7ff95409d160 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff948013470 con 0x7ff95409d160 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff942ffd640 10 monclient: dump: 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:22.806 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:22.807 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 10 monclient: _renew_subs 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff954094260 con 0x7ff95409d160 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.785+0000 7ff950d99640 4 set_mon_vals no callback set 2024-09-10T16:30:22.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff94801a070 con 0x7ff95409d160 2024-09-10T16:30:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff958e7c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff958e7c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff958e7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff95409d930 con 0x7ff95409d160 2024-09-10T16:30:22.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff942ffd640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 0x7ff934040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(832..832 src has 677..832) ==== 4783+0+0 (secure 0 0 0) 0x7ff948052d30 con 0x7ff95409d160 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.786+0000 7ff95259c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 0x7ff934040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.787+0000 7ff95259c640 10 monclient: get_auth_request con 0x7ff93403e240 auth_method 0 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.789+0000 7ff95259c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 0x7ff934040700 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff944007fd0 tx=0x7ff944009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.791+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff948051730 con 0x7ff95409d160 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.791+0000 7ff942ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:22.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:22.791+0000 7ff942ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:23.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff958e7c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff958e7c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff958e7c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9540963a0 con 0x7ff95409d160 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff942ffd640 1 -- 172.21.15.164:0/3974209084 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7ff948023d00 con 0x7ff95409d160 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff942ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.032+0000 7ff942ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:23.034 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":236,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:23.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 msgr2=0x7ff934040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:23.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 0x7ff934040700 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7ff944007fd0 tx=0x7ff944009040 comp rx=0 tx=0).stop 2024-09-10T16:30:23.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 10 monclient: shutdown 2024-09-10T16:30:23.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 msgr2=0x7ff954092d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.034+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7ff94800e980 tx=0x7ff94800ee40 comp rx=0 tx=0).stop 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 shutdown_connections 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff93403e240 0x7ff934040700 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff95413a5c0 0x7ff9540932c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff95409d160 0x7ff954092d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 --2- 172.21.15.164:0/3974209084 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff95409c180 0x7ff954092840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 >> 172.21.15.164:0/3974209084 conn(0x7ff9540a3110 msgr2=0x7ff95409db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 shutdown_connections 2024-09-10T16:30:23.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:23.035+0000 7ff958e7c640 1 -- 172.21.15.164:0/3974209084 wait complete. 2024-09-10T16:30:23.406 INFO:tasks.rados.rados.0.smithi165.stdout:3699: done (1 left) 2024-09-10T16:30:23.406 INFO:tasks.rados.rados.0.smithi165.stdout:3700: read oid 14 snap -1 2024-09-10T16:30:23.406 INFO:tasks.rados.rados.0.smithi165.stdout:3700: expect (ObjNum 1274 snap 408 seq_num 1274) 2024-09-10T16:30:23.406 INFO:tasks.rados.rados.0.smithi165.stdout:3701: snap_remove snap 386 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3701: done (2 left) 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3702: read oid 15 snap -1 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3702: expect (ObjNum 1257 snap 405 seq_num 1257) 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3703: write oid 49 current snap is 412 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3697: finishing rollback tid 0 to smithi16534841-40 2024-09-10T16:30:24.407 INFO:tasks.rados.rados.0.smithi165.stdout:3703: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2024-09-10T16:30:24.414 INFO:tasks.rados.rados.0.smithi165.stdout:3703: writing smithi16534841-49 from 712960 to 1378984 tid 1 2024-09-10T16:30:24.421 INFO:tasks.rados.rados.0.smithi165.stdout:3703: writing smithi16534841-49 from 1985775 to 2567226 tid 2 2024-09-10T16:30:24.426 INFO:tasks.rados.rados.0.smithi165.stdout:3703: writing smithi16534841-49 from 3234019 to 3512960 tid 3 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:3704: rollback oid 34 current snap is 412 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 372 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:3697: finishing rollback tid 1 to smithi16534841-40 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:3697: finishing rollback tid 2 to smithi16534841-40 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1912 (ObjNum 1088 snap 332 seq_num 1088) dirty exists 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:3700: expect (ObjNum 1274 snap 408 seq_num 1274) 2024-09-10T16:30:24.427 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:24.499 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2156047 2024-09-10T16:30:24.523 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 437124 2024-09-10T16:30:24.572 INFO:tasks.rados.rados.0.smithi165.stdout:3702: expect (ObjNum 1257 snap 405 seq_num 1257) 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: finishing write tid 1 to smithi16534841-49 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: finishing write tid 2 to smithi16534841-49 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3704: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: finishing write tid 3 to smithi16534841-49 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: finishing write tid 4 to smithi16534841-49 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: finishing write tid 5 to smithi16534841-49 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: oid 49 updating version 0 to 1851 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: oid 49 version 1851 is already newer than 1848 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: oid 49 version 1851 is already newer than 1850 2024-09-10T16:30:24.659 INFO:tasks.rados.rados.0.smithi165.stdout:3703: oid 49 version 1851 is already newer than 1849 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1851 (ObjNum 1280 snap 412 seq_num 1280) dirty exists 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3703: left oid 49 (ObjNum 1280 snap 412 seq_num 1280) 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3704: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3697: done (4 left) 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3700: done (3 left) 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3702: done (2 left) 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3703: done (1 left) 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3705: write (excl) oid 9 current snap is 412 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3704: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1599 (ObjNum 1154 snap 362 seq_num 1154) dirty exists 2024-09-10T16:30:24.660 INFO:tasks.rados.rados.0.smithi165.stdout:3705: seq_num 1281 ranges {729767=741273,1927904=725109,3145089=384678} 2024-09-10T16:30:24.666 INFO:tasks.rados.rados.0.smithi165.stdout:3705: writing smithi16534841-9 from 729767 to 1471040 tid 1 2024-09-10T16:30:24.673 INFO:tasks.rados.rados.0.smithi165.stdout:3705: writing smithi16534841-9 from 1927904 to 2653013 tid 2 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3705: writing smithi16534841-9 from 3145089 to 3529767 tid 3 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3704: done (1 left) 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3706: read oid 50 snap -1 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3706: expect (ObjNum 1275 snap 408 seq_num 1275) 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3707: rollback oid 30 current snap is 412 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 311 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3705: finishing write tid 1 to smithi16534841-9 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3705: finishing write tid 2 to smithi16534841-9 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3708: rollback oid 15 current snap is 412 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 389 2024-09-10T16:30:24.679 INFO:tasks.rados.rados.0.smithi165.stdout:3709: snap_create 2024-09-10T16:30:24.681 INFO:tasks.rados.rados.0.smithi165.stdout:3705: finishing write tid 3 to smithi16534841-9 2024-09-10T16:30:24.681 INFO:tasks.rados.rados.0.smithi165.stdout:3708: finishing rollback tid 0 to smithi16534841-15 2024-09-10T16:30:24.686 INFO:tasks.rados.rados.0.smithi165.stdout:3705: finishing write tid 4 to smithi16534841-9 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: finishing write tid 5 to smithi16534841-9 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: oid 9 updating version 0 to 1914 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: oid 9 updating version 1914 to 1915 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: oid 9 updating version 1915 to 1916 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: oid 9 version 1916 is already newer than 1913 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1916 (ObjNum 1281 snap 412 seq_num 1281) dirty exists 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3705: left oid 9 (ObjNum 1281 snap 412 seq_num 1281) 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3707: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3708: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1865 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:3707: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:30:24.687 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2412 (ObjNum 977 snap 297 seq_num 977) dirty exists 2024-09-10T16:30:24.690 INFO:tasks.rados.rados.0.smithi165.stdout:3706: expect (ObjNum 1275 snap 408 seq_num 1275) 2024-09-10T16:30:24.718 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2397837 2024-09-10T16:30:24.769 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2103569 2024-09-10T16:30:24.798 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:25.411 INFO:tasks.rados.rados.0.smithi165.stdout:3705: done (4 left) 2024-09-10T16:30:25.411 INFO:tasks.rados.rados.0.smithi165.stdout:3706: done (3 left) 2024-09-10T16:30:25.411 INFO:tasks.rados.rados.0.smithi165.stdout:3707: done (2 left) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3708: done (1 left) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3709: done (0 left) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3710: read oid 9 snap -1 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3710: expect (ObjNum 1281 snap 412 seq_num 1281) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3711: read oid 30 snap 344 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3711: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3712: read oid 1 snap -1 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3712: expect deleted 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3713: read oid 32 snap -1 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3713: expect (ObjNum 1276 snap 408 seq_num 1276) 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3714: read oid 23 snap 391 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3714: expect deleted 2024-09-10T16:30:25.412 INFO:tasks.rados.rados.0.smithi165.stdout:3715: snap_create 2024-09-10T16:30:25.421 INFO:tasks.rados.rados.0.smithi165.stdout:3710: expect (ObjNum 1281 snap 412 seq_num 1281) 2024-09-10T16:30:25.474 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:25.549 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2368145 2024-09-10T16:30:25.565 INFO:tasks.rados.rados.0.smithi165.stdout:3711: expect (ObjNum 1102 snap 336 seq_num 1102) 2024-09-10T16:30:25.593 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:25.646 INFO:tasks.rados.rados.0.smithi165.stdout:3713: expect (ObjNum 1276 snap 408 seq_num 1276) 2024-09-10T16:30:25.650 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 750345 2024-09-10T16:30:25.713 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2619723 2024-09-10T16:30:26.047 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 10 monclient: get_monmap_and_config 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 10 monclient: monmap: 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.157 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:26.156910+0000 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:26.156910+0000 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:26.158 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 1 Processor -- start 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 1 -- start start 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.156+0000 7f79cd8e6640 10 monclient: init 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient: _add_conns ranks=[0,4,3,2,1] 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f79c80a3e60 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 0x7f79c813d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f79c806d280 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79c806d690 0x7f79c813fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f79c806d690 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c80a5db0 con 0x7f79c80a3e60 2024-09-10T16:30:26.161 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f79c809bd10 con 0x7f79c806d690 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f79c809bf40 con 0x7f79c806d280 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79cd8e6640 10 monclient(hunting): authenticate will time out at 2204.644183s 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79c806d690 0x7f79c813fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f79c806d690 0x7f79c813fb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55084/0 (socket says 172.21.15.164:55084) 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c6ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f79c80a3e60 0x7f79c806cb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38552/0 (socket says 172.21.15.164:38552) 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 -- 172.21.15.164:0/2503075296 learned_addr learned my addr 172.21.15.164:0/2503075296 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): get_auth_request con 0x7f79c806d690 auth_method 0 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.162 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2503075296 >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 0x7f79c813d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 10 monclient(hunting): get_auth_request con 0x7f79c806d280 auth_method 0 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient(hunting): handle_auth_done global_id 254289 payload 274 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 -- 172.21.15.164:0/2503075296 >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 msgr2=0x7f79c813d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 --2- 172.21.15.164:0/2503075296 >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 0x7f79c813d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 -- 172.21.15.164:0/2503075296 >> v1:172.21.15.164:6789/0 conn(0x7f79c80a3e60 legacy=0x7f79c806cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:26.163 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c77fe640 1 -- 172.21.15.164:0/2503075296 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c80a4810 con 0x7f79c806d690 2024-09-10T16:30:26.164 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.157+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2503075296 >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 0x7f79c813d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c77fe640 1 -- 172.21.15.164:0/2503075296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d690 msgr2=0x7f79c813fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c77fe640 1 --2- 172.21.15.164:0/2503075296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d690 0x7f79c813fb40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f79c80a4f20 tx=0x7f79c002a610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5c311c8bc807b45 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 1 -- 172.21.15.164:0/2503075296 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0036070 con 0x7f79c806d690 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: dump: 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.170 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 1 -- 172.21.15.164:0/2503075296 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79c0007800 con 0x7f79c806d690 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79cd8e6640 5 monclient: authenticate success, global_id 254289 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79cd8e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:26.171 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79cd8e6640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 1 -- 172.21.15.164:0/2503075296 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79c0007bd0 con 0x7f79c806d690 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79cd8e6640 4 set_mon_vals no callback set 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.160+0000 7f79c5ffb640 10 monclient: dump: 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.172 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 10 monclient: shutdown 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2503075296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d690 msgr2=0x7f79c813fb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2503075296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d690 0x7f79c813fb40 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f79c80a4f20 tx=0x7f79c002a610 comp rx=0 tx=0).stop 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2503075296 shutdown_connections 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2503075296 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d690 0x7f79c813fb40 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2503075296 >> v2:172.21.15.165:3300/0 conn(0x7f79c806d280 0x7f79c813d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2503075296 >> 172.21.15.164:0/2503075296 conn(0x7f79c80a4f50 msgr2=0x7f79c80a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:26.173 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2503075296 shutdown_connections 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2503075296 wait complete. 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.161+0000 7f79cd8e6640 10 monclient: monmap: 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:26.174 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 Processor -- start 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 -- start start 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient: init 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-a con 0x7f79c806d280 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-c con 0x7f79c806d690 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79c80a3e60 0x7f79c8094290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): picked mon.noname-b con 0x7f79c80a3e60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f79c80a4240 con 0x7f79c806d280 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f79c806e3d0 con 0x7f79c80a3e60 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f79c80a43c0 con 0x7f79c806d690 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79cd8e6640 10 monclient(hunting): authenticate will time out at 2204.649848s 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55094/0 (socket says 172.21.15.164:55094) 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c67fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59924/0 (socket says 172.21.15.164:59924) 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 1 -- 172.21.15.164:0/2150270057 learned_addr learned my addr 172.21.15.164:0/2150270057 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 10 monclient(hunting): get_auth_request con 0x7f79c806d280 auth_method 0 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.162+0000 7f79c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 10 monclient(hunting): get_auth_request con 0x7f79c806d690 auth_method 0 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 msgr2=0x7f79c8093d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 msgr2=0x7f79c8093d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c77fe640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79c80a3e60 0x7f79c8094290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c77fe640 10 monclient(hunting): get_auth_request con 0x7f79c80a3e60 auth_method 0 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient(hunting): handle_auth_done global_id 254292 payload 824 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 msgr2=0x7f79c8093d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79c80a3e60 msgr2=0x7f79c8094290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79c80a3e60 0x7f79c8094290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.163+0000 7f79c6ffd640 1 -- 172.21.15.164:0/2150270057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c8097890 con 0x7f79c806d280 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79c6ffd640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 msgr2=0x7f79c809a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79c6ffd640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00c480 tx=0x7f79bc00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc017070 con 0x7f79c806d280 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: dump: 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.179 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79bc005810 con 0x7f79c806d280 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 5 monclient: authenticate success, global_id 254292 2024-09-10T16:30:26.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79bc007690 con 0x7f79c806d280 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 10 monclient: _renew_subs 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c8097b70 con 0x7f79c806d280 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79affff640 10 monclient: dump: 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.181 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79c57fa640 4 set_mon_vals no callback set 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 10 monclient: _renew_subs 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.165+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c809c140 con 0x7f79c806d280 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f79bc01a070 con 0x7f79c806d280 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79cd8e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79cd8e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79cd8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c806cb70 con 0x7f79c806d280 2024-09-10T16:30:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.166+0000 7f79affff640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 0x7f79b40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.167+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(835..835 src has 677..835) ==== 4787+0+0 (secure 0 0 0) 0x7f79bc0521b0 con 0x7f79c806d280 2024-09-10T16:30:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.167+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 0x7f79b40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.167+0000 7f79c67fc640 10 monclient: get_auth_request con 0x7f79b403e210 auth_method 0 2024-09-10T16:30:26.183 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.169+0000 7f79c67fc640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 0x7f79b40406d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f79c8096300 tx=0x7f79b8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.171+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f79bc010300 con 0x7f79c806d280 2024-09-10T16:30:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.171+0000 7f79affff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.184 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.171+0000 7f79affff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3710: done (5 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3711: done (4 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3712: done (3 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3713: done (2 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3714: done (1 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3715: done (0 left) 2024-09-10T16:30:26.380 INFO:tasks.rados.rados.0.smithi165.stdout:3716: write (excl) oid 36 current snap is 414 2024-09-10T16:30:26.381 INFO:tasks.rados.rados.0.smithi165.stdout:3716: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2024-09-10T16:30:26.386 INFO:tasks.rados.rados.0.smithi165.stdout:3716: writing smithi16534841-36 from 746574 to 1163096 tid 1 2024-09-10T16:30:26.390 INFO:tasks.rados.rados.0.smithi165.stdout:3716: writing smithi16534841-36 from 1586386 to 2055153 tid 2 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3716: writing smithi16534841-36 from 2772512 to 3402056 tid 3 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3717: copy_from oid 1 from oid 43 current snap is 414 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3716: finishing write tid 1 to smithi16534841-36 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3716: finishing write tid 2 to smithi16534841-36 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3718: rollback oid 16 current snap is 414 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 344 2024-09-10T16:30:26.399 INFO:tasks.rados.rados.0.smithi165.stdout:3719: write oid 25 current snap is 414 2024-09-10T16:30:26.400 INFO:tasks.rados.rados.0.smithi165.stdout:3719: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2024-09-10T16:30:26.406 INFO:tasks.rados.rados.0.smithi165.stdout:3719: writing smithi16534841-25 from 763381 to 1255152 tid 1 2024-09-10T16:30:26.416 INFO:tasks.rados.rados.0.smithi165.stdout:3719: writing smithi16534841-25 from 2044868 to 2773646 tid 2 2024-09-10T16:30:26.421 INFO:tasks.rados.rados.0.smithi165.stdout:3719: writing smithi16534841-25 from 3199935 to 3563381 tid 3 2024-09-10T16:30:26.421 INFO:tasks.rados.rados.0.smithi165.stdout:3716: finishing write tid 3 to smithi16534841-36 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: finishing write tid 4 to smithi16534841-36 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3720: read oid 13 snap -1 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3720: expect deleted 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3721: rollback oid 44 current snap is 414 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 344 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: finishing write tid 5 to smithi16534841-36 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: oid 36 updating version 0 to 1699 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: oid 36 version 1699 is already newer than 1698 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: oid 36 version 1699 is already newer than 1697 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: oid 36 version 1699 is already newer than 1696 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 36 v 1699 (ObjNum 1282 snap 414 seq_num 1282) dirty exists 2024-09-10T16:30:26.422 INFO:tasks.rados.rados.0.smithi165.stdout:3716: left oid 36 (ObjNum 1282 snap 414 seq_num 1282) 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3718: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3718: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2415 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3719: finishing write tid 1 to smithi16534841-25 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3717: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3717: finishing copy_from to smithi16534841-1 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1713 (ObjNum 1152 snap 362 seq_num 1152) dirty exists 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3716: done (5 left) 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3717: done (4 left) 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3718: done (3 left) 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3722: copy_from oid 10 from oid 11 current snap is 414 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3723: read oid 18 snap -1 2024-09-10T16:30:26.423 INFO:tasks.rados.rados.0.smithi165.stdout:3723: expect (ObjNum 1277 snap 409 seq_num 1277) 2024-09-10T16:30:26.424 INFO:tasks.rados.rados.0.smithi165.stdout:3724: write oid 1 current snap is 414 2024-09-10T16:30:26.424 INFO:tasks.rados.rados.0.smithi165.stdout:3724: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2024-09-10T16:30:26.428 INFO:tasks.rados.rados.0.smithi165.stdout:3724: writing smithi16534841-1 from 780188 to 1347208 tid 1 2024-09-10T16:30:26.435 INFO:tasks.rados.rados.0.smithi165.stdout:3724: writing smithi16534841-1 from 2103350 to 2575786 tid 2 2024-09-10T16:30:26.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79cd8e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79cd8e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79cd8e6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f79c8073870 con 0x7f79c806d280 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79affff640 1 -- 172.21.15.164:0/2150270057 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f79bc010e20 con 0x7f79c806d280 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79affff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.436+0000 7f79affff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:26.438 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:26.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 msgr2=0x7f79b40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 0x7f79b40406d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7f79c8096300 tx=0x7f79b8008040 comp rx=0 tx=0).stop 2024-09-10T16:30:26.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 10 monclient: shutdown 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 msgr2=0x7f79c809a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.438+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f79bc00c480 tx=0x7f79bc00c940 comp rx=0 tx=0).stop 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 shutdown_connections 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f79b403e210 0x7f79b40406d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f79c80a3e60 0x7f79c8094290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f79c806d690 0x7f79c8093d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 --2- 172.21.15.164:0/2150270057 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f79c806d280 0x7f79c809a0b0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 >> 172.21.15.164:0/2150270057 conn(0x7f79c80a4f50 msgr2=0x7f79c809eab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 shutdown_connections 2024-09-10T16:30:26.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.439+0000 7f79cd8e6640 1 -- 172.21.15.164:0/2150270057 wait complete. 2024-09-10T16:30:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:3724: writing smithi16534841-1 from 3227358 to 3580188 tid 3 2024-09-10T16:30:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:3725: read oid 24 snap 391 2024-09-10T16:30:26.442 INFO:tasks.rados.rados.0.smithi165.stdout:3725: expect deleted 2024-09-10T16:30:26.443 INFO:tasks.rados.rados.0.smithi165.stdout:3726: rollback oid 23 current snap is 414 2024-09-10T16:30:26.443 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 23 to 407 2024-09-10T16:30:26.443 INFO:tasks.rados.rados.0.smithi165.stdout:3721: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:30:26.443 INFO:tasks.rados.rados.0.smithi165.stdout:3723: expect (ObjNum 1277 snap 409 seq_num 1277) 2024-09-10T16:30:26.443 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:26.451 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:30:26.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:30:26.452 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:30:26.518 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2342750 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.549+0000 7fcb6a991640 10 monclient: get_monmap_and_config 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.549+0000 7fcb6a991640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.549+0000 7fcb6a991640 10 monclient: monmap: 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:26.550450+0000 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:26.550450+0000 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.551 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:26.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:26.552 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:26.552 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:26.552 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:26.552 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 Processor -- start 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 -- start start 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient: init 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:30:26.556 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 0x7fcb6409ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6409c310 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcb6409d190 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb6413a5c0 0x7fcb6413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb6409fbc0 con 0x7fcb6409d190 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fcb640a2a20 con 0x7fcb6413a5c0 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fcb6409c6f0 con 0x7fcb6409c310 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb6a991640 10 monclient(hunting): authenticate will time out at 2205.037829s 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb68f2e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb6413a5c0 0x7fcb6413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.557 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.550+0000 7fcb63fff640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 0x7fcb6409ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcb6413a5c0 0x7fcb6413c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55106/0 (socket says 172.21.15.164:55106) 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 -- 172.21.15.164:0/3677293937 learned_addr learned my addr 172.21.15.164:0/3677293937 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): get_auth_request con 0x7fcb6413a5c0 auth_method 0 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): get_auth_request con 0x7fcb6409c310 auth_method 0 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.558 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_done global_id 254295 payload 274 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 -- 172.21.15.164:0/3677293937 >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 msgr2=0x7fcb6409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 --2- 172.21.15.164:0/3677293937 >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 0x7fcb6409ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 -- 172.21.15.164:0/3677293937 >> v1:172.21.15.165:6789/0 conn(0x7fcb6409d190 legacy=0x7fcb6409ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.559 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb68f2e640 1 -- 172.21.15.164:0/3677293937 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb6409a4d0 con 0x7fcb6413a5c0 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.551+0000 7fcb63fff640 1 --2- 172.21.15.164:0/3677293937 >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 0x7fcb6409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb68f2e640 1 -- 172.21.15.164:0/3677293937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6413a5c0 msgr2=0x7fcb6413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb68f2e640 1 --2- 172.21.15.164:0/3677293937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6413a5c0 0x7fcb6413c9b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406d4d0 tx=0x7fcb5c02c780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834e7d449ecc9588 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 1 -- 172.21.15.164:0/3677293937 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c009800 con 0x7fcb6413a5c0 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: dump: 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.560 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.561 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 1 -- 172.21.15.164:0/3677293937 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5c0099a0 con 0x7fcb6413a5c0 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 5 monclient: authenticate success, global_id 254295 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 1 -- 172.21.15.164:0/3677293937 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5c009d70 con 0x7fcb6413a5c0 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 4 set_mon_vals no callback set 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.562 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb62ffd640 10 monclient: dump: 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 10 monclient: shutdown 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:26.563 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 -- 172.21.15.164:0/3677293937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6413a5c0 msgr2=0x7fcb6413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 --2- 172.21.15.164:0/3677293937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6413a5c0 0x7fcb6413c9b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb6406d4d0 tx=0x7fcb5c02c780 comp rx=0 tx=0).stop 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 -- 172.21.15.164:0/3677293937 shutdown_connections 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 --2- 172.21.15.164:0/3677293937 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6413a5c0 0x7fcb6413c9b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 --2- 172.21.15.164:0/3677293937 >> v2:172.21.15.165:3300/0 conn(0x7fcb6409c310 0x7fcb6409ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 -- 172.21.15.164:0/3677293937 >> 172.21.15.164:0/3677293937 conn(0x7fcb640a3110 msgr2=0x7fcb640a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 -- 172.21.15.164:0/3677293937 shutdown_connections 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.554+0000 7fcb6a991640 1 -- 172.21.15.164:0/3677293937 wait complete. 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.555+0000 7fcb6a991640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.555+0000 7fcb6a991640 10 monclient: monmap: 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:26.564 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:26.565 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.571 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.555+0000 7fcb6a991640 1 Processor -- start 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.555+0000 7fcb6a991640 1 -- start start 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.555+0000 7fcb6a991640 10 monclient: init 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 0x7fcb640995b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-a con 0x7fcb6409c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 0x7fcb6408fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-c con 0x7fcb6413a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb640903a0 0x7fcb640907f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): picked mon.noname-b con 0x7fcb640903a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb640a2230 con 0x7fcb6409c310 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcb640a20b0 con 0x7fcb640903a0 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcb64074e00 con 0x7fcb6413a5c0 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb6a991640 10 monclient(hunting): authenticate will time out at 2205.043359s 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 0x7fcb640995b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 0x7fcb6408fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.573 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 0x7fcb640995b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55108/0 (socket says 172.21.15.164:55108) 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 -- 172.21.15.164:0/11732460 learned_addr learned my addr 172.21.15.164:0/11732460 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 10 monclient(hunting): get_auth_request con 0x7fcb6413a5c0 auth_method 0 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): get_auth_request con 0x7fcb6409c310 auth_method 0 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.574 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb640903a0 0x7fcb640907f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): get_auth_request con 0x7fcb640903a0 auth_method 0 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 msgr2=0x7fcb6408fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 msgr2=0x7fcb6408fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 0x7fcb6408fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 0x7fcb6408fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.575 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb68f2e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient(hunting): handle_auth_done global_id 254298 payload 824 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 msgr2=0x7fcb6408fe60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcb6413a5c0 0x7fcb6408fe60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb640903a0 msgr2=0x7fcb640907f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcb640903a0 0x7fcb640907f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.576 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.556+0000 7fcb63fff640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb64093bf0 con 0x7fcb6409c310 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.558+0000 7fcb63fff640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 msgr2=0x7fcb640995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.558+0000 7fcb63fff640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 0x7fcb640995b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400ebe0 tx=0x7fcb5400c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54017070 con 0x7fcb6409c310 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: dump: 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.577 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb5400ce40 con 0x7fcb6409c310 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 5 monclient: authenticate success, global_id 254298 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 10 monclient: _renew_subs 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.578 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb6409ee10 con 0x7fcb6409c310 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb54013690 con 0x7fcb6409c310 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb60ff9640 10 monclient: dump: 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.579 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 10 monclient: _renew_subs 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb64091890 con 0x7fcb6409c310 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.559+0000 7fcb627fc640 4 set_mon_vals no callback set 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb6a991640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb6a991640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 0x7fcb64071ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.580 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb6409f720 con 0x7fcb6409d190 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 0x7fcb64071ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient: get_auth_request con 0x7fcb6409d190 auth_method 0 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcb5401a070 con 0x7fcb6409c310 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 10 monclient(hunting): handle_auth_done global_id 254301 payload 824 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb6409d620 con 0x7fcb6409d190 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb60ff9640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb5003e210 0x7fcb500406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.581 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(836..836 src has 677..836) ==== 4799+0+0 (secure 0 0 0) 0x7fcb54052cf0 con 0x7fcb6409c310 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb68f2e640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb5003e210 0x7fcb500406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.560+0000 7fcb68f2e640 10 monclient: get_auth_request con 0x7fcb5003e210 auth_method 0 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 msgr2=0x7fcb64071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 0x7fcb64071ee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb64093200 tx=0x7fcb5800e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb58007450 con 0x7fcb6409d190 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb60ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.562+0000 7fcb60ff9640 10 monclient: dump: 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.582 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb58018070 con 0x7fcb6409d190 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:26.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 msgr2=0x7fcb64071ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 0x7fcb64071ee0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb64093200 tx=0x7fcb5800e8b0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 msgr2=0x7fcb64071ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb60ff9640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409d190 0x7fcb64071ee0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcb64093200 tx=0x7fcb5800e8b0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.563+0000 7fcb68f2e640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb5003e210 0x7fcb500406d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcb640a0280 tx=0x7fcb5c030000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb6a991640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb6a991640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 0x7fcb64072830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb6a991640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcb64090d30 con 0x7fcb64072420 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 0x7fcb64072830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient: get_auth_request con 0x7fcb64072420 auth_method 0 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.584 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 10 monclient(hunting): handle_auth_done global_id 254304 payload 824 2024-09-10T16:30:26.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.569+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb64077260 con 0x7fcb64072420 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb637fe640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 msgr2=0x7fcb64072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb637fe640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 0x7fcb64072830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb58017000 tx=0x7fcb58003760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb5801d7c0 con 0x7fcb64072420 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb60ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb60ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.571+0000 7fcb60ff9640 10 monclient: dump: 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.586 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7fcb58013420 con 0x7fcb64072420 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 msgr2=0x7fcb64072830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 0x7fcb64072830 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb58017000 tx=0x7fcb58003760 comp rx=0 tx=0).stop 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 msgr2=0x7fcb64072830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:26.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb60ff9640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb64072420 0x7fcb64072830 secure :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcb58017000 tx=0x7fcb58003760 comp rx=0 tx=0).stop 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 115222, 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:26.588 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:26.589 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:26.590 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:26.591 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:26.592 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:26.593 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:26.594 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.595 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:26.596 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:26.597 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.598 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.599 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:26.600 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:26.601 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb5003e210 msgr2=0x7fcb500406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcb5003e210 0x7fcb500406d0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fcb640a0280 tx=0x7fcb5c030000 comp rx=0 tx=0).stop 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 10 monclient: shutdown 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 msgr2=0x7fcb640995b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 --2- 172.21.15.164:0/11732460 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcb6409c310 0x7fcb640995b0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fcb5400ebe0 tx=0x7fcb5400c520 comp rx=0 tx=0).stop 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb68f2e640 1 -- 172.21.15.164:0/11732460 reap_dead start 2024-09-10T16:30:26.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb68f2e640 1 -- 172.21.15.164:0/11732460 reap_dead start 2024-09-10T16:30:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 shutdown_connections 2024-09-10T16:30:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 >> 172.21.15.164:0/11732460 conn(0x7fcb640a3110 msgr2=0x7fcb6409d830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 shutdown_connections 2024-09-10T16:30:26.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.572+0000 7fcb6a991640 1 -- 172.21.15.164:0/11732460 wait complete. 2024-09-10T16:30:26.604 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:26.604 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:30:26.604 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: finishing write tid 2 to smithi16534841-25 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3721: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1841 (ObjNum 1030 snap 313 seq_num 1030) dirty exists 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: finishing write tid 3 to smithi16534841-25 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: finishing write tid 4 to smithi16534841-25 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: finishing write tid 5 to smithi16534841-25 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: oid 25 updating version 0 to 2064 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: oid 25 updating version 2064 to 2065 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: oid 25 version 2065 is already newer than 2062 2024-09-10T16:30:26.606 INFO:tasks.rados.rados.0.smithi165.stdout:3719: oid 25 version 2065 is already newer than 2063 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 2065 (ObjNum 1283 snap 414 seq_num 1283) dirty exists 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3719: left oid 25 (ObjNum 1283 snap 414 seq_num 1283) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3722: finishing copy_from racing read to smithi16534841-10 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3722: finishing copy_from to smithi16534841-10 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3724: finishing write tid 1 to smithi16534841-1 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3719: done (7 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3720: done (6 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3721: done (5 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3722: done (4 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3723: done (3 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3725: done (2 left) 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3727: snap_create 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3724: finishing write tid 2 to smithi16534841-1 2024-09-10T16:30:26.607 INFO:tasks.rados.rados.0.smithi165.stdout:3724: finishing write tid 3 to smithi16534841-1 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: finishing write tid 4 to smithi16534841-1 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: finishing write tid 5 to smithi16534841-1 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: oid 1 updating version 0 to 2039 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: oid 1 version 2039 is already newer than 2038 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: oid 1 version 2039 is already newer than 2037 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: oid 1 updating version 2039 to 2040 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2040 (ObjNum 1284 snap 414 seq_num 1284) dirty exists 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3724: left oid 1 (ObjNum 1284 snap 414 seq_num 1284) 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:3726: finishing rollback tid 1 to smithi16534841-23 2024-09-10T16:30:26.608 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 2011 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.705+0000 7f5e8a95d640 10 monclient: get_monmap_and_config 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.705+0000 7f5e8a95d640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.705+0000 7f5e8a95d640 10 monclient: monmap: 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:26.706718+0000 2024-09-10T16:30:26.707 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:26.706718+0000 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:26.708 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.706+0000 7f5e8a95d640 1 Processor -- start 2024-09-10T16:30:26.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.706+0000 7f5e8a95d640 1 -- start start 2024-09-10T16:30:26.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.706+0000 7f5e8a95d640 10 monclient: init 2024-09-10T16:30:26.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e8409c180 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5e8413a5c0 0x7f5e8413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e8409a430 con 0x7f5e8409c180 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f5e840a24e0 con 0x7f5e8413a5c0 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e840a2bf0 con 0x7f5e8409d160 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e8a95d640 10 monclient(hunting): authenticate will time out at 2205.194137s 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f5e8413a5c0 0x7f5e8413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:59946/0 (socket says 172.21.15.164:59946) 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 -- 172.21.15.164:0/1436286660 learned_addr learned my addr 172.21.15.164:0/1436286660 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 10 monclient(hunting): get_auth_request con 0x7f5e8409d160 auth_method 0 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): get_auth_request con 0x7f5e8413a5c0 auth_method 0 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 -- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e8409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 -- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e8409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e837fe640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient(hunting): handle_auth_done global_id 254307 payload 274 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 -- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e8409f150 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 -- 172.21.15.164:0/1436286660 >> v1:172.21.15.165:6789/0 conn(0x7f5e8409c180 legacy=0x7f5e8409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.707+0000 7f5e88efa640 1 -- 172.21.15.164:0/1436286660 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8409c560 con 0x7f5e8413a5c0 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e88efa640 1 -- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 msgr2=0x7f5e8413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e88efa640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e8413c9b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e8409abb0 tx=0x7f5e7c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aecdd3c15addfaee server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 1 -- 172.21.15.164:0/1436286660 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c007950 con 0x7f5e8413a5c0 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: dump: 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.720 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 1 -- 172.21.15.164:0/1436286660 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7c007af0 con 0x7f5e8413a5c0 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e8a95d640 5 monclient: authenticate success, global_id 254307 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e8a95d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:26.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e8a95d640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 1 -- 172.21.15.164:0/1436286660 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c031400 con 0x7f5e8413a5c0 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e8a95d640 4 set_mon_vals no callback set 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.710+0000 7f5e82ffd640 10 monclient: dump: 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.722 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 10 monclient: shutdown 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 -- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 msgr2=0x7f5e8413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e8413c9b0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f5e8409abb0 tx=0x7f5e7c02a480 comp rx=0 tx=0).stop 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 -- 172.21.15.164:0/1436286660 shutdown_connections 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e8409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/1436286660 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e8413c9b0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 -- 172.21.15.164:0/1436286660 >> 172.21.15.164:0/1436286660 conn(0x7f5e840a3110 msgr2=0x7f5e840a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:26.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 -- 172.21.15.164:0/1436286660 shutdown_connections 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 1 -- 172.21.15.164:0/1436286660 wait complete. 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 10 monclient: build_initial_monmap 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.711+0000 7f5e8a95d640 10 monclient: monmap: 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:26.724 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 Processor -- start 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 -- start start 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient: init 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e8409c180 0x7f5e841cad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-b con 0x7f5e8409c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e841cb240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-c con 0x7f5e8409d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e841cb780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): picked mon.noname-a con 0x7f5e8413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84074f90 con 0x7f5e8413a5c0 2024-09-10T16:30:26.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5e84074e10 con 0x7f5e8409c180 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8409f690 con 0x7f5e8409d160 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e8a95d640 10 monclient(hunting): authenticate will time out at 2205.199858s 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e837fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e841cb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e841cb780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e841cb780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55148/0 (socket says 172.21.15.164:55148) 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 learned_addr learned my addr 172.21.15.164:0/342842144 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 10 monclient(hunting): get_auth_request con 0x7f5e8413a5c0 auth_method 0 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e88efa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.712+0000 7f5e837fe640 10 monclient(hunting): get_auth_request con 0x7f5e8409d160 auth_method 0 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e8409c180 0x7f5e841cad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): get_auth_request con 0x7f5e8409c180 auth_method 0 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e841cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e841cb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e841cb240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e837fe640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e841cb240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient(hunting): handle_auth_done global_id 254310 payload 824 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 msgr2=0x7f5e841cb240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e8409d160 0x7f5e841cb240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e8409c180 msgr2=0x7f5e841cad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5e8409c180 0x7f5e841cad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:26.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient: _finish_hunting 0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 monclient: found mon.noname-a 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.713+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e840a5030 con 0x7f5e8413a5c0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 msgr2=0x7f5e841cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e88efa640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e841cb780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c02aa70 tx=0x7f5e7c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c040070 con 0x7f5e8413a5c0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: dump: 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.729 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: _finish_auth 0 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 5 monclient: authenticate success, global_id 254310 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e7c045660 con 0x7f5e8413a5c0 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:26.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e7c0314c0 con 0x7f5e8413a5c0 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e80ff9640 10 monclient: dump: 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:26.731 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 10 monclient: _renew_subs 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e840951a0 con 0x7f5e8413a5c0 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e827fc640 4 set_mon_vals no callback set 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 10 monclient: _renew_subs 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.715+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e84093b20 con 0x7f5e8413a5c0 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e8a95d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8409ea40 con 0x7f5e841c77a0 2024-09-10T16:30:26.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c77a0 auth_method 0 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5e7c039040 con 0x7f5e8413a5c0 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e80ff9640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5e7003e210 0x7f5e700406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.716+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(836..836 src has 677..836) ==== 4799+0+0 (secure 0 0 0) 0x7f5e7c074670 con 0x7f5e8413a5c0 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.717+0000 7f5e837fe640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5e7003e210 0x7f5e700406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.717+0000 7f5e837fe640 10 monclient: get_auth_request con 0x7f5e7003e210 auth_method 0 2024-09-10T16:30:26.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.719+0000 7f5e837fe640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5e7003e210 0x7f5e700406d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e840a0070 tx=0x7f5e78008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c77a0 auth_method 0 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:26.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:26.917+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:30:27.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:27.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c77a0 auth_method 0 2024-09-10T16:30:27.319 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:27.320 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:27.318+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:30:27.421 INFO:tasks.rados.rados.0.smithi165.stdout:3724: done (2 left) 2024-09-10T16:30:27.421 INFO:tasks.rados.rados.0.smithi165.stdout:3726: done (1 left) 2024-09-10T16:30:27.421 INFO:tasks.rados.rados.0.smithi165.stdout:3727: done (0 left) 2024-09-10T16:30:27.422 INFO:tasks.rados.rados.0.smithi165.stdout:3728: copy_from oid 35 from oid 11 current snap is 415 2024-09-10T16:30:27.422 INFO:tasks.rados.rados.0.smithi165.stdout:3729: snap_create 2024-09-10T16:30:27.435 INFO:tasks.rados.rados.0.smithi165.stdout:3728: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:30:27.438 INFO:tasks.rados.rados.0.smithi165.stdout:3728: finishing copy_from to smithi16534841-35 2024-09-10T16:30:27.438 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c77a0 auth_method 0 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 10 monclient(hunting): handle_auth_done global_id 274100 payload 824 2024-09-10T16:30:28.122 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.120+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e84092e40 con 0x7f5e841c77a0 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.122+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74015db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74005470 con 0x7f5e841c77a0 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:28.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 10 monclient: dump: 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.125 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5e74005470 con 0x7f5e841c77a0 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74015db0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 msgr2=0x7f5e841c7b80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:28.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.123+0000 7f5e80ff9640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c77a0 0x7f5e841c7b80 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74015db0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e8a95d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e8a95d640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5e84077260 con 0x7f5e841c80c0 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c80c0 auth_method 0 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 msgr2=0x7f5e8408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:30:28.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 msgr2=0x7f5e8408df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:30:28.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:28.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.130+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient: get_auth_request con 0x7f5e841c80c0 auth_method 0 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 10 monclient(hunting): handle_auth_done global_id 274103 payload 824 2024-09-10T16:30:28.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.331+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5e841c84d0 con 0x7f5e841c80c0 2024-09-10T16:30:28.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.333+0000 7f5e83fff640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 msgr2=0x7f5e8408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.333+0000 7f5e83fff640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74003760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e74031020 con 0x7f5e841c80c0 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 10 monclient: dump: 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f5e740067d0 con 0x7f5e841c80c0 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 msgr2=0x7f5e8408df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74003760 comp rx=0 tx=0).stop 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 msgr2=0x7f5e8408df80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.334+0000 7f5e80ff9640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5e841c80c0 0x7f5e8408df80 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5e740158f0 tx=0x7f5e74003760 comp rx=0 tx=0).stop 2024-09-10T16:30:28.337 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 5576, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:28.338 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.339 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.340 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:28.341 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.342 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:28.343 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.344 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.345 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:28.346 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:28.347 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.348 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.349 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:28.350 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:28.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5e7003e210 msgr2=0x7f5e700406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5e7003e210 0x7f5e700406d0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f5e840a0070 tx=0x7f5e78008040 comp rx=0 tx=0).stop 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 10 monclient: shutdown 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 msgr2=0x7f5e841cb780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 --2- 172.21.15.164:0/342842144 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5e8413a5c0 0x7f5e841cb780 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5e7c02aa70 tx=0x7f5e7c004a40 comp rx=0 tx=0).stop 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 reap_dead start 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e88efa640 1 -- 172.21.15.164:0/342842144 reap_dead start 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 shutdown_connections 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 >> 172.21.15.164:0/342842144 conn(0x7f5e840a3110 msgr2=0x7f5e84139a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 shutdown_connections 2024-09-10T16:30:28.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.335+0000 7f5e8a95d640 1 -- 172.21.15.164:0/342842144 wait complete. 2024-09-10T16:30:28.353 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:28.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:30:28.353 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:30:28.429 INFO:tasks.rados.rados.0.smithi165.stdout:3728: done (1 left) 2024-09-10T16:30:28.429 INFO:tasks.rados.rados.0.smithi165.stdout:3729: done (0 left) 2024-09-10T16:30:28.430 INFO:tasks.rados.rados.0.smithi165.stdout:3730: write (excl) oid 16 current snap is 416 2024-09-10T16:30:28.430 INFO:tasks.rados.rados.0.smithi165.stdout:3730: seq_num 1285 ranges {796995=758622,2161832=732447} 2024-09-10T16:30:28.441 INFO:tasks.rados.rados.0.smithi165.stdout:3730: writing smithi16534841-16 from 796995 to 1555617 tid 1 2024-09-10T16:30:28.445 INFO:tasks.rados.rados.0.smithi165.stdout:3730: writing smithi16534841-16 from 2161832 to 2894279 tid 2 2024-09-10T16:30:28.445 INFO:tasks.rados.rados.0.smithi165.stdout:3731: read oid 33 snap -1 2024-09-10T16:30:28.445 INFO:tasks.rados.rados.0.smithi165.stdout:3731: expect (ObjNum 1271 snap 407 seq_num 1271) 2024-09-10T16:30:28.445 INFO:tasks.rados.rados.0.smithi165.stdout:3732: write oid 19 current snap is 416 2024-09-10T16:30:28.445 INFO:tasks.rados.rados.0.smithi165.stdout:3732: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2024-09-10T16:30:28.450 INFO:tasks.rados.rados.0.smithi165.stdout:3732: writing smithi16534841-19 from 413802 to 847673 tid 1 2024-09-10T16:30:28.456 INFO:tasks.rados.rados.0.smithi165.stdout:3732: writing smithi16534841-19 from 1420314 to 1896419 tid 2 2024-09-10T16:30:28.464 INFO:tasks.rados.rados.0.smithi165.stdout:3732: writing smithi16534841-19 from 2482204 to 3273189 tid 3 2024-09-10T16:30:28.464 INFO:tasks.rados.rados.0.smithi165.stdout:3733: rollback oid 41 current snap is 416 2024-09-10T16:30:28.464 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 359 2024-09-10T16:30:28.464 INFO:tasks.rados.rados.0.smithi165.stdout:3730: finishing write tid 1 to smithi16534841-16 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: finishing write tid 2 to smithi16534841-16 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: finishing write tid 3 to smithi16534841-16 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: finishing write tid 4 to smithi16534841-16 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: oid 16 updating version 0 to 2416 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: oid 16 updating version 2416 to 2418 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: oid 16 version 2418 is already newer than 2417 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2418 (ObjNum 1285 snap 416 seq_num 1285) dirty exists 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3730: left oid 16 (ObjNum 1285 snap 416 seq_num 1285) 2024-09-10T16:30:28.465 INFO:tasks.rados.rados.0.smithi165.stdout:3731: expect (ObjNum 1271 snap 407 seq_num 1271) 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 10 monclient: get_monmap_and_config 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 10 monclient: build_initial_monmap 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 10 monclient: monmap: 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:28.467912+0000 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:28.467912+0000 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:28.468 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:28.469 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 1 Processor -- start 2024-09-10T16:30:28.471 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 1 -- start start 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.467+0000 7f1592bec640 10 monclient: init 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f158c06d160 0x7f158c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-a con 0x7f158c06d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f158c06d540 0x7f158c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-c con 0x7f158c06d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c0a4c20 0x7f158c137ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-b con 0x7f158c0a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f158c09cb90 con 0x7f158c06d160 2024-09-10T16:30:28.472 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f158c09ce70 con 0x7f158c06d540 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f158c09d150 con 0x7f158c0a4c20 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1592bec640 10 monclient(hunting): authenticate will time out at 2206.955520s 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f158c06d160 0x7f158c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c0a4c20 0x7f158c137ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f158c06d160 0x7f158c134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55162/0 (socket says 172.21.15.164:55162) 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): get_auth_request con 0x7f158c0a4c20 auth_method 0 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 1 -- 172.21.15.164:0/2354719622 learned_addr learned my addr 172.21.15.164:0/2354719622 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.473 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 1 --2- 172.21.15.164:0/2354719622 >> v2:172.21.15.165:3300/0 conn(0x7f158c06d540 0x7f158c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): get_auth_request con 0x7f158c06d160 auth_method 0 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): get_auth_request con 0x7f158c06d540 auth_method 0 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.468+0000 7f1590988640 10 monclient(hunting): handle_auth_done global_id 254313 payload 274 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 -- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c0a4c20 msgr2=0x7f158c137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 --2- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c0a4c20 0x7f158c137ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 -- 172.21.15.164:0/2354719622 >> v2:172.21.15.165:3300/0 conn(0x7f158c06d540 msgr2=0x7f158c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 --2- 172.21.15.164:0/2354719622 >> v2:172.21.15.165:3300/0 conn(0x7f158c06d540 0x7f158c0a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 10 monclient: _finish_hunting 0 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 monclient: found mon.noname-a 2024-09-10T16:30:28.475 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:28.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.476 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.469+0000 7f1590988640 1 -- 172.21.15.164:0/2354719622 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c135340 con 0x7f158c06d160 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.471+0000 7f1590988640 1 -- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d160 msgr2=0x7f158c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.471+0000 7f1590988640 1 --2- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d160 0x7f158c134e00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f158c0a9d40 tx=0x7f158002a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec953934bf6f3f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.471+0000 7f158b7fe640 1 -- 172.21.15.164:0/2354719622 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580037070 con 0x7f158c06d160 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: dump: 2024-09-10T16:30:28.480 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:30:28.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 1 -- 172.21.15.164:0/2354719622 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1580032640 con 0x7f158c06d160 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 5 monclient: authenticate success, global_id 254313 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 1 -- 172.21.15.164:0/2354719622 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580032a10 con 0x7f158c06d160 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 4 set_mon_vals no callback set 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f158b7fe640 10 monclient: dump: 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.482 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 10 monclient: shutdown 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 -- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d160 msgr2=0x7f158c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 --2- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d160 0x7f158c134e00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f158c0a9d40 tx=0x7f158002a0b0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 -- 172.21.15.164:0/2354719622 shutdown_connections 2024-09-10T16:30:28.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 --2- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c0a4c20 0x7f158c137ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 --2- 172.21.15.164:0/2354719622 >> v2:172.21.15.165:3300/0 conn(0x7f158c06d540 0x7f158c0a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 --2- 172.21.15.164:0/2354719622 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d160 0x7f158c134e00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 -- 172.21.15.164:0/2354719622 >> 172.21.15.164:0/2354719622 conn(0x7f158c0aacb0 msgr2=0x7f158c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 -- 172.21.15.164:0/2354719622 shutdown_connections 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.472+0000 7f1592bec640 1 -- 172.21.15.164:0/2354719622 wait complete. 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.473+0000 7f1592bec640 10 monclient: build_initial_monmap 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.473+0000 7f1592bec640 10 monclient: monmap: 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.484 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.473+0000 7f1592bec640 1 Processor -- start 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.473+0000 7f1592bec640 1 -- start start 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.473+0000 7f1592bec640 10 monclient: init 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-c con 0x7f158c06d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f158c1c0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-a con 0x7f158c06d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:28.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f158c1c07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): picked mon.noname-b con 0x7f158c0a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1385b0 con 0x7f158c06d540 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f158c138430 con 0x7f158c0a4c20 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f158c138730 con 0x7f158c06d160 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1592bec640 10 monclient(hunting): authenticate will time out at 2206.961430s 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f158c1c0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60020/0 (socket says 172.21.15.164:60020) 2024-09-10T16:30:28.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 learned_addr learned my addr 172.21.15.164:0/2467600899 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f158c1c0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55164/0 (socket says 172.21.15.164:55164) 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): get_auth_request con 0x7f158c06d160 auth_method 0 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): get_auth_request con 0x7f158c06d540 auth_method 0 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f158c1c07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): get_auth_request con 0x7f158c0a4c20 auth_method 0 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient(hunting): handle_auth_done global_id 274109 payload 824 2024-09-10T16:30:28.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 msgr2=0x7f158c1c07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f158c1c07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 msgr2=0x7f158c1c0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f158c1c0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient: _finish_hunting 0 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 monclient: found mon.noname-c 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c09eaa0 con 0x7f158c06d160 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f158c1c0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 msgr2=0x7f158c09e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f15800090b0 tx=0x7f15800046c0 comp rx=0 tx=0).write compression request write failed r=-32 ((32) Broken pipe) 2024-09-10T16:30:28.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.474+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:28.673 INFO:tasks.rados.rados.0.smithi165.stdout:3732: finishing write tid 1 to smithi16534841-19 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3732: finishing write tid 2 to smithi16534841-19 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3733: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3733: finishing rollback tid 2 to smithi16534841-41 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 2042 (ObjNum 1139 snap 353 seq_num 1139) dirty exists 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3732: finishing write tid 3 to smithi16534841-19 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3730: done (3 left) 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3731: done (2 left) 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3733: done (1 left) 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3734: read oid 47 snap -1 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3734: expect (ObjNum 1138 snap 351 seq_num 1138) 2024-09-10T16:30:28.674 INFO:tasks.rados.rados.0.smithi165.stdout:3735: snap_remove snap 405 2024-09-10T16:30:28.679 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient: get_auth_request con 0x7f158c06d160 auth_method 0 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 0 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 msgr2=0x7f158c09e110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f158c06d160 0x7f158c09e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f15700054b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): picked mon.noname-a con 0x7f158c06d540 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:28.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f1570007fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): picked mon.noname-b con 0x7f158c0a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1570008a10 0x7f157000aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): picked mon.noname-c con 0x7f1570008a10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f157000b9a0 con 0x7f158c06d540 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f157000bb20 con 0x7f158c0a4c20 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f157000bd20 con 0x7f1570008a10 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f15897fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:28.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f15700054b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): get_auth_request con 0x7f158c06d540 auth_method 0 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1570008a10 0x7f157000aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): get_auth_request con 0x7f1570008a10 auth_method 0 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:30:28.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f1570007fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): get_auth_request con 0x7f158c0a4c20 auth_method 0 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1591189640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient(hunting): handle_auth_done global_id 274109 payload 293 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1570008a10 msgr2=0x7f157000aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1570008a10 0x7f157000aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 msgr2=0x7f1570007fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c0a4c20 0x7f1570007fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient: _finish_hunting 0 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 monclient: found mon.noname-a 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.675+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157000bf80 con 0x7f158c06d540 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1590988640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 msgr2=0x7f15700054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1590988640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f15700054b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f158002a0f0 tx=0x7f158002e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580037040 con 0x7f158c06d540 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.684 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: dump: 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: _finish_auth 0 2024-09-10T16:30:28.685 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: _check_auth_tickets 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f158003d9f0 con 0x7f158c06d540 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 5 monclient: authenticate success, global_id 274109 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 10 monclient: _renew_subs 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158c09f070 con 0x7f158c06d540 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f158003dda0 con 0x7f158c06d540 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f15897fa640 10 monclient: dump: 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.686 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 10 monclient: _renew_subs 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f158affd640 4 set_mon_vals no callback set 2024-09-10T16:30:28.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.678+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158c09f630 con 0x7f158c06d540 2024-09-10T16:30:28.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1592bec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1592bec640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 0x7f158c12fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0a46e0 con 0x7f158c09e110 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f158003b3d0 con 0x7f158c06d540 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 0x7f158c12fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient: get_auth_request con 0x7f158c09e110 auth_method 0 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158c06d160 0x7f15700485b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(838..838 src has 677..838) ==== 4771+0+0 (secure 0 0 0) 0x7f1580073270 con 0x7f158c06d540 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 10 monclient(hunting): handle_auth_done global_id 264196 payload 824 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f158bfff640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f158c1c13e0 con 0x7f158c09e110 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1591189640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158c06d160 0x7f15700485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.679+0000 7f1591189640 10 monclient: get_auth_request con 0x7f158c06d160 auth_method 0 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.681+0000 7f158bfff640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 msgr2=0x7f158c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.681+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 0x7f158c12fff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f158c09e0a0 tx=0x7f1578012700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578004450 con 0x7f158c09e110 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:28.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 10 monclient: dump: 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f1591189640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158c06d160 0x7f15700485b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f158c09e050 tx=0x7f158400b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1578004450 con 0x7f158c09e110 2024-09-10T16:30:28.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:28.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 msgr2=0x7f158c12fff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 0x7f158c12fff0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f158c09e0a0 tx=0x7f1578012700 comp rx=0 tx=0).stop 2024-09-10T16:30:28.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 msgr2=0x7f158c12fff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:28.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.682+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c09e110 0x7f158c12fff0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.689+0000 7f1592bec640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.689+0000 7f1592bec640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 0x7f158c130940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.689+0000 7f1592bec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.689+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f158c0aa1e0 con 0x7f158c130530 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 0x7f158c130940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient: get_auth_request con 0x7f158c130530 auth_method 0 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 10 monclient(hunting): handle_auth_done global_id 264199 payload 824 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.690+0000 7f158bfff640 1 -- 172.21.15.164:0/2467600899 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f158c0770b0 con 0x7f158c130530 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.692+0000 7f158bfff640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 msgr2=0x7f158c130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.692+0000 7f158bfff640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 0x7f158c130940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f157801e000 tx=0x7f1578004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1578025e60 con 0x7f158c130530 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 10 monclient: dump: 2024-09-10T16:30:28.696 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f1578003610 con 0x7f158c130530 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:28.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 msgr2=0x7f158c130940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 0x7f158c130940 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f157801e000 tx=0x7f1578004730 comp rx=0 tx=0).stop 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 msgr2=0x7f158c130940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f15897fa640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f158c130530 0x7f158c130940 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f157801e000 tx=0x7f1578004730 comp rx=0 tx=0).stop 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:28.698 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 50835, 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.699 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.700 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:28.701 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:28.702 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:28.702 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:28.702 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:28.704 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:28.705 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.706 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.707 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:28.708 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:28.709 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.710 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:28.711 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:28.712 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:28.713 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:28.713 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:28.713 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:28.713 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:28.713 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:28.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158c06d160 msgr2=0x7f15700485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f158c06d160 0x7f15700485b0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f158c09e050 tx=0x7f158400b040 comp rx=0 tx=0).stop 2024-09-10T16:30:28.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 10 monclient: shutdown 2024-09-10T16:30:28.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 msgr2=0x7f15700054b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 --2- 172.21.15.164:0/2467600899 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f158c06d540 0x7f15700054b0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f158002a0f0 tx=0x7f158002e000 comp rx=0 tx=0).stop 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1591189640 1 -- 172.21.15.164:0/2467600899 reap_dead start 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1591189640 1 -- 172.21.15.164:0/2467600899 reap_dead start 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 shutdown_connections 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 >> 172.21.15.164:0/2467600899 conn(0x7f158c0aacb0 msgr2=0x7f158c0a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 shutdown_connections 2024-09-10T16:30:28.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.693+0000 7f1592bec640 1 -- 172.21.15.164:0/2467600899 wait complete. 2024-09-10T16:30:28.715 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:28.715 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:30:28.716 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:30:28.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.812+0000 7f1b09100640 10 monclient: get_monmap_and_config 2024-09-10T16:30:28.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.813+0000 7f1b09100640 10 monclient: build_initial_monmap 2024-09-10T16:30:28.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.813+0000 7f1b09100640 10 monclient: monmap: 2024-09-10T16:30:28.814 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:28.814 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:28.813897+0000 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:28.813897+0000 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:28.815 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.813+0000 7f1b09100640 1 Processor -- start 2024-09-10T16:30:28.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.813+0000 7f1b09100640 1 -- start start 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.813+0000 7f1b09100640 10 monclient: init 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b0409d930 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b0409ab50 0x7f1b041397c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 0x7f1b0413bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0409e420 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b040a0870 con 0x7f1b0409d930 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f1b0409d3e0 con 0x7f1b0409ab50 2024-09-10T16:30:28.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f1b04074c70 con 0x7f1b0409e420 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b09100640 10 monclient(hunting): authenticate will time out at 2207.301276s 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f1b0409ab50 0x7f1b041397c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b02d9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1b0409d930 0x7f1b0409dd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38554/0 (socket says 172.21.15.164:38554) 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b02d9d640 1 -- 172.21.15.164:0/3357758786 learned_addr learned my addr 172.21.15.164:0/3357758786 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): get_auth_request con 0x7f1b0409ab50 auth_method 0 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b0359e640 1 --2- 172.21.15.164:0/3357758786 >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 0x7f1b0413bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient(hunting): handle_auth_done global_id 254319 payload 274 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 -- 172.21.15.164:0/3357758786 >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 msgr2=0x7f1b0413bd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 --2- 172.21.15.164:0/3357758786 >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 0x7f1b0413bd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 -- 172.21.15.164:0/3357758786 >> v1:172.21.15.164:6789/0 conn(0x7f1b0409d930 legacy=0x7f1b0409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 monclient: found mon.noname-a 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1afbfff640 1 -- 172.21.15.164:0/3357758786 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b0409ecb0 con 0x7f1b0409ab50 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b0359e640 10 monclient: get_auth_request con 0x7f1b0409e420 auth_method 0 2024-09-10T16:30:28.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.814+0000 7f1b0359e640 1 --2- 172.21.15.164:0/3357758786 >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 0x7f1b0413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:30:28.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1afbfff640 1 -- 172.21.15.164:0/3357758786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409ab50 msgr2=0x7f1b041397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1afbfff640 1 --2- 172.21.15.164:0/3357758786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409ab50 0x7f1b041397c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b0409af60 tx=0x7f1aec02b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd28cdb10f17ca8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 1 -- 172.21.15.164:0/3357758786 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec038070 con 0x7f1b0409ab50 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: dump: 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: _finish_auth 0 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: _check_auth_tickets 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 1 -- 172.21.15.164:0/3357758786 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aec0306b0 con 0x7f1b0409ab50 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b09100640 5 monclient: authenticate success, global_id 254319 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b09100640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 1 -- 172.21.15.164:0/3357758786 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aec00b040 con 0x7f1b0409ab50 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b09100640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b09100640 4 set_mon_vals no callback set 2024-09-10T16:30:28.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.817+0000 7f1b0259c640 10 monclient: dump: 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 10 monclient: shutdown 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 -- 172.21.15.164:0/3357758786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409ab50 msgr2=0x7f1b041397c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 --2- 172.21.15.164:0/3357758786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409ab50 0x7f1b041397c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1b0409af60 tx=0x7f1aec02b3a0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 -- 172.21.15.164:0/3357758786 shutdown_connections 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 --2- 172.21.15.164:0/3357758786 >> v2:172.21.15.165:3300/0 conn(0x7f1b0409e420 0x7f1b0413bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 --2- 172.21.15.164:0/3357758786 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409ab50 0x7f1b041397c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 -- 172.21.15.164:0/3357758786 >> 172.21.15.164:0/3357758786 conn(0x7f1b040a3110 msgr2=0x7f1b040a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 -- 172.21.15.164:0/3357758786 shutdown_connections 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 1 -- 172.21.15.164:0/3357758786 wait complete. 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 10 monclient: build_initial_monmap 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.818+0000 7f1b09100640 10 monclient: monmap: 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:28.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 Processor -- start 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 -- start start 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient: init 2024-09-10T16:30:28.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b0409ab50 0x7f1b041c7700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-b con 0x7f1b0409ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b0409d930 0x7f1b041c7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-c con 0x7f1b0409d930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): picked mon.noname-a con 0x7f1b0409e420 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f1b040a2940 con 0x7f1b0409e420 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f1b0409ef30 con 0x7f1b0409ab50 2024-09-10T16:30:28.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1b040a2ac0 con 0x7f1b0409d930 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b09100640 10 monclient(hunting): authenticate will time out at 2207.306727s 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55178/0 (socket says 172.21.15.164:55178) 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b0409d930 0x7f1b041c7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 1 -- 172.21.15.164:0/1702268206 learned_addr learned my addr 172.21.15.164:0/1702268206 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): get_auth_request con 0x7f1b0409e420 auth_method 0 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): get_auth_request con 0x7f1b0409d930 auth_method 0 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1b0359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:28.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.819+0000 7f1afbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b02d9d640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b0409ab50 0x7f1b041c7700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b02d9d640 10 monclient(hunting): get_auth_request con 0x7f1b0409ab50 auth_method 0 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b02d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b02d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b02d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 10 monclient(hunting): handle_auth_done global_id 254322 payload 824 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 -- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b0409d930 msgr2=0x7f1b041c7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b0409d930 0x7f1b041c7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 -- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b0409ab50 msgr2=0x7f1b041c7700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b0409ab50 0x7f1b041c7700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 10 monclient: _finish_hunting 0 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 monclient: found mon.noname-a 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:28.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.820+0000 7f1b0359e640 1 -- 172.21.15.164:0/1702268206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b04131720 con 0x7f1b0409e420 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b0359e640 1 -- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 msgr2=0x7f1b041c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b0359e640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00cef0 tx=0x7f1afc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc01a070 con 0x7f1b0409e420 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: dump: 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.835 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: _finish_auth 0 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 5 monclient: authenticate success, global_id 254322 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1afc00da70 con 0x7f1b0409e420 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 10 monclient: _renew_subs 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b0412f3c0 con 0x7f1b0409e420 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:28.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1afc00de60 con 0x7f1b0409e420 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1afb7fe640 10 monclient: dump: 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:28.837 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 10 monclient: _renew_subs 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b01d9b640 4 set_mon_vals no callback set 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.822+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b04130340 con 0x7f1b0409e420 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1b09100640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1b09100640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1b09100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b0409dd10 con 0x7f1b0409e420 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f1afc01d070 con 0x7f1b0409e420 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1afb7fe640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 0x7f1ae40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.823+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(838..838 src has 677..838) ==== 4771+0+0 (secure 0 0 0) 0x7f1afc053620 con 0x7f1b0409e420 2024-09-10T16:30:28.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.824+0000 7f1b02d9d640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 0x7f1ae40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:28.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.824+0000 7f1b02d9d640 10 monclient: get_auth_request con 0x7f1ae403e210 auth_method 0 2024-09-10T16:30:28.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.826+0000 7f1b02d9d640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 0x7f1ae40406d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c7f0 tx=0x7f1af4009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:28.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.828+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f1afc01ccd0 con 0x7f1b0409e420 2024-09-10T16:30:28.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.828+0000 7f1afb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:28.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:28.828+0000 7f1afb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:29.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.070+0000 7f1b09100640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:30:29.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.070+0000 7f1b09100640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:30:29.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.070+0000 7f1b09100640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:29.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.070+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1b0406daa0 con 0x7f1b0409e420 2024-09-10T16:30:29.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f1afb7fe640 1 -- 172.21.15.164:0/1702268206 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f1afc008320 con 0x7f1b0409e420 2024-09-10T16:30:29.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f1afb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:30:29.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f1afb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3070773286,config=2586702516,health=1799447023,kv=2249883377,logm=2068224881}) 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=2874250231,config=2586702516,health=1799447023,kv=1176791497,logm=37582893}) 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=29,config=2,health=12,kv=11,logm=46} crc {auth=3070773286,config=2586702516,health=1799447023,kv=2249883377,logm=2068224881}) 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.071+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=1,health=12,kv=11,logm=50} crc {auth=3235258585,config=1747058506,health=684638671,kv=805391194,logm=3466589375}) 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.074 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681374261}) 2024-09-10T16:30:29.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3929236305}) 2024-09-10T16:30:29.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2681374261}) 2024-09-10T16:30:29.075 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.072+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3805762048}) 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.073+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.073+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962203879}) 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.073+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117082439}) 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.073+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2962203879}) 2024-09-10T16:30:29.076 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3948690100}) 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 msgr2=0x7f1ae40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 0x7f1ae40406d0 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f1af400c7f0 tx=0x7f1af4009040 comp rx=0 tx=0).stop 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 10 monclient: shutdown 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 msgr2=0x7f1b041c8180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1afc00cef0 tx=0x7f1afc00a710 comp rx=0 tx=0).stop 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 shutdown_connections 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f1ae403e210 0x7f1ae40406d0 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f1b0409e420 0x7f1b041c8180 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:29.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1b0409d930 0x7f1b041c7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:29.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 --2- 172.21.15.164:0/1702268206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f1b0409ab50 0x7f1b041c7700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:29.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 >> 172.21.15.164:0/1702268206 conn(0x7f1b040a3110 msgr2=0x7f1b04073d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:29.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 shutdown_connections 2024-09-10T16:30:29.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:29.074+0000 7f1b09100640 1 -- 172.21.15.164:0/1702268206 wait complete. 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831265009}) 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1841467157}) 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2831265009}) 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.075+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3169119421}) 2024-09-10T16:30:29.080 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095052879}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3146615663}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095052879}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.076+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3981183873}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947331376}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=452748936}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947331376}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.077+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696069408}) 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.081 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366983753}) 2024-09-10T16:30:29.082 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4144811366}) 2024-09-10T16:30:29.082 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.082 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=366983753}) 2024-09-10T16:30:29.082 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.078+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443692875}) 2024-09-10T16:30:29.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:30:29.360 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.361 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369550593}) 2024-09-10T16:30:29.361 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1019779090}) 2024-09-10T16:30:29.361 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.361 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369550593}) 2024-09-10T16:30:29.361 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=711694834}) 2024-09-10T16:30:29.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091456763}) 2024-09-10T16:30:29.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4264093246}) 2024-09-10T16:30:29.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1091456763}) 2024-09-10T16:30:29.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.425+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=251857519}) 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243704746}) 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691011716}) 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3243704746}) 2024-09-10T16:30:29.432 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.431+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1205693777}) 2024-09-10T16:30:29.433 INFO:tasks.rados.rados.0.smithi165.stdout:3735: done (2 left) 2024-09-10T16:30:29.433 INFO:tasks.rados.rados.0.smithi165.stdout:3736: snap_remove snap 404 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293296940}) 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1745171889}) 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=293296940}) 2024-09-10T16:30:29.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:29.438+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1228201674}) 2024-09-10T16:30:30.089 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:30:30.089 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:30.152 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.152 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412256465}) 2024-09-10T16:30:30.152 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565274463}) 2024-09-10T16:30:30.152 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.152 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1412256465}) 2024-09-10T16:30:30.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.150+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224245307}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042296549}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3501691851}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4042296549}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.151+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022584010}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606747478}) 2024-09-10T16:30:30.154 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1697475894}) 2024-09-10T16:30:30.155 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.155 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1606747478}) 2024-09-10T16:30:30.155 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.152+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328389489}) 2024-09-10T16:30:30.156 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.156 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083981839}) 2024-09-10T16:30:30.156 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3789307659}) 2024-09-10T16:30:30.156 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.156 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2083981839}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.153+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2646208856}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872015217}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=435830973}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2872015217}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.154+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3553897802}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327520027}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1110403117}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1327520027}) 2024-09-10T16:30:30.157 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3118752816}) 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795054867}) 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=993250874}) 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795054867}) 2024-09-10T16:30:30.159 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.155+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=16399935}) 2024-09-10T16:30:30.160 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.160 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603696478}) 2024-09-10T16:30:30.160 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1893094137}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603696478}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.156+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=615422917}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789054594}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=307893217}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2789054594}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.157+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3559096285}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119782791}) 2024-09-10T16:30:30.161 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=333224246}) 2024-09-10T16:30:30.162 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.162 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119782791}) 2024-09-10T16:30:30.162 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3575201017}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983026138}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4236690797}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=983026138}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.158+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192043952}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123688783}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626996806}) 2024-09-10T16:30:30.165 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4123688783}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.159+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1904189586}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237609584}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4273427387}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4237609584}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.160+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3411400690}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133192345}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781998576}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3133192345}) 2024-09-10T16:30:30.166 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=10097143}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508354450}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3608218109}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1508354450}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.161+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4100777404}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872230212}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3558796439}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1872230212}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.162+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1638177990}) 2024-09-10T16:30:30.167 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.168 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109575184}) 2024-09-10T16:30:30.168 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=451430880}) 2024-09-10T16:30:30.168 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.168 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3109575184}) 2024-09-10T16:30:30.168 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.163+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010059009}) 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2806568594}) 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815247974}) 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2806568594}) 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.164+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2570956354}) 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.171 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415669865}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2182825147}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=415669865}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2136566558}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977021746}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=696332342}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1977021746}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.165+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2020836440}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515342641}) 2024-09-10T16:30:30.172 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1643635522}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1515342641}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.166+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1482344889}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839839323}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2683206945}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839839323}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.167+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=709777634}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966570587}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512420186}) 2024-09-10T16:30:30.173 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3966570587}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.168+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1141462243}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313677535}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1478340053}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313677535}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3847679037}) 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.174 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434223746}) 2024-09-10T16:30:30.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1425139503}) 2024-09-10T16:30:30.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.169+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1434223746}) 2024-09-10T16:30:30.175 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=952265606}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151579738}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3489982224}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3151579738}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.170+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=252931111}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917079177}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1895914364}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=917079177}) 2024-09-10T16:30:30.179 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.171+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2358636443}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106873718}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3360204745}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1106873718}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=733759228}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.172+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223998953}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339064787}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3223998953}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377479631}) 2024-09-10T16:30:30.180 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=628110376}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4087899851}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=628110376}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.173+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3660356598}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555774918}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2001915208}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1555774918}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.174+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3691836519}) 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.181 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315003796}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3020827705}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3315003796}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2897538686}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174799051}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.175+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3735275632}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.176+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.176+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174799051}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.176+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1078496696}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.176+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.176+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654906181}) 2024-09-10T16:30:30.182 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236457142}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1654906181}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3940308265}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020220762}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4168742188}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020220762}) 2024-09-10T16:30:30.183 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.177+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1618121246}) 2024-09-10T16:30:30.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.187 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849206458}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1041297199}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=849206458}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.178+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2557765995}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385206876}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4189773247}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385206876}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.179+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3356102277}) 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.188 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661399297}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2949444376}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=661399297}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.180+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4034294585}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458982539}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90182141}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458982539}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.181+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3253090113}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494425868}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=121671233}) 2024-09-10T16:30:30.189 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1494425868}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=6271753}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401995512}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3264834283}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2401995512}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.182+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2849049479}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043309195}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1846314483}) 2024-09-10T16:30:30.190 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4043309195}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.183+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3851972563}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899478896}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3832821934}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899478896}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.184+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078612907}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175103520}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1695286179}) 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.191 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=175103520}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425924037}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635643615}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2706785318}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635643615}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.185+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1423078850}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332648597}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1529612867}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1332648597}) 2024-09-10T16:30:30.192 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.186+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1432756941}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045367758}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3374842091}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4045367758}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3119689865}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529191465}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2431018049}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.187+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2529191465}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=829567982}) 2024-09-10T16:30:30.198 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765852961}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021658710}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1765852961}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.188+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=969634703}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536696204}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1426279260}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536696204}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.189+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=706733979}) 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.199 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152418800}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2490441407}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2152418800}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2279648865}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499271206}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497318493}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2499271206}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.190+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2332648181}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606179394}) 2024-09-10T16:30:30.200 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=31762162}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606179394}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.191+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=792849600}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620097160}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4185159395}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3620097160}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.192+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=197368814}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4155991644}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1934748520}) 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.201 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4155991644}) 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=810191846}) 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701798329}) 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1834068426}) 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=701798329}) 2024-09-10T16:30:30.202 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=996895396}) 2024-09-10T16:30:30.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f6d64b32640 10 monclient: get_monmap_and_config 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f6d64b32640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.193+0000 7f6d64b32640 10 monclient: monmap: 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:30.194835+0000 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:30.194835+0000 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:30.203 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:30.204 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:30.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f6d64b32640 1 Processor -- start 2024-09-10T16:30:30.204 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f6d64b32640 1 -- start start 2024-09-10T16:30:30.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f6d64b32640 10 monclient: init 2024-09-10T16:30:30.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073815728}) 2024-09-10T16:30:30.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2950938994}) 2024-09-10T16:30:30.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1073815728}) 2024-09-10T16:30:30.205 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.194+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3523260813}) 2024-09-10T16:30:30.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 0x7f6d60134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-b con 0x7f6d6006cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d600a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6d600a4a80 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d6009ce50 con 0x7f6d600a46a0 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6d60138ac0 con 0x7f6d600a4a80 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d6006db60 con 0x7f6d6006cb70 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d64b32640 10 monclient(hunting): authenticate will time out at 2208.682399s 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5dd9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f6d600a46a0 0x7f6d60137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38570/0 (socket says 172.21.15.164:38570) 2024-09-10T16:30:30.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5dd9b640 1 -- 172.21.15.164:0/2608257722 learned_addr learned my addr 172.21.15.164:0/2608257722 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:30.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 0x7f6d60134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.208 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649061284}) 2024-09-10T16:30:30.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): get_auth_request con 0x7f6d6006cb70 auth_method 0 2024-09-10T16:30:30.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.209 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2135497106}) 2024-09-10T16:30:30.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=649061284}) 2024-09-10T16:30:30.210 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3123177991}) 2024-09-10T16:30:30.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_done global_id 274115 payload 274 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 1 -- 172.21.15.164:0/2608257722 >> v1:172.21.15.165:6789/0 conn(0x7f6d600a4a80 legacy=0x7f6d601383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 1 -- 172.21.15.164:0/2608257722 >> v1:172.21.15.164:6789/0 conn(0x7f6d600a46a0 legacy=0x7f6d60137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 1 monclient: found mon.noname-b 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:30:30.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.195+0000 7f6d5e59c640 1 -- 172.21.15.164:0/2608257722 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d6009cb70 con 0x7f6d6006cb70 2024-09-10T16:30:30.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153445315}) 2024-09-10T16:30:30.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366801924}) 2024-09-10T16:30:30.212 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2153445315}) 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.196+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3984901102}) 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450027020}) 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1705764298}) 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3450027020}) 2024-09-10T16:30:30.213 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.197+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4262653339}) 2024-09-10T16:30:30.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5e59c640 1 -- 172.21.15.164:0/2608257722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 msgr2=0x7f6d60134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5e59c640 1 --2- 172.21.15.164:0/2608257722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 0x7f6d60134f70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d500097d0 tx=0x7f6d50029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b461588d7476c79e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198313437}) 2024-09-10T16:30:30.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4025954893}) 2024-09-10T16:30:30.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.214 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4198313437}) 2024-09-10T16:30:30.215 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885689207}) 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 1 -- 172.21.15.164:0/2608257722 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50037070 con 0x7f6d6006cb70 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: dump: 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 1 -- 172.21.15.164:0/2608257722 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d50032440 con 0x7f6d6006cb70 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d64b32640 5 monclient: authenticate success, global_id 274115 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d64b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d64b32640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 1 -- 172.21.15.164:0/2608257722 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d50032810 con 0x7f6d6006cb70 2024-09-10T16:30:30.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d64b32640 4 set_mon_vals no callback set 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.198+0000 7f6d5d59a640 10 monclient: dump: 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 10 monclient: shutdown 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 -- 172.21.15.164:0/2608257722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 msgr2=0x7f6d60134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 --2- 172.21.15.164:0/2608257722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 0x7f6d60134f70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f6d500097d0 tx=0x7f6d50029fb0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 -- 172.21.15.164:0/2608257722 shutdown_connections 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 --2- 172.21.15.164:0/2608257722 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d6006cb70 0x7f6d60134f70 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 -- 172.21.15.164:0/2608257722 >> 172.21.15.164:0/2608257722 conn(0x7f6d600aa900 msgr2=0x7f6d600aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 -- 172.21.15.164:0/2608257722 shutdown_connections 2024-09-10T16:30:30.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f6d64b32640 1 -- 172.21.15.164:0/2608257722 wait complete. 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322940285}) 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1743218162}) 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3322940285}) 2024-09-10T16:30:30.220 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.199+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3068782522}) 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f6d64b32640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f6d64b32640 10 monclient: monmap: 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:30.221 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:30.222 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:30.222 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.222 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079281589}) 2024-09-10T16:30:30.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1581886125}) 2024-09-10T16:30:30.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3079281589}) 2024-09-10T16:30:30.223 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1446052430}) 2024-09-10T16:30:30.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f6d64b32640 1 Processor -- start 2024-09-10T16:30:30.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f6d64b32640 1 -- start start 2024-09-10T16:30:30.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.200+0000 7f6d64b32640 10 monclient: init 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865248202}) 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=639891802}) 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2865248202}) 2024-09-10T16:30:30.224 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1639963060}) 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 0x7f6d601cd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-b con 0x7f6d6006cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 0x7f6d6013ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-a con 0x7f6d600a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d600a4a80 0x7f6d60140520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): picked mon.noname-c con 0x7f6d600a4a80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6d6006cf50 con 0x7f6d600a46a0 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6d60141050 con 0x7f6d6006cb70 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6d601411d0 con 0x7f6d600a4a80 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d64b32640 10 monclient(hunting): authenticate will time out at 2208.688291s 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 0x7f6d6013ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 0x7f6d601cd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d600a4a80 0x7f6d60140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 0x7f6d601cd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:56630/0 (socket says 172.21.15.164:56630) 2024-09-10T16:30:30.228 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 1 -- 172.21.15.164:0/106329833 learned_addr learned my addr 172.21.15.164:0/106329833 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): get_auth_request con 0x7f6d600a46a0 auth_method 0 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): get_auth_request con 0x7f6d600a4a80 auth_method 0 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): get_auth_request con 0x7f6d6006cb70 auth_method 0 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.229 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient(hunting): handle_auth_done global_id 254325 payload 824 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 -- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d600a4a80 msgr2=0x7f6d60140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d600a4a80 0x7f6d60140520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 -- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 msgr2=0x7f6d601cd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 0x7f6d601cd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 monclient: found mon.noname-a 2024-09-10T16:30:30.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.201+0000 7f6d5dd9b640 1 -- 172.21.15.164:0/106329833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d600a4000 con 0x7f6d600a46a0 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147673786}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1251954140}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=147673786}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1224038807}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338227812}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1994399839}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3338227812}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.202+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=61355614}) 2024-09-10T16:30:30.233 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475173753}) 2024-09-10T16:30:30.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2099382933}) 2024-09-10T16:30:30.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475173753}) 2024-09-10T16:30:30.234 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.203+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2647794664}) 2024-09-10T16:30:30.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f6d5dd9b640 1 -- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 msgr2=0x7f6d6013ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f6d5dd9b640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 0x7f6d6013ffe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c3e0 tx=0x7f6d5400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380296917}) 2024-09-10T16:30:30.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644565305}) 2024-09-10T16:30:30.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.241 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=380296917}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.204+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3495706480}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=88321289}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=850313787}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=88321289}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1608508557}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360843030}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2004492431}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2360843030}) 2024-09-10T16:30:30.242 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.205+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=9802677}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356359827}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1157796111}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1356359827}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.206+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=63673209}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468242762}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3978529359}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2468242762}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.207+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3265520907}) 2024-09-10T16:30:30.243 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124977800}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4244608181}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2124977800}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=589817000}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167249384}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3776557424}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1167249384}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.208+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2799780184}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827224825}) 2024-09-10T16:30:30.244 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2252737002}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3827224825}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.209+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1304296962}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830581833}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=339928121}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1830581833}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.210+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229128162}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202987059}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925724817}) 2024-09-10T16:30:30.245 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1202987059}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=381604530}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.211+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382247992}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745620800}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2382247992}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=240122561}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605958225}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3131228021}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1605958225}) 2024-09-10T16:30:30.246 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.212+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2003097221}) 2024-09-10T16:30:30.247 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d54017070 con 0x7f6d600a46a0 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: dump: 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.248 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634557562}) 2024-09-10T16:30:30.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d540070d0 con 0x7f6d600a46a0 2024-09-10T16:30:30.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 5 monclient: authenticate success, global_id 254325 2024-09-10T16:30:30.249 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2484632099}) 2024-09-10T16:30:30.249 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 10 monclient: _renew_subs 2024-09-10T16:30:30.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d6006e720 con 0x7f6d600a46a0 2024-09-10T16:30:30.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634557562}) 2024-09-10T16:30:30.250 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=431127346}) 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d540074c0 con 0x7f6d600a46a0 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d4effd640 10 monclient: dump: 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.251 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 10 monclient: _renew_subs 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d6013aa70 con 0x7f6d600a46a0 2024-09-10T16:30:30.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.213+0000 7f6d5cd99640 4 set_mon_vals no callback set 2024-09-10T16:30:30.252 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.252 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634798962}) 2024-09-10T16:30:30.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f6d64b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f6d64b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f6d64b32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3172492875}) 2024-09-10T16:30:30.253 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d60134f70 con 0x7f6d600a46a0 2024-09-10T16:30:30.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634798962}) 2024-09-10T16:30:30.254 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.214+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1486461694}) 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.215+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6d5401a070 con 0x7f6d600a46a0 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.215+0000 7f6d4effd640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 0x7f6d400406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.215+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(839..839 src has 677..839) ==== 4815+0+0 (secure 0 0 0) 0x7f6d54052590 con 0x7f6d600a46a0 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.215+0000 7f6d5e59c640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 0x7f6d400406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.215+0000 7f6d5e59c640 10 monclient: get_auth_request con 0x7f6d4003e210 auth_method 0 2024-09-10T16:30:30.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.218+0000 7f6d5e59c640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 0x7f6d400406d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6d50029ff0 tx=0x7f6d5002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916162626}) 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2318888515}) 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3916162626}) 2024-09-10T16:30:30.256 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.219+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1470565712}) 2024-09-10T16:30:30.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f6d54021020 con 0x7f6d600a46a0 2024-09-10T16:30:30.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f6d4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f6d4effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:30.263 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820713660}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3076079344}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820713660}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.220+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3778878062}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359039752}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1985030886}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359039752}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.221+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3069862961}) 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.264 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606597673}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1331541364}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3606597673}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.222+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192645111}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369735326}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2125187083}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369735326}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=497946702}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106178989}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1143940478}) 2024-09-10T16:30:30.265 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=106178989}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.223+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=699156076}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110146977}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1183606954}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=110146977}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.224+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2678202086}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271864574}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1366517609}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1271864574}) 2024-09-10T16:30:30.266 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.225+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1014616282}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556623199}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3121534662}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556623199}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.226+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1405551036}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=4293795454,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3939607973,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1704170810,osd_snap=1591159882}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=10} crc {logm=1805827729,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3321672722,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=1740073500,osd_pg_creating=1704170810,osd_snap=43778211}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=9} crc {logm=4293795454,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3939607973,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1704170810,osd_snap=1591159882}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=37,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=4289860725,mdsmap=931935893,mgr=2907349437,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=1453677847,mon_config_key=665552247,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1704170810,osd_snap=2397309466}) 2024-09-10T16:30:30.267 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3351769060}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=3685486020}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3351769060}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.227+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=2253836055}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=55,osdmap=45} crc {osd_snap=122219161,osdmap=1736416308}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=57,osdmap=43} crc {osd_snap=2777492192,osdmap=4201162871}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=55,osdmap=45} crc {osd_snap=122219161,osdmap=1736416308}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.228+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=51,osdmap=49} crc {osd_snap=2636374333,osdmap=1020468230}) 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.268 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=878581820}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1693492068}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=878581820}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.229+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1288668927}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=905638622}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=347339133}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=905638622}) 2024-09-10T16:30:30.269 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:30:30.231+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2842380000}) 2024-09-10T16:30:30.435 INFO:tasks.rados.rados.0.smithi165.stdout:3736: done (2 left) 2024-09-10T16:30:30.435 INFO:tasks.rados.rados.0.smithi165.stdout:3737: write (excl) oid 39 current snap is 416 2024-09-10T16:30:30.435 INFO:tasks.rados.rados.0.smithi165.stdout:3737: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2024-09-10T16:30:30.440 INFO:tasks.rados.rados.0.smithi165.stdout:3737: writing smithi16534841-39 from 430609 to 939729 tid 1 2024-09-10T16:30:30.448 INFO:tasks.rados.rados.0.smithi165.stdout:3737: writing smithi16534841-39 from 1478796 to 2214912 tid 2 2024-09-10T16:30:30.456 INFO:tasks.rados.rados.0.smithi165.stdout:3737: writing smithi16534841-39 from 2909627 to 3511884 tid 3 2024-09-10T16:30:30.456 INFO:tasks.rados.rados.0.smithi165.stdout:3738: snap_remove snap 413 2024-09-10T16:30:30.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d64b32640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:30.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d64b32640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:30.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d64b32640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6d600a89a0 con 0x7f6d600a46a0 2024-09-10T16:30:30.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d4effd640 1 -- 172.21.15.164:0/106329833 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6d54052260 con 0x7f6d600a46a0 2024-09-10T16:30:30.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:30.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.483+0000 7f6d4effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:30.485 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 msgr2=0x7f6d400406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 0x7f6d400406d0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f6d50029ff0 tx=0x7f6d5002e000 comp rx=0 tx=0).stop 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 10 monclient: shutdown 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 msgr2=0x7f6d6013ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 0x7f6d6013ffe0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6d5400c3e0 tx=0x7f6d5400c8a0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 shutdown_connections 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6d4003e210 0x7f6d400406d0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6d600a4a80 0x7f6d60140520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6d600a46a0 0x7f6d6013ffe0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 --2- 172.21.15.164:0/106329833 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6d6006cb70 0x7f6d601cd2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.485+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 >> 172.21.15.164:0/106329833 conn(0x7f6d600aa900 msgr2=0x7f6d60137960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.486+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 shutdown_connections 2024-09-10T16:30:30.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.486+0000 7f6d64b32640 1 -- 172.21.15.164:0/106329833 wait complete. 2024-09-10T16:30:30.497 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:30:30.497 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:30:30.498 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:30:30.498 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:30:30.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.596+0000 7ff009f64640 10 monclient: get_monmap_and_config 2024-09-10T16:30:30.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.596+0000 7ff009f64640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.596+0000 7ff009f64640 10 monclient: monmap: 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:30.597157+0000 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:30.597157+0000 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:30.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 Processor -- start 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 -- start start 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient: init 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff00406cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 0x7ff00413d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-b con 0x7ff004134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff00413d560 0x7ff00413f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-c con 0x7ff00413d560 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff00409ce50 con 0x7ff00406cb70 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff00406db60 con 0x7ff00413d560 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff00406cf50 con 0x7ff004134f70 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff009f64640 10 monclient(hunting): authenticate will time out at 2209.084824s 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7feffbfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 0x7ff00413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff0037fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff00406cb70 0x7ff0040a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38584/0 (socket says 172.21.15.164:38584) 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7ff0037fe640 1 -- 172.21.15.164:0/1118361103 learned_addr learned my addr 172.21.15.164:0/1118361103 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:30.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7feffbfff640 10 monclient(hunting): get_auth_request con 0x7ff004134f70 auth_method 0 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7feffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7feffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.597+0000 7feffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7ff003fff640 1 --2- 172.21.15.164:0/1118361103 >> v2:172.21.15.165:3300/0 conn(0x7ff00413d560 0x7ff00413f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7ff003fff640 10 monclient(hunting): get_auth_request con 0x7ff00413d560 auth_method 0 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7ff003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7ff003fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7ff003fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient(hunting): handle_auth_done global_id 274121 payload 274 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 1 -- 172.21.15.164:0/1118361103 >> v2:172.21.15.165:3300/0 conn(0x7ff00413d560 msgr2=0x7ff00413f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 1 --2- 172.21.15.164:0/1118361103 >> v2:172.21.15.165:3300/0 conn(0x7ff00413d560 0x7ff00413f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 1 -- 172.21.15.164:0/1118361103 >> v1:172.21.15.164:6789/0 conn(0x7ff00406cb70 legacy=0x7ff0040a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 1 monclient: found mon.noname-b 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:30:30.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.598+0000 7feffbfff640 1 -- 172.21.15.164:0/1118361103 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff004135380 con 0x7ff004134f70 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7feffbfff640 1 -- 172.21.15.164:0/1118361103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 msgr2=0x7ff00413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7feffbfff640 1 --2- 172.21.15.164:0/1118361103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 0x7ff00413d020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097a0 tx=0x7fefec029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=deeae1e2a17fe9c4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 1 -- 172.21.15.164:0/1118361103 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec037070 con 0x7ff004134f70 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: dump: 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 1 -- 172.21.15.164:0/1118361103 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fefec032440 con 0x7ff004134f70 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 5 monclient: authenticate success, global_id 274121 2024-09-10T16:30:30.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 1 -- 172.21.15.164:0/1118361103 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefec032810 con 0x7ff004134f70 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 4 set_mon_vals no callback set 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff002ffd640 10 monclient: dump: 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 10 monclient: shutdown 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 -- 172.21.15.164:0/1118361103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 msgr2=0x7ff00413d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 --2- 172.21.15.164:0/1118361103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 0x7ff00413d020 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097a0 tx=0x7fefec029fb0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 -- 172.21.15.164:0/1118361103 shutdown_connections 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 --2- 172.21.15.164:0/1118361103 >> v2:172.21.15.165:3300/0 conn(0x7ff00413d560 0x7ff00413f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 --2- 172.21.15.164:0/1118361103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff004134f70 0x7ff00413d020 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 -- 172.21.15.164:0/1118361103 >> 172.21.15.164:0/1118361103 conn(0x7ff0040aa900 msgr2=0x7ff0040aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.601+0000 7ff009f64640 1 -- 172.21.15.164:0/1118361103 shutdown_connections 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 1 -- 172.21.15.164:0/1118361103 wait complete. 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 10 monclient: monmap: 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 1 Processor -- start 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 1 -- start start 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.602+0000 7ff009f64640 10 monclient: init 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff00406cb70 0x7ff00413c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-b con 0x7ff00406cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 0x7ff00413c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-a con 0x7ff004134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 0x7ff0041cd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): picked mon.noname-c con 0x7ff00413d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040a4db0 con 0x7ff004134f70 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff00409d130 con 0x7ff00406cb70 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff00413fe90 con 0x7ff00413d560 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff009f64640 10 monclient(hunting): authenticate will time out at 2209.090410s 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 0x7ff00413c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 0x7ff0041cd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 0x7ff00413c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55190/0 (socket says 172.21.15.164:55190) 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 1 -- 172.21.15.164:0/3736274848 learned_addr learned my addr 172.21.15.164:0/3736274848 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): get_auth_request con 0x7ff00413d560 auth_method 0 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): get_auth_request con 0x7ff004134f70 auth_method 0 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff00406cb70 0x7ff00413c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): get_auth_request con 0x7ff00406cb70 auth_method 0 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient(hunting): handle_auth_done global_id 274124 payload 824 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff00406cb70 msgr2=0x7ff00413c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff00406cb70 0x7ff00413c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 msgr2=0x7ff00413c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 0x7ff00413c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 monclient: found mon.noname-c 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:30:30.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff00406e720 con 0x7ff00413d560 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.603+0000 7feffbfff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004134f70 0x7ff00413c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 msgr2=0x7ff0041cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff003fff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 0x7ff0041cd1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feffc007d90 tx=0x7feffc00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc003990 con 0x7ff00413d560 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: dump: 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.619 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feffc003b50 con 0x7ff00413d560 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feffc004020 con 0x7ff00413d560 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 5 monclient: authenticate success, global_id 274124 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:30.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff000ff9640 10 monclient: dump: 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 10 monclient: _renew_subs 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff0027fc640 4 set_mon_vals no callback set 2024-09-10T16:30:30.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff00413cc00 con 0x7ff00413d560 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 10 monclient: _renew_subs 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.606+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff004138160 con 0x7ff00413d560 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff009f64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff009f64640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 0x7ff0041cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0040a46a0 con 0x7ff004135d30 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7feffc01b020 con 0x7ff00413d560 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 0x7ff0041cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient: get_auth_request con 0x7ff004135d30 auth_method 0 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff000ff9640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefe403e290 0x7fefe4040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 10 monclient(hunting): handle_auth_done global_id 254331 payload 824 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff0037fe640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0041384e0 con 0x7ff004135d30 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7feffc052f20 con 0x7ff00413d560 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7feffbfff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefe403e290 0x7fefe4040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.607+0000 7feffbfff640 10 monclient: get_auth_request con 0x7fefe403e290 auth_method 0 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff0037fe640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 msgr2=0x7ff0041cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff0037fe640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 0x7ff0041cd6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feff400e7a0 tx=0x7feff400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4017070 con 0x7ff004135d30 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff000ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff000ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.609+0000 7ff000ff9640 10 monclient: dump: 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.626 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feff400ee70 con 0x7ff004135d30 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 msgr2=0x7ff0041cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 0x7ff0041cd6f0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feff400e7a0 tx=0x7feff400ec60 comp rx=0 tx=0).stop 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 msgr2=0x7ff0041cd6f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7ff000ff9640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff004135d30 0x7ff0041cd6f0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feff400e7a0 tx=0x7feff400ec60 comp rx=0 tx=0).stop 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.610+0000 7feffbfff640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefe403e290 0x7fefe4040750 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff004137b90 tx=0x7fefec02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff009f64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:30.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff009f64640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 0x7ff00412c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff009f64640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff0041ce040 con 0x7ff0041cdc30 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 0x7ff00412c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient: get_auth_request con 0x7ff0041cdc30 auth_method 0 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 10 monclient(hunting): handle_auth_done global_id 254334 payload 824 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.616+0000 7ff0037fe640 1 -- 172.21.15.164:0/3736274848 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0040770f0 con 0x7ff0041cdc30 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff0037fe640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 msgr2=0x7ff00412c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff0037fe640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 0x7ff00412c100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feff4019000 tx=0x7feff40058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feff4017050 con 0x7ff0041cdc30 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff000ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff000ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.618+0000 7ff000ff9640 10 monclient: dump: 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.629 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7feff4006460 con 0x7ff0041cdc30 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 msgr2=0x7ff00412c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 0x7ff00412c100 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feff4019000 tx=0x7feff40058d0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 msgr2=0x7ff00412c100 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff000ff9640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff0041cdc30 0x7ff00412c100 secure :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7feff4019000 tx=0x7feff40058d0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:30.630 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 119268, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:30.631 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:30.632 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.633 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:30.634 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.635 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.636 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.637 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.638 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.639 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.640 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.641 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.642 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:30.643 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:30.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefe403e290 msgr2=0x7fefe4040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fefe403e290 0x7fefe4040750 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7ff004137b90 tx=0x7fefec02e000 comp rx=0 tx=0).stop 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 10 monclient: shutdown 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 msgr2=0x7ff0041cd1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 --2- 172.21.15.164:0/3736274848 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff00413d560 0x7ff0041cd1b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7feffc007d90 tx=0x7feffc00e4b0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 reap_dead start 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff003fff640 1 -- 172.21.15.164:0/3736274848 reap_dead start 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 shutdown_connections 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 >> 172.21.15.164:0/3736274848 conn(0x7ff0040aa900 msgr2=0x7ff0040a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 shutdown_connections 2024-09-10T16:30:30.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.619+0000 7ff009f64640 1 -- 172.21.15.164:0/3736274848 wait complete. 2024-09-10T16:30:30.645 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:30.646 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:30:30.646 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:30:30.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.742+0000 7f4f9b41f640 10 monclient: get_monmap_and_config 2024-09-10T16:30:30.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.742+0000 7f4f9b41f640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.742+0000 7f4f9b41f640 10 monclient: monmap: 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:30.743153+0000 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:30.743153+0000 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:30.744 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.742+0000 7f4f9b41f640 1 Processor -- start 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 1 -- start start 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient: init 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient: _add_conns ranks=[1,0,4,3,2] 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f940a3b60 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f940a0530 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f4f940a0980 0x7f4f94140650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a0980 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:30.747 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f94138390 con 0x7f4f940a0530 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f940a0e10 con 0x7f4f940a3b60 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f4f9409cfe0 con 0x7f4f940a0980 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f9b41f640 10 monclient(hunting): authenticate will time out at 2209.230642s 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f989ba640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f4f940a0530 0x7f4f9413cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38586/0 (socket says 172.21.15.164:38586) 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f989ba640 1 -- 172.21.15.164:0/3488525206 learned_addr learned my addr 172.21.15.164:0/3488525206 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f999bc640 1 --2- 172.21.15.164:0/3488525206 >> v2:172.21.15.165:3300/0 conn(0x7f4f940a0980 0x7f4f94140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f999bc640 10 monclient(hunting): get_auth_request con 0x7f4f940a0980 auth_method 0 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f999bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f999bc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.743+0000 7f4f999bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient(hunting): handle_auth_done global_id 264214 payload 274 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 1 -- 172.21.15.164:0/3488525206 >> v1:172.21.15.165:6789/0 conn(0x7f4f940a3b60 legacy=0x7f4f94137c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 1 -- 172.21.15.164:0/3488525206 >> v1:172.21.15.164:6789/0 conn(0x7f4f940a0530 legacy=0x7f4f9413cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 1 monclient: found mon.noname-c 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:30:30.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.744+0000 7f4f999bc640 1 -- 172.21.15.164:0/3488525206 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f94074520 con 0x7f4f940a0980 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f999bc640 1 -- 172.21.15.164:0/3488525206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a0980 msgr2=0x7f4f94140650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f999bc640 1 --2- 172.21.15.164:0/3488525206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a0980 0x7f4f94140650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009d90 tx=0x7f4f8c02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6185a89a15eb36cb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 1 -- 172.21.15.164:0/3488525206 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c007a30 con 0x7f4f940a0980 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: dump: 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.756 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 5 monclient: authenticate success, global_id 264214 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 1 -- 172.21.15.164:0/3488525206 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f8c007bd0 con 0x7f4f940a0980 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 1 -- 172.21.15.164:0/3488525206 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f8c031400 con 0x7f4f940a0980 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:30.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 4 set_mon_vals no callback set 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f8bfff640 10 monclient: dump: 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.758 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 10 monclient: shutdown 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3488525206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a0980 msgr2=0x7f4f94140650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.747+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3488525206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a0980 0x7f4f94140650 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4f8c009d90 tx=0x7f4f8c02a490 comp rx=0 tx=0).stop 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3488525206 shutdown_connections 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3488525206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a0980 0x7f4f94140650 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3488525206 >> 172.21.15.164:0/3488525206 conn(0x7f4f940a6f80 msgr2=0x7f4f940a3450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3488525206 shutdown_connections 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3488525206 wait complete. 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 10 monclient: monmap: 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:30.759 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.748+0000 7f4f9b41f640 1 Processor -- start 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 -- start start 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient: init 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:30:30.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 0x7f4f941cd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-c con 0x7f4f940a0530 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 0x7f4f9413fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-a con 0x7f4f940a0980 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a3b60 0x7f4f941402a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): picked mon.noname-b con 0x7f4f940a3b60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f4f94140b90 con 0x7f4f940a0980 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4f94140d10 con 0x7f4f940a3b60 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f94140e90 con 0x7f4f940a0530 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f9b41f640 10 monclient(hunting): authenticate will time out at 2209.236462s 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 0x7f4f941cd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 0x7f4f9413fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 0x7f4f941cd2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:60094/0 (socket says 172.21.15.164:60094) 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 1 -- 172.21.15.164:0/3718352325 learned_addr learned my addr 172.21.15.164:0/3718352325 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): get_auth_request con 0x7f4f940a0980 auth_method 0 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): get_auth_request con 0x7f4f940a0530 auth_method 0 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a3b60 0x7f4f941402a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): get_auth_request con 0x7f4f940a3b60 auth_method 0 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f999bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient(hunting): handle_auth_done global_id 254337 payload 824 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 msgr2=0x7f4f941cd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 0x7f4f941cd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a3b60 msgr2=0x7f4f941402a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f4f940a3b60 0x7f4f941402a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 monclient: found mon.noname-a 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f989ba640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f940a13c0 con 0x7f4f940a0980 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.749+0000 7f4f991bb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f940a0530 0x7f4f941cd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f989ba640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 msgr2=0x7f4f9413fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f989ba640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 0x7f4f9413fd60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400c3e0 tx=0x7f4f8400c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f84017070 con 0x7f4f940a0980 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: dump: 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.765 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f840070d0 con 0x7f4f940a0980 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 5 monclient: authenticate success, global_id 254337 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 10 monclient: _renew_subs 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f9413e550 con 0x7f4f940a0980 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f840074c0 con 0x7f4f940a0980 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f89ffb640 10 monclient: dump: 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f8b7fe640 4 set_mon_vals no callback set 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 10 monclient: _renew_subs 2024-09-10T16:30:30.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.752+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f941407e0 con 0x7f4f940a0980 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f9b41f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f4f8401a070 con 0x7f4f940a0980 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 0x7f4f9413aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f94073b40 con 0x7f4f9413df90 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 0x7f4f9413aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient: get_auth_request con 0x7f4f9413df90 auth_method 0 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f89ffb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f7803e210 0x7f4f780406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7f4f840521d0 con 0x7f4f940a0980 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 10 monclient(hunting): handle_auth_done global_id 274130 payload 824 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f991bb640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4f94137c80 con 0x7f4f9413df90 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f999bc640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f7803e210 0x7f4f780406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.753+0000 7f4f999bc640 10 monclient: get_auth_request con 0x7f4f7803e210 auth_method 0 2024-09-10T16:30:30.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f991bb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 msgr2=0x7f4f9413aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f991bb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 0x7f4f9413aa10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000fae0 tx=0x7f4f900045a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90018020 con 0x7f4f9413df90 2024-09-10T16:30:30.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 10 monclient: dump: 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4f9000de20 con 0x7f4f9413df90 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 msgr2=0x7f4f9413aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 0x7f4f9413aa10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000fae0 tx=0x7f4f900045a0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 msgr2=0x7f4f9413aa10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f89ffb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413df90 0x7f4f9413aa10 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f4f9000fae0 tx=0x7f4f900045a0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.756+0000 7f4f999bc640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f7803e210 0x7f4f780406d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4f94074450 tx=0x7f4f8c02f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f9b41f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 0x7f4f9413b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f9b41f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4f9413b8a0 con 0x7f4f9413af50 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 0x7f4f9413b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient: get_auth_request con 0x7f4f9413af50 auth_method 0 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 10 monclient(hunting): handle_auth_done global_id 274133 payload 824 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.762+0000 7f4f991bb640 1 -- 172.21.15.164:0/3718352325 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4f9413c4d0 con 0x7f4f9413af50 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f991bb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 msgr2=0x7f4f9413b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f991bb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 0x7f4f9413b360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f90019000 tx=0x7f4f900036f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f90018030 con 0x7f4f9413af50 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:30.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 10 monclient: dump: 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f4f90007440 con 0x7f4f9413af50 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:30.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 msgr2=0x7f4f9413b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 0x7f4f9413b360 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f90019000 tx=0x7f4f900036f0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 msgr2=0x7f4f9413b360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f89ffb640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f4f9413af50 0x7f4f9413b360 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f4f90019000 tx=0x7f4f900036f0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 8007, 2024-09-10T16:30:30.776 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.777 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.778 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:30.779 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:30.780 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.781 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:30.782 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.783 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.784 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.785 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.786 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.787 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:30.788 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:30.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f7803e210 msgr2=0x7f4f780406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f4f7803e210 0x7f4f780406d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f4f94074450 tx=0x7f4f8c02f000 comp rx=0 tx=0).stop 2024-09-10T16:30:30.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 10 monclient: shutdown 2024-09-10T16:30:30.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 msgr2=0x7f4f9413fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 --2- 172.21.15.164:0/3718352325 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f4f940a0980 0x7f4f9413fd60 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f4f8400c3e0 tx=0x7f4f8400c8a0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f999bc640 1 -- 172.21.15.164:0/3718352325 reap_dead start 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f999bc640 1 -- 172.21.15.164:0/3718352325 reap_dead start 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 shutdown_connections 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 >> 172.21.15.164:0/3718352325 conn(0x7f4f940a6f80 msgr2=0x7f4f94135770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 shutdown_connections 2024-09-10T16:30:30.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.765+0000 7f4f9b41f640 1 -- 172.21.15.164:0/3718352325 wait complete. 2024-09-10T16:30:30.790 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:30.790 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:30:30.791 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.888+0000 7ff77baef640 10 monclient: get_monmap_and_config 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.888+0000 7ff77baef640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.888+0000 7ff77baef640 10 monclient: monmap: 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:30.889539+0000 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:30.889539+0000 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:30.890 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:30.891 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:30.891 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:30.891 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 Processor -- start 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 -- start start 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient: init 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff77409f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff7740a2550 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff7740a2980 0x7ff77413ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-a con 0x7ff7740a2980 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77409fd10 con 0x7ff77409f930 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77409cd20 con 0x7ff7740a2550 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff77409a0d0 con 0x7ff7740a2980 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77baef640 10 monclient(hunting): authenticate will time out at 2209.376912s 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.889+0000 7ff77a08c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff7740a2980 0x7ff77413ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77988b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff77409f930 0x7ff7740a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38600/0 (socket says 172.21.15.164:38600) 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77988b640 1 -- 172.21.15.164:0/3560566784 learned_addr learned my addr 172.21.15.164:0/3560566784 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): get_auth_request con 0x7ff7740a2980 auth_method 0 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient(hunting): handle_auth_done global_id 254340 payload 274 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 1 -- 172.21.15.164:0/3560566784 >> v1:172.21.15.165:6789/0 conn(0x7ff7740a2550 legacy=0x7ff77413b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 1 -- 172.21.15.164:0/3560566784 >> v1:172.21.15.164:6789/0 conn(0x7ff77409f930 legacy=0x7ff7740a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 1 monclient: found mon.noname-a 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.890+0000 7ff77a08c640 1 -- 172.21.15.164:0/3560566784 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77409e840 con 0x7ff7740a2980 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77a08c640 1 -- 172.21.15.164:0/3560566784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 msgr2=0x7ff77413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77a08c640 1 --2- 172.21.15.164:0/3560566784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff77413ed00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff77409cef0 tx=0x7ff76c02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67273e52663ded67 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 1 -- 172.21.15.164:0/3560566784 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c007a30 con 0x7ff7740a2980 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: dump: 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 1 -- 172.21.15.164:0/3560566784 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76c007bd0 con 0x7ff7740a2980 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 5 monclient: authenticate success, global_id 254340 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:30.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 1 -- 172.21.15.164:0/3560566784 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c031400 con 0x7ff7740a2980 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 4 set_mon_vals no callback set 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff778889640 10 monclient: dump: 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.904 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 10 monclient: shutdown 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 -- 172.21.15.164:0/3560566784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 msgr2=0x7ff77413ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 --2- 172.21.15.164:0/3560566784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff77413ed00 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff77409cef0 tx=0x7ff76c02a480 comp rx=0 tx=0).stop 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 -- 172.21.15.164:0/3560566784 shutdown_connections 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 --2- 172.21.15.164:0/3560566784 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff77413ed00 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 -- 172.21.15.164:0/3560566784 >> 172.21.15.164:0/3560566784 conn(0x7ff7740a3110 msgr2=0x7ff7740a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 -- 172.21.15.164:0/3560566784 shutdown_connections 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.893+0000 7ff77baef640 1 -- 172.21.15.164:0/3560566784 wait complete. 2024-09-10T16:30:30.905 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.894+0000 7ff77baef640 10 monclient: build_initial_monmap 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.894+0000 7ff77baef640 10 monclient: monmap: 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.894+0000 7ff77baef640 1 Processor -- start 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.894+0000 7ff77baef640 1 -- start start 2024-09-10T16:30:30.906 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.894+0000 7ff77baef640 10 monclient: init 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff77409f930 0x7ff7741bad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-b con 0x7ff77409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7740a2550 0x7ff7741b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-c con 0x7ff7740a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): picked mon.noname-a con 0x7ff7740a2980 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff77409d930 con 0x7ff7740a2980 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff77409ebe0 con 0x7ff77409f930 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff77409c8b0 con 0x7ff7740a2550 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77baef640 10 monclient(hunting): authenticate will time out at 2209.382222s 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55234/0 (socket says 172.21.15.164:55234) 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 learned_addr learned my addr 172.21.15.164:0/2292819693 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): get_auth_request con 0x7ff7740a2980 auth_method 0 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7740a2550 0x7ff7741b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): get_auth_request con 0x7ff7740a2550 auth_method 0 2024-09-10T16:30:30.908 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff77409f930 0x7ff7741bad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77908a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): get_auth_request con 0x7ff77409f930 auth_method 0 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.909 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient(hunting): handle_auth_done global_id 254343 payload 824 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7740a2550 msgr2=0x7ff7741b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff7740a2550 0x7ff7741b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff77409f930 msgr2=0x7ff7741bad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff77409f930 0x7ff7741bad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient: _finish_hunting 0 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 monclient: found mon.noname-a 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.895+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7741b4200 con 0x7ff7740a2980 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.897+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 msgr2=0x7ff7741b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.910 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.897+0000 7ff77a08c640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff76c02aa70 tx=0x7ff76c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c040070 con 0x7ff7740a2980 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: dump: 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.911 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76c045660 con 0x7ff7740a2980 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 5 monclient: authenticate success, global_id 254343 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 monclient: _renew_subs 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7741b23e0 con 0x7ff7740a2980 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c0314c0 con 0x7ff7740a2980 2024-09-10T16:30:30.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 10 monclient: dump: 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76bfff640 4 set_mon_vals no callback set 2024-09-10T16:30:30.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 monclient: _renew_subs 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7741b3f20 con 0x7ff7740a2980 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 0x7ff7740704f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff7741b1620 con 0x7ff7741b1240 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.898+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff76c039040 con 0x7ff7740a2980 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 0x7ff7740704f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient: get_auth_request con 0x7ff7741b1240 auth_method 0 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.914 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff76a7fc640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff75803e260 0x7ff758040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7ff76c0747d0 con 0x7ff7740a2980 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 10 monclient(hunting): handle_auth_done global_id 264223 payload 824 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77988b640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff77406e9d0 con 0x7ff7741b1240 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77908a640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff75803e260 0x7ff758040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.915 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.899+0000 7ff77908a640 10 monclient: get_auth_request con 0x7ff75803e260 auth_method 0 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff77988b640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 msgr2=0x7ff7740704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff77988b640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 0x7ff7740704f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff77000ea70 tx=0x7ff77000ef30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77001a070 con 0x7ff7741b1240 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff76a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff76a7fc640 10 monclient: dump: 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.918 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.901+0000 7ff77908a640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff75803e260 0x7ff758040720 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff764004f60 tx=0x7ff764007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff77000cc50 con 0x7ff7741b1240 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 msgr2=0x7ff7740704f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 0x7ff7740704f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff77000ea70 tx=0x7ff77000ef30 comp rx=0 tx=0).stop 2024-09-10T16:30:30.919 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 msgr2=0x7ff7740704f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.902+0000 7ff76a7fc640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff7741b1240 0x7ff7740704f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77baef640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77baef640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 0x7ff774070e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77baef640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff77406cc70 con 0x7ff774070a30 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 0x7ff774070e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient: get_auth_request con 0x7ff774070a30 auth_method 0 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:30.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.908+0000 7ff77988b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.909+0000 7ff77988b640 10 monclient(hunting): handle_auth_done global_id 264226 payload 824 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.909+0000 7ff77988b640 1 -- 172.21.15.164:0/2292819693 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff7740a0760 con 0x7ff774070a30 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff77988b640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 msgr2=0x7ff774070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff77988b640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 0x7ff774070e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff770017000 tx=0x7ff770004940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff77001a050 con 0x7ff774070a30 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 10 monclient: dump: 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:30.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7ff77001b040 con 0x7ff774070a30 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 msgr2=0x7ff774070e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 0x7ff774070e10 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff770017000 tx=0x7ff770004940 comp rx=0 tx=0).stop 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 msgr2=0x7ff774070e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:30.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.911+0000 7ff76a7fc640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff774070a30 0x7ff774070e10 secure :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7ff770017000 tx=0x7ff770004940 comp rx=0 tx=0).stop 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 238, 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 53053, 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:30.923 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.924 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.925 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:30.926 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:30.927 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.928 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:30.929 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.930 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:30.931 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.932 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.933 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:30.934 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:30.935 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:30.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff75803e260 msgr2=0x7ff758040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff75803e260 0x7ff758040720 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7ff764004f60 tx=0x7ff764007e30 comp rx=0 tx=0).stop 2024-09-10T16:30:30.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 10 monclient: shutdown 2024-09-10T16:30:30.936 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 msgr2=0x7ff7741b0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff76c02aa70 tx=0x7ff76c004a40 comp rx=0 tx=0).stop 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77a08c640 1 -- 172.21.15.164:0/2292819693 reap_dead start 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 shutdown_connections 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 --2- 172.21.15.164:0/2292819693 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff7740a2980 0x7ff7741b0d00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 >> 172.21.15.164:0/2292819693 conn(0x7ff7740a3110 msgr2=0x7ff7740a4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 shutdown_connections 2024-09-10T16:30:30.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:30.912+0000 7ff77baef640 1 -- 172.21.15.164:0/2292819693 wait complete. 2024-09-10T16:30:30.937 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:30:30.937 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-10T16:30:30.938 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:30:30.938 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-10T16:30:30.938 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-10T16:30:30.938 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:30:30.938 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:30:30.952 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:30:30.952 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:30:30.952 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.048+0000 7fcdc7b84640 10 monclient: get_monmap_and_config 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.048+0000 7fcdc7b84640 10 monclient: build_initial_monmap 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.048+0000 7fcdc7b84640 10 monclient: monmap: 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:31.049122+0000 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:31.049122+0000 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:31.050 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.048+0000 7fcdc7b84640 1 Processor -- start 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 1 -- start start 2024-09-10T16:30:31.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient: init 2024-09-10T16:30:31.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:31.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-10T16:30:31.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcdc006cb70 0x7fcdc0134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc006cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcdc00a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fcdc00a4ab0 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc009ce50 con 0x7fcdc00a46a0 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdc0138ac0 con 0x7fcdc00a4ab0 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fcdc006db60 con 0x7fcdc006cb70 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc7b84640 10 monclient(hunting): authenticate will time out at 2209.536580s 2024-09-10T16:30:31.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc6121640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fcdc00a4ab0 legacy=0x7fcdc01383b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fcdc006cb70 0x7fcdc0134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc511f640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fcdc00a46a0 0x7fcdc0137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38610/0 (socket says 172.21.15.164:38610) 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc511f640 1 -- 172.21.15.164:0/3868455219 learned_addr learned my addr 172.21.15.164:0/3868455219 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): get_auth_request con 0x7fcdc006cb70 auth_method 0 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient(hunting): handle_auth_done global_id 254346 payload 274 2024-09-10T16:30:31.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 1 -- 172.21.15.164:0/3868455219 >> v1:172.21.15.165:6789/0 conn(0x7fcdc00a4ab0 legacy=0x7fcdc01383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 1 -- 172.21.15.164:0/3868455219 >> v1:172.21.15.164:6789/0 conn(0x7fcdc00a46a0 legacy=0x7fcdc0137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient: _finish_hunting 0 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 1 monclient: found mon.noname-a 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.049+0000 7fcdc5920640 1 -- 172.21.15.164:0/3868455219 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc009cb70 con 0x7fcdc006cb70 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.052+0000 7fcdc5920640 1 -- 172.21.15.164:0/3868455219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 msgr2=0x7fcdc0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.052+0000 7fcdc5920640 1 --2- 172.21.15.164:0/3868455219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc0134f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0097d0 tx=0x7fcdbc02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85e5834fa68bf85e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.052+0000 7fcdc491e640 1 -- 172.21.15.164:0/3868455219 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc037070 con 0x7fcdc006cb70 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.052+0000 7fcdc491e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:31.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: dump: 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: _finish_auth 0 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: _check_auth_tickets 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 1 -- 172.21.15.164:0/3868455219 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc032440 con 0x7fcdc006cb70 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 5 monclient: authenticate success, global_id 254346 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 4 set_mon_vals no callback set 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 1 -- 172.21.15.164:0/3868455219 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc032810 con 0x7fcdc006cb70 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:31.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc491e640 10 monclient: dump: 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:31.061 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 10 monclient: shutdown 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 -- 172.21.15.164:0/3868455219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 msgr2=0x7fcdc0134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/3868455219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc0134f70 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0097d0 tx=0x7fcdbc02a000 comp rx=0 tx=0).stop 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 -- 172.21.15.164:0/3868455219 shutdown_connections 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/3868455219 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc0134f70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 -- 172.21.15.164:0/3868455219 >> 172.21.15.164:0/3868455219 conn(0x7fcdc00aa900 msgr2=0x7fcdc00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 -- 172.21.15.164:0/3868455219 shutdown_connections 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.053+0000 7fcdc7b84640 1 -- 172.21.15.164:0/3868455219 wait complete. 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient: build_initial_monmap 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient: monmap: 2024-09-10T16:30:31.062 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 1 Processor -- start 2024-09-10T16:30:31.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 1 -- start start 2024-09-10T16:30:31.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient: init 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-a con 0x7fcdc006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.054+0000 7fcdc7b84640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 0x7fcdc009ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-b con 0x7fcdc00a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a4ab0 0x7fcdc009f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): picked mon.noname-c con 0x7fcdc00a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:31.069 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc0141050 con 0x7fcdc006cb70 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fcdc006cf50 con 0x7fcdc00a46a0 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fcdc01411d0 con 0x7fcdc00a4ab0 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc7b84640 10 monclient(hunting): authenticate will time out at 2209.542070s 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:55242/0 (socket says 172.21.15.164:55242) 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 -- 172.21.15.164:0/2420496542 learned_addr learned my addr 172.21.15.164:0/2420496542 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:31.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): get_auth_request con 0x7fcdc006cb70 auth_method 0 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a4ab0 0x7fcdc009f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc511f640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 msgr2=0x7fcdc009ed00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc511f640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 0x7fcdc009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): get_auth_request con 0x7fcdc00a4ab0 auth_method 0 2024-09-10T16:30:31.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc6121640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient(hunting): handle_auth_done global_id 254349 payload 824 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a4ab0 msgr2=0x7fcdc009f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a4ab0 0x7fcdc009f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 msgr2=0x7fcdc009ed00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 0x7fcdc009ed00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient: _finish_hunting 0 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 monclient: found mon.noname-a 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.055+0000 7fcdc5920640 1 -- 172.21.15.164:0/2420496542 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdc00a2210 con 0x7fcdc006cb70 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc5920640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 msgr2=0x7fcdc009e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc5920640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0098f0 tx=0x7fcdbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc041070 con 0x7fcdc006cb70 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: dump: 2024-09-10T16:30:31.073 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: _finish_auth 0 2024-09-10T16:30:31.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcdbc03c670 con 0x7fcdc006cb70 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 5 monclient: authenticate success, global_id 254349 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 10 monclient: _renew_subs 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdc00a24f0 con 0x7fcdc006cb70 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdbc037040 con 0x7fcdc006cb70 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:31.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb67fc640 10 monclient: dump: 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:31.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 10 monclient: _renew_subs 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdc00a1f30 con 0x7fcdc006cb70 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.057+0000 7fcdb7fff640 4 set_mon_vals no callback set 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fcdbc044070 con 0x7fcdc006cb70 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdc7b84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdc7b84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdc7b84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdc0135210 con 0x7fcdc006cb70 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.058+0000 7fcdb67fc640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 0x7fcda4040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.059+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7fcdbc072fd0 con 0x7fcdc006cb70 2024-09-10T16:30:31.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.059+0000 7fcdc511f640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 0x7fcda4040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.059+0000 7fcdc511f640 10 monclient: get_auth_request con 0x7fcda403e1c0 auth_method 0 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.061+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.061+0000 7fcdc511f640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 0x7fcda4040680 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0007fd0 tx=0x7fcdb0007b20 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.063+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fcdc0135210 con 0x7fcdc006cb70 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.064+0000 7fcdb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:31.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.064+0000 7fcdb67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.304+0000 7fcdc7b84640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.304+0000 7fcdc7b84640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.304+0000 7fcdc7b84640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.304+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcdc0134f70 con 0x7fcdc006cb70 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.305+0000 7fcdb67fc640 1 -- 172.21.15.164:0/2420496542 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fcdbc04b070 con 0x7fcdc006cb70 2024-09-10T16:30:31.306 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.305+0000 7fcdb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:31.307 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.305+0000 7fcdb67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:31.307 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 msgr2=0x7fcda4040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 0x7fcda4040680 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0007fd0 tx=0x7fcdb0007b20 comp rx=0 tx=0).stop 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 10 monclient: shutdown 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 msgr2=0x7fcdc009e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcdbc0098f0 tx=0x7fcdbc0046c0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 shutdown_connections 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fcda403e1c0 0x7fcda4040680 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fcdc00a4ab0 0x7fcdc009f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.309 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fcdc00a46a0 0x7fcdc009ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 --2- 172.21.15.164:0/2420496542 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fcdc006cb70 0x7fcdc009e7c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 >> 172.21.15.164:0/2420496542 conn(0x7fcdc00aa900 msgr2=0x7fcdc00a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 shutdown_connections 2024-09-10T16:30:31.310 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:31.307+0000 7fcdc7b84640 1 -- 172.21.15.164:0/2420496542 wait complete. 2024-09-10T16:30:34.320 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:34.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.427+0000 7f8549b32640 10 monclient: get_monmap_and_config 2024-09-10T16:30:34.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.427+0000 7f8549b32640 10 monclient: build_initial_monmap 2024-09-10T16:30:34.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.427+0000 7f8549b32640 10 monclient: monmap: 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:34.429162+0000 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:34.429162+0000 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:34.430 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:34.431 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:34.431 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:34.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 Processor -- start 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 -- start start 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient: init 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient: _add_conns ranks=[0,2,3,4,1] 2024-09-10T16:30:34.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f854409c310 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 0x7f854409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-b con 0x7f854409d190 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f854413a5c0 0x7f854413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-a con 0x7f854413a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854409fbc0 con 0x7f854409c310 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f85440a2a20 con 0x7f854413a5c0 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f854409c6f0 con 0x7f854409d190 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8549b32640 10 monclient(hunting): authenticate will time out at 2212.916671s 2024-09-10T16:30:34.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 0x7f854409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f854413a5c0 0x7f854413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f85437fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f854409c310 0x7f854409ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49128/0 (socket says 172.21.15.164:49128) 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f85437fe640 1 -- 172.21.15.164:0/1708940345 learned_addr learned my addr 172.21.15.164:0/1708940345 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): get_auth_request con 0x7f854409d190 auth_method 0 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): get_auth_request con 0x7f854413a5c0 auth_method 0 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:34.435 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.428+0000 7f8542ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 10 monclient(hunting): handle_auth_done global_id 254352 payload 274 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 1 -- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 msgr2=0x7f854409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 0x7f854409f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 1 -- 172.21.15.164:0/1708940345 >> v1:172.21.15.164:6789/0 conn(0x7f854409c310 legacy=0x7f854409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 10 monclient: _finish_hunting 0 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 1 monclient: found mon.noname-a 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:34.436 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8543fff640 1 -- 172.21.15.164:0/1708940345 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854409a4d0 con 0x7f854413a5c0 2024-09-10T16:30:34.437 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.429+0000 7f8542ffd640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 0x7f854409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:34.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.431+0000 7f8543fff640 1 -- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854413a5c0 msgr2=0x7f854413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:34.440 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.431+0000 7f8543fff640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854413a5c0 0x7f854413c9b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f853c009d90 tx=0x7f853c02a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eaa5056c4598852 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 1 -- 172.21.15.164:0/1708940345 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c007950 con 0x7f854413a5c0 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: dump: 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.441 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: _finish_auth 0 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: _check_auth_tickets 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 1 -- 172.21.15.164:0/1708940345 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c007af0 con 0x7f854413a5c0 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 5 monclient: authenticate success, global_id 254352 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 1 -- 172.21.15.164:0/1708940345 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c031400 con 0x7f854413a5c0 2024-09-10T16:30:34.442 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 4 set_mon_vals no callback set 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f85427fc640 10 monclient: dump: 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:34.443 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 10 monclient: shutdown 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 -- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854413a5c0 msgr2=0x7f854413c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854413a5c0 0x7f854413c9b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f853c009d90 tx=0x7f853c02a530 comp rx=0 tx=0).stop 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 -- 172.21.15.164:0/1708940345 shutdown_connections 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f854413a5c0 0x7f854413c9b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 --2- 172.21.15.164:0/1708940345 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409d190 0x7f854409f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 -- 172.21.15.164:0/1708940345 >> 172.21.15.164:0/1708940345 conn(0x7f85440a3110 msgr2=0x7f85440a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 -- 172.21.15.164:0/1708940345 shutdown_connections 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.432+0000 7f8549b32640 1 -- 172.21.15.164:0/1708940345 wait complete. 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.433+0000 7f8549b32640 10 monclient: build_initial_monmap 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.433+0000 7f8549b32640 10 monclient: monmap: 2024-09-10T16:30:34.444 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.433+0000 7f8549b32640 1 Processor -- start 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.433+0000 7f8549b32640 1 -- start start 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.433+0000 7f8549b32640 10 monclient: init 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:34.445 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-c con 0x7f854409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 0x7f85440933e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-b con 0x7f854413a5c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 0x7f8544093db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): picked mon.noname-a con 0x7f8544093920 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f85440a2230 con 0x7f8544093920 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85440a20b0 con 0x7f854413a5c0 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8544074e00 con 0x7f854409c310 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8549b32640 10 monclient(hunting): authenticate will time out at 2212.922427s 2024-09-10T16:30:34.446 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 0x7f8544093db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50464/0 (socket says 172.21.15.164:50464) 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 -- 172.21.15.164:0/992533252 learned_addr learned my addr 172.21.15.164:0/992533252 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): get_auth_request con 0x7f854409c310 auth_method 0 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): get_auth_request con 0x7f8544093920 auth_method 0 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:34.447 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8542ffd640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 msgr2=0x7f85440933e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8542ffd640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 0x7f85440933e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient(hunting): handle_auth_done global_id 274145 payload 824 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 msgr2=0x7f85440933e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 0x7f85440933e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 msgr2=0x7f8544093db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:34.448 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 0x7f8544093db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient: _finish_hunting 0 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 monclient: found mon.noname-c 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f85437fe640 1 -- 172.21.15.164:0/992533252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85440960f0 con 0x7f854409c310 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.434+0000 7f8543fff640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 0x7f8544093db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.437+0000 7f85437fe640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 msgr2=0x7f8544073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:34.449 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.437+0000 7f85437fe640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f853400c490 tx=0x7f853400c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534007070 con 0x7f854409c310 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: dump: 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.451 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: _finish_auth 0 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8534007230 con 0x7f854409c310 2024-09-10T16:30:34.452 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 5 monclient: authenticate success, global_id 274145 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 10 monclient: _renew_subs 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854409d620 con 0x7f854409c310 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8534007620 con 0x7f854409c310 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f852bfff640 10 monclient: dump: 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.453 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 10 monclient: _renew_subs 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f85441d37c0 con 0x7f854409c310 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.439+0000 7f8541ffb640 4 set_mon_vals no callback set 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f8549b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f8549b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f8549b32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:34.454 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f853401b400 con 0x7f854409c310 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f854409f720 con 0x7f854409c310 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f852bfff640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 0x7f85300406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.440+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7f8534052900 con 0x7f854409c310 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.441+0000 7f8542ffd640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 0x7f85300406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.441+0000 7f8542ffd640 10 monclient: get_auth_request con 0x7f853003e210 auth_method 0 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.443+0000 7f8542ffd640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 0x7f85300406d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85440a1790 tx=0x7f8538009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.445+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f85340512e0 con 0x7f854409c310 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.445+0000 7f852bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:34.455 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.445+0000 7f852bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:34.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.684+0000 7f8549b32640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:34.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.684+0000 7f8549b32640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:34.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.684+0000 7f8549b32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:34.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.684+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f854409f0a0 con 0x7f854409c310 2024-09-10T16:30:34.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.685+0000 7f852bfff640 1 -- 172.21.15.164:0/992533252 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f8534012400 con 0x7f854409c310 2024-09-10T16:30:34.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.685+0000 7f852bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:34.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.685+0000 7f852bfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:34.688 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:34.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 msgr2=0x7f85300406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:34.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 0x7f85300406d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f85440a1790 tx=0x7f8538009190 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 10 monclient: shutdown 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 msgr2=0x7f8544073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f853400c490 tx=0x7f853400c950 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 shutdown_connections 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f853003e210 0x7f85300406d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8544093920 0x7f8544093db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f854413a5c0 0x7f85440933e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 --2- 172.21.15.164:0/992533252 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f854409c310 0x7f8544073dc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 >> 172.21.15.164:0/992533252 conn(0x7f85440a3110 msgr2=0x7f85441387d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 shutdown_connections 2024-09-10T16:30:34.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:34.687+0000 7f8549b32640 1 -- 172.21.15.164:0/992533252 wait complete. 2024-09-10T16:30:37.701 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:37.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.809+0000 7fa7bed32640 10 monclient: get_monmap_and_config 2024-09-10T16:30:37.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.809+0000 7fa7bed32640 10 monclient: build_initial_monmap 2024-09-10T16:30:37.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.809+0000 7fa7bed32640 10 monclient: monmap: 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:37.811122+0000 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:37.811122+0000 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:37.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 Processor -- start 2024-09-10T16:30:37.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 -- start start 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient: init 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 0x7fa7b8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa7b80a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 0x7fa7b81383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b80a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7b809ce50 con 0x7fa7b80a46a0 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fa7b806db60 con 0x7fa7b806cb70 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:37.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b806cf50 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bed32640 10 monclient(hunting): authenticate will time out at 2216.298703s 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bcace640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 msgr2=0x7fa7b8134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7affff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa7b80a46a0 0x7fa7b8137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49134/0 (socket says 172.21.15.164:49134) 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bcace640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 0x7fa7b8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7affff640 1 -- 172.21.15.164:0/1880901212 learned_addr learned my addr 172.21.15.164:0/1880901212 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 1 --2- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 0x7fa7b81383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): get_auth_request con 0x7fa7b80a4ab0 auth_method 0 2024-09-10T16:30:37.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.810+0000 7fa7bd2cf640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 10 monclient(hunting): handle_auth_done global_id 274148 payload 274 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 1 -- 172.21.15.164:0/1880901212 >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 msgr2=0x7fa7b8134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 1 --2- 172.21.15.164:0/1880901212 >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 0x7fa7b8134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 1 -- 172.21.15.164:0/1880901212 >> v1:172.21.15.164:6789/0 conn(0x7fa7b80a46a0 legacy=0x7fa7b8137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 10 monclient: _finish_hunting 0 2024-09-10T16:30:37.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 1 monclient: found mon.noname-b 2024-09-10T16:30:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:30:37.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.811+0000 7fa7bd2cf640 1 -- 172.21.15.164:0/1880901212 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b809cb70 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.813+0000 7fa7bd2cf640 1 -- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 msgr2=0x7fa7b81383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:37.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.813+0000 7fa7bd2cf640 1 --2- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 0x7fa7b81383b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009dc0 tx=0x7fa7b002a500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=813be1337f8ec37e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 1 -- 172.21.15.164:0/1880901212 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0007a30 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: dump: 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:37.824 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: _finish_auth 0 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 1 -- 172.21.15.164:0/1880901212 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b0007bd0 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 5 monclient: authenticate success, global_id 274148 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 4 set_mon_vals no callback set 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 1 -- 172.21.15.164:0/1880901212 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b0031400 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:37.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7af7fe640 10 monclient: dump: 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 10 monclient: shutdown 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 -- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 msgr2=0x7fa7b81383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 --2- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 0x7fa7b81383b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa7b0009dc0 tx=0x7fa7b002a500 comp rx=0 tx=0).stop 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 -- 172.21.15.164:0/1880901212 shutdown_connections 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 --2- 172.21.15.164:0/1880901212 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b80a4ab0 0x7fa7b81383b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 --2- 172.21.15.164:0/1880901212 >> v2:172.21.15.165:3300/0 conn(0x7fa7b806cb70 0x7fa7b8134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 -- 172.21.15.164:0/1880901212 >> 172.21.15.164:0/1880901212 conn(0x7fa7b80aa900 msgr2=0x7fa7b80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 -- 172.21.15.164:0/1880901212 shutdown_connections 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.814+0000 7fa7bed32640 1 -- 172.21.15.164:0/1880901212 wait complete. 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.815+0000 7fa7bed32640 10 monclient: build_initial_monmap 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.815+0000 7fa7bed32640 10 monclient: monmap: 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:37.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.815+0000 7fa7bed32640 1 Processor -- start 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.815+0000 7fa7bed32640 1 -- start start 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.815+0000 7fa7bed32640 10 monclient: init 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:30:37.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-c con 0x7fa7b806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 0x7fa7b81cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-a con 0x7fa7b80a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 0x7fa7b81cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): picked mon.noname-b con 0x7fa7b80a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b8138a70 con 0x7fa7b80a46a0 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa7b81388f0 con 0x7fa7b80a4ab0 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa7b8140040 con 0x7fa7b806cb70 2024-09-10T16:30:37.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bed32640 10 monclient(hunting): authenticate will time out at 2216.304376s 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bd2cf640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 msgr2=0x7fa7b81cdd50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 0x7fa7b81cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50480/0 (socket says 172.21.15.164:50480) 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bd2cf640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 0x7fa7b81cdd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 -- 172.21.15.164:0/93356562 learned_addr learned my addr 172.21.15.164:0/93356562 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): get_auth_request con 0x7fa7b806cb70 auth_method 0 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): get_auth_request con 0x7fa7b80a46a0 auth_method 0 2024-09-10T16:30:37.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient(hunting): handle_auth_done global_id 274151 payload 824 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 -- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 msgr2=0x7fa7b81cdd50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 0x7fa7b81cdd50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 -- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 msgr2=0x7fa7b81cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 0x7fa7b81cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:37.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient: _finish_hunting 0 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 monclient: found mon.noname-c 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7bcace640 1 -- 172.21.15.164:0/93356562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7b80a33a0 con 0x7fa7b806cb70 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.816+0000 7fa7affff640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 0x7fa7b81cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.818+0000 7fa7bcace640 1 -- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 msgr2=0x7fa7b81cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.818+0000 7fa7bcace640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400b550 tx=0x7fa7b400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4013020 con 0x7fa7b806cb70 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: dump: 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:37.832 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: _finish_auth 0 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 5 monclient: authenticate success, global_id 274151 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa7b4005070 con 0x7fa7b806cb70 2024-09-10T16:30:37.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 10 monclient: _renew_subs 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7b4005480 con 0x7fa7b806cb70 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7b80a3680 con 0x7fa7b806cb70 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7ad7fa640 10 monclient: dump: 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:37.834 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7aeffd640 4 set_mon_vals no callback set 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 10 monclient: _renew_subs 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.819+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7b80a30c0 con 0x7fa7b806cb70 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7bed32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa7b4016020 con 0x7fa7b806cb70 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7bed32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7bed32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:37.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7b80a1c60 con 0x7fa7b806cb70 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7ad7fa640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 0x7fa7a40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(840..840 src has 677..840) ==== 4815+0+0 (secure 0 0 0) 0x7fa7b40526d0 con 0x7fa7b806cb70 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7affff640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 0x7fa7a40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.820+0000 7fa7affff640 10 monclient: get_auth_request con 0x7fa7a403e210 auth_method 0 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.823+0000 7fa7affff640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 0x7fa7a40406d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8007c90 tx=0x7fa7a8007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.824+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fa7b80a1c60 con 0x7fa7b806cb70 2024-09-10T16:30:37.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.824+0000 7fa7ad7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:37.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:37.824+0000 7fa7ad7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:38.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7bed32640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:38.070 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7bed32640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7bed32640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa7b8051000 con 0x7fa7b806cb70 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7ad7fa640 1 -- 172.21.15.164:0/93356562 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7fa7b8051000 con 0x7fa7b806cb70 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7ad7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.068+0000 7fa7ad7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:38.071 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":238,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 msgr2=0x7fa7a40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 0x7fa7a40406d0 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7fa7a8007c90 tx=0x7fa7a8007be0 comp rx=0 tx=0).stop 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 10 monclient: shutdown 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 msgr2=0x7fa7b81cd2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:38.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa7b400b550 tx=0x7fa7b400ba10 comp rx=0 tx=0).stop 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 shutdown_connections 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa7a403e210 0x7fa7a40406d0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa7b80a4ab0 0x7fa7b81cdd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa7b80a46a0 0x7fa7b81cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 --2- 172.21.15.164:0/93356562 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa7b806cb70 0x7fa7b81cd2d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 >> 172.21.15.164:0/93356562 conn(0x7fa7b80aa900 msgr2=0x7fa7b80a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 shutdown_connections 2024-09-10T16:30:38.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:38.070+0000 7fa7bed32640 1 -- 172.21.15.164:0/93356562 wait complete. 2024-09-10T16:30:41.085 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:30:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 10 monclient: get_monmap_and_config 2024-09-10T16:30:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 10 monclient: build_initial_monmap 2024-09-10T16:30:41.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 10 monclient: monmap: 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:41.193125+0000 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:41.193125+0000 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:41.194 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 1 Processor -- start 2024-09-10T16:30:41.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 1 -- start start 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.191+0000 7f46557da640 10 monclient: init 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): picked mon.noname-c con 0x7f465006cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f46500a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): picked mon.noname-b con 0x7f46500a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465009ce50 con 0x7f46500a46a0 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f465006db60 con 0x7f465006cb70 2024-09-10T16:30:41.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f465006cf50 con 0x7f46500a4ab0 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46557da640 10 monclient(hunting): authenticate will time out at 2219.680529s 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464effd640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 msgr2=0x7f4650134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f46467fc640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f46500a46a0 legacy=0x7f4650137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464effd640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:50486/0 (socket says 172.21.15.164:50486) 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 learned_addr learned my addr 172.21.15.164:0/2611055391 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f46500a4ab0 auth_method 0 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:41.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:30:41.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:41.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:41.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.192+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 msgr2=0x7f4650134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f46467fc640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.165:6789/0 conn(0x7f46500a46a0 legacy=0x7f4650137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f46500a4ab0 auth_method 0 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:41.396 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:30:41.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:41.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:41.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.393+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:30:41.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 msgr2=0x7f4650134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:41.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f46467fc640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.165:6789/0 conn(0x7f46500a46a0 legacy=0x7f4650137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:41.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f46500a4ab0 auth_method 0 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.794+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.795+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.795+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:41.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:41.795+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:30:42.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.595+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 msgr2=0x7f4650134f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:42.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.595+0000 7f46467fc640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.165:6789/0 conn(0x7f46500a46a0 legacy=0x7f4650137ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:42.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.595+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:30:42.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:42.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f46500a4ab0 auth_method 0 2024-09-10T16:30:42.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:42.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:42.597+0000 7f464f7fe640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:30:44.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): tick 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 monclient(hunting): continuing hunt 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 msgr2=0x7f46501383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f46500a4ab0 0x7f46501383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 msgr2=0x7f4650134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.165:3300/0 conn(0x7f465006cb70 0x7f4650134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.165:6789/0 conn(0x7f46500a46a0 legacy=0x7f4650137ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient: _add_conns ranks=[3,0] 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f46300036f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:44.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4630006240 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 -- 172.21.15.164:0/2611055391 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f465009cb70 con 0x7f4630006240 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 1 -- 172.21.15.164:0/2611055391 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f465006cf50 con 0x7f46300036f0 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464d7fa640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464effd640 10 monclient(hunting): get_auth_request con 0x7f46300036f0 auth_method 0 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.193+0000 7f464effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.194+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.194+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:44.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.194+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:44.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.194+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:44.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.196+0000 7f46467fc640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.164:6789/0 conn(0x7f4630006240 legacy=0x7f4630008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:44.397 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 10 monclient(hunting): get_auth_request con 0x7f46300036f0 auth_method 0 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:44.398 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.395+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 10 monclient(hunting): get_auth_request con 0x7f46300036f0 auth_method 0 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:44.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:30:44.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:30:44.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:30:44.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:44.796+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.472+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1964933781 0 0) 0x7f465009cb70 con 0x7f4630006240 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): _negotiate 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): my global_id is 254361 2024-09-10T16:30:45.474 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f463c003520 con 0x7f4630006240 2024-09-10T16:30:45.476 INFO:tasks.rados.rados.0.smithi165.stdout:3738: done (3 left) 2024-09-10T16:30:45.476 INFO:tasks.rados.rados.0.smithi165.stdout:3739: read oid 2 snap -1 2024-09-10T16:30:45.476 INFO:tasks.rados.rados.0.smithi165.stdout:3739: expect (ObjNum 1084 snap 331 seq_num 1084) 2024-09-10T16:30:45.476 INFO:tasks.rados.rados.0.smithi165.stdout:3740: read oid 3 snap -1 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3740: expect deleted 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3741: rollback oid 46 current snap is 416 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 46 to 394 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: finishing write tid 4 to smithi16534841-19 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: finishing write tid 5 to smithi16534841-19 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: oid 19 updating version 0 to 2422 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: oid 19 version 2422 is already newer than 2420 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: oid 19 version 2422 is already newer than 2421 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: oid 19 updating version 2422 to 2423 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2423 (ObjNum 1286 snap 416 seq_num 1286) dirty exists 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3732: left oid 19 (ObjNum 1286 snap 416 seq_num 1286) 2024-09-10T16:30:45.477 INFO:tasks.rados.rados.0.smithi165.stdout:3734: expect (ObjNum 1138 snap 351 seq_num 1138) 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1207550234 0 0) 0x7f463c003520 con 0x7f4630006240 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 msgr2=0x7f4630005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 --2- 172.21.15.164:0/2611055391 >> v2:172.21.15.164:3300/0 conn(0x7f46300036f0 0x7f4630005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:30:45.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463000b010 con 0x7f4630006240 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f4638004240 con 0x7f4630006240 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:30:45.480 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: dump: 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:30:45.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 >> v1:172.21.15.164:6789/0 conn(0x7f4630006240 legacy=0x7f4630008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464f7fe640 1 -- 172.21.15.164:0/2611055391 reap_dead start 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 0x7f463c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): picked mon.c con 0x7f463c003a20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 0x7f463c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 10 monclient(hunting): picked mon.b con 0x7f463c004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.473+0000 7f464e7fc640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f463c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): picked mon.a con 0x7f463c0052e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f465006cf50 con 0x7f463c0052e0 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f463c00c200 con 0x7f463c004800 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f463c00c400 con 0x7f463c003a20 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f46467fc640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 msgr2=0x7f463c004c80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f46467fc640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 0x7f463c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:45.483 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 0x7f463c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): get_auth_request con 0x7f463c003a20 auth_method 0 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.474+0000 7f464effd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 10 monclient(hunting): handle_auth_done global_id 254361 payload 293 2024-09-10T16:30:45.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 msgr2=0x7f463c004c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 0x7f463c004c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 msgr2=0x7f463c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f463c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 10 monclient: _finish_hunting 0 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 monclient: found mon.c 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f463000b010 con 0x7f463c003a20 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 10 monclient: _finish_auth 0 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f464effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:45.485 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.475+0000 7f46557da640 5 monclient: authenticate success, global_id 254361 2024-09-10T16:30:45.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.476+0000 7f46557da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:45.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.479+0000 7f464effd640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 msgr2=0x7f463c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.479+0000 7f464effd640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 0x7f463c003e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4630006180 tx=0x7f463c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a3982f202a419b1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640038070 con 0x7f463c003a20 2024-09-10T16:30:45.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: dump: 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.492 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f46557da640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f464002bcf0 con 0x7f463c003a20 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f46557da640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 1 -- 172.21.15.164:0/2611055391 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4640009a70 con 0x7f463c003a20 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f46557da640 4 set_mon_vals no callback set 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.480+0000 7f464e7fc640 10 monclient: dump: 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.493 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 10 monclient: shutdown 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 -- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 msgr2=0x7f463c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 0x7f463c003e20 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4630006180 tx=0x7f463c005830 comp rx=0 tx=0).stop 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 -- 172.21.15.164:0/2611055391 shutdown_connections 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f463c00b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c004800 0x7f463c004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 --2- 172.21.15.164:0/2611055391 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c003a20 0x7f463c003e20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 -- 172.21.15.164:0/2611055391 >> 172.21.15.164:0/2611055391 conn(0x7f46500aa900 msgr2=0x7f46500aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 -- 172.21.15.164:0/2611055391 shutdown_connections 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.481+0000 7f46557da640 1 -- 172.21.15.164:0/2611055391 wait complete. 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.482+0000 7f46557da640 10 monclient: build_initial_monmap 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.482+0000 7f46557da640 10 monclient: monmap: 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.495 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 Processor -- start 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 -- start start 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient: init 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 0x7f46501cce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): picked mon.noname-b con 0x7f463c003a20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): picked mon.noname-c con 0x7f463c004800 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): picked mon.noname-a con 0x7f463c0052e0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f46501350f0 con 0x7f463c0052e0 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f4650134f70 con 0x7f463c003a20 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.483+0000 7f46557da640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f4650135270 con 0x7f463c004800 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46557da640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46557da640 10 monclient(hunting): authenticate will time out at 2223.971017s 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464effd640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 msgr2=0x7f46501cce20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:45.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464effd640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 0x7f46501cce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39456/0 (socket says 172.21.15.164:39456) 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 1 -- 172.21.15.164:0/3166298891 learned_addr learned my addr 172.21.15.164:0/3166298891 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53762/0 (socket says 172.21.15.164:53762) 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): get_auth_request con 0x7f463c004800 auth_method 0 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): get_auth_request con 0x7f463c0052e0 auth_method 0 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient(hunting): handle_auth_done global_id 274154 payload 824 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 -- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 msgr2=0x7f46501cce20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 0x7f46501cce20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 -- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 msgr2=0x7f46500a22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient: _finish_hunting 0 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 monclient: found mon.noname-c 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f46467fc640 1 -- 172.21.15.164:0/3166298891 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4650122870 con 0x7f463c004800 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.484+0000 7f464f7fe640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.486+0000 7f46467fc640 1 -- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 msgr2=0x7f46501cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.486+0000 7f46467fc640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4638003970 tx=0x7f463800cb90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4638018070 con 0x7f463c004800 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: dump: 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.500 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: _finish_auth 0 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: _check_auth_tickets 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f463800a610 con 0x7f463c004800 2024-09-10T16:30:45.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 5 monclient: authenticate success, global_id 274154 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 10 monclient: _renew_subs 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4650131870 con 0x7f463c004800 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f463800a9e0 con 0x7f463c004800 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f4647fff640 10 monclient: dump: 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.502 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f464dffb640 4 set_mon_vals no callback set 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 10 monclient: _renew_subs 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.487+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f46501cd8a0 con 0x7f463c004800 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f463801b070 con 0x7f463c004800 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f46557da640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:45.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f46557da640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f46557da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f465006e770 con 0x7f463c004800 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f4647fff640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 0x7f4630040e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.488+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(841..841 src has 677..841) ==== 4843+0+0 (secure 0 0 0) 0x7f4638052220 con 0x7f463c004800 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.489+0000 7f464effd640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 0x7f4630040e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.489+0000 7f464effd640 10 monclient: get_auth_request con 0x7f463003e9c0 auth_method 0 2024-09-10T16:30:45.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.491+0000 7f464effd640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 0x7f4630040e80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4640032000 tx=0x7f464002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.493+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f465006e770 con 0x7f463c004800 2024-09-10T16:30:45.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.493+0000 7f4647fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:45.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.493+0000 7f4647fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: finishing write tid 1 to smithi16534841-39 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: finishing write tid 2 to smithi16534841-39 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: finishing write tid 3 to smithi16534841-39 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: finishing write tid 4 to smithi16534841-39 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: finishing write tid 5 to smithi16534841-39 2024-09-10T16:30:45.696 INFO:tasks.rados.rados.0.smithi165.stdout:3737: oid 39 updating version 0 to 2046 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3737: oid 39 version 2046 is already newer than 2045 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3737: oid 39 version 2046 is already newer than 2044 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3737: oid 39 version 2046 is already newer than 2043 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2046 (ObjNum 1287 snap 416 seq_num 1287) dirty exists 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3737: left oid 39 (ObjNum 1287 snap 416 seq_num 1287) 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3732: done (5 left) 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3734: done (4 left) 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3737: done (3 left) 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3742: copy_from oid 6 from oid 11 current snap is 416 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3741: finishing rollback tid 1 to smithi16534841-46 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3741: finishing rollback tid 2 to smithi16534841-46 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1843 (ObjNum 1220 snap 393 seq_num 1220) dirty exists 2024-09-10T16:30:45.697 INFO:tasks.rados.rados.0.smithi165.stdout:3739: expect (ObjNum 1084 snap 331 seq_num 1084) 2024-09-10T16:30:45.698 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 64694 2024-09-10T16:30:45.740 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 988223 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.749+0000 7f46557da640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.749+0000 7f46557da640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.749+0000 7f46557da640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.749+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4650137ca0 con 0x7f463c004800 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.750+0000 7f4647fff640 1 -- 172.21.15.164:0/3166298891 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f4638011360 con 0x7f463c004800 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.750+0000 7f4647fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.750+0000 7f4647fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:45.751 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":242,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 msgr2=0x7f4630040e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 0x7f4630040e80 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f4640032000 tx=0x7f464002f000 comp rx=0 tx=0).stop 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 10 monclient: shutdown 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 msgr2=0x7f46501cd360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4638003970 tx=0x7f463800cb90 comp rx=0 tx=0).stop 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 shutdown_connections 2024-09-10T16:30:45.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f463003e9c0 0x7f4630040e80 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f463c0052e0 0x7f46500a22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f463c004800 0x7f46501cd360 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 --2- 172.21.15.164:0/3166298891 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f463c003a20 0x7f46501cce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 >> 172.21.15.164:0/3166298891 conn(0x7f46500aa900 msgr2=0x7f4650133260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 shutdown_connections 2024-09-10T16:30:45.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.752+0000 7f46557da640 1 -- 172.21.15.164:0/3166298891 wait complete. 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3742: finishing copy_from racing read to smithi16534841-6 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3742: finishing copy_from to smithi16534841-6 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3739: done (3 left) 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3740: done (2 left) 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3741: done (1 left) 2024-09-10T16:30:45.756 INFO:tasks.rados.rados.0.smithi165.stdout:3742: done (0 left) 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:3743: rollback oid 9 current snap is 416 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 9 to 382 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:3744: read oid 35 snap -1 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:3744: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:3745: read oid 50 snap -1 2024-09-10T16:30:45.757 INFO:tasks.rados.rados.0.smithi165.stdout:3745: expect (ObjNum 1275 snap 408 seq_num 1275) 2024-09-10T16:30:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:3746: delete oid 7 current snap is 416 2024-09-10T16:30:45.758 INFO:tasks.rados.rados.0.smithi165.stdout:3745: expect (ObjNum 1275 snap 408 seq_num 1275) 2024-09-10T16:30:45.768 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:30:45.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T16:30:45.769 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:30:45.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.866+0000 7f85b4975640 10 monclient: get_monmap_and_config 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.866+0000 7f85b4975640 10 monclient: build_initial_monmap 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.866+0000 7f85b4975640 10 monclient: monmap: 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:45.867270+0000 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:45.867270+0000 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:45.868 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:45.869 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:45.869 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.866+0000 7f85b4975640 1 Processor -- start 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 -- start start 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient: init 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient: _add_conns ranks=[3,1,4,0,2] 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85b01329f0 0x7f85b0132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-a con 0x7f85b01329f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:30:45.871 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85b00a2f10 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 0x7f85b013dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-c con 0x7f85b009d520 addr v2:172.21.15.165:3300/0 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85b00a0730 con 0x7f85b00a2f10 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f85b00a21a0 con 0x7f85b01329f0 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.872 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f85b00a2480 con 0x7f85b009d520 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85b4975640 10 monclient(hunting): authenticate will time out at 2224.354774s 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85aed9d640 1 -- >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 msgr2=0x7f85b013dd90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85add9b640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7f85b00a2f10 legacy=0x7f85b009ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85b01329f0 0x7f85b0132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85aed9d640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 0x7f85b013dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f85b01329f0 0x7f85b0132dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39472/0 (socket says 172.21.15.164:39472) 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 1 -- 172.21.15.164:0/3516294702 learned_addr learned my addr 172.21.15.164:0/3516294702 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 10 monclient(hunting): get_auth_request con 0x7f85b01329f0 auth_method 0 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.867+0000 7f85ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient(hunting): handle_auth_done global_id 254367 payload 274 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 1 -- 172.21.15.164:0/3516294702 >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 msgr2=0x7f85b013dd90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 1 --2- 172.21.15.164:0/3516294702 >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 0x7f85b013dd90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 1 -- 172.21.15.164:0/3516294702 >> v1:172.21.15.165:6789/0 conn(0x7f85b00a2f10 legacy=0x7f85b009ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient: _finish_hunting 0 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 1 monclient: found mon.noname-a 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:45.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.868+0000 7f85ae59c640 1 -- 172.21.15.164:0/3516294702 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b00a1060 con 0x7f85b01329f0 2024-09-10T16:30:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ae59c640 1 -- 172.21.15.164:0/3516294702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 msgr2=0x7f85b0132dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ae59c640 1 --2- 172.21.15.164:0/3516294702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b0132dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85b00a7f10 tx=0x7f85a002a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee2190d602d52642 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 1 -- 172.21.15.164:0/3516294702 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0037070 con 0x7f85b01329f0 2024-09-10T16:30:45.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: dump: 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.882 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: _finish_auth 0 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: _check_auth_tickets 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 5 monclient: authenticate success, global_id 254367 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 1 -- 172.21.15.164:0/3516294702 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a0032640 con 0x7f85b01329f0 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:45.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 1 -- 172.21.15.164:0/3516294702 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0032a10 con 0x7f85b01329f0 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 4 set_mon_vals no callback set 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85ad59a640 10 monclient: dump: 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.884 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 10 monclient: shutdown 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 -- 172.21.15.164:0/3516294702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 msgr2=0x7f85b0132dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 --2- 172.21.15.164:0/3516294702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b0132dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f85b00a7f10 tx=0x7f85a002a0d0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 -- 172.21.15.164:0/3516294702 shutdown_connections 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 --2- 172.21.15.164:0/3516294702 >> v2:172.21.15.165:3300/0 conn(0x7f85b009d520 0x7f85b013dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 --2- 172.21.15.164:0/3516294702 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b0132dd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 -- 172.21.15.164:0/3516294702 >> 172.21.15.164:0/3516294702 conn(0x7f85b00a7320 msgr2=0x7f85b00a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:45.885 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.871+0000 7f85b4975640 1 -- 172.21.15.164:0/3516294702 shutdown_connections 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.872+0000 7f85b4975640 1 -- 172.21.15.164:0/3516294702 wait complete. 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.872+0000 7f85b4975640 10 monclient: build_initial_monmap 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.872+0000 7f85b4975640 10 monclient: monmap: 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.886 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.872+0000 7f85b4975640 1 Processor -- start 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 -- start start 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient: init 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85b009d520 0x7f85b01cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-b con 0x7f85b009d520 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85b00a2f10 0x7f85b0131910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-c con 0x7f85b00a2f10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b00723b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): picked mon.noname-a con 0x7f85b01329f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0133490 con 0x7f85b01329f0 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0133310 con 0x7f85b009d520 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f85b009d900 con 0x7f85b00a2f10 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85b4975640 10 monclient(hunting): authenticate will time out at 2224.360536s 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85ae59c640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85b009d520 msgr2=0x7f85b01cada0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b00723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85ae59c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85b009d520 0x7f85b01cada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b00723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39488/0 (socket says 172.21.15.164:39488) 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 learned_addr learned my addr 172.21.15.164:0/2609143372 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85b00a2f10 0x7f85b0131910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): get_auth_request con 0x7f85b01329f0 auth_method 0 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): get_auth_request con 0x7f85b00a2f10 auth_method 0 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85add9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient(hunting): handle_auth_done global_id 254370 payload 824 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85b00a2f10 msgr2=0x7f85b0131910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f85b00a2f10 0x7f85b0131910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85b009d520 msgr2=0x7f85b01cada0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:45.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f85b009d520 0x7f85b01cada0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 monclient: found mon.noname-a 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.873+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85b011f010 con 0x7f85b01329f0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 msgr2=0x7f85b00723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85aed9d640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b00723b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85a800be50 tx=0x7f85a8009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8016630 con 0x7f85b01329f0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: dump: 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.891 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: _finish_auth 0 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:30:45.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 5 monclient: authenticate success, global_id 254370 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85a8010030 con 0x7f85b01329f0 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a8016a10 con 0x7f85b01329f0 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 10 monclient: _renew_subs 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85b011f2f0 con 0x7f85b01329f0 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a7fff640 4 set_mon_vals no callback set 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:45.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85a6ffd640 10 monclient: dump: 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 10 monclient: _renew_subs 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.876+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85b011ed30 con 0x7f85b01329f0 2024-09-10T16:30:45.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85b4975640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85b4975640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 0x7f85b0072cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f85b0073210 con 0x7f85b00728f0 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 0x7f85b0072cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient: get_auth_request con 0x7f85b00728f0 auth_method 0 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f85a80153e0 con 0x7f85b01329f0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 10 monclient(hunting): handle_auth_done global_id 254373 payload 824 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85ae59c640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85b0132290 con 0x7f85b00728f0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85a6ffd640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f859003e1c0 0x7f8590040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(841..841 src has 677..841) ==== 4843+0+0 (secure 0 0 0) 0x7f85a8053700 con 0x7f85b01329f0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85add9b640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f859003e1c0 0x7f8590040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.877+0000 7f85add9b640 10 monclient: get_auth_request con 0x7f859003e1c0 auth_method 0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.879+0000 7f85ae59c640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 msgr2=0x7f85b0072cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.879+0000 7f85ae59c640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 0x7f85b0072cd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85a0034000 tx=0x7f85a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a0037040 con 0x7f85b00728f0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 10 monclient: dump: 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f85a0009490 con 0x7f85b00728f0 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:45.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:45.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 msgr2=0x7f85b0072cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 0x7f85b0072cd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f85a0034000 tx=0x7f85a00046c0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 msgr2=0x7f85b0072cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:45.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85a6ffd640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b00728f0 0x7f85b0072cd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:45.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.880+0000 7f85add9b640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f859003e1c0 0x7f8590040680 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f85b0133ca0 tx=0x7f8598009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.898 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 648277 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85b4975640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85b4975640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 0x7f85b01bdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85b4975640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f85b00a68b0 con 0x7f85b01bd860 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 0x7f85b01bdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient: get_auth_request con 0x7f85b01bd860 auth_method 0 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:45.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 10 monclient(hunting): handle_auth_done global_id 254376 payload 824 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.886+0000 7f85ae59c640 1 -- 172.21.15.164:0/2609143372 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f85b0132dd0 con 0x7f85b01bd860 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85ae59c640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 msgr2=0x7f85b01bdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85ae59c640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 0x7f85b01bdc70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85a002a110 tx=0x7f85a0009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85a00482e0 con 0x7f85b01bd860 2024-09-10T16:30:45.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 10 monclient: dump: 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:45.902 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4712 (secure 0 0 0) 0x7f85a003e880 con 0x7f85b01bd860 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 msgr2=0x7f85b01bdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 0x7f85b01bdc70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85a002a110 tx=0x7f85a0009240 comp rx=0 tx=0).stop 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 msgr2=0x7f85b01bdc70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:45.903 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85a6ffd640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01bd860 0x7f85b01bdc70 secure :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f85a002a110 tx=0x7f85a0009240 comp rx=0 tx=0).stop 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 242, 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 134540, 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:45.904 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:45.905 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:45.906 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:45.907 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:45.908 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:45.909 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:45.910 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:45.911 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.912 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:45.913 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:45.914 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.915 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:45.916 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:45.917 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:45.918 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f859003e1c0 msgr2=0x7f8590040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f859003e1c0 0x7f8590040680 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f85b0133ca0 tx=0x7f8598009110 comp rx=0 tx=0).stop 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 10 monclient: shutdown 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 msgr2=0x7f85b00723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 --2- 172.21.15.164:0/2609143372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f85b01329f0 0x7f85b00723b0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f85a800be50 tx=0x7f85a8009f40 comp rx=0 tx=0).stop 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 reap_dead start 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85aed9d640 1 -- 172.21.15.164:0/2609143372 reap_dead start 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 shutdown_connections 2024-09-10T16:30:45.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 >> 172.21.15.164:0/2609143372 conn(0x7f85b00a7320 msgr2=0x7f85b0133cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 shutdown_connections 2024-09-10T16:30:45.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:45.889+0000 7f85b4975640 1 -- 172.21.15.164:0/2609143372 wait complete. 2024-09-10T16:30:45.921 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:30:45.921 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:30:45.921 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:30:45.948 INFO:tasks.rados.rados.0.smithi165.stdout:3743: finishing rollback tid 0 to smithi16534841-9 2024-09-10T16:30:45.949 INFO:tasks.rados.rados.0.smithi165.stdout:3743: finishing rollback tid 1 to smithi16534841-9 2024-09-10T16:30:45.949 INFO:tasks.rados.rados.0.smithi165.stdout:3743: finishing rollback tid 2 to smithi16534841-9 2024-09-10T16:30:45.949 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1919 (ObjNum 1162 snap 365 seq_num 1162) dirty exists 2024-09-10T16:30:45.949 INFO:tasks.rados.rados.0.smithi165.stdout:3744: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:30:46.021 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 10 monclient: get_monmap_and_config 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 10 monclient: build_initial_monmap 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 10 monclient: monmap: 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:30:46.020914+0000 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:30:46.020914+0000 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:30:46.022 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:30:46.023 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:30:46.023 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:30:46.023 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:30:46.023 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 1 Processor -- start 2024-09-10T16:30:46.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 1 -- start start 2024-09-10T16:30:46.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.020+0000 7fb803eb8640 10 monclient: init 2024-09-10T16:30:46.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient: _add_conns ranks=[2,0,1,4,3] 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb7fc0a40d0 addr v1:172.21.15.164:6789/0 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb7fc1367d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7fc0a4ac0 con 0x7fb7fc0a40d0 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7fc0a1120 con 0x7fb7fc1367d0 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0a1400 con 0x7fb7fc135ce0 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:46.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb803eb8640 10 monclient(hunting): authenticate will time out at 2224.508441s 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb802455640 1 -- >> v1:172.21.15.165:6789/0 conn(0x7fb7fc1367d0 legacy=0x7fb7fc13d290 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.165:6789/0 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:53780/0 (socket says 172.21.15.164:53780) 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 -- 172.21.15.164:0/1082728249 learned_addr learned my addr 172.21.15.164:0/1082728249 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): get_auth_request con 0x7fb7fc135ce0 auth_method 0 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient(hunting): handle_auth_done global_id 274160 payload 274 2024-09-10T16:30:46.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 -- 172.21.15.164:0/1082728249 >> v1:172.21.15.165:6789/0 conn(0x7fb7fc1367d0 legacy=0x7fb7fc13d290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 -- 172.21.15.164:0/1082728249 >> v1:172.21.15.164:6789/0 conn(0x7fb7fc0a40d0 legacy=0x7fb7fc1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient: _finish_hunting 0 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 monclient: found mon.noname-b 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.021+0000 7fb801c54640 1 -- 172.21.15.164:0/1082728249 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc0a28e0 con 0x7fb7fc135ce0 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb801c54640 1 -- 172.21.15.164:0/1082728249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 msgr2=0x7fb7fc0a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb801c54640 1 --2- 172.21.15.164:0/1082728249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009e80 tx=0x7fb7ec02a520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52d5d9435fffb394 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 1 -- 172.21.15.164:0/1082728249 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec036070 con 0x7fb7fc135ce0 2024-09-10T16:30:46.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: dump: 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.031 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: _finish_auth 0 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: _check_auth_tickets 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 1 -- 172.21.15.164:0/1082728249 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7ec007800 con 0x7fb7fc135ce0 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb803eb8640 5 monclient: authenticate success, global_id 274160 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb803eb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb803eb8640 10 monclient: get_monmap_and_config success 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 1 -- 172.21.15.164:0/1082728249 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec007bd0 con 0x7fb7fc135ce0 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb803eb8640 4 set_mon_vals no callback set 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.024+0000 7fb800c52640 10 monclient: dump: 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 10 monclient: shutdown 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 -- 172.21.15.164:0/1082728249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 msgr2=0x7fb7fc0a23a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1082728249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec009e80 tx=0x7fb7ec02a520 comp rx=0 tx=0).stop 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 -- 172.21.15.164:0/1082728249 shutdown_connections 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1082728249 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc135ce0 0x7fb7fc0a23a0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 -- 172.21.15.164:0/1082728249 >> 172.21.15.164:0/1082728249 conn(0x7fb7fc0a4f50 msgr2=0x7fb7fc0a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 -- 172.21.15.164:0/1082728249 shutdown_connections 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.025+0000 7fb803eb8640 1 -- 172.21.15.164:0/1082728249 wait complete. 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient: build_initial_monmap 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient: monmap: 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:30:46.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 Processor -- start 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 -- start start 2024-09-10T16:30:46.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient: init 2024-09-10T16:30:46.036 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0a40d0 0x7fb7fc131920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-b con 0x7fb7fc0a40d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 0x7fb7fc1cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-a con 0x7fb7fc135ce0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc1367d0 0x7fb7fc1cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): picked mon.noname-c con 0x7fb7fc1367d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc13ed40 con 0x7fb7fc135ce0 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb7fc13ebc0 con 0x7fb7fc0a40d0 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc13eec0 con 0x7fb7fc1367d0 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb803eb8640 10 monclient(hunting): authenticate will time out at 2224.513842s 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801c54640 1 -- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0a40d0 msgr2=0x7fb7fc131920 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.165:3300/0 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801c54640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0a40d0 0x7fb7fc131920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 0x7fb7fc1cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb802455640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc1367d0 0x7fb7fc1cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 0x7fb7fc1cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:39520/0 (socket says 172.21.15.164:39520) 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 1 -- 172.21.15.164:0/1840038405 learned_addr learned my addr 172.21.15.164:0/1840038405 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 10 monclient(hunting): get_auth_request con 0x7fb7fc135ce0 auth_method 0 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb801453640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb802455640 10 monclient(hunting): get_auth_request con 0x7fb7fc1367d0 auth_method 0 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb802455640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb802455640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.026+0000 7fb802455640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:46.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb802455640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb802455640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb802455640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient(hunting): handle_auth_done global_id 254379 payload 824 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc1367d0 msgr2=0x7fb7fc1cdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc1367d0 0x7fb7fc1cdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0a40d0 msgr2=0x7fb7fc131920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb7fc0a40d0 0x7fb7fc131920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient: _finish_hunting 0 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 monclient: found mon.noname-a 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:30:46.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb801453640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7fc132260 con 0x7fb7fc135ce0 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.027+0000 7fb802455640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc1367d0 0x7fb7fc1cdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb801453640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 msgr2=0x7fb7fc1cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb801453640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 0x7fb7fc1cd980 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000c430 tx=0x7fb7f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f00119a0 con 0x7fb7fc135ce0 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: dump: 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.048 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: _finish_auth 0 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 5 monclient: authenticate success, global_id 254379 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7f0017070 con 0x7fb7fc135ce0 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 10 monclient: _renew_subs 2024-09-10T16:30:46.049 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7fc06e910 con 0x7fb7fc135ce0 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7f0011d80 con 0x7fb7fc135ce0 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fa7fc640 10 monclient: dump: 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.050 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 10 monclient: _renew_subs 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb7fbfff640 4 set_mon_vals no callback set 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.029+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7fc1338d0 con 0x7fb7fc135ce0 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb803eb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 0x7fb7fc06cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc0a23a0 con 0x7fb7fc09beb0 2024-09-10T16:30:46.051 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 0x7fb7fc06cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient: get_auth_request con 0x7fb7fc09beb0 auth_method 0 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb7f001c460 con 0x7fb7fc135ce0 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 10 monclient(hunting): handle_auth_done global_id 274166 payload 824 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb801c54640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7fc13e790 con 0x7fb7fc09beb0 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb7fa7fc640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb7e803e1f0 0x7fb7e80406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.030+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(841..841 src has 677..841) ==== 4843+0+0 (secure 0 0 0) 0x7fb7f0052b40 con 0x7fb7fc135ce0 2024-09-10T16:30:46.052 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.031+0000 7fb802455640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb7e803e1f0 0x7fb7e80406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.031+0000 7fb802455640 10 monclient: get_auth_request con 0x7fb7e803e1f0 auth_method 0 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb801c54640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 msgr2=0x7fb7fc06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb801c54640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 0x7fb7fc06cb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec033000 tx=0x7fb7ec004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec041020 con 0x7fb7fc09beb0 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 10 monclient: dump: 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.053 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb7ec03c880 con 0x7fb7fc09beb0 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 msgr2=0x7fb7fc06cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 0x7fb7fc06cb70 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec033000 tx=0x7fb7ec004a40 comp rx=0 tx=0).stop 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 msgr2=0x7fb7fc06cb70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb7fa7fc640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc09beb0 0x7fb7fc06cb70 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.033+0000 7fb802455640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb7e803e1f0 0x7fb7e80406b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0751d0 tx=0x7fb7f40090e0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb803eb8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:30:46.054 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 0x7fb7fc06d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb803eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb7fc06da00 con 0x7fb7fc06d0b0 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 0x7fb7fc06d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient: get_auth_request con 0x7fb7fc06d0b0 auth_method 0 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 10 monclient(hunting): handle_auth_done global_id 274169 payload 824 2024-09-10T16:30:46.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.039+0000 7fb801c54640 1 -- 172.21.15.164:0/1840038405 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7fc06e690 con 0x7fb7fc06d0b0 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb801c54640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 msgr2=0x7fb7fc06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb801c54640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 0x7fb7fc06d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec02a560 tx=0x7fb7ec009810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7ec041050 con 0x7fb7fc06d0b0 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 10 monclient: dump: 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:30:46.056 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4709 (secure 0 0 0) 0x7fb7ec048860 con 0x7fb7fc06d0b0 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 msgr2=0x7fb7fc06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 0x7fb7fc06d4c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec02a560 tx=0x7fb7ec009810 comp rx=0 tx=0).stop 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 msgr2=0x7fb7fc06d4c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb7fa7fc640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb7fc06d0b0 0x7fb7fc06d4c0 secure :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fb7ec02a560 tx=0x7fb7ec009810 comp rx=0 tx=0).stop 2024-09-10T16:30:46.057 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 242, 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 23285, 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:46.058 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:46.059 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:46.060 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:30:46.061 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:30:46.062 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:46.063 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:46.064 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.065 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:46.066 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.067 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.068 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:30:46.069 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:30:46.070 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:30:46.071 INFO:tasks.rados.rados.0.smithi165.stdout:3743: done (3 left) 2024-09-10T16:30:46.071 INFO:tasks.rados.rados.0.smithi165.stdout:3744: done (2 left) 2024-09-10T16:30:46.071 INFO:tasks.rados.rados.0.smithi165.stdout:3745: done (1 left) 2024-09-10T16:30:46.071 INFO:tasks.rados.rados.0.smithi165.stdout:3746: done (0 left) 2024-09-10T16:30:46.072 INFO:tasks.rados.rados.0.smithi165.stdout:3747: snap_create 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb7e803e1f0 msgr2=0x7fb7e80406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb7e803e1f0 0x7fb7e80406b0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fb7fc0751d0 tx=0x7fb7f40090e0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 10 monclient: shutdown 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 msgr2=0x7fb7fc1cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 --2- 172.21.15.164:0/1840038405 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb7fc135ce0 0x7fb7fc1cd980 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fb7f000c430 tx=0x7fb7f000c8f0 comp rx=0 tx=0).stop 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb802455640 1 -- 172.21.15.164:0/1840038405 reap_dead start 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb802455640 1 -- 172.21.15.164:0/1840038405 reap_dead start 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 shutdown_connections 2024-09-10T16:30:46.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 >> 172.21.15.164:0/1840038405 conn(0x7fb7fc0a4f50 msgr2=0x7fb7fc0a2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:30:46.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 shutdown_connections 2024-09-10T16:30:46.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:30:46.042+0000 7fb803eb8640 1 -- 172.21.15.164:0/1840038405 wait complete. 2024-09-10T16:30:46.074 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:30:46.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:30:46.909 INFO:tasks.rados.rados.0.smithi165.stdout:3747: done (0 left) 2024-09-10T16:30:46.909 INFO:tasks.rados.rados.0.smithi165.stdout:3748: snap_create 2024-09-10T16:30:47.915 INFO:tasks.rados.rados.0.smithi165.stdout:3748: done (0 left) 2024-09-10T16:30:47.915 INFO:tasks.rados.rados.0.smithi165.stdout:3749: read oid 40 snap -1 2024-09-10T16:30:47.915 INFO:tasks.rados.rados.0.smithi165.stdout:3749: expect (ObjNum 1088 snap 332 seq_num 1088) 2024-09-10T16:30:47.915 INFO:tasks.rados.rados.0.smithi165.stdout:3750: snap_remove snap 397 2024-09-10T16:30:48.917 INFO:tasks.rados.rados.0.smithi165.stdout:3750: done (1 left) 2024-09-10T16:30:48.918 INFO:tasks.rados.rados.0.smithi165.stdout:3751: write oid 22 current snap is 418 2024-09-10T16:30:48.918 INFO:tasks.rados.rados.0.smithi165.stdout:3751: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2024-09-10T16:30:48.924 INFO:tasks.rados.rados.0.smithi165.stdout:3751: writing smithi16534841-22 from 464223 to 1123841 tid 1 2024-09-10T16:30:48.933 INFO:tasks.rados.rados.0.smithi165.stdout:3751: writing smithi16534841-22 from 1879407 to 2619192 tid 2 2024-09-10T16:30:48.939 INFO:tasks.rados.rados.0.smithi165.stdout:3751: writing smithi16534841-22 from 3248120 to 3664223 tid 3 2024-09-10T16:30:48.939 INFO:tasks.rados.rados.0.smithi165.stdout:3752: write oid 45 current snap is 418 2024-09-10T16:30:48.939 INFO:tasks.rados.rados.0.smithi165.stdout:3752: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2024-09-10T16:30:48.946 INFO:tasks.rados.rados.0.smithi165.stdout:3752: writing smithi16534841-45 from 481030 to 1215897 tid 1 2024-09-10T16:30:48.952 INFO:tasks.rados.rados.0.smithi165.stdout:3752: writing smithi16534841-45 from 1937889 to 2421332 tid 2 2024-09-10T16:30:48.959 INFO:tasks.rados.rados.0.smithi165.stdout:3752: writing smithi16534841-45 from 2875543 to 3427969 tid 3 2024-09-10T16:30:48.959 INFO:tasks.rados.rados.0.smithi165.stdout:3753: copy_from oid 1 from oid 9 current snap is 418 2024-09-10T16:30:48.959 INFO:tasks.rados.rados.0.smithi165.stdout:3749: expect (ObjNum 1088 snap 332 seq_num 1088) 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: finishing write tid 1 to smithi16534841-22 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: finishing write tid 2 to smithi16534841-22 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: finishing write tid 3 to smithi16534841-22 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: finishing write tid 4 to smithi16534841-22 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: finishing write tid 5 to smithi16534841-22 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: oid 22 updating version 0 to 2021 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: oid 22 version 2021 is already newer than 2018 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: oid 22 version 2021 is already newer than 2020 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: oid 22 version 2021 is already newer than 2019 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 2021 (ObjNum 1289 snap 418 seq_num 1289) dirty exists 2024-09-10T16:30:49.032 INFO:tasks.rados.rados.0.smithi165.stdout:3751: left oid 22 (ObjNum 1289 snap 418 seq_num 1289) 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: finishing write tid 1 to smithi16534841-45 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3749: done (3 left) 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3751: done (2 left) 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3754: snap_create 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: finishing write tid 2 to smithi16534841-45 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: finishing write tid 3 to smithi16534841-45 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: finishing write tid 4 to smithi16534841-45 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: finishing write tid 5 to smithi16534841-45 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: oid 45 updating version 0 to 1718 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: oid 45 updating version 1718 to 1719 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: oid 45 updating version 1719 to 1720 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: oid 45 updating version 1720 to 1721 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1721 (ObjNum 1290 snap 418 seq_num 1290) dirty exists 2024-09-10T16:30:49.033 INFO:tasks.rados.rados.0.smithi165.stdout:3752: left oid 45 (ObjNum 1290 snap 418 seq_num 1290) 2024-09-10T16:30:49.034 INFO:tasks.rados.rados.0.smithi165.stdout:3753: finishing copy_from racing read to smithi16534841-1 2024-09-10T16:30:49.034 INFO:tasks.rados.rados.0.smithi165.stdout:3753: finishing copy_from to smithi16534841-1 2024-09-10T16:30:49.034 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 1919 (ObjNum 1162 snap 365 seq_num 1162) dirty exists 2024-09-10T16:30:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:3752: done (2 left) 2024-09-10T16:30:49.925 INFO:tasks.rados.rados.0.smithi165.stdout:3753: done (1 left) 2024-09-10T16:30:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:3754: done (0 left) 2024-09-10T16:30:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:3755: write (excl) oid 5 current snap is 419 2024-09-10T16:30:49.926 INFO:tasks.rados.rados.0.smithi165.stdout:3755: seq_num 1291 ranges {497837=410116,1596371=743454,2902966=763698} 2024-09-10T16:30:49.929 INFO:tasks.rados.rados.0.smithi165.stdout:3755: writing smithi16534841-5 from 497837 to 907953 tid 1 2024-09-10T16:30:49.937 INFO:tasks.rados.rados.0.smithi165.stdout:3755: writing smithi16534841-5 from 1596371 to 2339825 tid 2 2024-09-10T16:30:49.944 INFO:tasks.rados.rados.0.smithi165.stdout:3755: writing smithi16534841-5 from 2902966 to 3666664 tid 3 2024-09-10T16:30:49.944 INFO:tasks.rados.rados.0.smithi165.stdout:3756: snap_remove snap 344 2024-09-10T16:30:50.926 INFO:tasks.rados.rados.0.smithi165.stdout:3756: done (1 left) 2024-09-10T16:30:50.926 INFO:tasks.rados.rados.0.smithi165.stdout:3757: read oid 31 snap -1 2024-09-10T16:30:50.926 INFO:tasks.rados.rados.0.smithi165.stdout:3757: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3758: snap_create 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: finishing write tid 1 to smithi16534841-5 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: finishing write tid 2 to smithi16534841-5 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: finishing write tid 3 to smithi16534841-5 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: finishing write tid 4 to smithi16534841-5 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: finishing write tid 5 to smithi16534841-5 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: oid 5 updating version 0 to 1889 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: oid 5 updating version 1889 to 1890 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: oid 5 updating version 1890 to 1891 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: oid 5 version 1891 is already newer than 1888 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1891 (ObjNum 1291 snap 419 seq_num 1291) dirty exists 2024-09-10T16:30:50.927 INFO:tasks.rados.rados.0.smithi165.stdout:3755: left oid 5 (ObjNum 1291 snap 419 seq_num 1291) 2024-09-10T16:30:50.944 INFO:tasks.rados.rados.0.smithi165.stdout:3757: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:30:50.944 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:51.027 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1516188 2024-09-10T16:30:51.384 INFO:tasks.rados.rados.0.smithi165.stdout:3755: done (2 left) 2024-09-10T16:30:51.385 INFO:tasks.rados.rados.0.smithi165.stdout:3757: done (1 left) 2024-09-10T16:30:51.385 INFO:tasks.rados.rados.0.smithi165.stdout:3758: done (0 left) 2024-09-10T16:30:51.385 INFO:tasks.rados.rados.0.smithi165.stdout:3759: copy_from oid 35 from oid 47 current snap is 420 2024-09-10T16:30:51.385 INFO:tasks.rados.rados.0.smithi165.stdout:3760: write (excl) oid 2 current snap is 420 2024-09-10T16:30:51.385 INFO:tasks.rados.rados.0.smithi165.stdout:3760: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2024-09-10T16:30:51.390 INFO:tasks.rados.rados.0.smithi165.stdout:3760: writing smithi16534841-2 from 514644 to 1000009 tid 1 2024-09-10T16:30:51.396 INFO:tasks.rados.rados.0.smithi165.stdout:3760: writing smithi16534841-2 from 1654853 to 2141965 tid 2 2024-09-10T16:30:51.404 INFO:tasks.rados.rados.0.smithi165.stdout:3760: writing smithi16534841-2 from 2930389 to 3505359 tid 3 2024-09-10T16:30:51.404 INFO:tasks.rados.rados.0.smithi165.stdout:3761: write (excl) oid 23 current snap is 420 2024-09-10T16:30:51.404 INFO:tasks.rados.rados.0.smithi165.stdout:3760: finishing write tid 1 to smithi16534841-2 2024-09-10T16:30:51.404 INFO:tasks.rados.rados.0.smithi165.stdout:3761: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2024-09-10T16:30:51.410 INFO:tasks.rados.rados.0.smithi165.stdout:3761: writing smithi16534841-23 from 531451 to 1208418 tid 1 2024-09-10T16:30:51.418 INFO:tasks.rados.rados.0.smithi165.stdout:3761: writing smithi16534841-23 from 1713335 to 2460458 tid 2 2024-09-10T16:30:51.423 INFO:tasks.rados.rados.0.smithi165.stdout:3761: writing smithi16534841-23 from 2957812 to 3460407 tid 3 2024-09-10T16:30:51.423 INFO:tasks.rados.rados.0.smithi165.stdout:3762: write (excl) oid 10 current snap is 420 2024-09-10T16:30:51.423 INFO:tasks.rados.rados.0.smithi165.stdout:3759: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:30:51.423 INFO:tasks.rados.rados.0.smithi165.stdout:3762: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2024-09-10T16:30:51.430 INFO:tasks.rados.rados.0.smithi165.stdout:3762: writing smithi16534841-10 from 548258 to 1300474 tid 1 2024-09-10T16:30:51.435 INFO:tasks.rados.rados.0.smithi165.stdout:3762: writing smithi16534841-10 from 1771817 to 2262598 tid 2 2024-09-10T16:30:51.444 INFO:tasks.rados.rados.0.smithi165.stdout:3762: writing smithi16534841-10 from 2985235 to 3699102 tid 3 2024-09-10T16:30:51.444 INFO:tasks.rados.rados.0.smithi165.stdout:3763: copy_from oid 34 from oid 5 current snap is 420 2024-09-10T16:30:51.444 INFO:tasks.rados.rados.0.smithi165.stdout:3759: finishing copy_from to smithi16534841-35 2024-09-10T16:30:51.444 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1967 (ObjNum 1138 snap 351 seq_num 1138) dirty exists 2024-09-10T16:30:51.444 INFO:tasks.rados.rados.0.smithi165.stdout:3760: finishing write tid 2 to smithi16534841-2 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: finishing write tid 3 to smithi16534841-2 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: finishing write tid 4 to smithi16534841-2 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: finishing write tid 5 to smithi16534841-2 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: oid 2 updating version 0 to 2094 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: oid 2 updating version 2094 to 2095 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: oid 2 updating version 2095 to 2096 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: oid 2 updating version 2096 to 2097 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 2097 (ObjNum 1292 snap 420 seq_num 1292) dirty exists 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3760: left oid 2 (ObjNum 1292 snap 420 seq_num 1292) 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3761: finishing write tid 1 to smithi16534841-23 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3761: finishing write tid 2 to smithi16534841-23 2024-09-10T16:30:51.445 INFO:tasks.rados.rados.0.smithi165.stdout:3761: finishing write tid 3 to smithi16534841-23 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: finishing write tid 4 to smithi16534841-23 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: finishing write tid 5 to smithi16534841-23 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: oid 23 updating version 0 to 2047 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: oid 23 updating version 2047 to 2050 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: oid 23 version 2050 is already newer than 2048 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: oid 23 version 2050 is already newer than 2049 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 2050 (ObjNum 1293 snap 420 seq_num 1293) dirty exists 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: left oid 23 (ObjNum 1293 snap 420 seq_num 1293) 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3762: finishing write tid 1 to smithi16534841-10 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3762: finishing write tid 2 to smithi16534841-10 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3759: done (4 left) 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3760: done (3 left) 2024-09-10T16:30:51.446 INFO:tasks.rados.rados.0.smithi165.stdout:3761: done (2 left) 2024-09-10T16:30:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:3764: read oid 31 snap 365 2024-09-10T16:30:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:3764: expect deleted 2024-09-10T16:30:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:3765: read oid 9 snap 391 2024-09-10T16:30:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:3765: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:30:51.447 INFO:tasks.rados.rados.0.smithi165.stdout:3766: snap_remove snap 402 2024-09-10T16:30:52.391 INFO:tasks.rados.rados.0.smithi165.stdout:3766: done (4 left) 2024-09-10T16:30:52.391 INFO:tasks.rados.rados.0.smithi165.stdout:3767: snap_remove snap 334 2024-09-10T16:30:53.398 INFO:tasks.rados.rados.0.smithi165.stdout:3767: done (4 left) 2024-09-10T16:30:53.398 INFO:tasks.rados.rados.0.smithi165.stdout:3768: delete oid 3 current snap is 420 2024-09-10T16:30:53.398 INFO:tasks.rados.rados.0.smithi165.stdout:3762: finishing write tid 3 to smithi16534841-10 2024-09-10T16:30:53.398 INFO:tasks.rados.rados.0.smithi165.stdout:3765: expect (ObjNum 1162 snap 365 seq_num 1162) 2024-09-10T16:30:53.553 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 365605 2024-09-10T16:30:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:3762: finishing write tid 4 to smithi16534841-10 2024-09-10T16:30:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:3762: finishing write tid 5 to smithi16534841-10 2024-09-10T16:30:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:3762: oid 10 updating version 0 to 1888 2024-09-10T16:30:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:3762: oid 10 updating version 1888 to 1889 2024-09-10T16:30:53.622 INFO:tasks.rados.rados.0.smithi165.stdout:3762: oid 10 updating version 1889 to 1890 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3762: oid 10 updating version 1890 to 1891 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1891 (ObjNum 1294 snap 420 seq_num 1294) dirty exists 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3762: left oid 10 (ObjNum 1294 snap 420 seq_num 1294) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3763: finishing copy_from racing read to smithi16534841-34 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3762: done (4 left) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3764: done (3 left) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3765: done (2 left) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3768: done (1 left) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3769: rollback oid 1 current snap is 420 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 389 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3763: finishing copy_from to smithi16534841-34 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1891 (ObjNum 1291 snap 419 seq_num 1291) dirty exists 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3763: done (1 left) 2024-09-10T16:30:53.623 INFO:tasks.rados.rados.0.smithi165.stdout:3770: read oid 36 snap -1 2024-09-10T16:30:53.624 INFO:tasks.rados.rados.0.smithi165.stdout:3770: expect (ObjNum 1282 snap 414 seq_num 1282) 2024-09-10T16:30:53.624 INFO:tasks.rados.rados.0.smithi165.stdout:3771: rollback oid 41 current snap is 420 2024-09-10T16:30:53.624 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 41 to 311 2024-09-10T16:30:53.624 INFO:tasks.rados.rados.0.smithi165.stdout:3772: write (excl) oid 25 current snap is 420 2024-09-10T16:30:53.624 INFO:tasks.rados.rados.0.smithi165.stdout:3772: seq_num 1296 ranges {581872=502714,1488781=494450,2640081=736411} 2024-09-10T16:30:53.627 INFO:tasks.rados.rados.0.smithi165.stdout:3772: writing smithi16534841-25 from 581872 to 1084586 tid 1 2024-09-10T16:30:53.633 INFO:tasks.rados.rados.0.smithi165.stdout:3772: writing smithi16534841-25 from 1488781 to 1983231 tid 2 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3772: writing smithi16534841-25 from 2640081 to 3376492 tid 3 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3773: read oid 24 snap -1 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3773: expect (ObjNum 1254 snap 405 seq_num 1254) 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3774: write oid 14 current snap is 420 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3769: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:30:53.641 INFO:tasks.rados.rados.0.smithi165.stdout:3774: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2024-09-10T16:30:53.647 INFO:tasks.rados.rados.0.smithi165.stdout:3774: writing smithi16534841-14 from 598679 to 1176642 tid 1 2024-09-10T16:30:53.655 INFO:tasks.rados.rados.0.smithi165.stdout:3774: writing smithi16534841-14 from 1947263 to 2585371 tid 2 2024-09-10T16:30:53.662 INFO:tasks.rados.rados.0.smithi165.stdout:3774: writing smithi16534841-14 from 3351151 to 3798679 tid 3 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3775: copy_from oid 23 from oid 12 current snap is 420 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3769: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3769: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2053 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3771: finishing rollback tid 0 to smithi16534841-41 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3771: finishing rollback tid 1 to smithi16534841-41 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3771: finishing rollback tid 2 to smithi16534841-41 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 2056 (ObjNum 719 snap 221 seq_num 719) dirty exists 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3772: finishing write tid 1 to smithi16534841-25 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:3770: expect (ObjNum 1282 snap 414 seq_num 1282) 2024-09-10T16:30:53.663 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:53.970 INFO:tasks.rados.rados.0.smithi165.stdout:3772: finishing write tid 2 to smithi16534841-25 2024-09-10T16:30:53.970 INFO:tasks.rados.rados.0.smithi165.stdout:3773: expect (ObjNum 1254 snap 405 seq_num 1254) 2024-09-10T16:30:53.970 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:54.016 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:54.063 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: finishing write tid 3 to smithi16534841-25 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: finishing write tid 4 to smithi16534841-25 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: finishing write tid 5 to smithi16534841-25 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: oid 25 updating version 0 to 2110 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: oid 25 updating version 2110 to 2111 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: oid 25 updating version 2111 to 2112 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: oid 25 updating version 2112 to 2113 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 25 v 2113 (ObjNum 1296 snap 420 seq_num 1296) dirty exists 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3772: left oid 25 (ObjNum 1296 snap 420 seq_num 1296) 2024-09-10T16:30:54.110 INFO:tasks.rados.rados.0.smithi165.stdout:3774: finishing write tid 1 to smithi16534841-14 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: finishing write tid 2 to smithi16534841-14 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: finishing write tid 3 to smithi16534841-14 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: finishing write tid 4 to smithi16534841-14 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: finishing write tid 5 to smithi16534841-14 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: oid 14 updating version 0 to 1922 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: oid 14 version 1922 is already newer than 1921 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: oid 14 updating version 1922 to 1923 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: oid 14 version 1923 is already newer than 1920 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1923 (ObjNum 1297 snap 420 seq_num 1297) dirty exists 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3774: left oid 14 (ObjNum 1297 snap 420 seq_num 1297) 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3775: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:30:54.111 INFO:tasks.rados.rados.0.smithi165.stdout:3769: done (6 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3770: done (5 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3771: done (4 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3772: done (3 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3773: done (2 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3774: done (1 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3776: copy_from oid 48 from oid 8 current snap is 420 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3775: finishing copy_from to smithi16534841-23 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1658 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3775: done (1 left) 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3777: read oid 13 snap -1 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3777: expect deleted 2024-09-10T16:30:54.112 INFO:tasks.rados.rados.0.smithi165.stdout:3778: snap_create 2024-09-10T16:30:54.113 INFO:tasks.rados.rados.0.smithi165.stdout:3776: finishing copy_from to smithi16534841-48 2024-09-10T16:30:54.113 INFO:tasks.rados.rados.0.smithi165.stdout:3776: got expected ENOENT (src dne) 2024-09-10T16:30:54.113 INFO:tasks.rados.rados.0.smithi165.stdout:3776: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:30:54.400 INFO:tasks.rados.rados.0.smithi165.stdout:3776: done (2 left) 2024-09-10T16:30:54.400 INFO:tasks.rados.rados.0.smithi165.stdout:3777: done (1 left) 2024-09-10T16:30:54.400 INFO:tasks.rados.rados.0.smithi165.stdout:3778: done (0 left) 2024-09-10T16:30:54.400 INFO:tasks.rados.rados.0.smithi165.stdout:3779: copy_from oid 13 from oid 7 current snap is 421 2024-09-10T16:30:54.400 INFO:tasks.rados.rados.0.smithi165.stdout:3780: delete oid 21 current snap is 421 2024-09-10T16:30:54.404 INFO:tasks.rados.rados.0.smithi165.stdout:3780: done (1 left) 2024-09-10T16:30:54.404 INFO:tasks.rados.rados.0.smithi165.stdout:3781: read oid 27 snap -1 2024-09-10T16:30:54.404 INFO:tasks.rados.rados.0.smithi165.stdout:3781: expect deleted 2024-09-10T16:30:54.404 INFO:tasks.rados.rados.0.smithi165.stdout:3782: snap_create 2024-09-10T16:30:54.404 INFO:tasks.rados.rados.0.smithi165.stdout:3779: finishing copy_from to smithi16534841-13 2024-09-10T16:30:54.405 INFO:tasks.rados.rados.0.smithi165.stdout:3779: got expected ENOENT (src dne) 2024-09-10T16:30:54.405 INFO:tasks.rados.rados.0.smithi165.stdout:3779: finishing copy_from racing read to smithi16534841-13 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3779: done (2 left) 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3781: done (1 left) 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3782: done (0 left) 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3783: read oid 29 snap -1 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3783: expect deleted 2024-09-10T16:30:55.409 INFO:tasks.rados.rados.0.smithi165.stdout:3784: snap_create 2024-09-10T16:30:56.389 INFO:tasks.rados.rados.0.smithi165.stdout:3783: done (1 left) 2024-09-10T16:30:56.389 INFO:tasks.rados.rados.0.smithi165.stdout:3784: done (0 left) 2024-09-10T16:30:56.389 INFO:tasks.rados.rados.0.smithi165.stdout:3785: write (excl) oid 28 current snap is 423 2024-09-10T16:30:56.389 INFO:tasks.rados.rados.0.smithi165.stdout:3785: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2024-09-10T16:30:56.395 INFO:tasks.rados.rados.0.smithi165.stdout:3785: writing smithi16534841-28 from 632293 to 1360754 tid 1 2024-09-10T16:30:56.402 INFO:tasks.rados.rados.0.smithi165.stdout:3785: writing smithi16534841-28 from 1947874 to 2589651 tid 2 2024-09-10T16:30:56.409 INFO:tasks.rados.rados.0.smithi165.stdout:3785: writing smithi16534841-28 from 3005997 to 3692577 tid 3 2024-09-10T16:30:56.409 INFO:tasks.rados.rados.0.smithi165.stdout:3786: write oid 39 current snap is 423 2024-09-10T16:30:56.409 INFO:tasks.rados.rados.0.smithi165.stdout:3785: finishing write tid 1 to smithi16534841-28 2024-09-10T16:30:56.409 INFO:tasks.rados.rados.0.smithi165.stdout:3785: finishing write tid 2 to smithi16534841-28 2024-09-10T16:30:56.409 INFO:tasks.rados.rados.0.smithi165.stdout:3786: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2024-09-10T16:30:56.414 INFO:tasks.rados.rados.0.smithi165.stdout:3786: writing smithi16534841-39 from 649100 to 1052810 tid 1 2024-09-10T16:30:56.420 INFO:tasks.rados.rados.0.smithi165.stdout:3786: writing smithi16534841-39 from 1606356 to 2108144 tid 2 2024-09-10T16:30:56.426 INFO:tasks.rados.rados.0.smithi165.stdout:3786: writing smithi16534841-39 from 2633420 to 3131272 tid 3 2024-09-10T16:30:56.431 INFO:tasks.rados.rados.0.smithi165.stdout:3786: writing smithi16534841-39 from 3636005 to 3849100 tid 4 2024-09-10T16:30:56.431 INFO:tasks.rados.rados.0.smithi165.stdout:3787: copy_from oid 48 from oid 38 current snap is 423 2024-09-10T16:30:56.431 INFO:tasks.rados.rados.0.smithi165.stdout:3785: finishing write tid 3 to smithi16534841-28 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: finishing write tid 4 to smithi16534841-28 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: finishing write tid 5 to smithi16534841-28 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: oid 28 updating version 0 to 1292 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: oid 28 updating version 1292 to 1293 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: oid 28 version 1293 is already newer than 1290 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: oid 28 version 1293 is already newer than 1291 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 28 v 1293 (ObjNum 1299 snap 423 seq_num 1299) dirty exists 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: left oid 28 (ObjNum 1299 snap 423 seq_num 1299) 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 1 to smithi16534841-39 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 2 to smithi16534841-39 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 3 to smithi16534841-39 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3785: done (2 left) 2024-09-10T16:30:56.432 INFO:tasks.rados.rados.0.smithi165.stdout:3788: rollback oid 44 current snap is 423 2024-09-10T16:30:56.433 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 389 2024-09-10T16:30:56.433 INFO:tasks.rados.rados.0.smithi165.stdout:3789: write oid 30 current snap is 423 2024-09-10T16:30:56.433 INFO:tasks.rados.rados.0.smithi165.stdout:3789: seq_num 1301 ranges {665907=595312,1781191=645446,3177196=688711} 2024-09-10T16:30:56.438 INFO:tasks.rados.rados.0.smithi165.stdout:3789: writing smithi16534841-30 from 665907 to 1261219 tid 1 2024-09-10T16:30:56.445 INFO:tasks.rados.rados.0.smithi165.stdout:3789: writing smithi16534841-30 from 1781191 to 2426637 tid 2 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3789: writing smithi16534841-30 from 3177196 to 3865907 tid 3 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3790: write oid 17 current snap is 423 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3787: finishing copy_from to smithi16534841-48 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3787: got expected ENOENT (src dne) 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3787: finishing copy_from racing read to smithi16534841-48 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 4 to smithi16534841-39 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 5 to smithi16534841-39 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3786: finishing write tid 6 to smithi16534841-39 2024-09-10T16:30:56.454 INFO:tasks.rados.rados.0.smithi165.stdout:3786: oid 39 updating version 0 to 2057 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3786: oid 39 updating version 2057 to 2060 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3786: oid 39 version 2060 is already newer than 2059 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3786: oid 39 version 2060 is already newer than 2058 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3786: oid 39 updating version 2060 to 2061 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 39 v 2061 (ObjNum 1300 snap 423 seq_num 1300) dirty exists 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3786: left oid 39 (ObjNum 1300 snap 423 seq_num 1300) 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3788: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3788: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3788: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1846 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3789: finishing write tid 1 to smithi16534841-30 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3789: finishing write tid 2 to smithi16534841-30 2024-09-10T16:30:56.455 INFO:tasks.rados.rados.0.smithi165.stdout:3790: seq_num 1302 ranges {682714=670561,2123320=505457,3088266=636749} 2024-09-10T16:30:56.460 INFO:tasks.rados.rados.0.smithi165.stdout:3790: writing smithi16534841-17 from 682714 to 1353275 tid 1 2024-09-10T16:30:56.467 INFO:tasks.rados.rados.0.smithi165.stdout:3790: writing smithi16534841-17 from 2123320 to 2628777 tid 2 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3790: writing smithi16534841-17 from 3088266 to 3725015 tid 3 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3786: done (4 left) 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3787: done (3 left) 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3788: done (2 left) 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3791: rollback oid 13 current snap is 423 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 411 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3789: finishing write tid 3 to smithi16534841-30 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3789: finishing write tid 4 to smithi16534841-30 2024-09-10T16:30:56.474 INFO:tasks.rados.rados.0.smithi165.stdout:3789: finishing write tid 5 to smithi16534841-30 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: oid 30 updating version 0 to 2484 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: oid 30 updating version 2484 to 2485 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: oid 30 version 2485 is already newer than 2483 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: oid 30 version 2485 is already newer than 2482 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2485 (ObjNum 1301 snap 423 seq_num 1301) dirty exists 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: left oid 30 (ObjNum 1301 snap 423 seq_num 1301) 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3790: finishing write tid 1 to smithi16534841-17 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3790: finishing write tid 2 to smithi16534841-17 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3789: done (2 left) 2024-09-10T16:30:56.475 INFO:tasks.rados.rados.0.smithi165.stdout:3792: snap_create 2024-09-10T16:30:56.476 INFO:tasks.rados.rados.0.smithi165.stdout:3791: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:30:56.476 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 2017 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-10T16:30:56.477 INFO:tasks.rados.rados.0.smithi165.stdout:3790: finishing write tid 3 to smithi16534841-17 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: finishing write tid 4 to smithi16534841-17 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: finishing write tid 5 to smithi16534841-17 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: oid 17 updating version 0 to 1895 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: oid 17 version 1895 is already newer than 1892 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: oid 17 version 1895 is already newer than 1893 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: oid 17 version 1895 is already newer than 1894 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 1895 (ObjNum 1302 snap 423 seq_num 1302) dirty exists 2024-09-10T16:30:56.478 INFO:tasks.rados.rados.0.smithi165.stdout:3790: left oid 17 (ObjNum 1302 snap 423 seq_num 1302) 2024-09-10T16:30:57.417 INFO:tasks.rados.rados.0.smithi165.stdout:3790: done (2 left) 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:3791: done (1 left) 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:3792: done (0 left) 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:3793: rollback oid 3 current snap is 424 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 3 to 419 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:3794: write (excl) oid 33 current snap is 424 2024-09-10T16:30:57.418 INFO:tasks.rados.rados.0.smithi165.stdout:3794: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2024-09-10T16:30:57.425 INFO:tasks.rados.rados.0.smithi165.stdout:3794: writing smithi16534841-33 from 699521 to 1445331 tid 1 2024-09-10T16:30:57.433 INFO:tasks.rados.rados.0.smithi165.stdout:3794: writing smithi16534841-33 from 2181802 to 2830917 tid 2 2024-09-10T16:30:57.439 INFO:tasks.rados.rados.0.smithi165.stdout:3794: writing smithi16534841-33 from 3515689 to 3899521 tid 3 2024-09-10T16:30:57.439 INFO:tasks.rados.rados.0.smithi165.stdout:3795: delete oid 36 current snap is 424 2024-09-10T16:30:57.439 INFO:tasks.rados.rados.0.smithi165.stdout:3793: finishing rollback tid 1 to smithi16534841-3 2024-09-10T16:30:57.439 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 1935 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-10T16:30:57.439 INFO:tasks.rados.rados.0.smithi165.stdout:3794: finishing write tid 1 to smithi16534841-33 2024-09-10T16:30:57.440 INFO:tasks.rados.rados.0.smithi165.stdout:3794: finishing write tid 2 to smithi16534841-33 2024-09-10T16:30:57.443 INFO:tasks.rados.rados.0.smithi165.stdout:3794: finishing write tid 3 to smithi16534841-33 2024-09-10T16:30:57.444 INFO:tasks.rados.rados.0.smithi165.stdout:3793: done (2 left) 2024-09-10T16:30:57.444 INFO:tasks.rados.rados.0.smithi165.stdout:3795: done (1 left) 2024-09-10T16:30:57.444 INFO:tasks.rados.rados.0.smithi165.stdout:3796: delete oid 19 current snap is 424 2024-09-10T16:30:57.444 INFO:tasks.rados.rados.0.smithi165.stdout:3794: finishing write tid 4 to smithi16534841-33 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: finishing write tid 5 to smithi16534841-33 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: oid 33 updating version 0 to 2120 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: oid 33 updating version 2120 to 2121 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: oid 33 version 2121 is already newer than 2118 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: oid 33 version 2121 is already newer than 2119 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 2121 (ObjNum 1303 snap 424 seq_num 1303) dirty exists 2024-09-10T16:30:57.445 INFO:tasks.rados.rados.0.smithi165.stdout:3794: left oid 33 (ObjNum 1303 snap 424 seq_num 1303) 2024-09-10T16:30:57.448 INFO:tasks.rados.rados.0.smithi165.stdout:3794: done (1 left) 2024-09-10T16:30:57.448 INFO:tasks.rados.rados.0.smithi165.stdout:3796: done (0 left) 2024-09-10T16:30:57.448 INFO:tasks.rados.rados.0.smithi165.stdout:3797: snap_create 2024-09-10T16:30:58.422 INFO:tasks.rados.rados.0.smithi165.stdout:3797: done (0 left) 2024-09-10T16:30:58.422 INFO:tasks.rados.rados.0.smithi165.stdout:3798: read oid 6 snap -1 2024-09-10T16:30:58.423 INFO:tasks.rados.rados.0.smithi165.stdout:3798: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:30:58.423 INFO:tasks.rados.rados.0.smithi165.stdout:3799: delete oid 36 current snap is 425 2024-09-10T16:30:58.426 INFO:tasks.rados.rados.0.smithi165.stdout:3799: done (1 left) 2024-09-10T16:30:58.426 INFO:tasks.rados.rados.0.smithi165.stdout:3800: snap_create 2024-09-10T16:30:58.428 INFO:tasks.rados.rados.0.smithi165.stdout:3798: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:30:58.437 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1602511 2024-09-10T16:30:59.427 INFO:tasks.rados.rados.0.smithi165.stdout:3798: done (1 left) 2024-09-10T16:30:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:3800: done (0 left) 2024-09-10T16:30:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:3801: read oid 27 snap -1 2024-09-10T16:30:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:3801: expect deleted 2024-09-10T16:30:59.428 INFO:tasks.rados.rados.0.smithi165.stdout:3802: delete oid 13 current snap is 426 2024-09-10T16:30:59.432 INFO:tasks.rados.rados.0.smithi165.stdout:3801: done (1 left) 2024-09-10T16:30:59.432 INFO:tasks.rados.rados.0.smithi165.stdout:3802: done (0 left) 2024-09-10T16:30:59.432 INFO:tasks.rados.rados.0.smithi165.stdout:3803: write (excl) oid 3 current snap is 426 2024-09-10T16:30:59.432 INFO:tasks.rados.rados.0.smithi165.stdout:3803: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2024-09-10T16:30:59.438 INFO:tasks.rados.rados.0.smithi165.stdout:3803: writing smithi16534841-3 from 783556 to 1221964 tid 1 2024-09-10T16:30:59.443 INFO:tasks.rados.rados.0.smithi165.stdout:3803: writing smithi16534841-3 from 1674212 to 2074323 tid 2 2024-09-10T16:30:59.449 INFO:tasks.rados.rados.0.smithi165.stdout:3803: writing smithi16534841-3 from 2736451 to 3157185 tid 3 2024-09-10T16:30:59.454 INFO:tasks.rados.rados.0.smithi165.stdout:3803: writing smithi16534841-3 from 3882270 to 3983556 tid 4 2024-09-10T16:30:59.454 INFO:tasks.rados.rados.0.smithi165.stdout:3804: write oid 20 current snap is 426 2024-09-10T16:30:59.454 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 1 to smithi16534841-3 2024-09-10T16:30:59.454 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 2 to smithi16534841-3 2024-09-10T16:30:59.455 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 3 to smithi16534841-3 2024-09-10T16:30:59.455 INFO:tasks.rados.rados.0.smithi165.stdout:3804: seq_num 1309 ranges {400363=513657,1332694=660122} 2024-09-10T16:30:59.459 INFO:tasks.rados.rados.0.smithi165.stdout:3804: writing smithi16534841-20 from 400363 to 914020 tid 1 2024-09-10T16:30:59.466 INFO:tasks.rados.rados.0.smithi165.stdout:3804: writing smithi16534841-20 from 1332694 to 1992816 tid 2 2024-09-10T16:30:59.466 INFO:tasks.rados.rados.0.smithi165.stdout:3805: delete oid 32 current snap is 426 2024-09-10T16:30:59.466 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 4 to smithi16534841-3 2024-09-10T16:30:59.466 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 5 to smithi16534841-3 2024-09-10T16:30:59.466 INFO:tasks.rados.rados.0.smithi165.stdout:3803: finishing write tid 6 to smithi16534841-3 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: oid 3 updating version 0 to 2123 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: oid 3 updating version 2123 to 2126 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: oid 3 version 2126 is already newer than 2125 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: oid 3 version 2126 is already newer than 2124 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: oid 3 version 2126 is already newer than 2122 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 3 v 2126 (ObjNum 1308 snap 426 seq_num 1308) dirty exists 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3803: left oid 3 (ObjNum 1308 snap 426 seq_num 1308) 2024-09-10T16:30:59.467 INFO:tasks.rados.rados.0.smithi165.stdout:3804: finishing write tid 1 to smithi16534841-20 2024-09-10T16:30:59.470 INFO:tasks.rados.rados.0.smithi165.stdout:3804: finishing write tid 2 to smithi16534841-20 2024-09-10T16:30:59.470 INFO:tasks.rados.rados.0.smithi165.stdout:3804: finishing write tid 3 to smithi16534841-20 2024-09-10T16:30:59.470 INFO:tasks.rados.rados.0.smithi165.stdout:3804: finishing write tid 4 to smithi16534841-20 2024-09-10T16:30:59.471 INFO:tasks.rados.rados.0.smithi165.stdout:3804: oid 20 updating version 0 to 2130 2024-09-10T16:30:59.471 INFO:tasks.rados.rados.0.smithi165.stdout:3804: oid 20 version 2130 is already newer than 2129 2024-09-10T16:30:59.471 INFO:tasks.rados.rados.0.smithi165.stdout:3804: oid 20 version 2130 is already newer than 2128 2024-09-10T16:30:59.471 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 20 v 2130 (ObjNum 1309 snap 426 seq_num 1309) dirty exists 2024-09-10T16:30:59.471 INFO:tasks.rados.rados.0.smithi165.stdout:3804: left oid 20 (ObjNum 1309 snap 426 seq_num 1309) 2024-09-10T16:30:59.472 INFO:tasks.rados.rados.0.smithi165.stdout:3803: done (2 left) 2024-09-10T16:30:59.472 INFO:tasks.rados.rados.0.smithi165.stdout:3804: done (1 left) 2024-09-10T16:30:59.472 INFO:tasks.rados.rados.0.smithi165.stdout:3805: done (0 left) 2024-09-10T16:30:59.472 INFO:tasks.rados.rados.0.smithi165.stdout:3806: write (excl) oid 45 current snap is 426 2024-09-10T16:30:59.472 INFO:tasks.rados.rados.0.smithi165.stdout:3806: seq_num 1311 ranges {433977=664155,1733305=300672} 2024-09-10T16:30:59.478 INFO:tasks.rados.rados.0.smithi165.stdout:3806: writing smithi16534841-45 from 433977 to 1098132 tid 1 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3806: writing smithi16534841-45 from 1733305 to 2033977 tid 2 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3807: read oid 17 snap -1 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3807: expect (ObjNum 1302 snap 423 seq_num 1302) 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3808: rollback oid 8 current snap is 426 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 415 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3809: copy_from oid 30 from oid 12 current snap is 426 2024-09-10T16:30:59.484 INFO:tasks.rados.rados.0.smithi165.stdout:3810: copy_from oid 14 from oid 38 current snap is 426 2024-09-10T16:30:59.485 INFO:tasks.rados.rados.0.smithi165.stdout:3806: finishing write tid 1 to smithi16534841-45 2024-09-10T16:30:59.485 INFO:tasks.rados.rados.0.smithi165.stdout:3811: write (excl) oid 9 current snap is 426 2024-09-10T16:30:59.485 INFO:tasks.rados.rados.0.smithi165.stdout:3811: seq_num 1312 ranges {450784=739404,1791787=258997} 2024-09-10T16:30:59.490 INFO:tasks.rados.rados.0.smithi165.stdout:3811: writing smithi16534841-9 from 450784 to 1190188 tid 1 2024-09-10T16:30:59.496 INFO:tasks.rados.rados.0.smithi165.stdout:3811: writing smithi16534841-9 from 1791787 to 2050784 tid 2 2024-09-10T16:30:59.496 INFO:tasks.rados.rados.0.smithi165.stdout:3812: write (excl) oid 2 current snap is 426 2024-09-10T16:30:59.496 INFO:tasks.rados.rados.0.smithi165.stdout:3810: finishing copy_from to smithi16534841-14 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3810: got expected ENOENT (src dne) 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3810: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3808: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1911 (ObjNum 129 snap 0 seq_num 1162276688) dirty dne 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3806: finishing write tid 2 to smithi16534841-45 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3806: finishing write tid 3 to smithi16534841-45 2024-09-10T16:30:59.497 INFO:tasks.rados.rados.0.smithi165.stdout:3812: seq_num 1313 ranges {467591=414653,1450269=617322} 2024-09-10T16:30:59.500 INFO:tasks.rados.rados.0.smithi165.stdout:3812: writing smithi16534841-2 from 467591 to 882244 tid 1 2024-09-10T16:30:59.507 INFO:tasks.rados.rados.0.smithi165.stdout:3812: writing smithi16534841-2 from 1450269 to 2067591 tid 2 2024-09-10T16:30:59.507 INFO:tasks.rados.rados.0.smithi165.stdout:3808: done (6 left) 2024-09-10T16:30:59.507 INFO:tasks.rados.rados.0.smithi165.stdout:3810: done (5 left) 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3813: write (excl) oid 35 current snap is 426 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3806: finishing write tid 4 to smithi16534841-45 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3806: oid 45 updating version 0 to 1723 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3806: oid 45 version 1723 is already newer than 1722 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3806: oid 45 updating version 1723 to 1724 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1724 (ObjNum 1311 snap 426 seq_num 1311) dirty exists 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3806: left oid 45 (ObjNum 1311 snap 426 seq_num 1311) 2024-09-10T16:30:59.508 INFO:tasks.rados.rados.0.smithi165.stdout:3813: seq_num 1314 ranges {484398=489902,1392398=411118} 2024-09-10T16:30:59.512 INFO:tasks.rados.rados.0.smithi165.stdout:3813: writing smithi16534841-35 from 484398 to 974300 tid 1 2024-09-10T16:30:59.517 INFO:tasks.rados.rados.0.smithi165.stdout:3813: writing smithi16534841-35 from 1392398 to 1803516 tid 2 2024-09-10T16:30:59.517 INFO:tasks.rados.rados.0.smithi165.stdout:3806: done (5 left) 2024-09-10T16:30:59.517 INFO:tasks.rados.rados.0.smithi165.stdout:3814: read oid 1 snap -1 2024-09-10T16:30:59.518 INFO:tasks.rados.rados.0.smithi165.stdout:3814: expect (ObjNum 1028 snap 313 seq_num 1028) 2024-09-10T16:30:59.518 INFO:tasks.rados.rados.0.smithi165.stdout:3815: snap_create 2024-09-10T16:30:59.518 INFO:tasks.rados.rados.0.smithi165.stdout:3807: expect (ObjNum 1302 snap 423 seq_num 1302) 2024-09-10T16:30:59.530 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1649333 2024-09-10T16:30:59.620 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3809: finishing copy_from racing read to smithi16534841-30 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: finishing write tid 1 to smithi16534841-9 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: finishing write tid 2 to smithi16534841-9 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: finishing write tid 3 to smithi16534841-9 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: finishing write tid 4 to smithi16534841-9 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: oid 9 updating version 0 to 1926 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: oid 9 version 1926 is already newer than 1925 2024-09-10T16:30:59.678 INFO:tasks.rados.rados.0.smithi165.stdout:3811: oid 9 version 1926 is already newer than 1924 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1926 (ObjNum 1312 snap 426 seq_num 1312) dirty exists 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3811: left oid 9 (ObjNum 1312 snap 426 seq_num 1312) 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3809: finishing copy_from to smithi16534841-30 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 1658 (ObjNum 1103 snap 336 seq_num 1103) dirty exists 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: finishing write tid 1 to smithi16534841-2 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: finishing write tid 2 to smithi16534841-2 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: finishing write tid 3 to smithi16534841-2 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: finishing write tid 4 to smithi16534841-2 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: oid 2 updating version 0 to 2118 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: oid 2 version 2118 is already newer than 2116 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: oid 2 version 2118 is already newer than 2117 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 2 v 2118 (ObjNum 1313 snap 426 seq_num 1313) dirty exists 2024-09-10T16:30:59.679 INFO:tasks.rados.rados.0.smithi165.stdout:3812: left oid 2 (ObjNum 1313 snap 426 seq_num 1313) 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: finishing write tid 1 to smithi16534841-35 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: finishing write tid 2 to smithi16534841-35 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: finishing write tid 3 to smithi16534841-35 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: finishing write tid 4 to smithi16534841-35 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: oid 35 updating version 0 to 2022 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: oid 35 updating version 2022 to 2023 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: oid 35 updating version 2023 to 2024 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2024 (ObjNum 1314 snap 426 seq_num 1314) dirty exists 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3813: left oid 35 (ObjNum 1314 snap 426 seq_num 1314) 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:3814: expect (ObjNum 1028 snap 313 seq_num 1028) 2024-09-10T16:30:59.680 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:00.433 INFO:tasks.rados.rados.0.smithi165.stdout:3807: done (6 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3809: done (5 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3811: done (4 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3812: done (3 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3813: done (2 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3814: done (1 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3815: done (0 left) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3816: read oid 15 snap -1 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3816: expect deleted 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3817: read oid 11 snap -1 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3817: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3818: read oid 25 snap -1 2024-09-10T16:31:00.434 INFO:tasks.rados.rados.0.smithi165.stdout:3818: expect (ObjNum 1296 snap 420 seq_num 1296) 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3819: rollback oid 10 current snap is 427 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 389 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3820: read oid 7 snap -1 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3820: expect deleted 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3821: rollback oid 34 current snap is 427 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 417 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3822: write (excl) oid 41 current snap is 427 2024-09-10T16:31:00.435 INFO:tasks.rados.rados.0.smithi165.stdout:3822: seq_num 1315 ranges {501205=565151,1850880=250325} 2024-09-10T16:31:00.439 INFO:tasks.rados.rados.0.smithi165.stdout:3822: writing smithi16534841-41 from 501205 to 1066356 tid 1 2024-09-10T16:31:00.445 INFO:tasks.rados.rados.0.smithi165.stdout:3822: writing smithi16534841-41 from 1850880 to 2101205 tid 2 2024-09-10T16:31:00.446 INFO:tasks.rados.rados.0.smithi165.stdout:3823: rollback oid 12 current snap is 427 2024-09-10T16:31:00.446 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 408 2024-09-10T16:31:00.446 INFO:tasks.rados.rados.0.smithi165.stdout:3817: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:00.446 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3821: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3819: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3819: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3821: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3819: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 1898 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-10T16:31:00.581 INFO:tasks.rados.rados.0.smithi165.stdout:3821: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1894 (ObjNum 1154 snap 362 seq_num 1154) dirty exists 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3822: finishing write tid 1 to smithi16534841-41 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3816: done (7 left) 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3817: done (6 left) 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3819: done (5 left) 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3820: done (4 left) 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3821: done (3 left) 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3824: write oid 14 current snap is 427 2024-09-10T16:31:00.582 INFO:tasks.rados.rados.0.smithi165.stdout:3818: expect (ObjNum 1296 snap 420 seq_num 1296) 2024-09-10T16:31:00.667 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 660792 2024-09-10T16:31:00.794 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 3561631 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: finishing write tid 2 to smithi16534841-41 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3823: finishing rollback tid 0 to smithi16534841-12 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: finishing write tid 3 to smithi16534841-41 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: finishing write tid 4 to smithi16534841-41 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: oid 41 updating version 0 to 2062 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: oid 41 updating version 2062 to 2064 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: oid 41 version 2064 is already newer than 2063 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 41 v 2064 (ObjNum 1315 snap 427 seq_num 1315) dirty exists 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3822: left oid 41 (ObjNum 1315 snap 427 seq_num 1315) 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3823: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:31:00.802 INFO:tasks.rados.rados.0.smithi165.stdout:3823: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:31:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1661 (ObjNum 1230 snap 398 seq_num 1230) dirty exists 2024-09-10T16:31:00.803 INFO:tasks.rados.rados.0.smithi165.stdout:3824: seq_num 1316 ranges {518012=756753,2025715=92297} 2024-09-10T16:31:00.808 INFO:tasks.rados.rados.0.smithi165.stdout:3824: writing smithi16534841-14 from 518012 to 1274765 tid 1 2024-09-10T16:31:00.813 INFO:tasks.rados.rados.0.smithi165.stdout:3824: writing smithi16534841-14 from 2025715 to 2118012 tid 2 2024-09-10T16:31:00.813 INFO:tasks.rados.rados.0.smithi165.stdout:3818: done (3 left) 2024-09-10T16:31:00.813 INFO:tasks.rados.rados.0.smithi165.stdout:3822: done (2 left) 2024-09-10T16:31:00.814 INFO:tasks.rados.rados.0.smithi165.stdout:3823: done (1 left) 2024-09-10T16:31:00.814 INFO:tasks.rados.rados.0.smithi165.stdout:3825: rollback oid 44 current snap is 427 2024-09-10T16:31:00.814 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 359 2024-09-10T16:31:00.814 INFO:tasks.rados.rados.0.smithi165.stdout:3826: snap_remove snap 424 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3826: done (2 left) 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3827: copy_from oid 5 from oid 29 current snap is 427 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3824: finishing write tid 1 to smithi16534841-14 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3825: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3824: finishing write tid 2 to smithi16534841-14 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3825: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3824: finishing write tid 3 to smithi16534841-14 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3824: finishing write tid 4 to smithi16534841-14 2024-09-10T16:31:01.387 INFO:tasks.rados.rados.0.smithi165.stdout:3824: oid 14 updating version 0 to 1928 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3824: oid 14 version 1928 is already newer than 1927 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3824: oid 14 updating version 1928 to 1929 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1929 (ObjNum 1316 snap 427 seq_num 1316) dirty exists 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3824: left oid 14 (ObjNum 1316 snap 427 seq_num 1316) 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3825: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1849 (ObjNum 1114 snap 341 seq_num 1114) dirty exists 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3824: done (2 left) 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3825: done (1 left) 2024-09-10T16:31:01.388 INFO:tasks.rados.rados.0.smithi165.stdout:3828: snap_remove snap 382 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3828: done (1 left) 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3829: rollback oid 24 current snap is 427 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 24 to 426 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3827: finishing copy_from to smithi16534841-5 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3827: got expected ENOENT (src dne) 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3827: finishing copy_from racing read to smithi16534841-5 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3827: done (1 left) 2024-09-10T16:31:02.389 INFO:tasks.rados.rados.0.smithi165.stdout:3830: snap_create 2024-09-10T16:31:02.401 INFO:tasks.rados.rados.0.smithi165.stdout:3829: finishing rollback tid 0 to smithi16534841-24 2024-09-10T16:31:02.401 INFO:tasks.rados.rados.0.smithi165.stdout:3829: finishing rollback tid 1 to smithi16534841-24 2024-09-10T16:31:02.401 INFO:tasks.rados.rados.0.smithi165.stdout:3829: finishing rollback tid 2 to smithi16534841-24 2024-09-10T16:31:02.401 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 24 v 2124 (ObjNum 1254 snap 405 seq_num 1254) dirty exists 2024-09-10T16:31:03.389 INFO:tasks.rados.rados.0.smithi165.stdout:3829: done (1 left) 2024-09-10T16:31:03.389 INFO:tasks.rados.rados.0.smithi165.stdout:3830: done (0 left) 2024-09-10T16:31:03.390 INFO:tasks.rados.rados.0.smithi165.stdout:3831: snap_remove snap 359 2024-09-10T16:31:04.394 INFO:tasks.rados.rados.0.smithi165.stdout:3831: done (0 left) 2024-09-10T16:31:04.394 INFO:tasks.rados.rados.0.smithi165.stdout:3832: write (excl) oid 49 current snap is 428 2024-09-10T16:31:04.394 INFO:tasks.rados.rados.0.smithi165.stdout:3832: seq_num 1317 ranges {534819=432002,1684197=450622} 2024-09-10T16:31:04.399 INFO:tasks.rados.rados.0.smithi165.stdout:3832: writing smithi16534841-49 from 534819 to 966821 tid 1 2024-09-10T16:31:04.406 INFO:tasks.rados.rados.0.smithi165.stdout:3832: writing smithi16534841-49 from 1684197 to 2134819 tid 2 2024-09-10T16:31:04.406 INFO:tasks.rados.rados.0.smithi165.stdout:3833: rollback oid 1 current snap is 428 2024-09-10T16:31:04.407 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 1 to 387 2024-09-10T16:31:04.407 INFO:tasks.rados.rados.0.smithi165.stdout:3834: write oid 16 current snap is 428 2024-09-10T16:31:04.407 INFO:tasks.rados.rados.0.smithi165.stdout:3834: seq_num 1318 ranges {551626=507251,1626326=418456} 2024-09-10T16:31:04.412 INFO:tasks.rados.rados.0.smithi165.stdout:3834: writing smithi16534841-16 from 551626 to 1058877 tid 1 2024-09-10T16:31:04.418 INFO:tasks.rados.rados.0.smithi165.stdout:3834: writing smithi16534841-16 from 1626326 to 2044782 tid 2 2024-09-10T16:31:04.418 INFO:tasks.rados.rados.0.smithi165.stdout:3835: delete oid 50 current snap is 428 2024-09-10T16:31:04.418 INFO:tasks.rados.rados.0.smithi165.stdout:3832: finishing write tid 1 to smithi16534841-49 2024-09-10T16:31:04.418 INFO:tasks.rados.rados.0.smithi165.stdout:3832: finishing write tid 2 to smithi16534841-49 2024-09-10T16:31:04.418 INFO:tasks.rados.rados.0.smithi165.stdout:3832: finishing write tid 3 to smithi16534841-49 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3832: finishing write tid 4 to smithi16534841-49 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3832: oid 49 updating version 0 to 1901 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3832: oid 49 version 1901 is already newer than 1899 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3832: oid 49 version 1901 is already newer than 1900 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 1901 (ObjNum 1317 snap 428 seq_num 1317) dirty exists 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3832: left oid 49 (ObjNum 1317 snap 428 seq_num 1317) 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3833: finishing rollback tid 0 to smithi16534841-1 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3833: finishing rollback tid 1 to smithi16534841-1 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:3833: finishing rollback tid 2 to smithi16534841-1 2024-09-10T16:31:04.419 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 1 v 2067 (ObjNum 1028 snap 313 seq_num 1028) dirty exists 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3832: done (3 left) 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3833: done (2 left) 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3835: done (1 left) 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3836: read oid 8 snap 365 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3836: expect (ObjNum 1146 snap 355 seq_num 1146) 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3837: write (excl) oid 5 current snap is 428 2024-09-10T16:31:04.422 INFO:tasks.rados.rados.0.smithi165.stdout:3837: seq_num 1320 ranges {585240=657749,1743290=422125} 2024-09-10T16:31:04.429 INFO:tasks.rados.rados.0.smithi165.stdout:3837: writing smithi16534841-5 from 585240 to 1242989 tid 1 2024-09-10T16:31:04.434 INFO:tasks.rados.rados.0.smithi165.stdout:3837: writing smithi16534841-5 from 1743290 to 2165415 tid 2 2024-09-10T16:31:04.434 INFO:tasks.rados.rados.0.smithi165.stdout:3838: write oid 38 current snap is 428 2024-09-10T16:31:04.434 INFO:tasks.rados.rados.0.smithi165.stdout:3834: finishing write tid 1 to smithi16534841-16 2024-09-10T16:31:04.434 INFO:tasks.rados.rados.0.smithi165.stdout:3838: seq_num 1321 ranges {602047=732998,1801772=400275} 2024-09-10T16:31:04.441 INFO:tasks.rados.rados.0.smithi165.stdout:3838: writing smithi16534841-38 from 602047 to 1335045 tid 1 2024-09-10T16:31:04.446 INFO:tasks.rados.rados.0.smithi165.stdout:3838: writing smithi16534841-38 from 1801772 to 2202047 tid 2 2024-09-10T16:31:04.446 INFO:tasks.rados.rados.0.smithi165.stdout:3839: read oid 19 snap -1 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3839: expect deleted 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3840: copy_from oid 31 from oid 13 current snap is 428 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: finishing write tid 2 to smithi16534841-16 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: finishing write tid 3 to smithi16534841-16 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: finishing write tid 4 to smithi16534841-16 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: oid 16 updating version 0 to 2510 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: oid 16 updating version 2510 to 2511 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: oid 16 version 2511 is already newer than 2509 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2511 (ObjNum 1318 snap 428 seq_num 1318) dirty exists 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3834: left oid 16 (ObjNum 1318 snap 428 seq_num 1318) 2024-09-10T16:31:04.447 INFO:tasks.rados.rados.0.smithi165.stdout:3836: expect (ObjNum 1146 snap 355 seq_num 1146) 2024-09-10T16:31:04.628 INFO:tasks.rados.rados.0.smithi165.stdout:3840: finishing copy_from to smithi16534841-31 2024-09-10T16:31:04.628 INFO:tasks.rados.rados.0.smithi165.stdout:3840: got expected ENOENT (src dne) 2024-09-10T16:31:04.628 INFO:tasks.rados.rados.0.smithi165.stdout:3840: finishing copy_from racing read to smithi16534841-31 2024-09-10T16:31:04.628 INFO:tasks.rados.rados.0.smithi165.stdout:3837: finishing write tid 1 to smithi16534841-5 2024-09-10T16:31:04.628 INFO:tasks.rados.rados.0.smithi165.stdout:3837: finishing write tid 2 to smithi16534841-5 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: finishing write tid 3 to smithi16534841-5 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: finishing write tid 4 to smithi16534841-5 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: oid 5 updating version 0 to 1895 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: oid 5 version 1895 is already newer than 1893 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: oid 5 version 1895 is already newer than 1894 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 5 v 1895 (ObjNum 1320 snap 428 seq_num 1320) dirty exists 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: left oid 5 (ObjNum 1320 snap 428 seq_num 1320) 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3838: finishing write tid 1 to smithi16534841-38 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3838: finishing write tid 2 to smithi16534841-38 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3838: finishing write tid 3 to smithi16534841-38 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3834: done (5 left) 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3836: done (4 left) 2024-09-10T16:31:04.629 INFO:tasks.rados.rados.0.smithi165.stdout:3837: done (3 left) 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3839: done (2 left) 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3840: done (1 left) 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3841: snap_create 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3838: finishing write tid 4 to smithi16534841-38 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3838: oid 38 updating version 0 to 1701 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3838: oid 38 updating version 1701 to 1702 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3838: oid 38 updating version 1702 to 1703 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 1703 (ObjNum 1321 snap 428 seq_num 1321) dirty exists 2024-09-10T16:31:04.630 INFO:tasks.rados.rados.0.smithi165.stdout:3838: left oid 38 (ObjNum 1321 snap 428 seq_num 1321) 2024-09-10T16:31:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3838: done (1 left) 2024-09-10T16:31:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3841: done (0 left) 2024-09-10T16:31:05.410 INFO:tasks.rados.rados.0.smithi165.stdout:3842: snap_remove snap 373 2024-09-10T16:31:06.074 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-10T16:31:06.075 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T16:31:06.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T16:31:06.078 INFO:tasks.ceph.mon.b:Started 2024-09-10T16:31:06.078 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:31:06.078 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.186+0000 7f8f8a582640 10 monclient: get_monmap_and_config 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.186+0000 7f8f8a582640 10 monclient: build_initial_monmap 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.186+0000 7f8f8a582640 10 monclient: monmap: 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:06.187744+0000 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:06.187744+0000 2024-09-10T16:31:06.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:06.189 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.187+0000 7f8f8a582640 1 Processor -- start 2024-09-10T16:31:06.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.187+0000 7f8f8a582640 1 -- start start 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.187+0000 7f8f8a582640 10 monclient: init 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-c con 0x7f8f841329f0 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f840a2f10 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f8409d520 0x7f8f8413dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-a con 0x7f8f8409d520 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f840a0730 con 0x7f8f840a2f10 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f8f840a21a0 con 0x7f8f8409d520 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:06.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8f840a2480 con 0x7f8f841329f0 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f8a582640 10 monclient(hunting): authenticate will time out at 2244.675361s 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f8409d520 0x7f8f8413dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8f8409d520 0x7f8f8413dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56392/0 (socket says 172.21.15.164:56392) 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 -- 172.21.15.164:0/2842640321 learned_addr learned my addr 172.21.15.164:0/2842640321 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 1 --2- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): get_auth_request con 0x7f8f8409d520 auth_method 0 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 10 monclient(hunting): get_auth_request con 0x7f8f841329f0 auth_method 0 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:06.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 1 -- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 msgr2=0x7f8f84132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 1 -- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 msgr2=0x7f8f84132dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 1 --2- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f83fff640 1 --2- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_done global_id 254382 payload 274 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 -- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 msgr2=0x7f8f84132dd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 --2- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 -- 172.21.15.164:0/2842640321 >> v1:172.21.15.165:6789/0 conn(0x7f8f840a2f10 legacy=0x7f8f8409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient: _finish_hunting 0 2024-09-10T16:31:06.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 monclient: found mon.noname-a 2024-09-10T16:31:06.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:06.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.188+0000 7f8f88b1f640 1 -- 172.21.15.164:0/2842640321 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f840a1060 con 0x7f8f8409d520 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f88b1f640 1 -- 172.21.15.164:0/2842640321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f8409d520 msgr2=0x7f8f8413dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f88b1f640 1 --2- 172.21.15.164:0/2842640321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f8409d520 0x7f8f8413dd90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f840a1470 tx=0x7f8f7c02a540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6ed374d43e16dbb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 1 -- 172.21.15.164:0/2842640321 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c036070 con 0x7f8f8409d520 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: dump: 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.202 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: _finish_auth 0 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 1 -- 172.21.15.164:0/2842640321 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f7c007800 con 0x7f8f8409d520 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f8a582640 5 monclient: authenticate success, global_id 254382 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f8a582640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 1 -- 172.21.15.164:0/2842640321 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c007bd0 con 0x7f8f8409d520 2024-09-10T16:31:06.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f8a582640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f8a582640 4 set_mon_vals no callback set 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.191+0000 7f8f82ffd640 10 monclient: dump: 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:06.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 10 monclient: shutdown 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 -- 172.21.15.164:0/2842640321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f8409d520 msgr2=0x7f8f8413dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 --2- 172.21.15.164:0/2842640321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f8409d520 0x7f8f8413dd90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f8f840a1470 tx=0x7f8f7c02a540 comp rx=0 tx=0).stop 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 -- 172.21.15.164:0/2842640321 shutdown_connections 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 --2- 172.21.15.164:0/2842640321 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f8409d520 0x7f8f8413dd90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 --2- 172.21.15.164:0/2842640321 >> v2:172.21.15.165:3300/0 conn(0x7f8f841329f0 0x7f8f84132dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 -- 172.21.15.164:0/2842640321 >> 172.21.15.164:0/2842640321 conn(0x7f8f840a7320 msgr2=0x7f8f840a7730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 -- 172.21.15.164:0/2842640321 shutdown_connections 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.192+0000 7f8f8a582640 1 -- 172.21.15.164:0/2842640321 wait complete. 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.193+0000 7f8f8a582640 10 monclient: build_initial_monmap 2024-09-10T16:31:06.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.193+0000 7f8f8a582640 10 monclient: monmap: 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.193+0000 7f8f8a582640 1 Processor -- start 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.193+0000 7f8f8a582640 1 -- start start 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.193+0000 7f8f8a582640 10 monclient: init 2024-09-10T16:31:06.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f8409d520 0x7f8f841caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-b con 0x7f8f8409d520 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 0x7f8f84131a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-a con 0x7f8f840a2f10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): picked mon.noname-c con 0x7f8f841329f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84133490 con 0x7f8f840a2f10 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8f84133310 con 0x7f8f8409d520 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8f8409d900 con 0x7f8f841329f0 2024-09-10T16:31:06.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f8a582640 10 monclient(hunting): authenticate will time out at 2244.681072s 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 0x7f8f84131a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57666/0 (socket says 172.21.15.164:57666) 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 1 -- 172.21.15.164:0/166954953 learned_addr learned my addr 172.21.15.164:0/166954953 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): get_auth_request con 0x7f8f841329f0 auth_method 0 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): get_auth_request con 0x7f8f840a2f10 auth_method 0 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:06.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f83fff640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f8409d520 0x7f8f841caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f83fff640 10 monclient(hunting): get_auth_request con 0x7f8f8409d520 auth_method 0 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f83fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f83fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f83fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient(hunting): handle_auth_done global_id 254385 payload 824 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 -- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 msgr2=0x7f8f84072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:06.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 -- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f8409d520 msgr2=0x7f8f841caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f8409d520 0x7f8f841caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient: _finish_hunting 0 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 monclient: found mon.noname-a 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f837fe640 1 -- 172.21.15.164:0/166954953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f84136370 con 0x7f8f840a2f10 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.194+0000 7f8f88b1f640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f837fe640 1 -- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 msgr2=0x7f8f84131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f837fe640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 0x7f8f84131a10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c440 tx=0x7f8f7800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:06.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f780119a0 con 0x7f8f840a2f10 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: dump: 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: _finish_auth 0 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f78017070 con 0x7f8f840a2f10 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f8a582640 5 monclient: authenticate success, global_id 254385 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f8a582640 10 monclient: _renew_subs 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f8a582640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f8409e0f0 con 0x7f8f840a2f10 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f78011d80 con 0x7f8f840a2f10 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f80ff9640 10 monclient: dump: 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:06.212 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 10 monclient: _renew_subs 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.196+0000 7f8f827fc640 4 set_mon_vals no callback set 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f8411eb30 con 0x7f8f840a2f10 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8f7801c460 con 0x7f8f840a2f10 2024-09-10T16:31:06.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.197+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f840a48e0 con 0x7f8f840a2f10 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.198+0000 7f8f80ff9640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 0x7f8f700406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.198+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(861..861 src has 772..861) ==== 4771+0+0 (secure 0 0 0) 0x7f8f78052ad0 con 0x7f8f840a2f10 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.198+0000 7f8f83fff640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 0x7f8f700406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.198+0000 7f8f83fff640 10 monclient: get_auth_request con 0x7f8f7003e210 auth_method 0 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.200+0000 7f8f83fff640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 0x7f8f700406d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8f74006fd0 tx=0x7f8f74008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:06.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.202+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8f840a48e0 con 0x7f8f840a2f10 2024-09-10T16:31:06.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.202+0000 7f8f80ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:06.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.202+0000 7f8f80ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:06.417 INFO:tasks.rados.rados.0.smithi165.stdout:3842: done (0 left) 2024-09-10T16:31:06.417 INFO:tasks.rados.rados.0.smithi165.stdout:3843: read oid 13 snap -1 2024-09-10T16:31:06.417 INFO:tasks.rados.rados.0.smithi165.stdout:3843: expect deleted 2024-09-10T16:31:06.418 INFO:tasks.rados.rados.0.smithi165.stdout:3844: snap_remove snap 421 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.454+0000 7f8f8a582640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.454+0000 7f8f8a582640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.454+0000 7f8f8a582640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.454+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8f8411d950 con 0x7f8f840a2f10 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.455+0000 7f8f80ff9640 1 -- 172.21.15.164:0/166954953 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f8f780514d0 con 0x7f8f840a2f10 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.455+0000 7f8f80ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:06.456 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.455+0000 7f8f80ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:06.457 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":242,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 msgr2=0x7f8f700406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 0x7f8f700406d0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8f74006fd0 tx=0x7f8f74008040 comp rx=0 tx=0).stop 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 10 monclient: shutdown 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 msgr2=0x7f8f84131a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 0x7f8f84131a10 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8f7800c440 tx=0x7f8f7800c900 comp rx=0 tx=0).stop 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 shutdown_connections 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8f7003e210 0x7f8f700406d0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8f841329f0 0x7f8f84072390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8f840a2f10 0x7f8f84131a10 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.459 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 --2- 172.21.15.164:0/166954953 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8f8409d520 0x7f8f841caeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:06.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 >> 172.21.15.164:0/166954953 conn(0x7f8f840a7320 msgr2=0x7f8f84134a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:06.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 shutdown_connections 2024-09-10T16:31:06.460 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:06.457+0000 7f8f8a582640 1 -- 172.21.15.164:0/166954953 wait complete. 2024-09-10T16:31:07.790 INFO:tasks.rados.rados.0.smithi165.stdout:3844: done (1 left) 2024-09-10T16:31:07.790 INFO:tasks.rados.rados.0.smithi165.stdout:3845: write (excl) oid 19 current snap is 429 2024-09-10T16:31:07.790 INFO:tasks.rados.rados.0.smithi165.stdout:3845: seq_num 1322 ranges {618854=408247,1743901=425794} 2024-09-10T16:31:07.794 INFO:tasks.rados.rados.0.smithi165.stdout:3845: writing smithi16534841-19 from 618854 to 1027101 tid 1 2024-09-10T16:31:07.801 INFO:tasks.rados.rados.0.smithi165.stdout:3845: writing smithi16534841-19 from 1743901 to 2169695 tid 2 2024-09-10T16:31:07.801 INFO:tasks.rados.rados.0.smithi165.stdout:3846: write (excl) oid 44 current snap is 429 2024-09-10T16:31:07.801 INFO:tasks.rados.rados.0.smithi165.stdout:3846: seq_num 1323 ranges {635661=599849,1918736=316925} 2024-09-10T16:31:07.808 INFO:tasks.rados.rados.0.smithi165.stdout:3846: writing smithi16534841-44 from 635661 to 1235510 tid 1 2024-09-10T16:31:07.815 INFO:tasks.rados.rados.0.smithi165.stdout:3846: writing smithi16534841-44 from 1918736 to 2235661 tid 2 2024-09-10T16:31:07.815 INFO:tasks.rados.rados.0.smithi165.stdout:3847: write (excl) oid 7 current snap is 429 2024-09-10T16:31:07.815 INFO:tasks.rados.rados.0.smithi165.stdout:3847: seq_num 1324 ranges {652468=675098,1977218=275250} 2024-09-10T16:31:07.822 INFO:tasks.rados.rados.0.smithi165.stdout:3847: writing smithi16534841-7 from 652468 to 1327566 tid 1 2024-09-10T16:31:07.828 INFO:tasks.rados.rados.0.smithi165.stdout:3847: writing smithi16534841-7 from 1977218 to 2252468 tid 2 2024-09-10T16:31:07.828 INFO:tasks.rados.rados.0.smithi165.stdout:3848: snap_remove snap 422 2024-09-10T16:31:08.429 INFO:tasks.rados.rados.0.smithi165.stdout:3848: done (4 left) 2024-09-10T16:31:08.429 INFO:tasks.rados.rados.0.smithi165.stdout:3849: snap_remove snap 387 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3849: done (4 left) 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3850: delete oid 33 current snap is 429 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3845: finishing write tid 1 to smithi16534841-19 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3845: finishing write tid 2 to smithi16534841-19 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3845: finishing write tid 3 to smithi16534841-19 2024-09-10T16:31:09.432 INFO:tasks.rados.rados.0.smithi165.stdout:3845: finishing write tid 4 to smithi16534841-19 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3845: oid 19 updating version 0 to 2529 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3845: oid 19 version 2529 is already newer than 2528 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3845: oid 19 updating version 2529 to 2530 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2530 (ObjNum 1322 snap 429 seq_num 1322) dirty exists 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3845: left oid 19 (ObjNum 1322 snap 429 seq_num 1322) 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: finishing write tid 1 to smithi16534841-44 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: finishing write tid 2 to smithi16534841-44 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: finishing write tid 3 to smithi16534841-44 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: finishing write tid 4 to smithi16534841-44 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: oid 44 updating version 0 to 1852 2024-09-10T16:31:09.433 INFO:tasks.rados.rados.0.smithi165.stdout:3846: oid 44 version 1852 is already newer than 1851 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3846: oid 44 version 1852 is already newer than 1850 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1852 (ObjNum 1323 snap 429 seq_num 1323) dirty exists 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3846: left oid 44 (ObjNum 1323 snap 429 seq_num 1323) 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: finishing write tid 1 to smithi16534841-7 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: finishing write tid 2 to smithi16534841-7 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: finishing write tid 3 to smithi16534841-7 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: finishing write tid 4 to smithi16534841-7 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: oid 7 updating version 0 to 2531 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: oid 7 updating version 2531 to 2532 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: oid 7 updating version 2532 to 2533 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2533 (ObjNum 1324 snap 429 seq_num 1324) dirty exists 2024-09-10T16:31:09.434 INFO:tasks.rados.rados.0.smithi165.stdout:3847: left oid 7 (ObjNum 1324 snap 429 seq_num 1324) 2024-09-10T16:31:09.441 INFO:tasks.rados.rados.0.smithi165.stdout:3843: done (4 left) 2024-09-10T16:31:09.441 INFO:tasks.rados.rados.0.smithi165.stdout:3845: done (3 left) 2024-09-10T16:31:09.441 INFO:tasks.rados.rados.0.smithi165.stdout:3846: done (2 left) 2024-09-10T16:31:09.442 INFO:tasks.rados.rados.0.smithi165.stdout:3847: done (1 left) 2024-09-10T16:31:09.442 INFO:tasks.rados.rados.0.smithi165.stdout:3850: done (0 left) 2024-09-10T16:31:09.442 INFO:tasks.rados.rados.0.smithi165.stdout:3851: copy_from oid 17 from oid 22 current snap is 429 2024-09-10T16:31:09.442 INFO:tasks.rados.rados.0.smithi165.stdout:3852: snap_remove snap 311 2024-09-10T16:31:09.470 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:09.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 10 monclient: get_monmap_and_config 2024-09-10T16:31:09.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 10 monclient: build_initial_monmap 2024-09-10T16:31:09.572 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 10 monclient: monmap: 2024-09-10T16:31:09.572 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:09.572 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:09.571912+0000 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:09.571912+0000 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:09.576 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:09.577 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:09.577 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 1 Processor -- start 2024-09-10T16:31:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 1 -- start start 2024-09-10T16:31:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.571+0000 7fd9e7a7f640 10 monclient: init 2024-09-10T16:31:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:09.585 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fd9e006d160 0x7fd9e0134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e006d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd9e006d540 0x7fd9e00a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e006d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9e00a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9e009cb90 con 0x7fd9e00a4c20 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e009ce70 con 0x7fd9e006d540 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fd9e009d150 con 0x7fd9e006d160 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e7a7f640 10 monclient(hunting): authenticate will time out at 2248.059380s 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fd9e006d540 0x7fd9e00a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.586 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e601c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fd9e00a4c20 0x7fd9e0137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57090/0 (socket says 172.21.15.164:57090) 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e601c640 1 -- 172.21.15.164:0/4232358269 learned_addr learned my addr 172.21.15.164:0/4232358269 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): get_auth_request con 0x7fd9e006d540 auth_method 0 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e581b640 1 --2- 172.21.15.164:0/4232358269 >> v2:172.21.15.165:3300/0 conn(0x7fd9e006d160 0x7fd9e0134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e581b640 10 monclient(hunting): get_auth_request con 0x7fd9e006d160 auth_method 0 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e581b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e581b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:09.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e581b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient(hunting): handle_auth_done global_id 254388 payload 274 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 -- 172.21.15.164:0/4232358269 >> v2:172.21.15.165:3300/0 conn(0x7fd9e006d160 msgr2=0x7fd9e0134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 --2- 172.21.15.164:0/4232358269 >> v2:172.21.15.165:3300/0 conn(0x7fd9e006d160 0x7fd9e0134e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 -- 172.21.15.164:0/4232358269 >> v1:172.21.15.164:6789/0 conn(0x7fd9e00a4c20 legacy=0x7fd9e0137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient: _finish_hunting 0 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 monclient: found mon.noname-a 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.572+0000 7fd9e501a640 1 -- 172.21.15.164:0/4232358269 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e0135340 con 0x7fd9e006d540 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e501a640 1 -- 172.21.15.164:0/4232358269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e006d540 msgr2=0x7fd9e00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:09.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e501a640 1 --2- 172.21.15.164:0/4232358269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e006d540 0x7fd9e00a46e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd9e006d130 tx=0x7fd9d002a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86228dd674ce908 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 1 -- 172.21.15.164:0/4232358269 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0037070 con 0x7fd9e006d540 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: dump: 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:09.589 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: _finish_auth 0 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: _check_auth_tickets 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 1 -- 172.21.15.164:0/4232358269 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d0032440 con 0x7fd9e006d540 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e7a7f640 5 monclient: authenticate success, global_id 254388 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e7a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e7a7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e7a7f640 4 set_mon_vals no callback set 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 1 -- 172.21.15.164:0/4232358269 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0032810 con 0x7fd9e006d540 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:09.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.575+0000 7fd9e4819640 10 monclient: dump: 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:09.591 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 10 monclient: shutdown 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/4232358269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e006d540 msgr2=0x7fd9e00a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/4232358269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e006d540 0x7fd9e00a46e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd9e006d130 tx=0x7fd9d002a000 comp rx=0 tx=0).stop 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/4232358269 shutdown_connections 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/4232358269 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e006d540 0x7fd9e00a46e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/4232358269 >> v2:172.21.15.165:3300/0 conn(0x7fd9e006d160 0x7fd9e0134e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/4232358269 >> 172.21.15.164:0/4232358269 conn(0x7fd9e00aacb0 msgr2=0x7fd9e00ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/4232358269 shutdown_connections 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.576+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/4232358269 wait complete. 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient: build_initial_monmap 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient: monmap: 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.593 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 Processor -- start 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 -- start start 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient: init 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9e006d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 0x7fd9e01cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9e006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9e00a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e0138780 con 0x7fd9e00a4c20 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fd9e0138600 con 0x7fd9e006d160 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fd9e0138900 con 0x7fd9e006d540 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.577+0000 7fd9e7a7f640 10 monclient(hunting): authenticate will time out at 2248.064926s 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 0x7fd9e01cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.595 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56422/0 (socket says 172.21.15.164:56422) 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 -- 172.21.15.164:0/3398632568 learned_addr learned my addr 172.21.15.164:0/3398632568 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): get_auth_request con 0x7fd9e00a4c20 auth_method 0 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): get_auth_request con 0x7fd9e006d540 auth_method 0 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:09.596 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 10 monclient(hunting): get_auth_request con 0x7fd9e006d160 auth_method 0 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient(hunting): handle_auth_done global_id 254391 payload 824 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 msgr2=0x7fd9e01cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 0x7fd9e01cd770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 msgr2=0x7fd9e00a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:31:09.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 msgr2=0x7fd9e00a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 msgr2=0x7fd9e00a78d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e581b640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient: _finish_hunting 0 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 monclient: found mon.noname-a 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e601c640 1 -- 172.21.15.164:0/3398632568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9e013b870 con 0x7fd9e00a4c20 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.578+0000 7fd9e501a640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 0x7fd9e01cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e601c640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 msgr2=0x7fd9e01cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e601c640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800cb20 tx=0x7fd9d800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8012400 con 0x7fd9e00a4c20 2024-09-10T16:31:09.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: dump: 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:09.599 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9d80125c0 con 0x7fd9e00a4c20 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9d80129b0 con 0x7fd9e00a4c20 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 5 monclient: authenticate success, global_id 254391 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 10 monclient: _renew_subs 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9e01ce1f0 con 0x7fd9e00a4c20 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d67fc640 10 monclient: dump: 2024-09-10T16:31:09.600 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9d7fff640 4 set_mon_vals no callback set 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 10 monclient: _renew_subs 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.580+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9e0136450 con 0x7fd9e00a4c20 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.581+0000 7fd9e7a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.581+0000 7fd9e7a7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.581+0000 7fd9e7a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.581+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9e00a4900 con 0x7fd9e00a4c20 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.581+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fd9d8012b90 con 0x7fd9e00a4c20 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.582+0000 7fd9d67fc640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 0x7fd9c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.582+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(865..865 src has 772..865) ==== 4847+0+0 (secure 0 0 0) 0x7fd9d8053380 con 0x7fd9e00a4c20 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.582+0000 7fd9e581b640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 0x7fd9c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:09.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.582+0000 7fd9e581b640 10 monclient: get_auth_request con 0x7fd9c403e210 auth_method 0 2024-09-10T16:31:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.584+0000 7fd9e581b640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 0x7fd9c40406d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc006fd0 tx=0x7fd9dc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.586+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fd9d8051d40 con 0x7fd9e00a4c20 2024-09-10T16:31:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.586+0000 7fd9d67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:09.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.586+0000 7fd9d67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:09.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.829+0000 7fd9e7a7f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:09.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.829+0000 7fd9e7a7f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:09.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.829+0000 7fd9e7a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:09.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.829+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd9e0134e00 con 0x7fd9e00a4c20 2024-09-10T16:31:09.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.830+0000 7fd9d67fc640 1 -- 172.21.15.164:0/3398632568 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7fd9d8053080 con 0x7fd9e00a4c20 2024-09-10T16:31:09.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.830+0000 7fd9d67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:09.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.830+0000 7fd9d67fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:09.832 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":242,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:09.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 msgr2=0x7fd9c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:09.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 0x7fd9c40406d0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc006fd0 tx=0x7fd9dc008040 comp rx=0 tx=0).stop 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 10 monclient: shutdown 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 msgr2=0x7fd9e01cdcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800cb20 tx=0x7fd9d800a060 comp rx=0 tx=0).stop 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 shutdown_connections 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fd9c403e210 0x7fd9c40406d0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fd9e00a4c20 0x7fd9e01cdcb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fd9e006d540 0x7fd9e01cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 --2- 172.21.15.164:0/3398632568 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fd9e006d160 0x7fd9e00a78d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:09.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 >> 172.21.15.164:0/3398632568 conn(0x7fd9e00aacb0 msgr2=0x7fd9e0132780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:09.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 shutdown_connections 2024-09-10T16:31:09.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:09.832+0000 7fd9e7a7f640 1 -- 172.21.15.164:0/3398632568 wait complete. 2024-09-10T16:31:10.449 INFO:tasks.rados.rados.0.smithi165.stdout:3852: done (1 left) 2024-09-10T16:31:10.450 INFO:tasks.rados.rados.0.smithi165.stdout:3853: snap_create 2024-09-10T16:31:10.450 INFO:tasks.rados.rados.0.smithi165.stdout:3851: finishing copy_from racing read to smithi16534841-17 2024-09-10T16:31:10.450 INFO:tasks.rados.rados.0.smithi165.stdout:3851: finishing copy_from to smithi16534841-17 2024-09-10T16:31:10.450 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 2021 (ObjNum 1289 snap 418 seq_num 1289) dirty exists 2024-09-10T16:31:11.389 INFO:tasks.rados.rados.0.smithi165.stdout:3851: done (1 left) 2024-09-10T16:31:11.390 INFO:tasks.rados.rados.0.smithi165.stdout:3853: done (0 left) 2024-09-10T16:31:11.390 INFO:tasks.rados.rados.0.smithi165.stdout:3854: write oid 27 current snap is 430 2024-09-10T16:31:11.390 INFO:tasks.rados.rados.0.smithi165.stdout:3854: seq_num 1326 ranges {686082=425596,1577829=433132} 2024-09-10T16:31:11.395 INFO:tasks.rados.rados.0.smithi165.stdout:3854: writing smithi16534841-27 from 686082 to 1111678 tid 1 2024-09-10T16:31:11.400 INFO:tasks.rados.rados.0.smithi165.stdout:3854: writing smithi16534841-27 from 1577829 to 2010961 tid 2 2024-09-10T16:31:11.401 INFO:tasks.rados.rados.0.smithi165.stdout:3855: snap_create 2024-09-10T16:31:11.404 INFO:tasks.rados.rados.0.smithi165.stdout:3854: finishing write tid 1 to smithi16534841-27 2024-09-10T16:31:11.406 INFO:tasks.rados.rados.0.smithi165.stdout:3854: finishing write tid 2 to smithi16534841-27 2024-09-10T16:31:11.406 INFO:tasks.rados.rados.0.smithi165.stdout:3854: finishing write tid 3 to smithi16534841-27 2024-09-10T16:31:11.406 INFO:tasks.rados.rados.0.smithi165.stdout:3854: finishing write tid 4 to smithi16534841-27 2024-09-10T16:31:11.406 INFO:tasks.rados.rados.0.smithi165.stdout:3854: oid 27 updating version 0 to 2171 2024-09-10T16:31:11.406 INFO:tasks.rados.rados.0.smithi165.stdout:3854: oid 27 version 2171 is already newer than 2169 2024-09-10T16:31:11.407 INFO:tasks.rados.rados.0.smithi165.stdout:3854: oid 27 version 2171 is already newer than 2170 2024-09-10T16:31:11.407 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 2171 (ObjNum 1326 snap 430 seq_num 1326) dirty exists 2024-09-10T16:31:11.407 INFO:tasks.rados.rados.0.smithi165.stdout:3854: left oid 27 (ObjNum 1326 snap 430 seq_num 1326) 2024-09-10T16:31:12.611 INFO:tasks.rados.rados.0.smithi165.stdout:3854: done (1 left) 2024-09-10T16:31:12.611 INFO:tasks.rados.rados.0.smithi165.stdout:3855: done (0 left) 2024-09-10T16:31:12.611 INFO:tasks.rados.rados.0.smithi165.stdout:3856: snap_remove snap 410 2024-09-10T16:31:12.845 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.952+0000 7fc6d0eb8640 10 monclient: get_monmap_and_config 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.952+0000 7fc6d0eb8640 10 monclient: build_initial_monmap 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.952+0000 7fc6d0eb8640 10 monclient: monmap: 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:12.953847+0000 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:12.953847+0000 2024-09-10T16:31:12.954 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:12.955 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.953+0000 7fc6d0eb8640 1 Processor -- start 2024-09-10T16:31:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.953+0000 7fc6d0eb8640 1 -- start start 2024-09-10T16:31:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.953+0000 7fc6d0eb8640 10 monclient: init 2024-09-10T16:31:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:12.973 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6cc09f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6cc0a2550 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc6cc0a2980 0x7fc6cc13ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc0a2980 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09fd10 con 0x7fc6cc09f930 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6cc09cd20 con 0x7fc6cc0a2550 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc6cc09a0d0 con 0x7fc6cc0a2980 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:12.974 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6d0eb8640 10 monclient(hunting): authenticate will time out at 2251.441307s 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6ca59c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc6cc09f930 0x7fc6cc0a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57106/0 (socket says 172.21.15.164:57106) 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3695127352 learned_addr learned my addr 172.21.15.164:0/3695127352 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6cad9d640 1 --2- 172.21.15.164:0/3695127352 >> v2:172.21.15.165:3300/0 conn(0x7fc6cc0a2980 0x7fc6cc13ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request con 0x7fc6cc0a2980 auth_method 0 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:12.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.954+0000 7fc6cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3695127352 >> v1:172.21.15.164:6789/0 conn(0x7fc6cc09f930 legacy=0x7fc6cc0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 212556885 0 0) 0x7fc6cc09fd10 con 0x7fc6cc09f930 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 10 monclient(hunting): _negotiate 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 10 monclient(hunting): my global_id is 254394 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 --> v1:172.21.15.164:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6b4003520 con 0x7fc6cc09f930 2024-09-10T16:31:12.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2684581578 0 0) 0x7fc6b4003520 con 0x7fc6cc09f930 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 >> v2:172.21.15.165:3300/0 conn(0x7fc6cc0a2980 msgr2=0x7fc6cc13ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.956+0000 7fc6c959a640 1 --2- 172.21.15.164:0/3695127352 >> v2:172.21.15.165:3300/0 conn(0x7fc6cc0a2980 0x7fc6cc13ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 >> v1:172.21.15.165:6789/0 conn(0x7fc6cc0a2550 legacy=0x7fc6cc13b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _finish_hunting 0 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:31:12.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc09e840 con 0x7fc6cc09f930 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _finish_auth 0 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _check_auth_tickets 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fc6c00019a0 con 0x7fc6cc09f930 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6d0eb8640 5 monclient: authenticate success, global_id 254394 2024-09-10T16:31:12.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6d0eb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:31:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: dump: 2024-09-10T16:31:12.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:12.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:12.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 >> v1:172.21.15.164:6789/0 conn(0x7fc6cc09f930 legacy=0x7fc6cc0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6b4003ac0 0x7fc6b4003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): picked mon.b con 0x7fc6b4003ac0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): picked mon.a con 0x7fc6b4004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:12.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6b4005410 0x7fc6b400ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): picked mon.c con 0x7fc6b4005410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc09a0d0 con 0x7fc6b4004800 2024-09-10T16:31:12.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc6b400c350 con 0x7fc6b4003ac0 2024-09-10T16:31:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc6b400c550 con 0x7fc6b4005410 2024-09-10T16:31:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:12.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c959a640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6d0eb8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6b4005410 0x7fc6b400ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6b4003ac0 0x7fc6b4003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): get_auth_request con 0x7fc6b4004800 auth_method 0 2024-09-10T16:31:12.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): get_auth_request con 0x7fc6b4003ac0 auth_method 0 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:12.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request con 0x7fc6b4005410 auth_method 0 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:12.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient(hunting): handle_auth_done global_id 254394 payload 293 2024-09-10T16:31:12.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 -- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6b4005410 msgr2=0x7fc6b400ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6b4005410 0x7fc6b400ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 -- 172.21.15.164:0/3695127352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6b4003ac0 msgr2=0x7fc6b4003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6b4003ac0 0x7fc6b4003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 monclient: found mon.a 2024-09-10T16:31:12.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 1 -- 172.21.15.164:0/3695127352 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6b4003520 con 0x7fc6b4004800 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6cad9d640 1 -- 172.21.15.164:0/3695127352 reap_dead start 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient: _finish_auth 0 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.957+0000 7fc6c9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:12.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c9d9b640 1 -- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 msgr2=0x7fc6b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6b4004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc003db0 tx=0x7fc6b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4729155097febfa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc037070 con 0x7fc6b4004800 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: dump: 2024-09-10T16:31:12.991 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:12.992 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6d0eb8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6bc02ac00 con 0x7fc6b4004800 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:12.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6d0eb8640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6d0eb8640 4 set_mon_vals no callback set 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 1 -- 172.21.15.164:0/3695127352 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc003980 con 0x7fc6b4004800 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.960+0000 7fc6c959a640 10 monclient: dump: 2024-09-10T16:31:12.994 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:12.995 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 10 monclient: shutdown 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 msgr2=0x7fc6b4004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:12.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6b4004c80 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc003db0 tx=0x7fc6b4005830 comp rx=0 tx=0).stop 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3695127352 shutdown_connections 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3695127352 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6b4004c80 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3695127352 >> 172.21.15.164:0/3695127352 conn(0x7fc6cc0a3110 msgr2=0x7fc6cc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3695127352 shutdown_connections 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3695127352 wait complete. 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 10 monclient: build_initial_monmap 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.961+0000 7fc6d0eb8640 10 monclient: monmap: 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:12.997 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 Processor -- start 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 -- start start 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient: init 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-a con 0x7fc6b4004800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6cc09a970 0x7fc6cc09ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-b con 0x7fc6cc09a970 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6cc1ccb60 0x7fc6cc1ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): picked mon.noname-c con 0x7fc6cc1ccb60 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc09d930 con 0x7fc6b4004800 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc6cc09eb80 con 0x7fc6cc09a970 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc6cc09c8b0 con 0x7fc6cc1ccb60 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6d0eb8640 10 monclient(hunting): authenticate will time out at 2251.449779s 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56440/0 (socket says 172.21.15.164:56440) 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3904076607 learned_addr learned my addr 172.21.15.164:0/3904076607 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:12.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): get_auth_request con 0x7fc6b4004800 auth_method 0 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6cc1ccb60 0x7fc6cc1ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request con 0x7fc6cc1ccb60 auth_method 0 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.962+0000 7fc6cad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6cc09a970 0x7fc6cc09ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6c9d9b640 10 monclient(hunting): get_auth_request con 0x7fc6cc09a970 auth_method 0 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6c9d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6c9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6c9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 10 monclient(hunting): handle_auth_done global_id 254397 payload 824 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6cc1ccb60 msgr2=0x7fc6cc1ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6cc1ccb60 0x7fc6cc1ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6cc09a970 msgr2=0x7fc6cc09ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6cc09a970 0x7fc6cc09ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.963+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3904076607 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6cc09d620 con 0x7fc6b4004800 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6ca59c640 1 -- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 msgr2=0x7fc6cc09a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6ca59c640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00010e0 tx=0x7fc6c0004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000f8c0 con 0x7fc6b4004800 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: dump: 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.002 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 5 monclient: authenticate success, global_id 254397 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 10 monclient: _renew_subs 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c0018070 con 0x7fc6b4004800 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6cc0a5800 con 0x7fc6b4004800 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c000fce0 con 0x7fc6b4004800 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6baffd640 10 monclient: dump: 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 10 monclient: _renew_subs 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6c8d99640 4 set_mon_vals no callback set 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.965+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6cc098120 con 0x7fc6b4004800 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6d0eb8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6d0eb8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6d0eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc6c001d460 con 0x7fc6b4004800 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6cc0a0760 con 0x7fc6b4004800 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6baffd640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 0x7fc6ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.966+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(868..868 src has 772..868) ==== 4787+0+0 (secure 0 0 0) 0x7fc6c0052c40 con 0x7fc6b4004800 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.967+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 0x7fc6ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.967+0000 7fc6c9d9b640 10 monclient: get_auth_request con 0x7fc6ac03e210 auth_method 0 2024-09-10T16:31:13.005 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.969+0000 7fc6c9d9b640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 0x7fc6ac0406d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc034000 tx=0x7fc6bc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.971+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc6c0051630 con 0x7fc6b4004800 2024-09-10T16:31:13.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.971+0000 7fc6baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.006 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:12.971+0000 7fc6baffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.216+0000 7fc6d0eb8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.216+0000 7fc6d0eb8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.216+0000 7fc6d0eb8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.216+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc6cc096920 con 0x7fc6b4004800 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.217+0000 7fc6baffd640 1 -- 172.21.15.164:0/3904076607 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fc6c0012d60 con 0x7fc6b4004800 2024-09-10T16:31:13.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.217+0000 7fc6baffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:13.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.217+0000 7fc6baffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:13.219 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":244,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 msgr2=0x7fc6ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 0x7fc6ac0406d0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc034000 tx=0x7fc6bc02e000 comp rx=0 tx=0).stop 2024-09-10T16:31:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 10 monclient: shutdown 2024-09-10T16:31:13.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 msgr2=0x7fc6cc09a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fc6c00010e0 tx=0x7fc6c0004f90 comp rx=0 tx=0).stop 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 shutdown_connections 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc6ac03e210 0x7fc6ac0406d0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc6cc1ccb60 0x7fc6cc1ccfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc6cc09a970 0x7fc6cc09ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 --2- 172.21.15.164:0/3904076607 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc6b4004800 0x7fc6cc09a430 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 >> 172.21.15.164:0/3904076607 conn(0x7fc6cc0a3110 msgr2=0x7fc6cc072100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 shutdown_connections 2024-09-10T16:31:13.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.219+0000 7fc6d0eb8640 1 -- 172.21.15.164:0/3904076607 wait complete. 2024-09-10T16:31:13.231 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:31:13.231 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:31:13.231 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:31:13.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.328+0000 7f6a7b700640 10 monclient: get_monmap_and_config 2024-09-10T16:31:13.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.328+0000 7f6a7b700640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.328+0000 7f6a7b700640 10 monclient: monmap: 2024-09-10T16:31:13.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:13.329651+0000 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:13.329651+0000 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:13.331 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:13.332 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:13.332 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:13.332 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:13.332 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.329+0000 7f6a7b700640 1 Processor -- start 2024-09-10T16:31:13.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.329+0000 7f6a7b700640 1 -- start start 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.329+0000 7f6a7b700640 10 monclient: init 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.329+0000 7f6a7b700640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.329+0000 7f6a7b700640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a7409c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a7409d190 0x7f6a7409f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-a con 0x7f6a7409d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f6a7413a5c0 0x7f6a7413c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-c con 0x7f6a7413a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a7409fbc0 con 0x7f6a7409c310 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f6a740a2a20 con 0x7f6a7409d190 2024-09-10T16:31:13.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f6a7409c6f0 con 0x7f6a7413a5c0 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a7b700640 10 monclient(hunting): authenticate will time out at 2251.817078s 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a7409d190 0x7f6a7409f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f6a7409d190 0x7f6a7409f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56454/0 (socket says 172.21.15.164:56454) 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 -- 172.21.15.164:0/3425725711 learned_addr learned my addr 172.21.15.164:0/3425725711 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): get_auth_request con 0x7f6a7409d190 auth_method 0 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a79c9d640 1 --2- 172.21.15.164:0/3425725711 >> v2:172.21.15.165:3300/0 conn(0x7f6a7413a5c0 0x7f6a7413c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a79c9d640 10 monclient(hunting): get_auth_request con 0x7f6a7413a5c0 auth_method 0 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a79c9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a79c9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a79c9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_done global_id 254400 payload 274 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 -- 172.21.15.164:0/3425725711 >> v2:172.21.15.165:3300/0 conn(0x7f6a7413a5c0 msgr2=0x7f6a7413c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 --2- 172.21.15.164:0/3425725711 >> v2:172.21.15.165:3300/0 conn(0x7f6a7413a5c0 0x7f6a7413c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 -- 172.21.15.164:0/3425725711 >> v1:172.21.15.165:6789/0 conn(0x7f6a7409c310 legacy=0x7f6a7409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.330+0000 7f6a78c9b640 1 -- 172.21.15.164:0/3425725711 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a7409a4d0 con 0x7f6a7409d190 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a78c9b640 1 -- 172.21.15.164:0/3425725711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 msgr2=0x7f6a7409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a78c9b640 1 --2- 172.21.15.164:0/3425725711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7409f150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6a7406dcd0 tx=0x7f6a6402a0b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fcdf6f956ecdbf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 1 -- 172.21.15.164:0/3425725711 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64037070 con 0x7f6a7409d190 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: dump: 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.348 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 1 -- 172.21.15.164:0/3425725711 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a64032640 con 0x7f6a7409d190 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a7b700640 5 monclient: authenticate success, global_id 254400 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a7b700640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a7b700640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:13.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 1 -- 172.21.15.164:0/3425725711 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64032a10 con 0x7f6a7409d190 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a7b700640 4 set_mon_vals no callback set 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.333+0000 7f6a6bfff640 10 monclient: dump: 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.350 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 10 monclient: shutdown 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 -- 172.21.15.164:0/3425725711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 msgr2=0x7f6a7409f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 --2- 172.21.15.164:0/3425725711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7409f150 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f6a7406dcd0 tx=0x7f6a6402a0b0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 -- 172.21.15.164:0/3425725711 shutdown_connections 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 --2- 172.21.15.164:0/3425725711 >> v2:172.21.15.165:3300/0 conn(0x7f6a7413a5c0 0x7f6a7413c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 --2- 172.21.15.164:0/3425725711 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7409f150 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 -- 172.21.15.164:0/3425725711 >> 172.21.15.164:0/3425725711 conn(0x7f6a740a3110 msgr2=0x7f6a740a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 -- 172.21.15.164:0/3425725711 shutdown_connections 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.334+0000 7f6a7b700640 1 -- 172.21.15.164:0/3425725711 wait complete. 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient: monmap: 2024-09-10T16:31:13.351 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 Processor -- start 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 -- start start 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient: init 2024-09-10T16:31:13.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a7409c310 0x7f6a74139870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-c con 0x7f6a7409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 0x7f6a7413cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-a con 0x7f6a7413a5c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a7413d430 0x7f6a7413d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): picked mon.noname-b con 0x7f6a7413d430 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740a2230 con 0x7f6a7413a5c0 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740a20b0 con 0x7f6a7413d430 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f6a74074e00 con 0x7f6a7409c310 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7b700640 10 monclient(hunting): authenticate will time out at 2251.822850s 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 0x7f6a7413cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7949c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a7409c310 0x7f6a74139870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 0x7f6a7413cef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56460/0 (socket says 172.21.15.164:56460) 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 1 -- 172.21.15.164:0/120068098 learned_addr learned my addr 172.21.15.164:0/120068098 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 10 monclient(hunting): get_auth_request con 0x7f6a7413a5c0 auth_method 0 2024-09-10T16:31:13.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a78c9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7949c640 10 monclient(hunting): get_auth_request con 0x7f6a7409c310 auth_method 0 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7949c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7949c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.335+0000 7f6a7949c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a79c9d640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a7413d430 0x7f6a7413d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient(hunting): handle_auth_done global_id 254403 payload 824 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a7409c310 msgr2=0x7f6a74139870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a7409c310 0x7f6a74139870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a7413d430 msgr2=0x7f6a7413d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a7413d430 0x7f6a7413d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a78c9b640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a74093a70 con 0x7f6a7413a5c0 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a79c9d640 10 monclient: get_auth_request con 0x7f6a7413d430 auth_method 0 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a79c9d640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6a7413d430 0x7f6a7413d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:31:13.356 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.336+0000 7f6a7949c640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6a7409c310 0x7f6a74139870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a78c9b640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 msgr2=0x7f6a7413cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a78c9b640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 0x7f6a7413cef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6a64000c00 tx=0x7f6a640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64041070 con 0x7f6a7413a5c0 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: dump: 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.357 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a6403c880 con 0x7f6a7413a5c0 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 5 monclient: authenticate success, global_id 254403 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a64037040 con 0x7f6a7413a5c0 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.358 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a69ffb640 10 monclient: dump: 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 10 monclient: _renew_subs 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.359 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a6b7fe640 4 set_mon_vals no callback set 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a7413cba0 con 0x7f6a7413a5c0 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 10 monclient: _renew_subs 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.338+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a7413ed00 con 0x7f6a7413a5c0 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7b700640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7b700640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7402d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.360 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6a7409f720 con 0x7f6a7409d190 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6a64044070 con 0x7f6a7413a5c0 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7402d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient: get_auth_request con 0x7f6a7409d190 auth_method 0 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.361 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.339+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a69ffb640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6a5803e290 0x7f6a58040750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a7949c640 10 monclient(hunting): handle_auth_done global_id 254406 payload 824 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a7949c640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6a74070ef0 con 0x7f6a7409d190 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(868..868 src has 772..868) ==== 4787+0+0 (secure 0 0 0) 0x7f6a64072e40 con 0x7f6a7413a5c0 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a79c9d640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6a5803e290 0x7f6a58040750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.362 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.340+0000 7f6a79c9d640 10 monclient: get_auth_request con 0x7f6a5803e290 auth_method 0 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a7949c640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 msgr2=0x7f6a7402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a7949c640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7402d440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a7000d830 con 0x7f6a7409d190 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 10 monclient: dump: 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.363 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.364 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6a70018070 con 0x7f6a7409d190 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 msgr2=0x7f6a7402d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7402d440 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001cf0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 msgr2=0x7f6a7402d440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a69ffb640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7409d190 0x7f6a7402d440 secure :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001cf0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.365 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.342+0000 7f6a79c9d640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6a5803e290 0x7f6a58040750 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009f70 tx=0x7f6a6c00b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7b700640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:13.368 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7b700640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 0x7f6a7402d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7b700640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6a740936f0 con 0x7f6a7413e5c0 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 0x7f6a7402d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient: get_auth_request con 0x7f6a7413e5c0 auth_method 0 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.348+0000 7f6a7949c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.349+0000 7f6a7949c640 10 monclient(hunting): handle_auth_done global_id 254409 payload 824 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.349+0000 7f6a7949c640 1 -- 172.21.15.164:0/120068098 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6a74077260 con 0x7f6a7413e5c0 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7949c640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 msgr2=0x7f6a7402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7949c640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 0x7f6a7402d980 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a700035f0 con 0x7f6a7413e5c0 2024-09-10T16:31:13.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 10 monclient: dump: 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.371 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7f6a700035f0 con 0x7f6a7413e5c0 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:13.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 msgr2=0x7f6a7402d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 0x7f6a7402d980 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001020 comp rx=0 tx=0).stop 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 msgr2=0x7f6a7402d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a69ffb640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413e5c0 0x7f6a7402d980 secure :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6a7000fb10 tx=0x7f6a70001020 comp rx=0 tx=0).stop 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:13.373 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 162002, 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.374 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.375 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.376 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:13.377 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.378 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.379 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.380 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:13.381 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.382 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.383 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:13.384 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.385 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.386 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.387 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:13.387 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6a5803e290 msgr2=0x7f6a58040750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7b700640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f6a5803e290 0x7f6a58040750 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f6a6c009f70 tx=0x7f6a6c00b040 comp rx=0 tx=0).stop 2024-09-10T16:31:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7b700640 10 monclient: shutdown 2024-09-10T16:31:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.351+0000 7f6a7b700640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 msgr2=0x7f6a7413cef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 --2- 172.21.15.164:0/120068098 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6a7413a5c0 0x7f6a7413cef0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6a64000c00 tx=0x7f6a640046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a79c9d640 1 -- 172.21.15.164:0/120068098 reap_dead start 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a79c9d640 1 -- 172.21.15.164:0/120068098 reap_dead start 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 shutdown_connections 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 >> 172.21.15.164:0/120068098 conn(0x7f6a740a3110 msgr2=0x7f6a7409eea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 shutdown_connections 2024-09-10T16:31:13.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.352+0000 7f6a7b700640 1 -- 172.21.15.164:0/120068098 wait complete. 2024-09-10T16:31:13.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:13.389 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:31:13.390 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:31:13.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.486+0000 7fb6f4648640 10 monclient: get_monmap_and_config 2024-09-10T16:31:13.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.486+0000 7fb6f4648640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.486+0000 7fb6f4648640 10 monclient: monmap: 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:13.487553+0000 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:13.487553+0000 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.489 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:13.490 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 Processor -- start 2024-09-10T16:31:13.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 -- start start 2024-09-10T16:31:13.491 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient: init 2024-09-10T16:31:13.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 0x7fb6ec0a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-c con 0x7fb6ec09f930 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 0x7fb6ec0a27b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-b con 0x7fb6ec0a2380 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-a con 0x7fb6ec13a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ec09fd10 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fb6ec09cd20 con 0x7fb6ec09f930 2024-09-10T16:31:13.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec09a0d0 con 0x7fb6ec0a2380 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.487+0000 7fb6f4648640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f4648640 10 monclient(hunting): authenticate will time out at 2251.975008s 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 0x7fb6ec0a27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 0x7fb6ec0a27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57710/0 (socket says 172.21.15.164:57710) 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 1 -- 172.21.15.164:0/26309193 learned_addr learned my addr 172.21.15.164:0/26309193 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request con 0x7fb6ec0a2380 auth_method 0 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 0x7fb6ec0a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request con 0x7fb6ec13a5c0 auth_method 0 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request con 0x7fb6ec09f930 auth_method 0 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 msgr2=0x7fb6ec0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 msgr2=0x7fb6ec0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 0x7fb6ec0a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 0x7fb6ec0a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_done global_id 254412 payload 274 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 msgr2=0x7fb6ec0a27b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 0x7fb6ec0a27b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 msgr2=0x7fb6ec0a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> v2:172.21.15.165:3300/0 conn(0x7fb6ec09f930 0x7fb6ec0a1e40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec09e840 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.488+0000 7fb6f1be3640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec0a2380 0x7fb6ec0a27b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec06d470 tx=0x7fb6e402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce376bb0f824182e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec06d470 tx=0x7fb6e402a4b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec06d470 tx=0x7fb6e402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient: ms_handle_reset current mon [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec13c9b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient: _add_conns ranks=[1,0] 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb6d8003500 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6d8007490 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d800b400 con 0x7fb6d8007490 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d800c600 con 0x7fb6d8003500 2024-09-10T16:31:13.522 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.491+0000 7fb6f13e2640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f1be3640 1 -- 172.21.15.164:0/26309193 >> v1:172.21.15.164:6789/0 conn(0x7fb6d8007490 legacy=0x7fb6d8009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v1:172.21.15.164:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2459423607 0 0) 0x7fb6d800b400 con 0x7fb6d8007490 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient(hunting): _negotiate 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> v1:172.21.15.164:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fb6d800b5d0 con 0x7fb6d8007490 2024-09-10T16:31:13.523 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v1:172.21.15.164:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 440913745 0 0) 0x7fb6d800b5d0 con 0x7fb6d8007490 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 >> v1:172.21.15.165:6789/0 conn(0x7fb6d8003500 legacy=0x7fb6d8005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> v1:172.21.15.165:6789/0 conn(0x7fb6d8003500 legacy=0x7fb6d8005980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 monclient: found mon.noname-a-legacy 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.164:6789/0 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> v1:172.21.15.164:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d800d910 con 0x7fb6d8007490 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.524 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v1:172.21.15.164:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7fb6dc0010d0 con 0x7fb6d8007490 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f4648640 5 monclient: authenticate success, global_id 254412 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f4648640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: dump: 2024-09-10T16:31:13.525 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.526 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] but i'm connected to v1:172.21.15.164:6789/0, reconnecting 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.493+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 >> v1:172.21.15.164:6789/0 conn(0x7fb6d8007490 legacy=0x7fb6d8009950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 1 -- 172.21.15.164:0/26309193 reap_dead start 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 0x7fb6d800c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): picked mon.a con 0x7fb6d800bdd0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800cd10 0x7fb6d800d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): picked mon.c con 0x7fb6d800cd10 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6d8006150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): picked mon.b con 0x7fb6d80058f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8005e40 con 0x7fb6d800bdd0 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.528 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8006690 con 0x7fb6d80058f0 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d8006890 con 0x7fb6d800cd10 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f13e2640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f4648640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 0x7fb6d800c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request con 0x7fb6d800bdd0 auth_method 0 2024-09-10T16:31:13.529 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800cd10 0x7fb6d800d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request con 0x7fb6d800cd10 auth_method 0 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:13.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6d8006150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request con 0x7fb6d80058f0 auth_method 0 2024-09-10T16:31:13.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_done global_id 254412 payload 293 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800cd10 msgr2=0x7fb6d800d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800cd10 0x7fb6d800d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 msgr2=0x7fb6d8006150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6d8006150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 monclient: found mon.a 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d800b5d0 con 0x7fb6d800bdd0 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f23e4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.494+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6d8006150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f23e4640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 msgr2=0x7fb6d800c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 0x7fb6d800c7d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8006fd0 tx=0x7fb6d800b810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab76e36166ce15 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8021070 con 0x7fb6d800bdd0 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: dump: 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.534 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f4648640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6e801c600 con 0x7fb6d800bdd0 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.535 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f4648640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 1 -- 172.21.15.164:0/26309193 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e8010cd0 con 0x7fb6d800bdd0 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f4648640 4 set_mon_vals no callback set 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.496+0000 7fb6f13e2640 10 monclient: dump: 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.536 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 10 monclient: shutdown 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 -- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 msgr2=0x7fb6d800c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 0x7fb6d800c7d0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8006fd0 tx=0x7fb6d800b810 comp rx=0 tx=0).stop 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 -- 172.21.15.164:0/26309193 shutdown_connections 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6d8006150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800cd10 0x7fb6d800d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800bdd0 0x7fb6d800c7d0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 --2- 172.21.15.164:0/26309193 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6ec13a5c0 0x7fb6ec13c9b0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.537 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 -- 172.21.15.164:0/26309193 >> 172.21.15.164:0/26309193 conn(0x7fb6ec0a3110 msgr2=0x7fb6ec0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 -- 172.21.15.164:0/26309193 shutdown_connections 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.497+0000 7fb6f4648640 1 -- 172.21.15.164:0/26309193 wait complete. 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient: monmap: 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:13.538 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 1 Processor -- start 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 1 -- start start 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient: init 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6ec073d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d80058f0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.498+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d800bdd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 0x7fb6ec1c9390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d800cd10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec09d930 con 0x7fb6d800cd10 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec09ee00 con 0x7fb6d80058f0 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec09c8b0 con 0x7fb6d800bdd0 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f4648640 10 monclient(hunting): authenticate will time out at 2251.986040s 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 0x7fb6ec1c9390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:57730/0 (socket says 172.21.15.164:57730) 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 -- 172.21.15.164:0/3120311808 learned_addr learned my addr 172.21.15.164:0/3120311808 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request con 0x7fb6d800bdd0 auth_method 0 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 0x7fb6ec1c9390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56514/0 (socket says 172.21.15.164:56514) 2024-09-10T16:31:13.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request con 0x7fb6d800cd10 auth_method 0 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6ec073d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request con 0x7fb6d80058f0 auth_method 0 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f23e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f23e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient(hunting): handle_auth_done global_id 274187 payload 824 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 msgr2=0x7fb6ec073d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb6d80058f0 0x7fb6ec073d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 msgr2=0x7fb6ec1c9390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 0x7fb6ec1c9390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 monclient: found mon.noname-c 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f1be3640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec12e360 con 0x7fb6d800bdd0 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.499+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb6d800cd10 0x7fb6ec1c9390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f1be3640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 msgr2=0x7fb6ec1c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f1be3640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0019c0 tx=0x7fb6dc00de80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc018400 con 0x7fb6d800bdd0 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:13.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: dump: 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.544 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6dc0185a0 con 0x7fb6d800bdd0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 5 monclient: authenticate success, global_id 274187 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 10 monclient: _renew_subs 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec074680 con 0x7fb6d800bdd0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc018970 con 0x7fb6d800bdd0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6e2ffd640 10 monclient: dump: 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.546 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f0be1640 4 set_mon_vals no callback set 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 10 monclient: _renew_subs 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.501+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec09d620 con 0x7fb6d800bdd0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f4648640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb6dc021440 con 0x7fb6d800bdd0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f4648640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 0x7fb6ec12b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec12dc50 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 0x7fb6ec12b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient: get_auth_request con 0x7fb6ec13a5c0 auth_method 0 2024-09-10T16:31:13.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.502+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6e2ffd640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6d003e240 0x7fb6d0040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_done global_id 274190 payload 824 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(868..868 src has 772..868) ==== 4787+0+0 (secure 0 0 0) 0x7fb6dc053070 con 0x7fb6d800bdd0 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6f23e4640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6ec0a0760 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6d003e240 0x7fb6d0040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.503+0000 7fb6f2be5640 10 monclient: get_auth_request con 0x7fb6d003e240 auth_method 0 2024-09-10T16:31:13.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6f23e4640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec12b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 0x7fb6ec12b4f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8006fd0 tx=0x7fb6e8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e802b020 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 10 monclient: dump: 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.549 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6e8004250 con 0x7fb6ec13a5c0 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec12b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 0x7fb6ec12b4f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8006fd0 tx=0x7fb6e8004730 comp rx=0 tx=0).stop 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 msgr2=0x7fb6ec12b4f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6e2ffd640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec13a5c0 0x7fb6ec12b4f0 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8006fd0 tx=0x7fb6e8004730 comp rx=0 tx=0).stop 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.505+0000 7fb6f2be5640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6d003e240 0x7fb6d0040700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0770d0 tx=0x7fb6e402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.511+0000 7fb6f4648640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.511+0000 7fb6f4648640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 0x7fb6ec12be40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.511+0000 7fb6f4648640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.511+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb6ec12c380 con 0x7fb6ec12ba30 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.511+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 0x7fb6ec12be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient: get_auth_request con 0x7fb6ec12ba30 auth_method 0 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 10 monclient(hunting): handle_auth_done global_id 274193 payload 824 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.512+0000 7fb6f23e4640 1 -- 172.21.15.164:0/3120311808 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6ec077260 con 0x7fb6ec12ba30 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6f23e4640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 msgr2=0x7fb6ec12be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6f23e4640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 0x7fb6ec12be40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6e802b040 con 0x7fb6ec12ba30 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 10 monclient: dump: 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.552 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fb6e802fdf0 con 0x7fb6ec12ba30 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 msgr2=0x7fb6ec12be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 0x7fb6ec12be40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8014000 comp rx=0 tx=0).stop 2024-09-10T16:31:13.553 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 msgr2=0x7fb6ec12be40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.514+0000 7fb6e2ffd640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6ec12ba30 0x7fb6ec12be40 secure :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb6e8000c00 tx=0x7fb6e8014000 comp rx=0 tx=0).stop 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 50757, 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:13.554 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 1, 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.555 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:13.556 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.557 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:13.558 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:13.559 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:13.560 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.561 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:13.562 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:13.563 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:13.564 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.565 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.566 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:13.567 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6d003e240 msgr2=0x7fb6d0040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb6d003e240 0x7fb6d0040700 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fb6ec0770d0 tx=0x7fb6e402f000 comp rx=0 tx=0).stop 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 10 monclient: shutdown 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 msgr2=0x7fb6ec1c8e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc0019c0 tx=0x7fb6dc00de80 comp rx=0 tx=0).stop 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f2be5640 1 -- 172.21.15.164:0/3120311808 reap_dead start 2024-09-10T16:31:13.569 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 shutdown_connections 2024-09-10T16:31:13.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 --2- 172.21.15.164:0/3120311808 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb6d800bdd0 0x7fb6ec1c8e50 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 >> 172.21.15.164:0/3120311808 conn(0x7fb6ec0a3110 msgr2=0x7fb6ec1376b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 shutdown_connections 2024-09-10T16:31:13.570 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.515+0000 7fb6f4648640 1 -- 172.21.15.164:0/3120311808 wait complete. 2024-09-10T16:31:13.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:13.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:31:13.570 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:31:13.617 INFO:tasks.rados.rados.0.smithi165.stdout:3856: done (0 left) 2024-09-10T16:31:13.618 INFO:tasks.rados.rados.0.smithi165.stdout:3857: copy_from oid 38 from oid 2 current snap is 431 2024-09-10T16:31:13.618 INFO:tasks.rados.rados.0.smithi165.stdout:3858: write (excl) oid 49 current snap is 431 2024-09-10T16:31:13.618 INFO:tasks.rados.rados.0.smithi165.stdout:3858: seq_num 1327 ranges {702889=500845,1636311=576790} 2024-09-10T16:31:13.624 INFO:tasks.rados.rados.0.smithi165.stdout:3858: writing smithi16534841-49 from 702889 to 1203734 tid 1 2024-09-10T16:31:13.631 INFO:tasks.rados.rados.0.smithi165.stdout:3858: writing smithi16534841-49 from 1636311 to 2213101 tid 2 2024-09-10T16:31:13.631 INFO:tasks.rados.rados.0.smithi165.stdout:3859: read oid 29 snap -1 2024-09-10T16:31:13.631 INFO:tasks.rados.rados.0.smithi165.stdout:3859: expect deleted 2024-09-10T16:31:13.631 INFO:tasks.rados.rados.0.smithi165.stdout:3860: snap_create 2024-09-10T16:31:13.633 INFO:tasks.rados.rados.0.smithi165.stdout:3857: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:31:13.636 INFO:tasks.rados.rados.0.smithi165.stdout:3858: finishing write tid 1 to smithi16534841-49 2024-09-10T16:31:13.636 INFO:tasks.rados.rados.0.smithi165.stdout:3857: finishing copy_from to smithi16534841-38 2024-09-10T16:31:13.636 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 38 v 2118 (ObjNum 1313 snap 426 seq_num 1313) dirty exists 2024-09-10T16:31:13.636 INFO:tasks.rados.rados.0.smithi165.stdout:3858: finishing write tid 2 to smithi16534841-49 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:3858: finishing write tid 3 to smithi16534841-49 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:3858: finishing write tid 4 to smithi16534841-49 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:3858: oid 49 updating version 0 to 2022 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:3858: oid 49 updating version 2022 to 2023 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:3858: oid 49 updating version 2023 to 2024 2024-09-10T16:31:13.637 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 49 v 2024 (ObjNum 1327 snap 431 seq_num 1327) dirty exists 2024-09-10T16:31:13.638 INFO:tasks.rados.rados.0.smithi165.stdout:3858: left oid 49 (ObjNum 1327 snap 431 seq_num 1327) 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.678+0000 7f33cb27f640 10 monclient: get_monmap_and_config 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.678+0000 7f33cb27f640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.678+0000 7f33cb27f640 10 monclient: monmap: 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:13.679305+0000 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:13.679305+0000 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.680 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.681 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.678+0000 7f33cb27f640 1 Processor -- start 2024-09-10T16:31:13.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 -- start start 2024-09-10T16:31:13.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient: init 2024-09-10T16:31:13.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.688 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f33c406d160 0x7f33c4134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-a con 0x7f33c406d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 0x7f33c40a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-c con 0x7f33c406d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33c40a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33c409cb90 con 0x7f33c40a4c20 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f33c409ce70 con 0x7f33c406d160 2024-09-10T16:31:13.689 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f33c409d150 con 0x7f33c406d540 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33cb27f640 10 monclient(hunting): authenticate will time out at 2252.166805s 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f33c406d160 0x7f33c4134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c981c640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f33c40a4c20 0x7f33c4137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:57124/0 (socket says 172.21.15.164:57124) 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c981c640 1 -- 172.21.15.164:0/2668169238 learned_addr learned my addr 172.21.15.164:0/2668169238 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): get_auth_request con 0x7f33c406d160 auth_method 0 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.690 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.679+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 10 monclient(hunting): handle_auth_done global_id 254418 payload 274 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 1 -- 172.21.15.164:0/2668169238 >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 msgr2=0x7f33c40a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c881a640 1 --2- 172.21.15.164:0/2668169238 >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 0x7f33c40a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 1 --2- 172.21.15.164:0/2668169238 >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 0x7f33c40a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 1 -- 172.21.15.164:0/2668169238 >> v1:172.21.15.164:6789/0 conn(0x7f33c40a4c20 legacy=0x7f33c4137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c901b640 1 -- 172.21.15.164:0/2668169238 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c4135340 con 0x7f33c406d160 2024-09-10T16:31:13.691 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c406d540 auth_method 0 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.680+0000 7f33c881a640 1 --2- 172.21.15.164:0/2668169238 >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 0x7f33c40a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.682+0000 7f33c901b640 1 -- 172.21.15.164:0/2668169238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 msgr2=0x7f33c4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.682+0000 7f33c901b640 1 --2- 172.21.15.164:0/2668169238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c4134e00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f33c40a9d40 tx=0x7f33b802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30e66fdb0dda1bc3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 1 -- 172.21.15.164:0/2668169238 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8037070 con 0x7f33c406d160 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: dump: 2024-09-10T16:31:13.692 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 1 -- 172.21.15.164:0/2668169238 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b8032640 con 0x7f33c406d160 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 5 monclient: authenticate success, global_id 254418 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 4 set_mon_vals no callback set 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 1 -- 172.21.15.164:0/2668169238 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8032a10 con 0x7f33c406d160 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.694 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33b7fff640 10 monclient: dump: 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 10 monclient: shutdown 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 -- 172.21.15.164:0/2668169238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 msgr2=0x7f33c4134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 --2- 172.21.15.164:0/2668169238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c4134e00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f33c40a9d40 tx=0x7f33b802a000 comp rx=0 tx=0).stop 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 -- 172.21.15.164:0/2668169238 shutdown_connections 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 --2- 172.21.15.164:0/2668169238 >> v2:172.21.15.165:3300/0 conn(0x7f33c406d540 0x7f33c40a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 --2- 172.21.15.164:0/2668169238 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c4134e00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 -- 172.21.15.164:0/2668169238 >> 172.21.15.164:0/2668169238 conn(0x7f33c40aacb0 msgr2=0x7f33c40ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 -- 172.21.15.164:0/2668169238 shutdown_connections 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.683+0000 7f33cb27f640 1 -- 172.21.15.164:0/2668169238 wait complete. 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.684+0000 7f33cb27f640 10 monclient: build_initial_monmap 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.684+0000 7f33cb27f640 10 monclient: monmap: 2024-09-10T16:31:13.696 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.684+0000 7f33cb27f640 1 Processor -- start 2024-09-10T16:31:13.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.684+0000 7f33cb27f640 1 -- start start 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.684+0000 7f33cb27f640 10 monclient: init 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c41ccf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-a con 0x7f33c406d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33c406d540 0x7f33c41cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-c con 0x7f33c406d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c40a4c20 0x7f33c41cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): picked mon.noname-b con 0x7f33c40a4c20 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4138780 con 0x7f33c406d160 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4138600 con 0x7f33c40a4c20 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f33c4138900 con 0x7f33c406d540 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33cb27f640 10 monclient(hunting): authenticate will time out at 2252.172104s 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c41ccf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33c406d540 0x7f33c41cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c41ccf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56528/0 (socket says 172.21.15.164:56528) 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 -- 172.21.15.164:0/3555962435 learned_addr learned my addr 172.21.15.164:0/3555962435 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:13.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): get_auth_request con 0x7f33c406d160 auth_method 0 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): get_auth_request con 0x7f33c406d540 auth_method 0 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:13.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c981c640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c40a4c20 0x7f33c41cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c981c640 10 monclient(hunting): get_auth_request con 0x7f33c40a4c20 auth_method 0 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c981c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c981c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c981c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient(hunting): handle_auth_done global_id 254421 payload 824 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33c406d540 msgr2=0x7f33c41cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f33c406d540 0x7f33c41cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c40a4c20 msgr2=0x7f33c41cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c40a4c20 0x7f33c41cda00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:13.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 monclient: found mon.noname-a 2024-09-10T16:31:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.685+0000 7f33c901b640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c4136490 con 0x7f33c406d160 2024-09-10T16:31:13.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33c901b640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 msgr2=0x7f33c41ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:13.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33c901b640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c41ccf80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33b8000c00 tx=0x7f33b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8037040 con 0x7f33c406d160 2024-09-10T16:31:13.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: dump: 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.709 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b8042070 con 0x7f33c406d160 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33cb27f640 5 monclient: authenticate success, global_id 254421 2024-09-10T16:31:13.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33cb27f640 10 monclient: _renew_subs 2024-09-10T16:31:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33cb27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c40a4970 con 0x7f33c406d160 2024-09-10T16:31:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b8030d60 con 0x7f33c406d160 2024-09-10T16:31:13.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b5ffb640 10 monclient: dump: 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.712 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:13.713 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 10 monclient: _renew_subs 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.687+0000 7f33b77fe640 4 set_mon_vals no callback set 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c40a3b80 con 0x7f33c406d160 2024-09-10T16:31:13.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.688+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f33c40a3270 con 0x7f33c412dfb0 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f33b8032500 con 0x7f33c406d160 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33b5ffb640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33a803e210 0x7f33a80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:13.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(869..869 src has 772..869) ==== 4831+0+0 (secure 0 0 0) 0x7f33b8073710 con 0x7f33c406d160 2024-09-10T16:31:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412dfb0 auth_method 0 2024-09-10T16:31:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c981c640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33a803e210 0x7f33a80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:13.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.689+0000 7f33c981c640 10 monclient: get_auth_request con 0x7f33a803e210 auth_method 0 2024-09-10T16:31:13.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.692+0000 7f33c981c640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33a803e210 0x7f33a80406d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007ae0 tx=0x7f33bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:13.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:13.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412dfb0 auth_method 0 2024-09-10T16:31:13.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:13.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:13.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:13.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:31:13.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:13.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:13.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:13.890+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412dfb0 auth_method 0 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:14.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:31:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:14.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:14.291+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3857: done (3 left) 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3858: done (2 left) 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3859: done (1 left) 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3860: done (0 left) 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3861: read oid 34 snap 407 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3861: expect (ObjNum 1268 snap 406 seq_num 1268) 2024-09-10T16:31:14.630 INFO:tasks.rados.rados.0.smithi165.stdout:3862: snap_create 2024-09-10T16:31:14.653 INFO:tasks.rados.rados.0.smithi165.stdout:3861: expect (ObjNum 1268 snap 406 seq_num 1268) 2024-09-10T16:31:14.655 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 534020 2024-09-10T16:31:14.697 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:14.755 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1278626 2024-09-10T16:31:15.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.092+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:15.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.092+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412dfb0 auth_method 0 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.092+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.092+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.092+0000 7f33c881a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.093+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.093+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.093+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:15.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:15.093+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:3861: done (1 left) 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:3862: done (0 left) 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:3863: rollback oid 48 current snap is 433 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 48 to 420 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:3864: write (excl) oid 27 current snap is 433 2024-09-10T16:31:15.636 INFO:tasks.rados.rados.0.smithi165.stdout:3864: seq_num 1328 ranges {719696=576094,2094793=224903} 2024-09-10T16:31:15.643 INFO:tasks.rados.rados.0.smithi165.stdout:3864: writing smithi16534841-27 from 719696 to 1295790 tid 1 2024-09-10T16:31:15.649 INFO:tasks.rados.rados.0.smithi165.stdout:3864: writing smithi16534841-27 from 2094793 to 2319696 tid 2 2024-09-10T16:31:15.649 INFO:tasks.rados.rados.0.smithi165.stdout:3865: snap_remove snap 431 2024-09-10T16:31:16.393 INFO:tasks.rados.rados.0.smithi165.stdout:3865: done (2 left) 2024-09-10T16:31:16.394 INFO:tasks.rados.rados.0.smithi165.stdout:3866: write oid 7 current snap is 433 2024-09-10T16:31:16.394 INFO:tasks.rados.rados.0.smithi165.stdout:3863: finishing rollback tid 0 to smithi16534841-48 2024-09-10T16:31:16.394 INFO:tasks.rados.rados.0.smithi165.stdout:3866: seq_num 1329 ranges {736503=651343,2153275=183228} 2024-09-10T16:31:16.401 INFO:tasks.rados.rados.0.smithi165.stdout:3866: writing smithi16534841-7 from 736503 to 1387846 tid 1 2024-09-10T16:31:16.406 INFO:tasks.rados.rados.0.smithi165.stdout:3866: writing smithi16534841-7 from 2153275 to 2336503 tid 2 2024-09-10T16:31:16.406 INFO:tasks.rados.rados.0.smithi165.stdout:3867: write oid 12 current snap is 433 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:3863: finishing rollback tid 1 to smithi16534841-48 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:3863: finishing rollback tid 2 to smithi16534841-48 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 2569 (ObjNum 1064 snap 328 seq_num 1064) dirty exists 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:3864: finishing write tid 1 to smithi16534841-27 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:3864: finishing write tid 2 to smithi16534841-27 2024-09-10T16:31:16.407 INFO:tasks.rados.rados.0.smithi165.stdout:3867: seq_num 1330 ranges {753310=726592,2095404=257906} 2024-09-10T16:31:16.414 INFO:tasks.rados.rados.0.smithi165.stdout:3867: writing smithi16534841-12 from 753310 to 1479902 tid 1 2024-09-10T16:31:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:3867: writing smithi16534841-12 from 2095404 to 2353310 tid 2 2024-09-10T16:31:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:3863: done (3 left) 2024-09-10T16:31:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:3868: rollback oid 37 current snap is 433 2024-09-10T16:31:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 37 to 411 2024-09-10T16:31:16.420 INFO:tasks.rados.rados.0.smithi165.stdout:3864: finishing write tid 3 to smithi16534841-27 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3864: finishing write tid 4 to smithi16534841-27 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3864: oid 27 updating version 0 to 2178 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3864: oid 27 updating version 2178 to 2179 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3864: oid 27 version 2179 is already newer than 2177 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 2179 (ObjNum 1328 snap 433 seq_num 1328) dirty exists 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3864: left oid 27 (ObjNum 1328 snap 433 seq_num 1328) 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: finishing write tid 1 to smithi16534841-7 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: finishing write tid 2 to smithi16534841-7 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: finishing write tid 3 to smithi16534841-7 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: finishing write tid 4 to smithi16534841-7 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: oid 7 updating version 0 to 2575 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: oid 7 version 2575 is already newer than 2574 2024-09-10T16:31:16.421 INFO:tasks.rados.rados.0.smithi165.stdout:3866: oid 7 version 2575 is already newer than 2573 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2575 (ObjNum 1329 snap 433 seq_num 1329) dirty exists 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3866: left oid 7 (ObjNum 1329 snap 433 seq_num 1329) 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3864: done (3 left) 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3866: done (2 left) 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3869: copy_from oid 22 from oid 24 current snap is 433 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3870: read oid 46 snap -1 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3870: expect (ObjNum 1220 snap 393 seq_num 1220) 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3871: write oid 30 current snap is 433 2024-09-10T16:31:16.422 INFO:tasks.rados.rados.0.smithi165.stdout:3871: seq_num 1331 ranges {770117=518194,1870239=499878} 2024-09-10T16:31:16.427 INFO:tasks.rados.rados.0.smithi165.stdout:3871: writing smithi16534841-30 from 770117 to 1288311 tid 1 2024-09-10T16:31:16.434 INFO:tasks.rados.rados.0.smithi165.stdout:3871: writing smithi16534841-30 from 1870239 to 2370117 tid 2 2024-09-10T16:31:16.434 INFO:tasks.rados.rados.0.smithi165.stdout:3872: read oid 47 snap -1 2024-09-10T16:31:16.434 INFO:tasks.rados.rados.0.smithi165.stdout:3872: expect (ObjNum 1138 snap 351 seq_num 1138) 2024-09-10T16:31:16.435 INFO:tasks.rados.rados.0.smithi165.stdout:3873: read oid 48 snap 432 2024-09-10T16:31:16.435 INFO:tasks.rados.rados.0.smithi165.stdout:3873: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:31:16.435 INFO:tasks.rados.rados.0.smithi165.stdout:3874: write oid 40 current snap is 433 2024-09-10T16:31:16.435 INFO:tasks.rados.rados.0.smithi165.stdout:3874: seq_num 1332 ranges {786924=593443,1928721=458203} 2024-09-10T16:31:16.442 INFO:tasks.rados.rados.0.smithi165.stdout:3874: writing smithi16534841-40 from 786924 to 1380367 tid 1 2024-09-10T16:31:16.449 INFO:tasks.rados.rados.0.smithi165.stdout:3874: writing smithi16534841-40 from 1928721 to 2386924 tid 2 2024-09-10T16:31:16.449 INFO:tasks.rados.rados.0.smithi165.stdout:3875: copy_from oid 14 from oid 6 current snap is 433 2024-09-10T16:31:16.449 INFO:tasks.rados.rados.0.smithi165.stdout:3870: expect (ObjNum 1220 snap 393 seq_num 1220) 2024-09-10T16:31:16.449 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:16.490 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1948962 2024-09-10T16:31:16.504 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1673961 2024-09-10T16:31:16.513 INFO:tasks.rados.rados.0.smithi165.stdout:3867: finishing write tid 1 to smithi16534841-12 2024-09-10T16:31:16.513 INFO:tasks.rados.rados.0.smithi165.stdout:3867: finishing write tid 2 to smithi16534841-12 2024-09-10T16:31:16.513 INFO:tasks.rados.rados.0.smithi165.stdout:3867: finishing write tid 3 to smithi16534841-12 2024-09-10T16:31:16.513 INFO:tasks.rados.rados.0.smithi165.stdout:3867: finishing write tid 4 to smithi16534841-12 2024-09-10T16:31:16.513 INFO:tasks.rados.rados.0.smithi165.stdout:3867: oid 12 updating version 0 to 1662 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3867: oid 12 updating version 1662 to 1663 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3867: oid 12 updating version 1663 to 1664 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1664 (ObjNum 1330 snap 433 seq_num 1330) dirty exists 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3867: left oid 12 (ObjNum 1330 snap 433 seq_num 1330) 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3867: done (8 left) 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3870: done (7 left) 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3876: copy_from oid 9 from oid 45 current snap is 433 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3871: finishing write tid 1 to smithi16534841-30 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3868: finishing rollback tid 0 to smithi16534841-37 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3868: finishing rollback tid 1 to smithi16534841-37 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:3868: finishing rollback tid 2 to smithi16534841-37 2024-09-10T16:31:16.514 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 37 v 1897 (ObjNum 1240 snap 402 seq_num 1240) dirty exists 2024-09-10T16:31:16.515 INFO:tasks.rados.rados.0.smithi165.stdout:3872: expect (ObjNum 1138 snap 351 seq_num 1138) 2024-09-10T16:31:16.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.685+0000 7f33b4ff9640 10 monclient: tick 2024-09-10T16:31:16.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.685+0000 7f33b4ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:31:16.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.685+0000 7f33b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:16.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.685+0000 7f33b4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:16.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412dfb0 auth_method 0 2024-09-10T16:31:16.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:16.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 10 monclient(hunting): handle_auth_done global_id 284098 payload 824 2024-09-10T16:31:16.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.694+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33c406e6b0 con 0x7f33c412dfb0 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f33c0018b30 tx=0x7f33c000af80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c00225f0 con 0x7f33c412dfb0 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:16.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 10 monclient: dump: 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.699 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f33c00225f0 con 0x7f33c412dfb0 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f33c0018b30 tx=0x7f33c000af80 comp rx=0 tx=0).stop 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 msgr2=0x7f33c412e390 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:16.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.697+0000 7f33b5ffb640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412dfb0 0x7f33c412e390 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33cb27f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:16.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33cb27f640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 0x7f33c41301e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33cb27f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:16.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f33c4134e00 con 0x7f33c412e8d0 2024-09-10T16:31:16.706 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 0x7f33c41301e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient: get_auth_request con 0x7f33c412e8d0 auth_method 0 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.707 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 10 monclient(hunting): handle_auth_done global_id 284101 payload 824 2024-09-10T16:31:16.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.704+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f33c40aa1e0 con 0x7f33c412e8d0 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.706+0000 7f33c881a640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 msgr2=0x7f33c41301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.706+0000 7f33c881a640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 0x7f33c41301e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33c0018b30 tx=0x7f33c0022cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c0004230 con 0x7f33c412e8d0 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 10 monclient: dump: 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.709 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f33c0004230 con 0x7f33c412e8d0 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 msgr2=0x7f33c41301e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 0x7f33c41301e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33c0018b30 tx=0x7f33c0022cd0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 msgr2=0x7f33c41301e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33b5ffb640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f33c412e8d0 0x7f33c41301e0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f33c0018b30 tx=0x7f33c0022cd0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:16.711 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 10538, 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 4, 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:16.712 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:16.713 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:16.714 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:16.715 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:16.716 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:16.717 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:16.718 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.719 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:16.720 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:16.721 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:16.722 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.723 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:16.724 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:16.725 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.728 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.728 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.728 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:16.728 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.729 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:16.730 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:16.731 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:16.732 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:16.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33a803e210 msgr2=0x7f33a80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33a803e210 0x7f33a80406d0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f33bc007ae0 tx=0x7f33bc009040 comp rx=0 tx=0).stop 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 10 monclient: shutdown 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 msgr2=0x7f33c41ccf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 --2- 172.21.15.164:0/3555962435 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f33c406d160 0x7f33c41ccf80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f33b8000c00 tx=0x7f33b80046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33c981c640 1 -- 172.21.15.164:0/3555962435 reap_dead start 2024-09-10T16:31:16.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33c981c640 1 -- 172.21.15.164:0/3555962435 reap_dead start 2024-09-10T16:31:16.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 shutdown_connections 2024-09-10T16:31:16.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 >> 172.21.15.164:0/3555962435 conn(0x7f33c40aacb0 msgr2=0x7f33c4132730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:16.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 shutdown_connections 2024-09-10T16:31:16.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.707+0000 7f33cb27f640 1 -- 172.21.15.164:0/3555962435 wait complete. 2024-09-10T16:31:16.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:16.737 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:31:16.737 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:31:16.739 INFO:tasks.rados.rados.0.smithi165.stdout:3874: finishing write tid 1 to smithi16534841-40 2024-09-10T16:31:16.739 INFO:tasks.rados.rados.0.smithi165.stdout:3869: finishing copy_from racing read to smithi16534841-22 2024-09-10T16:31:16.739 INFO:tasks.rados.rados.0.smithi165.stdout:3874: finishing write tid 2 to smithi16534841-40 2024-09-10T16:31:16.739 INFO:tasks.rados.rados.0.smithi165.stdout:3874: finishing write tid 3 to smithi16534841-40 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:3874: finishing write tid 4 to smithi16534841-40 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:3874: oid 40 updating version 0 to 1932 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:3874: oid 40 version 1932 is already newer than 1931 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:3874: oid 40 version 1932 is already newer than 1930 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 40 v 1932 (ObjNum 1332 snap 433 seq_num 1332) dirty exists 2024-09-10T16:31:16.740 INFO:tasks.rados.rados.0.smithi165.stdout:3874: left oid 40 (ObjNum 1332 snap 433 seq_num 1332) 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:3869: finishing copy_from to smithi16534841-22 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 2124 (ObjNum 1254 snap 405 seq_num 1254) dirty exists 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:3871: finishing write tid 2 to smithi16534841-30 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:3868: done (7 left) 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:3869: done (6 left) 2024-09-10T16:31:16.741 INFO:tasks.rados.rados.0.smithi165.stdout:3872: done (5 left) 2024-09-10T16:31:16.742 INFO:tasks.rados.rados.0.smithi165.stdout:3874: done (4 left) 2024-09-10T16:31:16.742 INFO:tasks.rados.rados.0.smithi165.stdout:3877: read oid 34 snap -1 2024-09-10T16:31:16.742 INFO:tasks.rados.rados.0.smithi165.stdout:3877: expect (ObjNum 1154 snap 362 seq_num 1154) 2024-09-10T16:31:16.742 INFO:tasks.rados.rados.0.smithi165.stdout:3878: write (excl) oid 13 current snap is 433 2024-09-10T16:31:16.742 INFO:tasks.rados.rados.0.smithi165.stdout:3871: finishing write tid 3 to smithi16534841-30 2024-09-10T16:31:16.743 INFO:tasks.rados.rados.0.smithi165.stdout:3878: seq_num 1333 ranges {403731=668692,1587203=587797} 2024-09-10T16:31:16.744 INFO:tasks.rados.rados.0.smithi165.stdout:3878: writing smithi16534841-13 from 403731 to 1072423 tid 1 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3878: writing smithi16534841-13 from 1587203 to 2175000 tid 2 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3879: snap_create 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3871: finishing write tid 4 to smithi16534841-30 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3871: oid 30 updating version 0 to 2578 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3871: oid 30 version 2578 is already newer than 2577 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3871: oid 30 updating version 2578 to 2579 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2579 (ObjNum 1331 snap 433 seq_num 1331) dirty exists 2024-09-10T16:31:16.749 INFO:tasks.rados.rados.0.smithi165.stdout:3871: left oid 30 (ObjNum 1331 snap 433 seq_num 1331) 2024-09-10T16:31:16.750 INFO:tasks.rados.rados.0.smithi165.stdout:3875: finishing copy_from racing read to smithi16534841-14 2024-09-10T16:31:16.750 INFO:tasks.rados.rados.0.smithi165.stdout:3875: finishing copy_from to smithi16534841-14 2024-09-10T16:31:16.750 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 14 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:31:16.750 INFO:tasks.rados.rados.0.smithi165.stdout:3873: expect (ObjNum 1064 snap 328 seq_num 1064) 2024-09-10T16:31:16.770 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1550257 2024-09-10T16:31:16.825 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 608359 2024-09-10T16:31:16.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.841+0000 7f58e93ec640 10 monclient: get_monmap_and_config 2024-09-10T16:31:16.842 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.841+0000 7f58e93ec640 10 monclient: build_initial_monmap 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.841+0000 7f58e93ec640 10 monclient: monmap: 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:16.842339+0000 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:16.842339+0000 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:16.843 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:16.844 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 Processor -- start 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 -- start start 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient: init 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58e4135ce0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:16.846 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58e40a40d0 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f58e41367d0 0x7f58e413e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-c con 0x7f58e41367d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e40a4ac0 con 0x7f58e40a40d0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58e40a1120 con 0x7f58e4135ce0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f58e40a1400 con 0x7f58e41367d0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e93ec640 10 monclient(hunting): authenticate will time out at 2255.329760s 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58dbfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f58e40a40d0 0x7f58e41360c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:43034/0 (socket says 172.21.15.164:43034) 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58dbfff640 1 -- 172.21.15.164:0/3848673162 learned_addr learned my addr 172.21.15.164:0/3848673162 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e37fe640 1 --2- 172.21.15.164:0/3848673162 >> v2:172.21.15.165:3300/0 conn(0x7f58e41367d0 0x7f58e413e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.847 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e37fe640 10 monclient(hunting): get_auth_request con 0x7f58e41367d0 auth_method 0 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.842+0000 7f58e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient(hunting): handle_auth_done global_id 284104 payload 274 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 1 -- 172.21.15.164:0/3848673162 >> v1:172.21.15.165:6789/0 conn(0x7f58e4135ce0 legacy=0x7f58e40a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 1 -- 172.21.15.164:0/3848673162 >> v1:172.21.15.164:6789/0 conn(0x7f58e40a40d0 legacy=0x7f58e41360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 1 monclient: found mon.noname-c 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:16.848 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:16.849 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.843+0000 7f58e37fe640 1 -- 172.21.15.164:0/3848673162 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e4140800 con 0x7f58e41367d0 2024-09-10T16:31:16.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e37fe640 1 -- 172.21.15.164:0/3848673162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e41367d0 msgr2=0x7f58e413e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e37fe640 1 --2- 172.21.15.164:0/3848673162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e41367d0 0x7f58e413e2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009d90 tx=0x7f58dc02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=456f38ecfc490954 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 1 -- 172.21.15.164:0/3848673162 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc007a30 con 0x7f58e41367d0 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: dump: 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.855 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 5 monclient: authenticate success, global_id 284104 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 1 -- 172.21.15.164:0/3848673162 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58dc007bd0 con 0x7f58e41367d0 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 4 set_mon_vals no callback set 2024-09-10T16:31:16.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 1 -- 172.21.15.164:0/3848673162 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58dc031400 con 0x7f58e41367d0 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e27fc640 10 monclient: dump: 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.857 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 10 monclient: shutdown 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 1 -- 172.21.15.164:0/3848673162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e41367d0 msgr2=0x7f58e413e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.846+0000 7f58e93ec640 1 --2- 172.21.15.164:0/3848673162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e41367d0 0x7f58e413e2a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f58dc009d90 tx=0x7f58dc02a490 comp rx=0 tx=0).stop 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 -- 172.21.15.164:0/3848673162 shutdown_connections 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 --2- 172.21.15.164:0/3848673162 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e41367d0 0x7f58e413e2a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 -- 172.21.15.164:0/3848673162 >> 172.21.15.164:0/3848673162 conn(0x7f58e40a4f50 msgr2=0x7f58e40a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 -- 172.21.15.164:0/3848673162 shutdown_connections 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 -- 172.21.15.164:0/3848673162 wait complete. 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 10 monclient: build_initial_monmap 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 10 monclient: monmap: 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:16.858 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 Processor -- start 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 1 -- start start 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.847+0000 7f58e93ec640 10 monclient: init 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:16.859 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-a con 0x7f58e40a40d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 0x7f58e406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-b con 0x7f58e4135ce0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 0x7f58e406d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): picked mon.noname-c con 0x7f58e41367d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f58e413eb70 con 0x7f58e40a40d0 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f58e413e9f0 con 0x7f58e4135ce0 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f58e413ecf0 con 0x7f58e41367d0 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e93ec640 10 monclient(hunting): authenticate will time out at 2255.335422s 2024-09-10T16:31:16.860 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54234/0 (socket says 172.21.15.164:54234) 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 -- 172.21.15.164:0/1232582649 learned_addr learned my addr 172.21.15.164:0/1232582649 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): get_auth_request con 0x7f58e40a40d0 auth_method 0 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 0x7f58e406d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): get_auth_request con 0x7f58e41367d0 auth_method 0 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 0x7f58e406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.861 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): get_auth_request con 0x7f58e4135ce0 auth_method 0 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient(hunting): handle_auth_done global_id 254424 payload 824 2024-09-10T16:31:16.862 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 -- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 msgr2=0x7f58e406d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 0x7f58e406d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 -- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 msgr2=0x7f58e406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 0x7f58e406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 monclient: found mon.noname-a 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e2ffd640 1 -- 172.21.15.164:0/1232582649 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58e4073270 con 0x7f58e40a40d0 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58e37fe640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 0x7f58e406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.848+0000 7f58dbfff640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 0x7f58e406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e2ffd640 1 -- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 msgr2=0x7f58e4131920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e2ffd640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c430 tx=0x7f58d400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:16.863 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d40119a0 con 0x7f58e40a40d0 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: dump: 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.864 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 5 monclient: authenticate success, global_id 254424 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f58d4017070 con 0x7f58e40a40d0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 10 monclient: _renew_subs 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58e4073550 con 0x7f58e40a40d0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58d4011d80 con 0x7f58e40a40d0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58db7fe640 10 monclient: dump: 2024-09-10T16:31:16.865 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:16.866 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 10 monclient: _renew_subs 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e1ffb640 4 set_mon_vals no callback set 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.851+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58e409c0d0 con 0x7f58e40a40d0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f58d401c460 con 0x7f58e40a40d0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58e93ec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58e93ec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58e93ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e41360c0 con 0x7f58e40a40d0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58db7fe640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 0x7f58c40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(872..872 src has 772..872) ==== 4859+0+0 (secure 0 0 0) 0x7f58d4052bc0 con 0x7f58e40a40d0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58dbfff640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 0x7f58c40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:16.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.852+0000 7f58dbfff640 10 monclient: get_auth_request con 0x7f58c403e210 auth_method 0 2024-09-10T16:31:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.855+0000 7f58dbfff640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 0x7f58c40406d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007c90 tx=0x7f58cc007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.857+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f58d4051570 con 0x7f58e40a40d0 2024-09-10T16:31:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.857+0000 7f58db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:16.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:16.857+0000 7f58db7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:3876: finishing copy_from racing read to smithi16534841-9 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:3876: finishing copy_from to smithi16534841-9 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 9 v 1724 (ObjNum 1311 snap 426 seq_num 1311) dirty exists 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:3878: finishing write tid 1 to smithi16534841-13 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:3877: expect (ObjNum 1154 snap 362 seq_num 1154) 2024-09-10T16:31:16.974 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:17.056 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.095+0000 7f58e93ec640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:31:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.095+0000 7f58e93ec640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:31:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.095+0000 7f58e93ec640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.095+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f58e409da60 con 0x7f58e40a40d0 2024-09-10T16:31:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f58db7fe640 1 -- 172.21.15.164:0/1232582649 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f58d4012420 con 0x7f58e40a40d0 2024-09-10T16:31:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f58db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:31:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f58db7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:17.098 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=1848162327,config=2586702516,health=2768383227,kv=805391194,logm=692513243}) 2024-09-10T16:31:17.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=219379897,config=2586702516,health=2095120397,kv=1634912798,logm=380781502}) 2024-09-10T16:31:17.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=1848162327,config=2586702516,health=2768383227,kv=805391194,logm=692513243}) 2024-09-10T16:31:17.099 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.096+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=26,config=2,health=11,kv=11,logm=50} crc {auth=219379897,config=575665002,health=1271696426,kv=1176791497,logm=1224123342}) 2024-09-10T16:31:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 msgr2=0x7f58c40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 0x7f58c40406d0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f58cc007c90 tx=0x7f58cc007be0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 10 monclient: shutdown 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 msgr2=0x7f58e4131920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.098+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c430 tx=0x7f58d400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 shutdown_connections 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f58c403e210 0x7f58c40406d0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58e41367d0 0x7f58e406d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f58e4135ce0 0x7f58e406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 --2- 172.21.15.164:0/1232582649 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58e40a40d0 0x7f58e4131920 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 >> 172.21.15.164:0/1232582649 conn(0x7f58e40a4f50 msgr2=0x7f58e40a6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 shutdown_connections 2024-09-10T16:31:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:17.099+0000 7f58e93ec640 1 -- 172.21.15.164:0/1232582649 wait complete. 2024-09-10T16:31:17.111 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:31:17.150 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 950041 2024-09-10T16:31:17.217 INFO:tasks.rados.rados.0.smithi165.stdout:3878: finishing write tid 2 to smithi16534841-13 2024-09-10T16:31:17.217 INFO:tasks.rados.rados.0.smithi165.stdout:3878: finishing write tid 3 to smithi16534841-13 2024-09-10T16:31:17.217 INFO:tasks.rados.rados.0.smithi165.stdout:3878: finishing write tid 4 to smithi16534841-13 2024-09-10T16:31:17.217 INFO:tasks.rados.rados.0.smithi165.stdout:3878: oid 13 updating version 0 to 2125 2024-09-10T16:31:17.217 INFO:tasks.rados.rados.0.smithi165.stdout:3878: oid 13 updating version 2125 to 2127 2024-09-10T16:31:17.218 INFO:tasks.rados.rados.0.smithi165.stdout:3878: oid 13 version 2127 is already newer than 2126 2024-09-10T16:31:17.218 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 2127 (ObjNum 1333 snap 433 seq_num 1333) dirty exists 2024-09-10T16:31:17.218 INFO:tasks.rados.rados.0.smithi165.stdout:3878: left oid 13 (ObjNum 1333 snap 433 seq_num 1333) 2024-09-10T16:31:17.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.237 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705367616}) 2024-09-10T16:31:17.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3187114659}) 2024-09-10T16:31:17.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3705367616}) 2024-09-10T16:31:17.238 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.236+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2220390393}) 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495033965}) 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=652173508}) 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2495033965}) 2024-09-10T16:31:17.591 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.590+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1204074491}) 2024-09-10T16:31:17.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899111072}) 2024-09-10T16:31:17.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1430247623}) 2024-09-10T16:31:17.592 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.593 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2899111072}) 2024-09-10T16:31:17.593 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.591+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852790600}) 2024-09-10T16:31:17.593 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.593 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370121697}) 2024-09-10T16:31:17.593 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132212425}) 2024-09-10T16:31:17.594 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.594 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1370121697}) 2024-09-10T16:31:17.594 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3811288473}) 2024-09-10T16:31:17.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823244724}) 2024-09-10T16:31:17.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1433414152}) 2024-09-10T16:31:17.595 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1823244724}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.592+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2620552428}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229000349}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=524164787}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229000349}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.593+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=844127032}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177675670}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3693080298}) 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.596 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=177675670}) 2024-09-10T16:31:17.597 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.594+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3178470937}) 2024-09-10T16:31:17.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863129400}) 2024-09-10T16:31:17.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3868328102}) 2024-09-10T16:31:17.598 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1863129400}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1377272828}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184506976}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=674003859}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2184506976}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.595+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101225458}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433919635}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967491578}) 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.599 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1433919635}) 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.596+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3770167104}) 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486538802}) 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2067412493}) 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=486538802}) 2024-09-10T16:31:17.600 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=463355459}) 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116307557}) 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1725496524}) 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116307557}) 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.597+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1581792160}) 2024-09-10T16:31:17.604 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864291311}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1730858214}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1864291311}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4104537826}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530754870}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=56268514}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=530754870}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.598+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622311454}) 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.605 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369326865}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3612653692}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=369326865}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.599+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3737473058}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820342413}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3007886315}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3820342413}) 2024-09-10T16:31:17.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2997174157}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183360998}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368026869}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2183360998}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.600+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4250849598}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296840948}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2212897755}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3296840948}) 2024-09-10T16:31:17.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.601+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1814382902}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379335656}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=309021804}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2379335656}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3109038816}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749375035}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=301025457}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1749375035}) 2024-09-10T16:31:17.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.602+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=481998715}) 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101713232}) 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=65746079}) 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4101713232}) 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.603+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3613818060}) 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443076564}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=976411380}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=443076564}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2564873123}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.604+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425794749}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2117990673}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425794749}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1206788288}) 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742934711}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3038014143}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742934711}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.605+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2317910849}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849151375}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3125230382}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1849151375}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.606+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=83264592}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1206832736}) 2024-09-10T16:31:17.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590196048}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1206832736}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.607+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3917545140}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2627663242}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766674916}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2627663242}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2123976666}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421205373}) 2024-09-10T16:31:17.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.608+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3075687200}) 2024-09-10T16:31:17.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.609+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.609+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3421205373}) 2024-09-10T16:31:17.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.609+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2526756}) 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39921673}) 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1196135692}) 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=39921673}) 2024-09-10T16:31:17.640 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.638+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1874246247}) 2024-09-10T16:31:17.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.647 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274297604}) 2024-09-10T16:31:17.648 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=853733267}) 2024-09-10T16:31:17.648 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.648 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2274297604}) 2024-09-10T16:31:17.648 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.646+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=163221641}) 2024-09-10T16:31:17.653 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.653 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695506802}) 2024-09-10T16:31:17.653 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1866084840}) 2024-09-10T16:31:17.653 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.654 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2695506802}) 2024-09-10T16:31:17.654 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.651+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765668203}) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3871: done (6 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3873: done (5 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3875: done (4 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3876: done (3 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3877: done (2 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3878: done (1 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3879: done (0 left) 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3880: write (excl) oid 46 current snap is 434 2024-09-10T16:31:17.654 INFO:tasks.rados.rados.0.smithi165.stdout:3880: seq_num 1334 ranges {420538=743941,1929332=491206} 2024-09-10T16:31:17.659 INFO:tasks.rados.rados.0.smithi165.stdout:3880: writing smithi16534841-46 from 420538 to 1164479 tid 1 2024-09-10T16:31:17.666 INFO:tasks.rados.rados.0.smithi165.stdout:3880: writing smithi16534841-46 from 1929332 to 2420538 tid 2 2024-09-10T16:31:17.666 INFO:tasks.rados.rados.0.smithi165.stdout:3881: snap_remove snap 412 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045549654}) 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=149746679}) 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3045549654}) 2024-09-10T16:31:17.780 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:17.778+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120015078}) 2024-09-10T16:31:18.112 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:31:18.112 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 10 monclient: get_monmap_and_config 2024-09-10T16:31:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 10 monclient: monmap: 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:18.214984+0000 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:18.214984+0000 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:18.216 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:18.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:18.217 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:18.217 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:18.217 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 1 Processor -- start 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 1 -- start start 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.214+0000 7f23257f3640 10 monclient: init 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient: _add_conns ranks=[4,3,1,0,2] 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 0x7f232009ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-c con 0x7f232009c180 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:18.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f232009d160 0x7f232009ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-a con 0x7f232009d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f232009f4c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f232009a430 con 0x7f232009f4c0 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f23200a24e0 con 0x7f232009d160 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f23200a2bf0 con 0x7f232009c180 2024-09-10T16:31:18.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f23257f3640 10 monclient(hunting): authenticate will time out at 2256.702415s 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f232009d160 0x7f232009ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f232009d160 0x7f232009ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54240/0 (socket says 172.21.15.164:54240) 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 -- 172.21.15.164:0/1067966931 learned_addr learned my addr 172.21.15.164:0/1067966931 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): get_auth_request con 0x7f232009d160 auth_method 0 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 1 --2- 172.21.15.164:0/1067966931 >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 0x7f232009ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 10 monclient(hunting): get_auth_request con 0x7f232009c180 auth_method 0 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient(hunting): handle_auth_done global_id 254427 payload 274 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 -- 172.21.15.164:0/1067966931 >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 msgr2=0x7f232009ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 --2- 172.21.15.164:0/1067966931 >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 0x7f232009ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 -- 172.21.15.164:0/1067966931 >> v1:172.21.15.165:6789/0 conn(0x7f232009f4c0 legacy=0x7f232013b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231e7fc640 1 -- 172.21.15.164:0/1067966931 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f232009c560 con 0x7f232009d160 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.215+0000 7f231effd640 1 --2- 172.21.15.164:0/1067966931 >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 0x7f232009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231e7fc640 1 -- 172.21.15.164:0/1067966931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 msgr2=0x7f232009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231e7fc640 1 --2- 172.21.15.164:0/1067966931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f232009ef80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2314009870 tx=0x7f231402a100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=926d092a1f3785e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 1 -- 172.21.15.164:0/1067966931 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314037070 con 0x7f232009d160 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: dump: 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.224 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f23257f3640 5 monclient: authenticate success, global_id 254427 2024-09-10T16:31:18.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 1 -- 172.21.15.164:0/1067966931 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2314032640 con 0x7f232009d160 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f23257f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f23257f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f23257f3640 4 set_mon_vals no callback set 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 1 -- 172.21.15.164:0/1067966931 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314032a10 con 0x7f232009d160 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.218+0000 7f231dffb640 10 monclient: dump: 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.226 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.227 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 10 monclient: shutdown 2024-09-10T16:31:18.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 -- 172.21.15.164:0/1067966931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 msgr2=0x7f232009ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 --2- 172.21.15.164:0/1067966931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f232009ef80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2314009870 tx=0x7f231402a100 comp rx=0 tx=0).stop 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 -- 172.21.15.164:0/1067966931 shutdown_connections 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 --2- 172.21.15.164:0/1067966931 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f232009ef80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 --2- 172.21.15.164:0/1067966931 >> v2:172.21.15.165:3300/0 conn(0x7f232009c180 0x7f232009ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 -- 172.21.15.164:0/1067966931 >> 172.21.15.164:0/1067966931 conn(0x7f23200a3110 msgr2=0x7f23200a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 -- 172.21.15.164:0/1067966931 shutdown_connections 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.219+0000 7f23257f3640 1 -- 172.21.15.164:0/1067966931 wait complete. 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.220+0000 7f23257f3640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.220+0000 7f23257f3640 10 monclient: monmap: 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:18.239 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.220+0000 7f23257f3640 1 Processor -- start 2024-09-10T16:31:18.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.220+0000 7f23257f3640 1 -- start start 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.220+0000 7f23257f3640 10 monclient: init 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 0x7f2320072a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-c con 0x7f232009c180 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-a con 0x7f232009d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f232009f4c0 0x7f232006e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): picked mon.noname-b con 0x7f232009f4c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2320074f90 con 0x7f232009d160 2024-09-10T16:31:18.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2320074e10 con 0x7f232009f4c0 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f232013cce0 con 0x7f232009c180 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f23257f3640 10 monclient(hunting): authenticate will time out at 2256.708312s 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 0x7f2320072a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54248/0 (socket says 172.21.15.164:54248) 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 -- 172.21.15.164:0/935964863 learned_addr learned my addr 172.21.15.164:0/935964863 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): get_auth_request con 0x7f232009d160 auth_method 0 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): get_auth_request con 0x7f232009c180 auth_method 0 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f232009f4c0 0x7f232006e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): get_auth_request con 0x7f232009f4c0 auth_method 0 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient(hunting): handle_auth_done global_id 254430 payload 824 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 -- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 msgr2=0x7f2320072a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 0x7f2320072a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 -- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f232009f4c0 msgr2=0x7f232006e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f232009f4c0 0x7f232006e7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231e7fc640 1 -- 172.21.15.164:0/935964863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2320073400 con 0x7f232009d160 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.221+0000 7f231effd640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 0x7f2320072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.223+0000 7f231e7fc640 1 -- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 msgr2=0x7f23201369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.223+0000 7f231e7fc640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2314009870 tx=0x7f23140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314041030 con 0x7f232009d160 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: dump: 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.246 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f231403d6f0 con 0x7f232009d160 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 5 monclient: authenticate success, global_id 254430 2024-09-10T16:31:18.247 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 10 monclient: _renew_subs 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23200a6090 con 0x7f232009d160 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2314037040 con 0x7f232009d160 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23077fe640 10 monclient: dump: 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.248 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 10 monclient: _renew_subs 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f231d7fa640 4 set_mon_vals no callback set 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.224+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23201cd5c0 con 0x7f232009d160 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2314044070 con 0x7f232009d160 2024-09-10T16:31:18.249 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23257f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23257f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23257f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2320130240 con 0x7f232009d160 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23077fe640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 0x7f230c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(873..873 src has 772..873) ==== 4771+0+0 (secure 0 0 0) 0x7f2314072fa0 con 0x7f232009d160 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f231effd640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 0x7f230c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.225+0000 7f231effd640 10 monclient: get_auth_request con 0x7f230c03e210 auth_method 0 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.228+0000 7f231effd640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 0x7f230c0406d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2310006e80 tx=0x7f2310008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.229+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f231404b070 con 0x7f232009d160 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.229+0000 7f23077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.250 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.229+0000 7f23077fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038220604}) 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1201039943}) 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4038220604}) 2024-09-10T16:31:18.345 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.343+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3249853810}) 2024-09-10T16:31:18.347 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.347 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806292370}) 2024-09-10T16:31:18.347 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=99369190}) 2024-09-10T16:31:18.347 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.347 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3806292370}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.344+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849434087}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50934079}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1281751449}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50934079}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.345+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1359934610}) 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.348 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397070142}) 2024-09-10T16:31:18.349 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1188086177}) 2024-09-10T16:31:18.349 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.349 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397070142}) 2024-09-10T16:31:18.349 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1409143403}) 2024-09-10T16:31:18.395 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.395 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547359682}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=352619704}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1547359682}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.346+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3239219002}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128741820}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2692567362}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4128741820}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.347+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2738351098}) 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.396 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095419857}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793115832}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2095419857}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.348+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=270409025}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816167982}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3340251270}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=816167982}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4045380046}) 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.397 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094767748}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2000180706}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2094767748}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.349+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2640516551}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2708175215}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=845810899}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2708175215}) 2024-09-10T16:31:18.398 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.350+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2098822523}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417845422}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=312814095}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1417845422}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3350118283}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240015753}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.351+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177925186}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=240015753}) 2024-09-10T16:31:18.399 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3273647140}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574309048}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2112110870}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1574309048}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.352+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3740351307}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391293722}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068434127}) 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.400 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=391293722}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.353+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4224754862}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859794796}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2119855226}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859794796}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3894953238}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152920827}) 2024-09-10T16:31:18.401 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1407356176}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4152920827}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.354+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2475334887}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316277463}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=981135079}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=316277463}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.355+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3788180117}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049318165}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1463870807}) 2024-09-10T16:31:18.402 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3049318165}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.356+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=754638784}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604737464}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2339049932}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1604737464}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371667261}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947275010}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2003695137}) 2024-09-10T16:31:18.403 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947275010}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.357+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4105763802}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654327637}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1793768065}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=654327637}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.358+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3035103294}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356645081}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430328663}) 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.404 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3356645081}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.359+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662679174}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239994718}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2968152957}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3239994718}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4132442861}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363324736}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619994947}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=363324736}) 2024-09-10T16:31:18.405 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.360+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3799056234}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435256778}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450639654}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435256778}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.361+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2682276834}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102714965}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2888742371}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3102714965}) 2024-09-10T16:31:18.406 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1547720949}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739416684}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684542356}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3739416684}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.362+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1629899145}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331034383}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483606306}) 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.407 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=331034383}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.363+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3144019258}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877462657}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=585309219}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877462657}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.364+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2877723076}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665388569}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4048637754}) 2024-09-10T16:31:18.408 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2665388569}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441704995}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078428302}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3345187670}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1078428302}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.365+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2187196152}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153166849}) 2024-09-10T16:31:18.409 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1407750830}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=153166849}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.366+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=874969090}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289095781}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2568792545}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=289095781}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3046891851}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493616514}) 2024-09-10T16:31:18.410 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.367+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1101706681}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1493616514}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3147817315}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528231074}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3132862848}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=528231074}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.368+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1528498940}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833623176}) 2024-09-10T16:31:18.411 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1802914989}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1833623176}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.369+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2129007109}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693654144}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3215033691}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693654144}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=401899588}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293220962}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2953799722}) 2024-09-10T16:31:18.412 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293220962}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.370+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1701382752}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192798731}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3254181323}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192798731}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.371+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2217299317}) 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.413 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40481979}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=905126357}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=40481979}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1733254403}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006058730}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1638715349}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.372+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006058730}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1375762155}) 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.414 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027368633}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147973592}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2027368633}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.373+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3417553472}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565242016}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=691832446}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1565242016}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.374+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3935977473}) 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.415 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963861164}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1228504203}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1963861164}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2885596245}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598815065}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2554416262}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1598815065}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.375+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=39286068}) 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.416 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3140853603}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=384150139}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3140853603}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.376+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197208900}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654095327}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4117520168}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654095327}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2847826256}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.377+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418560897}) 2024-09-10T16:31:18.417 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3958023906}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3418560897}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1173287208}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057058395}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2300135249}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3057058395}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.378+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=995314150}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441481481}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=693765764}) 2024-09-10T16:31:18.418 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3441481481}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.379+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2153089712}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170530357}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3067190428}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2170530357}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4079698548}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455085721}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2550013266}) 2024-09-10T16:31:18.419 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455085721}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.380+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2785536643}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919577760}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3311762262}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2919577760}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.381+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2721114528}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174165149}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3731588484}) 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.420 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1174165149}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.382+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3676432198}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086332824}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3790487853}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1086332824}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=907428134}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617317329}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=678419081}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3617317329}) 2024-09-10T16:31:18.421 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.383+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484489430}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426318629}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1714904327}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426318629}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.384+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017547613}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314834144}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=64950409}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2314834144}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3096615626}) 2024-09-10T16:31:18.422 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816757997}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4102713224}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.385+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2816757997}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3612606962}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988563940}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3499265081}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1988563940}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.386+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2960022928}) 2024-09-10T16:31:18.423 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303828303}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530462374}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3303828303}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.387+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1682076534}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691616925}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896477862}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691616925}) 2024-09-10T16:31:18.424 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=424414877}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105339323}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3349838921}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3105339323}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.388+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3909275548}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293859602}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3398720625}) 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.425 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1293859602}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.389+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154082724}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601005877}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1447965659}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2601005877}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4261047293}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853124850}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1601460867}) 2024-09-10T16:31:18.426 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853124850}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.390+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3883210031}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178729400}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=766069472}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=178729400}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.391+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2151980681}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023118199}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1765399907}) 2024-09-10T16:31:18.427 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1023118199}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.392+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2852546737}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462624540}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1749740030}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2462624540}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477726471}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205794594}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=581693325}) 2024-09-10T16:31:18.428 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2205794594}) 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.393+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2695299960}) 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148962049}) 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=729217984}) 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=148962049}) 2024-09-10T16:31:18.429 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.394+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550463845}) 2024-09-10T16:31:18.435 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.435 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857289482}) 2024-09-10T16:31:18.435 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2519481510}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3857289482}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4101270418}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606221806}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4282158268}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2606221806}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.395+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2832708638}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915515505}) 2024-09-10T16:31:18.436 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3955647098}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915515505}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.396+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1802589924}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638615500}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3258983749}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638615500}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955137638}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957252206}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=794142531}) 2024-09-10T16:31:18.437 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3957252206}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.397+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3816066317}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9} crc {logm=1980265997,mdsmap=931935893,mgr=2880672550,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4011353633,mon_config_key=3329957694}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=7} crc {logm=1546361371,mdsmap=931935893,mgr=3312148215,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2505165003,mon_config_key=2897421619}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=9} crc {logm=1980265997,mdsmap=931935893,mgr=2880672550,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=4011353633,mon_config_key=3329957694}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.398+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=72,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=5} crc {logm=3885558445,mdsmap=931935893,mgr=1154067016,mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2505165003,mon_config_key=231642688}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=3195224601,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=3014334576,osd_pg_creating=2536538039,osd_snap=3106967142}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=8,monmap=2,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=82} crc {mon_config_key=2128850857,monmap=3140514009,nvmeofgw=3140514009,osd_metadata=4124331855,osd_pg_creating=2536538039,osd_snap=2383952780}) 2024-09-10T16:31:18.438 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=6,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=83} crc {mon_config_key=3195224601,monmap=1433040829,nvmeofgw=1596038039,osd_metadata=3014334576,osd_pg_creating=2536538039,osd_snap=3106967142}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.399+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=8,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=80} crc {mon_config_key=2128850856,monmap=1433040829,nvmeofgw=3140514009,osd_metadata=947046174,osd_pg_creating=2536538039,osd_snap=1048987971}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=86,osdmap=14} crc {osd_snap=1507761005,osdmap=2164075711}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=90,osdmap=10} crc {osd_snap=582708017,osdmap=1108831496}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=86,osdmap=14} crc {osd_snap=1507761005,osdmap=2164075711}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.400+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=87,osdmap=13} crc {osd_snap=1720800903,osdmap=3936727900}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=959930897}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3774304065}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=959930897}) 2024-09-10T16:31:18.439 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.401+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1450432452}) 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=296502076}) 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=49} crc {osdmap=2245502636}) 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=48} crc {osdmap=296502076}) 2024-09-10T16:31:18.440 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:31:18.405+0000 7f7ebb95f640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=2892198482}) 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23257f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23257f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23257f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f232006fd10 con 0x7f232009d160 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23077fe640 1 -- 172.21.15.164:0/935964863 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f231402ae20 con 0x7f232009d160 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.484+0000 7f23077fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:18.486 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":244,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 msgr2=0x7f230c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 0x7f230c0406d0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f2310006e80 tx=0x7f2310008040 comp rx=0 tx=0).stop 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 10 monclient: shutdown 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 msgr2=0x7f23201369e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f2314009870 tx=0x7f23140046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 shutdown_connections 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f230c03e210 0x7f230c0406d0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f232009f4c0 0x7f232006e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f232009d160 0x7f23201369e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 --2- 172.21.15.164:0/935964863 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f232009c180 0x7f2320072a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 >> 172.21.15.164:0/935964863 conn(0x7f23200a3110 msgr2=0x7f232009ae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 shutdown_connections 2024-09-10T16:31:18.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.487+0000 7f23257f3640 1 -- 172.21.15.164:0/935964863 wait complete. 2024-09-10T16:31:18.499 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:31:18.499 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T16:31:18.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:31:18.499 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:31:18.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 10 monclient: get_monmap_and_config 2024-09-10T16:31:18.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 10 monclient: monmap: 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:18.597053+0000 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:18.597053+0000 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:18.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 1 Processor -- start 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 1 -- start start 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.596+0000 7ff58b4b1640 10 monclient: init 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff58406cb70 0x7ff584134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-c con 0x7ff58406cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:18.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff5840a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5840a4ab0 0x7ff5841383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5840a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff58409ce50 con 0x7ff5840a46a0 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff58406db60 con 0x7ff5840a4ab0 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff58406cf50 con 0x7ff58406cb70 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58b4b1640 10 monclient(hunting): authenticate will time out at 2257.084498s 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5840a4ab0 0x7ff5841383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff5840a4ab0 0x7ff5841383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54264/0 (socket says 172.21.15.164:54264) 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 -- 172.21.15.164:0/4132268592 learned_addr learned my addr 172.21.15.164:0/4132268592 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): get_auth_request con 0x7ff5840a4ab0 auth_method 0 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58924d640 1 --2- 172.21.15.164:0/4132268592 >> v2:172.21.15.165:3300/0 conn(0x7ff58406cb70 0x7ff584134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58924d640 10 monclient(hunting): get_auth_request con 0x7ff58406cb70 auth_method 0 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58924d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58924d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff58924d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient(hunting): handle_auth_done global_id 254433 payload 274 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 -- 172.21.15.164:0/4132268592 >> v2:172.21.15.165:3300/0 conn(0x7ff58406cb70 msgr2=0x7ff584134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 --2- 172.21.15.164:0/4132268592 >> v2:172.21.15.165:3300/0 conn(0x7ff58406cb70 0x7ff584134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 -- 172.21.15.164:0/4132268592 >> v1:172.21.15.165:6789/0 conn(0x7ff5840a46a0 legacy=0x7ff584137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.597+0000 7ff589a4e640 1 -- 172.21.15.164:0/4132268592 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58409cb70 con 0x7ff5840a4ab0 2024-09-10T16:31:18.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff589a4e640 1 -- 172.21.15.164:0/4132268592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a4ab0 msgr2=0x7ff5841383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff589a4e640 1 --2- 172.21.15.164:0/4132268592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a4ab0 0x7ff5841383b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0098a0 tx=0x7ff57c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbcf9fc80904089 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 1 -- 172.21.15.164:0/4132268592 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c037070 con 0x7ff5840a4ab0 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: dump: 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.612 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 1 -- 172.21.15.164:0/4132268592 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff57c032640 con 0x7ff5840a4ab0 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 1 -- 172.21.15.164:0/4132268592 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff57c032a10 con 0x7ff5840a4ab0 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff58b4b1640 5 monclient: authenticate success, global_id 254433 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff58b4b1640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff58b4b1640 4 set_mon_vals no callback set 2024-09-10T16:31:18.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.600+0000 7ff583fff640 10 monclient: dump: 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 10 monclient: shutdown 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 -- 172.21.15.164:0/4132268592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a4ab0 msgr2=0x7ff5841383b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/4132268592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a4ab0 0x7ff5841383b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7ff57c0098a0 tx=0x7ff57c02a110 comp rx=0 tx=0).stop 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 -- 172.21.15.164:0/4132268592 shutdown_connections 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/4132268592 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a4ab0 0x7ff5841383b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/4132268592 >> v2:172.21.15.165:3300/0 conn(0x7ff58406cb70 0x7ff584134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 -- 172.21.15.164:0/4132268592 >> 172.21.15.164:0/4132268592 conn(0x7ff5840aa900 msgr2=0x7ff5840aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 -- 172.21.15.164:0/4132268592 shutdown_connections 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.601+0000 7ff58b4b1640 1 -- 172.21.15.164:0/4132268592 wait complete. 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient: monmap: 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.615 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 Processor -- start 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 -- start start 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient: init 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:31:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 0x7ff58412e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-c con 0x7ff58406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-a con 0x7ff5840a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5840a4ab0 0x7ff5841cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): picked mon.noname-b con 0x7ff5840a4ab0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff584138a70 con 0x7ff5840a46a0 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff5841388f0 con 0x7ff5840a4ab0 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff584140040 con 0x7ff58406cb70 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58b4b1640 10 monclient(hunting): authenticate will time out at 2257.089868s 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff588a4c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58924d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 0x7ff58412e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff58924d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 0x7ff58412e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47458/0 (socket says 172.21.15.164:47458) 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.602+0000 7ff588a4c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54270/0 (socket says 172.21.15.164:54270) 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 1 -- 172.21.15.164:0/2457740537 learned_addr learned my addr 172.21.15.164:0/2457740537 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): get_auth_request con 0x7ff58406cb70 auth_method 0 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): get_auth_request con 0x7ff5840a46a0 auth_method 0 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff589a4e640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5840a4ab0 0x7ff5841cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient(hunting): handle_auth_done global_id 254436 payload 824 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 msgr2=0x7ff58412e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 0x7ff58412e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5840a4ab0 msgr2=0x7ff5841cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5840a4ab0 0x7ff5841cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff588a4c640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff58413e8b0 con 0x7ff5840a46a0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff589a4e640 10 monclient: get_auth_request con 0x7ff5840a4ab0 auth_method 0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff589a4e640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff5840a4ab0 0x7ff5841cd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.603+0000 7ff58924d640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff58406cb70 0x7ff58412e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff588a4c640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 msgr2=0x7ff58412e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff588a4c640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c410 tx=0x7ff57800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff578017070 con 0x7ff5840a46a0 2024-09-10T16:31:18.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: dump: 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 5 monclient: authenticate success, global_id 254436 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff578005230 con 0x7ff5840a46a0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 10 monclient: _renew_subs 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5841344f0 con 0x7ff5840a46a0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5780117a0 con 0x7ff5840a46a0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff581ffb640 10 monclient: dump: 2024-09-10T16:31:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 10 monclient: _renew_subs 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff5837fe640 4 set_mon_vals no callback set 2024-09-10T16:31:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.605+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff58413e5d0 con 0x7ff5840a46a0 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58b4b1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 0x7ff5841cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff58413d390 con 0x7ff5841cd710 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff57801a070 con 0x7ff5840a46a0 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58924d640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 0x7ff5841cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58924d640 10 monclient: get_auth_request con 0x7ff5841cd710 auth_method 0 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58924d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58924d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.606+0000 7ff58924d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff581ffb640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57003e260 0x7ff570040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff58924d640 10 monclient(hunting): handle_auth_done global_id 254439 payload 824 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(873..873 src has 772..873) ==== 4771+0+0 (secure 0 0 0) 0x7ff578051ff0 con 0x7ff5840a46a0 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff58924d640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5841ce030 con 0x7ff5841cd710 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff589a4e640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57003e260 0x7ff570040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.607+0000 7ff589a4e640 10 monclient: get_auth_request con 0x7ff57003e260 auth_method 0 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff58924d640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 msgr2=0x7ff5841cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff58924d640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 0x7ff5841cdaf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff574007360 tx=0x7ff574007820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574014880 con 0x7ff5841cd710 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 10 monclient: dump: 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff57401a070 con 0x7ff5841cd710 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 msgr2=0x7ff5841cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 0x7ff5841cdaf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff574007360 tx=0x7ff574007820 comp rx=0 tx=0).stop 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 msgr2=0x7ff5841cdaf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff581ffb640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5841cd710 0x7ff5841cdaf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.609+0000 7ff589a4e640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57003e260 0x7ff570040720 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff58406e110 tx=0x7ff57c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58b4b1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 0x7ff584130660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58b4b1640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff584134f70 con 0x7ff584130250 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58924d640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 0x7ff584130660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58924d640 10 monclient: get_auth_request con 0x7ff584130250 auth_method 0 2024-09-10T16:31:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58924d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58924d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.615+0000 7ff58924d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.616+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.616+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.616+0000 7ff58924d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.616+0000 7ff58924d640 10 monclient(hunting): handle_auth_done global_id 254442 payload 824 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.616+0000 7ff58924d640 1 -- 172.21.15.164:0/2457740537 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff58413cf10 con 0x7ff584130250 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff58924d640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 msgr2=0x7ff584130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff58924d640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 0x7ff584130660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff574007360 tx=0x7ff574004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff574015420 con 0x7ff584130250 2024-09-10T16:31:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 10 monclient: dump: 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4723 (secure 0 0 0) 0x7ff574015420 con 0x7ff584130250 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 msgr2=0x7ff584130660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 0x7ff584130660 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff574007360 tx=0x7ff574004a40 comp rx=0 tx=0).stop 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 msgr2=0x7ff584130660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.618+0000 7ff581ffb640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff584130250 0x7ff584130660 secure :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff574007360 tx=0x7ff574004a40 comp rx=0 tx=0).stop 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:18.632 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 167269, 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 6, 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.633 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.634 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:18.635 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:18.636 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.637 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.638 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.639 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.640 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:18.641 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.642 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.643 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:18.644 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 6 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:18.645 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57003e260 msgr2=0x7ff570040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff57003e260 0x7ff570040720 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7ff58406e110 tx=0x7ff57c02e000 comp rx=0 tx=0).stop 2024-09-10T16:31:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 10 monclient: shutdown 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 msgr2=0x7ff58412e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7ff57800c410 tx=0x7ff57800c8d0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff589a4e640 1 -- 172.21.15.164:0/2457740537 reap_dead start 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 shutdown_connections 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 --2- 172.21.15.164:0/2457740537 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff5840a46a0 0x7ff58412e9b0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 >> 172.21.15.164:0/2457740537 conn(0x7ff5840aa900 msgr2=0x7ff5840a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 shutdown_connections 2024-09-10T16:31:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.619+0000 7ff58b4b1640 1 -- 172.21.15.164:0/2457740537 wait complete. 2024-09-10T16:31:18.647 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:18.648 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:31:18.648 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:31:18.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 10 monclient: get_monmap_and_config 2024-09-10T16:31:18.748 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 10 monclient: monmap: 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:18.748170+0000 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:18.748170+0000 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:18.749 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 1 Processor -- start 2024-09-10T16:31:18.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 1 -- start start 2024-09-10T16:31:18.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.747+0000 7fa2bc553640 10 monclient: init 2024-09-10T16:31:18.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient: _add_conns ranks=[2,3,1,4,0] 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 0x7fa2b4134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b406cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-a con 0x7fa2b40a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2b4137ca0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2b409ce50 con 0x7fa2b4137ca0 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa2b406db60 con 0x7fa2b40a46a0 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b406cf50 con 0x7fa2b406cb70 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2bc553640 10 monclient(hunting): authenticate will time out at 2257.235565s 2024-09-10T16:31:18.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 0x7fa2b4134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54304/0 (socket says 172.21.15.164:54304) 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 1 -- 172.21.15.164:0/3129394103 learned_addr learned my addr 172.21.15.164:0/3129394103 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): get_auth_request con 0x7fa2b406cb70 auth_method 0 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request con 0x7fa2b40a46a0 auth_method 0 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_done global_id 274208 payload 274 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/3129394103 >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 msgr2=0x7fa2b40a4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 --2- 172.21.15.164:0/3129394103 >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/3129394103 >> v1:172.21.15.165:6789/0 conn(0x7fa2b4137ca0 legacy=0x7fa2b4138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 monclient: found mon.noname-b 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.748+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/3129394103 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b409cb70 con 0x7fa2b406cb70 2024-09-10T16:31:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.749+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/3129394103 >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/3129394103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 msgr2=0x7fa2b4134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2ba2ef640 1 --2- 172.21.15.164:0/3129394103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 0x7fa2b4134f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097a0 tx=0x7fa2b0029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9cd5086d58e6e491 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 1 -- 172.21.15.164:0/3129394103 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0037070 con 0x7fa2b406cb70 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: dump: 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2bc553640 5 monclient: authenticate success, global_id 274208 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 1 -- 172.21.15.164:0/3129394103 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b0032440 con 0x7fa2b406cb70 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2bc553640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 1 -- 172.21.15.164:0/3129394103 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0032810 con 0x7fa2b406cb70 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2bc553640 4 set_mon_vals no callback set 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.751+0000 7fa2b92ed640 10 monclient: dump: 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 10 monclient: shutdown 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 -- 172.21.15.164:0/3129394103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 msgr2=0x7fa2b4134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 --2- 172.21.15.164:0/3129394103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 0x7fa2b4134f70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa2b00097a0 tx=0x7fa2b0029fd0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 -- 172.21.15.164:0/3129394103 shutdown_connections 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 --2- 172.21.15.164:0/3129394103 >> v2:172.21.15.164:3300/0 conn(0x7fa2b40a46a0 0x7fa2b40a4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 --2- 172.21.15.164:0/3129394103 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b406cb70 0x7fa2b4134f70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 -- 172.21.15.164:0/3129394103 >> 172.21.15.164:0/3129394103 conn(0x7fa2b40aa900 msgr2=0x7fa2b40aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 -- 172.21.15.164:0/3129394103 shutdown_connections 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.752+0000 7fa2bc553640 1 -- 172.21.15.164:0/3129394103 wait complete. 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient: monmap: 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 1 Processor -- start 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 1 -- start start 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient: init 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 0x7fa2b412e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-a con 0x7fa2b406cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b40a46a0 0x7fa2b412e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-b con 0x7fa2b40a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.753+0000 7fa2bc553640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4137ca0 0x7fa2b41cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): picked mon.noname-c con 0x7fa2b4137ca0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b41389c0 con 0x7fa2b406cb70 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa2b4138840 con 0x7fa2b40a46a0 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b4140040 con 0x7fa2b4137ca0 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2bc553640 10 monclient(hunting): authenticate will time out at 2257.241032s 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 0x7fa2b412e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 0x7fa2b412e370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54318/0 (socket says 172.21.15.164:54318) 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/2492999716 learned_addr learned my addr 172.21.15.164:0/2492999716 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): get_auth_request con 0x7fa2b406cb70 auth_method 0 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4137ca0 0x7fa2b41cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): get_auth_request con 0x7fa2b4137ca0 auth_method 0 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2baaf0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.770 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b40a46a0 0x7fa2b412e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request con 0x7fa2b40a46a0 auth_method 0 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2b9aee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2b9aee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient(hunting): handle_auth_done global_id 254448 payload 824 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4137ca0 msgr2=0x7fa2b41cd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4137ca0 0x7fa2b41cd1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b40a46a0 msgr2=0x7fa2b412e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa2b40a46a0 0x7fa2b412e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.771 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.754+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2b406e720 con 0x7fa2b406cb70 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2ba2ef640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 msgr2=0x7fa2b412e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2ba2ef640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 0x7fa2b412e370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b0037040 con 0x7fa2b406cb70 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: dump: 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.772 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2b0042070 con 0x7fa2b406cb70 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 5 monclient: authenticate success, global_id 254448 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 10 monclient: _renew_subs 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2b40a2f80 con 0x7fa2b406cb70 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2b003bb00 con 0x7fa2b406cb70 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2aaffd640 10 monclient: dump: 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 10 monclient: _renew_subs 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2b409f720 con 0x7fa2b406cb70 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.756+0000 7fa2b8aec640 4 set_mon_vals no callback set 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2bc553640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2bc553640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 0x7fa2b41cdaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b41334f0 con 0x7fa2b41cd710 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 0x7fa2b41cdaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient: get_auth_request con 0x7fa2b41cd710 auth_method 0 2024-09-10T16:31:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa2b0047460 con 0x7fa2b406cb70 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_done global_id 274214 payload 824 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2b9aee640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2b41ce030 con 0x7fa2b41cd710 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2aaffd640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa29803e240 0x7fa298040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.757+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(873..873 src has 772..873) ==== 4771+0+0 (secure 0 0 0) 0x7fa2b0073480 con 0x7fa2b406cb70 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.758+0000 7fa2baaf0640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa29803e240 0x7fa298040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.758+0000 7fa2baaf0640 10 monclient: get_auth_request con 0x7fa29803e240 auth_method 0 2024-09-10T16:31:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2b9aee640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 msgr2=0x7fa2b41cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 0x7fa2b41cdaf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4011a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a400fcc0 con 0x7fa2b41cd710 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 10 monclient: dump: 2024-09-10T16:31:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa2a400fcc0 con 0x7fa2b41cd710 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 msgr2=0x7fa2b41cdaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 0x7fa2b41cdaf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4011a50 comp rx=0 tx=0).stop 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 msgr2=0x7fa2b41cdaf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2aaffd640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b41cd710 0x7fa2b41cdaf0 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4011a50 comp rx=0 tx=0).stop 2024-09-10T16:31:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.760+0000 7fa2baaf0640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa29803e240 0x7fa298040700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40a1ad0 tx=0x7fa2ac003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2bc553640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:18.783 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2bc553640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 0x7fa2b4130630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2bc553640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2b4134f70 con 0x7fa2b4130250 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 0x7fa2b4130630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient: get_auth_request con 0x7fa2b4130250 auth_method 0 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 10 monclient(hunting): handle_auth_done global_id 274217 payload 824 2024-09-10T16:31:18.784 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.766+0000 7fa2b9aee640 1 -- 172.21.15.164:0/2492999716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa2b4132f30 con 0x7fa2b4130250 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2b9aee640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 msgr2=0x7fa2b4130630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2b9aee640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 0x7fa2b4130630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a4006460 con 0x7fa2b4130250 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 10 monclient: dump: 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.785 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fa2a4006460 con 0x7fa2b4130250 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 msgr2=0x7fa2b4130630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 0x7fa2b4130630 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4004730 comp rx=0 tx=0).stop 2024-09-10T16:31:18.786 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 msgr2=0x7fa2b4130630 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2aaffd640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2b4130250 0x7fa2b4130630 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa2a4011590 tx=0x7fa2a4004730 comp rx=0 tx=0).stop 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 56012, 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:18.787 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 6, 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.788 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.789 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.790 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.791 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:18.792 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.793 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.794 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.795 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.796 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.797 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.798 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:18.799 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:18.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa29803e240 msgr2=0x7fa298040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa29803e240 0x7fa298040700 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40a1ad0 tx=0x7fa2ac003d10 comp rx=0 tx=0).stop 2024-09-10T16:31:18.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 10 monclient: shutdown 2024-09-10T16:31:18.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.801 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 msgr2=0x7fa2b412e370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 --2- 172.21.15.164:0/2492999716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa2b406cb70 0x7fa2b412e370 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0000c00 tx=0x7fa2b00046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2baaf0640 1 -- 172.21.15.164:0/2492999716 reap_dead start 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2baaf0640 1 -- 172.21.15.164:0/2492999716 reap_dead start 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 shutdown_connections 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 >> 172.21.15.164:0/2492999716 conn(0x7fa2b40aa900 msgr2=0x7fa2b40a73b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 shutdown_connections 2024-09-10T16:31:18.802 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.769+0000 7fa2bc553640 1 -- 172.21.15.164:0/2492999716 wait complete. 2024-09-10T16:31:18.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:18.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:31:18.803 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3881: done (1 left) 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3882: delete oid 27 current snap is 434 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3880: finishing write tid 1 to smithi16534841-46 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3880: finishing write tid 2 to smithi16534841-46 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3880: finishing write tid 3 to smithi16534841-46 2024-09-10T16:31:18.893 INFO:tasks.rados.rados.0.smithi165.stdout:3880: finishing write tid 4 to smithi16534841-46 2024-09-10T16:31:18.894 INFO:tasks.rados.rados.0.smithi165.stdout:3880: oid 46 updating version 0 to 1853 2024-09-10T16:31:18.894 INFO:tasks.rados.rados.0.smithi165.stdout:3880: oid 46 updating version 1853 to 1854 2024-09-10T16:31:18.894 INFO:tasks.rados.rados.0.smithi165.stdout:3880: oid 46 updating version 1854 to 1855 2024-09-10T16:31:18.894 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 46 v 1855 (ObjNum 1334 snap 434 seq_num 1334) dirty exists 2024-09-10T16:31:18.894 INFO:tasks.rados.rados.0.smithi165.stdout:3880: left oid 46 (ObjNum 1334 snap 434 seq_num 1334) 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3880: done (1 left) 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3882: done (0 left) 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3883: read oid 31 snap -1 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3883: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3884: copy_from oid 43 from oid 42 current snap is 434 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3885: write (excl) oid 21 current snap is 434 2024-09-10T16:31:18.902 INFO:tasks.rados.rados.0.smithi165.stdout:3885: seq_num 1336 ranges {454152=494439,1646296=735124} 2024-09-10T16:31:18.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.906+0000 7f7b63a7f640 10 monclient: get_monmap_and_config 2024-09-10T16:31:18.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.906+0000 7f7b63a7f640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.907 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.906+0000 7f7b63a7f640 10 monclient: monmap: 2024-09-10T16:31:18.907 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.907 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:18.907142+0000 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:18.907142+0000 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:18.908 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.909 INFO:tasks.rados.rados.0.smithi165.stdout:3885: writing smithi16534841-21 from 454152 to 948591 tid 1 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.906+0000 7f7b63a7f640 1 Processor -- start 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 1 -- start start 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient: init 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7b5c06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7b5c134f70 0x7f7b5c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7b5c13d560 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b5c09ce50 con 0x7f7b5c06cb70 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b5c135380 con 0x7f7b5c13d560 2024-09-10T16:31:18.911 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f7b5c06db60 con 0x7f7b5c134f70 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b63a7f640 10 monclient(hunting): authenticate will time out at 2257.394835s 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b6101a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7b5c134f70 0x7f7b5c13d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b6101a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f7b5c134f70 0x7f7b5c13d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54320/0 (socket says 172.21.15.164:54320) 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.907+0000 7f7b6181b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7b5c06cb70 0x7f7b5c0a46a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:43038/0 (socket says 172.21.15.164:43038) 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 1 -- 172.21.15.164:0/2330679745 learned_addr learned my addr 172.21.15.164:0/2330679745 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): get_auth_request con 0x7f7b5c134f70 auth_method 0 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.912 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient(hunting): handle_auth_done global_id 254451 payload 274 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 1 -- 172.21.15.164:0/2330679745 >> v1:172.21.15.165:6789/0 conn(0x7f7b5c13d560 legacy=0x7f7b5c13f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 1 -- 172.21.15.164:0/2330679745 >> v1:172.21.15.164:6789/0 conn(0x7f7b5c06cb70 legacy=0x7f7b5c0a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.913 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.908+0000 7f7b6101a640 1 -- 172.21.15.164:0/2330679745 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5c09cb70 con 0x7f7b5c134f70 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b6101a640 1 -- 172.21.15.164:0/2330679745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c134f70 msgr2=0x7f7b5c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b6101a640 1 --2- 172.21.15.164:0/2330679745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c134f70 0x7f7b5c13d020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0098a0 tx=0x7f7b4c02a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=855ac4d8fefa460f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 1 -- 172.21.15.164:0/2330679745 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c037070 con 0x7f7b5c134f70 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: dump: 2024-09-10T16:31:18.920 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.921 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 1 -- 172.21.15.164:0/2330679745 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b4c032640 con 0x7f7b5c134f70 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b63a7f640 5 monclient: authenticate success, global_id 254451 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b63a7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b63a7f640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 1 -- 172.21.15.164:0/2330679745 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b4c032a10 con 0x7f7b5c134f70 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b63a7f640 4 set_mon_vals no callback set 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.911+0000 7f7b60819640 10 monclient: dump: 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.922 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 10 monclient: shutdown 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 -- 172.21.15.164:0/2330679745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c134f70 msgr2=0x7f7b5c13d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/2330679745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c134f70 0x7f7b5c13d020 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b4c0098a0 tx=0x7f7b4c02a0d0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 -- 172.21.15.164:0/2330679745 shutdown_connections 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/2330679745 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c134f70 0x7f7b5c13d020 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.923 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 -- 172.21.15.164:0/2330679745 >> 172.21.15.164:0/2330679745 conn(0x7f7b5c0aa900 msgr2=0x7f7b5c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 -- 172.21.15.164:0/2330679745 shutdown_connections 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 1 -- 172.21.15.164:0/2330679745 wait complete. 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 10 monclient: build_initial_monmap 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.912+0000 7f7b63a7f640 10 monclient: monmap: 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:18.924 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 Processor -- start 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 -- start start 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient: init 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 0x7f7b5c1cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-c con 0x7f7b5c06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c134f70 0x7f7b5c0a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-b con 0x7f7b5c134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): picked mon.noname-a con 0x7f7b5c13d560 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:18.925 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c0a4db0 con 0x7f7b5c13d560 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c06cf50 con 0x7f7b5c134f70 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7b5c141070 con 0x7f7b5c06cb70 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b63a7f640 10 monclient(hunting): authenticate will time out at 2257.400741s 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 0x7f7b5c1cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:54326/0 (socket says 172.21.15.164:54326) 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 learned_addr learned my addr 172.21.15.164:0/735522466 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 0x7f7b5c1cd530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47514/0 (socket says 172.21.15.164:47514) 2024-09-10T16:31:18.926 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): get_auth_request con 0x7f7b5c13d560 auth_method 0 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 10 monclient(hunting): get_auth_request con 0x7f7b5c06cb70 auth_method 0 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6101a640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c134f70 0x7f7b5c0a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6201c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.913+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.927 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): get_auth_request con 0x7f7b5c134f70 auth_method 0 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6101a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 10 monclient(hunting): handle_auth_done global_id 254454 payload 824 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 msgr2=0x7f7b5c1cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 0x7f7b5c1cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c134f70 msgr2=0x7f7b5c0a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c134f70 0x7f7b5c0a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.928 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 10 monclient: _finish_hunting 0 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 monclient: found mon.noname-a 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b5c09dbe0 con 0x7f7b5c13d560 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.914+0000 7f7b6181b640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f7b5c06cb70 0x7f7b5c1cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 msgr2=0x7f7b5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b6201c640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400c700 tx=0x7f7b5400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54016020 con 0x7f7b5c13d560 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: dump: 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.929 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5400ac60 con 0x7f7b5c13d560 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 5 monclient: authenticate success, global_id 254454 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:18.930 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b54012660 con 0x7f7b5c13d560 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 10 monclient: _renew_subs 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b5c06e720 con 0x7f7b5c13d560 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b527fc640 10 monclient: dump: 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.931 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b53fff640 4 set_mon_vals no callback set 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 10 monclient: _renew_subs 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.916+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b5c09e180 con 0x7f7b5c13d560 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b63a7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7b54019020 con 0x7f7b5c13d560 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 0x7f7b5c13a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c0a46a0 con 0x7f7b5c13a060 2024-09-10T16:31:18.932 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b527fc640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b4003e210 0x7f7b400406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(874..874 src has 772..874) ==== 4815+0+0 (secure 0 0 0) 0x7f7b54052f60 con 0x7f7b5c13d560 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b6181b640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 0x7f7b5c13a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.917+0000 7f7b6181b640 10 monclient: get_auth_request con 0x7f7b5c13a060 auth_method 0 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6101a640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b4003e210 0x7f7b400406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6101a640 10 monclient: get_auth_request con 0x7f7b4003e210 auth_method 0 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 10 monclient(hunting): handle_auth_done global_id 284125 payload 824 2024-09-10T16:31:18.933 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.918+0000 7f7b6181b640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b5c09e520 con 0x7f7b5c13a060 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3885: writing smithi16534841-21 from 1646296 to 2381420 tid 2 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3886: delete oid 32 current snap is 434 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3884: finishing copy_from to smithi16534841-43 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3884: got expected ENOENT (src dne) 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3884: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:3883: expect (ObjNum 1166 snap 366 seq_num 1166) 2024-09-10T16:31:18.935 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:18.937 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b6181b640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 msgr2=0x7f7b5c13a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b6181b640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 0x7f7b5c13a440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800eae0 tx=0x7f7b58009eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b6101a640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b4003e210 0x7f7b400406d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c06e110 tx=0x7f7b4c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58017070 con 0x7f7b5c13a060 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.920+0000 7f7b527fc640 10 monclient: dump: 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.938 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b5800cd60 con 0x7f7b5c13a060 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 msgr2=0x7f7b5c13a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 0x7f7b5c13a440 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800eae0 tx=0x7f7b58009eb0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 msgr2=0x7f7b5c13a440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.921+0000 7f7b527fc640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a060 0x7f7b5c13a440 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800eae0 tx=0x7f7b58009eb0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b63a7f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:18.939 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 0x7f7b5c13d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b63a7f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f7b5c0770f0 con 0x7f7b5c13a980 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 0x7f7b5c13d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient: get_auth_request con 0x7f7b5c13a980 auth_method 0 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 10 monclient(hunting): handle_auth_done global_id 284128 payload 824 2024-09-10T16:31:18.940 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.927+0000 7f7b6181b640 1 -- 172.21.15.164:0/735522466 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b5c13ad90 con 0x7f7b5c13a980 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b6181b640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 msgr2=0x7f7b5c13d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b6181b640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 0x7f7b5c13d940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b58019000 tx=0x7f7b580058c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58017070 con 0x7f7b5c13a980 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 10 monclient: dump: 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:18.941 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f7b58005030 con 0x7f7b5c13a980 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 msgr2=0x7f7b5c13d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 0x7f7b5c13d940 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b58019000 tx=0x7f7b580058c0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 msgr2=0x7f7b5c13d940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b527fc640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f7b5c13a980 0x7f7b5c13d940 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7b58019000 tx=0x7f7b580058c0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.942 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 244, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 12761, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 7, 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.943 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.944 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.945 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:18.946 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:18.947 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.948 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.949 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.950 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:18.951 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.952 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.953 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:18.954 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b4003e210 msgr2=0x7f7b400406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f7b4003e210 0x7f7b400406d0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f7b5c06e110 tx=0x7f7b4c02e000 comp rx=0 tx=0).stop 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 10 monclient: shutdown 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 msgr2=0x7f7b5c0a3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b6201c640 1 -- 172.21.15.164:0/735522466 reap_dead start 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.930+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f7b5400c700 tx=0x7f7b5400cbc0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.931+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 shutdown_connections 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.931+0000 7f7b63a7f640 1 --2- 172.21.15.164:0/735522466 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7b5c13d560 0x7f7b5c0a3ef0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.931+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 >> 172.21.15.164:0/735522466 conn(0x7f7b5c0aa900 msgr2=0x7f7b5c1335f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:18.956 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.931+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 shutdown_connections 2024-09-10T16:31:18.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:18.931+0000 7f7b63a7f640 1 -- 172.21.15.164:0/735522466 wait complete. 2024-09-10T16:31:18.957 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:31:18.957 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T16:31:18.957 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T16:31:18.957 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T16:31:18.957 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T16:31:18.957 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:31:18.958 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:31:18.971 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:31:18.971 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T16:31:18.972 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:19.000 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: finishing write tid 1 to smithi16534841-21 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: finishing write tid 2 to smithi16534841-21 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: finishing write tid 3 to smithi16534841-21 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: finishing write tid 4 to smithi16534841-21 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: oid 21 updating version 0 to 2202 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: oid 21 updating version 2202 to 2203 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: oid 21 version 2203 is already newer than 2197 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 2203 (ObjNum 1336 snap 434 seq_num 1336) dirty exists 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: left oid 21 (ObjNum 1336 snap 434 seq_num 1336) 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3883: done (3 left) 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3884: done (2 left) 2024-09-10T16:31:19.040 INFO:tasks.rados.rados.0.smithi165.stdout:3885: done (1 left) 2024-09-10T16:31:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:3886: done (0 left) 2024-09-10T16:31:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:3887: copy_from oid 35 from oid 14 current snap is 434 2024-09-10T16:31:19.041 INFO:tasks.rados.rados.0.smithi165.stdout:3888: delete oid 17 current snap is 434 2024-09-10T16:31:19.043 INFO:tasks.rados.rados.0.smithi165.stdout:3888: done (1 left) 2024-09-10T16:31:19.043 INFO:tasks.rados.rados.0.smithi165.stdout:3889: write oid 47 current snap is 434 2024-09-10T16:31:19.043 INFO:tasks.rados.rados.0.smithi165.stdout:3889: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2024-09-10T16:31:19.049 INFO:tasks.rados.rados.0.smithi165.stdout:3889: writing smithi16534841-47 from 504573 to 941112 tid 1 2024-09-10T16:31:19.055 INFO:tasks.rados.rados.0.smithi165.stdout:3889: writing smithi16534841-47 from 1421742 to 2020546 tid 2 2024-09-10T16:31:19.058 INFO:tasks.rados.rados.0.smithi165.stdout:3889: writing smithi16534841-47 from 2437505 to 2504573 tid 3 2024-09-10T16:31:19.058 INFO:tasks.rados.rados.0.smithi165.stdout:3890: read oid 11 snap -1 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:3890: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:3891: write (excl) oid 43 current snap is 434 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:3887: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:3887: finishing copy_from to smithi16534841-35 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 1887 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:31:19.059 INFO:tasks.rados.rados.0.smithi165.stdout:3889: finishing write tid 1 to smithi16534841-47 2024-09-10T16:31:19.060 INFO:tasks.rados.rados.0.smithi165.stdout:3891: seq_num 1340 ranges {521380=511788,1480224=742462} 2024-09-10T16:31:19.063 INFO:tasks.rados.rados.0.smithi165.stdout:3891: writing smithi16534841-43 from 521380 to 1033168 tid 1 2024-09-10T16:31:19.071 INFO:tasks.rados.rados.0.smithi165.stdout:3891: writing smithi16534841-43 from 1480224 to 2222686 tid 2 2024-09-10T16:31:19.071 INFO:tasks.rados.rados.0.smithi165.stdout:3887: done (3 left) 2024-09-10T16:31:19.071 INFO:tasks.rados.rados.0.smithi165.stdout:3892: write oid 26 current snap is 434 2024-09-10T16:31:19.071 INFO:tasks.rados.rados.0.smithi165.stdout:3892: seq_num 1341 ranges {538187=587037,1538706=602473} 2024-09-10T16:31:19.077 INFO:tasks.rados.rados.0.smithi165.stdout:3892: writing smithi16534841-26 from 538187 to 1125224 tid 1 2024-09-10T16:31:19.082 INFO:tasks.rados.rados.0.smithi165.stdout:3892: writing smithi16534841-26 from 1538706 to 2141179 tid 2 2024-09-10T16:31:19.082 INFO:tasks.rados.rados.0.smithi165.stdout:3893: snap_remove snap 391 2024-09-10T16:31:19.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.111+0000 7ff51bc8c640 10 monclient: get_monmap_and_config 2024-09-10T16:31:19.124 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.111+0000 7ff51bc8c640 10 monclient: build_initial_monmap 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.111+0000 7ff51bc8c640 10 monclient: monmap: 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:19.112831+0000 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:19.112831+0000 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:19.125 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.112+0000 7ff51bc8c640 1 Processor -- start 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.112+0000 7ff51bc8c640 1 -- start start 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.112+0000 7ff51bc8c640 10 monclient: init 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff51409d930 0x7ff51409dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff51409d930 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff51409ab50 0x7ff51409e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff51409ab50 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff5140a1e40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff51409d3e0 con 0x7ff51409ab50 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff51409e790 con 0x7ff51409d930 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5140a0870 con 0x7ff5140a1e40 2024-09-10T16:31:19.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51bc8c640 10 monclient(hunting): authenticate will time out at 2257.600253s 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47548/0 (socket says 172.21.15.164:47548) 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 -- 172.21.15.164:0/1186047232 learned_addr learned my addr 172.21.15.164:0/1186047232 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): get_auth_request con 0x7ff5140a1e40 auth_method 0 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:19.128 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff519a28640 1 --2- 172.21.15.164:0/1186047232 >> v2:172.21.15.165:3300/0 conn(0x7ff51409d930 0x7ff51409dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff519a28640 10 monclient(hunting): get_auth_request con 0x7ff51409d930 auth_method 0 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff519a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff519a28640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff519a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient(hunting): handle_auth_done global_id 274223 payload 274 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 -- 172.21.15.164:0/1186047232 >> v2:172.21.15.165:3300/0 conn(0x7ff51409d930 msgr2=0x7ff51409dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 --2- 172.21.15.164:0/1186047232 >> v2:172.21.15.165:3300/0 conn(0x7ff51409d930 0x7ff51409dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 -- 172.21.15.164:0/1186047232 >> v2:172.21.15.164:3300/0 conn(0x7ff51409ab50 msgr2=0x7ff51409e250 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 --2- 172.21.15.164:0/1186047232 >> v2:172.21.15.164:3300/0 conn(0x7ff51409ab50 0x7ff51409e250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient: _finish_hunting 0 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 monclient: found mon.noname-b 2024-09-10T16:31:19.129 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.113+0000 7ff51a229640 1 -- 172.21.15.164:0/1186047232 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff514074c70 con 0x7ff5140a1e40 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51a229640 1 -- 172.21.15.164:0/1186047232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 msgr2=0x7ff514134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51a229640 1 --2- 172.21.15.164:0/1186047232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff51406d500 tx=0x7ff50c02a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7017367d4799b400 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 1 -- 172.21.15.164:0/1186047232 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c036070 con 0x7ff5140a1e40 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: dump: 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:19.130 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: _finish_auth 0 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: _check_auth_tickets 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 1 -- 172.21.15.164:0/1186047232 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff50c007800 con 0x7ff5140a1e40 2024-09-10T16:31:19.131 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51bc8c640 5 monclient: authenticate success, global_id 274223 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51bc8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51bc8c640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 1 -- 172.21.15.164:0/1186047232 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c007bd0 con 0x7ff5140a1e40 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff51bc8c640 4 set_mon_vals no callback set 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.116+0000 7ff518a26640 10 monclient: dump: 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.132 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 10 monclient: shutdown 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1186047232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 msgr2=0x7ff514134580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1186047232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff51406d500 tx=0x7ff50c02a590 comp rx=0 tx=0).stop 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1186047232 shutdown_connections 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1186047232 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff514134580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1186047232 >> v2:172.21.15.164:3300/0 conn(0x7ff51409ab50 0x7ff51409e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1186047232 >> v2:172.21.15.165:3300/0 conn(0x7ff51409d930 0x7ff51409dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1186047232 >> 172.21.15.164:0/1186047232 conn(0x7ff5140a3110 msgr2=0x7ff5140a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1186047232 shutdown_connections 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1186047232 wait complete. 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 10 monclient: build_initial_monmap 2024-09-10T16:31:19.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.117+0000 7ff51bc8c640 10 monclient: monmap: 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:19.135 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 Processor -- start 2024-09-10T16:31:19.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 -- start start 2024-09-10T16:31:19.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient: init 2024-09-10T16:31:19.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:19.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:31:19.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff51409ab50 0x7ff51413b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-b con 0x7ff51409ab50 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 0x7ff51413b710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-a con 0x7ff51409d930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5140a1e40 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff51409ef30 con 0x7ff51409d930 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff5140a02b0 con 0x7ff51409ab50 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff5140a2940 con 0x7ff5140a1e40 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:19.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51bc8c640 10 monclient(hunting): authenticate will time out at 2257.605566s 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519227640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 msgr2=0x7ff51413b710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519227640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 0x7ff51413b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47556/0 (socket says 172.21.15.164:47556) 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 -- 172.21.15.164:0/1207377110 learned_addr learned my addr 172.21.15.164:0/1207377110 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:19.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): get_auth_request con 0x7ff5140a1e40 auth_method 0 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:19.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519a28640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff51409ab50 0x7ff51413b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519a28640 10 monclient(hunting): get_auth_request con 0x7ff51409ab50 auth_method 0 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519a28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519a28640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff519a28640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient(hunting): handle_auth_done global_id 274226 payload 824 2024-09-10T16:31:19.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 -- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff51409ab50 msgr2=0x7ff51413b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff51409ab50 0x7ff51413b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 -- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 msgr2=0x7ff51413b710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 0x7ff51413b710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient: _finish_hunting 0 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 monclient: found mon.noname-c 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.118+0000 7ff51a229640 1 -- 172.21.15.164:0/1207377110 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51412b900 con 0x7ff5140a1e40 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51a229640 1 -- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 msgr2=0x7ff51413bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:19.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51a229640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009f40 tx=0x7ff50c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c03f020 con 0x7ff5140a1e40 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: dump: 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:19.144 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff50c037040 con 0x7ff5140a1e40 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 5 monclient: authenticate success, global_id 274226 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 10 monclient: _renew_subs 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51412b620 con 0x7ff5140a1e40 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff50c036070 con 0x7ff5140a1e40 2024-09-10T16:31:19.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50a7fc640 10 monclient: dump: 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:19.146 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff50bfff640 4 set_mon_vals no callback set 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 10 monclient: _renew_subs 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.121+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51413d6d0 con 0x7ff5140a1e40 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff50c042020 con 0x7ff5140a1e40 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff51bc8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff51bc8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff51bc8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5140a5f00 con 0x7ff5140a1e40 2024-09-10T16:31:19.147 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff50a7fc640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 0x7ff4f80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(874..874 src has 772..874) ==== 4815+0+0 (secure 0 0 0) 0x7ff50c073120 con 0x7ff5140a1e40 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff519a28640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 0x7ff4f80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.122+0000 7ff519a28640 10 monclient: get_auth_request con 0x7ff4f803e210 auth_method 0 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.125+0000 7ff519a28640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 0x7ff4f80406d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff510006fd0 tx=0x7ff510008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.127+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff50c049070 con 0x7ff5140a1e40 2024-09-10T16:31:19.148 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.127+0000 7ff50a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:19.149 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.127+0000 7ff50a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff51bc8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff51bc8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff51bc8c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5140a22d0 con 0x7ff5140a1e40 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff50a7fc640 1 -- 172.21.15.164:0/1207377110 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7ff50c03be20 con 0x7ff5140a1e40 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff50a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.370+0000 7ff50a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:19.372 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":244,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 msgr2=0x7ff4f80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 0x7ff4f80406d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7ff510006fd0 tx=0x7ff510008040 comp rx=0 tx=0).stop 2024-09-10T16:31:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 10 monclient: shutdown 2024-09-10T16:31:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:19.374 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 msgr2=0x7ff51413bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff50c009f40 tx=0x7ff50c004a40 comp rx=0 tx=0).stop 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 shutdown_connections 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff4f803e210 0x7ff4f80406d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff5140a1e40 0x7ff51413bc50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff51409d930 0x7ff51413b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.372+0000 7ff51bc8c640 1 --2- 172.21.15.164:0/1207377110 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff51409ab50 0x7ff51413b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.373+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 >> 172.21.15.164:0/1207377110 conn(0x7ff5140a3110 msgr2=0x7ff5140994e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.373+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 shutdown_connections 2024-09-10T16:31:19.375 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:19.373+0000 7ff51bc8c640 1 -- 172.21.15.164:0/1207377110 wait complete. 2024-09-10T16:31:22.386 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.494+0000 7f5896f9f640 10 monclient: get_monmap_and_config 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.494+0000 7f5896f9f640 10 monclient: build_initial_monmap 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.494+0000 7f5896f9f640 10 monclient: monmap: 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:22.495728+0000 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:22.495728+0000 2024-09-10T16:31:22.496 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:22.497 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.495+0000 7f5896f9f640 1 Processor -- start 2024-09-10T16:31:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.495+0000 7f5896f9f640 1 -- start start 2024-09-10T16:31:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.495+0000 7f5896f9f640 10 monclient: init 2024-09-10T16:31:22.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f58901354c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 0x7f589009d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f589009d450 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5890137cd0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f589006e180 con 0x7f5890137cd0 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f58900a65d0 con 0x7f589009d450 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5890135090 con 0x7f58901354c0 2024-09-10T16:31:22.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5896f9f640 10 monclient(hunting): authenticate will time out at 2260.983240s 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47566/0 (socket says 172.21.15.164:47566) 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 -- 172.21.15.164:0/1839722635 learned_addr learned my addr 172.21.15.164:0/1839722635 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): get_auth_request con 0x7f58901354c0 auth_method 0 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 1 --2- 172.21.15.164:0/1839722635 >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 0x7f589009d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 10 monclient(hunting): get_auth_request con 0x7f589009d450 auth_method 0 2024-09-10T16:31:22.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient(hunting): handle_auth_done global_id 274229 payload 274 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 -- 172.21.15.164:0/1839722635 >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 msgr2=0x7f589009d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 --2- 172.21.15.164:0/1839722635 >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 0x7f589009d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 -- 172.21.15.164:0/1839722635 >> v1:172.21.15.165:6789/0 conn(0x7f5890137cd0 legacy=0x7f5890138160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient: _finish_hunting 0 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 monclient: found mon.noname-b 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f5895f9d640 1 -- 172.21.15.164:0/1839722635 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f589006e460 con 0x7f58901354c0 2024-09-10T16:31:22.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.496+0000 7f589579c640 1 --2- 172.21.15.164:0/1839722635 >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 0x7f589009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5895f9d640 1 -- 172.21.15.164:0/1839722635 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 msgr2=0x7f58900a5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5895f9d640 1 --2- 172.21.15.164:0/1839722635 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f588c009870 tx=0x7f588c02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2bcac5c16bf9ab6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 1 -- 172.21.15.164:0/1839722635 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c037070 con 0x7f58901354c0 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:22.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: dump: 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: _finish_auth 0 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: _check_auth_tickets 2024-09-10T16:31:22.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 1 -- 172.21.15.164:0/1839722635 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588c032440 con 0x7f58901354c0 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 1 -- 172.21.15.164:0/1839722635 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588c032810 con 0x7f58901354c0 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5896f9f640 5 monclient: authenticate success, global_id 274229 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5896f9f640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5896f9f640 4 set_mon_vals no callback set 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.499+0000 7f5894f9b640 10 monclient: dump: 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.510 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 10 monclient: shutdown 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 -- 172.21.15.164:0/1839722635 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 msgr2=0x7f58900a5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 --2- 172.21.15.164:0/1839722635 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f588c009870 tx=0x7f588c02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 -- 172.21.15.164:0/1839722635 shutdown_connections 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 --2- 172.21.15.164:0/1839722635 >> v2:172.21.15.165:3300/0 conn(0x7f589009d450 0x7f589009d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 --2- 172.21.15.164:0/1839722635 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f58901354c0 0x7f58900a5ff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 -- 172.21.15.164:0/1839722635 >> 172.21.15.164:0/1839722635 conn(0x7f58900a70e0 msgr2=0x7f58900a74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 -- 172.21.15.164:0/1839722635 shutdown_connections 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 1 -- 172.21.15.164:0/1839722635 wait complete. 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 10 monclient: build_initial_monmap 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.500+0000 7f5896f9f640 10 monclient: monmap: 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:22.512 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 Processor -- start 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 -- start start 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient: init 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589009d450 0x7f58901ccf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-b con 0x7f589009d450 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 0x7f58901cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-a con 0x7f58901354c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): picked mon.noname-c con 0x7f5890137cd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:22.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f589009e040 con 0x7f58901354c0 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f589009dec0 con 0x7f589009d450 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5890138870 con 0x7f5890137cd0 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f5896f9f640 10 monclient(hunting): authenticate will time out at 2260.988832s 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589579c640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 msgr2=0x7f58901cd470 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589579c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 0x7f58901cd470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:47570/0 (socket says 172.21.15.164:47570) 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 1 -- 172.21.15.164:0/3844416655 learned_addr learned my addr 172.21.15.164:0/3844416655 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:22.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 10 monclient(hunting): get_auth_request con 0x7f5890137cd0 auth_method 0 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.501+0000 7f589679e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f5895f9d640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589009d450 0x7f58901ccf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient(hunting): handle_auth_done global_id 274232 payload 824 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 -- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589009d450 msgr2=0x7f58901ccf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589009d450 0x7f58901ccf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 -- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 msgr2=0x7f58901cd470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 0x7f58901cd470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient: _finish_hunting 0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 monclient: found mon.noname-c 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.502+0000 7f589679e640 1 -- 172.21.15.164:0/3844416655 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58901c0a10 con 0x7f5890137cd0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f589679e640 1 -- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 msgr2=0x7f58901cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f589679e640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f588800c780 tx=0x7f588800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f588800ac60 con 0x7f5890137cd0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: dump: 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:22.516 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588800ae00 con 0x7f5890137cd0 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 5 monclient: authenticate success, global_id 274232 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:22.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 10 monclient: _renew_subs 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888004ba0 con 0x7f5890137cd0 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58901c0170 con 0x7f5890137cd0 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5887fff640 4 set_mon_vals no callback set 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f58867fc640 10 monclient: dump: 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:22.518 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 10 monclient: _renew_subs 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.504+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f58901bf210 con 0x7f5890137cd0 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5888005030 con 0x7f5890137cd0 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5896f9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5896f9f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5896f9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f589009d830 con 0x7f5890137cd0 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f58867fc640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 0x7f5874040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:22.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(874..874 src has 772..874) ==== 4815+0+0 (secure 0 0 0) 0x7f58880530f0 con 0x7f5890137cd0 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5895f9d640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 0x7f5874040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.505+0000 7f5895f9d640 10 monclient: get_auth_request con 0x7f587403e1c0 auth_method 0 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.508+0000 7f5895f9d640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 0x7f5874040680 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f589009dd70 tx=0x7f588c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.510+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f5888020070 con 0x7f5890137cd0 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.510+0000 7f58867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:22.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.510+0000 7f58867fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:22.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.749+0000 7f5896f9f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:22.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.749+0000 7f5896f9f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:22.751 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.749+0000 7f5896f9f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:22.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.749+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5890076eb0 con 0x7f5890137cd0 2024-09-10T16:31:22.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.750+0000 7f58867fc640 1 -- 172.21.15.164:0/3844416655 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f588801a3a0 con 0x7f5890137cd0 2024-09-10T16:31:22.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.750+0000 7f58867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:22.752 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.750+0000 7f58867fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:22.752 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":244,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:22.753 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 msgr2=0x7f5874040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 0x7f5874040680 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f589009dd70 tx=0x7f588c02e000 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 10 monclient: shutdown 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 msgr2=0x7f58901cd9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f588800c780 tx=0x7f588800cc40 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 shutdown_connections 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f587403e1c0 0x7f5874040680 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5890137cd0 0x7f58901cd9b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f58901354c0 0x7f58901cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 --2- 172.21.15.164:0/3844416655 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f589009d450 0x7f58901ccf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 >> 172.21.15.164:0/3844416655 conn(0x7f58900a70e0 msgr2=0x7f589009b430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 shutdown_connections 2024-09-10T16:31:22.754 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:22.752+0000 7f5896f9f640 1 -- 172.21.15.164:0/3844416655 wait complete. 2024-09-10T16:31:25.765 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 10 monclient: get_monmap_and_config 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 10 monclient: build_initial_monmap 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 10 monclient: monmap: 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:25.874 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:25.873945+0000 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:25.873945+0000 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:25.875 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 1 Processor -- start 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 1 -- start start 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.873+0000 7f99138fe640 10 monclient: init 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f990c06cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 0x7f990c13d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-a con 0x7f990c134f70 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f990c13d560 0x7f990c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-c con 0x7f990c13d560 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f990c09ce50 con 0x7f990c06cb70 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f990c06db60 con 0x7f990c134f70 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f990c06cf50 con 0x7f990c13d560 2024-09-10T16:31:25.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f99138fe640 10 monclient(hunting): authenticate will time out at 2264.361401s 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9910e99640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 msgr2=0x7f990c13d020 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f991169a640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f990c06cb70 legacy=0x7f990c0a46a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9910e99640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 0x7f990c13d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f990c13d560 0x7f990c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f990c13d560 0x7f990c13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:51918/0 (socket says 172.21.15.164:51918) 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 1 -- 172.21.15.164:0/3606418956 learned_addr learned my addr 172.21.15.164:0/3606418956 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): get_auth_request con 0x7f990c13d560 auth_method 0 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:25.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.874+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 10 monclient(hunting): handle_auth_done global_id 284140 payload 274 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 1 -- 172.21.15.164:0/3606418956 >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 msgr2=0x7f990c13d020 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 1 --2- 172.21.15.164:0/3606418956 >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 0x7f990c13d020 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 1 -- 172.21.15.164:0/3606418956 >> v1:172.21.15.164:6789/0 conn(0x7f990c06cb70 legacy=0x7f990c0a46a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 1 monclient: found mon.noname-c 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:25.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.875+0000 7f9911e9b640 1 -- 172.21.15.164:0/3606418956 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c135380 con 0x7f990c13d560 2024-09-10T16:31:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f9911e9b640 1 -- 172.21.15.164:0/3606418956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c13d560 msgr2=0x7f990c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:25.887 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f9911e9b640 1 --2- 172.21.15.164:0/3606418956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c13d560 0x7f990c13f950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9904009870 tx=0x7f990402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96983bd9eea44108 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 1 -- 172.21.15.164:0/3606418956 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904037070 con 0x7f990c13d560 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: dump: 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:25.888 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:25.889 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: _finish_auth 0 2024-09-10T16:31:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:31:25.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 1 -- 172.21.15.164:0/3606418956 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9904032640 con 0x7f990c13d560 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 1 -- 172.21.15.164:0/3606418956 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904032a10 con 0x7f990c13d560 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f990bfff640 10 monclient: dump: 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:25.890 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 5 monclient: authenticate success, global_id 284140 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 4 set_mon_vals no callback set 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 10 monclient: shutdown 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.878+0000 7f99138fe640 1 -- 172.21.15.164:0/3606418956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c13d560 msgr2=0x7f990c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:25.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 --2- 172.21.15.164:0/3606418956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c13d560 0x7f990c13f950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f9904009870 tx=0x7f990402a110 comp rx=0 tx=0).stop 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 -- 172.21.15.164:0/3606418956 shutdown_connections 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 --2- 172.21.15.164:0/3606418956 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c13d560 0x7f990c13f950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 --2- 172.21.15.164:0/3606418956 >> v2:172.21.15.164:3300/0 conn(0x7f990c134f70 0x7f990c13d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 -- 172.21.15.164:0/3606418956 >> 172.21.15.164:0/3606418956 conn(0x7f990c0aa900 msgr2=0x7f990c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 -- 172.21.15.164:0/3606418956 shutdown_connections 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 1 -- 172.21.15.164:0/3606418956 wait complete. 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 10 monclient: build_initial_monmap 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.879+0000 7f99138fe640 10 monclient: monmap: 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:25.892 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 Processor -- start 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 -- start start 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient: init 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c06cb70 0x7f990c12f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-b con 0x7f990c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 0x7f990c12fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-a con 0x7f990c134f70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): picked mon.noname-c con 0x7f990c13d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f990c0a4db0 con 0x7f990c134f70 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f990c09d130 con 0x7f990c06cb70 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f990c13fe90 con 0x7f990c13d560 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f99138fe640 10 monclient(hunting): authenticate will time out at 2264.367597s 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9910e99640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 msgr2=0x7f990c12fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:25.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9910e99640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 0x7f990c12fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:55704/0 (socket says 172.21.15.164:55704) 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 1 -- 172.21.15.164:0/421409464 learned_addr learned my addr 172.21.15.164:0/421409464 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): get_auth_request con 0x7f990c13d560 auth_method 0 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c06cb70 0x7f990c12f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): get_auth_request con 0x7f990c06cb70 auth_method 0 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:25.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f991169a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.880+0000 7f9911e9b640 10 monclient(hunting): handle_auth_done global_id 274235 payload 824 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 -- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c06cb70 msgr2=0x7f990c12f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c06cb70 0x7f990c12f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 -- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 msgr2=0x7f990c12fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 0x7f990c12fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 10 monclient: _finish_hunting 0 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 monclient: found mon.noname-c 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:25.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.881+0000 7f9911e9b640 1 -- 172.21.15.164:0/421409464 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f990c13d030 con 0x7f990c13d560 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9911e9b640 1 -- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 msgr2=0x7f990c1cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9911e9b640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9904009870 tx=0x7f99040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904041070 con 0x7f990c13d560 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: dump: 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:25.897 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 5 monclient: authenticate success, global_id 274235 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f990403c880 con 0x7f990c13d560 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9904037040 con 0x7f990c13d560 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 10 monclient: _renew_subs 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:25.898 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f990c06e720 con 0x7f990c13d560 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f9909ffb640 10 monclient: dump: 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:25.899 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f990b7fe640 4 set_mon_vals no callback set 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 10 monclient: _renew_subs 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.883+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f990c130990 con 0x7f990c13d560 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9904044070 con 0x7f990c13d560 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f99138fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f99138fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f99138fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f990c132ec0 con 0x7f990c13d560 2024-09-10T16:31:25.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f9909ffb640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 0x7f98f8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.884+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(874..874 src has 772..874) ==== 4815+0+0 (secure 0 0 0) 0x7f9904073120 con 0x7f990c13d560 2024-09-10T16:31:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.885+0000 7f991169a640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 0x7f98f8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.885+0000 7f991169a640 10 monclient: get_auth_request con 0x7f98f803e1c0 auth_method 0 2024-09-10T16:31:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.887+0000 7f991169a640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 0x7f98f8040680 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f98fc003c30 tx=0x7f98fc00d040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:25.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:25.889+0000 7f9911e9b640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9904009870 tx=0x7f99040046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f990c132ec0 mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) 2024-09-10T16:31:26.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.718+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f990c132ec0 con 0x7f990c13d560 2024-09-10T16:31:26.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.718+0000 7f9909ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:26.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.718+0000 7f9909ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f99138fe640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f99138fe640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f99138fe640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f990c138570 con 0x7f990c13d560 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f9909ffb640 1 -- 172.21.15.164:0/421409464 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1570 (secure 0 0 0) 0x7f990404b070 con 0x7f990c13d560 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f9909ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:26.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.964+0000 7f9909ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:26.967 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":244,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:26.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 msgr2=0x7f98f8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:26.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 0x7f98f8040680 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f98fc003c30 tx=0x7f98fc00d040 comp rx=0 tx=0).stop 2024-09-10T16:31:26.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 10 monclient: shutdown 2024-09-10T16:31:26.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 msgr2=0x7f990c1cd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.966+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9904009870 tx=0x7f99040046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 shutdown_connections 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f98f803e1c0 0x7f98f8040680 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f990c13d560 0x7f990c1cd300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f990c134f70 0x7f990c12fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 --2- 172.21.15.164:0/421409464 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f990c06cb70 0x7f990c12f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 >> 172.21.15.164:0/421409464 conn(0x7f990c0aa900 msgr2=0x7f990c0a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 shutdown_connections 2024-09-10T16:31:26.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:26.967+0000 7f99138fe640 1 -- 172.21.15.164:0/421409464 wait complete. 2024-09-10T16:31:29.979 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:30.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.089+0000 7f2ee76e6640 10 monclient: get_monmap_and_config 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.089+0000 7f2ee76e6640 10 monclient: build_initial_monmap 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.089+0000 7f2ee76e6640 10 monclient: monmap: 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:30.090171+0000 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:30.090171+0000 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:30.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:30.092 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:30.092 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:30.092 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:30.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.089+0000 7f2ee76e6640 1 Processor -- start 2024-09-10T16:31:30.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 1 -- start start 2024-09-10T16:31:30.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient: init 2024-09-10T16:31:30.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2ee006cb70 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2ee0134f70 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ee013e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee009ce50 con 0x7f2ee006cb70 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ee0135380 con 0x7f2ee0134f70 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2ee006db60 con 0x7f2ee013e740 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee76e6640 10 monclient(hunting): authenticate will time out at 2268.577644s 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5482640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f2ee006cb70 legacy=0x7f2ee00a46a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:30.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee4c81640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f2ee0134f70 0x7f2ee013d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:34370/0 (socket says 172.21.15.164:34370) 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 learned_addr learned my addr 172.21.15.164:0/525173541 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:51938/0 (socket says 172.21.15.164:51938) 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request con 0x7f2ee013e740 auth_method 0 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.090+0000 7f2ee5c83640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.091+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.091+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.091+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.091+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:30.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.093+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.165:6789/0 conn(0x7f2ee0134f70 legacy=0x7f2ee013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:30.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.291+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.164:6789/0 conn(0x7f2ee006cb70 legacy=0x7f2ee00a46a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request con 0x7f2ee013e740 auth_method 0 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:30.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:30.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:30.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:30.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.292+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:30.693 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.692+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.164:6789/0 conn(0x7f2ee006cb70 legacy=0x7f2ee00a46a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request con 0x7f2ee013e740 auth_method 0 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:30.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:30.693+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:31:31.494 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.493+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.164:6789/0 conn(0x7f2ee006cb70 legacy=0x7f2ee00a46a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:31.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:31.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request con 0x7f2ee013e740 auth_method 0 2024-09-10T16:31:31.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:31.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:31.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:31.495+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): tick 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 msgr2=0x7f2ee0140b30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ee013e740 0x7f2ee0140b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.165:6789/0 conn(0x7f2ee0134f70 legacy=0x7f2ee013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 >> v1:172.21.15.164:6789/0 conn(0x7f2ee006cb70 legacy=0x7f2ee00a46a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:33.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient: _add_conns ranks=[2,3] 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec4003770 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec4006280 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec4008db0 con 0x7f2ec4006280 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec4008f60 con 0x7f2ec4003770 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ed2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 msgr2=0x7f2ec4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:33.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003770 auth_method 0 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:33.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 msgr2=0x7f2ec4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003770 auth_method 0 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:33.293 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:33.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:33.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:33.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.291+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003770 auth_method 0 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:33.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:33.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:33.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:33.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:31:33.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 msgr2=0x7f2ec4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:33.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:33.692+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:31:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003770 auth_method 0 2024-09-10T16:31:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:34.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 msgr2=0x7f2ec4008720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:34.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:34.493+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T16:31:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): tick 2024-09-10T16:31:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-10T16:31:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 monclient(hunting): continuing hunt 2024-09-10T16:31:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T16:31:36.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 msgr2=0x7f2ec4005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003770 0x7f2ec4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 msgr2=0x7f2ec4008720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec4006280 0x7f2ec4008720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 reap_dead start 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient: _add_conns ranks=[2,4,3,0,1] 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec4003b90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec4006c50 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec40076c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec4008db0 con 0x7f2ec40076c0 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f2ec4008f60 con 0x7f2ec4006c50 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 1 -- 172.21.15.164:0/525173541 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2ec40082c0 con 0x7f2ec4003b90 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ed2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 msgr2=0x7f2ec4007b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003b90 auth_method 0 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ec4006710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:36.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ec4006710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request con 0x7f2ec4006c50 auth_method 0 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:36.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.090+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:36.294 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request con 0x7f2ec4006c50 auth_method 0 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 msgr2=0x7f2ec4007b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.291+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.295 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003b90 auth_method 0 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ec4006710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ec4006710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:36.296 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.292+0000 7f2ee5482640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T16:31:36.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee5c83640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 msgr2=0x7f2ec4007b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:36.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T16:31:36.695 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.165:3300/0 conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request con 0x7f2ec4006c50 auth_method 0 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient(hunting): handle_auth_done global_id 284146 payload 274 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ec4006710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 msgr2=0x7f2ec4007b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:36.696 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient: _finish_hunting 0 2024-09-10T16:31:36.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 monclient: found mon.noname-c 2024-09-10T16:31:36.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:36.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:36.697 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.692+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee009cb70 con 0x7f2ec4006c50 2024-09-10T16:31:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.695+0000 7f2ee4c81640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:36.698 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec4006c50 0x7f2ec4007030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00a5390 tx=0x7f2ed402a840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e26b129ebd7e37b3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 1 -- 172.21.15.164:0/525173541 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4004b60 con 0x7f2ec4006c50 2024-09-10T16:31:36.699 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: dump: 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:36.700 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: _finish_auth 0 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 1 -- 172.21.15.164:0/525173541 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ed4004d00 con 0x7f2ec4006c50 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 1 -- 172.21.15.164:0/525173541 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ed4005250 con 0x7f2ec4006c50 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ed3fff640 10 monclient: dump: 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.701 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ee76e6640 5 monclient: authenticate success, global_id 284146 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ee76e6640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.696+0000 7f2ee76e6640 4 set_mon_vals no callback set 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 10 monclient: shutdown 2024-09-10T16:31:36.702 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 -- 172.21.15.164:0/525173541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec4006c50 msgr2=0x7f2ec4007030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec4006c50 0x7f2ec4007030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2ee00a5390 tx=0x7f2ed402a840 comp rx=0 tx=0).stop 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 -- 172.21.15.164:0/525173541 shutdown_connections 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/525173541 >> v2:172.21.15.164:3300/0 conn(0x7f2ec40076c0 0x7f2ec4007b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec4006c50 0x7f2ec4007030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/525173541 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ec4006710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 -- 172.21.15.164:0/525173541 >> 172.21.15.164:0/525173541 conn(0x7f2ee00aa900 msgr2=0x7f2ee00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 -- 172.21.15.164:0/525173541 shutdown_connections 2024-09-10T16:31:36.703 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.697+0000 7f2ee76e6640 1 -- 172.21.15.164:0/525173541 wait complete. 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.698+0000 7f2ee76e6640 10 monclient: build_initial_monmap 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.698+0000 7f2ee76e6640 10 monclient: monmap: 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:36.708 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.698+0000 7f2ee76e6640 1 Processor -- start 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.698+0000 7f2ee76e6640 1 -- start start 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.698+0000 7f2ee76e6640 10 monclient: init 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.709 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec4003b90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 0x7f2ee013d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec4006c50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec40076c0 0x7f2ee013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec40076c0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee00a46a0 con 0x7f2ec4006c50 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f2ee006cf50 con 0x7f2ec40076c0 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2ee00a4820 con 0x7f2ec4003b90 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee76e6640 10 monclient(hunting): authenticate will time out at 2275.187195s 2024-09-10T16:31:36.710 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee4c81640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 msgr2=0x7f2ee013d430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee4c81640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 0x7f2ee013d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46090/0 (socket says 172.21.15.164:46090) 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 -- 172.21.15.164:0/2293017225 learned_addr learned my addr 172.21.15.164:0/2293017225 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): get_auth_request con 0x7f2ec4003b90 auth_method 0 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec40076c0 0x7f2ee013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request con 0x7f2ec40076c0 auth_method 0 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:36.711 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5c83640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient(hunting): handle_auth_done global_id 274238 payload 824 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 -- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec40076c0 msgr2=0x7f2ee013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec40076c0 0x7f2ee013d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 -- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 msgr2=0x7f2ee013d430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:36.712 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 0x7f2ee013d430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient: _finish_hunting 0 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 monclient: found mon.noname-c 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.699+0000 7f2ee5482640 1 -- 172.21.15.164:0/2293017225 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ee013a9f0 con 0x7f2ec4003b90 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee5482640 1 -- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ee01c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee5482640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003890 tx=0x7f2edc005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc01a660 con 0x7f2ec4003b90 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: dump: 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:36.713 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2edc01a800 con 0x7f2ec4003b90 2024-09-10T16:31:36.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 5 monclient: authenticate success, global_id 274238 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 10 monclient: _renew_subs 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ee013a720 con 0x7f2ec4003b90 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2edc01abf0 con 0x7f2ec4003b90 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed1ffb640 10 monclient: dump: 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:36.715 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 10 monclient: _renew_subs 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ed37fe640 4 set_mon_vals no callback set 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.702+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ee013b960 con 0x7f2ec4003b90 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.703+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f2edc017050 con 0x7f2ec4003b90 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.703+0000 7f2ed1ffb640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 0x7f2ec40425e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.703+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(874..874 src has 772..874) ==== 4815+0+0 (secure 0 0 0) 0x7f2edc054320 con 0x7f2ec4003b90 2024-09-10T16:31:36.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee76e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee76e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee76e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ee00a9780 con 0x7f2ec4003b90 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 0x7f2ec40425e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.704+0000 7f2ee4c81640 10 monclient: get_auth_request con 0x7f2ec4040120 auth_method 0 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.706+0000 7f2ee4c81640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 0x7f2ec40425e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4032000 tx=0x7f2ed402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:36.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.709+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2ee00a9780 con 0x7f2ec4003b90 2024-09-10T16:31:36.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.709+0000 7f2ed1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:36.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.709+0000 7f2ed1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ee76e6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ee76e6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ee76e6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ee00a6140 con 0x7f2ec4003b90 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ed1ffb640 1 -- 172.21.15.164:0/2293017225 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f2ee00a6140 con 0x7f2ec4003b90 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ed1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.952+0000 7f2ed1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:36.955 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":246,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:36.957 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 msgr2=0x7f2ec40425e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 0x7f2ec40425e0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f2ed4032000 tx=0x7f2ed402f000 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 10 monclient: shutdown 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 msgr2=0x7f2ee01c9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2edc003890 tx=0x7f2edc005480 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 shutdown_connections 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f2ec4040120 0x7f2ec40425e0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f2ec40076c0 0x7f2ee013d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f2ec4006c50 0x7f2ee013d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 --2- 172.21.15.164:0/2293017225 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f2ec4003b90 0x7f2ee01c9eb0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 >> 172.21.15.164:0/2293017225 conn(0x7f2ee00aa900 msgr2=0x7f2ee0133210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:36.958 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 shutdown_connections 2024-09-10T16:31:36.959 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:36.955+0000 7f2ee76e6640 1 -- 172.21.15.164:0/2293017225 wait complete. 2024-09-10T16:31:36.968 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T16:31:36.969 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T16:31:36.969 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:31:37.024 INFO:tasks.rados.rados.0.smithi165.stdout:3893: done (4 left) 2024-09-10T16:31:37.024 INFO:tasks.rados.rados.0.smithi165.stdout:3894: read oid 19 snap -1 2024-09-10T16:31:37.025 INFO:tasks.rados.rados.0.smithi165.stdout:3894: expect (ObjNum 1322 snap 429 seq_num 1322) 2024-09-10T16:31:37.025 INFO:tasks.rados.rados.0.smithi165.stdout:3895: rollback oid 15 current snap is 434 2024-09-10T16:31:37.025 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 15 to 409 2024-09-10T16:31:37.025 INFO:tasks.rados.rados.0.smithi165.stdout:3890: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:37.029 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 850417 2024-09-10T16:31:37.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.071+0000 7fa20d216640 10 monclient: get_monmap_and_config 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.071+0000 7fa20d216640 10 monclient: build_initial_monmap 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.071+0000 7fa20d216640 10 monclient: monmap: 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:37.073210+0000 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:37.073210+0000 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:37.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:37.075 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:37.075 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 Processor -- start 2024-09-10T16:31:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 -- start start 2024-09-10T16:31:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient: init 2024-09-10T16:31:37.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient: _add_conns ranks=[2,3,0,4,1] 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-b con 0x7fa20809c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 0x7fa20809ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-a con 0x7fa20809d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa20809f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa20809fbc0 con 0x7fa20809f4c0 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fa2080a2a20 con 0x7fa20809d190 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa20809c6f0 con 0x7fa20809c310 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20d216640 10 monclient(hunting): authenticate will time out at 2275.560655s 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa20759e640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7fa20809f4c0 legacy=0x7fa20813b5c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa1fffff640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 msgr2=0x7fa20809ef80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa1fffff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 0x7fa20809ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46106/0 (socket says 172.21.15.164:46106) 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 1 -- 172.21.15.164:0/496586460 learned_addr learned my addr 172.21.15.164:0/496586460 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:37.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): get_auth_request con 0x7fa20809c310 auth_method 0 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 10 monclient(hunting): handle_auth_done global_id 274241 payload 274 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 1 -- 172.21.15.164:0/496586460 >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 msgr2=0x7fa20809ef80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.072+0000 7fa206d9d640 1 --2- 172.21.15.164:0/496586460 >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 0x7fa20809ef80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 1 -- 172.21.15.164:0/496586460 >> v1:172.21.15.164:6789/0 conn(0x7fa20809f4c0 legacy=0x7fa20813b5c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:37.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 1 monclient: found mon.noname-b 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.073+0000 7fa206d9d640 1 -- 172.21.15.164:0/496586460 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa20809a4d0 con 0x7fa20809c310 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.075+0000 7fa206d9d640 1 -- 172.21.15.164:0/496586460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 msgr2=0x7fa20809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.075+0000 7fa206d9d640 1 --2- 172.21.15.164:0/496586460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2547f40518abc1fc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 1 -- 172.21.15.164:0/496586460 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8037070 con 0x7fa20809c310 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:37.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: dump: 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: _finish_auth 0 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: _check_auth_tickets 2024-09-10T16:31:37.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 1 -- 172.21.15.164:0/496586460 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f8032640 con 0x7fa20809c310 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 5 monclient: authenticate success, global_id 274241 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 1 -- 172.21.15.164:0/496586460 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8032a10 con 0x7fa20809c310 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 4 set_mon_vals no callback set 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20659c640 10 monclient: dump: 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.088 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 10 monclient: shutdown 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 -- 172.21.15.164:0/496586460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 msgr2=0x7fa20809ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 --2- 172.21.15.164:0/496586460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.089 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 -- 172.21.15.164:0/496586460 shutdown_connections 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 --2- 172.21.15.164:0/496586460 >> v2:172.21.15.164:3300/0 conn(0x7fa20809d190 0x7fa20809ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 --2- 172.21.15.164:0/496586460 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809c310 0x7fa20809ea40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 -- 172.21.15.164:0/496586460 >> 172.21.15.164:0/496586460 conn(0x7fa2080a3110 msgr2=0x7fa2080a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 -- 172.21.15.164:0/496586460 shutdown_connections 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.076+0000 7fa20d216640 1 -- 172.21.15.164:0/496586460 wait complete. 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.077+0000 7fa20d216640 10 monclient: build_initial_monmap 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.077+0000 7fa20d216640 10 monclient: monmap: 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:37.090 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.077+0000 7fa20d216640 1 Processor -- start 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.077+0000 7fa20d216640 1 -- start start 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.077+0000 7fa20d216640 10 monclient: init 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20809c310 0x7fa2081cb050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-a con 0x7fa20809c310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 0x7fa20813cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-c con 0x7fa20809f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa20813d0b0 0x7fa208070a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): picked mon.noname-b con 0x7fa20813d0b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa2080a2230 con 0x7fa20809c310 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa2080a20b0 con 0x7fa20813d0b0 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa208074e00 con 0x7fa20809f4c0 2024-09-10T16:31:37.092 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20d216640 10 monclient(hunting): authenticate will time out at 2275.566299s 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa206d9d640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20809c310 msgr2=0x7fa2081cb050 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa206d9d640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20809c310 0x7fa2081cb050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 0x7fa20813cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 0x7fa20813cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46116/0 (socket says 172.21.15.164:46116) 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 -- 172.21.15.164:0/2368279984 learned_addr learned my addr 172.21.15.164:0/2368279984 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): get_auth_request con 0x7fa20809f4c0 auth_method 0 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.093 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20759e640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa20813d0b0 0x7fa208070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20759e640 10 monclient(hunting): get_auth_request con 0x7fa20813d0b0 auth_method 0 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa20759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient(hunting): handle_auth_done global_id 274244 payload 824 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa20813d0b0 msgr2=0x7fa208070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa20813d0b0 0x7fa208070a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20809c310 msgr2=0x7fa2081cb050 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa20809c310 0x7fa2081cb050 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient: _finish_hunting 0 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 monclient: found mon.noname-c 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.078+0000 7fa1fffff640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2080a42a0 con 0x7fa20809f4c0 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.080+0000 7fa1fffff640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 msgr2=0x7fa20813cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.080+0000 7fa1fffff640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 0x7fa20813cb70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000b550 tx=0x7fa1f000ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.080+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0005070 con 0x7fa20809f4c0 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.080+0000 7fa1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: dump: 2024-09-10T16:31:37.101 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:37.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1f0015070 con 0x7fa20809f4c0 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 5 monclient: authenticate success, global_id 274244 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 monclient: _renew_subs 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa20809d620 con 0x7fa20809f4c0 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f0005430 con 0x7fa20809f4c0 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa1ff7fe640 10 monclient: dump: 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.103 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 monclient: _renew_subs 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa205d9b640 4 set_mon_vals no callback set 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa208072fa0 con 0x7fa20809f4c0 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 0x7fa208070f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.081+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2081cbd30 con 0x7fa20809d190 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa1f0005610 con 0x7fa20809f4c0 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 0x7fa208070f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient: get_auth_request con 0x7fa20809d190 auth_method 0 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa1ff7fe640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa1e803e1c0 0x7fa1e8040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(875..875 src has 772..875) ==== 4831+0+0 (secure 0 0 0) 0x7fa1f0052b00 con 0x7fa20809f4c0 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 10 monclient(hunting): handle_auth_done global_id 274247 payload 824 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa206d9d640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2080a47d0 con 0x7fa20809d190 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa20759e640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa1e803e1c0 0x7fa1e8040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.082+0000 7fa20759e640 10 monclient: get_auth_request con 0x7fa1e803e1c0 auth_method 0 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa206d9d640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 msgr2=0x7fa208070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa206d9d640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 0x7fa208070f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8034000 tx=0x7fa1f80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8037040 con 0x7fa20809d190 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa1ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa1ff7fe640 10 monclient: dump: 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.106 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa20759e640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa1e803e1c0 0x7fa1e8040680 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa200003cb0 tx=0x7fa200009430 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.084+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa1f8009490 con 0x7fa20809d190 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:37.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 msgr2=0x7fa208070f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 0x7fa208070f90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8034000 tx=0x7fa1f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 msgr2=0x7fa208070f90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.085+0000 7fa1ff7fe640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809d190 0x7fa208070f90 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8034000 tx=0x7fa1f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.090+0000 7fa20d216640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.090+0000 7fa20d216640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 0x7fa208098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.090+0000 7fa20d216640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.090+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa2080718e0 con 0x7fa2080714d0 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 0x7fa208098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient: get_auth_request con 0x7fa2080714d0 auth_method 0 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 10 monclient(hunting): handle_auth_done global_id 274250 payload 824 2024-09-10T16:31:37.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.091+0000 7fa206d9d640 1 -- 172.21.15.164:0/2368279984 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa20809d930 con 0x7fa2080714d0 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa206d9d640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 msgr2=0x7fa208098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa206d9d640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 0x7fa208098220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1f8037040 con 0x7fa2080714d0 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 10 monclient: dump: 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.110 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7fa1f804c070 con 0x7fa2080714d0 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 msgr2=0x7fa208098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 0x7fa208098220 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f8009240 comp rx=0 tx=0).stop 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 msgr2=0x7fa208098220 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:37.111 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.093+0000 7fa1ff7fe640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa2080714d0 0x7fa208098220 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa1f8009870 tx=0x7fa1f8009240 comp rx=0 tx=0).stop 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 246, 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 74336, 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:37.112 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.113 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.114 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:37.115 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.116 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:37.117 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.118 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.119 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:37.120 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.121 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.122 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.123 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.124 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:37.125 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:37.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa1e803e1c0 msgr2=0x7fa1e8040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa1e803e1c0 0x7fa1e8040680 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fa200003cb0 tx=0x7fa200009430 comp rx=0 tx=0).stop 2024-09-10T16:31:37.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 10 monclient: shutdown 2024-09-10T16:31:37.126 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 msgr2=0x7fa20813cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 --2- 172.21.15.164:0/2368279984 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa20809f4c0 0x7fa20813cb70 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa1f000b550 tx=0x7fa1f000ba10 comp rx=0 tx=0).stop 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20759e640 1 -- 172.21.15.164:0/2368279984 reap_dead start 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20759e640 1 -- 172.21.15.164:0/2368279984 reap_dead start 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 shutdown_connections 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 >> 172.21.15.164:0/2368279984 conn(0x7fa2080a3110 msgr2=0x7fa20809db40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 shutdown_connections 2024-09-10T16:31:37.127 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.094+0000 7fa20d216640 1 -- 172.21.15.164:0/2368279984 wait complete. 2024-09-10T16:31:37.127 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:31:37.127 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T16:31:37.128 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: finishing write tid 2 to smithi16534841-47 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: finishing write tid 3 to smithi16534841-47 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: finishing write tid 4 to smithi16534841-47 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: finishing write tid 5 to smithi16534841-47 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: oid 47 updating version 0 to 2190 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: oid 47 updating version 2190 to 2192 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: oid 47 version 2192 is already newer than 2189 2024-09-10T16:31:37.147 INFO:tasks.rados.rados.0.smithi165.stdout:3889: oid 47 version 2192 is already newer than 2191 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 47 v 2192 (ObjNum 1339 snap 434 seq_num 1339) dirty exists 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3889: left oid 47 (ObjNum 1339 snap 434 seq_num 1339) 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3891: finishing write tid 1 to smithi16534841-43 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3889: done (5 left) 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3890: done (4 left) 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3896: write (excl) oid 18 current snap is 434 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3891: finishing write tid 2 to smithi16534841-43 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3891: finishing write tid 3 to smithi16534841-43 2024-09-10T16:31:37.148 INFO:tasks.rados.rados.0.smithi165.stdout:3896: seq_num 1342 ranges {554994=662286,1997188=557806} 2024-09-10T16:31:37.153 INFO:tasks.rados.rados.0.smithi165.stdout:3896: writing smithi16534841-18 from 554994 to 1217280 tid 1 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3896: writing smithi16534841-18 from 1997188 to 2554994 tid 2 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3897: rollback oid 31 current snap is 434 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 31 to 419 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3891: finishing write tid 4 to smithi16534841-43 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3891: oid 43 updating version 0 to 1726 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3891: oid 43 updating version 1726 to 1727 2024-09-10T16:31:37.162 INFO:tasks.rados.rados.0.smithi165.stdout:3891: oid 43 version 1727 is already newer than 1725 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 1727 (ObjNum 1340 snap 434 seq_num 1340) dirty exists 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3891: left oid 43 (ObjNum 1340 snap 434 seq_num 1340) 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: finishing write tid 1 to smithi16534841-26 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: finishing write tid 2 to smithi16534841-26 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: finishing write tid 3 to smithi16534841-26 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: finishing write tid 4 to smithi16534841-26 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: oid 26 updating version 0 to 2590 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: oid 26 updating version 2590 to 2591 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: oid 26 version 2591 is already newer than 2589 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2591 (ObjNum 1341 snap 434 seq_num 1341) dirty exists 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3892: left oid 26 (ObjNum 1341 snap 434 seq_num 1341) 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3895: finishing rollback tid 1 to smithi16534841-15 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:3895: finishing rollback tid 2 to smithi16534841-15 2024-09-10T16:31:37.163 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 1867 (ObjNum 1257 snap 405 seq_num 1257) dirty exists 2024-09-10T16:31:37.164 INFO:tasks.rados.rados.0.smithi165.stdout:3894: expect (ObjNum 1322 snap 429 seq_num 1322) 2024-09-10T16:31:37.225 INFO:tasks.rados.rados.0.smithi165.stdout:3896: finishing write tid 1 to smithi16534841-18 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: finishing write tid 2 to smithi16534841-18 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: finishing write tid 3 to smithi16534841-18 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: finishing write tid 4 to smithi16534841-18 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: oid 18 updating version 0 to 2102 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: oid 18 updating version 2102 to 2103 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: oid 18 version 2103 is already newer than 2101 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 18 v 2103 (ObjNum 1342 snap 434 seq_num 1342) dirty exists 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3896: left oid 18 (ObjNum 1342 snap 434 seq_num 1342) 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3897: finishing rollback tid 0 to smithi16534841-31 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3891: done (5 left) 2024-09-10T16:31:37.226 INFO:tasks.rados.rados.0.smithi165.stdout:3892: done (4 left) 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3894: done (3 left) 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3895: done (2 left) 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3896: done (1 left) 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3898: delete oid 42 current snap is 434 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3897: finishing rollback tid 1 to smithi16534841-31 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:3897: finishing rollback tid 2 to smithi16534841-31 2024-09-10T16:31:37.227 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 2607 (ObjNum 1166 snap 366 seq_num 1166) dirty exists 2024-09-10T16:31:37.228 INFO:tasks.rados.rados.0.smithi165.stdout:3897: done (1 left) 2024-09-10T16:31:37.228 INFO:tasks.rados.rados.0.smithi165.stdout:3898: done (0 left) 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3899: read oid 32 snap -1 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3899: expect deleted 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3900: read oid 29 snap 420 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3900: expect deleted 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3901: read oid 13 snap -1 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3901: expect (ObjNum 1333 snap 433 seq_num 1333) 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3902: read oid 10 snap -1 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3902: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3903: read oid 15 snap -1 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3903: expect (ObjNum 1257 snap 405 seq_num 1257) 2024-09-10T16:31:37.229 INFO:tasks.rados.rados.0.smithi165.stdout:3904: delete oid 9 current snap is 434 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.226+0000 7f341f072640 10 monclient: get_monmap_and_config 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.226+0000 7f341f072640 10 monclient: build_initial_monmap 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.226+0000 7f341f072640 10 monclient: monmap: 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.230 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:37.228636+0000 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:37.228636+0000 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.227+0000 7f341f072640 1 Processor -- start 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.227+0000 7f341f072640 1 -- start start 2024-09-10T16:31:37.231 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.227+0000 7f341f072640 10 monclient: init 2024-09-10T16:31:37.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:37.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T16:31:37.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f341809aba0 0x7f341809af80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient(hunting): picked mon.noname-a con 0x7f341809aba0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f341806df00 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f34180a41b0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341806d590 con 0x7f341806df00 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f341806d240 con 0x7f34180a41b0 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341f072640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f341809bdb0 con 0x7f341809aba0 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341f072640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.228+0000 7f341ce0e640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f341809aba0 msgr2=0x7f341809af80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341f072640 10 monclient(hunting): authenticate will time out at 2275.717015s 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341ce0e640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f341809aba0 0x7f341809af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f3417fff640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f341806df00 legacy=0x7f341809b4c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341d60f640 1 --1- >> v1:172.21.15.165:6789/0 conn(0x7f34180a41b0 0x7f34181439a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:33150/0 (socket says 172.21.15.164:33150) 2024-09-10T16:31:37.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.229+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 learned_addr learned my addr 172.21.15.164:0/838190164 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:37.236 INFO:tasks.rados.rados.0.smithi165.stdout:3901: expect (ObjNum 1333 snap 433 seq_num 1333) 2024-09-10T16:31:37.248 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1721484 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.232+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 >> v1:172.21.15.165:6789/0 conn(0x7f34180a41b0 legacy=0x7f34181439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.232+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 DispatchQueue::entry inject delay of 0.002000 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333929687 0 0) 0x7f341806d240 con 0x7f34180a41b0 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): _negotiate 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): my global_id is 284155 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3408003520 con 0x7f34180a41b0 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3774122744 0 0) 0x7f3408003520 con 0x7f34180a41b0 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 >> v2:172.21.15.164:3300/0 conn(0x7f341809aba0 msgr2=0x7f341809af80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:37.252 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 --2- 172.21.15.164:0/838190164 >> v2:172.21.15.164:3300/0 conn(0x7f341809aba0 0x7f341809af80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 >> v1:172.21.15.164:6789/0 conn(0x7f341806df00 legacy=0x7f341809b4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _finish_hunting 0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f341806cf60 con 0x7f34180a41b0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f3410001d20 con 0x7f34180a41b0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:31:37.253 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: dump: 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 >> v1:172.21.15.165:6789/0 conn(0x7f34180a41b0 legacy=0x7f34181439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.254 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3408003a20 0x7f3408003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): picked mon.a con 0x7f3408003a20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3408004800 0x7f3408004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): picked mon.b con 0x7f3408004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 0x7f340800bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): picked mon.c con 0x7f34080099b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f340800c390 con 0x7f3408003a20 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f340800c510 con 0x7f3408004800 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.255 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f340800c710 con 0x7f34080099b0 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f34177fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f341ce0e640 1 -- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3408003a20 msgr2=0x7f3408003e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.234+0000 7f341ce0e640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3408003a20 0x7f3408003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 0x7f340800bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): get_auth_request con 0x7f34080099b0 auth_method 0 2024-09-10T16:31:37.256 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3408004800 0x7f3408004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): get_auth_request con 0x7f3408004800 auth_method 0 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:31:37.257 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient(hunting): handle_auth_done global_id 284155 payload 293 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3408004800 msgr2=0x7f3408004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3408004800 0x7f3408004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.258 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3408003a20 msgr2=0x7f3408003e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3408003a20 0x7f3408003e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient: _finish_hunting 0 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 monclient: found mon.c 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3408003520 con 0x7f34080099b0 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient: _finish_auth 0 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 10 monclient: _check_auth_tickets 2024-09-10T16:31:37.259 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341f072640 5 monclient: authenticate success, global_id 284155 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341f072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.235+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 reap_dead start 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f341d60f640 1 -- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 msgr2=0x7f340800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f341d60f640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 0x7f340800bda0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34180a45b0 tx=0x7f3408005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcdcaf75c5978d26 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341000a690 con 0x7f34080099b0 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.260 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: dump: 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.261 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f341f072640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f341002f070 con 0x7f34080099b0 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 1 -- 172.21.15.164:0/838190164 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f341002bd20 con 0x7f34080099b0 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.262 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f34177fe640 10 monclient: dump: 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.263 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f341f072640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.238+0000 7f341f072640 4 set_mon_vals no callback set 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 10 monclient: shutdown 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:37.264 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 -- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 msgr2=0x7f340800bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 0x7f340800bda0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f34180a45b0 tx=0x7f3408005830 comp rx=0 tx=0).stop 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 -- 172.21.15.164:0/838190164 shutdown_connections 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 --2- 172.21.15.164:0/838190164 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f34080099b0 0x7f340800bda0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 -- 172.21.15.164:0/838190164 >> 172.21.15.164:0/838190164 conn(0x7f34180a5030 msgr2=0x7f34180a5440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 -- 172.21.15.164:0/838190164 shutdown_connections 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 1 -- 172.21.15.164:0/838190164 wait complete. 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 10 monclient: build_initial_monmap 2024-09-10T16:31:37.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.239+0000 7f341f072640 10 monclient: monmap: 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:37.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 Processor -- start 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 -- start start 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient: init 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34080099b0 0x7f341808e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.267 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): picked mon.noname-b con 0x7f34080099b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 0x7f341808eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): picked mon.noname-c con 0x7f341808e730 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f341808f0d0 0x7f34181d77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): picked mon.noname-a con 0x7f341808f0d0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f341809b310 con 0x7f341808f0d0 2024-09-10T16:31:37.268 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f341809b190 con 0x7f34080099b0 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f341809b490 con 0x7f341808e730 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341f072640 10 monclient(hunting): authenticate will time out at 2275.728954s 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341d60f640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f341808f0d0 msgr2=0x7f34181d77a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.240+0000 7f341d60f640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f341808f0d0 0x7f34181d77a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 0x7f341808eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 0x7f341808eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46146/0 (socket says 172.21.15.164:46146) 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 -- 172.21.15.164:0/3806759827 learned_addr learned my addr 172.21.15.164:0/3806759827 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): get_auth_request con 0x7f341808e730 auth_method 0 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f341ce0e640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34080099b0 0x7f341808e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f341ce0e640 10 monclient(hunting): get_auth_request con 0x7f34080099b0 auth_method 0 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f341ce0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f341ce0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f341ce0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient(hunting): handle_auth_done global_id 274253 payload 824 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34080099b0 msgr2=0x7f341808e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f34080099b0 0x7f341808e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f341808f0d0 msgr2=0x7f34181d77a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:37.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f341808f0d0 0x7f34181d77a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient: _finish_hunting 0 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 monclient: found mon.noname-c 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.241+0000 7f3417fff640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34180914e0 con 0x7f341808e730 2024-09-10T16:31:37.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f3417fff640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 msgr2=0x7f341808eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f3417fff640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 0x7f341808eb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f340400e840 tx=0x7f340400ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3404016020 con 0x7f341808e730 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: dump: 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.273 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: _finish_auth 0 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: _check_auth_tickets 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 5 monclient: authenticate success, global_id 274253 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3404009e00 con 0x7f341808e730 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34040050d0 con 0x7f341808e730 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.274 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f34157fa640 10 monclient: dump: 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.275 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 10 monclient: _renew_subs 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f3416ffd640 4 set_mon_vals no callback set 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34180917c0 con 0x7f341808e730 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 10 monclient: _renew_subs 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.243+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f341809c670 con 0x7f341808e730 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341f072640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341f072640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 0x7f3418071550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f341809f780 con 0x7f3418071170 2024-09-10T16:31:37.276 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3404019020 con 0x7f341808e730 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 0x7f3418071550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient: get_auth_request con 0x7f3418071170 auth_method 0 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.244+0000 7f34157fa640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33fc03e1f0 0x7f33fc0406b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.245+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(875..875 src has 772..875) ==== 4831+0+0 (secure 0 0 0) 0x7f34040526a0 con 0x7f341808e730 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.245+0000 7f341ce0e640 10 monclient(hunting): handle_auth_done global_id 284161 payload 824 2024-09-10T16:31:37.277 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.245+0000 7f341ce0e640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f341809b820 con 0x7f3418071170 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.245+0000 7f341d60f640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33fc03e1f0 0x7f33fc0406b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.245+0000 7f341d60f640 10 monclient: get_auth_request con 0x7f33fc03e1f0 auth_method 0 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f341ce0e640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 msgr2=0x7f3418071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f341ce0e640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 0x7f3418071550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d700 tx=0x7f340c00dbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c005c50 con 0x7f3418071170 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 10 monclient: dump: 2024-09-10T16:31:37.278 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f341d60f640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33fc03e1f0 0x7f33fc0406b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f34180a6330 tx=0x7f3410007000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f340c005c50 con 0x7f3418071170 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:37.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:37.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 msgr2=0x7f3418071550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 0x7f3418071550 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d700 tx=0x7f340c00dbc0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 msgr2=0x7f3418071550 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:37.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.247+0000 7f34157fa640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071170 0x7f3418071550 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f340c00d700 tx=0x7f340c00dbc0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341f072640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341f072640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 0x7f34181dbd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341f072640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f341808fd40 con 0x7f3418071a90 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 0x7f34181dbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient: get_auth_request con 0x7f3418071a90 auth_method 0 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:37.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.254+0000 7f341ce0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.255+0000 7f341ce0e640 10 monclient(hunting): handle_auth_done global_id 284164 payload 824 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.255+0000 7f341ce0e640 1 -- 172.21.15.164:0/3806759827 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f34180770d0 con 0x7f3418071a90 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341ce0e640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 msgr2=0x7f34181dbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341ce0e640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 0x7f34181dbd00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f340c015000 tx=0x7f340c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f340c01bdb0 con 0x7f3418071a90 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 10 monclient: dump: 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:37.283 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f340c005110 con 0x7f3418071a90 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 msgr2=0x7f34181dbd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 0x7f34181dbd00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f340c015000 tx=0x7f340c0059d0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 msgr2=0x7f34181dbd00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:31:37.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f34157fa640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3418071a90 0x7f34181dbd00 secure :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f340c015000 tx=0x7f340c0059d0 comp rx=0 tx=0).stop 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 246, 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 31089, 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:31:37.285 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 0, 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.286 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.287 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:31:37.288 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:31:37.289 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:31:37.290 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.291 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.292 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.293 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:31:37.294 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:31:37.295 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:31:37.296 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:31:37.297 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.298 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.299 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:31:37.300 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:31:37.301 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:31:37.301 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33fc03e1f0 msgr2=0x7f33fc0406b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f33fc03e1f0 0x7f33fc0406b0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f34180a6330 tx=0x7f3410007000 comp rx=0 tx=0).stop 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 10 monclient: shutdown 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 msgr2=0x7f341808eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:37.303 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341f072640 1 --2- 172.21.15.164:0/3806759827 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f341808e730 0x7f341808eb90 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f340400e840 tx=0x7f340400ed00 comp rx=0 tx=0).stop 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.257+0000 7f341d60f640 1 -- 172.21.15.164:0/3806759827 reap_dead start 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.258+0000 7f341d60f640 1 -- 172.21.15.164:0/3806759827 reap_dead start 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.258+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 shutdown_connections 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.258+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 >> 172.21.15.164:0/3806759827 conn(0x7f34180a5030 msgr2=0x7f341809ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.258+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 shutdown_connections 2024-09-10T16:31:37.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:37.258+0000 7f341f072640 1 -- 172.21.15.164:0/3806759827 wait complete. 2024-09-10T16:31:37.305 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T16:31:37.305 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-09-10T16:31:37.305 INFO:tasks.rados.rados.0.smithi165.stdout:3902: expect (ObjNum 1091 snap 332 seq_num 1091) 2024-09-10T16:31:37.306 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:37.462 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:37.543 INFO:tasks.rados.rados.0.smithi165.stdout:3903: expect (ObjNum 1257 snap 405 seq_num 1257) 2024-09-10T16:31:37.557 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 2630223 2024-09-10T16:31:37.566 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1679076 2024-09-10T16:31:37.591 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1008195 2024-09-10T16:31:37.614 INFO:tasks.rados.rados.0.smithi165.stdout:3899: done (5 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3900: done (4 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3901: done (3 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3902: done (2 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3903: done (1 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3904: done (0 left) 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3905: rollback oid 50 current snap is 434 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 50 to 426 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3906: write oid 7 current snap is 434 2024-09-10T16:31:37.615 INFO:tasks.rados.rados.0.smithi165.stdout:3906: seq_num 1345 ranges {605415=488033,1656281=493458} 2024-09-10T16:31:37.620 INFO:tasks.rados.rados.0.smithi165.stdout:3906: writing smithi16534841-7 from 605415 to 1093448 tid 1 2024-09-10T16:31:37.628 INFO:tasks.rados.rados.0.smithi165.stdout:3906: writing smithi16534841-7 from 1656281 to 2149739 tid 2 2024-09-10T16:31:37.628 INFO:tasks.rados.rados.0.smithi165.stdout:3907: copy_from oid 23 from oid 43 current snap is 434 2024-09-10T16:31:37.628 INFO:tasks.rados.rados.0.smithi165.stdout:3905: finishing rollback tid 1 to smithi16534841-50 2024-09-10T16:31:37.628 INFO:tasks.rados.rados.0.smithi165.stdout:3905: finishing rollback tid 2 to smithi16534841-50 2024-09-10T16:31:37.629 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 50 v 1895 (ObjNum 1275 snap 408 seq_num 1275) dirty exists 2024-09-10T16:31:37.629 INFO:tasks.rados.rados.0.smithi165.stdout:3906: finishing write tid 1 to smithi16534841-7 2024-09-10T16:31:37.629 INFO:tasks.rados.rados.0.smithi165.stdout:3905: done (2 left) 2024-09-10T16:31:37.629 INFO:tasks.rados.rados.0.smithi165.stdout:3908: write oid 17 current snap is 434 2024-09-10T16:31:37.629 INFO:tasks.rados.rados.0.smithi165.stdout:3908: seq_num 1346 ranges {622222=679635,1831116=753469} 2024-09-10T16:31:37.633 INFO:tasks.rados.rados.0.smithi165.stdout:3908: writing smithi16534841-17 from 622222 to 1301857 tid 1 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3908: writing smithi16534841-17 from 1831116 to 2584585 tid 2 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3909: read oid 6 snap -1 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3909: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3910: write (excl) oid 26 current snap is 434 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3906: finishing write tid 2 to smithi16534841-7 2024-09-10T16:31:37.641 INFO:tasks.rados.rados.0.smithi165.stdout:3910: seq_num 1347 ranges {639029=754884,2173245=465784} 2024-09-10T16:31:37.648 INFO:tasks.rados.rados.0.smithi165.stdout:3910: writing smithi16534841-26 from 639029 to 1393913 tid 1 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3910: writing smithi16534841-26 from 2173245 to 2639029 tid 2 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3911: snap_create 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3906: finishing write tid 3 to smithi16534841-7 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3906: finishing write tid 4 to smithi16534841-7 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3906: oid 7 updating version 0 to 2611 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3906: oid 7 version 2611 is already newer than 2609 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:3906: oid 7 version 2611 is already newer than 2610 2024-09-10T16:31:37.656 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2611 (ObjNum 1345 snap 434 seq_num 1345) dirty exists 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:3906: left oid 7 (ObjNum 1345 snap 434 seq_num 1345) 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:3908: finishing write tid 1 to smithi16534841-17 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:3907: finishing copy_from racing read to smithi16534841-23 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:3907: finishing copy_from to smithi16534841-23 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 1727 (ObjNum 1340 snap 434 seq_num 1340) dirty exists 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:3909: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:37.657 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:37.753 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1678398 2024-09-10T16:31:37.780 INFO:tasks.rados.rados.0.smithi165.stdout:3908: finishing write tid 2 to smithi16534841-17 2024-09-10T16:31:37.780 INFO:tasks.rados.rados.0.smithi165.stdout:3908: finishing write tid 3 to smithi16534841-17 2024-09-10T16:31:37.780 INFO:tasks.rados.rados.0.smithi165.stdout:3908: finishing write tid 4 to smithi16534841-17 2024-09-10T16:31:37.780 INFO:tasks.rados.rados.0.smithi165.stdout:3908: oid 17 updating version 0 to 2028 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3908: oid 17 version 2028 is already newer than 2027 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3908: oid 17 version 2028 is already newer than 2026 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 2028 (ObjNum 1346 snap 434 seq_num 1346) dirty exists 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3908: left oid 17 (ObjNum 1346 snap 434 seq_num 1346) 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: finishing write tid 1 to smithi16534841-26 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: finishing write tid 2 to smithi16534841-26 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: finishing write tid 3 to smithi16534841-26 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: finishing write tid 4 to smithi16534841-26 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: oid 26 updating version 0 to 2614 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: oid 26 version 2614 is already newer than 2612 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: oid 26 version 2614 is already newer than 2613 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2614 (ObjNum 1347 snap 434 seq_num 1347) dirty exists 2024-09-10T16:31:37.781 INFO:tasks.rados.rados.0.smithi165.stdout:3910: left oid 26 (ObjNum 1347 snap 434 seq_num 1347) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3906: done (5 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3907: done (4 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3908: done (3 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3909: done (2 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3910: done (1 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3911: done (0 left) 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3912: write (excl) oid 4 current snap is 435 2024-09-10T16:31:38.021 INFO:tasks.rados.rados.0.smithi165.stdout:3912: seq_num 1348 ranges {655836=430133,1831727=757138} 2024-09-10T16:31:38.026 INFO:tasks.rados.rados.0.smithi165.stdout:3912: writing smithi16534841-4 from 655836 to 1085969 tid 1 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3912: writing smithi16534841-4 from 1831727 to 2588865 tid 2 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3913: copy_from oid 32 from oid 12 current snap is 435 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3912: finishing write tid 1 to smithi16534841-4 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3914: copy_from oid 44 from oid 15 current snap is 435 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3915: read oid 3 snap -1 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3915: expect (ObjNum 1308 snap 426 seq_num 1308) 2024-09-10T16:31:38.035 INFO:tasks.rados.rados.0.smithi165.stdout:3916: write (excl) oid 26 current snap is 435 2024-09-10T16:31:38.036 INFO:tasks.rados.rados.0.smithi165.stdout:3916: seq_num 1349 ranges {672643=505382,1890209=500796} 2024-09-10T16:31:38.041 INFO:tasks.rados.rados.0.smithi165.stdout:3916: writing smithi16534841-26 from 672643 to 1178025 tid 1 2024-09-10T16:31:38.049 INFO:tasks.rados.rados.0.smithi165.stdout:3916: writing smithi16534841-26 from 1890209 to 2391005 tid 2 2024-09-10T16:31:38.049 INFO:tasks.rados.rados.0.smithi165.stdout:3917: snap_remove snap 409 2024-09-10T16:31:39.026 INFO:tasks.rados.rados.0.smithi165.stdout:3917: done (5 left) 2024-09-10T16:31:39.026 INFO:tasks.rados.rados.0.smithi165.stdout:3918: write (excl) oid 48 current snap is 435 2024-09-10T16:31:39.026 INFO:tasks.rados.rados.0.smithi165.stdout:3918: seq_num 1350 ranges {689450=580631,1948691=740759} 2024-09-10T16:31:39.033 INFO:tasks.rados.rados.0.smithi165.stdout:3918: writing smithi16534841-48 from 689450 to 1270081 tid 1 2024-09-10T16:31:39.042 INFO:tasks.rados.rados.0.smithi165.stdout:3918: writing smithi16534841-48 from 1948691 to 2689450 tid 2 2024-09-10T16:31:39.042 INFO:tasks.rados.rados.0.smithi165.stdout:3919: rollback oid 33 current snap is 435 2024-09-10T16:31:39.042 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 33 to 365 2024-09-10T16:31:39.042 INFO:tasks.rados.rados.0.smithi165.stdout:3912: finishing write tid 2 to smithi16534841-4 2024-09-10T16:31:39.042 INFO:tasks.rados.rados.0.smithi165.stdout:3915: expect (ObjNum 1308 snap 426 seq_num 1308) 2024-09-10T16:31:39.049 INFO:tasks.ceph.osd.0.smithi164.stderr:2024-09-10T16:31:39.047+0000 7f41cd239640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 1a6 after trimming was started 2024-09-10T16:31:39.112 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:39.159 INFO:tasks.rados.rados.0.smithi165.stdout:3912: finishing write tid 3 to smithi16534841-4 2024-09-10T16:31:39.159 INFO:tasks.rados.rados.0.smithi165.stdout:3912: finishing write tid 4 to smithi16534841-4 2024-09-10T16:31:39.159 INFO:tasks.rados.rados.0.smithi165.stdout:3912: oid 4 updating version 0 to 1870 2024-09-10T16:31:39.159 INFO:tasks.rados.rados.0.smithi165.stdout:3912: oid 4 version 1870 is already newer than 1868 2024-09-10T16:31:39.159 INFO:tasks.rados.rados.0.smithi165.stdout:3912: oid 4 version 1870 is already newer than 1869 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1870 (ObjNum 1348 snap 435 seq_num 1348) dirty exists 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3912: left oid 4 (ObjNum 1348 snap 435 seq_num 1348) 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: finishing write tid 1 to smithi16534841-26 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: finishing write tid 2 to smithi16534841-26 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3913: finishing copy_from racing read to smithi16534841-32 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: finishing write tid 3 to smithi16534841-26 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3912: done (6 left) 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3915: done (5 left) 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3920: read oid 42 snap -1 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3920: expect deleted 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3921: write oid 19 current snap is 435 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: finishing write tid 4 to smithi16534841-26 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: oid 26 updating version 0 to 2617 2024-09-10T16:31:39.160 INFO:tasks.rados.rados.0.smithi165.stdout:3916: oid 26 updating version 2617 to 2618 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3916: oid 26 version 2618 is already newer than 2616 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 26 v 2618 (ObjNum 1349 snap 435 seq_num 1349) dirty exists 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3916: left oid 26 (ObjNum 1349 snap 435 seq_num 1349) 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3914: finishing copy_from racing read to smithi16534841-44 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3913: finishing copy_from to smithi16534841-32 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 32 v 1664 (ObjNum 1330 snap 433 seq_num 1330) dirty exists 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3914: finishing copy_from to smithi16534841-44 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1867 (ObjNum 1257 snap 405 seq_num 1257) dirty exists 2024-09-10T16:31:39.161 INFO:tasks.rados.rados.0.smithi165.stdout:3921: seq_num 1351 ranges {706257=655880,1890820=504465} 2024-09-10T16:31:39.165 INFO:tasks.rados.rados.0.smithi165.stdout:3921: writing smithi16534841-19 from 706257 to 1362137 tid 1 2024-09-10T16:31:39.172 INFO:tasks.rados.rados.0.smithi165.stdout:3921: writing smithi16534841-19 from 1890820 to 2395285 tid 2 2024-09-10T16:31:39.172 INFO:tasks.rados.rados.0.smithi165.stdout:3913: done (6 left) 2024-09-10T16:31:39.172 INFO:tasks.rados.rados.0.smithi165.stdout:3914: done (5 left) 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3916: done (4 left) 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3922: read oid 43 snap 427 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3922: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3923: rollback oid 13 current snap is 435 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 13 to 423 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3918: finishing write tid 1 to smithi16534841-48 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3919: finishing rollback tid 1 to smithi16534841-33 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 2166 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-10T16:31:39.173 INFO:tasks.rados.rados.0.smithi165.stdout:3918: finishing write tid 2 to smithi16534841-48 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: finishing write tid 3 to smithi16534841-48 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: finishing write tid 4 to smithi16534841-48 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: oid 48 updating version 0 to 2620 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: oid 48 updating version 2620 to 2630 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: oid 48 version 2630 is already newer than 2629 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 48 v 2630 (ObjNum 1350 snap 435 seq_num 1350) dirty exists 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: left oid 48 (ObjNum 1350 snap 435 seq_num 1350) 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3918: done (5 left) 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3919: done (4 left) 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3924: write oid 45 current snap is 435 2024-09-10T16:31:39.174 INFO:tasks.rados.rados.0.smithi165.stdout:3924: seq_num 1352 ranges {723064=731129,1949302=764476} 2024-09-10T16:31:39.180 INFO:tasks.rados.rados.0.smithi165.stdout:3924: writing smithi16534841-45 from 723064 to 1454193 tid 1 2024-09-10T16:31:39.187 INFO:tasks.rados.rados.0.smithi165.stdout:3924: writing smithi16534841-45 from 1949302 to 2713778 tid 2 2024-09-10T16:31:39.187 INFO:tasks.rados.rados.0.smithi165.stdout:3925: snap_create 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3921: finishing write tid 1 to smithi16534841-19 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3923: finishing rollback tid 0 to smithi16534841-13 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3923: finishing rollback tid 1 to smithi16534841-13 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 13 v 2129 (ObjNum 161 snap 0 seq_num 4062158192) dirty dne 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3921: finishing write tid 2 to smithi16534841-19 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3921: finishing write tid 3 to smithi16534841-19 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3921: finishing write tid 4 to smithi16534841-19 2024-09-10T16:31:39.188 INFO:tasks.rados.rados.0.smithi165.stdout:3921: oid 19 updating version 0 to 2636 2024-09-10T16:31:39.189 INFO:tasks.rados.rados.0.smithi165.stdout:3921: oid 19 version 2636 is already newer than 2635 2024-09-10T16:31:39.189 INFO:tasks.rados.rados.0.smithi165.stdout:3921: oid 19 version 2636 is already newer than 2634 2024-09-10T16:31:39.189 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 19 v 2636 (ObjNum 1351 snap 435 seq_num 1351) dirty exists 2024-09-10T16:31:39.189 INFO:tasks.rados.rados.0.smithi165.stdout:3921: left oid 19 (ObjNum 1351 snap 435 seq_num 1351) 2024-09-10T16:31:39.189 INFO:tasks.rados.rados.0.smithi165.stdout:3922: expect (ObjNum 1152 snap 362 seq_num 1152) 2024-09-10T16:31:39.242 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:39.277 INFO:tasks.rados.rados.0.smithi165.stdout:3924: finishing write tid 1 to smithi16534841-45 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: finishing write tid 2 to smithi16534841-45 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: finishing write tid 3 to smithi16534841-45 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: finishing write tid 4 to smithi16534841-45 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: oid 45 updating version 0 to 1730 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: oid 45 version 1730 is already newer than 1729 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: oid 45 version 1730 is already newer than 1728 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 45 v 1730 (ObjNum 1352 snap 435 seq_num 1352) dirty exists 2024-09-10T16:31:39.278 INFO:tasks.rados.rados.0.smithi165.stdout:3924: left oid 45 (ObjNum 1352 snap 435 seq_num 1352) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3920: done (5 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3921: done (4 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3922: done (3 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3923: done (2 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3924: done (1 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3925: done (0 left) 2024-09-10T16:31:40.033 INFO:tasks.rados.rados.0.smithi165.stdout:3926: snap_create 2024-09-10T16:31:41.032 INFO:tasks.rados.rados.0.smithi165.stdout:3926: done (0 left) 2024-09-10T16:31:41.032 INFO:tasks.rados.rados.0.smithi165.stdout:3927: read oid 35 snap -1 2024-09-10T16:31:41.032 INFO:tasks.rados.rados.0.smithi165.stdout:3927: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:41.032 INFO:tasks.rados.rados.0.smithi165.stdout:3928: write oid 33 current snap is 437 2024-09-10T16:31:41.032 INFO:tasks.rados.rados.0.smithi165.stdout:3928: seq_num 1353 ranges {739871=406378,1607784=508134} 2024-09-10T16:31:41.038 INFO:tasks.rados.rados.0.smithi165.stdout:3928: writing smithi16534841-33 from 739871 to 1146249 tid 1 2024-09-10T16:31:41.044 INFO:tasks.rados.rados.0.smithi165.stdout:3928: writing smithi16534841-33 from 1607784 to 2115918 tid 2 2024-09-10T16:31:41.044 INFO:tasks.rados.rados.0.smithi165.stdout:3929: rollback oid 21 current snap is 437 2024-09-10T16:31:41.044 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 433 2024-09-10T16:31:41.044 INFO:tasks.rados.rados.0.smithi165.stdout:3927: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:41.079 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 545654 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: finishing write tid 1 to smithi16534841-33 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: finishing write tid 2 to smithi16534841-33 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: finishing write tid 3 to smithi16534841-33 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: finishing write tid 4 to smithi16534841-33 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: oid 33 updating version 0 to 2223 2024-09-10T16:31:41.159 INFO:tasks.rados.rados.0.smithi165.stdout:3928: oid 33 version 2223 is already newer than 2221 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3928: oid 33 version 2223 is already newer than 2222 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 33 v 2223 (ObjNum 1353 snap 437 seq_num 1353) dirty exists 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3928: left oid 33 (ObjNum 1353 snap 437 seq_num 1353) 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3929: finishing rollback tid 0 to smithi16534841-21 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3927: done (2 left) 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3928: done (1 left) 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3930: write (excl) oid 11 current snap is 437 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3929: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 2226 (ObjNum 161 snap 0 seq_num 1162409184) dirty dne 2024-09-10T16:31:41.160 INFO:tasks.rados.rados.0.smithi165.stdout:3930: seq_num 1354 ranges {756678=597980,1782619=768145} 2024-09-10T16:31:41.166 INFO:tasks.rados.rados.0.smithi165.stdout:3930: writing smithi16534841-11 from 756678 to 1354658 tid 1 2024-09-10T16:31:41.173 INFO:tasks.rados.rados.0.smithi165.stdout:3930: writing smithi16534841-11 from 1782619 to 2550764 tid 2 2024-09-10T16:31:41.173 INFO:tasks.rados.rados.0.smithi165.stdout:3929: done (1 left) 2024-09-10T16:31:41.173 INFO:tasks.rados.rados.0.smithi165.stdout:3931: snap_create 2024-09-10T16:31:41.173 INFO:tasks.rados.rados.0.smithi165.stdout:3930: finishing write tid 1 to smithi16534841-11 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: finishing write tid 2 to smithi16534841-11 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: finishing write tid 3 to smithi16534841-11 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: finishing write tid 4 to smithi16534841-11 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: oid 11 updating version 0 to 1897 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: oid 11 updating version 1897 to 1898 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: oid 11 version 1898 is already newer than 1896 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1898 (ObjNum 1354 snap 437 seq_num 1354) dirty exists 2024-09-10T16:31:41.177 INFO:tasks.rados.rados.0.smithi165.stdout:3930: left oid 11 (ObjNum 1354 snap 437 seq_num 1354) 2024-09-10T16:31:41.185 INFO:tasks.rados.rados.0.smithi165.stdout:3930: done (1 left) 2024-09-10T16:31:41.185 INFO:tasks.rados.rados.0.smithi165.stdout:3931: done (0 left) 2024-09-10T16:31:41.185 INFO:tasks.rados.rados.0.smithi165.stdout:3932: write oid 23 current snap is 438 2024-09-10T16:31:41.185 INFO:tasks.rados.rados.0.smithi165.stdout:3932: seq_num 1355 ranges {773485=673229,2124748=511803} 2024-09-10T16:31:41.193 INFO:tasks.rados.rados.0.smithi165.stdout:3932: writing smithi16534841-23 from 773485 to 1446714 tid 1 2024-09-10T16:31:41.199 INFO:tasks.rados.rados.0.smithi165.stdout:3932: writing smithi16534841-23 from 2124748 to 2636551 tid 2 2024-09-10T16:31:41.200 INFO:tasks.rados.rados.0.smithi165.stdout:3933: write (excl) oid 34 current snap is 438 2024-09-10T16:31:41.200 INFO:tasks.rados.rados.0.smithi165.stdout:3932: finishing write tid 1 to smithi16534841-23 2024-09-10T16:31:41.200 INFO:tasks.rados.rados.0.smithi165.stdout:3933: seq_num 1356 ranges {790292=748478,2183230=607062} 2024-09-10T16:31:41.208 INFO:tasks.rados.rados.0.smithi165.stdout:3933: writing smithi16534841-34 from 790292 to 1538770 tid 1 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3933: writing smithi16534841-34 from 2183230 to 2790292 tid 2 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3934: read oid 8 snap -1 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3934: expect deleted 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3935: write (excl) oid 27 current snap is 438 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3932: finishing write tid 2 to smithi16534841-23 2024-09-10T16:31:41.215 INFO:tasks.rados.rados.0.smithi165.stdout:3935: seq_num 1357 ranges {407099=423727,1441712=515472,2582060=225039} 2024-09-10T16:31:41.219 INFO:tasks.rados.rados.0.smithi165.stdout:3935: writing smithi16534841-27 from 407099 to 830826 tid 1 2024-09-10T16:31:41.226 INFO:tasks.rados.rados.0.smithi165.stdout:3935: writing smithi16534841-27 from 1441712 to 1957184 tid 2 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3935: writing smithi16534841-27 from 2582060 to 2807099 tid 3 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3936: write (excl) oid 4 current snap is 438 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: finishing write tid 3 to smithi16534841-23 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: finishing write tid 4 to smithi16534841-23 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: oid 23 updating version 0 to 2113 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: oid 23 version 2113 is already newer than 2112 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: oid 23 version 2113 is already newer than 2111 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 2113 (ObjNum 1355 snap 438 seq_num 1355) dirty exists 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3932: left oid 23 (ObjNum 1355 snap 438 seq_num 1355) 2024-09-10T16:31:41.231 INFO:tasks.rados.rados.0.smithi165.stdout:3936: seq_num 1358 ranges {423906=498976,1500194=659130,2609483=214423} 2024-09-10T16:31:41.235 INFO:tasks.rados.rados.0.smithi165.stdout:3936: writing smithi16534841-4 from 423906 to 922882 tid 1 2024-09-10T16:31:41.242 INFO:tasks.rados.rados.0.smithi165.stdout:3936: writing smithi16534841-4 from 1500194 to 2159324 tid 2 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:3936: writing smithi16534841-4 from 2609483 to 2823906 tid 3 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:3932: done (4 left) 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:3937: rollback oid 21 current snap is 438 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 21 to 420 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:3933: finishing write tid 1 to smithi16534841-34 2024-09-10T16:31:41.248 INFO:tasks.rados.rados.0.smithi165.stdout:3933: finishing write tid 2 to smithi16534841-34 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: finishing write tid 3 to smithi16534841-34 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: finishing write tid 4 to smithi16534841-34 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: oid 34 updating version 0 to 1900 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: oid 34 version 1900 is already newer than 1898 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: oid 34 version 1900 is already newer than 1899 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1900 (ObjNum 1356 snap 438 seq_num 1356) dirty exists 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3933: left oid 34 (ObjNum 1356 snap 438 seq_num 1356) 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3935: finishing write tid 1 to smithi16534841-27 2024-09-10T16:31:41.249 INFO:tasks.rados.rados.0.smithi165.stdout:3935: finishing write tid 2 to smithi16534841-27 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3933: done (4 left) 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3934: done (3 left) 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3938: rollback oid 16 current snap is 438 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 16 to 365 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: finishing write tid 3 to smithi16534841-27 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: finishing write tid 4 to smithi16534841-27 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: finishing write tid 5 to smithi16534841-27 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: oid 27 updating version 0 to 2208 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: oid 27 updating version 2208 to 2210 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: oid 27 version 2210 is already newer than 2207 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: oid 27 version 2210 is already newer than 2209 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 27 v 2210 (ObjNum 1357 snap 438 seq_num 1357) dirty exists 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3935: left oid 27 (ObjNum 1357 snap 438 seq_num 1357) 2024-09-10T16:31:41.250 INFO:tasks.rados.rados.0.smithi165.stdout:3936: finishing write tid 1 to smithi16534841-4 2024-09-10T16:31:41.251 INFO:tasks.rados.rados.0.smithi165.stdout:3936: finishing write tid 2 to smithi16534841-4 2024-09-10T16:31:41.251 INFO:tasks.rados.rados.0.smithi165.stdout:3935: done (3 left) 2024-09-10T16:31:41.251 INFO:tasks.rados.rados.0.smithi165.stdout:3939: read oid 30 snap -1 2024-09-10T16:31:41.251 INFO:tasks.rados.rados.0.smithi165.stdout:3939: expect (ObjNum 1331 snap 433 seq_num 1331) 2024-09-10T16:31:41.251 INFO:tasks.rados.rados.0.smithi165.stdout:3940: snap_remove snap 436 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3940: done (4 left) 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3941: snap_create 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: finishing write tid 3 to smithi16534841-4 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: finishing write tid 4 to smithi16534841-4 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: finishing write tid 5 to smithi16534841-4 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: oid 4 updating version 0 to 1874 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: oid 4 version 1874 is already newer than 1871 2024-09-10T16:31:42.195 INFO:tasks.rados.rados.0.smithi165.stdout:3936: oid 4 version 1874 is already newer than 1872 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3936: oid 4 version 1874 is already newer than 1873 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1874 (ObjNum 1358 snap 438 seq_num 1358) dirty exists 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3936: left oid 4 (ObjNum 1358 snap 438 seq_num 1358) 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3937: finishing rollback tid 1 to smithi16534841-21 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3937: finishing rollback tid 2 to smithi16534841-21 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 21 v 2228 (ObjNum 1265 snap 406 seq_num 1265) dirty exists 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3938: finishing rollback tid 0 to smithi16534841-16 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3938: finishing rollback tid 1 to smithi16534841-16 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2639 (ObjNum 33 snap 0 seq_num 4) dirty dne 2024-09-10T16:31:42.196 INFO:tasks.rados.rados.0.smithi165.stdout:3939: expect (ObjNum 1331 snap 433 seq_num 1331) 2024-09-10T16:31:42.205 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1003129 2024-09-10T16:31:43.196 INFO:tasks.rados.rados.0.smithi165.stdout:3936: done (4 left) 2024-09-10T16:31:43.197 INFO:tasks.rados.rados.0.smithi165.stdout:3937: done (3 left) 2024-09-10T16:31:43.197 INFO:tasks.rados.rados.0.smithi165.stdout:3938: done (2 left) 2024-09-10T16:31:43.197 INFO:tasks.rados.rados.0.smithi165.stdout:3939: done (1 left) 2024-09-10T16:31:43.197 INFO:tasks.rados.rados.0.smithi165.stdout:3941: done (0 left) 2024-09-10T16:31:43.197 INFO:tasks.rados.rados.0.smithi165.stdout:3942: snap_create 2024-09-10T16:31:44.208 INFO:tasks.rados.rados.0.smithi165.stdout:3942: done (0 left) 2024-09-10T16:31:44.208 INFO:tasks.rados.rados.0.smithi165.stdout:3943: read oid 12 snap -1 2024-09-10T16:31:44.208 INFO:tasks.rados.rados.0.smithi165.stdout:3943: expect (ObjNum 1330 snap 433 seq_num 1330) 2024-09-10T16:31:44.208 INFO:tasks.rados.rados.0.smithi165.stdout:3944: snap_create 2024-09-10T16:31:44.213 INFO:tasks.rados.rados.0.smithi165.stdout:3943: expect (ObjNum 1330 snap 433 seq_num 1330) 2024-09-10T16:31:44.263 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3943: done (1 left) 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3944: done (0 left) 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3945: rollback oid 6 current snap is 441 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 6 to 429 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3946: read oid 29 snap -1 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3946: expect deleted 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3947: write oid 17 current snap is 441 2024-09-10T16:31:45.205 INFO:tasks.rados.rados.0.smithi165.stdout:3947: seq_num 1359 ranges {440713=574225,1442323=519141,2636906=203807} 2024-09-10T16:31:45.208 INFO:tasks.rados.rados.0.smithi165.stdout:3947: writing smithi16534841-17 from 440713 to 1014938 tid 1 2024-09-10T16:31:45.214 INFO:tasks.rados.rados.0.smithi165.stdout:3947: writing smithi16534841-17 from 1442323 to 1961464 tid 2 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3947: writing smithi16534841-17 from 2636906 to 2840713 tid 3 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3948: rollback oid 44 current snap is 441 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 44 to 439 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3945: finishing rollback tid 0 to smithi16534841-6 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3945: finishing rollback tid 1 to smithi16534841-6 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3947: finishing write tid 1 to smithi16534841-17 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3945: finishing rollback tid 2 to smithi16534841-6 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 6 v 1898 (ObjNum 1244 snap 402 seq_num 1244) dirty exists 2024-09-10T16:31:45.219 INFO:tasks.rados.rados.0.smithi165.stdout:3947: finishing write tid 2 to smithi16534841-17 2024-09-10T16:31:45.220 INFO:tasks.rados.rados.0.smithi165.stdout:3945: done (3 left) 2024-09-10T16:31:45.220 INFO:tasks.rados.rados.0.smithi165.stdout:3946: done (2 left) 2024-09-10T16:31:45.220 INFO:tasks.rados.rados.0.smithi165.stdout:3949: snap_create 2024-09-10T16:31:45.221 INFO:tasks.rados.rados.0.smithi165.stdout:3947: finishing write tid 3 to smithi16534841-17 2024-09-10T16:31:45.223 INFO:tasks.rados.rados.0.smithi165.stdout:3947: finishing write tid 4 to smithi16534841-17 2024-09-10T16:31:45.223 INFO:tasks.rados.rados.0.smithi165.stdout:3947: finishing write tid 5 to smithi16534841-17 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:3947: oid 17 updating version 0 to 2029 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:3947: oid 17 updating version 2029 to 2032 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:3947: oid 17 version 2032 is already newer than 2030 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:3947: oid 17 version 2032 is already newer than 2031 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 17 v 2032 (ObjNum 1359 snap 441 seq_num 1359) dirty exists 2024-09-10T16:31:45.224 INFO:tasks.rados.rados.0.smithi165.stdout:3947: left oid 17 (ObjNum 1359 snap 441 seq_num 1359) 2024-09-10T16:31:45.225 INFO:tasks.rados.rados.0.smithi165.stdout:3948: finishing rollback tid 0 to smithi16534841-44 2024-09-10T16:31:45.225 INFO:tasks.rados.rados.0.smithi165.stdout:3948: finishing rollback tid 1 to smithi16534841-44 2024-09-10T16:31:45.225 INFO:tasks.rados.rados.0.smithi165.stdout:3948: finishing rollback tid 2 to smithi16534841-44 2024-09-10T16:31:45.225 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 44 v 1870 (ObjNum 1257 snap 405 seq_num 1257) dirty exists 2024-09-10T16:31:46.184 INFO:tasks.rados.rados.0.smithi165.stdout:3947: done (2 left) 2024-09-10T16:31:46.185 INFO:tasks.rados.rados.0.smithi165.stdout:3948: done (1 left) 2024-09-10T16:31:46.185 INFO:tasks.rados.rados.0.smithi165.stdout:3949: done (0 left) 2024-09-10T16:31:46.185 INFO:tasks.rados.rados.0.smithi165.stdout:3950: snap_remove snap 406 2024-09-10T16:31:47.186 INFO:tasks.rados.rados.0.smithi165.stdout:3950: done (0 left) 2024-09-10T16:31:47.186 INFO:tasks.rados.rados.0.smithi165.stdout:3951: delete oid 13 current snap is 442 2024-09-10T16:31:47.195 INFO:tasks.rados.rados.0.smithi165.stdout:3951: done (0 left) 2024-09-10T16:31:47.196 INFO:tasks.rados.rados.0.smithi165.stdout:3952: write (excl) oid 31 current snap is 442 2024-09-10T16:31:47.196 INFO:tasks.rados.rados.0.smithi165.stdout:3952: seq_num 1361 ranges {474327=441076,1675640=522810,2808105=66222} 2024-09-10T16:31:47.201 INFO:tasks.rados.rados.0.smithi165.stdout:3952: writing smithi16534841-31 from 474327 to 915403 tid 1 2024-09-10T16:31:47.208 INFO:tasks.rados.rados.0.smithi165.stdout:3952: writing smithi16534841-31 from 1675640 to 2198450 tid 2 2024-09-10T16:31:47.212 INFO:tasks.rados.rados.0.smithi165.stdout:3952: writing smithi16534841-31 from 2808105 to 2874327 tid 3 2024-09-10T16:31:47.212 INFO:tasks.rados.rados.0.smithi165.stdout:3953: delete oid 16 current snap is 442 2024-09-10T16:31:47.212 INFO:tasks.rados.rados.0.smithi165.stdout:3952: finishing write tid 1 to smithi16534841-31 2024-09-10T16:31:47.212 INFO:tasks.rados.rados.0.smithi165.stdout:3952: finishing write tid 2 to smithi16534841-31 2024-09-10T16:31:47.215 INFO:tasks.rados.rados.0.smithi165.stdout:3952: finishing write tid 3 to smithi16534841-31 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: finishing write tid 4 to smithi16534841-31 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: finishing write tid 5 to smithi16534841-31 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: oid 31 updating version 0 to 2656 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: oid 31 version 2656 is already newer than 2651 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: oid 31 version 2656 is already newer than 2655 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: oid 31 version 2656 is already newer than 2654 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 2656 (ObjNum 1361 snap 442 seq_num 1361) dirty exists 2024-09-10T16:31:47.216 INFO:tasks.rados.rados.0.smithi165.stdout:3952: left oid 31 (ObjNum 1361 snap 442 seq_num 1361) 2024-09-10T16:31:47.217 INFO:tasks.rados.rados.0.smithi165.stdout:3952: done (1 left) 2024-09-10T16:31:47.217 INFO:tasks.rados.rados.0.smithi165.stdout:3953: done (0 left) 2024-09-10T16:31:47.217 INFO:tasks.rados.rados.0.smithi165.stdout:3954: read oid 38 snap 439 2024-09-10T16:31:47.217 INFO:tasks.rados.rados.0.smithi165.stdout:3954: expect (ObjNum 1313 snap 426 seq_num 1313) 2024-09-10T16:31:47.217 INFO:tasks.rados.rados.0.smithi165.stdout:3955: snap_create 2024-09-10T16:31:47.224 INFO:tasks.rados.rados.0.smithi165.stdout:3954: expect (ObjNum 1313 snap 426 seq_num 1313) 2024-09-10T16:31:47.224 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 203003 2024-09-10T16:31:47.288 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:48.194 INFO:tasks.rados.rados.0.smithi165.stdout:3954: done (1 left) 2024-09-10T16:31:48.194 INFO:tasks.rados.rados.0.smithi165.stdout:3955: done (0 left) 2024-09-10T16:31:48.194 INFO:tasks.rados.rados.0.smithi165.stdout:3956: copy_from oid 7 from oid 48 current snap is 443 2024-09-10T16:31:48.194 INFO:tasks.rados.rados.0.smithi165.stdout:3957: delete oid 12 current snap is 443 2024-09-10T16:31:48.198 INFO:tasks.rados.rados.0.smithi165.stdout:3957: done (1 left) 2024-09-10T16:31:48.198 INFO:tasks.rados.rados.0.smithi165.stdout:3958: read oid 45 snap -1 2024-09-10T16:31:48.198 INFO:tasks.rados.rados.0.smithi165.stdout:3958: expect (ObjNum 1352 snap 435 seq_num 1352) 2024-09-10T16:31:48.198 INFO:tasks.rados.rados.0.smithi165.stdout:3959: copy_from oid 16 from oid 39 current snap is 443 2024-09-10T16:31:48.199 INFO:tasks.rados.rados.0.smithi165.stdout:3960: rollback oid 34 current snap is 443 2024-09-10T16:31:48.199 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 34 to 407 2024-09-10T16:31:48.199 INFO:tasks.rados.rados.0.smithi165.stdout:3961: snap_remove snap 432 2024-09-10T16:31:49.196 INFO:tasks.rados.rados.0.smithi165.stdout:3961: done (4 left) 2024-09-10T16:31:49.196 INFO:tasks.rados.rados.0.smithi165.stdout:3962: snap_remove snap 381 2024-09-10T16:31:50.201 INFO:tasks.rados.rados.0.smithi165.stdout:3962: done (4 left) 2024-09-10T16:31:50.201 INFO:tasks.rados.rados.0.smithi165.stdout:3963: rollback oid 30 current snap is 443 2024-09-10T16:31:50.202 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 30 to 417 2024-09-10T16:31:50.202 INFO:tasks.rados.rados.0.smithi165.stdout:3958: expect (ObjNum 1352 snap 435 seq_num 1352) 2024-09-10T16:31:50.202 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:50.254 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3960: finishing rollback tid 0 to smithi16534841-34 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3960: finishing rollback tid 1 to smithi16534841-34 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3960: finishing rollback tid 2 to smithi16534841-34 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 34 v 1903 (ObjNum 1268 snap 406 seq_num 1268) dirty exists 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3959: finishing copy_from racing read to smithi16534841-16 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3956: finishing copy_from to smithi16534841-7 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 7 v 2630 (ObjNum 1350 snap 435 seq_num 1350) dirty exists 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3956: finishing copy_from racing read to smithi16534841-7 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3959: finishing copy_from to smithi16534841-16 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 16 v 2061 (ObjNum 1300 snap 423 seq_num 1300) dirty exists 2024-09-10T16:31:50.353 INFO:tasks.rados.rados.0.smithi165.stdout:3956: done (4 left) 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3958: done (3 left) 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3959: done (2 left) 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3960: done (1 left) 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3964: rollback oid 12 current snap is 443 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 12 to 441 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3963: finishing rollback tid 0 to smithi16534841-30 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3963: finishing rollback tid 1 to smithi16534841-30 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3963: finishing rollback tid 2 to smithi16534841-30 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 30 v 2678 (ObjNum 977 snap 297 seq_num 977) dirty exists 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3963: done (1 left) 2024-09-10T16:31:50.354 INFO:tasks.rados.rados.0.smithi165.stdout:3965: delete oid 6 current snap is 443 2024-09-10T16:31:50.355 INFO:tasks.rados.rados.0.smithi165.stdout:3965: done (1 left) 2024-09-10T16:31:50.355 INFO:tasks.rados.rados.0.smithi165.stdout:3966: snap_remove snap 429 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3964: finishing rollback tid 1 to smithi16534841-12 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3964: finishing rollback tid 2 to smithi16534841-12 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 12 v 1667 (ObjNum 1330 snap 433 seq_num 1330) dirty exists 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3964: done (1 left) 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3966: done (0 left) 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3967: read oid 11 snap -1 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3967: expect (ObjNum 1354 snap 437 seq_num 1354) 2024-09-10T16:31:51.186 INFO:tasks.rados.rados.0.smithi165.stdout:3968: write (excl) oid 4 current snap is 443 2024-09-10T16:31:51.187 INFO:tasks.rados.rados.0.smithi165.stdout:3968: seq_num 1365 ranges {541555=742072,1793215=530148,2801444=140111} 2024-09-10T16:31:51.194 INFO:tasks.rados.rados.0.smithi165.stdout:3968: writing smithi16534841-4 from 541555 to 1283627 tid 1 2024-09-10T16:31:51.200 INFO:tasks.rados.rados.0.smithi165.stdout:3968: writing smithi16534841-4 from 1793215 to 2323363 tid 2 2024-09-10T16:31:51.204 INFO:tasks.rados.rados.0.smithi165.stdout:3968: writing smithi16534841-4 from 2801444 to 2941555 tid 3 2024-09-10T16:31:51.204 INFO:tasks.rados.rados.0.smithi165.stdout:3969: delete oid 30 current snap is 443 2024-09-10T16:31:51.204 INFO:tasks.rados.rados.0.smithi165.stdout:3967: expect (ObjNum 1354 snap 437 seq_num 1354) 2024-09-10T16:31:51.252 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:51.298 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:51.345 INFO:tasks.rados.rados.0.smithi165.stdout:3968: finishing write tid 1 to smithi16534841-4 2024-09-10T16:31:51.345 INFO:tasks.rados.rados.0.smithi165.stdout:3968: finishing write tid 2 to smithi16534841-4 2024-09-10T16:31:51.345 INFO:tasks.rados.rados.0.smithi165.stdout:3968: finishing write tid 3 to smithi16534841-4 2024-09-10T16:31:51.345 INFO:tasks.rados.rados.0.smithi165.stdout:3968: finishing write tid 4 to smithi16534841-4 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: finishing write tid 5 to smithi16534841-4 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: oid 4 updating version 0 to 1877 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: oid 4 version 1877 is already newer than 1875 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: oid 4 version 1877 is already newer than 1876 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: oid 4 updating version 1877 to 1878 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 1878 (ObjNum 1365 snap 443 seq_num 1365) dirty exists 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: left oid 4 (ObjNum 1365 snap 443 seq_num 1365) 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3967: done (2 left) 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3968: done (1 left) 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3969: done (0 left) 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3970: rollback oid 11 current snap is 443 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 11 to 394 2024-09-10T16:31:51.346 INFO:tasks.rados.rados.0.smithi165.stdout:3971: copy_from oid 29 from oid 9 current snap is 443 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:3972: rollback oid 8 current snap is 443 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 8 to 428 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:3973: read oid 19 snap 426 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:3973: expect deleted 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:3974: snap_create 2024-09-10T16:31:51.347 INFO:tasks.rados.rados.0.smithi165.stdout:3971: finishing copy_from to smithi16534841-29 2024-09-10T16:31:51.348 INFO:tasks.rados.rados.0.smithi165.stdout:3971: got expected ENOENT (src dne) 2024-09-10T16:31:51.348 INFO:tasks.rados.rados.0.smithi165.stdout:3971: finishing copy_from racing read to smithi16534841-29 2024-09-10T16:31:51.348 INFO:tasks.rados.rados.0.smithi165.stdout:3972: finishing rollback tid 1 to smithi16534841-8 2024-09-10T16:31:51.348 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 8 v 1911 (ObjNum 1162327120 snap 21859 seq_num 304) dirty dne 2024-09-10T16:31:51.349 INFO:tasks.rados.rados.0.smithi165.stdout:3970: finishing rollback tid 0 to smithi16534841-11 2024-09-10T16:31:51.349 INFO:tasks.rados.rados.0.smithi165.stdout:3970: finishing rollback tid 1 to smithi16534841-11 2024-09-10T16:31:51.350 INFO:tasks.rados.rados.0.smithi165.stdout:3970: finishing rollback tid 2 to smithi16534841-11 2024-09-10T16:31:51.350 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 11 v 1901 (ObjNum 1187 snap 374 seq_num 1187) dirty exists 2024-09-10T16:31:52.207 INFO:tasks.rados.rados.0.smithi165.stdout:3970: done (4 left) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3971: done (3 left) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3972: done (2 left) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3973: done (1 left) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3974: done (0 left) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3975: read oid 35 snap -1 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3975: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3976: read oid 7 snap -1 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3976: expect (ObjNum 1350 snap 435 seq_num 1350) 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3977: write (excl) oid 22 current snap is 444 2024-09-10T16:31:52.208 INFO:tasks.rados.rados.0.smithi165.stdout:3977: seq_num 1367 ranges {575169=492570,1793826=417464,2623584=351585} 2024-09-10T16:31:52.213 INFO:tasks.rados.rados.0.smithi165.stdout:3977: writing smithi16534841-22 from 575169 to 1067739 tid 1 2024-09-10T16:31:52.220 INFO:tasks.rados.rados.0.smithi165.stdout:3977: writing smithi16534841-22 from 1793826 to 2211290 tid 2 2024-09-10T16:31:52.225 INFO:tasks.rados.rados.0.smithi165.stdout:3977: writing smithi16534841-22 from 2623584 to 2975169 tid 3 2024-09-10T16:31:52.225 INFO:tasks.rados.rados.0.smithi165.stdout:3978: snap_create 2024-09-10T16:31:52.225 INFO:tasks.rados.rados.0.smithi165.stdout:3975: expect (ObjNum 1244 snap 402 seq_num 1244) 2024-09-10T16:31:52.225 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:52.350 INFO:tasks.rados.rados.0.smithi165.stdout:3976: expect (ObjNum 1350 snap 435 seq_num 1350) 2024-09-10T16:31:52.350 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 195719 2024-09-10T16:31:52.488 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1589797 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: finishing write tid 1 to smithi16534841-22 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: finishing write tid 2 to smithi16534841-22 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: finishing write tid 3 to smithi16534841-22 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: finishing write tid 4 to smithi16534841-22 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: finishing write tid 5 to smithi16534841-22 2024-09-10T16:31:52.511 INFO:tasks.rados.rados.0.smithi165.stdout:3977: oid 22 updating version 0 to 2133 2024-09-10T16:31:52.512 INFO:tasks.rados.rados.0.smithi165.stdout:3977: oid 22 version 2133 is already newer than 2130 2024-09-10T16:31:52.512 INFO:tasks.rados.rados.0.smithi165.stdout:3977: oid 22 version 2133 is already newer than 2131 2024-09-10T16:31:52.512 INFO:tasks.rados.rados.0.smithi165.stdout:3977: oid 22 version 2133 is already newer than 2132 2024-09-10T16:31:52.512 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 22 v 2133 (ObjNum 1367 snap 444 seq_num 1367) dirty exists 2024-09-10T16:31:52.512 INFO:tasks.rados.rados.0.smithi165.stdout:3977: left oid 22 (ObjNum 1367 snap 444 seq_num 1367) 2024-09-10T16:31:53.213 INFO:tasks.rados.rados.0.smithi165.stdout:3975: done (3 left) 2024-09-10T16:31:53.213 INFO:tasks.rados.rados.0.smithi165.stdout:3976: done (2 left) 2024-09-10T16:31:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3977: done (1 left) 2024-09-10T16:31:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3978: done (0 left) 2024-09-10T16:31:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3979: write (excl) oid 31 current snap is 445 2024-09-10T16:31:53.214 INFO:tasks.rados.rados.0.smithi165.stdout:3979: seq_num 1368 ranges {591976=567819,1852308=677475} 2024-09-10T16:31:53.219 INFO:tasks.rados.rados.0.smithi165.stdout:3979: writing smithi16534841-31 from 591976 to 1159795 tid 1 2024-09-10T16:31:53.227 INFO:tasks.rados.rados.0.smithi165.stdout:3979: writing smithi16534841-31 from 1852308 to 2529783 tid 2 2024-09-10T16:31:53.227 INFO:tasks.rados.rados.0.smithi165.stdout:3980: delete oid 34 current snap is 445 2024-09-10T16:31:53.228 INFO:tasks.rados.rados.0.smithi165.stdout:3979: finishing write tid 1 to smithi16534841-31 2024-09-10T16:31:53.232 INFO:tasks.rados.rados.0.smithi165.stdout:3980: done (1 left) 2024-09-10T16:31:53.232 INFO:tasks.rados.rados.0.smithi165.stdout:3981: write oid 23 current snap is 445 2024-09-10T16:31:53.232 INFO:tasks.rados.rados.0.smithi165.stdout:3981: seq_num 1370 ranges {625590=434670,1685625=681144,2822206=203384} 2024-09-10T16:31:53.238 INFO:tasks.rados.rados.0.smithi165.stdout:3981: writing smithi16534841-23 from 625590 to 1060260 tid 1 2024-09-10T16:31:53.245 INFO:tasks.rados.rados.0.smithi165.stdout:3981: writing smithi16534841-23 from 1685625 to 2366769 tid 2 2024-09-10T16:31:53.249 INFO:tasks.rados.rados.0.smithi165.stdout:3981: writing smithi16534841-23 from 2822206 to 3025590 tid 3 2024-09-10T16:31:53.249 INFO:tasks.rados.rados.0.smithi165.stdout:3982: copy_from oid 4 from oid 22 current snap is 445 2024-09-10T16:31:53.249 INFO:tasks.rados.rados.0.smithi165.stdout:3979: finishing write tid 2 to smithi16534841-31 2024-09-10T16:31:53.249 INFO:tasks.rados.rados.0.smithi165.stdout:3979: finishing write tid 3 to smithi16534841-31 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: finishing write tid 4 to smithi16534841-31 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: oid 31 updating version 0 to 2707 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: oid 31 version 2707 is already newer than 2706 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: oid 31 version 2707 is already newer than 2705 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 31 v 2707 (ObjNum 1368 snap 445 seq_num 1368) dirty exists 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: left oid 31 (ObjNum 1368 snap 445 seq_num 1368) 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3981: finishing write tid 1 to smithi16534841-23 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3981: finishing write tid 2 to smithi16534841-23 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3979: done (2 left) 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3983: write oid 29 current snap is 445 2024-09-10T16:31:53.250 INFO:tasks.rados.rados.0.smithi165.stdout:3983: seq_num 1371 ranges {642397=509919,1627754=424802,2733276=309121} 2024-09-10T16:31:53.255 INFO:tasks.rados.rados.0.smithi165.stdout:3983: writing smithi16534841-29 from 642397 to 1152316 tid 1 2024-09-10T16:31:53.260 INFO:tasks.rados.rados.0.smithi165.stdout:3983: writing smithi16534841-29 from 1627754 to 2052556 tid 2 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:3983: writing smithi16534841-29 from 2733276 to 3042397 tid 3 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:3984: read oid 2 snap -1 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:3984: expect (ObjNum 1313 snap 426 seq_num 1313) 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:3985: rollback oid 10 current snap is 445 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:rollback oid 10 to 408 2024-09-10T16:31:53.267 INFO:tasks.rados.rados.0.smithi165.stdout:3981: finishing write tid 3 to smithi16534841-23 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: finishing write tid 4 to smithi16534841-23 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: finishing write tid 5 to smithi16534841-23 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: oid 23 updating version 0 to 2142 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: oid 23 version 2142 is already newer than 2141 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: oid 23 updating version 2142 to 2144 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: oid 23 version 2144 is already newer than 2143 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 23 v 2144 (ObjNum 1370 snap 445 seq_num 1370) dirty exists 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3981: left oid 23 (ObjNum 1370 snap 445 seq_num 1370) 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3983: finishing write tid 1 to smithi16534841-29 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3982: finishing copy_from racing read to smithi16534841-4 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3983: finishing write tid 2 to smithi16534841-29 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:3982: finishing copy_from to smithi16534841-4 2024-09-10T16:31:53.268 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 4 v 2133 (ObjNum 1367 snap 444 seq_num 1367) dirty exists 2024-09-10T16:31:53.269 INFO:tasks.rados.rados.0.smithi165.stdout:3981: done (4 left) 2024-09-10T16:31:53.269 INFO:tasks.rados.rados.0.smithi165.stdout:3982: done (3 left) 2024-09-10T16:31:53.269 INFO:tasks.rados.rados.0.smithi165.stdout:3986: copy_from oid 43 from oid 47 current snap is 445 2024-09-10T16:31:53.269 INFO:tasks.rados.rados.0.smithi165.stdout:3987: snap_create 2024-09-10T16:31:53.272 INFO:tasks.rados.rados.0.smithi165.stdout:3985: finishing rollback tid 0 to smithi16534841-10 2024-09-10T16:31:53.272 INFO:tasks.rados.rados.0.smithi165.stdout:3985: finishing rollback tid 1 to smithi16534841-10 2024-09-10T16:31:53.272 INFO:tasks.rados.rados.0.smithi165.stdout:3985: finishing rollback tid 2 to smithi16534841-10 2024-09-10T16:31:53.272 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 10 v 2035 (ObjNum 1091 snap 332 seq_num 1091) dirty exists 2024-09-10T16:31:53.274 INFO:tasks.rados.rados.0.smithi165.stdout:3983: finishing write tid 3 to smithi16534841-29 2024-09-10T16:31:53.274 INFO:tasks.rados.rados.0.smithi165.stdout:3983: finishing write tid 4 to smithi16534841-29 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: finishing write tid 5 to smithi16534841-29 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: oid 29 updating version 0 to 2231 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: oid 29 updating version 2231 to 2233 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: oid 29 version 2233 is already newer than 2232 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: oid 29 version 2233 is already newer than 2230 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 2233 (ObjNum 1371 snap 445 seq_num 1371) dirty exists 2024-09-10T16:31:53.275 INFO:tasks.rados.rados.0.smithi165.stdout:3983: left oid 29 (ObjNum 1371 snap 445 seq_num 1371) 2024-09-10T16:31:53.276 INFO:tasks.rados.rados.0.smithi165.stdout:3984: expect (ObjNum 1313 snap 426 seq_num 1313) 2024-09-10T16:31:53.276 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:53.350 INFO:tasks.rados.rados.0.smithi165.stdout:3986: finishing copy_from racing read to smithi16534841-43 2024-09-10T16:31:53.351 INFO:tasks.rados.rados.0.smithi165.stdout:3986: finishing copy_from to smithi16534841-43 2024-09-10T16:31:53.351 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 43 v 2192 (ObjNum 1339 snap 434 seq_num 1339) dirty exists 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3983: done (4 left) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3984: done (3 left) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3985: done (2 left) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3986: done (1 left) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3987: done (0 left) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3988: read oid 20 snap -1 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3988: expect (ObjNum 1309 snap 426 seq_num 1309) 2024-09-10T16:31:54.219 INFO:tasks.rados.rados.0.smithi165.stdout:3989: snap_create 2024-09-10T16:31:54.225 INFO:tasks.rados.rados.0.smithi165.stdout:3988: expect (ObjNum 1309 snap 426 seq_num 1309) 2024-09-10T16:31:54.225 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:54.284 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 1050445 2024-09-10T16:31:55.220 INFO:tasks.rados.rados.0.smithi165.stdout:3988: done (1 left) 2024-09-10T16:31:55.221 INFO:tasks.rados.rados.0.smithi165.stdout:3989: done (0 left) 2024-09-10T16:31:55.221 INFO:tasks.rados.rados.0.smithi165.stdout:3990: copy_from oid 38 from oid 42 current snap is 447 2024-09-10T16:31:55.221 INFO:tasks.rados.rados.0.smithi165.stdout:3991: copy_from oid 35 from oid 9 current snap is 447 2024-09-10T16:31:55.221 INFO:tasks.rados.rados.0.smithi165.stdout:3992: snap_create 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3990: finishing copy_from to smithi16534841-38 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3990: got expected ENOENT (src dne) 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3991: finishing copy_from to smithi16534841-35 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3991: got expected ENOENT (src dne) 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3990: finishing copy_from racing read to smithi16534841-38 2024-09-10T16:31:55.225 INFO:tasks.rados.rados.0.smithi165.stdout:3991: finishing copy_from racing read to smithi16534841-35 2024-09-10T16:31:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:3990: done (2 left) 2024-09-10T16:31:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:3991: done (1 left) 2024-09-10T16:31:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:3992: done (0 left) 2024-09-10T16:31:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:3993: write (excl) oid 29 current snap is 448 2024-09-10T16:31:56.188 INFO:tasks.rados.rados.0.smithi165.stdout:3993: seq_num 1372 ranges {659204=585168,1686236=684813} 2024-09-10T16:31:56.194 INFO:tasks.rados.rados.0.smithi165.stdout:3993: writing smithi16534841-29 from 659204 to 1244372 tid 1 2024-09-10T16:31:56.201 INFO:tasks.rados.rados.0.smithi165.stdout:3993: writing smithi16534841-29 from 1686236 to 2371049 tid 2 2024-09-10T16:31:56.201 INFO:tasks.rados.rados.0.smithi165.stdout:3994: read oid 13 snap -1 2024-09-10T16:31:56.201 INFO:tasks.rados.rados.0.smithi165.stdout:3994: expect deleted 2024-09-10T16:31:56.201 INFO:tasks.rados.rados.0.smithi165.stdout:3995: write (excl) oid 15 current snap is 448 2024-09-10T16:31:56.201 INFO:tasks.rados.rados.0.smithi165.stdout:3995: seq_num 1373 ranges {676011=660417,1744718=428471,2788122=287889} 2024-09-10T16:31:56.208 INFO:tasks.rados.rados.0.smithi165.stdout:3995: writing smithi16534841-15 from 676011 to 1336428 tid 1 2024-09-10T16:31:56.213 INFO:tasks.rados.rados.0.smithi165.stdout:3995: writing smithi16534841-15 from 1744718 to 2173189 tid 2 2024-09-10T16:31:56.218 INFO:tasks.rados.rados.0.smithi165.stdout:3995: writing smithi16534841-15 from 2788122 to 3076011 tid 3 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3996: read oid 49 snap -1 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3996: expect (ObjNum 1327 snap 431 seq_num 1327) 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3997: read oid 45 snap 403 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3997: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3998: write (excl) oid 35 current snap is 448 2024-09-10T16:31:56.219 INFO:tasks.rados.rados.0.smithi165.stdout:3998: seq_num 1374 ranges {692818=735666,2203200=688482} 2024-09-10T16:31:56.227 INFO:tasks.rados.rados.0.smithi165.stdout:3998: writing smithi16534841-35 from 692818 to 1428484 tid 1 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3998: writing smithi16534841-35 from 2203200 to 2891682 tid 2 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3999: read oid 16 snap -1 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3999: expect (ObjNum 1300 snap 423 seq_num 1300) 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 7 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: finishing write tid 1 to smithi16534841-29 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: finishing write tid 2 to smithi16534841-29 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: finishing write tid 3 to smithi16534841-29 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: finishing write tid 4 to smithi16534841-29 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: oid 29 updating version 0 to 2236 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: oid 29 version 2236 is already newer than 2235 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:3993: oid 29 updating version 2236 to 2237 2024-09-10T16:31:56.236 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 29 v 2237 (ObjNum 1372 snap 448 seq_num 1372) dirty exists 2024-09-10T16:31:56.237 INFO:tasks.rados.rados.0.smithi165.stdout:3993: left oid 29 (ObjNum 1372 snap 448 seq_num 1372) 2024-09-10T16:31:56.237 INFO:tasks.rados.rados.0.smithi165.stdout:3993: done (6 left) 2024-09-10T16:31:56.237 INFO:tasks.rados.rados.0.smithi165.stdout:3994: done (5 left) 2024-09-10T16:31:56.237 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 5 2024-09-10T16:31:56.237 INFO:tasks.rados.rados.0.smithi165.stdout:3996: expect (ObjNum 1327 snap 431 seq_num 1327) 2024-09-10T16:31:56.277 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:56.361 INFO:tasks.rados.rados.0.smithi165.stdout:3995: finishing write tid 1 to smithi16534841-15 2024-09-10T16:31:56.362 INFO:tasks.rados.rados.0.smithi165.stdout:3995: finishing write tid 2 to smithi16534841-15 2024-09-10T16:31:56.362 INFO:tasks.rados.rados.0.smithi165.stdout:3996: done (4 left) 2024-09-10T16:31:56.362 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 4 2024-09-10T16:31:56.362 INFO:tasks.rados.rados.0.smithi165.stdout:3995: finishing write tid 3 to smithi16534841-15 2024-09-10T16:31:56.362 INFO:tasks.rados.rados.0.smithi165.stdout:3997: expect (ObjNum 1208 snap 385 seq_num 1208) 2024-09-10T16:31:56.424 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 0 2024-09-10T16:31:56.539 INFO:tasks.rados.rados.0.smithi165.stdout:3995: finishing write tid 4 to smithi16534841-15 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3997: done (3 left) 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 3 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: finishing write tid 5 to smithi16534841-15 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: oid 15 updating version 0 to 2136 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: oid 15 version 2136 is already newer than 2135 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: oid 15 version 2136 is already newer than 2134 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: oid 15 updating version 2136 to 2137 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 15 v 2137 (ObjNum 1373 snap 448 seq_num 1373) dirty exists 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: left oid 15 (ObjNum 1373 snap 448 seq_num 1373) 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3998: finishing write tid 1 to smithi16534841-35 2024-09-10T16:31:56.540 INFO:tasks.rados.rados.0.smithi165.stdout:3995: done (2 left) 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout: waiting on 2 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: finishing write tid 2 to smithi16534841-35 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: finishing write tid 3 to smithi16534841-35 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: finishing write tid 4 to smithi16534841-35 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: oid 35 updating version 0 to 2135 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: oid 35 updating version 2135 to 2136 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: oid 35 version 2136 is already newer than 2134 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:update_object_version oid 35 v 2136 (ObjNum 1374 snap 448 seq_num 1374) dirty exists 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3998: left oid 35 (ObjNum 1374 snap 448 seq_num 1374) 2024-09-10T16:31:56.541 INFO:tasks.rados.rados.0.smithi165.stdout:3999: expect (ObjNum 1300 snap 423 seq_num 1300) 2024-09-10T16:31:56.751 INFO:tasks.rados.rados.0.smithi165.stdout:seeking to 847839 2024-09-10T16:31:56.847 INFO:tasks.rados.rados.0.smithi165.stdout:3998: done (1 left) 2024-09-10T16:31:56.847 INFO:tasks.rados.rados.0.smithi165.stdout:3999: done (0 left) 2024-09-10T16:31:56.848 INFO:tasks.rados.rados.0.smithi165.stderr:0 errors. 2024-09-10T16:31:56.848 INFO:tasks.rados.rados.0.smithi165.stderr: 2024-09-10T16:31:56.851 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.959+0000 7f3d371c6640 10 monclient: get_monmap_and_config 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.959+0000 7f3d371c6640 10 monclient: build_initial_monmap 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.959+0000 7f3d371c6640 10 monclient: monmap: 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:56.960772+0000 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:56.960772+0000 2024-09-10T16:31:56.961 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:56.962 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.960+0000 7f3d371c6640 1 Processor -- start 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.960+0000 7f3d371c6640 1 -- start start 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.960+0000 7f3d371c6640 10 monclient: init 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient: _add_conns ranks=[0,2,4,3,1] 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3d300a0410 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d300a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 0x7f3d3013db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d300a29b0 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d300a0210 con 0x7f3d300a0410 2024-09-10T16:31:56.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f3d3009d3f0 con 0x7f3d300a29b0 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d3009ed90 con 0x7f3d300a2550 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d34f62640 1 -- >> v1:172.21.15.164:6789/0 conn(0x7f3d300a0410 legacy=0x7f3d300a1e40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.164:6789/0 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d371c6640 10 monclient(hunting): authenticate will time out at 2295.448318s 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48126/0 (socket says 172.21.15.164:48126) 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 -- 172.21.15.164:0/2882992090 learned_addr learned my addr 172.21.15.164:0/2882992090 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): get_auth_request con 0x7f3d300a2550 auth_method 0 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:56.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d35763640 1 --2- 172.21.15.164:0/2882992090 >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 0x7f3d3013db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient(hunting): handle_auth_done global_id 274256 payload 274 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 -- 172.21.15.164:0/2882992090 >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 msgr2=0x7f3d3013db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 --2- 172.21.15.164:0/2882992090 >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 0x7f3d3013db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 -- 172.21.15.164:0/2882992090 >> v1:172.21.15.164:6789/0 conn(0x7f3d300a0410 legacy=0x7f3d300a1e40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 monclient: found mon.noname-b 2024-09-10T16:31:56.967 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:56.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:31:56.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d2ffff640 1 -- 172.21.15.164:0/2882992090 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d3009d930 con 0x7f3d300a2550 2024-09-10T16:31:56.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d35763640 10 monclient: get_auth_request con 0x7f3d300a29b0 auth_method 0 2024-09-10T16:31:56.968 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.961+0000 7f3d35763640 1 --2- 172.21.15.164:0/2882992090 >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 0x7f3d3013db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:31:56.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2ffff640 1 -- 172.21.15.164:0/2882992090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 msgr2=0x7f3d3013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:56.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2ffff640 1 --2- 172.21.15.164:0/2882992090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d30074db0 tx=0x7f3d1c02a080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1973e35fa8da3583 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:56.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 1 -- 172.21.15.164:0/2882992090 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c037070 con 0x7f3d300a2550 2024-09-10T16:31:56.975 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: dump: 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.976 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 1 -- 172.21.15.164:0/2882992090 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c032640 con 0x7f3d300a2550 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 1 -- 172.21.15.164:0/2882992090 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c032a10 con 0x7f3d300a2550 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d371c6640 5 monclient: authenticate success, global_id 274256 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d371c6640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d371c6640 4 set_mon_vals no callback set 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.964+0000 7f3d2f7fe640 10 monclient: dump: 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:56.977 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 10 monclient: shutdown 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 -- 172.21.15.164:0/2882992090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 msgr2=0x7f3d3013b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 --2- 172.21.15.164:0/2882992090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3d30074db0 tx=0x7f3d1c02a080 comp rx=0 tx=0).stop 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 -- 172.21.15.164:0/2882992090 shutdown_connections 2024-09-10T16:31:56.978 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 --2- 172.21.15.164:0/2882992090 >> v2:172.21.15.165:3300/0 conn(0x7f3d300a29b0 0x7f3d3013db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 --2- 172.21.15.164:0/2882992090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a2550 0x7f3d3013b5d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 -- 172.21.15.164:0/2882992090 >> 172.21.15.164:0/2882992090 conn(0x7f3d300a3110 msgr2=0x7f3d300a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 -- 172.21.15.164:0/2882992090 shutdown_connections 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 1 -- 172.21.15.164:0/2882992090 wait complete. 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 10 monclient: build_initial_monmap 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.965+0000 7f3d371c6640 10 monclient: monmap: 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:56.979 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 Processor -- start 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 -- start start 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient: init 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.980 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-c con 0x7f3d300a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 0x7f3d300997e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-a con 0x7f3d300a2550 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d300a29b0 0x7f3d301b0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): picked mon.noname-b con 0x7f3d300a29b0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3d300a07f0 con 0x7f3d300a2550 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3009c910 con 0x7f3d300a29b0 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3d300a0970 con 0x7f3d300a0410 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.966+0000 7f3d371c6640 10 monclient(hunting): authenticate will time out at 2295.453995s 2024-09-10T16:31:56.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d2ffff640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 msgr2=0x7f3d300997e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d2ffff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 0x7f3d300997e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48128/0 (socket says 172.21.15.164:48128) 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 -- 172.21.15.164:0/11466829 learned_addr learned my addr 172.21.15.164:0/11466829 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): get_auth_request con 0x7f3d300a0410 auth_method 0 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d300a29b0 0x7f3d301b0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:56.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): get_auth_request con 0x7f3d300a29b0 auth_method 0 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d35763640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient(hunting): handle_auth_done global_id 274259 payload 824 2024-09-10T16:31:56.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 -- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d300a29b0 msgr2=0x7f3d301b0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d300a29b0 0x7f3d301b0200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 -- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 msgr2=0x7f3d300997e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 0x7f3d300997e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient: _finish_hunting 0 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 monclient: found mon.noname-c 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.967+0000 7f3d34f62640 1 -- 172.21.15.164:0/11466829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d301b34d0 con 0x7f3d300a0410 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d34f62640 1 -- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 msgr2=0x7f3d300992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d34f62640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b550 tx=0x7f3d2400ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24005070 con 0x7f3d300a0410 2024-09-10T16:31:56.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: dump: 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.985 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d24015070 con 0x7f3d300a0410 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 5 monclient: authenticate success, global_id 274259 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24005450 con 0x7f3d300a0410 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 10 monclient: _renew_subs 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d301b37b0 con 0x7f3d300a0410 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2d7fa640 10 monclient: dump: 2024-09-10T16:31:56.986 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d2effd640 4 set_mon_vals no callback set 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 10 monclient: _renew_subs 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.969+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d301bb200 con 0x7f3d300a0410 2024-09-10T16:31:56.987 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.970+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3d24005630 con 0x7f3d300a0410 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.970+0000 7f3d371c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.970+0000 7f3d371c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.970+0000 7f3d371c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.970+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d3009fb30 con 0x7f3d300a0410 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.971+0000 7f3d2d7fa640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 0x7f3d14040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.971+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(895..895 src has 772..895) ==== 4799+0+0 (secure 0 0 0) 0x7f3d24052ae0 con 0x7f3d300a0410 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.971+0000 7f3d2ffff640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 0x7f3d14040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.971+0000 7f3d2ffff640 10 monclient: get_auth_request con 0x7f3d1403e1c0 auth_method 0 2024-09-10T16:31:56.988 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.973+0000 7f3d2ffff640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 0x7f3d14040680 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3d30075190 tx=0x7f3d1c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:56.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.975+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f3d240514d0 con 0x7f3d300a0410 2024-09-10T16:31:56.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.975+0000 7f3d2d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:56.989 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:56.975+0000 7f3d2d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:57.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.152+0000 7f3d371c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:31:57.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.152+0000 7f3d371c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:31:57.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.152+0000 7f3d371c6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:57.154 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.152+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3d30077260 con 0x7f3d300a0410 2024-09-10T16:31:57.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.153+0000 7f3d2d7fa640 1 -- 172.21.15.164:0/11466829 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v895) ==== 74+0+14386 (secure 0 0 0) 0x7f3d2401a460 con 0x7f3d300a0410 2024-09-10T16:31:57.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.153+0000 7f3d2d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:31:57.155 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.153+0000 7f3d2d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:57.155 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:31:57.155 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":895,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:31:56.181584+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T16:06:20.696007+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"unique_pool_0","create_time":"2024-09-10T16:06:45.857820+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"895","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":460,"snap_epoch":895,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":442,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":391,"length":1},{"begin":419,"length":1},{"begin":445,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 msgr2=0x7f3d14040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 0x7f3d14040680 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f3d30075190 tx=0x7f3d1c02e000 comp rx=0 tx=0).stop 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 10 monclient: shutdown 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 msgr2=0x7f3d300992a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3d2400b550 tx=0x7f3d2400ba10 comp rx=0 tx=0).stop 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 shutdown_connections 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3d1403e1c0 0x7f3d14040680 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.157 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3d300a29b0 0x7f3d301b0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3d300a2550 0x7f3d300997e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 --2- 172.21.15.164:0/11466829 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3d300a0410 0x7f3d300992a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 >> 172.21.15.164:0/11466829 conn(0x7f3d300a3110 msgr2=0x7f3d301bd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 shutdown_connections 2024-09-10T16:31:57.158 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.155+0000 7f3d371c6640 1 -- 172.21.15.164:0/11466829 wait complete. 2024-09-10T16:31:57.168 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:31:57.265 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.264+0000 7f0ba969b640 10 monclient: get_monmap_and_config 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.264+0000 7f0ba969b640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.264+0000 7f0ba969b640 10 monclient: monmap: 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:57.265397+0000 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:57.265397+0000 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:57.266 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:57.267 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:57.267 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 Processor -- start 2024-09-10T16:31:57.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 -- start start 2024-09-10T16:31:57.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient: init 2024-09-10T16:31:57.269 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 0x7f0ba4135000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba406cd00 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ba40a46e0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ba40a4b30 0x7f0ba4138300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba40a4b30 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ba406df10 con 0x7f0ba40a46e0 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba40a4f10 con 0x7f0ba406cd00 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f0ba406d0e0 con 0x7f0ba40a4b30 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.270 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba969b640 10 monclient(hunting): authenticate will time out at 2295.752655s 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba2ffd640 1 -- >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 msgr2=0x7f0ba4135000 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba2ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 0x7f0ba4135000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ba40a4b30 0x7f0ba4138300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f0ba40a4b30 0x7f0ba4138300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.165:3300/0 says I am v2:172.21.15.164:50198/0 (socket says 172.21.15.164:50198) 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 1 -- 172.21.15.164:0/1260624502 learned_addr learned my addr 172.21.15.164:0/1260624502 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 10 monclient(hunting): get_auth_request con 0x7f0ba40a4b30 auth_method 0 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba27fc640 1 --1- 172.21.15.164:0/1260624502 >> v1:172.21.15.165:6789/0 conn(0x7f0ba40a46e0 0x7f0ba4137bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.165:6789/0 says I am v1:172.21.15.164:59356/0 (socket says 172.21.15.164:59356) 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.265+0000 7f0ba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.271 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient(hunting): handle_auth_done global_id 284170 payload 274 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 1 -- 172.21.15.164:0/1260624502 >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 msgr2=0x7f0ba4135000 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 1 --2- 172.21.15.164:0/1260624502 >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 0x7f0ba4135000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 1 -- 172.21.15.164:0/1260624502 >> v1:172.21.15.165:6789/0 conn(0x7f0ba40a46e0 legacy=0x7f0ba4137bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 1 monclient: found mon.noname-c 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.272 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.266+0000 7f0ba37fe640 1 -- 172.21.15.164:0/1260624502 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba409cb90 con 0x7f0ba40a4b30 2024-09-10T16:31:57.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba37fe640 1 -- 172.21.15.164:0/1260624502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba40a4b30 msgr2=0x7f0ba4138300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba37fe640 1 --2- 172.21.15.164:0/1260624502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40a5090 tx=0x7f0b9c02a580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8bfadb62d4c96ebf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 1 -- 172.21.15.164:0/1260624502 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c036070 con 0x7f0ba40a4b30 2024-09-10T16:31:57.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.279 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: dump: 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.280 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 5 monclient: authenticate success, global_id 284170 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 1 -- 172.21.15.164:0/1260624502 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b9c007800 con 0x7f0ba40a4b30 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 1 -- 172.21.15.164:0/1260624502 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b9c007bd0 con 0x7f0ba40a4b30 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 4 set_mon_vals no callback set 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba1ffb640 10 monclient: dump: 2024-09-10T16:31:57.281 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 10 monclient: shutdown 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 -- 172.21.15.164:0/1260624502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba40a4b30 msgr2=0x7f0ba4138300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1260624502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138300 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f0ba40a5090 tx=0x7f0b9c02a580 comp rx=0 tx=0).stop 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 -- 172.21.15.164:0/1260624502 shutdown_connections 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1260624502 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138300 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1260624502 >> v2:172.21.15.164:3300/0 conn(0x7f0ba406cd00 0x7f0ba4135000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 -- 172.21.15.164:0/1260624502 >> 172.21.15.164:0/1260624502 conn(0x7f0ba40aaa70 msgr2=0x7f0ba40aae80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 -- 172.21.15.164:0/1260624502 shutdown_connections 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.269+0000 7f0ba969b640 1 -- 172.21.15.164:0/1260624502 wait complete. 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.270+0000 7f0ba969b640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.270+0000 7f0ba969b640 10 monclient: monmap: 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:57.283 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.270+0000 7f0ba969b640 1 Processor -- start 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.270+0000 7f0ba969b640 1 -- start start 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.270+0000 7f0ba969b640 10 monclient: init 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:31:57.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406cd00 0x7f0ba413eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-b con 0x7f0ba406cd00 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-c con 0x7f0ba40a46e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): picked mon.noname-a con 0x7f0ba40a4b30 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba4138840 con 0x7f0ba40a4b30 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0ba4134b80 con 0x7f0ba406cd00 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0ba41389c0 con 0x7f0ba40a46e0 2024-09-10T16:31:57.285 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba969b640 10 monclient(hunting): authenticate will time out at 2295.758464s 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba37fe640 1 -- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 msgr2=0x7f0ba4138d20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.164:3300/0 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba37fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48142/0 (socket says 172.21.15.164:48142) 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 -- 172.21.15.164:0/1051091598 learned_addr learned my addr 172.21.15.164:0/1051091598 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): get_auth_request con 0x7f0ba40a46e0 auth_method 0 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406cd00 0x7f0ba413eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.286 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): get_auth_request con 0x7f0ba406cd00 auth_method 0 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient(hunting): handle_auth_done global_id 274262 payload 824 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 -- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406cd00 msgr2=0x7f0ba413eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406cd00 0x7f0ba413eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 -- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 msgr2=0x7f0ba4138d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:57.287 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138d20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 monclient: found mon.noname-c 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.271+0000 7f0ba27fc640 1 -- 172.21.15.164:0/1051091598 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ba40a7df0 con 0x7f0ba40a46e0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba27fc640 1 -- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 msgr2=0x7f0ba413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba27fc640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c450 tx=0x7f0b8c00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c017070 con 0x7f0ba40a46e0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: dump: 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.288 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 5 monclient: authenticate success, global_id 274262 2024-09-10T16:31:57.289 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c005810 con 0x7f0ba40a46e0 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 10 monclient: _renew_subs 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ba413ba60 con 0x7f0ba40a46e0 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c005c00 con 0x7f0ba40a46e0 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0b9b7fe640 10 monclient: dump: 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.290 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba17fa640 4 set_mon_vals no callback set 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 10 monclient: _renew_subs 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.274+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ba413a260 con 0x7f0ba40a46e0 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0b8c01a070 con 0x7f0ba40a46e0 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba969b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba969b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba969b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ba4135000 con 0x7f0ba40a46e0 2024-09-10T16:31:57.291 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0b9b7fe640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 0x7f0b84040680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(895..895 src has 772..895) ==== 4799+0+0 (secure 0 0 0) 0x7f0b8c052170 con 0x7f0ba40a46e0 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba2ffd640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 0x7f0b84040680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.275+0000 7f0ba2ffd640 10 monclient: get_auth_request con 0x7f0b8403e1c0 auth_method 0 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.278+0000 7f0ba2ffd640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 0x7f0b84040680 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0b94003980 tx=0x7f0b94009110 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.280+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0b8c050b60 con 0x7f0ba40a46e0 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.280+0000 7f0b9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.292 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.280+0000 7f0b9b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:57.306 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T16:31:57.306 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T16:31:57.306 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T16:31:57.309 INFO:tasks.ceph.mon.a:Started 2024-09-10T16:31:57.309 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:31:57.309 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:31:57.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.405+0000 7f271fc79640 10 monclient: get_monmap_and_config 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.405+0000 7f271fc79640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.405+0000 7f271fc79640 10 monclient: monmap: 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:57.406337+0000 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:57.406337+0000 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:57.413 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.405+0000 7f271fc79640 1 Processor -- start 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 -- start start 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient: init 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f271806cb70 0x7f2718134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-c con 0x7f271806cb70 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f27180a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:31:57.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-a con 0x7f27180a4ab0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f271809ce50 con 0x7f27180a46a0 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f271806db60 con 0x7f27180a4ab0 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f271806cf50 con 0x7f271806cb70 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271fc79640 10 monclient(hunting): authenticate will time out at 2295.893731s 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37600/0 (socket says 172.21.15.164:37600) 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 -- 172.21.15.164:0/778376201 learned_addr learned my addr 172.21.15.164:0/778376201 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 10 monclient(hunting): get_auth_request con 0x7f27180a4ab0 auth_method 0 2024-09-10T16:31:57.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 -- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 msgr2=0x7f27181383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 -- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 msgr2=0x7f27181383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 --2- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271e216640 1 --2- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.406+0000 7f271da15640 1 --2- 172.21.15.164:0/778376201 >> v2:172.21.15.165:3300/0 conn(0x7f271806cb70 0x7f2718134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): get_auth_request con 0x7f271806cb70 auth_method 0 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.416 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient(hunting): handle_auth_done global_id 284176 payload 274 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 1 -- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 msgr2=0x7f27181383b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 1 --2- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 1 -- 172.21.15.164:0/778376201 >> v1:172.21.15.165:6789/0 conn(0x7f27180a46a0 legacy=0x7f2718137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 1 monclient: found mon.noname-c 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.407+0000 7f271da15640 1 -- 172.21.15.164:0/778376201 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f271809cb70 con 0x7f271806cb70 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271da15640 1 -- 172.21.15.164:0/778376201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271806cb70 msgr2=0x7f2718134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.417 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271da15640 1 --2- 172.21.15.164:0/778376201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271806cb70 0x7f2718134f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402a010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e71290b6cb6ef769 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 1 -- 172.21.15.164:0/778376201 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714037070 con 0x7f271806cb70 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: dump: 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.418 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 1 -- 172.21.15.164:0/778376201 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2714032440 con 0x7f271806cb70 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 1 -- 172.21.15.164:0/778376201 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2714032810 con 0x7f271806cb70 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271ca13640 10 monclient: dump: 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.419 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 5 monclient: authenticate success, global_id 284176 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 4 set_mon_vals no callback set 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 10 monclient: shutdown 2024-09-10T16:31:57.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 -- 172.21.15.164:0/778376201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271806cb70 msgr2=0x7f2718134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 --2- 172.21.15.164:0/778376201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271806cb70 0x7f2718134f70 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f27140097a0 tx=0x7f271402a010 comp rx=0 tx=0).stop 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 -- 172.21.15.164:0/778376201 shutdown_connections 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 --2- 172.21.15.164:0/778376201 >> v2:172.21.15.164:3300/0 conn(0x7f27180a4ab0 0x7f27181383b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 --2- 172.21.15.164:0/778376201 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f271806cb70 0x7f2718134f70 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 -- 172.21.15.164:0/778376201 >> 172.21.15.164:0/778376201 conn(0x7f27180aa900 msgr2=0x7f27180aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 -- 172.21.15.164:0/778376201 shutdown_connections 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.410+0000 7f271fc79640 1 -- 172.21.15.164:0/778376201 wait complete. 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.411+0000 7f271fc79640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.411+0000 7f271fc79640 10 monclient: monmap: 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.421 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.411+0000 7f271fc79640 1 Processor -- start 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.411+0000 7f271fc79640 1 -- start start 2024-09-10T16:31:57.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.411+0000 7f271fc79640 10 monclient: init 2024-09-10T16:31:57.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:57.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-a con 0x7f271806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:57.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 0x7f271812e4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-b con 0x7f27180a46a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 0x7f271812ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): picked mon.noname-c con 0x7f27180a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2718138a70 con 0x7f271806cb70 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f27181388f0 con 0x7f27180a46a0 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f2718140040 con 0x7f27180a4ab0 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271fc79640 10 monclient(hunting): authenticate will time out at 2295.899170s 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 0x7f271812ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 0x7f271812e4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37614/0 (socket says 172.21.15.164:37614) 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): get_auth_request con 0x7f27180a4ab0 auth_method 0 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 -- 172.21.15.164:0/3413420372 learned_addr learned my addr 172.21.15.164:0/3413420372 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): get_auth_request con 0x7f27180a46a0 auth_method 0 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 10 monclient(hunting): get_auth_request con 0x7f271806cb70 auth_method 0 2024-09-10T16:31:57.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 msgr2=0x7f27180a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 msgr2=0x7f27180a3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271da15640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient(hunting): handle_auth_done global_id 284179 payload 824 2024-09-10T16:31:57.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 msgr2=0x7f271812ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 0x7f271812ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 msgr2=0x7f27180a3ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 monclient: found mon.noname-b 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271d214640 1 -- 172.21.15.164:0/3413420372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27180a13f0 con 0x7f27180a46a0 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.412+0000 7f271e216640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 0x7f271812ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.414+0000 7f271d214640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 msgr2=0x7f271812e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.414+0000 7f271d214640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 0x7f271812e4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f270800c410 tx=0x7f270800c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2708017070 con 0x7f27180a46a0 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: dump: 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2708005230 con 0x7f27180a46a0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 5 monclient: authenticate success, global_id 284179 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _renew_subs 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f271812ef60 con 0x7f27180a46a0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27080117a0 con 0x7f27180a46a0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270e7fc640 10 monclient: dump: 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _renew_subs 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f270ffff640 4 set_mon_vals no callback set 2024-09-10T16:31:57.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27180a2eb0 con 0x7f27180a46a0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.415+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f27180ab190 con 0x7f27180a46a0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.416+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f270801a070 con 0x7f27180a46a0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.416+0000 7f270e7fc640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 0x7f26fc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.416+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(895..895 src has 772..895) ==== 4799+0+0 (secure 0 0 0) 0x7f2708052130 con 0x7f27180a46a0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.416+0000 7f271da15640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 0x7f26fc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.416+0000 7f271da15640 10 monclient: get_auth_request con 0x7f26fc03e210 auth_method 0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.418+0000 7f271da15640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 0x7f26fc0406d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2714035040 tx=0x7f271402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.420+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f27180ab190 con 0x7f27180a46a0 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.420+0000 7f270e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.434 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.420+0000 7f270e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:57.463 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.462+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0ba41381e0 con 0x7f0b8403e1c0 2024-09-10T16:31:57.465 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.464+0000 7f0b9b7fe640 1 -- 172.21.15.164:0/1051091598 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+78477 (secure 0 0 0) 0x7f0ba41381e0 con 0x7f0b8403e1c0 2024-09-10T16:31:57.466 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:31:57.466 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 msgr2=0x7f0b84040680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 0x7f0b84040680 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0b94003980 tx=0x7f0b94009110 comp rx=0 tx=0).stop 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 10 monclient: shutdown 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 msgr2=0x7f0ba413f3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c00c450 tx=0x7f0b8c00c850 comp rx=0 tx=0).stop 2024-09-10T16:31:57.468 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 shutdown_connections 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0b8403e1c0 0x7f0b84040680 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0ba40a4b30 0x7f0ba4138d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0ba40a46e0 0x7f0ba413f3f0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 --2- 172.21.15.164:0/1051091598 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0ba406cd00 0x7f0ba413eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 >> 172.21.15.164:0/1051091598 conn(0x7f0ba40aaa70 msgr2=0x7f0ba4133400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 shutdown_connections 2024-09-10T16:31:57.469 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.466+0000 7f0ba969b640 1 -- 172.21.15.164:0/1051091598 wait complete. 2024-09-10T16:31:57.479 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1662,"stamp":"2024-09-10T16:31:56.184291+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":581555894,"num_objects":259,"num_object_clones":205,"num_object_copies":518,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":8,"num_read":7664,"num_read_kb":5847035,"num_write":5476,"num_write_kb":2073626,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20081,"ondisk_log_size":20081,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":922372,"kb_used_data":640872,"kb_used_omap":9,"kb_used_meta":281334,"kb_avail":565308668,"statfs":{"total":579820584960,"available":578876076032,"internally_reserved":0,"allocated":656252928,"data_stored":937154587,"data_compressed":20157022,"data_compressed_allocated":13537280,"data_compressed_original":41758838,"omap_allocated":9565,"internal_metadata":288086691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14598403,"num_objects":8,"num_object_clones":8,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":2,"num_read":53,"num_read_kb":40588,"num_write":30,"num_write_kb":10398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.979720"},"pg_stats":[{"pgid":"3.f","version":"890'1030","reported_seq":5302,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.222120+0000","last_change":"2024-09-10T16:31:51.200264+0000","last_active":"2024-09-10T16:31:53.222120+0000","last_peered":"2024-09-10T16:31:53.222120+0000","last_clean":"2024-09-10T16:31:53.222120+0000","last_became_active":"2024-09-10T16:06:47.871547+0000","last_became_peered":"2024-09-10T16:06:47.871547+0000","last_unstale":"2024-09-10T16:31:53.222120+0000","last_undegraded":"2024-09-10T16:31:53.222120+0000","last_fullsized":"2024-09-10T16:31:53.222120+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":1030,"log_dups_size":0,"ondisk_log_size":1030,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:01:10.100129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011010265999999999,"stat_sum":{"num_bytes":24826022,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":497,"num_read_kb":413382,"num_write":292,"num_write_kb":104064,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.e","version":"890'606","reported_seq":4838,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.250958+0000","last_change":"2024-09-10T16:31:51.190358+0000","last_active":"2024-09-10T16:31:53.250958+0000","last_peered":"2024-09-10T16:31:53.250958+0000","last_clean":"2024-09-10T16:31:53.250958+0000","last_became_active":"2024-09-10T16:06:47.872121+0000","last_became_peered":"2024-09-10T16:06:47.872121+0000","last_unstale":"2024-09-10T16:31:53.250958+0000","last_undegraded":"2024-09-10T16:31:53.250958+0000","last_fullsized":"2024-09-10T16:31:53.250958+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":606,"log_dups_size":0,"ondisk_log_size":606,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:43:32.227157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040537510000000004,"stat_sum":{"num_bytes":15021494,"num_objects":7,"num_object_clones":5,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":193,"num_read_kb":132735,"num_write":171,"num_write_kb":81157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.d","version":"892'789","reported_seq":4999,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.326521+0000","last_change":"2024-09-10T16:31:51.199511+0000","last_active":"2024-09-10T16:31:55.326521+0000","last_peered":"2024-09-10T16:31:55.326521+0000","last_clean":"2024-09-10T16:31:55.326521+0000","last_became_active":"2024-09-10T16:06:47.872188+0000","last_became_peered":"2024-09-10T16:06:47.872188+0000","last_unstale":"2024-09-10T16:31:55.326521+0000","last_undegraded":"2024-09-10T16:31:55.326521+0000","last_fullsized":"2024-09-10T16:31:55.326521+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":789,"log_dups_size":0,"ondisk_log_size":789,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:21:11.843977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047318560000000004,"stat_sum":{"num_bytes":29761437,"num_objects":12,"num_object_clones":10,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":280,"num_read_kb":254511,"num_write":219,"num_write_kb":98378,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.c","version":"892'2707","reported_seq":6900,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.233241+0000","last_change":"2024-09-10T16:31:51.213103+0000","last_active":"2024-09-10T16:31:53.233241+0000","last_peered":"2024-09-10T16:31:53.233241+0000","last_clean":"2024-09-10T16:31:53.233241+0000","last_became_active":"2024-09-10T16:06:47.873329+0000","last_became_peered":"2024-09-10T16:06:47.873329+0000","last_unstale":"2024-09-10T16:31:53.233241+0000","last_undegraded":"2024-09-10T16:31:53.233241+0000","last_fullsized":"2024-09-10T16:31:53.233241+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":2707,"log_dups_size":0,"ondisk_log_size":2707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:53.172359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.024349869999999999,"stat_sum":{"num_bytes":79347482,"num_objects":38,"num_object_clones":31,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":1,"num_read":1107,"num_read_kb":905287,"num_write":725,"num_write_kb":292276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.b","version":"890'2139","reported_seq":6290,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-09-10T16:31:52.205573+0000","last_change":"2024-09-10T16:31:51.193278+0000","last_active":"2024-09-10T16:31:52.205573+0000","last_peered":"2024-09-10T16:31:52.205573+0000","last_clean":"2024-09-10T16:31:52.205573+0000","last_became_active":"2024-09-10T16:06:47.872511+0000","last_became_peered":"2024-09-10T16:06:47.872511+0000","last_unstale":"2024-09-10T16:31:52.205573+0000","last_undegraded":"2024-09-10T16:31:52.205573+0000","last_fullsized":"2024-09-10T16:31:52.205573+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":2139,"log_dups_size":0,"ondisk_log_size":2139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:15:08.375090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0096686989999999993,"stat_sum":{"num_bytes":72313388,"num_objects":28,"num_object_clones":23,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":0,"num_read":738,"num_read_kb":574915,"num_write":541,"num_write_kb":212456,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.a","version":"892'1135","reported_seq":5394,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.271307+0000","last_change":"2024-09-10T16:31:51.197005+0000","last_active":"2024-09-10T16:31:53.271307+0000","last_peered":"2024-09-10T16:31:53.271307+0000","last_clean":"2024-09-10T16:31:53.271307+0000","last_became_active":"2024-09-10T16:06:47.872105+0000","last_became_peered":"2024-09-10T16:06:47.872105+0000","last_unstale":"2024-09-10T16:31:53.271307+0000","last_undegraded":"2024-09-10T16:31:53.271307+0000","last_fullsized":"2024-09-10T16:31:53.271307+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":1135,"log_dups_size":0,"ondisk_log_size":1135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T19:06:46.668154+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0081489920000000007,"stat_sum":{"num_bytes":40265420,"num_objects":17,"num_object_clones":14,"num_object_copies":34,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":461,"num_read_kb":350626,"num_write":312,"num_write_kb":100440,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.9","version":"892'819","reported_seq":5079,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.261670+0000","last_change":"2024-09-10T16:31:51.192439+0000","last_active":"2024-09-10T16:31:53.261670+0000","last_peered":"2024-09-10T16:31:53.261670+0000","last_clean":"2024-09-10T16:31:53.261670+0000","last_became_active":"2024-09-10T16:06:47.875517+0000","last_became_peered":"2024-09-10T16:06:47.875517+0000","last_unstale":"2024-09-10T16:31:53.261670+0000","last_undegraded":"2024-09-10T16:31:53.261670+0000","last_fullsized":"2024-09-10T16:31:53.261670+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:43:52.568061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0037511179999999999,"stat_sum":{"num_bytes":24592261,"num_objects":10,"num_object_clones":8,"num_object_copies":20,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":377,"num_read_kb":315055,"num_write":227,"num_write_kb":85144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.8","version":"890'1209","reported_seq":5326,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219269+0000","last_change":"2024-09-10T16:31:51.192465+0000","last_active":"2024-09-10T16:31:55.219269+0000","last_peered":"2024-09-10T16:31:55.219269+0000","last_clean":"2024-09-10T16:31:55.219269+0000","last_became_active":"2024-09-10T16:06:47.871471+0000","last_became_peered":"2024-09-10T16:06:47.871471+0000","last_unstale":"2024-09-10T16:31:55.219269+0000","last_undegraded":"2024-09-10T16:31:55.219269+0000","last_fullsized":"2024-09-10T16:31:55.219269+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":1209,"log_dups_size":0,"ondisk_log_size":1209,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:06.832276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0079949249999999999,"stat_sum":{"num_bytes":44181609,"num_objects":20,"num_object_clones":17,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":324,"num_read_kb":210226,"num_write":292,"num_write_kb":102778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":1833,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.326196+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:31:55.326196+0000","last_peered":"2024-09-10T16:31:55.326196+0000","last_clean":"2024-09-10T16:31:55.326196+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:31:55.326196+0000","last_undegraded":"2024-09-10T16:31:55.326196+0000","last_fullsized":"2024-09-10T16:31:55.326196+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1743,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.326228+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:31:55.326228+0000","last_peered":"2024-09-10T16:31:55.326228+0000","last_clean":"2024-09-10T16:31:55.326228+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:31:55.326228+0000","last_undegraded":"2024-09-10T16:31:55.326228+0000","last_fullsized":"2024-09-10T16:31:55.326228+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2","version":"890'1215","reported_seq":5434,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-09-10T16:31:52.205527+0000","last_change":"2024-09-10T16:31:51.197068+0000","last_active":"2024-09-10T16:31:52.205527+0000","last_peered":"2024-09-10T16:31:52.205527+0000","last_clean":"2024-09-10T16:31:52.205527+0000","last_became_active":"2024-09-10T16:06:47.874261+0000","last_became_peered":"2024-09-10T16:06:47.874261+0000","last_unstale":"2024-09-10T16:31:52.205527+0000","last_undegraded":"2024-09-10T16:31:52.205527+0000","last_fullsized":"2024-09-10T16:31:52.205527+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":1215,"log_dups_size":0,"ondisk_log_size":1215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T18:20:01.893122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013450821,"stat_sum":{"num_bytes":23842857,"num_objects":11,"num_object_clones":8,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":439,"num_read_kb":339500,"num_write":312,"num_write_kb":124289,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":1747,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.250879+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:31:53.250879+0000","last_peered":"2024-09-10T16:31:53.250879+0000","last_clean":"2024-09-10T16:31:53.250879+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:31:53.250879+0000","last_undegraded":"2024-09-10T16:31:53.250879+0000","last_fullsized":"2024-09-10T16:31:53.250879+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"892'862","reported_seq":5016,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219283+0000","last_change":"2024-09-10T16:31:51.189056+0000","last_active":"2024-09-10T16:31:55.219283+0000","last_peered":"2024-09-10T16:31:55.219283+0000","last_clean":"2024-09-10T16:31:55.219283+0000","last_became_active":"2024-09-10T16:06:47.871558+0000","last_became_peered":"2024-09-10T16:06:47.871558+0000","last_unstale":"2024-09-10T16:31:55.219283+0000","last_undegraded":"2024-09-10T16:31:55.219283+0000","last_fullsized":"2024-09-10T16:31:55.219283+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":862,"log_dups_size":0,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:06:26.837248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0047550989999999996,"stat_sum":{"num_bytes":23522923,"num_objects":11,"num_object_clones":9,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":271,"num_read_kb":193012,"num_write":223,"num_write_kb":97445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":1757,"reported_epoch":891,"state":"active+clean","last_fresh":"2024-09-10T16:31:52.205441+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:31:52.205441+0000","last_peered":"2024-09-10T16:31:52.205441+0000","last_clean":"2024-09-10T16:31:52.205441+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:31:52.205441+0000","last_undegraded":"2024-09-10T16:31:52.205441+0000","last_fullsized":"2024-09-10T16:31:52.205441+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.0","version":"892'2233","reported_seq":6350,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.274587+0000","last_change":"2024-09-10T16:31:51.199026+0000","last_active":"2024-09-10T16:31:53.274587+0000","last_peered":"2024-09-10T16:31:53.274587+0000","last_clean":"2024-09-10T16:31:53.274587+0000","last_became_active":"2024-09-10T16:06:47.874102+0000","last_became_peered":"2024-09-10T16:06:47.874102+0000","last_unstale":"2024-09-10T16:31:53.274587+0000","last_undegraded":"2024-09-10T16:31:53.274587+0000","last_fullsized":"2024-09-10T16:31:53.274587+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":2233,"log_dups_size":0,"ondisk_log_size":2233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:07:53.645749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061149960000000001,"stat_sum":{"num_bytes":51377472,"num_objects":25,"num_object_clones":19,"num_object_copies":50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":904,"num_read_kb":662306,"num_write":568,"num_write_kb":229790,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":1757,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.221902+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:31:53.221902+0000","last_peered":"2024-09-10T16:31:53.221902+0000","last_clean":"2024-09-10T16:31:53.221902+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:31:53.221902+0000","last_undegraded":"2024-09-10T16:31:53.221902+0000","last_fullsized":"2024-09-10T16:31:53.221902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3","version":"889'460","reported_seq":4733,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.222024+0000","last_change":"2024-09-10T16:31:51.203594+0000","last_active":"2024-09-10T16:31:53.222024+0000","last_peered":"2024-09-10T16:31:53.222024+0000","last_clean":"2024-09-10T16:31:53.222024+0000","last_became_active":"2024-09-10T16:06:47.874258+0000","last_became_peered":"2024-09-10T16:06:47.874258+0000","last_unstale":"2024-09-10T16:31:53.222024+0000","last_undegraded":"2024-09-10T16:31:53.222024+0000","last_fullsized":"2024-09-10T16:31:53.222024+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:40:30.137518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011862513999999999,"stat_sum":{"num_bytes":5083773,"num_objects":3,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":108043,"num_write":129,"num_write_kb":51138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"3.5","version":"891'1350","reported_seq":5613,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.253653+0000","last_change":"2024-09-10T16:31:51.192309+0000","last_active":"2024-09-10T16:31:53.253653+0000","last_peered":"2024-09-10T16:31:53.253653+0000","last_clean":"2024-09-10T16:31:53.253653+0000","last_became_active":"2024-09-10T16:06:47.875463+0000","last_became_peered":"2024-09-10T16:06:47.875463+0000","last_unstale":"2024-09-10T16:31:53.253653+0000","last_undegraded":"2024-09-10T16:31:53.253653+0000","last_fullsized":"2024-09-10T16:31:53.253653+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":1350,"log_dups_size":0,"ondisk_log_size":1350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:57:55.032113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0060053600000000004,"stat_sum":{"num_bytes":51095702,"num_objects":20,"num_object_clones":17,"num_object_copies":40,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":659,"num_read_kb":589086,"num_write":359,"num_write_kb":133147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":1720,"reported_epoch":892,"state":"active+clean","last_fresh":"2024-09-10T16:31:53.212741+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:31:53.212741+0000","last_peered":"2024-09-10T16:31:53.212741+0000","last_clean":"2024-09-10T16:31:53.212741+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:31:53.212741+0000","last_undegraded":"2024-09-10T16:31:53.212741+0000","last_fullsized":"2024-09-10T16:31:53.212741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"890'706","reported_seq":4858,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219332+0000","last_change":"2024-09-10T16:31:51.196652+0000","last_active":"2024-09-10T16:31:55.219332+0000","last_peered":"2024-09-10T16:31:55.219332+0000","last_clean":"2024-09-10T16:31:55.219332+0000","last_became_active":"2024-09-10T16:06:47.871238+0000","last_became_peered":"2024-09-10T16:06:47.871238+0000","last_unstale":"2024-09-10T16:31:55.219332+0000","last_undegraded":"2024-09-10T16:31:55.219332+0000","last_fullsized":"2024-09-10T16:31:55.219332+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":706,"log_dups_size":0,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:44:35.655755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012125693,"stat_sum":{"num_bytes":21767604,"num_objects":11,"num_object_clones":9,"num_object_copies":22,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":191,"num_read_kb":123929,"num_write":181,"num_write_kb":58405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":1741,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.326297+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:31:55.326297+0000","last_peered":"2024-09-10T16:31:55.326297+0000","last_clean":"2024-09-10T16:31:55.326297+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:31:55.326297+0000","last_undegraded":"2024-09-10T16:31:55.326297+0000","last_fullsized":"2024-09-10T16:31:55.326297+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"890'498","reported_seq":4718,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219493+0000","last_change":"2024-09-10T16:31:51.200021+0000","last_active":"2024-09-10T16:31:55.219493+0000","last_peered":"2024-09-10T16:31:55.219493+0000","last_clean":"2024-09-10T16:31:55.219493+0000","last_became_active":"2024-09-10T16:06:47.872826+0000","last_became_peered":"2024-09-10T16:06:47.872826+0000","last_unstale":"2024-09-10T16:31:55.219493+0000","last_undegraded":"2024-09-10T16:31:55.219493+0000","last_fullsized":"2024-09-10T16:31:55.219493+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":498,"log_dups_size":0,"ondisk_log_size":498,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:36:47.360907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.015085751,"stat_sum":{"num_bytes":10776641,"num_objects":7,"num_object_clones":6,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":197,"num_read_kb":111371,"num_write":141,"num_write_kb":51791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":1724,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219499+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:31:55.219499+0000","last_peered":"2024-09-10T16:31:55.219499+0000","last_clean":"2024-09-10T16:31:55.219499+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:31:55.219499+0000","last_undegraded":"2024-09-10T16:31:55.219499+0000","last_fullsized":"2024-09-10T16:31:55.219499+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"889'2248","reported_seq":6351,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.219398+0000","last_change":"2024-09-10T16:31:51.196994+0000","last_active":"2024-09-10T16:31:55.219398+0000","last_peered":"2024-09-10T16:31:55.219398+0000","last_clean":"2024-09-10T16:31:55.219398+0000","last_became_active":"2024-09-10T16:06:47.875286+0000","last_became_peered":"2024-09-10T16:06:47.875286+0000","last_unstale":"2024-09-10T16:31:55.219398+0000","last_undegraded":"2024-09-10T16:31:55.219398+0000","last_fullsized":"2024-09-10T16:31:55.219398+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:46.850807+0000","last_clean_scrub_stamp":"2024-09-10T16:06:46.850807+0000","objects_scrubbed":0,"log_size":2248,"log_dups_size":0,"ondisk_log_size":2248,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:27:50.800590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011460796000000001,"stat_sum":{"num_bytes":63189422,"num_objects":24,"num_object_clones":19,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":796,"num_read_kb":562992,"num_write":681,"num_write_kb":249298,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"187","length":"1"},{"start":"1a3","length":"1"},{"start":"1ba","length":"1"},{"start":"1bd","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":1740,"reported_epoch":894,"state":"active+clean","last_fresh":"2024-09-10T16:31:55.326330+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:31:55.326330+0000","last_peered":"2024-09-10T16:31:55.326330+0000","last_clean":"2024-09-10T16:31:55.326330+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:31:55.326330+0000","last_undegraded":"2024-09-10T16:31:55.326330+0000","last_fullsized":"2024-09-10T16:31:55.326330+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":16,"stat_sum":{"num_bytes":580965507,"num_objects":255,"num_object_clones":205,"num_object_copies":510,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":255,"num_whiteouts":8,"num_read":7594,"num_read_kb":5846976,"num_write":5373,"num_write_kb":2071996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":656166912,"data_stored":945119549,"data_compressed":20313698,"data_compressed_allocated":13623296,"data_compressed_original":41319740,"omap_allocated":0,"internal_metadata":0},"log_size":20006,"ondisk_log_size":20006,"up":32,"acting":32,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575162,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":138048,"kb_used_data":88804,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94233792,"statfs":{"total":96636764160,"available":96495403008,"internally_reserved":0,"allocated":90935296,"data_stored":132749761,"data_compressed":2650365,"data_compressed_allocated":1761280,"data_compressed_original":5529600,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575161,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":79068,"kb_used_data":52036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94292772,"statfs":{"total":96636764160,"available":96555798528,"internally_reserved":0,"allocated":53284864,"data_stored":76986950,"data_compressed":1807575,"data_compressed_allocated":1150976,"data_compressed_original":4226631,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607863,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":75512,"kb_used_data":36004,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94296328,"statfs":{"total":96636764160,"available":96559439872,"internally_reserved":0,"allocated":36868096,"data_stored":57264938,"data_compressed":1281841,"data_compressed_allocated":843776,"data_compressed_original":2716122,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607864,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":163000,"kb_used_data":117916,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94208840,"statfs":{"total":96636764160,"available":96469852160,"internally_reserved":0,"allocated":120745984,"data_stored":166404915,"data_compressed":4033224,"data_compressed_allocated":2637824,"data_compressed_original":8333580,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607865,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":214496,"kb_used_data":165252,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94157344,"statfs":{"total":96636764160,"available":96417120256,"internally_reserved":0,"allocated":169218048,"data_stored":239538695,"data_compressed":5034109,"data_compressed_allocated":3371008,"data_compressed_original":10133295,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640567,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":252248,"kb_used_data":180860,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94119592,"statfs":{"total":96636764160,"available":96378462208,"internally_reserved":0,"allocated":185200640,"data_stored":264209328,"data_compressed":5349908,"data_compressed_allocated":3772416,"data_compressed_original":10819610,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":172191744,"data_stored":247196872,"data_compressed":5167741,"data_compressed_allocated":3452928,"data_compressed_original":10521589,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":51728384,"data_stored":75809015,"data_compressed":1778018,"data_compressed_allocated":1110016,"data_compressed_original":3714631,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122523648,"data_stored":169116434,"data_compressed":4115382,"data_compressed_allocated":2723840,"data_compressed_original":8530188,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183717888,"data_stored":263565648,"data_compressed":5349908,"data_compressed_allocated":3772416,"data_compressed_original":10819610,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":89423872,"data_stored":132095408,"data_compressed":2650365,"data_compressed_allocated":1761280,"data_compressed_original":5529600,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36581376,"data_stored":57336172,"data_compressed":1252284,"data_compressed_allocated":802816,"data_compressed_original":2204122,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:31:57.480 INFO:tasks.ceph.ceph_manager.ceph:removing pool_name unique_pool_0 2024-09-10T16:31:57.481 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2024-09-10T16:31:57.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 10 monclient: get_monmap_and_config 2024-09-10T16:31:57.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.583 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 10 monclient: monmap: 2024-09-10T16:31:57.583 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.583 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:31:57.582941+0000 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:31:57.582941+0000 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:31:57.584 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 1 Processor -- start 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 1 -- start start 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.582+0000 7f8dfbefe640 10 monclient: init 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f8df40a9800 0x7f8df4134ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-c con 0x7f8df40a9800 addr v2:172.21.15.165:3300/0 2024-09-10T16:31:57.587 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-a con 0x7f8df40a45c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8df4137b80 addr v1:172.21.15.164:6789/0 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df40a4f10 con 0x7f8df4137b80 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f8df4135130 con 0x7f8df40a45c0 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f8df409ca90 con 0x7f8df40a9800 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfbefe640 10 monclient(hunting): authenticate will time out at 2296.070400s 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfa49b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f8df4137b80 0x7f8df4137fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:44062/0 (socket says 172.21.15.164:44062) 2024-09-10T16:31:57.588 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8dfa49b640 1 -- 172.21.15.164:0/1843747275 learned_addr learned my addr 172.21.15.164:0/1843747275 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 10 monclient(hunting): get_auth_request con 0x7f8df40a45c0 auth_method 0 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 1 -- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 msgr2=0x7f8df40a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 1 -- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 msgr2=0x7f8df40a49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 1 --2- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/1843747275 >> v2:172.21.15.165:3300/0 conn(0x7f8df40a9800 0x7f8df4134ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9499640 1 --2- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): get_auth_request con 0x7f8df40a9800 auth_method 0 2024-09-10T16:31:57.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient(hunting): handle_auth_done global_id 284182 payload 274 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 -- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 msgr2=0x7f8df40a49d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 -- 172.21.15.164:0/1843747275 >> v1:172.21.15.164:6789/0 conn(0x7f8df4137b80 legacy=0x7f8df4137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 monclient: found mon.noname-c 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.591 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.583+0000 7f8df9c9a640 1 -- 172.21.15.164:0/1843747275 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df409cd70 con 0x7f8df40a9800 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.586+0000 7f8df9c9a640 1 -- 172.21.15.164:0/1843747275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 msgr2=0x7f8df4134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.586+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/1843747275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4134ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8df40a8890 tx=0x7f8de802a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4e98f81eff65c0d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 1 -- 172.21.15.164:0/1843747275 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8037070 con 0x7f8df40a9800 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: dump: 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.597 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 1 -- 172.21.15.164:0/1843747275 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8032440 con 0x7f8df40a9800 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 5 monclient: authenticate success, global_id 284182 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:31:57.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 10 monclient: get_monmap_and_config success 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 4 set_mon_vals no callback set 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 1 -- 172.21.15.164:0/1843747275 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8032810 con 0x7f8df40a9800 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8df8c98640 10 monclient: dump: 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.599 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 10 monclient: shutdown 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 -- 172.21.15.164:0/1843747275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 msgr2=0x7f8df4134ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/1843747275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4134ae0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8df40a8890 tx=0x7f8de802a000 comp rx=0 tx=0).stop 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 -- 172.21.15.164:0/1843747275 shutdown_connections 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/1843747275 >> v2:172.21.15.164:3300/0 conn(0x7f8df40a45c0 0x7f8df40a49d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/1843747275 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4134ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 -- 172.21.15.164:0/1843747275 >> 172.21.15.164:0/1843747275 conn(0x7f8df40aab70 msgr2=0x7f8df40aaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 -- 172.21.15.164:0/1843747275 shutdown_connections 2024-09-10T16:31:57.600 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.587+0000 7f8dfbefe640 1 -- 172.21.15.164:0/1843747275 wait complete. 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.588+0000 7f8dfbefe640 10 monclient: build_initial_monmap 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.588+0000 7f8dfbefe640 10 monclient: monmap: 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.601 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.588+0000 7f8dfbefe640 1 Processor -- start 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.588+0000 7f8dfbefe640 1 -- start start 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.588+0000 7f8dfbefe640 10 monclient: init 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient: _reopen_session rank -1 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-a con 0x7f8df40a45c0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4131970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-b con 0x7f8df40a9800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 0x7f8df41ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): picked mon.noname-c con 0x7f8df4137b80 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f8df40a9240 con 0x7f8df40a45c0 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.602 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f8df41386e0 con 0x7f8df40a9800 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f8df40a3590 con 0x7f8df4137b80 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): _renew_subs 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfbefe640 10 monclient(hunting): authenticate will time out at 2296.076228s 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37622/0 (socket says 172.21.15.164:37622) 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 -- 172.21.15.164:0/3089176068 learned_addr learned my addr 172.21.15.164:0/3089176068 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 10 monclient(hunting): get_auth_request con 0x7f8df40a45c0 auth_method 0 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 0x7f8df41ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.603 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): get_auth_request con 0x7f8df4137b80 auth_method 0 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4131970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): get_auth_request con 0x7f8df40a9800 auth_method 0 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 msgr2=0x7f8df4131430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 msgr2=0x7f8df4131430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:31:57.604 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient(hunting): handle_auth_done global_id 284185 payload 824 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 msgr2=0x7f8df41ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 0x7f8df41ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 msgr2=0x7f8df4131430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient: _finish_hunting 0 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 monclient: found mon.noname-b 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:31:57.605 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8df9499640 1 -- 172.21.15.164:0/3089176068 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df4137670 con 0x7f8df40a9800 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.589+0000 7f8dfa49b640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 0x7f8df41ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8df9499640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 msgr2=0x7f8df4131970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8df9499640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4131970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d5a0 tx=0x7f8df000da60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0012800 con 0x7f8df40a9800 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: dump: 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.606 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: _finish_auth 0 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: _check_auth_tickets 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8df0019070 con 0x7f8df40a9800 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 5 monclient: authenticate success, global_id 284185 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 10 monclient: _renew_subs 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df406e780 con 0x7f8df40a9800 2024-09-10T16:31:57.607 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8df0012be0 con 0x7f8df40a9800 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de67fc640 10 monclient: dump: 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:31:57.608 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8de7fff640 4 set_mon_vals no callback set 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 10 monclient: _renew_subs 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.592+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df4131eb0 con 0x7f8df40a9800 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8dfbefe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8dfbefe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8dfbefe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f8df001e460 con 0x7f8df40a9800 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df40a49d0 con 0x7f8df40a9800 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8de67fc640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 0x7f8dd80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 5 ==== osd_map(895..895 src has 772..895) ==== 4799+0+0 (secure 0 0 0) 0x7f8df0053130 con 0x7f8df40a9800 2024-09-10T16:31:57.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.593+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 0x7f8dd80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:31:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.594+0000 7f8df9c9a640 10 monclient: get_auth_request con 0x7f8dd803e210 auth_method 0 2024-09-10T16:31:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.596+0000 7f8df9c9a640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 0x7f8dd80406d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8de8035040 tx=0x7f8de802e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:31:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.598+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f8df001ce30 con 0x7f8df40a9800 2024-09-10T16:31:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.598+0000 7f8de67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:31:57.610 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.598+0000 7f8de67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:31:57.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.671+0000 7f271fc79640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:31:57.673 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.671+0000 7f271fc79640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.671+0000 7f271fc79640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.671+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2718134f70 con 0x7f27180a46a0 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.672+0000 7f270e7fc640 1 -- 172.21.15.164:0/3413420372 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f27080102e0 con 0x7f27180a46a0 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.672+0000 7f270e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.672+0000 7f270e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:31:57.674 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":246,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 msgr2=0x7f26fc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 0x7f26fc0406d0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f2714035040 tx=0x7f271402e000 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 10 monclient: shutdown 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 msgr2=0x7f271812e4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 0x7f271812e4e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f270800c410 tx=0x7f270800c8d0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 shutdown_connections 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f26fc03e210 0x7f26fc0406d0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f27180a4ab0 0x7f271812ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f27180a46a0 0x7f271812e4e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 --2- 172.21.15.164:0/3413420372 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f271806cb70 0x7f27180a3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:57.676 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 >> 172.21.15.164:0/3413420372 conn(0x7f27180aa900 msgr2=0x7f27180a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:57.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 shutdown_connections 2024-09-10T16:31:57.677 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.674+0000 7f271fc79640 1 -- 172.21.15.164:0/3413420372 wait complete. 2024-09-10T16:31:57.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.777+0000 7f8dfbefe640 10 start_mon_command cmd=[{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-10T16:31:57.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.777+0000 7f8dfbefe640 10 monclient: _send_command 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-10T16:31:57.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.777+0000 7f8dfbefe640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.165:3300/0 2024-09-10T16:31:57.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:57.777+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8df40771d0 con 0x7f8df40a9800 2024-09-10T16:31:58.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.626+0000 7f8de67fc640 1 -- 172.21.15.164:0/3089176068 <== mon.1 v2:172.21.15.165:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}]=0 pool 'unique_pool_0' removed v896) ==== 175+0+0 (secure 0 0 0) 0x7f8df0051b20 con 0x7f8df40a9800 2024-09-10T16:31:58.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.626+0000 7f8de67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool rm", "pool": "unique_pool_0", "pool2": "unique_pool_0", "yes_i_really_really_mean_it": true}] 2024-09-10T16:31:58.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.626+0000 7f8de67fc640 10 monclient: _finish_command 2 = system:0 pool 'unique_pool_0' removed 2024-09-10T16:31:58.628 INFO:teuthology.orchestra.run.smithi164.stderr:pool 'unique_pool_0' removed 2024-09-10T16:31:58.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 msgr2=0x7f8dd80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:58.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 0x7f8dd80406d0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f8de8035040 tx=0x7f8de802e000 comp rx=0 tx=0).stop 2024-09-10T16:31:58.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 10 monclient: shutdown 2024-09-10T16:31:58.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:31:58.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 msgr2=0x7f8df4131970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.628+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4131970 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8df000d5a0 tx=0x7f8df000da60 comp rx=0 tx=0).stop 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 shutdown_connections 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f8dd803e210 0x7f8dd80406d0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f8df4137b80 0x7f8df41ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f8df40a9800 0x7f8df4131970 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 --2- 172.21.15.164:0/3089176068 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f8df40a45c0 0x7f8df4131430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 >> 172.21.15.164:0/3089176068 conn(0x7f8df40aab70 msgr2=0x7f8df41bf860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 shutdown_connections 2024-09-10T16:31:58.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:31:58.629+0000 7f8dfbefe640 1 -- 172.21.15.164:0/3089176068 wait complete. 2024-09-10T16:31:58.645 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-10T16:31:58.657 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-10T16:32:00.686 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:32:00.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.794+0000 7f9ac58ed640 10 monclient: get_monmap_and_config 2024-09-10T16:32:00.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.794+0000 7f9ac58ed640 10 monclient: build_initial_monmap 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.794+0000 7f9ac58ed640 10 monclient: monmap: 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:00.795254+0000 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:00.795254+0000 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:00.796 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:00.797 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:00.797 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.794+0000 7f9ac58ed640 1 Processor -- start 2024-09-10T16:32:00.807 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 -- start start 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient: init 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9ac006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9ac0134f70 0x7f9ac013d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac0134f70 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac009ce50 con 0x7f9ac006cb70 2024-09-10T16:32:00.808 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9ac006db60 con 0x7f9ac0134f70 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac006cf50 con 0x7f9ac013d560 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9ac58ed640 10 monclient(hunting): authenticate will time out at 2299.282561s 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48230/0 (socket says 172.21.15.164:48230) 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 1 -- 172.21.15.164:0/236008782 learned_addr learned my addr 172.21.15.164:0/236008782 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): get_auth_request con 0x7f9ac013d560 auth_method 0 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:00.809 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 1 --2- 172.21.15.164:0/236008782 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0134f70 0x7f9ac013d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): get_auth_request con 0x7f9ac0134f70 auth_method 0 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abe7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:00.810 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.795+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_done global_id 274271 payload 274 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 1 -- 172.21.15.164:0/236008782 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0134f70 msgr2=0x7f9ac013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/236008782 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0134f70 0x7f9ac013d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 1 -- 172.21.15.164:0/236008782 >> v1:172.21.15.165:6789/0 conn(0x7f9ac006cb70 legacy=0x7f9ac00a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 1 monclient: found mon.noname-b 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.796+0000 7f9abf7fe640 1 -- 172.21.15.164:0/236008782 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac0135380 con 0x7f9ac013d560 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abf7fe640 1 -- 172.21.15.164:0/236008782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 msgr2=0x7f9ac013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/236008782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80098a0 tx=0x7f9ab802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74e8f225ff413126 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 1 -- 172.21.15.164:0/236008782 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8037070 con 0x7f9ac013d560 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:00.811 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: dump: 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.812 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 1 -- 172.21.15.164:0/236008782 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab8032440 con 0x7f9ac013d560 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9ac58ed640 5 monclient: authenticate success, global_id 274271 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9ac58ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9ac58ed640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9ac58ed640 4 set_mon_vals no callback set 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 1 -- 172.21.15.164:0/236008782 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8032810 con 0x7f9ac013d560 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.798+0000 7f9abdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9abdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9abdffb640 10 monclient: dump: 2024-09-10T16:32:00.813 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 10 monclient: shutdown 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:00.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 -- 172.21.15.164:0/236008782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 msgr2=0x7f9ac013f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/236008782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ab80098a0 tx=0x7f9ab802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 -- 172.21.15.164:0/236008782 shutdown_connections 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/236008782 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac013f950 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/236008782 >> v2:172.21.15.165:3300/0 conn(0x7f9ac0134f70 0x7f9ac013d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 -- 172.21.15.164:0/236008782 >> 172.21.15.164:0/236008782 conn(0x7f9ac00aa900 msgr2=0x7f9ac00aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 -- 172.21.15.164:0/236008782 shutdown_connections 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.799+0000 7f9ac58ed640 1 -- 172.21.15.164:0/236008782 wait complete. 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.800+0000 7f9ac58ed640 10 monclient: build_initial_monmap 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.800+0000 7f9ac58ed640 10 monclient: monmap: 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:00.815 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.800+0000 7f9ac58ed640 1 Processor -- start 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.800+0000 7f9ac58ed640 1 -- start start 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.800+0000 7f9ac58ed640 10 monclient: init 2024-09-10T16:32:00.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac0134f70 0x7f9ac013c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac0134f70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac01c9f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac013d560 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac00a4db0 con 0x7f9ac006cb70 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9ac009d130 con 0x7f9ac0134f70 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:00.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9ac013fe90 con 0x7f9ac013d560 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9ac58ed640 10 monclient(hunting): authenticate will time out at 2299.288350s 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37634/0 (socket says 172.21.15.164:37634) 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 -- 172.21.15.164:0/4264775291 learned_addr learned my addr 172.21.15.164:0/4264775291 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 10 monclient(hunting): get_auth_request con 0x7f9ac006cb70 auth_method 0 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac01c9f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): get_auth_request con 0x7f9ac013d560 auth_method 0 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:00.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 msgr2=0x7f9ac01cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 msgr2=0x7f9ac01cd930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abeffd640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abe7fc640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac0134f70 0x7f9ac013c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abe7fc640 10 monclient(hunting): get_auth_request con 0x7f9ac0134f70 auth_method 0 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abe7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:00.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient(hunting): handle_auth_done global_id 274274 payload 824 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac0134f70 msgr2=0x7f9ac013c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac0134f70 0x7f9ac013c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 msgr2=0x7f9ac01cd930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 monclient: found mon.noname-c 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.801+0000 7f9abf7fe640 1 -- 172.21.15.164:0/4264775291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac013cf80 con 0x7f9ac013d560 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.803+0000 7f9abf7fe640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 msgr2=0x7f9ac01c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9abf7fe640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac01c9f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ac006da80 tx=0x7f9ab80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:00.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8037040 con 0x7f9ac013d560 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: dump: 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:00.821 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: _finish_auth 0 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ab8042070 con 0x7f9ac013d560 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 5 monclient: authenticate success, global_id 274274 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 10 monclient: _renew_subs 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac0138300 con 0x7f9ac013d560 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:00.822 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ab803ba40 con 0x7f9ac013d560 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9aa7fff640 10 monclient: dump: 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:00.823 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 10 monclient: _renew_subs 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ac0131b10 con 0x7f9ac013d560 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.804+0000 7f9abd7fa640 4 set_mon_vals no callback set 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9ab803bbe0 con 0x7f9ac013d560 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9ac58ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9ac58ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9ac58ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ac00a9810 con 0x7f9ac013d560 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9aa7fff640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 0x7f9aac040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:00.824 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(896..896 src has 772..896) ==== 4374+0+0 (secure 0 0 0) 0x7f9ab8073290 con 0x7f9ac013d560 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9abeffd640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 0x7f9aac040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.805+0000 7f9abeffd640 10 monclient: get_auth_request con 0x7f9aac03e260 auth_method 0 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.808+0000 7f9abeffd640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 0x7f9aac040720 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009590 tx=0x7f9ab40095c0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.810+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9ac00a9810 con 0x7f9ac013d560 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.810+0000 7f9aa7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:00.825 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:00.810+0000 7f9aa7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:01.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.054+0000 7f9ac58ed640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:32:01.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.054+0000 7f9ac58ed640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.054+0000 7f9ac58ed640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.054+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9ac00a46a0 con 0x7f9ac013d560 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.055+0000 7f9aa7fff640 1 -- 172.21.15.164:0/4264775291 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f9ac009ce50 con 0x7f9ac013d560 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.055+0000 7f9aa7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.055+0000 7f9aa7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:01.057 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":246,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":24,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 msgr2=0x7f9aac040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 0x7f9aac040720 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f9ab4009590 tx=0x7f9ab40095c0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 10 monclient: shutdown 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 msgr2=0x7f9ac01c9f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac01c9f00 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9ac006da80 tx=0x7f9ab80046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 shutdown_connections 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9aac03e260 0x7f9aac040720 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9ac013d560 0x7f9ac01c9f00 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9ac0134f70 0x7f9ac013c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 --2- 172.21.15.164:0/4264775291 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9ac006cb70 0x7f9ac01cd930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 >> 172.21.15.164:0/4264775291 conn(0x7f9ac00aa900 msgr2=0x7f9ac012e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 shutdown_connections 2024-09-10T16:32:01.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:01.057+0000 7f9ac58ed640 1 -- 172.21.15.164:0/4264775291 wait complete. 2024-09-10T16:32:04.070 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:32:04.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.177+0000 7f5a07917640 10 monclient: get_monmap_and_config 2024-09-10T16:32:04.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.177+0000 7f5a07917640 10 monclient: build_initial_monmap 2024-09-10T16:32:04.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.177+0000 7f5a07917640 10 monclient: monmap: 2024-09-10T16:32:04.179 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:04.178549+0000 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:04.178549+0000 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:04.180 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:04.181 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:04.181 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:04.181 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:04.181 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 1 Processor -- start 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 1 -- start start 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 10 monclient: init 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 10 monclient: _add_conns ranks=[2,4,1,3,0] 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.178+0000 7f5a07917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-b con 0x7f5a0009f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:04.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5a000a2380 0x7f5a000a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-c con 0x7f5a000a2380 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5a0013a5c0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a0009fd10 con 0x7f5a0013a5c0 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5a000a2cc0 con 0x7f5a000a2380 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0009cd70 con 0x7f5a0009f930 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a07917640 10 monclient(hunting): authenticate will time out at 2302.666154s 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48262/0 (socket says 172.21.15.164:48262) 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 -- 172.21.15.164:0/3676736841 learned_addr learned my addr 172.21.15.164:0/3676736841 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:04.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): get_auth_request con 0x7f5a0009f930 auth_method 0 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a04eb2640 1 --2- 172.21.15.164:0/3676736841 >> v2:172.21.15.165:3300/0 conn(0x7f5a000a2380 0x7f5a000a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a04eb2640 10 monclient(hunting): get_auth_request con 0x7f5a000a2380 auth_method 0 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a04eb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a04eb2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a04eb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient(hunting): handle_auth_done global_id 274277 payload 274 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 -- 172.21.15.164:0/3676736841 >> v2:172.21.15.165:3300/0 conn(0x7f5a000a2380 msgr2=0x7f5a000a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:04.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 --2- 172.21.15.164:0/3676736841 >> v2:172.21.15.165:3300/0 conn(0x7f5a000a2380 0x7f5a000a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 -- 172.21.15.164:0/3676736841 >> v1:172.21.15.165:6789/0 conn(0x7f5a0013a5c0 legacy=0x7f5a0013c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:04.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient: _finish_hunting 0 2024-09-10T16:32:04.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 monclient: found mon.noname-b 2024-09-10T16:32:04.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:04.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.179+0000 7f5a056b3640 1 -- 172.21.15.164:0/3676736841 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a0009a120 con 0x7f5a0009f930 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f5a056b3640 1 -- 172.21.15.164:0/3676736841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 msgr2=0x7f5a000a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f5a056b3640 1 --2- 172.21.15.164:0/3676736841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009870 tx=0x7f59fc02a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e810db59b53e29e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 1 -- 172.21.15.164:0/3676736841 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc037070 con 0x7f5a0009f930 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:04.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:04.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: dump: 2024-09-10T16:32:04.197 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:04.197 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:04.197 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.197 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:04.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: _finish_auth 0 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 1 -- 172.21.15.164:0/3676736841 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59fc032440 con 0x7f5a0009f930 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:04.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 1 -- 172.21.15.164:0/3676736841 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59fc032810 con 0x7f5a0009f930 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f5a07917640 5 monclient: authenticate success, global_id 274277 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f59f3fff640 10 monclient: dump: 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:04.200 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:04.201 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.201 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.201 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:04.201 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.201 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f5a07917640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.182+0000 7f5a07917640 4 set_mon_vals no callback set 2024-09-10T16:32:04.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 10 monclient: shutdown 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 -- 172.21.15.164:0/3676736841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 msgr2=0x7f5a000a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 --2- 172.21.15.164:0/3676736841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009870 tx=0x7f59fc02a0a0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 -- 172.21.15.164:0/3676736841 shutdown_connections 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 --2- 172.21.15.164:0/3676736841 >> v2:172.21.15.165:3300/0 conn(0x7f5a000a2380 0x7f5a000a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 --2- 172.21.15.164:0/3676736841 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0009f930 0x7f5a000a1e40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 -- 172.21.15.164:0/3676736841 >> 172.21.15.164:0/3676736841 conn(0x7f5a000a3110 msgr2=0x7f5a000a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 -- 172.21.15.164:0/3676736841 shutdown_connections 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 1 -- 172.21.15.164:0/3676736841 wait complete. 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 10 monclient: build_initial_monmap 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.183+0000 7f5a07917640 10 monclient: monmap: 2024-09-10T16:32:04.204 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.205 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 Processor -- start 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 -- start start 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient: init 2024-09-10T16:32:04.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:04.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:04.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-a con 0x7f5a0009f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:04.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 0x7f5a0009a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-b con 0x7f5a000a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): picked mon.noname-c con 0x7f5a0013a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0009d930 con 0x7f5a0009f930 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5a0009ee00 con 0x7f5a000a2380 2024-09-10T16:32:04.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5a0009c8b0 con 0x7f5a0013a5c0 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a07917640 10 monclient(hunting): authenticate will time out at 2302.671704s 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:37638/0 (socket says 172.21.15.164:37638) 2024-09-10T16:32:04.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:48272/0 (socket says 172.21.15.164:48272) 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 -- 172.21.15.164:0/746749576 learned_addr learned my addr 172.21.15.164:0/746749576 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 10 monclient(hunting): get_auth_request con 0x7f5a0009f930 auth_method 0 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): get_auth_request con 0x7f5a0013a5c0 auth_method 0 2024-09-10T16:32:04.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a05eb4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 msgr2=0x7f5a0009b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T16:32:04.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 msgr2=0x7f5a0009b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a056b3640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.184+0000 7f5a04eb2640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 0x7f5a0009a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a04eb2640 10 monclient(hunting): get_auth_request con 0x7f5a000a2380 auth_method 0 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a04eb2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:04.212 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a04eb2640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a04eb2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 10 monclient(hunting): handle_auth_done global_id 274280 payload 824 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 msgr2=0x7f5a0009a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 0x7f5a0009a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 msgr2=0x7f5a0009b8f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:32:04.213 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 10 monclient: _finish_hunting 0 2024-09-10T16:32:04.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 monclient: found mon.noname-c 2024-09-10T16:32:04.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:04.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.214 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a05eb4640 1 -- 172.21.15.164:0/746749576 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a00131770 con 0x7f5a0013a5c0 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.185+0000 7f5a04eb2640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 0x7f5a0009a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a05eb4640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 msgr2=0x7f5a0009a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a05eb4640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f59f800c780 tx=0x7f59f800cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8016020 con 0x7f5a0013a5c0 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:04.215 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:04.216 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: dump: 2024-09-10T16:32:04.216 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:04.216 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:04.216 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.216 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.217 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:04.217 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.217 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:04.217 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:04.217 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f59f800ac60 con 0x7f5a0013a5c0 2024-09-10T16:32:04.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 5 monclient: authenticate success, global_id 274280 2024-09-10T16:32:04.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 10 monclient: _renew_subs 2024-09-10T16:32:04.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a00131ab0 con 0x7f5a0013a5c0 2024-09-10T16:32:04.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:04.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59f8012660 con 0x7f5a0013a5c0 2024-09-10T16:32:04.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:04.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:04.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f1ffb640 10 monclient: dump: 2024-09-10T16:32:04.220 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:04.221 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 10 monclient: _renew_subs 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f59f37fe640 4 set_mon_vals no callback set 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.187+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a00136270 con 0x7f5a0013a5c0 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f59f8019020 con 0x7f5a0013a5c0 2024-09-10T16:32:04.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f5a07917640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f5a07917640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f5a07917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a000a0760 con 0x7f5a0013a5c0 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.188+0000 7f59f1ffb640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 0x7f59e4040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.189+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 772..897) ==== 4374+0+0 (secure 0 0 0) 0x7f59f8052b50 con 0x7f5a0013a5c0 2024-09-10T16:32:04.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.189+0000 7f5a056b3640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 0x7f59e4040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:04.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.189+0000 7f5a056b3640 10 monclient: get_auth_request con 0x7f59e403e260 auth_method 0 2024-09-10T16:32:04.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.191+0000 7f5a056b3640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 0x7f59e4040720 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009630 tx=0x7f59fc02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:04.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.193+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f59f8020030 con 0x7f5a0013a5c0 2024-09-10T16:32:04.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.193+0000 7f59f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:04.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.193+0000 7f59f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:04.439 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f5a07917640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:32:04.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f5a07917640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:04.478 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f5a07917640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:04.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5a000718e0 con 0x7f5a0013a5c0 2024-09-10T16:32:04.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f59f1ffb640 1 -- 172.21.15.164:0/746749576 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f5a00131ab0 con 0x7f5a0013a5c0 2024-09-10T16:32:04.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f59f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:04.479 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.437+0000 7f59f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:04.479 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":246,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 msgr2=0x7f59e4040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 0x7f59e4040720 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f59fc009630 tx=0x7f59fc02e000 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 10 monclient: shutdown 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 msgr2=0x7f5a0009a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f59f800c780 tx=0x7f59f800cc40 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 shutdown_connections 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f59e403e260 0x7f59e4040720 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5a0013a5c0 0x7f5a0009a970 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5a000a2380 0x7f5a0009a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 --2- 172.21.15.164:0/746749576 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5a0009f930 0x7f5a0009b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:04.481 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 >> 172.21.15.164:0/746749576 conn(0x7f5a000a3110 msgr2=0x7f5a00138bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:04.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 shutdown_connections 2024-09-10T16:32:04.482 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:04.440+0000 7f5a07917640 1 -- 172.21.15.164:0/746749576 wait complete. 2024-09-10T16:32:07.483 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:32:07.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.592+0000 7f622a5f8640 10 monclient: get_monmap_and_config 2024-09-10T16:32:07.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.592+0000 7f622a5f8640 10 monclient: build_initial_monmap 2024-09-10T16:32:07.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.592+0000 7f622a5f8640 10 monclient: monmap: 2024-09-10T16:32:07.593 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:07.593001+0000 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:07.593001+0000 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:07.594 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:07.595 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:07.595 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 Processor -- start 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 -- start start 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient: init 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-a con 0x7f622406cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f62240a46a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f62240a4ab0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:07.597 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f622409ce50 con 0x7f62240a4ab0 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6224138ac0 con 0x7f62240a46a0 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f622406db60 con 0x7f622406cb70 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f622a5f8640 10 monclient(hunting): authenticate will time out at 2306.080645s 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42552/0 (socket says 172.21.15.164:42552) 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6228b95640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f62240a4ab0 0x7f62241383b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38006/0 (socket says 172.21.15.164:38006) 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 learned_addr learned my addr 172.21.15.164:0/1701364893 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 10 monclient(hunting): get_auth_request con 0x7f622406cb70 auth_method 0 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.598 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 msgr2=0x7f6224134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 msgr2=0x7f6224134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.593+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.595+0000 7f6228b95640 1 -- 172.21.15.164:0/1701364893 >> v1:172.21.15.164:6789/0 conn(0x7f62240a4ab0 legacy=0x7f62241383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:07.599 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f621bfff640 1 -- 172.21.15.164:0/1701364893 >> v1:172.21.15.165:6789/0 conn(0x7f62240a46a0 legacy=0x7f6224137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.1 v1:172.21.15.165:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 189522862 0 0) 0x7f6224138ac0 con 0x7f62240a46a0 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): _negotiate 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): my global_id is 284200 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 --> v1:172.21.15.165:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6210003520 con 0x7f62240a46a0 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.1 v1:172.21.15.165:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3226220380 0 0) 0x7f6210003520 con 0x7f62240a46a0 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 msgr2=0x7f6224134f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 --2- 172.21.15.164:0/1701364893 >> v2:172.21.15.164:3300/0 conn(0x7f622406cb70 0x7f6224134f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 >> v1:172.21.15.164:6789/0 conn(0x7f62240a4ab0 legacy=0x7f62241383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:07.608 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 monclient: found mon.noname-c-legacy 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.165:6789/0 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 --> v1:172.21.15.165:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622409cb70 con 0x7f62240a46a0 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.1 v1:172.21.15.165:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3229242726 0 0) 0x7f620c001ab0 con 0x7f62240a46a0 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f622a5f8640 5 monclient: authenticate success, global_id 284200 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f622a5f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: dump: 2024-09-10T16:32:07.609 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:07.610 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] but i'm connected to v1:172.21.15.165:6789/0, reconnecting 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 >> v1:172.21.15.165:6789/0 conn(0x7f62240a46a0 legacy=0x7f6224137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 0x7f6210003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): picked mon.a con 0x7f6210003ac0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210004800 0x7f6210004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): picked mon.b con 0x7f6210004800 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6210005410 0x7f621000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): picked mon.c con 0x7f6210005410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:07.611 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f621000c320 con 0x7f6210003ac0 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f621000c4d0 con 0x7f6210004800 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f621000c6d0 con 0x7f6210005410 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f62237fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f622a5f8640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 0x7f6210003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f6223fff640 10 monclient(hunting): get_auth_request con 0x7f6210003ac0 auth_method 0 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f6223fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.596+0000 7f6223fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.612 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6210005410 0x7f621000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210004800 0x7f6210004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): get_auth_request con 0x7f6210004800 auth_method 0 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): get_auth_request con 0x7f6210005410 auth_method 0 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.613 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient(hunting): handle_auth_done global_id 284200 payload 293 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6210005410 msgr2=0x7f621000ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:07.614 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f6210005410 0x7f621000ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210004800 msgr2=0x7f6210004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210004800 0x7f6210004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 monclient: found mon.a 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6210003520 con 0x7f6210003ac0 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient: _finish_auth 0 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6228b95640 1 -- 172.21.15.164:0/1701364893 reap_dead start 2024-09-10T16:32:07.615 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.597+0000 7f6223fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f6223fff640 1 -- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 msgr2=0x7f6210003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f6223fff640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 0x7f6210003ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6214009890 tx=0x7f6210005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d180045d11404144 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6214037070 con 0x7f6210003ac0 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: dump: 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6214032440 con 0x7f6210003ac0 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 1 -- 172.21.15.164:0/1701364893 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f621400f040 con 0x7f6210003ac0 2024-09-10T16:32:07.617 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 4 set_mon_vals no callback set 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f62237fe640 10 monclient: dump: 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:07.618 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 10 monclient: shutdown 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 -- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 msgr2=0x7f6210003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 0x7f6210003ee0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6214009890 tx=0x7f6210005830 comp rx=0 tx=0).stop 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 -- 172.21.15.164:0/1701364893 shutdown_connections 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 --2- 172.21.15.164:0/1701364893 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f6210003ac0 0x7f6210003ee0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 -- 172.21.15.164:0/1701364893 >> 172.21.15.164:0/1701364893 conn(0x7f62240aa900 msgr2=0x7f62240aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:07.619 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.600+0000 7f622a5f8640 1 -- 172.21.15.164:0/1701364893 shutdown_connections 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 1 -- 172.21.15.164:0/1701364893 wait complete. 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 10 monclient: build_initial_monmap 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 10 monclient: monmap: 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:07.620 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 1 Processor -- start 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 1 -- start start 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.601+0000 7f622a5f8640 10 monclient: init 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 0x7f62241cd210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-b con 0x7f6210003ac0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:07.621 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-c con 0x7f62240a3830 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): picked mon.noname-a con 0x7f622409e680 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f6224134f70 con 0x7f622409e680 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f622406cf50 con 0x7f6210003ac0 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f62241350f0 con 0x7f62240a3830 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:07.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f622a5f8640 10 monclient(hunting): authenticate will time out at 2306.089401s 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42570/0 (socket says 172.21.15.164:42570) 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46852/0 (socket says 172.21.15.164:46852) 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 -- 172.21.15.164:0/791469318 learned_addr learned my addr 172.21.15.164:0/791469318 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 10 monclient(hunting): get_auth_request con 0x7f622409e680 auth_method 0 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:07.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): get_auth_request con 0x7f62240a3830 auth_method 0 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 msgr2=0x7f622409eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 msgr2=0x7f622409eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:07.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6228b95640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6223fff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 0x7f62241cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient(hunting): handle_auth_done global_id 274283 payload 824 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 msgr2=0x7f62241cd210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 0x7f62241cd210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 msgr2=0x7f622409eb10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 monclient: found mon.noname-c 2024-09-10T16:32:07.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f621bfff640 1 -- 172.21.15.164:0/791469318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f622409ffa0 con 0x7f62240a3830 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6223fff640 10 monclient: get_auth_request con 0x7f6210003ac0 auth_method 0 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.602+0000 7f6223fff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 0x7f62241cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f621bfff640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 msgr2=0x7f62240a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f621bfff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f620c001520 tx=0x7f620c007410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c017020 con 0x7f62240a3830 2024-09-10T16:32:07.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: dump: 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: _finish_auth 0 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: _check_auth_tickets 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f620c00f9c0 con 0x7f62240a3830 2024-09-10T16:32:07.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 5 monclient: authenticate success, global_id 274283 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 10 monclient: _renew_subs 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62240a8f50 con 0x7f62240a3830 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f620c00fd90 con 0x7f62240a3830 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f62217fa640 10 monclient: dump: 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.629 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f6222ffd640 4 set_mon_vals no callback set 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 10 monclient: _renew_subs 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.605+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62240a2300 con 0x7f62240a3830 2024-09-10T16:32:07.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f622a5f8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f622a5f8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f622a5f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f620c01a020 con 0x7f62240a3830 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6224137ca0 con 0x7f62240a3830 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f62217fa640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 0x7f62040406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f620c052e10 con 0x7f62240a3830 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f6223fff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 0x7f62040406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:07.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.606+0000 7f6223fff640 10 monclient: get_auth_request con 0x7f620403e210 auth_method 0 2024-09-10T16:32:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.609+0000 7f6223fff640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 0x7f62040406d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6214009100 tx=0x7f621402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.610+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f620c021020 con 0x7f62240a3830 2024-09-10T16:32:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.610+0000 7f62217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:07.632 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.610+0000 7f62217fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f622a5f8640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f622a5f8640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f622a5f8640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f62240a0b00 con 0x7f62240a3830 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f62217fa640 1 -- 172.21.15.164:0/791469318 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1564 (secure 0 0 0) 0x7f620c054030 con 0x7f62240a3830 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f62217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.852+0000 7f62217fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:07.854 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":246,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":31,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:32:07.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 msgr2=0x7f62040406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 0x7f62040406d0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f6214009100 tx=0x7f621402e000 comp rx=0 tx=0).stop 2024-09-10T16:32:07.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 10 monclient: shutdown 2024-09-10T16:32:07.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:07.856 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 msgr2=0x7f62240a3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f620c001520 tx=0x7f620c007410 comp rx=0 tx=0).stop 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 shutdown_connections 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f620403e210 0x7f62040406d0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f622409e680 0x7f622409eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f62240a3830 0x7f62240a3cb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 --2- 172.21.15.164:0/791469318 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f6210003ac0 0x7f62241cd210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 >> 172.21.15.164:0/791469318 conn(0x7f62240aa900 msgr2=0x7f6224132db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.854+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 shutdown_connections 2024-09-10T16:32:07.857 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:07.855+0000 7f622a5f8640 1 -- 172.21.15.164:0/791469318 wait complete. 2024-09-10T16:32:10.868 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.975+0000 7f48c9375640 10 monclient: get_monmap_and_config 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.975+0000 7f48c9375640 10 monclient: build_initial_monmap 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.975+0000 7f48c9375640 10 monclient: monmap: 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:10.977 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:10.976815+0000 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:10.976815+0000 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:10.978 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 1 Processor -- start 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 1 -- start start 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 10 monclient: init 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.976+0000 7f48c9375640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 0x7f48c406cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-c con 0x7f48c40a3e60 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f48c406d0b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:10.981 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f48c413d5f0 0x7f48c413f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-a con 0x7f48c413d5f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48c40a4600 con 0x7f48c406d0b0 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f48c40a49b0 con 0x7f48c413d5f0 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f48c40a4240 con 0x7f48c40a3e60 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c9375640 10 monclient(hunting): authenticate will time out at 2309.464066s 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f48c413d5f0 0x7f48c413f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c27fc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f48c406d0b0 0x7f48c406d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38008/0 (socket says 172.21.15.164:38008) 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): get_auth_request con 0x7f48c413d5f0 auth_method 0 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c27fc640 1 -- 172.21.15.164:0/1533349188 learned_addr learned my addr 172.21.15.164:0/1533349188 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:10.982 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/1533349188 >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 0x7f48c406cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 10 monclient(hunting): get_auth_request con 0x7f48c40a3e60 auth_method 0 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient(hunting): handle_auth_done global_id 294099 payload 274 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 -- 172.21.15.164:0/1533349188 >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 msgr2=0x7f48c406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:10.983 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 --2- 172.21.15.164:0/1533349188 >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 0x7f48c406cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 -- 172.21.15.164:0/1533349188 >> v1:172.21.15.164:6789/0 conn(0x7f48c406d0b0 legacy=0x7f48c406d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 monclient: found mon.noname-a 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c37fe640 1 -- 172.21.15.164:0/1533349188 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c406dba0 con 0x7f48c413d5f0 2024-09-10T16:32:10.984 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.977+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/1533349188 >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 0x7f48c406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c37fe640 1 -- 172.21.15.164:0/1533349188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c413d5f0 msgr2=0x7f48c413f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c37fe640 1 --2- 172.21.15.164:0/1533349188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c413d5f0 0x7f48c413f9e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f48c406e6b0 tx=0x7f48bc02a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03f6f84286a2e41 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 1 -- 172.21.15.164:0/1533349188 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc007950 con 0x7f48c413d5f0 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: dump: 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.990 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 1 -- 172.21.15.164:0/1533349188 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48bc007af0 con 0x7f48c413d5f0 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c9375640 5 monclient: authenticate success, global_id 294099 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:10.991 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c9375640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c9375640 4 set_mon_vals no callback set 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 1 -- 172.21.15.164:0/1533349188 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48bc031400 con 0x7f48c413d5f0 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.980+0000 7f48c1ffb640 10 monclient: dump: 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:10.992 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 10 monclient: shutdown 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 -- 172.21.15.164:0/1533349188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c413d5f0 msgr2=0x7f48c413f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 --2- 172.21.15.164:0/1533349188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c413d5f0 0x7f48c413f9e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f48c406e6b0 tx=0x7f48bc02a480 comp rx=0 tx=0).stop 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 -- 172.21.15.164:0/1533349188 shutdown_connections 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 --2- 172.21.15.164:0/1533349188 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c413d5f0 0x7f48c413f9e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 --2- 172.21.15.164:0/1533349188 >> v2:172.21.15.165:3300/0 conn(0x7f48c40a3e60 0x7f48c406cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 -- 172.21.15.164:0/1533349188 >> 172.21.15.164:0/1533349188 conn(0x7f48c40a4f50 msgr2=0x7f48c40a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 -- 172.21.15.164:0/1533349188 shutdown_connections 2024-09-10T16:32:10.993 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 1 -- 172.21.15.164:0/1533349188 wait complete. 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 10 monclient: build_initial_monmap 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.981+0000 7f48c9375640 10 monclient: monmap: 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 Processor -- start 2024-09-10T16:32:10.994 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 -- start start 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient: init 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-a con 0x7f48c406d0b0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48c40a3e60 0x7f48c4092390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-b con 0x7f48c40a3e60 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): picked mon.noname-c con 0x7f48c413d5f0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f48c409bd30 con 0x7f48c406d0b0 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:10.995 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f48c406e340 con 0x7f48c40a3e60 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f48c413ff20 con 0x7f48c413d5f0 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c9375640 10 monclient(hunting): authenticate will time out at 2309.469772s 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42614/0 (socket says 172.21.15.164:42614) 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 1 -- 172.21.15.164:0/4084841927 learned_addr learned my addr 172.21.15.164:0/4084841927 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46876/0 (socket says 172.21.15.164:46876) 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 10 monclient(hunting): get_auth_request con 0x7f48c406d0b0 auth_method 0 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:10.996 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 10 monclient(hunting): get_auth_request con 0x7f48c413d5f0 auth_method 0 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.982+0000 7f48c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48c40a3e60 0x7f48c4092390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): get_auth_request con 0x7f48c40a3e60 auth_method 0 2024-09-10T16:32:10.997 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c27fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient(hunting): handle_auth_done global_id 294102 payload 824 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 -- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 msgr2=0x7f48c40928d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 -- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48c40a3e60 msgr2=0x7f48c4092390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48c40a3e60 0x7f48c4092390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 monclient: found mon.noname-a 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:10.998 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c2ffd640 1 -- 172.21.15.164:0/4084841927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c4094eb0 con 0x7f48c406d0b0 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.983+0000 7f48c37fe640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c2ffd640 1 -- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 msgr2=0x7f48c409ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f48b400d7d0 tx=0x7f48b400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b4004030 con 0x7f48c406d0b0 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: dump: 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:10.999 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48b40041d0 con 0x7f48c406d0b0 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 5 monclient: authenticate success, global_id 294102 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 10 monclient: _renew_subs 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:11.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48c4093990 con 0x7f48c406d0b0 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48b400b8d0 con 0x7f48c406d0b0 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48ab7fe640 10 monclient: dump: 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:11.001 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c17fa640 4 set_mon_vals no callback set 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 10 monclient: _renew_subs 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.985+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48c4074c90 con 0x7f48c406d0b0 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48c2ffd640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f48b400d7d0 tx=0x7f48b400dc90 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f48b4052e00 osd_map(897..897 src has 869..897) 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f48b400ba70 con 0x7f48c406d0b0 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48c9375640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48c9375640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48c9375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48c406cb70 con 0x7f48c406d0b0 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.986+0000 7f48ab7fe640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 0x7f48b00406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:11.002 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.987+0000 7f48c27fc640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 0x7f48b00406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.987+0000 7f48c27fc640 10 monclient: get_auth_request con 0x7f48b003e210 auth_method 0 2024-09-10T16:32:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.989+0000 7f48c27fc640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 0x7f48b00406d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f48b80092a0 tx=0x7f48b8009230 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:11.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:10.991+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f48b4052e00 con 0x7f48c406d0b0 2024-09-10T16:32:11.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:11.814+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f48b4051990 con 0x7f48c406d0b0 2024-09-10T16:32:11.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:11.814+0000 7f48ab7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:11.816 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:11.814+0000 7f48ab7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:12.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.059+0000 7f48c9375640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:32:12.061 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48c9375640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48c9375640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f48c40949b0 con 0x7f48c406d0b0 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48ab7fe640 1 -- 172.21.15.164:0/4084841927 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f48b4003dc0 con 0x7f48c406d0b0 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48ab7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.060+0000 7f48ab7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:12.062 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":250,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 msgr2=0x7f48b00406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 0x7f48b00406d0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f48b80092a0 tx=0x7f48b8009230 comp rx=0 tx=0).stop 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 10 monclient: shutdown 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 msgr2=0x7f48c409ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f48b400d7d0 tx=0x7f48b400dc90 comp rx=0 tx=0).stop 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 shutdown_connections 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f48b003e210 0x7f48b00406d0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f48c413d5f0 0x7f48c40928d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f48c40a3e60 0x7f48c4092390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 --2- 172.21.15.164:0/4084841927 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f48c406d0b0 0x7f48c409ab50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 >> 172.21.15.164:0/4084841927 conn(0x7f48c40a4f50 msgr2=0x7f48c40a7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 shutdown_connections 2024-09-10T16:32:12.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.062+0000 7f48c9375640 1 -- 172.21.15.164:0/4084841927 wait complete. 2024-09-10T16:32:12.075 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:32:12.075 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T16:32:12.075 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.173+0000 7fbd2ae40640 10 monclient: get_monmap_and_config 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.173+0000 7fbd2ae40640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.173+0000 7fbd2ae40640 10 monclient: monmap: 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:12.174807+0000 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:12.174807+0000 2024-09-10T16:32:12.175 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:12.176 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.174+0000 7fbd2ae40640 1 Processor -- start 2024-09-10T16:32:12.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.174+0000 7fbd2ae40640 1 -- start start 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.174+0000 7fbd2ae40640 10 monclient: init 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd2409f930 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbd240a2550 0x7fbd2413b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240a2550 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fbd240a2980 0x7fbd2413db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-c con 0x7fbd240a2980 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd2409fd10 con 0x7fbd2409f930 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2409cd20 con 0x7fbd240a2550 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fbd2409a0d0 con 0x7fbd240a2980 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd2ae40640 10 monclient(hunting): authenticate will time out at 2310.662213s 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fbd240a2550 0x7fbd2413b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd28bdc640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fbd2409f930 0x7fbd240a1e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38014/0 (socket says 172.21.15.164:38014) 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd28bdc640 1 -- 172.21.15.164:0/69442440 learned_addr learned my addr 172.21.15.164:0/69442440 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 --2- 172.21.15.164:0/69442440 >> v2:172.21.15.164:3300/0 conn(0x7fbd240a2550 0x7fbd2413b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42630/0 (socket says 172.21.15.164:42630) 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): get_auth_request con 0x7fbd240a2550 auth_method 0 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.180 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd293dd640 1 --2- 172.21.15.164:0/69442440 >> v2:172.21.15.165:3300/0 conn(0x7fbd240a2980 0x7fbd2413db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd293dd640 10 monclient(hunting): get_auth_request con 0x7fbd240a2980 auth_method 0 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd293dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd293dd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd293dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient(hunting): handle_auth_done global_id 294105 payload 274 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 -- 172.21.15.164:0/69442440 >> v2:172.21.15.165:3300/0 conn(0x7fbd240a2980 msgr2=0x7fbd2413db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 --2- 172.21.15.164:0/69442440 >> v2:172.21.15.165:3300/0 conn(0x7fbd240a2980 0x7fbd2413db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 -- 172.21.15.164:0/69442440 >> v1:172.21.15.164:6789/0 conn(0x7fbd2409f930 legacy=0x7fbd240a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 monclient: found mon.noname-a 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.181 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.182 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.175+0000 7fbd23fff640 1 -- 172.21.15.164:0/69442440 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd2409e840 con 0x7fbd240a2550 2024-09-10T16:32:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd23fff640 1 -- 172.21.15.164:0/69442440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2550 msgr2=0x7fbd2413b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd23fff640 1 --2- 172.21.15.164:0/69442440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2550 0x7fbd2413b5d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd2409f620 tx=0x7fbd1402a4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b11d92e554ded23 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 1 -- 172.21.15.164:0/69442440 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd14007950 con 0x7fbd240a2550 2024-09-10T16:32:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: dump: 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd2ae40640 5 monclient: authenticate success, global_id 294105 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd2ae40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 1 -- 172.21.15.164:0/69442440 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd14007af0 con 0x7fbd240a2550 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd2ae40640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd2ae40640 4 set_mon_vals no callback set 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 1 -- 172.21.15.164:0/69442440 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd14031400 con 0x7fbd240a2550 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.178+0000 7fbd237fe640 10 monclient: dump: 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.189 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 10 monclient: shutdown 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 -- 172.21.15.164:0/69442440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2550 msgr2=0x7fbd2413b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/69442440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2550 0x7fbd2413b5d0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbd2409f620 tx=0x7fbd1402a4b0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 -- 172.21.15.164:0/69442440 shutdown_connections 2024-09-10T16:32:12.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/69442440 >> v2:172.21.15.165:3300/0 conn(0x7fbd240a2980 0x7fbd2413db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/69442440 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2550 0x7fbd2413b5d0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 -- 172.21.15.164:0/69442440 >> 172.21.15.164:0/69442440 conn(0x7fbd240a3110 msgr2=0x7fbd240a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 -- 172.21.15.164:0/69442440 shutdown_connections 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 1 -- 172.21.15.164:0/69442440 wait complete. 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.179+0000 7fbd2ae40640 10 monclient: monmap: 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:12.191 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 Processor -- start 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 -- start start 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient: init 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd2409f930 0x7fbd241bb480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-b con 0x7fbd2409f930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd240a2550 0x7fbd241bb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-c con 0x7fbd240a2550 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): picked mon.noname-a con 0x7fbd240a2980 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2409d930 con 0x7fbd240a2980 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fbd2409eb80 con 0x7fbd2409f930 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fbd2409c8b0 con 0x7fbd240a2550 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd2ae40640 10 monclient(hunting): authenticate will time out at 2310.667677s 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd240a2550 0x7fbd241bb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42646/0 (socket says 172.21.15.164:42646) 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 learned_addr learned my addr 172.21.15.164:0/3590023053 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): get_auth_request con 0x7fbd240a2550 auth_method 0 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): get_auth_request con 0x7fbd240a2980 auth_method 0 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd293dd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd23fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.180+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd2409f930 0x7fbd241bb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 10 monclient(hunting): handle_auth_done global_id 294108 payload 824 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd240a2550 msgr2=0x7fbd241bb9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd240a2550 0x7fbd241bb9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd2409f930 msgr2=0x7fbd241bb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd2409f930 0x7fbd241bb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 monclient: found mon.noname-a 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd241b2ca0 con 0x7fbd240a2980 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd28bdc640 10 monclient: get_auth_request con 0x7fbd2409f930 auth_method 0 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd23fff640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fbd240a2550 0x7fbd241bb9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.181+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fbd2409f930 0x7fbd241bb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 msgr2=0x7fbd241b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd293dd640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00d830 tx=0x7fbd0c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c00b8d0 con 0x7fbd240a2980 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: dump: 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.196 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd0c019070 con 0x7fbd240a2980 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd0c00bcd0 con 0x7fbd240a2980 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 5 monclient: authenticate success, global_id 294108 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd217fa640 10 monclient: dump: 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 10 monclient: _renew_subs 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd22ffd640 4 set_mon_vals no callback set 2024-09-10T16:32:12.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd2409a810 con 0x7fbd240a2980 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 10 monclient: _renew_subs 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.183+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd241bbf00 con 0x7fbd240a2980 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd2ae40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 0x7fbd241b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbd240a0760 con 0x7fbd241c8d60 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 0x7fbd241b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fbd0c01e460 con 0x7fbd240a2980 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient: get_auth_request con 0x7fbd241c8d60 auth_method 0 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_done global_id 294111 payload 824 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd217fa640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbd0403e240 0x7fbd04040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd28bdc640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd2409d5a0 con 0x7fbd241c8d60 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.184+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fbd0c052fe0 con 0x7fbd240a2980 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.185+0000 7fbd23fff640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbd0403e240 0x7fbd04040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.185+0000 7fbd23fff640 10 monclient: get_auth_request con 0x7fbd0403e240 auth_method 0 2024-09-10T16:32:12.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd28bdc640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 msgr2=0x7fbd241b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 0x7fbd241b1870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00d4f0 tx=0x7fbd1c00d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c0070f0 con 0x7fbd241c8d60 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 10 monclient: dump: 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.203 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbd1c017070 con 0x7fbd241c8d60 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 msgr2=0x7fbd241b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 0x7fbd241b1870 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00d4f0 tx=0x7fbd1c00d9b0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 msgr2=0x7fbd241b1870 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd217fa640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241c8d60 0x7fbd241b1870 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.187+0000 7fbd23fff640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbd0403e240 0x7fbd04040700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbd2406d470 tx=0x7fbd1402f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd2ae40640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 0x7fbd240717f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd2ae40640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fbd241b21c0 con 0x7fbd241b1db0 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 0x7fbd240717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient: get_auth_request con 0x7fbd241b1db0 auth_method 0 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 10 monclient(hunting): handle_auth_done global_id 294114 payload 824 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.193+0000 7fbd28bdc640 1 -- 172.21.15.164:0/3590023053 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbd241b4860 con 0x7fbd241b1db0 2024-09-10T16:32:12.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd28bdc640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 msgr2=0x7fbd240717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd28bdc640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 0x7fbd240717f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00d4f0 tx=0x7fbd1c004940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd1c025450 con 0x7fbd241b1db0 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 10 monclient: dump: 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.206 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 <== mon.0 v2:172.21.15.164:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7fbd1c012410 con 0x7fbd241b1db0 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 msgr2=0x7fbd240717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 0x7fbd240717f0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00d4f0 tx=0x7fbd1c004940 comp rx=0 tx=0).stop 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 msgr2=0x7fbd240717f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd217fa640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd241b1db0 0x7fbd240717f0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fbd1c00d4f0 tx=0x7fbd1c004940 comp rx=0 tx=0).stop 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "leader", 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 250, 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 14808, 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:32:12.208 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.209 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.210 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:32:12.211 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.212 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.213 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.214 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.215 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:32:12.216 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.217 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:32:12.218 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 2 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:32:12.219 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbd0403e240 msgr2=0x7fbd04040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fbd0403e240 0x7fbd04040700 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fbd2406d470 tx=0x7fbd1402f000 comp rx=0 tx=0).stop 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 10 monclient: shutdown 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 msgr2=0x7fbd241b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd293dd640 1 -- 172.21.15.164:0/3590023053 reap_dead start 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbd0c00d830 tx=0x7fbd0c00dcf0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 shutdown_connections 2024-09-10T16:32:12.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 --2- 172.21.15.164:0/3590023053 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fbd240a2980 0x7fbd241b1330 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 >> 172.21.15.164:0/3590023053 conn(0x7fbd240a3110 msgr2=0x7fbd241bc330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 shutdown_connections 2024-09-10T16:32:12.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.196+0000 7fbd2ae40640 1 -- 172.21.15.164:0/3590023053 wait complete. 2024-09-10T16:32:12.222 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:32:12.222 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T16:32:12.222 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T16:32:12.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.320+0000 7f87d21ad640 10 monclient: get_monmap_and_config 2024-09-10T16:32:12.321 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.320+0000 7f87d21ad640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.320+0000 7f87d21ad640 10 monclient: monmap: 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:12.321176+0000 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:12.321176+0000 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:12.322 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 Processor -- start 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 -- start start 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient: init 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient: _add_conns ranks=[3,2,1,4,0] 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc09d930 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:12.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc09e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f87cc134580 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87cc0a0870 con 0x7f87cc134580 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f87cc09d3e0 con 0x7f87cc09d930 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc09e790 con 0x7f87cc09ab50 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87d21ad640 10 monclient(hunting): authenticate will time out at 2310.808810s 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42668/0 (socket says 172.21.15.164:42668) 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87c3fff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc09e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 1 -- 172.21.15.164:0/1214614277 learned_addr learned my addr 172.21.15.164:0/1214614277 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:12.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 10 monclient(hunting): get_auth_request con 0x7f87cc09d930 auth_method 0 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87c3fff640 10 monclient(hunting): get_auth_request con 0x7f87cc09ab50 auth_method 0 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.321+0000 7f87c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient(hunting): handle_auth_done global_id 274292 payload 274 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 1 -- 172.21.15.164:0/1214614277 >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 msgr2=0x7f87cc09dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 1 --2- 172.21.15.164:0/1214614277 >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 1 -- 172.21.15.164:0/1214614277 >> v1:172.21.15.165:6789/0 conn(0x7f87cc134580 legacy=0x7f87cc134a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 1 monclient: found mon.noname-b 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87c3fff640 1 -- 172.21.15.164:0/1214614277 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc074c70 con 0x7f87cc09ab50 2024-09-10T16:32:12.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.322+0000 7f87cb7fe640 1 --2- 172.21.15.164:0/1214614277 >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.324+0000 7f87c3fff640 1 -- 172.21.15.164:0/1214614277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 msgr2=0x7f87cc09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.324+0000 7f87c3fff640 1 --2- 172.21.15.164:0/1214614277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc09e250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87b40097a0 tx=0x7f87b402a050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e71a182527fce49 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 1 -- 172.21.15.164:0/1214614277 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4037070 con 0x7f87cc09ab50 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: dump: 2024-09-10T16:32:12.335 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 1 -- 172.21.15.164:0/1214614277 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87b4032440 con 0x7f87cc09ab50 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 1 -- 172.21.15.164:0/1214614277 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4032810 con 0x7f87cc09ab50 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 5 monclient: authenticate success, global_id 274292 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 4 set_mon_vals no callback set 2024-09-10T16:32:12.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87caffd640 10 monclient: dump: 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 10 monclient: shutdown 2024-09-10T16:32:12.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 -- 172.21.15.164:0/1214614277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 msgr2=0x7f87cc09e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 --2- 172.21.15.164:0/1214614277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc09e250 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f87b40097a0 tx=0x7f87b402a050 comp rx=0 tx=0).stop 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 -- 172.21.15.164:0/1214614277 shutdown_connections 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 --2- 172.21.15.164:0/1214614277 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc09e250 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 --2- 172.21.15.164:0/1214614277 >> v2:172.21.15.164:3300/0 conn(0x7f87cc09d930 0x7f87cc09dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 -- 172.21.15.164:0/1214614277 >> 172.21.15.164:0/1214614277 conn(0x7f87cc0a3110 msgr2=0x7f87cc0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 -- 172.21.15.164:0/1214614277 shutdown_connections 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.325+0000 7f87d21ad640 1 -- 172.21.15.164:0/1214614277 wait complete. 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.326+0000 7f87d21ad640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.326+0000 7f87d21ad640 10 monclient: monmap: 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:12.339 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.326+0000 7f87d21ad640 1 Processor -- start 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.326+0000 7f87d21ad640 1 -- start start 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.326+0000 7f87d21ad640 10 monclient: init 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc13dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-c con 0x7f87cc09ab50 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87cc09d930 0x7f87cc13e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-b con 0x7f87cc09d930 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87cc134580 0x7f87cc1caea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): picked mon.noname-a con 0x7f87cc134580 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc09ef30 con 0x7f87cc134580 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f87cc0a02b0 con 0x7f87cc09d930 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc0a2940 con 0x7f87cc09ab50 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87d21ad640 10 monclient(hunting): authenticate will time out at 2310.814153s 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc13dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc13dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46892/0 (socket says 172.21.15.164:46892) 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 -- 172.21.15.164:0/3348397553 learned_addr learned my addr 172.21.15.164:0/3348397553 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): get_auth_request con 0x7f87cc09ab50 auth_method 0 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87cc134580 0x7f87cc1caea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): get_auth_request con 0x7f87cc134580 auth_method 0 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient(hunting): handle_auth_done global_id 274295 payload 824 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87c3fff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87cc09d930 0x7f87cc13e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87cc09d930 msgr2=0x7f87cc13e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f87cc09d930 0x7f87cc13e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87cc134580 msgr2=0x7f87cc1caea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87cc134580 0x7f87cc1caea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 monclient: found mon.noname-c 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cb7fe640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87cc13e660 con 0x7f87cc09ab50 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.327+0000 7f87cbfff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f87cc134580 0x7f87cc1caea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87cb7fe640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 msgr2=0x7f87cc13dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87cb7fe640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc13dbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00e9d0 tx=0x7f87bc00ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc017070 con 0x7f87cc09ab50 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: dump: 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.344 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87bc00cc70 con 0x7f87cc09ab50 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 5 monclient: authenticate success, global_id 274295 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 10 monclient: _renew_subs 2024-09-10T16:32:12.345 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87cc1352c0 con 0x7f87cc09ab50 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87bc013670 con 0x7f87cc09ab50 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87c8ff9640 10 monclient: dump: 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.346 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87ca7fc640 4 set_mon_vals no callback set 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 10 monclient: _renew_subs 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.329+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f87cc073460 con 0x7f87cc09ab50 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87d21ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87d21ad640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 0x7f87cc02d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc0a1e40 con 0x7f87cc02d440 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 0x7f87cc02d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.347 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient: get_auth_request con 0x7f87cc02d440 auth_method 0 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f87bc01a070 con 0x7f87cc09ab50 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.330+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87c3fff640 10 monclient(hunting): handle_auth_done global_id 274298 payload 824 2024-09-10T16:32:12.348 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87c3fff640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87cc06dac0 con 0x7f87cc02d440 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87c8ff9640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87ac03e210 0x7f87ac0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f87bc052b70 con 0x7f87cc09ab50 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87cbfff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87ac03e210 0x7f87ac0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.331+0000 7f87cbfff640 10 monclient: get_auth_request con 0x7f87ac03e210 auth_method 0 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c3fff640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 msgr2=0x7f87cc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c3fff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 0x7f87cc02d820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f87b4034000 tx=0x7f87b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4042070 con 0x7f87cc02d440 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 10 monclient: dump: 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.349 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f87b403d670 con 0x7f87cc02d440 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 msgr2=0x7f87cc02d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 0x7f87cc02d820 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f87b4034000 tx=0x7f87b40046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 msgr2=0x7f87cc02d820 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.350 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87c8ff9640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02d440 0x7f87cc02d820 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.351 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.333+0000 7f87cbfff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87ac03e210 0x7f87ac0406d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0741d0 tx=0x7f87c4003d10 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.352 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87d21ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87d21ad640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 0x7f87cc02e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87d21ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f87cc0916a0 con 0x7f87cc02dd60 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 0x7f87cc02e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient: get_auth_request con 0x7f87cc02dd60 auth_method 0 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.339+0000 7f87c3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.340+0000 7f87c3fff640 10 monclient(hunting): handle_auth_done global_id 274301 payload 824 2024-09-10T16:32:12.353 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.340+0000 7f87c3fff640 1 -- 172.21.15.164:0/3348397553 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87cc077260 con 0x7f87cc02dd60 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c3fff640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 msgr2=0x7f87cc02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c3fff640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 0x7f87cc02e170 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f87b402a090 tx=0x7f87b4008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87b4037040 con 0x7f87cc02dd60 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 10 monclient: dump: 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.354 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 <== mon.2 v2:172.21.15.164:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4888 (secure 0 0 0) 0x7f87b40477b0 con 0x7f87cc02dd60 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 msgr2=0x7f87cc02e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 0x7f87cc02e170 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f87b402a090 tx=0x7f87b4008030 comp rx=0 tx=0).stop 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 msgr2=0x7f87cc02e170 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.342+0000 7f87c8ff9640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc02dd60 0x7f87cc02e170 secure :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f87b402a090 tx=0x7f87b4008030 comp rx=0 tx=0).stop 2024-09-10T16:32:12.355 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 250, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 109585, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.356 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:32:12.357 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.358 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:32:12.359 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.360 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.361 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.362 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.363 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.364 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.365 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.366 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 5 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 3 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.367 INFO:teuthology.orchestra.run.smithi164.stdout: "mgr": [ 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:32:12.368 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:12.369 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87ac03e210 msgr2=0x7f87ac0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f87ac03e210 0x7f87ac0406d0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f87cc0741d0 tx=0x7f87c4003d10 comp rx=0 tx=0).stop 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 10 monclient: shutdown 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 msgr2=0x7f87cc13dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 --2- 172.21.15.164:0/3348397553 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f87cc09ab50 0x7f87cc13dbe0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f87bc00e9d0 tx=0x7f87bc00ee90 comp rx=0 tx=0).stop 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87cbfff640 1 -- 172.21.15.164:0/3348397553 reap_dead start 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87cbfff640 1 -- 172.21.15.164:0/3348397553 reap_dead start 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 shutdown_connections 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 >> 172.21.15.164:0/3348397553 conn(0x7f87cc0a3110 msgr2=0x7f87cc099770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 shutdown_connections 2024-09-10T16:32:12.370 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.343+0000 7f87d21ad640 1 -- 172.21.15.164:0/3348397553 wait complete. 2024-09-10T16:32:12.371 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:32:12.371 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T16:32:12.371 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T16:32:12.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 10 monclient: get_monmap_and_config 2024-09-10T16:32:12.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.484 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 10 monclient: monmap: 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:12.484199+0000 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:12.484199+0000 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:12.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 1 Processor -- start 2024-09-10T16:32:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 1 -- start start 2024-09-10T16:32:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.483+0000 7ff66d1c9640 10 monclient: init 2024-09-10T16:32:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.487 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient: _add_conns ranks=[1,3,0,4,2] 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff66809c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff66809d190 0x7ff66809f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff66809d190 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff66813a5c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff66809fbc0 con 0x7ff66813a5c0 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6680a2a20 con 0x7ff66809c310 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff66809c6f0 con 0x7ff66809d190 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66d1c9640 10 monclient(hunting): authenticate will time out at 2310.971433s 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff66809d190 0x7ff66809f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66759e640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff66813a5c0 0x7ff66813c9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38030/0 (socket says 172.21.15.164:38030) 2024-09-10T16:32:12.488 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66759e640 1 -- 172.21.15.164:0/2721303874 learned_addr learned my addr 172.21.15.164:0/2721303874 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): get_auth_request con 0x7ff66809d190 auth_method 0 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient(hunting): handle_auth_done global_id 294123 payload 274 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 1 -- 172.21.15.164:0/2721303874 >> v1:172.21.15.165:6789/0 conn(0x7ff66809c310 legacy=0x7ff66809ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 1 -- 172.21.15.164:0/2721303874 >> v1:172.21.15.164:6789/0 conn(0x7ff66813a5c0 legacy=0x7ff66813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 1 monclient: found mon.noname-a 2024-09-10T16:32:12.489 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.490 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.484+0000 7ff66659c640 1 -- 172.21.15.164:0/2721303874 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66809a4d0 con 0x7ff66809d190 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66659c640 1 -- 172.21.15.164:0/2721303874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff66809d190 msgr2=0x7ff66809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66659c640 1 --2- 172.21.15.164:0/2721303874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff66809d190 0x7ff66809f150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6500097a0 tx=0x7ff650029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc17c42e88454f11 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 1 -- 172.21.15.164:0/2721303874 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650037070 con 0x7ff66809d190 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: dump: 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.497 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 1 -- 172.21.15.164:0/2721303874 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff650032640 con 0x7ff66809d190 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66d1c9640 5 monclient: authenticate success, global_id 294123 2024-09-10T16:32:12.498 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66d1c9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66d1c9640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 1 -- 172.21.15.164:0/2721303874 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650032a10 con 0x7ff66809d190 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff66d1c9640 4 set_mon_vals no callback set 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.487+0000 7ff665d9b640 10 monclient: dump: 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.499 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 10 monclient: shutdown 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 -- 172.21.15.164:0/2721303874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff66809d190 msgr2=0x7ff66809f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/2721303874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff66809d190 0x7ff66809f150 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff6500097a0 tx=0x7ff650029fd0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 -- 172.21.15.164:0/2721303874 shutdown_connections 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/2721303874 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff66809d190 0x7ff66809f150 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 -- 172.21.15.164:0/2721303874 >> 172.21.15.164:0/2721303874 conn(0x7ff6680a3110 msgr2=0x7ff6680a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 -- 172.21.15.164:0/2721303874 shutdown_connections 2024-09-10T16:32:12.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.488+0000 7ff66d1c9640 1 -- 172.21.15.164:0/2721303874 wait complete. 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.489+0000 7ff66d1c9640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.489+0000 7ff66d1c9640 10 monclient: monmap: 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.489+0000 7ff66d1c9640 1 Processor -- start 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.489+0000 7ff66d1c9640 1 -- start start 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.489+0000 7ff66d1c9640 10 monclient: init 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809c310 0x7ff66809d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-b con 0x7ff66809c310 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-c con 0x7ff66813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 0x7ff66812daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): picked mon.noname-a con 0x7ff668137090 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680a2230 con 0x7ff668137090 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff6680a20b0 con 0x7ff66809c310 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff668074e00 con 0x7ff66813a5c0 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66d1c9640 10 monclient(hunting): authenticate will time out at 2310.977172s 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 0x7ff66812daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:46934/0 (socket says 172.21.15.164:46934) 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 0x7ff66812daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42690/0 (socket says 172.21.15.164:42690) 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 -- 172.21.15.164:0/1034960024 learned_addr learned my addr 172.21.15.164:0/1034960024 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): get_auth_request con 0x7ff66813a5c0 auth_method 0 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): get_auth_request con 0x7ff668137090 auth_method 0 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809c310 0x7ff66809d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): get_auth_request con 0x7ff66809c310 auth_method 0 2024-09-10T16:32:12.504 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient(hunting): handle_auth_done global_id 274304 payload 824 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809c310 msgr2=0x7ff66809d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809c310 0x7ff66809d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 msgr2=0x7ff66812daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 0x7ff66812daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 monclient: found mon.noname-c 2024-09-10T16:32:12.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66659c640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff66812efd0 con 0x7ff66813a5c0 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.490+0000 7ff66759e640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff668137090 0x7ff66812daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.492+0000 7ff66659c640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 msgr2=0x7ff668136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.492+0000 7ff66659c640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff6500098c0 tx=0x7ff6500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650041070 con 0x7ff66813a5c0 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: dump: 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.506 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff65003c880 con 0x7ff66813a5c0 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 5 monclient: authenticate success, global_id 274304 2024-09-10T16:32:12.507 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 monclient: _renew_subs 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff650037040 con 0x7ff66813a5c0 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6681cb0c0 con 0x7ff66813a5c0 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff65f7fe640 10 monclient: dump: 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.508 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66559a640 4 set_mon_vals no callback set 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 monclient: _renew_subs 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6680a5160 con 0x7ff66813a5c0 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 0x7ff66812e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.493+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff66809e210 con 0x7ff66809d190 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff650044070 con 0x7ff66813a5c0 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 0x7ff66812e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff65f7fe640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff64803e240 0x7ff648040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient: get_auth_request con 0x7ff66809d190 auth_method 0 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7ff6500735c0 con 0x7ff66813a5c0 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff66759e640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff64803e240 0x7ff648040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 10 monclient(hunting): handle_auth_done global_id 284215 payload 824 2024-09-10T16:32:12.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff666d9d640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff66809f7b0 con 0x7ff66809d190 2024-09-10T16:32:12.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.494+0000 7ff66759e640 10 monclient: get_auth_request con 0x7ff64803e240 auth_method 0 2024-09-10T16:32:12.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.496+0000 7ff666d9d640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 msgr2=0x7ff66812e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.496+0000 7ff666d9d640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 0x7ff66812e030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660005800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff66759e640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff64803e240 0x7ff648040700 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff658007c90 tx=0x7ff658007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660018030 con 0x7ff66809d190 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 10 monclient: dump: 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.514 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff66000dd30 con 0x7ff66809d190 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 msgr2=0x7ff66812e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 0x7ff66812e030 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660005800 comp rx=0 tx=0).stop 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 msgr2=0x7ff66812e030 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.497+0000 7ff65f7fe640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66809d190 0x7ff66812e030 secure :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660005800 comp rx=0 tx=0).stop 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff66d1c9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 0x7ff6681393c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff66d1c9640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff66812e980 con 0x7ff66812e570 2024-09-10T16:32:12.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 0x7ff6681393c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient: get_auth_request con 0x7ff66812e570 auth_method 0 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 10 monclient(hunting): handle_auth_done global_id 284218 payload 824 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.503+0000 7ff666d9d640 1 -- 172.21.15.164:0/1034960024 --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff66809d620 con 0x7ff66812e570 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff666d9d640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 msgr2=0x7ff6681393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff666d9d640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 0x7ff6681393c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660003ae0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660014b90 con 0x7ff66812e570 2024-09-10T16:32:12.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 10 monclient: dump: 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 <== mon.1 v2:172.21.15.165:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7ff660014b90 con 0x7ff66812e570 2024-09-10T16:32:12.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 msgr2=0x7ff6681393c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 0x7ff6681393c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660003ae0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 msgr2=0x7ff6681393c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff65f7fe640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff66812e570 0x7ff6681393c0 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff66000fb10 tx=0x7ff660003ae0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stdout: "state": "peon", 2024-09-10T16:32:12.518 INFO:teuthology.orchestra.run.smithi164.stdout: "election_epoch": 250, 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "uptime": 66337, 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum": [ 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: 0, 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: 1, 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: 2 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_age": 2, 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "required_con": "2450521708007866372", 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "required_mon": [ 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.519 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_con": "4540719139924082687", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "quorum_mon": [ 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.520 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "outside_quorum": [], 2024-09-10T16:32:12.521 INFO:teuthology.orchestra.run.smithi164.stdout: "extra_probe_peers": [], 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "sync_provider": [], 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "monmap": { 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "epoch": 1, 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "fsid": "b134de91-f819-41d3-9bbd-da58ec0ed6be", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "modified": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "created": "2024-09-10T16:05:30.502390Z", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release": 19, 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "min_mon_release_name": "squid", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "election_strategy": 1, 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "disallowed_leaders: ": "", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false, 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "tiebreaker_mon": "", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "removed_ranks: ": "", 2024-09-10T16:32:12.522 INFO:teuthology.orchestra.run.smithi164.stdout: "features": { 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "persistent": [ 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "kraken", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "luminous", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "mimic", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "osdmap-prune", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "nautilus", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "octopus", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "pacific", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "elector-pinging", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "quincy", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "reef", 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: "squid" 2024-09-10T16:32:12.523 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "optional": [] 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "mons": [ 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 0, 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "a", 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3300", 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.524 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789", 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6789/0", 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.525 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 1, 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "b", 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:3300", 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789", 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.526 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.165:6789/0", 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "rank": 2, 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "name": "c", 2024-09-10T16:32:12.527 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addrs": { 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "addrvec": [ 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v2", 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:3301", 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "type": "v1", 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790", 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: "nonce": 0 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.528 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "public_addr": "172.21.15.164:6790/0", 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "priority": 0, 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "weight": 0, 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "crush_location": "{}" 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "feature_map": { 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "mon": [ 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.529 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "osd": [ 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 1 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: ], 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "client": [ 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: { 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T16:32:12.530 INFO:teuthology.orchestra.run.smithi164.stdout: "release": "squid", 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: "num": 4 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: } 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: ] 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: }, 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout: "stretch_mode": false 2024-09-10T16:32:12.531 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:12.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff64803e240 msgr2=0x7ff648040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff64803e240 0x7ff648040700 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7ff658007c90 tx=0x7ff658007be0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 10 monclient: shutdown 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 msgr2=0x7ff668136b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff6500098c0 tx=0x7ff6500046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66759e640 1 -- 172.21.15.164:0/1034960024 reap_dead start 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 shutdown_connections 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 --2- 172.21.15.164:0/1034960024 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff66813a5c0 0x7ff668136b50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 >> 172.21.15.164:0/1034960024 conn(0x7ff6680a3110 msgr2=0x7ff66809af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 shutdown_connections 2024-09-10T16:32:12.533 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.506+0000 7ff66d1c9640 1 -- 172.21.15.164:0/1034960024 wait complete. 2024-09-10T16:32:12.533 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T16:32:12.534 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T16:32:12.534 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T16:32:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 10 monclient: get_monmap_and_config 2024-09-10T16:32:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 10 monclient: monmap: 2024-09-10T16:32:12.631 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:12.630915+0000 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:12.630915+0000 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:12.632 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:12.633 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:12.633 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 1 Processor -- start 2024-09-10T16:32:12.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 1 -- start start 2024-09-10T16:32:12.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.630+0000 7f8292a9b640 10 monclient: init 2024-09-10T16:32:12.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient: _add_conns ranks=[3,0,1,4,2] 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f828c06cb70 0x7f828c134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f828c06cb70 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f828c0a46a0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f828c0a4ab0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828c09ce50 con 0x7f828c0a46a0 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f828c138ac0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f828c06db60 con 0x7f828c06cb70 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8292a9b640 10 monclient(hunting): authenticate will time out at 2311.118463s 2024-09-10T16:32:12.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f828c06cb70 0x7f828c134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f828bfff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f828c0a46a0 0x7f828c137ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38034/0 (socket says 172.21.15.164:38034) 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f828bfff640 1 -- 172.21.15.164:0/4226153415 learned_addr learned my addr 172.21.15.164:0/4226153415 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): get_auth_request con 0x7f828c06cb70 auth_method 0 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient(hunting): handle_auth_done global_id 294129 payload 274 2024-09-10T16:32:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 1 -- 172.21.15.164:0/4226153415 >> v1:172.21.15.165:6789/0 conn(0x7f828c0a4ab0 legacy=0x7f828c1383b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 1 -- 172.21.15.164:0/4226153415 >> v1:172.21.15.164:6789/0 conn(0x7f828c0a46a0 legacy=0x7f828c137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 1 monclient: found mon.noname-a 2024-09-10T16:32:12.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:12.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.631+0000 7f8290837640 1 -- 172.21.15.164:0/4226153415 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c09cb70 con 0x7f828c06cb70 2024-09-10T16:32:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f8290837640 1 -- 172.21.15.164:0/4226153415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c06cb70 msgr2=0x7f828c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f8290837640 1 --2- 172.21.15.164:0/4226153415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c06cb70 0x7f828c134f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82840097d0 tx=0x7f828402a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20f373d01ad9697 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 1 -- 172.21.15.164:0/4226153415 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284037070 con 0x7f828c06cb70 2024-09-10T16:32:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: dump: 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f8292a9b640 5 monclient: authenticate success, global_id 294129 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 1 -- 172.21.15.164:0/4226153415 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8284032440 con 0x7f828c06cb70 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f8292a9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.634+0000 7f828b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 4 set_mon_vals no callback set 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f828b7fe640 1 -- 172.21.15.164:0/4226153415 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8284032810 con 0x7f828c06cb70 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f828b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f828b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f828b7fe640 10 monclient: dump: 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.650 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 10 monclient: shutdown 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 -- 172.21.15.164:0/4226153415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c06cb70 msgr2=0x7f828c134f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 --2- 172.21.15.164:0/4226153415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c06cb70 0x7f828c134f70 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f82840097d0 tx=0x7f828402a000 comp rx=0 tx=0).stop 2024-09-10T16:32:12.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 -- 172.21.15.164:0/4226153415 shutdown_connections 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 --2- 172.21.15.164:0/4226153415 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c06cb70 0x7f828c134f70 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 -- 172.21.15.164:0/4226153415 >> 172.21.15.164:0/4226153415 conn(0x7f828c0aa900 msgr2=0x7f828c0aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 -- 172.21.15.164:0/4226153415 shutdown_connections 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.635+0000 7f8292a9b640 1 -- 172.21.15.164:0/4226153415 wait complete. 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.636+0000 7f8292a9b640 10 monclient: build_initial_monmap 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.636+0000 7f8292a9b640 10 monclient: monmap: 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:12.652 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.636+0000 7f8292a9b640 1 Processor -- start 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.636+0000 7f8292a9b640 1 -- start start 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.636+0000 7f8292a9b640 10 monclient: init 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f828c06cb70 0x7f828c0a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-b con 0x7f828c06cb70 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-a con 0x7f828c0a46a0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 0x7f828c0a3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): picked mon.noname-c con 0x7f828c0a4ab0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f828c141050 con 0x7f828c0a46a0 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f828c06cf50 con 0x7f828c06cb70 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f828c1411d0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8292a9b640 10 monclient(hunting): authenticate will time out at 2311.124208s 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42702/0 (socket says 172.21.15.164:42702) 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 0x7f828c0a3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 1 -- 172.21.15.164:0/1020175918 learned_addr learned my addr 172.21.15.164:0/1020175918 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): get_auth_request con 0x7f828c0a46a0 auth_method 0 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): get_auth_request con 0x7f828c0a4ab0 auth_method 0 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8290837640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f828c06cb70 0x7f828c0a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8290837640 10 monclient(hunting): get_auth_request con 0x7f828c06cb70 auth_method 0 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8290837640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8290837640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8290837640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient(hunting): handle_auth_done global_id 274307 payload 824 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 -- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f828c06cb70 msgr2=0x7f828c0a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f828c06cb70 0x7f828c0a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 -- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 msgr2=0x7f828c0a2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient: _finish_hunting 0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 monclient: found mon.noname-c 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f8291038640 1 -- 172.21.15.164:0/1020175918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f828c06e720 con 0x7f828c0a4ab0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.637+0000 7f828bfff640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f8291038640 1 -- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 msgr2=0x7f828c0a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f8291038640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 0x7f828c0a3030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f827400c410 tx=0x7f827400c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274017070 con 0x7f828c0a4ab0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: dump: 2024-09-10T16:32:12.657 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: _finish_auth 0 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: _check_auth_tickets 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:12.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82740037f0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8274003be0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f8292a9b640 5 monclient: authenticate success, global_id 274307 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f82897fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f82897fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f82897fa640 10 monclient: dump: 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:12.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 10 monclient: _renew_subs 2024-09-10T16:32:12.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.639+0000 7f828affd640 4 set_mon_vals no callback set 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f828c13a520 con 0x7f828c0a4ab0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 10 monclient: _renew_subs 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.640+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f828c1392e0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8292a9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8292a9b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8292a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f828c134f70 con 0x7f828c0a4ab0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f827401a360 con 0x7f828c0a4ab0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f82897fa640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 0x7f826c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f8274051d80 con 0x7f828c0a4ab0 2024-09-10T16:32:12.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8290837640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 0x7f826c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:12.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.641+0000 7f8290837640 10 monclient: get_auth_request con 0x7f826c03e210 auth_method 0 2024-09-10T16:32:12.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.644+0000 7f8290837640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 0x7f826c0406d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:12.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.645+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f828c06e720 con 0x7f828c0a4ab0 2024-09-10T16:32:12.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.645+0000 7f82897fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:12.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.645+0000 7f82897fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.888+0000 7f8292a9b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T16:32:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.888+0000 7f8292a9b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T16:32:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.888+0000 7f8292a9b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:12.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.888+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f828c13abc0 con 0x7f828c0a4ab0 2024-09-10T16:32:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f82897fa640 1 -- 172.21.15.164:0/1020175918 <== mon.2 v2:172.21.15.164:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8274003e00 con 0x7f828c0a4ab0 2024-09-10T16:32:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f82897fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T16:32:12.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f82897fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=11,kv=11,logm=50} crc {auth=4004376530,config=1228700967,health=1304483648,kv=1680660932,logm=3142874466}) 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=26,config=2,health=12,kv=11,logm=49} crc {auth=2292868711,config=2586702516,health=2115129483,kv=1176791497,logm=4260650343}) 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=27,config=1,health=11,kv=11,logm=50} crc {auth=4004376530,config=1228700967,health=1304483648,kv=1680660932,logm=3142874466}) 2024-09-10T16:32:12.891 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.889+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=12,kv=12,logm=49} crc {auth=1040992807,config=2586702516,health=2115129483,kv=805391194,logm=646135894}) 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961192409}) 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=316989440}) 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2961192409}) 2024-09-10T16:32:12.892 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3991481278}) 2024-09-10T16:32:12.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 msgr2=0x7f826c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 0x7f826c0406d0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f82840097a0 tx=0x7f828402e000 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 10 monclient: shutdown 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 msgr2=0x7f828c0a3030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 0x7f828c0a3030 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f827400c410 tx=0x7f827400c8d0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 shutdown_connections 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f826c03e210 0x7f826c0406d0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f828c0a4ab0 0x7f828c0a3030 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f828c0a46a0 0x7f828c0a2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 --2- 172.21.15.164:0/1020175918 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f828c06cb70 0x7f828c0a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.891+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 >> 172.21.15.164:0/1020175918 conn(0x7f828c0aa900 msgr2=0x7f828c137a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.892+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 shutdown_connections 2024-09-10T16:32:12.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:12.892+0000 7f8292a9b640 1 -- 172.21.15.164:0/1020175918 wait complete. 2024-09-10T16:32:12.904 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T16:32:13.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440099262}) 2024-09-10T16:32:13.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2364696758}) 2024-09-10T16:32:13.034 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=440099262}) 2024-09-10T16:32:13.035 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.033+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1862219189}) 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934485429}) 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3861530187}) 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2934485429}) 2024-09-10T16:32:13.121 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.119+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2365901528}) 2024-09-10T16:32:13.125 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3372633183}) 2024-09-10T16:32:13.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1586567531}) 2024-09-10T16:32:13.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3372633183}) 2024-09-10T16:32:13.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.124+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3756399830}) 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753502021}) 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1314933457}) 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=753502021}) 2024-09-10T16:32:13.606 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.604+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=716071659}) 2024-09-10T16:32:13.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638022574}) 2024-09-10T16:32:13.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2429207524}) 2024-09-10T16:32:13.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.607 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1638022574}) 2024-09-10T16:32:13.608 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.605+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=635383523}) 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891767530}) 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=26652143}) 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2891767530}) 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.606+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4051992789}) 2024-09-10T16:32:13.609 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225552226}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021362940}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=225552226}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.607+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1008337251}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441614472}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173228680}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441614472}) 2024-09-10T16:32:13.610 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2907469540}) 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129685827}) 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2395404537}) 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=129685827}) 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.608+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=771825515}) 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.612 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291327970}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995690950}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1291327970}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.609+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3415411467}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447871732}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1708269751}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1447871732}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.610+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2930873268}) 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.613 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006146707}) 2024-09-10T16:32:13.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=763524869}) 2024-09-10T16:32:13.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3006146707}) 2024-09-10T16:32:13.614 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4280357930}) 2024-09-10T16:32:13.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213426443}) 2024-09-10T16:32:13.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1733443050}) 2024-09-10T16:32:13.615 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1213426443}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.611+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435745394}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839753401}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=587436887}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=839753401}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.612+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4112218083}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229208510}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1138957574}) 2024-09-10T16:32:13.616 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4229208510}) 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.613+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=748248807}) 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806451454}) 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4065655775}) 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1806451454}) 2024-09-10T16:32:13.617 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.614+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3435642501}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477230894}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2594963372}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1477230894}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2341219021}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2684915360}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=497883967}) 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.620 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2684915360}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.615+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1301964939}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362464873}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=469348348}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=362464873}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.616+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534614805}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222875071}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2444617046}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2222875071}) 2024-09-10T16:32:13.621 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.617+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2423454941}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861937868}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3795122979}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1861937868}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018438715}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397321862}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1832385808}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397321862}) 2024-09-10T16:32:13.622 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.618+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3120689413}) 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104067673}) 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3740396508}) 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2104067673}) 2024-09-10T16:32:13.625 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.619+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2256588011}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3817158975}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=933224241}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3817158975}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=979376769}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374294158}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3781003104}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2374294158}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.620+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=688947941}) 2024-09-10T16:32:13.626 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685914178}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3149564333}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3685914178}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.621+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130534666}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448999693}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904513984}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448999693}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.622+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=531809559}) 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.627 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133308217}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=619205074}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4133308217}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2712684928}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725218580}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1519076663}) 2024-09-10T16:32:13.628 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.629 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3725218580}) 2024-09-10T16:32:13.629 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.623+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3907261322}) 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431405690}) 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2523323419}) 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=431405690}) 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.624+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578778873}) 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.634 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983130184}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1828403053}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2983130184}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.625+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702876792}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234099890}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3514663225}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234099890}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2727529308}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.626+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046722113}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2649058841}) 2024-09-10T16:32:13.635 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2046722113}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3580016873}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265328426}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2648981949}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3265328426}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.627+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454156993}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790450542}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3839828471}) 2024-09-10T16:32:13.636 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=790450542}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.628+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=271689495}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590276569}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3684373431}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=590276569}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3326379408}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219297647}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3752542105}) 2024-09-10T16:32:13.637 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219297647}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.629+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1849585393}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603992566}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1616688352}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=603992566}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.630+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3115273442}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173572824}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2621368760}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=173572824}) 2024-09-10T16:32:13.638 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.631+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1520082361}) 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128080621}) 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3720550815}) 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2128080621}) 2024-09-10T16:32:13.639 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:13.632+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=722209345}) 2024-09-10T16:32:13.905 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T16:32:13.905 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.011+0000 7f0f5c00c640 10 monclient: get_monmap_and_config 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.011+0000 7f0f5c00c640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.011+0000 7f0f5c00c640 10 monclient: monmap: 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:14.012482+0000 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:14.012482+0000 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:14.013 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:14.014 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:14.014 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:14.014 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:14.014 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 Processor -- start 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 -- start start 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient: init 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient: _add_conns ranks=[1,0,3,4,2] 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f5409a430 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0f54074cb0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0f540a2820 0x7f0f54144f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.016 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-a con 0x7f0f540a2820 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f5409cbd0 con 0x7f0f54074cb0 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f5409c2c0 con 0x7f0f5409a430 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0f540a2320 con 0x7f0f540a2820 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.012+0000 7f0f5c00c640 10 monclient(hunting): authenticate will time out at 2312.499989s 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f595a7640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0f54074cb0 0x7f0f54141840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:38044/0 (socket says 172.21.15.164:38044) 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f595a7640 1 -- 172.21.15.164:0/2757054097 learned_addr learned my addr 172.21.15.164:0/2757054097 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 1 --2- 172.21.15.164:0/2757054097 >> v2:172.21.15.164:3300/0 conn(0x7f0f540a2820 0x7f0f54144f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): get_auth_request con 0x7f0f540a2820 auth_method 0 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.017 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_done global_id 294135 payload 274 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 1 -- 172.21.15.164:0/2757054097 >> v1:172.21.15.165:6789/0 conn(0x7f0f5409a430 legacy=0x7f0f5409a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 1 -- 172.21.15.164:0/2757054097 >> v1:172.21.15.164:6789/0 conn(0x7f0f54074cb0 legacy=0x7f0f54141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 1 monclient: found mon.noname-a 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.018 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.013+0000 7f0f5a5a9640 1 -- 172.21.15.164:0/2757054097 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f540a2ca0 con 0x7f0f540a2820 2024-09-10T16:32:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5a5a9640 1 -- 172.21.15.164:0/2757054097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f540a2820 msgr2=0x7f0f54144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5a5a9640 1 --2- 172.21.15.164:0/2757054097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f540a2820 0x7f0f54144f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009d90 tx=0x7f0f4c02a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ab22a55e32ba27d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 1 -- 172.21.15.164:0/2757054097 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c007a30 con 0x7f0f540a2820 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: dump: 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.025 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 1 -- 172.21.15.164:0/2757054097 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f4c007bd0 con 0x7f0f540a2820 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 5 monclient: authenticate success, global_id 294135 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 1 -- 172.21.15.164:0/2757054097 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f4c031400 con 0x7f0f540a2820 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 4 set_mon_vals no callback set 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f58da6640 10 monclient: dump: 2024-09-10T16:32:14.026 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 10 monclient: shutdown 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 -- 172.21.15.164:0/2757054097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f540a2820 msgr2=0x7f0f54144f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/2757054097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f540a2820 0x7f0f54144f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c009d90 tx=0x7f0f4c02a490 comp rx=0 tx=0).stop 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 -- 172.21.15.164:0/2757054097 shutdown_connections 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/2757054097 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f540a2820 0x7f0f54144f70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 -- 172.21.15.164:0/2757054097 >> 172.21.15.164:0/2757054097 conn(0x7f0f540a3110 msgr2=0x7f0f540a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 -- 172.21.15.164:0/2757054097 shutdown_connections 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.016+0000 7f0f5c00c640 1 -- 172.21.15.164:0/2757054097 wait complete. 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.017+0000 7f0f5c00c640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.017+0000 7f0f5c00c640 10 monclient: monmap: 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.028 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.017+0000 7f0f5c00c640 1 Processor -- start 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.017+0000 7f0f5c00c640 1 -- start start 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.017+0000 7f0f5c00c640 10 monclient: init 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f54074cb0 0x7f0f5412b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-b con 0x7f0f54074cb0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5409a430 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 0x7f0f5412bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): picked mon.noname-c con 0x7f0f540a2820 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5409b0a0 con 0x7f0f5409a430 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0f5409af20 con 0x7f0f54074cb0 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0f5409b220 con 0x7f0f540a2820 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5c00c640 10 monclient(hunting): authenticate will time out at 2312.505384s 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:42720/0 (socket says 172.21.15.164:42720) 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 -- 172.21.15.164:0/3783030284 learned_addr learned my addr 172.21.15.164:0/3783030284 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): get_auth_request con 0x7f0f5409a430 auth_method 0 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 0x7f0f5412bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): get_auth_request con 0x7f0f540a2820 auth_method 0 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f54074cb0 0x7f0f5412b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): get_auth_request con 0x7f0f54074cb0 auth_method 0 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f59da8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient(hunting): handle_auth_done global_id 294138 payload 824 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 -- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 msgr2=0x7f0f5412bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 0x7f0f5412bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 -- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f54074cb0 msgr2=0x7f0f5412b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f54074cb0 0x7f0f5412b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 monclient: found mon.noname-a 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f595a7640 1 -- 172.21.15.164:0/3783030284 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f5412dad0 con 0x7f0f5409a430 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.018+0000 7f0f5a5a9640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 0x7f0f5412bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f595a7640 1 -- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 msgr2=0x7f0f5412ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f595a7640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800c490 tx=0x7f0f4800c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48007070 con 0x7f0f5409a430 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: dump: 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f48007230 con 0x7f0f5409a430 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 5 monclient: authenticate success, global_id 294138 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 10 monclient: _renew_subs 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f5412ddb0 con 0x7f0f5409a430 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f48007620 con 0x7f0f5409a430 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f467fc640 10 monclient: dump: 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.036 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f47fff640 4 set_mon_vals no callback set 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 10 monclient: _renew_subs 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.021+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f54073f40 con 0x7f0f5409a430 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0f4801b400 con 0x7f0f5409a430 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f5c00c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f5c00c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f5c00c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f54096bf0 con 0x7f0f5409a430 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f467fc640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 0x7f0f380406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f0f48052820 con 0x7f0f5409a430 2024-09-10T16:32:14.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f59da8640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 0x7f0f380406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.022+0000 7f0f59da8640 10 monclient: get_auth_request con 0x7f0f3803e210 auth_method 0 2024-09-10T16:32:14.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.025+0000 7f0f59da8640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 0x7f0f380406d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f50004f60 tx=0x7f0f50007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.027+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f0f54096bf0 con 0x7f0f5409a430 2024-09-10T16:32:14.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.027+0000 7f0f467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.027+0000 7f0f467fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:14.039 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.039 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416403134}) 2024-09-10T16:32:14.039 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414335455}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3416403134}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.035+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4237466361}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479733731}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1312288766}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2479733731}) 2024-09-10T16:32:14.040 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.036+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2900286305}) 2024-09-10T16:32:14.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489713445}) 2024-09-10T16:32:14.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3714476507}) 2024-09-10T16:32:14.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489713445}) 2024-09-10T16:32:14.051 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=828805911}) 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181256663}) 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860398047}) 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3181256663}) 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=372457274}) 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.052 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446838422}) 2024-09-10T16:32:14.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2935414281}) 2024-09-10T16:32:14.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=446838422}) 2024-09-10T16:32:14.053 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1487218640}) 2024-09-10T16:32:14.054 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670819793}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2397944469}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670819793}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1896604787}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2718140067}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2720018359}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2718140067}) 2024-09-10T16:32:14.055 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1010992824}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651482620}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2129253914}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1651482620}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=718269354}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237396144}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116283279}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=237396144}) 2024-09-10T16:32:14.056 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780687676}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036248654}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3176737659}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1036248654}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154785025}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861829270}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602532944}) 2024-09-10T16:32:14.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3861829270}) 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2032377001}) 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063183420}) 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4267848436}) 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3063183420}) 2024-09-10T16:32:14.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=634688925}) 2024-09-10T16:32:14.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2915057875}) 2024-09-10T16:32:14.123 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1941084182}) 2024-09-10T16:32:14.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2915057875}) 2024-09-10T16:32:14.124 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.122+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=512067732}) 2024-09-10T16:32:14.126 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.127 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458435134}) 2024-09-10T16:32:14.127 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=903848589}) 2024-09-10T16:32:14.127 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.127 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1458435134}) 2024-09-10T16:32:14.127 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.125+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1504951524}) 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839354059}) 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1247392344}) 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1839354059}) 2024-09-10T16:32:14.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3662600767}) 2024-09-10T16:32:14.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.278+0000 7f0f5c00c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T16:32:14.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.278+0000 7f0f5c00c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:14.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.278+0000 7f0f5c00c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.278+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0f5409a890 con 0x7f0f5409a430 2024-09-10T16:32:14.281 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.279+0000 7f0f467fc640 1 -- 172.21.15.164:0/3783030284 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f0f480513b0 con 0x7f0f5409a430 2024-09-10T16:32:14.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.279+0000 7f0f467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T16:32:14.282 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.279+0000 7f0f467fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:14.282 INFO:teuthology.orchestra.run.smithi164.stdout:{"election_epoch":250,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","modified":"2024-09-10T16:05:30.502390Z","created":"2024-09-10T16:05:30.502390Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3300","nonce":0},{"type":"v1","addr":"172.21.15.164:6789","nonce":0}]},"addr":"172.21.15.164:6789/0","public_addr":"172.21.15.164:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:3300","nonce":0},{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:3301","nonce":0},{"type":"v1","addr":"172.21.15.164:6790","nonce":0}]},"addr":"172.21.15.164:6790/0","public_addr":"172.21.15.164:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T16:32:14.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 msgr2=0x7f0f380406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 0x7f0f380406d0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f0f50004f60 tx=0x7f0f50007e30 comp rx=0 tx=0).stop 2024-09-10T16:32:14.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 10 monclient: shutdown 2024-09-10T16:32:14.283 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 msgr2=0x7f0f5412ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0f4800c490 tx=0x7f0f4800c950 comp rx=0 tx=0).stop 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 shutdown_connections 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0f3803e210 0x7f0f380406d0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0f540a2820 0x7f0f5412bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0f5409a430 0x7f0f5412ba30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 --2- 172.21.15.164:0/3783030284 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0f54074cb0 0x7f0f5412b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 >> 172.21.15.164:0/3783030284 conn(0x7f0f540a3110 msgr2=0x7f0f540a51c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 shutdown_connections 2024-09-10T16:32:14.284 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.281+0000 7f0f5c00c640 1 -- 172.21.15.164:0/3783030284 wait complete. 2024-09-10T16:32:14.293 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T16:32:14.293 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-10T16:32:14.304 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T16:32:14.304 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.401+0000 7f52897bb640 10 monclient: get_monmap_and_config 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.401+0000 7f52897bb640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.401+0000 7f52897bb640 10 monclient: monmap: 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:14.402657+0000 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:14.402657+0000 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.403 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:14.404 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 1 Processor -- start 2024-09-10T16:32:14.406 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 1 -- start start 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 10 monclient: init 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.402+0000 7f52897bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-b con 0x7f5284135ce0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52840a40d0 0x7f52841360c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-a con 0x7f52840a40d0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5284136600 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:14.407 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52840a4ac0 con 0x7f5284136600 2024-09-10T16:32:14.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f52840a1400 con 0x7f52840a40d0 2024-09-10T16:32:14.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f52840a1120 con 0x7f5284135ce0 2024-09-10T16:32:14.408 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f52897bb640 10 monclient(hunting): authenticate will time out at 2312.890067s 2024-09-10T16:32:14.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f52840a40d0 0x7f52841360c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32870/0 (socket says 172.21.15.164:32870) 2024-09-10T16:32:14.409 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 1 -- 172.21.15.164:0/3747843968 learned_addr learned my addr 172.21.15.164:0/3747843968 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): get_auth_request con 0x7f52840a40d0 auth_method 0 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): get_auth_request con 0x7f5284135ce0 auth_method 0 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.410 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient(hunting): handle_auth_done global_id 294141 payload 274 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 -- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 msgr2=0x7f52840a23a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 -- 172.21.15.164:0/3747843968 >> v1:172.21.15.164:6789/0 conn(0x7f5284136600 legacy=0x7f528409b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 monclient: found mon.noname-a 2024-09-10T16:32:14.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f527bfff640 1 -- 172.21.15.164:0/3747843968 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52840a28e0 con 0x7f52840a40d0 2024-09-10T16:32:14.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.403+0000 7f5282ffd640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:14.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f527bfff640 1 -- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52840a40d0 msgr2=0x7f52841360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f527bfff640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52840a40d0 0x7f52841360c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097a0 tx=0x7f526c029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fe33df28064688d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 1 -- 172.21.15.164:0/3747843968 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c037070 con 0x7f52840a40d0 2024-09-10T16:32:14.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.419 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: dump: 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.420 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 1 -- 172.21.15.164:0/3747843968 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f526c032440 con 0x7f52840a40d0 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 5 monclient: authenticate success, global_id 294141 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 1 -- 172.21.15.164:0/3747843968 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f526c032810 con 0x7f52840a40d0 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 4 set_mon_vals no callback set 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52827fc640 10 monclient: dump: 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.421 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 10 monclient: shutdown 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 1 -- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52840a40d0 msgr2=0x7f52841360c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.406+0000 7f52897bb640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52840a40d0 0x7f52841360c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f526c0097a0 tx=0x7f526c029fd0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.422 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 -- 172.21.15.164:0/3747843968 shutdown_connections 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f52840a40d0 0x7f52841360c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 --2- 172.21.15.164:0/3747843968 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5284135ce0 0x7f52840a23a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 -- 172.21.15.164:0/3747843968 >> 172.21.15.164:0/3747843968 conn(0x7f52840a4f50 msgr2=0x7f52840a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 -- 172.21.15.164:0/3747843968 shutdown_connections 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 -- 172.21.15.164:0/3747843968 wait complete. 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 10 monclient: monmap: 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.423 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.407+0000 7f52897bb640 1 Processor -- start 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 -- start start 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient: init 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-c con 0x7f52840a40d0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5284135ce0 0x7f52841cb810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-b con 0x7f5284135ce0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5284136600 0x7f528409d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.424 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): picked mon.noname-a con 0x7f5284136600 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f528409b970 con 0x7f5284136600 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f528409b7f0 con 0x7f5284135ce0 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f528409baf0 con 0x7f52840a40d0 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52897bb640 10 monclient(hunting): authenticate will time out at 2312.895512s 2024-09-10T16:32:14.425 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32874/0 (socket says 172.21.15.164:32874) 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 -- 172.21.15.164:0/3625198394 learned_addr learned my addr 172.21.15.164:0/3625198394 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): get_auth_request con 0x7f52840a40d0 auth_method 0 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5284136600 0x7f528409d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.426 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): get_auth_request con 0x7f5284136600 auth_method 0 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f52837fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f527bfff640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5284135ce0 0x7f52841cb810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.427 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f527bfff640 10 monclient(hunting): get_auth_request con 0x7f5284135ce0 auth_method 0 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f527bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f527bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f527bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient(hunting): handle_auth_done global_id 274316 payload 824 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 -- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5284135ce0 msgr2=0x7f52841cb810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5284135ce0 0x7f52841cb810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 -- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5284136600 msgr2=0x7f528409d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5284136600 0x7f528409d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 monclient: found mon.noname-c 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.428 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.408+0000 7f5282ffd640 1 -- 172.21.15.164:0/3625198394 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5284072a40 con 0x7f52840a40d0 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f5282ffd640 1 -- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 msgr2=0x7f52841cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f5282ffd640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f527400e9d0 tx=0x7f527400ee90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274017070 con 0x7f52840a40d0 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: dump: 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.429 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f527400cc70 con 0x7f52840a40d0 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 5 monclient: authenticate success, global_id 274316 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 10 monclient: _renew_subs 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5284072d20 con 0x7f52840a40d0 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.430 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5274013670 con 0x7f52840a40d0 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f527b7fe640 10 monclient: dump: 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.431 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f5281ffb640 4 set_mon_vals no callback set 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 10 monclient: _renew_subs 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.411+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f528409c140 con 0x7f52840a40d0 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f527401a070 con 0x7f52840a40d0 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f52897bb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f52897bb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f52897bb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f528409bf00 con 0x7f52840a40d0 2024-09-10T16:32:14.432 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f527b7fe640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 0x7f5264040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f5274052b10 con 0x7f52840a40d0 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.412+0000 7f527bfff640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 0x7f5264040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.413+0000 7f527bfff640 10 monclient: get_auth_request con 0x7f526403e260 auth_method 0 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.415+0000 7f527bfff640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 0x7f5264040720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f526c034000 tx=0x7f526c02e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.417+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f527400ce20 con 0x7f52840a40d0 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.417+0000 7f527b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.433 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.417+0000 7f527b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:14.589 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.587+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f52840770e0 con 0x7f526403e260 2024-09-10T16:32:14.590 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.589+0000 7f527b7fe640 1 -- 172.21.15.164:0/3625198394 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34028 (secure 0 0 0) 0x7f52840770e0 con 0x7f526403e260 2024-09-10T16:32:14.591 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:14.591 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:32:14.592 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 msgr2=0x7f5264040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 0x7f5264040720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f526c034000 tx=0x7f526c02e000 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 10 monclient: shutdown 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 msgr2=0x7f52841cb2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f527400e9d0 tx=0x7f527400ee90 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 shutdown_connections 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f526403e260 0x7f5264040720 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5284136600 0x7f528409d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5284135ce0 0x7f52841cb810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 --2- 172.21.15.164:0/3625198394 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f52840a40d0 0x7f52841cb2d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 >> 172.21.15.164:0/3625198394 conn(0x7f52840a4f50 msgr2=0x7f52840a6250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.593 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 shutdown_connections 2024-09-10T16:32:14.594 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.591+0000 7f52897bb640 1 -- 172.21.15.164:0/3625198394 wait complete. 2024-09-10T16:32:14.603 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-09-10T16:32:13.688676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":290736,"kb_used_data":9272,"kb_used_omap":9,"kb_used_meta":281334,"kb_avail":565940304,"statfs":{"total":579820584960,"available":579522871296,"internally_reserved":0,"allocated":9494528,"data_stored":5105380,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":288086691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1839,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326935+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:32:02.326935+0000","last_peered":"2024-09-10T16:32:02.326935+0000","last_clean":"2024-09-10T16:32:02.326935+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:32:02.326935+0000","last_undegraded":"2024-09-10T16:32:02.326935+0000","last_fullsized":"2024-09-10T16:32:02.326935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1749,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326946+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:32:02.326946+0000","last_peered":"2024-09-10T16:32:02.326946+0000","last_clean":"2024-09-10T16:32:02.326946+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:32:02.326946+0000","last_undegraded":"2024-09-10T16:32:02.326946+0000","last_fullsized":"2024-09-10T16:32:02.326946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1757,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326957+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:32:02.326957+0000","last_peered":"2024-09-10T16:32:02.326957+0000","last_clean":"2024-09-10T16:32:02.326957+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:32:02.326957+0000","last_undegraded":"2024-09-10T16:32:02.326957+0000","last_fullsized":"2024-09-10T16:32:02.326957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1769,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.188520+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:32:01.188520+0000","last_peered":"2024-09-10T16:32:01.188520+0000","last_clean":"2024-09-10T16:32:01.188520+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:32:01.188520+0000","last_undegraded":"2024-09-10T16:32:01.188520+0000","last_fullsized":"2024-09-10T16:32:01.188520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1767,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326850+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:32:02.326850+0000","last_peered":"2024-09-10T16:32:02.326850+0000","last_clean":"2024-09-10T16:32:02.326850+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:32:02.326850+0000","last_undegraded":"2024-09-10T16:32:02.326850+0000","last_fullsized":"2024-09-10T16:32:02.326850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326638+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:32:02.326638+0000","last_peered":"2024-09-10T16:32:02.326638+0000","last_clean":"2024-09-10T16:32:02.326638+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:32:02.326638+0000","last_undegraded":"2024-09-10T16:32:02.326638+0000","last_fullsized":"2024-09-10T16:32:02.326638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1747,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327123+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:32:02.327123+0000","last_peered":"2024-09-10T16:32:02.327123+0000","last_clean":"2024-09-10T16:32:02.327123+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:32:02.327123+0000","last_undegraded":"2024-09-10T16:32:02.327123+0000","last_fullsized":"2024-09-10T16:32:02.327123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.187848+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:32:01.187848+0000","last_peered":"2024-09-10T16:32:01.187848+0000","last_clean":"2024-09-10T16:32:01.187848+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:32:01.187848+0000","last_undegraded":"2024-09-10T16:32:01.187848+0000","last_fullsized":"2024-09-10T16:32:01.187848+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1746,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327125+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:32:02.327125+0000","last_peered":"2024-09-10T16:32:02.327125+0000","last_clean":"2024-09-10T16:32:02.327125+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:32:02.327125+0000","last_undegraded":"2024-09-10T16:32:02.327125+0000","last_fullsized":"2024-09-10T16:32:02.327125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28612,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343228,"statfs":{"total":96636764160,"available":96607465472,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41092,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330748,"statfs":{"total":96636764160,"available":96594685952,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46600,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325240,"statfs":{"total":96636764160,"available":96589045760,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:32:14.604 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:32:14.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.699+0000 7ff42409b640 10 monclient: get_monmap_and_config 2024-09-10T16:32:14.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.699+0000 7ff42409b640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.701 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.699+0000 7ff42409b640 10 monclient: monmap: 2024-09-10T16:32:14.702 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.702 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.702 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:14.700689+0000 2024-09-10T16:32:14.702 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:14.700689+0000 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:14.703 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.700+0000 7ff42409b640 1 Processor -- start 2024-09-10T16:32:14.713 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.700+0000 7ff42409b640 1 -- start start 2024-09-10T16:32:14.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.700+0000 7ff42409b640 10 monclient: init 2024-09-10T16:32:14.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient: _add_conns ranks=[0,4,3,1,2] 2024-09-10T16:32:14.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff41c09c180 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:14.714 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ff41c09d160 0x7ff41c09f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-c con 0x7ff41c09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff41c13a5c0 0x7ff41c13c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-a con 0x7ff41c13a5c0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff41c09a430 con 0x7ff41c09c180 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7ff41c0a24e0 con 0x7ff41c13a5c0 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.715 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ff41c0a2bf0 con 0x7ff41c09d160 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff42409b640 10 monclient(hunting): authenticate will time out at 2313.188172s 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7ff41c13a5c0 0x7ff41c13c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff421e37640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7ff41c09c180 0x7ff41c09ea40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49632/0 (socket says 172.21.15.164:49632) 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff421e37640 1 -- 172.21.15.164:0/1067930653 learned_addr learned my addr 172.21.15.164:0/1067930653 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): get_auth_request con 0x7ff41c13a5c0 auth_method 0 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.716 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff421636640 1 --2- 172.21.15.164:0/1067930653 >> v2:172.21.15.165:3300/0 conn(0x7ff41c09d160 0x7ff41c09f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient(hunting): handle_auth_done global_id 294147 payload 274 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 -- 172.21.15.164:0/1067930653 >> v2:172.21.15.165:3300/0 conn(0x7ff41c09d160 msgr2=0x7ff41c09f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 --2- 172.21.15.164:0/1067930653 >> v2:172.21.15.165:3300/0 conn(0x7ff41c09d160 0x7ff41c09f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.717 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 -- 172.21.15.164:0/1067930653 >> v1:172.21.15.164:6789/0 conn(0x7ff41c09c180 legacy=0x7ff41c09ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 monclient: found mon.noname-a 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.701+0000 7ff422638640 1 -- 172.21.15.164:0/1067930653 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c09c560 con 0x7ff41c13a5c0 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff422638640 1 -- 172.21.15.164:0/1067930653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c13a5c0 msgr2=0x7ff41c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff422638640 1 --2- 172.21.15.164:0/1067930653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c13a5c0 0x7ff41c13c9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a33b75ae9c6b0491 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 1 -- 172.21.15.164:0/1067930653 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414037070 con 0x7ff41c13a5c0 2024-09-10T16:32:14.718 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: dump: 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.719 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.720 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 1 -- 172.21.15.164:0/1067930653 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff414032640 con 0x7ff41c13a5c0 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff42409b640 5 monclient: authenticate success, global_id 294147 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff42409b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 1 -- 172.21.15.164:0/1067930653 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414032a10 con 0x7ff41c13a5c0 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff42409b640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff42409b640 4 set_mon_vals no callback set 2024-09-10T16:32:14.721 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.704+0000 7ff420e35640 10 monclient: dump: 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.722 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 10 monclient: shutdown 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.723 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 -- 172.21.15.164:0/1067930653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c13a5c0 msgr2=0x7ff41c13c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 --2- 172.21.15.164:0/1067930653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c13a5c0 0x7ff41c13c9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff4140097a0 tx=0x7ff41402a010 comp rx=0 tx=0).stop 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 -- 172.21.15.164:0/1067930653 shutdown_connections 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 --2- 172.21.15.164:0/1067930653 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c13a5c0 0x7ff41c13c9b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 --2- 172.21.15.164:0/1067930653 >> v2:172.21.15.165:3300/0 conn(0x7ff41c09d160 0x7ff41c09f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 -- 172.21.15.164:0/1067930653 >> 172.21.15.164:0/1067930653 conn(0x7ff41c0a3110 msgr2=0x7ff41c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 -- 172.21.15.164:0/1067930653 shutdown_connections 2024-09-10T16:32:14.724 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 1 -- 172.21.15.164:0/1067930653 wait complete. 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 10 monclient: build_initial_monmap 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.705+0000 7ff42409b640 10 monclient: monmap: 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:14.725 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 Processor -- start 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 -- start start 2024-09-10T16:32:14.726 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient: init 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff41c09c180 0x7ff41c092600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-b con 0x7ff41c09c180 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-a con 0x7ff41c09d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:14.727 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 0x7ff41c1ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): picked mon.noname-c con 0x7ff41c13a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c074f90 con 0x7ff41c09d160 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ff41c074e10 con 0x7ff41c09c180 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:14.728 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ff41c09f690 con 0x7ff41c13a5c0 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff42409b640 10 monclient(hunting): authenticate will time out at 2313.193730s 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 0x7ff41c1ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47740/0 (socket says 172.21.15.164:47740) 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 1 -- 172.21.15.164:0/4026964055 learned_addr learned my addr 172.21.15.164:0/4026964055 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): get_auth_request con 0x7ff41c09d160 auth_method 0 2024-09-10T16:32:14.729 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): get_auth_request con 0x7ff41c13a5c0 auth_method 0 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.730 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff421636640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.706+0000 7ff422638640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421e37640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff41c09c180 0x7ff41c092600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421e37640 10 monclient(hunting): get_auth_request con 0x7ff41c09c180 auth_method 0 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421e37640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421e37640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421e37640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:14.731 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 10 monclient(hunting): handle_auth_done global_id 274319 payload 824 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 -- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff41c09c180 msgr2=0x7ff41c092600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff41c09c180 0x7ff41c092600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 -- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 msgr2=0x7ff41c1c9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 10 monclient: _finish_hunting 0 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 monclient: found mon.noname-c 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.732 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff422638640 1 -- 172.21.15.164:0/4026964055 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff41c093920 con 0x7ff41c13a5c0 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.707+0000 7ff421636640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff422638640 1 -- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 msgr2=0x7ff41c1ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff422638640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 0x7ff41c1ca0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff414031000 tx=0x7ff4140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414037040 con 0x7ff41c13a5c0 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: dump: 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.733 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:14.734 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff414032e50 con 0x7ff41c13a5c0 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff414030ac0 con 0x7ff41c13a5c0 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 5 monclient: authenticate success, global_id 274319 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 10 monclient: _renew_subs 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff41c093c00 con 0x7ff41c13a5c0 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41a7fc640 10 monclient: dump: 2024-09-10T16:32:14.735 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff41bfff640 4 set_mon_vals no callback set 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 10 monclient: _renew_subs 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.709+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff41c1ca630 con 0x7ff41c13a5c0 2024-09-10T16:32:14.736 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ff414044020 con 0x7ff41c13a5c0 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff42409b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff42409b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff42409b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff41c093f90 con 0x7ff41c13a5c0 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff41a7fc640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 0x7ff4080406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.710+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7ff414072c30 con 0x7ff41c13a5c0 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.711+0000 7ff421e37640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 0x7ff4080406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:14.737 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.711+0000 7ff421e37640 10 monclient: get_auth_request con 0x7ff40803e210 auth_method 0 2024-09-10T16:32:14.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.713+0000 7ff421e37640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 0x7ff4080406d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007b70 tx=0x7ff40c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:14.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.715+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7ff4140717c0 con 0x7ff41c13a5c0 2024-09-10T16:32:14.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.715+0000 7ff41a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:14.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.715+0000 7ff41a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:14.872 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819564910}) 2024-09-10T16:32:14.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3634033227}) 2024-09-10T16:32:14.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1819564910}) 2024-09-10T16:32:14.873 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.871+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=896321383}) 2024-09-10T16:32:14.875 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526932748}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3052857713}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526932748}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.872+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2076906881}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550540442}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=73602104}) 2024-09-10T16:32:14.876 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2550540442}) 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.873+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581604833}) 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597143613}) 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=24090125}) 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1597143613}) 2024-09-10T16:32:14.877 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.874+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658473597}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310633177}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2988700896}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310633177}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.875+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2550234438}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276098708}) 2024-09-10T16:32:14.885 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2159715748}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4276098708}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.876+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=897605427}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758438303}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149469988}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2758438303}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.877+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=227066711}) 2024-09-10T16:32:14.886 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868709150}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1159948039}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1868709150}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.878+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2091400314}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982899882}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3202071528}) 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.887 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1982899882}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.879+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3993602307}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288305582}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=820867583}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288305582}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.880+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2156816078}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274640054}) 2024-09-10T16:32:14.888 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3281299218}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=274640054}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=588373874}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408269132}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1482652884}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.881+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3408269132}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.882+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2243793818}) 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.882+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.889 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.883+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033402487}) 2024-09-10T16:32:14.890 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.883+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1593808365}) 2024-09-10T16:32:14.890 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.883+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:14.890 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.883+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1033402487}) 2024-09-10T16:32:14.890 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:14.883+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1094237872}) 2024-09-10T16:32:14.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.894+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff41c0942c0 con 0x7ff40803e210 2024-09-10T16:32:14.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.895+0000 7ff41a7fc640 1 -- 172.21.15.164:0/4026964055 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34028 (secure 0 0 0) 0x7ff41c0942c0 con 0x7ff40803e210 2024-09-10T16:32:14.897 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:14.897 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:32:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 msgr2=0x7ff4080406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.899 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 0x7ff4080406d0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7ff40c007b70 tx=0x7ff40c009040 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 10 monclient: shutdown 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 msgr2=0x7ff41c1ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 0x7ff41c1ca0f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7ff414031000 tx=0x7ff4140046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 shutdown_connections 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ff40803e210 0x7ff4080406d0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ff41c13a5c0 0x7ff41c1ca0f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ff41c09d160 0x7ff41c1c9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 --2- 172.21.15.164:0/4026964055 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ff41c09c180 0x7ff41c092600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 >> 172.21.15.164:0/4026964055 conn(0x7ff41c0a3110 msgr2=0x7ff41c1399f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:14.900 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 shutdown_connections 2024-09-10T16:32:14.901 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:14.897+0000 7ff42409b640 1 -- 172.21.15.164:0/4026964055 wait complete. 2024-09-10T16:32:14.909 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-09-10T16:32:13.688676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":290736,"kb_used_data":9272,"kb_used_omap":9,"kb_used_meta":281334,"kb_avail":565940304,"statfs":{"total":579820584960,"available":579522871296,"internally_reserved":0,"allocated":9494528,"data_stored":5105380,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":288086691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1839,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326935+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:32:02.326935+0000","last_peered":"2024-09-10T16:32:02.326935+0000","last_clean":"2024-09-10T16:32:02.326935+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:32:02.326935+0000","last_undegraded":"2024-09-10T16:32:02.326935+0000","last_fullsized":"2024-09-10T16:32:02.326935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1749,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326946+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:32:02.326946+0000","last_peered":"2024-09-10T16:32:02.326946+0000","last_clean":"2024-09-10T16:32:02.326946+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:32:02.326946+0000","last_undegraded":"2024-09-10T16:32:02.326946+0000","last_fullsized":"2024-09-10T16:32:02.326946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1757,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326957+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:32:02.326957+0000","last_peered":"2024-09-10T16:32:02.326957+0000","last_clean":"2024-09-10T16:32:02.326957+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:32:02.326957+0000","last_undegraded":"2024-09-10T16:32:02.326957+0000","last_fullsized":"2024-09-10T16:32:02.326957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1769,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.188520+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:32:01.188520+0000","last_peered":"2024-09-10T16:32:01.188520+0000","last_clean":"2024-09-10T16:32:01.188520+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:32:01.188520+0000","last_undegraded":"2024-09-10T16:32:01.188520+0000","last_fullsized":"2024-09-10T16:32:01.188520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1767,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326850+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:32:02.326850+0000","last_peered":"2024-09-10T16:32:02.326850+0000","last_clean":"2024-09-10T16:32:02.326850+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:32:02.326850+0000","last_undegraded":"2024-09-10T16:32:02.326850+0000","last_fullsized":"2024-09-10T16:32:02.326850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326638+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:32:02.326638+0000","last_peered":"2024-09-10T16:32:02.326638+0000","last_clean":"2024-09-10T16:32:02.326638+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:32:02.326638+0000","last_undegraded":"2024-09-10T16:32:02.326638+0000","last_fullsized":"2024-09-10T16:32:02.326638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1747,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327123+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:32:02.327123+0000","last_peered":"2024-09-10T16:32:02.327123+0000","last_clean":"2024-09-10T16:32:02.327123+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:32:02.327123+0000","last_undegraded":"2024-09-10T16:32:02.327123+0000","last_fullsized":"2024-09-10T16:32:02.327123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.187848+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:32:01.187848+0000","last_peered":"2024-09-10T16:32:01.187848+0000","last_clean":"2024-09-10T16:32:01.187848+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:32:01.187848+0000","last_undegraded":"2024-09-10T16:32:01.187848+0000","last_fullsized":"2024-09-10T16:32:01.187848+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1746,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327125+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:32:02.327125+0000","last_peered":"2024-09-10T16:32:02.327125+0000","last_clean":"2024-09-10T16:32:02.327125+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:32:02.327125+0000","last_undegraded":"2024-09-10T16:32:02.327125+0000","last_fullsized":"2024-09-10T16:32:02.327125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28612,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343228,"statfs":{"total":96636764160,"available":96607465472,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41092,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330748,"statfs":{"total":96636764160,"available":96594685952,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46600,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325240,"statfs":{"total":96636764160,"available":96589045760,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:32:14.910 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T16:32:14.911 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:32:15.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.006+0000 7fb882ab6640 10 monclient: get_monmap_and_config 2024-09-10T16:32:15.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.006+0000 7fb882ab6640 10 monclient: build_initial_monmap 2024-09-10T16:32:15.008 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.006+0000 7fb882ab6640 10 monclient: monmap: 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:15.007802+0000 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:15.007802+0000 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.009 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:15.010 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:15.010 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:15.010 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:15.010 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:15.010 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.007+0000 7fb882ab6640 1 Processor -- start 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.007+0000 7fb882ab6640 1 -- start start 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.007+0000 7fb882ab6640 10 monclient: init 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient: _add_conns ranks=[2,1,3,0,4] 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-b con 0x7fb87c0a0410 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb87c0a2380 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:15.022 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb87c0a27e0 0x7fb87c13dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-a con 0x7fb87c0a27e0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb87c0a0210 con 0x7fb87c0a2380 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fb87c09d3f0 con 0x7fb87c0a27e0 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c09ed90 con 0x7fb87c0a0410 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb882ab6640 10 monclient(hunting): authenticate will time out at 2313.495277s 2024-09-10T16:32:15.023 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fb87c0a27e0 0x7fb87c13dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32896/0 (socket says 172.21.15.164:32896) 2024-09-10T16:32:15.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 1 -- 172.21.15.164:0/1363636639 learned_addr learned my addr 172.21.15.164:0/1363636639 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:15.024 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): get_auth_request con 0x7fb87c0a0410 auth_method 0 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): get_auth_request con 0x7fb87c0a27e0 auth_method 0 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.025 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient(hunting): handle_auth_done global_id 294153 payload 274 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 -- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 msgr2=0x7fb87c0a1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 -- 172.21.15.164:0/1363636639 >> v1:172.21.15.165:6789/0 conn(0x7fb87c0a2380 legacy=0x7fb87c13a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:15.026 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient: _finish_hunting 0 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 monclient: found mon.noname-a 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb881053640 1 -- 172.21.15.164:0/1363636639 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c09d930 con 0x7fb87c0a27e0 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.008+0000 7fb880852640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb881053640 1 -- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a27e0 msgr2=0x7fb87c13dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb881053640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a27e0 0x7fb87c13dcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb87c074db0 tx=0x7fb874029fb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9997bf248caf5724 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.027 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 1 -- 172.21.15.164:0/1363636639 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874037070 con 0x7fb87c0a27e0 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: dump: 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.028 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:15.029 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 1 -- 172.21.15.164:0/1363636639 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb874032440 con 0x7fb87c0a27e0 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb882ab6640 5 monclient: authenticate success, global_id 294153 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb882ab6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb882ab6640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb882ab6640 4 set_mon_vals no callback set 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 1 -- 172.21.15.164:0/1363636639 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb874032810 con 0x7fb87c0a27e0 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.030 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.011+0000 7fb87b7fe640 10 monclient: dump: 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.031 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 10 monclient: shutdown 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 -- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a27e0 msgr2=0x7fb87c13dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a27e0 0x7fb87c13dcf0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb87c074db0 tx=0x7fb874029fb0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 -- 172.21.15.164:0/1363636639 shutdown_connections 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a27e0 0x7fb87c13dcf0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1363636639 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a0410 0x7fb87c0a1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 -- 172.21.15.164:0/1363636639 >> 172.21.15.164:0/1363636639 conn(0x7fb87c0a3110 msgr2=0x7fb87c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:15.032 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 -- 172.21.15.164:0/1363636639 shutdown_connections 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 1 -- 172.21.15.164:0/1363636639 wait complete. 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 10 monclient: build_initial_monmap 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.012+0000 7fb882ab6640 10 monclient: monmap: 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:15.033 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 Processor -- start 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 -- start start 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient: init 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:15.034 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-a con 0x7fb87c0a0410 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb87c0a2380 0x7fb87c1ba550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-b con 0x7fb87c0a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 0x7fb87c1baa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): picked mon.noname-c con 0x7fb87c0a27e0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c09c910 con 0x7fb87c0a0410 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fb87c09eab0 con 0x7fb87c0a2380 2024-09-10T16:32:15.035 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fb87c0a07f0 con 0x7fb87c0a27e0 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb882ab6640 10 monclient(hunting): authenticate will time out at 2313.500898s 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47756/0 (socket says 172.21.15.164:47756) 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 1 -- 172.21.15.164:0/1712493806 learned_addr learned my addr 172.21.15.164:0/1712493806 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 10 monclient(hunting): get_auth_request con 0x7fb87c0a0410 auth_method 0 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb881053640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 0x7fb87c1baa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.036 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.013+0000 7fb880852640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): get_auth_request con 0x7fb87c0a27e0 auth_method 0 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb87c0a2380 0x7fb87c1ba550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.037 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.038 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): get_auth_request con 0x7fb87c0a2380 auth_method 0 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb87bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.039 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient(hunting): handle_auth_done global_id 294156 payload 824 2024-09-10T16:32:15.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 -- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 msgr2=0x7fb87c1baa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 0x7fb87c1baa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 -- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb87c0a2380 msgr2=0x7fb87c1ba550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb87c0a2380 0x7fb87c1ba550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.040 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient: _finish_hunting 0 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 monclient: found mon.noname-a 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb880852640 1 -- 172.21.15.164:0/1712493806 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb87c1b3fd0 con 0x7fb87c0a0410 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.014+0000 7fb881053640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 0x7fb87c1baa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb880852640 1 -- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 msgr2=0x7fb87c1cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb880852640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00efc0 tx=0x7fb86c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c014680 con 0x7fb87c0a0410 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: dump: 2024-09-10T16:32:15.041 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.042 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: _finish_auth 0 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: _check_auth_tickets 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 5 monclient: authenticate success, global_id 294156 2024-09-10T16:32:15.043 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb86c019070 con 0x7fb87c0a0410 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb86c014a20 con 0x7fb87c0a0410 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 10 monclient: _renew_subs 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb87c1cb340 con 0x7fb87c0a0410 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb87affd640 4 set_mon_vals no callback set 2024-09-10T16:32:15.044 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb8797fa640 10 monclient: dump: 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.045 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 10 monclient: _renew_subs 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.016+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb87c1b3a10 con 0x7fb87c0a0410 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fb86c005d10 con 0x7fb87c0a0410 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb8797fa640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 0x7fb85c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.046 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb882ab6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fb86c053160 con 0x7fb87c0a0410 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb882ab6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb882ab6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.017+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb87c09faa0 con 0x7fb87c0a0410 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.018+0000 7fb87bfff640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 0x7fb85c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.018+0000 7fb87bfff640 10 monclient: get_auth_request con 0x7fb85c03e210 auth_method 0 2024-09-10T16:32:15.047 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.020+0000 7fb87bfff640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 0x7fb85c0406d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb87c1b2db0 tx=0x7fb864007e30 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.022+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fb86c010040 con 0x7fb87c0a0410 2024-09-10T16:32:15.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.022+0000 7fb8797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.048 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.022+0000 7fb8797fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:15.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332426413}) 2024-09-10T16:32:15.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2419915550}) 2024-09-10T16:32:15.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=332426413}) 2024-09-10T16:32:15.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.032+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3425330988}) 2024-09-10T16:32:15.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247884346}) 2024-09-10T16:32:15.131 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2874339099}) 2024-09-10T16:32:15.131 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.131 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4247884346}) 2024-09-10T16:32:15.131 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=736957891}) 2024-09-10T16:32:15.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562197103}) 2024-09-10T16:32:15.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3128890799}) 2024-09-10T16:32:15.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=562197103}) 2024-09-10T16:32:15.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.133+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3663887442}) 2024-09-10T16:32:15.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.196+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb87c0008d0 con 0x7fb85c03e210 2024-09-10T16:32:15.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.198+0000 7fb8797fa640 1 -- 172.21.15.164:0/1712493806 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34028 (secure 0 0 0) 0x7fb87c0008d0 con 0x7fb85c03e210 2024-09-10T16:32:15.200 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:15.201 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:32:15.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 msgr2=0x7fb85c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 0x7fb85c0406d0 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fb87c1b2db0 tx=0x7fb864007e30 comp rx=0 tx=0).stop 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 10 monclient: shutdown 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 msgr2=0x7fb87c1cae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb86c00efc0 tx=0x7fb86c00c490 comp rx=0 tx=0).stop 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 shutdown_connections 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fb85c03e210 0x7fb85c0406d0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fb87c0a27e0 0x7fb87c1baa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fb87c0a2380 0x7fb87c1ba550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 --2- 172.21.15.164:0/1712493806 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fb87c0a0410 0x7fb87c1cae00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 >> 172.21.15.164:0/1712493806 conn(0x7fb87c0a3110 msgr2=0x7fb87c0a50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:15.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 shutdown_connections 2024-09-10T16:32:15.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.200+0000 7fb882ab6640 1 -- 172.21.15.164:0/1712493806 wait complete. 2024-09-10T16:32:15.213 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1673,"stamp":"2024-09-10T16:32:13.688676+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":290736,"kb_used_data":9272,"kb_used_omap":9,"kb_used_meta":281334,"kb_avail":565940304,"statfs":{"total":579820584960,"available":579522871296,"internally_reserved":0,"allocated":9494528,"data_stored":5105380,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":288086691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002640"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1839,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326935+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:32:02.326935+0000","last_peered":"2024-09-10T16:32:02.326935+0000","last_clean":"2024-09-10T16:32:02.326935+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:32:02.326935+0000","last_undegraded":"2024-09-10T16:32:02.326935+0000","last_fullsized":"2024-09-10T16:32:02.326935+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1749,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326946+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:32:02.326946+0000","last_peered":"2024-09-10T16:32:02.326946+0000","last_clean":"2024-09-10T16:32:02.326946+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:32:02.326946+0000","last_undegraded":"2024-09-10T16:32:02.326946+0000","last_fullsized":"2024-09-10T16:32:02.326946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1757,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326957+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:32:02.326957+0000","last_peered":"2024-09-10T16:32:02.326957+0000","last_clean":"2024-09-10T16:32:02.326957+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:32:02.326957+0000","last_undegraded":"2024-09-10T16:32:02.326957+0000","last_fullsized":"2024-09-10T16:32:02.326957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1769,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.188520+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:32:01.188520+0000","last_peered":"2024-09-10T16:32:01.188520+0000","last_clean":"2024-09-10T16:32:01.188520+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:32:01.188520+0000","last_undegraded":"2024-09-10T16:32:01.188520+0000","last_fullsized":"2024-09-10T16:32:01.188520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1767,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326850+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:32:02.326850+0000","last_peered":"2024-09-10T16:32:02.326850+0000","last_clean":"2024-09-10T16:32:02.326850+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:32:02.326850+0000","last_undegraded":"2024-09-10T16:32:02.326850+0000","last_fullsized":"2024-09-10T16:32:02.326850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326638+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:32:02.326638+0000","last_peered":"2024-09-10T16:32:02.326638+0000","last_clean":"2024-09-10T16:32:02.326638+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:32:02.326638+0000","last_undegraded":"2024-09-10T16:32:02.326638+0000","last_fullsized":"2024-09-10T16:32:02.326638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1747,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327123+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:32:02.327123+0000","last_peered":"2024-09-10T16:32:02.327123+0000","last_clean":"2024-09-10T16:32:02.327123+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:32:02.327123+0000","last_undegraded":"2024-09-10T16:32:02.327123+0000","last_fullsized":"2024-09-10T16:32:02.327123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.187848+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:32:01.187848+0000","last_peered":"2024-09-10T16:32:01.187848+0000","last_clean":"2024-09-10T16:32:01.187848+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:32:01.187848+0000","last_undegraded":"2024-09-10T16:32:01.187848+0000","last_fullsized":"2024-09-10T16:32:01.187848+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1746,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327125+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:32:02.327125+0000","last_peered":"2024-09-10T16:32:02.327125+0000","last_clean":"2024-09-10T16:32:02.327125+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:32:02.327125+0000","last_undegraded":"2024-09-10T16:32:02.327125+0000","last_fullsized":"2024-09-10T16:32:02.327125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575165,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575164,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28612,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343228,"statfs":{"total":96636764160,"available":96607465472,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41092,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330748,"statfs":{"total":96636764160,"available":96594685952,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46600,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325240,"statfs":{"total":96636764160,"available":96589045760,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640571,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:32:15.214 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 10 monclient: get_monmap_and_config 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 10 monclient: build_initial_monmap 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 10 monclient: monmap: 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:15.311050+0000 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:15.311050+0000 2024-09-10T16:32:15.312 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 1 Processor -- start 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 1 -- start start 2024-09-10T16:32:15.313 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.310+0000 7f65343ad640 10 monclient: init 2024-09-10T16:32:15.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:15.322 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652c06d160 0x7f652c134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-a con 0x7f652c06d160 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f652c06d540 0x7f652c0a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-c con 0x7f652c06d540 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f652c0a4c20 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f652c09cb90 con 0x7f652c0a4c20 2024-09-10T16:32:15.323 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f652c09ce70 con 0x7f652c06d160 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f652c09d150 con 0x7f652c06d540 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f65343ad640 10 monclient(hunting): authenticate will time out at 2313.798583s 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652c06d160 0x7f652c134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f653294a640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f652c0a4c20 0x7f652c137ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49642/0 (socket says 172.21.15.164:49642) 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f652c06d160 0x7f652c134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47766/0 (socket says 172.21.15.164:47766) 2024-09-10T16:32:15.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f653294a640 1 -- 172.21.15.164:0/3824334018 learned_addr learned my addr 172.21.15.164:0/3824334018 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): get_auth_request con 0x7f652c06d160 auth_method 0 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient(hunting): handle_auth_done global_id 294159 payload 274 2024-09-10T16:32:15.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 -- 172.21.15.164:0/3824334018 >> v2:172.21.15.165:3300/0 conn(0x7f652c06d540 msgr2=0x7f652c0a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6531948640 1 --2- 172.21.15.164:0/3824334018 >> v2:172.21.15.165:3300/0 conn(0x7f652c06d540 0x7f652c0a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 --2- 172.21.15.164:0/3824334018 >> v2:172.21.15.165:3300/0 conn(0x7f652c06d540 0x7f652c0a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 -- 172.21.15.164:0/3824334018 >> v1:172.21.15.164:6789/0 conn(0x7f652c0a4c20 legacy=0x7f652c137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient: _finish_hunting 0 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 monclient: found mon.noname-a 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:15.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.311+0000 7f6532149640 1 -- 172.21.15.164:0/3824334018 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c135340 con 0x7f652c06d160 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.314+0000 7f6532149640 1 -- 172.21.15.164:0/3824334018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 msgr2=0x7f652c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.314+0000 7f6532149640 1 --2- 172.21.15.164:0/3824334018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c134e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a9d40 tx=0x7f652802a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b16ca7a7e5e260 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 1 -- 172.21.15.164:0/3824334018 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528037070 con 0x7f652c06d160 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: dump: 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.327 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: _finish_auth 0 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: _check_auth_tickets 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 1 -- 172.21.15.164:0/3824334018 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6528032640 con 0x7f652c06d160 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 5 monclient: authenticate success, global_id 294159 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 1 -- 172.21.15.164:0/3824334018 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528032a10 con 0x7f652c06d160 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f6531147640 10 monclient: dump: 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.330 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 4 set_mon_vals no callback set 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 10 monclient: shutdown 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 -- 172.21.15.164:0/3824334018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 msgr2=0x7f652c134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 --2- 172.21.15.164:0/3824334018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c134e00 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f652c0a9d40 tx=0x7f652802a000 comp rx=0 tx=0).stop 2024-09-10T16:32:15.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 -- 172.21.15.164:0/3824334018 shutdown_connections 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 --2- 172.21.15.164:0/3824334018 >> v2:172.21.15.165:3300/0 conn(0x7f652c06d540 0x7f652c0a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 --2- 172.21.15.164:0/3824334018 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c134e00 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 -- 172.21.15.164:0/3824334018 >> 172.21.15.164:0/3824334018 conn(0x7f652c0aacb0 msgr2=0x7f652c0ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 -- 172.21.15.164:0/3824334018 shutdown_connections 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.315+0000 7f65343ad640 1 -- 172.21.15.164:0/3824334018 wait complete. 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.316+0000 7f65343ad640 10 monclient: build_initial_monmap 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.316+0000 7f65343ad640 10 monclient: monmap: 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:15.332 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.316+0000 7f65343ad640 1 Processor -- start 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.316+0000 7f65343ad640 1 -- start start 2024-09-10T16:32:15.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.316+0000 7f65343ad640 10 monclient: init 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-a con 0x7f652c06d160 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652c06d540 0x7f652c12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-b con 0x7f652c06d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652c0a4c20 0x7f652c1cd080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): picked mon.noname-c con 0x7f652c0a4c20 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f652c138780 con 0x7f652c06d160 2024-09-10T16:32:15.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f652c138600 con 0x7f652c06d540 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f652c138900 con 0x7f652c0a4c20 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f65343ad640 10 monclient(hunting): authenticate will time out at 2313.804296s 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47772/0 (socket says 172.21.15.164:47772) 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 -- 172.21.15.164:0/1161053062 learned_addr learned my addr 172.21.15.164:0/1161053062 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): get_auth_request con 0x7f652c06d160 auth_method 0 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652c0a4c20 0x7f652c1cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): get_auth_request con 0x7f652c0a4c20 auth_method 0 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:15.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f653294a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient(hunting): handle_auth_done global_id 294162 payload 824 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6531948640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652c06d540 0x7f652c12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 -- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652c0a4c20 msgr2=0x7f652c1cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652c0a4c20 0x7f652c1cd080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 -- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652c06d540 msgr2=0x7f652c12e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652c06d540 0x7f652c12e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient: _finish_hunting 0 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 monclient: found mon.noname-a 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:15.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.317+0000 7f6532149640 1 -- 172.21.15.164:0/1161053062 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f652c0a1de0 con 0x7f652c06d160 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.319+0000 7f6532149640 1 -- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 msgr2=0x7f652c12e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.319+0000 7f6532149640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6528000c00 tx=0x7f65280046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528037040 con 0x7f652c06d160 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: dump: 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.338 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6528042070 con 0x7f652c06d160 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 5 monclient: authenticate success, global_id 294162 2024-09-10T16:32:15.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 10 monclient: _renew_subs 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f652c06e6b0 con 0x7f652c06d160 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6528030d60 con 0x7f652c06d160 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6522ffd640 10 monclient: dump: 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:15.340 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f6530946640 4 set_mon_vals no callback set 2024-09-10T16:32:15.341 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 10 monclient: _renew_subs 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f652c0a3020 con 0x7f652c06d160 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.320+0000 7f65343ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f65343ad640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f65343ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f6528032500 con 0x7f652c06d160 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f652c134e00 con 0x7f652c06d160 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f6522ffd640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 0x7f65100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f6528073610 con 0x7f652c06d160 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f6531948640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 0x7f65100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:15.342 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.321+0000 7f6531948640 10 monclient: get_auth_request con 0x7f651003e210 auth_method 0 2024-09-10T16:32:15.343 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.324+0000 7f6531948640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 0x7f65100406d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f651c006fd0 tx=0x7f651c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:15.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.325+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f65280721a0 con 0x7f652c06d160 2024-09-10T16:32:15.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.325+0000 7f6522ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:15.344 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.325+0000 7f6522ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.503+0000 7f65343ad640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:32:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.503+0000 7f65343ad640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:32:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.503+0000 7f65343ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:15.505 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.503+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f652c0aa1e0 con 0x7f652c06d160 2024-09-10T16:32:15.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.504+0000 7f6522ffd640 1 -- 172.21.15.164:0/1161053062 <== mon.0 v2:172.21.15.164:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v897) ==== 74+0+12399 (secure 0 0 0) 0x7f652803d440 con 0x7f652c06d160 2024-09-10T16:32:15.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.504+0000 7f6522ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T16:32:15.506 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.504+0000 7f6522ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:15.506 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:15.506 INFO:teuthology.orchestra.run.smithi164.stdout:{"epoch":897,"fsid":"b134de91-f819-41d3-9bbd-da58ec0ed6be","created":"2024-09-10T16:06:03.055717+0000","modified":"2024-09-10T16:32:01.182962+0000","last_up_change":"2024-09-10T16:06:14.366957+0000","last_in_change":"2024-09-10T16:06:05.574588+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":6,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T16:06:13.375364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T16:06:20.696007+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"d38e4bd2-6da0-4c53-8a8c-26ae27d74b04","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6800","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6801","nonce":3366597746}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6802","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6803","nonce":3366597746}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6806","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6807","nonce":3366597746}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6804","nonce":3366597746},{"type":"v1","addr":"172.21.15.164:6805","nonce":3366597746}]},"public_addr":"172.21.15.164:6801/3366597746","cluster_addr":"172.21.15.164:6803/3366597746","heartbeat_back_addr":"172.21.15.164:6807/3366597746","heartbeat_front_addr":"172.21.15.164:6805/3366597746","state":["exists","up"]},{"osd":1,"uuid":"f75ad5a2-93bb-4997-9998-7bc4da2f5c07","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6816","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6817","nonce":1609205011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6818","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6819","nonce":1609205011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6822","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6823","nonce":1609205011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6820","nonce":1609205011},{"type":"v1","addr":"172.21.15.164:6821","nonce":1609205011}]},"public_addr":"172.21.15.164:6817/1609205011","cluster_addr":"172.21.15.164:6819/1609205011","heartbeat_back_addr":"172.21.15.164:6823/1609205011","heartbeat_front_addr":"172.21.15.164:6821/1609205011","state":["exists","up"]},{"osd":2,"uuid":"3a0b4d74-3a8e-41d6-ba2e-f18fa1a72de7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6808","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6809","nonce":1539912567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6810","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6811","nonce":1539912567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6814","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6815","nonce":1539912567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.164:6812","nonce":1539912567},{"type":"v1","addr":"172.21.15.164:6813","nonce":1539912567}]},"public_addr":"172.21.15.164:6809/1539912567","cluster_addr":"172.21.15.164:6811/1539912567","heartbeat_back_addr":"172.21.15.164:6815/1539912567","heartbeat_front_addr":"172.21.15.164:6813/1539912567","state":["exists","up"]},{"osd":3,"uuid":"48093655-e3b0-45b9-961b-c11d709531a2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6800","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6801","nonce":4161381544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6802","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6803","nonce":4161381544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6806","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6807","nonce":4161381544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6804","nonce":4161381544},{"type":"v1","addr":"172.21.15.165:6805","nonce":4161381544}]},"public_addr":"172.21.15.165:6801/4161381544","cluster_addr":"172.21.15.165:6803/4161381544","heartbeat_back_addr":"172.21.15.165:6807/4161381544","heartbeat_front_addr":"172.21.15.165:6805/4161381544","state":["exists","up"]},{"osd":4,"uuid":"b0070757-0f2c-43d9-9fb7-deadfab0ea10","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6808","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6809","nonce":592433527}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6810","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6811","nonce":592433527}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6814","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6815","nonce":592433527}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6812","nonce":592433527},{"type":"v1","addr":"172.21.15.165:6813","nonce":592433527}]},"public_addr":"172.21.15.165:6809/592433527","cluster_addr":"172.21.15.165:6811/592433527","heartbeat_back_addr":"172.21.15.165:6815/592433527","heartbeat_front_addr":"172.21.15.165:6813/592433527","state":["exists","up"]},{"osd":5,"uuid":"8de7e69d-412c-4983-912b-baa09f77d37c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6816","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6817","nonce":3196812989}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6818","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6819","nonce":3196812989}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6822","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6823","nonce":3196812989}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.165:6820","nonce":3196812989},{"type":"v1","addr":"172.21.15.165:6821","nonce":3196812989}]},"public_addr":"172.21.15.165:6817/3196812989","cluster_addr":"172.21.15.165:6819/3196812989","heartbeat_back_addr":"172.21.15.165:6823/3196812989","heartbeat_front_addr":"172.21.15.165:6821/3196812989","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.666056+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.366169+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.822367+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:11.333174+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.514984+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T16:06:12.335416+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":442,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T16:32:15.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.506+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 msgr2=0x7f65100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.506+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 0x7f65100406d0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f651c006fd0 tx=0x7f651c008040 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 10 monclient: shutdown 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 msgr2=0x7f652c12e300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f6528000c00 tx=0x7f65280046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 shutdown_connections 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f651003e210 0x7f65100406d0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f652c0a4c20 0x7f652c1cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f652c06d540 0x7f652c12e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 --2- 172.21.15.164:0/1161053062 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f652c06d160 0x7f652c12e300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 >> 172.21.15.164:0/1161053062 conn(0x7f652c0aacb0 msgr2=0x7f652c132730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 shutdown_connections 2024-09-10T16:32:15.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:15.507+0000 7f65343ad640 1 -- 172.21.15.164:0/1161053062 wait complete. 2024-09-10T16:32:15.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2591350636}) 2024-09-10T16:32:15.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3148532559}) 2024-09-10T16:32:15.532 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2591350636}) 2024-09-10T16:32:15.533 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.530+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=151950769}) 2024-09-10T16:32:15.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.691 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202128736}) 2024-09-10T16:32:15.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1777945106}) 2024-09-10T16:32:15.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:15.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202128736}) 2024-09-10T16:32:15.692 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:15.689+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=981430504}) 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438934863}) 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=466469393}) 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1438934863}) 2024-09-10T16:32:16.041 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.039+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3056852566}) 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603482112}) 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3710128768}) 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3603482112}) 2024-09-10T16:32:16.042 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.040+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2696290415}) 2024-09-10T16:32:16.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.043 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734710750}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=204951764}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1734710750}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.041+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2077917545}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217238470}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=325243501}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2217238470}) 2024-09-10T16:32:16.044 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.042+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=963525906}) 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200012951}) 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4149337685}) 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3200012951}) 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=703825680}) 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.048 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410530164}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2092150450}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=410530164}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.043+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=287822805}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594058666}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4087285850}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=594058666}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.044+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2078114646}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889270118}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=761693258}) 2024-09-10T16:32:16.049 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3889270118}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.045+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=20579291}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013059356}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2107133271}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1013059356}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=528913903}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882636430}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2969790670}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2882636430}) 2024-09-10T16:32:16.050 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.046+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3210112469}) 2024-09-10T16:32:16.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642853704}) 2024-09-10T16:32:16.057 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3089779905}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2642853704}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.047+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1776505635}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376028631}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2483166905}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2376028631}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.048+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3070940409}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470494620}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1182192806}) 2024-09-10T16:32:16.058 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3470494620}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=535568000}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690049834}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1632581286}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2690049834}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.049+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1147091257}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398573376}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3570878918}) 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.059 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2398573376}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.050+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=331257754}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212893686}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2930244615}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=212893686}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.051+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4078444437}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1715628780}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=6633110}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1715628780}) 2024-09-10T16:32:16.060 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=403652142}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025856220}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=344785511}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1025856220}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.052+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3749783993}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494104419}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1924793463}) 2024-09-10T16:32:16.061 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3494104419}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.053+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2233421605}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2099189083}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3239427591}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2099189083}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.054+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4134234624}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747709120}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3722534044}) 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.062 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1747709120}) 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.055+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=831567596}) 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565032210}) 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3711626250}) 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=565032210}) 2024-09-10T16:32:16.063 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=885026463}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561730734}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3463495114}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=561730734}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.056+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2689936799}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240480516}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=527905666}) 2024-09-10T16:32:16.066 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1240480516}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.057+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2442061953}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353666330}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=869771112}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3353666330}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.058+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4020597606}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552404397}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3602351324}) 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.067 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552404397}) 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2850027351}) 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304561198}) 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1466233310}) 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2304561198}) 2024-09-10T16:32:16.068 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.059+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225840796}) 2024-09-10T16:32:16.129 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983670397}) 2024-09-10T16:32:16.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=491787266}) 2024-09-10T16:32:16.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3983670397}) 2024-09-10T16:32:16.130 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.128+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3465643548}) 2024-09-10T16:32:16.131 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605539458}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4190474367}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3605539458}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=865771994}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949361743}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=343773526}) 2024-09-10T16:32:16.132 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.129+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.133 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.130+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2949361743}) 2024-09-10T16:32:16.133 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.130+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3373570643}) 2024-09-10T16:32:16.133 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689237714}) 2024-09-10T16:32:16.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3280546039}) 2024-09-10T16:32:16.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3689237714}) 2024-09-10T16:32:16.134 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.131+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909195493}) 2024-09-10T16:32:16.135 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204372903}) 2024-09-10T16:32:16.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1533566076}) 2024-09-10T16:32:16.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1204372903}) 2024-09-10T16:32:16.136 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.134+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1182565635}) 2024-09-10T16:32:16.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586196966}) 2024-09-10T16:32:16.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1810308075}) 2024-09-10T16:32:16.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.142 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586196966}) 2024-09-10T16:32:16.143 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=320817662}) 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=641105221}) 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753174823}) 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=641105221}) 2024-09-10T16:32:16.146 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.141+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1506150324}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723633489}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173103431}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=723633489}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1997335429}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860683424}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1821486655}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860683424}) 2024-09-10T16:32:16.147 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.142+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2347132429}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951440020}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4092355114}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2951440020}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.143+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3382493102}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635005364}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=533285024}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3635005364}) 2024-09-10T16:32:16.148 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2558419907}) 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3098968855,mdsmap=931935893,mgr=2907349437}) 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=789543315,mdsmap=931935893,mgr=2907349437}) 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=90,mdsmap=3,mgr=7} crc {logm=3098968855,mdsmap=931935893,mgr=2907349437}) 2024-09-10T16:32:16.149 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.144+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=3491827503,mdsmap=931935893,mgr=3162150554}) 2024-09-10T16:32:16.150 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.150 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgrstat=2744719555,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1301567229,osd_snap=3814312392}) 2024-09-10T16:32:16.150 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=2744719555,mon_config_key=157930937,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186996,osd_pg_creating=1301567229,osd_snap=1221915054}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr_command_descs=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=60} crc {mgr_command_descs=3192186153,mgrstat=2744719555,mon_config_key=1413546875,monmap=1433040829,nvmeofgw=598337686,osd_metadata=4005186995,osd_pg_creating=1301567229,osd_snap=3814312392}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.145+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=5,osd_pg_creating=1,osd_snap=63} crc {mgr_command_descs=3192186153,mgr_metadata=2169216343,mgrstat=3268966337,mon_config_key=1698199361,monmap=1433040829,nvmeofgw=3140514009,osd_metadata=3297165161,osd_pg_creating=1301567229,osd_snap=840587829}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3902583033}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=100} crc {osd_snap=307136989}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=100} crc {osd_snap=3902583033}) 2024-09-10T16:32:16.151 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.146+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=100} crc {osd_snap=2032401087}) 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=61} crc {osd_snap=1340070008,osdmap=3954530601}) 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=17,osdmap=57} crc {osd_snap=1864356900,osdmap=2921763931}) 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=16,osdmap=61} crc {osd_snap=1340070008,osdmap=3954530601}) 2024-09-10T16:32:16.153 INFO:tasks.ceph.mon.a.smithi164.stderr:2024-09-10T16:32:16.151+0000 7f7cf7592640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=17,osdmap=60} crc {osd_snap=1492766152,osdmap=3920825653}) 2024-09-10T16:32:16.520 INFO:tasks.ceph:Scrubbing osd.0 2024-09-10T16:32:16.520 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:16.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.634+0000 7f9150071640 10 monclient: get_monmap_and_config 2024-09-10T16:32:16.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.634+0000 7f9150071640 10 monclient: build_initial_monmap 2024-09-10T16:32:16.635 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.634+0000 7f9150071640 10 monclient: monmap: 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:16.635123+0000 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:16.635123+0000 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:16.636 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.634+0000 7f9150071640 1 Processor -- start 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 1 -- start start 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient: init 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f91480a9800 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f91480a4cd0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9148134ae0 0x7f9148140630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): picked mon.noname-c con 0x7f9148134ae0 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9148135130 con 0x7f91480a9800 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f914809ca90 con 0x7f91480a4cd0 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f91480a9be0 con 0x7f9148134ae0 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f9150071640 10 monclient(hunting): authenticate will time out at 2315.122691s 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f914de0d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f91480a9800 0x7f91480a45c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49646/0 (socket says 172.21.15.164:49646) 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.635+0000 7f914de0d640 1 -- 172.21.15.164:0/861445307 learned_addr learned my addr 172.21.15.164:0/861445307 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 1 --2- 172.21.15.164:0/861445307 >> v2:172.21.15.165:3300/0 conn(0x7f9148134ae0 0x7f9148140630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): get_auth_request con 0x7f9148134ae0 auth_method 0 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient(hunting): handle_auth_done global_id 284236 payload 274 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 1 -- 172.21.15.164:0/861445307 >> v1:172.21.15.165:6789/0 conn(0x7f91480a4cd0 legacy=0x7f914813cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 1 -- 172.21.15.164:0/861445307 >> v1:172.21.15.164:6789/0 conn(0x7f91480a9800 legacy=0x7f91480a45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient: _finish_hunting 0 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 1 monclient: found mon.noname-c 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:32:16.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.636+0000 7f914e60e640 1 -- 172.21.15.164:0/861445307 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f914809cd70 con 0x7f9148134ae0 2024-09-10T16:32:16.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914e60e640 1 -- 172.21.15.164:0/861445307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9148134ae0 msgr2=0x7f9148140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:16.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914e60e640 1 --2- 172.21.15.164:0/861445307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9148134ae0 0x7f9148140630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9140009d90 tx=0x7f914002a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=405922ab3682cf08 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 1 -- 172.21.15.164:0/861445307 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140007a30 con 0x7f9148134ae0 2024-09-10T16:32:16.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: dump: 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: _finish_auth 0 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: _check_auth_tickets 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 1 -- 172.21.15.164:0/861445307 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9140007bd0 con 0x7f9148134ae0 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 1 -- 172.21.15.164:0/861445307 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140031400 con 0x7f9148134ae0 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f914ce0b640 10 monclient: dump: 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.649 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f9150071640 5 monclient: authenticate success, global_id 284236 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f9150071640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.639+0000 7f9150071640 4 set_mon_vals no callback set 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 10 monclient: shutdown 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 -- 172.21.15.164:0/861445307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9148134ae0 msgr2=0x7f9148140630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 --2- 172.21.15.164:0/861445307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9148134ae0 0x7f9148140630 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f9140009d90 tx=0x7f914002a490 comp rx=0 tx=0).stop 2024-09-10T16:32:16.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 -- 172.21.15.164:0/861445307 shutdown_connections 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 --2- 172.21.15.164:0/861445307 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9148134ae0 0x7f9148140630 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 -- 172.21.15.164:0/861445307 >> 172.21.15.164:0/861445307 conn(0x7f91480aab70 msgr2=0x7f91480aaf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 -- 172.21.15.164:0/861445307 shutdown_connections 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 1 -- 172.21.15.164:0/861445307 wait complete. 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 10 monclient: build_initial_monmap 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.640+0000 7f9150071640 10 monclient: monmap: 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.651 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 Processor -- start 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 -- start start 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient: init 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91480a4cd0 0x7f91481316a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): picked mon.noname-b con 0x7f91480a4cd0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 0x7f9148131be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): picked mon.noname-a con 0x7f91480a9800 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:16.652 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 0x7f9148136620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): picked mon.noname-c con 0x7f9148134ae0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9148140cf0 con 0x7f91480a9800 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9148140b70 con 0x7f91480a4cd0 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9148140e70 con 0x7f9148134ae0 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f9150071640 10 monclient(hunting): authenticate will time out at 2315.128593s 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 0x7f9148131be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 0x7f9148136620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 0x7f9148131be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47784/0 (socket says 172.21.15.164:47784) 2024-09-10T16:32:16.653 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 1 -- 172.21.15.164:0/2037288564 learned_addr learned my addr 172.21.15.164:0/2037288564 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): get_auth_request con 0x7f91480a9800 auth_method 0 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): get_auth_request con 0x7f9148134ae0 auth_method 0 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914d60c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.654 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914e60e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914de0d640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91480a4cd0 0x7f91481316a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914de0d640 10 monclient(hunting): get_auth_request con 0x7f91480a4cd0 auth_method 0 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914de0d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914de0d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.641+0000 7f914de0d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 10 monclient(hunting): handle_auth_done global_id 274331 payload 824 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91480a4cd0 msgr2=0x7f91481316a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f91480a4cd0 0x7f91481316a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 msgr2=0x7f9148131be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 0x7f9148131be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 10 monclient: _finish_hunting 0 2024-09-10T16:32:16.655 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 monclient: found mon.noname-c 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914e60e640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9148138090 con 0x7f9148134ae0 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.642+0000 7f914d60c640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f91480a9800 0x7f9148131be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f914e60e640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 msgr2=0x7f9148136620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:16.656 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f914e60e640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 0x7f9148136620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9140031000 tx=0x7f9140004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9140040070 con 0x7f9148134ae0 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: dump: 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.657 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9140045660 con 0x7f9148134ae0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 5 monclient: authenticate success, global_id 274331 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 10 monclient: _renew_subs 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f914813a3f0 con 0x7f9148134ae0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f91400314c0 con 0x7f9148134ae0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.658 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913e7fc640 10 monclient: dump: 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 10 monclient: _renew_subs 2024-09-10T16:32:16.659 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f913ffff640 4 set_mon_vals no callback set 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.644+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9148138bf0 con 0x7f9148134ae0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.645+0000 7f9150071640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.645+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91480a45c0 con 0x7f9148134ae0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.645+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9140039040 con 0x7f9148134ae0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.645+0000 7f913e7fc640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f912c03e210 0x7f912c0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.645+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f9140074540 con 0x7f9148134ae0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f913e7fc640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f912c044010 0x7f912c046430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f912c046b00 con 0x7f912c044010 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (secure 0 0 0) 0x7f914007b050 con 0x7f9148134ae0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f913e7fc640 10 monclient: handle_get_version_reply finishing 1 version 897 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f914de0d640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f912c03e210 0x7f912c0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.660 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f914d60c640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f912c044010 0x7f912c046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f914de0d640 10 monclient: get_auth_request con 0x7f912c03e210 auth_method 0 2024-09-10T16:32:16.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.646+0000 7f914d60c640 10 monclient: get_auth_request con 0x7f912c044010 auth_method 0 2024-09-10T16:32:16.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.648+0000 7f914de0d640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f912c03e210 0x7f912c0406d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f914400c7f0 tx=0x7f9144009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.661 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.650+0000 7f914d60c640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f912c044010 0x7f912c046430 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.662 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.651+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== osd.0 v2:172.21.15.164:6800/3366597746 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f912c046b00 con 0x7f912c044010 2024-09-10T16:32:16.664 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.663+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 --> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f9148133aa0 con 0x7f912c044010 2024-09-10T16:32:16.665 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.663+0000 7f913e7fc640 1 -- 172.21.15.164:0/2037288564 <== osd.0 v2:172.21.15.164:6800/3366597746 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f9148133aa0 con 0x7f912c044010 2024-09-10T16:32:16.665 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:16.665 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:16.665 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f912c044010 msgr2=0x7f912c046430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:6800/3366597746,v1:172.21.15.164:6801/3366597746] conn(0x7f912c044010 0x7f912c046430 crc :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f912c03e210 msgr2=0x7f912c0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f912c03e210 0x7f912c0406d0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f914400c7f0 tx=0x7f9144009040 comp rx=0 tx=0).stop 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 10 monclient: shutdown 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 msgr2=0x7f9148136620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 --2- 172.21.15.164:0/2037288564 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9148134ae0 0x7f9148136620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9140031000 tx=0x7f9140004a40 comp rx=0 tx=0).stop 2024-09-10T16:32:16.666 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f914e60e640 1 -- 172.21.15.164:0/2037288564 reap_dead start 2024-09-10T16:32:16.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 shutdown_connections 2024-09-10T16:32:16.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 >> 172.21.15.164:0/2037288564 conn(0x7f91480aab70 msgr2=0x7f9148133390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:16.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 shutdown_connections 2024-09-10T16:32:16.667 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.664+0000 7f9150071640 1 -- 172.21.15.164:0/2037288564 wait complete. 2024-09-10T16:32:16.675 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 10 monclient: get_monmap_and_config 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 10 monclient: build_initial_monmap 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 10 monclient: monmap: 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.774 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:16.773925+0000 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:16.773925+0000 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:16.775 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 1 Processor -- start 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 1 -- start start 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.773+0000 7f2535b16640 10 monclient: init 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-b con 0x7f253006d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-a con 0x7f253006d540 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f25300a4c20 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253009cb90 con 0x7f25300a4c20 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f253009ce70 con 0x7f253006d540 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f253009d150 con 0x7f253006d160 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2535b16640 10 monclient(hunting): authenticate will time out at 2315.261427s 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32940/0 (socket says 172.21.15.164:32940) 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47792/0 (socket says 172.21.15.164:47792) 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 -- 172.21.15.164:0/817543873 learned_addr learned my addr 172.21.15.164:0/817543873 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): get_auth_request con 0x7f253006d160 auth_method 0 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): get_auth_request con 0x7f253006d540 auth_method 0 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient(hunting): handle_auth_done global_id 274334 payload 274 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 -- 172.21.15.164:0/817543873 >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 msgr2=0x7f25300a46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 --2- 172.21.15.164:0/817543873 >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 -- 172.21.15.164:0/817543873 >> v1:172.21.15.165:6789/0 conn(0x7f25300a4c20 legacy=0x7f2530137ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 monclient: found mon.noname-b 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f252f7fe640 1 -- 172.21.15.164:0/817543873 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2530135340 con 0x7f253006d160 2024-09-10T16:32:16.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.774+0000 7f2526ffd640 1 --2- 172.21.15.164:0/817543873 >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252f7fe640 1 -- 172.21.15.164:0/817543873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 msgr2=0x7f2530134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252f7fe640 1 --2- 172.21.15.164:0/817543873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2520009870 tx=0x7f252002a0a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78777239f1fc7694 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 1 -- 172.21.15.164:0/817543873 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520037070 con 0x7f253006d160 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: dump: 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.787 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: _finish_auth 0 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f2535b16640 5 monclient: authenticate success, global_id 274334 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f2535b16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 1 -- 172.21.15.164:0/817543873 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2520032440 con 0x7f253006d160 2024-09-10T16:32:16.788 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f2535b16640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f2535b16640 4 set_mon_vals no callback set 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 1 -- 172.21.15.164:0/817543873 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2520032810 con 0x7f253006d160 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.777+0000 7f252effd640 10 monclient: dump: 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.789 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 10 monclient: shutdown 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 -- 172.21.15.164:0/817543873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 msgr2=0x7f2530134e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 --2- 172.21.15.164:0/817543873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f2520009870 tx=0x7f252002a0a0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 -- 172.21.15.164:0/817543873 shutdown_connections 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 --2- 172.21.15.164:0/817543873 >> v2:172.21.15.164:3300/0 conn(0x7f253006d540 0x7f25300a46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 --2- 172.21.15.164:0/817543873 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d160 0x7f2530134e00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 -- 172.21.15.164:0/817543873 >> 172.21.15.164:0/817543873 conn(0x7f25300aacb0 msgr2=0x7f25300ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 -- 172.21.15.164:0/817543873 shutdown_connections 2024-09-10T16:32:16.790 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 1 -- 172.21.15.164:0/817543873 wait complete. 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 10 monclient: build_initial_monmap 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.778+0000 7f2535b16640 10 monclient: monmap: 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 Processor -- start 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 -- start start 2024-09-10T16:32:16.791 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient: init 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f253006d160 0x7f25301cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-b con 0x7f253006d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-c con 0x7f253006d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25300a4c20 0x7f253013dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): picked mon.noname-a con 0x7f25300a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f2530138830 con 0x7f25300a4c20 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.792 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f25301386b0 con 0x7f253006d160 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f25301389b0 con 0x7f253006d540 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2535b16640 10 monclient(hunting): authenticate will time out at 2315.266906s 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.779+0000 7f2526ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32952/0 (socket says 172.21.15.164:32952) 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 -- 172.21.15.164:0/4269768335 learned_addr learned my addr 172.21.15.164:0/4269768335 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): get_auth_request con 0x7f253006d540 auth_method 0 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25300a4c20 0x7f253013dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.793 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): get_auth_request con 0x7f25300a4c20 auth_method 0 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient(hunting): handle_auth_done global_id 274337 payload 824 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f252f7fe640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f253006d160 0x7f25301cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 -- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f253006d160 msgr2=0x7f25301cd280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f253006d160 0x7f25301cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.794 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 -- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25300a4c20 msgr2=0x7f253013dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25300a4c20 0x7f253013dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 monclient: found mon.noname-c 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.780+0000 7f2526ffd640 1 -- 172.21.15.164:0/4269768335 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25300a44a0 con 0x7f253006d540 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2526ffd640 1 -- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 msgr2=0x7f25300a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2526ffd640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f251800ebb0 tx=0x7f251800c520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518017070 con 0x7f253006d540 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: dump: 2024-09-10T16:32:16.795 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: _finish_auth 0 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: _check_auth_tickets 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f251800ce40 con 0x7f253006d540 2024-09-10T16:32:16.796 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 5 monclient: authenticate success, global_id 274337 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 10 monclient: _renew_subs 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2518013670 con 0x7f253006d540 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253006e6b0 con 0x7f253006d540 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252cff9640 10 monclient: dump: 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:16.797 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f252e7fc640 4 set_mon_vals no callback set 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 10 monclient: _renew_subs 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.782+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f253009f4d0 con 0x7f253006d540 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.783+0000 7f2535b16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.783+0000 7f2535b16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.783+0000 7f2535b16640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.783+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2530134e00 con 0x7f253006d540 2024-09-10T16:32:16.798 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.783+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f251801a070 con 0x7f253006d540 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.784+0000 7f252cff9640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 0x7f25100406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.784+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f2518052b70 con 0x7f253006d540 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.784+0000 7f252f7fe640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 0x7f25100406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.784+0000 7f252f7fe640 10 monclient: get_auth_request con 0x7f251003e210 auth_method 0 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.786+0000 7f252f7fe640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 0x7f25100406d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f25300773f0 tx=0x7f252002e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.788+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f2518051700 con 0x7f253006d540 2024-09-10T16:32:16.799 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.788+0000 7f252cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:16.800 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.788+0000 7f252cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:16.965 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.964+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f25300aa1e0 con 0x7f251003e210 2024-09-10T16:32:16.966 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.964+0000 7f252cff9640 1 -- 172.21.15.164:0/4269768335 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f25300aa1e0 con 0x7f251003e210 2024-09-10T16:32:16.966 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 0 to deep-scrub 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 msgr2=0x7f25100406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 0x7f25100406d0 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f25300773f0 tx=0x7f252002e000 comp rx=0 tx=0).stop 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 10 monclient: shutdown 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 msgr2=0x7f25300a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f251800ebb0 tx=0x7f251800c520 comp rx=0 tx=0).stop 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 shutdown_connections 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f251003e210 0x7f25100406d0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f25300a4c20 0x7f253013dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.969 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f253006d540 0x7f25300a3a30 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 --2- 172.21.15.164:0/4269768335 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f253006d160 0x7f25301cd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:16.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 >> 172.21.15.164:0/4269768335 conn(0x7f25300aacb0 msgr2=0x7f2530136c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:16.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 shutdown_connections 2024-09-10T16:32:16.970 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:16.967+0000 7f2535b16640 1 -- 172.21.15.164:0/4269768335 wait complete. 2024-09-10T16:32:16.979 INFO:tasks.ceph:Scrubbing osd.1 2024-09-10T16:32:16.979 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.079+0000 7f3b4ca52640 10 monclient: get_monmap_and_config 2024-09-10T16:32:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.079+0000 7f3b4ca52640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.079+0000 7f3b4ca52640 10 monclient: monmap: 2024-09-10T16:32:17.083 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:17.080445+0000 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:17.080445+0000 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 Processor -- start 2024-09-10T16:32:17.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 -- start start 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient: init 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient: _add_conns ranks=[3,0,2,4,1] 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b4809c9f0 0x7f3b48135ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-a con 0x7f3b4809c9f0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b4809cdd0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-b con 0x7f3b4813b040 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b480a1af0 con 0x7f3b4809cdd0 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f3b4809dc00 con 0x7f3b4809c9f0 2024-09-10T16:32:17.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3b480a1dd0 con 0x7f3b4813b040 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4ca52640 10 monclient(hunting): authenticate will time out at 2315.567834s 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b4809c9f0 0x7f3b48135ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:32954/0 (socket says 172.21.15.164:32954) 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b4659c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f3b4809c9f0 0x7f3b48135ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47802/0 (socket says 172.21.15.164:47802) 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b45d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f3b4809cdd0 0x7f3b4809d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49660/0 (socket says 172.21.15.164:49660) 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 1 -- 172.21.15.164:0/4046352126 learned_addr learned my addr 172.21.15.164:0/4046352126 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 10 monclient(hunting): get_auth_request con 0x7f3b4813b040 auth_method 0 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.080+0000 7f3b46d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): get_auth_request con 0x7f3b4809c9f0 auth_method 0 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b46d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b46d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b46d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient(hunting): handle_auth_done global_id 294174 payload 274 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 1 -- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 msgr2=0x7f3b4813d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 1 -- 172.21.15.164:0/4046352126 >> v1:172.21.15.164:6789/0 conn(0x7f3b4809cdd0 legacy=0x7f3b4809d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 1 monclient: found mon.noname-a 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b4659c640 1 -- 172.21.15.164:0/4046352126 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b48136530 con 0x7f3b4809c9f0 2024-09-10T16:32:17.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.081+0000 7f3b46d9d640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.083+0000 7f3b4659c640 1 -- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 msgr2=0x7f3b48135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.083+0000 7f3b4659c640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b48135ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3b380097a0 tx=0x7f3b38029fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c142e3e217f5dc0e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 1 -- 172.21.15.164:0/4046352126 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38037070 con 0x7f3b4809c9f0 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:17.094 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: dump: 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.095 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 1 -- 172.21.15.164:0/4046352126 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b38032440 con 0x7f3b4809c9f0 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 5 monclient: authenticate success, global_id 294174 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 1 -- 172.21.15.164:0/4046352126 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38032810 con 0x7f3b4809c9f0 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 4 set_mon_vals no callback set 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.096 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4559a640 10 monclient: dump: 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.097 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 10 monclient: shutdown 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 -- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 msgr2=0x7f3b48135ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b48135ff0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3b380097a0 tx=0x7f3b38029fd0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 -- 172.21.15.164:0/4046352126 shutdown_connections 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4813b040 0x7f3b4813d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/4046352126 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b48135ff0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 -- 172.21.15.164:0/4046352126 >> 172.21.15.164:0/4046352126 conn(0x7f3b480a7050 msgr2=0x7f3b480a7460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 -- 172.21.15.164:0/4046352126 shutdown_connections 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.084+0000 7f3b4ca52640 1 -- 172.21.15.164:0/4046352126 wait complete. 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.085+0000 7f3b4ca52640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.098 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.085+0000 7f3b4ca52640 10 monclient: monmap: 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.085+0000 7f3b4ca52640 1 Processor -- start 2024-09-10T16:32:17.099 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.085+0000 7f3b4ca52640 1 -- start start 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.085+0000 7f3b4ca52640 10 monclient: init 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-a con 0x7f3b4809c9f0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4809cdd0 0x7f3b4809e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-c con 0x7f3b4809cdd0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b4813b040 0x7f3b4809eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): picked mon.noname-b con 0x7f3b4813b040 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:17.100 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4813daf0 con 0x7f3b4809c9f0 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f3b4813d970 con 0x7f3b4813b040 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f3b4813dc70 con 0x7f3b4809cdd0 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4ca52640 10 monclient(hunting): authenticate will time out at 2315.573389s 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47812/0 (socket says 172.21.15.164:47812) 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 -- 172.21.15.164:0/3417828526 learned_addr learned my addr 172.21.15.164:0/3417828526 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:17.101 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): get_auth_request con 0x7f3b4809c9f0 auth_method 0 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4809cdd0 0x7f3b4809e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): get_auth_request con 0x7f3b4809cdd0 auth_method 0 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.102 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b45d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b46d9d640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b4813b040 0x7f3b4809eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient(hunting): handle_auth_done global_id 294177 payload 824 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4809cdd0 msgr2=0x7f3b4809e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f3b4809cdd0 0x7f3b4809e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b4813b040 msgr2=0x7f3b4809eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b4813b040 0x7f3b4809eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.103 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 monclient: found mon.noname-a 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b4659c640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b480728f0 con 0x7f3b4809c9f0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b46d9d640 10 monclient: get_auth_request con 0x7f3b4813b040 auth_method 0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.086+0000 7f3b46d9d640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f3b4813b040 0x7f3b4809eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.088+0000 7f3b4659c640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 msgr2=0x7f3b481333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.088+0000 7f3b4659c640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b38000c00 tx=0x7f3b380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b38037040 con 0x7f3b4809c9f0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:17.104 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: dump: 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.105 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b38042070 con 0x7f3b4809c9f0 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 5 monclient: authenticate success, global_id 294177 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 10 monclient: _renew_subs 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b480a5fd0 con 0x7f3b4809c9f0 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b3803bb00 con 0x7f3b4809c9f0 2024-09-10T16:32:17.106 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b36ffd640 10 monclient: dump: 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.107 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b44d99640 4 set_mon_vals no callback set 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 10 monclient: _renew_subs 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.089+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b4809e240 con 0x7f3b4809c9f0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b4ca52640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3b480a42a0 con 0x7f3b4809c9f0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f3b38047460 con 0x7f3b4809c9f0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b4659c640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b38000c00 tx=0x7f3b380046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f3b3807a050 mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b36ffd640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b2803e210 0x7f3b280406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f3b38073400 con 0x7f3b4809c9f0 2024-09-10T16:32:17.108 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b36ffd640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7f3b28044010 0x7f3b28046430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b3804de50 con 0x7f3b28044010 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b46d9d640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7f3b28044010 0x7f3b28046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b46d9d640 10 monclient: get_auth_request con 0x7f3b28044010 auth_method 0 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b45d9b640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b2803e210 0x7f3b280406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.090+0000 7f3b45d9b640 10 monclient: get_auth_request con 0x7f3b2803e210 auth_method 0 2024-09-10T16:32:17.109 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.093+0000 7f3b45d9b640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b2803e210 0x7f3b280406d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005fd0 tx=0x7f3b3c007e60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.095+0000 7f3b46d9d640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7f3b28044010 0x7f3b28046430 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.095+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== osd.1 v2:172.21.15.164:6816/1609205011 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3b3804de50 con 0x7f3b28044010 2024-09-10T16:32:17.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 --> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3b48135ff0 con 0x7f3b28044010 2024-09-10T16:32:17.110 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== osd.1 v2:172.21.15.164:6816/1609205011 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3b48135ff0 con 0x7f3b28044010 2024-09-10T16:32:17.110 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:17.111 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:17.111 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7f3b28044010 msgr2=0x7f3b28046430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:6816/1609205011,v1:172.21.15.164:6817/1609205011] conn(0x7f3b28044010 0x7f3b28046430 crc :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b2803e210 msgr2=0x7f3b280406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f3b2803e210 0x7f3b280406d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f3b3c005fd0 tx=0x7f3b3c007e60 comp rx=0 tx=0).stop 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 10 monclient: shutdown 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 20 monclient: shutdown canceling and discarding version request 1 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.112 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 msgr2=0x7f3b481333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 --2- 172.21.15.164:0/3417828526 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f3b4809c9f0 0x7f3b481333b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3b38000c00 tx=0x7f3b380046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b46d9d640 1 -- 172.21.15.164:0/3417828526 reap_dead start 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b36ffd640 1 -- 172.21.15.164:0/3417828526 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (unknown 0 0 0) 0x7f3b3807a050 con 0x7f3b4809c9f0 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b36ffd640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 shutdown_connections 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 >> 172.21.15.164:0/3417828526 conn(0x7f3b480a7050 msgr2=0x7f3b481343c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 shutdown_connections 2024-09-10T16:32:17.113 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.108+0000 7f3b4ca52640 1 -- 172.21.15.164:0/3417828526 wait complete. 2024-09-10T16:32:17.120 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-10T16:32:17.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 10 monclient: get_monmap_and_config 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 10 monclient: monmap: 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:17.219166+0000 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:17.219166+0000 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:17.220 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:17.221 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:17.221 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 1 Processor -- start 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 1 -- start start 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.218+0000 7f19749f3640 10 monclient: init 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f197006cb70 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1970134f70 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f197013e740 0x7f1970140b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-c con 0x7f197013e740 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:17.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f197009ce50 con 0x7f1970134f70 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1970135380 con 0x7f197006cb70 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f197006db60 con 0x7f197013e740 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f19749f3640 10 monclient(hunting): authenticate will time out at 2315.706706s 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196dd9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f1970134f70 0x7f197013d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49670/0 (socket says 172.21.15.164:49670) 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196dd9b640 1 -- 172.21.15.164:0/2196262510 learned_addr learned my addr 172.21.15.164:0/2196262510 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196ed9d640 1 --2- 172.21.15.164:0/2196262510 >> v2:172.21.15.165:3300/0 conn(0x7f197013e740 0x7f1970140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196ed9d640 10 monclient(hunting): get_auth_request con 0x7f197013e740 auth_method 0 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.219+0000 7f196ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient(hunting): handle_auth_done global_id 284242 payload 274 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 1 -- 172.21.15.164:0/2196262510 >> v1:172.21.15.165:6789/0 conn(0x7f197006cb70 legacy=0x7f19700a46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 1 -- 172.21.15.164:0/2196262510 >> v1:172.21.15.164:6789/0 conn(0x7f1970134f70 legacy=0x7f197013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 1 monclient: found mon.noname-c 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:32:17.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.220+0000 7f196ed9d640 1 -- 172.21.15.164:0/2196262510 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197009cb70 con 0x7f197013e740 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196ed9d640 1 -- 172.21.15.164:0/2196262510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 msgr2=0x7f1970140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196ed9d640 1 --2- 172.21.15.164:0/2196262510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970140b30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f19600097d0 tx=0x7f196002a000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65e6f99b26802893 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 1 -- 172.21.15.164:0/2196262510 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960037070 con 0x7f197013e740 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: dump: 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.232 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.233 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 1 -- 172.21.15.164:0/2196262510 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1960032440 con 0x7f197013e740 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 1 -- 172.21.15.164:0/2196262510 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1960032810 con 0x7f197013e740 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f196d59a640 10 monclient: dump: 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.234 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 5 monclient: authenticate success, global_id 284242 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 4 set_mon_vals no callback set 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 10 monclient: shutdown 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 1 -- 172.21.15.164:0/2196262510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 msgr2=0x7f1970140b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.235 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.223+0000 7f19749f3640 1 --2- 172.21.15.164:0/2196262510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970140b30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f19600097d0 tx=0x7f196002a000 comp rx=0 tx=0).stop 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 1 -- 172.21.15.164:0/2196262510 shutdown_connections 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 1 --2- 172.21.15.164:0/2196262510 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970140b30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 1 -- 172.21.15.164:0/2196262510 >> 172.21.15.164:0/2196262510 conn(0x7f19700aa900 msgr2=0x7f19700aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 1 -- 172.21.15.164:0/2196262510 shutdown_connections 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 1 -- 172.21.15.164:0/2196262510 wait complete. 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.224+0000 7f19749f3640 10 monclient: monmap: 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.236 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 Processor -- start 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 -- start start 2024-09-10T16:32:17.237 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient: init 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-a con 0x7f197006cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 0x7f197013d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-c con 0x7f1970134f70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970130180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): picked mon.noname-b con 0x7f197013e740 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f19700a4db0 con 0x7f197006cb70 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.238 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f197006cf50 con 0x7f197013e740 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f1970141070 con 0x7f1970134f70 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f19749f3640 10 monclient(hunting): authenticate will time out at 2315.712538s 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 0x7f197013d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47826/0 (socket says 172.21.15.164:47826) 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 1 -- 172.21.15.164:0/1964835608 learned_addr learned my addr 172.21.15.164:0/1964835608 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): get_auth_request con 0x7f197006cb70 auth_method 0 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.239 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): get_auth_request con 0x7f1970134f70 auth_method 0 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196ed9d640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970130180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient(hunting): handle_auth_done global_id 274346 payload 824 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 -- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 msgr2=0x7f1970130180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970130180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.240 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 -- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 msgr2=0x7f197013cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 monclient: found mon.noname-c 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196dd9b640 1 -- 172.21.15.164:0/1964835608 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197013a110 con 0x7f1970134f70 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.225+0000 7f196e59c640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f196dd9b640 1 -- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 msgr2=0x7f197013d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f196dd9b640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 0x7f197013d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f195800c3e0 tx=0x7f195800c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1958017070 con 0x7f1970134f70 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: dump: 2024-09-10T16:32:17.241 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19580070d0 con 0x7f1970134f70 2024-09-10T16:32:17.242 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 5 monclient: authenticate success, global_id 274346 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 10 monclient: _renew_subs 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197013a450 con 0x7f1970134f70 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19580074c0 con 0x7f1970134f70 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f1966ffd640 10 monclient: dump: 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.243 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 10 monclient: _renew_subs 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1970138db0 con 0x7f1970134f70 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.228+0000 7f196cd99640 4 set_mon_vals no callback set 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f195801a070 con 0x7f1970134f70 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f19749f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f19749f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f19749f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.244 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19700a46a0 con 0x7f1970134f70 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f1966ffd640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 0x7f1950040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f1958052000 con 0x7f1970134f70 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f196e59c640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 0x7f1950040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.229+0000 7f196e59c640 10 monclient: get_auth_request con 0x7f195003e240 auth_method 0 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.232+0000 7f196e59c640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 0x7f1950040700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f196800c550 tx=0x7f196800b040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.234+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f197013a450 con 0x7f1970134f70 2024-09-10T16:32:17.245 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.234+0000 7f1966ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.246 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.234+0000 7f1966ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:17.411 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.409+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f197013a630 con 0x7f195003e240 2024-09-10T16:32:17.412 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.410+0000 7f1966ffd640 1 -- 172.21.15.164:0/1964835608 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f19680153f0 con 0x7f195003e240 2024-09-10T16:32:17.412 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 1 to deep-scrub 2024-09-10T16:32:17.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 msgr2=0x7f1950040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 0x7f1950040700 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f196800c550 tx=0x7f196800b040 comp rx=0 tx=0).stop 2024-09-10T16:32:17.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 10 monclient: shutdown 2024-09-10T16:32:17.414 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 msgr2=0x7f197013d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 0x7f197013d330 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f195800c3e0 tx=0x7f195800c8a0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 shutdown_connections 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f195003e240 0x7f1950040700 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f197013e740 0x7f1970130180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f1970134f70 0x7f197013d330 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 --2- 172.21.15.164:0/1964835608 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f197006cb70 0x7f197013cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.412+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 >> 172.21.15.164:0/1964835608 conn(0x7f19700aa900 msgr2=0x7f19700a8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.413+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 shutdown_connections 2024-09-10T16:32:17.415 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.413+0000 7f19749f3640 1 -- 172.21.15.164:0/1964835608 wait complete. 2024-09-10T16:32:17.425 INFO:tasks.ceph:Scrubbing osd.2 2024-09-10T16:32:17.425 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.525+0000 7fa646c95640 10 monclient: get_monmap_and_config 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.525+0000 7fa646c95640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.525+0000 7fa646c95640 10 monclient: monmap: 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:17.526475+0000 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:17.526475+0000 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:17.527 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:17.528 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:17.528 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:17.528 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:17.528 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 Processor -- start 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 -- start start 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient: init 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 0x7fa64009eb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-b con 0x7fa64009c440 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.530 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa64009d270 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa64013a6a0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa64009c820 con 0x7fa64009d270 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6400a09b0 con 0x7fa64013a6a0 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa64009a600 con 0x7fa64009c440 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.526+0000 7fa646c95640 10 monclient(hunting): authenticate will time out at 2316.013944s 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa63ffff640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fa64009d270 0x7fa64009f060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49686/0 (socket says 172.21.15.164:49686) 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 0x7fa64009eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa63ffff640 1 -- 172.21.15.164:0/3882091324 learned_addr learned my addr 172.21.15.164:0/3882091324 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): get_auth_request con 0x7fa64009c440 auth_method 0 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.531 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient(hunting): handle_auth_done global_id 274349 payload 274 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 1 -- 172.21.15.164:0/3882091324 >> v1:172.21.15.165:6789/0 conn(0x7fa64013a6a0 legacy=0x7fa64013ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 1 -- 172.21.15.164:0/3882091324 >> v1:172.21.15.164:6789/0 conn(0x7fa64009d270 legacy=0x7fa64009f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 1 monclient: found mon.noname-b 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.532 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.527+0000 7fa644a31640 1 -- 172.21.15.164:0/3882091324 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa64009a890 con 0x7fa64009c440 2024-09-10T16:32:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa644a31640 1 -- 172.21.15.164:0/3882091324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 msgr2=0x7fa64009eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa644a31640 1 --2- 172.21.15.164:0/3882091324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 0x7fa64009eb20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa630009dc0 tx=0x7fa63002a480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f4180f92def66bf server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 1 -- 172.21.15.164:0/3882091324 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630007a30 con 0x7fa64009c440 2024-09-10T16:32:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.538 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: dump: 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.539 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 1 -- 172.21.15.164:0/3882091324 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa630007bd0 con 0x7fa64009c440 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 5 monclient: authenticate success, global_id 274349 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 1 -- 172.21.15.164:0/3882091324 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa630031400 con 0x7fa64009c440 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 4 set_mon_vals no callback set 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa63f7fe640 10 monclient: dump: 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.540 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 10 monclient: shutdown 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 -- 172.21.15.164:0/3882091324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 msgr2=0x7fa64009eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 --2- 172.21.15.164:0/3882091324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 0x7fa64009eb20 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa630009dc0 tx=0x7fa63002a480 comp rx=0 tx=0).stop 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 -- 172.21.15.164:0/3882091324 shutdown_connections 2024-09-10T16:32:17.541 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 --2- 172.21.15.164:0/3882091324 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009c440 0x7fa64009eb20 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 -- 172.21.15.164:0/3882091324 >> 172.21.15.164:0/3882091324 conn(0x7fa6400a31f0 msgr2=0x7fa6400a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 -- 172.21.15.164:0/3882091324 shutdown_connections 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.530+0000 7fa646c95640 1 -- 172.21.15.164:0/3882091324 wait complete. 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.531+0000 7fa646c95640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.531+0000 7fa646c95640 10 monclient: monmap: 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:17.542 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.531+0000 7fa646c95640 1 Processor -- start 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.531+0000 7fa646c95640 1 -- start start 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.531+0000 7fa646c95640 10 monclient: init 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 0x7fa640091690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-a con 0x7fa64009c440 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 0x7fa640091bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-c con 0x7fa64009d270 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64013a6a0 0x7fa640092110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): picked mon.noname-b con 0x7fa64013a6a0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:17.543 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fa640075100 con 0x7fa64009c440 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fa640074f80 con 0x7fa64013a6a0 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fa64009f770 con 0x7fa64009d270 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa646c95640 10 monclient(hunting): authenticate will time out at 2316.019385s 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 0x7fa640091690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 0x7fa640091690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47842/0 (socket says 172.21.15.164:47842) 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 0x7fa640091bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.544 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 1 -- 172.21.15.164:0/1372546659 learned_addr learned my addr 172.21.15.164:0/1372546659 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): get_auth_request con 0x7fa64009c440 auth_method 0 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): get_auth_request con 0x7fa64009d270 auth_method 0 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.545 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64013a6a0 0x7fa640092110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient(hunting): handle_auth_done global_id 274352 payload 824 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64013a6a0 msgr2=0x7fa640092110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64013a6a0 0x7fa640092110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 msgr2=0x7fa640091690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 0x7fa640091690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 monclient: found mon.noname-c 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa63ffff640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa640095090 con 0x7fa64009d270 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa645232640 10 monclient: get_auth_request con 0x7fa64013a6a0 auth_method 0 2024-09-10T16:32:17.546 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa644a31640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fa64009c440 0x7fa640091690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.532+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fa64013a6a0 0x7fa640092110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.534+0000 7fa63ffff640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 msgr2=0x7fa640091bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.534+0000 7fa63ffff640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 0x7fa640091bd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6400a2a20 tx=0x7fa62800c870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa628017070 con 0x7fa64009d270 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: dump: 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.547 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6280070d0 con 0x7fa64009d270 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 5 monclient: authenticate success, global_id 274352 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 10 monclient: _renew_subs 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa640093b70 con 0x7fa64009d270 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6280074c0 con 0x7fa64009d270 2024-09-10T16:32:17.548 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63d7fa640 10 monclient: dump: 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa63effd640 4 set_mon_vals no callback set 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 10 monclient: _renew_subs 2024-09-10T16:32:17.549 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.535+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa64009d700 con 0x7fa64009d270 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa646c95640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa640093f30 con 0x7fa64009d270 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fa62801a070 con 0x7fa64009d270 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa62003e210 0x7fa6200406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fa628051ff0 con 0x7fa64009d270 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa620046b00 con 0x7fa620044010 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (secure 0 0 0) 0x7fa6280545c0 con 0x7fa64009d270 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa63d7fa640 10 monclient: handle_get_version_reply finishing 1 version 897 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa645232640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 msgr2=0x7fa620046430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T16:32:17.550 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa645232640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 msgr2=0x7fa620046430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T16:32:17.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:17.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:17.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa644a31640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa62003e210 0x7fa6200406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.536+0000 7fa644a31640 10 monclient: get_auth_request con 0x7fa62003e210 auth_method 0 2024-09-10T16:32:17.551 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.539+0000 7fa644a31640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa62003e210 0x7fa6200406d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa630009bd0 tx=0x7fa63002f000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.737+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.738 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.737+0000 7fa645232640 10 monclient: get_auth_request con 0x7fa620044010 auth_method 0 2024-09-10T16:32:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.741+0000 7fa645232640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.743 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.742+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== osd.2 v2:172.21.15.164:6808/1539912567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa620046b00 con 0x7fa620044010 2024-09-10T16:32:17.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.755+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 --> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa64009eb20 con 0x7fa620044010 2024-09-10T16:32:17.757 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.755+0000 7fa63d7fa640 1 -- 172.21.15.164:0/1372546659 <== osd.2 v2:172.21.15.164:6808/1539912567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa64009eb20 con 0x7fa620044010 2024-09-10T16:32:17.757 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:17.757 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:17.757 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:17.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.755+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 msgr2=0x7fa620046430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.755+0000 7fa646c95640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:6808/1539912567,v1:172.21.15.164:6809/1539912567] conn(0x7fa620044010 0x7fa620046430 crc :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.758 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa62003e210 msgr2=0x7fa6200406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fa62003e210 0x7fa6200406d0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fa630009bd0 tx=0x7fa63002f000 comp rx=0 tx=0).stop 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 10 monclient: shutdown 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 msgr2=0x7fa640091bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 --2- 172.21.15.164:0/1372546659 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fa64009d270 0x7fa640091bd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa6400a2a20 tx=0x7fa62800c870 comp rx=0 tx=0).stop 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa645232640 1 -- 172.21.15.164:0/1372546659 reap_dead start 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 shutdown_connections 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 >> 172.21.15.164:0/1372546659 conn(0x7fa6400a31f0 msgr2=0x7fa640139a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 shutdown_connections 2024-09-10T16:32:17.759 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.756+0000 7fa646c95640 1 -- 172.21.15.164:0/1372546659 wait complete. 2024-09-10T16:32:17.768 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-10T16:32:17.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.865+0000 7fc17b964640 10 monclient: get_monmap_and_config 2024-09-10T16:32:17.867 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.865+0000 7fc17b964640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.866+0000 7fc17b964640 10 monclient: monmap: 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:17.866888+0000 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:17.866888+0000 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:17.868 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.869 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.866+0000 7fc17b964640 1 Processor -- start 2024-09-10T16:32:17.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.866+0000 7fc17b964640 1 -- start start 2024-09-10T16:32:17.873 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.866+0000 7fc17b964640 10 monclient: init 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc17406d160 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 0x7fc1740a46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-b con 0x7fc17406d540 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc1740a4c20 0x7fc17413f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-c con 0x7fc1740a4c20 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc174135510 con 0x7fc17406d160 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc17409cb90 con 0x7fc1740a4c20 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.874 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc17409ce70 con 0x7fc17406d540 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc17b964640 10 monclient(hunting): authenticate will time out at 2316.354511s 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179700640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc17406d160 0x7fc174134e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49694/0 (socket says 172.21.15.164:49694) 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179700640 1 -- 172.21.15.164:0/541997763 learned_addr learned my addr 172.21.15.164:0/541997763 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 --2- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 0x7fc1740a46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): get_auth_request con 0x7fc17406d540 auth_method 0 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.875 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 1 --2- 172.21.15.164:0/541997763 >> v2:172.21.15.165:3300/0 conn(0x7fc1740a4c20 0x7fc17413f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): get_auth_request con 0x7fc1740a4c20 auth_method 0 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient(hunting): handle_auth_done global_id 274355 payload 274 2024-09-10T16:32:17.876 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 -- 172.21.15.164:0/541997763 >> v2:172.21.15.165:3300/0 conn(0x7fc1740a4c20 msgr2=0x7fc17413f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 --2- 172.21.15.164:0/541997763 >> v2:172.21.15.165:3300/0 conn(0x7fc1740a4c20 0x7fc17413f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 -- 172.21.15.164:0/541997763 >> v1:172.21.15.164:6789/0 conn(0x7fc17406d160 legacy=0x7fc174134e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 monclient: found mon.noname-b 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.867+0000 7fc178eff640 1 -- 172.21.15.164:0/541997763 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc17409d150 con 0x7fc17406d540 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc178eff640 1 -- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 msgr2=0x7fc1740a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc178eff640 1 --2- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 0x7fc1740a46e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1640097a0 tx=0x7fc164029fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a63545e4ce2cd93a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 1 -- 172.21.15.164:0/541997763 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164037070 con 0x7fc17406d540 2024-09-10T16:32:17.877 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: dump: 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.878 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc17b964640 5 monclient: authenticate success, global_id 274355 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 1 -- 172.21.15.164:0/541997763 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc164032440 con 0x7fc17406d540 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc17b964640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc17b964640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 1 -- 172.21.15.164:0/541997763 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc164032810 con 0x7fc17406d540 2024-09-10T16:32:17.879 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc17b964640 4 set_mon_vals no callback set 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.870+0000 7fc16bfff640 10 monclient: dump: 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.880 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 10 monclient: shutdown 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 -- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 msgr2=0x7fc1740a46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 --2- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 0x7fc1740a46e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc1640097a0 tx=0x7fc164029fd0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 -- 172.21.15.164:0/541997763 shutdown_connections 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 --2- 172.21.15.164:0/541997763 >> v2:172.21.15.165:3300/0 conn(0x7fc1740a4c20 0x7fc17413f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 --2- 172.21.15.164:0/541997763 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d540 0x7fc1740a46e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.881 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 -- 172.21.15.164:0/541997763 >> 172.21.15.164:0/541997763 conn(0x7fc1740aacb0 msgr2=0x7fc1740ab0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 -- 172.21.15.164:0/541997763 shutdown_connections 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.871+0000 7fc17b964640 1 -- 172.21.15.164:0/541997763 wait complete. 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient: build_initial_monmap 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient: monmap: 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:17.882 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 Processor -- start 2024-09-10T16:32:17.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 -- start start 2024-09-10T16:32:17.883 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient: init 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-c con 0x7fc17406d160 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17406d540 0x7fc1740a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-b con 0x7fc17406d540 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 0x7fc1740a3fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.888 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): picked mon.noname-a con 0x7fc1740a4c20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc17413fb20 con 0x7fc1740a4c20 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc17413f9a0 con 0x7fc17406d540 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc17413fca0 con 0x7fc17406d160 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.872+0000 7fc17b964640 10 monclient(hunting): authenticate will time out at 2316.359990s 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 0x7fc1740a3fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.889 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33020/0 (socket says 172.21.15.164:33020) 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 -- 172.21.15.164:0/1179855986 learned_addr learned my addr 172.21.15.164:0/1179855986 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): get_auth_request con 0x7fc17406d160 auth_method 0 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): get_auth_request con 0x7fc1740a4c20 auth_method 0 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.890 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17406d540 0x7fc1740a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): get_auth_request con 0x7fc17406d540 auth_method 0 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc178eff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient(hunting): handle_auth_done global_id 274358 payload 824 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 -- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17406d540 msgr2=0x7fc1740a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.891 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17406d540 0x7fc1740a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 -- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 msgr2=0x7fc1740a3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 0x7fc1740a3fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient: _finish_hunting 0 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 monclient: found mon.noname-c 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179700640 1 -- 172.21.15.164:0/1179855986 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1740a4510 con 0x7fc17406d160 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.873+0000 7fc179f01640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 0x7fc1740a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc179700640 1 -- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 msgr2=0x7fc1741cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc179700640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c490 tx=0x7fc17000c950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc170017070 con 0x7fc17406d160 2024-09-10T16:32:17.892 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: dump: 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:17.893 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1700070d0 con 0x7fc17406d160 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 5 monclient: authenticate success, global_id 274358 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 10 monclient: _renew_subs 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1740a2560 con 0x7fc17406d160 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1700074c0 con 0x7fc17406d160 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc169ffb640 10 monclient: dump: 2024-09-10T16:32:17.894 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 10 monclient: _renew_subs 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc16b7fe640 4 set_mon_vals no callback set 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.875+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1740a0d60 con 0x7fc17406d160 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.876+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc17001a070 con 0x7fc17406d160 2024-09-10T16:32:17.895 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc17b964640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc17b964640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc17b964640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1740a46e0 con 0x7fc17406d160 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc169ffb640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 0x7fc1580406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fc1700520c0 con 0x7fc17406d160 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc178eff640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 0x7fc1580406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.877+0000 7fc178eff640 10 monclient: get_auth_request con 0x7fc15803e210 auth_method 0 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.880+0000 7fc178eff640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 0x7fc1580406d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc17406d130 tx=0x7fc16402e000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.881+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc170050c50 con 0x7fc17406d160 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.881+0000 7fc169ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:17.896 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:17.881+0000 7fc169ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:18.055 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.054+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fc1740a22e0 con 0x7fc15803e210 2024-09-10T16:32:18.056 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.054+0000 7fc169ffb640 1 -- 172.21.15.164:0/1179855986 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc1740a22e0 con 0x7fc15803e210 2024-09-10T16:32:18.056 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 2 to deep-scrub 2024-09-10T16:32:18.058 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 msgr2=0x7fc1580406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 0x7fc1580406d0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc17406d130 tx=0x7fc16402e000 comp rx=0 tx=0).stop 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 10 monclient: shutdown 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 msgr2=0x7fc1741cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc17000c490 tx=0x7fc17000c950 comp rx=0 tx=0).stop 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 shutdown_connections 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc15803e210 0x7fc1580406d0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc1740a4c20 0x7fc1740a3fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc17406d540 0x7fc1740a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 --2- 172.21.15.164:0/1179855986 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc17406d160 0x7fc1741cd280 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 >> 172.21.15.164:0/1179855986 conn(0x7fc1740aacb0 msgr2=0x7fc17412e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 shutdown_connections 2024-09-10T16:32:18.060 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.057+0000 7fc17b964640 1 -- 172.21.15.164:0/1179855986 wait complete. 2024-09-10T16:32:18.069 INFO:tasks.ceph:Scrubbing osd.3 2024-09-10T16:32:18.069 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:18.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.169+0000 7ffb01584640 10 monclient: get_monmap_and_config 2024-09-10T16:32:18.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.169+0000 7ffb01584640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.174 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.169+0000 7ffb01584640 10 monclient: monmap: 2024-09-10T16:32:18.174 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:18.170766+0000 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:18.170766+0000 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.175 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.170+0000 7ffb01584640 1 Processor -- start 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.170+0000 7ffb01584640 1 -- start start 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.170+0000 7ffb01584640 10 monclient: init 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-b con 0x7ffafc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffafc0a4700 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7ffafc0a4b20 0x7ffafc138500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-c con 0x7ffafc0a4b20 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:18.176 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc06dd80 con 0x7ffafc0a4700 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7ffafc0a9d60 con 0x7ffafc0a4b20 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc06cf50 con 0x7ffafc06cb70 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffb01584640 10 monclient(hunting): authenticate will time out at 2316.658134s 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33028/0 (socket says 172.21.15.164:33028) 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 -- 172.21.15.164:0/2181046299 learned_addr learned my addr 172.21.15.164:0/2181046299 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): get_auth_request con 0x7ffafc06cb70 auth_method 0 2024-09-10T16:32:18.177 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 1 --2- 172.21.15.164:0/2181046299 >> v2:172.21.15.165:3300/0 conn(0x7ffafc0a4b20 0x7ffafc138500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): get_auth_request con 0x7ffafc0a4b20 auth_method 0 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.178 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient(hunting): handle_auth_done global_id 274361 payload 274 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 -- 172.21.15.164:0/2181046299 >> v2:172.21.15.165:3300/0 conn(0x7ffafc0a4b20 msgr2=0x7ffafc138500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 --2- 172.21.15.164:0/2181046299 >> v2:172.21.15.165:3300/0 conn(0x7ffafc0a4b20 0x7ffafc138500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 -- 172.21.15.164:0/2181046299 >> v1:172.21.15.165:6789/0 conn(0x7ffafc0a4700 legacy=0x7ffafc137df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 monclient: found mon.noname-b 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.179 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.171+0000 7ffafaffd640 1 -- 172.21.15.164:0/2181046299 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc09cbd0 con 0x7ffafc06cb70 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffafaffd640 1 -- 172.21.15.164:0/2181046299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 msgr2=0x7ffafc134d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffafaffd640 1 --2- 172.21.15.164:0/2181046299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffafc06dd00 tx=0x7ffaec02a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=624f500e00e5fb22 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 1 -- 172.21.15.164:0/2181046299 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec037070 con 0x7ffafc06cb70 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: dump: 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.186 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.187 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 1 -- 172.21.15.164:0/2181046299 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaec032440 con 0x7ffafc06cb70 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 5 monclient: authenticate success, global_id 274361 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 4 set_mon_vals no callback set 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 1 -- 172.21.15.164:0/2181046299 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec032810 con 0x7ffafc06cb70 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffaf9ffb640 10 monclient: dump: 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.188 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 10 monclient: shutdown 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.189 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.174+0000 7ffb01584640 1 -- 172.21.15.164:0/2181046299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 msgr2=0x7ffafc134d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 --2- 172.21.15.164:0/2181046299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffafc06dd00 tx=0x7ffaec02a060 comp rx=0 tx=0).stop 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 -- 172.21.15.164:0/2181046299 shutdown_connections 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 --2- 172.21.15.164:0/2181046299 >> v2:172.21.15.165:3300/0 conn(0x7ffafc0a4b20 0x7ffafc138500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 --2- 172.21.15.164:0/2181046299 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc134d30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 -- 172.21.15.164:0/2181046299 >> 172.21.15.164:0/2181046299 conn(0x7ffafc0aa990 msgr2=0x7ffafc0aada0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 -- 172.21.15.164:0/2181046299 shutdown_connections 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 1 -- 172.21.15.164:0/2181046299 wait complete. 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.175+0000 7ffb01584640 10 monclient: monmap: 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:18.190 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 Processor -- start 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 -- start start 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient: init 2024-09-10T16:32:18.191 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-c con 0x7ffafc06cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc0a4700 0x7ffafc131a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-b con 0x7ffafc0a4700 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffafc0a4b20 0x7ffafc1ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): picked mon.noname-a con 0x7ffafc0a4b20 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc138a40 con 0x7ffafc0a4b20 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.192 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc135270 con 0x7ffafc0a4700 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc138bc0 con 0x7ffafc06cb70 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffb01584640 10 monclient(hunting): authenticate will time out at 2316.663673s 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33044/0 (socket says 172.21.15.164:33044) 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffafc0a4b20 0x7ffafc1ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 1 -- 172.21.15.164:0/3764899716 learned_addr learned my addr 172.21.15.164:0/3764899716 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): get_auth_request con 0x7ffafc06cb70 auth_method 0 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.193 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): get_auth_request con 0x7ffafc0a4b20 auth_method 0 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.194 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafa7fc640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc0a4700 0x7ffafc131a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafa7fc640 10 monclient(hunting): get_auth_request con 0x7ffafc0a4700 auth_method 0 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafa7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.176+0000 7ffafa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 10 monclient(hunting): handle_auth_done global_id 274364 payload 824 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc0a4700 msgr2=0x7ffafc131a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7ffafc0a4700 0x7ffafc131a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffafc0a4b20 msgr2=0x7ffafc1ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffafc0a4b20 0x7ffafc1ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.195 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafaffd640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc0a3630 con 0x7ffafc06cb70 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.177+0000 7ffafb7fe640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7ffafc0a4b20 0x7ffafc1ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffafaffd640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 msgr2=0x7ffafc1314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffafaffd640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0090b0 tx=0x7ffaec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec041020 con 0x7ffafc06cb70 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: dump: 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.196 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 5 monclient: authenticate success, global_id 274364 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaec03d830 con 0x7ffafc06cb70 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 10 monclient: _renew_subs 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec037040 con 0x7ffafc06cb70 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1368f0 con 0x7ffafc06cb70 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffae37fe640 10 monclient: dump: 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffaf97fa640 4 set_mon_vals no callback set 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 10 monclient: _renew_subs 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.179+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc0a12d0 con 0x7ffafc06cb70 2024-09-10T16:32:18.199 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7ffaec044020 con 0x7ffafc06cb70 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffb01584640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffafc134d30 con 0x7ffafc06cb70 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffafaffd640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0090b0 tx=0x7ffaec0046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7ffaec078420 mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffae37fe640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffae803e240 0x7ffae8040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7ffaec072e90 con 0x7ffafc06cb70 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffae37fe640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7ffae80440b0 0x7ffae80464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.180+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffaec032440 con 0x7ffae80440b0 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.181+0000 7ffafa7fc640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffae803e240 0x7ffae8040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.181+0000 7ffafa7fc640 10 monclient: get_auth_request con 0x7ffae803e240 auth_method 0 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.181+0000 7ffafb7fe640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7ffae80440b0 0x7ffae80464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.200 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.181+0000 7ffafb7fe640 10 monclient: get_auth_request con 0x7ffae80440b0 auth_method 0 2024-09-10T16:32:18.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.183+0000 7ffafa7fc640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffae803e240 0x7ffae8040700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0004ea0 tx=0x7ffaf0009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.201 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.186+0000 7ffafb7fe640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7ffae80440b0 0x7ffae80464d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.187+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== osd.3 v2:172.21.15.165:6800/4161381544 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffaec032440 con 0x7ffae80440b0 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.199+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 --> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffafc0a2f20 con 0x7ffae80440b0 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== osd.3 v2:172.21.15.165:6800/4161381544 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffaf400e830 con 0x7ffae80440b0 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:18.202 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:18.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7ffae80440b0 msgr2=0x7ffae80464d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6800/4161381544,v1:172.21.15.165:6801/4161381544] conn(0x7ffae80440b0 0x7ffae80464d0 crc :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffae803e240 msgr2=0x7ffae8040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7ffae803e240 0x7ffae8040700 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0004ea0 tx=0x7ffaf0009040 comp rx=0 tx=0).stop 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 10 monclient: shutdown 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 20 monclient: shutdown canceling and discarding version request 1 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 msgr2=0x7ffafc1314c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 --2- 172.21.15.164:0/3764899716 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7ffafc06cb70 0x7ffafc1314c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0090b0 tx=0x7ffaec0046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffafb7fe640 1 -- 172.21.15.164:0/3764899716 reap_dead start 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffae37fe640 1 -- 172.21.15.164:0/3764899716 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (unknown 0 0 0) 0x7ffaec078420 con 0x7ffafc06cb70 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffae37fe640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:18.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 shutdown_connections 2024-09-10T16:32:18.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 >> 172.21.15.164:0/3764899716 conn(0x7ffafc0aa990 msgr2=0x7ffafc137e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 shutdown_connections 2024-09-10T16:32:18.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.200+0000 7ffb01584640 1 -- 172.21.15.164:0/3764899716 wait complete. 2024-09-10T16:32:18.212 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.309+0000 7fc939bbb640 10 monclient: get_monmap_and_config 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.309+0000 7fc939bbb640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.309+0000 7fc939bbb640 10 monclient: monmap: 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:18.310439+0000 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:18.310439+0000 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.311 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:18.312 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 Processor -- start 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 -- start start 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient: init 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc93409c310 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc93409d190 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc93413b5d0 0x7fc93413d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc93413b5d0 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc93409fbc0 con 0x7fc93409d190 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9340a2a20 con 0x7fc93409c310 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc93409f860 con 0x7fc93413b5d0 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc939bbb640 10 monclient(hunting): authenticate will time out at 2316.797825s 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc932ffd640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc93409d190 0x7fc93409f150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49700/0 (socket says 172.21.15.164:49700) 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.310+0000 7fc932ffd640 1 -- 172.21.15.164:0/3146133151 learned_addr learned my addr 172.21.15.164:0/3146133151 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 1 --2- 172.21.15.164:0/3146133151 >> v2:172.21.15.165:3300/0 conn(0x7fc93413b5d0 0x7fc93413d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): get_auth_request con 0x7fc93413b5d0 auth_method 0 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.316 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient(hunting): handle_auth_done global_id 284257 payload 274 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 1 -- 172.21.15.164:0/3146133151 >> v1:172.21.15.165:6789/0 conn(0x7fc93409c310 legacy=0x7fc93409ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 1 -- 172.21.15.164:0/3146133151 >> v1:172.21.15.164:6789/0 conn(0x7fc93409d190 legacy=0x7fc93409f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:32:18.317 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.311+0000 7fc933fff640 1 -- 172.21.15.164:0/3146133151 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc93409c6f0 con 0x7fc93413b5d0 2024-09-10T16:32:18.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc933fff640 1 -- 172.21.15.164:0/3146133151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 msgr2=0x7fc93413d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.324 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc933fff640 1 --2- 172.21.15.164:0/3146133151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc93413d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009d90 tx=0x7fc92c02a490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7ce522e15b4049d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 1 -- 172.21.15.164:0/3146133151 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c007a30 con 0x7fc93413b5d0 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: dump: 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.325 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 1 -- 172.21.15.164:0/3146133151 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc92c007bd0 con 0x7fc93413b5d0 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 1 -- 172.21.15.164:0/3146133151 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc92c031400 con 0x7fc93413b5d0 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc9327fc640 10 monclient: dump: 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.326 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc939bbb640 5 monclient: authenticate success, global_id 284257 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc939bbb640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.314+0000 7fc939bbb640 4 set_mon_vals no callback set 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 10 monclient: shutdown 2024-09-10T16:32:18.327 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 -- 172.21.15.164:0/3146133151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 msgr2=0x7fc93413d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 --2- 172.21.15.164:0/3146133151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc93413d9c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc92c009d90 tx=0x7fc92c02a490 comp rx=0 tx=0).stop 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 -- 172.21.15.164:0/3146133151 shutdown_connections 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 --2- 172.21.15.164:0/3146133151 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc93413d9c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 -- 172.21.15.164:0/3146133151 >> 172.21.15.164:0/3146133151 conn(0x7fc9340a3110 msgr2=0x7fc9340a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 -- 172.21.15.164:0/3146133151 shutdown_connections 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 1 -- 172.21.15.164:0/3146133151 wait complete. 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.315+0000 7fc939bbb640 10 monclient: monmap: 2024-09-10T16:32:18.328 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 Processor -- start 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 -- start start 2024-09-10T16:32:18.329 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient: init 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-c con 0x7fc93409c310 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc934099a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-b con 0x7fc93413b5d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 0x7fc934091320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): picked mon.noname-a con 0x7fc9341cad50 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc9340a20b0 con 0x7fc9341cad50 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.330 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc93409a530 con 0x7fc93413b5d0 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc9340a2230 con 0x7fc93409c310 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc939bbb640 10 monclient(hunting): authenticate will time out at 2316.803764s 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc933fff640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 0x7fc934091320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33060/0 (socket says 172.21.15.164:33060) 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 1 -- 172.21.15.164:0/445407090 learned_addr learned my addr 172.21.15.164:0/445407090 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): get_auth_request con 0x7fc93409c310 auth_method 0 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.331 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc933fff640 10 monclient(hunting): get_auth_request con 0x7fc9341cad50 auth_method 0 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc933fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc933fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc933fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc932ffd640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc934099a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc9337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc932ffd640 10 monclient(hunting): get_auth_request con 0x7fc93413b5d0 auth_method 0 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc932ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.332 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc932ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.316+0000 7fc932ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc933fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc932ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc932ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc932ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 10 monclient(hunting): handle_auth_done global_id 274367 payload 824 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 -- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 msgr2=0x7fc934099a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.333 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc934099a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 -- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 msgr2=0x7fc934091320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 0x7fc934091320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc9337fe640 1 -- 172.21.15.164:0/445407090 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc934094fc0 con 0x7fc93409c310 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.317+0000 7fc933fff640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 0x7fc934091320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc9337fe640 1 -- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 msgr2=0x7fc934099520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc9337fe640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c430 tx=0x7fc92400c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.334 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9240119a0 con 0x7fc93409c310 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: dump: 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.335 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc924017070 con 0x7fc93409c310 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 5 monclient: authenticate success, global_id 274367 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.336 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc924011d80 con 0x7fc93409c310 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 10 monclient: _renew_subs 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc934093da0 con 0x7fc93409c310 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc92bfff640 10 monclient: dump: 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.337 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc931ffb640 4 set_mon_vals no callback set 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 10 monclient: _renew_subs 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.319+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc934094a00 con 0x7fc93409c310 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc92401c460 con 0x7fc93409c310 2024-09-10T16:32:18.338 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc939bbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc939bbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc939bbb640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc92bfff640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 0x7fc9140406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc93409ea40 con 0x7fc93409c310 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.320+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fc924052a70 con 0x7fc93409c310 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.321+0000 7fc932ffd640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 0x7fc9140406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.321+0000 7fc932ffd640 10 monclient: get_auth_request con 0x7fc91403e210 auth_method 0 2024-09-10T16:32:18.339 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.323+0000 7fc932ffd640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 0x7fc9140406d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007c90 tx=0x7fc91c007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.325+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc92401ac60 con 0x7fc93409c310 2024-09-10T16:32:18.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.325+0000 7fc92bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.340 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.325+0000 7fc92bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:18.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.498+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fc9340937c0 con 0x7fc91403e210 2024-09-10T16:32:18.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.499+0000 7fc92bfff640 1 -- 172.21.15.164:0/445407090 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc9340937c0 con 0x7fc91403e210 2024-09-10T16:32:18.500 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 3 to deep-scrub 2024-09-10T16:32:18.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 msgr2=0x7fc9140406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 0x7fc9140406d0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7fc91c007c90 tx=0x7fc91c007be0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 10 monclient: shutdown 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 msgr2=0x7fc934099520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc92400c430 tx=0x7fc92400c8f0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 shutdown_connections 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc91403e210 0x7fc9140406d0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc9341cad50 0x7fc934091320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc93413b5d0 0x7fc934099a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 --2- 172.21.15.164:0/445407090 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc93409c310 0x7fc934099520 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 >> 172.21.15.164:0/445407090 conn(0x7fc9340a3110 msgr2=0x7fc93413a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 shutdown_connections 2024-09-10T16:32:18.503 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.501+0000 7fc939bbb640 1 -- 172.21.15.164:0/445407090 wait complete. 2024-09-10T16:32:18.513 INFO:tasks.ceph:Scrubbing osd.4 2024-09-10T16:32:18.513 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.614+0000 7f5d20918640 10 monclient: get_monmap_and_config 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.614+0000 7f5d20918640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.614+0000 7f5d20918640 10 monclient: monmap: 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:18.615419+0000 2024-09-10T16:32:18.616 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:18.615419+0000 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:18.617 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 Processor -- start 2024-09-10T16:32:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 -- start start 2024-09-10T16:32:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient: init 2024-09-10T16:32:18.622 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f5d1c09da10 0x7f5d1c09ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1c09da10 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c09ac90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5d1c138680 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d1c074d50 con 0x7f5d1c138680 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f5d1c0a0a30 con 0x7f5d1c09da10 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1c09e870 con 0x7f5d1c09ac90 2024-09-10T16:32:18.623 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d20918640 10 monclient(hunting): authenticate will time out at 2317.102773s 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33070/0 (socket says 172.21.15.164:33070) 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 1 -- 172.21.15.164:0/3105568007 learned_addr learned my addr 172.21.15.164:0/3105568007 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 10 monclient(hunting): get_auth_request con 0x7f5d1c09ac90 auth_method 0 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.615+0000 7f5d19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d19d9b640 1 -- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 msgr2=0x7f5d1c09e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d19d9b640 1 -- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 msgr2=0x7f5d1c09e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d19d9b640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T16:32:18.624 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d19d9b640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/3105568007 >> v2:172.21.15.165:3300/0 conn(0x7f5d1c09da10 0x7f5d1c09ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): get_auth_request con 0x7f5d1c09da10 auth_method 0 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_done global_id 284263 payload 274 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 -- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 msgr2=0x7f5d1c09e330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 -- 172.21.15.164:0/3105568007 >> v1:172.21.15.165:6789/0 conn(0x7f5d1c138680 legacy=0x7f5d1c13aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.625 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.165:3300/0 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.616+0000 7f5d1a59c640 1 -- 172.21.15.164:0/3105568007 --> v2:172.21.15.165:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c09ede0 con 0x7f5d1c09da10 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1a59c640 1 -- 172.21.15.164:0/3105568007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c09da10 msgr2=0x7f5d1c09ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c09da10 0x7f5d1c09ddf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c06e620 tx=0x7f5d0c02a040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c32560268904d74a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 1 -- 172.21.15.164:0/3105568007 <== mon.1 v2:172.21.15.165:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c037070 con 0x7f5d1c09da10 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: dump: 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.626 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 1 -- 172.21.15.164:0/3105568007 <== mon.1 v2:172.21.15.165:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c032440 con 0x7f5d1c09da10 2024-09-10T16:32:18.627 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 1 -- 172.21.15.164:0/3105568007 <== mon.1 v2:172.21.15.165:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c032810 con 0x7f5d1c09da10 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d1959a640 10 monclient: dump: 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.628 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d20918640 5 monclient: authenticate success, global_id 284263 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d20918640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.619+0000 7f5d20918640 4 set_mon_vals no callback set 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 10 monclient: shutdown 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 -- 172.21.15.164:0/3105568007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c09da10 msgr2=0x7f5d1c09ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c09da10 0x7f5d1c09ddf0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c06e620 tx=0x7f5d0c02a040 comp rx=0 tx=0).stop 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 -- 172.21.15.164:0/3105568007 shutdown_connections 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c09e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 --2- 172.21.15.164:0/3105568007 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c09da10 0x7f5d1c09ddf0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.629 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 -- 172.21.15.164:0/3105568007 >> 172.21.15.164:0/3105568007 conn(0x7f5d1c0a31f0 msgr2=0x7f5d1c0a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 -- 172.21.15.164:0/3105568007 shutdown_connections 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.620+0000 7f5d20918640 1 -- 172.21.15.164:0/3105568007 wait complete. 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient: monmap: 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:18.630 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:18.631 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 Processor -- start 2024-09-10T16:32:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 -- start start 2024-09-10T16:32:18.631 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient: init 2024-09-10T16:32:18.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T16:32:18.636 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1c09ac90 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d1c09da10 0x7f5d1c073930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-a con 0x7f5d1c09da10 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c138680 0x7f5d1c073e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): picked mon.noname-b con 0x7f5d1c138680 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c0a2a20 con 0x7f5d1c09da10 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f5d1c0a0190 con 0x7f5d1c138680 2024-09-10T16:32:18.637 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f5d1c0a2ba0 con 0x7f5d1c09ac90 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.621+0000 7f5d20918640 10 monclient(hunting): authenticate will time out at 2317.108984s 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33078/0 (socket says 172.21.15.164:33078) 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d1c09da10 0x7f5d1c073930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 -- 172.21.15.164:0/2914852206 learned_addr learned my addr 172.21.15.164:0/2914852206 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): get_auth_request con 0x7f5d1c09ac90 auth_method 0 2024-09-10T16:32:18.638 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): get_auth_request con 0x7f5d1c09da10 auth_method 0 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.639 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c138680 0x7f5d1c073e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 10 monclient(hunting): get_auth_request con 0x7f5d1c138680 auth_method 0 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient(hunting): handle_auth_done global_id 274373 payload 824 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c138680 msgr2=0x7f5d1c073e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.640 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c138680 0x7f5d1c073e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d1c09da10 msgr2=0x7f5d1c073930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d1c09da10 0x7f5d1c073930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1a59c640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d1c090a30 con 0x7f5d1c09ac90 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d1ad9d640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f5d1c138680 0x7f5d1c073e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.622+0000 7f5d19d9b640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f5d1c09da10 0x7f5d1c073930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:18.641 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d1a59c640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 msgr2=0x7f5d1c0733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02a630 tx=0x7f5d0c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c041070 con 0x7f5d1c09ac90 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: dump: 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.642 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d0c03c8b0 con 0x7f5d1c09ac90 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 5 monclient: authenticate success, global_id 274373 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 10 monclient: _renew_subs 2024-09-10T16:32:18.643 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d1c0a6050 con 0x7f5d1c09ac90 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c037040 con 0x7f5d1c09ac90 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d12ffd640 10 monclient: dump: 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.644 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 10 monclient: _renew_subs 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d18d99640 4 set_mon_vals no callback set 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.624+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d1c0925a0 con 0x7f5d1c09ac90 2024-09-10T16:32:18.645 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.625+0000 7f5d20918640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.625+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5d1c0a2120 con 0x7f5d1c09ac90 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.625+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f5d0c044070 con 0x7f5d1c09ac90 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.625+0000 7f5d1a59c640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02a630 tx=0x7f5d0c0046c0 comp rx=0 tx=0).queue_received will delay after 1.72599e+09 on 0x7f5d0c07a050 mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d12ffd640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cfc03e210 0x7f5cfc0406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f5d0c073470 con 0x7f5d1c09ac90 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d12ffd640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7f5cfc044010 0x7f5cfc046430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5d0c032440 con 0x7f5cfc044010 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d19d9b640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cfc03e210 0x7f5cfc0406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.646 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d19d9b640 10 monclient: get_auth_request con 0x7f5cfc03e210 auth_method 0 2024-09-10T16:32:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d1ad9d640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7f5cfc044010 0x7f5cfc046430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.626+0000 7f5d1ad9d640 10 monclient: get_auth_request con 0x7f5cfc044010 auth_method 0 2024-09-10T16:32:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.628+0000 7f5d19d9b640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cfc03e210 0x7f5cfc0406d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0921f0 tx=0x7f5d04008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.631+0000 7f5d1ad9d640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7f5cfc044010 0x7f5cfc046430 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.647 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.632+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== osd.4 v2:172.21.15.165:6808/592433527 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5d0c032440 con 0x7f5cfc044010 2024-09-10T16:32:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.644+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 --> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f5d1c074600 con 0x7f5cfc044010 2024-09-10T16:32:18.648 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.644+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== osd.4 v2:172.21.15.165:6808/592433527 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f5d1c074600 con 0x7f5cfc044010 2024-09-10T16:32:18.648 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:18.648 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:18.648 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:18.649 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7f5cfc044010 msgr2=0x7f5cfc046430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6808/592433527,v1:172.21.15.165:6809/592433527] conn(0x7f5cfc044010 0x7f5cfc046430 crc :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cfc03e210 msgr2=0x7f5cfc0406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f5cfc03e210 0x7f5cfc0406d0 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f5d1c0921f0 tx=0x7f5d04008040 comp rx=0 tx=0).stop 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 10 monclient: shutdown 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 20 monclient: shutdown canceling and discarding version request 1 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 msgr2=0x7f5d1c0733f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 --2- 172.21.15.164:0/2914852206 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f5d1c09ac90 0x7f5d1c0733f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c02a630 tx=0x7f5d0c0046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d1ad9d640 1 -- 172.21.15.164:0/2914852206 reap_dead start 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d12ffd640 1 -- 172.21.15.164:0/2914852206 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (unknown 0 0 0) 0x7f5d0c07a050 con 0x7f5d1c09ac90 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d12ffd640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=897) 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 shutdown_connections 2024-09-10T16:32:18.650 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 >> 172.21.15.164:0/2914852206 conn(0x7f5d1c0a31f0 msgr2=0x7f5d1c099690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 shutdown_connections 2024-09-10T16:32:18.651 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.645+0000 7f5d20918640 1 -- 172.21.15.164:0/2914852206 wait complete. 2024-09-10T16:32:18.657 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 10 monclient: get_monmap_and_config 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 10 monclient: monmap: 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:18.754068+0000 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:18.754068+0000 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:18.755 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 1 Processor -- start 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 1 -- start start 2024-09-10T16:32:18.756 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.753+0000 7f17e005a640 10 monclient: init 2024-09-10T16:32:18.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-10T16:32:18.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 0x7f17d8134f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.760 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-b con 0x7f17d806cb70 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17d80a46a0 0x7f17d80a4a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-a con 0x7f17d80a46a0 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f17d8137ca0 0x7f17d8138130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-c con 0x7f17d8137ca0 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f17d806db60 con 0x7f17d80a46a0 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f17d806cf50 con 0x7f17d8137ca0 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17d809ce50 con 0x7f17d806cb70 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17e005a640 10 monclient(hunting): authenticate will time out at 2317.241501s 2024-09-10T16:32:18.761 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17d80a46a0 0x7f17d80a4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 0x7f17d8134f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f17d80a46a0 0x7f17d80a4a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47898/0 (socket says 172.21.15.164:47898) 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 -- 172.21.15.164:0/1815434794 learned_addr learned my addr 172.21.15.164:0/1815434794 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): get_auth_request con 0x7f17d806cb70 auth_method 0 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): get_auth_request con 0x7f17d80a46a0 auth_method 0 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.762 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17de5f7640 1 --2- 172.21.15.164:0/1815434794 >> v2:172.21.15.165:3300/0 conn(0x7f17d8137ca0 0x7f17d8138130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17de5f7640 10 monclient(hunting): get_auth_request con 0x7f17d8137ca0 auth_method 0 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17de5f7640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17de5f7640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17de5f7640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_done global_id 294198 payload 274 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 -- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 msgr2=0x7f17d8134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.763 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 0x7f17d8134f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 -- 172.21.15.164:0/1815434794 >> v2:172.21.15.165:3300/0 conn(0x7f17d8137ca0 msgr2=0x7f17d8138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/1815434794 >> v2:172.21.15.165:3300/0 conn(0x7f17d8137ca0 0x7f17d8138130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 monclient: found mon.noname-a 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dd5f5640 1 -- 172.21.15.164:0/1815434794 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d809cb70 con 0x7f17d80a46a0 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.754+0000 7f17dddf6640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 0x7f17d8134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dd5f5640 1 -- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d80a46a0 msgr2=0x7f17d80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d80a46a0 0x7f17d80a4a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17d80a6cc0 tx=0x7f17c802a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb91729385e73db server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 1 -- 172.21.15.164:0/1815434794 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8037070 con 0x7f17d80a46a0 2024-09-10T16:32:18.764 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: dump: 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.765 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 1 -- 172.21.15.164:0/1815434794 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c8032440 con 0x7f17d80a46a0 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17e005a640 5 monclient: authenticate success, global_id 294198 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17e005a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17e005a640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 1 -- 172.21.15.164:0/1815434794 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8032810 con 0x7f17d80a46a0 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17e005a640 4 set_mon_vals no callback set 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.757+0000 7f17dcdf4640 10 monclient: dump: 2024-09-10T16:32:18.766 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 10 monclient: shutdown 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 -- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d80a46a0 msgr2=0x7f17d80a4a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d80a46a0 0x7f17d80a4a80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f17d80a6cc0 tx=0x7f17c802a0d0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.767 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 -- 172.21.15.164:0/1815434794 shutdown_connections 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 --2- 172.21.15.164:0/1815434794 >> v2:172.21.15.165:3300/0 conn(0x7f17d8137ca0 0x7f17d8138130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d80a46a0 0x7f17d80a4a80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 --2- 172.21.15.164:0/1815434794 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d806cb70 0x7f17d8134f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 -- 172.21.15.164:0/1815434794 >> 172.21.15.164:0/1815434794 conn(0x7f17d80aa900 msgr2=0x7f17d80aad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 -- 172.21.15.164:0/1815434794 shutdown_connections 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.758+0000 7f17e005a640 1 -- 172.21.15.164:0/1815434794 wait complete. 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient: build_initial_monmap 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient: monmap: 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:18.768 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 1 Processor -- start 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 1 -- start start 2024-09-10T16:32:18.769 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient: init 2024-09-10T16:32:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:32:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.773 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-a con 0x7f17d806cb70 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 0x7f17d809e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-c con 0x7f17d80a46a0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.759+0000 7f17e005a640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 0x7f17d809e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): picked mon.noname-b con 0x7f17d8137ca0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f17d81387f0 con 0x7f17d806cb70 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f17d8138670 con 0x7f17d8137ca0 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f17d8138970 con 0x7f17d80a46a0 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17e005a640 10 monclient(hunting): authenticate will time out at 2317.247044s 2024-09-10T16:32:18.774 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 0x7f17d809e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47906/0 (socket says 172.21.15.164:47906) 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 1 -- 172.21.15.164:0/381452826 learned_addr learned my addr 172.21.15.164:0/381452826 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): get_auth_request con 0x7f17d80a46a0 auth_method 0 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): get_auth_request con 0x7f17d806cb70 auth_method 0 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.775 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17de5f7640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 0x7f17d809e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient(hunting): handle_auth_done global_id 274379 payload 824 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 -- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 msgr2=0x7f17d809e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 0x7f17d809e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 -- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 msgr2=0x7f17d8131610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient: _finish_hunting 0 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 monclient: found mon.noname-c 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:18.776 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dd5f5640 1 -- 172.21.15.164:0/381452826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17d80a1d10 con 0x7f17d80a46a0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17de5f7640 10 monclient: get_auth_request con 0x7f17d8137ca0 auth_method 0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17de5f7640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 0x7f17d809e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.760+0000 7f17dddf6640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17dd5f5640 1 -- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 msgr2=0x7f17d809e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17dd5f5640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 0x7f17d809e2a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17d806da80 tx=0x7f17c80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c8037040 con 0x7f17d80a46a0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: dump: 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.777 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: _finish_auth 0 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 5 monclient: authenticate success, global_id 274379 2024-09-10T16:32:18.778 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17c8042070 con 0x7f17d80a46a0 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 10 monclient: _renew_subs 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17d809fef0 con 0x7f17d80a46a0 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17c803ba60 con 0x7f17d80a46a0 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17ce7fc640 10 monclient: dump: 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:18.779 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17cffff640 4 set_mon_vals no callback set 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 10 monclient: _renew_subs 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.762+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f17d806e720 con 0x7f17d80a46a0 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.763+0000 7f17e005a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.763+0000 7f17e005a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.763+0000 7f17e005a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.763+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f17c8032d10 con 0x7f17d80a46a0 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.763+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17d80a4a80 con 0x7f17d80a46a0 2024-09-10T16:32:18.780 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.764+0000 7f17ce7fc640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 0x7f17bc040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.764+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f17c8072f10 con 0x7f17d80a46a0 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.764+0000 7f17dddf6640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 0x7f17bc040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.764+0000 7f17dddf6640 10 monclient: get_auth_request con 0x7f17bc03e240 auth_method 0 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.766+0000 7f17dddf6640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 0x7f17bc040700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f17d40095e0 tx=0x7f17d4009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.768+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f17c803c3a0 con 0x7f17d80a46a0 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.768+0000 7f17ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:18.781 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.768+0000 7f17ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:18.943 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.942+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f17d80770f0 con 0x7f17bc03e240 2024-09-10T16:32:18.944 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.943+0000 7f17ce7fc640 1 -- 172.21.15.164:0/381452826 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f17d80770f0 con 0x7f17bc03e240 2024-09-10T16:32:18.944 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 4 to deep-scrub 2024-09-10T16:32:18.946 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 msgr2=0x7f17bc040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 0x7f17bc040700 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f17d40095e0 tx=0x7f17d4009190 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 10 monclient: shutdown 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 msgr2=0x7f17d809e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 0x7f17d809e2a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f17d806da80 tx=0x7f17c80046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 shutdown_connections 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f17bc03e240 0x7f17bc040700 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f17d8137ca0 0x7f17d809e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f17d80a46a0 0x7f17d809e2a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 --2- 172.21.15.164:0/381452826 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f17d806cb70 0x7f17d8131610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 >> 172.21.15.164:0/381452826 conn(0x7f17d80aa900 msgr2=0x7f17d80a8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 shutdown_connections 2024-09-10T16:32:18.947 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:18.945+0000 7f17e005a640 1 -- 172.21.15.164:0/381452826 wait complete. 2024-09-10T16:32:18.957 INFO:tasks.ceph:Scrubbing osd.5 2024-09-10T16:32:18.957 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.057+0000 7f0c949f3640 10 monclient: get_monmap_and_config 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.057+0000 7f0c949f3640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.057+0000 7f0c949f3640 10 monclient: monmap: 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.059 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:19.058699+0000 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:19.058699+0000 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:19.060 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.058+0000 7f0c949f3640 1 Processor -- start 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.058+0000 7f0c949f3640 1 -- start start 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.058+0000 7f0c949f3640 10 monclient: init 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient: _add_conns ranks=[3,1,0,2,4] 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c900a0520 0x7f0c900a1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c900a0520 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c9009d070 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:19.063 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0c9013b6b0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:19.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9009e8b0 con 0x7f0c9013b6b0 2024-09-10T16:32:19.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c9009eb90 con 0x7f0c9009d070 2024-09-10T16:32:19.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f0c9009fb50 con 0x7f0c900a0520 2024-09-10T16:32:19.064 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c949f3640 10 monclient(hunting): authenticate will time out at 2317.546241s 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f0c900a0520 0x7f0c900a1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8ed9d640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f0c9013b6b0 0x7f0c9013daa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49716/0 (socket says 172.21.15.164:49716) 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8ed9d640 1 -- 172.21.15.164:0/326895024 learned_addr learned my addr 172.21.15.164:0/326895024 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 --2- 172.21.15.164:0/326895024 >> v2:172.21.15.164:3300/0 conn(0x7f0c900a0520 0x7f0c900a1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47922/0 (socket says 172.21.15.164:47922) 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request con 0x7f0c900a0520 auth_method 0 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.065 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_done global_id 294204 payload 274 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 -- 172.21.15.164:0/326895024 >> v1:172.21.15.165:6789/0 conn(0x7f0c9009d070 legacy=0x7f0c900a2460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 -- 172.21.15.164:0/326895024 >> v1:172.21.15.164:6789/0 conn(0x7f0c9013b6b0 legacy=0x7f0c9013daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 monclient: found mon.noname-a 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.066 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.059+0000 7f0c8e59c640 1 -- 172.21.15.164:0/326895024 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c9009da10 con 0x7f0c900a0520 2024-09-10T16:32:19.072 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8e59c640 1 -- 172.21.15.164:0/326895024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c900a0520 msgr2=0x7f0c900a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8e59c640 1 --2- 172.21.15.164:0/326895024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c900a0520 0x7f0c900a1f20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c9009dc40 tx=0x7f0c8002a480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2aac44007cc3ef5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 1 -- 172.21.15.164:0/326895024 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80007a30 con 0x7f0c900a0520 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: dump: 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.073 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 1 -- 172.21.15.164:0/326895024 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c80007bd0 con 0x7f0c900a0520 2024-09-10T16:32:19.074 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 5 monclient: authenticate success, global_id 294204 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 4 set_mon_vals no callback set 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 1 -- 172.21.15.164:0/326895024 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80031400 con 0x7f0c900a0520 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c8d59a640 10 monclient: dump: 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.075 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 10 monclient: shutdown 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 1 -- 172.21.15.164:0/326895024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c900a0520 msgr2=0x7f0c900a1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.076 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.062+0000 7f0c949f3640 1 --2- 172.21.15.164:0/326895024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c900a0520 0x7f0c900a1f20 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f0c9009dc40 tx=0x7f0c8002a480 comp rx=0 tx=0).stop 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 1 -- 172.21.15.164:0/326895024 shutdown_connections 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 1 --2- 172.21.15.164:0/326895024 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c900a0520 0x7f0c900a1f20 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 1 -- 172.21.15.164:0/326895024 >> 172.21.15.164:0/326895024 conn(0x7f0c900a31f0 msgr2=0x7f0c900a3600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 1 -- 172.21.15.164:0/326895024 shutdown_connections 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 1 -- 172.21.15.164:0/326895024 wait complete. 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.063+0000 7f0c949f3640 10 monclient: monmap: 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:19.077 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 Processor -- start 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 -- start start 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient: init 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 0x7f0c9013afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-a con 0x7f0c9009d070 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c900a0520 0x7f0c900922c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.078 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-b con 0x7f0c900a0520 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c9013b6b0 0x7f0c90092800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): picked mon.noname-c con 0x7f0c9013b6b0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9009a060 con 0x7f0c9009d070 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f0c9009cad0 con 0x7f0c900a0520 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f0c9009a1e0 con 0x7f0c9013b6b0 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c949f3640 10 monclient(hunting): authenticate will time out at 2317.551745s 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 0x7f0c9013afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 0x7f0c9013afd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47924/0 (socket says 172.21.15.164:47924) 2024-09-10T16:32:19.079 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 1 -- 172.21.15.164:0/3054285422 learned_addr learned my addr 172.21.15.164:0/3054285422 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request con 0x7f0c9009d070 auth_method 0 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c9013b6b0 0x7f0c90092800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c9013b6b0 auth_method 0 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.080 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.064+0000 7f0c8dd9b640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c900a0520 0x7f0c900922c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request con 0x7f0c900a0520 auth_method 0 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 10 monclient(hunting): handle_auth_done global_id 294207 payload 824 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c9013b6b0 msgr2=0x7f0c90092800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c9013b6b0 0x7f0c90092800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c900a0520 msgr2=0x7f0c900922c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c900a0520 0x7f0c900922c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.081 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 monclient: found mon.noname-a 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8e59c640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c900a5e90 con 0x7f0c9009d070 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8dd9b640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f0c900a0520 0x7f0c900922c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.065+0000 7f0c8ed9d640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f0c9013b6b0 0x7f0c90092800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c8e59c640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 msgr2=0x7f0c9013afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c8e59c640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 0x7f0c9013afd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009e80 tx=0x7f0c80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c80040070 con 0x7f0c9009d070 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: dump: 2024-09-10T16:32:19.082 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.083 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 5 monclient: authenticate success, global_id 294207 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c8003c620 con 0x7f0c9009d070 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c8003ca10 con 0x7f0c9009d070 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 10 monclient: _renew_subs 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c90095e00 con 0x7f0c9009d070 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c8cd99640 4 set_mon_vals no callback set 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c86ffd640 10 monclient: dump: 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.084 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 10 monclient: _renew_subs 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.067+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c900746f0 con 0x7f0c9009d070 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.068+0000 7f0c949f3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.068+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0c900a2ce0 con 0x7f0c9009d070 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.068+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f0c80039040 con 0x7f0c9009d070 2024-09-10T16:32:19.085 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.068+0000 7f0c86ffd640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7003e240 0x7f0c70040700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.068+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f0c800742f0 con 0x7f0c9009d070 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c86ffd640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7f0c700440b0 0x7f0c700464d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c80040e50 con 0x7f0c700440b0 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_get_version_reply(handle=1 version=897) ==== 24+0+0 (secure 0 0 0) 0x7f0c8007b050 con 0x7f0c9009d070 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c86ffd640 10 monclient: handle_get_version_reply finishing 1 version 897 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c8dd9b640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7003e240 0x7f0c70040700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c8dd9b640 10 monclient: get_auth_request con 0x7f0c7003e240 auth_method 0 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c8ed9d640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7f0c700440b0 0x7f0c700464d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.069+0000 7f0c8ed9d640 10 monclient: get_auth_request con 0x7f0c700440b0 auth_method 0 2024-09-10T16:32:19.086 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.071+0000 7f0c8dd9b640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7003e240 0x7f0c70040700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0c90094990 tx=0x7f0c78009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.073+0000 7f0c8ed9d640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7f0c700440b0 0x7f0c700464d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.087 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.074+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== osd.5 v2:172.21.15.165:6816/3196812989 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0c80040e50 con 0x7f0c700440b0 2024-09-10T16:32:19.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.087+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 --> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0c90094660 con 0x7f0c700440b0 2024-09-10T16:32:19.088 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.087+0000 7f0c86ffd640 1 -- 172.21.15.164:0/3054285422 <== osd.5 v2:172.21.15.165:6816/3196812989 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f0c90094660 con 0x7f0c700440b0 2024-09-10T16:32:19.089 INFO:teuthology.orchestra.run.smithi164.stdout:{ 2024-09-10T16:32:19.089 INFO:teuthology.orchestra.run.smithi164.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T16:32:19.089 INFO:teuthology.orchestra.run.smithi164.stdout:} 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.087+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7f0c700440b0 msgr2=0x7f0c700464d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.087+0000 7f0c949f3640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6816/3196812989,v1:172.21.15.165:6817/3196812989] conn(0x7f0c700440b0 0x7f0c700464d0 crc :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7003e240 msgr2=0x7f0c70040700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f0c7003e240 0x7f0c70040700 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f0c90094990 tx=0x7f0c78009040 comp rx=0 tx=0).stop 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 10 monclient: shutdown 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 msgr2=0x7f0c9013afd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 --2- 172.21.15.164:0/3054285422 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f0c9009d070 0x7f0c9013afd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0c80009e80 tx=0x7f0c80004a40 comp rx=0 tx=0).stop 2024-09-10T16:32:19.090 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c8ed9d640 1 -- 172.21.15.164:0/3054285422 reap_dead start 2024-09-10T16:32:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 shutdown_connections 2024-09-10T16:32:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 >> 172.21.15.164:0/3054285422 conn(0x7f0c900a31f0 msgr2=0x7f0c90072060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 shutdown_connections 2024-09-10T16:32:19.091 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.088+0000 7f0c949f3640 1 -- 172.21.15.164:0/3054285422 wait complete. 2024-09-10T16:32:19.099 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-10T16:32:19.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.196+0000 7fc80e052640 10 monclient: get_monmap_and_config 2024-09-10T16:32:19.197 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.196+0000 7fc80e052640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.196+0000 7fc80e052640 10 monclient: monmap: 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:19.197087+0000 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:19.197087+0000 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:19.198 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:19.199 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:19.199 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 Processor -- start 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 -- start start 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient: init 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-b con 0x7fc80809f930 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:19.203 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-a con 0x7fc8080a2380 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc80813a5c0 0x7fc80813c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-c con 0x7fc80813a5c0 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc8080a2cc0 con 0x7fc8080a2380 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc80809fd10 con 0x7fc80813a5c0 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc80809cd20 con 0x7fc80809f930 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc80e052640 10 monclient(hunting): authenticate will time out at 2317.684561s 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.204 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:33124/0 (socket says 172.21.15.164:33124) 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 1 -- 172.21.15.164:0/2434624601 learned_addr learned my addr 172.21.15.164:0/2434624601 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): get_auth_request con 0x7fc80809f930 auth_method 0 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47936/0 (socket says 172.21.15.164:47936) 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): get_auth_request con 0x7fc8080a2380 auth_method 0 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.205 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.165:3300/0 conn(0x7fc80813a5c0 0x7fc80813c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): get_auth_request con 0x7fc80813a5c0 auth_method 0 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 10 monclient(hunting): handle_auth_done global_id 274385 payload 274 2024-09-10T16:32:19.206 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 1 -- 172.21.15.164:0/2434624601 >> v2:172.21.15.165:3300/0 conn(0x7fc80813a5c0 msgr2=0x7fc80813c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.197+0000 7fc8077fe640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.165:3300/0 conn(0x7fc80813a5c0 0x7fc80813c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 1 -- 172.21.15.164:0/2434624601 >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 msgr2=0x7fc8080a2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 1 monclient: found mon.noname-b 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc8077fe640 1 -- 172.21.15.164:0/2434624601 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc80809a0d0 con 0x7fc80809f930 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.198+0000 7fc7feffd640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc8077fe640 1 -- 172.21.15.164:0/2434624601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 msgr2=0x7fc8080a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc8077fe640 1 --2- 172.21.15.164:0/2434624601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc80809f5c0 tx=0x7fc7f802a0b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3023a575a7f9a924 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 1 -- 172.21.15.164:0/2434624601 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8037070 con 0x7fc80809f930 2024-09-10T16:32:19.207 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: dump: 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.208 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 1 -- 172.21.15.164:0/2434624601 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f8032640 con 0x7fc80809f930 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc80e052640 5 monclient: authenticate success, global_id 274385 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc80e052640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 1 -- 172.21.15.164:0/2434624601 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8032a10 con 0x7fc80809f930 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc80e052640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc80e052640 4 set_mon_vals no callback set 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.200+0000 7fc806ffd640 10 monclient: dump: 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.209 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 10 monclient: shutdown 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 -- 172.21.15.164:0/2434624601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 msgr2=0x7fc8080a1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 --2- 172.21.15.164:0/2434624601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc80809f5c0 tx=0x7fc7f802a0b0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 -- 172.21.15.164:0/2434624601 shutdown_connections 2024-09-10T16:32:19.210 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.165:3300/0 conn(0x7fc80813a5c0 0x7fc80813c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 --2- 172.21.15.164:0/2434624601 >> v2:172.21.15.164:3300/0 conn(0x7fc8080a2380 0x7fc8080a2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 --2- 172.21.15.164:0/2434624601 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80809f930 0x7fc8080a1e40 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 -- 172.21.15.164:0/2434624601 >> 172.21.15.164:0/2434624601 conn(0x7fc8080a3110 msgr2=0x7fc8080a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 -- 172.21.15.164:0/2434624601 shutdown_connections 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.201+0000 7fc80e052640 1 -- 172.21.15.164:0/2434624601 wait complete. 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.202+0000 7fc80e052640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.202+0000 7fc80e052640 10 monclient: monmap: 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.211 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:19.212 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:19.212 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:19.212 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.217 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.202+0000 7fc80e052640 1 Processor -- start 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.202+0000 7fc80e052640 1 -- start start 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.202+0000 7fc80e052640 10 monclient: init 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-a con 0x7fc80809f930 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8080a2380 0x7fc808099720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-b con 0x7fc8080a2380 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80813a5c0 0x7fc8081caca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.218 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): picked mon.noname-c con 0x7fc80813a5c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc80809d930 con 0x7fc80809f930 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc80809ee00 con 0x7fc8080a2380 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc80809c8b0 con 0x7fc80813a5c0 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc80e052640 10 monclient(hunting): authenticate will time out at 2317.690270s 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47946/0 (socket says 172.21.15.164:47946) 2024-09-10T16:32:19.219 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 -- 172.21.15.164:0/4187668611 learned_addr learned my addr 172.21.15.164:0/4187668611 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): get_auth_request con 0x7fc80809f930 auth_method 0 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80813a5c0 0x7fc8081caca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): get_auth_request con 0x7fc80813a5c0 auth_method 0 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.220 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8080a2380 0x7fc808099720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): get_auth_request con 0x7fc8080a2380 auth_method 0 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc807fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.221 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc7feffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient(hunting): handle_auth_done global_id 294213 payload 824 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 -- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80813a5c0 msgr2=0x7fc8081caca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80813a5c0 0x7fc8081caca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 -- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8080a2380 msgr2=0x7fc808099720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8080a2380 0x7fc808099720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 monclient: found mon.noname-a 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.203+0000 7fc8077fe640 1 -- 172.21.15.164:0/4187668611 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8080941b0 con 0x7fc80809f930 2024-09-10T16:32:19.222 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc8077fe640 1 -- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 msgr2=0x7fc8081c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc8077fe640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80090b0 tx=0x7fc7f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8041020 con 0x7fc80809f930 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: dump: 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.223 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f803d830 con 0x7fc80809f930 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc80e052640 5 monclient: authenticate success, global_id 294213 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.205+0000 7fc804ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _renew_subs 2024-09-10T16:32:19.224 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f8037040 con 0x7fc80809f930 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8080a1d10 con 0x7fc80809f930 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc804ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc804ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc804ff9640 10 monclient: dump: 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.225 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc8067fc640 4 set_mon_vals no callback set 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _renew_subs 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8080929b0 con 0x7fc80809f930 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc7f8044020 con 0x7fc80809f930 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.226 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.206+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8080a0760 con 0x7fc80809f930 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.207+0000 7fc804ff9640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 0x7fc7e80406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.207+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fc7f8073040 con 0x7fc80809f930 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.207+0000 7fc7feffd640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 0x7fc7e80406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.207+0000 7fc7feffd640 10 monclient: get_auth_request con 0x7fc7e803e210 auth_method 0 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.209+0000 7fc7feffd640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 0x7fc7e80406d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009560 tx=0x7fc7f0009190 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.211+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc7f804b030 con 0x7fc80809f930 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.211+0000 7fc804ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.227 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.211+0000 7fc804ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:19.385 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.384+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fc808091880 con 0x7fc7e803e210 2024-09-10T16:32:19.386 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.384+0000 7fc804ff9640 1 -- 172.21.15.164:0/4187668611 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fc808091880 con 0x7fc7e803e210 2024-09-10T16:32:19.386 INFO:teuthology.orchestra.run.smithi164.stderr:instructed osd(s) 5 to deep-scrub 2024-09-10T16:32:19.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 msgr2=0x7fc7e80406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 0x7fc7e80406d0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7fc7f0009560 tx=0x7fc7f0009190 comp rx=0 tx=0).stop 2024-09-10T16:32:19.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 10 monclient: shutdown 2024-09-10T16:32:19.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.388 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 msgr2=0x7fc8081c77a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc7f80090b0 tx=0x7fc7f80046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 shutdown_connections 2024-09-10T16:32:19.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc7e803e210 0x7fc7e80406d0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.389 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc80813a5c0 0x7fc8081caca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc8080a2380 0x7fc808099720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 --2- 172.21.15.164:0/4187668611 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc80809f930 0x7fc8081c77a0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 >> 172.21.15.164:0/4187668611 conn(0x7fc8080a3110 msgr2=0x7fc8081388f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.392 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 shutdown_connections 2024-09-10T16:32:19.393 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.386+0000 7fc80e052640 1 -- 172.21.15.164:0/4187668611 wait complete. 2024-09-10T16:32:19.399 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:32:19.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 10 monclient: get_monmap_and_config 2024-09-10T16:32:19.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.496 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 10 monclient: monmap: 2024-09-10T16:32:19.496 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.496 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:19.495944+0000 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:19.495944+0000 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:19.497 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.499 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 1 Processor -- start 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 1 -- start start 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.495+0000 7fc573789640 10 monclient: init 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient: _add_conns ranks=[3,4,0,1,2] 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc56c09c180 0x7fc56c09ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): picked mon.noname-a con 0x7fc56c09c180 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7fc56c09d160 0x7fc56c09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): picked mon.noname-c con 0x7fc56c09d160 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc56c09f4c0 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc56c09a430 con 0x7fc56c09f4c0 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.500 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7fc56c0a24e0 con 0x7fc56c09c180 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7fc56c0a2bf0 con 0x7fc56c09d160 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc573789640 10 monclient(hunting): authenticate will time out at 2317.983437s 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7fc56c09c180 0x7fc56c09ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571d26640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7fc56c09f4c0 0x7fc56c13b5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:49730/0 (socket says 172.21.15.164:49730) 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571d26640 1 -- 172.21.15.164:0/527166906 learned_addr learned my addr 172.21.15.164:0/527166906 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): get_auth_request con 0x7fc56c09c180 auth_method 0 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.501 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc570d24640 1 --2- 172.21.15.164:0/527166906 >> v2:172.21.15.165:3300/0 conn(0x7fc56c09d160 0x7fc56c09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc570d24640 10 monclient(hunting): get_auth_request con 0x7fc56c09d160 auth_method 0 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc570d24640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc570d24640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc570d24640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient(hunting): handle_auth_done global_id 294216 payload 274 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 -- 172.21.15.164:0/527166906 >> v2:172.21.15.165:3300/0 conn(0x7fc56c09d160 msgr2=0x7fc56c09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 --2- 172.21.15.164:0/527166906 >> v2:172.21.15.165:3300/0 conn(0x7fc56c09d160 0x7fc56c09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 -- 172.21.15.164:0/527166906 >> v1:172.21.15.164:6789/0 conn(0x7fc56c09f4c0 legacy=0x7fc56c13b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 monclient: found mon.noname-a 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.502 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.496+0000 7fc571525640 1 -- 172.21.15.164:0/527166906 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c09c560 con 0x7fc56c09c180 2024-09-10T16:32:19.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc571525640 1 -- 172.21.15.164:0/527166906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 msgr2=0x7fc56c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc571525640 1 --2- 172.21.15.164:0/527166906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c09ea40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc5680097a0 tx=0x7fc56802a010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3224402b0b514429 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 1 -- 172.21.15.164:0/527166906 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568037070 con 0x7fc56c09c180 2024-09-10T16:32:19.508 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: dump: 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.509 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc573789640 5 monclient: authenticate success, global_id 294216 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 1 -- 172.21.15.164:0/527166906 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc568032440 con 0x7fc56c09c180 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc573789640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc573789640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc573789640 4 set_mon_vals no callback set 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 1 -- 172.21.15.164:0/527166906 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568032810 con 0x7fc56c09c180 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.499+0000 7fc563fff640 10 monclient: dump: 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.510 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 10 monclient: shutdown 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 -- 172.21.15.164:0/527166906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 msgr2=0x7fc56c09ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 --2- 172.21.15.164:0/527166906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c09ea40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc5680097a0 tx=0x7fc56802a010 comp rx=0 tx=0).stop 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 -- 172.21.15.164:0/527166906 shutdown_connections 2024-09-10T16:32:19.511 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 --2- 172.21.15.164:0/527166906 >> v2:172.21.15.165:3300/0 conn(0x7fc56c09d160 0x7fc56c09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 --2- 172.21.15.164:0/527166906 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c09ea40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 -- 172.21.15.164:0/527166906 >> 172.21.15.164:0/527166906 conn(0x7fc56c0a3110 msgr2=0x7fc56c0a3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 -- 172.21.15.164:0/527166906 shutdown_connections 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 1 -- 172.21.15.164:0/527166906 wait complete. 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 10 monclient: build_initial_monmap 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.500+0000 7fc573789640 10 monclient: monmap: 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.512 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 Processor -- start 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 -- start start 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient: init 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): picked mon.noname-a con 0x7fc56c09c180 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56c09d160 0x7fc56c136f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): picked mon.noname-b con 0x7fc56c09d160 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc56c09f4c0 0x7fc56c071410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.513 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): picked mon.noname-c con 0x7fc56c09f4c0 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c074f90 con 0x7fc56c09c180 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7fc56c074e10 con 0x7fc56c09d160 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7fc56c13cce0 con 0x7fc56c09f4c0 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc573789640 10 monclient(hunting): authenticate will time out at 2317.988761s 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:47968/0 (socket says 172.21.15.164:47968) 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 1 -- 172.21.15.164:0/3418520870 learned_addr learned my addr 172.21.15.164:0/3418520870 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): get_auth_request con 0x7fc56c09c180 auth_method 0 2024-09-10T16:32:19.514 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571d26640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc56c09f4c0 0x7fc56c071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571525640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571d26640 10 monclient(hunting): get_auth_request con 0x7fc56c09f4c0 auth_method 0 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571d26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571d26640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.501+0000 7fc571d26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571d26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571d26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571d26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:19.515 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 10 monclient(hunting): handle_auth_done global_id 294219 payload 824 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 -- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc56c09f4c0 msgr2=0x7fc56c071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc56c09f4c0 0x7fc56c071410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 -- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56c09d160 msgr2=0x7fc56c136f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56c09d160 0x7fc56c136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 10 monclient: _finish_hunting 0 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 monclient: found mon.noname-a 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.502+0000 7fc571525640 1 -- 172.21.15.164:0/3418520870 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc56c0a5050 con 0x7fc56c09c180 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc571525640 1 -- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 msgr2=0x7fc56c1369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc571525640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568041020 con 0x7fc56c09c180 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.516 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: dump: 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:19.517 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 5 monclient: authenticate success, global_id 294219 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc56803d8f0 con 0x7fc56c09c180 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 10 monclient: _renew_subs 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc56c073c30 con 0x7fc56c09c180 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc568037040 con 0x7fc56c09c180 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc561ffb640 10 monclient: dump: 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:19.518 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc5637fe640 4 set_mon_vals no callback set 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 10 monclient: _renew_subs 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.504+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc56c072710 con 0x7fc56c09c180 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7fc568044020 con 0x7fc56c09c180 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc573789640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.519 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc573789640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc573789640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc561ffb640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 0x7fc550040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc56c0008d0 con 0x7fc56c09c180 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.505+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7fc568073200 con 0x7fc56c09c180 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.506+0000 7fc570d24640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 0x7fc550040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:19.520 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.506+0000 7fc570d24640 10 monclient: get_auth_request con 0x7fc55003e260 auth_method 0 2024-09-10T16:32:19.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.508+0000 7fc570d24640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 0x7fc550040720 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc56c09e830 tx=0x7fc55c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:19.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.510+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7fc56804b030 con 0x7fc56c09c180 2024-09-10T16:32:19.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.510+0000 7fc561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:19.521 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.510+0000 7fc561ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:19.682 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.681+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc56c137470 con 0x7fc55003e260 2024-09-10T16:32:19.683 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.682+0000 7fc561ffb640 1 -- 172.21.15.164:0/3418520870 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34028 (secure 0 0 0) 0x7fc56c137470 con 0x7fc55003e260 2024-09-10T16:32:19.684 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:19.684 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:32:19.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 msgr2=0x7fc550040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 0x7fc550040720 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fc56c09e830 tx=0x7fc55c008040 comp rx=0 tx=0).stop 2024-09-10T16:32:19.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 10 monclient: shutdown 2024-09-10T16:32:19.686 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 msgr2=0x7fc56c1369f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.684+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc568000c00 tx=0x7fc5680046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 shutdown_connections 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7fc55003e260 0x7fc550040720 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7fc56c09f4c0 0x7fc56c071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7fc56c09d160 0x7fc56c136f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 --2- 172.21.15.164:0/3418520870 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7fc56c09c180 0x7fc56c1369f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 >> 172.21.15.164:0/3418520870 conn(0x7fc56c0a3110 msgr2=0x7fc56c099230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 shutdown_connections 2024-09-10T16:32:19.687 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:19.685+0000 7fc573789640 1 -- 172.21.15.164:0/3418520870 wait complete. 2024-09-10T16:32:19.696 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1675,"stamp":"2024-09-10T16:32:17.689572+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":290736,"kb_used_data":9272,"kb_used_omap":9,"kb_used_meta":281334,"kb_avail":565940304,"statfs":{"total":579820584960,"available":579522871296,"internally_reserved":0,"allocated":9494528,"data_stored":5105380,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":288086691},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002677"},"pg_stats":[{"pgid":"1.0","version":"15'72","reported_seq":1840,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326999+0000","last_change":"2024-09-10T16:06:16.273506+0000","last_active":"2024-09-10T16:32:02.326999+0000","last_peered":"2024-09-10T16:32:02.326999+0000","last_clean":"2024-09-10T16:32:02.326999+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:32:02.326999+0000","last_undegraded":"2024-09-10T16:32:02.326999+0000","last_fullsized":"2024-09-10T16:32:02.326999+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:14.366957+0000","last_clean_scrub_stamp":"2024-09-10T16:06:14.366957+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:19:43.590460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1749,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326946+0000","last_change":"2024-09-10T16:06:24.428654+0000","last_active":"2024-09-10T16:32:02.326946+0000","last_peered":"2024-09-10T16:32:02.326946+0000","last_clean":"2024-09-10T16:32:02.326946+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:32:02.326946+0000","last_undegraded":"2024-09-10T16:32:02.326946+0000","last_fullsized":"2024-09-10T16:32:02.326946+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:40:31.289747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1757,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326957+0000","last_change":"2024-09-10T16:06:24.427754+0000","last_active":"2024-09-10T16:32:02.326957+0000","last_peered":"2024-09-10T16:32:02.326957+0000","last_clean":"2024-09-10T16:32:02.326957+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:32:02.326957+0000","last_undegraded":"2024-09-10T16:32:02.326957+0000","last_fullsized":"2024-09-10T16:32:02.326957+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:04:50.763828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1769,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.188520+0000","last_change":"2024-09-10T16:06:24.430219+0000","last_active":"2024-09-10T16:32:01.188520+0000","last_peered":"2024-09-10T16:32:01.188520+0000","last_clean":"2024-09-10T16:32:01.188520+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:32:01.188520+0000","last_undegraded":"2024-09-10T16:32:01.188520+0000","last_fullsized":"2024-09-10T16:32:01.188520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:36:31.328302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1767,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326850+0000","last_change":"2024-09-10T16:06:24.432116+0000","last_active":"2024-09-10T16:32:02.326850+0000","last_peered":"2024-09-10T16:32:02.326850+0000","last_clean":"2024-09-10T16:32:02.326850+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:32:02.326850+0000","last_undegraded":"2024-09-10T16:32:02.326850+0000","last_fullsized":"2024-09-10T16:32:02.326850+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:32:42.728193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.326638+0000","last_change":"2024-09-10T16:06:24.520508+0000","last_active":"2024-09-10T16:32:02.326638+0000","last_peered":"2024-09-10T16:32:02.326638+0000","last_clean":"2024-09-10T16:32:02.326638+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:32:02.326638+0000","last_undegraded":"2024-09-10T16:32:02.326638+0000","last_fullsized":"2024-09-10T16:32:02.326638+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:17:34.094664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1747,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327123+0000","last_change":"2024-09-10T16:06:24.428235+0000","last_active":"2024-09-10T16:32:02.327123+0000","last_peered":"2024-09-10T16:32:02.327123+0000","last_clean":"2024-09-10T16:32:02.327123+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:32:02.327123+0000","last_undegraded":"2024-09-10T16:32:02.327123+0000","last_fullsized":"2024-09-10T16:32:02.327123+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T16:51:31.242732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1730,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:01.187848+0000","last_change":"2024-09-10T16:06:24.437270+0000","last_active":"2024-09-10T16:32:01.187848+0000","last_peered":"2024-09-10T16:32:01.187848+0000","last_clean":"2024-09-10T16:32:01.187848+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:32:01.187848+0000","last_undegraded":"2024-09-10T16:32:01.187848+0000","last_fullsized":"2024-09-10T16:32:01.187848+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:40:28.629696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1746,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:02.327125+0000","last_change":"2024-09-10T16:06:24.428562+0000","last_active":"2024-09-10T16:32:02.327125+0000","last_peered":"2024-09-10T16:32:02.327125+0000","last_clean":"2024-09-10T16:32:02.327125+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:32:02.327125+0000","last_undegraded":"2024-09-10T16:32:02.327125+0000","last_fullsized":"2024-09-10T16:32:02.327125+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:06:21.408405+0000","last_clean_scrub_stamp":"2024-09-10T16:06:21.408405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:12.072849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575166,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575165,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28612,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343228,"statfs":{"total":96636764160,"available":96607465472,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607867,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41092,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330748,"statfs":{"total":96636764160,"available":96594685952,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607868,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46600,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325240,"statfs":{"total":96636764160,"available":96589045760,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607869,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50764,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321076,"statfs":{"total":96636764160,"available":96584781824,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640572,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72904,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298936,"statfs":{"total":96636764160,"available":96562110464,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:32:19.700 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-10T16:06:14.366957+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=14, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.700 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-10T16:06:21.408405+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=6, tm_sec=21, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=16, tm_min=32, tm_sec=15, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T16:32:19.701 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-10T16:32:39.702 DEBUG:teuthology.orchestra.run.smithi164:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T16:32:39.813 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 10 monclient: get_monmap_and_config 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 10 monclient: build_initial_monmap 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 10 monclient: monmap: 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:39.812940+0000 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:39.812940+0000 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:39.814 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:39.815 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:39.815 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 1 Processor -- start 2024-09-10T16:32:39.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 1 -- start start 2024-09-10T16:32:39.815 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.812+0000 7f9de8cb0640 10 monclient: init 2024-09-10T16:32:39.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:39.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient: _add_conns ranks=[4,0,3,1,2] 2024-09-10T16:32:39.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 1 --2- >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 0x7f9de4137e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f9de4074af0 addr v2:172.21.15.165:3300/0 2024-09-10T16:32:39.817 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9de40a1090 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9de40a1500 0x7f9de40a35d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f9de40a1500 addr v2:172.21.15.164:3300/0 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9de409ebc0 con 0x7f9de40a1090 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 1 -- --> v2:172.21.15.164:3300/0 -- mon_getmap magic: 0 -- 0x7f9de40698a0 con 0x7f9de40a1500 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 1 -- --> v2:172.21.15.165:3300/0 -- mon_getmap magic: 0 -- 0x7f9de406bfb0 con 0x7f9de4074af0 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:39.818 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de8cb0640 10 monclient(hunting): authenticate will time out at 2338.300406s 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 --2- >> v2:172.21.15.164:3300/0 conn(0x7f9de40a1500 0x7f9de40a35d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de1d9b640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f9de40a1090 0x7f9de40a2ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:56306/0 (socket says 172.21.15.164:56306) 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de1d9b640 1 -- 172.21.15.164:0/125382054 learned_addr learned my addr 172.21.15.164:0/125382054 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): get_auth_request con 0x7f9de40a1500 auth_method 0 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de259c640 1 --2- 172.21.15.164:0/125382054 >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 0x7f9de4137e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient(hunting): handle_auth_done global_id 294222 payload 274 2024-09-10T16:32:39.819 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 -- 172.21.15.164:0/125382054 >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 msgr2=0x7f9de4137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 --2- 172.21.15.164:0/125382054 >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 0x7f9de4137e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 -- 172.21.15.164:0/125382054 >> v1:172.21.15.164:6789/0 conn(0x7f9de40a1090 legacy=0x7f9de40a2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient: _finish_hunting 0 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 monclient: found mon.noname-a 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de2d9d640 1 -- 172.21.15.164:0/125382054 --> v2:172.21.15.164:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de40a1950 con 0x7f9de40a1500 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de259c640 10 monclient: get_auth_request con 0x7f9de4074af0 auth_method 0 2024-09-10T16:32:39.820 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.813+0000 7f9de259c640 1 --2- 172.21.15.164:0/125382054 >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 0x7f9de4137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T16:32:39.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.816+0000 7f9de2d9d640 1 -- 172.21.15.164:0/125382054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de40a1500 msgr2=0x7f9de40a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:39.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.816+0000 7f9de2d9d640 1 --2- 172.21.15.164:0/125382054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de40a1500 0x7f9de40a35d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de40a3ed0 tx=0x7f9ddc02a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b984f429a962dda1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:39.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 1 -- 172.21.15.164:0/125382054 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc037070 con 0x7f9de40a1500 2024-09-10T16:32:39.826 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: dump: 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.827 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: _finish_auth 0 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: _check_auth_tickets 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 5 monclient: authenticate success, global_id 294222 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 1 -- 172.21.15.164:0/125382054 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ddc032640 con 0x7f9de40a1500 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 4 set_mon_vals no callback set 2024-09-10T16:32:39.828 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 1 -- 172.21.15.164:0/125382054 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ddc032a10 con 0x7f9de40a1500 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de159a640 10 monclient: dump: 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:39.829 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 10 monclient: shutdown 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 -- 172.21.15.164:0/125382054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de40a1500 msgr2=0x7f9de40a35d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/125382054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de40a1500 0x7f9de40a35d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9de40a3ed0 tx=0x7f9ddc02a000 comp rx=0 tx=0).stop 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 -- 172.21.15.164:0/125382054 shutdown_connections 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/125382054 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de40a1500 0x7f9de40a35d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/125382054 >> v2:172.21.15.165:3300/0 conn(0x7f9de4074af0 0x7f9de4137e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 -- 172.21.15.164:0/125382054 >> 172.21.15.164:0/125382054 conn(0x7f9de40a3fb0 msgr2=0x7f9de40a43c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 -- 172.21.15.164:0/125382054 shutdown_connections 2024-09-10T16:32:39.830 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.817+0000 7f9de8cb0640 1 -- 172.21.15.164:0/125382054 wait complete. 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.818+0000 7f9de8cb0640 10 monclient: build_initial_monmap 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.818+0000 7f9de8cb0640 10 monclient: monmap: 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:39.831 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.818+0000 7f9de8cb0640 1 Processor -- start 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.818+0000 7f9de8cb0640 1 -- start start 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.818+0000 7f9de8cb0640 10 monclient: init 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-a con 0x7f9de4074af0 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 0x7f9de41bdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-c con 0x7f9de40a1090 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:39.832 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de40a1500 0x7f9de41be200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): picked mon.noname-b con 0x7f9de40a1500 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f9de4069370 con 0x7f9de4074af0 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f9de4074ed0 con 0x7f9de40a1500 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f9de40a3b10 con 0x7f9de40a1090 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de8cb0640 10 monclient(hunting): authenticate will time out at 2338.306385s 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3300/0 says I am v2:172.21.15.164:56724/0 (socket says 172.21.15.164:56724) 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 -- 172.21.15.164:0/1619399890 learned_addr learned my addr 172.21.15.164:0/1619399890 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:39.833 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): get_auth_request con 0x7f9de4074af0 auth_method 0 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 0x7f9de41bdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): get_auth_request con 0x7f9de40a1090 auth_method 0 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:39.834 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de2d9d640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de40a1500 0x7f9de41be200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de2d9d640 10 monclient(hunting): get_auth_request con 0x7f9de40a1500 auth_method 0 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de2d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de2d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de2d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient(hunting): handle_auth_done global_id 294225 payload 824 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 -- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 msgr2=0x7f9de41bdcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 0x7f9de41bdcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 -- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de40a1500 msgr2=0x7f9de41be200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de40a1500 0x7f9de41be200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient: _finish_hunting 0 2024-09-10T16:32:39.835 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 monclient: found mon.noname-a 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.164:3300/0 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de259c640 1 -- 172.21.15.164:0/1619399890 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9de41c1970 con 0x7f9de4074af0 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.819+0000 7f9de1d9b640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 0x7f9de41bdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de259c640 1 -- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 msgr2=0x7f9de409a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de259c640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400cf60 tx=0x7f9dd400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd401a070 con 0x7f9de4074af0 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: dump: 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:39.836 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: _finish_auth 0 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: _check_auth_tickets 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 5 monclient: authenticate success, global_id 294225 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9dd400da50 con 0x7f9de4074af0 2024-09-10T16:32:39.837 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dd400de20 con 0x7f9de4074af0 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 10 monclient: _renew_subs 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9de40a5360 con 0x7f9de4074af0 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9ddaffd640 10 monclient: dump: 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:39.838 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de0d99640 4 set_mon_vals no callback set 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 10 monclient: _renew_subs 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.822+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9de41c1690 con 0x7f9de4074af0 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de8cb0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de8cb0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de8cb0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.164:3300/0 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9de41321c0 con 0x7f9de4074af0 2024-09-10T16:32:39.839 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f9dd401d070 con 0x7f9de4074af0 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9ddaffd640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 0x7f9dc40406d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f9dd4053410 con 0x7f9de4074af0 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de1d9b640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 0x7f9dc40406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.823+0000 7f9de1d9b640 10 monclient: get_auth_request con 0x7f9dc403e210 auth_method 0 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.826+0000 7f9de1d9b640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 0x7f9dc40406d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007c90 tx=0x7f9dcc007be0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:39.840 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.828+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mon.0 v2:172.21.15.164:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f9dd4051fa0 con 0x7f9de4074af0 2024-09-10T16:32:39.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.828+0000 7f9ddaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:39.841 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.828+0000 7f9ddaffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:40.000 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:39.999+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 --> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9de4137e70 con 0x7f9dc403e210 2024-09-10T16:32:40.001 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.000+0000 7f9ddaffd640 1 -- 172.21.15.164:0/1619399890 <== mgr.4102 v2:172.21.15.165:6824/3840871261 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34045 (secure 0 0 0) 0x7f9de4137e70 con 0x7f9dc403e210 2024-09-10T16:32:40.002 INFO:teuthology.orchestra.run.smithi164.stdout: 2024-09-10T16:32:40.002 INFO:teuthology.orchestra.run.smithi164.stderr:dumped all 2024-09-10T16:32:40.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 msgr2=0x7f9dc40406d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.003 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 0x7f9dc40406d0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f9dcc007c90 tx=0x7f9dcc007be0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 10 monclient: shutdown 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 msgr2=0x7f9de409a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f9dd400cf60 tx=0x7f9dd400cf90 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 shutdown_connections 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f9dc403e210 0x7f9dc40406d0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f9de40a1500 0x7f9de41be200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f9de40a1090 0x7f9de41bdcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 --2- 172.21.15.164:0/1619399890 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f9de4074af0 0x7f9de409a770 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 >> 172.21.15.164:0/1619399890 conn(0x7f9de40a3fb0 msgr2=0x7f9de41328e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 shutdown_connections 2024-09-10T16:32:40.004 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.002+0000 7f9de8cb0640 1 -- 172.21.15.164:0/1619399890 wait complete. 2024-09-10T16:32:40.014 INFO:teuthology.orchestra.run.smithi164.stdout:{"pg_ready":true,"pg_map":{"version":1686,"stamp":"2024-09-10T16:32:39.694178+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":79,"ondisk_log_size":79,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":294872,"kb_used_data":9272,"kb_used_omap":9,"kb_used_meta":285494,"kb_avail":565936168,"statfs":{"total":579820584960,"available":579518636032,"internally_reserved":0,"allocated":9494528,"data_stored":5105380,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9565,"internal_metadata":292346531},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002505"},"pg_stats":[{"pgid":"1.0","version":"897'74","reported_seq":1850,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:17.867622+0000","last_change":"2024-09-10T16:32:17.867622+0000","last_active":"2024-09-10T16:32:17.867622+0000","last_peered":"2024-09-10T16:32:17.867622+0000","last_clean":"2024-09-10T16:32:17.867622+0000","last_became_active":"2024-09-10T16:06:16.272384+0000","last_became_peered":"2024-09-10T16:06:16.272384+0000","last_unstale":"2024-09-10T16:32:17.867622+0000","last_undegraded":"2024-09-10T16:32:17.867622+0000","last_fullsized":"2024-09-10T16:32:17.867622+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'74","last_scrub_stamp":"2024-09-10T16:32:17.867551+0000","last_deep_scrub":"897'74","last_deep_scrub_stamp":"2024-09-10T16:32:17.867551+0000","last_clean_scrub_stamp":"2024-09-10T16:32:17.867551+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:52:41.718672+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"897'2","reported_seq":1758,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:18.838091+0000","last_change":"2024-09-10T16:32:18.838091+0000","last_active":"2024-09-10T16:32:18.838091+0000","last_peered":"2024-09-10T16:32:18.838091+0000","last_clean":"2024-09-10T16:32:18.838091+0000","last_became_active":"2024-09-10T16:06:22.421712+0000","last_became_peered":"2024-09-10T16:06:22.421712+0000","last_unstale":"2024-09-10T16:32:18.838091+0000","last_undegraded":"2024-09-10T16:32:18.838091+0000","last_fullsized":"2024-09-10T16:32:18.838091+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'2","last_scrub_stamp":"2024-09-10T16:32:18.838038+0000","last_deep_scrub":"897'2","last_deep_scrub_stamp":"2024-09-10T16:32:18.838038+0000","last_clean_scrub_stamp":"2024-09-10T16:32:18.838038+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:52:24.668157+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00050161300000000004,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1766,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:18.653328+0000","last_change":"2024-09-10T16:32:18.653283+0000","last_active":"2024-09-10T16:32:18.653328+0000","last_peered":"2024-09-10T16:32:18.653328+0000","last_clean":"2024-09-10T16:32:18.653328+0000","last_became_active":"2024-09-10T16:06:22.421698+0000","last_became_peered":"2024-09-10T16:06:22.421698+0000","last_unstale":"2024-09-10T16:32:18.653328+0000","last_undegraded":"2024-09-10T16:32:18.653328+0000","last_fullsized":"2024-09-10T16:32:18.653328+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:18.653213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:18.653213+0000","last_clean_scrub_stamp":"2024-09-10T16:32:18.653213+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:47.507632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00056707999999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1777,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:19.270249+0000","last_change":"2024-09-10T16:32:19.270249+0000","last_active":"2024-09-10T16:32:19.270249+0000","last_peered":"2024-09-10T16:32:19.270249+0000","last_clean":"2024-09-10T16:32:19.270249+0000","last_became_active":"2024-09-10T16:06:22.421649+0000","last_became_peered":"2024-09-10T16:06:22.421649+0000","last_unstale":"2024-09-10T16:32:19.270249+0000","last_undegraded":"2024-09-10T16:32:19.270249+0000","last_fullsized":"2024-09-10T16:32:19.270249+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:19.270160+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:19.270160+0000","last_clean_scrub_stamp":"2024-09-10T16:32:19.270160+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:17:43.624458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043761199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"897'3","reported_seq":1776,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:17.753200+0000","last_change":"2024-09-10T16:32:17.753200+0000","last_active":"2024-09-10T16:32:17.753200+0000","last_peered":"2024-09-10T16:32:17.753200+0000","last_clean":"2024-09-10T16:32:17.753200+0000","last_became_active":"2024-09-10T16:06:22.420958+0000","last_became_peered":"2024-09-10T16:06:22.420958+0000","last_unstale":"2024-09-10T16:32:17.753200+0000","last_undegraded":"2024-09-10T16:32:17.753200+0000","last_fullsized":"2024-09-10T16:32:17.753200+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"897'3","last_scrub_stamp":"2024-09-10T16:32:17.753112+0000","last_deep_scrub":"897'3","last_deep_scrub_stamp":"2024-09-10T16:32:17.753112+0000","last_clean_scrub_stamp":"2024-09-10T16:32:17.753112+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T18:05:20.214124+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0012952300000000001,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1738,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:19.448776+0000","last_change":"2024-09-10T16:32:19.448776+0000","last_active":"2024-09-10T16:32:19.448776+0000","last_peered":"2024-09-10T16:32:19.448776+0000","last_clean":"2024-09-10T16:32:19.448776+0000","last_became_active":"2024-09-10T16:06:22.422779+0000","last_became_peered":"2024-09-10T16:06:22.422779+0000","last_unstale":"2024-09-10T16:32:19.448776+0000","last_undegraded":"2024-09-10T16:32:19.448776+0000","last_fullsized":"2024-09-10T16:32:19.448776+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:19.448673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:19.448673+0000","last_clean_scrub_stamp":"2024-09-10T16:32:19.448673+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:00:31.083754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037338600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1755,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:19.823178+0000","last_change":"2024-09-10T16:32:19.823178+0000","last_active":"2024-09-10T16:32:19.823178+0000","last_peered":"2024-09-10T16:32:19.823178+0000","last_clean":"2024-09-10T16:32:19.823178+0000","last_became_active":"2024-09-10T16:06:22.421769+0000","last_became_peered":"2024-09-10T16:06:22.421769+0000","last_unstale":"2024-09-10T16:32:19.823178+0000","last_undegraded":"2024-09-10T16:32:19.823178+0000","last_fullsized":"2024-09-10T16:32:19.823178+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:19.823081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:19.823081+0000","last_clean_scrub_stamp":"2024-09-10T16:32:19.823081+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:12.637479+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00060586199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1739,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:19.076966+0000","last_change":"2024-09-10T16:32:19.076913+0000","last_active":"2024-09-10T16:32:19.076966+0000","last_peered":"2024-09-10T16:32:19.076966+0000","last_clean":"2024-09-10T16:32:19.076966+0000","last_became_active":"2024-09-10T16:06:22.420962+0000","last_became_peered":"2024-09-10T16:06:22.420962+0000","last_unstale":"2024-09-10T16:32:19.076966+0000","last_undegraded":"2024-09-10T16:32:19.076966+0000","last_fullsized":"2024-09-10T16:32:19.076966+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:19.076812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:19.076812+0000","last_clean_scrub_stamp":"2024-09-10T16:32:19.076812+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:06:13.520005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041148900000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1755,"reported_epoch":897,"state":"active+clean","last_fresh":"2024-09-10T16:32:20.842044+0000","last_change":"2024-09-10T16:32:20.841982+0000","last_active":"2024-09-10T16:32:20.842044+0000","last_peered":"2024-09-10T16:32:20.842044+0000","last_clean":"2024-09-10T16:32:20.842044+0000","last_became_active":"2024-09-10T16:06:22.421891+0000","last_became_peered":"2024-09-10T16:06:22.421891+0000","last_unstale":"2024-09-10T16:32:20.842044+0000","last_undegraded":"2024-09-10T16:32:20.842044+0000","last_fullsized":"2024-09-10T16:32:20.842044+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T16:32:20.841888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T16:32:20.841888+0000","last_clean_scrub_stamp":"2024-09-10T16:32:20.841888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:27:45.198300+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.00089635099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":59,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59114,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":4,"up_from":13,"seq":55834575170,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54920,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316920,"statfs":{"total":96636764160,"available":96580526080,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834575169,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28608,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343232,"statfs":{"total":96636764160,"available":96607469568,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":12,"seq":51539607872,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41088,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330752,"statfs":{"total":96636764160,"available":96594690048,"internally_reserved":0,"allocated":1622016,"data_stored":1203509,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607873,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46596,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325244,"statfs":{"total":96636764160,"available":96589049856,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607874,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50760,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94321080,"statfs":{"total":96636764160,"available":96584785920,"internally_reserved":0,"allocated":1564672,"data_stored":674600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640576,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72900,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":71358,"kb_avail":94298940,"statfs":{"total":96636764160,"available":96562114560,"internally_reserved":0,"allocated":1560576,"data_stored":674581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":73071047},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29557,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T16:32:40.016 DEBUG:teuthology.orchestra.run.smithi164:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-10T16:32:40.114 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.113+0000 7f743927c640 10 monclient: get_monmap_and_config 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.113+0000 7f743927c640 10 monclient: build_initial_monmap 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.113+0000 7f743927c640 10 monclient: monmap: 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:32:40.114327+0000 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:32:40.114327+0000 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:0: v1:172.21.15.164:6789/0 mon.noname-a-legacy 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:1: v1:172.21.15.165:6789/0 mon.noname-c-legacy 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-b 2024-09-10T16:32:40.115 INFO:teuthology.orchestra.run.smithi164.stderr:3: v2:172.21.15.164:3300/0 mon.noname-a 2024-09-10T16:32:40.116 INFO:teuthology.orchestra.run.smithi164.stderr:4: v2:172.21.15.165:3300/0 mon.noname-c 2024-09-10T16:32:40.116 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.113+0000 7f743927c640 1 Processor -- start 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 1 -- start start 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient: init 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): picked mon.noname-b con 0x7f74340a4520 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74340a66d0 addr v1:172.21.15.165:6789/0 2024-09-10T16:32:40.118 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7434138310 addr v1:172.21.15.164:6789/0 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 1 -- --> v1:172.21.15.164:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74340a64c0 con 0x7f7434138310 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 1 -- --> v1:172.21.15.165:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743409c9f0 con 0x7f74340a66d0 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f743409ccb0 con 0x7f74340a4520 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f743927c640 10 monclient(hunting): authenticate will time out at 2338.601564s 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f74337fe640 1 --1- >> v1:172.21.15.164:6789/0 conn(0x7f7434138310 0x7f743413ffa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.164:6789/0 says I am v1:172.21.15.164:56314/0 (socket says 172.21.15.164:56314) 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): get_auth_request con 0x7f74340a4520 auth_method 0 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f74337fe640 1 -- 172.21.15.164:0/342734397 learned_addr learned my addr 172.21.15.164:0/342734397 (peer_addr_for_me v1:172.21.15.164:0/0) 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:40.119 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient(hunting): handle_auth_done global_id 274397 payload 274 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 1 -- 172.21.15.164:0/342734397 >> v1:172.21.15.165:6789/0 conn(0x7f74340a66d0 legacy=0x7f7434137c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 1 -- 172.21.15.164:0/342734397 >> v1:172.21.15.164:6789/0 conn(0x7f7434138310 legacy=0x7f743413ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 1 monclient: found mon.noname-b 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.120 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.114+0000 7f7432ffd640 1 -- 172.21.15.164:0/342734397 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74340a7200 con 0x7f74340a4520 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.117+0000 7f7432ffd640 1 -- 172.21.15.164:0/342734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 msgr2=0x7f74340a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.117+0000 7f7432ffd640 1 --2- 172.21.15.164:0/342734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a4900 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74240097d0 tx=0x7f7424029fb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae4d2b675f874266 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 1 -- 172.21.15.164:0/342734397 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424037070 con 0x7f74340a4520 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: dump: 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.132 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: _finish_auth 0 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: _check_auth_tickets 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 1 -- 172.21.15.164:0/342734397 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7424032440 con 0x7f74340a4520 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 5 monclient: authenticate success, global_id 274397 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T16:32:40.133 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 10 monclient: get_monmap_and_config success 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 1 -- 172.21.15.164:0/342734397 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424032810 con 0x7f74340a4520 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 4 set_mon_vals no callback set 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f7431ffb640 10 monclient: dump: 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.134 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 10 monclient: shutdown 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 -- 172.21.15.164:0/342734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 msgr2=0x7f74340a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 --2- 172.21.15.164:0/342734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a4900 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f74240097d0 tx=0x7f7424029fb0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 -- 172.21.15.164:0/342734397 shutdown_connections 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 --2- 172.21.15.164:0/342734397 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a4900 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 -- 172.21.15.164:0/342734397 >> 172.21.15.164:0/342734397 conn(0x7f74340aac50 msgr2=0x7f74340ab060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 -- 172.21.15.164:0/342734397 shutdown_connections 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.118+0000 7f743927c640 1 -- 172.21.15.164:0/342734397 wait complete. 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.119+0000 7f743927c640 10 monclient: build_initial_monmap 2024-09-10T16:32:40.135 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.119+0000 7f743927c640 10 monclient: monmap: 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 0 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 0.000000 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:created 0.000000 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 0 (unknown) 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.noname-a 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.noname-b 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.noname-c 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.119+0000 7f743927c640 1 Processor -- start 2024-09-10T16:32:40.136 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.119+0000 7f743927c640 1 -- start start 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.119+0000 7f743927c640 10 monclient: init 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient: _reopen_session rank -1 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): picked mon.noname-c con 0x7f74340a4520 addr [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 --2- >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74340a66d0 0x7f74340a3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): picked mon.noname-b con 0x7f74340a66d0 addr [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 0x7f74341c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): picked mon.noname-a con 0x7f7434138310 addr [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 -- --> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] -- mon_getmap magic: 0 -- 0x7f74340a4fc0 con 0x7f7434138310 2024-09-10T16:32:40.137 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 -- --> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] -- mon_getmap magic: 0 -- 0x7f74340a4e40 con 0x7f74340a66d0 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): start opening mon connection 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 1 -- --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_getmap magic: 0 -- 0x7f7434134930 con 0x7f74340a4520 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): _renew_subs 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f743927c640 10 monclient(hunting): authenticate will time out at 2338.607214s 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 --2- >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.164:3301/0 says I am v2:172.21.15.164:54008/0 (socket says 172.21.15.164:54008) 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 -- 172.21.15.164:0/412272777 learned_addr learned my addr 172.21.15.164:0/412272777 (peer_addr_for_me v2:172.21.15.164:0/0) 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): get_auth_request con 0x7f74340a4520 auth_method 0 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:40.138 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 1 --2- >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 0x7f74341c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): get_auth_request con 0x7f7434138310 auth_method 0 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74327fc640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74340a66d0 0x7f74340a3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74327fc640 10 monclient(hunting): get_auth_request con 0x7f74340a66d0 auth_method 0 2024-09-10T16:32:40.139 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74327fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient(hunting): handle_auth_done global_id 274400 payload 824 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 -- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74340a66d0 msgr2=0x7f74340a3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74340a66d0 0x7f74340a3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 -- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 msgr2=0x7f74341c9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 0x7f74341c9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient: _finish_hunting 0 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 monclient: found mon.noname-c 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.140 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f7432ffd640 1 -- 172.21.15.164:0/412272777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7434137e40 con 0x7f74340a4520 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.120+0000 7f74337fe640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 0x7f74341c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.122+0000 7f7432ffd640 1 -- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 msgr2=0x7f74340a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.122+0000 7f7432ffd640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7424009720 tx=0x7f74240046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.122+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424041070 con 0x7f74340a4520 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: dump: 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:40.141 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: _finish_auth 0 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: _check_auth_tickets 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f742403c660 con 0x7f74340a4520 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 5 monclient: authenticate success, global_id 274400 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 10 monclient: _renew_subs 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743406e8b0 con 0x7f74340a4520 2024-09-10T16:32:40.142 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7424037040 con 0x7f74340a4520 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f742b7fe640 10 monclient: dump: 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:epoch 1 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:fsid b134de91-f819-41d3-9bbd-da58ec0ed6be 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:last_changed 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:created 2024-09-10T16:05:30.502390+0000 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:min_mon_release 19 (squid) 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:election_strategy: 1 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:0: [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] mon.a 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:1: [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] mon.b 2024-09-10T16:32:40.143 INFO:teuthology.orchestra.run.smithi164.stderr:2: [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] mon.c 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 10 monclient: _renew_subs 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f74317fa640 4 set_mon_vals no callback set 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.123+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f74340a2500 con 0x7f74340a4520 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f743927c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f743927c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f743927c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7434051000 con 0x7f74340a4520 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 4 ==== mgrmap(e 5) ==== 50609+0+0 (secure 0 0 0) 0x7f7424044070 con 0x7f74340a4520 2024-09-10T16:32:40.144 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f742b7fe640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 0x7f7414040720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 5 ==== osd_map(897..897 src has 869..897) ==== 4374+0+0 (secure 0 0 0) 0x7f7424073230 con 0x7f74340a4520 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f74327fc640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 0x7f7414040720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.124+0000 7f74327fc640 10 monclient: get_auth_request con 0x7f741403e260 auth_method 0 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.127+0000 7f74327fc640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 0x7f7414040720 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f74340a4de0 tx=0x7f741c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.128+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+195771 (secure 0 0 0) 0x7f7434051000 con 0x7f74340a4520 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.128+0000 7f742b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T16:32:40.145 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.128+0000 7f742b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T16:32:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.303+0000 7f743927c640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T16:32:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.303+0000 7f743927c640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T16:32:40.304 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.303+0000 7f743927c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.164:3301/0 2024-09-10T16:32:40.305 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.303+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 --> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f74341388c0 con 0x7f74340a4520 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f742404ec00 con 0x7f74340a4520 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f742b7fe640 10 monclient: handle_config config(1 keys) 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f74317fa640 4 set_mon_vals no callback set 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f74317fa640 10 set_mon_vals mon_health_to_clog = false 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f742b7fe640 1 -- 172.21.15.164:0/412272777 <== mon.2 v2:172.21.15.164:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (secure 0 0 0) 0x7f742403d3f0 con 0x7f74340a4520 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f742b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T16:32:40.311 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.309+0000 7f742b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 msgr2=0x7f7414040720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 0x7f7414040720 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f74340a4de0 tx=0x7f741c009040 comp rx=0 tx=0).stop 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 10 monclient: shutdown 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 msgr2=0x7f74340a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7424009720 tx=0x7f74240046c0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 shutdown_connections 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:6824/3840871261,v1:172.21.15.165:6825/3840871261] conn(0x7f741403e260 0x7f7414040720 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3300/0,v1:172.21.15.164:6789/0] conn(0x7f7434138310 0x7f74341c9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.165:3300/0,v1:172.21.15.165:6789/0] conn(0x7f74340a66d0 0x7f74340a3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.314 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 --2- 172.21.15.164:0/412272777 >> [v2:172.21.15.164:3301/0,v1:172.21.15.164:6790/0] conn(0x7f74340a4520 0x7f74340a3a30 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T16:32:40.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 >> 172.21.15.164:0/412272777 conn(0x7f74340aac50 msgr2=0x7f74340aa040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T16:32:40.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 shutdown_connections 2024-09-10T16:32:40.315 INFO:teuthology.orchestra.run.smithi164.stderr:2024-09-10T16:32:40.312+0000 7f743927c640 1 -- 172.21.15.164:0/412272777 wait complete. 2024-09-10T16:32:40.324 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-10T16:32:40.324 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-10T16:32:40.325 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T16:32:40.325 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.359 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T16:32:40.359 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T16:32:40.359 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.433 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T16:32:40.433 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-10T16:32:40.433 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.504 INFO:tasks.ceph.osd.2:Stopped 2024-09-10T16:32:40.504 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T16:32:40.504 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.540 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T16:32:40.540 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-10T16:32:40.540 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.609 INFO:tasks.ceph.osd.4:Stopped 2024-09-10T16:32:40.610 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-10T16:32:40.610 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.680 INFO:tasks.ceph.osd.5:Stopped 2024-09-10T16:32:40.680 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-10T16:32:40.681 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-10T16:32:40.681 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.750 INFO:tasks.ceph.mgr.x:Stopped 2024-09-10T16:32:40.750 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-10T16:32:40.750 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T16:32:40.750 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.761 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T16:32:40.761 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T16:32:40.762 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.814 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T16:32:40.814 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T16:32:40.814 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T16:32:40.826 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T16:32:40.826 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-10T16:32:40.827 DEBUG:teuthology.orchestra.run.smithi164:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2024-09-10T16:32:40.887 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:32:40.887 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-10T16:32:41.100 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:32:41.100 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-10T16:32:41.253 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:32:41.254 DEBUG:teuthology.orchestra.run.smithi164:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-10T16:32:41.304 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:32:41.304 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-10T16:32:41.449 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:32:41.449 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-10T16:32:41.544 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:32:41.544 DEBUG:teuthology.orchestra.run.smithi165:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-10T16:32:41.600 INFO:tasks.ceph:Archiving mon data... 2024-09-10T16:32:41.602 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/data/mon.a.tgz 2024-09-10T16:32:41.602 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-09-10T16:32:41.618 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.BqcX99WCyz 2024-09-10T16:32:41.619 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.BqcX99WCyz 2024-09-10T16:32:41.836 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.BqcX99WCyz 2024-09-10T16:32:41.919 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.BqcX99WCyz is 872KB 2024-09-10T16:32:42.023 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.BqcX99WCyz 2024-09-10T16:32:42.039 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/data/mon.c.tgz 2024-09-10T16:32:42.039 DEBUG:teuthology.orchestra.run.smithi164:> mktemp 2024-09-10T16:32:42.093 INFO:teuthology.orchestra.run.smithi164.stdout:/tmp/tmp.zkgTCXr4Mb 2024-09-10T16:32:42.093 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.zkgTCXr4Mb 2024-09-10T16:32:42.283 DEBUG:teuthology.orchestra.run.smithi164:> sudo chmod 0666 /tmp/tmp.zkgTCXr4Mb 2024-09-10T16:32:42.367 DEBUG:teuthology.orchestra.remote:smithi164:/tmp/tmp.zkgTCXr4Mb is 873KB 2024-09-10T16:32:42.470 DEBUG:teuthology.orchestra.run.smithi164:> rm -fr /tmp/tmp.zkgTCXr4Mb 2024-09-10T16:32:42.487 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/data/mon.b.tgz 2024-09-10T16:32:42.487 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-10T16:32:42.508 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.pl5DAal4ir 2024-09-10T16:32:42.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.pl5DAal4ir 2024-09-10T16:32:42.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.pl5DAal4ir 2024-09-10T16:32:42.833 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.pl5DAal4ir is 876KB 2024-09-10T16:32:42.939 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.pl5DAal4ir 2024-09-10T16:32:42.955 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-10T16:32:42.955 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T16:32:42.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T16:32:43.037 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T16:32:43.076 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T16:32:43.076 INFO:tasks.ceph:Archiving crash dumps... 2024-09-10T16:32:43.077 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi164/crash 2024-09-10T16:32:43.078 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T16:32:43.103 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi165/crash 2024-09-10T16:32:43.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T16:32:43.140 INFO:tasks.ceph:Compressing logs... 2024-09-10T16:32:43.141 DEBUG:teuthology.orchestra.run.smithi164:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T16:32:43.148 DEBUG:teuthology.orchestra.run.smithi165:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T16:32:43.170 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-10T16:32:43.170 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46256.log 2024-09-10T16:32:43.219 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32020.log 2024-09-10T16:32:43.219 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31980.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.34841.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32020.log: /var/log/ceph/ceph-client.admin.31980.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.32020.log.gz 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.0.34841.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31980.log.gz 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31940.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31860.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.31940.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31900.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.31860.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31940.log.gz 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T16:32:43.220 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.31900.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.31860.log.gz 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.31900.log.gz 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32071.log 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 90.0% -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-client.admin.32071.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-10T16:32:43.221 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-mon.b.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32071.log.gz 2024-09-10T16:32:43.237 INFO:teuthology.orchestra.run.smithi165.stderr:/var/log/ceph/ceph-osd.3.log: 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T16:32:43.295 INFO:teuthology.orchestra.run.smithi165.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.0.34841.log.gz 2024-09-10T16:32:43.411 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42130.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.0.log: /var/log/ceph/ceph-client.admin.46256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51660.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53555.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46256.log.gz 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51660.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41318.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.42130.log.gz 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38050.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51660.log.gz 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43630.log 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53555.log.gz 2024-09-10T16:32:43.412 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41318.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44631.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38050.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43630.log: /var/log/ceph/ceph-client.admin.41444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43039.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50971.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43630.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43039.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44631.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44362.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43039.log.gz 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54735.log 2024-09-10T16:32:43.413 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50971.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38092.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39227.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44362.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54017.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54735.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35309.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38092.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48762.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54017.log: /var/log/ceph/ceph-client.admin.35309.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.39227.log.gz 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48762.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.54017.log.gz -5 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.43672.log 2024-09-10T16:32:43.414 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35309.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54525.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43672.log: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.48762.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47249.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54525.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43672.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57211.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42560.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54525.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52616.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42560.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.47249.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45387.log 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57211.log.gz 2024-09-10T16:32:43.415 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40570.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52616.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.42560.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55840.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41950.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45387.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52616.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50845.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40570.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40315.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.55840.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50845.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47711.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.41950.log.gz 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50761.log 2024-09-10T16:32:43.416 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40315.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.50845.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53513.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.40315.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38218.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47711.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50677.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.50761.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38218.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48493.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50677.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53891.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38218.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 93.3% -- replaced with /var/log/ceph/ceph-client.admin.53513.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48493.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57127.log 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.50677.log.gz 2024-09-10T16:32:43.417 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39832.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53244.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48493.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53891.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47118.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.57127.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53244.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37377.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39832.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38470.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.53244.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39496.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47118.log.gz 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40612.log 2024-09-10T16:32:43.418 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56664.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0%/var/log/ceph/ceph-client.admin.40612.log: -- replaced with /var/log/ceph/ceph-client.admin.38470.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.37377.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39058.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40654.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56664.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.40612.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46718.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40654.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.56664.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39058.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr: 93.0% -- replaced with /var/log/ceph/ceph-client.admin.39496.log.gz 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46718.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54059.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55798.log 2024-09-10T16:32:43.419 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40654.log.gz 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5/var/log/ceph/ceph-client.admin.54059.log: --verbose -- /var/log/ceph/ceph-client.admin.46676.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46718.log.gz 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55798.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42172.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.54059.log.gzgzip 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56184.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46676.log: /var/log/ceph/ceph-client.admin.42172.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.55798.log.gz 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55321.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56184.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.46676.log.gz -5 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49504.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42172.log.gz 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45751.log 2024-09-10T16:32:43.420 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56184.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53639.log 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.55321.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36685.log 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49504.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56142.log 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45751.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57337.log 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.53639.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55195.log 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.36685.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57337.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56142.log.gz 2024-09-10T16:32:43.421 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43504.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51576.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57337.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47669.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55195.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51576.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43504.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56058.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47669.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54651.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51576.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41234.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47669.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43941.log 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54651.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56058.log.gz 2024-09-10T16:32:43.422 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41234.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46298.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43941.log: gzip 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54651.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.44967.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46298.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.41234.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.43941.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41488.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45836.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44967.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46298.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41488.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54483.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37109.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.44967.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41488.log.gz 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34831.log 2024-09-10T16:32:43.423 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.45836.log.gz 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37109.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.54483.log.gz 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.34831.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41402.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37109.log.gz 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48678.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.34831.log.gz 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57631.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53389.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.41402.log.gz 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57631.log: gzip -5 --verbose 89.8% -- -- replaced with /var/log/ceph/ceph-client.admin.48678.log.gz /var/log/ceph/ceph-client.admin.36052.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36983.log 2024-09-10T16:32:43.424 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57631.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53389.log: /var/log/ceph/ceph-client.admin.36052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50027.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36983.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53389.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50719.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.36052.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46550.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.36983.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.50027.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56268.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50719.log.gz 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48972.log 2024-09-10T16:32:43.425 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41656.log: /var/log/ceph/ceph-client.admin.56268.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.46550.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40001.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44883.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56268.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41056.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.48972.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44883.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.40001.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56100.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54399.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41056.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44883.log.gz 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57673.log 2024-09-10T16:32:43.426 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41056.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54399.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.56100.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52124.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42088.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57673.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54399.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37193.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42088.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57673.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46844.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37193.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.52124.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39874.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.42088.log.gz 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56396.log 2024-09-10T16:32:43.427 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.37193.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39874.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49462.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46844.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49210.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56396.log: /var/log/ceph/ceph-client.admin.49462.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.39874.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55756.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56396.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45513.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49462.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47291.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49210.log.gz 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49378.log 2024-09-10T16:32:43.428 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.55756.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48325.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.45513.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35853.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47291.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36262.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48325.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49378.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50365.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48325.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38596.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36262.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35853.log.gz 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44589.log 2024-09-10T16:32:43.429 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36262.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49714.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50365.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44589.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44151.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38596.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35804.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44589.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39142.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.49714.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44151.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57463.log 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35804.log.gz 2024-09-10T16:32:43.430 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52950.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.39142.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45794.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52950.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35543.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57463.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52502.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52950.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40483.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35543.log: /var/log/ceph/ceph-client.admin.52502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40169.log 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.45794.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0%/var/log/ceph/ceph-client.admin.40483.log: -- replaced with /var/log/ceph/ceph-client.admin.35543.log.gz 2024-09-10T16:32:43.431 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55489.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52502.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35541.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40483.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47585.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.40169.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45962.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55489.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39706.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35541.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38848.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45962.log: /var/log/ceph/ceph-client.admin.39706.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47753.log 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.47585.log.gz 2024-09-10T16:32:43.432 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38848.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45962.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36324.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39706.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40930.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47753.log: /var/log/ceph/ceph-client.admin.36324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46466.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.38848.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47753.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36324.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56226.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46466.log: /var/log/ceph/ceph-client.admin.41614.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40930.log.gz 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51534.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53849.log 2024-09-10T16:32:43.433 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.46466.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32144.log 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53849.log: gzip -5 --verbose -- 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51534.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.56310.log 91.5% -- replaced with /var/log/ceph/ceph-client.admin.56226.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.32144.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36515.log 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53849.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56310.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32144.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40738.log 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46088.log 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36515.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56310.log.gz 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49336.log 2024-09-10T16:32:43.434 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46088.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.36515.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47627.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.40738.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49336.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46088.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43084.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49336.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47627.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35856.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50155.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43084.log: /var/log/ceph/ceph-client.admin.35856.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.47627.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35544.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43084.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57713.log 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.35856.log.gz 2024-09-10T16:32:43.435 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44465.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35544.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50155.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49672.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.35544.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38344.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44465.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57713.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48720.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44465.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38344.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48199.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49672.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39538.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48720.log: /var/log/ceph/ceph-client.admin.48199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42476.log 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38344.log.gz 2024-09-10T16:32:43.436 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39538.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41782.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48720.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.48199.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51954.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39538.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44404.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42476.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35716.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41782.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44404.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51954.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38764.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5%/var/log/ceph/ceph.audit.log: -- replaced with /var/log/ceph/ceph-client.admin.44404.log.gz 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36941.log 2024-09-10T16:32:43.437 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35716.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47795.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36941.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57001.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38764.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51998.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.36941.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45597.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51998.log: 90.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37277.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47795.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57001.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39453.log 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37277.log: gzip -5 --verbose 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45597.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.51998.log.gz 2024-09-10T16:32:43.438 INFO:teuthology.orchestra.run.smithi164.stderr: -- /var/log/ceph/ceph-client.admin.41530.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39453.log: 94.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50533.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.37277.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49099.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.39453.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50533.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48888.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49099.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43336.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50533.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.48888.log: -- /var/log/ceph/ceph-client.admin.49014.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49099.log.gz 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38512.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42004.log 2024-09-10T16:32:43.439 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48888.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49014.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.43336.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38512.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52866.log 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49057.log 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42004.log: 90.8% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.38512.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49014.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52866.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44109.log 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42004.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57547.log 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52866.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44109.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49057.log.gz 2024-09-10T16:32:43.440 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55069.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56959.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40399.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.44109.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.57547.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56959.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57421.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39100.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.55069.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44757.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56959.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39311.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49630.log 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39100.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.57421.log.gz 2024-09-10T16:32:43.441 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51744.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49630.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39100.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55363.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.39311.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51744.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49630.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51491.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47501.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55363.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.51744.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53807.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51491.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55363.log.gz 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46886.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47501.log: /var/log/ceph/ceph-client.admin.53807.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54820.log 2024-09-10T16:32:43.442 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.51491.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.47501.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51223.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48115.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.53807.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54820.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46886.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50113.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51912.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54820.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48115.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51223.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49943.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40230.log 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51912.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.48115.log.gz/var/log/ceph/ceph-client.admin.49943.log: 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50113.log.gz 2024-09-10T16:32:43.443 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38176.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51912.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37504.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49943.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37672.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40230.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52292.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.38176.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36389.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37504.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52292.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37151.log 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.37672.log.gz 2024-09-10T16:32:43.444 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55714.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52292.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35564.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36389.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48157.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.37151.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35564.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35311.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48157.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55714.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51449.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35564.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/ceph-client.admin.35311.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.34993.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48157.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47375.log 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35311.log.gz 2024-09-10T16:32:43.445 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41276.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.34993.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51449.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36599.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41276.log: 88.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.34993.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.42770.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47375.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36599.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55279.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41276.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42770.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41908.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55279.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42770.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48283.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.36599.log.gz 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32073.log 2024-09-10T16:32:43.446 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41908.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55279.log.gz 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51265.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41908.log.gz 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.32073.log: gzip 89.3% -5 -- replaced with /var/log/ceph/ceph-client.admin.48283.log.gz --verbose 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr: -- /var/log/ceph/ceph-client.admin.43546.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.32073.log.gz 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43546.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54101.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51265.log.gz 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42262.log: gzip 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43546.log.gz -5 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42686.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54101.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46130.log 2024-09-10T16:32:43.447 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45429.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46130.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54101.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37067.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42686.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54778.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.46130.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36136.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.45429.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50595.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.37067.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54882.log 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54778.log.gz 2024-09-10T16:32:43.448 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36136.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54882.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50595.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40525.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42518.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41698.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54882.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56580.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42518.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40525.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47013.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47942.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56580.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42518.log.gz 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.47013.log: /var/log/ceph/ceph-mon.c.log 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47942.log: gzip 88.7% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.56580.log.gz -- 2024-09-10T16:32:43.449 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.37966.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47013.log.gz 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.57379.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47942.log.gz 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51307.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37966.log: /var/log/ceph/ceph-client.admin.57379.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52782.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7%/var/log/ceph/ceph-client.admin.51307.log: -- replaced with /var/log/ceph/ceph-client.admin.37966.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48930.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57379.log.gz 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37924.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51307.log.gz 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43983.log 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52782.log.gz 2024-09-10T16:32:43.450 INFO:teuthology.orchestra.run.smithi164.stderr:gzip/var/log/ceph/ceph-client.admin.37924.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.36643.log 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43983.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48930.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36178.log 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.37924.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36643.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.43983.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46172.log 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.36178.log: /var/log/ceph/ceph-client.admin.35307.log 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36643.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37798.log 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.36178.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49756.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.46172.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37798.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35307.log.gz 2024-09-10T16:32:43.451 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56707.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53470.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.37798.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40972.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49756.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53470.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53328.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56707.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55671.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53470.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52908.log 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.40972.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55671.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.53328.log.gz 2024-09-10T16:32:43.452 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40780.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52908.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55671.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48804.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40780.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52908.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34912.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48804.log: gzip 89.8% -- replaced with /var/log/ceph/ceph-client.admin.40780.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51368.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.34912.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38386.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48804.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53681.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.34912.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38386.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.51368.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52697.log 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53681.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.38386.log.gz 2024-09-10T16:32:43.453 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52376.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52697.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.53681.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36431.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52376.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55932.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.52697.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36431.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49546.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52376.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54188.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49546.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.36431.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51181.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55932.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42896.log 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.49546.log.gz 2024-09-10T16:32:43.454 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47333.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54188.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42896.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.36773.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.51181.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57085.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.42896.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36773.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47333.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45093.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38806.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36773.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42602.log 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.57085.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38806.log: gzip 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45093.log.gz 2024-09-10T16:32:43.455 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37588.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45051.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38806.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51139.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.42602.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35044.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37588.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51139.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45051.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37630.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52418.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35044.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51139.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55629.log 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35044.log.gz 2024-09-10T16:32:43.456 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52418.log: gzip -5 --verbose -- 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37630.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-mon.a.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50491.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52418.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-mon.a.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55629.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57295.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55027.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46929.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50491.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50407.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.57295.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5/var/log/ceph/ceph-client.admin.46929.log: --verbose -- /var/log/ceph/ceph-client.admin.39269.log 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.55027.log.gz 2024-09-10T16:32:43.457 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55237.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46929.log.gz -5 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50407.log.gz 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39269.log: /var/log/ceph/ceph-client.admin.55237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44025.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47837.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.39269.log.gz 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.55237.log.gz 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44025.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56455.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44025.log.gz 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36220.log 2024-09-10T16:32:43.458 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.47837.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56455.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41740.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36220.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43252.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41740.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.36220.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42812.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43252.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41740.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37714.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42812.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43252.log.gz 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53597.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52208.log 2024-09-10T16:32:43.459 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.42812.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53597.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.37714.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39748.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.52208.log: -- 89.9% /var/log/ceph/ceph-client.admin.54143.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53597.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46214.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.52208.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47459.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39748.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54143.log: /var/log/ceph/ceph-client.admin.46214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46046.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46508.log 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46214.log.gz 2024-09-10T16:32:43.460 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46046.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54143.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.47459.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43294.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46508.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46046.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53160.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38554.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46508.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53160.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51013.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43294.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45555.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr: 88.1%/var/log/ceph/ceph-client.admin.38554.log: -- replaced with /var/log/ceph/ceph-client.admin.53160.log.gz 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46592.log 2024-09-10T16:32:43.461 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48846.log 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.51013.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46592.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.38554.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.45555.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52250.log 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48846.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.46592.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46802.log 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52250.log: gzip -5 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48846.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.48636.log 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36010.log 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52250.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48636.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.46802.log.gz 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52166.log 2024-09-10T16:32:43.462 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36010.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48636.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40273.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52166.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.36010.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35736.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40273.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52166.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35736.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40273.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51702.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-osd.2.log: gzip 88.9% -- replaced with /var/log/ceph/ceph-client.admin.35736.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.36473.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56833.log 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36473.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51702.log.gz 2024-09-10T16:32:43.463 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35968.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36473.log.gz 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.41572.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35968.log: /var/log/ceph/ceph-client.admin.56833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54357.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41572.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35968.log.gz 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48535.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.56833.log.gz 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6%/var/log/ceph/ceph-client.admin.54357.log: -- replaced with /var/log/ceph/ceph-client.admin.41572.log.gz 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51828.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 90.8% -- replaced with /var/log/ceph/ceph-client.admin.54357.log.gz -5 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.38260.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48535.log: gzip/var/log/ceph/ceph-client.admin.51828.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48241.log 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38260.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.51828.log.gz 2024-09-10T16:32:43.464 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.48535.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.38260.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45659.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56875.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44320.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48241.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45659.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.49985.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44320.log: /var/log/ceph/ceph-client.admin.56875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35170.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49985.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.56875.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.44320.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.49985.log.gz 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36557.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36815.log 2024-09-10T16:32:43.465 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36557.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35170.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38008.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- 89.0% -- replaced with /var/log/ceph/ceph-client.admin.36557.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.42046.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38008.log: /var/log/ceph/ceph-client.admin.36815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51055.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42046.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38008.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.36815.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42046.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.56791.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56917.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56791.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51055.log.gz 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37840.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45177.log 2024-09-10T16:32:43.466 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.56791.log.gz/var/log/ceph/ceph-client.admin.56917.log: 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53933.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56917.log.gz 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45177.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.37840.log.gz 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52544.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45177.log.gz -5 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40357.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53933.log: gzip/var/log/ceph/ceph-client.admin.52544.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.38428.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40357.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53933.log.gz 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.52544.log.gz 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.40357.log.gz 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46634.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38890.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38428.log: gzip/var/log/ceph/ceph-client.admin.46634.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.43210.log 2024-09-10T16:32:43.467 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38890.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.46634.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.38428.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.38890.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43126.log 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46760.log 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35308.log 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43126.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43210.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46760.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49294.log 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43126.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35308.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.46760.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49420.log 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35308.log.gz 2024-09-10T16:32:43.468 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46971.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49294.log: /var/log/ceph/ceph-client.admin.49420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50887.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46971.log: 88.7% 89.8% -- replaced with /var/log/ceph/ceph-client.admin.49294.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49420.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52040.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.46971.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37882.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50887.log: /var/log/ceph/ceph-client.admin.52040.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37882.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50887.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.52040.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45303.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.37882.log.gz 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35816.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.45303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40085.log 2024-09-10T16:32:43.469 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35816.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45303.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35816.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36857.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40085.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35537.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36857.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.40085.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.36857.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43733.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35537.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44067.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43733.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.35537.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48023.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43733.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40696.log 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48023.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44067.log.gz 2024-09-10T16:32:43.470 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37025.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40696.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48023.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38638.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37025.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.37025.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38302.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.admin.40696.log.gz/var/log/ceph/ceph-client.admin.38638.log: 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49798.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38638.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53202.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49798.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.38302.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43899.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49798.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37546.log 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43899.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53202.log.gz 2024-09-10T16:32:43.471 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56749.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr: 90.3%/var/log/ceph/ceph-client.admin.37546.log: -- replaced with /var/log/ceph/ceph-client.admin.43899.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.37546.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.54963.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44673.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54963.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56749.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52460.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6%/var/log/ceph/ceph-client.admin.44673.log: -- replaced with /var/log/ceph/ceph-client.admin.54963.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44673.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51870.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39958.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51870.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52460.log.gz 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50929.log 2024-09-10T16:32:43.472 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39958.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.51870.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54567.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39958.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51097.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54567.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50929.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54567.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37420.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52082.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51097.log: /var/log/ceph/ceph-client.admin.37420.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51097.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.37420.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52824.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54231.log 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52082.log: /var/log/ceph/ceph-client.admin.52824.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.52082.log.gz 2024-09-10T16:32:43.473 INFO:teuthology.orchestra.run.smithi164.stderr: 90.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45009.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.52824.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47879.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45009.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54231.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50239.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47879.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.45009.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49861.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.47879.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50239.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36094.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49861.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50239.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48367.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49861.log.gz 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-09-10T16:32:43.474 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.36094.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54273.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42304.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.48367.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42958.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57169.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54273.log: /var/log/ceph/ceph-client.admin.42958.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54273.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.42958.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56536.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54693.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57169.log: /var/log/ceph/ceph-client.admin.56536.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57169.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56536.log.gz 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57253.log 2024-09-10T16:32:43.475 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31844.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54693.log: /var/log/ceph/ceph-client.admin.57253.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54693.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.57253.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47543.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48451.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47543.log: /var/log/ceph/ceph-client.admin.31844.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47543.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.31844.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38932.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52992.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38932.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.48451.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.38932.log.gz 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47055.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56622.log 2024-09-10T16:32:43.476 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8%/var/log/ceph/ceph-client.admin.47055.log: -- replaced with /var/log/ceph/ceph-client.admin.52992.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35542.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56622.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.47055.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43420.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56622.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44715.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43420.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.35542.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37756.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43420.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45471.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr: 91.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37756.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44715.log.gzgzip 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.43588.log 2024-09-10T16:32:43.477 INFO:teuthology.orchestra.run.smithi164.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41866.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37756.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45471.log: /var/log/ceph/ceph-client.admin.43588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51618.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41866.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45471.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.43588.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55531.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41866.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50281.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51618.log: /var/log/ceph/ceph-client.admin.55531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35086.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50281.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51618.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.55531.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38680.log 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50281.log.gz 2024-09-10T16:32:43.478 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40043.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35086.log: gzip/var/log/ceph/ceph-client.admin.38680.log: -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.27461.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40043.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35086.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.38680.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.40043.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40886.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41192.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph.tmp-client.admin.27461.log: /var/log/ceph/ceph-client.admin.40886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56016.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.27461.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50197.log 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56016.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.41192.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.40886.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.56016.log.gz 2024-09-10T16:32:43.479 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46382.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50197.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57589.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41098.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50197.log.gz 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36729.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57589.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46382.log.gz 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38974.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57589.log.gz 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36729.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41098.log.gz 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44925.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39580.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.36729.log.gz 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38974.log: /var/log/ceph/ceph-client.admin.44925.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46004.log 2024-09-10T16:32:43.480 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39580.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.38974.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44925.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45878.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.39580.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43462.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46004.log: /var/log/ceph/ceph-client.admin.45878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43462.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.46004.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.45878.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55111.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.43462.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41824.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42430.log: /var/log/ceph/ceph-client.admin.55111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42644.log 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41824.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-09-10T16:32:43.481 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41824.log.gz 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49252.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45135.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.55111.log.gz 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.49252.log: /var/log/ceph/ceph-client.admin.42644.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55890.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49252.log.gz/var/log/ceph/ceph-client.admin.45135.log: 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.42644.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55573.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9%/var/log/ceph/ceph-client.admin.55890.log: -- replaced with /var/log/ceph/ceph-client.admin.45135.log.gz 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39016.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55890.log.gz 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52740.log 2024-09-10T16:32:43.482 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55573.log: /var/log/ceph/ceph-client.admin.39016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38134.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52740.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55573.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.39016.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45345.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52740.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54315.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.38134.log: /var/log/ceph/ceph-client.admin.45345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39372.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54315.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.38134.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.45345.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54315.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43378.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44193.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39372.log: gzip/var/log/ceph/ceph-client.admin.43378.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53034.log 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44193.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.43378.log.gz 2024-09-10T16:32:43.483 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39372.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.44193.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39184.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50323.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.53034.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42388.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39184.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53765.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53118.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53765.log: 89.5% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50323.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53765.log.gz 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr: /var/log/ceph/ceph-client.admin.55447.log 2024-09-10T16:32:43.484 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50449.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55447.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.53118.log.gz 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38722.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 88.3% -5 --verbose -- /var/log/ceph/ceph-client.admin.44235.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.55447.log.gz 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50449.log: /var/log/ceph/ceph-client.admin.38722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41014.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44235.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.38722.log.gz 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43168.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr: 90.5% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44235.log.gz 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50449.log.gz 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41014.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51786.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43168.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.41014.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45261.log 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.485 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42854.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.43168.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.51786.log: /var/log/ceph/ceph-client.admin.45261.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45261.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51786.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45219.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54609.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45219.log: 89.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53286.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42854.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.45219.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.48073.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54609.log: /var/log/ceph/ceph-client.admin.53286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50803.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% 88.2%/var/log/ceph/ceph-client.admin.48073.log: -- replaced with /var/log/ceph/ceph-client.admin.54609.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.53286.log.gz 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42220.log 2024-09-10T16:32:43.486 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39622.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48073.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47206.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.50803.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39622.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50071.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.42220.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57505.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.50071.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.39622.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.47206.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36323.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.50071.log.gz 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48409.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.57505.log: /var/log/ceph/ceph-client.admin.36323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42728.log 2024-09-10T16:32:43.487 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.48409.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.57505.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.36323.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57043.log 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.42728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56354.log 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48409.log.gz/var/log/ceph/ceph-client.admin.57043.log: 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.42728.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55153.log 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57043.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39916.log 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.56354.log: /var/log/ceph/ceph-client.admin.55153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35310.log 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39916.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.56354.log.gz 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55153.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr:gzip 89.5% -- replaced with /var/log/ceph/ceph-client.admin.39916.log.gz 2024-09-10T16:32:43.488 INFO:teuthology.orchestra.run.smithi164.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52334.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.35310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49588.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.52334.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35310.log.gz 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43857.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54441.log 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52334.log.gz 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43857.log: /var/log/ceph/ceph-client.admin.49588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40441.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.49588.log.gz 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43857.log.gz 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.54441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46340.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.40441.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54441.log.gz 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37462.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45920.log 2024-09-10T16:32:43.489 INFO:teuthology.orchestra.run.smithi164.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.40441.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.37462.log: /var/log/ceph/ceph-client.admin.46340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43814.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.45920.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.37462.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.46340.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34968.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.45920.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.43814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44278.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.34968.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43814.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35312.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41360.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.34968.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.44278.log: gzip/var/log/ceph/ceph-client.admin.35312.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.53975.log 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.41360.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.35312.log.gz 2024-09-10T16:32:43.490 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44278.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53076.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53975.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.41360.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35128.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47417.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53975.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53076.log: /var/log/ceph/ceph-client.admin.35128.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39790.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.47417.log: 88.8% 90.3% -- replaced with /var/log/ceph/ceph-client.admin.53076.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35128.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44547.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.47417.log.gz 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36899.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.39790.log: /var/log/ceph/ceph-client.admin.44547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37235.log 2024-09-10T16:32:43.491 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.36899.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.39790.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.44547.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46424.log 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.36899.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53723.log 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.46424.log: /var/log/ceph/ceph-client.admin.37235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55405.log 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.53723.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46424.log.gz 89.6% -- replaced with /var/log/ceph/ceph-client.admin.37235.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55974.log 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53723.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr:/var/log/ceph/ceph-client.admin.55405.log: /var/log/ceph/ceph-client.admin.55974.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55405.log.gz 2024-09-10T16:32:43.492 INFO:teuthology.orchestra.run.smithi164.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.55974.log.gz 2024-09-10T16:32:43.493 INFO:teuthology.orchestra.run.smithi165.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-10T16:32:43.877 INFO:teuthology.orchestra.run.smithi165.stderr: 91.0% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-10T16:32:44.016 INFO:teuthology.orchestra.run.smithi164.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-10T16:32:44.324 INFO:teuthology.orchestra.run.smithi164.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-10T16:32:45.631 INFO:teuthology.orchestra.run.smithi165.stderr: 91.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-10T16:32:46.041 INFO:teuthology.orchestra.run.smithi164.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-10T16:32:47.385 INFO:teuthology.orchestra.run.smithi165.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-10T16:32:48.182 INFO:teuthology.orchestra.run.smithi164.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-10T16:32:48.927 INFO:teuthology.orchestra.run.smithi165.stderr: 92.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-10T16:32:48.929 INFO:teuthology.orchestra.run.smithi165.stderr: 2024-09-10T16:32:48.929 INFO:teuthology.orchestra.run.smithi165.stderr:real 0m5.732s 2024-09-10T16:32:48.930 INFO:teuthology.orchestra.run.smithi165.stderr:user 0m12.911s 2024-09-10T16:32:48.930 INFO:teuthology.orchestra.run.smithi165.stderr:sys 0m0.454s 2024-09-10T16:32:49.507 INFO:teuthology.orchestra.run.smithi164.stderr: 92.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-10T16:32:49.509 INFO:teuthology.orchestra.run.smithi164.stderr: 2024-09-10T16:32:49.509 INFO:teuthology.orchestra.run.smithi164.stderr:real 0m6.349s 2024-09-10T16:32:49.509 INFO:teuthology.orchestra.run.smithi164.stderr:user 0m15.654s 2024-09-10T16:32:49.510 INFO:teuthology.orchestra.run.smithi164.stderr:sys 0m0.942s 2024-09-10T16:32:49.510 INFO:tasks.ceph:Archiving logs... 2024-09-10T16:32:49.510 DEBUG:teuthology.misc:Transferring archived files from smithi164:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi164/log 2024-09-10T16:32:49.511 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T16:32:52.493 DEBUG:teuthology.misc:Transferring archived files from smithi165:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi165/log 2024-09-10T16:32:52.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T16:32:54.747 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T16:32:54.756 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T16:32:54.757 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T16:32:54.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T16:32:54.854 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T16:32:54.863 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T16:32:54.864 DEBUG:teuthology.orchestra.run.smithi164:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T16:32:54.867 DEBUG:teuthology.orchestra.run.smithi165:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T16:32:54.882 INFO:teuthology.orchestra.run.smithi164.stderr:bash: line 1: ntpq: command not found 2024-09-10T16:32:54.883 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:=============================================================================== 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:^* hv01.front.sepia.ceph.com 3 6 377 32 +416us[ +416us] +/- 43ms 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 32 -79us[ -79us] +/- 55ms 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -752us[ -752us] +/- 87ms 2024-09-10T16:32:54.886 INFO:teuthology.orchestra.run.smithi164.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:^* hv01.front.sepia.ceph.com 3 6 377 31 +420us[ +418us] +/- 43ms 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -75us[ -75us] +/- 55ms 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 -748us[ -748us] +/- 87ms 2024-09-10T16:32:54.888 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-10T16:32:54.888 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T16:32:54.908 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T16:32:54.909 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T16:32:54.967 DEBUG:teuthology.orchestra.run.smithi164:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T16:32:55.044 DEBUG:teuthology.orchestra.run.smithi165:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-10T16:32:55.110 DEBUG:teuthology.orchestra.run.smithi164:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T16:32:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:32:55.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 2024-09-10T16:32:55.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:32:55.169 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T16:32:55.178 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T16:32:55.234 INFO:teuthology.task.internal:Duration was 2008.277034 seconds 2024-09-10T16:32:55.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T16:32:55.304 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T16:32:55.305 DEBUG:teuthology.orchestra.run.smithi164:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T16:32:55.307 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T16:32:55.346 INFO:teuthology.orchestra.run.smithi164.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T16:32:55.347 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-10T16:32:55.697 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T16:32:55.698 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi164.front.sepia.ceph.com 2024-09-10T16:32:55.698 DEBUG:teuthology.orchestra.run.smithi164:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T16:32:55.719 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi165.front.sepia.ceph.com 2024-09-10T16:32:55.719 DEBUG:teuthology.orchestra.run.smithi165:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T16:32:55.763 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T16:32:55.763 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T16:32:55.766 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T16:32:55.830 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T16:32:55.830 DEBUG:teuthology.orchestra.run.smithi164:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T16:32:55.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T16:32:55.975 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T16:32:55.984 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T16:32:55.985 DEBUG:teuthology.orchestra.run.smithi164:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T16:32:56.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T16:32:56.035 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T16:32:56.046 DEBUG:teuthology.orchestra.run.smithi164:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:32:56.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:32:56.073 INFO:teuthology.orchestra.run.smithi164.stdout:kernel.core_pattern = core 2024-09-10T16:32:56.098 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = core 2024-09-10T16:32:56.122 DEBUG:teuthology.orchestra.run.smithi164:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:32:56.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:32:56.150 DEBUG:teuthology.orchestra.run.smithi165:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T16:32:56.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T16:32:56.179 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T16:32:56.190 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T16:32:56.192 DEBUG:teuthology.misc:Transferring archived files from smithi164:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi164 2024-09-10T16:32:56.192 DEBUG:teuthology.orchestra.run.smithi164:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T16:32:56.226 DEBUG:teuthology.misc:Transferring archived files from smithi165:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899276/remote/smithi165 2024-09-10T16:32:56.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T16:32:56.261 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T16:32:56.262 DEBUG:teuthology.orchestra.run.smithi164:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T16:32:56.264 DEBUG:teuthology.orchestra.run.smithi165:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T16:32:56.311 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T16:32:56.322 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T16:32:56.322 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T16:32:56.360 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T16:32:56.360 DEBUG:teuthology.orchestra.run.smithi164:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T16:32:56.364 DEBUG:teuthology.orchestra.run.smithi165:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T16:32:56.376 INFO:teuthology.orchestra.run.smithi164.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 16:32 /home/ubuntu/cephtest 2024-09-10T16:32:56.379 INFO:teuthology.orchestra.run.smithi165.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 16:32 /home/ubuntu/cephtest 2024-09-10T16:32:56.380 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T16:32:56.397 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T16:32:56.463 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{centos_latest} thrashers/one workloads/snaps-few-objects} duration: 2008.2770335674286 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-10T16:32:56.463 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T16:32:56.537 INFO:teuthology.run:pass